[External] Re: Question - Livepatch/Kprobe Coexistence on Ftrace-enabled Functions (Ubuntu kernel based on Linux stable 5.15.30)

[Andrey Grodzovsky]

On 10/20/25 15:10, Andrey Grodzovsky wrote:
> On 10/20/25 14:53, Song Liu wrote:
>> On Mon, Oct 20, 2025 at 9:45 AM Andrey Grodzovsky
>> <andrey.grodzovsky at crowdstrike.com> wrote:
>>> On 10/20/25 12:03, Song Liu wrote:
>>>> On Mon, Oct 20, 2025 at 7:56 AM Andrey Grodzovsky
>>>> <andrey.grodzovsky at crowdstrike.com> wrote:
>>>> [...]
>>>>>> If you build the kernel from source code, there are some samples in
>>>>>> samples/livepatch that you can use for testing. PS: You need to 
>>>>>> enable
>>>>>>
>>>>>>      CONFIG_SAMPLE_LIVEPATCH=m
>>>>>>
>>>>>> I hope this helps.
>>>>> Thanks Song, working on repro, kernel rebuilt, test module is loading
>>>>> but, bpftrace is refusing to attach now to fentries/fexits 
>>>>> claiming the
>>>>> costum kernel is not supporting it. It did
>>>>> attach in the case of stock AWS kernel i copied the .config from. So
>>>>> just trying to figure out now if some Kcofnig flags are missing or
>>>>> different . Let me know in case you manage to confirm yourself in the
>>>>> meanwhile the fix works for
>>>>> you.
>>>> Yes, it worked in my tests.
>>>>
>>>> [root@(none) /]# kpatch load 
>>>> linux/samples/livepatch/livepatch-sample.ko
>>>> loading patch module: linux/samples/livepatch/livepatch-sample.ko
>>>> [root@(none) /]# bpftrace.real -e 'fexit:cmdline_proc_show
>>>> {printf("fexit\n");}' &
>>>> [1] 388
>>>> [root@(none) /]# Attached 1 probe
>>>> [root@(none) /]# bpftrace.real -e 'fentry:cmdline_proc_show
>>>> {printf("fentry\n");}' &
>>>> [2] 397
>>>> [root@(none) /]# Attached 1 probe
>>>>
>>>> [root@(none) /]# cat /proc/cmdline
>>>> this has been live patched
>>>> fentry
>>>> fexit
>>>>
>>>> Thanks,
>>>> Song
>>>>
>>> Verified the failures I observe when trying to attach with BPF trace 
>>> are
>>> only in presence of patch you provided.
>>> Please see attached dmesg for failures. Initial warning on boot.
>>> Subsequebt warnings and errors at the point i try to run
>>> sudo bpftrace -e "fexit:cmdline_proc_show { printf(\"fexit hit\\n\");
>>> exit(); }"
>>>
>>> sudo: unable to resolve host ip-10-10-115-238: Temporary failure in 
>>> name
>>> resolution
>>> stdin:1:1-25: ERROR: kfunc/kretfunc not available for your kernel 
>>> version.
>>>
>>> ubuntu at ip-10-10-115-238:~/linux-6.8.1$ sudo cat
>>> /sys/kernel/debug/tracing/available_filter_functions | grep
>>> cmdline_proc_show
>>> sudo: unable to resolve host ip-10-10-115-238: Temporary failure in 
>>> name
>>> resolution
>>> cat: /sys/kernel/debug/tracing/available_filter_functions: No such 
>>> device
>>>
>>> After reboot and before trying to attacg with bpftrace,
>>> /sys/kernel/debug/tracing/available_filter_functions is available and
>>> shows all function.
>>>
>>> Using stable kernel from
>>> https://urldefense.com/v3/__https://cdn.kernel.org/pub/linux/kernel/v6.x/linux-6.8.1.tar.gz__;!!BmdzS3_lV9HdKG8!1ZJe4jY49_xIzp4h4i4AbqpkLKoAqrXLFX2wDxhoSUDg2kSeTjy3COy9MngNDRlZhJ1oUKgf1yPqmnTY9-Y50TkA$ 
>>> for build.
>>> FTRACE related KCONFIGs bellow
>> I can see the similar issue with the upstream kernel. I was testing on
>> stable 6.17 before just know because of another issue with upstream
>> kernel, and somehow 6.17 kernel doesn't seem to have the issue.
>>
>> To fix this, I think we should land a fix similar to the earlier diff:
>>
>> diff --git i/kernel/trace/ftrace.c w/kernel/trace/ftrace.c
>> index 42bd2ba68a82..8f320df0ac52 100644
>> --- i/kernel/trace/ftrace.c
>> +++ w/kernel/trace/ftrace.c
>> @@ -6049,6 +6049,9 @@ int register_ftrace_direct(struct ftrace_ops
>> *ops, unsigned long addr)
>>
>>          err = register_ftrace_function_nolock(ops);
>>
>> +       if (err)
>> +               remove_direct_functions_hash(hash, addr);
>> +
>>    out_unlock:
>>          mutex_unlock(&direct_mutex);
>>
>>
>> Steven,
>>
>> Does this change look good to you?
>>
>>
>
> Seems reasonable to me, we are simply cleaning the entry on failure so 
> we don't encounter it late anymore.
> So I will apply this patch ONLY and retest - correct ?
>
> Another question - it seems you found where it broke ? I saw 'Cc: 
> stable at vger.kernel.org # v6.6+' in your prev. patch.'
> If so , can you please point me to the offending patch so I add this 
> to my records of my discovery work of bpf coexistence
> livepatching ?
>
> Thanks,
>
> Andrey


Update - with latest fix work find, both after loading the livepatch 
.ko  no conflicts and hooks work and,
before loading it, pre exsisting hooks keep working after the patch is 
loaded

You can add Acked-and-tested-by: Andrey Grodzovsky 
<andrey.grodzovsky at crowdstrike.com>

Once again, in case you now the exact commit that broke it, please let 
me know.

Thanks,
Andrey

>
>>
>> Thanks,
>> Song
>
>