[External] Re: Question - Livepatch/Kprobe Coexistence on Ftrace-enabled Functions (Ubuntu kernel based on Linux stable 5.15.30)

[Song Liu]

On Thu, Oct 16, 2025 at 3:56 AM Petr Mladek <pmladek at suse.com> wrote:
>
> Added Song Liu and Steven into Cc because they are more familiar with
> the ftrace trampolines.
>
> On Wed 2025-10-15 17:11:31, Andrey Grodzovsky wrote:
> > On 10/15/25 07:53, Petr Mladek wrote:
> > > On Tue 2025-10-14 21:37:49, Andrey Grodzovsky wrote:
> > > > Dear Upstream Livepatch  team and Ubuntu Kernel team - I included you both in this since the issue lies on the boundary between Ubuntu kernel and upstream.
> > > >
> > > > According to official kernel documentation  - https://urldefense.com/v3/__https://docs.kernel.org/livepatch/livepatch.html*livepatch__;Iw!!BmdzS3_lV9HdKG8!z3Y4vlE7RcCriT3z4Hg7cVaojZPN-ysQTbjDJVXyO_MoRRlkKsymUTDP4PGvvPaV0TDVYhziOYMm9WnUGu5TeFxUxQ$ , section 7, Limitations -
> > > > 1 - Kretprobes using the ftrace framework conflict with the patched functions.
> > > > 2 - Kprobes in the original function are ignored when the code is redirected to the new implementation.
> > > >
> > > > Yet, when testing on my Ubuntu 5.15.0-1005.7-aws (based on 5.15.30 stable kernel) machine, I have no problem applying Livepatch and then setting krpobes and kretprobes on a patched function using bpftrace (or directly by coding a BPF program with kprobe/kretprobe attachment)and can confirm both execute without issues. Also the opposite works fine, running my krpobe and kretprobe hooks doesn't prevent from livepatch to be applied successfully.
> > > >
> > > > fentry/fexit probes do fail in in both directions - but this is expected according to my understanding as coexistence of livepatching and ftrace based BPF hooks are mutually exclusive until 6.0 based kernels
> > > >
> > > > libbpf: prog 'begin_new_exec_fentry': failed to attach: Device or resource busy
> > > > libbpf: prog 'begin_new_exec_fentry': failed to auto-attach: -16
> > > >
> > > >
> > > > Please help me understand this contradiction about kprobes - is this because the KPROBES are FTRACE based  or any other reason ?
> > > Heh, it seems that we have discussed this 10 years ago and I already
> > > forgot most details.
> > >
> > > Yes, the conflict is detected when KPROBES are using FTRACE
> > > infrastructure. But it happens only when the KPROBE needs to redirect
> > > the function call, namely when it needs to modify IP address which will be used
> > > when all attached ftrace callbacks are proceed.
> > >
> > > It is related to the FTRACE_OPS_FL_IPMODIFY flag.
> >
> >
> > I see, that explains my case as my probes are simple, print only probes that
> > defently don't that the ip pointer.
> >
> > But i still don't understand limitation 2 above doesn't show itself - how my
> > kprobes and especially kretprobes, continue execute even after livepatch
> > applied to the function  i am attached to ? The code execution is redirected
> > to a different function then to which i attached my probes...
>
> The code is rather complicated and it is not obvious to me.
>
> But I expect that kretprobes modify return addresses which are
> stored on the stack. They replace them
> with a helper function (trampoline) which would process
> the kretprobe callback and jump to the original return address.
>
> It works even with livepatching. The only requirement is that
> the function returns using a "ret" instruction which reads
> the return address from the stack.
>
> > Also - can you please confirm that as far as i checked, starting with kernel
> > 6.0 fentry/fexit on x86 should not have any conflicts with livepatch per
> > merge of this RFC -
> > https://lkml.iu.edu/hypermail/linux/kernel/2207.2/00858.html ?
>
> My understanding is that the above mentioned patchset allowed to use
> livepatching and BPF on the same function at the same time. BPF used
> to redirect the function to a BPF trampoline. And the patchset allowed
> to call the BPF trampoline/callback directly from the ftrace callback.

Yes, the patch set above allows BPF trampoline to work on the same
function at the same time with livepatch. After this patchset, BPF
trampoline no longer sets FTRACE_OPS_FL_IPMODIFY.

Thanks,
Song



> Best Regards,
> Petr
>
> > >
> > > More details can be found in the discussion at
> > > https://urldefense.com/v3/__https://lore.kernel.org/all/20141121102516.11844.27829.stgit@localhost.localdomain/T/*re746846b6b16c49a55c89b4c63b7995fe5972111__;Iw!!BmdzS3_lV9HdKG8!z3Y4vlE7RcCriT3z4Hg7cVaojZPN-ysQTbjDJVXyO_MoRRlkKsymUTDP4PGvvPaV0TDVYhziOYMm9WnUGu6pjuIgig$
> > >
> > > I seems that I made some analyze when it worked and it did not work,
> > > see https://urldefense.com/v3/__https://lore.kernel.org/all/20141121102516.11844.27829.stgit@localhost.localdomain/T/*mffd8c8bf4325b473d89876f2805f42f1af7c82d7__;Iw!!BmdzS3_lV9HdKG8!z3Y4vlE7RcCriT3z4Hg7cVaojZPN-ysQTbjDJVXyO_MoRRlkKsymUTDP4PGvvPaV0TDVYhziOYMm9WnUGu5xbeoulA$
> > > But I am not 100% sure that it was correct. Also it was before the
> > > BPF-boom.
> > >
> > > Also you might look at the selftest in the todays Linus' tree:
> > >
> > >    + tools/testing/selftests/livepatch/https://urldefense.com/v3/__http://test-kprobe.sh__;!!BmdzS3_lV9HdKG8!z3Y4vlE7RcCriT3z4Hg7cVaojZPN-ysQTbjDJVXyO_MoRRlkKsymUTDP4PGvvPaV0TDVYhziOYMm9WnUGu5RXF-AnA$
> > >    + tools/testing/selftests/livepatch/test_modules/test_klp_kprobe.c
> > >    + tools/testing/selftests/livepatch/test_modules/test_klp_livepatch.c
> > >
> > > The parallel load fails when the Kprobe is using a post_handler.
> > >
> > > Sigh, we should fix the livepatch documentation. The kretprobes
> > > obviously work. register_kretprobe() even explicitely sets:
> > >
> > >     rp->kp.post_handler = NULL;
> > >
> > > It seems that .post_handler is called after all ftrace handlers.
> > > And it sets IP after the NOPs, see kprobe_ftrace_handler().
> > > I am not sure about the use case.
> > >
> > > Best Regards,
> > > Petr
> >
> >