NACK: [PATCH 0/1][SRU][U/P/O/N/J/F] distribute sources list generation in parallel

[You-Sheng Yang]

To follow https://lists.ubuntu.com/archives/kernel-team/2025-March/158350.html

On Sat, Mar 29, 2025 at 2:39 AM You-Sheng Yang
<vicamo.yang at canonical.com> wrote:
>
> BugLink: https://bugs.launchpad.net/bugs/2104911
>
> [Impact]
>
> In debian/rules.d/2-binary-arch.mk, stamp-build-% targets, we have:
> ```
> # Collect the list of kernel source files used for this build. Need to do this early
> # before modules are stripped. Fail if the resulting file is empty.
> find $(build_dir) \( -name vmlinux -o -name \*.ko \) -exec dwarfdump -i {} \; | \
>         grep -E 'DW_AT_(call|decl)_file' | sed -n 's|.*\s/|/|p' | sort -u > \
>         $(build_dir)/sources.list
> test -s $(build_dir)/sources.list
> ```
>
> The process is single threaded and may takes up to half an hour to complete:
> ```
> $ time ...
>
> real 28m15.167s
> user 54m32.917s
> sys 4m32.531s
> ```
>
> The best case (don't sort, ignore output overlapping from find) takes about 4
> minutes.
>
> [Fix]
>
> Let `find` output only kernel module paths and use `xargs -P $(nproc)` to invoke
> dwarfdump pipelines in parallel. This shorten the time for the whole process to
> 13 minutes.
>
> Then pre-sort output from each of the dwarfdump pipeline, this shorten the time
> to 6 and half minutes.
>
> Finally, use `flock` to guard output control.
>
> [Test Case]
>
> 1. Add `time` at the beginning of the `find ...` line to measure time spent.
>    Also dump the checksum of the generated sources.list file for later
>    comparison.
>
> 2. Rebuild kernel, check the time spent for generating sources list
>
> 3. Compare the checksum of sources.list generated.
>
> [Where problems could occur]
>
> Build recipt changes only.
>
> [Other Info]
>
> This affects only series with dwarfdump usage. Nominate for U/P/O/N/J/F.
>
> You-Sheng Yang (1):
>   UBUNTU: [Packaging] distribute sources list generation in parallel
>
>  debian/rules.d/2-binary-arch.mk | 9 +++++++--
>  1 file changed, 7 insertions(+), 2 deletions(-)
>
> --
> 2.48.1
>
>
> --
> kernel-team mailing list
> kernel-team at lists.ubuntu.com
> https://lists.ubuntu.com/mailman/listinfo/kernel-team



-- 
Regards,
You-Sheng Yang