[Bug 1895106] Re: output non reproducible inhibiting verification of changes

[Christian Ehrhardt ]

_rescue_includes does check if a package is already present and
continues.

But in a run with some debug between flaky and sorted output I found
modemmanager-dev toggling between "Rescued from modemmanager" (unsorted)
and "libmm-glib-dev" (sorted)

I many times see "_add_dependency modemmanager" before we get to the rescue check.
And I'd assume that this makes it not rescuing it and be skipped.

unsorted/germinate.log:11974:  D: rescue-check modemmanager-doc
unsorted/germinate.log:11975:  D: Rescued modemmanager-doc from extra to supported
unsorted/germinate.log:11977:  D: rescue-check gir1.2-modemmanager-1.0
unsorted/germinate.log:11978:  D: Rescued gir1.2-modemmanager-1.0 from extra to supported
unsorted/germinate.log:11980:  D: rescue-check modemmanager-dev
unsorted/germinate.log:11981:  D: Rescued modemmanager-dev from extra to supported
unsorted/germinate.log:15661:  D: rescue-check modemmanager-doc
unsorted/germinate.log:15663:  D: rescue-check gir1.2-modemmanager-1.0
unsorted/germinate.log:15665:  D: rescue-check modemmanager-dev

sorted/germinate.log:11970:  D: rescue-check gir1.2-modemmanager-1.0
sorted/germinate.log:11971:  D: Rescued gir1.2-modemmanager-1.0 from extra to supported
sorted/germinate.log:11980:  D: rescue-check modemmanager-dev
sorted/germinate.log:11982:  D: rescue-check modemmanager-doc
sorted/germinate.log:11983:  D: Rescued modemmanager-doc from extra to supported
sorted/germinate.log:15658:  D: rescue-check gir1.2-modemmanager-1.0
sorted/germinate.log:15664:  D: rescue-check modemmanager-dev
sorted/germinate.log:15666:  D: rescue-check modemmanager-doc

First of all we see that multiple rescue checks are done multiple times,
there would be an optimization to skip duplicates here.

But more interestingly if we see that in one case it already was in
output._all and skipped while it wasn't in the other. Looking further we
see that "libmm-glib-dev" also is a rescued dependency. And in one case
it was processed earlier, in the other later.

Due to that we all the time had flickering output, as the set was unsorted those that came early got the "Rescued from" entry and were added including their dependency tree.
But also they were further filling output._all and due to that reducing the chance for further entries to be marked as "Rescued"

-- 
You received this bug notification because you are a member of Ubuntu
Foundations Bugs, which is subscribed to germinate in Ubuntu.
https://bugs.launchpad.net/bugs/1895106

Title:
  output non reproducible inhibiting verification of changes

Status in germinate package in Ubuntu:
  New

Bug description:
  Hi,
  there are multiple classes of issues that make germinate output hard to compare for "what changed".

  1. ordering issues
  Even the output of the running program changes. I didn't see subprocess spawning or any other asynchronicity. So I'd assume that there are lists generated/gathered that we process as-is.
  Example:
    1120 Resolving supported dependencies ...                                             
    1121 * Chose dict to satisfy dictd                                                    
    1122 * Chose probert-storage to satisfy curtin                                        
    1123   Rescued python-requests-toolbelt-doc from extra to supported                                                                                                                          
    1124   Rescued libmemcached-dbg from extra to supported 
  Rerun on same content:
    1120 Resolving supported dependencies ...                                             
    1121 * Chose dict to satisfy dictd                                                    
    1122 * Chose probert-storage to satisfy curtin                                        
    1123   Rescued default-libmysqlclient-dev from extra to supported                                                                                                                            
    1124   Rescued libmpc-dev from extra to supported                                                                                                                                            
    1125   Rescued python-openstackclient-doc from extra to supported                                                                                                                            
    1126   Rescued libgnome-menu-3-dev from extra to supported 

  Why is that - could we just sort any of the gathered lists before we iterate on them.
  That could in turn make many other things in the output reproducible.

  
  2. If a package is depended on by multiple packages or seeds a random one is reported.

  Example (run with the same seeds multiple times):
  all:
  -binutils-multiarch                          | binutils                             | binutils-multiarch-dev
  +binutils-multiarch                          | binutils                             | binutils-multiarch-dbg

  It is correct that both packages and seeds depend on them but I'd think it could be much better if we'd either:
  - report an ordered full list of dependency sources (could become very long but complete)
  or
  - report the first element out of a sorted list (as short as today, but reproducible)

  We could even get "best of both worlds" if we sort the list of
  dependency anchors, then concat the fist X (an arbitrary limit we set)
  of them and if some are left append ", ..." to reflect that.

  This way we would be reproducible, in many cases even complete, and in
  corner cases don't explode the list size.

  
  There could be more sources of non reproducibility that come to mind for experts in germinate (I'm not) and I'd appreciate if there could be an overhaul to get as close as reasonable to reproducible output.
  Because then if someone modifies seeds (or checks any follow on output based on it) it would stay the same and indicate the actual impact a change has caused.

  
  Well, maybe this was already discussed but I didn't find anything. If it was and considered undoable please point me to a log of the discussion if possible.

To manage notifications about this bug go to:
https://bugs.launchpad.net/ubuntu/+source/germinate/+bug/1895106/+subscriptions