[Bug 1765933] Re: Allow building livefses against a view of the archive at a fixed point in time

[Dimitri John Ledkov]

@colin watson

After fighting with magicproxy & iptables issues. again. I am interested
in getting magic proxy doing more-or-less things that might one day make
things "nice".

I.e.
make magic proxy, talk to launchpad proxy to access authenticated archives without explicit username/password.

drop iptables nat rules, and instead try to simply export proxy for apt
to use.

try to drop proxy, and instead make magic-proxy write out in-release-
path stanzas for apt to use based on the passed in timestamp, and make
livebuild / debootstrap use all that.

eventually "just" accept the mapping of archive/suites = hash as
collected via launchpad api before dispatching the build.

Hopefully above things will make certain things easier as we go along.

Alternatively, maybe we could merge magic-proxy into the proxy that
launchpad exports for the builds? I guess probably not, cause all of
that is quite hackish.

-- 
You received this bug notification because you are a member of Ubuntu
Foundations Bugs, which is subscribed to livecd-rootfs in Ubuntu.
https://bugs.launchpad.net/bugs/1765933

Title:
  Allow building livefses against a view of the archive at a fixed point
  in time

Status in Launchpad itself:
  In Progress
Status in livecd-rootfs package in Ubuntu:
  Confirmed

Bug description:
  It would be useful to be able to dispatch a set of livefs builds that
  all reliably build from the same point in time in the archive's
  history, even if they're dispatched at slightly different times.  We
  did the InRelease-by-hash work (https://bugs.debian.org/886745, and
  subsequent Launchpad publisher changes) as a prerequisite for this.
  We're now ready to consider the next stage.  I believe that there are
  three major chunks of work here:

  1) Somebody needs to work out the livecd-rootfs/live-build bits, which
  I think ought to be a Foundations job (I don't expect to do this part
  myself).  It needs to pick up the inrelease-path option from the
  sources.list in the environment where livecd-rootfs is run and
  propagate that through while building the image, but make sure that
  the inrelease-path option *doesn't* end up in the sources.list in the
  final image (just like the way we build from ftpmaster.internal but
  don't actually leave that in the final image, since it wouldn't work).

  2) In parallel with 1), we need to improve Launchpad's internal
  tracking of the history of archive index files.  At the moment, the
  only feasible interface we could offer would be that the entity
  scheduling livefs builds would need to work out the SHA-256 hashes of
  all the relevant InRelease files and tell Launchpad what to use.  This
  is cumbersome and hard to automate.  Instead, it would be much better
  to make it work by timestamp (perhaps even simply having an option to
  pick whatever's current when the builds are scheduled), but to do that
  we need to convert ArchiveFile into a history table, adding
  date_created and date_superseded columns so that we can work out which
  version was live at any given time.

  3) Once both 1) and 2) are done, we can add the relevant options to
  Launchpad's livefs build code to cause the correct sources.list to be
  dispatched.

To manage notifications about this bug go to:
https://bugs.launchpad.net/launchpad/+bug/1765933/+subscriptions