CD optimization

[John]

Colin Watson wrote:

>On Sat, Sep 18, 2004 at 11:16:29AM +0000, John wrote:
>  
>
>>Matt Zimmerman wrote:
>>    
>>
>>>On Fri, Sep 17, 2004 at 03:24:22PM +0000, John wrote:
>>>      
>>>
>>>>I was thinking about install speed a little while ago. It's long seemed 
>>>>to me that using debootstrap at install time is flawed: surely just 
>>>>unrolling a tarball is faster?
>>>>        
>>>>
>>>This is how Debian used to do things before the Woody release, and it was a
>>>maintenance headache.  It certainly was faster, though.
>>>      
>>>
>>I can see how it would be a customisation headache, but deboostrap as 
>>implemented is that.
>>
>>What makes the tarball so much more difficullt than the non-customisable 
>>package list?
>>    
>>
>
>It's completely trivial to customize. You can drop /.disk/base_include
>and /.disk/base_exclude files onto the CD.
>  
>

I don't see that in any debootstrap package I've ever installed, 
including the Ubunto version.


>  
>
>>Apart from speed of modification?
>>    
>>
>
>Installer developers often drop in a single new .deb in place of the old
>one and build a new image based on it; I lost count of how many times I
>did this for people at the Canonical conference last month. This process
>becomes rather more painful when you have more levels of archives to dig
>your way through, and the images are typically much less rsyncable when
>you change an entire gzipped tarball rather than a single file in an
>ISO9660 image.
>
>Also, the tarball approach means that somebody has to actually *build*
>the tarball, usually using debootstrap. Currently this essentially
>requires root privileges (I'd thought fakechroot would help, but having
>looked at it recently it's still too much of a hack), and therefore
>requires a special build daemon job unlike any other, plus a
>several-hundred-megabyte upload to the archive every day. If that
>breaks, somebody with privileged access to the build daemons has to go
>and spend time fixing it, and in the meantime critical bug fixes to the
>base system (e.g. security fixes) aren't being applied. This approach
>would stand a very good chance of delaying releases at exactly the wrong
>times.
>
>For netboot installs, you need debootstrap anyway. You don't want to
>have to download a single monolithic tarball and curse when wget has to
>be restarted 90% of the way through; you just want to have the installer
>download each .deb independently. Thus, base-installer needs to support
>the debootstrap approach anyway.
>  
>

wget does restart (thought the busybox version might not), and even when 
I installed Fedora Core 2.90 through my modem and getting cut off I had 
no apparent problems with downloads restarting.

>debootstrap really is the Right Answer to all sorts of questions, and
>it's therefore an essential component of the installer. The solution to
>problems encountered with it is to improve debootstrap and the code
>around it, not to throw it away.
>  
>

remember any time that gets saved by users gets multiplied out to 
enormous numbers.

You might be releasing new images frantically now, but that won't always 
be the case.

A tool that exposes the content of a CD image and updates those contents 
& contructs a new CD might work well.

mount -o loop ...
tar xjpf ... -C .
dpkg -x ...
tar cjf ... -C ... .
mkisofs ..

It's what I had in mind when I asked a few months ago when asking about 
hacking on Sarge install images. I noticed someone else asked a similar 
question later.

This won't create a bit-for-bit image, of course, but it will be 
functionally equivalent.