[Merge] lp:~rbalint/livecd-rootfs/minimize-unminimize into lp:~vorlon/livecd-rootfs/minimizing

[Balint Reczey]

Diff comments:

> === modified file 'live-build/auto/build'
> --- live-build/auto/build	2017-05-09 03:15:14 +0000
> +++ live-build/auto/build	2017-05-09 10:23:04 +0000
> @@ -72,6 +72,62 @@
>  
>  		# Remove docs installed by bootstrap
>  		Chroot chroot dpkg-query -f '${binary:Package}\n' -W | Chroot chroot xargs apt-get install --reinstall
> +
> +                # Add unminimizer script which restores default image behavior
> +                mkdir -p chroot/usr/local/bin

I think everything under /usr/*bin is dpkg-managed in practice and here we should not break this expectation.
Packaging unminimize would surely help with managing it and SRU-ing once does not seem to be a huge problem, but how can we keep it in sync with the minimization steps taken in livecd-rootfs?

As I understand it "minimization" is a set of steps taken to make Ubuntu images smaller by sometimes breaking a set of expectations like having working man pages. In that sense "unminimization" should revert those steps and the easiest way of keeping the reversion steps is generating them in livecd-rootfs, too, or keeping both "minimize" and "unminimize" in the same package.

In case we don't go with packaged scripts I agree that /usr/local/ is for sysadmins and if we would like to respect FHS in that sense we could put the script in /opt/bin which is a bit less user-friendly since /opt/bin is not on the path: LP: #1075860 .

> +                cat > chroot/usr/local/bin/unminimize <<'EOF'
> +#!/bin/sh
> +
> +set -e
> +
> +TMP=$(mktemp -d)
> +cleanup() {
> +  if [ -f $TMP/excludes ]; then
> +    echo "Re-enabling installation of all documentation in dpkg failed." 1>&2

No problem, your version sound better and is easier to understand indeed.

> +    mv $TMP/excludes /etc/dpkg/dpkg.cfg.d/excludes
> +  fi
> +  rm -rf "$TMP"
> +}
> +trap cleanup EXIT
> +
> +if [ -f /usr/sbin/update-initramfs.divert-minimize ]; then

OK, dropping it.

> +    echo "Re-enabling intiramfs..."
> +    rm -f /usr/sbin/update-initramfs
> +    dpkg-divert --remove --rename /usr/sbin/update-initramfs
> +    update-initramfs -u -k all
> +else
> +    echo "Initramfs is already enabled, skipping re-enabling it."
> +fi
> +
> +if [ -f /etc/dpkg/dpkg.cfg.d/excludes ]; then
> +    echo "Re-enabling installation of all documentation in dpkg..."
> +    mv /etc/dpkg/dpkg.cfg.d/excludes $TMP/excludes
> +    echo "Updating package list and upgrading packages..."
> +    apt-get update
> +    apt-get upgrade
> +    echo "Installing missing documentation from packages..."
> +    (cd $TMP

When I tried the 'apt-get install --reinstall $pkglist' style in the chroot dir without /proc mounted it failed for grub-pc and grub-gfxpayload-lists. It also ran triggers, restarted services, etc, thus I went with this more complicated method updating only the files.
I have just checked and in lxc the "dpkg -S '/usr/share/man' /usr/share/doc |sed 's|, |\n|g;s|: [^:]*$||' | sort | uniq | DEBIAN_FRONTEND=noninteractive xargs apt-get install -y --reinstall" command succeeded thus we can choose this more intrusive, but package-manager-only solution.

> +        echo "Downloading packages with files in /usr/share/man/ ..."
> +        dpkg -S /usr/share/man/ |sed 's|, |\n|g;s|: [^:]*$||' | xargs apt-get download
> +        echo "Downloading packages with missing files in /usr/share/doc/ .."
> +        dpkg --verify | awk '/\?\?5\?\?\?\?\?\?   \/usr\/share\/doc/ {print $2}' | sed 's|/[^/]*$||' | sort |uniq \
> +             | xargs dpkg -S | sed 's|, |\n|g;s|: [^:]*$||' | uniq | xargs apt-get download

> This part is cleverly done, +1, and now I've learned about dpkg --verify.  I just think the end of the pipeline should be 'DEBIAN_FRONTEND=noninteractive apt-get install --reinstall -y'.

Agreed, using noninteractive is better in case of using reinstall.

> From the documentation, I note that the other columns may at some later date have other output when dpkg starts to support them.  So to future-proof this, I recommend doing 'awk '/..5......   \/usr\/share\/doc/' instead of hard-coding the ? for each column.

Agreed.

> Also, I guess you should be able to handle both /usr/share/man and /usr/share/doc with a single dpkg --verify command and avoid calling dpkg -S entirely.


I tried the single dpkg --verified command earlier and it generated very long parameter list for "xargs dpkg -S" and I don't like risking going over ARG_MAX. Since many packages have man pages the second download handles a much smaller amount of packages.

> +        mkdir target
> +        echo "Extracting packages to temporary location..."
> +        for i in *.deb; do dpkg-deb -x $i target/; done
> +        echo "Restoring missing documentation on the system ..."
> +        cp -rpn target/usr/share/man/* /usr/share/man/
> +        cp -rpn target/usr/share/doc/* /usr/share/doc/
> +        if dpkg --verify > /dev/null; then
> +           echo "Documentation is restored successfully."
> +           rm $TMP/excludes
> +        fi
> +    )
> +else
> +    echo "Dpkg already installs documentation, skipping re-enabling it."
> +fi
> +EOF

OK, will do.

> +                chmod +x chroot/usr/local/bin/unminimize
>          fi
>  
>  	Chroot chroot "dpkg-divert --quiet --add \
> 
> === modified file 'live-build/ubuntu-cpc/hooks/032-disk-image.binary'
> --- live-build/ubuntu-cpc/hooks/032-disk-image.binary	2017-05-09 03:15:14 +0000
> +++ live-build/ubuntu-cpc/hooks/032-disk-image.binary	2017-05-09 10:23:04 +0000
> @@ -131,6 +131,8 @@
>  		cat > mountpoint/usr/sbin/update-initramfs <<'EOF'
>  #! /bin/sh
>  echo "initramfs disabled on this system.  To reenable, run:" >&2
> +echo "  sudo unminimize" >&2
> +echo "or" >&2

OK.

>  echo "  sudo rm -f /usr/sbin/update-initramfs" >&2
>  echo "  sudo dpkg-divert --remove --rename /usr/sbin/update-initramfs" >&2
>  exit 0


-- 
https://code.launchpad.net/~rbalint/livecd-rootfs/minimize-unminimize/+merge/323157
Your team Ubuntu Core Development Team is requested to review the proposed merge of lp:~rbalint/livecd-rootfs/minimize-unminimize into lp:~vorlon/livecd-rootfs/minimizing.