[Bug 1599459] [NEW] cannot bootstrap from repositories without InRelease files when --debian-installer set

[David McBride]

Public bug reported:

Version: 1.0.78+nmu1ubuntu1

I am using debootstrap in my own minimal system bootstrapper, and am
making use of the progress information reported to FH 3 as enabled by
the --debian-installer command-line flag.

Unfortunately, using this flag causes the behaviour of debootstrap to
change, specifically in its handling of InRelease / Release files.

During normal operation, debootstrap will first attempt to fetch an
InRelease file from the repository; if this is unavailable, i.e. the
fetch fails with 404, then debootstrap will normally fall back to
fetching a Release file instead.  Indeed, this is the behaviour of
debootstrap if --debian-installer is not passed on the command-line.

debootstrap uses the wgetprogress() function to fetch these URLs.  It
reads:

wgetprogress () {
        [ ! "$VERBOSE" ] && QSWITCH="-q"
        local ret=0
        if [ "$USE_DEBIANINSTALLER_INTERACTION" ] && [ "$PROGRESS_NEXT" ]; then
                wget "$@" 2>&1 >/dev/null | $PKGDETAILS "WGET%" $PROGRESS_NOW $PROGRESS_NEXT $PROGRESS_END >&3
                ret=$?
        else
                wget $QSWITCH "$@"
                ret=$?
        fi
        return $ret
}

When the --debian-installer command-line flag is set, the first path of
the if branch will be taken - running the output of wget through a pipe,
so that the $PKGDETAILS command can be used to parse progress
information provided by wget and report it to FH 3 in a format usable by
debian-installer.

However, if the URL passed to wget returns 404, then while the wget command
will fail, the $PKGDETAILS command, and thus the pipeline as a whole, does not, and the wgetprocess () function thus erroneously returns success.

As a consequence, debootstrap does not fall back to fetching and using a
Release file as it should in this case, and the bootstrapping attempt as a
whole fails.

Because this is a POSIX shell-script, I don't believe there is a
straightforward mechanism to fetch the exit status of the wget command when it is part of a pipeline.  (See: http://cfajohnson.com/shell/cus-faq-2.html#Q11). It might be possible to enhance the code invoked by $PKGDETAILS to return a fatal exit status if it does not definitely see a successful file retrieval?

A more direct work-around is to modify this function to unconditionally
use the second code path that does not attempt to invoke wget as part of
a pipeline. However, this does mean that you lose out on intra-file
download progress reporting.

** Affects: debootstrap (Ubuntu)
     Importance: Undecided
         Status: New

** Description changed:

  Version: 1.0.78+nmu1ubuntu1
  
- I am using debootstrap in my own minimal system bootstrapper, and am making use
- of the progress information reported to FH 3 as enabled by the
- --debian-installer command-line flag.
+ I am using debootstrap in my own minimal system bootstrapper, and am
+ making use of the progress information reported to FH 3 as enabled by
+ the --debian-installer command-line flag.
  
- Unfortunately, using this flag causes the behaviour of debootstrap to change,
- specifically in its handling of InRelease / Release files.
+ Unfortunately, using this flag causes the behaviour of debootstrap to
+ change, specifically in its handling of InRelease / Release files.
  
- During normal operation, debootstrap will first attempt to fetch an InRelease
- file from the repository; if this is unavailable, i.e. the fetch fails with
- 404, then debootstrap will normally fall back to fetching a Release file
- instead.  Indeed, this is the behaviour of debootstrap if --debian-installer is
- not passed on the command-line.
+ During normal operation, debootstrap will first attempt to fetch an
+ InRelease file from the repository; if this is unavailable, i.e. the
+ fetch fails with 404, then debootstrap will normally fall back to
+ fetching a Release file instead.  Indeed, this is the behaviour of
+ debootstrap if --debian-installer is not passed on the command-line.
  
  debootstrap uses the wgetprogress() function to fetch these URLs.  It
  reads:
  
  wgetprogress () {
-         [ ! "$VERBOSE" ] && QSWITCH="-q"
-         local ret=0
-         if [ "$USE_DEBIANINSTALLER_INTERACTION" ] && [ "$PROGRESS_NEXT" ]; then
-                 wget "$@" 2>&1 >/dev/null | $PKGDETAILS "WGET%" $PROGRESS_NOW $PROGRESS_NEXT $PROGRESS_END >&3
-                 ret=$?
-         else
-                 wget $QSWITCH "$@"
-                 ret=$?
-         fi
-         return $ret
+         [ ! "$VERBOSE" ] && QSWITCH="-q"
+         local ret=0
+         if [ "$USE_DEBIANINSTALLER_INTERACTION" ] && [ "$PROGRESS_NEXT" ]; then
+                 wget "$@" 2>&1 >/dev/null | $PKGDETAILS "WGET%" $PROGRESS_NOW $PROGRESS_NEXT $PROGRESS_END >&3
+                 ret=$?
+         else
+                 wget $QSWITCH "$@"
+                 ret=$?
+         fi
+         return $ret
  }
  
- When the --debian-installer command-line flag is set, the first path of the if
- branch will be taken - running the output of wget through a pipe, so that the
- $PKGDETAILS command can be used to parse progress information provided by wget
- and report it to FH 3 in a format usable by debian-installer.
+ When the --debian-installer command-line flag is set, the first path of
+ the if branch will be taken - running the output of wget through a pipe,
+ so that the $PKGDETAILS command can be used to parse progress
+ information provided by wget and report it to FH 3 in a format usable by
+ debian-installer.
  
  However, if the URL passed to wget returns 404, then while the wget command
- will fail, the $PKGDETAILS command, and thus the pipeline as a whole, does not,
- and the wgetprocess () function thus erroneously returns success.
+ will fail, the $PKGDETAILS command, and thus the pipeline as a whole, does not, and the wgetprocess () function thus erroneously returns success.
  
  As a consequence, debootstrap does not fall back to fetching and using a
  Release file as it should in this case, and the bootstrapping attempt as a
  whole fails.
  
  Because this is a POSIX shell-script, I don't believe there is a
- straightforward mechanism to fetch the exit status of the wget command when it
- is part of a pipeline.  (See: http://cfajohnson.com/shell/cus-faq-2.html#Q11).
- It might be possible to enhance the code invoked by $PKGDETAILS to return a
- fatal exit status if it does not definitely see a successful file retrieval?
+ straightforward mechanism to fetch the exit status of the wget command when it is part of a pipeline.  (See: http://cfajohnson.com/shell/cus-faq-2.html#Q11). It might be possible to enhance the code invoked by $PKGDETAILS to return a fatal exit status if it does not definitely see a successful file retrieval?
  
- A more direct work-around is to modify this function to unconditionally use the
- second code path that does not attempt to invoke wget as part of a pipeline.
- However, this does mean that you lose out on intra-file download progress
- reporting.
+ A more direct work-around is to modify this function to unconditionally
+ use the second code path that does not attempt to invoke wget as part of
+ a pipeline. However, this does mean that you lose out on intra-file
+ download progress reporting.

-- 
You received this bug notification because you are a member of Ubuntu
Foundations Bugs, which is subscribed to debootstrap in Ubuntu.
https://bugs.launchpad.net/bugs/1599459

Title:
  cannot bootstrap from repositories without InRelease files when
  --debian-installer set

Status in debootstrap package in Ubuntu:
  New

Bug description:
  Version: 1.0.78+nmu1ubuntu1

  I am using debootstrap in my own minimal system bootstrapper, and am
  making use of the progress information reported to FH 3 as enabled by
  the --debian-installer command-line flag.

  Unfortunately, using this flag causes the behaviour of debootstrap to
  change, specifically in its handling of InRelease / Release files.

  During normal operation, debootstrap will first attempt to fetch an
  InRelease file from the repository; if this is unavailable, i.e. the
  fetch fails with 404, then debootstrap will normally fall back to
  fetching a Release file instead.  Indeed, this is the behaviour of
  debootstrap if --debian-installer is not passed on the command-line.

  debootstrap uses the wgetprogress() function to fetch these URLs.  It
  reads:

  wgetprogress () {
          [ ! "$VERBOSE" ] && QSWITCH="-q"
          local ret=0
          if [ "$USE_DEBIANINSTALLER_INTERACTION" ] && [ "$PROGRESS_NEXT" ]; then
                  wget "$@" 2>&1 >/dev/null | $PKGDETAILS "WGET%" $PROGRESS_NOW $PROGRESS_NEXT $PROGRESS_END >&3
                  ret=$?
          else
                  wget $QSWITCH "$@"
                  ret=$?
          fi
          return $ret
  }

  When the --debian-installer command-line flag is set, the first path
  of the if branch will be taken - running the output of wget through a
  pipe, so that the $PKGDETAILS command can be used to parse progress
  information provided by wget and report it to FH 3 in a format usable
  by debian-installer.

  However, if the URL passed to wget returns 404, then while the wget command
  will fail, the $PKGDETAILS command, and thus the pipeline as a whole, does not, and the wgetprocess () function thus erroneously returns success.

  As a consequence, debootstrap does not fall back to fetching and using a
  Release file as it should in this case, and the bootstrapping attempt as a
  whole fails.

  Because this is a POSIX shell-script, I don't believe there is a
  straightforward mechanism to fetch the exit status of the wget command when it is part of a pipeline.  (See: http://cfajohnson.com/shell/cus-faq-2.html#Q11). It might be possible to enhance the code invoked by $PKGDETAILS to return a fatal exit status if it does not definitely see a successful file retrieval?

  A more direct work-around is to modify this function to
  unconditionally use the second code path that does not attempt to
  invoke wget as part of a pipeline. However, this does mean that you
  lose out on intra-file download progress reporting.

To manage notifications about this bug go to:
https://bugs.launchpad.net/ubuntu/+source/debootstrap/+bug/1599459/+subscriptions