[Bug 1197005] Re: Add "expect stderr" restriction

[Launchpad Bug Tracker]

This bug was fixed in the package autopkgtest - 2.3

---------------
autopkgtest (2.3) unstable; urgency=low


  * runner/adt-run.1: Fix bad "affect modify" wording. (LP: #1199500)
  * Drop hosts/chinstrap. We haven't actually used this in many years, it has
    been replaced with https://launchpad.net/auto-package-testing, Jenkins,
    and blocking migration to release until tests pass. (LP: #381210)
  * Promote apt-utils from Recommends to Depends:, to avoid test failures due
    to not finding apt-ftparchive. (LP: #1065874)
  * Promote pbuilder from Recommends: to Depends:, as we always call
    pbuilder-satisfydepends-classic. (LP: #1065874)
  * runner/adt-run: Stop explicitly checking for $HOME (with broken error
    message) when giving --gnupg-home, use os.path.expanduser() instead.
    (LP: #1065873)
  * runner/adt-run: Fix timeout --help output.
  * runner/adt-run: Fix bomb() to clean up the testbed before removing it.
    Afterwards (in cleanup()) we don't have a testbed object any more to run
    commands in. (LP: #1157292)
  * runner/adt-run: Fix UnicodeDecodeError crash if stderr contains non-ASCII
    characters. (LP: #1100283)
  * Add "allow-stderr" restriction for tests where re-routing or disabling
    stderr is impractical. (LP: #1197005)
  * runner-adt-run: Support architecture specifiers in Depends: field.
    (LP: #1165128)
  * runner/adt-run: Remove/fix some unused variables, spotted by pyflakes.

 -- Martin Pitt <mpitt at debian.org>  Fri, 19 Jul 2013 11:58:15 +0200

** Changed in: autopkgtest (Ubuntu)
       Status: Fix Committed => Fix Released

-- 
You received this bug notification because you are a member of Ubuntu
Foundations Bugs, which is subscribed to autopkgtest in Ubuntu.
https://bugs.launchpad.net/bugs/1197005

Title:
  Add "expect stderr" restriction

Status in “autopkgtest” package in Ubuntu:
  Fix Released

Bug description:
  The specification currently says:

      ...if a test exits nonzero, or prints to stderr, it is considered
  to have failed.

  This is backwards. Plenty of programs write status to stderr and then
  exit with a zero status to indicate success. Examples: wget, curl. I'm
  sure there are many others. Unix convention is to use the exit status
  to indicate success/failure, so it would be far more consistent to
  make this behaviour the default.

  This is a problem because now I have to catch every instance of this
  happening and redirect stderr to /dev/null. Worse, this violates the
  principle that when a test fails it should tell me why, since now all
  of stderr which might have told me has been redirected away.

  A workaround might be to capture stderr to a temporary file and then
  only print it if the exit status was non-zero, but this is painful to
  do in every test and there is no good solution for logging
  stderr/stdout ordering (eg. annotate-output's manpage warns about
  this).

  Before the specification is set in stone, IMHO the sense of this
  should be reversed. Tests should exit with a zero status on success.
  If they actually failed because some program failed but returned a
  zero exit status, then that's a bug that should be fixed. It might be
  sensible to define a "restriction" that changes this to the original
  behaviour of writes to stderr being treated as errors, but the default
  should be to use the exit status only.

  If you don't agree, then at the very least please define and implement
  a "stderr-is-not-a-failure" restriction, so capturing stderr and
  handling the exit status does not have to be done in every test.

To manage notifications about this bug go to:
https://bugs.launchpad.net/ubuntu/+source/autopkgtest/+bug/1197005/+subscriptions