[Bug 1197005] [NEW] Printing to stderr should not be considered a test failure

[Robie Basak]

Public bug reported:

The specification currently says:

    ...if a test exits nonzero, or prints to stderr, it is considered to
have failed.

This is backwards. Plenty of programs write status to stderr and then
exit with a zero status to indicate success. Examples: wget, curl. I'm
sure there are many others. Unix convention is to use the exit status to
indicate success/failure, so it would be far more consistent to make
this behaviour the default.

This is a problem because now I have to catch every instance of this
happening and redirect stderr to /dev/null. Worse, this violates the
principle that when a test fails it should tell me why, since now all of
stderr which might have told me has been redirected away.

A workaround might be to capture stderr to a temporary file and then
only print it if the exit status was non-zero, but this is painful to do
in every test and there is no good solution for logging stderr/stdout
ordering (eg. annotate-output's manpage warns about this).

Before the specification is set in stone, IMHO the sense of this should
be reversed. Tests should exit with a zero status on success. If they
actually failed because some program failed but returned a zero exit
status, then that's a bug that should be fixed. It might be sensible to
define a "restriction" that changes this to the original behaviour of
writes to stderr being treated as errors, but the default should be to
use the exit status only.

If you don't agree, then at the very least please define and implement a
"stderr-is-not-a-failure" restriction, so capturing stderr and handling
the exit status does not have to be done in every test.

** Affects: autopkgtest (Ubuntu)
     Importance: Undecided
         Status: New

-- 
You received this bug notification because you are a member of Ubuntu
Foundations Bugs, which is subscribed to autopkgtest in Ubuntu.
https://bugs.launchpad.net/bugs/1197005

Title:
  Printing to stderr should not be considered a test failure

Status in “autopkgtest” package in Ubuntu:
  New

Bug description:
  The specification currently says:

      ...if a test exits nonzero, or prints to stderr, it is considered
  to have failed.

  This is backwards. Plenty of programs write status to stderr and then
  exit with a zero status to indicate success. Examples: wget, curl. I'm
  sure there are many others. Unix convention is to use the exit status
  to indicate success/failure, so it would be far more consistent to
  make this behaviour the default.

  This is a problem because now I have to catch every instance of this
  happening and redirect stderr to /dev/null. Worse, this violates the
  principle that when a test fails it should tell me why, since now all
  of stderr which might have told me has been redirected away.

  A workaround might be to capture stderr to a temporary file and then
  only print it if the exit status was non-zero, but this is painful to
  do in every test and there is no good solution for logging
  stderr/stdout ordering (eg. annotate-output's manpage warns about
  this).

  Before the specification is set in stone, IMHO the sense of this
  should be reversed. Tests should exit with a zero status on success.
  If they actually failed because some program failed but returned a
  zero exit status, then that's a bug that should be fixed. It might be
  sensible to define a "restriction" that changes this to the original
  behaviour of writes to stderr being treated as errors, but the default
  should be to use the exit status only.

  If you don't agree, then at the very least please define and implement
  a "stderr-is-not-a-failure" restriction, so capturing stderr and
  handling the exit status does not have to be done in every test.

To manage notifications about this bug go to:
https://bugs.launchpad.net/ubuntu/+source/autopkgtest/+bug/1197005/+subscriptions