[Bug 1734447] [NEW] BN_cmp regards negative and positive zero as different numbers

[Guido]

Public bug reported:

"BN_cmp() returns -1 if a < b, 0 if a == b and 1 if a > b." (
https://wiki.openssl.org/index.php/Manual:BN_cmp(3) )

The libcrypto.a shipped with Ubuntu makes a distinction between a
negative 0 ("-0") and a positive zero ("0"). This means that BN_cmp(-0,
0) returns -1 and BN_cmp(0, -0) returns 1. The latest versions of
OpenSSL (1.0.2m and 1.1.0g) both return 0 for both comparisons, which is
what you would expect.

lsb_release -rd:

Description:	Ubuntu 16.04.3 LTS
Release:	16.04

Tested on both the 32 bit and 64 bit versions of this Ubuntu release.

Attached is a proof of concept.

The anomaly may also be caused by BN_dec2bn rather than BN_cmp.

The bug is unlikely to have direct security consequences, but may cause
unexpected behaviour in other applications that depend on this
widespread library.

Found with https://github.com/guidovranken/bignum-fuzzer

** Affects: openssl (Ubuntu)
     Importance: Undecided
         Status: New

** Attachment added: "proof of concept of the bug"
   https://bugs.launchpad.net/bugs/1734447/+attachment/5014408/+files/osslcmp.c

-- 
You received this bug notification because you are a member of Ubuntu
Foundations Bugs, which is subscribed to openssl in Ubuntu.
https://bugs.launchpad.net/bugs/1734447

Title:
  BN_cmp regards negative and positive zero as different numbers

Status in openssl package in Ubuntu:
  New

Bug description:
  "BN_cmp() returns -1 if a < b, 0 if a == b and 1 if a > b." (
  https://wiki.openssl.org/index.php/Manual:BN_cmp(3) )

  The libcrypto.a shipped with Ubuntu makes a distinction between a
  negative 0 ("-0") and a positive zero ("0"). This means that
  BN_cmp(-0, 0) returns -1 and BN_cmp(0, -0) returns 1. The latest
  versions of OpenSSL (1.0.2m and 1.1.0g) both return 0 for both
  comparisons, which is what you would expect.

  lsb_release -rd:

  Description:	Ubuntu 16.04.3 LTS
  Release:	16.04

  Tested on both the 32 bit and 64 bit versions of this Ubuntu release.

  Attached is a proof of concept.

  The anomaly may also be caused by BN_dec2bn rather than BN_cmp.

  The bug is unlikely to have direct security consequences, but may
  cause unexpected behaviour in other applications that depend on this
  widespread library.

  Found with https://github.com/guidovranken/bignum-fuzzer

To manage notifications about this bug go to:
https://bugs.launchpad.net/ubuntu/+source/openssl/+bug/1734447/+subscriptions