[Bug 1594748] Re: CRYPTO_set_mem_functions() is broken

[Dave Chiluk]

@Joy

It looks like the upstream bug has been rejected.  Do you know what the
resolution for this issue was?  Can you work with upstream to figure out
what's going on?

Thanks,

-- 
You received this bug notification because you are a member of Ubuntu
Foundations Bugs, which is subscribed to openssl in Ubuntu.
https://bugs.launchpad.net/bugs/1594748

Title:
  CRYPTO_set_mem_functions() is broken

Status in OpenSSL:
  Unknown
Status in openssl package in Ubuntu:
  Confirmed

Bug description:
  Description:	Ubuntu 16.04 LTS
  Release:	16.04

  openssl:
    Installed: 1.0.2g-1ubuntu4.1
    Candidate: 1.0.2g-1ubuntu4.1
    Version table:
   *** 1.0.2g-1ubuntu4.1 500
          500 http://fi.archive.ubuntu.com/ubuntu xenial-updates/main amd64 Packages
          500 http://security.ubuntu.com/ubuntu xenial-security/main amd64 Packages
          100 /var/lib/dpkg/status
       1.0.2g-1ubuntu4 500
          500 http://fi.archive.ubuntu.com/ubuntu xenial/main amd64 Packages

  CRYPTO_set_mem_functions() always returns 0 because library
  initialization already calls CRYPTO_malloc() and disables it:

  #0  CRYPTO_malloc (num=num at entry=1168, file=file at entry=0x7ffff70ae02c "fips_drbg_lib.c",
      line=line at entry=106) at mem.c:329
  #1  0x00007ffff70596df in FIPS_drbg_new (type=type at entry=0, flags=flags at entry=0)
      at fips_drbg_lib.c:106
  #2  0x00007ffff705aeb9 in FIPS_drbg_health_check (
      dctx=dctx at entry=0x7ffff731c960 <ossl_dctx>) at fips_drbg_selftest.c:760
  #3  0x00007ffff70595f0 in FIPS_drbg_init (dctx=dctx at entry=0x7ffff731c960 <ossl_dctx>,
      type=<optimized out>, flags=<optimized out>) at fips_drbg_lib.c:94
  #4  0x00007ffff6fe38f3 in RAND_init_fips () at rand_lib.c:287
  #5  0x00007ffff6f26f7a in OPENSSL_init_library () at o_init.c:119
  #6  0x00007ffff7de74ea in call_init (l=<optimized out>, argc=argc at entry=1,
      argv=argv at entry=0x7fffffffe5e8, env=env at entry=0x7fffffffe5f8) at dl-init.c:72
  #7  0x00007ffff7de75fb in call_init (env=0x7fffffffe5f8, argv=0x7fffffffe5e8, argc=1,
      l=<optimized out>) at dl-init.c:30
  #8  _dl_init (main_map=main_map at entry=0x640380, argc=1, argv=0x7fffffffe5e8,
      env=0x7fffffffe5f8) at dl-init.c:120

  This doesn't happen in upstream OpenSSL or in Debian's OpenSSL.
  Looking at the patches, this is caused by FIPS_drbg_init() in
  openssl-1.0.2g-fips.patch:

  +    if (!(dctx->xflags & DRBG_FLAG_TEST)) {
  +        if (!FIPS_drbg_health_check(dctx)) {
  +            FIPSerr(FIPS_F_FIPS_DRBG_INIT, FIPS_R_SELFTEST_FAILURE);
  +            return 0;
  +        }
  +    }

  I don't want any FIPS mode enabled though, so does it really even need
  to call RAND_init_fips() then?

To manage notifications about this bug go to:
https://bugs.launchpad.net/openssl/+bug/1594748/+subscriptions