[Bug 1020210] Re: Race condition using ATOMIC_FASTBINS in _int_free causes crash or heap corruption

[Bug Watch Updater]

Launchpad has imported 1 comments from the remote bug at
http://sourceware.org/bugzilla/show_bug.cgi?id=15073.

If you reply to an imported comment from within Launchpad, your comment
will be sent to the remote bug automatically. Read more about
Launchpad's inter-bugtracker facilities at
https://help.launchpad.net/InterBugTracking.

------------------------------------------------------------------------
On 2013-01-28T16:17:29+00:00 Josh Pieper wrote:

Created attachment 6833
Reproduction recipe

I reported the following bug in ubuntu's libc about 6 months ago where
the ATOMIC_FASTBINS feature can crash or cause heap corruption.

https://bugs.launchpad.net/ubuntu/+source/eglibc/+bug/1020210

I'm pasting the content of that problem here and will attach the
reproduction recipe.  Unfortunately, the reproduction recipe is only
reliable with the exact version of eglibc in ubuntu, given enough time
to build an arbitrary version, I should be able to create one that works
with any version.

----

We have an application which makes heavy allocation and de-allocation
demands from multiple threads. We run this application continuously on
many servers, and once every several CPU months or years, we were
getting a crash in _int_free that did not look like vanilla heap
corruption. I believe I have narrowed it down to a race condition in
_int_free due to the ATOMIC_FASTBINS feature. Basically, in the lockless
FASTBIN _int_free path, a chunk is pulled into a local variable with the
intent to add it to the fastbins list. However, the heap
consolidation/trim code can race with this, and can coalesce the entire
block and/or give it back to the OS before _int_free has a chance to try
and store it into the fastbins list.

The problem is very challenging to reproduce in situ, but using gdb I
have a recipe which demonstrates the crash 100% of the time on my 12.04
x64 system running eglibc 2.15. It relies on malloc_trim, although in
our in situ data, the consolidation is triggered as a result of a normal
free. malloc_trim is just easier to control.

While I am not a glibc developer, I could not see any easy ways to fix
the situation shy of disabling ATOMIC_FASTBINS.

I am attaching the reproduction source. Other pertinent information
follows:

> jpieper at calculon:~/downloads$ lsb_release -rd
> Description:	Ubuntu 12.04 LTS
> Release:	12.04

> jpieper at calculon:~/downloads$ apt-cache policy libc6
> libc6:
> Installed: 2.15-0ubuntu10
> Candidate: 2.15-0ubuntu10
> Version table:
> *** 2.15-0ubuntu10 0
> 500 http://us.archive.ubuntu.com/ubuntu/ precise/main amd64 Packages
> 100 /var/lib/dpkg/status

What I expect: I expect the attached application, when run using the gdb script in the comments, to complete with no failures.
What happened: A SIGSEGV after the final continue.

Reply at:
https://bugs.launchpad.net/ubuntu/+source/eglibc/+bug/1020210/comments/3


** Changed in: eglibc
       Status: Unknown => Confirmed

** Changed in: eglibc
   Importance: Unknown => Medium

-- 
You received this bug notification because you are a member of Ubuntu
Foundations Bugs, which is subscribed to eglibc in Ubuntu.
https://bugs.launchpad.net/bugs/1020210

Title:
  Race condition using ATOMIC_FASTBINS in _int_free causes crash or heap
  corruption

Status in Embedded GLIBC:
  Confirmed
Status in “eglibc” package in Ubuntu:
  Confirmed

Bug description:
  We have an application which makes heavy allocation and de-allocation
  demands from multiple threads.  We run this application continuously
  on many servers, and once every several CPU months or years, we were
  getting a crash in _int_free that did not look like vanilla heap
  corruption.  I believe I have narrowed it down to a race condition in
  _int_free due to the ATOMIC_FASTBINS feature.  Basically, in the
  lockless FASTBIN _int_free path, a chunk is pulled into a local
  variable with the intent to add it to the fastbins list.  However, the
  heap consolidation/trim code can race with this, and can coalesce the
  entire block and/or give it back to the OS before _int_free has a
  chance to try and store it into the fastbins list.

  The problem is very challenging to reproduce in situ, but using gdb I
  have a recipe which demonstrates the crash 100% of the time on my
  12.04 x64 system running eglibc 2.15.  It relies on malloc_trim,
  although in our in situ data, the consolidation is triggered as a
  result of a normal free.  malloc_trim is just easier to control.

  While I am not a glibc developer, I could not see any easy ways to fix
  the situation shy of disabling ATOMIC_FASTBINS.

  I am attaching the reproduction source.  Other pertinent information
  follows:

  > jpieper at calculon:~/downloads$ lsb_release -rd
  > Description:	Ubuntu 12.04 LTS
  > Release:	12.04

  > jpieper at calculon:~/downloads$ apt-cache policy libc6
  > libc6:
  >   Installed: 2.15-0ubuntu10
  >   Candidate: 2.15-0ubuntu10
  >   Version table:
  >  *** 2.15-0ubuntu10 0
  >        500 http://us.archive.ubuntu.com/ubuntu/ precise/main amd64 Packages
  >        100 /var/lib/dpkg/status

  What I expect: I expect the attached application, when run using the gdb script in the comments, to complete with no failures.
  What happened: A SIGSEGV after the final continue.

To manage notifications about this bug go to:
https://bugs.launchpad.net/eglibc/+bug/1020210/+subscriptions