[Bug 1983554] [NEW] gmon memory corruption due wrong calculation of required buffer size

[Leo Yuriev]

Public bug reported:

The `__monstartup()` allocates a buffer used to store all the data
accumulated by the monitor.

The size of this buffer depends on the size of the internal structures
used and the address range for which the monitor is activated, as well
as on the maximum density of call instuctions and/or callable functions
that could be potentially on a segment of executable code.

In particular a hash table of arcs is placed at the end of this buffer.
The size of this hash table is calculated in bytes as `p->fromssize =
p->textsize / HASHFRACTION`, but actually should be `p->fromssize =
ROUNDUP(p->textsize / HASHFRACTION, sizeof(*p->froms))`.

This results in writing beyond the end of the allocated buffer when an
added arc corresponds to a call near from the end of the monitored
address range, since `_mcount()` check the incoming caller address for
monitored range but not the intermediate result hash-like index that
uses to write into the table.

It should be noted that when the results are output to `gmon.out`, the
table is read to the last element calculated from the allocated size in
bytes, so the arcs stored outside the buffer boundary did not fall into
`gprof` for analysis. Thus this "feature" help me to found this bug
during working with DSO-profiling
(https://sourceware.org/bugzilla/show_bug.cgi?id=29438).

Another minor error seems a related typo in the calculation of
`kcountsize`.

Upstream bugzilla ref:
https://sourceware.org/bugzilla/show_bug.cgi?id=29444

** Affects: glibc (Ubuntu)
     Importance: Undecided
         Status: New

** Patch added: "0001-gmon-fix-memory-corruption-due-wrong-calculation-of-.patch"
   https://bugs.launchpad.net/bugs/1983554/+attachment/5606920/+files/0001-gmon-fix-memory-corruption-due-wrong-calculation-of-.patch

** Description changed:

  The `__monstartup()` allocates a buffer used to store all the data
  accumulated by the monitor.
  
  The size of this buffer depends on the size of the internal structures
  used and the address range for which the monitor is activated, as well
  as on the maximum density of call instuctions and/or callable functions
  that could be potentially on a segment of executable code.
  
  In particular a hash table of arcs is placed at the end of this buffer.
  The size of this hash table is calculated in bytes as `p->fromssize =
  p->textsize / HASHFRACTION`, but actually should be `p->fromssize =
  ROUNDUP(p->textsize / HASHFRACTION, sizeof(*p->froms))`.
  
  This results in writing beyond the end of the allocated buffer when an
  added arc corresponds to a call near from the end of the monitored
  address range, since `_mcount()` check the incoming caller address for
  monitored range but not the intermediate result hash-like index that
  uses to write into the table.
  
  It should be noted that when the results are output to `gmon.out`, the
  table is read to the last element calculated from the allocated size in
  bytes, so the arcs stored outside the buffer boundary did not fall into
- `gprof` for analysis. Thus  this "feature" help me to found this bug
- during working with Bug 29438.
+ `gprof` for analysis. Thus this "feature" help me to found this bug
+ during working with DSO-profiling
+ (https://sourceware.org/bugzilla/show_bug.cgi?id=29438).
  
  Another minor error seems a related typo in the calculation of
  `kcountsize`.
  
+ Upstream bugzilla ref:
  https://sourceware.org/bugzilla/show_bug.cgi?id=29444

-- 
You received this bug notification because you are a member of Ubuntu
Foundations Bugs, which is subscribed to glibc in Ubuntu.
https://bugs.launchpad.net/bugs/1983554

Title:
  gmon memory corruption due wrong calculation of required buffer size

Status in glibc package in Ubuntu:
  New

Bug description:
  The `__monstartup()` allocates a buffer used to store all the data
  accumulated by the monitor.

  The size of this buffer depends on the size of the internal structures
  used and the address range for which the monitor is activated, as well
  as on the maximum density of call instuctions and/or callable
  functions that could be potentially on a segment of executable code.

  In particular a hash table of arcs is placed at the end of this
  buffer. The size of this hash table is calculated in bytes as
  `p->fromssize = p->textsize / HASHFRACTION`, but actually should be
  `p->fromssize = ROUNDUP(p->textsize / HASHFRACTION,
  sizeof(*p->froms))`.

  This results in writing beyond the end of the allocated buffer when an
  added arc corresponds to a call near from the end of the monitored
  address range, since `_mcount()` check the incoming caller address for
  monitored range but not the intermediate result hash-like index that
  uses to write into the table.

  It should be noted that when the results are output to `gmon.out`, the
  table is read to the last element calculated from the allocated size
  in bytes, so the arcs stored outside the buffer boundary did not fall
  into `gprof` for analysis. Thus this "feature" help me to found this
  bug during working with DSO-profiling
  (https://sourceware.org/bugzilla/show_bug.cgi?id=29438).

  Another minor error seems a related typo in the calculation of
  `kcountsize`.

  Upstream bugzilla ref:
  https://sourceware.org/bugzilla/show_bug.cgi?id=29444

To manage notifications about this bug go to:
https://bugs.launchpad.net/ubuntu/+source/glibc/+bug/1983554/+subscriptions