[Bug 1655447] [NEW] IEEE indexes in netaddr package may not match data in ieee-data package
Mike Pontillo
mike.pontillo at canonical.com
Tue Jan 10 20:27:50 UTC 2017
Public bug reported:
The Debian package for `netaddr` ships with a pre-calculated index for
entries in the IEEE OUI and IAB files, which the `ieee-data` package
places here:
$ dpkg -L python-netaddr | grep idx$
/usr/lib/python2.7/dist-packages/netaddr/eui/oui.idx
/usr/lib/python2.7/dist-packages/netaddr/eui/iab.idx
The index files only work if they match the IEEE data at the time the
`netaddr` package was built. If anything changes in the IEEE data, the
seek locations in the index will be thrown off, causing undefined
behavior.
However, the way `netaddr` is packaged doesn't guarantee that the index
will be kept in sync with the data files in the `ieee-data` pacakge. In
the `netaddr` packages, symbolic links are created to the data files in
the `ieee-data` package as follows:
$ dpkg -L ieee-data | grep usr.share.*.txt
/usr/share/ieee-data/iab.txt
/usr/share/ieee-data/oui.txt
$ ls -la /usr/lib/python2.7/dist-packages/netaddr/eui/*.txt
lrwxrwxrwx 1 root root 38 Oct 23 2015 /usr/lib/python2.7/dist-packages/netaddr/eui/iab.txt -> ../../../../../share/ieee-data/iab.txt
lrwxrwxrwx 1 root root 38 Oct 23 2015 /usr/lib/python2.7/dist-packages/netaddr/eui/oui.txt -> ../../../../../share/ieee-data/oui.txt
Upstream, `netaddr` ships with its own copy of these files. During the
build process, `netaddr` generates an index file for fast lookups into
the OUI and IAB text files.
However, there is a problem with this arrangement that only occurs in
the Debian package: if the `ieee-data` on a given system is out-of-sync
with the index in the `python-netaddr` or `python3-netaddr`, the
undefined behavior begins.
In MAAS, we make heavy use of netaddr, and the symptoms of the index
file being out of date have ranged from an IndexError being raised to
UnicodeDecodeError, since `netaddr` assumes that the index will always
match the data file, and caches the index for the life of the
process.[1]
Symptoms
========
This bug can cause `netaddr` to raise tracebacks during operation, such
as `IndexError`, and possibly others, as a result of the incorrect
indexes. For examples, see also:
Bug #1655049
Bug #1628761
Suggested ways to fix
=====================
(1) The `netaddr` package MUST always be in a state where the index file
is consistent with the IEEE data.
(1a) The `netaddr` package COULD ship with a copy of the IEEE data files
as-designed, so that there is never a mismatch regarding the on-disk
data. (This is the simplest fix, but is likely a Debian policy violation
due to the duplication of data.)
(1b) The `netaddr` package COULD make a copy of the current state of the
OUI and IAB indexes in `ieee-data` when the package is installed or
updated. (Perhaps hard links would work, so that `netaddr` can continue
working with the old data until the indexes are regenerated. But it
would need to fall back to a data copy in case of cross-device links.)
(1c) If the `ieee-data` package changes, and the `netaddr` is using the
data from it, the `netaddr` package MUST correspondingly update its data
files and index.
(2) The `netaddr` upstream code MUST detect if the index file changes
and reload the index (to be usable in a long-running process backed by a
Debian package that could be updated at any time).
Short term, the most important thing needed is a fix for (1), so that
will be the focus for this bug.
Workaround
==========
Regenerate the indexes (for python 2.x and 3.x) as root by running:
sudo python -m netaddr.eui.ieee
sudo python3 -m netaddr.eui.ieee
Either or both commands may be needed, depending on which `netaddr`
packages are installed.
Then, restart any long-running processes (such as MAAS) which may rely
on `netaddr` for OUI lookups.
Note that even if this bug is fixed, the "long-running process" issue
still exists. To fix that, (2) above needs to be resolved.
---
[1]: That means there is a related issue: long-running processes will be
exposed to the same symptoms of this bug, if the `netaddr` package is
upgraded while the long-running process is continues to run. Arguably,
this is a bug that upstream should address by recalculating the index if
the data file changes, and/or forcing an index recalculation if an
integrity check fails, such as by storing a size and hash of the file. I
assume it would be counter-productive to do too many integrity checks,
though, because by the time you get done with them, you may as well have
recalculated the entire index. (And on a slow embedded system, this
might be unacceptable.)
** Affects: maas
Importance: Critical
Status: Won't Fix
** Affects: maas/2.1
Importance: Critical
Status: Won't Fix
** Affects: netaddr
Importance: Undecided
Status: New
** Affects: python-netaddr (Ubuntu)
Importance: Undecided
Status: Confirmed
** Also affects: netaddr
Importance: Undecided
Status: New
** Also affects: maas
Importance: Undecided
Status: New
** Also affects: maas/2.1
Importance: Undecided
Status: New
** Also affects: maas/2.2
Importance: Undecided
Status: New
** Changed in: maas
Status: New => Won't Fix
** Changed in: maas/2.1
Status: New => Won't Fix
** No longer affects: maas/2.2
** Changed in: python-netaddr (Ubuntu)
Status: New => Confirmed
** Description changed:
The Debian package for `netaddr` ships with a pre-calculated index for
entries in the IEEE OUI and IAB files, which the `ieee-data` package
places here:
$ dpkg -L python-netaddr | grep idx$
/usr/lib/python2.7/dist-packages/netaddr/eui/oui.idx
/usr/lib/python2.7/dist-packages/netaddr/eui/iab.idx
The index files only work if they match the IEEE data at the time the
`netaddr` package was built. If anything changes in the IEEE data, the
seek locations in the index will be thrown off, causing undefined
behavior.
However, the way `netaddr` is packaged doesn't guarantee that the index
will be kept in sync with the data files in the `ieee-data` pacakge. In
the `netaddr` packages, symbolic links are created to the data files in
the `ieee-data` package as follows:
$ dpkg -L ieee-data | grep usr.share.*.txt
/usr/share/ieee-data/iab.txt
/usr/share/ieee-data/oui.txt
$ ls -la /usr/lib/python2.7/dist-packages/netaddr/eui/*.txt
lrwxrwxrwx 1 root root 38 Oct 23 2015 /usr/lib/python2.7/dist-packages/netaddr/eui/iab.txt -> ../../../../../share/ieee-data/iab.txt
lrwxrwxrwx 1 root root 38 Oct 23 2015 /usr/lib/python2.7/dist-packages/netaddr/eui/oui.txt -> ../../../../../share/ieee-data/oui.txt
Upstream, `netaddr` ships with its own copy of these files. During the
build process, `netaddr` generates an index file for fast lookups into
- the OUE and IAB text files.
+ the OUI and IAB text files.
However, there is a problem with this arrangement that only occurs in
the Debian package: if the `ieee-data` on a given system is out-of-sync
with the index in the `python-netaddr` or `python3-netaddr`, the
undefined behavior begins.
In MAAS, we make heavy use of netaddr, and the symptoms of the index
file being out of date have ranged from an IndexError being raised to
UnicodeDecodeError, since `netaddr` assumes that the index will always
match the data file, and caches the index for the life of the
process.[1]
-
Symptoms
========
This bug can cause `netaddr` to raise tracebacks during operation, such
as `IndexError`, and possibly others, as a result of the incorrect
indexes. For examples, see also:
- Bug #1655049
- Bug #1628761
-
+ Bug #1655049
+ Bug #1628761
Suggested ways to fix
=====================
(1) The `netaddr` package MUST always be in a state where the index file
is consistent with the IEEE data.
(1a) The `netaddr` package COULD ship with a copy of the IEEE data files
as-designed, so that there is never a mismatch regarding the on-disk
data. (This is the simplest fix, but is likely a Debian policy violation
due to the duplication of data.)
(1b) The `netaddr` package COULD make a copy of the current state of the
OUI and IAB indexes in `ieee-data` when the package is installed or
updated. (Perhaps hard links would work, so that `netaddr` can continue
working with the old data until the indexes are regenerated. But it
would need to fall back to a data copy in case of cross-device links.)
(1c) If the `ieee-data` package changes, and the `netaddr` is using the
data from it, the `netaddr` package MUST correspondingly update its data
files and index.
(2) The `netaddr` upstream code MUST detect if the index file changes
and reload the index (to be usable in a long-running process backed by a
Debian package that could be updated at any time).
Short term, the most important thing needed is a fix for (1), so that
will be the focus for this bug.
-
Workaround
==========
Regenerate the indexes (for python 2.x and 3.x) as root by running:
- sudo python -m netaddr.eui.ieee
+ sudo python -m netaddr.eui.ieee
- sudo python3 -m netaddr.eui.ieee
+ sudo python3 -m netaddr.eui.ieee
Either or both commands may be needed, depending on which `netaddr`
packages are installed.
Then, restart any long-running processes (such as MAAS) which may rely
on `netaddr` for OUI lookups.
Note that even if this bug is fixed, the "long-running process" issue
still exists. To fix that, (2) above needs to be resolved.
---
[1]: That means there is a related issue: long-running processes will be
exposed to the same symptoms of this bug, if the `netaddr` package is
upgraded while the long-running process is continues to run. Arguably,
this is a bug that upstream should address by recalculating the index if
the data file changes, and/or forcing an index recalculation if an
integrity check fails, such as by storing a size and hash of the file. I
assume it would be counter-productive to do too many integrity checks,
though, because by the time you get done with them, you may as well have
recalculated the entire index. (And on a slow embedded system, this
might be unacceptable.)
** Changed in: maas
Importance: Undecided => Critical
** Changed in: maas/2.1
Importance: Undecided => Critical
--
You received this bug notification because you are a member of Ubuntu
OpenStack, which is subscribed to python-netaddr in Ubuntu.
https://bugs.launchpad.net/bugs/1655447
Title:
IEEE indexes in netaddr package may not match data in ieee-data
package
Status in MAAS:
Won't Fix
Status in MAAS 2.1 series:
Won't Fix
Status in netaddr:
New
Status in python-netaddr package in Ubuntu:
Confirmed
Bug description:
The Debian package for `netaddr` ships with a pre-calculated index for
entries in the IEEE OUI and IAB files, which the `ieee-data` package
places here:
$ dpkg -L python-netaddr | grep idx$
/usr/lib/python2.7/dist-packages/netaddr/eui/oui.idx
/usr/lib/python2.7/dist-packages/netaddr/eui/iab.idx
The index files only work if they match the IEEE data at the time the
`netaddr` package was built. If anything changes in the IEEE data, the
seek locations in the index will be thrown off, causing undefined
behavior.
However, the way `netaddr` is packaged doesn't guarantee that the
index will be kept in sync with the data files in the `ieee-data`
pacakge. In the `netaddr` packages, symbolic links are created to the
data files in the `ieee-data` package as follows:
$ dpkg -L ieee-data | grep usr.share.*.txt
/usr/share/ieee-data/iab.txt
/usr/share/ieee-data/oui.txt
$ ls -la /usr/lib/python2.7/dist-packages/netaddr/eui/*.txt
lrwxrwxrwx 1 root root 38 Oct 23 2015 /usr/lib/python2.7/dist-packages/netaddr/eui/iab.txt -> ../../../../../share/ieee-data/iab.txt
lrwxrwxrwx 1 root root 38 Oct 23 2015 /usr/lib/python2.7/dist-packages/netaddr/eui/oui.txt -> ../../../../../share/ieee-data/oui.txt
Upstream, `netaddr` ships with its own copy of these files. During the
build process, `netaddr` generates an index file for fast lookups into
the OUI and IAB text files.
However, there is a problem with this arrangement that only occurs in
the Debian package: if the `ieee-data` on a given system is out-of-
sync with the index in the `python-netaddr` or `python3-netaddr`, the
undefined behavior begins.
In MAAS, we make heavy use of netaddr, and the symptoms of the index
file being out of date have ranged from an IndexError being raised to
UnicodeDecodeError, since `netaddr` assumes that the index will always
match the data file, and caches the index for the life of the
process.[1]
Symptoms
========
This bug can cause `netaddr` to raise tracebacks during operation,
such as `IndexError`, and possibly others, as a result of the
incorrect indexes. For examples, see also:
Bug #1655049
Bug #1628761
Suggested ways to fix
=====================
(1) The `netaddr` package MUST always be in a state where the index
file is consistent with the IEEE data.
(1a) The `netaddr` package COULD ship with a copy of the IEEE data
files as-designed, so that there is never a mismatch regarding the on-
disk data. (This is the simplest fix, but is likely a Debian policy
violation due to the duplication of data.)
(1b) The `netaddr` package COULD make a copy of the current state of
the OUI and IAB indexes in `ieee-data` when the package is installed
or updated. (Perhaps hard links would work, so that `netaddr` can
continue working with the old data until the indexes are regenerated.
But it would need to fall back to a data copy in case of cross-device
links.)
(1c) If the `ieee-data` package changes, and the `netaddr` is using
the data from it, the `netaddr` package MUST correspondingly update
its data files and index.
(2) The `netaddr` upstream code MUST detect if the index file changes
and reload the index (to be usable in a long-running process backed by
a Debian package that could be updated at any time).
Short term, the most important thing needed is a fix for (1), so that
will be the focus for this bug.
Workaround
==========
Regenerate the indexes (for python 2.x and 3.x) as root by running:
sudo python -m netaddr.eui.ieee
sudo python3 -m netaddr.eui.ieee
Either or both commands may be needed, depending on which `netaddr`
packages are installed.
Then, restart any long-running processes (such as MAAS) which may rely
on `netaddr` for OUI lookups.
Note that even if this bug is fixed, the "long-running process" issue
still exists. To fix that, (2) above needs to be resolved.
---
[1]: That means there is a related issue: long-running processes will
be exposed to the same symptoms of this bug, if the `netaddr` package
is upgraded while the long-running process is continues to run.
Arguably, this is a bug that upstream should address by recalculating
the index if the data file changes, and/or forcing an index
recalculation if an integrity check fails, such as by storing a size
and hash of the file. I assume it would be counter-productive to do
too many integrity checks, though, because by the time you get done
with them, you may as well have recalculated the entire index. (And on
a slow embedded system, this might be unacceptable.)
To manage notifications about this bug go to:
https://bugs.launchpad.net/maas/+bug/1655447/+subscriptions
More information about the Ubuntu-openstack-bugs
mailing list