[Bug 502002] [NEW] pbzip2 can fail to extract large bz2 files compressed using the traditional single threaded bzip2 tool

[Viktor Ferenczi]

Public bug reported:

Binary package hint: pbzip2

The pbzip2 manual says:

"If  you  have  a large file that was created with bzip2 (say 1.5GB for
example) you will likely not be able to decompress the file with pbzip2
since pbzip2 will try to allocate 1.5GB of memory to decompress it, and
that call might  fail  depending  on  your  system resources.   If  the
same 1.5GB file had of been compressed with pbzip2, it would decompress
fine with pbzip2.  If you are unable to decompress a file with pbzip2
due to its size, use the regular bzip2 instead."

You can easily run into such a problem on a memory limited VPS (with
multiple cores) if you want to decompress a large backup file there.

Why this problem is important:

I think it is pretty unreasonable that a decompression tool can fail
because of the above reason. This kind of failure is largely unexpected
by administrators and could cause serious problems in automated scripts,
which could result in a data loss if left unnoticed. It also means
excess memory usage while trying to decompress a file of a size more
than the amount of available physical memory, which could result in
severely reduced performance caused by excessive swapping.

Suggested solution:

Above a given "chunk" size pbzip2 should not try to allocate memory,
just decompress in single threaded mode without trying to load the whole
compressed data stream into memory. Maybe it should fall back to this
legacy mode in any low-memory condition automatically, since it is still
better to decompress the data in single threaded mode than failing or
slowing down due to too much memory allocation.

** Affects: pbzip2 (Ubuntu)
     Importance: Undecided
         Status: New


** Tags: allocation compression failure file large memory multicore multithreaded

-- 
pbzip2 can fail to extract large bz2 files compressed using the traditional single threaded bzip2 tool
https://bugs.launchpad.net/bugs/502002
You received this bug notification because you are a member of Ubuntu
Bugs, which is subscribed to Ubuntu.

-- 
ubuntu-bugs mailing list
ubuntu-bugs at lists.ubuntu.com
https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs