lp:bzr/2.0 upgrade --2a runs out of memory

[Brian de Alwis]

I tried using the tip from lp:bzr/2.0 to try upgrading a largish  
branch, but the upgrade failed as it ran out of memory.  I ran a  
parallel ps updating every 10 seconds, and the memory maxed out at  
3.5GB before it died.

The branch is formed of manually-made periodic snapshots from the  
pkgsrc.org CVS tree.  This tree is broad, but relatively shallow, with  
the latest revision containing 15297 directories and 79723.  There are  
only 21 revisions, but each revision has a large number of changes.   
The 1.9 .bzr is 129MB in size.  I can make this repository available,  
if it helps.

This is on MacOS X 10.5.8, with Python 2.5.1.

$ ulimit -d
unlimited
$ ~/tmp/bzr-2.0/bzr info -v
Standalone branch (format: 1.14 or 1.9)
Location:
  branch root: .

Related branches:
  parent branch: /scratch/dev/cvs-pkgsrc.org

Format:
       control: Meta directory format 1
        branch: Branch format 7
    repository: Packs 6 (uses btree indexes, requires bzr 1.9)

Branch history:
        21 revisions
       604 days old
   first revision: Wed 2008-01-09 09:38:08 -0600
  latest revision: Wed 2009-08-19 17:09:46 -0400

Repository:
        21 revisions

$ ~/tmp/bzr-2.0/bzr upgrade --2a
starting upgrade of file:///private/tmp/pkgsrc/
making backup of file:///private/tmp/pkgsrc/.bzr
  to file:///private/tmp/pkgsrc/backup.bzr
starting repository conversion
[##############-     ] Copying content into repository.:Transferring  
revisions 0/21
Python(1761) malloc: *** mmap(size=20480) failed (error code=12)
*** error: can't allocate region
*** set a breakpoint in malloc_error_break to debug
bzr: out of memory

.bzr.log:

Fri 2009-09-04 14:00:47 -0400
0.142  bzr arguments: [u'upgrade', u'--2a']
0.173  looking for plugins in /Users/bsd/.bazaar/plugins
0.757  looking for plugins in /Users/bsd/tmp/bzr-2.0/bzrlib/plugins
0.778  looking for plugins in /Library/Python/2.5/site-packages/bzrlib/ 
plugins
0.854  encoding stdout as sys.stdout encoding 'UTF-8'
7.249  creating repository in file:///private/tmp/pkgsrc/.bzr/.
17.871  Resizing the inventory entry cache from 10240 to 119264
2173.209  Traceback (most recent call last):
  File "/Users/bsd/tmp/bzr-2.0/bzrlib/commands.py", line 835, in  
exception_to_return_code
    return the_callable(*args, **kwargs)
  File "/Users/bsd/tmp/bzr-2.0/bzrlib/commands.py", line 1030, in  
run_bzr
    ret = run(*run_argv)
  File "/Users/bsd/tmp/bzr-2.0/bzrlib/commands.py", line 647, in  
run_argv_aliases
    return self.run(**all_cmd_args)
  File "/Users/bsd/tmp/bzr-2.0/bzrlib/builtins.py", line 3162, in run
    upgrade(url, format)
  File "/Users/bsd/tmp/bzr-2.0/bzrlib/upgrade.py", line 85, in upgrade
    Convert(url, format)
  File "/Users/bsd/tmp/bzr-2.0/bzrlib/upgrade.py", line 40, in __init__
    self.convert()
  File "/Users/bsd/tmp/bzr-2.0/bzrlib/upgrade.py", line 79, in convert
    self.bzrdir = converter.convert(self.bzrdir, self.pb)
  File "/Users/bsd/tmp/bzr-2.0/bzrlib/bzrdir.py", line 3023, in convert
    converter.convert(repo, pb)
  File "/Users/bsd/tmp/bzr-2.0/bzrlib/repository.py", line 4082, in  
convert
    self.source_repo.copy_content_into(converted)
  File "/Users/bsd/tmp/bzr-2.0/bzrlib/repository.py", line 1540, in  
copy_content_into
    return InterRepository.get(self,  
destination).copy_content(revision_id)
  File "/Users/bsd/tmp/bzr-2.0/bzrlib/decorators.py", line 192, in  
write_locked
    result = unbound(self, *args, **kwargs)
  File "/Users/bsd/tmp/bzr-2.0/bzrlib/repository.py", line 3393, in  
copy_content
    self.target.fetch(self.source, revision_id=revision_id)
  File "/Users/bsd/tmp/bzr-2.0/bzrlib/repository.py", line 1695, in  
fetch
    find_ghosts=find_ghosts, fetch_spec=fetch_spec)
  File "/Users/bsd/tmp/bzr-2.0/bzrlib/decorators.py", line 192, in  
write_locked
    result = unbound(self, *args, **kwargs)
  File "/Users/bsd/tmp/bzr-2.0/bzrlib/repository.py", line 4008, in  
fetch
    self._fetch_all_revisions(revision_ids, pb)
  File "/Users/bsd/tmp/bzr-2.0/bzrlib/repository.py", line 3963, in  
_fetch_all_revisions
    basis_id = self._fetch_batch(batch, basis_id, cache)
  File "/Users/bsd/tmp/bzr-2.0/bzrlib/repository.py", line 3898, in  
_fetch_batch
    not self.target._format._fetch_uses_deltas))
  File "/Users/bsd/tmp/bzr-2.0/bzrlib/groupcompress.py", line 1600, in  
insert_record_stream
    for _ in self._insert_record_stream(stream, random_id=False):
  File "/Users/bsd/tmp/bzr-2.0/bzrlib/groupcompress.py", line 1741, in  
_insert_record_stream
    flush()
  File "/Users/bsd/tmp/bzr-2.0/bzrlib/groupcompress.py", line 1635, in  
flush
    bytes = self._compressor.flush().to_bytes()
  File "/Users/bsd/tmp/bzr-2.0/bzrlib/groupcompress.py", line 326, in  
to_bytes
    self._create_z_content()
  File "/Users/bsd/tmp/bzr-2.0/bzrlib/groupcompress.py", line 319, in  
_create_z_content
    self._create_z_content_from_chunks()
  File "/Users/bsd/tmp/bzr-2.0/bzrlib/groupcompress.py", line 307, in  
_create_z_content_from_chunks
    compressed_chunks = map(compressor.compress, self._content_chunks)
MemoryError

2173.222  return code 3

-- 
"Amusement to an observing mind is study." - Benjamin Disraeli