Bazaar: Out of memory

[John Arbash Meinel]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

James Henstridge wrote:
| 2008/5/9 Ian Clatworthy <ian.clatworthy at internode.on.net>:
|> bazaar at ostkamp.fastmail.fm wrote:
|>
|>  >     parent_texts, left_matching_blocks, nostore_sha, random_id)
|>  >   File "/home/ostkamp/local/lib/python2.5/site-packages/bzrlib/knit.py",
|>  >   line 1035, in _add
|>  >     line_bytes = ''.join(lines)
|>
|>  Ah. That bug is now being encouraged in Bazaar itself, not in the
|>  fastimport plugin. It looks like the same bug though so the fix I
|>  applied may work there as well.
|>
|>  Try replacing line 1035 with these lines, taking care about the
|>  indentation (as Python requires):
|>
|>     lines_bytes = ''
|>     for line in lines:
|>         lines_bytes += line
|>
|>  FWIW, I'm pretty sure we're using the code we do because its faster.
|>  There's not much point to that though if it falls over on large files.
|>  I'd love to get this issue fixed ASAP in Bazaar's core so please let us
|>  know if it fixes your problem.
|
| The .join() method is faster because it does less memory allocations
| and copies than the for loop you list above.  It should allocate a
| single string for the final concatenated string.  In contrast, the for
| loop version does a string allocation and copy for every iteration (so
| in the final iteration you'd expect it to have allocated roughly twice
| the memory).
|
| If this change prevents MemoryErrors then something weird is going on.
|
| James.
|
|

Either way, this would indicate to me that he has a single file which can
consume all of his memory. Or, I should say 2 copies of it. One in the [lines],
and then one as ''.join(lines).

As that is a fairly known bad line, we probably could have an explicit trap for
MemoryError and raise something telling the user what text is causing the
problem, etc.

John
=:->

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.9 (Cygwin)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org

iEYEARECAAYFAkgkY/AACgkQJdeBCYSNAAPtZwCePdlBYLHEWHMOECrGlQ7mYc+n
nmEAnjuxL4WOCva0mkd6A70iKNtDW2Dl
=BgA3
-----END PGP SIGNATURE-----