Summary of memory consumption improvements

[John Arbash Meinel]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Ian Clatworthy wrote:
> John Arbash Meinel wrote:
> 
>> Anyway, the numbers are looking pretty good, though as mentioned I'm
>> only at 60%, and I'm shooting for 50% for the 2.1.0b2 release. I still
>> have the chk_map code to land, and some bits to clean up here and there.
> 
> Fantastic work!
> 
> FWIW, the biggest problem I have is memory consumption during
> fast-imports. Trying to pack OOo with 260k revisions is ugly - my 6G
> desktop just isn't enough to do it in 2.0.0. It's not always pack though
> - simply importing the first few thousand (hundred?) revisions of Qt is
> a major memory hog.

So #1 would certainly be 'pack'. I don't know the specific details here,
and certainly I'm surprised that 6GB is 'not enough'.

I *have* seen that when compressing a single large-object, we use about
2-3x memory for the 'DeltaIndex'. (So w/ a 100MB text, we spend 2-300MB
of memory to build the index for rabin hashes into that 100MB text,
along with leaving room for insertions, etc.)

I don't know the OOo codebase, but I don't think they have huge files
that are causing the specific memory blowout.

I certainly would like to look into memory wrt pack. I probably won't do
it in the 2.1.0b2 cycle, though. We'll see how things go with the CHK
code. Which I think is the current dominator for 'bzr branch'.


> 
> When you get a chance, perhaps you'll like to look into what happens
> during either of the above operations? I know imports are a one-off
> process but I suspect the problems are symptoms of problems we'd be
> seeing on commit/push/etc. if Bazaar was actually used on those projects.
> 
> Ian C.
> 

John
=:->
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.9 (Cygwin)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/

iEYEARECAAYFAkref1wACgkQJdeBCYSNAAM7vACgxawj40rzaxe70SKIY3v7k3me
Ej4AoIFdjBaHAlI3otZOct9Q6WMQF3yz
=f2WS
-----END PGP SIGNATURE-----