bzr memory usage

[Andrew Bennetts]

Dimitrios Apostolou wrote:
[…]
> >checkout --lightweight is currently only optimised for local use.  As you've
> >seen it currently performs pretty horridly over the network :(
[…]
> 
> I think the user should be warned about this, the normal expectation
> is to run --lightweight because you need less network utilisation.
> What I was actually expecting was no metadata transfers at all, all
> calculations being made server-side and me getting only the results
> of those.

Yes, that might be a good stop-gap until we make it efficient.

FWIW the intent of “lightweight” here is that it's lightweight in disk
consumption, and delegates all branch and repository storage elsewhere,
which is great when you have a local copy.  Specifically it's intended
to support cases like having tree-less branches in one directory, and
then as cheap as possible working trees for those branches (possibly
multiple per branch!) in another.

> >checkouts are designed for.  There is a use case of “I just want the current
> >state of the tree and then no further bzr operations” though, and we probably
> >can and should support that much better, although not necessarily via
> >lightweight checkouts.  Is that your case?
> 
> In my case I'd like to be able to run just a few bzr commands, but
> for now even being able to download the tree would be a huge
> improvement, actually it would make bzr usable for me.

Good to know.  Thanks for the data point.

> For the sake of completeness I have run all 4 available commands to
> download a tree:
> 
> (1) bzr branch lp:gcc
> (2) bzr branch --stacked lp:gcc
> (3) bzr checkout lp:gcc
> (4) bzr checkout --lightweight lp:gcc

I've got a patch to make a small improvement to case (2), and an idea
that might make a larger improvement it as well.  It'll probably be your
best bet in the medium term (aside from workarounds like scp'ing a
tarball).

> Here are the exact results I saw, which make bzr unusable equipment
> with little RAM:
> 
> (1) took 1h and maxed at 800MB VM usage
> (2) took 4.7h and maxed at 500MB VM usage
> (3) was OOM killed (!) after 1h with VM usage exceeding 850MB
> (4) took 5.5h, downloaded about 8GB of data, and used a maximum of 450MB

It's interesting that (1) and (3) should be so different, they're doing
essentially the same work!

> I should note that I did the branch or checkout, without being
> logged in launchpad, and without having done an "init" upfront.

You'll probably find that logging into Launchpad will help memory
consumption a bit: it lets the server do a bit of the work.  It will
also significantly reduce network roundtrips for at least case (1) and
possibly the others too.

> After these command here are the sizes of the local directories:
> 
> $ du -sh *
> 1.2G	gcc-branch
> 624M	gcc-branch-stacked
> 568M	gcc-co-heavy        # OOM killed!
> 624M	gcc-co-light

FWIW the lp:gcc branch on Launchpad is about 1.5G.  (You can see the
individual pack files at
<http://bazaar.launchpad.net/~vcs-imports/gcc/trunk/.bzr/repository/packs/>).

-Andrew.