Something gone awry, new stack filling too much in shared?

[John Arbash Meinel]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

I just issued a 'bzr pack' on my --1.9 format bzr.dev repository, and
the size shrunk from 220MB down to 174MB.

Since 1.9 doesn't actually rebuild text compression during 'bzr pack'
but just moves everything into a single file, some process has been
downloading 50MB of wasted data into my repo. Which is about 25% overhead...

My guess is the new code for filling in stacking parents was also
'filling-in' repositories that already had the data. Also of concern is
that this repository was 100MB a couple months ago. I know we had some
big changes, like the FSF copyright change, and some whitespace
cleanups. I don't think that quite accounts for the repository doubling
in size, though.

One concern is that if we insert redundant forms of a given key, the
'pack' code will arbitrarily pick one to use. Which means that if we
insert a fulltext (such as of a file text, or of an inventory) that
might get preserved in the repository, and thus transmitted forever more
as a fulltext.

In one sense, this is still 'correct' in that the data is always
available. In the other, it is causing 'bloat' because it is randomly
expanding certain texts.

I don't know that this is something to focus on, since we 'have a format
coming that will re-delta during "bzr pack"', but I know it bothered me
a little bit to see how big my repository was, and how much it shrank
via simple 'bzr pack'.

Anyway, having data go 'missing' is obvious, because it causes a fetch
failure. Having data be redundantly copied is not, but I'm thinking we
probably need adequate test coverage for that sort of thing.

John
=:->
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.9 (Cygwin)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org

iEYEARECAAYFAkoMQGkACgkQJdeBCYSNAAOCdACfajkkyqWGu2W+QSzEQYUcnBl7
E7kAnR6ue9bW99Gz5A76D5UAcLJ/oGOs
=HMxD
-----END PGP SIGNATURE-----