Too much data being downloaded with sftp

[Sowers, Brent - ES]

I went back to sftp to try to get some info for you.  Now, after I had run bzr pack on the repository on the server earlier this morning, the data transfer for a new checkout in a completely new directory is only 12 megs, versus the 430 megs before I ran bzr pack.

As far as the contents of the repository.  It's a Ruby on Rails web app.  There are no large files and there never have been.  Most of the repository is relatively small source code files, most of which haven't changed frequently, with a few megs of PNG graphics that never change (other than being added or removed).  I've made sure from the very beginning of the repository to put log files and temporary files in .bzrignore.  Let me know if there is anything that you'd like me to run on the repository to get more info.

________________________________
From: John Arbash Meinel [john at arbash-meinel.com]
Sent: Friday, September 10, 2010 11:41 AM
To: Martin Pool
Cc: bazaar at lists.canonical.com; Sowers, Brent - ES
Subject: Re: Too much data being downloaded with sftp

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

On 9/10/2010 10:40 AM, John Arbash Meinel wrote:
> On 9/9/2010 9:56 PM, Martin Pool wrote:
>> Hi Brent,
>
>> The short answer is if at all possible you should install bzr on the
>> server and use bzr+ssh instead; most of our performance work is now
>> focussed on that not sftp or ftp.
>
>> From your logs the problem may be that something is overflowing a
>> cache so it's repeatedly reading the same object from the server.  You
>> could file a bug on this at <https://bugs.launchpad.net/bzr/>.  This
>> sounds familiar but I can't find a dupe.
>
> It would certainly be good to understand why we aren't caching that
> object. It looks like a GroupCompressBlock that is bigger than we
> expect. Perhaps you just have a single file that is significant in size?
> (Though it would have to have several revisions or we wouldn't try to
> download it repeatedly...)
>
> John
> =:->
>

Just another thought. Maybe it is very redundant data that compresses
very well with zlib (consider 50MB of all NULLs.). It might be possible
that we see the uncompressed size of the GCBlock and decide it is too
big to hold in memory for now. I'd have to see the details.

John
=:->
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.9 (Cygwin)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/

iEYEARECAAYFAkyKUboACgkQJdeBCYSNAANHLACfQN9jRk8iA2yEUdh4D7KjGBes
XVYAoKeMw5QOVySsjCmjFNzReVsCZTZ3
=Jhjz
-----END PGP SIGNATURE-----

________________________________
This e-mail and any files transmitted with it may be proprietary and are intended solely for the use of the individual or entity to whom they are addressed. If you have received this e-mail in error please notify the sender.
Please note that any views or opinions presented in this e-mail are solely those of the author and do not necessarily represent those of ITT Corporation. The recipient should check this e-mail and any attachments for the presence of viruses. ITT accepts no liability for any damage caused by any virus transmitted by this e-mail.