[RFC] ztransport plugin

[John A Meinel]

Alexander Belchenko wrote:
> I'm working on plugin to create transport with transparent
> gzipping/ungzipping weave files. I named it ztransport.

I might point you in the direction of my 'compressed-weaves' branch.
http://bzr.arbash-meinel.com/branches/bzr/compressed-weaves

It was intended as a new branch format, because of specifically saving a
lot of space. 4x is pretty nice. And inventory.weave compresses
extremely well, because it has a lot of redundancy.

I don't think there was a lot of interest at the time.

Doing it at the transport level is interesting. I think it would be
easier to do it at the store level. Just do what we do know for the
TextStore. Check for a plain (or .gz) first, and if you don't find it,
look for the other one.

With that sort of system, it can be implemented in-place. Without
needing a format upgrade. And also means you don't need a new transport.

John
=:->

> 
> Background:
> 1) I have dial-up internet connection. *It's definitely my personal
> problem*. But when I need to pull some bzr.dev branch -- it every time
> very hard work. Each time when I need actually pull new revisions bzr
> download inventory.weave that have size of more than 2.5 MB. Plus
> additional weaves for changed files. It's very hard. Especially when
> connection breaks during pull.
> 2) I have not much free space on my site. *It's definitely my personal
> problem*. But sometimes people ask me why I not publish my working
> bzr.dev branch. Main problem is I have poor internet connection. So
> frequent upload changed weaves (at least inventory.weave that have size
> ~2.5 MB) is hard to me.
> 
> As I say it's all my personal problems.
> 
> I'm thinking about this and one day I try to compress .bzr directory
> with zip. Result was very encouraging: size reduced down to 4 times. And
> weave files was compressed very well. Because most of them have big size
> and regural structure.
> 
> So I start to write plugin to check my idea to compress weaves on the
> fly when I do push/pull operations. I wrote first draft version of
> ztransport plugin that working with local transport. Simply to check my
> idea. This transport use raw prefix 'zfile://' that simply stripped off
> in constructor of ZLocalTransport class (inherited from LocalTransport).
> In .put and .get methods I do transparent gzipping/ungzipping all files
> with names that endswith('.weave'). It's simple and dirty hack simply to
> check my idea.
> 
> After pushing my bzr.dev branch with this plugin I got this numbers:
> 
> pushed 3094 revisions (this number showed on progress bar during copying)
> 
> pure push (with LocalTransport):
>     push duration:    35 minutes
>     size of resulting .bzr dir:    20504215 bytes
>     size of resulting .bzr/weaves dir:    16297167 bytes
>     size of resulting .bzr/inventory.weave file:    2544162 bytes
> 
> push with 'zfile://' as destination:
>     push duration:    48 minutes
>     size of resulting .bzr dir:    4872308 bytes
>     size of resulting .bzr/weaves dir:    2708340 bytes
>     size of resulting .bzr/inventory.weave file:    530843 bytes
> 
> Overall compress ratio is about 1/4. And for inventory.weave this is
> about 1/5.
> 
> And reweaving each weave each time during pushing consuming much more
> time than gzipping/ungzipping weaves each time. So I think my experiment
> was successful.
> 
> The best thing with this approach is full support by this transport both
> gzipped and ungzipped weaves transparently (but probably not so fast as
> LocalTransport).
> 
> I want to create analogue ztransport for HttpTransport. But using
> modified transports with prefixes like 'zfile://' or 'zhttp://' is bad
> idea. Especially with regard to my previous post about bug with
> 'file://' prefix handling.
> 
> So I think it worth to extend my plugin with zpush/zpull/zbranch/zmerge
> commands. One more: push command store last push location in main bzr
> configuration dir in branches.conf file. But pull use location from
> .bzr/parent file and prefix 'zfile://' is lost (probably because
> transport.base store path without zfile-prefix).
> 
> I'd like to hear some suggestion about improving existing ztransport and
> future ZHttpTransport. What you think about z-prefixed urls or paths
> (zfile/zhhtp) and z-prefixed command?
> 
> If in the future I will publish my branch with gzipped weaves is this
> will not to be ugly or unusable for others?
> 
> 
> And BTW, you could branch ztransport plugin here:
> 
> bzr branch http://bzr.onembedding.com/bzr.win/plugins/ztransport
> 
> -- 
> Alexander
> 
> 


-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 249 bytes
Desc: OpenPGP digital signature
Url : https://lists.ubuntu.com/archives/bazaar/attachments/20060109/7da0c24d/attachment.pgp