2.0 upgrade experiences

[Robert Collins]

On Tue, 2009-08-11 at 16:20 -0500, Neil Martinsen-Burrell wrote:

> 1.  Running ``bzr check`` in a standalone branch that was originally in 
> format 1.9 gives me a load of output like:
> 
> [...]
> Missing inventory {('TREE_ROOT', 
> 'svn-v2:1179 at 8f9f66e6-09d4-0310-8052-96bf1e615463-')}
> Missing inventory {('TREE_ROOT', 
> 'nmb at wartburg.edu-20090417201757-chtr18w6ddnpcal3')}
> Missing inventory {('TREE_ROOT', 
> 'nmb at quaggy.local-20071106145233-vr33wd75mx91n97l')}
> Missing inventory {('TREE_ROOT', 
> 'svn-v2:724 at 8f9f66e6-09d4-0310-8052-96bf1e615463-')}
> Missing inventory {('TREE_ROOT', 
> 'nmb at wartburg.edu-20071109170840-o838ndsrs1prwl1m')}
> Missing inventory {('TREE_ROOT', 
> 'nmb at wartburg.edu-20071115142153-pkeip1evaq353ubc')}

What is the format of your source branch? you say 1.9, but those errors
(and the svn revids) make me think it was 1.9-rich-root.

> It is not clear from ``bzr help check`` whether or not a missing 
> inventory for the tree root "indicates a problem".

Its a problem - any output from check other than summary data is a
problem. We should make this clearer.

> 2. The space savings is not as big as my wildest dreams (This is a 
> standalone branch, even though the parent directory is called "repos".)
> 
> nmb at rameses[~/repos/career]$ du -sk .
> 6848	.
> nmb at rameses[~/repos/career]$ du -sk ../../repos.bak/career
> 7028	../../repos.bak/career

You will have content in .bzr/repository/obsolete-packs that will get
cleaned up when you make subsequent commits. We do this to deal with
various edge cases without requiring fsync/fdatasync operations (which
are either a) expensive, or b) [on XFS and MacOSX] don't work).

> When I thought that a pack operation might clear up additional space, I 
> got the following error:

bzr auto packs on conversion, there is no need to run pack. The general
rule for running 'bzr pack' is 'do not run this command'.

> nmb at rameses[~/repos/career]$ bzr pack
> bzr: ERROR: Pack '6ffbb186d5539f27e54609f64e5fb92e' already exists in 
> <bzrlib.repofmt.groupcompress_repo.GCRepositoryPackCollection object at 
> 0x126c3d0>
> 
> Is that error serious?

Its a known bug with 2a, and a fix is in-progress at the moment in a
branch of Andrews.

>   Does it mean that I cannot pack this branch 
> further?

It means that unlike 1.9 bzr is willing to attempt a pack when there is
only one pack present, and that it doesn't cleanly report the case when
the repository is already fully packed. 

>   Are there obsolete packs still sitting around somewhere? 

Possibly, you can look in .bzr/repository/obsolete_packs. The contents
of this directory can be removed (but be sure that your filesystem has
recorded the bzr dir to disk before doing the removal). Leave the
directory in place though.

>  Can 
> these be gotten rid of somehow?  This suggests that the space savings 
> could be quite a bit larger:
> 
> nmb at rameses[~/repos/career]$ du -sk .bzr/repository/obsolete_packs/
> 2260	.bzr/repository/obsolete_packs/
> 
> What processes can be used to empty out these obsolete packs?  When is 
> that process safe?

Once your new packs have hit disk, which is _extremely_ hard to tell
reliably. On some protocols we simply can't tell (SFTP/FTP/HTTP[S]), on
others asking for them to hit disk causes very expensive operations
globally rather than locally to that file (ext3), and on yet others the
operations required to make it hit disk are stubbed out (because tools
that didn't really need to ask for things to hit disk did so so often
the operating system vendor stubbed out the call and added new separate
calls for programs that really want to be sure data has hit disk).

> 3.  In a different branch, ``bzr check`` tells me that I have "sha1 
> mismatch"
...
> errors.  How does one go about fixing these SHA1 errors?  What will be 
> the future impact of them?  How did they get in there?  (This branch is 
> a bzr-svn conversion of part of an old SVN repository.)

Please file a bug on this one; on 'bzr' though it may be a bzr-svn
issue.

> 4.  The progress bar for upgrade goes 70% of the way to finished 
> immediately and then takes all of its time working on "Copying content 
> into repository.:Transferring revisions".  Is there any way to make the 
> progress on the progress bar corrspond more nearly to the time that 
> various parts of the process actually take?

Progress is hard :(. Feel free to file a bug on this, but I don't think
that we're likely to get time to really improve it before 2.0 - we have
lots of functional issues to fix.

> 5. Standalone branches are much easier to upgrade than those living in 
> shared repositories.  ...I hope that shared repository usage and policy will 
> be back on the table, because a single repository that large is taking a 
> *long* time to check before even starting an upgrade.

I use similar shared repositories. My recent check landing should have
made check faster for all repositories, but there is more that can be
done. Once the repository is checked, you have checked all the branches
as far as repository content goes - so there is no need to check the
repository. bzr check --no-repository will help there. (I think thats
how the option is spelt).

> Also around shared repositories, ``bzr check`` checks the branches 
> within that subdirectory when run at the repository level (prior to 
> upgrading the repository).  Does this mean that I don't need to run 
> check again when I am converting each branch in the following step?

Yes.

> Also for shared repositories, I find myself doing an unreasonable amount 
> of "for branch in `ls`; do bzr <something> $branch; done".  Can we 
> create a sensible recursive policy for this upgrade process?  Check is 
> already recursive.  Should upgrade be?

I believe Ian has an upgrade version that is.

> Overall, I'm pleased with the upgrade process and the smooth rhythm of
> 
> $ bzr check # WT... oh well
> $ bzr upgrade --2a
> 
> but I think that more effort needs to go into documentation and error 
> reporting so that the process is self-explanatory for the large part. 
> Just another data point...

Thanks! I really appreciate this feedback.

-Rob
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 197 bytes
Desc: This is a digitally signed message part
Url : https://lists.ubuntu.com/archives/bazaar/attachments/20090812/33ad8b70/attachment-0001.pgp