bzr for packaging

[James Westby]

Thanks for notifying of this thread Reinhard. I'd like to jump in if I
may.

To everyone else, hi, I am the author of bzr-builddeb. Reinhard has
helped me get the upload in to Debian and Ubuntu, and has suggested some
things to me that would help with Ubuntu packaging, notably easily
supporting -S and -sa. I still haven't got this point right, as I would
like to solve the problem, but in a way that isn't too specific, which
currently I think it is.

Anyway there are a few points from the thread that I would like to
comment upon if you would indulge me.

Martin Pitt wrote
> * Keeping bzr and Debian changelog in sync. I usually write the Debian
>   one and use [1] to automatically generate a bzr one. However, this
>   could become a more official and better integrated tool (plugin
>   instead of a shell wrapper).
>
> * Keeping the changelog release as 'UNRELEASED' while committing
>   changes. This makes it easy to see for other people that they must
>   not start a new package revision. As a very last step before
>   uploading, s/UNRELEASED/feisty/ and 
> 
>     bzr commit -m 'release as 1.2-3 to feisty'
> 
>   so that it is easy to retrieve older package versions just by
>   looking at bzr log.

These are interesting, and something I have considered. svn-buildpackage does
something like the second, with a release command to tag and change the
distribution. Martin F. Krafft, who gave me a lot of the starting ideas
was interested in keeping the plugin non-intrusive, and avoiding doing things
like this.

Also recently John Goerzen posted a script to parse a changelog entry
and extract the bug closings (like tagpending from devscripts), and send
an email that tags the bug pending with the commit message and patch.
This is just one of several things that could come from the integration.

Daniel Holbach wrote
> https://wiki.ubuntu.com/MOTU/Processes/REVU outlines the new review
> process we'll hope to implement soon. It describes the use of a simple
> script in a few scenarios, where the Debian packaging is stored in bzr
> and the tarball is either a) stored as .tar.gz in bzr or b) grabbed from
> s location that's specified somewhere. I'm not sure this is good
> practise for packaging in general, but I think it'll suit us quite well
> in reviewing packages or improving packages in their early days.

That is interesting. I plan to add a functionality like that to
builddeb. You will specify in the config the location (local or remote)
of a directory containing the tarballs for the project. Then when there
is a new version you just 'dch -v' or similar and it notices that a new
upstream is needed that it doesn't have locally yet, and so goes and
retrieves it. This can work well in team maintainance, where one member
of the team grabs the upstream tarball and verifies it then uploads it
somewhere central, then the next time the other members of the team
build they will not have to do that step. To be the most useful though
it would be able to point it at the upstream distribution location.
However it will normally be named wrong in that case, and so some
mapping of names needs to be done. This is possible, but I haven't
worked out the best way yet. It might be possible to integrate with
uscan here, as that does something similar.

Alexandre Vassalotti wrote
> I think bzr need to be better integrated with our packaging tools. For
> example, it would really useful to be able pull and build older
> packages versions using bzr.

This is something that I would really like to do. There are a few issues
with even knowing what branch and revision a certain package came from,
but there are some things that can help there, notably X-VCS-Bzr. With
tags in 0.15 then maybe they can be used to ease things somewhat.

Robert Collins noted that removing all trace of the branch from uploads
makes it unsolvable in every case, as the branch may no longer be
available. It would be possible to upload all the required stuff in the
source package. It seems like Ubuntu may be more amenable to that, but
as you use launchpad you are probably less concerned.

> This would probably require to create a new specialized utility, using
> bzrlib, instead of a plugin. I think cramming a full packaging tool
> into a plugin is not the best way to approch the problem. It certainly
> make the tool, in a way, less user-friendly.

A bzr plugin has full access to bzrlib and can change and extend it in
any way it likes. I don't quite see the problem that you are getting at.

>    bd source <package>   # fetch a package source code, from a bzr/apt-repo
>    bd source --distro <distro> --version <version-number> --revison \
>         <bzr-revision>

This is hoped for as I said above.

>    bd fastbuild  # build without using a chroot
>    bd build      # pbuilder-style build

This is 

  $ bzr bd --quick
  $ bzr bd

by default the builder doesn't use a chroot, but if you change that then
you can set quick-builder to a non chroot script, and the above command
does what you want

>    bd build --distro <distro> --version <version-number> --revison \
>         <bzr-revision>

This I'm not sure what you mean. Is it changing the changelog to that
which you specify? Or does it retrieve what you ask for an build it
straight away?

>    bd new        # create a new package template

This is planned.

>    bd release    # make a release (eg. create debian changelog, run
>tests, build, etc)

I have also considered this. However I haven't been convinced of the
need for much, except maybe having this as an alias of some script that
does the things that you want.

>    bd upload     # update to the archives, or REVU

This is quite specific to the maintainer, and the destination of the
package, but could be done if someone explains what is wanted.

>    bd log        # pretty-print bzr/debian changelog

I'm not sure what you want from this.

>    bd merge      # a merge helper?

As in merging a new upstream. This is also planned. The difficulty is
telling the difference between two ways of working that use the same
builddeb mode.

If you've never used builddeb it currently has 4 modes of building
packages. Normal, native, split and merge. The first is a full source
branch with upstream tarball as well. Native is full source but no
upstream tarball. Split is for non-native packages where the same person
is the author and maintainer, it is just a convenience thing that some
people like. Merge is where debian/ is the only versioned thing. Merge
has different ways of getting the tarball it needs, either from the
filesytem (or downloading it as mentioned earlier), or exporting it from
the upstream bzr branch.

For Ubuntu with the upstream imports either the first or last is
probably the one that you want, but both probably need some work to do
what you want.

>    bd import     # import a non-bzr enabled package

There is some code that dato wrote to try and do this, but it is hard
problem to do optimally (the parallel imports problem is one aspect).
Either solving those issues or using a non-optimal solution is possible.

>    bd bugs       # show open bugs from launchpad for this package

I'm reluctant to tie it to lauchpad, but it would be easy to have a
config option(s) for bug trackers, if the code to retrieve from them was
added. I know debbugs has a SOAP interface, and I assume launchpad has
something similar, so it is certainly possible.

>    bd install    # install a package for testing

It's possible obviously, but the requirements would need to be
explained.

> Yes, that's true. However, I think it could do a bit more, like
> picking the right orig.tar.gz when building a older revision.

This is already done.

> By the way, that's also an issue of our tools -- they are all
> fragmented and scattered a bit everywhere. Yes, I know that the Unix
> tradition is to small programs do one thing well. But, how about a
> program that glue in a clean interface all the useful packaging tools?
> Anyway, I'm just dreaming again.

This a dream of mine as well. Maybe it needs to be a level higher than
builddeb, and I would want it to be VCS-agnostic. There are a few people
poking around this area.

> There was bug in the changelog parser (I don't know if it's the same
> bug as the one you saw) that made bzr-builddeb fail when it
> encountered changelog entries in the legacy format:
> 
> <date> <author> (<email>)
>   * <change>
>   * <change>

Ah, I had no idea this format existed. A bug report on python-debian
would keep this tracked (preferably at bugs.debian.org where I can't
avoid it). Also a description of the format suitable to write a parser
from would be very much appreciated. I guess it can be extracted from
dpkg, as debian-policy doesn't include it.

Martin Pitt wrote
> That's what debcommit does (devscripts package)

Ah, I didn't know of that, thanks. Joey strikes again it seems.

> If we can get https://wiki.ubuntu.com/NoMoreSourcePackages working,
> and bzr loom becomes reality, then it would make a difference, I
> think.

I am interested in achieving this. Someone told me that
NoMoreSourcePackages in that form was no longer being considered
however.

A couple more pointers, the loom plugin reminds me a lot of StGit, which
is a great tool for use with git. It has its differences, but would be
worth a look. There's also mercurial queues which a lot of people say is
good, but that I have never tried.

Finally there is darcs-buildpackage and git-buildpackage. I have only
looked at the latter, but John Goerzen said that it was based on his
darcs-buildpackage code. This is interesting for the way that branches
are used for handling new upstreams and the like.

Apologies for the length of the email.

Please CC me on any replies, as I am not subscribed.

Thanks,

James

-- 
  James Westby   --    GPG Key ID: B577FE13    --     http://jameswestby.net/
  seccure key - (3+)k7|M*edCX/.A:n*N!>|&7U.L#9E)Tu)T0>AM - secp256r1/nistp256