Handling active dependencies in Go

[John Arbash Meinel]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

The primary underlying issue we are trying to solve is that we want to
make sure that juju-core's trunk branch always builds and passes its
test suite. So that people have a known-working reference when
developing their own code.

The secondary issue is that the main tool for downloading building and
installing go code is the "go get" tool. Which generally translates
import paths like: import "launchpad.net/goose" into "bzr branch
lp:goose $GOPATH/src/launchpad.net/goose" and installs the result.


This is a thread on golang-nuts about this:
 https://groups.google.com/d/topic/golang-nuts/0avuiWURSQk/discussion


It seems the current best practice is to make sure your 'trunk/master'
branch is always backwards compatible. And if you want to introduce
API breakage, then you should switch to a versioned import url. (For
example, you: import "labix.org/v2/mgo")


I'd like to bring up some limitations and possible solutions, and at
present what we've come up with as a medium term solution.

1. Using 'go get' means that any dependency can trivially accidentally
   break 'go get launchpad.net/juju-core'. If goyaml/gocheck/goose/mgo
   commit a new revision on trunk they risk breaking juju-core's test
   suite.

2. It would be nice if we break current trunk, that we can roll back to
   a previous version, and expect tests to pass. At present, there is
   no record of what version of dependencies was used when a given
   revision was committed on trunk (assuming we always have a green
   test suite before a new revision is committed).

   However, rolling back trunk implies rolling back dependencies if
   there was any API changes made.

3. At present it is desirable that "lp:goose" have a fair amount of
   flexbility wrt its API stability.
   Eg, we recently had a discussion about whether functions should
   return structs or pointers to structs. Once that is settled, it
   would be good to update most APIs to be consistent with that.


Some possible solutions:

A. Don't use 'go get'. Have a 3rd-party tool that works more like
   "buildout". Where you specify both what the dependency is, and what
   exact version of it to use (versions.cfg).
   This can be as trivial as a meta-branch that just has a "Makefile"
   with a list commands like:
      bzr branch -rXXX lp:FOO $GOPATH/src/launchpad.net/FOO

   In this case, dependencies are pinned, and you have to land a
   specific commit to update a dependency to a new version.

B. Put version numbers in your import statements, and publicly version
   your projects APIs.

   Eg, instead of doing:
     import "launchpad.net/goose/identity"

   You do:
     import "launchpad.net/~gophers/goose/v1/identity"

   The one caveat here, is that the code inside "lp:~gophers/goose/v1"
   *also* has to reference other bits of the project using the same
   identifier.

   So if you wanted to have a fluid 'trunk' branch, it will involve a
   bit of rewriting whenever you want to roll out your trunk branch to
   whatever is the current stable branch.

   For reference, this is what that looks like in lp:goose's case:
    https://code.launchpad.net/~gophers/goose/unstable-001/+merge/140140

   This doesn't insulate juju-core from dependencies breaking the
   build, but it does allow juju-core to say "bad dependency, if you
   want to break the API, you need to roll out a new URL."

C. Lock-step deployment. We wait on landing a patch to a dependency,
   until we have an approved patch on juju-core that uses the new API.

   So we don't guarantee that any revision of trunk will build an work,
   but we do guarantee that "go get launchpad.net/juju-core/..." at the
   current tip will work. (Note that this means that libraries should
   run the test suite of their dependencies before landing a trunk
   commit, so as to not accidentally break them.)

D. Slightly removed from (C) is to use a separate integration branch
   (which is effectively trunk, but not actually associated with
   'lp:FOO'). And then wait on updating trunk until we know the test
   suite passes, or we have a patch landing that makes it patch with
   the updated code.



My personal opinions:

(A) is tempting, because then things work more like they have worked
elsewhere. The main caveat is having yet-another tool to maintain.
(Though possibly this is something like poking at config-manager,
rather than writing something from scratch.) The primary downside is
you don't guarantee 'go get' just works, and it requires changes to
how the 'juju' team does their work.

(B) Has the very nice property of atomic updates to juju-core. (which
A also has). It lets you rollback trunk, and still get things working.
However, it adds a *lot* of process overhead to libraries.

The actual diff to juju-core isn't terrible:

https://code.launchpad.net/~jameinel/juju-core/goose-unstable-001/+merge/140141
But the diff inside lp:goose every time we make an unstable bump is at
the least ugly:
  https://code.launchpad.net/~gophers/goose/unstable-001/+merge/140140

I haven't quite sorted out how much of that could be scripted. Could
we still just develop on trunk, and script pulling trunk changes into
a 'current stable' API branch. Do the rewrite, run juju-core test
suite, if it passes, update the branch. Else set up a new stable
branch, put the code there.

It also potentially leaves behind a lot of 'unstable-XXX' branches. It
also means there is a bit of a disconnect of what branch you develop
on, vs what is being used.

It seems like a fair amount of overhead to have every dependency that
is undergoing active development follow this layout, though.

I suppose if you went into a bi-weekly minor release cycle (0.0.1,
0.0.2, etc) that might work out as a reasonable tradeoff of effort vs
benefit. You still are very tempted to re-use a branch if the API
hasn't actually changed, in which case you still lose the ability to
roll back to an exact state.

To get going, I'm looking to do (D: lock-step staged deployment) with
an end goal of getting to (B: versioned stable APIs)

John
=:->
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.12 (Cygwin)
Comment: Using GnuPG with undefined - http://www.enigmail.net/

iEYEARECAAYFAlDO7goACgkQJdeBCYSNAAN14wCfe15DJEXgIwnJyXHX4zNgKdTH
V94An09EeRpLogCw4Kg0Qmb82mLx8Np3
=pkbY
-----END PGP SIGNATURE-----