Handling active dependencies in Go

[David Cheney]

I vote for A. While I don't say this publicly, the go get tool is a toy. It's a good start, but we've clearly exceeded its capacity.


On Monday, 17 December 2012 at 9:03 PM, John Arbash Meinel wrote:

> -----BEGIN PGP SIGNED MESSAGE-----
> Hash: SHA1
> 
> The primary underlying issue we are trying to solve is that we want to
> make sure that juju-core's trunk branch always builds and passes its
> test suite. So that people have a known-working reference when
> developing their own code.
> 
> The secondary issue is that the main tool for downloading building and
> installing go code is the "go get" tool. Which generally translates
> import paths like: import "launchpad.net/goose (http://launchpad.net/goose)" into "bzr branch
> lp:goose $GOPATH/src/launchpad.net/goose (http://launchpad.net/goose)" and installs the result.
> 
> 
> This is a thread on golang-nuts about this:
> https://groups.google.com/d/topic/golang-nuts/0avuiWURSQk/discussion
> 
> 
> It seems the current best practice is to make sure your 'trunk/master'
> branch is always backwards compatible. And if you want to introduce
> API breakage, then you should switch to a versioned import url. (For
> example, you: import "labix.org/v2/mgo (http://labix.org/v2/mgo)")
> 
> 
> I'd like to bring up some limitations and possible solutions, and at
> present what we've come up with as a medium term solution.
> 
> 1. Using 'go get' means that any dependency can trivially accidentally
> break 'go get launchpad.net/juju-core' (http://launchpad.net/juju-core'). If goyaml/gocheck/goose/mgo
> commit a new revision on trunk they risk breaking juju-core's test
> suite.
> 
> 2. It would be nice if we break current trunk, that we can roll back to
> a previous version, and expect tests to pass. At present, there is
> no record of what version of dependencies was used when a given
> revision was committed on trunk (assuming we always have a green
> test suite before a new revision is committed).
> 
> However, rolling back trunk implies rolling back dependencies if
> there was any API changes made.
> 
> 3. At present it is desirable that "lp:goose" have a fair amount of
> flexbility wrt its API stability.
> Eg, we recently had a discussion about whether functions should
> return structs or pointers to structs. Once that is settled, it
> would be good to update most APIs to be consistent with that.
> 
> 
> Some possible solutions:
> 
> A. Don't use 'go get'. Have a 3rd-party tool that works more like
> "buildout". Where you specify both what the dependency is, and what
> exact version of it to use (versions.cfg).
> This can be as trivial as a meta-branch that just has a "Makefile"
> with a list commands like:
> bzr branch -rXXX lp:FOO $GOPATH/src/launchpad.net/FOO (http://launchpad.net/FOO)
> 
> In this case, dependencies are pinned, and you have to land a
> specific commit to update a dependency to a new version.
> 
> B. Put version numbers in your import statements, and publicly version
> your projects APIs.
> 
> Eg, instead of doing:
> import "launchpad.net/goose/identity (http://launchpad.net/goose/identity)"
> 
> You do:
> import "launchpad.net/~gophers/goose/v1/identity (http://launchpad.net/~gophers/goose/v1/identity)"
> 
> The one caveat here, is that the code inside "lp:~gophers/goose/v1"
> *also* has to reference other bits of the project using the same
> identifier.
> 
> So if you wanted to have a fluid 'trunk' branch, it will involve a
> bit of rewriting whenever you want to roll out your trunk branch to
> whatever is the current stable branch.
> 
> For reference, this is what that looks like in lp:goose's case:
> https://code.launchpad.net/~gophers/goose/unstable-001/+merge/140140
> 
> This doesn't insulate juju-core from dependencies breaking the
> build, but it does allow juju-core to say "bad dependency, if you
> want to break the API, you need to roll out a new URL."
> 
> C. Lock-step deployment. We wait on landing a patch to a dependency,
> until we have an approved patch on juju-core that uses the new API.
> 
> So we don't guarantee that any revision of trunk will build an work,
> but we do guarantee that "go get launchpad.net/juju-core/ (http://launchpad.net/juju-core/)..." at the
> current tip will work. (Note that this means that libraries should
> run the test suite of their dependencies before landing a trunk
> commit, so as to not accidentally break them.)
> 
> D. Slightly removed from (C) is to use a separate integration branch
> (which is effectively trunk, but not actually associated with
> 'lp:FOO'). And then wait on updating trunk until we know the test
> suite passes, or we have a patch landing that makes it patch with
> the updated code.
> 
> 
> 
> My personal opinions:
> 
> (A) is tempting, because then things work more like they have worked
> elsewhere. The main caveat is having yet-another tool to maintain.
> (Though possibly this is something like poking at config-manager,
> rather than writing something from scratch.) The primary downside is
> you don't guarantee 'go get' just works, and it requires changes to
> how the 'juju' team does their work.
> 
> (B) Has the very nice property of atomic updates to juju-core. (which
> A also has). It lets you rollback trunk, and still get things working.
> However, it adds a *lot* of process overhead to libraries.
> 
> The actual diff to juju-core isn't terrible:
> 
> https://code.launchpad.net/~jameinel/juju-core/goose-unstable-001/+merge/140141
> But the diff inside lp:goose every time we make an unstable bump is at
> the least ugly:
> https://code.launchpad.net/~gophers/goose/unstable-001/+merge/140140
> 
> I haven't quite sorted out how much of that could be scripted. Could
> we still just develop on trunk, and script pulling trunk changes into
> a 'current stable' API branch. Do the rewrite, run juju-core test
> suite, if it passes, update the branch. Else set up a new stable
> branch, put the code there.
> 
> It also potentially leaves behind a lot of 'unstable-XXX' branches. It
> also means there is a bit of a disconnect of what branch you develop
> on, vs what is being used.
> 
> It seems like a fair amount of overhead to have every dependency that
> is undergoing active development follow this layout, though.
> 
> I suppose if you went into a bi-weekly minor release cycle (0.0.1,
> 0.0.2, etc) that might work out as a reasonable tradeoff of effort vs
> benefit. You still are very tempted to re-use a branch if the API
> hasn't actually changed, in which case you still lose the ability to
> roll back to an exact state.
> 
> To get going, I'm looking to do (D: lock-step staged deployment) with
> an end goal of getting to (B: versioned stable APIs)
> 
> John
> =:->
> -----BEGIN PGP SIGNATURE-----
> Version: GnuPG v1.4.12 (Cygwin)
> Comment: Using GnuPG with undefined - http://www.enigmail.net/
> 
> iEYEARECAAYFAlDO7goACgkQJdeBCYSNAAN14wCfe15DJEXgIwnJyXHX4zNgKdTH
> V94An09EeRpLogCw4Kg0Qmb82mLx8Np3
> =pkbY
> -----END PGP SIGNATURE-----
> 
> -- 
> Juju-dev mailing list
> Juju-dev at lists.ubuntu.com (mailto:Juju-dev at lists.ubuntu.com)
> Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/juju-dev