store migration

[Francesco Banconi]

Hi all,

FYI this morning the UI team discussed about possibile paths to migrate the store code from lp:juju-core to Github.
We are sharing this plan so that we can synchronize/collaborate with other teams with similar needs in their todo list.

Here are the store package dependencies as obtained with go list:

Deps:
$ go list -f "{{range .Deps}}{{println .}}{{end}}" launchpad.net/juju-core/store | grep juju-core
launchpad.net/juju-core/charm
launchpad.net/juju-core/charm/hooks
launchpad.net/juju-core/juju/osenv
launchpad.net/juju-core/schema
launchpad.net/juju-core/thirdparty/pbkdf2
launchpad.net/juju-core/utils
launchpad.net/juju-core/utils/set
launchpad.net/juju-core/utils/zip

Tests deps:
$ for i in `go list -f "{{range .Deps}}{{println .}}{{end}}" launchpad.net/juju-core/store`; do go list -f "{{range .XTestImports}}{{println .}}{{end}}" $i; done | grep juju-core | sort | uniq
launchpad.net/juju-core/charm
launchpad.net/juju-core/charm/testing
launchpad.net/juju-core/environs/config
launchpad.net/juju-core/juju/osenv
launchpad.net/juju-core/schema
launchpad.net/juju-core/testing
launchpad.net/juju-core/testing/filetesting
launchpad.net/juju-core/testing/testbase
launchpad.net/juju-core/utils
launchpad.net/juju-core/utils/set
launchpad.net/juju-core/utils/zip

As you can see, there are some incremental steps we will need to follow to achieve our goal, I’ll try to describe them below including notes from William. I suppose we can encounter complications in this path, but hopefully at the end we’ll have a good starting point for the store.
William agreed on the goals of this migration (having a common/separate module which can be reused by both juju-core and the GUI).

Just two notes before sketching a possible plan:
- the store must be able to be configured to use its own db or the juju-core one based on the context it is used;
- we need a way to migrate partial Bazaar commit history to git (perhaps someone has already experience with this?).

A possible plan is as follow:

1) Migrate juju-core/thirdparty to juju/thirdparty (Github).
Since there are no dependencies here, this seems to be a good first candidate. 

2) Migrate juju-core/utils to juju/utils (Github).
package:
 launchpad.net/juju-core/utils
deps:
 launchpad.net/juju-core/juju/osenv
 launchpad.net/juju-core/thirdparty/pbkdf2
tests deps:
 launchpad.net/juju-core/juju/osenv
 launchpad.net/juju-core/testing/testbase

I did not look at juju/osenv: we might want to migrate that as well, or just refactor the code to remove the dependency. Note that each time we move a package we need to also move the relevant code in juju-core/testing to juju/testing. The latter already exists on Github. The juju-core/testing module has lots of dependencies on other packages in juju-core, so if we encounter those we will need to handle that (I presume by refactoring test code). In the utils case, juju-core/testing/testbase does not seem to depend on anything, so we should be ok.

3) Migrate juju-core/schema to juju/utils/schema. The schema package has no juju-core dependencies.

4) Migrate juju-core/charm. This has some pre-requisites. Basically IIUC charm defines config and meta and those definitions are tangled with the underlaying Mongo documents. William suggested to decouple that, implementing separate data structures to be used to (de)serialize data. This way, changes to charm database structure can be detected earlier and core developers are able to react accordingly. Soon this could also involve actions data. 

5) Move the store.

For each step AFAICT what we need to do is similar to the following:
1) possible preliminary work to move the testing stuff;
2) create Github project (if it does not exist);
3) add readme, copying, license files etc;
4) notify developers we are locking the package;
5) migrate the code;
6) fix package imports if required (e.g. for sub-packages);
7) fix package tests;
8) land a juju-core branch which:
   - removes the package;
   - fixes all the imports;
   - includes the new dependency info in the dependencies.tsv file;
9) notify developers about the new external dependency.

Thanks!

--
Francesco