charmrunner - automated charm testing tools

[Kapil Thangavelu]

Excerpts from Antonio Rosales's message of Fri Feb 17 16:57:10 -0500 2012:
> On 02/17/2012 01:43 PM, Robbie Williamson wrote:
> > On 02/17/2012 10:24 AM, Kapil Thangavelu wrote:
> >> Hi folks,
> >>
> >> After the testing discussion last week, i put together a python package for 
> >> generating automated test plans, running them, and recording the results. It 
> >> works by constructing a graph of the charm's dependencies and then creating a 
> >> plan consisting of charms and their versions, and the relations to instantiate 
> >> between the deployed services them to satisfy dependencies. 
> > 
> > <blink>...wow.
> > 
> >>
> >> The package is built as a set of standalone tools that can accomplish various 
> >> common practices when testing a charm (snapshot/restore environment, service 
> >> watching, environment loading), with a front end script to drive the 
> >> whole process. The front-end script is meant to be easy to drive from jenkins, 
> >> by hand, or via a REST queue.
> >>
> >> There's a readme overview of the tools and downloads available at.
> >> http://pypi.python.org/pypi/charmrunner 
> >>
> >> i've been able to deploy everything i thrown at it (mediawiki, 
> >> openstack/nova-compute, etc). Semantic deficiencies in relation metadata have 
> >> tripped up the tool, and i doubt its perfect. I expect it will see some 
> >> iteration as it expands beyond minimal dependency deployment to testing what a 
> >> charm provides.
> >>
> >> Code, bugs, etc.
> >> http://launchpad.net/charmrunner
> > 
> > This is pure awesomeness...thanks Kapil!
> 
> +1  good stuff, and I hear you are already working with Mark Mims on
> integrating this into this automated charm testing framework.
> 

yup, the plan is to try and use this as a testrunner in Mark's 
charmtester jenkins setup as a replacement for the current testrunner which just 
tests a charm's install and start hooks. 

There's some use as well of the various utilities to simplify the manual tests 
that have been written. Taking this mediawiki test as an example.

https://bazaar.launchpad.net/~charmers/charms/oneiric/mediawiki/trunk/view/head:/tests/100_deploy.test

Most of the code here for teardown, waiting for services to come up can be 
omitted using the charmrunner utility cli for the same, roughly cutting the 
size of the test by a 1/3. The environment deployment aspect from a json file 
might be of interest, but its mostly equivalent to the cli usage. The utility 
for recording the environment state for analysis post test run failure should 
also be useful for these manual tests.

cheers,

Kapil