Preventing IE10 regressions

[Gary Poster]

Thanks, Benji.  A couple of comments inline.

On 01/30/2013 09:58 AM, Benji York wrote:
> Goal
>
> Keep IE10 fixes from regressing (once we get the GUI running on IE10)

FF as well.

>
> Since we do not regularly develop or run the tests on IE10 (and that is
> unlikely to materially change) we would benefit from some form of
> continuous integration (CI) for (at a minimum) IE10.

same for FF

>
>
> Plan
>
> We will write browser tests that can be run via Sauce Labs
> (saucelabs.com) on IE10 against the Juju GUI charm.
>
> We will have one or more tests written in Python, using the Sauce Labs
> remote Selenium/Web Driver interface.  Those tests will operate against
> the Juju GUI charm running in EC2 (initially).  The tests will be
> triggered a hosted Jenkins instance (i.e., http://www.cloudbees.com/).

People will also be able to run this automation on their own developer 
machines, right?  As long as the branch they want to test has been 
pushed, I'd guess.

Would you deliver the developer tool first, and then the Jenkins 
integration separately?

Do you plan for Jenkins to trigger tests on trunk landing, or daily, do 
you think?  If it's just as easy, I would prefer trunk landing.

Our Jenkins would need to be persistent, and I bet that costs money on 
cloudbees.  IIUC, we would also need to install juju and set up a 
~/.juju/environments.yaml file to get that to work, which might not work 
well with their service.

Another alternative that might be even easier, and would hopefully 
address my cost and configuration concerns above, is to use cloud 
engineering's in-house jenkins instance.  Diogo Matsubara can get us 
hooked into that, and we have a much higher chance of having our way 
with it (maybe even using lxc eventually?).

>
>
> Rational
>
> Sauce Labs: They provide an easy to use system that is free to Open
> Source projects.  Since they use standard Selenium/Web Driver libraries
> we can transition to a different back end if the need presents itself.
>
> GUI Charm: Since this is the primary form in which our users will be
> consuming the software, it makes sense to be the form in which we test
> it (at least in part).  Additionally, deploying the charm is likely to
> be the easiest route to getting the software running in an automated
> fashion (and if it isn't we need to reconsider our charm design).

It also gives us easy access to improv and provides a pristine state for 
each test run.

>
> EC2: Getting the charm running in EC2 is a known quantity for most of
> the developers on the team.  Using Canonistack and/or LXC should be a
> short step from there.

And this would simply be controlled by the dev's default juju 
environment, I'm guessing?

>
> Jenkins: It is the dominant CI software, free hosting is available,

oh, free.  Cool.  My configuration concern still stands though, and 
Diogo is great to work with IME.

> and
> we can easily transition from a hosted environment to running it
> in-house if needed.
>
> Python tests: There is considerable infrastructure available to Python
> for us to leverage in this space.
>

Sounds great and sensible to me.  Thank you for investigating (you have 
a prototype of some of this, I think you said, yeah?) and for writing 
this up.

Gary