Preventing IE10 regressions

[Benji York]

On Wed, Jan 30, 2013 at 3:20 PM, Gary Poster <gary.poster at canonical.com> wrote:
> Thanks, Benji.  A couple of comments inline.
>
>
> On 01/30/2013 09:58 AM, Benji York wrote:
>>
>> Goal
>>
>> Keep IE10 fixes from regressing (once we get the GUI running on IE10)
>
>
> FF as well.

Yep.

>> Since we do not regularly develop or run the tests on IE10 (and that is
>> unlikely to materially change) we would benefit from some form of
>> continuous integration (CI) for (at a minimum) IE10.
>
>
> same for FF

Yep again.

>> Plan
>>
>> We will write browser tests that can be run via Sauce Labs
>> (saucelabs.com) on IE10 against the Juju GUI charm.
>>
>> We will have one or more tests written in Python, using the Sauce Labs
>> remote Selenium/Web Driver interface.  Those tests will operate against
>> the Juju GUI charm running in EC2 (initially).  The tests will be
>> triggered a hosted Jenkins instance (i.e., http://www.cloudbees.com/).
>
>
> People will also be able to run this automation on their own developer
> machines, right?  As long as the branch they want to test has been pushed,
> I'd guess.

Right, it will be easy to pass a non-trunk branch through to the charm
which will let us run the tests against that branch instead.

> Would you deliver the developer tool first, and then the Jenkins integration
> separately?

I am inclined to do so.

> Do you plan for Jenkins to trigger tests on trunk landing, or daily, do you
> think?  If it's just as easy, I would prefer trunk landing.

According to Diogo Matsubara setting up tarmac to be a gatekeeper is
easy, so that is another option as well.  Does anyone object to using
tarmac to run these tests and ensure they pass before a branch can land?
It would probably mean not using lbox submit any more (but we can still
use lbox propose as before).

> Our Jenkins would need to be persistent, and I bet that costs money on
> cloudbees.  IIUC, we would also need to install juju and set up a
> ~/.juju/environments.yaml file to get that to work, which might not work
> well with their service.

Quite possibly.  This appears to be moot though as Diogo is working with
us to get a Jenkins set up in the Canonical data center.

> Another alternative that might be even easier, and would hopefully address
> my cost and configuration concerns above, is to use cloud engineering's
> in-house jenkins instance.  Diogo Matsubara can get us hooked into that, and
> we have a much higher chance of having our way with it (maybe even using lxc
> eventually?).

Yep, I'm working with him on this.

>> EC2: Getting the charm running in EC2 is a known quantity for most of
>> the developers on the team.  Using Canonistack and/or LXC should be a
>> short step from there.
>
>
> And this would simply be controlled by the dev's default juju environment,
> I'm guessing?

I think the best approach is to use the default unless an environment is
otherwise specified.  We don't want to claim the default as our sole
property.  That only works for one claimant.  An alternative would be to
require an environment with a known name, since environments are cheap
to define that might be the best route.

[...]
> (you have a prototype of some of this, I think you said, yeah?)

Yep, I have a unittest.TestCase base class that does all the setup and
reporting of success/failure back to saucelabs.  It needs a little more
work, but it is a good start.
-- 
Benji York