[Merge] lp:~canonical-platform-qa/webbrowser-app/autopkgtest into lp:webbrowser-app

Tue Apr 21 16:00:11 UTC 2015

Thanks for the review Brendan. Reply inline. Please let me know what do you think.

Diff comments:

> === modified file 'README'
> --- README	2015-04-01 05:57:42 +0000
> +++ README	2015-04-20 20:28:58 +0000
> @@ -40,11 +40,11 @@
>  
>      $ ctest
>  
> -
>  = Automated UI tests =
>  
>  webbrowser-app uses autopilot (https://launchpad.net/autopilot) to test its UI.
> -To run the tests, you will need to install python3-autopilot and autopilot-qt5.
> +To run the tests locally, you will need to install python3-autopilot and
> +autopilot-qt5.
>  Then do the following:
>  
>      $ cd tests/autopilot/
> @@ -54,6 +54,8 @@
>  
>      $ autopilot3 list webbrowser_app
>  
> +In order to run the tests in a virtual machine with an environment closer to
> +what a user will get in Ubuntu Touch, see the Dep8 tests section.
>  
>  = Code coverage =
>  
> @@ -68,6 +70,50 @@
>  This will generate a coverage report in XML format (coverage.xml) and an
>  interactive human-readable report in HTML format (coveragereport/index.html).
>  
> += Dep8 tests =

That makes a lot of sense for when we have a single way to run all the dep8 tests.
But so far we have three packages with tests, and the three have small variations on the way we run them:
pay-ui provisions a click package
amazon-webapp provisions a click package and requires a ppa.
webbrowser provisions a deb, and still can't be run on a readonly phone.

I think we need to work to get a common way to define a test execution that takes care of all the difference we can't get rid of. For now, we are just finding those differences.

> +
> +Dep8 tests excercise the package "as-installed".
> +
> +Currently, the webbrowser-app has one suite of dep8 tests that uses autopilot
> +(https://launchpad.net/autopilot) to test from the point of view of the user.
> +
> +To run the tests you will need autopkgtest:
> +
> +    $ sudo apt-get install autopkgtest
> +
> +You can use multiple test beds to execute the tests. Below you will find
> +instructions to run them in a virtual machine
> +You can find more information with:
> +
> +    $ man adt-run
> +
> +== Run dep8 tests in a qemu virtual machine ==
> +
> +To run the tests in a qemu virtual machine, first you will have to create it. We output the image to ~/ rather than the current directory, so it will be in
> +a safer place to avoid rebuilding images every time. You can store it in any
> +directory you wish.
> +
> +    $ adt-buildvm-ubuntu-cloud -r vivid -a amd64 -o ~/
> +
> +Then run the tests using adt-run with the qemu virtualization host.
> +
> +    $ adt-run -B --unbuilt-tree . \
> +      -o /tmp/adt-browser-test \
> +      -U --apt-pocket proposed \
> +      --setup-commands ubuntu-touch-session \
> +      --- qemu ~/adt-vivid-amd64-cloud.img
> +
> +== Examine the dep8 autopilot results ==
> +
> +To examine the test results, which are in subunit format, additional tools are
> +required.
> +
> +    $ sudo add-apt-repository ppa:thomir/trv
> +    $ sudo apt-get update
> +    $ sudo apt-get install trv
> +    $ trv
> +
> +You can find the results file in the directory /tmp/adt-browser-test/artifacts.
>  
>  = Settings =
>  
> @@ -93,4 +139,3 @@
>  
>   - restoreSession: whether to restore the previous browsing session at startup
>     (defaults to true)
> -
> 
> === modified file 'debian/control'
> --- debian/control	2015-04-10 13:33:10 +0000
> +++ debian/control	2015-04-20 20:28:58 +0000
> @@ -26,6 +26,7 @@
>  # If you aren't a member of ~phablet-team but need to upload packaging changes,
>  # just go ahead.  ~phablet-team will notice and sync up the code again.
>  Vcs-Bzr: https://code.launchpad.net/~phablet-team/webbrowser-app/trunk
> +XS-Testsuite: autopkgtest
>  
>  Package: webbrowser-app
>  Architecture: any
> 
> === added directory 'debian/tests'
> === added file 'debian/tests/autopilot'
> --- debian/tests/autopilot	1970-01-01 00:00:00 +0000
> +++ debian/tests/autopilot	2015-04-20 20:28:58 +0000
> @@ -0,0 +1,19 @@
> +#!/bin/sh
> +
> +# Copyright 2015 Canonical
> +#
> +# This program is free software: you can redistribute it and/or modify it
> +# under the terms of the GNU General Public License version 3, as published
> +# by the Free Software Foundation.
> +#
> +# This program is distributed in the hope that it will be useful,
> +# but WITHOUT ANY WARRANTY; without even the implied warranty of
> +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
> +# GNU General Public License for more details.
> +#
> +# You should have received a copy of the GNU General Public License
> +# along with this program. If not, see <http://www.gnu.org/licenses/>.
> +
> +set -e
> +
> +autopilot3 run -v -f subunit -o $ADT_ARTIFACTS webbrowser_app
> 
> === added file 'debian/tests/control'
> --- debian/tests/control	1970-01-01 00:00:00 +0000
> +++ debian/tests/control	2015-04-20 20:28:58 +0000
> @@ -0,0 +1,17 @@
> +# Copyright 2015 Canonical
> +#
> +# This program is free software: you can redistribute it and/or modify it
> +# under the terms of the GNU General Public License version 3, as published
> +# by the Free Software Foundation.
> +#
> +# This program is distributed in the hope that it will be useful,
> +# but WITHOUT ANY WARRANTY; without even the implied warranty of
> +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
> +# GNU General Public License for more details.
> +#
> +# You should have received a copy of the GNU General Public License
> +# along with this program. If not, see <http://www.gnu.org/licenses/>.
> +
> +Tests: autopilot
> +Restrictions: allow-stderr
> +Depends: webbrowser-app-autopilot
> 

-- 
https://code.launchpad.net/~canonical-platform-qa/webbrowser-app/autopkgtest/+merge/256858
Your team Ubuntu Phablet Team is requested to review the proposed merge of lp:~canonical-platform-qa/webbrowser-app/autopkgtest into lp:webbrowser-app.