[ubuntu-studio-devel] Ferature Spec Discussion: Testing

[lukefromdc at hushmail.com]

Are package uploaders properly testing their own packages? When I 
wrote a  one passphrase/multi volume cryptsetup interface simply
to use it myself in systemd and dracut, I had to set up a dummy partition
with a keyfile so I could test that option, as otherwise I could not 
write it into the program and know if it worked or not.

I assume package devs are testing every option they include, or are they 
writing code they hope will work and packing it up into .debs untested?

On 5/21/2014 at 1:07 PM, "Elfy" <ub.untu at btinternet.com> wrote:
>
>Some comments in line ...
>
>It's likely to get a bit long, sorry about that ;)
>
>On 19/05/14 10:32, Kaj Ailomaa wrote:
>> If anyone is interested in helping out with writing and 
>performing tests
>> during this cycle, please answer this mail (and do read on).
>This is the most important bit here to be honest, if there are 
>only a 
>/few/ people that would be willing to run package tests then 
>anything 
>else is rather, struggling to find a word here that isn't 
>*pointless*
>
>When we (and for anyone reading this for the purposes of this mail 
>- 
>*we* is Xubuntu QA) started to write our testcases, there wasn't a 
>huge 
>crowd of people doing that - it took us a cycle to get the 
>testcases 
>written for us. We were then in a position to use those properly 
>during 
>the LTS cycle - and it went really well for us.
>
>Now, our applications are less complicated than many of yours. 
>Consequently, I'm not going to be able to do much in the way of 
>helping 
>to write testcases for you - what I could do - is start setting up 
>the 
>barebones of testcases for you, which someone with more experience 
>of an 
>application can flesh out.
>
>They aren't complicated to write - it just gets time consuming and 
>rather repetitive - certainly not a very glamorous job - but it is 
>one 
>that pays dividends in the end.
>> ----
>>
>> We hardly do any testing at all during our cycle, currently. 
>This needs
>> to be changed.
>>
>> Naturally, we do required tests for our releases, the Beta 
>releases and
>> the final release, but other than that, there's no structured 
>testing.
>>
>> There are two kinds of testing that we would like to do:
>>   * Quality Assurance Testing - to make sure there are no bugs 
>for a wide
>>   range of applications
>>   * performance testing (which is rather a big topic)
>>
>> The most urgent type of testing we need to deal with is the 
>first of
>> those.
>>
>> (So far, what we have in testing documentation can be found here
>> https://wiki.ubuntu.com/UbuntuStudio/TestingDocumentation)
>>
>> # QA testing
>>
>> I suggest we establish a plan for testing, write test cases, and 
>such,
>> until Debian Import Freeze 
>(https://wiki.ubuntu.com/DebianImportFreeze),
>> which is scheduled to happen Aug 7th this cycle.
>> Debian Import Freeze is a great time to do testing on Debian 
>imported
>> packages, since those packages won't be changing before release. 
>It also
>> gives us some time to find bugs, report them and fix them 
>(Testing can
>> of course be done from day one of our development cycle. The 
>more time
>> we have to spot bugs and fix them, the better, but we should 
>begin no
>> later than Debian Import Freeze).
>>
>> So:
>>   * Test writing may starts any time
>>   * Testing of applications should begin no later than at Debian 
>Import
>>   Freeze, Aug 7th
>I have a suggestion here, why not pick a handful of applications, 
>get 
>them landed in the manual testcase branch - then we can set up the 
>tracker and people can start testing.
>
>Doing this - people get practice at writing them, people can start 
>testing as soon as the tracker is up, you start to get results 
>sooner - 
>I would think it better to get reported bugs slowly to start with 
>than 
>to suddenly have 20 or 30 tests - all being run, all producing 
>results 
>at the same time.
>
>> Elfy has offered to give us a hand on this. If he likes, he 
>could take
>> the role of QA lead for Ubuntu Studio during the next cycle, and 
>mentor
>> us into set up testing. What do you think elfy?
>I am more than happy to help you with this goal, there are 
>probably some 
>infrastructure issues with the trackers that need to be sorted out 
>Launchpad wise, if you want me to do that I can talk to Nick 
>Skaggs 
>about what needs to be done.
>
>Let me know if you want me to do that please.
>
>As I alluded to earlier - 'we' took longer than a cycle - so I'm 
>happy 
>to help you all while you need the help, if that's longer than a 
>cycle - 
>so be it.
>
>
>>
>> The people who write the tests should know the applications they 
>write
>> the tests for. The test should be as simple as possible, but 
>still
>> designed to spot as many typical problems as possible for that
>> application.
>If anyone wants a look at how testcases are written for the 
>majority of 
>cases, then
>
>bzr branch lp:ubuntu-manual-tests
>
>and have a look in /testcases/packages/
>
>So, those are my thoughts at the moment - feel free to ask me 
>questions 
>about how we have worked our system.
>
>I tend to be about early morning for a while (06:00UTC ish) and 
>later in 
>the day 17:00UTC onward for 5 or so hours.
>
>My IRC nick is elfy - I've also dropped forestpiskie into your -
>devel 
>channel, so if elfy is missing you can ping forestpiskie and I'll 
>read 
>it in the morning.
>
>Obviously I am also on this list and will answer queries etc as 
>soon as 
>I can
>
>Hope that helps you,
>
>Elfy
>
>-- 
>Ubuntu Forum Council Member
>Xubuntu QA Lead