Automated Testing for Flavors -- Update

[Nicholas Skaggs]

On 03/09/2016 05:08 PM, flocculant at gmx.co.uk wrote:
> On 09/03/16 21:56, Simon Quigley wrote:
>> Flocculant,
>>
>> First of all, I would like to say thank you for providing input on this.
>>
>> There is no function implemented in the API to update the notice 
>> board (unless it is really REALLY subtle and I can't see the function).
>>
>> I actually think Nicholas' idea of having results on the tracker 
>> would be fine. I think there needs to be a comment on the submitted 
>> test case that says something like "this is automated" would be good.
> I don't agree.
>>   I'm not the release manager for a flavor, but my guess is, you 
>> don't just look at the test case completion and say, "oh, it's failing"
>
> no of course not - we would check what the person failing it has 
> reported as the cause - the bug
>>   and just leave it at that. Plus, if autopilot tests are failing, 
>> that's probably bad and should be addressed anyways.
>
> This is completely beside the point at the moment - the tests have 
> been failing for over a year now - so I see absolutely no rush to get 
> this causing issues now we're only a few weeks away from release.
>
>
>>   And it would be much easier to hack up a script to do this, I just 
>> need to fetch the autopilot results, then submit it. So I'm 
>> supporting Nicholas' original idea.
>>
>> Now, if someone, within the next hour or two (while I'm out eating), 
>> decides to hack together an API function for this and gets an MP out, 
>> I'm all for flocculant's solution, but for now, we don't have that. :)
>>
>> Let me know if you have another idea that uses the API and I will be 
>> glad to do it. :)
>>
>> Thanks,
>> Simon Quigley
>> tsimonq2 at ubuntu.com
>> tsimonq2 on Freenode
> If we are going to randomly add fails to the tracker for no other 
> reason than it seems like a good idea - then I'll have to ask that 
> Xubuntu isn't included - there are enough problems with the tracker 
> results without adding completely pointless fail results to it.
>
> When there is actually a sensible reason to add things to the tracker 
> then you'd find me much more accomodating.
>
Simon, while I do support you hacking on things and doing a proof of 
concept (we can delete the entries you make), I'm not yet ready to turn 
the beast loose just yet. We have to get it running on the same server 
for one, which needs a deployment, an MP to check in the code, etc.

We can explore some different ways to display the information, but 
getting a proof of concept done as entries on the tracker is one data 
point to consider. I will defer to the opinions of the release managers 
for flavors on this, as it's intended to make there lives easier. Even 
if we end up posting the results as just another entry, there is no 
point to spamming the tracker until the tests are fixed. We do know they 
are broken at the moment so there's no value in clogging up the tracker 
with a bunch of failures.

On the notice board updating, I think I prefer that suggestion myself, 
but clearly I'm still easily swayed. So perhaps we need to extend the 
API to support it. But first, the proof of concept.

Nicholas