[LTP] LTP - Include upstart whitebox / blackbox testing API's?

[Garrett Cooper]

On Wed, Oct 22, 2008 at 5:44 AM, Subrata Modak
<subrata at linux.vnet.ibm.com> wrote:
> Garrett,
>
> Is there any headway with upstart developers regarding this initiative.
> I dug out this mail from my mailbox to find this. Let me know if we can
> resume this discussion once again.
>
> Regards--
> Subrata
>
> On Fri, 2008-06-27 at 19:06 +0530, Subrata Modak wrote:
>> On Thu, 2008-06-26 at 05:26 -0700, Garrett Cooper wrote:
>> > Hello LTP gurus (and upstart gurus),
>> >       As I mentioned before on the upstart-devel list, one of the
>> > goals of the groups that I'm working with is to bring upstart -- the
>> > init replacement -- to Cisco's Linux based platform for process
>> > monitoring and management. As part of that we (my teammates and I)
>> > were thinking of including whitebox and blackbox tests with LTP (Linux
>> > test project) to try and unify testing of critical Linux components,
>> > and also provide deterministic output also with greater visibility in
>> > the testing community.
>> >       LTP has a number of whitebox and blackbox tests in place [3],
>> > most of the whitebox tests being C API's and the blackbox tests being
>> > shell invocations of Unix commands, as well as a well-defined set of
>> > test reporting API's and functions already in place.
>>
>> Ah!. That reminds me of the testcases for commands in LTP:
>>
>> http://ltp.cvs.sourceforge.net/ltp/ltp/testcases/commands/
>>
>> I have been merging lots of patches and we were totally engaged with our
>> white box test cases, that we completely forgot about those black box
>> test cases, which are of immense help for:
>>
>> 1) Increasing code coverage for the kernel,
>> 2) Testing the actual/mostly-used interfaces to the Linux OS.
>>
>> Thanks Garrett for reminding this valuable testcases piece. And the
>> important point here to make is:
>>
>> Writing white box test cases requires fair knowledge of Kernel
>> Internals, whereas the Blackbox test cases just requires user knowledge
>> of the OS. With guidance from the Man Pages information, a huge
>> community of administrators and normal users can write these black box
>> tests. And they are a huge group of people to count. I need to look into
>> this seriously from now.
>>
>> >       So, my question is two-fold:
>> >       1. Would the upstart project be willing to work with LTP (via my
>> > team as a proxy in the beginning) to enter some unit test code and
>> > other test cases into LTP's test framework / overall testsuite, and
>> > improve acceptance in the Linux testing community?
>>
>> I would be providing you the support with testing on the architectures i
>> have at my disposal and speedy patch merge to LTP. We definitely need to
>> do something to increase the code coverage.
>>
>> >       2. Would either group be willing to work with my team to help
>> > maintain these testcases and develop new ones?
>>
>> Of course, i will.
>>
>> > Thanks,
>> > -Garrett
>> >
>> > PS. Sorry for the cross-posting ; I try not to do this, but
>> > considering that both groups can benefit from the discussion I wanted
>> > to involve both.
>>
>> Nothing to worry about. When it comes to making Linux better, we need
>> collaboration on various fronts. The livest example being the work done
>> by Masatake Yamato from Red Hat in porting Crackerjack´s
>> (https://sourceforge.net/projects/crackerjack) regression tests to LTP
>> format. Thanks Garrett for taking this initiative. We need to
>> collaborate much more with others as well.
>>
>> Regards--
>> Subrata
>>
>> >
>> > 1. LTP -- Linux test project: http://ltp.sourceforge.net/
>> > 2. Upstart -- init(1) replacement: http://upstart.ubuntu.com/
>> > 3. LTP cvsweb -- http://ltp.cvs.sourceforge.net/ltp/ltp/ (see docs for
>> > relevant documentation items, lib/ltp for test lib API's, and
>> > testcases/commands for existing Linux command blackbox tests).

I haven't followed this up, but to be honest our group using upstart
has started using Python nose to write testcases for blackbox level
testing, and it's proven to be largely successful in finding basic
issues within the provided spec by the upstart folks.

I don't know if the test code can be easily committed back because it
has Cisco IP -- I'll talk to Sarvi (tech lead) and Corey (the manager)
about that.

As for whitebox testing, we should definitely follow up the intiative
for using tst_res.

-Garrett