The docs around charm and bundle testing need some love

[Merlijn Sebrechts]

Hi all


The testing docs need some love. The whole Charm testing ecosystem is very
complex. The docs should focus on guiding users to a best-practice
workflow. There is just too much choice to let the user figure it out
himself.

We want charmers to write Charm and Bundle tests that are compatible with

a) `bundletester` + (LXD | a cloud)
b) the review queue

c) the cwr-ci bundle


The docs should provide two best-practice workflows; one for testing charms
and one for testing bundles. Each workflow needs a working example.

The example includes a `tests.yaml` file that is compatible with a,b and c,
and it should include commands to deploy the tests locally. The example
should hide as much advanced config as possible. It's nice that you can
specify a custom glob pattern for the test files, but this is something for
advanced uses.

Some questions that the docs should answer:

- What tools should I use? (Bundletester + Amulet. cwr-ci if you want a ci
pipeline.)
- What's the best way to deploy a Charm/Bundle using amulet?
(`deployment.add` for a Charm and `find_bundle`
<https://github.com/juju-solutions/bundle-canonical-kubernetes/blob/master/tests/20-basic-component-check.py#L17>
for a bundle)
- Which options should I use in tests.yaml?
- Reset the environment or not? Let bundletester do cleanup, or clean it up
using Amulet?
- Deploy bundle by bundletester or by amulet?
- Where can I find Charms/Bundles that have working best-practice tests?

I think we need some discussion around what the best-practice workflows for
testing are, and then we need to standardize everything around these
workflows. An example of a choice that needs to be discussed is whether
bundle tests should first execute the individual Charm tests.

- The default behavior is to do this, so that implies "yes, bundle tests
should first execute charm tests"
- Charms and bundles are promulgated independently, and cwr-ci bundle is
also going this approach. This implies that the Charms are already tested
(either by cwr-ci or the review queue). This implies "no, it's useless to
first execute charm tests because we know they've succeeded".



Kind regards
Merlijn
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://lists.ubuntu.com/archives/juju/attachments/20170117/b731deed/attachment.html>