out of disk space on the bot

[David Cheney]

On Wed, Apr 2, 2014 at 6:04 PM, John Meinel <john at arbash-meinel.com> wrote:
> So I went ahead and did a few steps that will hopefully make this better.
>
> 1) I moved lp:goose into its own directory, we actually had it sharing the
> source dependencies of juju-core so we could know if we broke anything, but
> now we can trust dependencies.tsv so we can handle this a bit better.
>
> 2) I added a "killall mongod", which would help ensure that we don't have
> any mongo's running to block the flock.

That is an important point. If test's panic the mongos will remain
around and will pin their data files open even if /tmp is cleaned.

>
> 2) Changed the "verify_command" to both run godeps -u, as well as create a
> TMPDIR that gets cleaned up. The new logic is complex enough, it really
> feels like it should be a script inside of the source tree that we just run,
> but anyway, here it is:
>       verify_command = export GOPATH=/home/tarmac/trees; godeps -u
> dependencies.tsv && export TMPDIR=`mktemp -d --tmpdir juju-core-test.XXXXXX`
> && go fmt ./... && go install ./... && go test ./...; STATUS=$?; rm -rf
> $TMPDIR; killall -u tarmac mongod; exit $STATUS
>
> I made sure to submit a branch that actually fails, to ensure that the exit
> $STATUS stuff triggers correctly.
>
> This probably won't make tests any less flaky, but at least they'll leave a
> little less cruft behind.
>
> John
> =:->
>
>
> On Tue, Apr 1, 2014 at 1:12 PM, David Cheney <david.cheney at canonical.com>
> wrote:
>>
>> or use tmpwatcher ?
>>
>> On Tue, Apr 1, 2014 at 7:47 PM, Andrew Wilkins
>> <andrew.wilkins at canonical.com> wrote:
>> > Much as I'd like the tests to clean up after themselves, can we just
>> > create
>> > a temporary $TMPDIR and blow it away after the test run?
>> >
>> >
>> > On Tue, Apr 1, 2014 at 4:40 PM, John Meinel <john at arbash-meinel.com>
>> > wrote:
>> >>
>> >> It seems our test suite still leaks /tmp/test-* directories
>> >> occasionally.
>> >> Today I was working on setting up tarmac managing the 1.18 branch and
>> >> noticed it ran out of disk space running the "juju resolved --retry". I
>> >> ended up cleaning out approx 7GB of dead test directories.
>> >>
>> >> Anyway, if you ever have to kick the bot, please at least check 'df' to
>> >> see if we might be low on disk space.
>> >>
>> >> John
>> >> =:->
>> >>
>> >>
>> >> --
>> >> Juju-dev mailing list
>> >> Juju-dev at lists.ubuntu.com
>> >> Modify settings or unsubscribe at:
>> >> https://lists.ubuntu.com/mailman/listinfo/juju-dev
>> >>
>> >
>> >
>> > --
>> > Juju-dev mailing list
>> > Juju-dev at lists.ubuntu.com
>> > Modify settings or unsubscribe at:
>> > https://lists.ubuntu.com/mailman/listinfo/juju-dev
>> >
>
>