Controllers running out of disk space

[Jacek Nykis]

On 17/11/16 08:20, Uros Jovanovic wrote:
> Hi all,
> 
> I'd like to start a discussion on how to handle storage issues on
> controller machines, especially what we can do when storage is getting 95%
> or even 98% full. There are many processes that are storing data, we have
> at least:
> - charms and resources in the blobstore
> - logs in mongo
> - logs on disk
> - audit.log
> 
> Even if we put all models in the controller into read-only mode to prevent
> upload of new charms or resources to blobstore, we still have to deal with
> logs which can fill the space quickly.
> 
> While discussing about this with Rick, given the work on model migrations,
> maybe the path forward would be to allow admin to put the whole controller
> in "maintenance" mode and put all agents on "hold".
> 
> How to deal with storage issue after that is open to discussion, maybe we
> need tools to "clear" blobstore, or export/compress/truncate logs, etc.
> 
> Thoughts?

We have this bug for mongodb disk space:
https://bugs.launchpad.net/juju/+bug/1492237

There is a workaround for juju 1.25 but I don't know if it will work for
juju 2:
https://bugs.launchpad.net/juju/+bug/1492237/comments/6

Some time ago I also filed this one for fix log handling in juju:
https://bugs.launchpad.net/juju/+bug/1494661

Feel free to me too!

Jacek

-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 299 bytes
Desc: OpenPGP digital signature
URL: <https://lists.ubuntu.com/archives/juju/attachments/20161117/f31116b5/attachment.pgp>