millions of warnings per day, state DB grows by 4GB/day

[Peter Grandi]

> [ ... lots of notification logging of notification timeouts,
> Juju state DB growing large ... ]

> Juju 1.23 has some known issues with Lease operations. All of
> those have been fixed in the 1.24.4 (5?) release. It is good
> practice to take a backup before upgrading,

Ahh interesting thanks. I was considering a full database dump
and restore anyhow, as that seems to help with various MongoDB
issues I have seen other people have.

> [ ... ] sort out the specific growth to make sure it gets
> cleaned out after the upgrade. (Even just getting mongo stats
> on the collections as a whole is likely to help us understand
> where the size is coming from.)

Yes, thanks, I'll get them, what was really worrying me was the
thousands of logged events per second. The system seems to be
working well, but it has these very strange amounts of load.
Because the system is pretty static: we may create or delete a
VM instance perhaps once or twice a week. I would expect thus
the occasional state refresh traffic among the members.

> As an aside how could we have messaged you better to upgrade
> away from 1.23?

The system was setup in a small research oriented organization,
and then lightly looked at as it seemed to be "just working",
and OpenStack on top of it instead has had issues that needed
looking at, so I guess even better messaging might not have been
noticed. It is only during OpenStack issue investigations that a
smart colleague noticed strange resource usage levels in Juju.
And then I got worried by the flood of warnings.

> We never made it an official release in Trusty so clearly you
> were informed about our PPA.

That PPA gets mentioned on the mailing list and IRC channel
occasionally, plus some "unofficial" pages mention it too, but
I'll ask around. I am guilty myself of looking sometimes for a
PPA for "cool stuff" even if that's not exactly the spirit of
ULTS, that I have at home too :-).

> So there was some sort of communication between Canonical and
> your group, but I don't think our team was aware that you were
> running 1.23.

I don't know, I am looking at it as a new thing, I'll ask my
colleagues who set up the system.