[Maas-devel] Django, transactions, celery and provisioning jobs

[Julian Edwards]

I had a long call with Jeroen today about how to manage the PXE config writing 
Tasks.

There are quite a few problems, but to summarise them:
 1/ Celery tasks are not currently sent only on transaction commit
 2/ Writing PXE configs will eventually happen on a different worker to the 
boot Task so we need to worry about serialization

We think we can solve the first problem by using a package called django-
celery-transactions which installs its own middleware to handle task creation 
on commit.

Regarding the second point, this is a little subtle.  We've decided to change 
the way PXE configs are maintained on disk.  Initially we thought that we'd 
write them on every change that necessitates a new config but this is 
unnecessary and complicates other parts of the webapp (as recently discussed 
on a merge proposal).

So, the plan is to create a job to write PXE configs:
 * just before booting (observing the netboot flag)
 * when deleting a MAC address associated with a node (it's not something that 
can be synced later), we delete the file for that MAC
 * when decommissioning a node, we delete all its PXE config files

What this means is that we now have the problem of serializing two Tasks, 
possibly across different workers: writing configs and then booting.  
Hopefully we can solve this by using Celery's subtasks with callbacks.

If anyone has any questions or concerns, please reply now!

J