[Bug 1124384] Re: reload-configuration can confuse upstart

[Colin Watson]

16:54 <cjwatson> jodh: I *think* I may see the general structure of what's going on here
16:54 <cjwatson> jodh: This debugging should make it about as clear to you as it is to me so far, I think
16:54 <cjwatson> <7>init: conf_load_path_with_override: Loading configuration file /etc/init/rc-sysinit.conf
16:54 <cjwatson> ...
16:54 <cjwatson> <7>init: conf_file_destroy: Destroyed unused job rc-sysinit
16:54 <cjwatson> <7>init: event_unblock: name: 'filesystem', new blockers: 3
16:55 <cjwatson> jodh: When we tear down the old job, we end up unreferencing and destroying the event operators it refers to, at least enough to cause them to decrement various event->blockers
16:56 <cjwatson> jodh: So the 'filesystem' event ends up entering the finished state far too early because it's been wrongly unblocked
17:08 <jodh> cjwatson: gotcha - that is indeed subtle. Could be an interesting one to fix too ;)
17:10 <cjwatson> jodh: My feeling is that the nih_free in conf_file_destroy needs to be something more careful involving keeping references
17:10 <cjwatson> jodh: But that also perhaps job_class_reconsider shouldn't replace a job that has events that are blocking some other event?  I'm not quite sure, I don't know the structures quite well enough
17:10 <cjwatson> jodh: Is that enough to get you going for now though?
17:11 <jodh> cjwatson: sure is - thanks!!

-- 
You received this bug notification because you are a member of Ubuntu
Foundations Bugs, which is subscribed to upstart in Ubuntu.
https://bugs.launchpad.net/bugs/1124384

Title:
  reload-configuration can confuse upstart

Status in “cloud-init” package in Ubuntu:
  Confirmed
Status in “upstart” package in Ubuntu:
  Confirmed

Bug description:
  Under bug 1080841 we made cloud-init invoke 'initctl reload-
  configuration' after it wrote a upstart job.  This was necessary
  because inotify is not supported on all filesystems (overlayfs being
  the one of most current interst).

  This seems to be causing upstart some pain, and resulting in cloud-
  final (and 'rc') not being run.

  Easy user-data to reproduce the problem is:

  #cloud-config-archive
  - content: |
     #cloud-boothook
     #!/bin/sh
     touch /run/cloud-init-upstart-reload  # hack, see trunk commit 783
  - content: |
     #!/bin/sh
     echo "==== $(date -R): user-script run ===" | tee /run/user-script.log
  - content: |
     #upstart-job
     description "a test upstart job"
     start on stopped rc RUNLEVEL=[2345]
     console output
     task
     script
     echo "==== $(date -R): upstart job run ===" | tee /run/upstart-job.log
     end script

  You should (and do on quantal) end up with 2 files written to /run.

  I've verified that the same behavior is true on quantal.  If you
  change cloud-init to notify upstart about a job immediately after it
  writes it, then quantal's upstart gets confused also.

  Related bugs:
   * bug 1080841: should reload configuration if an upstart job is added
   * bug 1103881: cloud-final is never executed if upstart is upgraded during initialization of the image

To manage notifications about this bug go to:
https://bugs.launchpad.net/ubuntu/+source/cloud-init/+bug/1124384/+subscriptions