supervisord seems to go haywire under Upstart

Tue Mar 18 14:46:12 UTC 2014

Hi Dustin,

Sounds like an environment issue. Take a look at:

http://upstart.ubuntu.com/cookbook/#job-environment
http://upstart.ubuntu.com/cookbook/#see-the-environment-a-job-runs-in
http://upstart.ubuntu.com/cookbook/#checking-how-a-service-might-react-when-run-as-a-job
http://upstart.ubuntu.com/cookbook/#determining-why-your-service-fails-to-start

At a guess, I'd suggest that supervisord is expecting one or more
environment variables to be set, but not finding them and behaving
erratically as a result.

If you're running on Ubuntu or Debian, try installing my procenv tool and
diff'ing the logfiles in a "good" and "bad" environments (as shown in the
last link above) to help you track down the root cause.

Kind regards,

James.

2014-03-17 18:21 GMT+00:00 Dustin Oprea <myselfasunder at gmail.com>:

> This is an error having to do with a combination of Upstart and
> supervisor. Upstart works fine on other services, and supervisor works
> great on its own.
>
> When I start supervisor in the foreground, and start the job (without an
> expect value), the operation blocks. When I start the supervisor process as
> a daemon, it'll block with both "expect daemon" and "expect fork". I've
> also tried "expect stop", for good measure.
>
> When I try the instructions under "6.12.4 How to Establish Fork Count" (
> http://upstart.ubuntu.com/cookbook/#expect) while using supervisor in
> daemon mode, I get a massive number of clones up and until the process is
> killed:
>
> $ sudo egrep "\<(fork|clone)\>\(" /tmp/strace.log
> 18916 clone(child_stack=0,
> flags=CLONE_CHILD_CLEARTID|CLONE_CHILD_SETTID|SIGCHLD,
> child_tidptr=0x7ffdae9799d0) = 18917
> 18917 clone(child_stack=0,
> flags=CLONE_CHILD_CLEARTID|CLONE_CHILD_SETTID|SIGCHLD,
> child_tidptr=0x7fa9910d39d0) = 18918
> 18916 clone(child_stack=0,
> flags=CLONE_CHILD_CLEARTID|CLONE_CHILD_SETTID|SIGCHLD,
> child_tidptr=0x7ffdae9799d0) = 18919
> 18919 clone(child_stack=0,
> flags=CLONE_CHILD_CLEARTID|CLONE_CHILD_SETTID|SIGCHLD,
> child_tidptr=0x7ffdae9799d0) = 18920
> 18919 clone( <unfinished ...>
> 18919 clone( <unfinished ...>
> 18919 clone( <unfinished ...>
> 18919 clone( <unfinished ...>
> 18923 clone( <unfinished ...>
> 18925 clone( <unfinished ...>
> 18922 clone( <unfinished ...>
> 18926 clone( <unfinished ...>
> 18927 clone( <unfinished ...>
> 18929 clone( <unfinished ...>
> 18928 clone( <unfinished ...>
> 18930 clone( <unfinished ...>
> 18926 clone( <unfinished ...>
> 18929 clone( <unfinished ...>
> 18926 clone( <unfinished ...>
> 18929 clone( <unfinished ...>
> 18935 clone( <unfinished ...>
> 18936 clone( <unfinished ...>
> 18924 clone(child_stack=0,
> flags=CLONE_CHILD_CLEARTID|CLONE_CHILD_SETTID|SIGCHLD,
> child_tidptr=0x7f3c797889d0) = 18939
> 18923 clone( <unfinished ...>
> 18939 clone( <unfinished ...>
> 18941 clone( <unfinished ...>
> 18922 clone( <unfinished ...>
> 18942 clone( <unfinished ...>
> 18941 clone( <unfinished ...>
> 18941 clone( <unfinished ...>
> 18946 clone( <unfinished ...>
> 18924 clone( <unfinished ...>
> 18923 clone( <unfinished ...>
> 18924 clone( <unfinished ...>
> 18922 clone( <unfinished ...>
> 18919 clone( <unfinished ...>
> 18919 clone( <unfinished ...>
> 18921 clone( <unfinished ...>
> 18954 clone( <unfinished ...>
> 18955 clone( <unfinished ...>
> 18956 clone( <unfinished ...>
> 18955 clone( <unfinished ...>
> 18955 clone( <unfinished ...>
> 18959 clone( <unfinished ...>
> 18953 clone(child_stack=0,
> flags=CLONE_CHILD_CLEARTID|CLONE_CHILD_SETTID|SIGCHLD,
> child_tidptr=0x7f5871a0b9d0) = 18961
> 18961 clone( <unfinished ...>
> 18921 clone( <unfinished ...>
> 18962 clone( <unfinished ...>
> 18964 clone( <unfinished ...>
> 18962 clone( <unfinished ...>
> 18962 clone( <unfinished ...>
> 18967 clone( <unfinished ...>
> 18953 clone( <unfinished ...>
> 18921 clone( <unfinished ...>
> 18919 clone( <unfinished ...>
> 18953 clone( <unfinished ...>
> 18971 clone( <unfinished ...>
> 18973 clone( <unfinished ...>
> 18919 clone( <unfinished ...>
> 18974 clone( <unfinished ...>
> 18976 clone( <unfinished ...>
> 18975 clone(child_stack=0,
> flags=CLONE_CHILD_CLEARTID|CLONE_CHILD_SETTID|SIGCHLD,
> child_tidptr=0x7f537bc059d0) = 18978
> 18974 clone( <unfinished ...>
> 18978 clone( <unfinished ...>
> 18980 clone( <unfinished ...>
> 18974 clone( <unfinished ...>
> 18982 clone(child_stack=0,
> flags=CLONE_CHILD_CLEARTID|CLONE_CHILD_SETTID|SIGCHLD,
> child_tidptr=0x7fdf0db9f9d0) = 18983
> 18981 clone( <unfinished ...>
> 18980 clone( <unfinished ...>
> 18980 clone( <unfinished ...>
> 18971 clone( <unfinished ...>
> 18986 clone( <unfinished ...>
> 18975 clone( <unfinished ...>
> 18971 clone( <unfinished ...>
> 18975 clone( <unfinished ...>
> 18919 clone( <unfinished ...>
> 18992 clone(child_stack=0,
> flags=CLONE_CHILD_CLEARTID|CLONE_CHILD_SETTID|SIGCHLD,
> child_tidptr=0x7f11233fd9d0) = 18993
> 18993 clone( <unfinished ...>
> 18994 clone( <unfinished ...>
> 18995 clone( <unfinished ...>
> 18994 clone( <unfinished ...>
> 18994 clone( <unfinished ...>
> 18998 clone( <unfinished ...>
> 18992 clone( <unfinished ...>
> 18920 clone( <unfinished ...>
> 19001 clone( <unfinished ...>
> 19002 clone( <unfinished ...>
> 19003 clone( <unfinished ...>
> 19002 clone( <unfinished ...>
> 19002 clone( <unfinished ...>
> 19006 clone( <unfinished ...>
> 18992 clone(child_stack=0x7f8e10847ff0,
> flags=CLONE_VM|CLONE_FS|CLONE_FILES|CLONE_SIGHAND|CLONE_THREAD|CLONE_SYSVSEM|CLONE_SETTLS|CLONE_PARENT_SETTID|CLONE_CHILD_CLEARTID,
> parent_tidptr=0x7f8e108489d0, tls=0x7f8e10848700,
> child_tidptr=0x7f8e108489d0) = 19008
> 18920 clone( <unfinished ...>
> 18919 clone( <unfinished ...>
> 18919 clone( <unfinished ...>
> 18919 clone( <unfinished ...>
> 19011 clone( <unfinished ...>
> 19013 clone( <unfinished ...>
> 19014 clone( <unfinished ...>
> 19015 clone(child_stack=0,
> flags=CLONE_CHILD_CLEARTID|CLONE_CHILD_SETTID|SIGCHLD,
> child_tidptr=0x7f9d52dbe9d0) = 19016
> 19014 clone( <unfinished ...>
> 19014 clone( <unfinished ...>
> 19018 clone( <unfinished ...>
> 19011 clone( <unfinished ...>
> 19012 clone( <unfinished ...>
> 19021 clone( <unfinished ...>
> 19022 clone( <unfinished ...>
> 18919 clone( <unfinished ...>
> 19023 clone( <unfinished ...>
> 19022 clone( <unfinished ...>
> 19022 clone( <unfinished ...>
> 19027 clone( <unfinished ...>
> 19024 clone(child_stack=0,
> flags=CLONE_CHILD_CLEARTID|CLONE_CHILD_SETTID|SIGCHLD,
> child_tidptr=0x7f8e219879d0) = 19029
> 19029 clone( <unfinished ...>
> 19030 clone( <unfinished ...>
> 19031 clone( <unfinished ...>
> 19012 clone( <unfinished ...>
> 19030 clone( <unfinished ...>
> 19030 clone( <unfinished ...>
> 19035 clone( <unfinished ...>
> 19024 clone( <unfinished ...>
> 19012 clone( <unfinished ...>
> 18919 clone( <unfinished ...>
> 19024 clone( <unfinished ...>
> 19011 clone( <unfinished ...>
> 19039 clone( <unfinished ...>
> 19042 clone( <unfinished ...>
> 19043 clone( <unfinished ...>
> 19044 clone( <unfinished ...>
> 19043 clone( <unfinished ...>
> 19043 clone( <unfinished ...>
> 19047 clone( <unfinished ...>
> 19039 clone( <unfinished ...>
> 19039 clone( <unfinished ...>
> 18920 clone( <unfinished ...>
> 18919 clone(child_stack=0,
> flags=CLONE_CHILD_CLEARTID|CLONE_CHILD_SETTID|SIGCHLD,
> child_tidptr=0x7ffdae9799d0) = 19052
> 18919 clone( <unfinished ...>
> 19052 clone( <unfinished ...>
> 19054 clone( <unfinished ...>
> 19055 clone( <unfinished ...>
> 19056 clone( <unfinished ...>
> 19055 clone( <unfinished ...>
> 19055 clone( <unfinished ...>
> 19059 clone(child_stack=0,
> flags=CLONE_CHILD_CLEARTID|CLONE_CHILD_SETTID|SIGCHLD,
> child_tidptr=0x7fb5c74439d0) = 19060
> 19052 clone( <unfinished ...>
> 19053 clone( <unfinished ...>
> 19062 clone( <unfinished ...>
> 19063 clone( <unfinished ...>
> 19064 clone( <unfinished ...>
> 19063 clone( <unfinished ...>
> 19063 clone( <unfinished ...>
> 19067 clone( <unfinished ...>
> 19052 clone( <unfinished ...>
> 19053 clone( <unfinished ...>
> 19053 clone( <unfinished ...>
>
>
> Does anyone have any advice for what I'm missing?
>
>
>
> Dustin
>
>
>
> --
> upstart-devel mailing list
> upstart-devel at lists.ubuntu.com
> Modify settings or unsubscribe at:
> https://lists.ubuntu.com/mailman/listinfo/upstart-devel
>
>

-- 
James.
--
James Hunt
____________________________________
Ubuntu Foundations Team, Canonical.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://lists.ubuntu.com/archives/upstart-devel/attachments/20140318/1521682d/attachment-0001.html>