problem when building a charm for giraph

Thu Nov 24 07:12:58 UTC 2016

As it turned out this error was due to a version misalignment between the
juju-deployer and amulet. Its been fixed now. Please, let us know if you
face any more obstacles.

Thanks,
Konstantinos

On Wed, Nov 23, 2016 at 4:00 PM, Panagiotis Liakos <p.liakos at di.uoa.gr>
wrote:

> Thanks a lot for the clarifications Konstantinos!
>
> I opted to use the mahout interface and managed to overcome the previous
> issue.
>
> Then I bumped into this bug:
> https://bugs.launchpad.net/juju-deployer/+bug/1575863
>
> I followed the suggested workaround (mkdir ~/.juju) and now I get an
> "error getting env api endpoints". Any ideas?
>
> Panagiotis
>
> 2016-11-23 15:54:36 Starting deployment of lxd:admin/default
> 2016-11-23 15:54:36 Error getting env api endpoints, env bootstrapped?
> 2016-11-23 15:54:36 Command (juju api-endpoints -e lxd:admin/default)
> Output:
>
>
> 2016-11-23 15:54:36 Deployment stopped. run time: 0.15
> E
> ======================================================================
> ERROR: setUpClass (__main__.TestDeploy)
> ----------------------------------------------------------------------
> Traceback (most recent call last):
>   File "./tests/01-giraph-test.py", line 44, in setUpClass
>     cls.d.setup(timeout=3600)
>   File "/usr/lib/python3/dist-packages/amulet/deployer.py", line 704, in
> setup
>     subprocess.check_call(shlex.split(cmd))
>   File "/usr/lib/python3.5/subprocess.py", line 581, in check_call
>     raise CalledProcessError(retcode, cmd)
> subprocess.CalledProcessError: Command '['juju-deployer', '-W', '-c',
> '/tmp/amulet-juju-deployer-o1zyf3xd/deployer-schema.json', '-e',
> 'lxd:admin/default', '-t', '3700', 'lxd:admin/default']' returned
> non-zero exit status 1
>
> ----------------------------------------------------------------------
> Ran 0 tests in 4.738s
>
> FAILED (errors=1)
>
>
>
> 2016-11-22 16:31 GMT+02:00 Konstantinos Tsakalozos
> <kos.tsakalozos at canonical.com>:
> > Cool! The error that you are getting now says that the giraph charm has
> no
> > giraph relation [0].
> >
> > Looking at the metadata.yaml, you are using the mahout interface to
> relate
> > to the hadoop client. So the relation call should look like:
> > cls.d.relate('giraph:mahout', 'client:mahout') . This will work for now
> > since both Mahout and Giraph just want to add jars on the hadoop-client
> > charm. However, on the long run we should refactor charms such as
> > hadoop-client, spark, pig, etc, so that they have an interface exactly
> for
> > that purpose (adding jars to the file system). For now you have two
> options
> > either keep using the mahout interface or use the "juju-info" [1]
> interface
> > that is present in all charms. Note, however, that if you do use
> "juju-info"
> > your charm would be a subordinate to any other charm even if that does
> not
> > make sense.
> >
> > [0]
> > http://pythonhosted.org/amulet/amulet.html?highlight=
> relate#amulet.deployer.Deployment.relate
> > [1] https://jujucharms.com/docs/2.0/authors-implicit-relations
> >
> > On Tue, Nov 22, 2016 at 4:02 PM, Panagiotis Liakos <p.liakos at di.uoa.gr>
> > wrote:
> >>
> >> Thanks Konstantinos, you were right. Now the script progresses a little
> >> further:
> >>
> >> $ python3 ./tests/01-giraph-test.py
> >> E
> >> ======================================================================
> >> ERROR: setUpClass (__main__.TestDeploy)
> >> ----------------------------------------------------------------------
> >> Traceback (most recent call last):
> >>   File "./tests/01-giraph-test.py", line 42, in setUpClass
> >>     cls.d.relate('giraph:giraph', 'client:mahout')
> >>   File "/usr/lib/python3/dist-packages/amulet/deployer.py", line 431,
> in
> >> relate
> >>     raise ValueError('%s does not exist for %s' % (rel, srv))
> >> ValueError: giraph does not exist for giraph
> >>
> >> ----------------------------------------------------------------------
> >> Ran 0 tests in 2.285s
> >>
> >> FAILED (errors=1)
> >>
> >>
> >> Is this a namespace issue? Any ideas?
> >>
> >> Panagiotis
> >>
> >> 2016-11-22 14:16 GMT+02:00 Konstantinos Tsakalozos
> >> <kos.tsakalozos at canonical.com>:
> >> > Hi Panagiotis,
> >> >
> >> > The 404 error seems to be because the Giraph charm is not yet
> >> > promulgated
> >> > (your charm currently lives in cs:~panagiotisl/giraph). Could you try
> >> > the
> >> > same test only this time with a charm revision you have pushed under
> >> > your
> >> > namespace in line
> >> >
> >> > https://github.com/panagiotisl/bigtop/blob/
> master/bigtop-packages/src/charm/giraph/layer-giraph/
> tests/01-giraph-test.py#L29
> >> > ? You can also use a local path pointing to where your charm build
> >> > output
> >> > is.
> >> >
> >> > Glad the multiple SLF4J bindings are not causing any problem.
> >> >
> >> > Thanks,
> >> > Konstantinos
> >> >
> >> > On Tue, Nov 22, 2016 at 1:15 PM, Panagiotis Liakos <
> p.liakos at di.uoa.gr>
> >> > wrote:
> >> >>
> >> >> Dear Merlijn and Konstantinos,
> >> >>
> >> >> Thanks a lot for the very valuable information. I will have another
> >> >> look at the documentation for writing tests and debugging and I'll
> try
> >> >> to add some logs in my test.
> >> >>
> >> >> When I execute my smoke-test (through ssh to giraph) I get the
> >> >> following (which is exactly what I expect):
> >> >> https://pastebin.ubuntu.com/23516317/
> >> >>
> >> >> When I execute the 01-giraph-test.py (and after I installed amulet) I
> >> >> get the following: https://pastebin.ubuntu.com/23516415/
> >> >>
> >> >> The URL mentioned in the HTTPError exception is valid for mahout but
> >> >> not for giraph so I guess I am requesting something that is not there
> >> >> (yet).
> >> >>
> >> >> Thanks again, I'll keep you updated.
> >> >>
> >> >> Panagiotis
> >> >>
> >> >>
> >> >> 2016-11-22 12:08 GMT+02:00 Konstantinos Tsakalozos
> >> >> <kos.tsakalozos at canonical.com>:
> >> >> > Hi Panagiotis,
> >> >> >
> >> >> > Merlijn is right, the output of the smoke-test script is available
> in
> >> >> > the
> >> >> > juju debug logs (juju debug-log). Here is what I got when running
> >> >> > your
> >> >> > smoke-test: http://pastebin.ubuntu.com/23516167/ (multiple SLF4J
> >> >> > bindings)
> >> >> > It seems the slf4j-log4j12-1.7.5.jar jar from Giraph is slightly
> >> >> > older
> >> >> > that
> >> >> > the slf4j-log4j12-1.7.10.jar from Hadoop.
> >> >> >
> >> >> > In order to add some output to the smoke-test action you could call
> >> >> > "action
> >> >> > set" and/or "action fail" as we do here
> >> >> >
> >> >> >
> >> >> > https://github.com/panagiotisl/bigtop/blob/
> master/bigtop-packages/src/charm/hbase/layer-hbase/actions/smoke-test
> >> >> >
> >> >> > Thank you for your work,
> >> >> > Konstantinos
> >> >> >
> >> >> >
> >> >> > On Tue, Nov 22, 2016 at 11:06 AM, Merlijn Sebrechts
> >> >> > <merlijn.sebrechts at gmail.com> wrote:
> >> >> >>
> >> >> >> Hi Panagiotis Liakos
> >> >> >>
> >> >> >>
> >> >> >> Cool that you're charming Giraph! Feel free to let me know when
> you
> >> >> >> have a
> >> >> >> working prototype, some of my colleagues are interested in using
> >> >> >> Giraph.
> >> >> >>
> >> >> >> Now for your question; tests and actions are two different things.
> >> >> >>
> >> >> >> Actions are similar to hooks; they run on the charm unit itself.
> The
> >> >> >> mahout smoke-test is an action. You run them using `run-action`,
> you
> >> >> >> can
> >> >> >> debug them using `juju debug-hooks` and all output gets logged by
> >> >> >> juju
> >> >> >> (you
> >> >> >> can see the output in `juju debug-log`). More info on actions:
> >> >> >> https://jujucharms.com/docs/stable/actions
> >> >> >> Tests are scripts that run on your own machine that deploy and
> test
> >> >> >> charms. You run them using bundletester. More info on tests:
> >> >> >> https://jujucharms.com/docs/2.0/developer-testing
> >> >> >>
> >> >> >> So if you want to alter the `smoke-test` action; you should alter
> >> >> >> this
> >> >> >> file:
> >> >> >>
> >> >> >>
> >> >> >> https://github.com/apache/bigtop/blob/master/bigtop-
> packages/src/charm/mahout/layer-mahout/actions/smoke-test
> >> >> >>
> >> >> >> The actual tests of the Mahout Charm use the smoke-test action to
> >> >> >> verify
> >> >> >> the deployment. That is what you see at line 52:
> >> >> >>
> >> >> >>
> >> >> >> https://github.com/apache/bigtop/blob/master/bigtop-
> packages/src/charm/mahout/layer-mahout/tests/01-mahout-test.py#L52
> >> >> >> `self.mahout.run_action('smoke-test')` calls `juju run-action` in
> >> >> >> the
> >> >> >> background.
> >> >> >>
> >> >> >> Does that help you a bit?
> >> >> >>
> >> >> >>
> >> >> >>
> >> >> >> Kind regards
> >> >> >> Merlijn
> >> >> >>
> >> >> >> 2016-11-22 9:42 GMT+01:00 Panagiotis Liakos <p.liakos at di.uoa.gr>:
> >> >> >>>
> >> >> >>> Hi all,
> >> >> >>>
> >> >> >>> I am trying to build a charm for Apache Giraph. I have used the
> >> >> >>> mahout
> >> >> >>> charm (
> >> >> >>>
> >> >> >>>
> >> >> >>> https://github.com/apache/bigtop/tree/master/bigtop-
> packages/src/charm/mahout
> >> >> >>> ) and made the necessary changes to deploy giraph instead of
> >> >> >>> mahout.
> >> >> >>>
> >> >> >>> Deployment seems to work fine and I am able to submit giraph jobs
> >> >> >>> through an ssh connection.
> >> >> >>>
> >> >> >>> Now I am trying to create a test similar to the smoke-test of the
> >> >> >>> mahout charm (
> >> >> >>>
> >> >> >>>
> >> >> >>> https://github.com/apache/bigtop/blob/master/bigtop-
> packages/src/charm/mahout/layer-mahout/actions/smoke-test
> >> >> >>> )
> >> >> >>>
> >> >> >>> I have successfully run the script that I have included in my
> >> >> >>> smoke-test through ssh and I
> >> >> >>> would expect that the test would also execute without errors.
> >> >> >>> However, I am not at all familiar with the 'run-action' process
> of
> >> >> >>> juju and I have no idea how I should alter the following file:
> >> >> >>>
> >> >> >>>
> >> >> >>>
> >> >> >>> https://github.com/apache/bigtop/blob/master/bigtop-
> packages/src/charm/mahout/layer-mahout/tests/01-mahout-test.py
> >> >> >>>
> >> >> >>> In particular, I am puzzled by the way the result status is set
> in
> >> >> >>> this example. Simply changing the references of mahout to giraph
> >> >> >>> does
> >> >> >>> not seem to work. You can see my file here:
> >> >> >>>
> >> >> >>>
> >> >> >>>
> >> >> >>> https://github.com/panagiotisl/bigtop/blob/
> master/bigtop-packages/src/charm/giraph/layer-giraph/
> tests/01-giraph-test.py
> >> >> >>>
> >> >> >>> When I execute show-action-output I receive very limited
> >> >> >>> information:
> >> >> >>> message: exit status 1
> >> >> >>> status: failed
> >> >> >>> timing:
> >> >> >>>   completed: 2016-11-21 15:48:01 +0000 UTC
> >> >> >>>   enqueued: 2016-11-21 15:47:38 +0000 UTC
> >> >> >>>   started: 2016-11-21 15:47:40 +0000 UTC
> >> >> >>>
> >> >> >>> Is there a way I can view the full output of the 'smoke-test'
> >> >> >>> execution?
> >> >> >>>
> >> >> >>> Thank you,
> >> >> >>> Panagiotis Liakos
> >> >> >>>
> >> >> >>> --
> >> >> >>> Juju mailing list
> >> >> >>> Juju at lists.ubuntu.com
> >> >> >>> Modify settings or unsubscribe at:
> >> >> >>> https://lists.ubuntu.com/mailman/listinfo/juju
> >> >> >>
> >> >> >>
> >> >> >>
> >> >> >> --
> >> >> >> Juju mailing list
> >> >> >> Juju at lists.ubuntu.com
> >> >> >> Modify settings or unsubscribe at:
> >> >> >> https://lists.ubuntu.com/mailman/listinfo/juju
> >> >> >>
> >> >> >
> >> >
> >> >
> >
> >
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://lists.ubuntu.com/archives/juju/attachments/20161124/da4f68d1/attachment.html>