problem when building a charm for giraph

Thu Nov 24 11:58:55 UTC 2016

Thanks again Konstantinos for all your help.

I made some changes and am now able to pass the 01-giraph-test.py:
https://pastebin.ubuntu.com/23526882/

I have some more questions though:

1. I am using two environment variables ($HADOOP_CLASSPATH and
$LIBJARS) that I set in my smoke-test. However, I believe that it
would be more appropriate if these variables were set in .bashrc as
they are necessary for submitting giraph jobs to hadoop. Is this
possible?
2. I am quite sure that the first time I deployed the giraph charm
that I am building, there was more content in the
/usr/share/doc/giraph/ directory. As the installation of giraph is
pretty much automated through bigtop (if I am not missing something)
my question is where should I look to find out what gets installed and
where. Any ideas?
3. Is the documentation on the charm proof tool up-to-date? (
https://jujucharms.com/docs/2.0/authors-charm-writing#run-the-charm-proof-tool
)

Thank you,
Panagiotis

2016-11-24 9:12 GMT+02:00 Konstantinos Tsakalozos
<kos.tsakalozos at canonical.com>:
> As it turned out this error was due to a version misalignment between the
> juju-deployer and amulet. Its been fixed now. Please, let us know if you
> face any more obstacles.
>
> Thanks,
> Konstantinos
>
> On Wed, Nov 23, 2016 at 4:00 PM, Panagiotis Liakos <p.liakos at di.uoa.gr>
> wrote:
>>
>> Thanks a lot for the clarifications Konstantinos!
>>
>> I opted to use the mahout interface and managed to overcome the previous
>> issue.
>>
>> Then I bumped into this bug:
>> https://bugs.launchpad.net/juju-deployer/+bug/1575863
>>
>> I followed the suggested workaround (mkdir ~/.juju) and now I get an
>> "error getting env api endpoints". Any ideas?
>>
>> Panagiotis
>>
>> 2016-11-23 15:54:36 Starting deployment of lxd:admin/default
>> 2016-11-23 15:54:36 Error getting env api endpoints, env bootstrapped?
>> 2016-11-23 15:54:36 Command (juju api-endpoints -e lxd:admin/default)
>> Output:
>>
>>
>> 2016-11-23 15:54:36 Deployment stopped. run time: 0.15
>> E
>> ======================================================================
>> ERROR: setUpClass (__main__.TestDeploy)
>> ----------------------------------------------------------------------
>> Traceback (most recent call last):
>>   File "./tests/01-giraph-test.py", line 44, in setUpClass
>>     cls.d.setup(timeout=3600)
>>   File "/usr/lib/python3/dist-packages/amulet/deployer.py", line 704, in
>> setup
>>     subprocess.check_call(shlex.split(cmd))
>>   File "/usr/lib/python3.5/subprocess.py", line 581, in check_call
>>     raise CalledProcessError(retcode, cmd)
>> subprocess.CalledProcessError: Command '['juju-deployer', '-W', '-c',
>> '/tmp/amulet-juju-deployer-o1zyf3xd/deployer-schema.json', '-e',
>> 'lxd:admin/default', '-t', '3700', 'lxd:admin/default']' returned
>> non-zero exit status 1
>>
>> ----------------------------------------------------------------------
>> Ran 0 tests in 4.738s
>>
>> FAILED (errors=1)
>>
>>
>>
>> 2016-11-22 16:31 GMT+02:00 Konstantinos Tsakalozos
>> <kos.tsakalozos at canonical.com>:
>> > Cool! The error that you are getting now says that the giraph charm has
>> > no
>> > giraph relation [0].
>> >
>> > Looking at the metadata.yaml, you are using the mahout interface to
>> > relate
>> > to the hadoop client. So the relation call should look like:
>> > cls.d.relate('giraph:mahout', 'client:mahout') . This will work for now
>> > since both Mahout and Giraph just want to add jars on the hadoop-client
>> > charm. However, on the long run we should refactor charms such as
>> > hadoop-client, spark, pig, etc, so that they have an interface exactly
>> > for
>> > that purpose (adding jars to the file system). For now you have two
>> > options
>> > either keep using the mahout interface or use the "juju-info" [1]
>> > interface
>> > that is present in all charms. Note, however, that if you do use
>> > "juju-info"
>> > your charm would be a subordinate to any other charm even if that does
>> > not
>> > make sense.
>> >
>> > [0]
>> >
>> > http://pythonhosted.org/amulet/amulet.html?highlight=relate#amulet.deployer.Deployment.relate
>> > [1] https://jujucharms.com/docs/2.0/authors-implicit-relations
>> >
>> > On Tue, Nov 22, 2016 at 4:02 PM, Panagiotis Liakos <p.liakos at di.uoa.gr>
>> > wrote:
>> >>
>> >> Thanks Konstantinos, you were right. Now the script progresses a little
>> >> further:
>> >>
>> >> $ python3 ./tests/01-giraph-test.py
>> >> E
>> >> ======================================================================
>> >> ERROR: setUpClass (__main__.TestDeploy)
>> >> ----------------------------------------------------------------------
>> >> Traceback (most recent call last):
>> >>   File "./tests/01-giraph-test.py", line 42, in setUpClass
>> >>     cls.d.relate('giraph:giraph', 'client:mahout')
>> >>   File "/usr/lib/python3/dist-packages/amulet/deployer.py", line 431,
>> >> in
>> >> relate
>> >>     raise ValueError('%s does not exist for %s' % (rel, srv))
>> >> ValueError: giraph does not exist for giraph
>> >>
>> >> ----------------------------------------------------------------------
>> >> Ran 0 tests in 2.285s
>> >>
>> >> FAILED (errors=1)
>> >>
>> >>
>> >> Is this a namespace issue? Any ideas?
>> >>
>> >> Panagiotis
>> >>
>> >> 2016-11-22 14:16 GMT+02:00 Konstantinos Tsakalozos
>> >> <kos.tsakalozos at canonical.com>:
>> >> > Hi Panagiotis,
>> >> >
>> >> > The 404 error seems to be because the Giraph charm is not yet
>> >> > promulgated
>> >> > (your charm currently lives in cs:~panagiotisl/giraph). Could you try
>> >> > the
>> >> > same test only this time with a charm revision you have pushed under
>> >> > your
>> >> > namespace in line
>> >> >
>> >> >
>> >> > https://github.com/panagiotisl/bigtop/blob/master/bigtop-packages/src/charm/giraph/layer-giraph/tests/01-giraph-test.py#L29
>> >> > ? You can also use a local path pointing to where your charm build
>> >> > output
>> >> > is.
>> >> >
>> >> > Glad the multiple SLF4J bindings are not causing any problem.
>> >> >
>> >> > Thanks,
>> >> > Konstantinos
>> >> >
>> >> > On Tue, Nov 22, 2016 at 1:15 PM, Panagiotis Liakos
>> >> > <p.liakos at di.uoa.gr>
>> >> > wrote:
>> >> >>
>> >> >> Dear Merlijn and Konstantinos,
>> >> >>
>> >> >> Thanks a lot for the very valuable information. I will have another
>> >> >> look at the documentation for writing tests and debugging and I'll
>> >> >> try
>> >> >> to add some logs in my test.
>> >> >>
>> >> >> When I execute my smoke-test (through ssh to giraph) I get the
>> >> >> following (which is exactly what I expect):
>> >> >> https://pastebin.ubuntu.com/23516317/
>> >> >>
>> >> >> When I execute the 01-giraph-test.py (and after I installed amulet)
>> >> >> I
>> >> >> get the following: https://pastebin.ubuntu.com/23516415/
>> >> >>
>> >> >> The URL mentioned in the HTTPError exception is valid for mahout but
>> >> >> not for giraph so I guess I am requesting something that is not
>> >> >> there
>> >> >> (yet).
>> >> >>
>> >> >> Thanks again, I'll keep you updated.
>> >> >>
>> >> >> Panagiotis
>> >> >>
>> >> >>
>> >> >> 2016-11-22 12:08 GMT+02:00 Konstantinos Tsakalozos
>> >> >> <kos.tsakalozos at canonical.com>:
>> >> >> > Hi Panagiotis,
>> >> >> >
>> >> >> > Merlijn is right, the output of the smoke-test script is available
>> >> >> > in
>> >> >> > the
>> >> >> > juju debug logs (juju debug-log). Here is what I got when running
>> >> >> > your
>> >> >> > smoke-test: http://pastebin.ubuntu.com/23516167/ (multiple SLF4J
>> >> >> > bindings)
>> >> >> > It seems the slf4j-log4j12-1.7.5.jar jar from Giraph is slightly
>> >> >> > older
>> >> >> > that
>> >> >> > the slf4j-log4j12-1.7.10.jar from Hadoop.
>> >> >> >
>> >> >> > In order to add some output to the smoke-test action you could
>> >> >> > call
>> >> >> > "action
>> >> >> > set" and/or "action fail" as we do here
>> >> >> >
>> >> >> >
>> >> >> >
>> >> >> > https://github.com/panagiotisl/bigtop/blob/master/bigtop-packages/src/charm/hbase/layer-hbase/actions/smoke-test
>> >> >> >
>> >> >> > Thank you for your work,
>> >> >> > Konstantinos
>> >> >> >
>> >> >> >
>> >> >> > On Tue, Nov 22, 2016 at 11:06 AM, Merlijn Sebrechts
>> >> >> > <merlijn.sebrechts at gmail.com> wrote:
>> >> >> >>
>> >> >> >> Hi Panagiotis Liakos
>> >> >> >>
>> >> >> >>
>> >> >> >> Cool that you're charming Giraph! Feel free to let me know when
>> >> >> >> you
>> >> >> >> have a
>> >> >> >> working prototype, some of my colleagues are interested in using
>> >> >> >> Giraph.
>> >> >> >>
>> >> >> >> Now for your question; tests and actions are two different
>> >> >> >> things.
>> >> >> >>
>> >> >> >> Actions are similar to hooks; they run on the charm unit itself.
>> >> >> >> The
>> >> >> >> mahout smoke-test is an action. You run them using `run-action`,
>> >> >> >> you
>> >> >> >> can
>> >> >> >> debug them using `juju debug-hooks` and all output gets logged by
>> >> >> >> juju
>> >> >> >> (you
>> >> >> >> can see the output in `juju debug-log`). More info on actions:
>> >> >> >> https://jujucharms.com/docs/stable/actions
>> >> >> >> Tests are scripts that run on your own machine that deploy and
>> >> >> >> test
>> >> >> >> charms. You run them using bundletester. More info on tests:
>> >> >> >> https://jujucharms.com/docs/2.0/developer-testing
>> >> >> >>
>> >> >> >> So if you want to alter the `smoke-test` action; you should alter
>> >> >> >> this
>> >> >> >> file:
>> >> >> >>
>> >> >> >>
>> >> >> >>
>> >> >> >> https://github.com/apache/bigtop/blob/master/bigtop-packages/src/charm/mahout/layer-mahout/actions/smoke-test
>> >> >> >>
>> >> >> >> The actual tests of the Mahout Charm use the smoke-test action to
>> >> >> >> verify
>> >> >> >> the deployment. That is what you see at line 52:
>> >> >> >>
>> >> >> >>
>> >> >> >>
>> >> >> >> https://github.com/apache/bigtop/blob/master/bigtop-packages/src/charm/mahout/layer-mahout/tests/01-mahout-test.py#L52
>> >> >> >> `self.mahout.run_action('smoke-test')` calls `juju run-action` in
>> >> >> >> the
>> >> >> >> background.
>> >> >> >>
>> >> >> >> Does that help you a bit?
>> >> >> >>
>> >> >> >>
>> >> >> >>
>> >> >> >> Kind regards
>> >> >> >> Merlijn
>> >> >> >>
>> >> >> >> 2016-11-22 9:42 GMT+01:00 Panagiotis Liakos <p.liakos at di.uoa.gr>:
>> >> >> >>>
>> >> >> >>> Hi all,
>> >> >> >>>
>> >> >> >>> I am trying to build a charm for Apache Giraph. I have used the
>> >> >> >>> mahout
>> >> >> >>> charm (
>> >> >> >>>
>> >> >> >>>
>> >> >> >>>
>> >> >> >>> https://github.com/apache/bigtop/tree/master/bigtop-packages/src/charm/mahout
>> >> >> >>> ) and made the necessary changes to deploy giraph instead of
>> >> >> >>> mahout.
>> >> >> >>>
>> >> >> >>> Deployment seems to work fine and I am able to submit giraph
>> >> >> >>> jobs
>> >> >> >>> through an ssh connection.
>> >> >> >>>
>> >> >> >>> Now I am trying to create a test similar to the smoke-test of
>> >> >> >>> the
>> >> >> >>> mahout charm (
>> >> >> >>>
>> >> >> >>>
>> >> >> >>>
>> >> >> >>> https://github.com/apache/bigtop/blob/master/bigtop-packages/src/charm/mahout/layer-mahout/actions/smoke-test
>> >> >> >>> )
>> >> >> >>>
>> >> >> >>> I have successfully run the script that I have included in my
>> >> >> >>> smoke-test through ssh and I
>> >> >> >>> would expect that the test would also execute without errors.
>> >> >> >>> However, I am not at all familiar with the 'run-action' process
>> >> >> >>> of
>> >> >> >>> juju and I have no idea how I should alter the following file:
>> >> >> >>>
>> >> >> >>>
>> >> >> >>>
>> >> >> >>>
>> >> >> >>> https://github.com/apache/bigtop/blob/master/bigtop-packages/src/charm/mahout/layer-mahout/tests/01-mahout-test.py
>> >> >> >>>
>> >> >> >>> In particular, I am puzzled by the way the result status is set
>> >> >> >>> in
>> >> >> >>> this example. Simply changing the references of mahout to giraph
>> >> >> >>> does
>> >> >> >>> not seem to work. You can see my file here:
>> >> >> >>>
>> >> >> >>>
>> >> >> >>>
>> >> >> >>>
>> >> >> >>> https://github.com/panagiotisl/bigtop/blob/master/bigtop-packages/src/charm/giraph/layer-giraph/tests/01-giraph-test.py
>> >> >> >>>
>> >> >> >>> When I execute show-action-output I receive very limited
>> >> >> >>> information:
>> >> >> >>> message: exit status 1
>> >> >> >>> status: failed
>> >> >> >>> timing:
>> >> >> >>>   completed: 2016-11-21 15:48:01 +0000 UTC
>> >> >> >>>   enqueued: 2016-11-21 15:47:38 +0000 UTC
>> >> >> >>>   started: 2016-11-21 15:47:40 +0000 UTC
>> >> >> >>>
>> >> >> >>> Is there a way I can view the full output of the 'smoke-test'
>> >> >> >>> execution?
>> >> >> >>>
>> >> >> >>> Thank you,
>> >> >> >>> Panagiotis Liakos
>> >> >> >>>
>> >> >> >>> --
>> >> >> >>> Juju mailing list
>> >> >> >>> Juju at lists.ubuntu.com
>> >> >> >>> Modify settings or unsubscribe at:
>> >> >> >>> https://lists.ubuntu.com/mailman/listinfo/juju
>> >> >> >>
>> >> >> >>
>> >> >> >>
>> >> >> >> --
>> >> >> >> Juju mailing list
>> >> >> >> Juju at lists.ubuntu.com
>> >> >> >> Modify settings or unsubscribe at:
>> >> >> >> https://lists.ubuntu.com/mailman/listinfo/juju
>> >> >> >>
>> >> >> >
>> >> >
>> >> >
>> >
>> >
>
>