Regarding Hadoop Charm - Role based implementation

Tue Dec 15 10:53:36 UTC 2015

Hi Shilpa

The original Hadoop Charm used to do this, but the jujubigdata team rewrote
the Hadoop charms in the way Charles described. You can see the result of
this in the hadoop-spark bundle:
https://jujucharms.com/apache-hadoop-spark/bundle/4. The original Hadoop
charm is more or less deprecated...

As someone who has used both types of Charms, I found that creating a
separate Charm for each role is a lot easier to use, write and maintain.
I'd recommend against bundeling all components in one Charm with multiple
supporting roles. This makes it hard to figure out what is happening
exactly and creates complex and hard to read Charm code.
For example: Now we have separate Charms for Hadoop yarn-master and the
Hadoop hdfs-master role. In the original those roles were part of one
Charm. In the Hadoop ecosystem you have certain components that need to
connect to yarn-master. The new approach enabled the team to create a
'yarn-master' interface that only the yarn-master charm has. This way,
charms that need to connect to yarn-master cannot be accidentally connected
to hdfs-master. This is a lot clearer to users and guards them against
simple mistakes.

Doing the "multiple Charms" approach has been made a lot easier with the
addition of layers. You can create one base LSF layer that is shared among
all LSF Charms. Then you create a layer that implements each role on top of
that base layer.

Kind regards
Merlijn

2015-12-14 21:19 GMT+01:00 Charles Butler <charles.butler at canonical.com>:

> Greetings Shilpa,
>
> I'm cross posting this to the juju mailing list, as there are probably
> others who have had, or have the same question about using a single charm,
> with multiple supporting roles. I would highly encourage you to engage the
> mailing list as you develop questions about Juju, so that other users may
> chime in with their experience.
>
> With regard to changing how the charm deploys/interacts at runtime based
> on relations. It's been noted that while this may ease some of the
> maintenance burden of the charm author - this also causes confusion to
> users when they are deploying the stack. If this is along the lines of
> needing a service group leader, and each additional node becomes a follower
> unit participating in the cluster, I would encourage you to use the
> leadership features of juju
>
> https://jujucharms.com/docs/stable/authors-charm-leadership
>
>
> If you need to do raidcally different configuration of the software based
> on what 'role' the charm is deployed as, it may be worth investigating the
> work required to split the charm into layers, abstract the base delivery of
> the software into a shared layer, and encapsulating the 'roles' into an
> additional layer, and outputting 2 charms. As that will have a clear
> definition of how to deploy the software, especially when you distribute
> the recommended setup as a bundle.
>
> Now that i've spoken my peace against a single charm with multiple roles,
> in the spirit of answering the original question:
>
> To do this you will need to have a few concerns lined out before hand,
> such as what messaging you need to give to your user
>
> - if the user only deploys the LSF charm by itself, with no additional
> units, should it do anything? as in deploy in standalone mode
> - If the user adds units, will this depend on leader data?
>
> - If we need this secondary "role" to be related to enable the workload in
> its most basic form, what messaging is required to do so?
>
>
> With the pre-reqs sorted, and plotted - i'm going to assume the answer to
> the last bullet point was "we need the secondary role, and we have a
> relation for lsf-controller and lsf-node. both are encapsulated in the same
> charm
>
> As you deploy the LSF charm, you can check for the present of the
> relation. In reactive this is fairly trivial, as relations consume
> interfaces, and these interfaces establish states on the unit to identify
> that indeed the relationship is present.
>
> Looking over the HTTP interface, as an example - we see that the interface
>  sets a state for the relation consuming the interface to .available.
>
> https://github.com/juju-solutions/interface-http/blob/master/provides.py#L12
>
>
>
> You can then decorate a method body on the receiving side with this:
>
> @when('lsf-controller.available')
> setup_controller(){
>   # method body to setup the controller
> }
>
> @when('lsf-node.available')
> setup_node(){
>   # method body to setup the node
> }
>
> Each body is basically responsible for executing the setup portion that is
> specific to configuring the LSF daemon for that 'role'. and you've
> encapsulated 2 different roles in the same charm, dictated by
> relationships.
>
> All the best,
>
> Charles
>
> Charles Butler <charles.butler at canonical.com> - Juju Charmer
> Come see the future of datacenter orchestration: http://jujucharms.com
>
> On Mon, Dec 14, 2015 at 12:35 PM, Shilpa Kaul <shilkaul at in.ibm.com> wrote:
>
>> Hi Charles,
>>
>> I am working on IBM Platform LSF charm where I have to implement
>> different hosts types of LSF to form a LSF Cluster. I was going through the
>> juju docs where I read about the topic : charms-service-groups - Roles
>> where services acquire a role at runtime based on the relations that are
>> joined. Hadoop charm is an example where this role concept is implemented.I
>> am not so clear that how services can acquire role at run time?.
>> What I could understand is that I can deploy one charm but different
>> service names and then having relations between these services.
>>
>> I was thinking to implement something similar for my LSF charm ,where I
>> have one charm , say ibm-platform-lsf
>>                                                                     juju
>> deploy ibm-platform-lsf  lsf-master
>>                                                                     juju
>> deploy ibm-platform-lsf lsf-server
>>                                                                     juju
>> deploy ibm-platform-lsf lsf-master-candidate
>>
>> So based upon the host type, I am deploying one single charm, but
>> different service names during deployment and then adding relations among
>> them, for example
>>                                                           juju
>> add-relation lsf-server:server-host lsf-master:master-host
>>
>> I believe Hadoop charm is also implemented like this , please let me know
>> if my understanding is correct.
>>
>> Also, if this is the case ie deploying one charm but different service
>> names, then how we can do this type of deployment through Juju GUI, where
>> we cannot define a specific service name during drag and drop of charms on
>> GUI.
>>
>>
>>
>> Thanks and Regards,
>> Shilpa Kaul
>> IBM India Pvt. Ltd
>> Seat # EGLC 6F B067,
>> C-Block, Embassy Golf Link,
>> Intermediate Ring Road, Bangalore
>> Email : shilkaul at in.ibm.com
>> Mobile : 9900054908
>>
>
>
> --
> Juju mailing list
> Juju at lists.ubuntu.com
> Modify settings or unsubscribe at:
> https://lists.ubuntu.com/mailman/listinfo/juju
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://lists.ubuntu.com/archives/juju/attachments/20151215/d5bd90f8/attachment.html>