Anything providing Zookeeper?

Tom Barber tom at analytical-labs.com
Tue May 31 15:21:28 UTC 2016 

Sorry, forgot I mentioned that.

There is an unstable drill charm currently because I'm hacking it around
and pushing updates to the store whilst I test adding providers as hooks.
Bear with me until tomorrow and I'll dump something into the stable
channel. It does work, its very basic, missing auth and stuff, but the guts
will spin you up a scalable drill cluster. I'll post some stuff tomorrow
morning with details.

Tom

--------------

Director Meteorite.bi - Saiku Analytics Founder
Tel: +44(0)5603641316

(Thanks to the Saiku community we reached our Kickstart
<http://kickstarter.com/projects/2117053714/saiku-reporting-interactive-report-designer/>
goal, but you can always help by sponsoring the project
<http://www.meteorite.bi/products/saiku/sponsorship>)

On 31 May 2016 at 16:18, Merlijn Sebrechts <merlijn.sebrechts at gmail.com>
wrote:

> The Charms for a Drill cluster? Or did I misread that part?
>
> 2016-05-31 17:16 GMT+02:00 Tom Barber <tom at analytical-labs.com>:
>
>> Which charms?
>>
>> --------------
>>
>> Director Meteorite.bi - Saiku Analytics Founder
>> Tel: +44(0)5603641316
>>
>> (Thanks to the Saiku community we reached our Kickstart
>> <http://kickstarter.com/projects/2117053714/saiku-reporting-interactive-report-designer/>
>> goal, but you can always help by sponsoring the project
>> <http://www.meteorite.bi/products/saiku/sponsorship>)
>>
>> On 31 May 2016 at 16:16, Merlijn Sebrechts <merlijn.sebrechts at gmail.com>
>> wrote:
>>
>>> Cool! I'm interested in these Charms. Are they online somewhere?
>>>
>>> 2016-05-31 16:59 GMT+02:00 Tom Barber <tom at analytical-labs.com>:
>>>
>>>> Just to clear up this thread.
>>>>
>>>> Thanks to Kos we tracked the the condition, where Drill was setting its
>>>> config before the whole quorum was available.
>>>>
>>>> For reference I ended up implementing something a bit like:
>>>>
>>>> https://github.com/juju-solutions/layer-apache-spark/blob/master/reactive/spark.py#L159
>>>>
>>>> Thanks guys, I now have a working basic Apache Drill cluster \o/
>>>>
>>>> Tom
>>>>
>>>> --------------
>>>>
>>>> Director Meteorite.bi - Saiku Analytics Founder
>>>> Tel: +44(0)5603641316
>>>>
>>>> (Thanks to the Saiku community we reached our Kickstart
>>>> <http://kickstarter.com/projects/2117053714/saiku-reporting-interactive-report-designer/>
>>>> goal, but you can always help by sponsoring the project
>>>> <http://www.meteorite.bi/products/saiku/sponsorship>)
>>>>
>>>> On 31 May 2016 at 13:34, Konstantinos Tsakalozos <
>>>> kos.tsakalozos at canonical.com> wrote:
>>>>
>>>>> Hm... I do not see anything wrong with this part of the code. Looking
>>>>> at [0] you should be getting all three Zookeeper units/conversations.
>>>>>
>>>>> Would it be possible to share the charm you are building so that I can
>>>>> test it locally or ping us on IRC channel #juju at freenode for a pair
>>>>> debugging session?
>>>>>
>>>>>
>>>>> [0]
>>>>> https://github.com/juju-solutions/interface-zookeeper/blob/master/requires.py#L40
>>>>>
>>>>> On Tue, May 31, 2016 at 3:19 PM, Tom Barber <tom at analytical-labs.com>
>>>>> wrote:
>>>>>
>>>>>> Okay, slightly different question:
>>>>>>
>>>>>> zookeeper/0 active          idle        2.0-beta7 8
>>>>>> 2181/tcp,9998/tcp 10.106.143.126 Ready (3 zk units)
>>>>>> zookeeper/1 active          idle        2.0-beta7 9
>>>>>> 2181/tcp,9998/tcp 10.106.143.234 Ready (3 zk units)
>>>>>> zookeeper/2 active          idle        2.0-beta7 10
>>>>>>  2181/tcp,9998/tcp 10.106.143.133 Ready (3 zk units)
>>>>>>
>>>>>> I have 3 ZK units and I used the example here:
>>>>>> https://github.com/juju-solutions/interface-zookeeper for the client
>>>>>> config so my function looks like:
>>>>>>
>>>>>> @when('zookeeper.ready')
>>>>>> @when_not('charm.configured')
>>>>>> def configure(zookeeper):
>>>>>>     zklist = ''
>>>>>>     for zk_unit in zookeeper.zookeepers():
>>>>>>         zklist += add_zookeeper(zk_unit['host'], zk_unit['port'])
>>>>>>     zklist = zklist[:-1]
>>>>>>
>>>>>> But it only iterates once even though 3 are listed:
>>>>>>
>>>>>> a) did I do something wrong
>>>>>> b) is this due to some leader election thing and is actually correct?
>>>>>> c) another answer?
>>>>>>
>>>>>> Thanks
>>>>>>
>>>>>> Tom
>>>>>>
>>>>>> --------------
>>>>>>
>>>>>> Director Meteorite.bi - Saiku Analytics Founder
>>>>>> Tel: +44(0)5603641316
>>>>>>
>>>>>> (Thanks to the Saiku community we reached our Kickstart
>>>>>> <http://kickstarter.com/projects/2117053714/saiku-reporting-interactive-report-designer/>
>>>>>> goal, but you can always help by sponsoring the project
>>>>>> <http://www.meteorite.bi/products/saiku/sponsorship>)
>>>>>>
>>>>>> On 31 May 2016 at 12:37, Tom Barber <tom at analytical-labs.com> wrote:
>>>>>>
>>>>>>> Fair enough, that resolves that then!
>>>>>>>
>>>>>>> Thanks Kos.
>>>>>>>
>>>>>>> Tom
>>>>>>>
>>>>>>> --------------
>>>>>>>
>>>>>>> Director Meteorite.bi - Saiku Analytics Founder
>>>>>>> Tel: +44(0)5603641316
>>>>>>>
>>>>>>> (Thanks to the Saiku community we reached our Kickstart
>>>>>>> <http://kickstarter.com/projects/2117053714/saiku-reporting-interactive-report-designer/>
>>>>>>> goal, but you can always help by sponsoring the project
>>>>>>> <http://www.meteorite.bi/products/saiku/sponsorship>)
>>>>>>>
>>>>>>> On 31 May 2016 at 12:25, Konstantinos Tsakalozos <
>>>>>>> kos.tsakalozos at canonical.com> wrote:
>>>>>>>
>>>>>>>> Ah, it is clear now, thanks.
>>>>>>>>
>>>>>>>> The Apache Hadoop charms use Zookeeper when in HA mode. The
>>>>>>>> namenode is the charm that relates to  Zookeeper [0,1]. So to
>>>>>>>> answer your question the Hadoop charms use ZK but they do not
>>>>>>>> carry ZK under the hood. ZK is treated as separate service/charm.
>>>>>>>>
>>>>>>>>
>>>>>>>> [0] https://github.com/juju-solutions/layer-apache-hadoop-namenode
>>>>>>>> /blob/master/metadata.yaml
>>>>>>>> [1] https://github.com/juju-solutions/layer-apache-hadoop-namenode
>>>>>>>> /blob/master/reactive/namenode_failover.py
>>>>>>>>
>>>>>>>> On Tue, May 31, 2016 at 1:50 PM, Tom Barber <
>>>>>>>> tom at analytical-labs.com> wrote:
>>>>>>>>
>>>>>>>>> No worries Kos
>>>>>>>>>
>>>>>>>>> I'm rubbish at explaining stuff properly!
>>>>>>>>>
>>>>>>>>> My query was more along the lines of "I can see the ZK charms
>>>>>>>>> available, but surely the Hadoop charms use ZK under the hood somewhere, do
>>>>>>>>> any of them expose the ZK relation so you can hook into that", because it
>>>>>>>>> seems like overkill running a second ZK setup if one already exists inside
>>>>>>>>> an existing Juju Hadoop deployment.
>>>>>>>>>
>>>>>>>>> Tom
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> --------------
>>>>>>>>>
>>>>>>>>> Director Meteorite.bi - Saiku Analytics Founder
>>>>>>>>> Tel: +44(0)5603641316
>>>>>>>>>
>>>>>>>>> (Thanks to the Saiku community we reached our Kickstart
>>>>>>>>> <http://kickstarter.com/projects/2117053714/saiku-reporting-interactive-report-designer/>
>>>>>>>>> goal, but you can always help by sponsoring the project
>>>>>>>>> <http://www.meteorite.bi/products/saiku/sponsorship>)
>>>>>>>>>
>>>>>>>>> On 31 May 2016 at 11:47, Konstantinos Tsakalozos <
>>>>>>>>> kos.tsakalozos at canonical.com> wrote:
>>>>>>>>>
>>>>>>>>>> I am sorry Tom, I replied to your first email.
>>>>>>>>>>
>>>>>>>>>> Not sure I fully get your question. Not sure what "this" refers
>>>>>>>>>> to in "Is there a way to hook this".
>>>>>>>>>>
>>>>>>>>>> Charms interacting with a Zookeeper quorum need to use this
>>>>>>>>>> interface: http://interfaces.juju.solutions/interface/zookeeper/
>>>>>>>>>> . Through the zookeeper interface you can be informed about zookeeper units
>>>>>>>>>> joining and departing the quorum.
>>>>>>>>>>
>>>>>>>>>> You can have a single Zookeeper service that you can scale to an
>>>>>>>>>> odd number of units. Then you can have any charm "requiring" the above
>>>>>>>>>> relation use the zookeeper service. Typically you would need to construct a
>>>>>>>>>> connection string or edit some configuration file within your charm in
>>>>>>>>>> order to reference the zookeeper units. Have a look at  https://
>>>>>>>>>> github.com/juju-solutions/layer-apache
>>>>>>>>>> -spark/blob/master/lib/charms/layer/apache_spark.py#L175
>>>>>>>>>>
>>>>>>>>>> Thanks,
>>>>>>>>>> Konstantinos
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> On Tue, May 31, 2016 at 1:28 PM, Konstantinos Tsakalozos <
>>>>>>>>>> kos.tsakalozos at canonical.com> wrote:
>>>>>>>>>>
>>>>>>>>>>> Hi Tom,
>>>>>>>>>>>
>>>>>>>>>>> Have you looked at: https://jujucharms.com/apache
>>>>>>>>>>> -zookeeper/trusty/3
>>>>>>>>>>> The apache-zookeeper provides the zookeeper interface, see
>>>>>>>>>>> https://api.jujucharms.com/charmstore/v5/trusty/apache
>>>>>>>>>>> -zookeeper-3/archive/metadata.yaml
>>>>>>>>>>>
>>>>>>>>>>> Thanks,
>>>>>>>>>>> Konstantinos
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> On Tue, May 31, 2016 at 1:13 PM, Tom Barber <
>>>>>>>>>>> tom at analytical-labs.com> wrote:
>>>>>>>>>>>
>>>>>>>>>>>> Morning
>>>>>>>>>>>>
>>>>>>>>>>>> There is a zookeeper interface and a bucket load of Big Data
>>>>>>>>>>>> stuff which all must run ZK under the hood. Is there anything that provides
>>>>>>>>>>>> the relationit so my charm can hook into ZK? I can't see anything obvious
>>>>>>>>>>>> in the Charm Store.
>>>>>>>>>>>>
>>>>>>>>>>>> Tom
>>>>>>>>>>>> --------------
>>>>>>>>>>>>
>>>>>>>>>>>> Director Meteorite.bi - Saiku Analytics Founder
>>>>>>>>>>>> Tel: +44(0)5603641316
>>>>>>>>>>>>
>>>>>>>>>>>> (Thanks to the Saiku community we reached our Kickstart
>>>>>>>>>>>> <http://kickstarter.com/projects/2117053714/saiku-reporting-interactive-report-designer/>
>>>>>>>>>>>> goal, but you can always help by sponsoring the project
>>>>>>>>>>>> <http://www.meteorite.bi/products/saiku/sponsorship>)
>>>>>>>>>>>>
>>>>>>>>>>>> --
>>>>>>>>>>>> Juju mailing list
>>>>>>>>>>>> Juju at lists.ubuntu.com
>>>>>>>>>>>> Modify settings or unsubscribe at:
>>>>>>>>>>>> https://lists.ubuntu.com/mailman/listinfo/juju
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>>
>>>>>
>>>>
>>>> --
>>>> Juju mailing list
>>>> Juju at lists.ubuntu.com
>>>> Modify settings or unsubscribe at:
>>>> https://lists.ubuntu.com/mailman/listinfo/juju
>>>>
>>>>
>>>
>>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://lists.ubuntu.com/archives/juju/attachments/20160531/73022bbb/attachment.html>

More information about the Juju mailing list