Apache Hue charm

[Cory Johns]

Merlijn,

In general, I lean away from putting libraries in layers except if they're
required to bootstrap that layer or are otherwise very specific, but there
are benefits and drawbacks to each.

Putting the library in the layer means that it doesn't need to access the
network to install them.  In fact, they can be immediately available
without a separate install step (hence why they're good for bootstrap
libs).  If they're specific to the layer, it can also help keep the code
together instead of spread out over multiple projects.  However, this leads
to "fat" charms that are heavier to download and install.  It also means
that an update to the library, even for a minor bug fix, requires a rebuild
of every single charm built from that layer.

Having them separate means we can use SemVer ranges to automatically pull
in backwards compatible changes without touching each individual charm.  It
can also make it easier to iterate on the library itself, independent of
the layer(s) that use it.  It can also more easily be used by multiple
layers.  But you do then run into potential issues with network-restricted
charm installs, and charm auditing is harder.  The fetch & install step
also must be done before the library can be used, and that has to account
for the possibility that the install fails (could just let the charm error,
but if we can present a better user experience using extended status
reporting, we should, but the trade-off is charm complexity).

On Thu, Nov 5, 2015 at 3:30 AM, Merlijn Sebrechts <
merlijn.sebrechts at gmail.com> wrote:

> Hi Cory
>
>
> I think a lot of these utils are very useful even outside of the bigdata
> community. Afaik, the only "real" bigdata function is "wait_for_hdfs".
> Putting these tools in the charms.bigdata namespace seems to undersell the
> value of these tools. Although I'd rather have them in charms.bigdata if
> that means we have more freedom to iterate on these utils. Putting them in
> charmhelpers might hamper our "freedom" to improve the utils.
>
> charms.bigdata.hadoop,kafka,flume,... seems like a good idea to me.
> Although maybe a layer would be a better approach for this. Any thoughts on
> module vs layer?
>
> 2015-11-04 21:10 GMT+01:00 Cory Johns <cory.johns at canonical.com>:
>
>> Merlijn,
>>
>> I was thinking of splitting those helpers off into a separate, more
>> lightweight library.  Some of them are general enough that it might be
>> worth proposing them up to charm-helpers, though I know there's a push to
>> trim that down.
>>
>> We have a namespace for charms.X and I was thinking we could create a
>> sub-namespace for charms.bigdata.X and the utils part of jujubigdata could
>> live in charms.bigdata.utils, while the bits specific to the core Apache
>> Hadoop charms could move to charms.bigdata.hadoop or something.  Thoughts?
>>
>> On Wed, Nov 4, 2015 at 1:46 PM, Merlijn Sebrechts <
>> merlijn.sebrechts at gmail.com> wrote:
>>
>>> Thanks, Cory
>>>
>>> I was planning on using the jujubigdata library for useful tools (in
>>> utils) like `run_as` and the `re_edit_inline`. What do you propose I do in
>>> this case? Include a fork of only the utils or just pull in jujubigdata and
>>> don't use the interfaces and other non-reactive components?
>>>
>>> 2015-11-04 18:20 GMT+01:00 Cory Johns <cory.johns at canonical.com>:
>>>
>>>> Merlijn,
>>>>
>>>> I think that is unfortunately the most recent version of what has been
>>>> done so far for Apache Hue.  However, if you're going to work on it, I'd
>>>> like to suggest using the layered approach and the interface-hadoop-plugin
>>>> layer.
>>>>
>>>> I've got an in-progress example for Spark at
>>>> https://github.com/johnsca/layer-apache-spark that you could base it
>>>> off of, though I only just started on that yesterday and it still needs
>>>> some work.  Specifically, I wouldn't recommend using the bigdata base
>>>> layer, as it pulls in the jujubigdata library which I don't think is
>>>> necessary and currently doesn't work with layers.  Just using
>>>> 'interface:hadoop-plugin' and 'layer:basic' should be sufficient.
>>>>
>>>> Let me know if that seems feasible and if you have any questions or run
>>>> into any issues.
>>>>
>>>> Thanks!
>>>>
>>>> On Wed, Nov 4, 2015 at 8:55 AM, Merlijn Sebrechts <
>>>> merlijn.sebrechts at gmail.com> wrote:
>>>>
>>>>> Hi
>>>>>
>>>>>
>>>>> I'm going to Charm Apache Hue. From what I see the best place to start
>>>>> is the apache-hue charm from bigdata-dev
>>>>> <https://jujucharms.com/u/bigdata-dev/apache-hue/trusty/5>. This
>>>>> Charm is far from complete and development hasn't been continued for a
>>>>> while so I want to pick up development.
>>>>>
>>>>> I'll be working on this Charm, but before I start I'd like to ask if
>>>>> anyone has a more recent version of this Charm in a private repo that he is
>>>>> planning to publish? So I don't waste my time making something that already
>>>>> exist...
>>>>>
>>>>>
>>>>>
>>>>> Kind regards
>>>>> Merlijn Sebrechts
>>>>>
>>>>> --
>>>>> Bigdata mailing list
>>>>> Bigdata at lists.ubuntu.com
>>>>> Modify settings or unsubscribe at:
>>>>> https://lists.ubuntu.com/mailman/listinfo/bigdata
>>>>>
>>>>>
>>>>
>>>
>>
>> --
>> Bigdata mailing list
>> Bigdata at lists.ubuntu.com
>> Modify settings or unsubscribe at:
>> https://lists.ubuntu.com/mailman/listinfo/bigdata
>>
>>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://lists.ubuntu.com/archives/bigdata/attachments/20151105/3e914311/attachment.html>