Support for Livy Spark REST server

[Merlijn Sebrechts]

Wait, are you saying you were in a food coma for 7 days? :o That must be
one awesome turkey! I'm just getting into bed, I'll answer in full tomorrow
but subordinate livy + standalone Jupyter sounds about right. How does the
bigtop zeppelin talk to spark? Afaik, spark doesn't have a web api, right?

Op donderdag 1 december 2016 heeft Kevin Monroe <kevin.monroe at canonical.com>
het volgende geschreven:
> Hey Merlijn - apologies for the delay.  I, as a Texan, was still eating
mounds of turkey when you sent this.  Just woke up a few minutes ago.
> You mentioned writing Jupyter as a subordinate to Spark, like Zeppelin.
I want to clarify that 'apache-zeppelin' is indeed a subordinate to spark,
but the 'zeppelin' charm from bigtop [0] is standalone (source at [1]).
You could conceivably relate it to hive or hadoop with no spark at all.
Since you mentioned the benefit of a Jupyter charm outside of spark, I
thought you might want to consider making it a principal charm like we've
done with Zepp.
> As for bigtop spark/livy integration, I do not see any enablement in
Bigtop's spark offering, but we can certainly open a jira/PR to make that
happen.
> It seems like the first requirement is to get livy bits to the spark
unit.  Would it make sense to have a spark-livy subordinate that would add
livy to an existing spark deployment?  That subordinate would start the
livy-server process and open relevant ports -- and presumably have its own
interface to pass port/readiness to anyone that wanted to relate.  Does
that sound about right?
> [0] - https://jujucharms.com/zeppelin/
> [1] -
https://github.com/apache/bigtop/tree/master/bigtop-packages/src/charm/zeppelin/layer-zeppelin
> Thanks,
> -Kevin
> On Fri, Nov 25, 2016 at 4:36 AM, Merlijn Sebrechts <
merlijn.sebrechts at gmail.com> wrote:
>>
>> Hi
>>
>> We're writing a charm for Jupyter that connects to Spark. For the moment
we're writing it as a subordinate Charm for Spark; like the Zeppelin charm.
>> Spark subordinate isn't ideal since Jupyter can do a lot more than
connect to Spark and it is even useful on its own. To connect to Spark
remotely we need the Livy Spark REST server.
>> Any plans to include Livy in bigtop?
>>
>>
>> Kind regards
>> Merlijn
>> --
>> Bigdata mailing list
>> Bigdata at lists.ubuntu.com
>> Modify settings or unsubscribe at:
https://lists.ubuntu.com/mailman/listinfo/bigdata
>>
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://lists.ubuntu.com/archives/bigdata/attachments/20161202/1a6051d9/attachment.html>