How best to install Spark?
Andrew Wilkins
andrew.wilkins at canonical.com
Tue Feb 3 00:43:04 UTC 2015
On Mon, Feb 2, 2015 at 9:57 PM, Samuel Cozannet <
samuel.cozannet at canonical.com> wrote:
> Excellent! Happy to help you through your discovery of awesomeness with
> Juju :)
>
> Note that, if you have jq installed (which I advise, sudo apt-get install
> jq)
> juju stat | python -c 'import sys, yaml, json;
> json.dump(yaml.load(sys.stdin), sys.stdout, indent=4)' | jq
> '.services."<SERVICE NAME>".units."<SERVICE NAME>/0".machine' | tr -d "\""
>
FYI, you can do "juju status --format=json" and skip the Python bit.
will return the ID of the machine for <SERVICE NAME> (replace that by
> yarn-master or what ever the name you gave), which saves you the browsing
> of several pages of juju status...
>
> Let us know how your testing goes!
>
> Best,
> Sam
>
>
> Best,
> Samuel
>
> --
> Samuel Cozannet
> Cloud, Big Data and IoT Strategy Team
> Business Development - Cloud and ISV Ecosystem
> Changing the Future of Cloud
> Ubuntu <http://ubuntu.com> / Canonical UK LTD <http://canonical.com> /
> Juju <https://jujucharms.com>
> samuel.cozannet at canonical.com
> mob: +33 616 702 389
> skype: samnco
> Twitter: @SaMnCo_23
>
> On Mon, Feb 2, 2015 at 2:52 PM, Ken Williams <ken.w at theasi.co> wrote:
>
>> Hi Sam,
>>
>> Just to confirm that deploying the spark-master and the
>> yarn-hdfs-master to
>> the same machine seems to have worked ! :-)
>>
>> // use 'juju status' to find which machine yarn-hdfs-master is on
>> juju status
>> [ etc...]
>> // say...machine: "4"
>>
>> // deploy spark-master to same machine
>> juju deploy --to 4 spark-master
>>
>> // add relations
>> juju add-relation yarn-hdfs-master:resourcemanager spark-master
>> juju add-relation yarn-hdfs-master:namenode spark-master
>>
>>
>> // run test
>> root at ip-172-31-21-92:~# spark-submit --class
>> org.apache.spark.examples.SparkPi /tmp/spark-examples-1.2.0-hadoop2.4.0.jar
>> 25 --master yarn --num-executors 3 --driver-memory 1g
>> --executor-memory 1g --executor-cores 1 --deploy-mode cluster
>> --queue thequeue /tmp/spark-examples-1.2.0-hadoop2.4.0.jar
>> Spark assembly has been built with Hive, including Datanucleus jars on
>> classpath
>> 15/02/02 13:40:45 WARN NativeCodeLoader: Unable to load native-hadoop
>> library for your platform... using builtin-java classes where applicable
>> Pi is roughly 3.1405888
>>
>>
>> Many thanks again for all your help,
>>
>> Best Regards,
>>
>> Ken
>>
>>
>>
>> On 30 January 2015 at 18:11, Ken Williams <ken.w at theasi.co> wrote:
>>
>>>
>>> Ok - Sam, I'll try this and let you know.
>>>
>>> Thanks again for all your help,
>>>
>>> Best Regards,
>>>
>>> Ken
>>>
>>>
>>>
>>> On 30 January 2015 at 18:09, Samuel Cozannet <
>>> samuel.cozannet at canonical.com> wrote:
>>>
>>>> I'll have a look asap, but probably not before Tuesday.
>>>>
>>>> This may be "my guts tell me that" but, if you have the time, try to
>>>> collocate YARN and Spark, that will guarantee you have the YARN_CONF_DIR
>>>> set. I am 90% sure it will fix your problem.
>>>>
>>>> YARN itself will not eat much resources, you should be alright and it
>>>> may allow you to move forward instead of being stuck.
>>>>
>>>> Best,
>>>> Sam
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>> Best,
>>>> Samuel
>>>>
>>>> --
>>>> Samuel Cozannet
>>>> Cloud, Big Data and IoT Strategy Team
>>>> Business Development - Cloud and ISV Ecosystem
>>>> Changing the Future of Cloud
>>>> Ubuntu <http://ubuntu.com> / Canonical UK LTD <http://canonical.com> /
>>>> Juju <https://jujucharms.com>
>>>> samuel.cozannet at canonical.com
>>>> mob: +33 616 702 389
>>>> skype: samnco
>>>> Twitter: @SaMnCo_23
>>>>
>>>> On Fri, Jan 30, 2015 at 7:01 PM, Ken Williams <ken.w at theasi.co> wrote:
>>>>
>>>>> Hi Sam,
>>>>>
>>>>> Attached is my bundles.yaml file.
>>>>>
>>>>> Also, there is no file 'directories.sh' on my spark-master/0
>>>>> machine (see below),
>>>>>
>>>>> ubuntu at ip-172-31-54-245:~$ ls -l /etc/profile.d/
>>>>> total 12
>>>>> -rw-r--r-- 1 root root 1559 Jul 29 2014 Z97-byobu.sh
>>>>> -rwxr-xr-x 1 root root 2691 Oct 6 13:19 Z99-cloud-locale-test.sh
>>>>> -rw-r--r-- 1 root root 663 Apr 7 2014 bash_completion.sh
>>>>> ubuntu at ip-172-31-54-245:~$
>>>>>
>>>>>
>>>>> Many thanks again your help,
>>>>>
>>>>> Ken
>>>>>
>>>>>
>>>>> On 30 January 2015 at 15:45, Samuel Cozannet <
>>>>> samuel.cozannet at canonical.com> wrote:
>>>>>
>>>>>> Hey,
>>>>>>
>>>>>> can you send the bundle you're using (in the GUI, bottom right,
>>>>>> "export" button should give you a bundles.yaml file, please send that to
>>>>>> me, so I can bootstrap the same environment as you are playing with.
>>>>>>
>>>>>> also
>>>>>> * can you let me know if you have a file
>>>>>> /etc/profile.d/directories.sh?
>>>>>> * if yes, can you execute it from your command line, then do the
>>>>>> spark command again, and let me know?
>>>>>>
>>>>>> Thx,
>>>>>> Sam
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>> Best,
>>>>>> Samuel
>>>>>>
>>>>>> --
>>>>>> Samuel Cozannet
>>>>>> Cloud, Big Data and IoT Strategy Team
>>>>>> Business Development - Cloud and ISV Ecosystem
>>>>>> Changing the Future of Cloud
>>>>>> Ubuntu <http://ubuntu.com> / Canonical UK LTD <http://canonical.com> /
>>>>>> Juju <https://jujucharms.com>
>>>>>> samuel.cozannet at canonical.com
>>>>>> mob: +33 616 702 389
>>>>>> skype: samnco
>>>>>> Twitter: @SaMnCo_23
>>>>>>
>>>>>> On Fri, Jan 30, 2015 at 3:46 PM, Ken Williams <ken.w at theasi.co>
>>>>>> wrote:
>>>>>>
>>>>>>> Ok - I have been able to add the relation using this,
>>>>>>>
>>>>>>> juju add-relation yarn-hdfs-master:resourcemanager
>>>>>>> spark-master
>>>>>>>
>>>>>>> But I still cannot see a /etc/hadoop/conf directory on the
>>>>>>> spark-master machine
>>>>>>> so I still get the same error about HADOOP_CONF_DIR and
>>>>>>> YARN_CONF_DIR (below),
>>>>>>>
>>>>>>>
>>>>>>> root at ip-172-31-60-53:~# spark-submit --class
>>>>>>> org.apache.spark.examples.SparkPi --master yarn-client
>>>>>>> --num-executors 3 --driver-memory 1g --executor-memory 1g
>>>>>>> --executor-cores 1 --queue thequeue lib/spark-examples*.jar 10
>>>>>>> Spark assembly has been built with Hive, including Datanucleus jars
>>>>>>> on classpath
>>>>>>> Exception in thread "main" java.lang.Exception: When running with
>>>>>>> master 'yarn-client' either HADOOP_CONF_DIR or YARN_CONF_DIR must be set in
>>>>>>> the environment.
>>>>>>> at
>>>>>>> org.apache.spark.deploy.SparkSubmitArguments.checkRequiredArguments(SparkSubmitArguments.scala:177)
>>>>>>> at
>>>>>>> org.apache.spark.deploy.SparkSubmitArguments.<init>(SparkSubmitArguments.scala:81)
>>>>>>> at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:70)
>>>>>>> at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)
>>>>>>> root at ip-172-31-60-53:~#
>>>>>>>
>>>>>>> Should there be a /etc/hadoop/conf directory ?
>>>>>>>
>>>>>>> Thanks for any help,
>>>>>>>
>>>>>>> Ken
>>>>>>>
>>>>>>>
>>>>>>> On 30 January 2015 at 12:59, Samuel Cozannet <
>>>>>>> samuel.cozannet at canonical.com> wrote:
>>>>>>>
>>>>>>>> Have you tried without ':master":
>>>>>>>>
>>>>>>>> juju add-relation yarn-hdfs-master:resourcemanager spark-master
>>>>>>>>
>>>>>>>> I think Spark master consumes the relationship but doesn't have to
>>>>>>>> expose its master relationship.
>>>>>>>>
>>>>>>>> Rule of thumb, when a relation is non ambiguous on one of its ends,
>>>>>>>> there is no requirement to specify it when adding it.
>>>>>>>>
>>>>>>>> Another option if this doesn't work is to use the GUI to create the
>>>>>>>> relation. It will give you a dropdown of available relationships between
>>>>>>>> entities.
>>>>>>>>
>>>>>>>> Let me know how it goes,
>>>>>>>> Thx,
>>>>>>>> Sam
>>>>>>>>
>>>>>>>>
>>>>>>>> Best,
>>>>>>>> Samuel
>>>>>>>>
>>>>>>>> --
>>>>>>>> Samuel Cozannet
>>>>>>>> Cloud, Big Data and IoT Strategy Team
>>>>>>>> Business Development - Cloud and ISV Ecosystem
>>>>>>>> Changing the Future of Cloud
>>>>>>>> Ubuntu <http://ubuntu.com> / Canonical UK LTD
>>>>>>>> <http://canonical.com> / Juju <https://jujucharms.com>
>>>>>>>> samuel.cozannet at canonical.com
>>>>>>>> mob: +33 616 702 389
>>>>>>>> skype: samnco
>>>>>>>> Twitter: @SaMnCo_23
>>>>>>>>
>>>>>>>> On Fri, Jan 30, 2015 at 1:09 PM, Ken Williams <ken.w at theasi.co>
>>>>>>>> wrote:
>>>>>>>>
>>>>>>>>> Hi Sam,
>>>>>>>>>
>>>>>>>>> I understand what you are saying but when I try to add the 2
>>>>>>>>> relations I get this error,
>>>>>>>>>
>>>>>>>>> root at adminuser-VirtualBox:~# juju add-relation
>>>>>>>>> yarn-hdfs-master:resourcemanager spark-master:master
>>>>>>>>> ERROR no relations found
>>>>>>>>> root at adminuser-VirtualBox:~# juju add-relation
>>>>>>>>> yarn-hdfs-master:namenode spark-master:master
>>>>>>>>> ERROR no relations found
>>>>>>>>>
>>>>>>>>> Am I adding the relations right ?
>>>>>>>>>
>>>>>>>>> Attached is my 'juju status' file.
>>>>>>>>>
>>>>>>>>> Thanks for all your help,
>>>>>>>>>
>>>>>>>>> Ken
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> On 30 January 2015 at 11:16, Samuel Cozannet <
>>>>>>>>> samuel.cozannet at canonical.com> wrote:
>>>>>>>>>
>>>>>>>>>> Hey Ken,
>>>>>>>>>>
>>>>>>>>>> Yes, you need to create the relationship between the 2 entities
>>>>>>>>>> to they know about each other.
>>>>>>>>>>
>>>>>>>>>> Looking at the list of hooks for the charm
>>>>>>>>>> <https://github.com/Archethought/spark-charm/tree/master/hooks> you
>>>>>>>>>> can see there are 2 hooks named namenode-relation-changed
>>>>>>>>>> <https://github.com/Archethought/spark-charm/blob/master/hooks/namenode-relation-changed>
>>>>>>>>>> and resourcemanager-relation-changed
>>>>>>>>>> <https://github.com/Archethought/spark-charm/blob/master/hooks/resourcemanager-relation-changed> which
>>>>>>>>>> are related to YARN/Hadoop.
>>>>>>>>>> Looking deeper in the code, you'll notice they reference a
>>>>>>>>>> function found in bdutils.py called "setHadoopEnvVar()", which based on its
>>>>>>>>>> name should set the HADOOP_CONF_DIR.
>>>>>>>>>>
>>>>>>>>>> There are 2 relations, so add both of them.
>>>>>>>>>>
>>>>>>>>>> Note that I didn't test this myself, but I expect this should fix
>>>>>>>>>> the problem. If it doesn't please come back to us...
>>>>>>>>>>
>>>>>>>>>> Thanks!
>>>>>>>>>> Sam
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> Best,
>>>>>>>>>> Samuel
>>>>>>>>>>
>>>>>>>>>> --
>>>>>>>>>> Samuel Cozannet
>>>>>>>>>> Cloud, Big Data and IoT Strategy Team
>>>>>>>>>> Business Development - Cloud and ISV Ecosystem
>>>>>>>>>> Changing the Future of Cloud
>>>>>>>>>> Ubuntu <http://ubuntu.com> / Canonical UK LTD
>>>>>>>>>> <http://canonical.com> / Juju <https://jujucharms.com>
>>>>>>>>>> samuel.cozannet at canonical.com
>>>>>>>>>> mob: +33 616 702 389
>>>>>>>>>> skype: samnco
>>>>>>>>>> Twitter: @SaMnCo_23
>>>>>>>>>>
>>>>>>>>>> On Fri, Jan 30, 2015 at 11:51 AM, Ken Williams <ken.w at theasi.co>
>>>>>>>>>> wrote:
>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> Thanks, Kapil - this works :-)
>>>>>>>>>>>
>>>>>>>>>>> I can now run the SparkPi example successfully.
>>>>>>>>>>> root at ip-172-31-60-53:~# spark-submit --class
>>>>>>>>>>> org.apache.spark.examples.SparkPi /tmp/spark-examples-1.2.0-hadoop2.4.0.jar
>>>>>>>>>>> Spark assembly has been built with Hive, including Datanucleus
>>>>>>>>>>> jars on classpath
>>>>>>>>>>> 15/01/30 10:29:33 WARN NativeCodeLoader: Unable to load
>>>>>>>>>>> native-hadoop library for your platform... using builtin-java classes where
>>>>>>>>>>> applicable
>>>>>>>>>>> Pi is roughly 3.14318
>>>>>>>>>>>
>>>>>>>>>>> root at ip-172-31-60-53:~#
>>>>>>>>>>>
>>>>>>>>>>> I'm now trying to run the same example with the spark-submit
>>>>>>>>>>> '--master' option set to either 'yarn-cluster' or 'yarn-client'
>>>>>>>>>>> but I keep getting the same error :
>>>>>>>>>>>
>>>>>>>>>>> root at ip-172-31-60-53:~# spark-submit --class
>>>>>>>>>>> org.apache.spark.examples.SparkPi --master yarn-client
>>>>>>>>>>> --num-executors 3 --driver-memory 1g --executor-memory 1g
>>>>>>>>>>> --executor-cores 1 --queue thequeue lib/spark-examples*.jar 10
>>>>>>>>>>> Spark assembly has been built with Hive, including Datanucleus
>>>>>>>>>>> jars on classpath
>>>>>>>>>>> Exception in thread "main" java.lang.Exception: When running
>>>>>>>>>>> with master 'yarn-client' either HADOOP_CONF_DIR or YARN_CONF_DIR must be
>>>>>>>>>>> set in the environment.
>>>>>>>>>>>
>>>>>>>>>>> But on my spark-master/0 machine there is no /etc/hadoop/conf
>>>>>>>>>>> directory.
>>>>>>>>>>> So what should the HADOOP_CONF_DIR or YARN_CONF_DIR value be ?
>>>>>>>>>>> Do I need to add a juju relation between spark-master and ...
>>>>>>>>>>> yarn-hdfs-master to make them aware of each other ?
>>>>>>>>>>>
>>>>>>>>>>> Thanks for any help,
>>>>>>>>>>>
>>>>>>>>>>> Ken
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> On 28 January 2015 at 19:32, Kapil Thangavelu <
>>>>>>>>>>> kapil.thangavelu at canonical.com> wrote:
>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> On Wed, Jan 28, 2015 at 1:54 PM, Ken Williams <ken.w at theasi.co>
>>>>>>>>>>>> wrote:
>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> Hi Sam/Amir,
>>>>>>>>>>>>>
>>>>>>>>>>>>> I've been able to 'juju ssh spark-master/0' and I
>>>>>>>>>>>>> successfully ran the two
>>>>>>>>>>>>> simple examples for pyspark and spark-shell,
>>>>>>>>>>>>>
>>>>>>>>>>>>> ./bin/pyspark
>>>>>>>>>>>>> >>> sc.parallelize(range(1000)).count()
>>>>>>>>>>>>> 1000
>>>>>>>>>>>>>
>>>>>>>>>>>>> ./bin/spark-shell
>>>>>>>>>>>>> scala> sc.parallelize(1 to 1000).count()
>>>>>>>>>>>>> 1000
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> Now I want to run some of the spark examples in the
>>>>>>>>>>>>> spark-exampes*.jar
>>>>>>>>>>>>> file, which I have on my local machine. How do I copy the jar
>>>>>>>>>>>>> file from
>>>>>>>>>>>>> my local machine to the AWS machine ?
>>>>>>>>>>>>>
>>>>>>>>>>>>> I have tried 'scp' and 'juju scp' from the local command-line
>>>>>>>>>>>>> but both fail (below),
>>>>>>>>>>>>>
>>>>>>>>>>>>> root at adminuser:~# scp
>>>>>>>>>>>>> /tmp/spark-examples-1.2.0-hadoop2.4.0.jar ubuntu at ip-172-31-59
>>>>>>>>>>>>> :/tmp
>>>>>>>>>>>>> ssh: Could not resolve hostname ip-172-31-59: Name or service
>>>>>>>>>>>>> not known
>>>>>>>>>>>>> lost connection
>>>>>>>>>>>>> root at adminuser:~# juju scp
>>>>>>>>>>>>> /tmp/spark-examples-1.2.0-hadoop2.4.0.jar ubuntu at ip-172-31-59
>>>>>>>>>>>>> :/tmp
>>>>>>>>>>>>> ERROR exit status 1 (nc: getaddrinfo: Name or service not
>>>>>>>>>>>>> known)
>>>>>>>>>>>>>
>>>>>>>>>>>>> Any ideas ?
>>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> juju scp /tmp/spark-examples-1.2.0-hadoop2.4.0.jar
>>>>>>>>>>>> spark-master/0:/tmp
>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> Ken
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> On 28 January 2015 at 17:29, Samuel Cozannet <
>>>>>>>>>>>>> samuel.cozannet at canonical.com> wrote:
>>>>>>>>>>>>>
>>>>>>>>>>>>>> Glad it worked!
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> I'll make a merge request to the upstream so that it works
>>>>>>>>>>>>>> natively from the store asap.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Thanks for catching that!
>>>>>>>>>>>>>> Samuel
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Best,
>>>>>>>>>>>>>> Samuel
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> --
>>>>>>>>>>>>>> Samuel Cozannet
>>>>>>>>>>>>>> Cloud, Big Data and IoT Strategy Team
>>>>>>>>>>>>>> Business Development - Cloud and ISV Ecosystem
>>>>>>>>>>>>>> Changing the Future of Cloud
>>>>>>>>>>>>>> Ubuntu <http://ubuntu.com> / Canonical UK LTD
>>>>>>>>>>>>>> <http://canonical.com> / Juju <https://jujucharms.com>
>>>>>>>>>>>>>> samuel.cozannet at canonical.com
>>>>>>>>>>>>>> mob: +33 616 702 389
>>>>>>>>>>>>>> skype: samnco
>>>>>>>>>>>>>> Twitter: @SaMnCo_23
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> On Wed, Jan 28, 2015 at 6:15 PM, Ken Williams <
>>>>>>>>>>>>>> ken.w at theasi.co> wrote:
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Hi Sam (and Maarten),
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Cloning Spark 1.2.0 from github seems to have worked!
>>>>>>>>>>>>>>> I can install the Spark examples afterwards.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Thanks for all your help!
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Yes - Andrew and Angie both say 'hi' :-)
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Best Regards,
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Ken
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> On 28 January 2015 at 16:43, Samuel Cozannet <
>>>>>>>>>>>>>>> samuel.cozannet at canonical.com> wrote:
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> Hey Ken,
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> So I had a closer look to your Spark problem and found out
>>>>>>>>>>>>>>>> what went wrong.
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> The charm available on the charmstore is trying to download
>>>>>>>>>>>>>>>> Spark 1.0.2, and the versions available on the Apache website are 1.1.0,
>>>>>>>>>>>>>>>> 1.1.1 and 1.2.0.
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> There is another version of the charm available on GitHub
>>>>>>>>>>>>>>>> that actually will deploy 1.2.0
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> 1. On your computer, the below folders & get there:
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> cd ~
>>>>>>>>>>>>>>>> mkdir charms
>>>>>>>>>>>>>>>> mkdir charms/trusty
>>>>>>>>>>>>>>>> cd charms/trusty
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> 2. Branch the Spark charm.
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> git clone https://github.com/Archethought/spark-charm spark
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> 3. Deploy Spark from local repository
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> juju deploy --repository=~/charms local:trusty/spark
>>>>>>>>>>>>>>>> spark-master
>>>>>>>>>>>>>>>> juju deploy --repository=~/charms local:trusty/spark
>>>>>>>>>>>>>>>> spark-slave
>>>>>>>>>>>>>>>> juju add-relation spark-master:master spark-slave:slave
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> Worked on AWS for me just minutes ago. Let me know how it
>>>>>>>>>>>>>>>> goes for you. Note that this version of the charm does NOT install the
>>>>>>>>>>>>>>>> Spark examples. The files are present though, so you'll find them in
>>>>>>>>>>>>>>>> /var/lib/juju/agents/unit-spark-master-0/charm/files/archive
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> Hope that helps...
>>>>>>>>>>>>>>>> Let me know if it works for you!
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> Best,
>>>>>>>>>>>>>>>> Sam
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> Best,
>>>>>>>>>>>>>>>> Samuel
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> --
>>>>>>>>>>>>>>>> Samuel Cozannet
>>>>>>>>>>>>>>>> Cloud, Big Data and IoT Strategy Team
>>>>>>>>>>>>>>>> Business Development - Cloud and ISV Ecosystem
>>>>>>>>>>>>>>>> Changing the Future of Cloud
>>>>>>>>>>>>>>>> Ubuntu <http://ubuntu.com> / Canonical UK LTD
>>>>>>>>>>>>>>>> <http://canonical.com> / Juju <https://jujucharms.com>
>>>>>>>>>>>>>>>> samuel.cozannet at canonical.com
>>>>>>>>>>>>>>>> mob: +33 616 702 389
>>>>>>>>>>>>>>>> skype: samnco
>>>>>>>>>>>>>>>> Twitter: @SaMnCo_23
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> On Wed, Jan 28, 2015 at 4:44 PM, Ken Williams <
>>>>>>>>>>>>>>>> ken.w at theasi.co> wrote:
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> Hi folks,
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> I'm completely new to juju so any help is appreciated.
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> I'm trying to create a hadoop/analytics-type platform.
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> I've managed to install the 'data-analytics-with-sql-like'
>>>>>>>>>>>>>>>>> bundle
>>>>>>>>>>>>>>>>> (using this command)
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> juju quickstart
>>>>>>>>>>>>>>>>> bundle:data-analytics-with-sql-like/data-analytics-with-sql-like
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> This is very impressive, and gives me virtually everything
>>>>>>>>>>>>>>>>> that I want
>>>>>>>>>>>>>>>>> (hadoop, hive, etc) - but I also need Spark.
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> The Spark charm (
>>>>>>>>>>>>>>>>> http://manage.jujucharms.com/~asanjar/trusty/spark)
>>>>>>>>>>>>>>>>> and bundle (
>>>>>>>>>>>>>>>>> http://manage.jujucharms.com/bundle/~asanjar/spark/spark-cluster
>>>>>>>>>>>>>>>>> )
>>>>>>>>>>>>>>>>> however do not seem stable or available and I can't figure
>>>>>>>>>>>>>>>>> out how to install them.
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> Should I just download and install the Spark tar-ball on
>>>>>>>>>>>>>>>>> the nodes
>>>>>>>>>>>>>>>>> in my AWS cluster, or is there a better way to do this ?
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> Thanks in advance,
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> Ken
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> --
>>>>>>>>>>>>>>>>> Juju mailing list
>>>>>>>>>>>>>>>>> Juju at lists.ubuntu.com
>>>>>>>>>>>>>>>>> Modify settings or unsubscribe at:
>>>>>>>>>>>>>>>>> https://lists.ubuntu.com/mailman/listinfo/juju
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> --
>>>>>>>>>>>>> Juju mailing list
>>>>>>>>>>>>> Juju at lists.ubuntu.com
>>>>>>>>>>>>> Modify settings or unsubscribe at:
>>>>>>>>>>>>> https://lists.ubuntu.com/mailman/listinfo/juju
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>>
>>>>>
>>>>
>>>
>>
>
> --
> Juju mailing list
> Juju at lists.ubuntu.com
> Modify settings or unsubscribe at:
> https://lists.ubuntu.com/mailman/listinfo/juju
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://lists.ubuntu.com/archives/juju/attachments/20150203/b81e5ef5/attachment.html>
More information about the Juju
mailing list