Bigtop/Kafka versions

[Kevin Monroe]

Hey Merlijn and Gregory!

On Wed, Dec 7, 2016 at 10:31 AM, Merlijn Sebrechts <
merlijn.sebrechts at gmail.com> wrote:

> Hi Kevin
>
>
> We have a project that requires Kafka  0.10.x. The current stable (bigtop)
> kafka charm deploys kafka 0.8.x. Couple of questions:
>
> 1. Is it possible to deploy Kafka 0.10.x? If so, what's the best way to do
> this?
>
>
We currently don't have a good way to install a package that's different
than the one provided in the bigtop-1.1.0 release.  We do have a mechanism
in development that (when it works) will help you build local charms based
on bigtop-trunk vs bigtop-1.1.0.  In theory, you would override
bigtop_version [0] in your charm's local layer.yaml to point to 'trunk' and
it would use the trunk repository to pull bigtop related packages.  In
practice, there are 2 things wrong with this:

1 - I don't know of any bigtop release package [1] that we can use to pull
in bigtop-trunk puppet recipes.  Therefore, if the new package is
incompatible with the bigtop-1.1.0 puppet recipe, 'puppet apply' will
probably fail.
2 - the latest version of kafka available in the bigtop-trunk repo is still
0.8.1.1 [2]

Note for point 2 above, there *is* a .deb for kafka-0.10 [3], but it hasn't
made it to the trunk repository yet.  Soooo, back to your question.  If you
really wanted to do this, I would suggest:

- juju deploy kafka
- juju ssh kafka/0
- wget
https://ci.bigtop.apache.org/view/Packages/job/Bigtop-trunk-packages/BUILD_ENVIRONMENTS=ubuntu-16.04,COMPONENTS=kafka,label=docker-slave/lastSuccessfulBuild/artifact/output/kafka/kafka_0.10.0.0-1_all.deb
- sudo dpkg -i <deb from above>

That's so awful that I can't believe I typed it, but that's the only thing
I can think of that might get you kafka-0.10 in a bigtop-1.1.0 environment.


> 2. Where can we see what version of Kafka and what version of Bigtop a
> Charm will deploy? It would be very useful to us if this was always stated
> in the Charm's readme.
>
>
I'll take a todo to update our readmes with the version that will be
installed from the current stable bigtop release.  Until then, you can see
the bigtop-1.1.0 packages here (click a pkg to see deb versions):

https://ci.bigtop.apache.org/job/Bigtop-1.1.0/BUILD_ENVIRONMENTS=ubuntu-14.04,label=docker-slave/lastSuccessfulBuild/artifact/output/

And as alluded to above, you can see the latest bigtop packages (which may
or may not be in a usable repo) here:

https://ci.bigtop.apache.org/view/Packages/job/Bigtop-trunk-packages/

I'm sorry I don't have a better answer for you.  Deploying from trunk is
something that would be nice to enable, but we haven't had many requests
for unreleased bigtop packages yet.  I assume by your email that you would
be +1 for making this easier?

[0] - https://github.com/juju-solutions/layer-apache-bigtop-
base/blob/master/layer.yaml#L27
[1] - https://github.com/juju-solutions/layer-apache-bigtop-
base/blob/master/layer.yaml#L34
[2] - https://ci.bigtop.apache.org/job/Bigtop-trunk/BUILD_
ENVIRONMENTS=ubuntu-16.04,label=docker-slave/lastSuccessfulBuild/artifact/
output/apt/pool/contrib/k/kafka/
[3] -
https://ci.bigtop.apache.org/view/Packages/job/Bigtop-trunk-packages/BUILD_ENVIRONMENTS=ubuntu-16.04,COMPONENTS=kafka,label=docker-slave/

Thanks,
-Kevin


>
> Grergory, in cc, is working on this project.
>
>
>
> Kind regards
> Merlijn
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://lists.ubuntu.com/archives/bigdata/attachments/20161207/eb6385c0/attachment.html>