Does sftp eliminate the need to check sha1sum?
Marco Ceppi
marco.ceppi at canonical.com
Thu Jan 14 15:04:01 UTC 2016
Hey everyone!
Wow, great discussion so far. I think it's clear that repeat-ability and
reliability is very important and it's something Juju aims to do so we
should make sure charms follow suit. As Merlijn and Cory alluded to, this
is a pretty temporary problem as in Juju 2.0 we will get resources. In
brief, resources will allow you to, in the metadata, declare a set of
things the charm needs and where they map to. Here's a brief example of the
user experience that's being planned around this, *do note that it's
subject to change (and feedback welcome!)*:
# metadata.yaml
name: my-app
...
resources:
jdk:
type: file
application:
type: file
arbitrary-resource:
type: file
>From there, with the new juju charm command which has the publish workflow
you'll be able to directly push your charm to the store (and it's
resources). So, lets say i have /tmp/jdk.tar.gz /tmp/my-app.zip and
/tmp/package.deb
$ charm push ./my-app --resources "jdk=/tmp/jdk.tar.gz
application=/tmp/my-app.zip arbitrary-resource=/tmp/package.deb"
This will upload the charm and files to your personal namespace and map
each file blob to that resource id. When a user deploys the charm they will
get the payload delivered by juju just like the charm and you will be able
to just issue `resource-get <resource>` to retrieve the filepath on disk
that juju stored the resource. This will make software delivery very
reliable and make offline deployments of charms very easy.
Furthermore, at deploy time, you'll be able to modify resources inflight.
So, given the following example, where i have a different jdk
version/vendor that I want to deploy with:
$ juju deploy cs:~marcoceppi/trusty/my-app --resources
"jdk=/home/marco/other-jdk.tar.gz"
This gives me the charm I requested, the resources uploaded previously, but
the jdk resource will be taken from my local machine instead of what is in
the store. Finally, much like upgrade-charm, there will likely be an
upgrade-resource type command to manage resource versions independently to
charm code.
I do want to again stress that this is still in design, so while the
feature will be there, the command and formats might change. We'll be
talking more concretely about this at the summit and I'll follow up next
week with a roadmap and description of each new feature in Juju 2.0!
Marco
On Thu, Jan 14, 2016 at 4:27 PM Charles Butler <charles.butler at canonical.com>
wrote:
> Just as a word of calming against this our recommended patterns are to
> make the payload you're fetching configurable:
>
> such as fetch this particular tarball from server X with a shipping
> SHA1/SHA256 sum of the payload to ensure a) its the same payload b) it
> wasn't corrupted from the point of writing the charm to upgrading.
>
> Shipping these upgrades via charm config leaves existing deployments at
> their version, only new deployments will upgrade the components, and users
> are still left to "manually upgrade" by changing the charm config, which
> should keep you from seeing major breakage unless shipping components
> contain the defect as outlined above.
>
> The charm should handle upgrades, vs the user 'creating' the change.
>
> The GPG approach is interesting, as its replicates some of the "trusted
> registry" features i've been tracking on the app container side, where we
> have warm fuzzies knowing that our image passes a validity hash match, and
> further is verified against the developers signature w/ the hub, so we know
> they signed the release.
>
> I like these ideas, keep up the good work @cory
>
>
> Charles Butler <charles.butler at canonical.com> - Juju Charmer
> Come see the future of datacenter orchestration: http://jujucharms.com
>
> On Thu, Jan 14, 2016 at 7:36 AM, Merlijn Sebrechts <
> merlijn.sebrechts at gmail.com> wrote:
>
>> Please note that this doesn't have to be a burden on the vetting process.
>> The vetting process for an updated binary can be more or less automated,
>> especially with the upcoming juju resources. The important part is that
>> every time the binary changes, the charm version has to be bumped.
>>
>> 2016-01-14 13:29 GMT+01:00 Merlijn Sebrechts <merlijn.sebrechts at gmail.com
>> >:
>>
>>> hi all, Sorry to barge in like this, but this is very important for my
>>> use-case.
>>>
>>>
>>> Binaries that are downloaded always need to be checked using a checksum *included
>>> in the charm*.
>>>
>>> One Charm version should always deploy the *exact same version of that
>>> service*. If you want Juju to be used in production, *Charm deployment
>>> has to be reproducible*. Please do not force people who use Juju in
>>> production to fork all the Charms they use. Store Charms should be good
>>> enough so they can be used in production.
>>>
>>> Consider the use-case of a platform that automatically scales a service
>>> up and down depending on usage. This will break if we allow Charms to be
>>> changed between versions. This has bitten me once already. The Hadoop
>>> Charms downloads the jujubigdata pip dependency and uses code in this
>>> dependency for communication between units. Because of an oversight, two
>>> versions of this pip dependency were not compatible with each other. This
>>> meant that running `juju add-unit` on an existing Hadoop installation was
>>> successful one day and failed the next.
>>>
>>> I understand why the Hadoop Charms were build this way, it is a lot
>>> easier to maintain. However layers fix the maintenance issue.
>>>
>>> We do not know who uses our Charms and for what, so *there is no way we
>>> can reliably determine if a change would break a use case*. Yes, this
>>> change could fix a bug but there could be some service relying on this bug
>>> to be present. Because of this, one version of a Charm must always deploy
>>> the exact same thing. Let the users handle the upgrade process themselves.
>>>
>>> There are examples enough of cases where even minor version changes of
>>> binaries break relationships, so one version of a charm must always deploy
>>> the same binary.
>>>
>>>
>>>
>>> Kind regards
>>> Merlijn Sebrechts
>>>
>>> 2016-01-14 12:42 GMT+01:00 Cory Johns <cory.johns at canonical.com>:
>>>
>>>> My preference over hard-coding a checksum into the charm would be
>>>> hosting a GPG signature alongside the file and including the public key in
>>>> the charm. This allows the charm author to update a file if necessary
>>>> without having to also update the charm, but also allows the charm to
>>>> confirm that it got the file as intended by the author.
>>>>
>>>> Hopefully, though, this will become moot with the advent of resources
>>>> support in Juju 2.0.
>>>>
>>>> On Thu, Jan 14, 2016 at 1:48 AM, Andrew Wilkins <
>>>> andrew.wilkins at canonical.com> wrote:
>>>>
>>>>> On Thu, Jan 14, 2016 at 3:23 AM José Antonio Rey <jose at ubuntu.com>
>>>>> wrote:
>>>>>
>>>>>> I think this is more of a discusion on if you got 'what' you wanted or
>>>>>> if you got it from 'where' you wanted. Even if you used SFTP, the file
>>>>>> could've changed, and if it doesn't have a SHA1SUM it could result in
>>>>>> unexpected charm breakage.
>>>>>>
>>>>>> If it were me, I would always implement SHA1SUMS, just to make sure
>>>>>> that
>>>>>> the file is, in fact, what I wanted. It would make it easier to debug
>>>>>> and fix later down the road.
>>>>>>
>>>>>
>>>>> +1
>>>>>
>>>>> SFTP/SSH will (can?) ensure the integrity during transit, but can't
>>>>> tell you that the data wasn't tampered with outside of the SFTP transfer
>>>>> process. Someone could have replaced the file on the file server. The
>>>>> client needs to know ahead of time what to expect.
>>>>>
>>>>> On 01/13/2016 02:18 PM, Adam Israel wrote:
>>>>>> > Matt,
>>>>>> >
>>>>>> > For the charm in question, I would think adding the sha1sum check
>>>>>> to the
>>>>>> > process would be sufficient, especially in the scenario that the
>>>>>> binary
>>>>>> > is being self-hosted for the purposes of installing it via the
>>>>>> charm.
>>>>>> >
>>>>>> > Adam Israel - Software Engineer
>>>>>> > Canonical Ltd.
>>>>>> > http://juju.ubuntu.com/ - Automate your Cloud Infrastructure
>>>>>> >
>>>>>> >> On Jan 13, 2016, at 2:14 PM, Tom Barber <tom at analytical-labs.com
>>>>>> >> <mailto:tom at analytical-labs.com>> wrote:
>>>>>> >>
>>>>>> >> Yeah but as pointed out earlier, it verifies where you got it
>>>>>> from,
>>>>>> >> but not what you got. :)
>>>>>> >>
>>>>>> >> On 13 Jan 2016 19:11, "Jay Wren" <jay.wren at canonical.com
>>>>>> >> <mailto:jay.wren at canonical.com>> wrote:
>>>>>> >>
>>>>>> >> StrictHostKeyChecking and shipping the public key of the ssh
>>>>>> host with
>>>>>> >> the charm does seem to meet the criteria of verifying the
>>>>>> intended
>>>>>> >> source.
>>>>>> >>
>>>>>> >>
>>>>>> >> On Wed, Jan 13, 2016 at 1:46 PM, Matt Bruzek
>>>>>> >> <matthew.bruzek at canonical.com
>>>>>> >> <mailto:matthew.bruzek at canonical.com>> wrote:
>>>>>> >> > I recently reviewed a charm that is using sftp to download
>>>>>> the
>>>>>> >> binary files
>>>>>> >> > with a username and password. The charm does not check the
>>>>>> >> sha1sum of these
>>>>>> >> > files.
>>>>>> >> >
>>>>>> >> > The Charm Store Policy states: Must verify that any software
>>>>>> >> installed or
>>>>>> >> > utilized is verified as coming from the intended source
>>>>>> >> >
>>>>>> >> > https://jujucharms.com/docs/stable/authors-charm-policy
>>>>>> >> >
>>>>>> >> > Does using sftp eliminate the need to check the sha1sum of
>>>>>> the files
>>>>>> >> > downloaded?
>>>>>> >> >
>>>>>> >> > What does the Juju community say to this question?
>>>>>> >> >
>>>>>> >> > - Matt Bruzek <matthew.bruzek at canonical.com
>>>>>> >> <mailto:matthew.bruzek at canonical.com>>
>>>>>> >> >
>>>>>> >> > --
>>>>>> >> > Juju mailing list
>>>>>> >> > Juju at lists.ubuntu.com <mailto:Juju at lists.ubuntu.com>
>>>>>> >> > Modify settings or unsubscribe at:
>>>>>> >> > https://lists.ubuntu.com/mailman/listinfo/juju
>>>>>> >> >
>>>>>> >>
>>>>>> >> --
>>>>>> >> Juju mailing list
>>>>>> >> Juju at lists.ubuntu.com <mailto:Juju at lists.ubuntu.com>
>>>>>> >> Modify settings or unsubscribe at:
>>>>>> >> https://lists.ubuntu.com/mailman/listinfo/juju
>>>>>> >>
>>>>>> >> --
>>>>>> >> Juju mailing list
>>>>>> >> Juju at lists.ubuntu.com <mailto:Juju at lists.ubuntu.com>
>>>>>> >> Modify settings or unsubscribe at:
>>>>>> >> https://lists.ubuntu.com/mailman/listinfo/juju
>>>>>> >
>>>>>> >
>>>>>> >
>>>>>>
>>>>>>
>>>>>> --
>>>>>> José Antonio Rey
>>>>>>
>>>>>>
>>>>>> --
>>>>>> Juju mailing list
>>>>>> Juju at lists.ubuntu.com
>>>>>> Modify settings or unsubscribe at:
>>>>>> https://lists.ubuntu.com/mailman/listinfo/juju
>>>>>>
>>>>>
>>>>> --
>>>>> Juju mailing list
>>>>> Juju at lists.ubuntu.com
>>>>> Modify settings or unsubscribe at:
>>>>> https://lists.ubuntu.com/mailman/listinfo/juju
>>>>>
>>>>>
>>>>
>>>> --
>>>> Juju mailing list
>>>> Juju at lists.ubuntu.com
>>>> Modify settings or unsubscribe at:
>>>> https://lists.ubuntu.com/mailman/listinfo/juju
>>>>
>>>>
>>>
>>
>> --
>> Juju mailing list
>> Juju at lists.ubuntu.com
>> Modify settings or unsubscribe at:
>> https://lists.ubuntu.com/mailman/listinfo/juju
>>
>>
> --
> Juju mailing list
> Juju at lists.ubuntu.com
> Modify settings or unsubscribe at:
> https://lists.ubuntu.com/mailman/listinfo/juju
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://lists.ubuntu.com/archives/juju/attachments/20160114/44d300e4/attachment.html>
More information about the Juju
mailing list