Juju Secrets

[James Beedy]

John,

Thanks for your thoughtful response! I needed some time to digest what you
had to say, as well as do some further investigation into the storing of
secrets in general. Following some research, I would like to share and
discuss some findings of mine.

First off, I feel we could group methods of secrets distribution to more
clearly understand where/how they fit in. I’ve came up with four general
categories I think can be used to classify methods of secrets distribution,
they are:

   1. provider specific
   2. provider agnostic
   3. tooling specific
   4. tooling agnostic

How do you feel about this?

Ok, now let me give you some context to my use case to help you understand
what my motives are.

My company develops custom software for our client’s internal and external
operational needs. At the time when a client project nears completion e.g.
satisfies customer req spec, and is passing our QA, we often hand over the
software application to the client (client assumes responsibility for
deploying, and hosting the software at this point, we remain responsible
for maintenance and support of the software). Prior to my existence at the
company there was no deployment automation or packaging of the software
whatsoever - a great opportunity to slip Juju right in :-) I immediately
took it upon myself to charm up a few of our apps. One of the concerns I’ve
been facing whilst undergoing this process, is how to instrument the
distribution of secrets amongst our now Juju deployed applications keeping
in mind that these application will be packaged and distributed as charms
to individual clients.

While on my hunt, I’ve unveiled two methods of secrets distribution that I
feel have valuable strategies/components to take into consideration. The
first is provider specific to AWS using KMS + IAM + s3 in conjuction with
the metadata api to provision encrypted secrets from s3 directly to the
process environment ->
https://www.promptworks.com/blog/handling-environment-secrets-in-docker-on-the-aws-container-service?utm_source=devops&utm_medium=reddit.
The
second is something I had previously heard wind of, but had never really
dove into: castellan + barbican.

Secrets Matrix

 Secrets Dist. Method    | PA | PS | TA | TS |
 ----------------------- | -- | -- | -- | -- |
 1. AWS-KMS-IAM-ENV      |    | X  | X  |    |
 2. Castellan + Barbican |  X |    | X  |    |

I really like option #1, because it injects the secrets directly into the
process environment as environment variables using the IAM role to grant
access to the kms to get the key, and s3 to pull down the encrypted
details, following which, use the kms key decrypt them. What is notably
commendable about this?

### Pros

* Secrets never hit instance disk

* Secrets stored as encrypted value in access controlled storage

* Per instance access to storage and key based on role

* Secrets communicated through metadata api

### Cons

* Provider specific


Option #2 has the benefit of being provider and tooling agnostic, but
creates a dependency on barbican for any other entity who deploys our app
(this is the same dep problem that using Vault introduces; the acquisition
of a dependency on another technology).


For my use case neither of these implementations really fit the bill,
although they both seem to be great solutions in general.

@chismacnaughton made a great suggestion: to build an application or
tooling level switch that could handle provisioning secrets from different
sources. Possibly this could even be its own charm?

@john, taking your input into consideration alongside my findings, I feel
`juju secrets` could be modeled similarly to the AWS/KMS/IAM
implementation, where an "application" is granted a role which allows
access to the key needed to access, and decrypt the secrets.

Juju specific secrets distribution would be a huge win here (especially for
me), because it would allow the packaging of applications as charms without
the need to introduce a dependency on another technology or a specific
provider.

Hopefully this helps!

~James
​

On Sun, Aug 7, 2016 at 6:45 AM, John Meinel <john at arbash-meinel.com> wrote:

> ...
>>
>
>
>> Sure, an example:
>>
>> A web application needs creds (secrets) in order to access a 3rd party
>> api. These creds need to exist in a config file on the webserver/instance.
>> As a user, you/I know what the secrets are, but now we need to share them
>> with the web application, and only the web application in order for the
>> webapp to connect to the 3rd party api.
>>
>> Who are we hiding the information from?
>>
>> Considering the above example, restricting access to the “secrets” from
>> any entity outside of the one that would need access to them to provision
>> them seems like a reasonable approach. That being said, I feel like access
>> restrictions should be at the charm/layer level.
>>
> So how does a charm/layer prove who it is, such that nobody else can do
> that proof? Juju tracks what charms are deployed to what machine, and each
> unit agent ends up having its own login back to the Controller. However,
> all charms hooks are run as 'root' on the machine (they sort of have to,
> because they need to do stuff like install software). Which means they can
> read any files that exist on the machine.
>
> So there is very little we could do at the 'layer' level, aside from
> pretend that the information couldn't be read. (We could have a secret key
> for a layer, but where could that layer store its secret that any other
> software running as root on that machine couldn't get access to.)
>
> You also have to be a little careful if you ever install 2 charms on the
> same machine (juju deploy foo --to 1), because then both bits of software
> could read the secrets that were for the other software. (Worst case, they
> just read the configuration of the actual service, which must have some way
> to get the unencrypted form, or it couldn't do its job.)
>
> As is, we don't allow agents to read the data for other agents. (The agent
> for machine-1 could ask for the details of machine-2, but will get told
> EPERM).
>
> If we aren't hiding the values from the *user* then simple application
> config is already secret. There may be bugs, but by design the only thing
> that can read the config for an application is the agents for the units of
> that application, the controller itself, and the user.
>
> Now, there is a case for maybe storing the secrets encrypted on the
> controller, but again, all of the information for how to get the secrets
> out have to be in the system, so that *someone* could make use of them.
>
> One thought, we could improve things slightly over the status quo if we
> had each agent create a public/private key pair and register it, and then
> anything flagged 'secret' gets encrypted with the agents public key, and
> the agent stores its private key somewhere on the machine-local disk. Then
> when a user enters secrets, we encrypt the values with that public key, and
> not even the controller has access anymore. Still can't do per-layer
> secrets (cause where can a layer store a secret that another layer in the
> same charm can't get to it)? But we could cut out the ability for anything
> but software running on that machine to read it. I'm not sure if the
> cost/benefit is really there, as if you want to deploy 2 units of the
> software, you have to re-encrypt the original settings for the second unit,
> and if the Controller doesn't have access to the original text, then it
> can't encrypt it for the second unit. The User has to re-enter the
> software. Likely we also need a public/private key that the User tracks, if
> they ever want to 'get' the value back. And I guess we could make it so
> 'add-unit' interacts with the User, who can decrypt the content with the
> User's key, and then encrypt it back with the agents public key.
>
> Its a lot of hassle, with the one advantage that getting access to the
> Controller doesn't give you access to the secrets of all software that has
> been configured.
>
> But I have the feeling that for your actual use case, just the fact that
> the Controller doesn't let anyone but the agent-in-charge-of-that-application
> read the values for that application is actually enough.
>
>
>
>
>> Two possibilities I’ve thought of:
>>
>> A) Layer Sensitive Secrets - I’m thinking a secret might only be readable
>> to the layer in which it is configured to be used. A secret could be
>> context sensitive to the :, such that when you set a secret in a model, you
>> are setting it such that it can only be accessed by a named layer in named
>> charm.
>>
>> B) Shared Model Secrets - Less restrictive “secrets”, accessible
>> globally, at the model level.
>>
>> Are we hiding information from the Juju users?
>>
>> When a secret is set, it could be hashed before being stored in the
>> database, then other users who are members of the same model wouldn’t be
>> able to see what the secret was, but they could see that it exists by
>> listing secrets.
>>
>> Are we trying to hide information from other processes on the same
>> machines as the agents?
>>
>> I think the process isolation would come along for the ride inherently as
>> long as processes and directories have their permissions locked down
>> appropriately upon setup/configuration.
>>
>> Also, who generates and holds the keys?
>>
>> Here’s what I’m thinking, consider the following hypothetical situation:
>>
>> You basically want an action to do the following:
>>
>> sudo htpasswd -c /etc/nginx/htpasswd.users kibanaadmin
>>
>> Cloud admin of company X runs a Kibana instance that he desires to
>> enhance access control on. Kibana is deployed via Juju. Seeing as Juju
>> “secrets” exist, the cloud admin of company X can use the Juju CLI to set
>> the users and passwords (secrets) of those he desires to grant access to
>> the Kibana dashboard. The cloud admin wants to define the “secrets” in a
>> .yaml file, and feed it to his Juju environment using the juju
>> add-secrets command, after which he will trigger an “action” to write
>> out the htpasswd file with the latest secrets associated with the
>> environment. After running juju add-secrets, Juju hashes the secret
>> values and stores them in the model context of the controller, only to be
>> unencrypted (or read) upon a call to their getter function, called by the
>> layer or charm whom they are configured to in secrets.yaml. e.g.
>>
>>    1.
>>
>>    secrets.yaml
>>
>>    secrets:
>>    kibana:
>>     htpasswd-user: someuser
>>     htpasswd-pass: somepassword
>>
>>    2.
>>
>>    juju add-secret lxd-dev-controller:my-lxd-model -f secrets.yaml
>>    3. rm secrets.yaml
>>    4. juju run-action kibana/0 htpasswd access='allow'
>>
>> On #1 the user defines the secrets for the controller:model:charm in the
>> secrets.yaml.
>> On #2 the user updates the controller with the secrets to the
>> controller:model:charm, juju hashes the secrets and stores them in the
>> model context of the controller.
>> On #3 the user deletes the secrets.yaml file as he want to ensure no one
>> will be able to view the secrets if access is breached on his local machine.
>> On #4 the user runs an action that decrypts the secrets stored in the
>> state server, and puts them in the .htaccess file on the server.
>>
>> Starting to feel like I am rambling. Hopefully this will help clarify
>> what I am thinking a "secret" might be!
>>
>> Thoughts?
>>
> Hiding secrets from the user who uploaded them is possible, but poses some
> fairly significant usability costs. If we want to make the controller not a
> fault point, then you can see my above thoughts around creating per-unit
> and user public/private keys.
>
> A cheaper thing could be 'write only' values, which lets you set a config
> value, but doesn't let the user see it. We already have this idea floating
> around for read-only vs admin users. (So I can give you read-only access to
> my Model, but that doesn't let you read the private ssl keyfile.) This
> would take it one step further where you can't read it either. This
> generally suffers from 'do I have the right ssl cert there?' and not being
> able to read the value that you put there.
>
> Any sort of 'encrypting in the database' starts to suffer from 'where is
> the key that lets you read it', And how do you make sure that only the
> right places have access to that key. If you encrypt the whole database,
> but the keys for that database live next to it, you haven't really made
> anything more secure.
>
> John
> =:->
>
>
>
>>
>> On Sun, Jul 24, 2016 at 1:53 PM, Tim Penhey <tim.penhey at canonical.com>
>> wrote:
>>
>> On 25/07/16 06:32, James Beedy wrote:
>>>
>>>>   Proposed Solution: Juju Secrets
>>>>
>>>> To give Juju a combative edge on the privacy pinwheel of secrets
>>>> distribution in the realm of bleeding edge devops tooling, behold my
>>>> hypothetical proposed solution: |juju secrets|.
>>>> Juju secrets could be used like so:
>>>> |juju add-secret mycloud:mymodel -f secrets.yaml|
>>>>
>>>> I know you guys are pressing hard to get 2.0 out the door, so please
>>>> don’t mind my nonsense here. I just wanted to throw the idea out there
>>>> and possibly get some feedback, and have others weigh in on this topic.
>>>>
>>>> Thoughts?
>>>>
>>>
>>> Interesting idea. How would this really work in practice?
>>>
>>> For the secrets to be any use, the units need to be able to get access
>>> to the information right?
>>>
>>> Who are we hiding the information from? This is always a key question
>>> that needs to be answered in order to choose a good solution.
>>>
>>> Are we hiding information from the Juju users?
>>>
>>> Are we trying to hide information from other processes on the same
>>> machines as the agents?
>>>
>>> Also, who generates and holds the keys?
>>>
>>> Just some more questions.
>>>
>>>
>>> Tim
>>>
>> ​
>>
>> --
>> Juju-dev mailing list
>> Juju-dev at lists.ubuntu.com
>> Modify settings or unsubscribe at: https://lists.ubuntu.com/mailm
>> an/listinfo/juju-dev
>>
>>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://lists.ubuntu.com/archives/juju/attachments/20160829/c20c817d/attachment.html>