▲ 193 points
back
109 comments
This article is misguided. Environment variables can be more secure than files. Furthermore, in the presented case there's no improvement in security by switching to a file.
To address my second claim first: file permissions work at the user or group level. ACLs / MAC likewise. SELinux can be configured to assist in this case but it's not as trivial as it appears at first glance, it would be easier to use environment variables.
In the example case of spawning imagemagik, it's running as the same user and therefore has the same level of access to the properties file. That is, it can access the secrets without negotiating any authorisation to do so.
Depending on how imagemagik is launched and how the parent process handles the config file, it's possible that imagemagik could inherit a file handle, already open to the file.
Now to address my first claim, if the parent process is following best practice then it will sanitise the environment before exec'ing imagemagik, that should mean launching the imagemagik process with only the environment it needs.
To give a concrete example, the postfix mail transfer agent is extraordinarily high quality software, its spawn process owns the responsibility of launching external processes, potentially sysadmin supplied / external to postfix. This case would be very comparable to the web app invoking imagemagik.
We can see that it explicitly handles this case as I've suggested is best practice: https://github.com/vdukhovni/postfix/blob/master/postfix/src...
EDIT: accidentally posted before finishing.
If the parent sanitised the child's environment, then the only way for the child to access the data would be to read the parents memory. In practice this is quite easy - try the "ps auxe" command for a sample, however this access can much more easily be controlled by SELinux policy than can file access.
Any obfuscation technique applicable to a config file can similarly be applied to an environment variable.
This article is not talking about _malicious_ applications running on your system, but unintentionally-security-weakening ones. Think of the bash RCE bug from a few months back: the problem was not that Bash was malicious, but a mismatch in expectations about how to trust the environment variables. Similarly, lots of bits of code not written with the glimmer of a notion that the environment is sensitive may end up accidentally leaking them.
This is an extension of the "don't put secrets in the query string" advice for HTTP: it's not that it's more secure: anything along the way that reads the query string can also read the post data just as easily, but that more bits of software may inadvertently leak the query string but not the post data (in logs, for instance).
Thanks. I'm mainly looking at this from the point of how your secrets could be accidentally exposed.
I applaud to postfix for sanitising the ENV, and it's very good practice to do so. But are all the frameworks doing it correctly? Maybe some code is then also just spawning new processes without sanitising? You could argue that's a bug then (which I completely agree), but not all projects are run like postfix...
You are programming. You can do anything. But a guiding principle should be not to surprise other people. Environment contain information about the environment, search paths and the like. They don't contain secrets. That would surprise people.
Sure you can sanitize the environment, and make sure to wipe out sensitive data after reading it, but if that step fails for some reason, the only way you will notice is when you have a data leak. That's not a way to fail safe. You may the one to never make mistakes, but your coworkers aren't that perfect. That can only lead to security problems in the end.
When given the choice to store your secrets in regular config files or make up something with environment variables, choose the former. Doing it the expected way will buy you lots of goodness later on: You can use ACLs or policies to restrict access, you can allow just a limited number of reads, or pretty much anything else the VFS allows you to.
When you are not alone in your programming, do not inflict onto others what you wouldn't want inflicted on yourself. Avoid surprises.
How do you set env vars with out a file?
My company has an internal bit of infrastructure that I think is a somewhat novel approach that allows us to never have any secrets stored unencrypted on disk. There's a server (a set of servers, actually, for redundancy) called the secret server, and its only job is to run a daemon that owns all the secrets. When an app on another server is started up, it must be done from a shell (we use cap) which has an SSH agent forwarded to it. In order for the app to get its database passwords and various other secrets, it makes a request to the secret server (over a TLS-encrypted socket), which checks your SSH identity against an ACL (different identities can have access to different secrets) and does a signing challenge to verify the identity, and if all passes muster, it hands the secrets back. The app process keeps the secrets in memory and your cap shell disconnects, leaving the app unable to fetch any more secrets on your behalf.
The other kink is that the secret server itself reads the secrets from a symmetrically-encrypted file and when it boots, it doesn't actually know how to decrypt it. There's a master key for this that's stored GPG encrypted so that a small number of people can retrieve it and use a client tool that sends the secret server an "unlock" command containing the master key. So any time a secret server reboots, someone with access needs to gpg --decrypt mastersecret | secret_server_unlock_command someserver
There are some obvious drawbacks to this whole system (constraining pushes to require an SSH agent connection is a biggie and wouldn't fly some places, and agent forwarding is not without its security implications) and some obvious problems it doesn't solve (secrets are obviously still in RAM), but on the whole it works very well for distributing secrets to a large number of apps, and we have written tools that have basically completely eliminated any individual's need to ever actually lay eyes on a secret (e.g. if you want to run any tool in the mysql family, there's a tool that fetches the secret for you and spawns the tool you want with MYSQL_PWD temporarily set in the env, so you need not copy/paste it or be tempted to stick it in a .my.cnf).
This reminds me of OpenStack Barbican (Previously called CloudKeep.. kinda..) initially built by Rackspace. A good intro video at [1].
One of the interesting (and optional) things is does, is provide a agent to run on your instances that require the secrets, the agent implements a FUSE filesystem, and access to this filesytem is controlled by policy. For example - A policy can say "Allow exactly 1 read of /secrets/AWS.json within 120 seconds of boot". Any out of policy access attempts can cause the instance to be blacklisted, preventing any future secret access etc..
[1]: https://www.openstack.org/summit/portland-2013/session-video...
The system sounds very well thought-out, though probably not applicable at my $work location.
> When an app on another server is started up, it must be done from a shell
That's a no-go for many setups. It doesn't integrate well with how Linux distros usually start services (systemd, upstart, sysv init, ...), and means you have to have another way to manage dependencies between your services.
> When an app on another server is started up, it must be done from a shell (we use cap) which has an SSH agent forwarded to it. In order for the app to get its database passwords and various other secrets, it makes a request to the secret server (over a TLS-encrypted socket), which checks your SSH identity against an ACL
At this point you could have used ssh right away, no? Any reason you used TLS + checking SSH agent instead?
If anyone's interested in a somewhat out-of-the-box version of what's described above, using a Consul server/cluster to hold this information should give you basically everything ntucker listed. It's pretty trivial to setup and configuring it to store its data on an encrypted partition is also pretty simple. It's got ACLs and can support TLS connections as well. It's also got a bunch of features that the above system doesn't have, like being distributed (redundancy isn't the same thing as consensus) and datacenter-aware (I'd prefer to have different secrets per-datacenter, when possible).
We've been using it to store our application secrets for some time and had no complaints.
This sounds like a pretty standard bastion server configuration. The use of SSH is novel, usually I see the bastion address provided as a command-line option and a TLS certificate used to authenticate the client.
There's a not-widely-publicized feature of Linux that allows programs to store secrets directly in the kernel: https://www.kernel.org/doc/Documentation/security/keys.txt That has some advantages, including the guarantee that it can't be swapped to disk. Kerberos can use it for secret storage, I haven't seen it used elsewhere though.
It looks like process-private storage is one of its features.
I haven't looked to hard at the docs yet, but this seems kind of awesome. Is it something that you have to build your own kernel for, or is it configurable in a prebuilt kernel?
The main drawback of this ( and it's a minor drawback in the larger scheme of things ) is that this isn't a portable interface and isn't available on other POSIX-ish OS's. It looks like there are key agents available for Freebsd, OS X and Illumos though.
Thanks for pointing this out.
It seems like the security for that keyring is based on uid, which can cause problems in the world of containers.
Classic UNIX behavior was that environment variables were public (any user could see them with the right flags to "ps") so it was well-known not to put anything secret there.
Most (all?) of the current brand of UNIX variants have locked this down quite a while ago, which is a good thing. There are still a few old boxes kicking around though so if you're writing code that is meant to be widely deployed please don't put stuff there. For example: https://github.com/keithw/mosh/issues/156
Even if you are sure that your code will only be running on modern machines I think this article gives good advise. Unless you purge the secret environment variables when you launch they'll get sent to all of your subprocesses and it's quite possible that one of them won't consider their environment secret.
Classic UNIX behavior was also to have plaintext passwords in /etc/passwd, the shadow file was invented much later...
But why should we jump through hoops for something that was broken over 15 years ago?
Do you still filter ping packets on your router because back in the '90s large pings would crash[1] many operating systems?
This is the best argument for not doing this imo.
"Classic UNIX behavior was that environment variables were public (any user could see them with the right flags to "ps") so it was well-known not to put anything secret there."
The same logic applies to Windows environments and modern *nix too...
Storing private data in a public location is obviously a bad idea.
I've always been uncomfortable with the "store config in the environment" part of the 12 factor app thing, since it does imply storing things like database passwords and such, and the argument is that those shouldn't be in files. But, filesystem permissions are reasonably flexible and are easy to reason about (unlike the potential visibility of ENV).
I also don't really buy the arguments for ENV storage of even non-sensitive data. There's just not really any good reason to do so; your config has to be put in place by some tool, even if it is in ENV; why not make your tool produce a file, with reasonable permissions in a well-defined default location? The 12 Factor App article seems to believe that config files live in your source tree, and are thus easily accidentally checked into revision control. That's not where my config files live. My config files live in /etc; or, if I want it to isolate a service to a user, I make a /home/user/etc directory.
One could say, "Don't store passwords in the revision control system alongside your source." And that would be reasonable. But, there's no reason to throw the baby out with the bath water.
> filesystem permissions are reasonably flexible
Env makes things really flexible.For instance,you can call a program with env variables directly thus overriding the default ones,which simplify configuring applications. You dont have to have a test set-up,a production-setup or a staging set-up,just start a server or an app with different env variables in the command line.
You don't want your config or keys to depend on an OS,or a language. Finally Env variables can be restricted to a set of users,so third party process started with a different one cant access them.
I believe env variables are better than other solutions.
Config is ultra-sensitive information. It's like not having the right .gitignore if you're backing up /etc in git or not having enough password log filters.
The issue is that ultimately, one has to trust some infrastructure as being of an ultimately trusted network (whether it's Puppet/Chef/cfe2+ or offline authoritative CA's)
Installing secrets on disk exposes them to potential leakage through backups. This is a major issue, since much less attention is typically paid to access management for backups than to production servers. Therefore I support the approach of providing secrets through the environment.
Once an application has been written to get its secrets from the environment, there is a question of how the secrets are obtained. They can be sourced from a file in an init script, but today we are seeing a lot of momentum towards containerized architecture, and the use of service discovery and configuration systems like etcd, zookeeper and consul.
However, secrets require much more attention to security concerns than the data that these tools are designed to handle. Least privilege, separation of duties, encryption, and comprehensive audit are all necessary when dealing with secrets material. To this end, we have written a dedicated product which provides management and access for secrets and other infrastructure resources (e.g. SSH, LDAP, HTTP web services). The deployment model is similar to the HA systems provided by etcd, consul, SaltStack, etc. It's called Conjur (conjur.net).
Looks interesting. However, isn't the conjur API key stored in .netrc just another secret that can be easily leaked?
On the distribution as a virtual appliance: is it a black box? How do I back it up? How do I upgrade it? How do I check the integrity of the secrets database?
This is indeed an issue, though with a modern cloud based infrastructure and management systems there is no longer a need to create backups from your production servers.
They can be automatically recreated and no important data is stored on them.
Fundamentally, any secrets you store will have some mode of access - there's a downside to each and every way of distributing them.
If you're shelling out to commands you think might snarf credentials, the environment is easy for them to pick it out of, but if they're running as the same user then they could probably read the secrets from the config file. If they aren't running as the same user, you need a way of passing in the secrets - and we tend to come back to environment variables..
The good practice here is just to reset the environment when calling shell commands, as he notes. It's not hard to do.
Ultimately, secrets need to live somewhere and need to be accessed as plain text. Just make sure that the access as small window is as possible, and try to obliterate it after use, if possible.
If one absolutely needs to centralize secrets (TLS/SSL private keys, entropy sources, etc.) (at risk of SPOF or some HA setup), use some PSK style setup that delivers them directly, out-of-band (via separate NICs) or prioritized ahead of regular traffic. Keep it simple. Otherwise, prefer something like zookeeper with encrypted secrets (again PSK keying per box). Try to not deploy the same secret on every box, if possible. Also, try to avoid magic secrets if you can too (remove all local password hashes, use only auth keys).
If you're uncomfortable with plaintext secrets, encrypt them (as end-to-end as possible) and require an out-of-band decryption key at the last possible moment.
It's like having a secure document viewing system... ultimately, someone will need to browse just enough of the plaintext version, or it's not a document viewing system.
Isn't this what TPM was designed to avoid ?
Neither files nor env variables.
Most chipsets have a rather unused TPM function, and it should be possible to have developers and processes hook into that.
Perhaps using tmptool ? On master process startup ask user for passphrase, and use that to query the TPM stored values ?
Please don't use environment variables to store secrets. There are to many angles - as stated by others - where this data may leak into files or processes.
I would propose to use just one folder like /secret and put your config files in there. Exclude this folder from backup on all relevant hosts.
Then spend your time on security of your hosts, applications (OWASP) and monitoring / alerting. Something that you have to do anyway.
Assuming you arn't trying to go for top security, and just want a way to keep things safe from leaking due to errors and the such
why not just make use of the OS's secrets store? for example, like how https://pypi.python.org/pypi/keyring operates
we are in a similar situation and there is another approach I'd like to research. In order to have a distributed properties I was considering using something like consul[1] or etcd[2] which have some control access and load the required variables from upstart scripts
You could still store your secret keys in ENV but encrypt them. Only your program has the method to decrypt them so in the case of an ImageMagick sub process it would access only your encrypted secret key with no knowledge to decrypt it.
Same thing while debugging : only the encrypted key is printed.
Ansible has a neat feature, called Ansible Vault, which lets you encrypt sensitive files. This in combination with dotenv-deployment works pretty well for our Rails Apps. The only thing I’m worried about is someone gaining unauthorised access to our serves and thus being able to read all the credentials stored inside the .env file especially the username & password to our externally hosted db. Probably the only way to prevent this would be “to properly secure your server” and the use of an IDS? Anyone has any experience with someone hacking their servers and successfully preventing e.g. a db dump? In this particular case, how easy would it be to stop attackers in their tracks?
I typically store the env _name_ in the environment, and then use that in my apps to build a path to the file containing secrets (e.g. /etc/{mycompany}/{environment}/myapp.conf). The file is locked down by ACLs or permissions.
There's a (not widely publicized) feature of Linux that enables secure key storage inside the kernel: https://www.kernel.org/doc/Documentation/security/keys.txt Storing keys in the kernel has some advantages -- your key will never get inadvertently swapped to disk etc.
It's been too long since I used it to remember the details, but I believe process-private keys are one of this API's features.
I got bit by env vars a few years back, but due to performance issues in getenv() and ended up writing a whole bunch of PHP magic to ship config files safely and fast.
With pecl/hidef, I can hook a bunch of text .ini files into the PHP engine and defines constants as each requests comes in.
Originally, it was written for a nearly static website, which was spending a ridiculous amount of time defining constants, which rolled over every 30 minutes.
Plus those .ini files were only readable to root, which reads it and then forks off the unprivileged processes.
But with the hidef.per_request_ini, I could hook that into the Apache2 vhost settings, so that the exact same code would operate with different constants across vhosts without changing any code between them.
Used two different RPMs to push those two bits so that I could push code & critical config changes as a two-step process.
And with a bit of yum trickery (the yum-multiverse plugin), I could install two versions of code and one version of config, and other madness like that with the help of update-alternatives.
That served as an awesome A/B testing setup, to innoculate a new release with a fraction of users hitting a different vhost for the micro-service.
I'm rambling now, but the whole point is that you need per-request/per-user config overlay layers, for which env vars are horrible, slow and possibly printed everytime someone throws some diagnostics up.
Please consider the environment before printing this config?
We had to migrate our software from single tenant (per machine) to multiple tenant for our cloud offering, on a 11 year old code base.
We used Michael's trick: environment variables pointing to config files works unbelievably well if you ever need to implement a multiple tenant cloud offering.
So apart from the security aspect, there's the aspect that it is a more versatile design.
We've had good success with distributing our secrets using a GPG-encrypted file that we put in /etc, not in the source code tree. We then use an ENV setting to point the app to the file. This gives us good flexibility (because one server can have multiple GPG files if we want, such as alpha/beta/gamma) and good encryption.
Clearing your environment variables after reading them, and only passing the ENVs required to perform the new task are pretty basic security measures. This was pretty common practice in the 90's, and then I was hoping that would be one of the lessons out of ShellShock.
Ok I'm confused by 'environment variable' vs files. How does one set an environment variable without putting it in a file on the particular server. Or by 'file' in this article (and the 12 factor one) do they mean a file that in source control?
1. It's easy to grab the whole environment and print it out (can be useful for debugging) or send it as part of an error report for instance.
If you have software in your deployment that will send "error reports" to untrusted third parties then you have bigger problems than your shell environment.
2. The whole environment is passed down to child processes
If you don't trust your child processes then you have bigger problems than your shell environment.
3. External developers are not necessarily aware that your environment contains secret keys.
And?
I'm not sure what you mean by "external developer" and what you expect them to do with your environment. E-Mail it out when an error occurs?
If you tolerate that kind of developer on your project then you.. oh well, see above.
why not just split it up with an OTP, that you store in the code (or a file), then the other half in the environmental variable. combine in code (or include the file). seems like that would work. (you need both parts.)
I think this article is a response to people's practice of keeping API keys as an environmental variable so as to keep them off of the filesystem (or at least what git sees and checks in) so that they don't accidentally publish them, as happened in that article article where some gem he was using to respect .gitignore didn't work for some reason.
would this work as a solution?
ENV propagation to unwanted targets is a legit point.
Our most common crash report scenario is the airbrake gem sending crash reports from our rails app to errbit. I can confirm airbrake gem does not post any sensible env data.
Of course, this is only a good news for that specific case, others apps may transfer environment, and we can't just wonder for each app installed "what will it ever send ?".
I wrote a library to handle mixed configuration values by using asymmetric RSA encryption [1].
How do you suppose I do this in AWS where I have several auto-balanced servers? It kind of forces you to put it in environment variables.
while i agree that storing api keys in the code repository is not the best idea, i am curious about the suggestion of moving it into chef configs.
wouldnt that, in turn, also be stored in a code repository, likely accessible in the same way as the main coe repo? then, this feels like a non-solution to me.
While I could simply tell you to blank out ENV vars once you've internalized them, I will instead write an infinitely long essay on how they are "considered harmful" that contributes absolutely nothing back to society.
I created a quick gem (which you shouldn't install) that demonstrates having some untrusted code in your app which will post all of your environmental variables to a 3rd party server: https://github.com/tibbon/env_danger
Now, of course no one would install and run this... but I could imagine someone accidentally typing the name of a Gem wrong, someone accepting a bad PR (a sub-dependancy perhaps even doing so?), etc and somehow something untrusted getting in there. Yes, that means you have other problems, but it isn't outside the realm of possibility that accidental access like this is had.
Just because it shouldn't happen, doesn't mean it will never happen.