▲ 67 points
back
95 comments
Drives me nuts that somewhere along the devops journey people decided that SSHing into a private server used for internal tools is an antiquated and outrageous thing to expect. People for some reason are actually excited about the prospect -- "we're gonna make it so you never have to SSH!". Little do they know that I like SSH. A lot more than I like clicking on the AWS console. And then somehow we're expected to debug the bastard using logz.io or similar -- unconscionable to me but maybe I'm old
Not wanting to use SSH has nothing to do with not liking it. It's about getting your infrastructure to a state where everything you need to manage and troubleshoot it can be done _without_ SSH. This doesn't mean replacing it with click-ops, but having robust deployment and automation in place, an external and centralized log/metrics store, and everything else that minimizes the need to manually manage it at all.
Additionally, removing SSH means removing a large security risk, the need to create and rotate keys, and all the associated mess that we take for granted. It's a huge operational and security burden.
I think the use of IDEs really made this practice less simple. Terminal mode emacs and vi/m are simple with SSH, VSCode and other things start getting more complex.
I think the extensions to VSCode and others to run remotely with a browser are starting to bring some of this back into fashion.
Completely agree that ssh is simpler than clicking on a web console.
A good reason for not relying on ssh is 'cattle vs pets'--using ssh administration can make snowflakes where changes aren't tracked and diagnosable/reproducible. With versioned deployment of system changes you always know how things got to be and can configure many to be the same.
I find it a lot easier to manage a single Linux server than multiple AWS services and a separate third party service for every single thing. Stuff built on top of AWS like Heroku is even worse.
The problem is that a lot of people are now just not comfortable running things in-house. Subscribing to another service and adding an integration feels like the safe option.
I think it's because a lot of people don't actually know how to get full use out of a terminal. When you have to manually enter ssh commands and credentials every single time, browser-based solutions become very attractive.
it's just a bit sad that there was never a proper port knocking protocol established, which had features and was provably secure in theory and implementation, than no one would have issues de/activating ssh per on need-basis
Need to get a file from one box to another? Have SSH? `scp` is your friend: https://en.wikipedia.org/wiki/SSH_File_Transfer_Protocol
scp remote.host:path.txt local
scp local remote.host:path
ssh -D $port $host
Until we had proper VPN infrastructure to enter our VPC at AWS I would utilize this to view private vpc-only RMQ dashboards.
Love SSH!
If you use container tabs in Firefox, you can also configure proxies on a per-container basis! In case that’s useful to other people besides me.
> Need to get a file from one box to another? Have SSH? `scp` is your friend
For less trivial transfers (recursive structures, large collections that you want to sync with minimal time) rsync supports SSH as a transport medium out of the box so
scp remote.host:path.txt local
scp local remote.host:path
rsync remote.host:path.txt local
rsync local remote.host:path
rsync some/local/dir me@remote.host:/target/ --recursive --times
rsync some/local/dir me@remote.host:/target/ --archive # archive does recursive and also preserves times, ownership, etc.
rsync some/local/dir me@remote.host:/target/ --recursive --delete-before --dry-run
… … …
SCP (what you showed as an example) and SFTP (what you linked to a Wikipedia page for) are not the same thing. SCP is an older protocol and, while faster in some situations, is essentially abandoned. Newer versions of OpenSSH actually use SFTP even if you use the SCP command, so you might as well just use the SFTP command instead.
My company's VPN had some weird quirks with routing and DNS that were preventing me getting to certain web frontends, and being able to do a quick SOCKS proxy to get my Firefox accessing the corp network as if it was doing so from a machine in the office was a lifesaver.
sshuttle[0] is a great tool that turns a regular SSH connection into something that behaves more like a traditional VPN.
e.g. I can run "sshuttle -r myserver example.com", and when I next load example.com in my web browser (without any special configuration) it'll be routed via `myserver`.
I don't see how this gets around a VPN? I still need a host in your example. That's what I'm paying a VPN for, a bunch of hosts around the world that are relatively fast.
An alternative I am seeing mentioned with some frequency is Tailscale, which doesn't need port 22 open to the internet, since it's using its own network's connectivity to facilitate your "tailscale SSH" connectivity. From what I read it's very similar to Amazon's SSM Agent.
The usefulness here is that you're closing off ports and reducing your exposure, the downside is that you need proprietary agents installed on the remote devices, and clients (tailscale itself or the SSM extension to AWS CLI) on proprietary networks doing the routing for you. Which might be perfectly fine for your use cases.
I've done even less reading but is Cloudflare's WARP client the same thing for their own network?
It's a proprietary network but not proprietary agents (except the bits specific to proprietary platforms).
One handy feature this enabled is that you can include their open source go library in your program and avoid needing to install anything besides your own binary.
One of the major issues was that a VPN client requires additional configuration and software. Connectivity is the least of the problems they are trying to solve.
Sorry, I'm new to Tailscale, but I do setup and use WireGuard. What does Tailscale offers that WireGuard doesn't if I might ask?
If you enable taillock, I don’t see what a compromised coordination server could do.
You can do all sorts of things with SSH. My favorite is SSHFS, which is couplings for SFTP that treat it like a proper filesystem, and it works on everything that uses SSH. Quicker to setup than a VPN and SMB, and about as secure (you could also theoretically use PAM to authenticate with LDAP or newer MFA protocols)
I use SSHFS daily with my NAS on my home network. It's far faster and simpler to administer than SMB or NFS, with way less config overhead and AAA complications than the other two.
Unfortunately, it is currently semi-abandonware: https://github.com/libfuse/sshfs/blob/eadf7f104a479f0313ecd4... It works well enough for me, but there are certain issues to be aware of. For example, listening for file changes via inotify doesn't work-- and it's a double-whammy of neither SFTP nor FUSE supporting that, so it's unlikely to be fixed any time soon.
qemu lets you boot VMs over ssh:
$ qemu-kvm -hda ssh://example.com/var/tmp/fedora-39.img -m 2048
SSH port forwarding is one amazing aspect of this software. For one example, you can develop on a remote system by forwarding your local port 3000 to the remote 3000, using something like `ssh -nFL 3000:localhost:3000 user@remote`, all while going through SSH! It's an indispensable tool for modern development.
Ahh, good times. SSH tunnels and FoxyProxy to run X11 applications on remote servers.
I wish that SSH would be disaggregated further. SSH has become the suite du jour for file transfer, remote access, and a handful of other things. Unfortunately, simultaneously, innovations have been made in transport protocols and elsewhere in the stack that we're unable to take advantage of.
SFTP is a great example of a protocol which has a discrete server (look! There's sftp-server on your computer. Nothing prevents it from running over TLS, or a web socket). I wish that this was the way the entire suite worked. I wish the multiplexing, and underlying shell implementation was transport agnostic (perhaps relying on SOCK_STREAM, or SOCK_SEQPACKET semantics), and the authentication, encryption, etc was its own thing.
These are good asks. After xz I worry a little about how many eggs are
in the ssh basket.
I don't think that requiring a VPN to use SSH is good advice for big organizations, at least in terms of usability. My uni's (Leeds) comp sci department had this and it was extremely unpleasant to use. While it is "better" from a technical standpoint, I had peers who paid for private compute time instead of using the uni's free clusters. The reality is that even undergraduate comp sci students often don't know enough IT/sysadmin stuff to be able to figure out setting these things up and there's often a benefit to making the barrier to entry lower.
Something that few people remember is that if you have access to a filesystem through SSH, then you can have a remote Git repository with no configuration!
In the remote machine, you only need to create a bare repository:
git init --bare
And in your "client" machines you use it like any other remotes:
git remote add my_remote my_user@my_host:path_to_repo
It can be useful if for some reason you don't want to use GitHub/GitLab/Bitbucket/etc or as a glorified scp
For quite a while before I built my homelab, my git server was a flash drive plugged into my OpenWRT router.
Honestly I still kind of prefer that to gitlab et al. It's nice to not have to leave my terminal to setup a new repo. It takes so much more effort to log into a website and dismiss a bunch of notifications before I can click even more buttons to create a new repo.
I like having all my repos accessible through the website, but I really just want to create new projects through ssh like a civilized person.
Yes this is exactly how I do my private git hosting!
ssh example.com 'git init --bare git/foo.git'
git remote add origin example.com:git/foo.git
I often find out that SSH is blocked in a lot of networks. This is frustrating, since I usually clone git repositories trough SSH (and I'm considering for this reason switching to HTTPS), since I find stupid having to use a VPN just to work with git.
If you really want to have something that gives you access to remote resources in any network the only solution is to use a VPN over TLS on the port 443, to make it impossible to distinguish it from any other normal HTTPS traffic. This is the reason why I run an OpenVPN server at my company, where normally I use Wireguard that is more performant (but it's blocked in a lot of networks).
At the end of the day port 443 with TLS traffic on it is the only thing that is guaranteed to not have been blocked (on port 80, 25, etc firewalls may check that you are effectively transferring HTTP traffic, they could not on 443 since the traffic is encrypted, tough a smart firewall can assume from traffic patterns that the connection is unlikely HTTPS, to this day I've jet to se a firewall this smart).
I think Amazon does a good job here.
Default SSH is with certificates, passwords not used. I like that. Hard to brute force a certificates.
In the old flat 10.X.X.X network amazon days - your new hosts were absolutely hammered when being brought up. There must have been folks on the amazon network itself just portscanning like crazy.
> Hard to brute force a certificates.
I see this mentioned a lot. But is it any harder to brute force a 2048 certificate than a 2048 bit password? (Note: A 2048 bit password with base64 character set is 342 characters long)
Don't get me wrong, there are many advantages to certificates (server doesn't learn the secret, easier to enforce strong secrets, ability to issue centrally, ...) but there is nothing magically different between a certificate and password when it comes to resistance to brute force. If you generate a high-entropy password you are fine from this point of view.
What about an IP whitelist managed on some other website (say in AWS). If you need remote access while you are travelling, you login to that website, which will add your current IP to the whitelist. The server refreshes its firewall with the new whitelist every 5 minutes. So within 5 minutes you get access.
That creates another layer of protection (authentication to the website). I would assume a linux firewall is very hard to bypass so almost as good as not exposing the server to the WAN. And doesn't have all the problems and complexity associated with VPNs, works on any device from anywhere.
Apart from the complexities added by having to build this in the first place, this works, but requires a desktop environment, which is not true of ssh.
You could make the argument that this could be an off-the-shelf product you simply install, but then the default port for it because as big of a target as port 22 as soon as it becomes commonplace enough, except you don't have 20+ years of open-source security research in it, and now you're relying on AWS not being now in your region to connect to your servers.
This is what I do for my own network, but with DNS instead of a website.
I have a cron job that reads a DNS A/AAAA record every 5 minutes or so, and updates an ipset referred to by an iptable (now nftable) rule.
So I log in to my DNS provider's dash and update the record to my current IP address (I could automate this part too, but I'm lazy)
One neat feature of OpenSSH server is the AuthorizedKeysCommand config, which lets you fetch (or generate!) a user's keys from anywhere, e.g. a curl response.
With this you can easily set up a centralized SSH keys system without the pitfalls of decentralized systems or running a CA. Have the user register their public key on your website in the typical fashion, and then write a simple secure endpoint and use AuthorizedKeysCommand to instantly integrate all your OpenSSH servers.
It also lets you implement more exotic authorization schemes with the full capabilities of your internal backend, which is often a million times more enjoyable than fighting through Linux PAM.
If you use SSSD and LDAP and don't like the idea of relying on a curl every login, you can also centrally-manage the keys in LDAP for a similar effects.
SSH is great but rapidly becomes less so once companies decide to put their servers behind some times cascaded SSH gateways, so you need to know which magic invocation of jump hosts to put in to have your connection go through anyway.
it has been for 20 years IIRC. its designed to be a general purpose way to do remote access and command exec, with auth/auth. like thats its whole ballgame. I think the confusion comes when people assumed it was merely a successor to telnet.
but yes this is why its so important for OpenSSL client and server sides to be as bulletproof as they can. its a giant worldwide SPOF and therefore a drool-inducing pinata for hackers.
Privilege Access Management platforms do unify external access protocol for SSH, k8s, databases etc.
For eg.
Cyberark
Checkpoint harmony
Teleport [https://goteleport.com]
Strongdm [https://www.strongdm.com/]
Adaptive [https://adaptive.dev/]
and there are many other tools like these.
Exposing SSH to the world really bothers me. Personally I firewall to my own IP when connecting to EC2 instances, it's a pain but I don't feel comfortable knowing there may be zero days out there, plus I don't want my CPU cycles wasted by script kiddies.
It seems like there should be a better solution - something like port-knocking but done properly.
If you have SSH access you can use it as door to setup a simple VPN https://github.com/sshuttle/sshuttle
If ssh was less useful it would have less of an attack surface. ..... at least this is my theory.
Hence why I rather like it. github uses ssh keys - surely developers can learn to set those up?
That's nice and all, but as TFA already mentions, ssh has a large attack surface. The most critical one though is that you usually grant people access to a shell, so if an account gets breached, you need to worry about local root exploits, which are actually pretty common.
Also, believe it or not, setting up key-based authentication is quite the challenge for a lot of people, especially if you demand encryption of the private key and setting up an agent. However, you cannot enforce private key encryption server-side, so you can't even guarantee some kind of 2FA is in place. Yes, ssh does nowadays support FIDO, but that's even more complicated for users...
OpenSSH has seen one hole recently (and the failed xz attempt) and somehow SSH is less safe than a VPN? How the VPN configured? What is the client OS people are using to connect to the VPN? What's the track record security wise of the various VPN offering?
The "SSH has a wide attack surface as seen by RCE ..." is a bit dishonest IMO. How s any VPN more secure?
If you want configure SSH to be pubkey only and hand over Yubikey to your users.
I wonder: in all the recent data leaks leaking billions of users data where attackers were inside company's networks (and not just on online facing servers), was it through SSH holes that these attack too place? Or are we talking about a corporate culture of Windows+VPN?
I've come down to the same conclusion and automating with fabric and paramiko