When I started using Linux and I tried to host my first very own website, I remember thinking about how to protect it. And when you search around, you see a lot of terminology and “best practices” getting thrown around, like “use firewalls”, “don’t use ssh with password”, “do not use the root user, it’s dangerous”, “change the default ports”, and so forth. The problem I have is that these supposedly “best practices” are usually unquestioned.
They are presented as “here is the gospel, now follow my disciples”, without teaching one of the most important skills in hacking. Try not to assume things, always look deeper. When people do not explain the reasoning behind something, especially in IT security, it’s a MAJOR RED FLAG.
Probably they don’t understand it themselves and you should be sceptical. I guess people just copy from each other, because one person, one time, said it’s “best practice”. But do these recommendations really make sense (for security)?
Or is it snakeoil? For preparation of this video I looked at those typical recommendations by searching for stuff like “tips to secure a linux server”. I then tried to see what they have in common and here is the result.
If you want to directly support these videos, checkout liveoverflow. com/support SSH is the defacto standard to access a linux server remotely. For example ssh root@yourserverIP, and you get the shell.
A very typical recommendation is to disable password logins for SSH and instead use SSH keys. You do that by updating the sshd_config and disable PasswordAuthentication. Supposedly because passwords are insecure.
I mean, the comment in the actual ssh config here even states “disable clear text passwords”. So is that true? Let’s have a look into the SSH Protocol Architecture, to learn about how your local machine talks to the remote server.
And here is a section about “Password Authentication” where it clearly states the “weakness”: “If the server has been compromised, [. . .
]” at which point you can already stop, because here we talk about already the case where the server is already hacked. But still, there seem some caveats, so are ssh keys the better option? Well, they come with their own “issues”.
“The use of public key authentication assumes that the client host has not been compromised. It also assumes that the private key of the server host has not been compromised. ” You see, both are not “perfect”.
But we are talking here very small nuances. And this has nothing to do with a strong protection from hackers coming for your server. But you might still wonder about the clear text password, that sounds really scary.
Well, I kinda skipped over a word, and that is “tunnel”. Before ssh sends this kind of information, it will establish an encrypted channel with the server. And inside of that the “cleartext” password is sent.
So it’s not actually cleartext. That’s why the ssh-server has its own private-key, and that’s why you need to verify the fingerprint before you connect to a new server. Your machine remembers the server’s public key in a known-hosts file, and if for example a network attacker man in the middles you, or you mistype your IP or hostname, the ssh client will warn you that something is not right with the connection.
Maybe this is easier to think about: Https, so TLS or SSL is a very different protocol from ssh, but the result is very similar. When you login to YouTube, [ALARM] LiveOverflow from the future here, I thought I make an example with YouTube and Instagram. Turns out they do weird stuff when authenticating.
So I will continue show some twitter login. you also send a “cleartext password” in the HTTP request, but it’s INSIDE the encrypted TLS tunnel. Thanks to HTTPS.
So if you have a network attacker that man in the middles your connection, the browser will warn you about it and refuse to send your password. Like the SSH client would. And that already tells you, that SSH password authentication is not much different from using a password to log into gmail or Instagram.
And would you consider that to be dangerous too? But yes, it’s still passwords, so typical password recommendations apply, as they apply to your instagram account too. You don’t want an attacker to guess your password by bruteforcing, or because you re-use the same password everywhere.
This means, use a password manager to create unique, random and long passwords. But then you are totally secure. Now I still don’t use passwords for SSH, but only because I am lazy.
SSH keys are much more convenient. It just means I DON’T have to use a password manager to copy or type passwords all the time. So in the end “use SSH keys” is a useful recommendation, but not really for security.
it doesn’t make your server magically more secure. It’s a convenience thing. And yes, if convenience even decreases the likelihood of mistakes, is probably the preferred method.
Another SSH related server hardening recommendation is to disable direct root login. And instead create an unprivileged user without root permissions. Now generally it always makes sense to use the least amount of privileges you need, to do your job.
This is a very important security aspect for software. For example, the webserver nginx does not run as root, and instead creates a new unprivileged user, and when it’s started as root, it drops its own privilege. If now an attacker exploits the website that was running on nginx, the attacker would have those privileges, and not be root.
So it seems like this is a reasonable recommendation to do the same for when YOU are working on the server, but on second thought, it doesn’t make much sense. Two reasons. First of all, the purpose of a server is different than your local workstation laptop.
On a laptop you do a lot of stuff that doesn’t require root permissions. Browsing the internet, writing text documents, programming a python application, playing games, whatever. It’s unnecessary to run around as root, and if you accidentally execute a malicious program that you downloaded from a filesharing site, then that malware doesn’t run as root.
But a server you use very differently. Typically you use a server to setup a service, like a website, and then you let that run. You don’t really work on that server.
And to install those services and webservers, you need to be root. So in the end you mostly work as root on the server anyway. And that leads us to the second reason.
A typical recommendation that goes hand in hand with the disabling of the root login, is to add the unprivileged user to the sudo group. And now the user CAN execute commands as root, by adding sudo before a command. I guess to some people these two cases feel different.
“This is not the root user, must be more secure”, but from a pure security capabilities perspective, they are the same. By giving a user the sudo group, you essentially elevate that user to a root user. Indirectly, but security doesn’t care about that it’s indirect.
It’s just one additional step that has almost no security implications. Now some of you may say, “but sudo requires a password, and if an attacker is the user, they don’t know it, so can’t become root”. But there are tons of ways around that.
The quickest and easiest I came up with is, simply change the bashrc to add an entry for sudo. Where you execute a malicious command, indicated by the program id, everytime the user wants to use sudo. The next time the real user logs in, and wants to do something with sudo, they won’t realize, that they just executed a malicious command as root.
Again, here just to visualize, the user executed id without them wanting to. Now sudo can still help with a bit of auditing what happens on a system, through logging. It’s also easier to just give or remove the sudo group to users.
especially in a team environment it makes more sense than just to give everybody the root password. But as you can see, those are mostly convenience features again. But it doesn’t magically protect your server against hackers.
The next tip I want to address, is the changing of default ports. This one is a typical security by obscurity recommendation. It’s one of the best snakeoil examples.
Snake oil meaning: "a substance with no real medicinal value sold as a remedy for all diseases". The explanations for why we want to do that can range from anywhere like “everything can be hacked so we must hide ssh”, to “people will try to bruteforce it if they know there is ssh on port 22”. The term “everything can be hacked” deserves it’s own video to explore, my TLDR is “sure” I guess we can say that, but when people use it, especially in this context, most of the time it’s used for fear mongering to insinuate an urgency to this recommendation.
I’m telling you, an adversary who has the capabilities to just hack into SSH is so powerful, that we losers are not a target. But most importantly, a port change wouldn’t stop that kind of attacker. If such a crazy 0day is made public, or is used by some government agency, a port change won't help you.
A port change might only have an effect against script kiddies and automated scanners that look for ssh servers with weak passwords. So try to think about this yourself. If the “attack” is to try out weak passwords or brutforce passwords, the actual security “fix” is to have either a strong password as mentioned before, or only use ssh keys.
That’s the REAL defense against this. And that’s why moving the port is snakeoil - it doesn’t address the root medical issue. It just makes you believe you did something for security because somebody said so - unfortunately the placebo effect was not yet proven for computers.
But if you are looking for a master's thesis topic in computer science, that might be a cool area to research /s. Another weird ssh setting that some of these guides recommend is to disable ipv6, and only allow access via IPv4? The reasoning for that is also very funny.
They write: “IPv6 is better than IPv4, but you probably aren’t getting much out of it – because neither is anyone else. Hackers get something from it though – because they use it to send malicious traffic. ” What does that even mean?
! What “malicious traffic” happens in ipv6 that doesn’t happen in IPv4. Look.
Yes, there are a few caveats that immediately come to mind. For example IPv4 addresses are “more rare” so banning or blocking IPv4 is a lot more “expensive” for the attacker”. And there are so many more IPv6 addresses, that banning an IPv6 address is, I guess a bit less useful.
It’s “cheaper” to move to the next address. But I think that is a bad economical take. The costs are not that significant.
I think a more interesting issue could happen, if you use some kind of firewall, but the firewall only covers IPv4, and over IPv6 the attacker can reach anything, but then the root problem is your misconfigured firewall. Not the fact that you support ipv6. (Not to mention that server firewall recommendations are weird anyway, I get to that in a second).
IPv6 also has some concern for actual bigger networks. IPv4 nat is basically the best firewall we can have at home. You can open weird ports in your local network without the fear of somebody from the internet directly talking to it.
And that might be different with IPv6. But all of that doesn’t really apply to your single server you rent somewhere in a datacenter. Those are only concerns when you build up your own network, and if you searched for this video, you are proably not ready to setup your own professional network?
Anyway. When I saw this IPv6 recommendation I posted it as a joke on twitter, and I got an extremely funny response. “[.
. . ] For now, making sshd listen on IPv6 only stops automated login attempts more effectively than fail2ban.
”. This is basically the better “change default port recommendation”. Script kiddies don’t try as much IPv6 yet, so if your ssh server listens ONLY on IPv6 and you disable IPv4, that would be even more effective than making ssh listen on another port - I think that’s an awesome counter example.
Jokes aside. Lastly, attack surface reduction is in general a really good paradigm. So maybe disabling IPv6 on the whole network interface, not just just the ssh settings, maybe we could argue about that.
But the way these articles present this recommendation to disable IPv6, is also snakeoil. It doesn’t hurt, but I don’t see any reasonable positive effect for security. If anything you might get a false-sense of security.
Now let’s talk about firewalls. Many resources recommend you to use iptables or UWF (the Uncomplicated Firewall), to block ports. I think “firewall” is a really unfortunate term.
Because it sounds really cool, and really powerful, as if every hacker trying to walk through a firewall dies in burning agony. Firewalls come in different complexities ans features. But the basic firewall settings these resources recommend, are just about blocking all ports, and open the ones you need.
“Block everything, except SSH and the webserver on port 80 and 443. ” This is not different from simply just having ssh listening on port 22 and have the webserver listening on port 80 and 443. You can confirm that with ss or netstat, to see what is listening publicly.
To visualize thios. By listening on a port, you allow the outside to interact with the service listening. Here is a piece of paper with the three listening ports.
Send me a HTTP request to this window, and I will give you the response. Send me a bruteforce ssh attempt, and I tell you your password was wrong. Now let’s add the firewall as recommended.
Here it is. Everything is blocking except these opened windows, these ports. And look at that.
They match. You achieved nothing. Security wise you did nothing.
Again another snakeoil recommendation where you feel cool you did something with fire, but it was kinda useless. Of course firewalls in general are not useless, but you need to have the correct use case for them. For example let’s say you have a frontend server and a database server.
And for whatever reason you didn’t put them into an isolated private network, like a VPN, or you use a default password for it. Now you can use a firewall on the database server, to only allow connections from the frontend server IP. As you can see in my list, there are more things we could talk about.
But the video is getting a bit long, and so I thought to do one more. Auto updates. On a typical ubuntu server, you can update packages, or the operating system itself, with apt-get upgrade or even apt-get dist-upgrade.
And some resources recommend you enable unattended auto-updates. Which is again a lot more complex than these resources suggest. I think everybody agrees that generally, auto-updates are extremely vital for general public security.
Forcing auto-updates on Windows, Android and iPhones, is great. It’s a bit annoying, but keeping those systems up-to-date is very important. But again, the problem is that an actively used workstation, with tons of different software, and users that don’t understand the tech at all, is very different from the use-case of a server where you run a service.
If you want to “seriously” host something on a server, then enabling unattended updates is a great way to disrupt and break your server occasionally. Not every update is security relevant for you and that’s the difficult job of a system administrator, to think about what, how and when to patch systems. And for a larger deployment, this can be a full-time position.
If you host something for customers, you don’t want to accidentally kill your server. This sounds a bit like security is in conflict with business, and we always say “security must always win”, but reality is not that simple. Let’s be honest, if you just run a server with nginx and ssh.
The likelihood that a serious critical vulnerability is found in them is a bit low. Especially one that works against a basic default installation. I really wouldn’t worry much.
But still you could say, well at least I don’t have to care about it if it’s auto updating, so it’s still a win. BUT! Two reasons you should keep in mind.
First, when news hits of a new serious vulnerability, operating system packages that would be automatically updated, might not have the patch yet. So if it’s really serious, you might need to go into your server and do whatever is recommended to mitigate the issue anyway. This is a manual job.
And it’s a balancing act. In a business where customer data could be affected, you must have a responsable sysadmin who can deal with the situation. While on a private small server, you can say “whatever”.
There is nothing important on it, or I just turn it off for a few days. So if a vulnerability in nginx or ssh is not the big threat, then what is? And that’s the second thought.
Whatever webapp you are running on your server is much more attackable. Maybe it’s your own code you wrote that has vulnerabilities. Or code of another webapp you installed from GitHub.
These softwares are not covered by your auto updates. And most likely need to be updated by hand too. To be fair, there are webapps like wordpress, that are so widespread and have had such a huge struggle with vulnerabilities (and many inexperienced people run it), that they have implemented their own auto-update features, which does in general improve security.
But in my opinion, the benefits of unattended system auto-updates, given that it’s not the silver bullet that solves all your update headaches, is too little, compared to the accidental disruption of your service and manual work you have to do by running a server anyway. And that leads me to the conclusion. Yes, any of the opinions I shared are debatable.
You might not agree and you have better reasonings for doing what you do. And that is fine, and even awesome. Because that means you have actually thought about these things.
The whole point of the video is to showcase and critically question “best practices” many people just throw around, without any comments on reasoning, limitations and caveats. IT security is complicated, and not as simple as some people believe it is. And I think what you see here in this small example, is exactly what also happens in the larger IT security industry, with vendors selling you security appliances that supposedly protect you from those evil hackers.
Don’t get blinded, don’t let fear mongering get to you, and always try to dig deeper. If you then have good reasons for why you follow one of these recommendations, that’s totally fine. And one last comment.
The best advice I can give as an IT security professional on running your own server. Just don’t. Of course if you are still learning, and you never tried to run your own server, please do!
But if you are seriously considering to run a service in production. Maybe don’t host it yourself. That is a decision you have.
Being a system administrator is it’s own job and if that is not fun to you, don’t put that responsibility on yourself. That’s what you pay other people for and that’s why I run my blog on ghost, or why, if I would program a webapp, I would pay the higher prices for stuff like AppEngine or Heroku.
