How is "hacking" even possible if I "defend" properly?

Question

On a Linux-based server, I follow basic practices as below:

1. Make the admin account password long and complicated enough (i.e. theoretically speaking, password cannot be cracked within reasonable time).
2. Monitor all incoming network traffic to the administrative files.
3. To extend the layer of protection from #2 above, monitor local file changes (especially ones that have commands that require sudo privileges).
4. Validate all user input, so that all of the user input is guaranteed to be safe.

As a novice developer, I don't understand how a hacking is even feasible if the server admin practices the above.

Answer 1

Not making sure all security updates are applied? Remember, as the defender, you must win 100% of the time. A hacker only needs to win once.

The steps you listed are also a lot easier said than done (except the password thing... and yet people still choose horrible passwords!).

2) Also, what's a "credible source" for a public facing web server? The entire Internet? The entire Internet, sans China/Russia (/some/other/countries)? Automated systems can detect many types of attacks, but just like antivirus they can only go so far.

3) Monitoring local files is good, but, again, it's not a panacea. What if the attacker manages to inject code into the web server, and then uses a kernel bug to get code into the kernel... without ever writing a file to the disk? At that point, they could write files to the disk, and use a root kit to prevent most (theoretically all) online scans from noticing any changes to the system.

And even if they only manage to exploit the web server, they can do everything the web server can do (which might be all the attacker was interested in anyways).

4) You should always validate user input. Most developers know this (and many try to do it). Sadly, it's much easier said than done, which is why we continue to see issue after issue where user input isn't appropriately validated. You'll never be able to guarantee any real piece of software is correctly validating all user input. Read some PHP+MySQL questions on StackOverflow to see how many people think mysqli_real_escape_string() prevents all SQL injection attacks ("where ID = " . $val is vulnerable, even when $val is the output of mysqli_real_escape_string!).

Even if you could (you can't) ensure every known attack vector was guarded against, you can't do anything more than wildly swinging in the dark against and unknown-unknown (well, continually educating yourself helps).

As an example where your defenses wouldn't have done anything, I was taking part in a security course where we're doing "war games". I was able to root an opposing team's server by being able to get one of their user passwords off another machine (one of them screwed up and typed it into bash as a command by mistake, and they never thought to delete it from .bash_history).

From there, I spoofed the IP of the machine they usually logged in from, and SSHed in, inputting their username + password. I had limited access to the system. I then ran sudo vim, entered the same password again, and had vim spawn a bash shell. Tada! Root access, from a credible source, without modifying any local files in an unusual way, without exploiting a weak password (it was bad, but even the best password in the world wouldn't have helped), nor relying on unvalidated user input.

At that point, being mischievous me, I manually modified all the log files related to my legitimate login, and obliterated their IDS (I'm betting they won't be observant enough to notice I replaced all of its binaries with copies of /bin/true!). A 'real' hacker would likely be far better equipped to ensure their activity wasn't detected by more vigilant admins, but I'd already accomplished my goal, and a small part of me wanted them to find out that someone got it.

Answer 2

If I might mangle Schneier's law:

Anyone, from the most clueless amateur to the best systems administrator, can create a security system that he himself can't crack.

Answer 3

In short, to defend properly is not an easy task because...

Taking a quick stab at your inputs-->

Make admin account password long...

Mere shoulder surfing defeats any amount of endeavors on strong passwords.
Plus if your password storage mechanisms are weak (prone to SQL injections, or insecure hashing mechanisms or, worst, plain text password storage), you remain vulnerable.

Monitor every incoming network traffic

You will be monitoring an awful lot of traffic. What is the guarantee that you will not miss some important stuff? And what about zero-days?
What about encrypted traffic? What will you monitor on that?

...and only allow credible sources.

What if your credible sources get compromised/hacked? Then they are no more credible but you, the sys admin, do not know this!

...monitor local file changes...Validate every user input...

Again, how much can you monitor and how many logs can you analyse...

Moreover, assuming there are at least a few services running on the machine, vulnerabilities in the service may become the cause of your machine being compromised.

I hope I could give you a feel of how hacking is feasible :)

Ack: Image taken from www.quotesparade.com

Answer 4

If you could defend 'properly' then hackers could never get in. The problem is what properly means in this sort of situation.

You could turn the server off, and seal it in concrete half a mile under the earth and it is pretty secure, right? But it is unusable. And if the value of what is on it is high enough, an attacker may try to dig it up. So you can already see from this extreme example that security is a balance. It needs to be appropriate - and still allow the server to be used.

That being the case, problems you will find include:

• how do you ensure that all the executables are secure? Again, you can't. You can decide how much effort you want to spend checking them, and run various tests, but code bases are large enough for most applications that accidental errors creep in.
• does your proposed security option make the server unusable for its target audience? Unplugging it from the network will increase security, but will not please your users.
• do you actually know all the possible inputs your users may want to use? If so, brilliant - white list them. But remember to update every time your application is updated or has new features. And check every time you get a new user. And test that the white list works to stop all non-approved input. And so on...
• attacks may not need to alter any files on disk. Monitoring files won't pick up attacks that target code in memory directly, for example.
• adding technical security controls doesn't take into account social or physical attacks. If an attacker really wants access to a well protected server, they may deem it worthwhile bugging you, logging your keystrokes to gain passwords etc. Or it may be easier to bug a user - maybe they are less aware of security.
• etc etc

Simply put- you can secure to a level, depending on your budget. Decide how much you are willing to spend based on what you want to protect. Accept that there will be some residual risk.

And then plan for something going wrong - at some point your security will be broken and you need to know how to respond. For a simple file server this process may just be wipe and reinstall from backup, but for something more complex you may need to inform users, stakeholders, regulators, etc., store evidence, make forensic copies, rebuild, reconfigure or in some form remediate the incident...and then take the learning experiences from it to reassess your security plan.

...and then... Redo it all again. Regularly reassess your security and your security appetite, as its a rapidly changing world out there!

Answer 5

Let's say you have a web server, which takes one command:

GET /helloworld.txt

And that returns, very simply, "Hello, World!"

This is the easiest thing in the world to code, right? Assuming you take the input, check for the string "GET /helloworld.txt", and do the right thing, then ignore anything else, what are the possibilities for hacking? None?

OK. What size is your command buffer? 20 bytes? Just long enough to accept that command? What if someone gives a command that's two bytes longer? What if they give a command that's 2000 bytes longer? How do you actually receive the command? What talks to the network card? What does the packet decoding? What if the characters are sent in two packets, with no NUL termination between bytes?

In short, even this "simplest" of network programs, which doesn't even need password access, could have LOTS of security flaws.

Simply, it's about complexity. Every time a variable is introduced, the number of POSSIBLE attack vectors increases exponentially. Not all of those will be attack vectors, but they COULD be, and so you must cover them all, if you mean to be secure.

That's pretty much impossible, without a fully audited, fully unit tested, designed-by-contract, and maybe even mathematically proven system. As far as I know, no such system exists. OpenBSD might get close, but I doubt it's all the way there.

The very short answer is: with security, always assume the worst. It's MUCH smarter to know you're vulnerable, and to act in an appropriately humble manner, risking as little as possible, than to assume you're safe simply because you can't think of a possible attack vector.

Answer 6

If you do everything, and do it properly, you can prevent many different attacks. Unfortunately, hacking is more than just directly attack login/access.

A big part includes bypassing your protection, so an attacker does not have to know the password. Social engineering is another big part. Passwords do not protect anything, if they are known by attackers. Additionally, there are more damaging attacks and targets than just gaining access to the admin panel. If someone does not want you to run a webshop, because he has his own and wants to keep his customers (and you out of business), how do you want to prevent a DDoS attack, or simply just someone pulling the plug in your server room? (The cleaning lady, who just wanted to dust the mainboard? ;) )

Also, monitoring does not really help to prevent attacks.

There are different types of protection, tamper responding, tamper resisting and tamper evident systems. Tamper resisting systems want to slow attacks down (e.g. long passwords). Tamper responding tell you if there is an attack happening (e.g. an alarm that goes off is your IDS detects some SQL injection strings), and tamper evident system show you how the actual attacks did happen (e.g. log files)

Hackers try to crack or avoid these protection, and it is your job to be one step ahead. Of course this is very hard, as you can barely know any security holes in your server software, that may allow an attacker to access root information without a password. Usually attackers find these kind of weaknesses first, and it is your job to take care of your system and to slow their attacks down until the problem is fixed.

Answer 7

Make admin account password long and complicated enough (i.e. theoretically speaking, password cannot be cracked within reasonable time).

Cracked by brute force? No. Cracked by social engineering? Yep. If many people know the password, it's not too hard to get it via social engineering. (whenever more than one person knows the password, it is possible to use impersonation effectively to get the password). If only you have access, then, still, physical access can be achieved via social engineering. It's harder to manage this, though, and you need to wonder if it's really worth the effort.

Monitor every incoming network traffic to the administrative files.

This doesn't protect you from DDoS. And if you want to protect from DDoS, there will be too many false positives of users who get unjustly blocked.

Even if your password is long, a botnet can break in. If you have some sort of restriction on the number of times you can fail a password attempt, then they can lock you out.

Also, CAPTCHAs are no longer that secure, there are many entities that pay money for folks to sit all day and solve captchas.

To extend the layer of protection from #2 above, monitor local file changes (especially ones that have commands that require sudo privileges.

You can't fully monitor all file changes. There's just too much information. Most programs do lots of messing with files as they run. How would you differentiate between a program update and a malicious file?

Validate every user input, so that all of the user input are guaranteed to be safe.

Make sure you do this server side.

---

Finally, zero day exploits will always be a problem. If someone finds a security vulnerability in the framework you use, then you're sunk. To prevent this, use widely-used frameworks and hope for the best.

Protecting against hacking is like trying to protect an underwater bunker full of sodium. There are many places which you think need patching up. You may patch them all up. But that doesn't mean you're safe -- all it takes is a small leak in a place you didn't inspect for everything to come crashing down.

Answer 8

To "defend properly" means: "my code does not have bugs; none of the tools I use either". But software has bugs; hardware has bugs; nobody knows how to guarantee inexistence of bugs in a non-trivial piece of software or hardware (those who claim the contrary are delusional or lying, or both). See this previous question for some discussion.

The practical way attacks occur is when an attacker discovers or learns of a bug which can be exploited to his advantage, and before the system owner learns of it and fixes it.

From your list:

• A big, fat, random, unguessable admin password is fine -- as long as the machine on which the administrator types his password is not infected with some keylogger malware.

• Monitoring mostly helps for post mortem analysis; it does not prevent the attack, but it helps you determining what has happened, what bug was exploited, so that it can be fixed... for the next time.

• There is no such thing as "guaranteed safe user input". There is "user input which has been verified to conform to a specific set of rules, which should mean that the said user input may be used in a given construction without encountering inconvenient situations". For instance, a piece of text can be "safe for inclusion in a HTML file" (it does not contain some HTML-special characters which trigger advanced processing, such as '<' and '&'), but not safe at all for inclusion in a SQL query (the HTML-safe characters could encode a totally non-safe SQL query).

Answer 9

Just to add in a few points:

Make the admin account password long and complicated enough (i.e. theoretically speaking, password cannot be cracked within reasonable time).

Long and complicated passwords while good, increase the likelihood that they get written down. Same with passwords that must be changed often. Especially if more than one person needs the password, or it's a password that is not used often.

Monitor all incoming network traffic to the administrative files.

Privilege escalation attacks, will make that pointless, as changes will then seem to be coming from local traffic, not network. And too much monitoring will open up new issues. Buffer overflow Exploits are common. Heck, some consumer routers still choke on stalled traffic attempts clogging up it's connection tables.

To extend the layer of protection from #2 above, monitor local file changes (especially ones that have commands that require sudo privileges).

Better, but privilege escalation attacks could allow someone to upload a binary, and add suid/sgid sticky bits to it. It won't be on your list of files to check.

Validate all user input, so that all of the user input is guaranteed to be safe.

How much effort will you put into that? Sanitizing the user input is never 100% effective.

Basically, the point is that some things you do will cause more issues than not doing them, there are always ways around it, just try to do your best and hope you don't make a boneheaded mistake. Be like a disinfectant. You can only kill 99% of the problem.

"It is possible to commit no mistakes and still lose. That is not a weakness, that is life." - Picard

Answer 10

You have not defined what it means for your server to be hacked.

The first step is to define what is the scope of the legitimate access to the system. Who are the users and what does the system provide to the users which requires to be protected?

If the system is dedicated to a very specific, limited function (like serving a particular distributed application), and attackers can only access that system through the network, then pretty much just that application has to be secure, plus the network-exposed parts of the system: the LAN adapter, its driver and the network stack.

The real challenge in security in an OS like Linux is how to make the OS unhackable in a multi-user situation when people have almost full use of the system they can log in to operating system accounts and run applications directly.

A system can have security incidents without the superuser account being compromised. Suppose you have a perfectly uncrackable Linux system. Yet, user joe installs an insecure application /home/joe/bin. That application opens a malicious document, which exploits a flaw in the application, by means of which malicious code gains execution of arbitrary code in the security context of joe. The malicious code damages joe's data, and also steals CPU time from the machine. (Try as it might, though, it cannot get superuser execution privileges, because the machine is "unhackable").

Is that hacked? Or not hacked? How about from the perspective of user joe. Does joe care that root was never compromised in the incident?

Answer 11

If people (or at least your administrators) never made mistakes and computer programs were all 100% bug free all the time, then in theory you are correct. Unfortunately however, neither of these is ever true and it is easy enough for an attacker to attack at these points.

Even perfectly configured, the software is incapable of acting as it is configured all the time because of bugs. Under the right conditions, it may be possible to make the software do something it shouldn't.

Similarly, social engineering is probably the most common form of hacking. There is no need for complex technical measures when you can simply trick your opponent in to giving you what you want. Shoulder surfing the password, getting a key-logger on an a computer the administrator uses or tricking an administrator run your malicious software are all possible ways to sneak by under the radar.

You can compensate for a lot of this by locking a server in a room with no Internet connection and only allowing administrators to use the system while in the room with a guard at the door, and in fact, this is how many high security government networks operate, but it puts a huge barrier to usability.

Ultimately security is all about balancing usability and risk. The steps you outline require too much reduction in usability in order to be implemented securely and thus can't work by themselves in a real world situation. Security is a complex and always changing field that requires staying up to date on threats, checking your defenses against new threats and being sure to monitor for compromises that may get by as 0 day issues.

Answer 12

I've been trying to build bulletproof systems for almost ten years now. I had to learn about high assurance design techniques (e.g. EAL7), covert channels, subversion attacks, etc. Honestly, most system designers or administrators have neither the knowledge nor resources to totally secure a machine from known attacks. It's nearly impossible given business constraints. Most security decisions reflect tradeoffs between cost, features, usability, legacy compatibility, likely attacker profiles, etc.

If you want to know, here's one of my simplified breakdowns on the various levels and issues of security. (And it isn't the size of a book 8). It was addressed to a guy who thought top notch coders could make secure programs, but you can sub admin for coder and still same results. Have fun and learn the lessons less you repeat history like much of INFOSEC community does. ;)

http://www.schneier.com/blog/archives/2013/01/essay_on_fbi-ma.html#c1102869

This follow up comment has plenty of links that trace history, government factors, development requirements, etc. I also list a ton of exemplar systems and processes. Each might have flaws lurking but inspire more confidence than average product.

http://www.schneier.com/blog/archives/2013/01/essay_on_fbi-ma.html#c1105156

Answer 13

A system is only as strong as the weakest link

This is a phrase often repeated and you should remember that no matter how strong a password is - it's a single point of failure and a bruteforcer will be able to break through at some point.

Also, what if your sysadmin decided to accept a rather nice pay-off to accidently delete a load of files?

The germans thought the enigma was uncrackable and it's been argued that their reliance and trust in this single method of encryption cost them the war - over reliance on a single system or resource (human or otherwise) is the worst possible crime against security.

Answer 14

There's a few human aspects that aren't usually presented when something like this is asked.

1. Some people literally have nothing better to do and will try to hack your system out of sheer boredom.
2. Some people do it for fun (like a few white hats). White hats usually aren't bad because a lot of them will contact you and say "dude, I got into your system" and don't do anything malicious.
3. Some people out there will try to get your users' passwords in order to try them on other websites, like credit card ones. They pose to gain a huge profit from doing this
4. Some people will attack a server for revenge. "Jane Berman said that I'm a poopy head? HER SERVER SHALL GO DOWN!"

People especially in 2-4 tend to be incredibly passionate about getting into your system. Sometimes #1 turns into 2-4. Companies have to pay someone to secure a server, and it takes a lot of money, research, and man hours to prevent an army of these kinds of people from attacking it

Answer 15

Maybe because most systems and software are not security aware in the first place. Security is fastidious to teach, understand, and also to be updated about (news, new attack vectors, etc).

That's why it's not done entirely and extensively taught. For example, there are no security norm like there are ISO norms. Also because it's a potential market for domestic security (antivirus).

When computers will be more common, maybe one day it will become something that will be taken care of. Some future where there are something like 100 or more computers per human being.

If you really think about it, we don't really rely on computers enough to worry about internet security (at least I think, or at least the parts where it's critical is being taken good care of).