4

I'm experiencing a huge load on my server at the moment and I can't figure out why. When I use the 'top' command, there's hundreds of apache processes with the command "aux", but I can't find anything online that tells me what it means. The load is flapping between 50-150, which is a good 50-150 more than it usually is.

Netstat returns hundreds and hundreds of rows like this:

tcp  0  0 xxx.xxx.xxx.xxx:45216  61.155.202.205:80  CLOSE_WAIT  28863/aux

Almost all from 61.155.xxx.xxx (not sure if this is relevant information, but trying to give as much as possible).

The OS is CentOS: release 5.7 Final We just run LAMP stack on it with about 30 websites that don't get much load (or so I thought). I've checked the logs for all of the vHosts but none seem to be getting many/any requests (not nearly enough to cause this trouble). I'm not sure if there are other logs I should be checking?

It started a couple of days ago; no changes made on the server as far as I'm aware.

Does anyone have any ideas for how I can track down what's causing the huge spike in load? Are there other commands/logs that I've missed that might be able to help me track down what the problem is?

Khaled
  • 36,903
dKen
  • 193

2 Answers2

4

That's not a connection from 61.155.xxx.xxx. That's a connection to a webserver on 61.155.202.205.

It looks very much like your webserver is making HTTP requests to other webservers on ADSL connections in China. Try a tcpdump -n -A -s0 host 61.155.202.205 to see what kind of data you are collecting. I suspect it's malicious.

If it is malicious, refer to My server's been hacked! EMERGENCY.

The "many Apache processes" is most likely caused by the high load rather than causing the high load. Even at a load average of 50 I would expect to start seeing HTTP requests taking multiple seconds. At 150 it would be worse.

Community
  • 1
Ladadadada
  • 26,787
  • Thanks Ladadadada, I've tried running that command on a selection of the IP addresses, but they seem to be opening and closing connections so quickly I can't find one that's returning anything. I suspect we have been hacked though, so going through the link provided. – dKen Jun 14 '12 at 09:28
  • Ah, if you can't find individual IP addresses making requests, try a whole range: tcpdump -n -A -s0 net 61.155.202.0/24. Use a smaller range (say, /26 or /27) if you get too much data. – Ladadadada Jun 14 '12 at 09:31
  • Thanks Ladadadada, it looks like they're looping through IPs because they're slowly incrementing but I managed to preempt an IP range and got a lot of this:

    10:15:22.012238 IP xx.xx.xx.xx.56270 > 169.90.225.125.http: S 2429420269:2429420269(0) win 5840 <mss 1460,sackOK,timestamp 3753976813 0,nop,wscale 7> E..<!.@.@.. _.E@.Z.}...P....................... ..#.........

     – dKen Jun 14 '12 at 09:46
1

To possibly help anyone who hits the same thing, it was a Trojan (Trojan.Perl.Shellbot-2) that was causing the problem. Between the answer/comments here and on my other question at 398715, we did the following:

  • Installed and ran chkrootkit, but nothing found
  • Installed and ran clamav, which tracked down the name of the virus and where it was located
  • Searched for others with the same problem and found this post
  • Followed instructions on removing and cleaning up after the virus
  • Added apache to the cron.deny file and restarted crond

This is only part of the solution; the server still needs to be rebuilt after we track down where the vulnerability is, but this was a good start, and we got the server back up and running.

If anyone can think of anything I've missed, something I'm doing wrong or could do better, please let me know.

Community
  • 1
dKen
  • 193