High Load Alert

system · December 27, 2004, 6:05am

I’ve seen a few posts regarding this recently here in your forums. Compared to Windows, my Linux bag of tricks is a bit shallow.

I’m beginning to get pretty regular emails conatining:

1 minute: 16.07
5 minute: 13.49
15 minute: 11.42

from the box the last 2 days. Obviously, I’d like to nip this in the bud as the box is becoming slower and slower to respond. I’ve not added any new accounts to the box in the last couple of months, so it’s not due to any new known load.

I’ve run “top” and am not really sure what to look for. Another set of eyes would be welcome.

Ideas?

JB

system · December 27, 2004, 1:26pm

First of all, what are you running on the box? Any forums or CMS’s that are using a lot of processor power?

Check to make sure you haven’t been hacked.

Do you have a swap partitiion? Is it large enough?

Tim

system · December 28, 2004, 10:08am

Looks like it might have been a hack job.

I did a quick and dirty configuration of the APF firewall after noticing some unwanted IRCD “established” connections (… which I don’t run).

POW. CPU utilization dropped from the constant 100% for the last 3 days back down to the expected 10 - 20%.

FYI – I did enable the blocking of all egress (outgoing) connections from the box as well, so that if a script was running from the inside, it simply couldn’t connect to anything.

JB

IWorx-Chris · December 31, 2004, 4:49pm

This has been reported a bunch due to the recent Santy phpbb worm. Egress filtering will block the connections but you’ll need to kill any running procs as well and upgrade any old phpbb’s on the system to make sure the attacks don’t continue.

Check out this thread as well: http://interworx.info/forums/showthread.php?t=341&highlight=hack

Chris

system · January 5, 2005, 4:37am

Thanks for the heads-up. Of course I’ve got clients who haven’t upgraded their installations yet … still running 2.0.10 :mad:

It seems I’ve stopped the CPU usage, but 2 days ago, the associated traffic came back – even with all the non-essential ports closed in and out.

Might anyone have a quick listing of what procs to look for? I’m getting much more familiar with spelunking around the Linux environment, but nothing is a quick-fix for me yet. Any time someone can save me would be helpful.

JB

system · January 6, 2005, 4:32am

Here’s more of what I’ve done, but the box is still exhibiting the traffic –

I found LOADS of .txt files in /tmp. None of them named anything useful like worm or bot, but all .php-looking script containing wording that it was for the anti-sanity worm. Mmhmm. Deleted.

Still running APF. Blocking everything in and out that’s not web/iworx related.

Chmodded wget to 700.

“Installed” mod_security – I think. Not sure I’ve done that exactly right, though, I do think I grabbed the right version. I’m guessing rpm -i isn’t the only thing that needed to happen. Also, not exactly sure if the default security.conf you all have in your rpm for the iworx/rh9 flavor takes any additional tweaking.

As far as any rouge processes, frankly, it’s hard for me to tell. These green linux eyes don’t often know what shouldn’t be there. I do like the quick learning curve I’m getting troubleshooting, however.

If anyone has any additional steps I might try, I’m all ears.

JB

system · January 6, 2005, 4:51am

Hello,

It seems I’ve stopped the CPU usage, but 2 days ago, the associated traffic came back – even with all the non-essential ports closed in and out.

I’m not sure to understand well, sorry.

Do you mean your CPU usage seems to be fine, but you always encountered some CPU peeks ?

Anyway, here is what you may look at when your cpu usage and more your load avergae go up

First do “top”
Look at the jobs that takes a lot of cpu. They should be in first in the top list
Look for the PID number.

It maybe a perl script or something like this. You may also have a look at :
ps -axf or ps -axuw
you could look at cpu usage and memory usage of every processes running.

if the process is not a standard process you may kill it : kill -9 pid (where pid is the number process id)

You may look at log messages in /var/log/, as for example /var/log/messages, /var/log/secure/, /var/log/httpd/error_log

Maybe try to unset all processes not necessary from booting autostart for your init level in /etc/rc.d/rcX.d/ (where X is your init level : generaly 3)

You may for example disable all NFS, X, printer, RPC… processes you not need.
(check http://interworx.info/forums/showthread.php?t=353)

Generaly when my CPU usage and load average are very high is due to a non KILLUP perl pgm. So I find the PID with top and kill the process with kill -9 pid.

Php processes also give me sometimes a high cpu/loadaverage.
If you look at top, and wait until php(s) process(es) begin you’ll see that the cpu going high and the load average too.

now I have any solutions to figure out which user or which exactly script is responsible of this.

On a box with a lot of dynamic web site, you need a lot of ram, a good cpu, and fast HDD, it is less important on a box with static web site.

you may try to tweak your apache config with tools as AB, flood or httperf

Not sure my post will answer your question, and will help you

Try to give more details if you still have a pbm

Good luck

Pascal

system · January 6, 2005, 6:09am

Firstly – Thx Pascal for stepping in. Always good to have your input.

Do you mean your CPU usage seems to be fine, but you always encountered some CPU peeks ?

My CPU load average was only high last week. It was discerned that we probably had a phpBB hack somewhere on the box. After tightening up the firewall, the CPU load dropped to normal and has been fine ever since. Beginning Monday (2 days later), the network traffic spiked from a constant 2% (currently normal) to 20% (high) and has maintained that rate.

Maybe try to unset all processes not necessary from booting autostart for your init level in /etc/rc.d/rcX.d/ (where X is your init level : generaly 3)

Certainly a great idea, but I’m more concerned (at this point) to quickly kill the processes added by the hack. I’m not familiar enough with the standard processes (yet) to recognize a rogue one. Thoughts?

now I have any solutions to figure out which user or which exactly script is responsible of this.

Now, THAT would be great …

I’ll see if I can make any sense of anything in the logs early this afternoon. Again, thanks for your help, and if you have any other thoughts, I’ll take them.

JB

system · January 6, 2005, 6:55am

Hello,

Ok you speak about Network traffic, not cpu usage.

There is a lot of scanner that try to connect to your box or does DOS attack, etc …
Maybe the traffic come from this sort of pbm.

I think you’ve installed APF ?

If yes, Just look at your /var/log/messages and /var/log/secure and try to figure out if there is a lot of DROP packets.

I recommand you to also install BFD (brut force detection) from rxfn.

And finally you asked about mod_security.

So here is my rule set.
I have configured a file in /etc/httpd/conf.d/ named mod_security.conf

here is the file

mod_security

LoadModule security_module modules/mod_security.so

<IfModule mod_security.c>
# Turn the filtering engine On or Off
SecFilterEngine On

# Change Server: string
SecServerSignature " "

# Make sure that URL encoding is valid
SecFilterCheckURLEncoding On

# This setting should be set to On only if the Web site is
# using the Unicode encoding. Otherwise it may interfere with
# the normal Web site operation.
SecFilterCheckUnicodeEncoding Off

# Only allow bytes from this range
SecFilterForceByteRange 1 255

# The audit engine works independently and
# can be turned On of Off on the per-server or
# on the per-directory basis. "On" will log everything,
# "DynamicOrRelevant" will log dynamic requests or violations,
# and "RelevantOnly" will only log policy violations
SecAuditEngine RelevantOnly

# The name of the audit log file
SecAuditLog /var/log/httpd/audit_log

# Should mod_security inspect POST payloads
SecFilterScanPOST On

# Action to take by default
SecFilterDefaultAction "deny,log,status:500"

# Require HTTP_USER_AGENT and HTTP_HOST in all requests
SecFilterSelective "HTTP_USER_AGENT|HTTP_HOST" "^$"

# Weaker XSS protection but allows common HTML tags
SecFilter "&lt;[[:space:]]*script"

# Very crude filters to prevent SQL injection attacks
SecFilter "delete[[:space:]]+from"
SecFilter "insert[[:space:]]+into"

# Protecting from XSS attacks through the PHP session cookie
SecFilterSelective ARG_PHPSESSID "!^[0-9a-z]*$"
SecFilterSelective COOKIE_PHPSESSID "!^[0-9a-z]*$"

# Import our snort converted modsec rules

Include /etc/httpd/conf.d/mod_sec.snort.conf

</IfModule>

You may see that I have an include command which one include an other file (: mod_sec.snort.conf) to complete the rules

Why an other file ?
It easily allow me to remove these rules when my server is very busy

The mod_sec.snort.conf is my dedicated and personnal file from the snort rules

here is the file : mod_sec.snort.conf

WEB-ATTACKS ps command attempt

SecFilterSelective THE_REQUEST “/bin/ps”

WEB-ATTACKS /bin/ps command attempt

SecFilterSelective THE_REQUEST “ps\x20”

WEB-ATTACKS wget command attempt

SecFilter “wget\x20”

WEB-ATTACKS uname -a command attempt

SecFilter “uname\x20-a”

WEB-ATTACKS /usr/bin/id command attempt

SecFilterSelective THE_REQUEST “/usr/bin/id”

WEB-ATTACKS id command attempt

SecFilter “;id”

WEB-ATTACKS kill command attempt

SecFilterSelective THE_REQUEST “/bin/kill”

WEB-ATTACKS chsh command attempt

SecFilterSelective THE_REQUEST “/usr/bin/chsh”

WEB-ATTACKS tftp command attempt

SecFilter “tftp\x20”

WEB-ATTACKS /usr/bin/gcc command attempt

SecFilterSelective THE_REQUEST “/usr/bin/gcc”

WEB-ATTACKS gcc command attempt

SecFilter “gcc\x20-o”

WEB-ATTACKS /usr/bin/cc command attempt

SecFilterSelective THE_REQUEST “/usr/bin/cc”

WEB-ATTACKS cc command attempt

SecFilter “cc\x20”

WEB-ATTACKS /usr/bin/cpp command attempt

SecFilterSelective THE_REQUEST “/usr/bin/cpp”

WEB-ATTACKS cpp command attempt

SecFilter “cpp\x20”

WEB-ATTACKS /usr/bin/g++ command attempt

SecFilterSelective THE_REQUEST “/usr/bin/g++”

WEB-ATTACKS g++ command attempt

SecFilter “g++\x20”

WEB-ATTACKS bin/python access attempt

SecFilterSelective THE_REQUEST “bin/python”

WEB-ATTACKS python access attempt

SecFilter “python\x20”

WEB-ATTACKS bin/tclsh execution attempt

SecFilter “bin/tclsh”

WEB-ATTACKS tclsh execution attempt

SecFilter “tclsh8\x20”

WEB-ATTACKS bin/nasm command attempt

SecFilterSelective THE_REQUEST “bin/nasm”

WEB-ATTACKS nasm command attempt

SecFilter “nasm\x20”

WEB-ATTACKS perl execution attempt

SecFilter “perl\x20”

WEB-ATTACKS traceroute command attempt

SecFilter “traceroute\x20”

WEB-ATTACKS ping command attempt

SecFilterSelective THE_REQUEST “/bin/ping”

WEB-ATTACKS netcat command attempt

SecFilter “nc\x20”

WEB-ATTACKS nmap command attempt

SecFilter “nmap\x20”

WEB-ATTACKS xterm command attempt

SecFilterSelective THE_REQUEST “/usr/X11R6/bin/xterm”

WEB-ATTACKS X application to remote host attempt

SecFilter “\x20-display\x20”

WEB-ATTACKS lsof command attempt

SecFilter “lsof\x20”

WEB-ATTACKS rm command attempt

SecFilter “rm\x20”

WEB-ATTACKS mail command attempt

SecFilterSelective THE_REQUEST “/bin/mail”

WEB-ATTACKS mail command attempt

SecFilter “mail\x20”

WEB-ATTACKS /bin/ls command attempt

SecFilterSelective THE_REQUEST “/bin/ls”

WEB-ATTACKS /etc/shadow access

SecFilter “/etc/shadow”

WEB-ATTACKS .htgroup access

SecFilterSelective THE_REQUEST “.htgroup”

WEB-CGI websitepro path access

SecFilter " /HTTP/1."

WEB-CGI formmail arbitrary command execution attempt

SecFilterSelective THE_REQUEST “/formmail” chain
SecFilter “\x0a”

WEB-CGI formmail access

SecFilterSelective THE_REQUEST “/formmail” log,pass

WEB-CGI phf arbitrary command execution attempt

SecFilterSelective THE_REQUEST “/phf” chain
SecFilter “\x0a/”

WEB-CGI phf access

SecFilterSelective THE_REQUEST “/phf” log,pass

WEB-CGI rksh access

SecFilterSelective THE_REQUEST “/rksh”

WEB-CGI bash access

SecFilterSelective THE_REQUEST “/bash” log,pass

WEB-CGI zsh access

SecFilterSelective THE_REQUEST “/zsh”

WEB-CGI csh access

SecFilterSelective THE_REQUEST “/csh”

WEB-CGI tcsh access

SecFilterSelective THE_REQUEST “/tcsh”

WEB-CGI rsh access

SecFilterSelective THE_REQUEST “/rsh”

WEB-CGI ksh access

SecFilterSelective THE_REQUEST “/ksh”

WEB-CLIENT Javascript URL host spoofing attempt

SecFilter “javascript://”

WEB-MISC cross site scripting (img src=javascript) attempt

SecFilter “img src=javascript”

WEB-MISC .htpasswd access

SecFilter “.htpasswd”

WEB-MISC .htaccess access

SecFilter “.htaccess”

WEB-MISC cd…

SecFilter “cd..”

WEB-MISC ///cgi-bin access

SecFilterSelective THE_REQUEST “///cgi-bin”

WEB-MISC /cgi-bin/// access

SecFilterSelective THE_REQUEST “/cgi-bin///”

WEB-MISC /~root access

SecFilterSelective THE_REQUEST “/~root”

WEB-MISC /~ftp access

SecFilterSelective THE_REQUEST “/~ftp”

WEB-MISC cat%20 access

SecFilter “cat\x20”

WEB-MISC rpm_query access

SecFilterSelective THE_REQUEST “/rpm_query”

WEB-MISC htgrep attempt

SecFilterSelective THE_REQUEST “/htgrep” chain
SecFilter “hdr=/”

WEB-MISC htgrep access

SecFilterSelective THE_REQUEST “/htgrep” log,pass

WEB-MISC .history access

SecFilterSelective THE_REQUEST “/.history”

WEB-MISC .bash_history access

SecFilterSelective THE_REQUEST “/.bash_history”

WEB-MISC /~nobody access

SecFilterSelective THE_REQUEST “/~nobody”

WEB-MISC *%0a.pl access

SecFilterSelective THE_REQUEST “/*\x0a.pl”

WEB-MISC Apache Chunked-Encoding worm attempt

SecFilter “CCCCCCC: AAAAAAAAAAAAAAAAAAA”

WEB-MISC Transfer-Encoding: chunked

SecFilter “chunked”

WEB-PHP squirrel mail theme arbitrary command attempt

SecFilterSelective THE_REQUEST “/left_main.php” chain
SecFilter “cmdd=”

WEB-PHP DNSTools administrator authentication bypass attempt

SecFilterSelective THE_REQUEST “/dnstools.php” chain
SecFilter “user_dnstools_administrator=true”

WEB-PHP DNSTools authentication bypass attempt

SecFilterSelective THE_REQUEST “/dnstools.php” chain
SecFilter “user_logged_in=true”

WEB-PHP DNSTools access

SecFilterSelective THE_REQUEST “/dnstools.php” log,pass

WEB-PHP Blahz-DNS dostuff.php modify user attempt

SecFilterSelective THE_REQUEST “/dostuff.php?action=modify_user”

WEB-PHP Blahz-DNS dostuff.php access

SecFilterSelective THE_REQUEST “/dostuff.php” log,pass

WEB-PHP PHP-Wiki cross site scripting attempt

SecFilterSelective THE_REQUEST “<script”

WEB-PHP strings overflow

SecFilterSelective THE_REQUEST “?STRENGUR”

WEB-PHP PHPLIB remote command attempt

SecFilter “_PHPLIB[libdir]”

to create these files do :
first save your old security.conf file

cd /etc/httpd/conf.d/
cp security.conf security.back
rm -f security.conf

then create the mod_security.conf file and copy paste mine

touch mod_security.conf
vi mod_securtity.conf
press “Inser”
copy and paste my data

finally create the file mod_sec.snort.conf and copy and paste my data

touch mod_sec.snort.conf
vi mod_sec.snort.conf
press “Inser”
copy and paste my data

Hope this will help you

Pascal

system · January 6, 2005, 1:17pm

Thanks again Paschal for the LOADS of info. Here’s my 1st problem:

I downloaded the mod_security rpm from interworx in the iworx/rh9 location. At the time I only did a “rpm -i <package>” – I’m not sure that’s all I need to do to get mod_security to be enabled. Therefore, I don’t think it’s doing anything yet. How can I verify/fix that?

Once that happens, I’ll try your configs (and/or versions of them) and post the results.

JB

IWorx-Chris · January 6, 2005, 10:00pm

Verify that there’s a /etc/httpd/conf.d/security.conf file. If it’s there then you are running mod_security. You can also enable server-info in the main httpd.conf to view the loaded modules while apache is running.

Chris

system · January 7, 2005, 6:02am

Thx Chris for stepping in – I can use the help. I’m glad a few of you aren’t afraid to “lead this horse to water” … I promise, I’ll drink once I’m there.

After checking, I do have the security.conf file – and a quick phpinfo () shows the mod_security as loaded (… after enablilng ExtendedStatus). So far, my security.conf is the same as the one that ships in the .rpm courtesy of iworx.

Here’s the problem I’m dealing with:

http://www.battledomain.com/phpstuff/monitoring.php.png

Is is probable that mod_security can’t catch this sort of thing – or do I not have a certain feature of mod_security enabled to do so? I’m imagining I’ve got a rogue process lying around that I’m just not catching.

If anyone wants to take a look at the following and throw a guy a bone, I’d take it:

[root@web1 conf]# ps -axf
  PID TTY      STAT   TIME COMMAND
    1 ?        S      0:03 init
    2 ?        SW     0:00 [keventd]
    3 ?        SW     0:00 [kapmd]
    4 ?        SWN    0:01 [ksoftirqd_CPU0]
    9 ?        SW     0:00 [bdflush]
    5 ?        SW     0:01 [kswapd]
    6 ?        SW     0:00 [kscand/DMA]
    7 ?        SW     1:56 [kscand/Normal]
    8 ?        SW     0:00 [kscand/HighMem]
   10 ?        SW     0:00 [kupdated]
   11 ?        SW     0:00 [mdrecoveryd]
   15 ?        SW     0:08 [kjournald]
   73 ?        SW     0:00 [khubd]
 2261 ?        SW     0:00 [kjournald]
 2350 tty8     S      0:00 /bin/bash
 2474 ?        SW     0:00 [eth0]
 2667 ?        S      0:01 syslogd -m 0
 2671 ?        S      0:00 klogd -x
 2707 ?        S      0:00 /usr/sbin/apmd -p 10 -w 5 -W -P /etc/sysconfig/apm-sc
 3550 ?        S      0:00 /usr/sbin/sshd
26628 ?        S      0:00  \_ /usr/sbin/sshd
26654 ?        S      0:00      \_ /usr/sbin/sshd
26655 pts/0    S      0:00          \_ -bash
26689 pts/0    S      0:00              \_ su
26690 pts/0    S      0:04                  \_ bash
  513 pts/0    R      0:00                      \_ ps -axf
 3561 ?        S      0:00 xinetd -stayalive -pidfile /var/run/xinetd.pid
 3573 ?        SL     0:00 ntpd -U ntp -g
 3586 ?        S      0:00 /bin/sh /usr/bin/mysqld_safe --defaults-file=/etc/my.
 3621 ?        S      0:09  \_ /usr/sbin/mysqld --defaults-file=/etc/my.cnf --ba
 3632 ?        S      0:00 /bin/sh /home/interworx/bin/mysql/mysqld_safe --defau
 3670 ?        S      0:09  \_ /home/interworx/bin/iworx-db --defaults-file=/hom
 3676 ?        S      0:00 /home/interworx/bin/iworx-web -f /home/interworx/etc/
 3703 ?        S      0:11  \_ /home/interworx/bin/iworx-web -f /home/interworx/
 3704 ?        S      0:16  \_ /home/interworx/bin/iworx-web -f /home/interworx/
 3705 ?        S      0:15  \_ /home/interworx/bin/iworx-web -f /home/interworx/
 3706 ?        S      0:15  \_ /home/interworx/bin/iworx-web -f /home/interworx/
 3710 ?        S      2:47  \_ /home/interworx/bin/iworx-web -f /home/interworx/
 3711 ?        S      0:14  \_ /home/interworx/bin/iworx-web -f /home/interworx/
 3712 ?        S      0:20  \_ /home/interworx/bin/iworx-web -f /home/interworx/
 3713 ?        S      0:16  \_ /home/interworx/bin/iworx-web -f /home/interworx/
 3702 ?        S      0:00 crond
 3728 ?        S      0:00 /usr/sbin/atd
 3752 tty1     S      0:00 /sbin/mingetty tty1
 3753 tty2     S      0:00 /sbin/mingetty tty2
 3754 tty3     S      0:00 /sbin/mingetty tty3
 3755 tty4     S      0:00 /sbin/mingetty tty4
 3756 tty5     S      0:00 /sbin/mingetty tty5
 3757 tty6     S      0:00 /sbin/mingetty tty6
 3758 ?        S      0:00 /bin/sh /usr/bin/svscanboot
 3760 ?        S      0:00  \_ svscan /service
 3762 ?        S      0:00  |   \_ supervise send
 3808 ?        S      0:00  |   |   \_ qmail-send
 3822 ?        S      0:00  |   |       \_ qmail-lspawn ./Maildir/
 3823 ?        S      0:00  |   |       \_ qmail-rspawn
 3824 ?        S      0:00  |   |       \_ qmail-clean
 3763 ?        S      0:00  |   \_ supervise log
 3800 ?        S      0:00  |   |   \_ /usr/bin/multilog t /var/log/send
 3764 ?        S      0:00  |   \_ supervise smtp
 3779 ?        S      0:00  |   |   \_ /usr/bin/tcpserver -v -R -S -p -x /etc/tc
 3765 ?        S      0:00  |   \_ supervise log
 3801 ?        S      0:00  |   |   \_ /usr/bin/multilog t /var/log/smtp
 3766 ?        S      0:00  |   \_ supervise imap4
 3839 ?        S      0:00  |   |   \_ /usr/bin/tcpserver -v -R -H -l web1.battl
 3767 ?        S      0:00  |   \_ supervise log
 3802 ?        S      0:00  |   |   \_ /usr/bin/multilog t /var/log/imap4
 3768 ?        S      0:00  |   \_ supervise imap4-ssl
 3805 ?        S      0:00  |   |   \_ /usr/bin/tcpserver -v -R -H -l web1.battl
 3769 ?        S      0:00  |   \_ supervise log
 3806 ?        S      0:00  |   |   \_ /usr/bin/multilog t /var/log/imap4-ssl
 3770 ?        S      0:00  |   \_ supervise pop3-ssl
 3810 ?        S      0:00  |   |   \_ /usr/bin/tcpserver -v -R -H -l web1.battl
 3771 ?        S      0:00  |   \_ supervise log
 3811 ?        S      0:00  |   |   \_ /usr/bin/multilog t /var/log/pop3-ssl
 3772 ?        S      0:00  |   \_ supervise pop3
 3812 ?        S      0:07  |   |   \_ /usr/bin/tcpserver -v -R -c 200 0 110 /va
  510 ?        S      0:00  |   |       \_ /var/qmail/bin/qmail-popup web1.battl
  511 ?        S      0:00  |   |           \_ /var/qmail/bin/qmail-pop3d Maildi
 3773 ?        S      0:00  |   \_ supervise log
 3840 ?        S      0:06  |   |   \_ /usr/bin/multilog t /var/log/pop3
 3774 ?        S      0:00  |   \_ supervise dnscache
 3813 ?        S      0:11  |   |   \_ /usr/bin/dnscache
 3775 ?        S      0:00  |   \_ supervise log
 3814 ?        S      0:04  |   |   \_ multilog t ./main
 3776 ?        S      0:00  |   \_ supervise tinydns-xx.xx.xx.xx
28445 ?        S      0:00  |   |   \_ /usr/bin/tinydns
 3777 ?        S      0:00  |   \_ supervise log
 3841 ?        S      0:00  |   |   \_ multilog t ./main
 3778 ?        S      0:00  |   \_ supervise tinydns-xx.xx.xx.xx
28447 ?        S      0:00  |   |   \_ /usr/bin/tinydns
 3780 ?        S      0:00  |   \_ supervise log
 3826 ?        S      0:00  |   |   \_ multilog t ./main
 3781 ?        S      0:00  |   \_ supervise tinydns-xx.xx.xx.xx
28449 ?        S      0:00  |   |   \_ /usr/bin/tinydns
 3782 ?        S      0:00  |   \_ supervise log
 3842 ?        S      0:00  |   |   \_ multilog t ./main
 3783 ?        S      0:00  |   \_ supervise tinydns-xx.xx.xx.xx
28451 ?        S      0:00  |   |   \_ /usr/bin/tinydns
 3784 ?        S      0:00  |   \_ supervise log
 3829 ?        S      0:00  |   |   \_ multilog t ./main
 3785 ?        S      0:00  |   \_ supervise tinydns-xx.xx.xx.xx
28453 ?        S      0:00  |   |   \_ /usr/bin/tinydns
 3786 ?        S      0:00  |   \_ supervise log
 3831 ?        S      0:00  |   |   \_ multilog t ./main
 3787 ?        S      0:00  |   \_ supervise tinydns-xx.xx.xx.xx
28455 ?        S      0:00  |   |   \_ /usr/bin/tinydns
 3788 ?        S      0:00  |   \_ supervise log
 3833 ?        S      0:00  |   |   \_ multilog t ./main
 3789 ?        S      0:00  |   \_ supervise tinydns-xx.xx.xx.xx
28457 ?        S      0:00  |   |   \_ /usr/bin/tinydns
 3790 ?        S      0:00  |   \_ supervise log
 3835 ?        S      0:00  |   |   \_ multilog t ./main
 3791 ?        S      0:00  |   \_ supervise tinydns-xx.xx.xx.xx
28459 ?        S      0:00  |   |   \_ /usr/bin/tinydns
 3792 ?        S      0:00  |   \_ supervise log
 3845 ?        S      0:00  |   |   \_ multilog t ./main
 3793 ?        S      0:00  |   \_ supervise tinydns-xx.xx.xx.xx
28461 ?        S      0:00  |   |   \_ /usr/bin/tinydns
 3794 ?        S      0:00  |   \_ supervise log
 3846 ?        S      0:00  |   |   \_ multilog t ./main
 3795 ?        S      0:00  |   \_ supervise tinydns-xx.xx.xx.xx
28463 ?        S      0:00  |   |   \_ /usr/bin/tinydns
 3796 ?        S      0:00  |   \_ supervise log
 3838 ?        S      0:00  |       \_ multilog t ./main
 3761 ?        S      0:00  \_ readproctitle service errors: ...................
12731 ?        S      0:00 proftpd: (accepting connections)
27641 ?        S      0:00 /usr/sbin/httpd -DSSL
27644 ?        S      0:01  \_ /usr/sbin/httpd -DSSL
27646 ?        S      0:00  \_ /usr/sbin/httpd -DSSL
27647 ?        S      0:00  \_ /usr/sbin/httpd -DSSL
27648 ?        S      0:00  \_ /usr/sbin/httpd -DSSL
27651 ?        S      0:01  \_ /usr/sbin/httpd -DSSL
28893 ?        S      0:00  \_ /usr/sbin/httpd -DSSL
28949 ?        S      0:00  \_ /usr/sbin/httpd -DSSL
[root@web1 conf]#

Thanks again in advance.

JB

system · January 7, 2005, 6:44am

Hello,

First, there is nothing bad in your ps -axf. Maybe the ntpd (network time protocol - The ntpd program operates by exchanging messages with one or more configured servers at designated poll intervals) should be stop, except if you need it.

Mod_security has nothing to do with your high outgoing traffic. But you mat check the mod_security audit log in /var/log/httpd/audit_log (tail -n100 /var/log/httpd/audit_log)
also you may add this security filter concerning the last phpbb virus attack : secfilter “wget\x20” somewhere in you security.conf file

I’ll more check if there is not a new domain that is consumming the outgoing bandwidth.

Do you have only one domain on this box ? maybe few ? maybe you have a new client who have a lot of visitors.

I’ll check in nodeworx the bandwidth used per domain.
Login into nodeworx, click on “siteworx account” then check the field “bandwidth use” for every domains you host.

Also check all relevant logs in /var/log/messages and /var/log/secure

Hope this will help

Pascal

system · January 7, 2005, 10:00am

Thanks again Paschal, you’re being quite generous with your time.

Well, this sure is a good “hunt.” I’ve made a few more adjustments which I’ll post here – none of which have solved the issue yet. I promise I’ll post a “10 Condensed Steps on How I solved this Hack” sticky-able thread once I’ve conquered it.

What I’ve done since last post:

Tweaked mod_security a bit to include a) SecFilterEngine “On” instead of “DynamicOnly”. b) filters for XSS php Cookie attacks and c) an include list of converted modsec rules, similar to your snort ones.
Checked the mod_security audit_log and found no entries (none, zero, zilch, nada – even after running for a day or so now.)
Added the brute-force detection – which is working and denying via APF successfully.
Checked all my SiteWorx accounts, of which there are only a handful. Most are either my own, or a few select (trusted) friends. None of them reflect this bandwidth useage.

If there’s any other place you can think of I can dig, I’d appreciate it.

JB

system · January 7, 2005, 10:41am

Oh my goodness … I think I may have found it … give me a few to verify, then I’ll come back and (slightly) embarass myself.

JB

system · January 9, 2005, 8:45am

The update:

First of all, let me thank Paschal and others for providing all the info. It did allow me to put in place a few extra (and much needed) prevention measures (e.g. mod_security, bruteForce attack and APF). They also did aid me in bandaid-ing the phpBB exploit.

My most recent issue involving the bandwidth (coming right on the heels of the phpBB issue), turned out to be an endlessly looping pop3 message, caught in a cycle. I’ve since deleted the message server-side, and my bandwidth returned to normal.

I’m left with the remnants of the phpBB issue, however. Aside of getting all of my hosted phpBB sites upgraded, I firewalled all the Outgoing (egress) ports – except what’s needed for web/iworx functions. The problem is that my APF installation is logging many dropped packets on the way out of the box – which look as if they could be coming from some process left-over from the phpBB hack – attempting to make contact with the mother-ship.

What process? I have no idea – and that’s the rub. How can I be sure I’ve killed and removed what was changed/added by the worm? I’d like to be able to not filter egress ports, only inbound, as I’ve got some other web-scripts that would like to dynamically assign outgoing ports. Obviously, I can’t use them until I know I’m safe.

Thoughts?

JB

IWorx-Chris · January 9, 2005, 9:25am

Can you post the apf log entries that are suspicious JayBean?

Chris

system · January 9, 2005, 9:51am

Hey Chris – Certainly.

The ones that seem the most suspicious to me are when something appears to be port-hopping to find an exit. For example:


From 65.110.xx.xx - 51 packets (n.b. this is the source IP of the web box)
      To 66.23.xx.xx - 49 packets (n.b. In this instance, this happened to be *my* IP with which I was logged in via SSH .. !! )
         Service: 5601 (tcp/5601) (** OUT_TCP DROP **,none,eth0) - 1 packet
         Service: 5621 (tcp/5621) (** OUT_TCP DROP **,none,eth0) - 1 packet
         Service: 5653 (tcp/5653) (** OUT_TCP DROP **,none,eth0) - 1 packet
         Service: 5676 (tcp/5676) (** OUT_TCP DROP **,none,eth0) - 1 packet
         Service: 5698 (tcp/5698) (** OUT_TCP DROP **,none,eth0) - 1 packet
         Service: 5727 (tcp/5727) (** OUT_TCP DROP **,none,eth0) - 1 packet
         Service: 5752 (tcp/5752) (** OUT_TCP DROP **,none,eth0) - 1 packet
         Service: 5783 (tcp/5783) (** OUT_TCP DROP **,none,eth0) - 1 packet
         Service: 5811 (tcp/5811) (** OUT_TCP DROP **,none,eth0) - 1 packet
         Service: 5836 (tcp/5836) (** OUT_TCP DROP **,none,eth0) - 1 packet
         Service: 5855 (tcp/5855) (** OUT_TCP DROP **,none,eth0) - 1 packet
         Service: 5884 (tcp/5884) (** OUT_TCP DROP **,none,eth0) - 1 packet
         Service: 5917 (tcp/5917) (** OUT_TCP DROP **,none,eth0) - 1 packet
         Service: 5951 (tcp/5951) (** OUT_TCP DROP **,none,eth0) - 1 packet
         Service: 5984 (tcp/5984) (** OUT_TCP DROP **,none,eth0) - 1 packet
         Service: 6011 (tcp/6011) (** OUT_TCP DROP **,none,eth0) - 1 packet
         Service: 6039 (tcp/6039) (** OUT_TCP DROP **,none,eth0) - 1 packet
         Service: 6067 (tcp/6067) (** OUT_TCP DROP **,none,eth0) - 1 packet
         Service: 6092 (tcp/6092) (** OUT_TCP DROP **,none,eth0) - 1 packet
         Service: 6112 (tcp/6112) (** OUT_TCP DROP **,none,eth0) - 1 packet
         Service: 6141 (tcp/6141) (** OUT_TCP DROP **,none,eth0) - 1 packet
         Service: 6165 (tcp/6165) (** OUT_TCP DROP **,none,eth0) - 1 packet
         Service: 6184 (tcp/6184) (** OUT_TCP DROP **,none,eth0) - 1 packet
         Service: 6212 (tcp/6212) (** OUT_TCP DROP **,none,eth0) - 1 packet
         Service: 6236 (tcp/6236) (** OUT_TCP DROP **,none,eth0) - 1 packet
         Service: 6263 (tcp/6263) (** OUT_TCP DROP **,none,eth0) - 1 packet
         Service: 6292 (tcp/6292) (** OUT_TCP DROP **,none,eth0) - 1 packet
         Service: 6314 (tcp/6314) (** OUT_TCP DROP **,none,eth0) - 1 packet
         Service: 6336 (tcp/6336) (** OUT_TCP DROP **,none,eth0) - 1 packet
         Service: 6377 (tcp/6377) (** OUT_TCP DROP **,none,eth0) - 1 packet
         Service: 6398 (tcp/6398) (** OUT_TCP DROP **,none,eth0) - 1 packet
         Service: 6418 (tcp/6418) (** OUT_TCP DROP **,none,eth0) - 1 packet
         Service: 6447 (tcp/6447) (** OUT_TCP DROP **,none,eth0) - 1 packet
         Service: 6470 (tcp/6470) (** OUT_TCP DROP **,none,eth0) - 1 packet
         Service: 6492 (tcp/6492) (** OUT_TCP DROP **,none,eth0) - 1 packet
         Service: 6519 (tcp/6519) (** OUT_TCP DROP **,none,eth0) - 1 packet
         Service: 6541 (tcp/6541) (** OUT_TCP DROP **,none,eth0) - 1 packet
         Service: 6561 (tcp/6561) (** OUT_TCP DROP **,none,eth0) - 1 packet
         Service: 6600 (tcp/6600) (** OUT_TCP DROP **,none,eth0) - 1 packet
         Service: 6620 (tcp/6620) (** OUT_TCP DROP **,none,eth0) - 1 packet
         Service: 6648 (tcp/6648) (** OUT_TCP DROP **,none,eth0) - 1 packet
         Service: 6666 (tcp/6666) (** OUT_TCP DROP **,none,eth0) - 1 packet
         Service: 6687 (tcp/6687) (** OUT_TCP DROP **,none,eth0) - 1 packet
         Service: 6713 (tcp/6713) (** OUT_TCP DROP **,none,eth0) - 1 packet
         Service: 6732 (tcp/6732) (** OUT_TCP DROP **,none,eth0) - 1 packet
         Service: 6754 (tcp/6754) (** OUT_TCP DROP **,none,eth0) - 1 packet
         Service: 6780 (tcp/6780) (** OUT_TCP DROP **,none,eth0) - 1 packet
         Service: 6799 (tcp/6799) (** OUT_TCP DROP **,none,eth0) - 1 packet
         Service: 6824 (tcp/6824) (** OUT_TCP DROP **,none,eth0) - 1 packet

Then, the occasional:

   From 65.110.xx.xx - 2 packets (n.b. again, my web box -- diff IP)
      To 68.157.xx.xx - 2 packets (n.b. somewhere external, not me)
         Service: 50893 (tcp/50893) (** OUT_TCP DROP **,none,eth0) - 1 packet
         Service: 50924 (tcp/50924) (** OUT_TCP DROP **,none,eth0) - 1 packet

This is just a quick example of pieces from a particular day. I could provide more or different if it were helpful.

Thx again for helping.

JB

IWorx-Chris · January 9, 2005, 9:59am

Can you paste your current egress rules too for APF JayBean.

Chris

system · January 9, 2005, 11:10am

EGF=“1”

Common egress (outbound) TCP ports

EG_TCP_CPORTS=“21,25,80,443,43,51234,123,110,995,143,2080,2443,22,53,953,3306”