High Load Alert

[JayBaen]

I've seen a few posts regarding this recently here in your forums. Compared to Windows, my Linux bag of tricks is a bit shallow.

I'm beginning to get pretty regular emails conatining:

1 minute: 16.07
5 minute: 13.49
15 minute: 11.42

from the box the last 2 days. Obviously, I'd like to nip this in the bud as the box is becoming slower and slower to respond. I've *not* added any new accounts to the box in the last couple of months, so it's not due to any new *known* load.

I've run "top" and am not really sure what to look for. Another set of eyes would be welcome.

Ideas?

JB

[timryberg]

I've seen a few posts regarding this recently here in your forums. Compared to Windows, my Linux bag of tricks is a bit shallow.

I'm beginning to get pretty regular emails conatining:

1 minute: 16.07
5 minute: 13.49
15 minute: 11.42

from the box the last 2 days. Obviously, I'd like to nip this in the bud as the box is becoming slower and slower to respond. I've *not* added any new accounts to the box in the last couple of months, so it's not due to any new *known* load.

I've run "top" and am not really sure what to look for. Another set of eyes would be welcome.

Ideas?

JB


First of all, what are you running on the box? Any forums or CMS's that are using a lot of processor power?

Check to make sure you haven't been hacked.

Do you have a swap partitiion? Is it large enough?

Tim

[JayBaen]

Looks like it *might* have been a hack job.

I did a quick and dirty configuration of the APF firewall after noticing some unwanted IRCD "established" connections (... which I don't run).

POW. CPU utilization dropped from the constant 100% for the last 3 days back down to the expected 10 - 20%.

FYI -- I did enable the blocking of all egress (outgoing) connections from the box as well, so that if a script was running from the inside, it simply couldn't connect to anything.

JB

[IWorx-Chris]

This has been reported a bunch due to the recent Santy phpbb worm. Egress filtering will block the connections but you'll need to kill any running procs as well and upgrade any old phpbb's on the system to make sure the attacks don't continue.

Check out this thread as well: http://interworx.info/forums/showthread.php?t=341&highlight=hack

Chris

[JayBaen]

Thanks for the heads-up. Of course I've got clients who haven't upgraded their installations yet ... still running 2.0.10 :mad:

It seems I've stopped the CPU usage, but 2 days ago, the associated traffic came back -- even with all the non-essential ports closed in *and* out.

Might anyone have a quick listing of what procs to look for? I'm getting much more familiar with spelunking around the Linux environment, but nothing is a quick-fix for me yet. Any time someone can save me would be helpful.

JB

[JayBaen]

Here's more of what I've done, but the box is still exhibiting the traffic --

I found LOADS of .txt files in /tmp. None of them named anything useful like worm or bot, but all .php-looking script containing wording that it was for the anti-sanity worm. Mmhmm. Deleted.

Still running APF. Blocking everything in *and* out that's not web/iworx related.

Chmodded wget to 700.

"Installed" mod_security -- I *think.* Not sure I've done that exactly right, though, I do think I grabbed the right version. I'm guessing rpm -i isn't the only thing that needed to happen. Also, not exactly sure if the default security.conf you all have in your rpm for the iworx/rh9 flavor takes any additional tweaking.

As far as any rouge processes, frankly, it's hard for me to tell. These green linux eyes don't often know what shouldn't be there. I *do* like the quick learning curve I'm getting troubleshooting, however.

If anyone has any additional steps I might try, I'm all ears.

JB

[pascal]

Hello,


It seems I've stopped the CPU usage, but 2 days ago, the associated traffic came back -- even with all the non-essential ports closed in *and* out.


I'm not sure to understand well, sorry.

Do you mean your CPU usage seems to be fine, but you always encountered some CPU peeks ?

Anyway, here is what you may look at when your cpu usage and more your load avergae go up

First do "top"
Look at the jobs that takes a lot of cpu. They should be in first in the top list
Look for the PID number.

It maybe a perl script or something like this. You may also have a look at :
ps -axf or ps -axuw
you could look at cpu usage and memory usage of every processes running.

if the process is not a standard process you may kill it : kill -9 pid (where pid is the number process id)

You may look at log messages in /var/log/, as for example /var/log/messages, /var/log/secure/, /var/log/httpd/error_log

Maybe try to unset all processes not necessary from booting autostart for your init level in /etc/rc.d/rcX.d/ (where X is your init level : generaly 3)

You may for example disable all NFS, X, printer, RPC.... processes you not need.
(check http://interworx.info/forums/showthread.php?t=353)

Generaly when my CPU usage and load average are very high is due to a non KILLUP perl pgm. So I find the PID with top and kill the process with kill -9 pid.

Php processes also give me sometimes a high cpu/loadaverage.
If you look at top, and wait until php(s) process(es) begin you'll see that the cpu going high and the load average too.

now I have any solutions to figure out which user or which exactly script is responsible of this.

On a box with a lot of dynamic web site, you need a lot of ram, a good cpu, and fast HDD, it is less important on a box with static web site.

you may try to tweak your apache config with tools as AB, flood or httperf

Not sure my post will answer your question, and will help you

Try to give more details if you still have a pbm

Good luck

Pascal

[JayBaen]

Firstly -- Thx Pascal for stepping in. Always good to have your input.

Do you mean your CPU usage seems to be fine, but you always encountered some CPU peeks ?
My CPU load average was only high last week. It was discerned that we probably had a phpBB hack somewhere on the box. After tightening up the firewall, the CPU load dropped to normal and has been fine ever since. Beginning Monday (2 days later), the network traffic spiked from a constant 2% (currently normal) to 20% (high) and has maintained that rate.
Maybe try to unset all processes not necessary from booting autostart for your init level in /etc/rc.d/rcX.d/ (where X is your init level : generaly 3)
Certainly a great idea, but I'm more concerned (at this point) to quickly kill the processes added by the hack. I'm not familiar enough with the standard processes (yet) to recognize a rogue one. Thoughts?
now I have any solutions to figure out which user or which exactly script is responsible of this.
Now, THAT would be great ... :)

I'll see if I can make any sense of anything in the logs early this afternoon. Again, thanks for your help, and if you have any other thoughts, I'll take them.

JB

[pascal]

Hello,

Ok you speak about Network traffic, not cpu usage.

There is a lot of scanner that try to connect to your box or does DOS attack, etc ....
Maybe the traffic come from this sort of pbm.

I think you've installed APF ?

If yes, Just look at your /var/log/messages and /var/log/secure and try to figure out if there is a lot of DROP packets.

I recommand you to also install BFD (brut force detection) from rxfn.

And finally you asked about mod_security.

So here is my rule set.
I have configured a file in /etc/httpd/conf.d/ named mod_security.conf

here is the file

##
# mod_security
##

LoadModule security_module modules/mod_security.so

<IfModule mod_security.c>
# Turn the filtering engine On or Off
SecFilterEngine On

# Change Server: string
SecServerSignature " "

# Make sure that URL encoding is valid
SecFilterCheckURLEncoding On

# This setting should be set to On only if the Web site is
# using the Unicode encoding. Otherwise it may interfere with
# the normal Web site operation.
SecFilterCheckUnicodeEncoding Off

# Only allow bytes from this range
SecFilterForceByteRange 1 255

# The audit engine works independently and
# can be turned On of Off on the per-server or
# on the per-directory basis. "On" will log everything,
# "DynamicOrRelevant" will log dynamic requests or violations,
# and "RelevantOnly" will only log policy violations
SecAuditEngine RelevantOnly

# The name of the audit log file
SecAuditLog /var/log/httpd/audit_log

# Should mod_security inspect POST payloads
SecFilterScanPOST On

# Action to take by default
SecFilterDefaultAction "deny,log,status:500"

# Require HTTP_USER_AGENT and HTTP_HOST in all requests
SecFilterSelective "HTTP_USER_AGENT|HTTP_HOST" "^$"

# Weaker XSS protection but allows common HTML tags
SecFilter "<[[:space:]]*script"

# Very crude filters to prevent SQL injection attacks
SecFilter "delete[[:space:]]+from"
SecFilter "insert[[:space:]]+into"

# Protecting from XSS attacks through the PHP session cookie
SecFilterSelective ARG_PHPSESSID "!^[0-9a-z]*$"
SecFilterSelective COOKIE_PHPSESSID "!^[0-9a-z]*$"

# Import our snort converted modsec rules

Include /etc/httpd/conf.d/mod_sec.snort.conf
</IfModule>


You may see that I have an include command which one include an other file (: mod_sec.snort.conf) to complete the rules

Why an other file ?
It easily allow me to remove these rules when my server is very busy

The mod_sec.snort.conf is my dedicated and personnal file from the snort rules

here is the file : mod_sec.snort.conf

# WEB-ATTACKS ps command attempt
SecFilterSelective THE_REQUEST "/bin/ps"

# WEB-ATTACKS /bin/ps command attempt
SecFilterSelective THE_REQUEST "ps\x20"

# WEB-ATTACKS wget command attempt
SecFilter "wget\x20"

# WEB-ATTACKS uname -a command attempt
SecFilter "uname\x20-a"

# WEB-ATTACKS /usr/bin/id command attempt
SecFilterSelective THE_REQUEST "/usr/bin/id"

# WEB-ATTACKS id command attempt
SecFilter "\;id"

# WEB-ATTACKS kill command attempt
SecFilterSelective THE_REQUEST "/bin/kill"

# WEB-ATTACKS chsh command attempt
SecFilterSelective THE_REQUEST "/usr/bin/chsh"

# WEB-ATTACKS tftp command attempt
SecFilter "tftp\x20"

# WEB-ATTACKS /usr/bin/gcc command attempt
SecFilterSelective THE_REQUEST "/usr/bin/gcc"

# WEB-ATTACKS gcc command attempt
SecFilter "gcc\x20-o"

# WEB-ATTACKS /usr/bin/cc command attempt
SecFilterSelective THE_REQUEST "/usr/bin/cc"

# WEB-ATTACKS cc command attempt
SecFilter "cc\x20"

# WEB-ATTACKS /usr/bin/cpp command attempt
SecFilterSelective THE_REQUEST "/usr/bin/cpp"

# WEB-ATTACKS cpp command attempt
SecFilter "cpp\x20"

# WEB-ATTACKS /usr/bin/g++ command attempt
SecFilterSelective THE_REQUEST "/usr/bin/g\+\+"

# WEB-ATTACKS g++ command attempt
SecFilter "g\+\+\x20"

# WEB-ATTACKS bin/python access attempt
SecFilterSelective THE_REQUEST "bin/python"

# WEB-ATTACKS python access attempt
SecFilter "python\x20"

# WEB-ATTACKS bin/tclsh execution attempt
SecFilter "bin/tclsh"

# WEB-ATTACKS tclsh execution attempt
SecFilter "tclsh8\x20"

# WEB-ATTACKS bin/nasm command attempt
SecFilterSelective THE_REQUEST "bin/nasm"

# WEB-ATTACKS nasm command attempt
SecFilter "nasm\x20"

# WEB-ATTACKS perl execution attempt
SecFilter "perl\x20"

# WEB-ATTACKS traceroute command attempt
SecFilter "traceroute\x20"

# WEB-ATTACKS ping command attempt
SecFilterSelective THE_REQUEST "/bin/ping"

# WEB-ATTACKS netcat command attempt
SecFilter "nc\x20"

# WEB-ATTACKS nmap command attempt
SecFilter "nmap\x20"

# WEB-ATTACKS xterm command attempt
SecFilterSelective THE_REQUEST "/usr/X11R6/bin/xterm"

# WEB-ATTACKS X application to remote host attempt
SecFilter "\x20-display\x20"

# WEB-ATTACKS lsof command attempt
SecFilter "lsof\x20"

# WEB-ATTACKS rm command attempt
SecFilter "rm\x20"

# WEB-ATTACKS mail command attempt
SecFilterSelective THE_REQUEST "/bin/mail"

# WEB-ATTACKS mail command attempt
SecFilter "mail\x20"

# WEB-ATTACKS /bin/ls command attempt
SecFilterSelective THE_REQUEST "/bin/ls"

# WEB-ATTACKS /etc/shadow access
SecFilter "/etc/shadow"

# WEB-ATTACKS .htgroup access
SecFilterSelective THE_REQUEST "\.htgroup"

# WEB-CGI websitepro path access
SecFilter " /HTTP/1\."

# WEB-CGI formmail arbitrary command execution attempt
SecFilterSelective THE_REQUEST "/formmail" chain
SecFilter "\x0a"

# WEB-CGI formmail access
SecFilterSelective THE_REQUEST "/formmail" log,pass

# WEB-CGI phf arbitrary command execution attempt
SecFilterSelective THE_REQUEST "/phf" chain
SecFilter "\x0a/"

# WEB-CGI phf access
SecFilterSelective THE_REQUEST "/phf" log,pass

# WEB-CGI rksh access
SecFilterSelective THE_REQUEST "/rksh"

# WEB-CGI bash access
SecFilterSelective THE_REQUEST "/bash" log,pass

# WEB-CGI zsh access
SecFilterSelective THE_REQUEST "/zsh"

# WEB-CGI csh access
SecFilterSelective THE_REQUEST "/csh"

# WEB-CGI tcsh access
SecFilterSelective THE_REQUEST "/tcsh"

# WEB-CGI rsh access
SecFilterSelective THE_REQUEST "/rsh"

# WEB-CGI ksh access
SecFilterSelective THE_REQUEST "/ksh"

# WEB-CLIENT Javascript URL host spoofing attempt
SecFilter "javascript\://"

# WEB-MISC cross site scripting \(img src=javascript\) attempt
SecFilter "img src=javascript"

# WEB-MISC .htpasswd access
SecFilter "\.htpasswd"

# WEB-MISC .htaccess access
SecFilter "\.htaccess"

# WEB-MISC cd..
SecFilter "cd\.\."

# WEB-MISC ///cgi-bin access
SecFilterSelective THE_REQUEST "///cgi-bin"

# WEB-MISC /cgi-bin/// access
SecFilterSelective THE_REQUEST "/cgi-bin///"

# WEB-MISC /~root access
SecFilterSelective THE_REQUEST "/~root"

# WEB-MISC /~ftp access
SecFilterSelective THE_REQUEST "/~ftp"

# WEB-MISC cat%20 access
SecFilter "cat\x20"

# WEB-MISC rpm_query access
SecFilterSelective THE_REQUEST "/rpm_query"

# WEB-MISC htgrep attempt
SecFilterSelective THE_REQUEST "/htgrep" chain
SecFilter "hdr=/"

# WEB-MISC htgrep access
SecFilterSelective THE_REQUEST "/htgrep" log,pass

# WEB-MISC .history access
SecFilterSelective THE_REQUEST "/\.history"

# WEB-MISC .bash_history access
SecFilterSelective THE_REQUEST "/\.bash_history"

# WEB-MISC /~nobody access
SecFilterSelective THE_REQUEST "/~nobody"

# WEB-MISC *%0a.pl access
SecFilterSelective THE_REQUEST "/*\x0a\.pl"

# WEB-MISC Apache Chunked-Encoding worm attempt
SecFilter "CCCCCCC\: AAAAAAAAAAAAAAAAAAA"

# WEB-MISC Transfer-Encoding\: chunked
SecFilter "chunked"

# WEB-PHP squirrel mail theme arbitrary command attempt
SecFilterSelective THE_REQUEST "/left_main\.php" chain
SecFilter "cmdd="

# WEB-PHP DNSTools administrator authentication bypass attempt
SecFilterSelective THE_REQUEST "/dnstools\.php" chain
SecFilter "user_dnstools_administrator=true"

# WEB-PHP DNSTools authentication bypass attempt
SecFilterSelective THE_REQUEST "/dnstools\.php" chain
SecFilter "user_logged_in=true"

# WEB-PHP DNSTools access
SecFilterSelective THE_REQUEST "/dnstools\.php" log,pass

# WEB-PHP Blahz-DNS dostuff.php modify user attempt
SecFilterSelective THE_REQUEST "/dostuff\.php\?action=modify_user"

# WEB-PHP Blahz-DNS dostuff.php access
SecFilterSelective THE_REQUEST "/dostuff\.php" log,pass

# WEB-PHP PHP-Wiki cross site scripting attempt
SecFilterSelective THE_REQUEST "<script"

# WEB-PHP strings overflow
SecFilterSelective THE_REQUEST "\?STRENGUR"

# WEB-PHP PHPLIB remote command attempt
SecFilter "_PHPLIB\[libdir\]"


to create these files do :
first save your old security.conf file

cd /etc/httpd/conf.d/
cp security.conf security.back
rm -f security.conf


then create the mod_security.conf file and copy paste mine


touch mod_security.conf
vi mod_securtity.conf
press "Inser"
copy and paste my data


finally create the file mod_sec.snort.conf and copy and paste my data


touch mod_sec.snort.conf
vi mod_sec.snort.conf
press "Inser"
copy and paste my data



Hope this will help you

Pascal

[JayBaen]

Thanks again Paschal for the LOADS of info. Here's my 1st problem:

I downloaded the mod_security rpm from interworx in the iworx/rh9 location. At the time I only did a "rpm -i <package>" -- I'm not sure that's all I need to do to get mod_security to be enabled. Therefore, I don't think it's doing anything yet. How can I verify/fix that?

Once that happens, I'll try your configs (and/or versions of them) and post the results.

JB

[IWorx-Chris]

I downloaded the mod_security rpm from interworx in the iworx/rh9 location. At the time I only did a "rpm -i <package>" -- I'm not sure that's all I need to do to get mod_security to be enabled. Therefore, I don't think it's doing anything yet. How can I verify/fix that?


Verify that there's a /etc/httpd/conf.d/security.conf file. If it's there then you are running mod_security. You can also enable server-info in the main httpd.conf to view the loaded modules while apache is running.

Chris

[JayBaen]

Thx Chris for stepping in -- I can use the help. I'm glad a few of you aren't afraid to "lead this horse to water" .. :) I promise, I'll drink once I'm there.

After checking, I *do* have the security.conf file -- and a quick phpinfo () shows the mod_security as loaded (... after enablilng ExtendedStatus). So far, my security.conf is the same as the one that ships in the .rpm courtesy of iworx.

Here's the problem I'm dealing with:

http://www.battledomain.com/phpstuff/monitoring.php.png

Is is probable that mod_security can't catch this sort of thing -- or do I not have a certain feature of mod_security enabled to do so? I'm imagining I've got a rogue process lying around that I'm just not catching.

If anyone wants to take a look at the following and throw a guy a bone, I'd take it:

[root@web1 conf]# ps -axf
PID TTY STAT TIME COMMAND
1 ? S 0:03 init
2 ? SW 0:00 [keventd]
3 ? SW 0:00 [kapmd]
4 ? SWN 0:01 [ksoftirqd_CPU0]
9 ? SW 0:00 [bdflush]
5 ? SW 0:01 [kswapd]
6 ? SW 0:00 [kscand/DMA]
7 ? SW 1:56 [kscand/Normal]
8 ? SW 0:00 [kscand/HighMem]
10 ? SW 0:00 [kupdated]
11 ? SW 0:00 [mdrecoveryd]
15 ? SW 0:08 [kjournald]
73 ? SW 0:00 [khubd]
2261 ? SW 0:00 [kjournald]
2350 tty8 S 0:00 /bin/bash
2474 ? SW 0:00 [eth0]
2667 ? S 0:01 syslogd -m 0
2671 ? S 0:00 klogd -x
2707 ? S 0:00 /usr/sbin/apmd -p 10 -w 5 -W -P /etc/sysconfig/apm-sc
3550 ? S 0:00 /usr/sbin/sshd
26628 ? S 0:00 \_ /usr/sbin/sshd
26654 ? S 0:00 \_ /usr/sbin/sshd
26655 pts/0 S 0:00 \_ -bash
26689 pts/0 S 0:00 \_ su
26690 pts/0 S 0:04 \_ bash
513 pts/0 R 0:00 \_ ps -axf
3561 ? S 0:00 xinetd -stayalive -pidfile /var/run/xinetd.pid
3573 ? SL 0:00 ntpd -U ntp -g
3586 ? S 0:00 /bin/sh /usr/bin/mysqld_safe --defaults-file=/etc/my.
3621 ? S 0:09 \_ /usr/sbin/mysqld --defaults-file=/etc/my.cnf --ba
3632 ? S 0:00 /bin/sh /home/interworx/bin/mysql/mysqld_safe --defau
3670 ? S 0:09 \_ /home/interworx/bin/iworx-db --defaults-file=/hom
3676 ? S 0:00 /home/interworx/bin/iworx-web -f /home/interworx/etc/
3703 ? S 0:11 \_ /home/interworx/bin/iworx-web -f /home/interworx/
3704 ? S 0:16 \_ /home/interworx/bin/iworx-web -f /home/interworx/
3705 ? S 0:15 \_ /home/interworx/bin/iworx-web -f /home/interworx/
3706 ? S 0:15 \_ /home/interworx/bin/iworx-web -f /home/interworx/
3710 ? S 2:47 \_ /home/interworx/bin/iworx-web -f /home/interworx/
3711 ? S 0:14 \_ /home/interworx/bin/iworx-web -f /home/interworx/
3712 ? S 0:20 \_ /home/interworx/bin/iworx-web -f /home/interworx/
3713 ? S 0:16 \_ /home/interworx/bin/iworx-web -f /home/interworx/
3702 ? S 0:00 crond
3728 ? S 0:00 /usr/sbin/atd
3752 tty1 S 0:00 /sbin/mingetty tty1
3753 tty2 S 0:00 /sbin/mingetty tty2
3754 tty3 S 0:00 /sbin/mingetty tty3
3755 tty4 S 0:00 /sbin/mingetty tty4
3756 tty5 S 0:00 /sbin/mingetty tty5
3757 tty6 S 0:00 /sbin/mingetty tty6
3758 ? S 0:00 /bin/sh /usr/bin/svscanboot
3760 ? S 0:00 \_ svscan /service
3762 ? S 0:00 | \_ supervise send
3808 ? S 0:00 | | \_ qmail-send
3822 ? S 0:00 | | \_ qmail-lspawn ./Maildir/
3823 ? S 0:00 | | \_ qmail-rspawn
3824 ? S 0:00 | | \_ qmail-clean
3763 ? S 0:00 | \_ supervise log
3800 ? S 0:00 | | \_ /usr/bin/multilog t /var/log/send
3764 ? S 0:00 | \_ supervise smtp
3779 ? S 0:00 | | \_ /usr/bin/tcpserver -v -R -S -p -x /etc/tc
3765 ? S 0:00 | \_ supervise log
3801 ? S 0:00 | | \_ /usr/bin/multilog t /var/log/smtp
3766 ? S 0:00 | \_ supervise imap4
3839 ? S 0:00 | | \_ /usr/bin/tcpserver -v -R -H -l web1.battl
3767 ? S 0:00 | \_ supervise log
3802 ? S 0:00 | | \_ /usr/bin/multilog t /var/log/imap4
3768 ? S 0:00 | \_ supervise imap4-ssl
3805 ? S 0:00 | | \_ /usr/bin/tcpserver -v -R -H -l web1.battl
3769 ? S 0:00 | \_ supervise log
3806 ? S 0:00 | | \_ /usr/bin/multilog t /var/log/imap4-ssl
3770 ? S 0:00 | \_ supervise pop3-ssl
3810 ? S 0:00 | | \_ /usr/bin/tcpserver -v -R -H -l web1.battl
3771 ? S 0:00 | \_ supervise log
3811 ? S 0:00 | | \_ /usr/bin/multilog t /var/log/pop3-ssl
3772 ? S 0:00 | \_ supervise pop3
3812 ? S 0:07 | | \_ /usr/bin/tcpserver -v -R -c 200 0 110 /va
510 ? S 0:00 | | \_ /var/qmail/bin/qmail-popup web1.battl
511 ? S 0:00 | | \_ /var/qmail/bin/qmail-pop3d Maildi
3773 ? S 0:00 | \_ supervise log
3840 ? S 0:06 | | \_ /usr/bin/multilog t /var/log/pop3
3774 ? S 0:00 | \_ supervise dnscache
3813 ? S 0:11 | | \_ /usr/bin/dnscache
3775 ? S 0:00 | \_ supervise log
3814 ? S 0:04 | | \_ multilog t ./main
3776 ? S 0:00 | \_ supervise tinydns-xx.xx.xx.xx
28445 ? S 0:00 | | \_ /usr/bin/tinydns
3777 ? S 0:00 | \_ supervise log
3841 ? S 0:00 | | \_ multilog t ./main
3778 ? S 0:00 | \_ supervise tinydns-xx.xx.xx.xx
28447 ? S 0:00 | | \_ /usr/bin/tinydns
3780 ? S 0:00 | \_ supervise log
3826 ? S 0:00 | | \_ multilog t ./main
3781 ? S 0:00 | \_ supervise tinydns-xx.xx.xx.xx
28449 ? S 0:00 | | \_ /usr/bin/tinydns
3782 ? S 0:00 | \_ supervise log
3842 ? S 0:00 | | \_ multilog t ./main
3783 ? S 0:00 | \_ supervise tinydns-xx.xx.xx.xx
28451 ? S 0:00 | | \_ /usr/bin/tinydns
3784 ? S 0:00 | \_ supervise log
3829 ? S 0:00 | | \_ multilog t ./main
3785 ? S 0:00 | \_ supervise tinydns-xx.xx.xx.xx
28453 ? S 0:00 | | \_ /usr/bin/tinydns
3786 ? S 0:00 | \_ supervise log
3831 ? S 0:00 | | \_ multilog t ./main
3787 ? S 0:00 | \_ supervise tinydns-xx.xx.xx.xx
28455 ? S 0:00 | | \_ /usr/bin/tinydns
3788 ? S 0:00 | \_ supervise log
3833 ? S 0:00 | | \_ multilog t ./main
3789 ? S 0:00 | \_ supervise tinydns-xx.xx.xx.xx
28457 ? S 0:00 | | \_ /usr/bin/tinydns
3790 ? S 0:00 | \_ supervise log
3835 ? S 0:00 | | \_ multilog t ./main
3791 ? S 0:00 | \_ supervise tinydns-xx.xx.xx.xx
28459 ? S 0:00 | | \_ /usr/bin/tinydns
3792 ? S 0:00 | \_ supervise log
3845 ? S 0:00 | | \_ multilog t ./main
3793 ? S 0:00 | \_ supervise tinydns-xx.xx.xx.xx
28461 ? S 0:00 | | \_ /usr/bin/tinydns
3794 ? S 0:00 | \_ supervise log
3846 ? S 0:00 | | \_ multilog t ./main
3795 ? S 0:00 | \_ supervise tinydns-xx.xx.xx.xx
28463 ? S 0:00 | | \_ /usr/bin/tinydns
3796 ? S 0:00 | \_ supervise log
3838 ? S 0:00 | \_ multilog t ./main
3761 ? S 0:00 \_ readproctitle service errors: ...................
12731 ? S 0:00 proftpd: (accepting connections)
27641 ? S 0:00 /usr/sbin/httpd -DSSL
27644 ? S 0:01 \_ /usr/sbin/httpd -DSSL
27646 ? S 0:00 \_ /usr/sbin/httpd -DSSL
27647 ? S 0:00 \_ /usr/sbin/httpd -DSSL
27648 ? S 0:00 \_ /usr/sbin/httpd -DSSL
27651 ? S 0:01 \_ /usr/sbin/httpd -DSSL
28893 ? S 0:00 \_ /usr/sbin/httpd -DSSL
28949 ? S 0:00 \_ /usr/sbin/httpd -DSSL
[root@web1 conf]#

Thanks again in advance.

JB

[pascal]

Hello,

First, there is nothing bad in your ps -axf. Maybe the ntpd (network time protocol - The ntpd program operates by exchanging messages with one or more configured servers at designated poll intervals) should be stop, except if you need it.

Mod_security has nothing to do with your high outgoing traffic. But you mat check the mod_security audit log in /var/log/httpd/audit_log (tail -n100 /var/log/httpd/audit_log)
also you may add this security filter concerning the last phpbb virus attack : secfilter "wget\x20" somewhere in you security.conf file

I'll more check if there is not a new domain that is consumming the outgoing bandwidth.

Do you have only one domain on this box ? maybe few ? maybe you have a new client who have a lot of visitors.

I'll check in nodeworx the bandwidth used per domain.
Login into nodeworx, click on "siteworx account" then check the field "bandwidth use" for every domains you host.

Also check all relevant logs in /var/log/messages and /var/log/secure

Hope this will help

Pascal

[JayBaen]

Thanks again Paschal, you're being quite generous with your time.

Well, this sure is a good "hunt." I've made a few more adjustments which I'll post here -- none of which have solved the issue yet. I promise I'll post a "10 Condensed Steps on How I solved this Hack" sticky-able thread once I've conquered it.

What I've done since last post:

1) Tweaked mod_security a bit to include a) SecFilterEngine "On" instead of "DynamicOnly". b) filters for XSS php Cookie attacks and c) an include list of converted modsec rules, similar to your snort ones.

2) Checked the mod_security audit_log and found *no* entries (none, zero, zilch, nada -- even after running for a day or so now.)

3) Added the brute-force detection -- which is working and denying via APF successfully.

4) Checked all my SiteWorx accounts, of which there are only a handful. Most are either my own, or a few select (trusted) friends. None of them reflect this bandwidth useage.

If there's any other place you can think of I can dig, I'd appreciate it.

JB

[JayBaen]

Oh my goodness .. I think I may have found it ... give me a few to verify, then I'll come back and (slightly) embarass myself.

JB

[JayBaen]

The update:

First of all, let me thank Paschal and others for providing all the info. It did allow me to put in place a few extra (and much needed) prevention measures (e.g. mod_security, bruteForce attack and APF). They also *did aid me* in bandaid-ing the phpBB exploit.

My most recent issue involving the bandwidth (coming right on the heels of the phpBB issue), turned out to be an endlessly looping pop3 message, caught in a cycle. I've since deleted the message server-side, and my bandwidth returned to normal.

I'm left with the remnants of the phpBB issue, however. Aside of getting all of my hosted phpBB sites upgraded, I firewalled all the Outgoing (egress) ports -- except what's needed for web/iworx functions. The problem is that my APF installation is logging many dropped packets on the way out of the box -- which look as if they could be coming from some process left-over from the phpBB hack -- attempting to make contact with the mother-ship.

What process? I have no idea -- and that's the rub. How can I be sure I've killed and removed what was changed/added by the worm? I'd like to be able to *not* filter egress ports, only inbound, as I've got some other web-scripts that would like to dynamically assign outgoing ports. Obviously, I can't use them until I know I'm safe.

Thoughts?

JB

[IWorx-Chris]

Can you post the apf log entries that are suspicious JayBean?

Chris

[JayBaen]

Hey Chris -- Certainly.

The ones that seem the *most* suspicious to me are when something appears to be port-hopping to find an exit. For example:

From 65.110.xx.xx - 51 packets (n.b. this is the source IP of the web box)
To 66.23.xx.xx - 49 packets (n.b. In this instance, this happened to be *my* IP with which I was logged in via SSH .. !! )
Service: 5601 (tcp/5601) (** OUT_TCP DROP **,none,eth0) - 1 packet
Service: 5621 (tcp/5621) (** OUT_TCP DROP **,none,eth0) - 1 packet
Service: 5653 (tcp/5653) (** OUT_TCP DROP **,none,eth0) - 1 packet
Service: 5676 (tcp/5676) (** OUT_TCP DROP **,none,eth0) - 1 packet
Service: 5698 (tcp/5698) (** OUT_TCP DROP **,none,eth0) - 1 packet
Service: 5727 (tcp/5727) (** OUT_TCP DROP **,none,eth0) - 1 packet
Service: 5752 (tcp/5752) (** OUT_TCP DROP **,none,eth0) - 1 packet
Service: 5783 (tcp/5783) (** OUT_TCP DROP **,none,eth0) - 1 packet
Service: 5811 (tcp/5811) (** OUT_TCP DROP **,none,eth0) - 1 packet
Service: 5836 (tcp/5836) (** OUT_TCP DROP **,none,eth0) - 1 packet
Service: 5855 (tcp/5855) (** OUT_TCP DROP **,none,eth0) - 1 packet
Service: 5884 (tcp/5884) (** OUT_TCP DROP **,none,eth0) - 1 packet
Service: 5917 (tcp/5917) (** OUT_TCP DROP **,none,eth0) - 1 packet
Service: 5951 (tcp/5951) (** OUT_TCP DROP **,none,eth0) - 1 packet
Service: 5984 (tcp/5984) (** OUT_TCP DROP **,none,eth0) - 1 packet
Service: 6011 (tcp/6011) (** OUT_TCP DROP **,none,eth0) - 1 packet
Service: 6039 (tcp/6039) (** OUT_TCP DROP **,none,eth0) - 1 packet
Service: 6067 (tcp/6067) (** OUT_TCP DROP **,none,eth0) - 1 packet
Service: 6092 (tcp/6092) (** OUT_TCP DROP **,none,eth0) - 1 packet
Service: 6112 (tcp/6112) (** OUT_TCP DROP **,none,eth0) - 1 packet
Service: 6141 (tcp/6141) (** OUT_TCP DROP **,none,eth0) - 1 packet
Service: 6165 (tcp/6165) (** OUT_TCP DROP **,none,eth0) - 1 packet
Service: 6184 (tcp/6184) (** OUT_TCP DROP **,none,eth0) - 1 packet
Service: 6212 (tcp/6212) (** OUT_TCP DROP **,none,eth0) - 1 packet
Service: 6236 (tcp/6236) (** OUT_TCP DROP **,none,eth0) - 1 packet
Service: 6263 (tcp/6263) (** OUT_TCP DROP **,none,eth0) - 1 packet
Service: 6292 (tcp/6292) (** OUT_TCP DROP **,none,eth0) - 1 packet
Service: 6314 (tcp/6314) (** OUT_TCP DROP **,none,eth0) - 1 packet
Service: 6336 (tcp/6336) (** OUT_TCP DROP **,none,eth0) - 1 packet
Service: 6377 (tcp/6377) (** OUT_TCP DROP **,none,eth0) - 1 packet
Service: 6398 (tcp/6398) (** OUT_TCP DROP **,none,eth0) - 1 packet
Service: 6418 (tcp/6418) (** OUT_TCP DROP **,none,eth0) - 1 packet
Service: 6447 (tcp/6447) (** OUT_TCP DROP **,none,eth0) - 1 packet
Service: 6470 (tcp/6470) (** OUT_TCP DROP **,none,eth0) - 1 packet
Service: 6492 (tcp/6492) (** OUT_TCP DROP **,none,eth0) - 1 packet
Service: 6519 (tcp/6519) (** OUT_TCP DROP **,none,eth0) - 1 packet
Service: 6541 (tcp/6541) (** OUT_TCP DROP **,none,eth0) - 1 packet
Service: 6561 (tcp/6561) (** OUT_TCP DROP **,none,eth0) - 1 packet
Service: 6600 (tcp/6600) (** OUT_TCP DROP **,none,eth0) - 1 packet
Service: 6620 (tcp/6620) (** OUT_TCP DROP **,none,eth0) - 1 packet
Service: 6648 (tcp/6648) (** OUT_TCP DROP **,none,eth0) - 1 packet
Service: 6666 (tcp/6666) (** OUT_TCP DROP **,none,eth0) - 1 packet
Service: 6687 (tcp/6687) (** OUT_TCP DROP **,none,eth0) - 1 packet
Service: 6713 (tcp/6713) (** OUT_TCP DROP **,none,eth0) - 1 packet
Service: 6732 (tcp/6732) (** OUT_TCP DROP **,none,eth0) - 1 packet
Service: 6754 (tcp/6754) (** OUT_TCP DROP **,none,eth0) - 1 packet
Service: 6780 (tcp/6780) (** OUT_TCP DROP **,none,eth0) - 1 packet
Service: 6799 (tcp/6799) (** OUT_TCP DROP **,none,eth0) - 1 packet
Service: 6824 (tcp/6824) (** OUT_TCP DROP **,none,eth0) - 1 packet

Then, the occasional:
From 65.110.xx.xx - 2 packets (n.b. again, my web box -- diff IP)
To 68.157.xx.xx - 2 packets (n.b. somewhere external, not me)
Service: 50893 (tcp/50893) (** OUT_TCP DROP **,none,eth0) - 1 packet
Service: 50924 (tcp/50924) (** OUT_TCP DROP **,none,eth0) - 1 packet

This is just a quick example of pieces from a particular day. I could provide more or different if it were helpful.

Thx again for helping.

JB

[IWorx-Chris]

Can you paste your current egress rules too for APF JayBean.

Chris

[JayBaen]

EGF="1"

# Common egress (outbound) TCP ports
EG_TCP_CPORTS="21,25,80,443,43,51234,123,110,995,143,2080,2443,22 ,53,953,3306"

# Common egress (outbound) UDP ports
EG_UDP_CPORTS="20,21,53,123"

# Common ICMP egress (outbound) types
# 'internals/icmp.types' for type definition; 'all' is wildcard for any
EG_ICMP_TYPES="all"

[JayBaen]

Any other thoughts, Chris? Were you smelling something .. ;) ?

JB

[IWorx-Chris]

That does look a little odd JayBean but not totally telling, are there / have there been any other symptoms?

Chris

[JayBaen]

Only that I still have occasional packets being blocked egress on the box. I suppose I'll only know if it's really still a problem by allowing all outbound again and monitoring the traffic/cpu usage.