Segmentation Fault (11)

General Interworx Control Panel Info General Inquries
1

My apache error_log is filled with a reoccurrence of the following several times daily. Can anyone steer me in the right direction to address this. mod_watch, OpenSSL, and mod_ssl seem updated. Here’s what I am seeing:

[Thu Sep 03 03:18:37 2009] [notice] Apache/2.2.8 (Unix) DAV/2 PHP/5.1.6 mod_ssl/2.2.8 OpenSSL/0.9.8e-fips-rhel5 mod_watch/4.3 configured – resuming normal operations
[Thu Sep 03 03:18:40 2009] [notice] child pid 6823 exit signal Segmentation fault (11)
[Thu Sep 03 03:19:04 2009] [notice] child pid 6837 exit signal Segmentation fault (11)
[Thu Sep 03 03:19:26 2009] [notice] child pid 6858 exit signal Segmentation fault (11)
[Thu Sep 03 03:20:34 2009] [notice] child pid 6822 exit signal Segmentation fault (11)
[Thu Sep 03 03:20:35 2009] [notice] child pid 8551 exit signal Segmentation fault (11)
[Thu Sep 03 03:20:38 2009] [notice] child pid 6839 exit signal Segmentation fault (11)
[Thu Sep 03 03:23:55 2009] [notice] child pid 8543 exit signal Segmentation fault (11)

Any help would be greatly appreciated. Iworx 3.0.4

2

It’s possible that a PHP script can be causing a seg-fault. I’m not entirely sure if this is still true with the current version of PHP, but I have run into situations where a bad recursive function or an endless loop have caused apache to register a seg-fault error message.

Hope that helps.

3

thx jonathan. i went ahead and upgraded to cp ver 4 to see if this would take care of the issue, however it is still seg faulting. also, after the upgrade my axfrdns zones are not transferring to my secondaries, but that is another issue. i will continue to look on php. it is still showing 5.1.6 for the version. is this current?

4

PHP 5.1.6 is what is installed with CentOS by default, but we have made new RPMs available. Take a look at http://updates.interworx.com/iworx/RPMS/unsupported/.

5

ok, thanks for the info. do you know of anyone else having the seg fault (11) issue or is this a ‘known issue’ to any degree? also, are the seg faults worth persuing?

6

Which error log are we talking about here, /var/log/httpd/error_log? There really are a number of things that could be going on here. Maybe try increasing the log level of Apache and see if that gives you any more useful clues.

You aren’t running any kind of weird configuration, like running PHP4 and PHP5?

My only fear would be some script kitty using an exploit and a side effect being seg-faulting apache. I’m not saying that could be what is going on here, but that risk alone is enough for me to keep looking.

But as long as no one is complaining that their sites are down, that it isn’t too much of a problem.

Don’t take this advice as anything official, it’s just what I have done when I’ve seen similar issues.

7

As an aside, the concept of a “script kitty” is terrifying. :slight_smile:

I image some young punk LOLCat, chugging Bawls.

8

Oh crap, did I seriously type “kitty”, must have had “I Can Haz Cheezeburger” on the brain.

I CAN HAZ YOUR APACHEZ!!!

9

ok, here goes. i finally got the seg fault 11 (from /var/log/httpd/error_log) to dump a core and this is what gdb produces:

[root@host tmp]# gdb /usr/sbin/httpd /tmp/core.24052
GNU gdb Fedora (6.8-27.el5)
Copyright (C) 2008 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html&gt;
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law. Type “show copying”
and “show warranty” for details.
This GDB was configured as “x86_64-redhat-linux-gnu”…
(no debugging symbols found)
Reading symbols from /lib64/libm.so.6…(no debugging symbols found)…done.
Loaded symbols for /lib64/libm.so.6
Reading symbols from /lib64/libssl.so.6…(no debugging symbols found)…done.
Loaded symbols for /lib64/libssl.so.6
Reading symbols from /lib64/libcrypto.so.6…(no debugging symbols found)…done.
Loaded symbols for /lib64/libcrypto.so.6
Reading symbols from /usr/lib64/libz.so.1…
(no debugging symbols found)…done.
Loaded symbols for /usr/lib64/libz.so.1
Reading symbols from /usr/lib64/libaprutil-1.so.0…(no debugging symbols found)…done.
Loaded symbols for /usr/lib64/libaprutil-1.so.0
Reading symbols from /lib64/libcrypt.so.1…(no debugging symbols found)…done.
Loaded symbols for /lib64/libcrypt.so.1
Reading symbols from /usr/lib64/libdb-4.3.so…
(no debugging symbols found)…done.
Loaded symbols for /usr/lib64/libdb-4.3.so
Reading symbols from /usr/lib64/libldap-2.3.so.0…(no debugging symbols found)…done.
Loaded symbols for /usr/lib64/libldap-2.3.so.0
Reading symbols from /usr/lib64/liblber-2.3.so.0…(no debugging symbols found)…done.
Loaded symbols for /usr/lib64/liblber-2.3.so.0
Reading symbols from /lib64/libexpat.so.0…
(no debugging symbols found)…done.
Loaded symbols for /lib64/libexpat.so.0
Reading symbols from /usr/lib64/libapr-1.so.0…(no debugging symbols found)…done.
Loaded symbols for /usr/lib64/libapr-1.so.0
Reading symbols from /lib64/libpthread.so.0…(no debugging symbols found)…done.
Loaded symbols for /lib64/libpthread.so.0
Reading symbols from /lib64/libdl.so.2…
(no debugging symbols found)…done.
Loaded symbols for /lib64/libdl.so.2
Reading symbols from /lib64/libc.so.6…(no debugging symbols found)…done.
Loaded symbols for /lib64/libc.so.6
Reading symbols from /usr/lib64/libgssapi_krb5.so.2…(no debugging symbols found)…done.
Loaded symbols for /usr/lib64/libgssapi_krb5.so.2
Reading symbols from /usr/lib64/libkrb5.so.3…
(no debugging symbols found)…done.
Loaded symbols for /usr/lib64/libkrb5.so.3
Reading symbols from /lib64/libcom_err.so.2…(no debugging symbols found)…done.
Loaded symbols for /lib64/libcom_err.so.2
Reading symbols from /usr/lib64/libk5crypto.so.3…(no debugging symbols found)…done.
Loaded symbols for /usr/lib64/libk5crypto.so.3
Reading symbols from /lib64/libuuid.so.1…
(no debugging symbols found)…done.
Loaded symbols for /lib64/libuuid.so.1
Reading symbols from /usr/lib64/libpq.so.4…(no debugging symbols found)…done.
Loaded symbols for /usr/lib64/libpq.so.4
Reading symbols from /usr/lib64/libsqlite3.so.0…(no debugging symbols found)…done.
Loaded symbols for /usr/lib64/libsqlite3.so.0
Reading symbols from /lib64/libresolv.so.2…
(no debugging symbols found)…done.
Loaded symbols for /lib64/libresolv.so.2
Reading symbols from /usr/lib64/libsasl2.so.2…(no debugging symbols found)…done.
Loaded symbols for /usr/lib64/libsasl2.so.2
Reading symbols from /lib64/ld-linux-x86-64.so.2…(no debugging symbols found)…done.
Loaded symbols for /lib64/ld-linux-x86-64.so.2
Reading symbols from /usr/lib64/libkrb5support.so.0…
(no debugging symbols found)…done.
Loaded symbols for /usr/lib64/libkrb5support.so.0
Reading symbols from /lib64/libkeyutils.so.1…(no debugging symbols found)…done.
Loaded symbols for /lib64/libkeyutils.so.1
Reading symbols from /lib64/libnsl.so.1…(no debugging symbols found)…done.
Loaded symbols for /lib64/libnsl.so.1
Reading symbols from /lib64/libselinux.so.1…
(no debugging symbols found)…done.
Loaded symbols for /lib64/libselinux.so.1
Reading symbols from /lib64/libsepol.so.1…(no debugging symbols found)…done.
Loaded symbols for /lib64/libsepol.so.1
Reading symbols from /lib64/libnss_files.so.2…(no debugging symbols found)…done.
Loaded symbols for /lib64/libnss_files.so.2
Reading symbols from /usr/lib64/httpd/modules/mod_cgi.so…done.
Loaded symbols for /etc/httpd/modules/mod_cgi.so
Reading symbols from /usr/lib64/httpd/modules/mod_authz_user.so…done.
Loaded symbols for /usr/lib64/httpd/modules/mod_authz_user.so
Reading symbols from /usr/lib64/httpd/modules/mod_authn_file.so…done.
Loaded symbols for /usr/lib64/httpd/modules/mod_authn_file.so
Reading symbols from /usr/lib64/httpd/modules/mod_auth_basic.so…done.
Loaded symbols for /usr/lib64/httpd/modules/mod_auth_basic.so
Reading symbols from /usr/lib64/httpd/modules/mod_authz_groupfile.so…done.
Loaded symbols for /usr/lib64/httpd/modules/mod_authz_groupfile.so
Reading symbols from /usr/lib64/httpd/modules/mod_authz_host.so…done.
Loaded symbols for /usr/lib64/httpd/modules/mod_authz_host.so
Reading symbols from /usr/lib64/httpd/modules/mod_cache.so…done.
Loaded symbols for /usr/lib64/httpd/modules/mod_cache.so
Reading symbols from /usr/lib64/httpd/modules/mod_disk_cache.so…done.
Loaded symbols for /usr/lib64/httpd/modules/mod_disk_cache.so
Reading symbols from /usr/lib64/httpd/modules/mod_mem_cache.so…done.
Loaded symbols for /usr/lib64/httpd/modules/mod_mem_cache.so
Reading symbols from /usr/lib64/httpd/modules/mod_ext_filter.so…done.
Loaded symbols for /usr/lib64/httpd/modules/mod_ext_filter.so
Reading symbols from /usr/lib64/httpd/modules/mod_include.so…done.
Loaded symbols for /usr/lib64/httpd/modules/mod_include.so
Reading symbols from /usr/lib64/httpd/modules/mod_deflate.so…done.
Loaded symbols for /usr/lib64/httpd/modules/mod_deflate.so
Reading symbols from /usr/lib64/httpd/modules/mod_log_config.so…done.
Loaded symbols for /usr/lib64/httpd/modules/mod_log_config.so

etc…

Reading symbols from /usr/lib64/php/modules/mysql.so…done.
Loaded symbols for /usr/lib64/php/modules/mysql.so
Reading symbols from /usr/lib64/mysql/libmysqlclient.so.15…done.
Loaded symbols for /usr/lib64/mysql/libmysqlclient.so.15
Reading symbols from /usr/lib64/php/modules/mysqli.so…done.
Loaded symbols for /usr/lib64/php/modules/mysqli.so
Reading symbols from /usr/lib64/php/modules/pdo.so…done.
Loaded symbols for /usr/lib64/php/modules/pdo.so
Reading symbols from /usr/lib64/php/modules/pdo_mysql.so…done.
Loaded symbols for /usr/lib64/php/modules/pdo_mysql.so
Reading symbols from /usr/lib64/php/modules/pdo_sqlite.so…done.
Loaded symbols for /usr/lib64/php/modules/pdo_sqlite.so
Reading symbols from /lib64/libnss_dns.so.2…done.
Loaded symbols for /lib64/libnss_dns.so.2

Core was generated by `/usr/sbin/httpd -k start’.
Program terminated with signal 11, Segmentation fault.
[New process 24052]
#0 0x0000000000423add in ap_strcasecmp_match ()

(gdb) bt full
#0 0x0000000000423add in ap_strcasecmp_match ()
No symbol table info available.
#1 0x00002b7c8edd1cda in watchLog (r=0x0) at mod_watch.c:557
key = <value optimized out>
data = (struct shEntry *) 0x4f92720
#2 0x0000000000426cfa in ap_run_log_transaction ()
No symbol table info available.
#3 0x0000000000440fc7 in ap_process_request ()
No symbol table info available.
#4 0x000000000043e44c in ap_process_http_connection ()
No symbol table info available.
#5 0x000000000043a802 in ap_run_process_connection ()
No symbol table info available.
#6 0x0000000000444d11 in child_main ()
No symbol table info available.
#7 0x0000000000444f84 in make_child ()
No symbol table info available.
#8 0x0000000000445027 in startup_children ()
No symbol table info available.
#9 0x0000000000445aef in ap_mpm_run ()
No symbol table info available.
#10 0x0000000000421838 in main ()
No symbol table info available.

(gdb) thread apply all bt full

Thread 1 (process 24052):
#0 0x0000000000423add in ap_strcasecmp_match ()
No symbol table info available.
#1 0x00002b7c8edd1cda in watchLog (r=0x0) at mod_watch.c:557
key = <value optimized out>
data = (struct shEntry *) 0x4f92720
#2 0x0000000000426cfa in ap_run_log_transaction ()
No symbol table info available.
#3 0x0000000000440fc7 in ap_process_request ()
No symbol table info available.
#4 0x000000000043e44c in ap_process_http_connection ()
No symbol table info available.
#5 0x000000000043a802 in ap_run_process_connection ()
No symbol table info available.
#6 0x0000000000444d11 in child_main ()
No symbol table info available.
#7 0x0000000000444f84 in make_child ()
No symbol table info available.
#8 0x0000000000445027 in startup_children ()
No symbol table info available.
#9 0x0000000000445aef in ap_mpm_run ()
No symbol table info available.
#10 0x0000000000421838 in main ()
No symbol table info available.
(gdb)

while i can’t comprehend everything that the gdb backtrace is producing, it would seem that apache may have an issue with spawning children processes and possibly have a problem with mod_watch.

with this being a busy server, here are my settings for apache:

Server Limit: 1024
Max Clients: 256
Start Servers: 60
Spare Servers (min): 40
Spare Servers (max): 186
Max Requests / Server: 4000
Timeout: 300
Keepalive: Off
Keepalive Requests (max): 100
Keepalive Timeout: 15

Anyone have any ideas? I would really like to get this resolved. Your help is greatly appreciated.

j

10

Thanks j, that was very helpful.

I built a patched mod_watch rpm that should address the segfault. I’ve tested it, but if you don’t mind I’d like you to give it a shot as well since you were able to consistently reproduce the problem.

Assuming you server is centos5, 64 bit, do the following as root:

rpm -Uvh http://updates.interworx.com/iworx/RPMS/cos5x/x86_64/mod_watch-4.3-101.rhe5x.iworx.p1.x86_64.rpm

Let me know how it goes and we’ll get the update pushed out assuming gets rid of those annoying errors.

Paul

11

thx paul. i ran the rpm and this is what is in the httpd/error_log after restarting apache:

[Wed Sep 09 03:57:51 2009] [info] mod_unique_id: using ip addr 127.0.0.1

[Wed Sep 09 03:57:52 2009] [notice] suEXEC mechanism enabled (wrapper: /usr/sbin/suexec)

[Wed Sep 09 03:57:52 2009] [info] Init: Seeding PRNG with 144 bytes of entropy

[Wed Sep 09 03:57:52 2009] [info] Init: Generating temporary RSA private keys (512/1024 bits)

[Wed Sep 09 03:57:52 2009] [info] Init: Generating temporary DH parameters (512/1024 bits)

[Wed Sep 09 03:57:52 2009] [info] Init: Initializing (virtual) servers for SSL

[Wed Sep 09 03:57:52 2009] [info] mod_ssl/2.2.8 compiled against Server: Apache/2.2.8, Library: OpenSSL/0.9.8b

[Wed Sep 09 03:57:52 2009] [info] mod_unique_id: using ip addr 127.0.0.1

[Wed Sep 09 03:57:53 2009] [info] Init: Seeding PRNG with 144 bytes of entropy

[Wed Sep 09 03:57:53 2009] [info] Init: Generating temporary RSA private keys (512/1024 bits)

[Wed Sep 09 03:57:53 2009] [info] Init: Generating temporary DH parameters (512/1024 bits)

[Wed Sep 09 03:57:53 2009] [info] Init: Initializing (virtual) servers for SSL

[Wed Sep 09 03:57:53 2009] [info] mod_ssl/2.2.8 compiled against Server: Apache/2.2.8, Library: OpenSSL/0.9.8b

[Wed Sep 09 03:57:53 2009] [notice] Apache/2.2.8 (Unix) DAV/2 PHP/5.1.6 mod_ssl/2.2.8 OpenSSL/0.9.8e-fips-rhel5 mod_watch/4.3 configured – resuming normal operations I REALLY don’t get this one!

[Wed Sep 09 03:57:53 2009] [info] Server built: Feb 19 2008 15:49:01

i will let you know if the seg fault 11’s keep stacking up. i really appreciate your efforts on this one. i have additional core files if you need them or would like to take a look. btw, the Max Clients maximum on ver. 4 iworx is 256. this appears to have been a change from 3.0.4 where i could set it to 300 with no probs. i found the 256 max notice when restarting apache from a command line. i have been bumping the 256 max quite often with children processes.

12

6.5 hours later and no segmentation faults are showing! all good now on that, but i am seeing the complaint:

[Wed Sep 09 10:34:27 2009] [info] server seems busy, (you may need to increase StartServers, or Min/MaxSpareServers), spawning 8 children, there are 0 idle, and 38 total children

i am trying raising the settings gradually in the CP for StartServers and Min/Max SpareServers. I have 5GB of mem on this server and could go much higher on these, but Apache is complaining of 256 MaxClients.

13

more info from /var/log/http/error_log with current apache settings:

i restarted apache and w/i 4 seconds i had

[Wed Sep 09 13:21:17 2009] [info] Server built: Feb 19 2008 15:49:01

[Wed Sep 09 13:21:21 2009] [info] server seems busy, (you may need to increase StartServers, or Min/MaxSpareServers), spawning 8 children, there are 27 idle, and 47 total children

[Wed Sep 09 13:21:22 2009] [info] server seems busy, (you may need to increase StartServers, or Min/MaxSpareServers), spawning 16 children, there are 38 idle, and 55 total children

current apache settings in CP:

Server Limit: 1024
Server Limit: 1024
Max Clients: 256
Start Servers: 60
Spare Servers (min): 40
Spare Servers (max): 186
Max Requests / Server: 4000
Timeout: 300
Keepalive: Off
Keepalive Requests (max): 100
Keepalive Timeout: 15

14

seg faults have stayed away. patch worked! thanks.

15

Thanks for keeping us updated on your progress with this issue. :smiley: