Hello,
Since few weeks we have huge segfault11 in our httpd errors log. We tryed to remove all no default package (as mod_security, mod_evasive, some php, etc…) but we still encounter this problem. As sometimes segfault 11 is more an hardware pbm, we changed the memory, and one HDD. But still have these errors…
We tried a lot of things and finally we decided start HTTPD and strace it.
We have a PIV 3.0Ghz Dual Xeon with 3.0GB ram and 2x300GB scsi HDD in hardware RAID 1.
There is no others revelant logs in httpd/error_log the segfault appears without any kind of error before it.
Here is what we’ve done to strace these segfault
strace -f -o /tmp/http_trace /etc/init.d/httpd start
After few minutes, as expected, we might see some segfault in error_log
Here is the last lines in strace of the pid that has been in segfault11 state : pid 12431
12431 gettimeofday({1146246935, 850984}, NULL) = 0
12431 gettimeofday({1146246935, 851106}, NULL) = 0
12431 write(106, “[Fri Apr 28 19:55:35 2006] [erro”…, 133) = 133
12431 semop(2818050, 0, 213560479168) = 0
12431 semop(2818050, 0x42a367, 213560479174) = 0
12431 semop(2818050, 0x42a367, 213560479168) = 0
12431 semop(2818050, 0x42a367, 213560479174) = 0
12431 stat(“/home/emulezen/emule-zenzone.com/html/quizz”, {st_mode=S_IFDIR|0755, st_size=4096, …}) = 0
12431 stat(“/home”, {st_mode=S_IFDIR|0711, st_size=4096, …}) = 0
12431 stat(“/home/emulezen/emule-zenzone.com/html/quizz/error/HTTP_NOT_FOUND.html.var”, 0x7fbffff4c0) = -1 ENOENT (No such file or directory)
12431 lstat(“/home”, {st_mode=S_IFDIR|0711, st_size=4096, …}) = 0
12431 lstat(“/home/emulezen”, {st_mode=S_IFDIR|S_ISGID|0751, st_size=4096, …}) = 0
12431 lstat(“/home/emulezen/emule-zenzone.com”, {st_mode=S_IFDIR|S_ISGID|0711, st_size=4096, …}) = 0
12431 lstat(“/home/emulezen/emule-zenzone.com/html”, {st_mode=S_IFDIR|S_ISGID|0755, st_size=4096, …}) = 0
12431 lstat(“/home/emulezen/emule-zenzone.com/html/quizz”, {st_mode=S_IFDIR|0755, st_size=4096, …}) = 0
12431 lstat(“/home/emulezen/emule-zenzone.com/html/quizz/error”, 0x7fbffff4c0) = -1 ENOENT (No such file or directory)
12431 gettimeofday({1146246935, 852340}, NULL) = 0
12431 write(106, “[Fri Apr 28 19:55:35 2006] [erro”…, 128) = 128
12431 writev(279, [{“HTTP/1.1 404 Not Found
Date: Fr”…, 162}, {“<!DOCTYPE HTML PUBLIC "-//IETF//”…, 398}], 2) = 560
12431 write(241, “207.46.98.87 - - [28/Apr/2006:19”…, 132) = 132
12431 gettimeofday({1146246935, 852670}, NULL) = 0
12431 — SIGSEGV (Segmentation fault) @ 0 (0) —
12431 chdir(“/etc/httpd”) = 0
12431 rt_sigaction(SIGSEGV, {SIG_DFL}, {SIG_DFL}, 8) = 0
12431 kill(12431, SIGSEGV) = 0
12431 rt_sigreturn(0x308f) = 0
12431 — SIGSEGV (Segmentation fault) @ 0 (0) —
12011 <… wait4 resumed> [{WIFSIGNALED(s) && WTERMSIG(s) == SIGSEGV}], WNOHANG|WSTOPPED, NULL) = 12431
and here is strace lines of an other PID in segmentation fault 11
12094 <… gettimeofday resumed> {1146246841, 405045}, NULL) = 0
12094 gettimeofday( <unfinished …>
12094 <… gettimeofday resumed> {1146246841, 405181}, NULL) = 0
12094 write(104, “[Fri Apr 28 19:54:01 2006] [erro”…, 135 <unfinished …>
12094 <… write resumed> ) = 135
12094 semop(2818050, 0, 213560479168 <unfinished …>
12094 <… semop resumed> ) = 0
12094 semop(2818050, 0x42a367, 213560479174 <unfinished …>
12094 <… semop resumed> ) = 0
12094 semop(2818050, 0x42a367, 213560479168 <unfinished …>
12094 <… semop resumed> ) = 0
12094 semop(2818050, 0x1, 213560479174 <unfinished …>
12094 <… semop resumed> ) = 0
12094 stat(“/home/ericslif/ericslife.net/html/blog”, <unfinished …>
12094 <… stat resumed> {st_mode=S_IFDIR|0775, st_size=4096, …}) = 0
12094 stat(“/home”, <unfinished …>
12094 <… stat resumed> {st_mode=S_IFDIR|0711, st_size=4096, …}) = 0
12094 stat(“/home/ericslif/ericslife.net/html/blog/error/HTTP_NOT_FOUND.html.var”, <unfinished …>
12094 <… stat resumed> 0x7fbffff4c0) = -1 ENOENT (No such file or directory)
12094 lstat(“/home”, <unfinished …>
12094 <… lstat resumed> {st_mode=S_IFDIR|0711, st_size=4096, …}) = 0
12094 lstat(“/home/ericslif”, <unfinished …>
12094 <… lstat resumed> {st_mode=S_IFDIR|S_ISGID|0751, st_size=4096, …}) = 0
12094 lstat(“/home/ericslif/ericslife.net”, <unfinished …>
12094 <… lstat resumed> {st_mode=S_IFDIR|S_ISGID|0711, st_size=4096, …}) = 0
12094 lstat(“/home/ericslif/ericslife.net/html”, <unfinished …>
12094 <… lstat resumed> {st_mode=S_IFDIR|S_ISGID|0755, st_size=4096, …}) = 0
12094 lstat(“/home/ericslif/ericslife.net/html/blog”, <unfinished …>
12094 <… lstat resumed> {st_mode=S_IFDIR|0775, st_size=4096, …}) = 0
12094 lstat(“/home/ericslif/ericslife.net/html/blog/error”, <unfinished …>
12094 <… lstat resumed> 0x7fbffff4c0) = -1 ENOENT (No such file or directory)
12094 gettimeofday( <unfinished …>
12094 <… gettimeofday resumed> {1146246841, 406857}, NULL) = 0
12094 write(104, “[Fri Apr 28 19:54:01 2006] [erro”…, 125 <unfinished …>
12094 <… write resumed> ) = 125
12094 read(279, <unfinished …>
12094 <… read resumed> 0x10ef0a8, 8000) = -1 EAGAIN (Resource temporarily unavailable)
12094 writev(279, [{“HTTP/1.1 404 Not Found
Date: Fr”…, 198}, {“<!DOCTYPE HTML PUBLIC "-//IETF//”…, 398}], 2 <unfinished …>
12094 <… writev resumed> ) = 596
12094 write(239, “213.103.110.20 - - [28/Apr/2006:”…, 152 <unfinished …>
12094 <… write resumed> ) = 152
12094 gettimeofday( <unfinished …>
12094 <… gettimeofday resumed> {1146246841, 407418}, NULL) = 0
12094 — SIGSEGV (Segmentation fault) @ 0 (0) —
12094 chdir(“/etc/httpd” <unfinished …>
12094 <… chdir resumed> ) = 0
12094 rt_sigaction(SIGSEGV, {SIG_DFL}, <unfinished …>
12094 <… rt_sigaction resumed> {SIG_DFL}, 8) = 0
12094 kill(12094, SIGSEGV <unfinished …>
12094 <… kill resumed> ) = 0
12094 rt_sigreturn(0x2f3e <unfinished …>
12094 <… rt_sigreturn resumed> ) = 0
12094 — SIGSEGV (Segmentation fault) @ 0 (0) —
12055 gettimeofday({1146246842, 12094}, NULL) = 0
12011 <… wait4 resumed> [{WIFSIGNALED(s) && WTERMSIG(s) == SIGSEGV}], WNOHANG|WSTOPPED, NULL) = 12094
Any idea of what could cause this ?
Apparently it appears every time there is
write(104, “[Fri Apr 28 19:54:01 2006] [erro”…, 135 <unfinished …>
Thanks for your advises
Pascal