That Damn Lua SHM issue

Hi,

Anyone know what methods to use besides restarting the entire server just to get Apache to turn back on? I’ve tried rm-ing the file in question just so it’ll go, “Oh… it’s not there no moar, let’s use it!” but, nothing.

Here’s the error:

[Fri May 06 05:41:02.663233 2016] [suexec:notice] [pid 5764] AH01232: suEXEC mechanism enabled (wrapper: /usr/sbin/suexec)[Fri May 06 05:41:02.807632 2016] [auth_digest:notice] [pid 5764] AH01757: generating secret for digest authentication …
[Fri May 06 05:41:02.808685 2016] [lua:error] [pid 5764] (17)File exists: mod_lua: Failed to create shared memory segment on file /tmp/httpd_lua_shm.5764
[Fri May 06 05:41:02.808702 2016] [:emerg] [pid 5764] AH00020: Configuration Failed, exiting
[Fri May 06 05:46:43.772023 2016] [suexec:notice] [pid 1594] AH01232: suEXEC mechanism enabled (wrapper: /usr/sbin/suexec)
[Fri May 06 05:46:44.340583 2016] [auth_digest:notice] [pid 1594] AH01757: generating secret for digest authentication …
[Fri May 06 05:46:44.341673 2016] [lbmethod_heartbeat:notice] [pid 1594] AH02282: No slotmem from mod_heartmonitor
[Fri May 06 05:46:44.351675 2016] [mpm_prefork:notice] [pid 1594] AH00163: Apache/2.4.10 (CentOS) OpenSSL/1.0.1e-fips mod_watch/4.3 configured – resuming normal operations
[Fri May 06 05:46:44.351720 2016] [core:notice] [pid 1594] AH00094: Command line: ‘/usr/sbin/httpd -D SSL -D FOREGROUND’

Sidenote: Google is less than helpful, here. Either that, or my Google-fu has failed me. So I figure I may as well post here so that the solution is publicly available for anyone else on IW that encounters this… damn… issue!

Can you tell I’m frustrated with this issue? :stuck_out_tongue:

Hi kerio

If it’s just for restarting Apache, ssh into server and run sudo service httpd restart (or stop or start)

Also, are you on Apache 2.4

I maybe wrong, as I usually am, but I thought it was Apache 2.3. This maybe for centos 6 though

Many thanks

John

[QUOTE=d2d4j;28486]Hi kerio

If it’s just for restarting Apache, ssh into server and run sudo service httpd restart (or stop or start)

Also, are you on Apache 2.4

I maybe wrong, as I usually am, but I thought it was Apache 2.3. This maybe for centos 6 though

Many thanks

John[/QUOTE]

That does not help at all with Apache crashing due to that damn Lua SHM issue. Once it crashes, you have to restart the entire server. Restarting the web server will NOT work. I mentioned in the OP “besides restarting the entire server” for a reason. :slight_smile:

Also,

[root@helmsgate html]# httpd -v
Server version: Apache/2.4.10 (CentOS)
Server built: Apr 4 2016 10:23:24

We are running on CentOS 7. With InterWorx-CP v5.1.11-1084 as the version we’re running.

A restart does the job for me BUT this isn’t a solution. I got this issue twice in one week and what will happen if I dont have ssh access?

httpd -v
Server version: Apache/2.4.10 (CentOS)
Server built: Apr 4 2016 10:23:24
Centos 7
InterWorx Version: 5.1.14-1101

We’re hitting this issue as well and its not looking good. We’ve had to reboot server twice in the last month and just this morning again. We have customers threatening to leave.

john

Hi

I’m not sure if kerio found the answer and I am not too sure what this issue is but perhaps the following may help

Many thanks

John

service httpd reload

service httpd status

apachectl restart

[QUOTE=d2d4j;28945]Hi

I’m not sure if kerio found the answer and I am not too sure what this issue is but perhaps the following may help

Many thanks

John

service httpd reload

service httpd status

apachectl restart[/QUOTE]

That does not help. No, I have not found the answer. When this issue hits the server, the only way to fix it is to reboot it. Not even a restart of httpd will work. Specifically, it only delays the problem, it does not resolve the problem permanently.

No reloading http or even stop and start does not work. Theres zombie processes which we’ve tried to kill as well. The only solution we’ve found to work is to reboot the server. However, keep in mind that this is centos 7 so service commands are deprecated. We use systemctl instead.

Hi

Many thanks, and yes, aware service deprecated in centos 7, but still works, or appear too

Have you tried first command to reload httpd, then start httpd

Have you traced your zombie, and is it that you cannot kill the process and child

Have you opened a support ticket with IW

Also, it’s in my mind if soft restart of httpd is ticked, is this causing the zombie (sorry I forget the exact name but it is listed in nodeworx, services, web server

How would I be able to duplicate this issue on one of our centos 7 test servers.

I hope that helps

Many thanks

John

Hi

I’m sure you have seen this but it looks very similar to your issue

Many thanks

John

https://bz.apache.org/bugzilla/show_bug.cgi?id=54622

Hey all,

I’ve seen this error a couple times. So far we’ve recommended disabling mod_lua as a work around if it’s not in use. Javapipe or Keiro can you submit a support ticket with credentials for an affected server?

Hi

I see the change log shows a fix pushed out on the 22 August to resolve the lua issue with Apache

Kudos to IW for swift fix

You will need to update to release candidate to get the fix, unless IW have adjusted your server directly

Many thanks

John

[QUOTE=d2d4j;28970]Hi

I see the change log shows a fix pushed out on the 22 August to resolve the lua issue with Apache

Kudos to IW for swift fix

You will need to update to release candidate to get the fix, unless IW have adjusted your server directly

Many thanks

John[/QUOTE]

If that’s the case, apparently not… because I was just bitten by this issue again a couple hours ago.

I’m running InterWorx-CP v5.1.41 on the server in question. CentOS 7, as usual.

Edit: Just saw your request to provide access to an affected server, @IWorx-Nathan.

You have a ticket incoming.

Edit2: I have disabled LUA as per Nathan’s suggestion.

Hi keiro
Appreciate if you could update after IW have resolved your issue.
As we do not use lua or centos 7/apache 2.4 yet, I am not sure what the issue is, but aftre reading a little myself, I am thinking it maybe memory leak, and lua is rebooting but the zombie shutdowns are taking to long, or cannot be shutdown. I suppose a test would be to lower the session count for lua to reboot, and possible control the memory leaks quicker.
Apologies if I am totally wrong, but just a thought
Many thanks
John

[QUOTE=d2d4j;29456]Hi keiro
Appreciate if you could update after IW have resolved your issue.
As we do not use lua or centos 7/apache 2.4 yet, I am not sure what the issue is, but aftre reading a little myself, I am thinking it maybe memory leak, and lua is rebooting but the zombie shutdowns are taking to long, or cannot be shutdown. I suppose a test would be to lower the session count for lua to reboot, and possible control the memory leaks quicker.
Apologies if I am totally wrong, but just a thought
Many thanks
John[/QUOTE]

Yeah, I decided to unload it on the server experiencing the issue. We’ll see how it goes, as that issue is quite annoying.

But definitely, I will update after IW’s had a look at the server.

Nathan, perhaps the upcoming Apache update will resolve this issue?

Hi Justin

Do you mind me asking if you know if it is lua requesting a reboot or Apache

If lua, then I think it is lua which needs to resolve issue

Many thanks

John

No, I haven’t personally had this issue. Just had a PCI scan say that running LUA and my version of Apache had some kind of security bug. So I just removed mod_lua from loading. I only had my new CentOS 7 server running for about 2 weeks before removing LUA. Not sure how often this occurs, but maybe my disabling of LUA fixed it before it ever became a problem.

Here is the issue I was referring to: https://access.redhat.com/security/cve/cve-2014-8109