Since switching over to IW I have a recurring problem with the server running out of control about 4-5 times a week. CPU Load, Memory, Threads/Processes just go crazy. This usually happens overnight in the wee hours. I have tried to eliminate everything I can find that fires off overnight (backups, scripts, etc), but nothing so far has made a huge difference. When this happens the server becomes unreachable. Sometimes it comes back, sometimes (less seldom) not. Either way, not good.
Has anyone seen similar problems? Anyone have suggestions on things to look for?
Do you have any idea at all what is causing the spike? If not, I would try to catch the problem redhanded, and run “top” in SSH to try to determine what is going on and using up all the resouces.
No idea at all. Monday I shut down all log processing, tweaked the Apache server (min/max) settings and cleaned up some error log files. Helped on Tuesday morning, but the problem came back this AM. Problem is that it is inconsistent, which makes it hard to catch (not to mention that I am usually asleep at that point). I was hoping to find a combined log of server activity, but it appears that IW stores each site log separately. With 70+ sites, that makes it tough to find something…
ok, so I logged in this AM and saw your server maxxed out again, however today was the first time I was able to get into it.
i looked and there was 100s of HTTPD processes. So maybe that will help you track it down. I changed the memory on the fly first to 20GB and then to 40GB and it balloned up instantly to the max.
So maybe look at reducing the number of httpd servers that apache will spawn? The setting is in interworx.
Sounds like some website is just getting hammered with traffic, just strange that it occurs late at night. I would check which SiteWorx account has used the most BW for a clue as to which site is causing the problems.
I do see one site that seems disproportionate in the bandwidth department, but a review of the logs in Analog/Webalizer, etc nothing sticks out. Site is busy during business hours, drops off at night. The only oddity that I saw was the “country” list which showed a country called “network” using 52% of the hits/visits.
Hmm… So during the heavy traffic it’s fine, but at night it goes down. Kind of tosses out my guess.
When does your Iworx daily cron run? Maybe it’s causing some issue while creating the stats. If it’s around the time, you could try turning off some of the stats. Just leave one of them on and see if that helps. You can do this now right from NodeWorx under “Server > Settings”.