This morning I got a call from my boss. He was mad at me because none of our websites were working.
I quickly tried to access the Interworx-CP but I got a 404 error. Then I checked the server state (linode) and it was running fine. I also checked the DNS settings and everything was ok. I checked via SSH that the panel and the websites were actually there. Finally I restarted the server from the linode control panel and 5 minutes later everything was back on track again.
I would really like to know what was going on. Any clues? Could it have to do with the apache server?
InterWorx runs its own, separate instance of Apache, so it’s unlikely (but not impossible, of course) that both Apache servers failed simultaneously. For that matter…Apache has to be running in order to give the 404 in the first place.
None of which answers your question
Can you give more details about the setup? Is the /home directory part of the VM itself, or is it mounted from an external source? That would actually explain both webservers giving a 404 at the same time - if the mount had gotten screwed up, rebooting would have remounted and fixed the issue.
Tim, thanks for the answer Now that I think about it, it just said server not found. So there was no 404 error, my fault!
As for the setup. We are running interworx on a VPS with centos 5.2 32 bits. /home is on its own partition within the same VM as InterWorx-CP.
I’ve been looking into it and at around 4am we suddenly had nearly 10 thousand mails in the queue (!) I guess someone is sending spam through our mail server or something and that’s what caused our server to go down.
Ouch, bummer. That explains the InterWorx outage, I suppose pure load could explain the client Apache seeming to be offline, though I’m still not sure that’s the whole story.
The graphs seem to show the client Apache staying online until a few hours after the mail queue spiked. The two events may be unrelated, it’s hard to guess.
If you like, open a support ticket ( https://www.interworx.com/support/helpdesk/ ) and we can try to do a post-mortem. We may not be able to find anything concrete, though.
I had my mail queue look like that before. One client on one website had their account password stolen and were using my server to send out thousands and thousands of junk emails. I don’t remember it shutting down the whole server though, just email stopped working.
Now that I think about it, it just said 