Website Slow/Crashing with Increased Traffic

NodeWorx General Discussion
1

Hi Everyone

I’ve got InterWorx CP installed and is still my preferred CP for managing websites over the other CPs that exist.

Recently I started hosting another website through SiteWorx that receives 32K unique daily traffic.

The other SiteWorx accounts don’t seem to be impacted but this particular one have I have started hosting seems to crash a lot or run very slowly. I ran the “System Health” but couldn’t see anything in there.

Distribution: CentOS Linux release 7.6.1810 (Core)
Operating System: Linux 3.10.0-957.21.3.el7.x86_64 (SMP)
CPU: [12x] 2200.00Mhz Intel® Xeon® CPU D-1531 @ 2.20GHz (9216 KB Cache)
System Memory: 31910.51 MB
Load Average: 1.49 1.27 1.13

NodeWorx Server Info
10% to 20% CPU usage on average
16GB of memory used on average

Using PHP-FPM 7.3.7 and Apache 2.4.39 and 5.5.60-MariaDB

I decided to migrate this off SiteWorx onto a completely new box that has Intel Core i7-6700 Quad-Core with 64 GB DDR4 RAM. First I tried migrating the web server only and kept the DB on the SiteWorx box but it still happened so I thought I’d migrate the DB over too and again the same thing, it crashed on the new box and ran very slowly too or crashed.

As the CPU and RAM on the old and new box aren’t maxing out can I rule out that it’s not a hardware issue? and something else like configuration? I did try changing the web server and mysql server configuration in InterWorx CP but that didn’t make any difference (too be honest I was just experimenting and had no idea what I was doing). The site was still slow or crashed. I then used Cloudflare service and setup “I’m under attack” anti ddos solution on the article pages where this paricular website was receiving the most traffic and didnt make a difference.

The new box is running NGINX as the front end with Apache backend and MariaDB all the same versions as the InterWorx CP box but with better hardware and only one website. I tried Cloudflare service and setup “I’m under attack” anti ddos solution on the article pages where this paricular website was receiving the most traffic and now the website seems be stable.

Someone told me to run the following command to see if there was a possible ddos attack based on the number of connections: netstat -an | egrep ‘:80|:443’ | grep ESTABLISHED | awk ‘{print $5}’ | grep -o -E “([0-9]{1,3}[.]){3}[0-9]{1,3}” | sort -n | uniq -c | sort -nr

and so the new box when Cloudflare anti ddos was turned off it had 700+ connections (crashed, slow). With Cloudflare anti ddos turned on the connections dropped down to 40+ connections (didn’t crash, fast).

Any ideas on where I should start next to try and find what is causing the site crashing/slowness would be greatly appreciated. Can provide logs where required. There are no slow queries for mysql in the log, persistent connections used with innodb for all sites.

Thanks

2

Hi dirtydex

What is the website - Wordpress by any chance

Your posted figures don?t seem to point anything specific

Are the other websites running normally whilst this website is running slow or crashing

Are there anything in the website error log shown from siteworx, administration logs hosting error log

Have you set php error show to more detail

How big is the website and it?s MySQL db

Many thanks

John

3

Hi John

Sorry for the delayed reply.

The website is written in pure PHP (no framework).

The other websites run normally whilst this particular website runs slow or crashes. The other websites in SiteWorx use the same PHP code base but have less traffic (only 1/4 of this particular website).

SiteWorx for this particular website (web server error) log has nothing now - assuming the logs are now purged or archived now.

On the box for httpd error log I saw the following:
[mpm_event:error] [pid 6188:tid 140355014551744] AH00484: server reached MaxRequestWorkers setting, consider raising the MaxRequestWorkers setting
[mpm_event:error] [pid 3600:tid 140370674055360] AH10159: server is within MinSpareThreads of MaxRequestWorkers, consider raising the MaxRequestWorkers setting

Haven’t set PHP error to show more detail. Would you like me to do this?

The website code is around 200KB in size and the database is 2MB in size.

Appreciate your help.

Regards,
dirtydex

4

Hi Dirtydex

Many thanks and Please note - any advice offered below is at your own risk and I do not endorse any links or software. Use at your own risk.

That sounds like your apache is not tuned perhaps. Also, have you updated Apache2 as our Apache is a version behind yours, unless it is a mistype or your running IW-Beta and not IW-RC

A good place to start would be run the 2 commands below (remember, RSS needs to be divided by 1024 for Mib) and must be run as root or sudo

curl -sL https://raw.githubusercontent.com/richardforth/apache2buddy/master/apache2buddy.pl | sudo perl

ps -ylC httpd --sort:rss

on apache2buddy, at the end is alink to php-fpmpal from Github - I would also use that php-fpmpal.sh

I am sure you may have read the following, but here it is if not

The above is based on your log extract (server reached MaxRequestWorkers) and sounds likely that your apache is going into mem swap, then timing out but I could be entirely wrong.

I would also look to see if there any long running PHP scripts from top command or if there is any long running Mysql queries perhaps as well. Also, check to make sure keep alive is enabled.

Many thanks

John

5

You my friend are the best.

I tried:
curl -sL https://raw.githubusercontent.com/ri…pache2buddy.pl | sudo perl

But this didn’t give me any recommendations I needed to do.

I tried:
php-fpmpal.sh

And this is where I got some recommendations about “max children”. The website that was having the issues had around x12 more children than was set. I found in the SiteWorx accounts under the “Package PHP-FPM Settings” had “Max Children” set value. As soon as I increased this and restarted PHP-FPM it fixed the issue.

I then had a “too many connections” issue for MariaDB and that was due to not having enough max connections set and the timeout for some reason was way too high. After changing these that resolved the DB issue.

Thanks again for all your help John.

6

Hi dirtydex

Glad it resolved your issue

Hope you like IW

Many thanks

John