Site performance, Server errors, outages and tunings

[Like Ra]

If anybody experienced slowdowns this morning - they were caused by a monitoring script which went mad for some reason. Something like race conditions. This is the first time it happened in several years. Restarted. Look OK.

Nice graph:

   

[Like Ra]

Site rippers were at it again. Huge traffic, memory overflow, swapping, slow downs, etc. The average page access time became as twice as much. Actually longer for one-two page visitors. Had to block by IP again. Then decreased the amount of PHP processes by one. It helped. Yes, 1GB RAM is definitely not enough. Especially if fappers downloading all the gigabytes for days.

[Like Ra]

Another upgrade!! See http://www.likera.com/blog/wp/archives/15031 !!!

I'm still playing with various parameters, but this is the current changes:

Simultaneous connections (Apache threads) - 105 (was 50)
PHP5-FPM processes - 22 (was 17)
Keep-alive TTL - 5sec (was 3)
APC cache size - 128MB (was 64)
Mod_pagespeed cache size - 160MB (was 90)
New MySQL caches (no significant changes, e.g. query_cache_size from 36MB to 48)

Does not look like tremendous changes, but one additional GB is gone 😁

Let's see how stable this environment will be.

[Like Ra]

Strange, the network fell off this morning. Had to reboot the box. Pity, I lost the performance statistics for the last 12 hours.

[Like Ra]

And it happened again. I did not have these problems before the migration. A trouble ticket logged.

[Like Ra]

The guys responded very quickly and proposed to migrate to another server. What I did. Probably it was a known issue? The CPU is now different:

Intel® Xeon® CPU E5-2630L 0 @ 2.00GHz 15MB cache

Meanwhile I increased the APC cache to 164MB and reduced the amount of PHP servers to 20. Watching the performance.

[Like Ra]

Next step will be MyISAM to InnoDB migration. Should be interesting to compare 😉

[Like Ra]

I migrated almost all tables to InnoDB except those with FULLTEXT index. MySQL 5.6 supports FULLTEXT index in InnoDB, but there is no supported version for Ubuntu yet.

Only one WP (blog) plugin is currently incompatible - YARPP (related posts), and I'm not quite ready to disable it.

MyBB (forum) is a more complex case, because searching depends on the full text index.

Apparently for a full scale comparison I have to wait for MySQL 5.6 availability for Ubuntu, but I hope that even the first phase will bring some performance improvements.

[Like Ra]

Ooops... The maximum memory a PHP process could you was set to 128MB, but APC cache to 164MB. Hence the "cache full" issues while there were lots of unused cache. Let's see if "matching the levels" helps.

[Like Ra]

More fun. I removed the FULLTEXT index from the MyBB posts table (since the FULLTEXT search is buggy anyway) and converted it to InnoDB. But the same trick with the threads table did not work, so I had to revert to MyISAM.

Removed some unused tables what reduced the database by ~20MB - the idea is to cache the entire database and all PHP files (via APC cache) into the memory.