Site performance, Server errors, outages and tunings

[Like Ra]

The system has just easily survived load 12. Something what was not possible in the morning. So, I think I nailed the problem - memory.

[Like Ra]

The whole fun (suspending, upgrading, migration, troubleshooting, tuning, multiple tickets to fix access to the server, troubleshooting, tuning, scripting, troubleshooting, etc) took me 5 days. One business week + overtime 😁

[Like Ra]

Load 16.67! Still up and no delays!

[Like Ra]

Wow!

load average: 20.38, 10.99, 4.85

[Like Ra]

OK, I consider that the problem is solved 😉

Now time for more performance tuning. Need to check if mysql is using the assigned cache effectively, if not I can free up more memory.

[Like Ra]

There was another server overload an hour ago. The load jumped to 64 - something no servers can survive. The reason is the same - low memory:

Mem: 1024768k total, 1018484k used, 6284k free, 5756k buffers

6M is way too low for memory hungry processes like PHP:

PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
5787 likera 16 0 199m 85m 40m S 0.0 8.6 0:09.36 php
5766 likera 16 0 199m 85m 40m S 7.1 8.5 0:18.17 php
5093 likera 15 0 197m 78m 35m S 0.0 7.8 0:01.72 php
5095 likera 16 0 195m 77m 34m S 0.0 7.8 0:01.32 php

Next step is to try to reduce the memory used by PHP.

[Like Ra]

Simple operation and the size of mysqld reduced from 572MB to 363MB. The cache size remained the same. Yes, I do learn. Still can't do anything with PHP. They eat a lot of memory. Is it just the WordPress code?

[Like Ra]

Is anybody still experiencing serious problem accessing the site?

Since yesterday there was only one load spike (on Jan 25) at 17:30:17 CET or 16:30 GMT.

[krinlyc]

Hi,

got an server timeout a few hours ago.

What the heck do you with 363MB mysqld ? There is something megawrong.

And some over 60 Users... In one second ? Over what period ? This load can handle a low budget PC in these days.

I do php programming and mysql(It's my job). I havn't seen anything near what you have for probs.
Sorry for the bad english...

krinlyc

[Like Ra]

There is definitely something wrong. With everything. Let's start with PHP:

1.

Jan 25 13:24:41 likera kernel: php[768]: segfault at 000000001d6a8f80 rip 000000001d6a8f80 rsp 00007fff398aeb18 error 15

2.

PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
1126 likera 16 0 201m 88m 39m S 0.0 8.8 0:02.74 php

3. Sometimes I'm getting out of memory PHP error (I know how to allow PHP to allocate more memory, but that should not happen).

Most problems are caused by WordPress. I'm currently running 3.1RC2. It's definitely better than 3.0.4 (less errors).

Next step - httpd:

[Tue Jan 25 22:26:19 2011] [error] mod_fcgid: process /usr/local/cpanel/cgi-sys/php5(32568) exit(communication error), get unexpected signal 11
[Tue Jan 25 22:26:30 2011] [warn] [client x.x.x.x] (104)Connection reset by peer: mod_fcgid: error reading data from FastCGI server, referer: http://www.likera.com/blog/wp/archives/6203
[Tue Jan 25 22:26:30 2011] [warn] [client x.x.x.x] (104)Connection reset by peer: mod_fcgid: ap_pass_brigade failed in handle_request function, referer: http://www.likera.com/blog/wp/archives/6203

This is what I have for fcgid:

LoadModule fcgid_module modules/mod_fcgid.so
MaxRequestsPerProcess 500
MaxProcessCount 16
IdleTimeout 20

and for the rest:

Timeout 20
TraceEnable On
ServerSignature Off
ServerTokens Minimal
FileETag All
StartServers 5
<IfModule prefork.c>
MinSpareServers 2
</IfModule>
ServerLimit 200
MaxClients 100
MaxRequestsPerChild 200
KeepAlive Off
KeepAliveTimeout 5
MaxKeepAliveRequests 100


Too tight, but the idea is to get rid of the processes ASAP.

MySQL:

[mysqld]
query_cache_size = 10M
thread_cache_size = 16
table_cache = 200
key_buffer_size = 16M
skip-networking
skip-name-resolve

The whole DB is around 45MB and 160 tables.

PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
20474 mysql 15 0 428m 36m 3348 S 13.6 3.7 168:32.73 mysqld

The resident part is 36MB, what is not that bad.