AFF slow today ?

[GasoLane]

Is it just me, or is anyone else experiencing very slow page loading today/tonight ?

I also haven't received any email notifications from subscribed threads today either, could the two problems be connected ?

[Road Games]

Double post confirms it :/

[Road Games]

Since 10am mate but i thought it was just my new tablet or i had been put on miserable?

[tut0r]

I thought my internet was playing up but yeah very slow loads, not as bad as the 56k dial up days But my car can go faster than this site atm.....Thats not a good thing.

[ratter]

thought it was me as well

[duaned]

This website has been slow for months and out of the dozens of forums I visit, this one is the worst in relation to page loading etc. I recall the days it was super fast...

[russellw]

It is a bit slow at the moment so I'll restart the core services shortly.

Cheers
Russ

[Road Games]

Cheers Russ ... first world problems

[russellw]

... and restarted now.

Cheers
Russ

[MercurySilver]

yeah bit slow
got the cant post another post within 20secs thing once
thought it was my net as its quite overcast here
but other sites loading as normal

[Road Games]

Fixed now

[MercurySilver]

yup
moving heaps quicker
nice work russ

[GasoLane]

Definitely faster than yesterday but still no Email notification to Subscribed Threads though.

[prydey]

Originally Posted by GasOLane

Is it just me,

is it too late for me to say its just you that's slow? its called old age


oh, and no email notifications here either.

[GasoLane]

You're lucky my Coke Bottle reading glasses are broken and I couldn't read that Rob.

[russellw]

I'll restart the mail service which should fix email notifications.

Cheers
Russ

[Road Games]

Waaaaaaaaaaaaaaaaaaaay slow now Russ... has timed out a few times as well :/

[Trump]

Originally Posted by XR6menace

Waaaaaaaaaaaaaaaaaaaay slow now Russ... has timed out a few times as well :/

This. Have experienced several timeouts the past hour or so.

[Road Games]

Maybe block all of Hulk's new threads ???

[IngoStang]

yeah pages are taking 15-20 seconds to load for me but once there all info loads up straight away

[Geez Louise]

Appears to be getting really bad now.

I know you are working on it Russ. Hope you get it sorted soon

Cheers
Col

[Road Games]

Stoopid internetz :/

[russellw]

There is a rogue apache process running so the only option will be a server restart which I am about to do.

Current Time: Sunday, 25-May-2014 21:13:03 EST
CPU Usage: u31649.9 s141.1 cu74.97 cs0 - 468% CPU load
39.8 requests/sec - 0.6 MB/second - 15.8 kB/request
300 requests currently being processed, 0 idle workers

Cheers
Russ

[cs123]

Originally Posted by russellw

There is a rogue apache process running so the only option will be a server restart which I am about to do.

Current Time: Sunday, 25-May-2014 21:13:03 EST
CPU Usage: u31649.9 s141.1 cu74.97 cs0 - 468% CPU load
39.8 requests/sec - 0.6 MB/second - 15.8 kB/request
300 requests currently being processed, 0 idle workers

Cheers
Russ

0 idle workers might explain it. You can try jacking the number of worker processes up but it's a slippery slope if the machine is heavily loaded. No sure how many CPUs the box has but you want to keep the utilisation manageable.

With this type of web application you can pretty much assume that it is database service times that are causing the slow web response times and that is what you have to optimise. Not sure what optimisations you have already put into vbulletin but if the avatars or attachments are stored in the DB move them onto the file system. After that you need to look at SQL optimisation and possibly memcache. Then look at upgrading to the latest version of mysql as they normally include performance optimisations in the newer releases.

After you have looked at all that then have a look at process priorities for mysql. You can jack the process priority of the database processes to give them priority over other processes. This helps performance when running mixed workload (web server and database server) and the run queues are long.

There are other things you can do like setting noatime on the volumes for the database and webserver files. This just stops the OS from updating the last access time on the files. It is redundant for a server like this and slows things down. You need to modify the /etc/fstab file with the noatime option and remount.

As a first step monitor the worker processes and if you often have 0 free then increase it a bit at a time and monitor performance. Also grab a snapshot of the machine using vmstat 5 5 when performance is slow.

[russellw]

Originally Posted by cs123

0 idle workers might explain it. You can try jacking the number of worker processes up but it's a slippery slope if the machine is heavily loaded. No sure how many CPUs the box has but you want to keep the utilisation manageable.

With this type of web application you can pretty much assume that it is database service times that are causing the slow web response times and that is what you have to optimise. Not sure what optimisations you have already put into vbulletin but if the avatars or attachments are stored in the DB move them onto the file system. After that you need to look at SQL optimisation and possibly memcache. Then look at upgrading to the latest version of mysql as they normally include performance optimisations in the newer releases.

After you have looked at all that then have a look at process priorities for mysql. You can jack the process priority of the database processes to give them priority over other processes. This helps performance when running mixed workload (web server and database server) and the run queues are long.

There are other things you can do like setting noatime on the volumes for the database and webserver files. This just stops the OS from updating the last access time on the files. It is redundant for a server like this and slows things down. You need to modify the /etc/fstab file with the noatime option and remount.

As a first step monitor the worker processes and if you often have 0 free then increase it a bit at a time and monitor performance. Also grab a snapshot of the machine using vmstat 5 5 when performance is slow.

We have 16 cores in total. Avatars and attachments are already stored in the file system and we run APC as an opcode cache although I will check that it's running.

It's been running fine under this (and more) load for some time but I suspect that the last update to mysql replaced the tuned config file with a vanilla one. I am working on that now.

Apache is running out of workers as a result of mysql wait states and while there are no slow queries it is just the volume of queries that are creating short queues and keeping workers idle.

I've got the original conf files so they can go back and we can see what happens then.

There are 16 cores in the box and CPU utilisation has only been a problem this last 36 hours or so which suggests that something has changed and I have to nail down what.

Cheers
Russ

[my_gxl]

Its been telling me page can't be displayed

[russellw]

Restart completed. We'll watch and see over the next few hours.

Cheers
Russ

[russellw]

The server came up but then went down again - I have requested a hardware reboot and some troubleshooting of the issue once it is back up again (which it will be once you can read this).

CPU Usage: u5.39 s.1 cu0 cs0 - 28.9% CPU load

Cheers
Russ

[russellw]

The server came up but then went down again - I have requested a hardware reboot and some troubleshooting of the issue once it is back up again (which it will be once you can read this).

CPU Usage: u5.39 s.1 cu0 cs0 - 28.9% CPU load

Cheers
Russ

[Road Games]

I don't understand any of that but i hope it ain't too troublesome matey :/