CPU Limit

[interfasys]

I'd love to be able to set a CPU limit on some accounts, so that those don't take a server down.

[loopforever]

This isn't possible, since CPU load changes based on a number of factors. For example, you may have 100 users on your machine, each using MySQL - but MySQL runs as a single daemon, queuing up queries and executing them as a single user. We can say "Hey, MySQL is driving the load up." but we can't say "Hey, demouser is driving the load up." You can view MySQL processes, but this won't assist you in determining which user is increasing the load.

PHP and Apache is also another great example. You can see that Apache is driving the load up, but you can't see what script is doing it. The process just appears as Apache, doing something, which was commanded my someone, whom we cannot determine.

So really, it's not possible to limit CPU load on a per user basis.

[ProWebUK]

What it would be possible to do is install a process resouce monitor which monitors resource usage of applications then using variables set by you recieve email alerts etc.

If you wanted something like that i would be happy to put up a how-to guide :)

Just as a note - that would be server specific not user specific due to the reasons listed above.

Chris

[interfasys]

I think I saw some scripts that could could alert me in case Apache was overloading, but that script would let me know which domain was causing it, so I thought it would be possible to throttle it.

[S2S-Robert]

One of my users actually crashed my apache somehow with 150 apache processes and rising, so if you could tell me which scripts you found that would be great.

[interfasys]

Head over to WHT. In the service offer forum, you'll find script and services for sale that will help you (sorry for not having more info right now).

[ProWebUK]

Well some news that you may be suprised by

open httpd.conf with your fav text editor search for the line:

#ExtendedStatus On

once you find it uncomment the line and add the following:

<Location /serverstatus>
allow from YOURIP
deny from all
SetHandler server-status
</Location>

you should now have



ExtendedStatus On
<Location /serverstatus>
allow from YOURIP
deny from all
SetHandler server-status
</Location>


In your httpd.conf

Ensure you replace YOURIP with your actual IP address (this is so only you have access to the page - all other IP addresses will be disallowed access)

once your done run:

# service httpd restart

now go to:

yourdomain.com/serverstatus
SERVERIP/serverstatus

you will see a page showing each vhost whats happening etc.... aswell as CPU usage for each vhost.. so in reality... this feature is possible, although if you have followed this you will have noticed its off as default for performance issues - basically you have this but it will slow apache down and affect its performance.

This should come in handy when your servers resources seem to have shot up though :)

Chris

[l0rdphi1]

Very, very interesting. If only the 'CPU' column could be monitored, a script could VERY EASILY keep a running total per domain. Then you could "limit X CPU per minute" or something.

Also, if you could do this on a per-domain basis, you'd only set it up for people you know are high CPU eaters.

Hmm...

[ProWebUK]

Originally posted by l0rdphi1
Very, very interesting. If only the 'CPU' column could be monitored, a script could VERY EASILY keep a running total per domain. Then you could "limit X CPU per minute" or something.


Since your the php man...... would it be possible get that area out using php... dont forget you can use php in shell scripting..... so if its possible to split that part of the code up with any lang and basically leave with a domain variable with the CPU usage of that domain (could even interact with a database here)

The only thing i dont like about the idea is the performance issues... but then i have no idea what kind of performace effects this causes.

Chris

[l0rdphi1]

Yeah, that is all very feasible.

Also found this - http://httpd.apache.org/docs/mod/mod_status.html

"if you have lynx installed, you can also get a server statics report from the command line by running the command apachectl status, or, for the extended status report, apachectl fullstatus"

I need to isntall lynx and check that out :)


As for performance, I'm going to leave http://your.server.name/server-status?refresh=1 on for a bit and see what happens :)

[l0rdphi1]

Well, I have been invoking this every second for the past hour and I can say I'm seeing absolutely no increase in load. I think I may go a head and start playing with a program to monitor CPU on a per-domain basis.

[ProWebUK]

Performance issues wont neccessarily mean server load.... it could be page loading times (since each time a page is accessed its on that file etc.. if you have it running every 1 second for the application you are planning on developing this could cause problems there.... may want to do a bit of comparison between load times with pages etc before assuring yourself its not needed........

Chris

[l0rdphi1]

Hmm. Good idea.

Hopefully I can get apache to generate this stuff directly in a log file, but we'll see.

Thanks.

[l0rdphi1]

I decided to give a go at parsing the httpd data, and here's what I came up with:

http://www.liquenox.net/cpu.monitor.php?http://your.server.name/server-status

Replace 'http://your.server.name/server-status' with the URL to your stats page. Remember, this is only showing the totals for the data that would normally show on the page. Running totals over a set period of time is the next step, I guess.

[l0rdphi1]

Yes, and below is the source I'm running ...removed.. VB screws it up too bad...

http://www.l0rdphi1.com/junk/cpu.monitor.phps

[S2S-Robert]

Well this is looking tremendously good! The page is however still not working when I C&P, it does work using your php script.

And if this script is not a resource hog, you could just let it run every second and print out the results 'realtime'. Or run a graph next to it.

[l0rdphi1]

I wish you could get Apache itself to print to a log with the data found on the server-status page. It's impossible to get *everything* the way I'm doing now, even running the script every second. Seems silly that Apache wouldn't let you do that.

[S2S-Robert]

Perhaps there is some apache documentation that allows you to do this?

But even if you're not getting everything, it still provides a good way of seeing which sites take too much cpu time etc.

[l0rdphi1]

Yeah, even setting it to run every 5 minutes would give you a good idea. Haven't seen any docs on this (besides the one I linked), so I'll just go a head with the current method.

[ProWebUK]

Originally posted by S2S-Robert
And if this script is not a resource hog, you could just let it run every second and print out the results 'realtime'. Or run a graph next to it.

The graph could be done with mrtg, you just need the number (X) where X would be X% CPU usage...... thats not really wanted or needed for this though and just something that could be a nifty script for later on :p

lordphi: would it be possible to intergarte with mysql.... leaving the results to be used more easily

======================
Domain || <CPU USAGE>
======================
test.com || 0.01
your.com || 0.04
======================

etc

from there a shell script could simply be put in place with a 'max_cpu' variable.... a quick run through that table for a load higher than the var is specified as sends an email out to the admin (could also be a variable for the user to specify) possible admin@<DOMAIN_OVER_USAGE> and even suspend the account in directadmin for 2 minutes or a specified short period of time.. for the last idea we would need a command to suspend / unsuspend manually from command line

It may not be the best of things but I think its the best we could be getting without having individual daemons running for each domain.

Would make a brilliant 3rd party script if you ask me.... and it should work for all panels (at least everything but the domain suspension / unsuspension if thats possible :D)

Chris

[l0rdphi1]

Sounds good, Chris. All possible. I'll keep everyone here updated :D

[l0rdphi1]

I need some input.

As of the moment, I am taking a new CPU reading and averaging it with the existing value. In the case that there is no reading, I average a zero.

Is that how it should work? I can't decide.

[S2S-Robert]

Yes I guess that's how you'd do it. No running processes would mean a cpu load of 0.

Will this be a problem with the second reread though? The actual reading will take some time as well, and perhaps this way some domains can slip through. Although that won't be for the large ones eating up the resources that's true...

I'll think I'm really liking the way this project is taking! :D

[ProWebUK]

I would think the best way is:

=================================
MySQL
=================================
Domain || 1 || 2 || 3 || avg
=================================
test.com || 0.01 || 0.08 || 0.03 || 0.04
test2.com || 0.08 || 0.03 || 0.01 || 0.04
=================================

Basically the avg value is the average over 10 minutes which the script could use....

Every 5 minutes you get the cpu usage and do the following

1, 2 and 3 are the 5 minutes intervals - 1 being the most recent 2 being 5 minutes ago and 3 being 10 minutes ago.

Move the current value in 2 to 3
Move the current value in 1 to 2
Get the current CPU usage for this domain and insert that into 1

Basically every 5 minutes you move the previous value to 5 minutes later on.... after 10 minutes it gets overwirriten and removed.

Im getting the average by simply adding the last 3 usage amounts up then dividing it by 3 :)

Chris

[ProWebUK]

Originally posted by S2S-Robert
Yes I guess that's how you'd do it. No running processes would mean a cpu load of 0.

Will this be a problem with the second reread though? The actual reading will take some time as well, and perhaps this way some domains can slip through. Although that won't be for the large ones eating up the resources that's true...

I'll think I'm really liking the way this project is taking! :D

Have you tried the method?... if not, try it and view the page - the page gets updated every refresh, basically everytime http is accessed.

Chris

[S2S-Robert]

Why would you want a 5 minute 'screenshot' of the actual processes running? I see the second delay already as a problem, let alone the 5 minutes!

That would mean a lot of domains wouldn't have any cpu usage at all but will still be serving pages.

[edit]
So you would just set the refresh to 1 second then and put the calculated values into the mysql?

[l0rdphi1]

If you were updating @1 second I suppose your columns would be 1sec, 2sec and 3sec.

I like that method (of course the update interval will be changable).

[ProWebUK]

The php script can be set to get the values every X seconds... what you want to do with them is your choice - my idea to put them into mysql allows them to be stored neatly and easy access is then enabled.

An average over 10 minutes would be much more fair then an average over 3 seconds...... if there was something huggely wrong worth worrying about it would probably last more than 3 seconds.

Chris

[ProWebUK]

Dont like the idea of 1 second...... that means you have mysql running processes every second moving table rows every second along with that a php script parsing information of all data being accessed through apache to simply monitor the cpu usage of a domain.

As I stated in my previous post.... if they are doing something such as trying to kill your server or causing the server to slow down drastically for any reason it would surely last more than 3 seconds?!

Chris

[l0rdphi1]

Well it's updating into the DB now.

http://www.l0rdphi1.com/junk/cpu.monitor.tgz

You will have to enter your DB info at the top of the PHP file, and do remember to run the .sql file before the .php ;)

Next will be the account suspension stuff @X, I guess.

[ProWebUK]

Originally posted by l0rdphi1
Next will be the account suspension stuff @X, I guess.

I will do that if you want..... first we need John or Mark to get us a command to suspend and suspend for any reason

Could always finish it up by making a complete package.... with the idea:

make a copy / backup of your current php.ini
add whats needed to the php.ini
setup the sql tables
restart apache
setup cron jobs for php script
setup cron jobs for shell script to check usage and suspend appropritely.

Would make a neat little package i think :D

Chris

[DirectAdmin Support]

http://www.directadmin.com/api.html#suspend

It's not an offical "api" command.. but it works. Let me know if you want something better :)

John

[l0rdphi1]

Yeah, cool. There is one small problem though. To get a username based on their domain, I need to go through each of /usr/local/directadmin/data/users/*/domains.list until I find a match, correct? I guess that is the easiest way, is it not?

Thanks :D

[ProWebUK]

Would it be possibly to have a simple 1 line script.... I think the idea of an option to suspend accounts generally in the panel may also be a good idea..... if the site is an 'offender' you could simply tap that button instaed of removing their site totally or flooding the their quotas :p

/usr/local/directadmin/scripts/suspenduser suspend <user>
/usr/local/directadmin/scripts/suspenduser unsuspend <user>

Anything thats a 1 line argument and has 2 simple options of unsuspend or suspend and an option to specify the user will do me :D

Chris

[l0rdphi1]

That would make things easier, yes, but if we use a command line method like that, won't it render the script useless under safe_mode? I am *not* sure of that, let alone whether it's important or not, but it's something to find out.

On a side note, I set up a cron job this morning to run the script every 5 minutes, and let me tell you, I love this thing! :P

[ProWebUK]

No, this half would be totally seperate to the php.... the php has its own cron job and that would simply keep mysql up-to-date I assume and possibly just make a table of domains to suspend..... then the shell script simply checks that database for any domains, if its there it will suspend else nothing is affected.....

But dont forget everything is executed via shell and what scripts are run as can be chnaged..... aslong as nothing is accessable via the web (just set the area of your php.ini to deny all)

Chris

[l0rdphi1]

Ah yes, that makes sense. Great! :)

The script that disables/enables the accounts will be nothing more than:
SUSPENSION_DURATION = 300
MAX_CPU = 20

while "SELECT domain FROM cpu_monitor WHERE cpu_x1>MAX_CPU" {

user = GET_USER(domain)

`/usr/local/directadmin/scripts/suspenduser suspend <user>`

"INSERT INTO suspended (domain,user,time) VALUES (domain,user,TIME)"
// above could be cached and inserted all at once after the while()

}

while "SELECT user, domain FROM suspended WHERE time<TIME-SUSPENSION_DURATION" {

`/usr/local/directadmin/scripts/suspenduser unsuspend <user>`

"DELETE FROM suspended WHERE domain='domain' LIMIT 1"
// again, could be cached and ran all at once after the while()

}

[S2S-Robert]

The idea of suspending the entire domain is somewhat rough for my taste. If the domain exceeds a certain threshold I'd rather be e-mailed about that fact especially if the average of that domain exceeds the threshold for more than xxx hours.

I don't think load balancing will be in place right? Because that would be ideal, if the load of that particular domain is over the threshol level then just cap that domain at whatever you want.

[l0rdphi1]

Originally posted by S2S-Robert
The idea of suspending the entire domain is somewhat rough for my taste. If the domain exceeds a certain threshold I'd rather be e-mailed about that fact especially if the average of that domain exceeds the threshold for more than xxx hours.I like that. Sounds like a nice alternative to pissing off people :) However, since the script that would suspend or email is separate from the one that collects data, you can do whatever you want.


Originally posted by S2S-Robert
I don't think load balancing will be in place right? Because that would be ideal, if the load of that particular domain is over the threshol level then just cap that domain at whatever you want. Limiting the users' load would be ideal, but I'm not sure it's possible.

[ProWebUK]

Or an inbetween suggestion that covers both problems - set email alerts as soon as the usage goes to X then once it goes to X2 the site gets suspended..... after a certain load it would be quite stupid to leave the site up I would think

[ProWebUK]

I would say keep away from the following also:

SUSPENSION_DURATION = 300

have it keep checking mysql until the average has dropped to an ok amount (shouldnt take more than a few seconds) and then you simply re-enable the site.... maybe 3 attemps like that an hour then if they exceed the usage amount more than 3 times per hour its set to hold them down for a long duration or something.... not entirely should but leaving the suspend out totally after a certain point seems to remove the idea of the script..... and puttint it as the main feature seems to void it also as Robert stated..

Chris

[S2S-Robert]

And since you're already getting information out of the status page, would it be possible to get the number of pageviews as well? Because one of my clients accidentily created a loop from one index.php to another and back. The cpu wasn't very high, but since the number kept increasing is surely slowed down apache a lot.

If the number of processes an individual vhost has exceeds let's say 50 or 75 an e-mail would be sent as well. That would require 3 additional fields in the mysql database which would check the number of processes in the same way the cpu gets checked.

[l0rdphi1]

By "number of processes" which column are you referring to? 'Acc'? Forgive my ignorance :)

Should be able to finish this up tonight by the way.

[S2S-Robert]

Just the number of processes you see related to a specific vhost. What you do is add up all the loads of the processes, but I'm also interested in the actual number of the processes themselves.

In case of the include loop you'd soon be able to find out that there is an enormous amount of processes going on for a vhost.

[l0rdphi1]

Oh yes. Silly me. I do now see.

Thanks :D

[l0rdphi1]

The current tarball now has process monitoring. :)

http://www.l0rdphi1.com/junk/cpu.monitor.tgz

You will have to run the new sql file too (after dropping your old table).

I'll add the emailing bit when I get home later on today.

[S2S-Robert]

That's the part that check the database and see if a variable is higher than the threshold level? Because atm I'm still missing the actual purpose part.

And I'm not quite sure on how to run it, if I run it it simple add to the database, but how do I set it to automated runs say every minute?

If this program is finished I think this definately should be part of the chris' new server checklist ;)

[l0rdphi1]

Right now I'm running it via crontab every 5 minutes:
*/5 * * * * /usr/local/bin/php /home/........./cpu.monitor.php >/dev/null 2>&1

[ProWebUK]

Once its checked over release it in the how-to area and i'll link it up from the checklist :)

Chris

[l0rdphi1]

Well, I uploaded 1.0 of the new Domain Monitoring Tool (http://www.directadmin.com/forum/showthread.php?threadid=1650).