47

I operate a Linux system which has a lot of users but sometimes an abuse occurs; where a user might run a single process that uses up more than 80% of the CPU/Memory.

So is there a way to prevent this from happening by limiting the amount of CPU usage a process can use (to 10% for example)? I'm aware of cpulimit, but it unfortunately applies the limit to the processes I instruct it to limit (e.g single processes). So my question is, how can I apply the limit to all of the running processes and processes that will be run in the future without the need of providing their id/path for example?

Eduard Florinescu
  • 11,153
  • 18
  • 57
  • 67
Giovanni Mounir
  • 807
  • 2
  • 9
  • 12
  • Are you experiencing performance problems? or is it just the numbers that bother you? – ctrl-alt-delor Aug 24 '14 at 12:08
  • @richard Performance problems, so that's why I was trying to kill/limit/put an end to processes which seem to be using a lot of CPU, but I already did so by writing a bash script. This is also a virtual machine if that helps – Giovanni Mounir Aug 24 '14 at 12:16
  • 2
    Be careful of killing processes that may be 100% for very short time, also system processes. Consider `cpulimit` in conjunction with your search script. Have a policy and recommend the use of `cpulimit`, then search for over 10% and then limit to 5% (so users are encouraged to use `cpulimit`). Also make sure you can detect multiple processes adding up to more that 10% for a single user. – ctrl-alt-delor Aug 24 '14 at 12:28
  • @richard Thanks Richard for all of these pretty useful comments! They have helped me greatly! Your suggestion to use `cpulimit` is way better than just killing the process since it can be restarted by the user later on (as pointed in one of your comments). Thank you! – Giovanni Mounir Aug 24 '14 at 12:31
  • 1
    http://stackoverflow.com/questions/386945/limiting-certain-processes-to-cpu-linux – Ciro Santilli OurBigBook.com May 18 '16 at 09:02

7 Answers7

41

nice / renice

nice is a great tool for 'one off' tweaks to a system.

 nice COMMAND

cpulimit

cpulimit if you need to run a CPU intensive job and having free CPU time is essential for the responsiveness of a system.

cpulimit -l 50 -- COMMAND

cgroups

cgroups apply limits to a set of processes, rather than to just one

cgcreate -g cpu:/cpulimited
cgset -r cpu.shares=512 cpulimited
cgexec -g cpu:cpulimited COMMAND_1
cgexec -g cpu:cpulimited COMMAND_2
cgexec -g cpu:cpulimited COMMAND_3

Resources

http://blog.scoutapp.com/articles/2014/11/04/restricting-process-cpu-usage-using-nice-cpulimit-and-cgroups
http://manpages.ubuntu.com/manpages/xenial/man1/cpulimit.1.html

engelen
  • 103
  • 2
RafaSashi
  • 569
  • 5
  • 6
  • 5
    For those looking to set a hard limit on CPU usage, even when no other process is running, look at `cpu.cfs_quota_us` parameter (see [manual](https://access.redhat.com/documentation/en-US/Red_Hat_Enterprise_Linux/6/html/Resource_Management_Guide/sec-cpu.html#sect-cfs)) – Diego Jan 18 '16 at 14:59
  • cgroups are easier to use thanks to systemd directives... create a unit either system or user for that purpose is the best option – Yves Martin Dec 07 '16 at 09:03
  • for a running process ex.: `sudo cgclassify -g cpu:cpulimited 2315444` – Aquarius Power Aug 17 '17 at 21:27
  • 2
    my understanding is that nice only sets the relative CPU compared to other processes. If no other process is using CPU, then your process will use 100% CPU, not limit to 10%. – johny why Sep 19 '19 at 17:43
26

While it can be an abuse for memory, it isn't for CPU: when a CPU is idle, a running process (by "running", I mean that the process isn't waiting for I/O or something else) will take 100% CPU time by default. And there's no reason to enforce a limit.

Now, you can set up priorities thanks to nice. If you want them to apply to all processes for a given user, you just need to make sure that the user's login shell is run with nice: the child processes will inherit the nice value. This depends on how the users log in. See Prioritise ssh logins (nice) for instance.

Alternatively, you can set up virtual machines. Indeed setting a per-process limit doesn't make much sense since the user can start many processes, abusing the system. With a virtual machine, all the limits will be global to the virtual machine.

Another solution is to set /etc/security/limits.conf limits; see the limits.conf(5) man page. For instance, you can set the maximum CPU time per login and/or the maximum number of processes per login. You can also set maxlogins to 1 for each user.

Ishan Goel
  • 3
  • 2
vinc17
  • 11,912
  • 38
  • 45
  • 1
    @GiovanniMounir I meant: one virtual machine per user. – vinc17 Aug 24 '14 at 12:24
  • 1
    I see, but unfortunately this will be resource-consuming and not really useful for my purpose, as the users may require the usage of some common development packages, and I won't be installing this on every single new machine; I think it's better to leave it that way and do the monitoring automatically by a bash script. – Giovanni Mounir Aug 24 '14 at 12:26
  • 1
    @GiovanniMounir You can share a partition between several virtual machines. – vinc17 Aug 24 '14 at 12:35
  • @GiovanniMounir You can use [LXC](http://en.wikipedia.org/wiki/LXC) or [Docker](http://en.wikipedia.org/wiki/Docker_(software)) to decrease virtualization overhead to nearly zero. Also "liking" isn't a strong reason. For example I'd go with your solution if you're managing a shared PHP host because doing LXCs or Virtual Machines will require rewrite of a $15/$5 licensed software which is overkill. – Pooyan Khosravi Aug 25 '14 at 01:28
  • my understanding is that nice only sets the relative CPU compared to other processes. If no other process is using CPU, then your process will use 100% CPU, not limit to 10%. – johny why Sep 19 '19 at 17:42
12

Did you look at cgroups? There is some information on the Arch Wiki about them. Read the section about cpu.shares, it looks like it's doing what you need, and they can operate on a user-level, so you can limit all user processes at once.

Paul Schyska
  • 465
  • 4
  • 11
  • CGroups is the way to go, though. I also run (many) shared computer servers and we use cgroups to limit the maximum number of cores an *entire login session* can use. This way, if the person keeps starting new processes, each one gets a smaller slice. Same for memory use. You can have users automatically put into a cgroup with pam_cgroups and the cgrulesengd service. You can use a 'template' in the cgconfig file to put each user into their own cgroup. cgrulesengd acts like your script, except instead of killing processes, it just makes sure each process is in the right cgroup. – jsbillings Aug 24 '14 at 14:28
  • Even if you don't use cgroups to *limit* resource use, you can use it to *evaluate* how much resources an individual is using, by looking at the 'stat' file for each resource, then use that information for your 5 minute script. – jsbillings Aug 24 '14 at 14:31
7

For memory, what you are looking for is ulimit -v. Note that ulimit is inherited by child processes, so if you apply it to the login shell of the user at the time of login, it applies to all his processes.

If your users all use bash as login shell, putting the following line in /etc/profile should cause all user processes to have a hard limit of 1 gigabyte (more exactly, one million kilobytes):

ulimit -vH 1000000

The option H makes sure it's a hard limit, that is, the user cannot set it back up afterwards. Of course the user can still fill memory by starting sufficiently many processes at once.

For other shells, you'll have to find out what initialization files they read instead (and what other command instead of ulimit they use).

For CPU, what you wish for doesn't seem to make sense for me. What would be the use of letting 90% of the CPU unused when only one process is running? I think what you really want is nice (and possibly ionice). Note that, like ulimit, nice values are inherited by child processes, so applying it to the login shell at login time suffices. I guess that also applies to ionice but I'm not sure.

celtschk
  • 10,644
  • 1
  • 19
  • 27
  • Thanks for the memory suggestion! Is there any chance you can show me an example to apply this to the login shell of the user at the time of login? I'm not really sure how to do this. I'm also sorry for not being clear enough; what I'm trying to do is not allow any process to use more than 10% of the CPU. So do you think that `nice` will be nice enough to do this? If so, do you think you can show me an example to achieve this? – Giovanni Mounir Aug 24 '14 at 10:04
  • I still don't get the point of keeping the CPU 90% idle when only one process is running. – celtschk Aug 24 '14 at 10:07
  • 1
    If there are currently less than 10 processes running concurrently (and by running I mean *really* running, not just waiting for user input or disk I/O), then it is virtually *guaranteed* that one of them will have more than 10% of CPU. Otherwise the CPU would be virtually idling. And if you just kill any process that goes above 10%, I'm sure you'll have many users who will want to kill *you.* Or at least, will try to get you replaced by someone who has a clue about what those numbers mean, because you don't seem to. – celtschk Aug 24 '14 at 10:19
  • In contrast to @celtschk 's comment, if there are 11 or more processes running(cpu bound), then they will be at less than 9.09%. So if I am a user of a system that bans over 10% cpu usage I can run 11 or more processes, and hide under the radar. – ctrl-alt-delor Aug 24 '14 at 12:13
  • @richard You are right, perhaps it would be better if the script would sum up the total amount of memory/CPU used by a user, and then terminates all of this user's processes when the percentage reaches a specific amount (so would also log him out) – Giovanni Mounir Aug 24 '14 at 12:24
  • @GiovanniMounir There are various race conditions when doing the sum, and that's not accurate over some period. Taking a solution provided by the system is better. I've edited my answer to suggest `/etc/security/limits.conf` limits, for instance to set a maximum CPU time per login. – vinc17 Aug 24 '14 at 12:49
  • @vinc17 Thanks! I was aware of such function, but that wouldn't be very useful as the user can login multiple times and can workaround this limit, right? But I guess it's all fine! I have wrote a script to get the amount of RAM/CPU used by the users and terminate their session when the sum reaches a specific amount. The script would run every 4 minutes such that the processes the user is running shouldn't be using that much of RAM/CPU during the 4 minutes or the session gets closed. – Giovanni Mounir Aug 24 '14 at 12:59
  • @GiovanniMounir With the `maxlogins` type, you can restrict the number of simultaneous logins to 1 (though it doesn't always work due to a race condition… but see the following). You can also account the number/frequency of logins for each user via code in `/etc/profile`. If you detect that some user abuses, I suppose that you can deny the login by doing an `exit`. If you choose to do the work via your script, run it at random periods, otherwise users may know when it will be run and they will be able to stop most of their processes at this time. – vinc17 Aug 24 '14 at 17:21
  • @GiovanniMounir "the system specifications are high enough to make regular processes use less than 1% of the CPU usage, so using 10% is a huge number." <- That doesn't make any sense. If my process is the only one running, and it's CPU bound, then it will by definition use 100% of one core. It doesn't matter how fast that core is... – sapi Aug 25 '14 at 05:17
6

Since your tags have centos, you can use systemd.

For example if you want to limit user with ID of 1234:

sudo systemctl edit --force user-1234.slice

Then type and save this:

[Slice] CPUQuota=10%

Next time that user logs in, it will affect.

Man pages: systemctl, systemd.slice, systemd.resource-control...

stack-polices-are-trolls
  • 61
  • 1
  • 1
3

Since you are stating that cpulimit would not be practical in your case, then I suggest you look at nice, renice, and taskset, which may come close to what you want to achieve, although taskset allows to set a processes’s CPU affinity, so it might be not immediately helpful in your case.

R J
  • 812
  • 6
  • 9
  • 1
    `nice` and `renice`? That's nice! I have looked at their manual pages, but I still don't think they can help with this as you still have to set a process ID. If you could however give me an example which involves these packages to apply the limit on all running processes/future processes that would be awesome! – Giovanni Mounir Aug 24 '14 at 09:54
1

If you want to limit the processes that are already started, you will have to do it one by one by PID, but you can have a batch script to do that like the one below:

#!/bin/bash
LIMIT_PIDS=$(pgrep tesseract)   # PIDs in queue replace tesseract with your name
echo $LIMIT_PIDS
for i in $LIMIT_PIDS
do
    cpulimit -p $i -l 10 -z &   # to 10 percent processes
done

In my case pypdfocr launches the greedy tesseract.

Also in some cases were your CPU is pretty good you can just use a renice like this:

watch -n5 'pidof tesseract | xargs -L1 sudo renice +19'
Eduard Florinescu
  • 11,153
  • 18
  • 57
  • 67