[18253.961495] sched: RT throttling activatedThese simple 4 words were enough to put me on the right track and I asked, "Do you happen to have any processes using a Real Time scheduling policy?"...
Linux Scheduler
There's a "bit" of code in the kernel that decides which process will get to be executed next and that code is called Scheduler. In short, the scheduler is in charge of assigning CPU time to processes according to certain parameters. In Linux threads are scheduled considering two factors (ok, 2 + 1 if you want :P):
- The scheduling policy: there are basically two different types of scheduling policies, regular policies (SCHED_OTHER aka SCHED_NORMAL, SCHED_BATCH, SCHED_IDLE) and real time policies (SCHED_FIFO, SCHED_RR, SCHED_DEADLINE). The regular policies are implemented using the Completely Fair Scheduler (CFS):
- SCHED_NORMAL: regular scheduling policy, nothing fancy here, tasks get to be executed for a period of time and preemption is in place as well.
- SCHED_BATCH: allows tasks to run longer periods of time before preempting them, this improves for example the use of cache lines (but of course reducing interactivity).
- SCHED_IDLE: tasks with this policy have less priority than nice 19.
- SCHED_FIFO: is a simple scheduling algorithm without time slicing, based in a set of queues for the available priorities. Static priority must be higher than 0.
- SCHED_RR: unlike in SCHED_FIFO policy, threads under SCHED_RR run for a certain time slice called quantum and if they reach that time slice they are put at the end of its priority queue.
- SCHED_DEADLINE: uses three parameters, named "runtime", "period", and "deadline", to schedule tasks. A SCHED_DEADLINE task should receive "runtime" microseconds of execution time every "period" microseconds, and these "runtime" microseconds are available within "deadline" microseconds from the beginning of the period.
- Static priority: also known as sched_priority is a value between 1 and 99. This value only affects Real Time scheduling policies, it has no effect on regular policies and in fact should be set to 0.
Conceptually, the scheduler maintains a list of runnable threads for
each possible sched_priority value. In order to determine which
thread runs next, the scheduler looks for the nonempty list with the
highest static priority and selects the thread at the head of this
list.
There's another concept to mention when it comes to scheduling and is the "niceness" (this is the +1). This value, which only affects SCHED_OTHER and SCHED_BATCH policies, is used to influence the scheduler behavior to favor or disfavor certain threads. Linux's CFS (Completely Fair Scheduler) will then use the nice value to schedule the processes accordingly. Nice can take any integer value between [-20,19], being -20 the highest priority and 19 the lowest. This is the value you can change using nice command from your shell.
How do you find the Scheduling Policy of a process?
The easiest way to get the scheduling information from a process is by using our well known ps command. We can have a look at the process of the system like in the following example:
juan@test:~$ ps -eo pid,user,class,pri,nice,rtprio|awk '{print $3,$5,$6}'|sort|uniq -c
1 CLS NI RTPRIO
4 FF - 99
106 TS 0 -
1 TS 1 -
1 TS -10 -
1 TS 19 -
31 TS -20 -
1 TS 5 -
juan@test:~$
there we can see:
- 4 processes running with real time policy FF (SCHED_FIFO) with real time priority 99. Also note how the NI (nice) column is null for these processes as we described before.
- 141 processes with regular policy TS (SCHED_OTHER, you can see this in "man ps"). Also note how the RTPRIO (real time priority) column is null for these processes as we described before.
juan@test:~$ ps -eo pid,cmd,user,pcpu,class,pri,nice,rtprio|grep "FF\|PID"|grep -v grep PID CMD USER %CPU CLS PRI NI RTPRIO 11 [migration/0] root 0.0 FF 139 - 99 12 [watchdog/0] root 0.0 FF 139 - 99 13 [watchdog/1] root 0.0 FF 139 - 99 14 [migration/1] root 0.0 FF 139 - 99 juan@test:~$
- watchdog threads are there in order to identify software lockups conditions
- migration threads are there to handle load balance tasks among the available cores
Real time Policy
The core of the this post was actually real time policy right? Well lets have a look at the particularities of this policy and why it could become a headache easily.
- SCHED_FIFO policy doesn't use time slicing (SCHED_RR does though)
- So a SCHED_FIFO task will run until:
- A higher priority task goes in runable state and gets scheduled right away.
- It blocks due to IO
- Executes sched_yield call.
CPU Starvation
The problem a RT task can cause is called CPU starvation. The idea of the starvation situation is that there's a small number of tasks (even just one) consuming most or all of certain resource in this case the CPU time. Here you can see a simple example of a real time process that goes into an infinite loop causing CPU starvation to the rest (scheduling policy is set to SCHED_FIFO in line 23).
Code (remember you can pull this example or the others from https://github.com/jjpavlik):
#include <stdio.h>
#include <sched.h>
#include <errno.h>
#include <string.h>
#include <sys/time.h>
#include <sys/resource.h>
void main()
{
int sched,pri,pid,min,max,i,aux,error;
struct sched_param params;
pid = getpid();
sched = sched_getscheduler(pid);
pri = getpriority(PRIO_PROCESS,0);
printf("PID=%d\n",pid);
printf("Current scheduler is %d, current priority is %d\n",sched,pri);
printf("Priorities MIN=%d, MAX=%d, nice=%d\n",sched_get_priority_min(sched), sched_get_priority_max(sched),getpriority(PRIO_PROCESS,0));
printf("Changing scheduling class to SCHED_FIFO\n");
params.sched_priority=99;
aux = sched_setscheduler(pid,SCHED_FIFO,¶ms);
error=errno;
if( aux == -1)
{
//You need to run this as root :D, otherwise you will get a permission denied
printf("Setscheduler failed: %s\n",strerror(error));
return;
}
sched = sched_getscheduler(pid);
pri = getpriority(PRIO_PROCESS,0);
printf("Scheduler is %d, current priority is %d\n",sched,pri);
printf("Priorities MIN=%d, MAX=%d, nice=%d\n",sched_get_priority_min(sched), sched_get_priority_max(sched),getpriority(PRIO_PROCESS,0));
while(1)
{//Inifinite loop
i++;
}
}
Execution:
root@test:/home/juan/scheduling_tests# ./sched_details & [1] 3265 PID=3265 Current scheduler is 0, current priority is 0 Priorities MIN=0, MAX=0, nice=0 Changing scheduling class to SCHED_FIFO Scheduler is 1, current priority is 0 Priorities MIN=1, MAX=99, nice=0 root@test:/home/juan/scheduling_tests# root@test:/home/juan/scheduling_tests# ps -eo pid,cmd,user,pcpu,class,pri,nice,rtprio|grep "FF\|PID"|grep -v grep PID CMD USER %CPU CLS PRI NI RTPRIO 11 [migration/0] root 0.0 FF 139 - 99 12 [watchdog/0] root 0.0 FF 139 - 99 13 [watchdog/1] root 0.0 FF 139 - 99 14 [migration/1] root 0.0 FF 139 - 99 3265./sched_details root 99.3 FF 139 - 99 root@test:/home/juan/scheduling_tests#
Clearly the process is consuming almost completely one of the available cores in my system (99.3%), however we still don't see the throttling message on the logs because there's a second core available. So to push this even further I started a second run of the same binary to make sure the other core gets hammered as well and now:
root@test:/home/juan/scheduling_tests# ps -eo pid,cmd,user,pcpu,class,pri,nice,rtprio|grep "FF\|PID"|grep -v grep PID CMD USER %CPU CLS PRI NI RTPRIO 11 [migration/0] root 0.0 FF 139 - 99 12 [watchdog/0] root 0.0 FF 139 - 99 13 [watchdog/1] root 0.0 FF 139 - 99 14 [migration/1] root 0.0 FF 139 - 99 3299 ./sched_details root 97.9 FF 139 - 99 3338 ./sched_details root 95.8 FF 139 - 99 root@test:/home/juan/scheduling_tests#
After a few seconds you should see the throttling activated message on your logs:
root@test:/home/juan/scheduling_tests# dmesg |tail -1 [ 9640.055754] sched: RT throttling activated root@test:/home/juan/scheduling_tests#
at this stage the system is indeed crawling and barely responding to every command. But what does "sched: RT throttling activated" means after all?
There are two kernel parameters that help us reduce the impact of this situation with real time processes. Here you can see them:
root@test:/home/juan/scheduling_tests# cat /proc/sys/kernel/sched_rt_period_us 1000000 root@test:/home/juan/scheduling_tests# cat /proc/sys/kernel/sched_rt_runtime_us 950000 root@test:/home/juan/scheduling_tests#
- sched_rt_period_us specifies a scheduling period that is equivalent to 100% CPU bandwidth.
- sched_rt_runtime_us specifies how much of the rt_period can be used by real time and deadline tasks.
For example, changing 950000 to 500000 would allow RT tasks to use up to around 50% of the CPU time:
root@test:/home/juan/scheduling_tests# echo 500000 > /proc/sys/kernel/sched_rt_runtime_us root@test:/home/juan/scheduling_tests# cat /proc/sys/kernel/sched_rt_runtime_us 500000 root@test:/home/juan/scheduling_tests# root@test:/home/juan/scheduling_tests# ps -eo pid,cmd,user,pcpu,class,pri,nice,rtprio|grep "FF\|PID"|grep -v grep PID CMD USER %CPU CLS PRI NI RTPRIO 11 [migration/0] root 0.0 FF 139 - 99 12 [watchdog/0] root 0.0 FF 139 - 99 13 [watchdog/1] root 0.0 FF 139 - 99 14 [migration/1] root 0.0 FF 139 - 99 3394 ./sched_details root 58.7 FF 139 - 99 3395 ./sched_details root 50.0 FF 139 - 99 root@test:/home/juan/scheduling_tests#
so this way you can keep the real time processes under control.
However, considering real time tasks are supposed to be time sensitive you should avoid capping them like this unless it was really necessary. Maybe it's worth having a look at why they are consuming so much CPU time instead, usually real time tasks are operations that take really short periods of time but they need to be executed as soon as possible. So tools like strace or even perf should be really good starting points to identify the reason behind the CPU time, even better if you have access to the source code!
