New V2 CGroups inside Systemd using systemctl. Share user CPU and memory usage fairly

CPU & memory usage, can be hard limited or soft limited with CGroups.  Why is this great? It stops rogue users crashing the OS and ruining every other user's day.  There are lots of guides, but they are confusing because CGroups has V1 and V2,  with significant changes.  There doesn't seem to be many easy tutorials - this blog worked in November 2018 and uses V2 CGroups.

I recently had a problem on a multi user computer.  One user was demanding so much resource that the computer couldn't keep up.  *nix skills to the rescue!

Long Story about Multi-User Computer:

TL:DR FML.  Now skip to 'the Fix'.

Oh... you really care?  OK, imagine the scenario as follows, machine has 16 cores:

User 1(myself):  Running code on a single processor for long term test that take days to weeks of continuous computation
User 2  Running OMP tasks using 8 cores
User 3: Running OMP tasks using 13 cores

The computer (which is a linux box that is used for grunt processing) only has 16 cores, so obviously something gives.  Sadly  the computer  doesn't know that non of these processes are interactive, so it tries to carry out everything at once.   This means that each process gets a slices of 100ms in the ratio 1:8:13.  Worse still my code running on 1 core is now geting 72.7% of a core.  This wouldn't be the end of the world, but my code is quite RAM intensive, so instead of running for 100ms every 2.2s  it actually spends most of that time loading data in and out of the cache.  A run that I know used to take 40 hours went to over 8 days.  This affects everyone equally, but if you are using OMP tasks then the cache is shared and there is no need to load data in or out of it.  This means that you won't lose so much time when passing from one of your slices to another of your slices. My process only get one slice, so I'm always handing off to another user.

User 2 is under a lot of pressure and I have chosen not to push it.

Politely asking to User 3 to curtail their OMP task number to the number of cores available did not work.  Apparently the user with 13 cores has a 'bug' in their code that they can't explain or show me.  They claim not to understand it, but it magically requires 13 cores, they won't explain it to me or try to fix it.  They have also convinced our boss that this is reasonable behaviour.

User 3 routinely crashes the system when they decide to run code that leaks memory.  This obviously makes me lose weeks of work.  Yes, I should save state and be able to reload the code.  Its on my list, but it will take weeks to shoe horn in.

But wait... linux has an existing system for just my situation!  CGroups? Great! They create soft limits on CPU time, if there is no contention then processes are not curtailed, if there is, then time processor time is proportional to your CPUShare.  I talk to the sysadmin, he's sympathetic!  He asks me to find out about CGroups so that we share the load of putting them onto the system.

The Fix

Looking online it seems simple.  It turns out that it is, but most of the online documents are for V1 not V2!  Unless you are working with a legacy system, then the solution for giving users CPUShares and setting a memory limit are as follows.

Obligatory note: This is a fairly technical blog, it is your responsibility to know what you are doing. Take backups of all files that you are changing (e.g. sudo cp .safe) before you cut and paste a script from this blog, understand what it does, stay inside the magic circle.  Polite feedback welcome,  mistakes in this blog have happened.  It is up to you to decide if the commands you are executing are a good idea - I take no responsibility for lost files, forgotten passwords, accidental demon summoning and other problems encountered while reciting incantations described here.  You are now warned.

Optional: install cgroup-tools, but you need it if you want to run 'lscgroup' or network limits (which I haven't tried).

# apt install cgroup-tools

1. sudo nano /etc/default/grub, add the line:

GRUB_CMDLINE_LINUX="cgroup_enable=memory swapaccount=1"

$ sudo update-grub   (note the sysadmin also found update-grub2 on his system)

2. Close all programs and run:  (this will reboot your computer)

$ sudo reboot

3. set DefaultCPUAccounting=yes  and DefaultMemoryAccounting=yes in /etc/systemd/system.conf

4.  Each user is referred to in control groups by a file called 'user-.slice'.  Do you know your uids?  Find them out by running:

$ cat /etc/passwd |grep  

which outputs something like:

andrew:x:1000:1000:andrew,,,:/home/andrew:/bin/bash

My uid is 1000 (actually the first uid defaults to 1000 and counts upwards sequentially in most systems).
Alternatively, run lscgroup and look for the names that start 'user'  and all the other cgroups.

5. To give a user 1000 CPUShares, and limit them to 2G of memory.  Run the commands: 

# systemctl set-property user-1001.slice MemoryLimit=2G 
# sudo systemctl set-property user-1001.slice CPUShares=500

6. and check that they are set correctly with:

# systemctl show -p CPUShares user-1001.slice
# systemctl show -p MemoryLimit user-1001.slice

Repeat step 3 for each user.  That's it!
Now if you want to check what this does,  try installing a program like stress.  If you have two users with different CPUShares, they'll get core access in a ratio proportional to the number of shares.

$stress --cpu 2 --timeout 60

and to check if the memory limit works, this script should get killed if the limit is less than 3Gb:

$dd bs=3G if=/dev/zero of=/dev/null

In the picture below, I am running the above line of code twice as two different users.  One is getting two thirds of the two cpus on this laptop the other is getting 1 third,  the shares are 1000 & 500 respectively. :-)