There seems no officially supported way to do that. (This is incorrect. See the bottom)
An officially discouraged way (because it manipulates the cgroup)
is as follows:
Make the following file as /etc/systemd/system/[email protected]/set-memhigh.conf
[Service]
Type=simple
ExecStartPost=+/root/set-memoryhigh.sh %i
Then make the following file as "/root/set-memoryhigh.sh"
#!/bin/bash
exec >>/var/tmp/log.txt 2>&1 # for logging
set -x # for logging
for d in /sys/fs/cgroup /sys/fs/cgroup/user.slice /sys/fs/cgroup/user.slice/user-$1.slice; do
echo "+memory" >>${d}/cgroup.subtree_control
done
/bin/echo "24G" >> /sys/fs/cgroup/user.slice/user-$1.slice/memory.high
You can see if it works or not by running
cat /sys/fs/cgroup/user.slice/user-${UID}.slice/memory.high
If "/sys/fs/cgroup/user.slice" does not exist, then the unified cgroup hierarchy is not enabled. We have to enable it as https://unix.stackexchange.com/a/452728/297666
Although it works, I am not sure if you like this...
Note added on July 25: Making the following file as /etc/systemd/system/user-1000.slice for each user (replacing 1000 by user's UID) imposes a memory limitation on that user. I verified it on systemd 237 on ubuntu 18.04 and Debian strecth with systemd 237 installed from stretch-backports:
[Slice]
Slice=user.slice
MemoryHigh=24G
The inconvenience is that we have to make the above file for each user. With systemd 239, we can make the above file as /etc/systemd/system/user-.slice.d/memory.conf and the memory limitation is imposed on every user. But there is a bug in systemd 239 (this bug was corrected in 240) and it does not work as intended. To work around the bug, make the following file as user-0.slice and run systemctl enable user-0.slice. We do not have to make the following file for each user.
[Unit]
Before=systemd-logind.service
[Slice]
Slice=user.slice
[Install]
WantedBy=multi-user.target