How should I set up a systemd service to auto-start a server and let me pass commands to it?

Question

Goal:

I'm trying to get a Minecraft server to run on computer boot with systemd on Fedora. I have a few self-imposed criteria that I need to meet to be able to properly manage my server(s):

1. It has to run as the minecraft system user I made with the home dir /opt/minecraft. I attempted this one by addusering and then adding the line User=minecraft and WorkingDirectory=/opt/minecraft/
2. It has to be scalable and work with an arbitrary number of servers. I attempted this by using a template service and then changing the WorkingDirectory line to WorkingDirectory=/opt/minecraft/%i to let me pass in a directory.
3. I have to be able to pass commands into it somehow. This is the one I'm stuck on. I have tried using a socket unit and then hooking that up to /run/minecraft%I, but I haven't been able to get that to work. If you aren't familiar with Minecraft servers, they have this interactive console thingy that you can pass commands into. In the past, I have used tmux send with the server running in a tmux session, but the issue with that is that it doesn't start automatically and feels inelegant.

Attempted solution:

/usr/local/lib/systemd/system/[email protected]:

[Unit]
Description=Minecraft server: %i

# only run after networking is ready
After=network-online.target
Wants=network-online.target

[Service]
Type=simple

# restart if the server crashes
Restart=on-failure
RestartSec=5s

# set the input and outputs to a socket unit and the journal resp.
Sockets=minecraft@%i.socket
StandardInput=socket                     
StandardOutput=journal
StandardError=journal

# set the user and directory to the correct values
User=minecraft
WorkingDirectory=/opt/minecraft/%i/

# run the start script for the specified server
ExecStart=/bin/bash /opt/minecraft/%i/start.sh

[Install]
WantedBy=default.target

/usr/local/lib/systemd/system/[email protected]:

[Unit]
Description=Socket for Minecraft server: %i

[Socket]
# listen to a pipe for input
ListenFIFO=%t/minecraft%I.stdin

Service=minecraft@%i.service

Problem:

When I try to start the server with sudo systemctl start minecraft@1_17_1.service (I have the server installed in /opt/minecraft/1_17_1/), it fails:

Job for minecraft@1_17_1.service failed because of unavailable resources or another system error.
See "systemctl status minecraft@1_17_1.service" and "journalctl -xeu minecraft@1_17_1.service" for details.

This prompted me to run systemctl status minecraft@1_17_1.service:

● minecraft@1_17_1.service - Minecraft server: 1_17_1
     Loaded: loaded (/usr/local/lib/systemd/system/[email protected]; enabled; vendor preset: disabled)
     Active: activating (auto-restart) (Result: resources) since Thu 2021-11-04 14:37:27 EDT; 163ms ago
TriggeredBy: × minecraft@1_17_1.socket
        CPU: 0

And also journalctl -xeu minecraft@1_17_1.service

Nov 04 14:51:01 riley-fedora systemd[1]: minecraft@1_17_1.service: Got no socket.
Nov 04 14:51:01 riley-fedora systemd[1]: minecraft@1_17_1.service: Failed to run 'start' task: Invalid argument
Nov 04 14:51:01 riley-fedora systemd[1]: minecraft@1_17_1.service: Failed with result 'resources'.
░░ Subject: Unit failed
░░ Defined-By: systemd
░░ Support: https://lists.freedesktop.org/mailman/listinfo/systemd-devel
░░ 
░░ The unit minecraft@1_17_1.service has entered the 'failed' state with result 'resources'.
Nov 04 14:51:01 riley-fedora systemd[1]: Failed to start Minecraft server: 1_17_1.
░░ Subject: A start job for unit minecraft@1_17_1.service has failed
░░ Defined-By: systemd
░░ Support: https://lists.freedesktop.org/mailman/listinfo/systemd-devel
░░ 
░░ A start job for unit minecraft@1_17_1.service has finished with a failure.
░░ 
░░ The job identifier is 55890 and the job result is failed.

I saw that it was seemingly angry with my [email protected] file, so I ran systemctl status minecraft@1_17_1.socket:

× minecraft@1_17_1.socket - Socket for Minecraft server: 1_17_1
     Loaded: loaded (/usr/local/lib/systemd/system/[email protected]; static)
     Active: failed (Result: resources)
   Triggers: ● minecraft@1_17_1.service
     Listen: /run/minecraft1_17_1.stdin (FIFO)

Nov 04 14:52:35 riley-fedora systemd[1]: minecraft@1_17_1.socket: Failed with result 'resources'.
Nov 04 14:52:35 riley-fedora systemd[1]: Failed to listen on Socket for Minecraft server: 1_17_1.
Nov 04 14:52:41 riley-fedora systemd[1]: minecraft@1_17_1.socket: Failed to open FIFO /run/minecraft1_17_1.stdin: Permission denied
Nov 04 14:52:41 riley-fedora systemd[1]: minecraft@1_17_1.socket: Failed to listen on sockets: Permission denied
Nov 04 14:52:41 riley-fedora systemd[1]: minecraft@1_17_1.socket: Failed with result 'resources'.
Nov 04 14:52:41 riley-fedora systemd[1]: Failed to listen on Socket for Minecraft server: 1_17_1.
Nov 04 14:52:46 riley-fedora systemd[1]: minecraft@1_17_1.socket: Failed to open FIFO /run/minecraft1_17_1.stdin: Permission denied
Nov 04 14:52:46 riley-fedora systemd[1]: minecraft@1_17_1.socket: Failed to listen on sockets: Permission denied
Nov 04 14:52:46 riley-fedora systemd[1]: minecraft@1_17_1.socket: Failed with result 'resources'.
Nov 04 14:52:46 riley-fedora systemd[1]: Failed to listen on Socket for Minecraft server: 1_17_1.

So it seems like the issue has to do with permissions for the pipe I had it use.

For good measure, I ran journalctl -xeu minecraft@1_17_1.socket

Nov 04 14:52:46 riley-fedora systemd[1]: minecraft@1_17_1.socket: Failed to open FIFO /run/minecraft1_17_1.stdin: Permission denied
Nov 04 14:52:46 riley-fedora systemd[1]: minecraft@1_17_1.socket: Failed to listen on sockets: Permission denied
Nov 04 14:52:46 riley-fedora systemd[1]: minecraft@1_17_1.socket: Failed with result 'resources'.
░░ Subject: Unit failed
░░ Defined-By: systemd
░░ Support: https://lists.freedesktop.org/mailman/listinfo/systemd-devel
░░ 
░░ The unit minecraft@1_17_1.socket has entered the 'failed' state with result 'resources'.
Nov 04 14:52:46 riley-fedora systemd[1]: Failed to listen on Socket for Minecraft server: 1_17_1.
░░ Subject: A start job for unit minecraft@1_17_1.socket has failed
░░ Defined-By: systemd
░░ Support: https://lists.freedesktop.org/mailman/listinfo/systemd-devel
░░ 
░░ A start job for unit minecraft@1_17_1.socket has finished with a failure.
░░ 
░░ The job identifier is 58598 and the job result is failed.

Question:

What am I doing wrong? I have spent 4-ish hours on the gasp second, third, and even fourth pages of Google with no solution. I'm at a loss here, so any and all help is greatly appreciated.

Answer 1

I don't have a full answer for you because I can't reproduce the problem. But I've tried to put togeather a minimum verifiable example and had some ideas for troubleshooting:

# /etc/systemd/system/[email protected]
[Service]
ExecStart=cat -
User=stew
Sockets=sockinst@%i.socket
StandardInput=socket
StandardOutput=journal
StandardError=journal

# /etc/systemd/system/[email protected]
[Socket]
ListenFIFO=%t/sockinst%I.stdin
Service=sockinst@%i.service

Then running this works:

stew /etc/systemd/system $ systemctl start sockinst@1_1.service
stew /etc/systemd/system $ echo "Hello" > /run/sockinst1_1.stdin 
stew /etc/systemd/system $ systemctl status sockinst@1_1.{service,socket}
● sockinst@1_1.service - Testing instanced sockets 1_1
     Loaded: loaded (/etc/systemd/system/[email protected]; static)
     Active: active (running) since Fri 2021-11-05 10:35:41 CET; 37s ago
TriggeredBy: ● sockinst@1_1.socket

systemd[1]: Started Testing instanced sockets 1_1.
cat[11623]: Hello

● sockinst@1_1.socket - Socket for instance 1_1
     Loaded: loaded (/etc/systemd/system/[email protected]; static)
     Active: active (running) since Fri 2021-11-05 10:35:41 CET; 37s ago
   Triggers: ● sockinst@1_1.service
     Listen: /run/sockinst1_1.stdin (FIFO)

systemd[1]: Listening on Socket for instance 1_1.

There are some tricks that you might be running into here:

1. If you systemctl stop minecraft@1_17_1.service, then this will not stop the parallel socket. That means /run/minecraft1_17_1.stdin will continue to run. In fact, if you echo "hello" > /run/minecraft1_17_1.stdin, it will launch your service. I can imagine if you're troubleshooting your service, an old socket could be affecting your tests. I suggest one of two options:

   1. Use systemctl stop minecrat@1_17_1.{service,socket} to stop it,
   2. Add PartOf=minecraft@%i.service to the [Unit] section of the socket. This will cause it to automatically stop when the service is stopped.

2. You are using Restart=on-failure. This is great for your end result, but it could be hiding your initial failure. While you are troubleshooting, consider removing this temporarily. Then when it fails, you'll see the initial reason it failed. My guess is that you are having a legitimate problem with the server startup, but then your socket is bound to that first instance. Then each restart fails to bind to the old socket before even trying the ExecStart= and so you never discover why it originally failed in the first place. Be sure to stop the service and socket, then start it and check the output immediately. I wouldn't be surprised if your error message was different.

3. You described the minecraft server as having a gui. This is optional and suddenly depends on things like your DISPLAY and XAUTHORITY being set. That introduces a lot of stuff that can break. Consider calling your service with the nogui option in start.sh to eliminate this possible cause of the initial failure. If you REALLY REALLY want that gui, then you'll need to set the DISPLAY and XAUTHORITY environment variables in your script (but that's not necessarily the easiest thing to do). I recommend at least trying it without the GUI to see if that gets your service running.

4. It's possible that you've forgotten to run sudo systemctl daemon-reload while editing your files. As a reminder, be sure to run this so that systemd reads the latest version of each unit.