299

I have created a simple systemd service file for a custom application. The application works well when I run it manually, but my CPU gets maxed out when I run it with systemd.

I'm trying do track down where my problem is, but I don't know where to find the output (or how to configure systemd to put the output somewhere).

Here is my service file:

[Unit]
Description=Syncs files with a server when they change
Wants=network.target
After=network.target

[Service]
ExecStart=/usr/local/bin/filesync-client --port 2500
WorkingDirectory=/usr/local/lib/node_modules/filesync-client
Restart=always

[Install]
WantedBy=multi-user.target

Throughout the application, I output to stdout and stderr.

How can I read the output of my daemon?

Edit:

I found man systemd.exec, which mentioned the StandardOutput= option, but I'm not sure how to use it. From the man page:

StandardOutput=

Controls where file descriptor 1 (STDOUT) of the executed processes is connected to. Takes one of inherit, null, tty, syslog, kmsg, kmsg+console, syslog+console or socket.

If set to inherit the file descriptor of standard input is duplicated for standard output. If set to null standard output will be connected to /dev/null, i.e. everything written to it will be lost. If set to tty standard output will be connected to a tty (as configured via TTYPath=, see below). If the TTY is used for output only the executed process will not become the controlling process of the terminal, and will not fail or wait for other processes to release the terminal. syslog connects standard output to the syslog(3) system logger. kmsg connects it with the kernel log buffer which is accessible via dmesg(1). syslog+console and kmsg+console work similarly but copy the output to the system console as well. socket connects standard output to a socket from socket activation, semantics are similar to the respective option of StandardInput=. This setting defaults to inherit.

Does this mean that these are my only options? I would like, for example, to put output in /dev/shm or something. I suppose I could use a Unix domain socket and write a simple listener, but this seems a little unnecessary.

I just need this for debugging, and I'll probably end up removing most of the logs and change the output to syslog.

Community
  • 1
beatgammit
  • 7,453
  • 10
  • 31
  • 32
  • Have you tried checking `/var/log/syslog` for output? Most systems will log stuff into `/var/log/` so I'd start by checking there. You can use `grep` to search for text if you know the output: `grep "my output" /var/log` should do the trick. – sbtkd85 Sep 09 '11 at 19:10
  • @sbtkd85 - Well, I don't have `/var/log/syslog`, but `/var/log/messages` does the trick. The problem is, according to the logs, my daemon crashes on start, yet I can tell that it is still running because it has an HTTP server, and I can query it. It seems the rest of the logs are getting lost... – beatgammit Sep 10 '11 at 03:50
  • Why not try setting `StandardOutput=tty` so you can see what is happening when you launch your daemon. It should output the terminal (you may have to use `ttyS0` or similar to get the output on your screen). – sbtkd85 Sep 12 '11 at 13:46
  • 5
    Shouldn't standard IO redirection operators work in this context. Something like `ExecStart=/usr/local/bin/filesync-client --port 2500 2>/tmp/filesync.log` – Deepak Mittal Apr 10 '12 at 18:30
  • What is actually abusing your CPU? Is it systemd, your service or system (e.g. by spawning new copies of the service because systemd went crazy)? – peterph Nov 13 '12 at 16:03
  • @peterph - It was a problem with my service, which is why I wanted to look at the output. The problem is now solved though. – beatgammit Nov 14 '12 at 19:59
  • Appears that it is possible now with newer versions of systemd, see https://stackoverflow.com/a/48052152/32453 – rogerdpack May 18 '19 at 02:12

2 Answers2

299

Update

As mikemaccana notes, the systemd journal is now the standard logging device for most distros. To view the stdout and stderr of a systemd unit use the journalctl command.

sudo journalctl -u [unit]

Original Answer

By default stdout and stderr of a systemd unit are sent to syslog.

If you're using the full systemd, this will be accesible via journalctl. On Fedora, it should be /var/log/messages but syslog will put it where your rules say.

Due to the date of the post, and assuming most people that are exposed to systemd are via fedora, you were probably hit by the bug described here: https://bugzilla.redhat.com/show_bug.cgi?id=754938 It has a good explanation of how it all works too =) (This was a bug in selinux-policy that caused error messages to not be logged, and was fixed in selinux-policy-3.10.0-58.fc16)

Matt
  • 8,841
  • 1
  • 26
  • 32
  • 5
    Note that using the standard logging mechanism like this will *not* create persistent logs by default. To do that, you'll need to create /var/log/journal, and then run `sudo systemctl restart systemd-journald` – mlissner May 08 '15 at 02:21
  • 1
    what syslog facility and priority? – jrwren Jul 07 '15 at 16:31
  • 2
    This worked for me: `StandardOutput=syslog+console` and `StandardError=syslog+console` after that all output from my unit appeared in journalctl. The default setting was wrong apparently. (Such as DefaultStandardOutput in /etc/systemd/system.conf) – gregn3 Jul 22 '15 at 12:23
  • @gregn3 That does not help for me. It just keeps outputting stderr, no matter what I do :( – Joakim Apr 12 '16 at 22:29
  • @Joakim I have no idea right now... (but I would first check that the issue is not with the unit itself, that it works correctly when started from the command line (as a normal script)... then I would check systemd and journalctl settings... good luck :) ) – gregn3 Apr 13 '16 at 13:11
  • 2
    `-f` was helpful for me. Followed the log as changes occur (use case was following minecraft server which was running as a daemon) – blaughw Nov 21 '16 at 04:55
  • 3
    This is driving me crazy... On a standard Debian stretch journalctl won't show me any of the standard output whatsoever. I even use `/usr/bin/stdbuf -oL ` and an explicit `StandardOutput=journal`. Still nothing. – jlh Dec 18 '18 at 09:14
  • 1
    use `sudo journalctl -eu [unit]` to go the end of the logging – ignacio Jun 09 '20 at 08:26
  • I'm very confused, on OpenSUSE Leap 15.2 journalctl -u example.service does not show the stdout at all. It shows the times when the service was started and stopped but no stdout or stderr. – Parker Gibson Feb 24 '21 at 06:20
  • "Note that using the standard logging mechanism like this will not create persistent logs by default" what is does this means? – Viren Nov 04 '22 at 09:14
125

Shorter, simpler, non-legacy answer:

sudo journalctl -u [unitfile]

Where [unitfile] is the systemd .service name. Eg, to see messages from myapp.service, 

sudo journalctl --unit=myapp

To follow logs in real time: 

sudo journalctl -f -u myapp
mikemaccana
  • 1,723
  • 1
  • 12
  • 16
  • 5
    Note that you might have to `sudo` if you get a `No journal files found` error. – bigjosh Nov 17 '15 at 15:51
  • 9
    syslog is not legacy... – Miles Rout Jun 26 '16 at 05:18
  • 4
    It is in current Linux distros. You might really like syslog but that doesn't change what they ship with. – mikemaccana Jun 26 '16 at 11:13
  • So Ubuntu 16.04 LTS isn't a current distro in 2017? Maybe you're conflating the idea of sysvinit with syslog. – labyrinth Mar 09 '17 at 23:08
  • @JECompton Ubuntu 16.04 also uses journald. – mikemaccana Mar 10 '17 at 15:11
  • 3
    Sure. And it also uses syslog. My point was not that systemd is not used, but that syslog is not legacy. – labyrinth Mar 10 '17 at 16:46
  • 12
    @JECompton if all logging on current Linux distros uses journald, and syslog is not necessary and only used for compatibility, then it logically follows that syslog is legacy. – mikemaccana Mar 13 '17 at 11:27
  • 6
    @mikemaccana Nonsene. Debian/Ubuntu/RedHat != "all logging on current Linux". syslogd is very much alive and well and will continue to be relevant for a long long time. Just one example: BusyBox runs on hundreds of millions, if not *billions* of embedded devices the world over - far outstripping any desktop distro in popularity. Its system logging facility? Yeah. There are also many desktop distros which have managed to evade the SystemD malware infection, such as Devuan, Alpine Linux, Artix Linux, Gentoo, Slackware, etc - many of which will be using syslogd. – Ola Tuvesson Sep 13 '19 at 09:11