10

I've faced a really strange issue today, and am totally helpless about it.

Some of the servers I manage are monitored with Nagios. Recently I saw a disk usage probe failing with this error:

DISK CRITICAL - /sys/kernel/debug/tracing is not accessible: Permission denied

I wanted to investigate and my first try was to check this directory permissions, and compare these with another server (who's working well). Here are the commands I ran on the working server and you'll see that as soon as I cd into the directory, its permissions are changed:

# Here we've got 555 for /sys/kernel/debug/tracing
root@vps690079:/home/admin# cd /sys/kernel/debug
root@vps690079:/sys/kernel/debug# ll
total 0
drwx------ 30 root root 0 Jul 19 13:13 ./
drwxr-xr-x 13 root root 0 Jul 19 13:13 ../
…
dr-xr-xr-x  3 root root 0 Jul 19 13:13 tracing/
drwxr-xr-x  6 root root 0 Jul 19 13:13 usb/
drwxr-xr-x  2 root root 0 Jul 19 13:13 virtio-ports/
-r--r--r--  1 root root 0 Jul 19 13:13 wakeup_sources
drwxr-xr-x  2 root root 0 Jul 19 13:13 x86/
drwxr-xr-x  2 root root 0 Jul 19 13:13 zswap/

# I cd into the folder, and it (./) becomes 700!!
root@vps690079:/sys/kernel/debug# cd tracing/
root@vps690079:/sys/kernel/debug/tracing# ll
total 0
drwx------  8 root root 0 Jul 19 13:13 ./
drwx------ 30 root root 0 Jul 19 13:13 ../
-r--r--r--  1 root root 0 Jul 19 13:13 available_events
-r--r--r--  1 root root 0 Jul 19 13:13 available_filter_functions
-r--r--r--  1 root root 0 Jul 19 13:13 available_tracers
…

# Next commands are just a dumb test to double-check what I'm seeing
root@vps690079:/sys/kernel/debug/tracing# cd ..
root@vps690079:/sys/kernel/debug# ll
total 0
drwx------ 30 root root 0 Jul 19 13:13 ./
drwxr-xr-x 13 root root 0 Sep 27 10:57 ../
…
drwx------  8 root root 0 Jul 19 13:13 tracing/
drwxr-xr-x  6 root root 0 Jul 19 13:13 usb/
drwxr-xr-x  2 root root 0 Jul 19 13:13 virtio-ports/
-r--r--r--  1 root root 0 Jul 19 13:13 wakeup_sources
drwxr-xr-x  2 root root 0 Jul 19 13:13 x86/
drwxr-xr-x  2 root root 0 Jul 19 13:13 zswap/

Have you got any idea what could causes this behavior?
Side note, using chmod to re-etablish permissions does not seems to fix the probe.

Gilles 'SO- stop being evil'
  • 807,993
  • 194
  • 1,674
  • 2,175
zessx
  • 203
  • 2
  • 7
  • 2
    When providing samples of terminal input, consider replacing nonstandard aliases such as `ll` with the commands they stand for. – Roman Odaisky Sep 27 '19 at 18:26
  • 3
    @RomanOdaisky It's a default alias in Ubuntu, might not have known it wasn't default – GammaGames Sep 27 '19 at 22:20
  • On top of what @tecloM said, this looks like a kernel bug for your kernel version. On 4.19.0-4 the behavior is normal. – V13 Sep 28 '19 at 22:52

1 Answers1

20

/sys

/sys is sysfs, an entirely virtual view into kernel structures in memory that reflects the current system kernel and hardware configuration, and does not consume any real disk space. New files and directories cannot be written to it in the normal fashion.

Applying disk space monitoring to it does not produce useful information and is a waste of effort. It may have mount points for other RAM-based virtual filesystems inside, including...

/sys/kernel/debug

/sys/kernel/debug is the standard mount point for debugfs, which is an optional virtual filesystem for various kernel debugging and tracing features.

Because it's for debugging features, it is supposed to be unnecessary for production use (although you might choose to use some of the features for enhanced system statistics or similar).

Since using the features offered by debugfs will in most cases require being root anyway, and its primary purpose is to be an easy way for kernel developers to provide debug information, it may be a bit "rough around the edges".

When the kernel was loaded, the initialization routine for the kernel tracing subsystem registered /sys/kernel/debug/tracing as a debugfs access point for itself, deferring any further initialization until it's actually accessed for the first time (minimizing the resource usage of the tracing subsystem in case it turns out it's not needed). When you cd'd into the directory, this deferred initialization was triggered and the tracing subsystem readied itself for use. In effect, the original /sys/kernel/debug/tracing was initially a mirage with no substance, and it only became "real" when (and because) you accessed it with your cd command.

debugfs does not use any real disk space at all: all the information contained within it will vanish when the kernel is shut down.

/sys/fs/cgroup

/sys/fs/cgroup is a tmpfs-type RAM-based filesystem, used to group various running processes into control groups. It does not use real disk space at all. But if this filesystem is getting nearly full for some reason, it might be more serious than just running out of disk space: it might mean that

a) you're running out of free RAM,

b) some root-owned process is writing garbage to /sys/fs/cgroup, or

c) something is causing a truly absurd number of control groups to be created, possibly in the style of a classic "fork bomb" but with systemd-based services or similar.

Bottom line

A disk usage probe should have /sys excluded because nothing under /sys is stored on any disk whatsoever.

If you need to monitor /sys/fs/cgroup, you should provide a dedicated probe for it that will provide more meaningful alerts than a generic disk space probe.

telcoM
  • 87,318
  • 3
  • 112
  • 232
  • 1
    Thanks for this answer with as much details as I needed! I'll exclude `/sys` from my monitoring range. – zessx Sep 27 '19 at 12:21
  • 1
    @zessx: Also exclude `/proc` and probably `/dev` (because even if it isn't 100% RAM-backed, on the one hand it contains a number of files and directories that are "weird" in various ways, and on the other, if you actually consume a ton of disk space in `/dev`, your setup is horrifically broken and you should light the whole mess on fire). – Kevin Sep 28 '19 at 02:23
  • "`/sys` is `sysfs`, an entirely RAM-based virtual filesystem" – I'm pretty sure the contents of `sysfs` are 100% synthesized from in-kernel data structures and do *not* live in RAM somewhere. In fact, I would argue that "RAM-based virtual filesystem" is an oxymoron: either it is RAM-based, i.e. has a backing store (even if it is a very non-traditional backing store for a filesystem), then it is not virtual, or it is virtual, then it has no backing store. – Jörg W Mittag Sep 28 '19 at 11:27
  • 1
    @JörgWMittag Note that I very carefully avoided saying `sysfs` is a RAMdisk. Where would the in-kernel data structures live, if not in RAM, anyway? I agree that the word "virtual" is problematic here, as you may be aware that on top of *all* the filesystem drivers in the Linux kernel is the VFS (Virtual File System) layer, which uses "virtual" in yet another sense, as an uniform abstraction for all possible filesystems. But it is hard to *concisely* describe just how `proc` and `sysfs` are different from real filesystems, as this was just background information to make the main point stick. – telcoM Sep 28 '19 at 12:13
  • If you want to use these specific terms, then /sys and /proc are virtual as their contents are generated fully on demand and each object has custom handlers, while /dev and /run (tmpfs) are RAM-based as they have a semi-traditional backing store and standard file handlers. (Except /dev is *slightly* virtual in that the kernel automatically populates the backing store.) – u1686_grawity Sep 28 '19 at 16:56
  • 1
    @grawity I tweaked the wording a little, would it be better now? – telcoM Sep 28 '19 at 18:33