1

Yesterday /var/log/messages on one server was found empty. Looking into /var/ shows:

messages
messages-20161125
messages-20161127

But trying to access messages-20161127 hangs the process. tail, cat, etc, all never return, cannot be interrupted and even kill -9 does not help. (ps shows the process in IO wait)

lsof | grep messages shows: rsyslogd /var/messages-20161127

So rsyslog is still using the old version of the file and did not close it.

Is this a problem with the configuration of rsyslog? logrotate? Any tips on how to avoid this?

Thanks!

EDIT

After rebooting we could regain access to messages-20161127. There were messages about "task sync:30312 blocked for more than 120 seconds". The time of the the logrotate didn't match the sync error message but perhaps they are related.

So perhaps root cause is the filesystem got hung? /var/log/messages is on the root filesystem. Everything else seemed to be running normally...

See my other question slow/frozen ext4 // task sync blocked on big mostly write only server

Community
  • 1
Danny
  • 521
  • 8
  • 16
  • *"But trying to access messages-20161127 hangs the process. tail, cat, etc, all never return, cannot be interrupted and even kill -9 does not help. "* That is bad news. This is an OS problem. It could be a temporary condition in the kernel that goes away due to a reboot or worse: filesystem corruption and/or a hardware problem. There isn't anything that logrotate or rsyslog could do to make accesses on a file stuck in an uninterruptible sleep. – Kaz Nov 29 '16 at 02:15
  • Thanks. We'll fsck the filesystem today. See my edit about the sync problem. The times don't exactly match but could be related (??) – Danny Nov 29 '16 at 02:40

0 Answers0