1

My Solaris 11 cron appears to have stopped working.

Here is the last output of a cron job I run:

-rw-r--r--   1 root     root          60 Jul  2 20:30 locked_passwords.txt

I setup a test, like this:

* * * * * touch /tmp/testing.txt

It never touches the file

I checked if the service is running:

svcs cron
STATE          STIME    FMRI
online         Mar_09   svc:/system/cron:default

I truss the file and see this:

root   532     1   0   Mar 09 ?           3:08 /usr/sbin/cron
pfexec truss -f -p 532
532:    pollsys(0xFC7FC1A8, 1, 0xFC7FC750, 0x00030414) (sleeping...)

Is my cron process sleeping? why?

UPDATE: I have restarted cron 2 times and it continues to stop. The majority of my log reads:

! c queue max run limit reached Mon Jul 24 12:53:00 2017 
! rescheduling a cron job Mon Jul 24 12:53:00 2017 
! c queue max run limit reached Mon Jul 24 12:53:00 2017 
! rescheduling a cron job Mon Jul 24 12:53:00 2017 
! c queue max run limit reached Mon Jul 24 12:53:00 2017 
! rescheduling a cron job Mon Jul 24 12:53:00 2017

How do I diagnose this situation?

Stephen Rauch
  • 4,209
  • 14
  • 22
  • 32
Marinaio
  • 298
  • 5
  • 15
  • Well, the obvious question becomes: Have you tried restarting the cron daemon? – user Jul 19 '17 at 16:46
  • Wise advice. Here is why I did not do that - it doesn't answer, "Why has cron stopped processing commands?" So I did the restart and cron is processing as I would have guessed. But I don't have the root cause of the failure. Ultimately I want to know what made it stop or go to sleep so I can guard against that in the future. – Marinaio Jul 19 '17 at 17:10
  • 1
    Certainly fair enough. Root cause analysis is an underappreciated thing in many situations. If you want answers that focus on how to analyze the present situation in-place, I suggest to edit out the "How do re-enable cron?" at the end, perhaps replacing it by asking something like "how do I diagnose this situation?" (which can apply also to processes other than cron). I'm not familiar with Solaris, but I'd try attaching a debugger and looking at the current process state, including the call stack; and look up what those parameters to `pollsys()` mean. Then [edit] the question to add that data. – user Jul 19 '17 at 18:02
  • Another thing you do want to check is to look over your system logs. Try to pin down the time when cron jobs stopped running (you have a pretty good candidate already), and then go over the logs from around that time with a fine-toothed comb. You can then include also the log data, properly (but carefully!) anonymized or sanitized if required, in your question. – user Jul 19 '17 at 18:04
  • I would expect cron to be mostly sleeping, as it would only do something once a minute anyway. At this point, though, you've restarted cron and it's working, so I'm not sure we can actually answer this question any more. – Jeff Schaller Jul 19 '17 at 18:53

2 Answers2

1

Have you checked /var/cron/log? Perhaps the account is locked or added to cron.deny?
Is there a corresponding .au file in /var/spool/cron/crontabs? Historically, that file was needed for jobs to run. Since it's created by crontab -e, it's usually only missing if a cron file was copied to a new server. Possible issue with the script/binary being called?

The cron daemon is usually sleeping while it waits for when it does work.

If you're using v11.3, you could also look into using a scheduled service in the event that it is more useful for your needs?

sleepyweasel
  • 1,013
  • 5
  • 11
0

SOLVED!!!

/var/cron/log didn't solve my problem, but it did point me in the right direction. In the log I saw lots of

! c queue max run limit reached Thu May 27 15:45:01 2010 [[ Queue "c" is for cron]]

/etc/cron.d/queuedefs was the same as all my other servers so that was not the issue.

Here is what I did. ps -ef | grep cron ptree <PID> pgrep -f '/export/script.sh' # verify the listing before proceeding pkill -9 -f '/export/script.sh' ** Now cron is running again **

I saw hundreds of script.sh piling up.

script.sh connects to a database and I am now troubleshooting a change made to that script. However, cron is now running again.

thanks

Marinaio
  • 298
  • 5
  • 15