4

We have a java based web-app running on CentOS 6.0 64bit and it writes out many log files. Some of the files are rotated on daily basis using logrotate using the daily, copytruncate and other few options.

One such particular log file stops growing at exactly "102400000" bytes. The process continues working fine. But nothing gets written in the log then.

I did a stat on the file using watch to see if after reaching the "102400000" byte size the modified time changes, but it doesn't. Nothing happens to the file. It just stays there as it is. Other log files in the same directory continue to be written into.

The things mentioned in another StackExchange question here do not apply. I checked and rechecked if any where the mentioned ulimit option was set, but there is none any.

Doing strace -f -p PID showed the process to be trying to write but encountering the "File too large" and "File size limit exceeded" messages:

[pid  5679] write(1, "\n\n\n\n\n23-01-2013 12:00:46:921  Av"..., 128) = -1 EFBIG (File too large)
[pid  5679] --- SIGXFSZ (File size limit exceeded) @ 0 (0) ---

How do I find as to what is imposing this limit on the file size. Other log files in the same directory are quite large (greater than 1Gb in size) and have no issues as such.

Community
  • 1
Gautam Somani
  • 477
  • 1
  • 3
  • 12
  • Is this web-app home-brewn or something publicly available? (Maybe with a mailing list, documentation) – sr_ Jan 22 '13 at 09:28
  • @sr_ Its home brewn, but this problem is occurring with only two particular instances of deployment. And with same file. At other installations there is no problem. – Gautam Somani Jan 22 '13 at 09:34
  • 2
    If it was a ulimit thing, the application would get killed (unless it ignores the SIGXFSZ signal). More probably it's the application itself or java that sets the limit. You could `strace -f` it to see if it is attempting to write the log file. – Stéphane Chazelas Jan 22 '13 at 10:11
  • Presumably the application log doesn't show anything of interest? – Reed Kraft-Murphy Jan 22 '13 at 10:24
  • @ReedKraft-Murphy The log is important. It is monitored from time to time. Once it reaches the 102400000 size, I'm moving it myself, though I can use logrotate also, but I want to know the reason first. I believe rarely does one faces such issues, and I can learn a lot from it. – Gautam Somani Jan 22 '13 at 11:11
  • @StephaneChazelas Waiting for it to get to 102400000 size, will update soon then. – Gautam Somani Jan 22 '13 at 11:12
  • @GautamSomani sorry, I wasn't clear. I meant to say that I'm guessing that the application doesn't log anything relevant to this problem in the last few lines of the log before it hits 102400000 bytes? – Reed Kraft-Murphy Jan 22 '13 at 11:42
  • @ReedKraft-Murphy No offence taken :). No it doesn't, it keeps printing the app logs. No error at all. And no error in linux logs either. – Gautam Somani Jan 22 '13 at 11:44
  • It's not something silly like log4j being used, and the config is different for different deployments? In other words, log4j starts logging to a new file at the given size? – tink Jan 22 '13 at 19:08
  • @StephaneChazelas `strace -f` shows the process to be writing to the file BUT at the same time getting a response of "File too large" `[pid 5679] write(1, "\n\n\n\n\n23-01-2013 12:00:46:921 Av"..., 128) = -1 EFBIG (File too large)` `[pid 5679] --- SIGXFSZ (File size limit exceeded) @ 0 (0) ---` – Gautam Somani Jan 23 '13 at 07:28
  • @tink No. The line for that is commented out. `#log4j.appender.H.MaxFileSize = 100KB`. And the config is almost same, the only things that we change is logging level (INFO, DEBUG, etc.) – Gautam Somani Jan 23 '13 at 07:38

1 Answers1

3

So (from your comment), there is a ulimit after all (causing the SIGXFSZ and EFBIG) which explains why the size of the log can't get any larger.

You can confirm with:

grep 'Max file size' "/proc/$pid/limits"

(where $pid is the id of the java process meant to write to that file).

Possibly it is set by the process itself (check for setrlimit or ulimit in the strace output), It could also be set by a wrapper script that starts java or the java application (look for ulimit -f 100000).

Stéphane Chazelas
  • 522,931
  • 91
  • 1,010
  • 1,501
  • There is a `ulimit 100000` option set in the "limits" file under "/proc". But the script that is starting is using the command `ulimt 100000` only. There is no `-f` option being used. Also in the strace output, taken for 2 minutes, there is no mention of `ulimit` or `setrlimit`. Do I take more strace output? Would it come eventually? – Gautam Somani Jan 23 '13 at 07:53
  • `-f` is the default option, so `ulimit 100000` is the same as `ulimit -f 100000` – Stéphane Chazelas Jan 23 '13 at 07:56
  • Oh yeah u r right! They wanted to increase the number of open files, and they missed the `-n` option. Thanks a lot for pointing this out. :) – Gautam Somani Jan 23 '13 at 08:02