2

I am bringing in log files via rsyslog and my config looks like the following:

root@rhel:/etc/rsyslog.d# head mail_prod_logs.conf
if $fromhost-ip=="10.10.10.10" and $programname=="AMP_Logs" then -/var/log/mail_logs/amp.log

My logs are all stored in the /var/log/mail_logs/amp.log folder:

Oct 18 13:29:28 server.com AMP_Logs: Info: Begin Logfile
Oct 18 14:29:28 server.com AMP_Logs: Info: Version: 12.1.0-000 SN: .....
Oct 18 14:29:28 server.com AMP_Logs: Info: Time offset from UTC: -14400 seconds
Oct 18 15:29:23 server.com AMP_Logs: Info: Response received for.....
Oct 18 15:29:23 server.com AMP_Logs: Info: File reputation query.....
Oct 19 13:29:23 server.com AMP_Logs: Info: Response received for fil....
Oct 19 13:29:58 server.com AMP_Logs: Info: File reputation query ....
Oct 19 13:29:58 server.com AMP_Logs: Info: File reputation query ....

I would like to use the datetime portion of the log to put these in hourly folders inside of daily folders inside of the month while the data is coming in by editing the mail_prod_logs.conf.

So it would look like:

/var/log/mail_logs/Sep/30/23.log
/var/log/mail_logs/Oct/01/00.log
/var/log/mail_logs/Oct/01/01.log
/var/log/mail_logs/Oct/01/02.log
...

How can I do this?

sectechguy
  • 127
  • 1
  • 8

2 Answers2

1

Assuming your log lines are as well-formed as the snippet suggests, this would be a start:

awk '{
      dir=$1 "/" $2; log_file=dir "/" substr($3, 1, 2) ".log"
      if (!exists[dir]++) {system("mkdir -p " dir)}; 
      print > log_file
}' amp.log

You would execute this in /var/log/mail_logs/.

The system call to the external mkdir command creates the date-specific directory if one doesn't exist. Passing -p causes parents to be created as needed (so the directory Oct is created when mkdir -p /Oct/01 brings October into play for the first time). Also, -p ensures that mkdir does not report an error if the directory already exists.

The call to print writes log lines to a file whose name is crafted from log time components. Each new log file is created when first written to and appended to during the lifetime of the command.

You could change print > to print >> if you'd rather a datetime-based log file from a previous awk run is appended to than overwritten.

iruvar
  • 16,515
  • 8
  • 49
  • 81
1

You can do this with a dynamic file template. Use a property replacor to select parts of the %timestamp% property, in particular the option date-day and date-hour and characters 1 to 3 of date-rfc3164 (which is a string like "Oct 9 09:47:08"). Typically, in examples, the template is called DynFile:

$template DynFile,"/var/log/mail_logs/%timestamp:1:3:date-rfc3164%/%timestamp:::date-day%/%timestamp:::date-hour%.log"

To use the template, replace the ...then -/var/log/mail_logs/amp.log by ...then -?DynFile

Should you consider replacing the 3-letter month (Jan, Feb, ...) by the 2-digit month for easier handling, use instead

$template DynFile,"/var/log/mail_logs/%timestamp:::date-month%/%timestamp:::date-day%/%timestamp:::date-hour%.log
meuh
  • 49,672
  • 2
  • 52
  • 114