2

Where I work, we have a site-to-site VPN tunnel set up between our main data center and a third party data center that does some Oracle PaaS for us. Since they charge per-VM, we're monitoring the Oracle audit logs on a local VM that is running the analytics for the Security and Audits department. The third party (or rather a department within it) are complaining about the amount of network traffic going over the tunnel due to syslog.

It's currently running unencrypted over udp/514, so I think I need to do some compression. After some research it looks like both rsyslog (what the third party is using) and syslog-ng (what we're using) support TLS compression. We have the CPU cycle's to spare, the third party's networking team is just complaining because apparently it impacts their SLA's.

My question is this:

Is there any way for rsyslog to essentially batch together several messages at once (say over a five minute window) and have them sent to our syslog-ng server all at once?

The reason I'm interested in doing this is because we would be switching over to TCP there's going to be more overhead. It's my understanding that most compression methods will do substitution on redundant data, which I'd imagine would be the case with a stream of syslog messages. The goal is to batch together then remove the redundant parts prior to transmission.

Bratchley
  • 16,684
  • 13
  • 64
  • 103
  • TLS isn't compression, it's encryption. – kurtm Oct 10 '13 at 14:15
  • @kurtm It has compression options – Bratchley Oct 10 '13 at 14:15
  • [see also](http://www.ietf.org/rfc/rfc3749.txt) – Bratchley Oct 10 '13 at 14:16
  • I'd avoid using TCP for syslog. It's not only more overhead, it requires acknowledgement. So the clients sending to the syslog server will wait for acknowledgement, and start freezing until they get acknowledgements. I saw one department have their whole infrastructure freeze because their syslog server got bogged down. – kurtm Oct 10 '13 at 14:17
  • I stand corrected on the compression. – kurtm Oct 10 '13 at 14:17
  • Yeah the TCP switch is just so we can take advantage of the TLS compression to see if we can lower the bandwidth usage that way. I'd prefer to go ACK-less but if this is the only way to get the bandwidth down to an acceptable point, then it may be acceptable overhead (if we gain more through compression than we lose through the ACK's, I mean). Plus, since these are audit logs, we might benefit from encrypted, transmission even if it's purely backend stuff. – Bratchley Oct 10 '13 at 14:21
  • You say you are using VPN. Does the VPN have an option for compression? – kurtm Oct 10 '13 at 14:24
  • They're exploring that option but this handles ERP data and they'd have to downgrade the encryption to free the cycles to do compression. Apparently the encryption is happening on the Cisco devices, which apparently don't have the CPU cycles to spare. So they're interested in a host-based solution. – Bratchley Oct 10 '13 at 14:28
  • Ah, yes. Cisco devices tend to be very underpowered. – kurtm Oct 10 '13 at 14:36

1 Answers1

1

Sounds like rsyslog queueing might do what you want. Messages can also be stored for transmission during off-peak hours.

Specifically, the following:

The "$QueueDequeueSlowdown" directive allows to specify how long (in microseconds) dequeueing should be delayed.

Tom Damon
  • 151
  • 4
  • Yep this is the answer. It looks like `DequeueSlowdown` does the exact thing I was asking for. I want to accept this as the answer but right now it's just a link to another website (technically answers are supposed to be self-contained). Can you edit your post to illustrate `DequeueSlowdown` ? – Bratchley Oct 11 '13 at 21:12