3

I have a binary file on a Linux server that is being actively appended by a process (written in C with a constantly open file handler and flushing non-ASCII buffer to this file). I would like to replicate this file to another server without locking the write (C process), preferably, and not copying the entire file every time (file size ~1+GB and replication frequency < 1 sec).

I've explored the following: rsync: I believe rsync does a full replication, but not incremental.

filebeat by elasticsearch: it requires ASCII text and newlines (I have neither).

I would preferably like to leverage standard Linux tools, but I am open to any other 3rd party solution or creating a C program myself :).

rocketman
  • 33
  • 5

2 Answers2

4

If it's only being appended to (and not modified in the middle), you could just run tail -f on it. It should wait for any newly appended data and print it, and you can tell it what position to start at:

tail -c 0 -f datafile        # start at the current file end
tail -c +123 -f datafile     # start at byte 123

To actually move the data somewhere, piping through to ssh should work:

So if the remote end has the first 123456 bytes already:

tail -c +123456 -f datafile | ssh user@somehost 'cat >> datafile.copy'

(Though of course you need to go check the file size on the remote before starting the pipeline.)

If, instead, you have modifications to the middle of the file, you're going to need some sort of a logging layer in the program itself. Filesystem snapshots might do, but the one second interval may be too hard, especially since you'd need to scan the file for the changes anyway.

ilkkachu
  • 133,243
  • 15
  • 236
  • 397
  • thanks for the help. I'll take this for a test drive. And to confirm, the _local_ process will **only** append the file. – rocketman Mar 10 '17 at 16:26
0

Rsync should push deltas after the initial sync. Another option might be to snapshot the location of the file. This will depend on whether you are using LVM of have a file system that supports snapshots. You could take a snapshot, then sync the file to the remote location. Then remove the snapshot. I also like this idea because you are not touching the live file.

feeble
  • 27
  • 1
  • thanks for the quick response. A few questions on your solution. 1.) This would require me to run rsync as a daemon? I've checked the man page for rsync and the daemon and there was no polling interval described. Any guess on the frequency? 2.) How fast is the LVM snapshot (~milliseconds)? – rocketman Mar 09 '17 at 22:26
  • 1
    Just saw your replication frequency. In that case I would look at drbd. It works like realtime block level replication. – feeble Mar 10 '17 at 00:34