43

Is there any tool available to sync files between two or more linux servers immediately after writing the file in the disk? The rsync command does not suit me on this, because if I set rsync in cron, the minimum time I can set is 1 minute, but I need it on real-time basis.

Rui F Ribeiro
  • 55,929
  • 26
  • 146
  • 227
Sourav
  • 1,253
  • 2
  • 11
  • 16
  • 3
    Do you need synchronization from one read-write server to one or more read-only server(s), or do you need two-way synchronization (with propagation in both direction), or n-way? If you want two/n-way, how do you resolve conflicts? – Gilles 'SO- stop being evil' Sep 01 '16 at 21:43
  • 1
    *"The rsync command does not suit me on this, because if I set rsync in cron, the minimum time I can set is 1 minute"* - **so `cron` does not suit you, not `rsync`**. Don't blame `rsync` command with limitations of `cron`. Keep sentences logical. – bloody Dec 09 '22 at 12:05

8 Answers8

44

Haven't used it myself but read about it recently. There is a daemon called lsyncd, which I presume does exactly what you need.

Read more about it HERE

MelBurslan
  • 6,836
  • 2
  • 24
  • 35
  • 5
    IMHO, this should be the accepted answer. `lsyncd` uses `inotify` & should be the fastest at the fs level. More at https://github.com/axkibe/lsyncd. From that page: Lsyncd watches a local directory trees event monitor interface (inotify or fsevents). It aggregates and combines events for a few seconds and then spawns one (or more) process(es) to synchronize the changes. By default this is rsync. Lsyncd is thus a light-weight live mirror solution that is comparatively easy to install not requiring new filesystems or block devices and does not hamper local filesystem performance. – SACHIN GARG Sep 01 '16 at 00:59
  • I logged in, just to +1. Thanks for clarifying. This helped me make my decision for sync on two singleboards using SSDs. – GTodorov Oct 16 '20 at 04:29
21

Realtime File synchronisation between multiple servers in multi master mode

There is a good tool called lsyncd to sync files between multiple servers on real time basis. Here I have tried with two servers.

Hosts: Server1 and Server2

OS Used: CentOS 7

Install below packages on both the servers. 

# yum install -y epel-release
# yum -y install lua lua-devel pkgconfig gcc asciidoc lsyncd

generate ssh-key in both the servers and add to authorized_keys file. [add public key of server1 to server2's authorized_keys and public key of server2 to server1's authorized_keys file]

Server1 configuration

Open /etc/lsyncd.conf and comment out the default configuration using -- at the beginning of the line and add below configuration to the file. 

settings {
  logfile = "/var/log/lsyncd/lsyncd.log",
  delay = 1
}
sync {
  default.rsync,
  source="/home/test/public_html/",
  target="server2:/home/test/public_html/",
  rsync = {
    compress = true,
    acls = true,
    verbose = true,
    owner = true,
    group = true,
    perms = true,
    rsh = "/usr/bin/ssh -p 22 -o StrictHostKeyChecking=no"
  }
}

Change the target IP in target parameter.

You can change the delay parameter as per your requirement. Here it is set 1 second.

Now create the log directory.

# mkdir -p /var/log/lsyncd

Enable the lsyncd service to start automatically. 

# systemctl enable lsyncd.service

Start the service.

# systemctl start lsyncd.service

Server2 Configuration

Follow the same configuration as Server1 and change the target IP.

Now the synchronisation is set.

You can check the activity from tailf /var/log/lsyncd/lsyncd.log

Thanks to MelBurslan for his suggestion.

charmoniumQ
  • 143
  • 1
  • 6
Sourav
  • 1,253
  • 2
  • 11
  • 16
  • 2
    The delay parameter under settings function is not valid. Use **maxDelays** variable instead. I have implemented this tool. Its really powerful. – Hasanuzzaman Sattar Mar 31 '18 at 09:09
  • 1
    On Ubuntu (and probably on recents Debian releases), the config file must be placed in `/etc/lsyncd/lsyncd.conf.lua`. See [here](https://www.howtoforge.com/how-to-synchronize-directories-using-lsyncd-on-ubuntu/). – Cédric Oct 24 '20 at 21:53
  • 2
    As @HasanuzzamanSattar mentioned, the *delay* parameter should not be in the **settings** section of the configuration file. It should be in the **sync** section instead. – zavg Oct 29 '20 at 22:28
8

Syncthing might be an option. It's extremely fast, transfer is encrypted and there are clients for multiple platforms. It uses "inotify" to sync changed files instantly.

njam
  • 181
  • 1
  • 3
6

Inotify-tools

Provide an interface to inotify, consisting of:

inotifywait

This command simply blocks for inotify events, making it appropriate for use in shell scripts. It can watch any set of files and directories, and can recursively watch entire directory trees.

inotifywatch

This command collects filesystem usage statistics and outputs counts of each inotify event.

Community
  • 1
Pro Backup
  • 4,686
  • 12
  • 50
  • 82
2

You would need to approach this with a clustered filesystem type solution - a simple sync between the two machines won't give you real-time response.

John
  • 16,759
  • 1
  • 34
  • 43
  • Actually those servers are at AWS. So we cannot get a cluster environment! Isn't there any other way? – Sourav Aug 31 '16 at 18:48
1

"SIOS Protection Suite for Linux" can offer this, on AWS.

http://www.linuxclustering.net/2016/03/21/step-by-step-how-to-configure-a-linux-failover-cluster-in-amazon-ec2-without-shared-storage-amazon-aws-sanless-cluster/#InstallConfigureCluster

steve
  • 21,582
  • 5
  • 48
  • 75
0

Here is my working config, for this example i used /tmp/test both as a local and a distant directory.

I had an error that said ssh-askpass wasn't installed:

sudo apt install ssh-askpass

I also needed to be using a non standard port for ssh.

settings {
    logfile = "/var/log/lsyncd/lsyncd.log",
    statusFile = "/var/log/lsyncd/lsyncd-status.log",
    statusInterval = 1
}

sync {
    default.rsyncssh,
    delete = false,
    source = "/tmp/test",
    rsync = {
    binary = "/usr/bin/rsync",
    archive = true,
    perms = true,
    owner = true,
    _extra = {"-a"},
    update = true,
    protect_args = true,
    sparse = true,
    rsh = "/usr/bin/ssh -l src_username -i /home/src_username/.ssh/id_rsa",
    compress = true
},
ssh = {
    port = 6765,
    extra = {"-i","/home/src_username/.ssh/id_rsa"}
    },
host = "[email protected]",
targetdir = "/tmp/test"
}

And after 4 hours of wandering:

Wed Feb  2 04:22:52 2022 Normal: --- Startup, daemonizing ---
Wed Feb  2 04:22:52 2022 Normal: recursive startup rsync: /tmp/test/ -> [email protected]:/tmp/test/
Warning: Permanently added '[domain.xyz]:6765,[123.123.123.123]:6765' (ECDSA) to the list of known hosts.
Wed Feb  2 04:23:06 2022 Normal: Startup of "/tmp/test/" finished: 0
Wed Feb  2 04:24:13 2022 Normal: Calling rsync with filter-list of new/modified files/dirs 

sent 61 bytes  received 12 bytes  146.00 bytes/sec
total size is 57  speedup is 0.78
Wed Feb  2 04:24:13 2022 Normal: Finished a list after exitcode: 0
assayag.org
  • 103
  • 4
0

I tried lsyncd but had multiple issues, the daemon wouldnt sync files after a certain time, would only sync on a service restart.

I wrote a sync daemon called Poni - written in Crystal, very easy to configure Yaml config, listens on inotify and rsyncs, can specify additional options, see readme

https://github.com/perfecto25/poni

perfecto25
  • 101
  • 1