10

I'm working with a cluster of Linux servers sitting behind the Amazon cloud autoscaling feature (documentation). There are files that accumulate on each server, they are cache files for the application. I am searching for an automated way to have the servers sync that folder amongst themselves.

  • Rsync would work if someone had a clever way to do peer detection.
  • Same goes for Unison.
  • We have also considered using a SaaS such as Dropbox.
  • Gluster seems like overkill, and I'm not sure how easy it is to have a new computer auto-join the cluster.

Ideally what would happen is when a new server is launched, it discovers it's peers and begins syncing the folder. From that point on it's a latest filestamp wins P2P synchronization.

Anything out there that does this?

Caleb
  • 69,278
  • 18
  • 196
  • 226

5 Answers5

1

Rsync upstream provides support for SLP (Service location protocol). It is not enabled in vanilla rsync sources, but you can find the SLP support in rsync-patches tarball at rsync homepage (slp.diff). E.g. SUSE rsync package is built with this patch; I don't know about other distributions.

I'm not sure this is what you are looking for nor do I have any experience with SLP, though.

Jeff Schaller
  • 66,199
  • 35
  • 114
  • 250
Petr Uzel
  • 7,157
  • 4
  • 30
  • 26
1

You know as weird as it might sound but using rtorrent with DHT enabled on the private ec2 range could work

This would give you

  • Automatic Peer discovery
  • Verified file states (yay for hashes)
  • Minimal configuration requirements (add the magnet(or virtual ip for the latest torrent) for the torrent in the start up script)
  • scalability
Tacticus
  • 111
  • 2
0

Maybe I'm missing something here, but is there some reason you don't just setup a simple NFS directory and just have all the machines mount that? Anytime the app writes the new cache file it should just overwrite any old ones.

opsguy
  • 101
  • 1
  • 1
    Hi opsguy. What you're missing is what happens when the machine with the NFS fails? We're trying to eliminate single point of failure for the cache. –  May 25 '11 at 04:04
0

I'm pretty sure there is some Amazon management tool, such as Command Line Tools, that permits you to write a boot script to make nodes discover each others, via private or public IP addresses.

By the way, rsync and Unison are not the right tool, if they have to distribute changes to cache very frequently. In this case you should consider some really distributed file system.

Marco Solieri
  • 51
  • 5
0

Maybe DRBD (distributed block device)?

Gilles 'SO- stop being evil'
  • 807,993
  • 194
  • 1,674
  • 2,175
MSpike
  • 60
  • 3
  • 1
    Thanks for your answer, but it lacks substance. DRBD is not so well-known that it doesn't require at least a link. But really, you should give more information: why is DRBD suitable? How would it work? Especially considering that Kevin thought Gluster overkill: why would DRBD not be? – Gilles 'SO- stop being evil' Jun 17 '11 at 20:08
  • I'm just realized that you looking for automatic peer discovery, so this one is not suitable for you. DRBD is a block device on your real hardware or on the top of some volume, and than it's sync them over the network. – MSpike Jun 21 '11 at 08:07