6

I have a cluster with a bunch of servers with a shared disk containing a GFS global file system that all nodes access simultaneously.

Each node in the cluster run the same program (a shell script is the main core). The system processes files that appear in a couple of input directories, and it works like this:

  • the program loops through the input directories.
  • for each file found, check existence of a "lock file", if lock file exists skip to next file.
  • if no lock file found, create lock file. If lockfile creation failed (race lost), skip to next file
  • if "we" own the lock, process the file and move it out of the way when it is finished.

This all works very well, but I wonder if there are cheaper (less complex) solutions that would also work. I'm thinking NFS or SMB perhaps.

There are two reasons for my use of GFS:

  1. each file is stored in one place only (on redundant underlying hardware of course)
  2. file locking works reliably

I create the lockfile like this:

date '+%s:'${unid} > ${currlock}.${unid}
ln ${currlock}.${unid} ${currlock}
lockrc=$?
rm -f ${currlock}.${unid}

where $unid is a unique session identifier and $currlock is /gfs/tmp/lock.${file_to_process}

The beauty of ln is that it is atomic, so it fails for all but one that attempts the same thing at the same time.

So, I guess what I'm asking is: will NFS fill my needs? Does ln work reliably in the same way on NFS as on GFS?

Gilles 'SO- stop being evil'
  • 807,993
  • 194
  • 1,674
  • 2,175
MattBianco
  • 3,676
  • 6
  • 27
  • 43
  • I note you have one (at the time of writing) close vote, suggesting this is primarily opinion based. I don't necessarily agree - but you could avoid that issue by rewording question to something like, "Is `ln` reliable and atomic on NFS?" – EightBitTony Apr 22 '14 at 10:13
  • @EightBitTony, whoever flagged it as a question for opinions probably only read the question title. I took your advice and updated it to make it more clear that it is indeed a question for factual answers. – MattBianco Apr 22 '14 at 11:13

1 Answers1

3

The link() system call on the NFS client should map directly to the NFS LINK operation, which the server should implement using its link() system call. So as long as link() is atomic on the server, it will also be atomic on the clients.

Barmar
  • 9,648
  • 1
  • 19
  • 28
  • 2
    Quite a few "should" here... :-) Do you know of any issues with this in the "real world"? – MattBianco Apr 22 '14 at 14:06
  • 1
    I was being kind of wishy-washy because NFS is a protocol, not a specific implementation, so there may be variations. E.g. if the NFS server is not running Unix, it may not have an atomic `link` operation. In practice, with Unix clients and either Unix or specialized servers (e.g. Network Appliance), you can safely assume atomicity. – Barmar Apr 22 '14 at 16:20
  • 1
    Do you know if this holds true for what the client sees? That is, could a client conceivably report success before the operation fails on the server (maybe as a result of caching)? If the semantics observed by the server are not precisely observed by the clients as well, you could get into a situation where a client would *think* that they had obtained a lock, for example, when they really hadn't because the operation had actually failed, not taken effect, but the client observed it as having succeeded. – joshlf Aug 14 '15 at 19:38
  • @synful No. An operation like `link` is not idempotent, so it's not reasonable to cache it, and it must implement it synchronously. It simply forwards it to the server, and reports what it returned. – Barmar Aug 15 '15 at 06:47