1

In my GFS clusters I use the CDPN feature to have separate chrooted /dev/log directories on separate cluster nodes:

/home/ftpuser/foo:
lrwxrwxrwx   1 root root   18 Sep 26  2010 dev -> .sys/@hostname/dev

/home/ftpuser/foo/.sys:
drwx--x--x 3 root root 3864 Sep 26  2010 server1.example.com
drwx--x--x 3 root root 3864 Sep 26  2010 server2.example.com
drwx--x--x 3 root root 3864 Sep 26  2010 server3.example.com

/home/ftpuser/foo/.sys/server2.example.com:
drwx--x--x 2 root root 3864 Sep 25 09:34 dev

/home/ftpuser/foo/.sys/server2.example.com/dev:
srw-rw-rw- 1 root root    0 Sep 25 09:23 log

/home/ftpuser/foo/dev: (transparently picking 1 subdir depending on node name)
srw-rw-rw- 1 root root    0 Sep 25 09:23 log

I use this so the rsyslog daemon on each node doesn't interfere with eachother. It works because @hostname in a path is replaced with the hostname of the host that interprets it, so different hosts get a different directory. The clusters are active on all nodes simultaneously.

My questions:

  • Is there a way to get corresponding functionality on an NFS share?
  • Could it in theory be implemented in the linux kernel on all filesystems (via a mount option so it doesn't break stuff by default)?

This question is similar but not identical to this one: NFS file with same name but different content depending on host

Community
  • 1
MattBianco
  • 3,676
  • 6
  • 27
  • 43

1 Answers1

1

I don't think CDPN exists for NFS, but you can achieve something roughly equivalent with basic tools. The limitation is that you have to put all your node-specific files in the same location (or at least you have to keep a list of locations), you can't use the @hostname feature anywhere you like.

Mount a local filesystem on all nodes at the same location, e.g. /local. On that filesystem, create a symbolic link whose target varies between nodes and points to the node-specific area of the remote filesystem. You don't need any local storage for that, it can be an in-memory filesystem; since it only needs to store one symbolic link, the overhead is tiny.

mount -t tmpfs -o noexec,nodev,nosuid,mode=755,nr_inodes=2,nr_blocks=2 local-redirect /local
ln -s "/nfs/.sys/$HOSTNAME" /local/storage

Use /local/storage where you would use .sys/@hostname in your example.

A different, Linux-specific approach is to make a bind mount on each node. Have an empty directory on the shared filesystem, and bind-mount @hostname to it after mounting the NFS filesystem.

mount --bind "/.nfs/sys/$HOSTNAME" /nfs/.sys/@hostname
Gilles 'SO- stop being evil'
  • 807,993
  • 194
  • 1,674
  • 2,175
  • The thing that makes redhats own suggestion: bind mounts, awkward is that it (to my understanding) requires a mount for each CDPN location. I have one inside each (s)ftp user's chroot jail, which means an awful lot of mounts. These chroot environments also rules out symlinking outside the chroot, as in your first suggestion, which I think would be the cleaner solution otherwise. Any knowledge of a limitation to the number of concurrent mounts in the linux kernel? (I'll probably ask a new question about syslogging from chroots). Thanks Gilles. Always good unix answers from you! – MattBianco Nov 11 '14 at 08:08
  • @MattBianco I'm not aware of any limitation, but having lots of mount points isn't a common use case, so even if there's no hard limit you might run into efficiencies. But you must have at least a mechanism to mount `/proc` into each chroot, can't you piggyback `/local` onto that? – Gilles 'SO- stop being evil' Nov 11 '14 at 14:12
  • The chroots functions without `/proc`, since they are only used by the `internal-sftp` in `sshd` and `vsftpd`'s corresponding mechanism. No new/interactive processes are launched in the jails. The only contact with the world outside the jails has been through the `/dev/log` socket and the sshd/vsftpd process. – MattBianco Nov 12 '14 at 11:34
  • @MattBianco Ah, ok. Would [pam_mount](http://pam-mount.sourceforge.net/) do? – Gilles 'SO- stop being evil' Nov 12 '14 at 11:44
  • I need syslogging from inside each chroot on each active cluster node (which I asume doesn't work if `/dev/log` is shared with all nodes), and I need all nodes accesing the same chroots. I don't think `pam_mount` will do for this scenario, but I'll investigate it. Thanks again, Gilles – MattBianco Nov 12 '14 at 11:48