Here's the premise:
I have NFSv4 (only) server configured under Centos7 and running fine, here's the export (it's a sandboxed environment, so no security concerns):
server> cat /etc/exports
/export/dcs *(sec=sys,rw,no_root_squash,sync,no_subtree_check)
server> cat /proc/fs/nfsd/versions
-2 -3 +4 +4.1 +4.2
server>
I know that NFSv4 server and its configured share work fine, because I'm able to mount it from the MacOS (Monterey) and from a remote docker container (under Centos7).
Problem:
However, on one particular machine (also Centos 7), after installing nfs-utils, I'm unable to mount the same share:
bash# rpm -qa | grep nfs-utils
nfs-utils-1.3.0-0.65.el7.i686
bash#
bash# mkdir -p /mnt/nfs
bash#
bash# mount.nfs4 10.1.2.3:/export/dsc /mnt/nfs
mount.nfs4: Protocol not supported
bash#
A) I know that tcp & udp nfs ports (2049) are reachable from the client - verified with netcat
B) when s-tracing the mount.nfs4 I see this failure:
bash# strace mount.nfs4 10.1.2.3:/export/dsc /mnt/nfs
...
socket(AF_INET, SOCK_DGRAM, IPPROTO_UDP) = 3
bind(3, {sa_family=AF_INET, sin_port=htons(0), sin_addr=inet_addr("0.0.0.0")}, 16) = 0
connect(3, {sa_family=AF_INET, sin_port=htons(0), sin_addr=inet_addr("10.1.2.3")}, 16) = 0
getsockname(3, {sa_family=AF_INET, sin_port=htons(57709), sin_addr=inet_addr("10.4.5.6")}, [16]) = 0
close(3) = 0
mount("10.1.2.3:/export/dsc", "/mnt/nfs", "nfs4", 0, "vers=4.1,addr=10.1.2.3,client"...) = -1 EPROTONOSUPPORT (Protocol not supported)
...
The error that mount() call returns (EPROTONOSUPPORT) is pertain to some local / client's checks, because when I run tcpdump on the server, I observe that no single packet is sent by the client machine during mount operation.
I also tried mounting with: LIBMOUNT_DEBUG=0xffff mount.nfs4 -vvv 10.1.2.3:/export/dsc /mnt/nfs but it did not reveal any obvious failures:
bash# LIBMOUNT_DEBUG=0xffff mount.nfs4 -vvv 10.1.2.3:/export/dsc /mnt/nfs
5733: libmount: INIT: library debug mask: 0xffff
5733: libmount: INIT: library version: 2.23.0
5733: libmount: INIT: feature: selinux
5733: libmount: INIT: feature: debug
5733: libmount: INIT: feature: assert
5733: libmount: CXT: [0x576b2008]: ----> allocate
5733: libmount: UTILS: mtab: /etc/mtab
5733: libmount: UTILS: /etc/mtab: irregular/non-writable
5733: libmount: UTILS: utab: /run/mount/utab
5733: libmount: UTILS: try write /run/mount/utab dir: /run/mount
5733: libmount: UTILS: access OK [/run/mount]
5733: libmount: CXT: [0x576b2008]: enabling flag 0020
5733: libmount: CXT: [0x576b2008]: enabling flag 2000000
5733: libmount: CXT: [0x576b2008]: initialized for [u]mount.<type> helper [rc=0]
5733: libmount: CXT: [0x576b2008]: enabling flag 0010
5733: libmount: CXT: [0x576b2008]: enabling flag 0010
5733: libmount: CXT: [0x576b2008]: enabling flag 0010
5733: libmount: CACHE: [0x576b2158]: alloc
5733: libmount: CACHE: [0x576b2158]: add entry [ 1] (path): /mnt/nfs: /mnt/nfs
5733: libmount: CXT: [0x576b2008]: use default optmode
5733: libmount: CXT: [0x576b2008]: OPTSMODE: ignore=0, append=0, prepend=1, replace=0, force=0, fstab=1, mtab=1
5733: libmount: CXT: [0x576b2008]: fstab not required -- skip
5733: libmount: CXT: [0x576b2008]: mount: preparing
5733: libmount: CXT: [0x576b2008]: OPTSMODE: ignore=0, append=0, prepend=1, replace=0, force=0, fstab=1, mtab=1
5733: libmount: CXT: [0x576b2008]: fstab not required -- skip
5733: libmount: CXT: [0x576b2008]: merging mount flags
5733: libmount: CXT: [0x576b2008]: final flags: VFS=00000000 user=00000000
5733: libmount: CXT: [0x576b2008]: mount: evaluating permissions
5733: libmount: CXT: [0x576b2008]: mount: fixing optstr
5733: libmount: CXT: [0x576b2008]: mount: fixing vfs optstr
5733: libmount: CXT: applying 0x00000000 flags to '(null)'
5733: libmount: CXT: new optstr 'rw'
5733: libmount: CXT: [0x576b2008]: mount: fixing user optstr
5733: libmount: CXT: applying 0x00000000 flags to '(null)'
5733: libmount: CXT: new optstr '(null)'
5733: libmount: CXT: [0x576b2008]: fixed options [rc=0]: vfs: 'rw' fs: '(null)' user: '(null)', optstr: 'rw'
5733: libmount: CXT: [0x576b2008]: preparing source path
5733: libmount: CXT: [0x576b2008]: preparing target path
5733: libmount: CXT: [0x576b2008]: final target '/mnt/nfs'
5733: libmount: CXT: [0x576b2008]: FS type: nfs4 [rc=0]
mount.nfs4: timeout set for Tue Apr 18 11:20:16 2023
mount.nfs4: trying text-based options 'vers=4.1,addr=10.1.2.3,clientaddr=10.4.5.6'
mount.nfs4: mount(2): Protocol not supported
mount.nfs4: trying text-based options 'vers=4.0,addr=10.1.2.3,clientaddr=10.4.5.6'
mount.nfs4: mount(2): Protocol not supported
mount.nfs4: Protocol not supported
5733: libmount: CXT: [0x576b2008]: syscall status set to: -1
5733: libmount: CXT: [0x576b2008]: prepare update
5733: libmount: CXT: [0x576b2008]: skip update: syscall failed [status=-1]
5733: libmount: CXT: [0x576b2008]: don't update: no update prepared
5733: libmount: CXT: [0x576b2008]: <---- reset [status=0] ---->
5733: libmount: CXT: [0x576b2008]: tabfiler disabled
5733: libmount: CACHE: [0x576b2158]: free
5733: libmount: CXT: [0x576b2008]: <---- free
bash#
What would be a reason for that error on the client (knowing it's not a result of protocol version negotiation)? And how further to troubleshoot this issue?
UPD, extra debug info (as requested by @aviro):
here's rpcdebugs collected during mount.nfs4 operation:
Apr 24 14:52:52 nfsclient kernel: [13064.997882] NFS: nfs mount opts='vers=4.1,addr=10.1.2.3,clientaddr=10.4.5.6'
Apr 24 14:52:52 nfsclient kernel: [13064.997884] NFS: parsing nfs mount option 'vers=4.1'
Apr 24 14:52:52 nfsclient kernel: [13064.997890] NFS: parsing nfs mount option 'addr=10.1.2.3'
Apr 24 14:52:52 nfsclient kernel: [13064.997894] NFS: parsing nfs mount option 'clientaddr=10.4.5.6'
Apr 24 14:52:52 nfsclient kernel: [13064.997897] NFS: MNTPATH: '/export/disco'
Apr 24 14:52:52 nfsclient kernel: [13065.000040] NFS: nfs mount opts='vers=4.0,addr=10.1.2.3,clientaddr=10.4.5.6'
Apr 24 14:52:52 nfsclient kernel: [13065.000042] NFS: parsing nfs mount option 'vers=4.0'
Apr 24 14:52:52 nfsclient kernel: [13065.000047] NFS: parsing nfs mount option 'addr=10.1.2.3'
Apr 24 14:52:52 nfsclient kernel: [13065.000050] NFS: parsing nfs mount option 'clientaddr=10.4.5.6'
Apr 24 14:52:52 nfsclient kernel: [13065.000053] NFS: MNTPATH: '/export/disco'
And output with strace -s1024:
https://pastebin.ubuntu.com/p/8mMZmgR8w9/
(posted there as the output is too big to fit here)
PS. please do not offer upgrading the nfs-utils RPM - the same one works fine from another spin-up (docker environment), plus it's beyond my control.
