Just to give a bit of background - I have a 3 node GlusterFS cluster running, and I made sure to test it works by mounting a few directories on some servers. I also tested a mount on the Kubernetes cluster (currently just 1 node), to make sure it's not an issue there, and that the gluster-client package is installed.
Been using Rancher as the management overlay for Kubernetes. I manually created a Persistent Volume and selected Gluster plugin. For the endpoint, I had to create custom Endpoints and Service objects.
Here's the error I have been getting:
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Warning FailedMount 7m8s (x34 over 86m) kubelet, ssk8smstd01 Unable to mount volumes for pod "hello-7f66d8f588-whxwz_default(6a45cd5e-7259-11e9-ab24-005056aa5a40)": timeout expired waiting for volumes to attach or mount for pod "default"/"hello-7f66d8f588-whxwz". list of unmounted volumes=[pvc1]. list of unattached volumes=[pvc1 default-token-n7tlv]
Warning FailedMount 3m2s (x44 over 86m) kubelet, ssk8smstd01 (combined from similar events): MountVolume.SetUp failed for volume "gluster1" : mount failed: mount failed: exit status 1
Mounting command: mount
Mounting arguments: -t glusterfs -o backup-volfile-servers=ip1:ip2:ip3,log-file=/var/lib/kubelet/plugins/kubernetes.io/glusterfs/gluster1/hello-7f66d8f588-whxwz-glusterfs.log,log-level=ERROR ip1:gv0 /var/lib/kubelet/pods/6a45cd5e-7259-11e9-ab24-005056aa5a40/volumes/kubernetes.io~glusterfs/gluster1
Output: Mount failed. Please check the log file for more details.
the following error information was pulled from the glusterfs log to help diagnose this issue:
[2019-05-09 14:19:04.005668] E [MSGID: 100026] [glusterfsd.c:2307:glusterfs_process_volfp] 0-: failed to construct the graph
[2019-05-09 14:19:04.005900] E [graph.c:982:glusterfs_graph_destroy] (-->/usr/sbin/glusterfs(mgmt_getspec_cbk+0x12e) [0x5589853fc38e] -->/usr/sbin/glusterfs(glusterfs_process_volfp+0x148) [0x5589853f6dc8] -->/usr/lib/x86_64-linux-gnu/libglusterfs.so.0(glusterfs_graph_destroy+0x6b) [0x7fa9c305268b] ) 0-graph: invalid argument: graph [Invalid argument]
Has anyone seen something similar to this or have any ideas?
