MySQL DRBD Resource failing to start PaceMaker + Corosync

Question

So I have gotten to the point where I had all services running when I configured the cluster, but after reboot I am getting the following :

Full list of resources:

 virtual_ip     (ocf::heartbeat:IPaddr2):       Started node1
 webserver      (ocf::heartbeat:apache):        Stopped
 Master/Slave Set: WebDataClone [WebData]
     Masters: [ node2 ]
     Slaves: [ node1 ]
 WebFS  (ocf::heartbeat:Filesystem):    Started node2
 sqlfs  (ocf::heartbeat:Filesystem):    Stopped
 Master/Slave Set: SQLDataClone [SQLData]
     Masters: [ node1 ]
     Slaves: [ node2 ]

Failed actions:

    sqlfs_start_0 on node1 'unknown error' (1): call=26, status=complete, last-rc-change='Sat Jun  4 00:57:27 2016', queued=0ms, exec=32ms
    sqlfs_start_0 on node2 'unknown error' (1): call=34, status=complete, last-    rc-change='Sat Jun  4 00:57:27 2016', queued=0ms, exec=59ms

This is what I used for my SQL Configuration :

 resource sqldata {
 protocol C;
 disk   /dev/sdb2;
 device /dev/drbd1;

 startup {
wfc-timeout 30;
outdated-wfc-timeout 20;
degr-wfc-timeout 30;
  }

 net {
cram-hmac-alg sha1;
shared-secret sync_disk;
  }

 syncer {
rate 10M;
al-extents 257;
on-no-data-accessible io-error;
verify-alg sha1;
  }

 on node1.freesoftwareservers.com {
   address  192.168.1.218:7788;
   flexible-meta-disk internal; 
  }

 on node2.freesoftwareservers.com {
   address  192.168.1.221:7788;
   meta-disk internal;
  }

 }

Start DRBD and sync :

drbdadm -- --overwrite-data-of-peer primary sqldata
drbdadm primary --force sqldata
watch cat /proc/drbd

Modified my.cnf gets changed on BOTH servers :

service mysqld stop
sudo nano /etc/my.cnf
Change in /etc/my.cnf :

#bind-address = 127.0.0.1
#Make Virtual_IP
bind-address = 192.168.1.215
#datadir=/var/lib/mysql
#Where DRBD will Mount
datadir=/var/lib/mysql_drbd
Mount and Populate with Data :

Mount and Populate drbd1 :

/sbin/mkfs.ext4 /dev/drbd1
mkdir /mnt/drbd1
mount /dev/drbd1 /mnt/drbd1 
mv /var/lib/mysql_drbd/* /mnt/drbd1/
chcon -R --reference=/var/lib/mysql /mnt/drbd1
umount /dev/drbd1

THIS IS THE CONFIG THAT SEEMS TO FAIL :

pcs cluster start --all
pcs cluster cib sqlfs_cfg
pcs -f sqlfs_cfg resource create sqlfs ocf:heartbeat:Filesystem device="/dev/drbd1" directory="/var/lib/mysql_drbd" fstype="ext4"
pcs -f sqlfs_cfg constraint colocation add sqlfs with SQLDataClone INFINITY with-rsc-role=Master
pcs -f sqlfs_cfg constraint order promote SQLDataClone then start sqlfs
pcs -f sqlfs_cfg constraint colocation add webserver with sqlfs INFINITY
pcs -f sqlfs_cfg constraint order sqlfs then webserver
pcs -f sqlfs_cfg constraint
pcs -f sqlfs_cfg resource show
pcs cluster cib-push sqlfs_cfg

sqlfs_drbd_cfg :

pcs cluster start --all
pcs cluster cib sqlfs_drbd_cfg
pcs -f sqlfs_drbd_cfg resource create SQLData ocf:linbit:drbd drbd_resource=sqldata op monitor interval=60s
pcs -f sqlfs_drbd_cfg resource master SQLDataClone SQLData master-max=1 master-    node-max=1 clone-max=2 clone-node-max=1 notify=true
pcs -f sqlfs_drbd_cfg resource show
pcs cluster cib-push sqlfs_drbd_cfg

Set Prefered Location on node1 AKA Primary :

pcs constraint location sqlfs prefers node1=50

Any thoughts on how to troubleshoot this? Again, it worked during initial config, and broke after reboot...

Thanks!

I would grep the logs (/var/log/messages or /var/log/syslog) for 'sqlfs' and post any errors that were printed to stdout (gets logged by Pacemaker) by the start operation. That will probably give us more clues. — Matt Kereczman, Jun 05 '16 at 17:36

score 2 · Accepted Answer · answered Jun 05 '16 at 17:50

2

I don't see mysql defined in the Pacemaker; are you starting it manually after the cluster starts or does it start at boot?

If it's starting at boot, it could be starting before Pacemaker, and therefore holding a lock on the mountpoint (where it's data should be) which would cause the Filesystem resource to not be able to mount the filesystem there.

Kind of a shot in the dark, but if that's the case, you should disable mysql from starting at boot and then define it in the cluster.

answered Jun 05 '16 at 17:50

Matt Kereczman

669
3
7

@FreeSoftwareServers I'm glad it was helpful. I have alerts setup for certain keywords that are HA cluster related so I should see your new post when it comes along... perhaps you've noticed this pattern ;) – Matt Kereczman Jun 07 '16 at 21:59
disclaimer: I currently work for LINBIT (the original creators and maintainers of DRBD), so there would be a bit of an ethics problem if I accepted money for DRBD/Pacemaker assistance. Head over to https://www.linbit.com/en/p/products if you're interested in paid support, or you can get more focused assistance from the HA community using one of the various channels listed here: https://www.drbd.org/en/community/connecting. The #clusterlabs channel on freenode is also a very good place to ask clustering questions. – Matt Kereczman Jun 08 '16 at 14:30

MySQL DRBD Resource failing to start PaceMaker + Corosync

1 Answers1