7

I just upgraded my MariaDB installation to 10.2.
When I now try to start the service, it tells me it failed an checking the status shows me this rather strange message:

root@server # systemctl status mariadb
* mariadb.service - MariaDB 10.2.20 database server
   Loaded: loaded (/lib/systemd/system/mariadb.service; enabled; vendor preset: enabled)
  Drop-In: /etc/systemd/system/mariadb.service.d
           `-migrated-from-my.cnf-settings.conf
   Active: failed (Result: exit-code) since Wed 2018-12-26 01:54:50 CET; 9min ago
     Docs: man:mysqld(8)
           https://mariadb.com/kb/en/library/systemd/
  Process: 13690 ExecStartPre=/usr/bin/install -m 755 -o mysql -g root -d /var/run/mysqld (code=exited, status=127)
 Main PID: 10811 (code=exited, status=0/SUCCESS)
      CPU: 0

Dec 26 01:54:50 server systemd[1]: Starting MariaDB 10.2.20 database server...
Dec 26 01:54:50 server systemd[1]: mariadb.service: Control process exited, code=exited status=127
Dec 26 01:54:50 server systemd[1]: Failed to start MariaDB 10.2.20 database server.
Dec 26 01:54:50 server systemd[1]: mariadb.service: Unit entered failed state.
Dec 26 01:54:50 server systemd[1]: mariadb.service: Failed with result 'exit-code'.

Not that it says that ExecStartPre=/usr/bin/install -m 755 -o mysql -g root -d /var/run/mysqld (code=exited, status=127) (checking the mariabd.service file it is configured to run the Pre and Post start steps as root) returned the status code 127. From my knowledge that means command not found.
Running that exact same command manually as root just straight up works for me with exit status 0.

Also running mysqld_safe and mysqld works. So nothing wrong there.

Update:

As requested the contents of my /lib/systemd/system/mariadb.service

#
# /etc/systemd/system/mariadb.service
#
# This file is free software; you can redistribute it and/or modify it
# under the terms of the GNU Lesser General Public License as published by
# the Free Software Foundation; either version 2.1 of the License, or
# (at your option) any later version.
#
# Thanks to:
# Daniel Black
# Erkan Yanar
# David Strauss
# and probably others

[Unit]
Description=MariaDB 10.2.20 database server
Documentation=man:mysqld(8)
Documentation=https://mariadb.com/kb/en/library/systemd/
After=network.target
After=syslog.target

[Install]
WantedBy=multi-user.target
Alias=mysql.service
Alias=mysqld.service


[Service]

##############################################################################
## Core requirements
##

Type=notify

# Setting this to true can break replication and the Type=notify settings
# See also bind-address mysqld option.
PrivateNetwork=false

##############################################################################
## Package maintainers
##

User=mysql
Group=mysql

# To allow memlock to be used as non-root user if set in configuration
CapabilityBoundingSet=CAP_IPC_LOCK

# Prevent writes to /usr, /boot, and /etc
ProtectSystem=full

# Doesn't yet work properly with SELinux enabled
# NoNewPrivileges=true

PrivateDevices=true

# Prevent accessing /home, /root and /run/user
ProtectHome=true

# Execute pre and post scripts as root, otherwise it does it as User=
PermissionsStartOnly=true

ExecStartPre=/usr/bin/install -m 755 -o mysql -g root -d /var/run/mysqld

# Perform automatic wsrep recovery. When server is started without wsrep,
# galera_recovery simply returns an empty string. In any case, however,
# the script is not expected to return with a non-zero status.
# It is always safe to unset _WSREP_START_POSITION environment variable.
# Do not panic if galera_recovery script is not available. (MDEV-10538)
ExecStartPre=/bin/sh -c "systemctl unset-environment _WSREP_START_POSITION"
ExecStartPre=/bin/sh -c "[ ! -e /usr/bin/galera_recovery ] && VAR= || \
 VAR=`/usr/bin/galera_recovery`; [ $? -eq 0 ] \
 && systemctl set-environment _WSREP_START_POSITION=$VAR || exit 1"

# Needed to create system tables etc.
# ExecStartPre=/usr/bin/mysql_install_db -u mysql

# Start main service
# MYSQLD_OPTS here is for users to set in /etc/systemd/system/mariadb.service.d/MY_SPECIAL.conf
# Use the [Service] section and Environment="MYSQLD_OPTS=...".
# This isn't a replacement for my.cnf.
# _WSREP_NEW_CLUSTER is for the exclusive use of the script galera_new_cluster
ExecStart=/usr/sbin/mysqld $MYSQLD_OPTS $_WSREP_NEW_CLUSTER $_WSREP_START_POSITION
ExecStartPost=/etc/mysql/debian-start

# Unset _WSREP_START_POSITION environment variable.
ExecStartPost=/bin/sh -c "systemctl unset-environment _WSREP_START_POSITION"

KillSignal=SIGTERM

# Don't want to see an automated SIGKILL ever
SendSIGKILL=no

# Restart crashed server only, on-failure would also restart, for example, when
# my.cnf contains unknown option
Restart=on-abort
RestartSec=5s

UMask=007

##############################################################################
## USERs can override
##
##
## by creating a file in /etc/systemd/system/mariadb.service.d/MY_SPECIAL.conf
## and adding/setting the following under [Service] will override this file's
## settings.

# Useful options not previously available in [mysqld_safe]

# Kernels like killing mysqld when out of memory because its big.
# Lets temper that preference a little.
# OOMScoreAdjust=-600

# Explicitly start with high IO priority
# BlockIOWeight=1000

# If you don't use the /tmp directory for SELECT ... OUTFILE and
# LOAD DATA INFILE you can enable PrivateTmp=true for a little more security.
PrivateTmp=false

##
## Options previously available to be set via [mysqld_safe]
## that now needs to be set by systemd config files as mysqld_safe
## isn't executed.
##

# Number of files limit. previously [mysqld_safe] open-file-limit
LimitNOFILE=16364

# Maximium core size. previously [mysqld_safe] core-file-size
# LimitCore=

# Nice priority. previously [mysqld_safe] nice
# Nice=-5

# Timezone. previously [mysqld_safe] timezone
# Environment="TZ=UTC"

# Library substitutions. previously [mysqld_safe] malloc-lib with explicit paths
# (in LD_LIBRARY_PATH) and library name (in LD_PRELOAD).
# Environment="LD_LIBRARY_PATH=/path1 /path2" "LD_PRELOAD=

# Flush caches. previously [mysqld_safe] flush-caches=1
# ExecStartPre=sync
# ExecStartPre=sysctl -q -w vm.drop_caches=3

# numa-interleave=1 equalivant
# Change ExecStart=numactl --interleave=all /usr/sbin/mysqld......

# crash-script equalivent
# FailureAction=

Also I am running Debian 9.

Update 2:

All I did was upgrade MariaDB through apt. nothing out of the ordinary. So all files (except config) should be standard.

BrainStone
  • 3,534
  • 12
  • 32
  • 53
  • show your `/lib/systemd/system/mariadb.service` file. – Alexander Jan 02 '19 at 00:26
  • @Alexander there you go. – BrainStone Jan 02 '19 at 20:57
  • comment out that `ExecStartPre=/usr/bin/install .. /var/run/mysqld` line and restart the `mysql` service (that dir already exists and has the right permissions if you ran the command as root). This looks like systemd cannot tell its children apart -- let's see which of them it will complain about now. –  Jan 02 '19 at 21:29
  • I suspect that `galera_recovery` stuff; I suggest you try commenting it and see if it makes any difference; and please mention in your question **how** you did upgrade to mariadb 10.2 and the exact release of your debian (you can get it with `lsb_release -r`) –  Jan 02 '19 at 21:50
  • @BrainStone, I've no idea what is wrong, but I don't like `PermissionsStartOnly=true`, as I Can't see it in `https://www.freedesktop.org/software/systemd/man/systemd.directives.html` any more. Check Your man systemd.service if it's there. May be it was deleted/broken somehow in systemd. Try to replace it with `RuntimeDirectory=/var/run/mysqld, RuntimeDirectoryMode=755, RuntimeDirectoryPreserve=restart` – Alexander Jan 02 '19 at 22:50
  • @Alexander that's the standard file from debian stretch (except for the `After=syslog.target` line). My theory is that something else is failing (the `galera_recovery` script, whose status should be ignored) and systemd gets confused and mis-attributes its exit status to `install .. /var/run/mysqld`. The OP didn't tell yet how they upgraded to mariadb 10.2, or what other mods were done to the system, so it could be just some local mixup. –  Jan 03 '19 at 00:39
  • @UncleBilly. That's clear, but systemd said, that `/usr/bin/install command not found`, and BrainStone, said, that this command works fine, so binary exists. For me it points us to some bug in systemd or permission problem in that unit. And as I said. If something is only in Debian docs and not in Officcial systemd docs it's weered, and checking will not take much time – Alexander Jan 03 '19 at 01:01
  • @UncleBilly I updated the post. I will try that out later today to see if commenting out the `galera_recovery` stuff makes a difference – BrainStone Jan 04 '19 at 14:09
  • does running mysql_upgrade allows you to boot properly? – Rui F Ribeiro Jan 04 '19 at 22:25
  • @RuiFRibeiro yes it does. As mentioned in the question, I can start MariaDB just fine manually. Just not through `systemd`. – BrainStone Jan 04 '19 at 23:37
  • @BrainStone: could you please replace the content of `/lib/systemd/system/mariadb.service` with the output of `systemd cat mariadb.service` since there seems to be drop-in configuration at `/etc/systemd/system/mariadb.service.d/migrated-from-my.cnf-settings.conf`. Maybe looking at the logs with `journalctl -f` while trying to start `mariadb` reveals some more hints. – Thomas Jan 06 '19 at 15:40

5 Answers5

4

This happened to me, the problem was that systemd was not updated to a version compatible with the mariadb.service configuration. I updated systemd to 241 and it started working.

sbingner
  • 41
  • 4
  • That's was the soluction thanks. I got this problem upgrading from Ubuntu 18.04 to Ubuntu 20.04. Fixed thanks to your answer with an `apt install systemd` – NetVicious Feb 21 '23 at 08:51
4

in my case

systemctl daemon-reexec

then

service mysql start
Mouhcine MIMYA
  • 141
  • 1
3

The only command that worked for me was using aptitude this takes a look at all necessary dependencies etc. when installing new packages.

sudo aptitude install mariadb-server

Another problem you may have especially if you had another db server installed is your apparmor blocking connection. Try this:

sudo /etc/init.d/apparmor stop sudo /etc/init.d/apparmor teardown sudo update-rc.d -f apparmor remove

Josh
  • 131
  • 4
1

I had similar case. Workaround: if you have /etc/init.d/mysql, delete mariadb.service, mysqld.service, mysql.service.

systemctl daemon-reload
/etc/init.d/mysql start
TPS
  • 2,483
  • 5
  • 27
  • 45
tester
  • 11
  • 1
0

If you can make a copy of your data and reinstall mariadb you can try this:

Assuming you are using Debian amd64:

Download https://packages.debian.org/stretch/amd64/libmariadb-dev/download and apt-get install libmariadb2 after dpkg -i libmariadb-devxxxxx.deb ./configure --with-tls --with-virtualchroot --with-puredb --with-quotas --with-throttling --with-mysql make install-strip

JonLord
  • 644
  • 1
  • 4
  • 18
  • 3
    How would compiling from source fix issues with systemd? – BrainStone Jan 01 '19 at 21:27
  • I apologize. After rereading your question, I realize that maybe I misunderstood. Did you try to reinstall the package through the package manager? Please give more information about your system. – JonLord Jan 01 '19 at 22:02
  • I added more info, like me running Debian 9. If you need specific info, just ask :) – BrainStone Jan 02 '19 at 20:58