-
Problem report
-
Resolution: Commercial support required
-
Trivial
-
None
-
6.0.35, 6.4.19, 7.0.5
-
None
-
debian 12 stable, running on a VM.
Problem should happen on any x86 machine.
Motivation
I've been dealing with a complex bug in mariadb that causes the database to randomly 'seize up' and stop executing queries, filling up the query queue until full, then stopping all connection attempts. (MDEV-34445)
I've been using zabbix to monitor mariadb. Now the template is only tested up to version 10.6, but the developers have requested to upgrade the version. This broke the zabbix monitoring. **
Steps to reproduce:
This is for debian 12 stable;
Install a modern version of mariadb and zabbix-client; Add to apt list:
deb [arch=amd64] http://downloads.mariadb.com/Tools/debian bookworm main
Add the following to the keyring:
wget https://supplychain.mariadb.com/mariadb-keyring-2019.gpg
chmod 644 mariadb-keyring-2019.gpg
mv mariadb-keyring-2019.gpg /etc/apt/trusted.gpg.d/
Then install
apt-get -y install zabbix-agent mariadb-server mariadb-client
Setup a zabbix user. In this case, it will be 'zbx_monitor' with password 'ExamplePass$1'.
Edit /etc/passwd to give 'zabbix' user a home directory, then run:
mkhomedir_helper zabbix
Add this to .my.cnf in the new homedir as follows:
cat /home/zabbix/.my.cnf [client] protocol=tcp user='zbx_monitor' password='ExamplePass$1'
Create this user in mysql with the following command:
CREATE USER 'zbx_monitor'@'localhost' IDENTIFIED BY 'ExamplePass$1'; CREATE USER 'zbx_monitor'@'127.0.0.1' IDENTIFIED BY 'ExamplePass$1'; CREATE USER 'zbx_monitor'@'::1' IDENTIFIED BY 'ExamplePass$1'; GRANT ALL PRIVILEGES ON * . * TO 'zbx_monitor'@'localhost'; GRANT ALL PRIVILEGES ON * . * TO 'zbx_monitor'@'127.0.0.1'; GRANT ALL PRIVILEGES ON * . * TO 'zbx_monitor'@'::1';
Restart both zabbix-agent and mysql. Begin monitoring the error log;
systemctl restart zabbix-agent && systemctl restart mariadb
cd /var/log/mysql/
tail -f error.log
Next, login to the zabbix server, via both web and console. Add the client, connect it to the server (optionally configure certificate, but this has no impact on the error), etc.
Once regular 'ping' template is working, try the following command on the server:
zabbix_get -s sqlserver.example.com -k "mysql.get.status_variables[localhost,3306]"
Result:
The SQL error log will display this:
1680 [Warning] Access denied for user '-sNX'@'localhost' (using password: YES)
While the command displays:
mysql: Deprecated program name. It will be removed in a future release, use '/usr/bin/mariadb' instead ERROR 1045 (28000): Access denied for user '-sNX'@'localhost' (using password: YES)
A command line argument is being gobbled up / missing. I installed and configured auditd, and got the following, more detailed log of what zabbix_agent is doing:
type=SYSCALL msg=audit(1730975305.959:127197): arch=c000003e syscall=59 success=yes exit=0 a0=55a170c23720 a1=55a170c23680 a2=55a170c236c0 a3=8 items=3 ppid=1084568 pid=1084569 auid=4294967295 uid=103 gid=110 euid=103 suid=103 fsuid=103 egid=110 sgid=110 fsgid=110 tty=(none) ses=4294967295 comm="mysql" exe="/usr/bin/mariadb" subj=unconfined key=(null)ARCH=x86_64 SYSCALL=execve AUID="unset" UID="zabbix" GID="zabbix" EUID="zabbix" SUID="zabbix" FSUID="zabbix" EGID="zabbix" SGID="zabbix" FSGID="zabbix" type=EXECVE msg=audit(1730975305.959:127197): argc=7 a0="mysql" a1="-hlocalhost" a2="-P3306" a3="-u" a4="-sNX" a5="-e" a6=73686F7[REDACTED]
(Note that I redacted some of the hash at the end)
Diagnosis
It appears that for whatever reason the username is not being provided. This used to work with mariadb 10.6 and broke with mariadb 11, even though zabbix is the one running the command.
I'm not really sure how that is even possible, but the evidence speaks for itself.