[ZBX-25516] zabbix-agent fails to integrate with mariadb 11.x Created: 2024 Nov 07 Updated: 2024 Nov 12 Resolved: 2024 Nov 12 |
|
Status: | Closed |
Project: | ZABBIX BUGS AND ISSUES |
Component/s: | Agent (G) |
Affects Version/s: | 6.0.35, 6.4.19, 7.0.5 |
Fix Version/s: | None |
Type: | Problem report | Priority: | Trivial |
Reporter: | npr | Assignee: | Zabbix Support Team |
Resolution: | Commercial support required | Votes: | 0 |
Labels: | None | ||
Remaining Estimate: | Not Specified | ||
Time Spent: | Not Specified | ||
Original Estimate: | Not Specified | ||
Environment: |
debian 12 stable, running on a VM. |
Description |
Motivation I've been dealing with a complex bug in mariadb that causes the database to randomly 'seize up' and stop executing queries, filling up the query queue until full, then stopping all connection attempts. (MDEV-34445) I've been using zabbix to monitor mariadb. Now the template is only tested up to version 10.6, but the developers have requested to upgrade the version. This broke the zabbix monitoring. ** Steps to reproduce: This is for debian 12 stable; Install a modern version of mariadb and zabbix-client; Add to apt list:
deb [arch=amd64] http://downloads.mariadb.com/Tools/debian bookworm main
Add the following to the keyring:
wget https://supplychain.mariadb.com/mariadb-keyring-2019.gpg
chmod 644 mariadb-keyring-2019.gpg
mv mariadb-keyring-2019.gpg /etc/apt/trusted.gpg.d/
Then install
apt-get -y install zabbix-agent mariadb-server mariadb-client Setup a zabbix user. In this case, it will be 'zbx_monitor' with password 'ExamplePass$1'. Edit /etc/passwd to give 'zabbix' user a home directory, then run: mkhomedir_helper zabbix Add this to .my.cnf in the new homedir as follows: cat /home/zabbix/.my.cnf [client] protocol=tcp user='zbx_monitor' password='ExamplePass$1' Create this user in mysql with the following command: CREATE USER 'zbx_monitor'@'localhost' IDENTIFIED BY 'ExamplePass$1'; CREATE USER 'zbx_monitor'@'127.0.0.1' IDENTIFIED BY 'ExamplePass$1'; CREATE USER 'zbx_monitor'@'::1' IDENTIFIED BY 'ExamplePass$1'; GRANT ALL PRIVILEGES ON * . * TO 'zbx_monitor'@'localhost'; GRANT ALL PRIVILEGES ON * . * TO 'zbx_monitor'@'127.0.0.1'; GRANT ALL PRIVILEGES ON * . * TO 'zbx_monitor'@'::1'; Restart both zabbix-agent and mysql. Begin monitoring the error log; systemctl restart zabbix-agent && systemctl restart mariadb
cd /var/log/mysql/
tail -f error.log
Next, login to the zabbix server, via both web and console. Add the client, connect it to the server (optionally configure certificate, but this has no impact on the error), etc. Once regular 'ping' template is working, try the following command on the server:
zabbix_get -s sqlserver.example.com -k "mysql.get.status_variables[localhost,3306]"
Result: The SQL error log will display this: 1680 [Warning] Access denied for user '-sNX'@'localhost' (using password: YES) While the command displays: mysql: Deprecated program name. It will be removed in a future release, use '/usr/bin/mariadb' instead ERROR 1045 (28000): Access denied for user '-sNX'@'localhost' (using password: YES) A command line argument is being gobbled up / missing. I installed and configured auditd, and got the following, more detailed log of what zabbix_agent is doing: type=SYSCALL msg=audit(1730975305.959:127197): arch=c000003e syscall=59 success=yes exit=0 a0=55a170c23720 a1=55a170c23680 a2=55a170c236c0 a3=8 items=3 ppid=1084568 pid=1084569 auid=4294967295 uid=103 gid=110 euid=103 suid=103 fsuid=103 egid=110 sgid=110 fsgid=110 tty=(none) ses=4294967295 comm="mysql" exe="/usr/bin/mariadb" subj=unconfined key=(null)ARCH=x86_64 SYSCALL=execve AUID="unset" UID="zabbix" GID="zabbix" EUID="zabbix" SUID="zabbix" FSUID="zabbix" EGID="zabbix" SGID="zabbix" FSGID="zabbix" type=EXECVE msg=audit(1730975305.959:127197): argc=7 a0="mysql" a1="-hlocalhost" a2="-P3306" a3="-u" a4="-sNX" a5="-e" a6=73686F7[REDACTED] (Note that I redacted some of the hash at the end) Diagnosis It appears that for whatever reason the username is not being provided. This used to work with mariadb 10.6 and broke with mariadb 11, even though zabbix is the one running the command. I'm not really sure how that is even possible, but the evidence speaks for itself.
|
Comments |
Comment by npr [ 2024 Nov 07 ] |
This may be helpful in diagnosing the bug; Doing a simple test to see that the .my.cnf config is functional and zabbix-agent is somehow to blame (on the sql database server):
sudo -u zabbix mysql -e "SELECT 1;"
Properly executes a query, proving that the zabbix linux user is allowed into the database via its .my.cnf file, in contrast to whatever zabbix_agentd is actually doing.
sudo -u zabbix mysql -e "SHOW DATABASES;"
Prints the databases, proving that the user has the rights to them as well. Even a more complicated query;
sudo -u zabbix mysql -e "SELECT * FROM 1408_wp2020.wp_options;"
functions just fine. |
Comment by Tomasz Grzechulski [ 2024 Nov 08 ] |
Hello npr, Thank you for contacting with us,
Thank you, |
Comment by npr [ 2024 Nov 08 ] |
A quick > cat /etc/debian_version < 12.7 produces the info that this debian stable is otherwise up to date. Edit: Trying out agent2 isn't really an option because it's backwards incompatible. It utterly breaks the mysql template. mysql.get_status_variables appears to work entirely differently, for example. zabbix_get -s sqlserver.example.com -k "mysql.get_status_variables[localhost,3306]" ZBX_NOTSUPPORTED: Cannot fetch data: Error 1045: Access denied for user '3306'@'localhost' (using password: NO). I tried to mess around with it a bit and get it to load template_db_mysql.conf for testing, but this creates an issue where the agent won't start: zabbix_agent2 [1520024]: ERROR: cannot initialize user parameters: cannot register user parameter "mysql.ping[*] Some of the parameters added to zabbix_agent2 happen to be named the same as the ones used in the official template for zabbix_agent... with different parameter syntax. This is really bad for backwards compatibility. Note that I'm using an adapted version of the standard zabbix mysql integration template with some custom triggers/fields/etc. So going further with this would involve recreating all of that with the new version at the least. Agent2 is not really a viable solution to anyone who's made their own sql template. Renaming all the parameters though, still leaves you with a non-working template after starting the zabbix client for the following reason. All the dependent items now show this: Preprocessing failed for: mysql: Deprecated program name. It will be removed in a future release, use '/usr/bin/mariadb' in... 1. Failed: cannot extract XML value with xpath "/resultset/row[field/text()='Innodb_buffer_pool_pages_free']/field[@name='Value']/text()": cannot parse xml value: Start tag expected, '<' not found The reason for that is a problem with mariadb upstream: for whatever reason it now dumps a line in the stderr before giving the actual output about mysql being a deprecated program name. Zabbix reads both stdout and stderr, so this gets literally copied into zabbix, which then tries to read out that text as xml. (The real xml is on the next line). mysql -sNX -e "show global status" | head -n 10 mysql: Deprecated program name. It will be removed in a future release, use '/usr/bin/mariadb' instead <?xml version="1.0"?><resultset statement="show global status " xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"> <row> <field name="Variable_name">Aborted_clients</field> <field name="Value">3</field> </row> <row> Note the second line in the output above. This mangles the xml to an invalid state. It did lead me to a thing though: the test command used by me in the report isn't actually correct for the version 1 agent, it should be:
zabbix_get -s sqlserver.example.com -k "mysql.get_status_variables[localhost, 3306, zbx_monitor, $PASSWORD]" | head -n 10
This produces the exact same output as above (with the v2 agent), and leads to all SQL items being 'NOT SUPPORTED'. And this also leads me to a hack to solve it for me:
cp /etc/zabbix/zabbix_agentd.conf.d/template_db_mysql.conf /etc/zabbix/zabbix_agentd.conf.d/template_db_mysql.conf.bak
sed -i "s/mysql -h/mariadb -h/g" /etc/zabbix/zabbix_agentd.conf.d/template_db_mysql.conf
systemctl restart zabbix-agent
For the version 2 template, because it's written in go code, the go code would likely need to be modified to use the executable 'mariadb' rather than 'mysql' with mariadb v11+ if it otherwise works in the same way. Edit: It doesn't (uses mysql directly from go) so it shouldn't be a problem when using both v2 client and v2 template. For version 1, to make it work with other types of mysql (regular mysql rather than mariadb), I suppose the only solutions are multiple templates or suppressing that deprecation warning. I found some topics about this too: https://mariadb.com/kb/en/suppress-mariadb-11-deprecation-warnings/ and https://talk.plesk.com/threads/deprecated-program-name-warning-after-upgrading-to-mariadb-11-4.375785/ ... zabbix is not the only tool depending on formatted output that got burned by this.
|
Comment by Edgar Akhmetshin [ 2024 Nov 12 ] |
Debian 12.8/Agent 7.0.5/MariaDB 11.4.4, everything works: root@zero# zabbix_agentd -V |head -n 1 zabbix_agentd (daemon) (Zabbix) 7.0.5 root@zero# mariadb -V mariadb from 11.4.4-MariaDB, client 15.2 for debian-linux-gnu (aarch64) using EditLine wrapper root@zero# zabbix_get -s 127.0.0.1 -k mysql.get_status_variables[localhost,3306] |head -n 5 mysql: Deprecated program name. It will be removed in a future release, use '/usr/bin/mariadb' instead <?xml version="1.0"?> <resultset statement="show global status " xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"> |