[ZBX-25516] zabbix-agent fails to integrate with mariadb 11.x Created: 2024 Nov 07  Updated: 2024 Nov 12  Resolved: 2024 Nov 12

Status: Closed
Project: ZABBIX BUGS AND ISSUES
Component/s: Agent (G)
Affects Version/s: 6.0.35, 6.4.19, 7.0.5
Fix Version/s: None

Type: Problem report Priority: Trivial
Reporter: npr Assignee: Zabbix Support Team
Resolution: Commercial support required Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified
Environment:

debian 12 stable, running on a VM.
Problem should happen on any x86 machine.



 Description   

Motivation

I've been dealing with a complex bug in mariadb that causes the database to randomly 'seize up' and stop executing queries, filling up the query queue until full, then stopping all connection attempts. (MDEV-34445) 

I've been using zabbix to monitor mariadb. Now the template is only tested up to version 10.6, but the developers have requested to upgrade the version. This broke the zabbix monitoring. **

Steps to reproduce:

This is for debian 12 stable;

Install a modern version of mariadb and zabbix-client; Add to apt list:

 

deb [arch=amd64] http://downloads.mariadb.com/Tools/debian bookworm main

Add the following to the keyring:

 

 

wget https://supplychain.mariadb.com/mariadb-keyring-2019.gpg
chmod 644 mariadb-keyring-2019.gpg
mv mariadb-keyring-2019.gpg /etc/apt/trusted.gpg.d/ 

Then install

 

apt-get -y install zabbix-agent mariadb-server mariadb-client

Setup a zabbix user. In this case, it will be 'zbx_monitor' with password 'ExamplePass$1'.

Edit /etc/passwd to give 'zabbix' user a home directory, then run:

mkhomedir_helper zabbix

Add this to .my.cnf in the new homedir as follows: 

cat /home/zabbix/.my.cnf

[client]
protocol=tcp
user='zbx_monitor'
password='ExamplePass$1'

Create this user in mysql with the following command:

CREATE USER 'zbx_monitor'@'localhost' IDENTIFIED BY 'ExamplePass$1';
CREATE USER 'zbx_monitor'@'127.0.0.1' IDENTIFIED BY 'ExamplePass$1';
CREATE USER 'zbx_monitor'@'::1' IDENTIFIED BY 'ExamplePass$1';
GRANT ALL PRIVILEGES ON * . * TO 'zbx_monitor'@'localhost';
GRANT ALL PRIVILEGES ON * . * TO 'zbx_monitor'@'127.0.0.1';
GRANT ALL PRIVILEGES ON * . * TO 'zbx_monitor'@'::1';

Restart both zabbix-agent and mysql. Begin monitoring the error log;

systemctl restart zabbix-agent && systemctl restart mariadb
cd /var/log/mysql/
tail -f error.log

Next, login to the zabbix server, via both web and console. Add the client, connect it to the server (optionally configure certificate, but this has no impact on the error), etc.

Once regular 'ping' template is working, try the following command on the server:

 

zabbix_get -s sqlserver.example.com -k "mysql.get.status_variables[localhost,3306]"

Result:

The SQL error log will display this:

1680 [Warning] Access denied for user '-sNX'@'localhost' (using password: YES)

While the command displays:

mysql: Deprecated program name. It will be removed in a future release, use '/usr/bin/mariadb' instead
ERROR 1045 (28000): Access denied for user '-sNX'@'localhost' (using password: YES)

A command line argument is being gobbled up / missing. I installed and configured auditd, and got the following, more detailed log of what zabbix_agent is doing:

type=SYSCALL msg=audit(1730975305.959:127197): arch=c000003e syscall=59 success=yes exit=0 a0=55a170c23720 a1=55a170c23680 a2=55a170c236c0 a3=8 items=3 ppid=1084568 pid=1084569 auid=4294967295 uid=103 gid=110 euid=103 suid=103 fsuid=103 egid=110 sgid=110 fsgid=110 tty=(none) ses=4294967295 comm="mysql" exe="/usr/bin/mariadb" subj=unconfined key=(null)ARCH=x86_64 SYSCALL=execve AUID="unset" UID="zabbix" GID="zabbix" EUID="zabbix" SUID="zabbix" FSUID="zabbix" EGID="zabbix" SGID="zabbix" FSGID="zabbix"
type=EXECVE msg=audit(1730975305.959:127197): argc=7 a0="mysql" a1="-hlocalhost" a2="-P3306" a3="-u" a4="-sNX" a5="-e" a6=73686F7[REDACTED]

(Note that I redacted some of the hash at the end)

Diagnosis

It appears that for whatever reason the username is not being provided. This used to work with mariadb 10.6 and broke with mariadb 11, even though zabbix is the one running the command.

I'm not really sure how that is even possible, but the evidence speaks for itself.

 



 Comments   
Comment by npr [ 2024 Nov 07 ]

This may be helpful in diagnosing the bug;

Doing a simple test to see that the .my.cnf config is functional and zabbix-agent is somehow to blame (on the sql database server):

sudo -u zabbix mysql -e "SELECT 1;"

Properly executes a query, proving that the zabbix linux user is allowed into the database via its .my.cnf file, in contrast to whatever zabbix_agentd is actually doing.

sudo -u zabbix mysql -e "SHOW DATABASES;"

Prints the databases, proving that the user has the rights to them as well. Even a more complicated query;

sudo -u zabbix mysql -e  "SELECT * FROM 1408_wp2020.wp_options;" 

functions just fine.

Comment by Tomasz Grzechulski [ 2024 Nov 08 ]

Hello npr,

Thank you for contacting with us,
We will look into that, but two questions:

  • Which exact version of Debian 12 was used? the newest, 12.7?
  • Could you also please test how it does perform with Zabbix Agent2?

Thank you,
Regards,
Tomasz

Comment by npr [ 2024 Nov 08 ]

A quick

> cat /etc/debian_version
< 12.7 

produces the info that this debian stable is otherwise up to date.

Edit: Trying out agent2 isn't really an option because it's backwards incompatible. It utterly breaks the mysql template.

mysql.get_status_variables appears to work entirely differently, for example.

zabbix_get -s sqlserver.example.com -k "mysql.get_status_variables[localhost,3306]" 

ZBX_NOTSUPPORTED: Cannot fetch data: Error 1045: Access denied for user '3306'@'localhost' (using password: NO).

I tried to mess around with it a bit and get it to load template_db_mysql.conf for testing, but this creates an issue where the agent won't start:

zabbix_agent2 [1520024]: ERROR: cannot initialize user parameters: cannot register user parameter "mysql.ping[*]

Some of the parameters added to zabbix_agent2 happen to be named the same as the ones used in the official template for zabbix_agent... with different parameter syntax. This is really bad for backwards compatibility. Note that I'm using an adapted version of the standard zabbix mysql integration template with some custom triggers/fields/etc. So going further with this would involve recreating all of that with the new version at the least.

Agent2 is not really a viable solution to anyone who's made their own sql template.

Renaming all the parameters though, still leaves you with a non-working template after starting the zabbix client for the following reason. All the dependent items now show this:

Preprocessing failed for: mysql: Deprecated program name. It will be removed in a future release, use '/usr/bin/mariadb' in...
1. Failed: cannot extract XML value with xpath "/resultset/row[field/text()='Innodb_buffer_pool_pages_free']/field[@name='Value']/text()": cannot parse xml value: Start tag expected, '<' not found 

The reason for that is a problem with mariadb upstream: for whatever reason it now dumps a line in the stderr before giving the actual output about mysql being a deprecated program name.

Zabbix reads both stdout and stderr, so this gets literally copied into zabbix, which then tries to read out that text as xml. (The real xml is on the next line).

mysql -sNX -e "show global status" | head -n 10
mysql: Deprecated program name. It will be removed in a future release, use '/usr/bin/mariadb' instead
<?xml version="1.0"?><resultset statement="show global status
" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance">
  <row>
        <field name="Variable_name">Aborted_clients</field>
        <field name="Value">3</field>
  </row>  <row> 

Note the second line in the output above. This mangles the xml to an invalid state.

It did lead me to a thing though: the test command used by me in the report isn't actually correct for the version 1 agent, it should be:

zabbix_get -s sqlserver.example.com -k "mysql.get_status_variables[localhost, 3306, zbx_monitor, $PASSWORD]" | head -n 10

This produces the exact same output as above (with the v2 agent), and leads to all SQL items being 'NOT SUPPORTED'.

And this also leads me to a hack to solve it for me:

cp /etc/zabbix/zabbix_agentd.conf.d/template_db_mysql.conf /etc/zabbix/zabbix_agentd.conf.d/template_db_mysql.conf.bak
sed -i "s/mysql -h/mariadb -h/g" /etc/zabbix/zabbix_agentd.conf.d/template_db_mysql.conf
systemctl restart zabbix-agent

For the version 2 template, because it's written in go code, the go code would likely need to be modified to use the executable 'mariadb' rather than 'mysql' with mariadb v11+ if it otherwise works in the same way. Edit: It doesn't (uses mysql directly from go) so it shouldn't be a problem when using both v2 client and v2 template.

For version 1, to make it work with other types of mysql (regular mysql rather than mariadb), I suppose the only solutions are multiple templates or suppressing that deprecation warning.

I found some topics about this too: https://mariadb.com/kb/en/suppress-mariadb-11-deprecation-warnings/ and https://talk.plesk.com/threads/deprecated-program-name-warning-after-upgrading-to-mariadb-11-4.375785/ ... zabbix is not the only tool depending on formatted output that got burned by this.

 

 

 

Comment by Edgar Akhmetshin [ 2024 Nov 12 ]

Debian 12.8/Agent 7.0.5/MariaDB 11.4.4, everything works:

root@zero# zabbix_agentd -V |head -n 1
zabbix_agentd (daemon) (Zabbix) 7.0.5

root@zero# mariadb -V
mariadb from 11.4.4-MariaDB, client 15.2 for debian-linux-gnu (aarch64) using  EditLine wrapper

root@zero# zabbix_get -s 127.0.0.1 -k mysql.get_status_variables[localhost,3306] |head -n 5
mysql: Deprecated program name. It will be removed in a future release, use '/usr/bin/mariadb' instead
<?xml version="1.0"?>

<resultset statement="show global status
" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance">
Generated at Wed Apr 30 06:19:21 EEST 2025 using Jira 9.12.4#9120004-sha1:625303b708afdb767e17cb2838290c41888e9ff0.