-
Problem report
-
Resolution: Unresolved
-
Trivial
-
None
-
6.0.40, 6.0.41, 6.0.42rc1, 7.0.17, 7.0.18, 7.0.19rc1, 7.2.11, 7.2.12, 7.2.13rc1, 7.4.1, 7.4.2, 7.4.3rc1
-
Any modern enough linux.
This is about the official template "MySQL by Zabbix agent", and a problem for any v6 and v7 zabbix when combined with newer mariadb.
I found something curious after updating to debian13, an interaction between the newer mariadb and the way the zabbix mysql template works.
For example, the way the 'status variables' (from which a large amount of the zabbix items are derived) are retrieved is:
UserParameter=mysql.get_status_variables[*], export MYSQL_PWD="$4"; mysql -h"$1" -P"$2" -u"$3" -sNX -e "show global status"
This creates the following command, by default:
export MYSQL_PWD="zbx_password" mysql -h"localhost" -P"3306" -u"zbx_monitor" -sNX -e "show global status"
While in mariadb on debian 12 this plain outputs the status variables, the output in mariadb 10.11.14, gotten like so:
zabbix_get -s servername -k 'mysql.get_status_variables["localhost", "3306", "zbx_monitor", "zbx_password"]' > status_vars.xml
Will instead output:
WARNING: option --ssl-verify-server-cert is disabled, because of an insecure passwordless login. <?xml version="1.0"?><resultset statement="show global status " xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"> <row> <field name="Variable_name">Aborted_clients</field> <field name="Value">0</field> </row> <row> <field name="Variable_name">Aborted_connects</field> <field name="Value">19</field> </row> ...
This Warning is from stderr, not stdout. It goes in front of the XML... and changes it to be invalid xml, which basically breaks the data collection.
In this particular case, adding
--ssl-verify-server-cert=false
to each of the mysql commands within the file
/etc/zabbix/zabbix_agentd.conf.d/template_db_mysql.conf
will solve this problem and your template will work. The real issue though is, that it won't work out of the box on the most standard setup (a locally hosted mysql instance) because of this warning message that newer versions of mariaDB will output if you try to connect without SSL. This message is fine when you are an interactive command line user, not so much when you are a program trying to parse the output if it's in some rigid file format like XML.
The same problem really goes for any warning mysql might produce. It even applies in general to all custom templates that interact with other programs (pretty much any template). Discarding the stderr outputs entirely would be another solution, to do so you could very basically just add this to the command in the template:
2>/dev/null
this will trash any errors produced. But, here's a couple of problems with either solution:
- You may want to verify SSL. If you are using a zabbix agent on host bob.acme.com to report to zabbix.acme.com about alice.acme.com, then you may want bob to verify that the SQL server 'alice' is who it says it is and check its SSL certificate.
- If you discard this error you won't 'see' that SSL isn't properly being verified when, with a more complicated setup than the default, things are misconfigured.
- If you discard other warnings in general it might be more tricky for someone to debug a problem with their template. No data will be being sent and you won't notice something is wrong until the problem becomes serious, bypassing zabbix, so you have to 'know' to look and run the commands manually, which tends to invalidate the reason for having a monitoring system in the first place.
(It's why I created a 'general' template with an alert if there are any unsupported items on a host).
Possible solution:
For this particular example issue, perhaps you could simply add MYSQL.VERIFY_SERVER_CERT as a macro option with default value 'false' and instruct people who want to use zabbix to monitor an external server to use SSL and to turn this parameter to TRUE.
In that case, the example line would turn into:
UserParameter=mysql.get_status_variables[*], export MYSQL_PWD="$4"; mysql -h"$1" -P"$2" -u"$3" --ssl-verify-server-cert="$5" -sNX -e "show global status"
And then in the template the values would all have to change from
- mysql.get_status_variables["{$MYSQL.HOST}","{$MYSQL.PORT}","{$MYSQL.USER}","{$MYSQL.PASSWORD}"] + mysql.get_status_variables["{$MYSQL.HOST}","{$MYSQL.PORT}","{$MYSQL.USER}","{$MYSQL.PASSWORD}", {$MYSQL.VERIFY_SERVER_CERT}]
Though it won't solve the full problem, just this particular manifestation.
Some thoughts on the more general variant of the issue:
A general zabbix feature to send stderr to a different output or create a standard feature to send a trigger on receiving output on stderr in a userParameter (probably configurable) rather than collecting it into the standard result would allow for a much easier time separating data and errors. Once the data arrives at the zabbix server, it has no clue which part of it is from stdout, and which part is from stderr.
This information is destroyed by the zabbix client. For example, a feature that would accept this config:
ErrorParameter=mysql.version_error, userParameter=mysql.version
and define 'version_error' in the zabbix-server as a 'co-dependent' parameter of 'version', would then allow or require the client to always send version_error when it is sending version. Collect stderr and stdout in two different 'buckets' and output them in different zabbix items.
Extending this idea, a C-style return code (the third and final, somewhat old-fashioned, way programs report success or failure; where any non-zero value means error) could go in a third bucket.
If there's some kind of warning or error while the userparameter is being determined, a trigger on any truthy version_error or version_returncode value on the zabbix server side will catch any issues without trashing the data. After all, receiving anything on stdErr at all or any non-zero return code is an abnormal condition.
You could even do better and filter between the warnings and the errors at the zabbix server side by looking for output that begins with 'WARNING', though that's likely program specific and output language specific, so possibly better left to a template.
Technically you could already do this: define the same parameter twice, and on the value one, add 2>/dev/null, while on the error one you add 1>/dev/null. The main issue with that is that you are running the same code twice and doing double work. That might not be a problem until performance matters.
Second part of solution, incorporating this idea:
Apply the specific fix, then double all the parameters everywhere, creating a pattern like so:
UserParameter=mysql.get_status_variables[*], export MYSQL_PWD="$4"; mysql -h"$1" -P"$2" -u"$3" --ssl-verify-server-cert="$5" -sNX -e "show global status" 2>/dev/null UserParameter=mysql.get_status_variables_error[*], export MYSQL_PWD="$4"; mysql -h"$1" -P"$2" -u"$3" --ssl-verify-server-cert="$5" -sNX -e "show global status" 1>/dev/null
This still doesn't gel very well with me, because it seems redundant. Which is where the general suggestion to add this functionality to the client comes from.