[ZBX-22868] zabbix-agent2 upgrade breaks postgresql plugin monitoring Created: 2023 May 30  Updated: 2024 Apr 10  Resolved: 2023 Jun 20

Status: Closed
Project: ZABBIX BUGS AND ISSUES
Component/s: Agent2 plugin (G)
Affects Version/s: 6.0.18rc1
Fix Version/s: 5.0.36rc1, 6.0.19rc1, 6.4.4rc1, 7.0.0alpha2, 7.0 (plan)

Type: Problem report Priority: Blocker
Reporter: Sandro Bordacchini Assignee: Eriks Sneiders
Resolution: Fixed Votes: 30
Labels: agent, macros
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Attachments: Text File ZBX-22868.log    
Issue Links:
Duplicate
duplicates ZBX-22873 PostgreSQL monitoring is broken after... Closed
is duplicated by ZBX-22873 PostgreSQL monitoring is broken after... Closed
is duplicated by ZBX-22872 PostgreSQL by Zabbix agent 2 for 6.0.... Closed
is duplicated by ZBX-22947 [PostgreSQL] Connection failed: faile... Closed
is duplicated by ZBX-22973 Agent2 PostgreSQL plugin fails to aut... Closed
is duplicated by ZBX-23029 Problem with PostgreSQL by Zabbix age... Closed
is duplicated by ZBX-22870 Zabbix Agent 2 Postgres Template does... Need info
Team: Team INT
Sprint: Sprint 101 (Jun 2023)
Story Points: 1

 Description   

Starting scenario:

  • postgresql instance on debian10
  • zabbix-agent2 1:6.0.17-1+debian11 with same version psql plugin installed
  • created zbx_monitor role on pgsql db with proper permissions
  • configured {$PG.PASSWORD} e {$PG.USER} on zabbix server (v.1:6.0.13-1+debian11) accordingly to previous point
  • no configuration done in /etc/zabbix/zabbix_agent2.d/plugins.d/postgresql.conf
  • everything working fine...

Steps to reproduce:

  1. upgrade zabbix-agent2 (and its plugins) from 1:6.0.17-1+debian11 to 1:6.0.18-1+debian11 with usual apt upgrade command

Result:

It seems monitoring user is not taken from macros anymore... it fails back to default zabbix user and it fails authentication

Console output:
{{}}

Setting up zabbix-agent2-plugin-postgresql (6.0.18-1+debian11) ...
Installing new version of config file /etc/zabbix/zabbix_agent2.d/plugins.d/postgresql.conf ...
Setting up zabbix-agent2-plugin-mongodb (6.0.18-1+debian11) ...
Installing new version of config file /etc/zabbix/zabbix_agent2.d/plugins.d/mongodb.conf ...
Setting up zabbix-agent2 (1:6.0.18-1+debian11) ...
Installing new version of config file /etc/zabbix/zabbix_agent2.d/plugins.d/ceph.conf ...
Installing new version of config file /etc/zabbix/zabbix_agent2.d/plugins.d/memcached.conf ...
Installing new version of config file /etc/zabbix/zabbix_agent2.d/plugins.d/mysql.conf ...
Installing new version of config file /etc/zabbix/zabbix_agent2.d/plugins.d/oracle.conf ...
Installing new version of config file /etc/zabbix/zabbix_agent2.d/plugins.d/redis.conf ...

 

Postgresql log

 

2023-05-30 10:20:33.722 UTC [17672] zabbix@user=zbx_monitor FATAL:  password authentication failed for user "zabbix"
2023-05-30 10:20:33.722 UTC [17672] zabbix@user=zbx_monitor DETAIL:  Role "zabbix" does not exist.
        Connection matched pg_hba.conf line 92: "host    all             all             127.0.0.1/32            md5"
[... keeps repeating ...]
 
zabbix-agent2 log
2023/05/30 12:20:31.382440 Zabbix Agent 2 stopped. (6.0.17)
2023/05/30 12:20:31.473446 Starting Zabbix Agent 2 (6.0.18)
2023/05/30 12:20:31.474151 OpenSSL library (OpenSSL 1.1.1n  15 Mar 2022) initialized
2023/05/30 12:20:31.474233 using configuration file: /etc/zabbix/zabbix_agent2.custom.conf
2023/05/30 12:20:31.474349 using plugin 'Agent' (built-in) providing following interfaces: exporter
[...]
2023/05/30 12:20:31.476317 using plugin 'Modbus' (built-in) providing following interfaces: exporter, configurator
2023/05/30 12:20:31.476338 using plugin 'MongoDB' (/usr/sbin/zabbix-agent2-plugin/zabbix-agent2-plugin-mongodb) providing following interfaces: exporter, runner, configurator
2023/05/30 12:20:31.476352 using plugin 'Mysql' (built-in) providing following interfaces: exporter, runner, configurator
2023/05/30 12:20:31.476363 using plugin 'NetIf' (built-in) providing following interfaces: exporter
2023/05/30 12:20:31.476372 using plugin 'Oracle' (built-in) providing following interfaces: exporter, runner, configurator
2023/05/30 12:20:31.476389 using plugin 'PostgreSQL' (/usr/sbin/zabbix-agent2-plugin/zabbix-agent2-plugin-postgresql) providing following interfaces: exporter, runner, configurator
[...]
2023/05/30 12:20:31.476601 using plugin 'ZabbixStats' (built-in) providing following interfaces: exporter, configurator
2023/05/30 12:20:31.476608 lowering the plugin ZabbixSync capacity to 1 as the configured capacity 100 exceeds limits
2023/05/30 12:20:31.476615 using plugin 'ZabbixSync' (built-in) providing following interfaces: exporter
2023/05/30 12:20:31.478990 Plugin communication protocol version is 6.0.13
2023/05/30 12:20:32.020985 Zabbix Agent2 hostname: [node004.xxxxxxxxx.net]
2023/05/30 12:20:33.722582 [PostgreSQL] Connection failed: failed to connect to `host=localhost user=zabbix database=user=zbx_monitor`: server error (FATAL: password authentication failed for user "zabbix" (SQLSTATE 28P01)).
[... last line keeps repeating ...]

 

Expected:
No change.



 Comments   
Comment by Andrew Martin [ 2023 May 30 ]

I see this on 5.0.35-1 on Ubuntu too

Comment by Alessandro Duvia [ 2023 May 30 ]

Same problem here. Our logs outputs are identical to yours.

It seems like the PG.USER macro is in the database field....

Zabbix-agent2 1:6.0.18-1+debian10 amd64

Comment by Sandro Bordacchini [ 2023 May 30 ]

Ok, I tried to fix the authentication part (dropping the role zbx_monitor and creating a zabbix one) but, as it could be expected (reading accurately the logs above) now the plugin query a evidently non-existent db "user=zabbix".

Please see the attached postgres and zabbix-agent2 (level 3) debug log ZBX-22868.log.

Maybe the query's parameters are not passed correctly?

PS: <zabbix_server_ip_address>, <pgsql_password> replace the actual values for privacyZBX-22868.log

Comment by Sandro Bordacchini [ 2023 May 30 ]

I think that the only relevant work done between 6.0.17 and 6.0.18 is https://support.zabbix.com/browse/ZBXNEXT-8330

But I don't know how to relate that to this issue...

Comment by Lasse Osterild [ 2023 May 31 ]

Same issue on 6.4.3 - server and agent was installed with 6.4.2 where it worked fine, then after todays upgrade to 6.4.3 it breaks.

Seemingly something to do with  ZBXNEXT-8330 where defaults are now applied instead of macros, how the database ends up being "user=zbx_monitor" I am not sure.

Connection failed: failed to connect to `host=localhost user=zabbix database=user=zbx_monitor

Comment by Robbert Vercammen [ 2023 Jun 01 ]

Full zabbix-agent2 log line:

2023/06/01 12:00:53.816001 [PostgreSQL] Connection failed: failed to connect to `host=localhost user=zabbix database=user=nagios`: failed SASL auth (FATAL: password authentication failed for user "zabbix" (SQLSTATE 28P01)).

We use a nagios user (as set in the template macro) to connect to the PostgreSQL database.

Comment by Jonas Nilsson [ 2023 Jun 01 ]

Updating the postgres plugin but holding the zabbix agent on 6.4.2 produces the same results.

Comment by Tomasz Berner [ 2023 Jun 02 ]

Temporary solution is to modify plugin config:

Plugins.PostgreSQL.Default.Uri=tcp://localhost:5432
Plugins.PostgreSQL.Default.User=XXXXX
Plugins.PostgreSQL.Default.Password=YYYYY
Plugins.PostgreSQL.Default.Database=postgres

and set empty macros:

{$PG.URI}
{$PG.USER}
{$PG.PASSWORD}

 

Comment by Bruno Hostalery [ 2023 Jun 03 ]

Server on debian 11, zabbix server 6.4.3

Client on debian 11,ztabbix-agent2 and zabbix-agent2-plugin-postgresql.

Works with macro in server due to need of monitoring multiple instance on same machine.

Previous release: 6.4.2-1+debian11

Update to: 6.4.3-1+debian11

Break the monitoring with log:

  1. request received

created direct exporter task for plugin 'PostgreSQL' itemid:0 key 'pgsql.dbstat.sum["tcp://127.0.0.1:5432","USERNAME","PASSWORD"]'

created direct exporter task for plugin 'PostgreSQL' itemid:0 key 'pgsql.dbstat.sum["tcp://127.0.0.1:5432","USERNAME","PASSWORD"]'

  1. error on

failed to execute direct exporter task for key 'pgsql.dbstat.sum["tcp://127.0.0.1:5432","USERNAME","PASSWORD"]' error: '
Connection failed: failed to connect to `host=127.0.0.1 user=zabbix database=user=USERNAME`

Looks like plugin find no user and use user=zabbix, and use for db user=USERNAME

Back to version 6.4.2-1+debian11. Everything works fine again.

Comment by Evgeny Yurchenko [ 2023 Jun 04 ]

Experiencing the same.

Looking at PostgreSQL logs:

2023-06-03 22:48:44.669 UTC [52466] zabbix@user=zbx_monitor FATAL:  password authentication failed for user "zabbix"
2023-06-03 22:48:44.669 UTC [52466] zabbix@user=zbx_monitor DETAIL:  Connection matched pg_hba.conf line 101: "host    all             all             127.0.0.1/32            scram-sha-256" 

we can say that plugin suddenly uses user "zabbix" to connect to "user=zbx_monitor" database.

Comment by John Wayne [ 2023 Jun 06 ]

I really wonder how this one made it to the final 6.0.18 release as the problem was already reported with 6.0.18rc1?

At least the testing workflow may be improved for future releases?

Correct me if I'm wrong but the only "real" solution for now is to downgrade to 6.0.17 and wait for a fixed (and hopefully better tested) 6.0.19.

Comment by Sandro Bordacchini [ 2023 Jun 06 ]

Hello @JohnWayne , please note that i reported the bug on 6.0.18rc1 but i have discovered it when upgrading to 6.0.18 via deb packaging.

So the bug report has been made after the release of 6.0.18. I cannot recall now why I wrote 6.0.18rc1, probably that was the closest option available in the combo box .

 

@Tomasz Berner proposed a temporary workaround that i did not try yet, though.

Comment by Andris Zeila [ 2023 Jun 09 ]

As a temporary workaround adding Plugins.PostgreSQL.Default.Database=postgres to agent configuration file should be enough.

Comment by Andres Tarallo [ 2023 Jun 09 ]

We were working fine on 6.0.17, the upgrade broke posgres monitoring. We just installed the latest patch and still get errors.

 

failed: failed to connect to host=localhost user=zabbix database=user=zbx_monitor: failed SASL auth (FATAL: password authentication failed for user "zabbix" (SQLSTATE 28P01)).' to 'X.X.X.X'

Comment by Maksim Astapenko [ 2023 Jun 11 ]

On RHEL latest work zabbix-agent2 and PostgreSQL plugin 6.4.0 version.

Comment by Eriks Sneiders [ 2023 Jun 15 ]

Fixed in:

Generated at Wed Apr 30 06:47:17 EEST 2025 using Jira 9.12.4#9120004-sha1:625303b708afdb767e17cb2838290c41888e9ff0.