[ZBX-22948] New Hosts do not work until zabbix-server is restarted Created: 2023 Jun 09  Updated: 2024 Apr 10  Resolved: 2023 Aug 29

Status: Closed
Project: ZABBIX BUGS AND ISSUES
Component/s: Proxy (P), Server (S)
Affects Version/s: 6.4.3
Fix Version/s: None

Type: Problem report Priority: Blocker
Reporter: Andrew Landsverk Assignee: Edgar Akhmetshin
Resolution: Won't fix Votes: 1
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified
Environment:

Rocky Linux 9.1
Zabbix 6.4.3

1 App VM - 2 CPU, 4GB RAM
1 Proxy VM - 1 CPU, 4GB RAM - Active Proxy
2 Web VM - 2 CPU, 4GB RAM
3 DB VM - PostgreSQL 15, TimescaleDB 2.11

Issue Links:
Causes
Duplicate
is duplicated by ZBX-23414 Some hosts and items can not be found... Closed
is duplicated by ZBX-23866 zabbix search not work ,search host n... Closed
is duplicated by ZBX-23879 cannot send list of active checks to ... Closed
Team: Team A
Sprint: Support backlog

 Description   

Background

We recently migrated from MySQL 8.x to PostgreSQL 15 + TimescaleDB 2.11, everything appears to have gone as planned with the migration. We also upgraded from Zabbix 6.2.9 to Zabbix 6.4.3. I have tried with HA cluster enabled as well and the problem exists under HA and standalone.

Steps to reproduce:

  1. Create a new host via Zabbix ansible collection using Zabbix API
  2. Observe that new host is visible in Zabbix UI
  3. Watch proxy log to make sure configuration data was received from the server, datalen is very small
  4. Observe errors in the logs on the proxy about not being able to find the PSK
  5. Log in to the sqlite3 db on the proxy, query hosts table and observe that the new host is not present
  6. Restart zabbix-server process
  7. Watch proxy log to make sure new configuration is updated, datalen is very big
  8. Log in to sqlite3 db on the proxy, query hosts table, observe new host is now present

Expected:
Expected that new hosts will be pushed to proxies without having to restart zabbix-server

 

Server Configuration

AlertScriptsPath=/usr/lib/zabbix/alertscripts
AllowRoot=0
AllowUnsupportedDBVersions=0
CacheSize=128M
CacheUpdateFrequency=60
DBHost=192.168.4.131
DBName=zabbix
DBPassword=<password>
DBPort=5000
DBUser=zabbix
DebugLevel=3
ExportFileSize=1G
ExternalScripts=/usr/lib/zabbix/externalscripts
Fping6Location=/usr/sbin/fping6
FpingLocation=/usr/sbin/fping
HistoryStorageDateIndex=0
HistoryStorageTypes=uint,dbl,str,log,text
HousekeepingFrequency=1
Include=/etc/zabbix/zabbix_server.conf.d
JavaGatewayPort=10052
ListenPort=10051
LoadModulePath=${libdir}/modules
LogFile=/var/log/zabbix/zabbix_server.log
LogFileSize=10
LogSlowQueries=0
LogType=file
MaxHousekeeperDelete=500
PidFile=/var/run/zabbix/zabbix_server.pid
ProxyConfigFrequency=300
ProxyDataFrequency=1
SNMPTrapperFile=/tmp/zabbix_traps.tmp
SocketDir=/var/run/zabbix
SSLCertLocation=${datadir}/zabbix/ssl/certs
SSLKeyLocation=${datadir}/zabbix/ssl/keys
StartDBSyncers=4
StartDiscoverers=1
StartEscalators=1
StartHistoryPollers=5
StartHTTPPollers=1
StartIPMIPollers=0
StartJavaPollers=0
StartLLDProcessors=2
StartODBCPollers=1
StartPingers=1
StartPollers=5
StartPollersUnreachable=1
StartPreprocessors=3
StartProxyPollers=1
StartReportWriters=0
StartSNMPTrapper=0
StartTimers=1
StartTrappers=5
StartVMwareCollectors=0
Timeout=3
TmpDir=/tmp
TrapperTimeout=300
UnavailableDelay=60
UnreachableDelay=15
UnreachablePeriod=45
User=zabbix
ValueCacheSize=128M
VaultURL=https://127.0.0.1:8200
VMwareFrequency=60
VMwarePerfFrequency=60
VMwareTimeout=10

Proxy Configuration

AllowRoot=0
AllowUnsupportedDBVersions=0
CacheSize=128M
ConfigFrequency=60
DataSenderFrequency=1
DBHost=localhost
DBName=/var/lib/zabbix/zabbix_proxy.db
DBPassword=zabbix_proxy
DBUser=zabbix_proxy
DebugLevel=3
EnableRemoteCommands=1
ExternalScripts=/usr/lib/zabbix/externalscripts
Fping6Location=/usr/sbin/fping6
FpingLocation=/usr/sbin/fping
HistoryCacheSize=8M
HistoryIndexCacheSize=4M
Hostname=dev-zabbix-proxy-1.dev.internal
HousekeepingFrequency=1
Include=/etc/zabbix/zabbix_proxy.conf.d
JavaGatewayPort=10052
ListenPort=10051
LoadModulePath=/usr/lib/zabbix/modules
LogFile=/var/log/zabbix/zabbix_proxy.log
LogFileSize=10
LogRemoteCommands=1
LogType=file
PidFile=/var/run/zabbix/zabbix_proxy.pid
ProxyLocalBuffer=0
ProxyOfflineBuffer=96
Server=192.168.5.13
SNMPTrapperFile=/var/lib/zabbix/snmptraps/snmptraps.log
SocketDir=/var/run/zabbix
StartDBSyncers=4
StartDiscoverers=1
StartHTTPPollers=1
StartIPMIPollers=1
StartODBCPollers=1
StartPingers=1
StartPollers=20
StartPollersUnreachable=3
StartPreprocessors=3
StartSNMPTrapper=1
StartTrappers=10
StatsAllowedIP=127.0.0.1
Timeout=10
TLSConnect=psk
TLSPSKFile=/etc/zabbix/tls_psk_proxy.secret
TLSPSKIdentity=PSK : dev-zabbix-proxy-1
TmpDir=/tmp
TrapperTimeout=300
UnreachablePeriod=45
VaultURL=https://127.0.0.1:8200
VMwareCacheSize=8M
VMwareFrequency=60

Possible related ticket: ZBX-22657 

 Comments   
Comment by Vladislavs Sokurenko [ 2023 Jun 28 ]

Is it problem only on Zabbix proxy or also with host on Zabbix server ?
Please show output of:

SELECT
    trigger_schema,
    trigger_name,
    event_manipulation,
    action_statement
FROM information_schema.triggers;

Please also check if changelog is being updated when new host is added:

select * from changelog order by changelogid desc limit 10;
Comment by Vladislavs Sokurenko [ 2023 Jun 28 ]

It's also best to compare if schema matches vanilla one

Comment by Andrew Landsverk [ 2023 Jun 28 ]

Here is the output of the triggers query:

 

"trigger_schema"    "trigger_name"    "event_manipulation"    "action_statement"
"public"    "ts_insert_blocker"    "INSERT"    "EXECUTE FUNCTION _timescaledb_internal.insert_blocker()"
"public"    "ts_insert_blocker"    "INSERT"    "EXECUTE FUNCTION _timescaledb_internal.insert_blocker()"
"public"    "ts_insert_blocker"    "INSERT"    "EXECUTE FUNCTION _timescaledb_internal.insert_blocker()"
"public"    "ts_insert_blocker"    "INSERT"    "EXECUTE FUNCTION _timescaledb_internal.insert_blocker()"
"public"    "ts_insert_blocker"    "INSERT"    "EXECUTE FUNCTION _timescaledb_internal.insert_blocker()"
"public"    "ts_insert_blocker"    "INSERT"    "EXECUTE FUNCTION _timescaledb_internal.insert_blocker()"
"public"    "ts_insert_blocker"    "INSERT"    "EXECUTE FUNCTION _timescaledb_internal.insert_blocker()"
"_timescaledb_internal"    "ts_insert_blocker"    "INSERT"    "EXECUTE FUNCTION _timescaledb_internal.insert_blocker()"
"_timescaledb_internal"    "ts_insert_blocker"    "INSERT"    "EXECUTE FUNCTION _timescaledb_internal.insert_blocker()"
"_timescaledb_internal"    "ts_insert_blocker"    "INSERT"    "EXECUTE FUNCTION _timescaledb_internal.insert_blocker()"
"_timescaledb_internal"    "ts_insert_blocker"    "INSERT"    "EXECUTE FUNCTION _timescaledb_internal.insert_blocker()"
"_timescaledb_internal"    "ts_insert_blocker"    "INSERT"    "EXECUTE FUNCTION _timescaledb_internal.insert_blocker()"
"_timescaledb_internal"    "ts_insert_blocker"    "INSERT"    "EXECUTE FUNCTION _timescaledb_internal.insert_blocker()"
"_timescaledb_internal"    "ts_insert_blocker"    "INSERT"    "EXECUTE FUNCTION _timescaledb_internal.insert_blocker()"
"public"    "drules_insert"    "INSERT"    "EXECUTE FUNCTION changelog_drules_insert()"
"public"    "drules_update"    "UPDATE"    "EXECUTE FUNCTION changelog_drules_update()"
"public"    "drules_delete"    "DELETE"    "EXECUTE FUNCTION changelog_drules_delete()"
"public"    "dchecks_insert"    "INSERT"    "EXECUTE FUNCTION changelog_dchecks_insert()"
"public"    "dchecks_update"    "UPDATE"    "EXECUTE FUNCTION changelog_dchecks_update()"
"public"    "dchecks_delete"    "DELETE"    "EXECUTE FUNCTION changelog_dchecks_delete()"
"public"    "httptest_insert"    "INSERT"    "EXECUTE FUNCTION changelog_httptest_insert()"
"public"    "httptest_update"    "UPDATE"    "EXECUTE FUNCTION changelog_httptest_update()"
"public"    "httptest_delete"    "DELETE"    "EXECUTE FUNCTION changelog_httptest_delete()"
"public"    "httptest_field_insert"    "INSERT"    "EXECUTE FUNCTION changelog_httptest_field_insert()"
"public"    "httptest_field_update"    "UPDATE"    "EXECUTE FUNCTION changelog_httptest_field_update()"
"public"    "httptest_field_delete"    "DELETE"    "EXECUTE FUNCTION changelog_httptest_field_delete()"
"public"    "httptestitem_insert"    "INSERT"    "EXECUTE FUNCTION changelog_httptestitem_insert()"
"public"    "httptestitem_update"    "UPDATE"    "EXECUTE FUNCTION changelog_httptestitem_update()"
"public"    "httptestitem_delete"    "DELETE"    "EXECUTE FUNCTION changelog_httptestitem_delete()"
"public"    "httpstep_insert"    "INSERT"    "EXECUTE FUNCTION changelog_httpstep_insert()"
"public"    "httpstep_update"    "UPDATE"    "EXECUTE FUNCTION changelog_httpstep_update()"
"public"    "httpstep_delete"    "DELETE"    "EXECUTE FUNCTION changelog_httpstep_delete()"
"public"    "httpstep_field_insert"    "INSERT"    "EXECUTE FUNCTION changelog_httpstep_field_insert()"
"public"    "httpstep_field_update"    "UPDATE"    "EXECUTE FUNCTION changelog_httpstep_field_update()"
"public"    "httpstep_field_delete"    "DELETE"    "EXECUTE FUNCTION changelog_httpstep_field_delete()"
"public"    "httpstepitem_insert"    "INSERT"    "EXECUTE FUNCTION changelog_httpstepitem_insert()"
"public"    "httpstepitem_update"    "UPDATE"    "EXECUTE FUNCTION changelog_httpstepitem_update()"
"public"    "httpstepitem_delete"    "DELETE"    "EXECUTE FUNCTION changelog_httpstepitem_delete()"
"public"    "hosts_name_upper_insert"    "INSERT"    "EXECUTE FUNCTION hosts_name_upper_upper()"
"public"    "hosts_name_upper_update"    "UPDATE"    "EXECUTE FUNCTION hosts_name_upper_upper()"
"public"    "items_name_upper_insert"    "INSERT"    "EXECUTE FUNCTION items_name_upper_upper()"
"public"    "items_name_upper_update"    "UPDATE"    "EXECUTE FUNCTION items_name_upper_upper()"
"public"    "connector_insert"    "INSERT"    "EXECUTE FUNCTION changelog_connector_insert()"
"public"    "connector_update"    "UPDATE"    "EXECUTE FUNCTION changelog_connector_update()"
"public"    "connector_delete"    "DELETE"    "EXECUTE FUNCTION changelog_connector_delete()"
"public"    "connector_tag_insert"    "INSERT"    "EXECUTE FUNCTION changelog_connector_tag_insert()"
"public"    "connector_tag_update"    "UPDATE"    "EXECUTE FUNCTION changelog_connector_tag_update()"
"public"    "connector_tag_delete"    "DELETE"    "EXECUTE FUNCTION changelog_connector_tag_delete()"

The changelog query returns no results.
 
We've discovered that a bunch of changelog insert triggers and others are missing. I took the triggers out of server.sql.gz and ran them against our dev system. It appears that changelog entries are being created now. I will do some more testing today and report back.
 
Thank you very much for your help on this!
 

 

 

Comment by Andrew Landsverk [ 2023 Jun 28 ]

After testing new hosts with the triggers added, everything appears to be working as it should now and we no longer need to restart zabbix-server. I just wanted to say thank you again for the help narrowing this down!  

Generated at Wed Apr 30 06:42:02 EEST 2025 using Jira 9.12.4#9120004-sha1:625303b708afdb767e17cb2838290c41888e9ff0.