[ZBX-20172] Zabbix server keep sending duplicates erroneous configurations to proxies Created: 2021 Nov 03  Updated: 2021 Nov 07  Resolved: 2021 Nov 07

Status: Closed
Project: ZABBIX BUGS AND ISSUES
Component/s: Server (S)
Affects Version/s: 5.0.16
Fix Version/s: None

Type: Incident report Priority: Trivial
Reporter: Guillaume Fenollar Assignee: Alexey Pustovalov
Resolution: Won't fix Votes: 0
Labels: database
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified
Environment:

CentOS 8.1.1911
Kernel Linux 4.18.0-147.5.1
Postgresql 13.4
TimescaleDB 2 enabled
Zabbix proxies using SQLite
200 hosts / 12k items

Attachments: HTML File log    

 Description   

{}For more than a week now, my server instance is sending duplicate records of configuration (every ProxyConfigFrequency delay) to active and passive proxies, which can't update their configuration anymore. 

Proxy Logs

7763:20211025:102610.189 received configuration data from server at "192.168.128.14", datalen 8532
7763:20211025:102610.190 [Z3005] query failed: [0] UNIQUE constraint failed: globalmacro.macro [insert into globalmacro (globalmacroid,macro,value) values (2,'{$SNMP_COMMUNITY}','public');
insert into globalmacro (globalmacroid,macro,value) values (2,'{$SNMP_COMMUNITY}','public');
insert into globalmacro (globalmacroid,macro,value) values (3,'{$IOB_NET_INT_TRUNK}','^[Tt][Rr][Uu][Nn][Kk].*');
insert into globalmacro (globalmacroid,macro,value) values (3,'{$IOB_NET_INT_TRUNK}','^[Tt][Rr][Uu][Nn][Kk].*');
insert into globalmacro (globalmacroid,macro,value) values (4,'{$IOB_NET_INT_PO}','^Interface [Pp][Oo][0-9]$');
insert into globalmacro (globalmacroid,macro,value) values (4,'{$IOB_NET_INT_PO}','^Interface [Pp][Oo][0-9]$');
insert into globalmacro (globalmacroid,macro,value) values (5,'{$IOB_NET_INT_DUPLEX}','.*Duplex status.*');
insert into globalmacro (globalmacroid,macro,value) values (5,'{$IOB_NET_INT_DUPLEX}','.*Duplex status.*');
insert into globalmacro (globalmacroid,macro,value) values (6,'{$IOB_NET_INT_PO_CH}','^[Pp][Oo][Rr][Tt]-[Cc][Hh][Aa][Nn][Nn][Ee][Ll][0-9.]');
insert into globalmacro (globalmacroid,macro,value) values (6,'{$IOB_NET_INT_PO_CH}','^[Pp][Oo][Rr][Tt]-[Cc][Hh][Aa][Nn][Nn][Ee][Ll][0-9.]');
insert into globalmacro (globalmacroid,macro,value) values (7,'{$IOB_NET_INT_GI}','[Gg][Ii][0-9.].*');
insert into globalmacro (globalmacroid,macro,value) values (7,'{$IOB_NET_INT_GI}','[Gg][Ii][0-9.].*');
insert into globalmacro (globalmacroid,macro,value) values (8,'{$IOB_NET_INT_SUP}','^[Ss][Uu][Pp].*$');
insert into globalmacro (globalmacroid,macro,value) values (8,'{$IOB_NET_INT_SUP}','^[Ss][Uu][Pp].*$');
insert into globalmacro (globalmacroid,macro,value) values (9,'{$IF.ERRORS.WARN}','30');
insert into globalmacro (globalmacroid,macro,value) values (9,'{$IF.ERRORS.WARN}','30');
insert into globalmacro (globalmacroid,macro,value) values (10,'{$IF.UTIL.MAX}','50');
insert into globalmacro (globalmacroid,macro,value) values (10,'{$IF.UTIL.MAX}','50');
insert into globalmacro (globalmacroid,macro,value) values (12,'{$IOB_NET_INT_GIGA}','^GigabitEthernet[0-9.].*');
insert into globalmacro (globalmacroid,macro,value) values (12,'{$IOB_NET_INT_GIGA}','^GigabitEthernet[0-9.].*');
insert into globalmacro (globalmacroid,macro,value) values (13,'{$IOB_NET_INT_TUNNEL}','^Tunnel[0-9.].*');
insert into globalmacro (globalmacroid,macro,value) values (13,'{$IOB_NET_INT_TUNNEL}','^Tunnel[0-9.].*');
insert into globalmacro (globalmacroid,macro,value) values (14,'{$IF.UTIL.DUREE.MAX}','1m');
insert into globalmacro (globalmacroid,macro,value) values (14,'{$IF.UTIL.DUREE.MAX}','1m');
insert into globalmacro (globalmacroid,macro,value) values (15,'{$IF.UTIL.MAX.RECOVERY}','2');
insert into globalmacro (globalmacroid,macro,value) values (15,'{$IF.UTIL.MAX.RECOVERY}','2');
insert into globalmacro (globalmacroid,macro,value) values (16,'{$IF.ERRORS.DUREE.WARN}','5m');
insert into globalmacro (globalmacroid,macro,value) values (16,'{$IF.ERRORS.DUREE.WARN}','5m');
insert into globalmacro (globalmacroid,macro,value) values (17,'{$IF.ERRORS.WARN.RECOVERY}','5');
insert into globalmacro (globalmacroid,macro,value) values (17,'{$IF.ERRORS.WARN.RECOVERY}','5');
insert into globalmacro (globalmacroid,macro,value) values (18,'{$IOB_NET_INT_EMBED}','^[Ee]mbedded-[Ss]ervice-[Ee]ngine.*');
insert into globalmacro (globalmacroid,macro,value) values (18,'{$IOB_NET_INT_EMBED}','^[Ee]mbedded-[Ss]ervice-[Ee]ngine.*');
insert into globalmacro (globalmacroid,macro,value) values (19,'{$IOB_NET_INT_NULL}','^[Nn]ull.*');
insert into globalmacro (globalmacroid,macro,value) values (19,'{$IOB_NET_INT_NULL}','^[Nn]ull.*');
insert into globalmacro (globalmacroid,macro,value) values (20,'{$IOB_NET_INT_NU}','^[Nn][Uu].*');
insert into globalmacro (globalmacroid,macro,value) values (20,'{$IOB_NET_INT_NU}','^[Nn][Uu].*');
insert into globalmacro (globalmacroid,macro,value) values (21,'{$IOB_NET_INT_EM}','^[Ee][Mm].*');
insert into globalmacro (globalmacroid,macro,value) values (21,'{$IOB_NET_INT_EM}','^[Ee][Mm].*');
insert into globalmacro (globalmacroid,macro,value) values (23,'{$IOB_NET_INT_NV}','^[Nn][Vv].*');
insert into globalmacro (globalmacroid,macro,value) values (23,'{$IOB_NET_INT_NV}','^[Nn][Vv].*');
insert into globalmacro (globalmacroid,macro,value) values (24,'{$IOB_NET_INT_VIR}','(24|131|135)');
insert into globalmacro (globalmacroid,macro,value) values (24,'{$IOB_NET_INT_VIR}','(24|131|135)');
insert into globalmacro (globalmacroid,macro,value) values (26,'{$IOB_NET_INT_TYPE_1}','(1)');
insert into globalmacro (globalmacroid,macro,value) values (26,'{$IOB_NET_INT_TYPE_1}','(1)');
insert into globalmacro (globalmacroid,macro,value) values (27,'{$IOB_NET_INT_BRIO}','^[Bb][Rr][Ii][Oo].*');
insert into globalmacro (globalmacroid,macro,value) values (27,'{$IOB_NET_INT_BRIO}','^[Bb][Rr][Ii][Oo].*');
insert into globalmacro (globalmacroid,macro,value) values (28,'{$IOB_NET_INT_BRIO_IFNAME}','^[Bb][Rr][0-9].*');
insert into globalmacro (globalmacroid,macro,value) values (28,'{$IOB_NET_INT_BRIO_IFNAME}','^[Bb][Rr][0-9].*');
insert into globalmacro (globalmacroid,macro,value) values (30,'{$CPU.UTIL.CRIT}','99');
insert into globalmacro (globalmacroid,macro,value) values (30,'{$CPU.UTIL.CRIT}','99');
insert into globalmacro (globalmacroid,macro,value) values (31,'{$MEMORY.UTIL.MAX}','95');
insert into globalmacro (globalmacroid,macro,value) values (31,'{$MEMORY.UTIL.MAX}','95');
insert into globalmacro (globalmacroid,macro,value) values (32,'{$SWAP.FREE.ALERT}','30');
insert into globalmacro (globalmacroid,macro,value) values (32,'{$SWAP.FREE.ALERT}','30');
]
7763:20211025:102610.190 failed to update local proxy configuration copy: database error

Because of these errors, altought proxies continue to work with old items, they can't update their configuration, so the infrastructure is like "frozen" at the moment.

Server logs

On the server side, here's the debug log :

70754:20211027:095400.211 zbx_setproctitle() title:'proxy poller #1 [exchanged data with 2 proxies in 0.009213 sec, exchanging data]'
70754:20211027:095400.211 In process_proxy()
70754:20211027:095400.211 In DCconfig_get_proxypoller_hosts()
70754:20211027:095400.211 End of DCconfig_get_proxypoller_hosts():1
70754:20211027:095400.211 In substitute_simple_macros_impl() data:'10051'
70754:20211027:095400.211 In get_proxyconfig_data() proxy_hostid:10680
70754:20211027:095400.211 query [txnlev:1] [begin;]
70754:20211027:095400.211 query [txnlev:1] [select hostid from hosts where proxy_hostid=10680 and status in (0,1) and flags<>2]
70754:20211027:095400.211 query [txnlev:1] [select httptestid from httptest t,hosts h where t.hostid=h.hostid and t.status=0 and h.proxy_hostid=10680 and h.status=0]
70754:20211027:095400.211 In get_proxyconfig_table() proxy_hostid:10680 table:'globalmacro'
70754:20211027:095400.211 query [txnlev:1] [select t.globalmacroid,t.macro,t.value from globalmacro t order by t.globalmacroid]
70754:20211027:095400.212 End of get_proxyconfig_table():SUCCEED
70754:20211027:095400.212 In get_proxyconfig_table() proxy_hostid:10680 table:'hosts'
70754:20211027:095400.212 End of get_proxyconfig_table():SUCCEED
70754:20211027:095400.212 In get_proxyconfig_table() proxy_hostid:10680 table:'interface'
70754:20211027:095400.212 End of get_proxyconfig_table():SUCCEED
70754:20211027:095400.212 In get_proxyconfig_table() proxy_hostid:10680 table:'interface_snmp'
70754:20211027:095400.212 End of get_proxyconfig_table():SUCCEED
70754:20211027:095400.212 In get_proxyconfig_table() proxy_hostid:10680 table:'hosts_templates'
70754:20211027:095400.212 End of get_proxyconfig_table():SUCCEED
70754:20211027:095400.212 In get_proxyconfig_table() proxy_hostid:10680 table:'hostmacro'
70754:20211027:095400.212 End of get_proxyconfig_table():SUCCEED
70754:20211027:095400.212 In get_proxyconfig_table_items() proxy_hostid:10680
70754:20211027:095400.212 query [txnlev:1] [select t.itemid,t.type,t.snmp_oid,t.hostid,t.key_,t.delay ,t.status,t.value_type,t.trapper_hosts,t.logtimefm t,t.params,t.ipmi_sensor,t.authtype,t.username,t.p asswo
rd,t.publickey,t.privatekey,t.flags,t.interfaceid, t.jmx_endpoint,t.master_itemid,t.timeout,t.url,t.q uery_fields,t.posts,t.status_codes,t.follow_redire cts,t.post_type,t.http_proxy,t.headers,t.retrieve_ mode,t.
request_method,t.output_format,t.ssl_cert_file,t.s sl_key_file,t.ssl_key_password,t.verify_peer,t.ver ify_host,t.allow_traps from items t,hosts r where t.hostid=r.hostid and r.proxy_hostid=10680 and r.status in (0,1) and t.flags<>2 and t.type in (0,7,20,12,2,3,9,10,11,13,14,16,17,5,19,18) order by t.itemid]
70755:20211027:095400.212 zbx_setproctitle() title:'self-monitoring [processing data]'
70755:20211027:095400.212 In collect_selfmon_stats()
70755:20211027:095400.212 End of collect_selfmon_stats()
70755:20211027:095400.212 zbx_setproctitle() title:'self-monitoring [processed data in 0.000044 sec, idle 1 sec]'
70754:20211027:095400.212 End of get_proxyconfig_table_items():SUCCEED
70754:20211027:095400.212 In get_proxyconfig_table() proxy_hostid:10680 table:'drules'
70754:20211027:095400.212 query [txnlev:1] [select t.druleid,t.name,t.iprange,t.delay from drules t where t.proxy_hostid=10680 and t.status=0 order by t.druleid]
70754:20211027:095400.212 End of get_proxyconfig_table():SUCCEED
70754:20211027:095400.213 In get_proxyconfig_table() proxy_hostid:10680 table:'dchecks'
70754:20211027:095400.213 query [txnlev:1] [select t.dcheckid,t.druleid,t.type,t.key_,t.snmp_communit y,t.ports,t.snmpv3_securityname,t.snmpv3_securityl evel,t.snmpv3_authpassphrase,t.snmpv3_privpassphra se,t.uniq,t.snmpv3_authprotocol,t.snmpv3_privproto col,t.snmpv3_contextname,t.host_source,t.name_sour ce from dchecks t,drules r where t.druleid=r.druleid and r.proxy_hostid=10680 and r.status=0 order by t.dcheckid]
70754:20211027:095400.213 End of get_proxyconfig_table():SUCCEED
70754:20211027:095400.213 In get_proxyconfig_table() proxy_hostid:10680 table:'regexps'
70754:20211027:095400.213 query [txnlev:1] [select t.regexpid,t.name from regexps t order by t.regexpid]
70754:20211027:095400.213 End of get_proxyconfig_table():SUCCEED
70754:20211027:095400.213 In get_proxyconfig_table() proxy_hostid:10680 table:'expressions'
70754:20211027:095400.213 query [txnlev:1] [select t.expressionid,t.regexpid,t.expression,t.expressio n_type,t.exp_delimiter,t.case_sensitive from expressions t order by t.expressionid]
70754:20211027:095400.213 End of get_proxyconfig_table():SUCCEED
70754:20211027:095400.213 In get_proxyconfig_table() proxy_hostid:10680 table:'hstgrp'
70754:20211027:095400.213 query [txnlev:1] [select t.groupid from hstgrp t,config r where t.groupid=r.discovery_groupid order by t.groupid]
70754:20211027:095400.213 End of get_proxyconfig_table():SUCCEED
70754:20211027:095400.213 In get_proxyconfig_table() proxy_hostid:10680 table:'config'
70754:20211027:095400.213 query [txnlev:1] [select t.configid,t.refresh_unsupported,t.discovery_group id,t.snmptrap_logging,t.autoreg_tls_accept from config t order by t.configid]
70754:20211027:095400.213 End of get_proxyconfig_table():SUCCEED
70754:20211027:095400.213 In get_proxyconfig_table() proxy_hostid:10680 table:'httptest'
70754:20211027:095400.213 End of get_proxyconfig_table():SUCCEED
70754:20211027:095400.213 In get_proxyconfig_table() proxy_hostid:10680 table:'httptestitem'
70754:20211027:095400.213 End of get_proxyconfig_table():SUCCEED
70754:20211027:095400.213 In get_proxyconfig_table() proxy_hostid:10680 table:'httptest_field'
70754:20211027:095400.213 End of get_proxyconfig_table():SUCCEED
70754:20211027:095400.214 In get_proxyconfig_table() proxy_hostid:10680 table:'httpstep'
70754:20211027:095400.214 End of get_proxyconfig_table():SUCCEED
70754:20211027:095400.214 In get_proxyconfig_table() proxy_hostid:10680 table:'httpstepitem'
70754:20211027:095400.214 End of get_proxyconfig_table():SUCCEED
70754:20211027:095400.214 In get_proxyconfig_table() proxy_hostid:10680 table:'httpstep_field'
70754:20211027:095400.214 End of get_proxyconfig_table():SUCCEED
70754:20211027:095400.214 In get_proxyconfig_table() proxy_hostid:10680 table:'config_autoreg_tls'
70754:20211027:095400.214 query [txnlev:1] [select t.autoreg_tlsid,t.tls_psk_identity,t.tls_psk from config_autoreg_tls t order by t.autoreg_tlsid]
70754:20211027:095400.214 End of get_proxyconfig_table():SUCCEED
70754:20211027:095400.214 query [txnlev:1] [commit;]
70754:20211027:095400.214 End of get_proxyconfig_data():SUCCEED
70754:20211027:095400.214 In connect_to_proxy() address:192.168.128.220 port:10051 timeout:300 conn:1
70754:20211027:095400.215 End of connect_to_proxy():SUCCEED
70754:20211027:095400.215 sending configuration data to proxy "*********" at "192.168.128.220", datalen 8532
70754:20211027:095400.215 In send_data_to_proxy() data:'{"request":"proxy config","data":{"globalmacro":{"fields":["globalmacroid","macro","value"],"data":[[2,"{$SNMP_COMMUNITY}","public"],[2,"{$SNMP_COMMUNITY}","public"],[3,"{$IOB_NET_INT_TRUNK}","^[Tt][Rr][Uu][Nn][Kk].*"],[3,"{$IOB_NET_INT_TRUNK}","^[Tt][Rr][Uu][Nn][Kk].*"],[4,"{$IOB_NET_INT_PO}","^Interface [Pp][Oo][0-9]$"],[4,"{$IOB_NET_INT_PO}","^Interface [Pp][Oo][0-9]$"],[5,"{$IOB_NET_INT_DUPLEX}",".*Duplex status.*"],[5,"{$IOB_NET_INT_DUPLEX}",".*Duplex status.*"],[6,"{$IOB_NET_INT_PO_CH}","^[Pp][Oo][Rr][Tt]-[Cc][Hh][Aa][Nn][Nn][Ee][Ll][0-9.]"],[6,"{$IOB_NET_INT_PO_CH}","^[Pp][Oo][Rr][Tt]-[Cc][Hh][Aa][Nn][Nn][Ee][Ll][0-9.]"],[7,"{$IOB_NET_INT_GI}","[Gg][Ii][0-9.].*"],[7,"{$IOB_NET_INT_GI}","[Gg][Ii][0-9.].*"],[8,"{$IOB_NET_INT_SUP}","^[Ss][Uu][Pp].*$"],[8,"{$IOB_NET_INT_SUP}","^[Ss][Uu][Pp].*$"],[9,"{$IF.ERRORS.WARN}","30"],[9,"{$IF.ERRORS.WARN}","30"],[10,"{$IF.UTIL.MAX}","50"],[10,"{$IF.UTIL.MAX}","50"],[12,"{$IOB_NET_INT_GIGA}","^GigabitEthernet[0-9.].*"],[12,"{$IOB_NET_INT_GIGA}","^GigabitEthernet[0-9.].*"],[13,"{$IOB_NET_INT_TUNNEL}","^Tunnel[0-9.].*"],[13,"{$IOB_NET_INT_TUNNEL}","^Tunnel[0-9.].*"],[14,"{$IF.UTIL.DUREE.MAX}","1m"],[14,"{$IF.UTIL.DUREE.MAX}","1m"],[15,"{$IF.UTIL.MAX.RECOVERY}","2"],[15,"{$IF.UTIL.MAX.RECOVERY}","2"],[16,"{$IF.ERRORS.DUREE.WARN}","5m"],[16,"{$IF.ERRORS.DUREE.WARN}","5m"],[17,"{$IF.ERRORS.WARN.RECOVERY}","5"],[17,"{$IF.ERRORS.WARN.RECOVERY}","5"],[18,"{$IOB_NET_INT_EMBED}","^[Ee]mbedded-[Ss]ervice-[Ee]ngine.*"],[18,"{$IOB_NET_INT_EMBED}","^[Ee]mbedded-[Ss]ervice-[Ee]ngine.*"],[19,"{$IOB_NET_INT_NULL}","^[Nn]ull.*"],[19,"{$IOB_NET_INT_NULL}","^[Nn]ull.*"],[20,"{$IOB_NET_INT_NU}","^[Nn][Uu].*"],[20,"{$IOB_NET_INT_NU}","^[Nn][Uu].*"],[21,"{$IOB_NET_INT_EM}","^[Ee][Mm].*"],[21,"{$IOB_NET_INT_EM}","^[Ee][Mm].*"],[23,"{$IOB_NET_INT_NV}","^[Nn][Vv].*"],[23,"{$IOB_NET_INT_NV}","^[Nn][Vv].*"],[24,"{$IOB_NET_INT_VIR}","(24|131|135)"],[24,"{$IOB_NET_INT_VIR}","(24|131|135)"],[26,"{$IOB_NET_INT_TYPE_1}","(1)"],[26,"{$IOB_NET_INT_TYPE_1}","(1)"],[27,"{$IOB_NET_INT_BRIO}","^[Bb][Rr][Ii][Oo].*"],[27,"{$IOB_NET_INT_BRIO}","^[Bb][Rr][Ii][Oo].*"],[28,"{$IOB_NET_INT_BRIO_IFNAME}","^[Bb][Rr][0-9].*"],[28,"{$IOB_NET_INT_BRIO_IFNAME}","^[Bb][Rr][0-9].*"],[30,"{$CPU.UTIL.CRIT}","99"],[30,"{$CPU.UTIL.CRIT}","99"],[31,"{$MEMORY.UTIL.MAX}","95"],[31,"{$MEMORY.UTIL.MAX}","95"],[32,"{$SWAP.FREE.ALERT}","30"],[32,"{$SWAP.FREE.ALERT}","30"]]},"hosts":{"fields":["hostid","host","status","available","ipmi_authtyp e","ipmi_privilege","ipmi_username","ipmi_password ","ipmi_available","snmp_available","jmx_available ","name","tls_connect","tls_accept","tls_issuer"," tls_subject","tls_psk_identity","tls_psk"],"data":[]},"interface":{"fields":["interfaceid","hostid","main","type","useip","ip", "dns","port"],"data":[]},"interface_snmp":{"fields":["interfaceid","version","bulk","community","securi tyname","securitylevel","authpassphrase","privpass phrase","authprotocol","privprotocol","contextname "],"data":[]},"hosts_templates":{"fields":["hosttemplateid","hostid","templateid"],"data":[]},"hostmacro":{"fields":["hostmacroid","hostid","macro","value"],"data":[]},"items":{"fields":["itemid","type","snmp_oid","hostid","key_","delay" ,"status","value_type","trapper_hosts","logtimefmt ","params","ipmi_sensor","authtype","username","pa ssword","publickey","privatekey","flags","interfac eid","jmx_endpoint","master_itemid","timeout","url ","query_fields","posts","status_codes","follow_re directs","post_type","http_proxy","headers","retri eve_mode","request_method","output_format","ssl_ce rt_file","ssl_key_file","ssl_key_password","verify _peer","verify_host","allow_traps"],"data":[]},"drules":{"fields":["druleid","name","iprange","delay"],"data":[]},"dchecks":{"fields":["dcheckid","druleid","type","key_","snmp_community ","ports","snmpv3_securityname","snmpv3_securityle vel","snmpv3_authpassphrase","snmpv3_privpassphras e","uniq","snmpv3_authprotocol","snmpv3_privprotoc ol","snmpv3_contextname","host_source","name_sourc e"],"data":[]},"regexps":{"fields":["regexpid","name"],"data":[[1,"File systems for discovery"],[2,"Network interfaces for discovery"],[3,"Storage devices for SNMP discovery"],[4,"Windows service names for discovery"],[5,"Windows service startup states for discovery"],[6,"Network Interface for Discovery ALIAS Trunk"],[8,"Network Interface Discover Po"],[9,"Network Interface for Discovery Not DUPLEX"],[10,"Network Interface PO Discover"],[11,"Network Interface Gi Discovery"],[12,"Network Interface Discover ALIAS SUP"],[13,"Network Interface GigabitEthernet"],[14,"Network Interface Tunnel"],[15,"Network Interface Embedded-Service-Engine"],[16,"Network Interface Null0"],[17,"Network Interface GigabitEthernet Virtual"],[18,"Network Interface GigabitEthernet long"],[19,"Interface Virtuel Iftype"],[20,"Interface BRIO"],[21,"Interface BRIO IFNAME"],[22,"TESTE CSCB | SF2i"]]},"expressions":{"fields":["expressionid","regexpid","expression","expression _type","exp_delimiter","case_sensitive"],"data":[[1,1,"^(btrfs|ext2|ext3|ext4|reiser|xfs|ffs|ufs|jfs |jfs2|vxfs|hfs|apfs|refs|ntfs|fat32|zfs)$",3,",",0],[1,1,"^(btrfs|ext2|ext3|ext4|reiser|xfs|ffs|ufs|jfs |jfs2|vxfs|hfs|apfs|refs|ntfs|fat32|zfs)$",3,",",0],[3,3,"^(Physical memory|Virtual memory|Memory buffers|Cached memory|Swap space)$",4,",",1],[3,3,"^(Physical memory|Virtual memory|Memory buffers|Cached memory|Swap space)$",4,",",1],[5,4,"^(MMCSS|gupdate|SysmonLog|clr_optimization_v2 .0.50727_32|clr_optimization_v4.0.30319_32)$",4,", ",1],[5,4,"^(MMCSS|gupdate|SysmonLog|clr_optimization_v2 .0.50727_32|clr_optimization_v4.0.30319_32)$",4,", ",1],[6,5,"^(automatic|automatic delayed)$",3,",",1],[6,5,"^(automatic|automatic delayed)$",3,",",1],[7,2,"^Software Loopback Interface",4,",",1],[7,2,"^Software Loopback Interface",4,",",1],[8,2,"^(In)?[Ll]oop[Bb]ack[0-9._]*$",4,",",1],[8,2,"^(In)?[Ll]oop[Bb]ack[0-9._]*$",4,",",1],[9,2,"^NULL[0-9.]*$",4,",",1],[9,2,"^NULL[0-9.]*$",4,",",1],[10,2,"^[Ll]o[0-9.]*$",4,",",1],[10,2,"^[Ll]o[0-9.]*$",4,",",1],[11,2,"^[Ss]ystem$",4,",",1],[11,2,"^[Ss]ystem$",4,",",1],[12,2,"^Nu[0-9.]*$",4,",",1],[12,2,"^Nu[0-9.]*$",4,",",1],[13,6,"^[Tt][Rr][Uu][Nn][Kk].*",3,",",1],[13,6,"^[Tt][Rr][Uu][Nn][Kk].*",3,",",1],[17,6,"[.*]",4,",",0],[17,6,"[.*]",4,",",0],[18,8,"^Interface [Pp][Oo][0-9]$",3,",",0],[18,8,"^Interface [Pp][Oo][0-9]$",3,",",0],[19,9,".*Duplex status.*",3,",",0],[19,9,".*Duplex status.*",3,",",0],[20,10,"^[Pp][Oo][Rr][Tt]-[Cc][Hh][Aa][Nn][Nn][Ee][Ll][0-9.]",3,",",0],[20,10,"^[Pp][Oo][Rr][Tt]-[Cc][Hh][Aa][Nn][Nn][Ee][Ll][0-9.]",3,",",0],[21,11,"[Gg][Ii][0-9.].*",3,",",0],[21,11,"[Gg][Ii][0-9.].*",3,",",0],[22,12,"^[Ss][Uu][Pp].*$",3,",",0],[22,12,"^[Ss][Uu][Pp].*$",3,",",0],[23,13,"GigabitEthernet\\d{10}",3,",",0],[23,13,"GigabitEthernet\\d{10}",3,",",0],[24,14,"^Tunnel[0-9.].*",3,",",0],[24,14,"^Tunnel[0-9.].*",3,",",0],[25,15,"^[Ee]mbedded-[Ss]ervice-[Ee]ngine.*",3,",",0],[25,15,"^[Ee]mbedded-[Ss]ervice-[Ee]ngine.*",3,",",0],[26,16,"^[Nn]ull.*",3,",",0],[26,16,"^[Nn]ull.*",3,",",0],[27,17,"^GigabitEthernet[0-9.]/.*[.][0-9.].*$",3,",",0],[27,17,"^GigabitEthernet[0-9.]/.*[.][0-9.].*$",3,",",0],[28,18,"^GigabitEthernet[0-9.]/[0-9.]/[0-9.0-9]$",3,",",0],[28,18,"^GigabitEthernet[0-9.]/[0-9.]/[0-9.0-9]$",3,",",0],[29,19,"(24|131|135)",0,",",0],[29,19,"(24|131|135)",0,",",0],[30,20,"^[Bb][Rr][Ii][Oo].*",3,",",0],[30,20,"^[Bb][Rr][Ii][Oo].*",3,",",0],[31,21,"^[Bb][Rr][0-9].*",3,",",0],[31,21,"^[Bb][Rr][0-9].*",3,",",0],[32,22,"AC.1C.E7.05",4,",",0],[32,22,"AC.1C.E7.05",4,",",0]]},"hstgrp":{"fields":["groupid"],"data":[[5],[5]]},"config":{"fields":["configid","refresh_unsupported","discovery_groupi d","snmptrap_logging","autoreg_tls_accept"],"data":[[1,"10m",5,1,1],[1,"10m",5,1,1]]},"httptest":{"fields":["httptestid","name","delay","agent","authenticatio n","http_user","http_password","hostid","http_prox y","retries","ssl_cert_file","ssl_key_file","ssl_k ey_password","verify_peer","verify_host"],"data":[]},"httptestitem":{"fields":["httptestitemid","httptestid","itemid","type"],"data":[]},"httptest_field":{"fields":["httptest_fieldid","httptestid","type","name","val ue"],"data":[]},"httpstep":{"fields":["httpstepid","httptestid","name","no","url","timeo ut","posts","required","status_codes","follow_redi rects","retrieve_mode","post_type"],"data":[]},"httpstepitem":{"fields":["httpstepitemid","httpstepid","itemid","type"],"data":[]},"httpstep_field":{"fields":["httpstep_fieldid","httpstepid","type","name","val ue"],"data":[]},"config_autoreg_tls":{"fields":["autoreg_tlsid","tls_psk_identity","tls_psk"],"data":[[1,"",""],[1,"",""]]}}}'
70754:20211027:095400.215 End of send_data_to_proxy():SUCCEED

You can see the duplicate items being sent two lines before end of logs.

 

Steps to reproduce:

I don't know how to reproduce because it happened all of a sudden. Before that happened, I was replacing a bare-metal proxy instance with a new proxy with the same name, so it's probably related.

Since that time, I tried removing that proxy, restarting the server and several stuff with no effect at all, it's now affecting all (4) proxies, active and passive ones.

 

 Comments   
Comment by Alexey Pustovalov [ 2021 Nov 04 ]

Could you try to recreate schema on proxies? When you created DB for proxy, did you import schema only or data as well?

Comment by Guillaume Fenollar [ 2021 Nov 04 ]

Hello Alexey.

I forgot to mention that I indeed tried several times to remove the proxy DB (I'll update the issue with this information). I also created the proxy DB using the provided schema.sql but result is unchanged in all cases. I never imported any data while creating proxy DB, either I let the proxy service do it (I only removed the whole db file), or I created the schema myself but nothing more.

Comment by Alexey Pustovalov [ 2021 Nov 04 ]

I recommend you increase debug level for "configuration syncer" on active Zabbix proxy:

zabbix_proxy -R log_level_increase="configuration syncer"

then wait at least one iteration and attach the file to the issue.

Comment by Guillaume Fenollar [ 2021 Nov 04 ]

Thanks for the help. I attached the log which is for one sync iteration on a pretty buse active proxy server.

Comment by Alexey Pustovalov [ 2021 Nov 04 ]

could you attach output of the following SQL query from Zabbix server side:

select macro, count(*) from globalmacro group by macro;

or just check, count must be 1 for all records! It looks like Zabbix server has duplicated data.

Comment by Guillaume Fenollar [ 2021 Nov 04 ]

Hi,
Indeed all records from globalmacro are duplicated.

There's no way somebody in my company could do that, so it looks like it was inserted by zabbix server or frontend, I don't know.
I dropped the duplicated records and this issue disappeared but was instantly replaced with same problem concerning expressions columns (Perhaps it could be prevented by setting UNIQUE on those columns ?). After fixing globalmacro and expressions tables, another table issue took over.

2475:20211105:084959.853 [Z3005] query failed: [0] UNIQUE constraint failed: hstgrp.groupid [insert into hstgrp (groupid) values (5);
insert into hstgrp (groupid) values (5);

Problem is that I couldn't find any duplicate record in Zabbix database.

zabbix=# select groupid, count(*) from hstgrp group by groupid having count(*) <> 1;
 groupid | count
---------+-------
(0 rows)

Note than I dropped records from the database while zabbix-server and frontend where down for safety. Also, I tried deleting proxy database just in case, but same issue.

Nothing interesting with debug "configuration syncer" log on active proxy, it only tells about the duplicate row.

Comment by Alexey Pustovalov [ 2021 Nov 05 ]

could you check on Zabbix server:

select * from hstgrp where groupid = 5;

configuration syncer return duplicated data:

"hstgrp":{"fields":["groupid"],"data":[[5],[5]]}
Comment by Guillaume Fenollar [ 2021 Nov 05 ]

this hstgrp record is about Direcovered Hosts. It's the only one with internal column set to 1, I don't know what that means.

zabbix=# select * from hstgrp where groupid = 5;
 groupid |       name       | internal | flags
---------+------------------+----------+-------
       5 | Discovered hosts |        1 |     0
(1 row)

Thanks again for your help

Comment by Alexey Pustovalov [ 2021 Nov 05 ]

maybe you have duplicate records in "config" table on Zabbix server side?

Comment by Guillaume Fenollar [ 2021 Nov 05 ]

We indeed have 2 rows returned, exactly the same content (same configid rows).

How can that happen ? Is there a plan to put UNIQUE constraint on id columns ? 

I can drop one of these rows on monday, I'll comment this Jira issue then.

Thanks

Comment by Alexey Pustovalov [ 2021 Nov 05 ]

For example, in case some fails Yes, please remove one row!

Comment by Guillaume Fenollar [ 2021 Nov 07 ]

Problem solved after last duplicates where removed from config and config_autoreg_tls tables.

I'll let you decide if this issue can be closed or if you can prevent this from happening again by fixing the code

Thanks a bunch  Alexey for your help !

Comment by Alexey Pustovalov [ 2021 Nov 07 ]

Thank you!

Generated at Wed May 14 07:27:48 EEST 2025 using Jira 9.12.4#9120004-sha1:625303b708afdb767e17cb2838290c41888e9ff0.