Details
-
Problem report
-
Resolution: Unresolved
-
Trivial
-
None
-
3.4.14
-
actual version 3.4.10
-
Team B
-
3
Description
Last 28th during the time change to wintertime all communications between Zabbix server and Zabbix proxy suddenly stopped working.
11276:20181028:021904.224 sending configuration data to proxy "A" at "X.X.X.63", datalen 36831646 11271:20181028:022017.939 sending configuration data to proxy "B" at "X.X.X.74", datalen 14052878 11278:20181028:022022.380 sending configuration data to proxy "C" at "X.X.X.56", datalen 36070831 11271:20181028:022051.518 sending configuration data to proxy "D" at "X.X.X.53", datalen 37935485 11274:20181028:022127.225 sending configuration data to proxy "E" at "X.X.X.55", datalen 40935097 11274:20181028:022212.521 sending configuration data to proxy "F" at "X.X.X.49", datalen 35815498 11282:20181028:022252.250 sending configuration data to proxy "G" at "X.X.X.74", datalen 14684905 11279:20181028:022444.342 sending configuration data to proxy "H" at "X.X.X.54", datalen 41381057 11277:20181028:022911.060 sending configuration data to proxy "A" at "X.X.X.63", datalen 36831646 11275:20181028:023029.085 sending configuration data to proxy "C" at "X.X.X.56", datalen 36070831 11279:20181028:023044.988 sending configuration data to proxy "B" at "X.X.X.74", datalen 14052878 11274:20181028:023058.625 sending configuration data to proxy "D" at "X.X.X.53", datalen 37935485 11274:20181028:023134.601 sending configuration data to proxy "E" at "X.X.X.55", datalen 40935097 11276:20181028:023218.856 sending configuration data to proxy "F" at "X.X.X.49", datalen 35815498 11274:20181028:023259.784 sending configuration data to proxy "G" at "X.X.X.74", datalen 14684905 11274:20181028:023451.778 sending configuration data to proxy "H" at "X.X.X.54", datalen 41381057 11275:20181028:023918.121 sending configuration data to proxy "A" at "X.X.X.63", datalen 36831646 11277:20181028:024035.549 sending configuration data to proxy "C" at "X.X.X.56", datalen 36070831 11282:20181028:024105.781 sending configuration data to proxy "D" at "X.X.X.53", datalen 37935485 11280:20181028:024111.567 sending configuration data to proxy "B" at "X.X.X.74", datalen 14052878 11274:20181028:024142.038 sending configuration data to proxy "E" at "X.X.X.55", datalen 40935097 11281:20181028:024225.339 sending configuration data to proxy "F" at "X.X.X.49", datalen 35815498 11271:20181028:024307.378 sending configuration data to proxy "G" at "X.X.X.74", datalen 14684905 11279:20181028:024458.978 sending configuration data to proxy "H" at "X.X.X.54", datalen 41381057 11278:20181028:024925.014 sending configuration data to proxy "A" at "X.X.X.63", datalen 36831646 11280:20181028:025041.940 sending configuration data to proxy "C" at "X.X.X.56", datalen 36070831 11281:20181028:025112.798 sending configuration data to proxy "D" at "X.X.X.53", datalen 37935485 11272:20181028:025132.167 sending configuration data to proxy "B" at "X.X.X.74", datalen 14052878 11275:20181028:025149.643 sending configuration data to proxy "E" at "X.X.X.55", datalen 40935097 11276:20181028:025231.784 sending configuration data to proxy "F" at "X.X.X.49", datalen 35815498 11279:20181028:025314.788 sending configuration data to proxy "G" at "X.X.X.74", datalen 14684905 11275:20181028:025507.153 sending configuration data to proxy "H" at "X.X.X.54", datalen 41381057 11280:20181028:025931.875 sending configuration data to proxy "A" at "X.X.X.63", datalen 36831646 11280:20181028:020048.275 sending configuration data to proxy "C" at "X.X.X.56", datalen 36070831 11282:20181028:020120.518 sending configuration data to proxy "D" at "X.X.X.53", datalen 37935485 11279:20181028:020148.842 sending configuration data to proxy "B" at "X.X.X.74", datalen 14052878 11280:20181028:020157.451 sending configuration data to proxy "E" at "X.X.X.55", datalen 40935097 11274:20181028:020238.388 sending configuration data to proxy "F" at "X.X.X.49", datalen 35815498 11281:20181028:020322.559 sending configuration data to proxy "G" at "X.X.X.74", datalen 14684905 11278:20181028:020515.240 sending configuration data to proxy "H" at "X.X.X.54", datalen 41381057 11242:20181028:020935.444 executing housekeeper
No more useful logs until restart only housekeper logs.
In the proxy side the logs are quite similar:
9256:20181028:025917.057 SNMP agent item "authServerUptime.['X.X.X.70']" on host "A" failed: first network error, wait for 15 seconds 8895:20181028:025932.379 received configuration data from server at "server", datalen 36831646 9032:20181028:025950.648 SNMP agent item "wlsxSysExtPacketLossPercent" on host "B" failed: first network error, wait for 15 seconds 8898:20181028:020322.184 executing housekeeper 8898:20181028:020330.795 slow query: 8.593921 sec, "delete from proxy_history where id<13790379271 and (clock<1540685002 or (id<=13790379271 and clock<1540688602))" 8898:20181028:020330.834 housekeeper [deleted 1884192 records in 8.634713 sec, idle for 1 hour(s)] 8895:20181028:020940.637 received configuration data from server at "server", datalen 36831646 8898:20181028:030331.137 executing housekeeper 8898:20181028:030331.141 housekeeper [deleted 0 records in 0.002344 sec, idle for 1 hour(s)] 8898:20181028:040331.451 executing housekeeper 8898:20181028:040331.455 housekeeper [deleted 0 records in 0.002180 sec, idle for 1 hour(s)]
Again only housekeeping logs after restart
The first thing we tried was restarting the proxy without restarting the server
These are the logs in the server side and the proxy side after this:
Server log:
cannot send configuration data to proxy "A" at "X.X.X.54": TLS write set result code to 5:
Proxy log:
Unable to connect to the server [server]:10051 [TCP successful, cannot establish TLS to [[server]:10051]: SSL_connect() I/O error: [104] Connection reset by peer]
Restarting the agent shows similar errors:
active check configuration update from [A:10051] started to fail (TCP successful, cannot establish TLS to [[A]:10051]: SSL_connect() timed out)
We restarted the server and after this restart all the proxies started working again.