Uploaded image for project: 'ZABBIX BUGS AND ISSUES'
  1. ZABBIX BUGS AND ISSUES
  2. ZBX-15091

TLS communication between zabbix-server and zabbix-proxies stops working after change to wintertime

XMLWordPrintable

    • Icon: Problem report Problem report
    • Resolution: Unresolved
    • Icon: Trivial Trivial
    • None
    • 3.4.14
    • Proxy (P), Server (S)
    • actual version 3.4.10
    • Team B
    • 3

      Last 28th during the time change to wintertime all communications between Zabbix server and Zabbix proxy suddenly stopped working.

      11276:20181028:021904.224 sending configuration data to proxy "A" at "X.X.X.63", datalen 36831646
 11271:20181028:022017.939 sending configuration data to proxy "B" at "X.X.X.74", datalen 14052878
 11278:20181028:022022.380 sending configuration data to proxy "C" at "X.X.X.56", datalen 36070831
 11271:20181028:022051.518 sending configuration data to proxy "D" at "X.X.X.53", datalen 37935485
 11274:20181028:022127.225 sending configuration data to proxy "E" at "X.X.X.55", datalen 40935097
 11274:20181028:022212.521 sending configuration data to proxy "F" at "X.X.X.49", datalen 35815498
 11282:20181028:022252.250 sending configuration data to proxy "G" at "X.X.X.74", datalen 14684905
 11279:20181028:022444.342 sending configuration data to proxy "H" at "X.X.X.54", datalen 41381057
 11277:20181028:022911.060 sending configuration data to proxy "A" at "X.X.X.63", datalen 36831646
 11275:20181028:023029.085 sending configuration data to proxy "C" at "X.X.X.56", datalen 36070831
 11279:20181028:023044.988 sending configuration data to proxy "B" at "X.X.X.74", datalen 14052878
 11274:20181028:023058.625 sending configuration data to proxy "D" at "X.X.X.53", datalen 37935485
 11274:20181028:023134.601 sending configuration data to proxy "E" at "X.X.X.55", datalen 40935097
 11276:20181028:023218.856 sending configuration data to proxy "F" at "X.X.X.49", datalen 35815498
 11274:20181028:023259.784 sending configuration data to proxy "G" at "X.X.X.74", datalen 14684905
 11274:20181028:023451.778 sending configuration data to proxy "H" at "X.X.X.54", datalen 41381057
 11275:20181028:023918.121 sending configuration data to proxy "A" at "X.X.X.63", datalen 36831646
 11277:20181028:024035.549 sending configuration data to proxy "C" at "X.X.X.56", datalen 36070831
 11282:20181028:024105.781 sending configuration data to proxy "D" at "X.X.X.53", datalen 37935485
 11280:20181028:024111.567 sending configuration data to proxy "B" at "X.X.X.74", datalen 14052878
 11274:20181028:024142.038 sending configuration data to proxy "E" at "X.X.X.55", datalen 40935097
 11281:20181028:024225.339 sending configuration data to proxy "F" at "X.X.X.49", datalen 35815498
 11271:20181028:024307.378 sending configuration data to proxy "G" at "X.X.X.74", datalen 14684905
 11279:20181028:024458.978 sending configuration data to proxy "H" at "X.X.X.54", datalen 41381057
 11278:20181028:024925.014 sending configuration data to proxy "A" at "X.X.X.63", datalen 36831646
 11280:20181028:025041.940 sending configuration data to proxy "C" at "X.X.X.56", datalen 36070831
 11281:20181028:025112.798 sending configuration data to proxy "D" at "X.X.X.53", datalen 37935485
 11272:20181028:025132.167 sending configuration data to proxy "B" at "X.X.X.74", datalen 14052878
 11275:20181028:025149.643 sending configuration data to proxy "E" at "X.X.X.55", datalen 40935097
 11276:20181028:025231.784 sending configuration data to proxy "F" at "X.X.X.49", datalen 35815498
 11279:20181028:025314.788 sending configuration data to proxy "G" at "X.X.X.74", datalen 14684905
 11275:20181028:025507.153 sending configuration data to proxy "H" at "X.X.X.54", datalen 41381057
 11280:20181028:025931.875 sending configuration data to proxy "A" at "X.X.X.63", datalen 36831646
 11280:20181028:020048.275 sending configuration data to proxy "C" at "X.X.X.56", datalen 36070831
 11282:20181028:020120.518 sending configuration data to proxy "D" at "X.X.X.53", datalen 37935485
 11279:20181028:020148.842 sending configuration data to proxy "B" at "X.X.X.74", datalen 14052878
 11280:20181028:020157.451 sending configuration data to proxy "E" at "X.X.X.55", datalen 40935097
 11274:20181028:020238.388 sending configuration data to proxy "F" at "X.X.X.49", datalen 35815498
 11281:20181028:020322.559 sending configuration data to proxy "G" at "X.X.X.74", datalen 14684905
 11278:20181028:020515.240 sending configuration data to proxy "H" at "X.X.X.54", datalen 41381057
 11242:20181028:020935.444 executing housekeeper

      No more useful logs until restart only housekeper logs.

      In the proxy side the logs are quite similar:

      9256:20181028:025917.057 SNMP agent item "authServerUptime.['X.X.X.70']" on host "A" failed: first network error, wait for 15 seconds
  8895:20181028:025932.379 received configuration data from server at "server", datalen 36831646
  9032:20181028:025950.648 SNMP agent item "wlsxSysExtPacketLossPercent" on host "B" failed: first network error, wait for 15 seconds
  8898:20181028:020322.184 executing housekeeper
  8898:20181028:020330.795 slow query: 8.593921 sec, "delete from proxy_history where id<13790379271 and (clock<1540685002 or (id<=13790379271 and clock<1540688602))"
  8898:20181028:020330.834 housekeeper [deleted 1884192 records in 8.634713 sec, idle for 1 hour(s)]
  8895:20181028:020940.637 received configuration data from server at "server", datalen 36831646
  8898:20181028:030331.137 executing housekeeper
  8898:20181028:030331.141 housekeeper [deleted 0 records in 0.002344 sec, idle for 1 hour(s)]
  8898:20181028:040331.451 executing housekeeper
  8898:20181028:040331.455 housekeeper [deleted 0 records in 0.002180 sec, idle for 1 hour(s)]

      Again only housekeeping logs after restart

      The first thing we tried was restarting the proxy without restarting the server

      These are the logs in the server side and the proxy side after this:

      Server log:

      cannot send configuration data to proxy "A" at "X.X.X.54": TLS write set result code to 5:

      Proxy log:

      Unable to connect to the server [server]:10051 [TCP successful, cannot establish TLS to [[server]:10051]: SSL_connect() I/O error: [104] Connection reset by peer]

      Restarting the agent shows similar errors:

      active check configuration update from [A:10051] started to fail (TCP successful, cannot establish TLS to [[A]:10051]: SSL_connect() timed out)

      We restarted the server and after this restart all the proxies started working again.

            zabbix.dev Zabbix Development Team
            zux Edgars Melveris
            Team B
            Votes:
            2 Vote for this issue
            Watchers:
            9 Start watching this issue

              Created:
              Updated: