Configuration cache sync returning 0 triggers and functions intermittently

XMLWordPrintable

    • Type: Incident report
    • Resolution: Cannot Reproduce
    • Priority: Trivial
    • None
    • Affects Version/s: None
    • Component/s: Server (S)
    • None

       Hello,

      As part of a hardware refresh, we've recently failed over our database from one cluster, to another. Same OS version (Xenial 16.04.1, kernel 4.4.0-31-generic), same MySQL version (Percona-Server-5.7.21-20-Linux.x86_64.ssl100), the only difference is the underlying hardware. Since this was performed, we're seeing trigger processing fail intermittently. This occurs at some point after the configuration cache is synced (600s interval). We believe we've narrowed down why the processing failures occur, which are due to the configuration cache sync returning zero triggers and zero functions.

      Again, this occurs intermittently, but enough that it renders monitoring useless for the period of time, sometimes hitting up to 45 minutes. Once we failed back to the original cluster, the issue subsided and trigger processing is normal. Looking at the queries that are run during the config sync, nothing is failing and according to mysql, each query is running fine and returning the expected ~3m rows.

      I've attached the output from DCsync_configuration(), indicating what we are seeing. Any direction on how to debug this issue would be greatly appreciated.

      Zabbix Version: 3.0.5

        1. zabbix_new_hw_globals
          250 kB
        2. zabbix_new_hw_cfg
          6 kB
        3. good_zbx_sync2
          23 kB
        4. good_zbx_sync
          18 kB
        5. bad_zbx_sync2
          23 kB
        6. bad_zbx_sync
          17 kB

            Assignee:
            Kristians Pavars
            Reporter:
            Andrew Boren
            Votes:
            0 Vote for this issue
            Watchers:
            3 Start watching this issue

              Created:
              Updated:
              Resolved: