Uploaded image for project: 'ZABBIX BUGS AND ISSUES'
  1. ZABBIX BUGS AND ISSUES
  2. ZBX-7968

The Zabbix server pretty much just stops importing data from proxies

    Details

    • Type: Incident report
    • Status: Closed
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: 2.2.2
    • Fix Version/s: 2.2.4rc1, 2.3.0
    • Component/s: Server (S)
    • Labels:
    • Environment:
      CentOS 6.2 x64
      Percona MySQL 5.6.15
      24 DBSyncers

      Description

      I've run into this issue a few times, but I've never been able to replicate it at will. Here's what appears to cause the issue:

      1. My Zabbix proxies will suddenly start getting a massive backlog of data. I really do mean massive. I have 22 proxies and I've seen this issue occur when there are a total of 8+ million values backlogged between them all. This can be a result of various causes (lots of data sent in at once, server was offline for a while, etc.).
      2. When #1 occurs, the zabbix server begins accepting all the data (as expected).
      3. I can see all of my history syncers inserting data as fast as they can into the database (as expected).
      4. After a while (variable time), the database load will suddenly disappear. At any given time I might see one insert query for the DBSyncers, by that's it.
      5. After the load has disappeared, I can see that items that are polled/pushed to/from the zabbix server itself are updated.
      6. At the same time as #5, I can see that barely any data is being retrieved from the Zabbix proxies at all. The backlog of data keeps growing larger and larger.
      7. After waiting for a while without the problem going away, I'll give up and restart the zabbix_server process.
      8. After restarting the zabbix_server process, the server immediately begins to process data like crazy from the proxies (like it should have been doing BEFORE the restart).

      When this problem is occuring, the load on my database server is almost non-existant. It literally isn't doing anything. The same goes for the Zabbix server. There is nothing to indicate that the server is overly busy. The last time this happened (about 30 minutes ago), my monitoring on the various zabbix processes/caches didn't show anything odd. The history cache usage had grown quite a bit before #4 happened, but that was expected because so much data was coming in from my proxies. At no point did the free space for the history cache reach 0% (at least according to my graphs/data).

      I have attached a copy of my zabbix_server.conf and a screenshot of the cache, data gathering processes, and internal processes usage. Notice that my history syncers were maxed out, but as I mentioned earlier, they were not doing anything on the DB itself. The DB server was idle.

        Attachments

        1. Internal Processes.png
          Internal Processes.png
          61 kB
        2. zabbix_busy_procs.png
          zabbix_busy_procs.png
          147 kB
        3. zabbix_busy_procs2.png
          zabbix_busy_procs2.png
          157 kB
        4. Zabbix_server_stats.png
          Zabbix_server_stats.png
          90 kB
        5. zabbix_server.conf
          12 kB
        6. ZBX-7968-update-lastid-2.2.2.patch
          0.5 kB

          Activity

            People

            • Assignee:
              Unassigned
              Reporter:
              heaje Corey Shaw
            • Votes:
              3 Vote for this issue
              Watchers:
              8 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: