Uploaded image for project: 'ZABBIX BUGS AND ISSUES'
  2. ZBX-21866

Preprocessor workers getting stuck on certain items


    • Icon: Incident report Incident report
    • Resolution: Duplicate
    • Icon: Trivial Trivial
    • None
    • 6.0.10
    • Server (S)
    • None
    • Docker image: zabbix-server-pgsql:ubuntu-6.0.10
      DB: PostgreSQL 13.7 / TimescaleDB 2.7.2

      Steps to reproduce:

      1. Restart Zabbix server container
      2. Observe normal preprocessor performance. (several thousand items processed per 5s cycle of preprocessing manager)
        {{preprocessing manager #1 [queued 99, processed 4804 values, idle 4.963593 sec during 5.019956 sec]
        preprocessing manager #1 [queued 0, processed 4426 values, idle 4.967512 sec during 5.017058 sec]
      3. Wait awhile...


      After an unspecified amount of time, the preprocessing threads begin to run hot on CPU and process very few item values per 5 second cycle. Eventually the backlog becomes so great that all active agents begin alerting due to none of their associated items being inserted into the database.

      preprocessing manager #1 [queued 66660, processed 6 values, idle 2.757768 sec during 5.001833 sec]
      preprocessing manager #1 [queued 133332, processed 5 values, idle 0.000000 sec during 5.003844 sec]
      preprocessing manager #1 [queued 103065, processed 0 values, idle 0.000000 sec during 5.002716 sec]

      1. much later
        preprocessing manager #1 [queued 64577, processed 21 values, idle 0.000000 sec during 5.001084 sec]
        preprocessing manager #1 [queued 16933, processed 106003 values, idle 0.000000 sec during 5.000027 sec]
      2. and later still
        preprocessing manager #1 [queued 327975, processed 1 values, idle 0.000000 sec during 5.001024 sec]

      The low/zero idle time per cycle combined with the small number of metrics being passed seem to suggest that there is a bottleneck in the preprocessor workers that is causing them to get stuck on specific items. I was not able to find anything in the Zabbix documentation that specifies how to troubleshoot the contents of the preprocessing queue, i.e. which items are the ones that are taking so long to execute.

      I'm happy to provide additional information if told what commands to run or where to look inside of the database.

      See screenshot....
      See attached patch file...

            zux Edgars Melveris
            aboling Andrew Boling
            0 Vote for this issue
            2 Start watching this issue