ZBX 7.0.26 – Preprocessing queue grows indefinitely; workers not started by manager despite config

XMLWordPrintable

    • Type: Problem report
    • Resolution: Done
    • Priority: Trivial
    • 7.0.26
    • Affects Version/s: 7.0.26
    • Component/s: Server (S)
    • None
    • Environment:

      Starting around May 15, the preprocessing queue on the Zabbix server began growing uncontrollably, eventually exceeding 10 million queued values and 20 GB in size. The preprocessing manager (preprocessing manager #1) is running, but it does not start or maintain preprocessing worker processes, even though StartPreprocessors is set to 80 (or 1). Consequently, new values are accepted into the queue (queued count increases) but are never dispatched to workers for processing. After the threshold of 10 million, queues for data items start to grow dramatically > 10000 items.

      1. ps aux shows no preprocessing worker processes at all. Only the manager is present.
      1. zabbix[preprocessing] output:
        json
        Unknown macro: {"queued"}

        The queue field remains 0 while the internal queue bloats.

      1. zabbix_server -R diaginfo=preprocessing consistently reports:
        textpending tasks:0 finished tasks:0 task sequences:0 queued count:1244046 queued size:2366958346
        indicating that no tasks are being assigned or completed by workers.
      1. The manager’s debug log shows it cycling through preproc_flush_value_server(), receiving finished messages with values_num:0, and occasionally adding new requests via preprocessor_add_request(). However, no function related to pulling tasks from the queue and sending them to workers appears. For example:
        textzbx_setproctitle() title:'preprocessing manager #1 [queued 1532, processed 2299 values, idle 4.696601 sec during 5.002178 sec]'
        The manager acknowledges the queue but remains idle.
      1. Server logs do not contain any errors about workers crashing (terminated or killed), nor any OOM events. Workers simply never appear or, when they briefly do (after a restart with StartPreprocessors=1), they vanish later without a trace.
      1. zabbix_server -R prof_disable='preprocessing worker'
        Cannot redirect signal: "preprocessing worker" process does not exist

      Troubleshooting Steps Already Taken:

      1. Confirmed that the system has sufficient process limits (Max processes = 65536).
      1. Tried StartPreprocessors=200 – workers not started.
      1. Changed to StartPreprocessors=1 following known workarounds for similar bugs (ZBX-26610, ZBX-26693). After a restart, logs once showed:
        textserver #7 started preprocessing manager #1 [1] thread started preprocessing worker #1
        However, the worker disappeared soon after, without any termination log entry, and the queue continued to grow.
      1. Stopped the server, removed all Zabbix socket files (/var/run/zabbix/*.sock), and started again. The behavior did not improve.
      1. Performed multiple full stop/start cycles (with sleep 180 in between).
      1. Attempted runtime command zabbix_server -R preprocessing_worker_start, but it returned unknown option (command does not exist in this version).
      1. Ran zabbix_server -R log_level_increase="preprocessing manager" to gather debug logs. The collected debug output confirms the manager loops but never dispatches queued items to workers (see attached snippet).
      1. Verified that no SELinux/AppArmor denials are blocking socket creation or process spawning.

      Request:
      We suspect this is a regression or a not-fully-fixed variant of ZBX-26610 / ZBX-26693. Could you please:

      1. Confirm whether Zabbix 7.0.26 contains a known issue that would cause the preprocessing manager to fail to start/keep workers under certain conditions.
      1. Provide a workaround or hotfix that allows the preprocessing engine to function again (e.g., a configuration tweak or a runtime command) without waiting for the queue to drain (which is impossible without workers).
      1. If a newer stable release (7.0.27 or 7.2.x) resolves this, please advise on the recommended upgrade path and any special steps required given the current state of the preprocessing queue.

      We have already gathered extensive debug information and can supply full logs or a support bundle upon request.

      Thank you for your assistance.

        1. image-2026-05-19-10-42-50-319.png
          172 kB
          Sergey
        2. image-2026-05-19-10-48-16-643.png
          39 kB
          Sergey
        3. image-2026-05-19-10-51-06-360.png
          122 kB
          Sergey
        4. image-2026-05-19-10-52-47-245.png
          203 kB
          Sergey
        5. image-2026-05-19-10-54-01-727.png
          266 kB
          Sergey
        6. image-2026-05-19-16-48-08-985.png
          25 kB
          Sergey

            Assignee:
            Zabbix Support Team
            Reporter:
            Sergey
            Votes:
            0 Vote for this issue
            Watchers:
            2 Start watching this issue

              Created:
              Updated:
              Resolved: