Potential startup/shutdown deadlock between Zabbix Server and MariaDB when the database is unavailable during Zabbix startup.

XMLWordPrintable

    • Type: Problem report
    • Resolution: Unresolved
    • Priority: Trivial
    • None
    • Affects Version/s: 7.0.27
    • Component/s: Server (S)
    • Environment:
      - Zabbix Server 7.x
      - MariaDB 10.11.x
      - Debian-based Linux
      - systemd

      Description

      The issue was initially observed after an unattended upgrade. During the upgrade process, systemd automatically restarted several services, including MariaDB and Zabbix Server.

      After the automatic restart sequence, MariaDB failed to become available and Zabbix Server was left in an inconsistent state with multiple worker processes marked as terminated.

      Further testing indicated that the issue can be reproduced when Zabbix Server is started while MariaDB is unavailable.

      Steps to Reproduce

      1. Stop MariaDB:

      systemctl stop mariadb

      1. Start or restart Zabbix Server:

      systemctl restart zabbix-server

      1. Observe that Zabbix does not start cleanly and multiple worker processes remain in a terminated state.
      1. Attempt to start MariaDB:

      systemctl start mariadb

      1. Observe that MariaDB startup may become stuck or not complete normally.
      1. Terminate all remaining Zabbix Server processes.
      1. Start MariaDB again and then start Zabbix Server.

      Observed Behavior

      • Numerous Zabbix worker processes remain present with status similar to:

      {{zabbix_server: poller #1 [terminated] }}
      {{zabbix_server: trapper #1 [terminated] }}
      zabbix_server: alert manager #1 [terminated]

      • Zombie (<defunct>) processes may also appear.
      • MariaDB startup can become blocked until the remaining Zabbix processes are terminated.
      • Once all Zabbix processes are removed, MariaDB starts successfully and Zabbix can subsequently start normally.

      Expected Behavior

      If the database is unavailable:

      • Zabbix Server should fail cleanly and exit, or
      • Zabbix Server should remain in a controlled retry state without leaving partially terminated worker processes.

      The service should not remain in an inconsistent state that interferes with subsequent database startup.

      Additional Notes

      The issue appears related to service startup ordering and database availability. Adding the following systemd dependency to zabbix-server.service prevents the problem in my environment:

      [Unit]

      Requires=mariadb.service

       

      The default unit file already contains:

      After=mariadb.service

      However, After= only enforces startup order and does not require MariaDB to be running before Zabbix is started.

            Assignee:
            Zabbix Support Team
            Reporter:
            Andrei Naddour
            Votes:
            0 Vote for this issue
            Watchers:
            1 Start watching this issue

              Created:
              Updated:

                Estimated:
                Original Estimate - 1h
                1h
                Remaining:
                Remaining Estimate - 1h
                1h
                Logged:
                Time Spent - Not Specified
                Not Specified