Uploaded image for project: 'ZABBIX BUGS AND ISSUES'
  1. ZABBIX BUGS AND ISSUES
  2. ZBX-22122

Systemctl restart zabbix-server hangs

    XMLWordPrintable

Details

    • Problem report
    • Status: Closed
    • Trivial
    • Resolution: Duplicate
    • 6.2.6
    • None
    • Server (S)
    • None
    • Ubuntu 20.04
    • Team B
    • Sprint 95 (Dec 2022), Sprint 96 (Jan 2023)

    Description

      Steps to reproduce:

      1. Configuration: 
        LogFile=/var/log/zabbix/zabbix_server.log
        LogFileSize=0
        PidFile=/run/zabbix/zabbix_server.pid
        SocketDir=/run/zabbix
        DBHost=localhost
        DBName=zabbix
        DBUser=zabbix
        DBPassword=omitted
        DBPort=5000
        StartPollersUnreachable=3
        StartTrappers=5
        StartPingers=3
        StartDiscoverers=3
        SNMPTrapperFile=/var/log/snmptrap/snmptrap.log
        MaxHousekeeperDelete=50000
        CacheSize=512M
        TrendCacheSize=8M
        TrendFunctionCacheSize=4M
        ValueCacheSize=36M
        Timeout=4
        FpingLocation=/usr/bin/fping
        Fping6Location=/usr/bin/fping6
        LogSlowQueries=3000
        StatsAllowedIP=127.0.0.1
        HANodeName=zabbix-prod-van2
        NodeAddress=127.0.0.1:10051 
      1. Run:
        sudo systemctl restart zabbix-server
      1. Wait forever...?
      2. Do in another terminal:
        sudo pkill -9 zabbix_server

      Result:
      It's taking too long to restart. 

      $ ps auxww | grep zabbix_server
      
      zabbix   1970382  0.0  0.0 112972  6164 ?        S    15:47   0:00 /usr/sbin/zabbix_server: ha manager
      zabbix   1970389  0.0  0.0 678500  7280 ?        S    15:47   0:00 /usr/sbin/zabbix_server: service manager #1 [processed 0 events, updated 0 event tags, deleted 0 problems, synced 0 service updates, idle 4.002459 sec during 5.419148 sec]
      zabbix   1970390  0.2  1.3 783264 220944 ?       S    15:47   0:06 /usr/sbin/zabbix_server: configuration syncer #1 [terminated]
      zabbix   1970419  0.0  0.0 678236  5976 ?        S    15:48   0:00 /usr/sbin/zabbix_server: alert manager #1 [terminated]
      zabbix   1970420  0.0  0.0 678236  4220 ?        S    15:48   0:00 /usr/sbin/zabbix_server: alerter #1 started
      zabbix   1970421  0.0  0.0 678236  4220 ?        S    15:48   0:00 /usr/sbin/zabbix_server: alerter #2 started
      zabbix   1970422  0.0  0.0 678236  4220 ?        S    15:48   0:00 /usr/sbin/zabbix_server: alerter #3 started
      zabbix   1970423  0.1  0.8 761916 140084 ?       S    15:48   0:04 /usr/sbin/zabbix_server: preprocessing manager #1 [terminated]
      zabbix   1970424  0.0  0.0 679472  6612 ?        S    15:48   0:00 /usr/sbin/zabbix_server: preprocessing worker #1 started
      zabbix   1970425  0.0  0.0 679760  6876 ?        S    15:48   0:00 /usr/sbin/zabbix_server: preprocessing worker #2 started
      zabbix   1970427  0.0  0.0 679688  6876 ?        S    15:48   0:00 /usr/sbin/zabbix_server: preprocessing worker #3 started
      zabbix   1970428  0.0  0.0 678236  5212 ?        S    15:48   0:00 /usr/sbin/zabbix_server: lld manager #1 [terminated]
      zabbix   1970429  0.0  0.0 682820 14652 ?        S    15:48   0:00 /usr/sbin/zabbix_server: lld worker #1 [processed 1 LLD rules, idle 295.191830 sec during 300.755677 sec]
      zabbix   1970430  0.0  0.0 680956 12760 ?        S    15:48   0:00 /usr/sbin/zabbix_server: lld worker #2 [processed 1 LLD rules, idle 214.989587 sec during 220.226943 sec]
      zabbix   1970431  0.0  0.1 689680 18804 ?        S    15:48   0:00 /usr/sbin/zabbix_server: housekeeper [removing old history and trends]
      zabbix   1970432  0.0  0.0 678500  7512 ?        S    15:48   0:00 /usr/sbin/zabbix_server: timer #1 [terminated]
      zabbix   1970433  0.0  0.0 678236  5936 ?        S    15:48   0:00 /usr/sbin/zabbix_server: http poller #1 [terminated]
      zabbix   1970434  0.0  0.0 684144 13848 ?        S    15:48   0:00 /usr/sbin/zabbix_server: discoverer #1 [processed 0 rules in 0.000000 sec, performing discovery]
      zabbix   1970435  0.0  0.0 684148 13848 ?        S    15:48   0:00 /usr/sbin/zabbix_server: discoverer #2 [processed 0 rules in 0.000000 sec, performing discovery]
      zabbix   1970436  0.0  0.0 678236  7456 ?        S    15:48   0:00 /usr/sbin/zabbix_server: discoverer #3 [terminated]
      zabbix   1970437  0.1  1.0 793348 176636 ?       S    15:48   0:02 /usr/sbin/zabbix_server: history syncer #1 [processed 867 values, 1417 triggers in 93.567296 sec, syncing history]
      zabbix   1970438  0.1  0.9 772128 156484 ?       S    15:48   0:02 /usr/sbin/zabbix_server: history syncer #2 [processed 835 values, 1296 triggers in 87.592510 sec, syncing history]
      zabbix   1970439  0.1  1.0 772836 165108 ?       S    15:48   0:02 /usr/sbin/zabbix_server: history syncer #3 [processed 885 values, 1387 triggers in 76.867496 sec, syncing history]
      zabbix   1970440  0.1  1.1 790540 184104 ?       S    15:48   0:02 /usr/sbin/zabbix_server: history syncer #4 [processed 834 values, 1362 triggers in 92.601755 sec, syncing history]
      zabbix   1970441  0.0  0.0 678236  7456 ?        S    15:48   0:00 /usr/sbin/zabbix_server: escalator #1 [terminated]
      zabbix   1970442  0.0  0.0 678236  7456 ?        S    15:48   0:00 /usr/sbin/zabbix_server: proxy poller #1 [terminated]
      zabbix   1970443  0.0  0.0 678236  4236 ?        S    15:48   0:00 /usr/sbin/zabbix_server: self-monitoring #1 [terminated]
      zabbix   1970444  0.0  0.0 678236  5940 ?        S    15:48   0:00 /usr/sbin/zabbix_server: task manager #1 [terminated]
      zabbix   1970445  0.0  0.1 688392 29760 ?        S    15:48   0:00 /usr/sbin/zabbix_server: poller #1 [got 6 values in 0.285501 sec, getting values]
      zabbix   1970446  0.0  0.1 688360 29992 ?        S    15:48   0:00 /usr/sbin/zabbix_server: poller #2 [got 60 values in 0.734005 sec, getting values]
      zabbix   1970447  0.0  0.1 690916 31224 ?        S    15:48   0:00 /usr/sbin/zabbix_server: poller #3 [got 3 values in 0.142524 sec, getting values]
      zabbix   1970448  0.0  0.1 688488 29732 ?        S    15:48   0:00 /usr/sbin/zabbix_server: poller #4 [got 0 values in 0.000010 sec, getting values]
      zabbix   1970449  0.0  0.1 688376 29704 ?        S    15:48   0:00 /usr/sbin/zabbix_server: poller #5 [got 3 values in 0.212871 sec, getting values]
      zabbix   1970450  0.0  0.0 678236  8120 ?        S    15:48   0:00 /usr/sbin/zabbix_server: unreachable poller #1 [terminated]
      zabbix   1970451  0.0  0.0 678236  8644 ?        S    15:48   0:00 /usr/sbin/zabbix_server: unreachable poller #2 [terminated]
      zabbix   1970452  0.0  0.0 678236  8644 ?        S    15:48   0:00 /usr/sbin/zabbix_server: unreachable poller #3 [terminated]
      zabbix   1970453  0.2  0.3 711668 55552 ?        S    15:48   0:05 /usr/sbin/zabbix_server: trapper #1 [processing data]
      zabbix   1970454  0.2  0.2 700244 47196 ?        S    15:48   0:05 /usr/sbin/zabbix_server: trapper #2 [processing data]
      zabbix   1970455  0.2  0.4 700560 67724 ?        S    15:48   0:05 /usr/sbin/zabbix_server: trapper #3 [processing data]
      zabbix   1970456  0.2  0.3 711752 55624 ?        S    15:48   0:05 /usr/sbin/zabbix_server: trapper #4 [processing data]
      zabbix   1970457  0.2  0.3 711676 55336 ?        S    15:48   0:05 /usr/sbin/zabbix_server: trapper #5 [processing data]
      zabbix   1970458  0.0  0.0 681744  6756 ?        S    15:48   0:00 /usr/sbin/zabbix_server: icmp pinger #1 [terminated]
      zabbix   1970459  0.0  0.0 681744  6756 ?        S    15:48   0:00 /usr/sbin/zabbix_server: icmp pinger #2 [terminated]
      zabbix   1970460  0.0  0.0 681744  6756 ?        S    15:48   0:00 /usr/sbin/zabbix_server: icmp pinger #3 [terminated]
      zabbix   1970461  0.0  0.0 678236  5976 ?        S    15:48   0:00 /usr/sbin/zabbix_server: alert syncer #1 [terminated]
      zabbix   1970462  0.0  0.1 678380 20216 ?        S    15:48   0:00 /usr/sbin/zabbix_server: history poller #1 [got 1 values in 0.001266 sec, getting values]
      zabbix   1970463  0.0  0.1 678380 20476 ?        S    15:48   0:00 /usr/sbin/zabbix_server: history poller #2 [got 8 values in 0.001009 sec, getting values]
      zabbix   1970464  0.0  0.1 678380 20460 ?        S    15:48   0:00 /usr/sbin/zabbix_server: history poller #3 [got 13 values in 0.000897 sec, getting values]
      zabbix   1970465  0.0  0.1 678380 20196 ?        S    15:48   0:00 /usr/sbin/zabbix_server: history poller #4 [got 6 values in 0.001527 sec, getting values]
      zabbix   1970466  0.0  0.1 678380 20472 ?        S    15:48   0:00 /usr/sbin/zabbix_server: history poller #5 [got 7 values in 0.001050 sec, getting values]
      zabbix   1970472  0.0  0.0 678236  5936 ?        S    15:48   0:00 /usr/sbin/zabbix_server: trigger housekeeper #1 [terminated]
      zabbix   1970473  0.0  0.0 678236  5976 ?        S    15:48   0:00 /usr/sbin/zabbix_server: odbc poller #1 [terminated]
      root     1978597  0.0  0.0   9032   648 pts/0    R+   16:27   0:00 grep --color=auto zabbix_server
       

      Expected:

      $ sudo systemctl cat zabbix-server
      
      # /etc/systemd/system/zabbix-server.service
      [Unit]
      Description=Zabbix Server
      After=syslog.target
      After=network.target
      After=postgresql.service
      
      [Service]
      Environment="CONFFILE=/etc/zabbix/zabbix_server.conf"
      EnvironmentFile=-/etc/default/zabbix-server
      Type=forking
      Restart=on-failure
      PIDFile=/run/zabbix/zabbix_server.pid
      KillMode=control-group
      ExecStart=/usr/sbin/zabbix_server -c $CONFFILE
      ExecStop=/bin/sh -c '[ -n "$1" ] && kill -s TERM "$1"' -- "$MAINPID"
      RestartSec=10s
      TimeoutSec=infinity
      
      [Install]
      WantedBy=multi-user.target
      
      # /etc/systemd/system/zabbix-server.service.d/override.conf
      [Service]
      TimeoutSec=5s
       

      Adding the TimeoutSec=5s "fixed" it, but ZBX-15602 indicates that the infinity timeout is there to help diagnose missing database dependencies on shutdown.

      Also https://support.zabbix.com/browse/ZBX-16078

      Attachments

        Issue Links

          Activity

            People

              jlambda Juris Lambda
              c.h. C H
              Votes:
              0 Vote for this issue
              Watchers:
              3 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: