-
Incident report
-
Resolution: Commercial support required
-
Critical
-
None
-
3.4.3
-
None
Hello,
We are facing an issue with our Zabbix Server after upgrade from 3.2.9 to 3.4.3. Server is restarting continuously.
Firstly we didn't take into consideration that StartPreprocessors parameter should have been tuned up and Preprocessor was consuming 100% time of one of the CPUs. After changing to StartPreprocessors=300 this line appears in error_log:
zabbix_server [7327]: failed to open log file: [24] Too many open files zabbix_server [7327]: failed to write [cannot accept incoming IPC connection: [24] Too many open files] into log file
If changing StartPreprocessors to 100, Zabbix Server is still restarting with the same error message but once per 15-20 minutes.
These parameters had been changed but it doesn't solve the issue:
[root@zabbix ~]# cat /proc/sys/fs/file-max 2097152
[root@zabbix ~]# cat /etc/sysctl.conf # System default settings live in /usr/lib/sysctl.d/00-system.conf. # To override those settings, enter new settings here, or in an /etc/sysctl.d/<name>.conf file # # For more information, see sysctl.conf(5) and sysctl.d(5). dev.raid.speed_limit_min = 10000 dev.raid.speed_limit_max = 500000 vm.swappiness=1 vm.vfs_cache_pressure=50 net.ipv4.ip_no_pmtu_disc=1 fs.file-max = 2097152
[root@zabbix ~]# vim /etc/security/limits.conf * soft nproc 500000 * hard nproc 500000 * soft nofile 500000 * hard nofile 500000
[root@zabbix ~]# su - zabbix -c 'ulimit -aHS' -s '/bin/bash' su: warning: cannot change directory to /var/lib/zabbix: No such file or directory core file size (blocks, -c) 0 data seg size (kbytes, -d) unlimited scheduling priority (-e) 0 file size (blocks, -f) unlimited pending signals (-i) 256296 max locked memory (kbytes, -l) 64 max memory size (kbytes, -m) unlimited open files (-n) 500000 pipe size (512 bytes, -p) 8 POSIX message queues (bytes, -q) 819200 real-time priority (-r) 0 stack size (kbytes, -s) 8192 cpu time (seconds, -t) unlimited max user processes (-u) 4096 virtual memory (kbytes, -v) unlimited file locks (-x) unlimited
[root@zabbix ~]# su - root -c 'ulimit -aHS' -s '/bin/bash' core file size (blocks, -c) 0 data seg size (kbytes, -d) unlimited scheduling priority (-e) 0 file size (blocks, -f) unlimited pending signals (-i) 256296 max locked memory (kbytes, -l) 64 max memory size (kbytes, -m) unlimited open files (-n) 500000 pipe size (512 bytes, -p) 8 POSIX message queues (bytes, -q) 819200 real-time priority (-r) 0 stack size (kbytes, -s) 8192 cpu time (seconds, -t) unlimited max user processes (-u) 500000 virtual memory (kbytes, -v) unlimited file locks (-x) unlimited
zabbix_server.conf:
LogFile=/var/log/zabbix/zabbix_server.log LogFileSize=100 DebugLevel=3 PidFile=/var/run/zabbix/zabbix_server.pid DBName=zabbix DBUser=zabbix StartPollers=750 StartPreprocessors=300 StartPollersUnreachable=200 StartTrappers=20 StartPingers=20 StartDiscoverers=250 StartTimers=3 SNMPTrapperFile=/var/log/snmptrap/snmptrap.log HousekeepingFrequency=0 MaxHousekeeperDelete=1000 CacheSize=2048M CacheUpdateFrequency=360 StartDBSyncers=2 HistoryCacheSize=512M HistoryIndexCacheSize=128M TrendCacheSize=128M ValueCacheSize=512M Timeout=30 UnreachablePeriod=360 UnavailableDelay=600 UnreachableDelay=120 AlertScriptsPath=/usr/lib/zabbix/alertscripts ExternalScripts=/usr/lib/zabbix/externalscripts LogSlowQueries=3000
Zabbix System is all-in-one currently:
[root@zabbix ~]# df -h Filesystem Size Used Avail Use% Mounted on /dev/md127 280G 8.9G 272G 4% / devtmpfs 32G 0 32G 0% /dev tmpfs 32G 0 32G 0% /dev/shm tmpfs 32G 8.8M 32G 1% /run tmpfs 32G 0 32G 0% /sys/fs/cgroup /dev/md125 954G 179G 775G 19% /db /dev/md126 2.0G 254M 1.8G 13% /boot /dev/md2 1.6T 792G 783G 51% /mnt/lstorage /dev/sdb3 200M 0 200M 0% /boot/efi2 /dev/sda1 200M 9.8M 191M 5% /boot/efi tmpfs 6.3G 0 6.3G 0% /run/user/0
Intel(R) Xeon(R) CPU E5-2620 v4 @ 2.10GHz
[root@zabbix ~]# free
total used free shared buff/cache available
Mem: 65696748 24664656 39955132 460668 1076960 39956428
Swap: 5915644 0 5915644
Looking for help ASAP!
- is duplicated by
-
-
- Closed
-