I don't have any scheduled cronjob or simila that cleans /tmp
I've tried anyway to change the socketdir to /opt/zabbix (chowned zabbix.zabbix) with no success
Closed as duplicate of ZBX-14660.
Dear cotolez, pay attention to the Footnotes at the very bottom of the page. It's difficult to say for sure without the full log file and configuration file, but I'd guess you try to start too many Zabbix processes.
Thanks for point that out.
I dont' think that this is a duplicate of https://support.zabbix.com/browse/ZBX-14660 since SeLinux is Disabled
This zabbix server installation is running in HA with another instance (that instance is running fine).
The configurations are duplicated.
SeLinux is disabled
this is ulimit -a (identical on both machines)
core file size (blocks, -c) 0 data seg size (kbytes, -d) unlimited scheduling priority (-e) 0 file size (blocks, -f) unlimited pending signals (-i) 514599 max locked memory (kbytes, -l) 64 max memory size (kbytes, -m) unlimited open files (-n) 102400 pipe size (512 bytes, -p) 8 POSIX message queues (bytes, -q) 819200 real-time priority (-r) 0 stack size (kbytes, -s) 8192 cpu time (seconds, -t) unlimited max user processes (-u) 514599 virtual memory (kbytes, -v) unlimited file locks (-x) unlimited
Open files has been raised from 1024 to 102400
this is the output of "ps aux | grep zabbix_server"
zabbix 15604 1.3 1.1 389256 48024 ? S 14:27 0:01 /usr/local/sbin/zabbix_server zabbix 15607 0.6 1.0 389252 43256 ? S 14:27 0:00 /usr/local/sbin/zabbix_server: configuration syncer [synced configuration in 2.238462 sec, idle 60 sec] zabbix 15608 0.0 0.0 389256 2156 ? S 14:27 0:00 /usr/local/sbin/zabbix_server: alerter #1 started zabbix 15609 0.0 0.0 389256 2156 ? S 14:27 0:00 /usr/local/sbin/zabbix_server: alerter #2 started zabbix 15610 0.0 0.0 389256 2156 ? S 14:27 0:00 /usr/local/sbin/zabbix_server: alerter #3 started zabbix 15611 0.0 0.0 389256 1912 ? S 14:27 0:00 /usr/local/sbin/zabbix_server: housekeeper [startup idle for 30 minutes] zabbix 15612 2.1 1.4 416592 59328 ? S 14:27 0:01 /usr/local/sbin/zabbix_server: timer #1 [processed 4021 triggers, 0 events in 0.128787 sec, 0 maintenances in 0.000000 sec, idle 30 sec] zabbix 15613 0.7 0.2 406844 9556 ? S 14:27 0:00 /usr/local/sbin/zabbix_server: http poller #1 [got 2 values in 0.457350 sec, getting values] zabbix 15614 0.8 0.2 406500 9576 ? S 14:27 0:00 /usr/local/sbin/zabbix_server: http poller #2 [got 0 values in 0.000864 sec, idle 5 sec] zabbix 15615 0.0 0.0 486752 3892 ? S 14:27 0:00 /usr/local/sbin/zabbix_server: discoverer #1 [processed 0 rules in 0.000721 sec, idle 60 sec] zabbix 15616 0.8 1.0 397300 43684 ? S 14:27 0:00 /usr/local/sbin/zabbix_server: history syncer #1 [synced 0 items in 0.000045 sec, idle 1 sec] zabbix 15617 0.9 1.0 398672 45120 ? S 14:27 0:00 /usr/local/sbin/zabbix_server: history syncer #2 [synced 2 items in 0.002554 sec, syncing history] zabbix 15618 0.9 1.2 405252 52396 ? S 14:27 0:00 /usr/local/sbin/zabbix_server: history syncer #3 [synced 4 items in 0.012763 sec, idle 1 sec] zabbix 15619 0.9 1.4 413684 59472 ? S 14:27 0:00 /usr/local/sbin/zabbix_server: history syncer #4 [synced 0 items in 0.000046 sec, idle 1 sec] zabbix 15620 0.0 0.0 389256 2552 ? S 14:27 0:00 /usr/local/sbin/zabbix_server: escalator #1 [processed 0 escalations in 0.001681 sec, idle 3 sec] zabbix 15621 0.0 0.0 389256 2360 ? S 14:27 0:00 /usr/local/sbin/zabbix_server: snmp trapper [processed data in 0.000032 sec, idle 1 sec] zabbix 15622 0.0 0.0 389256 2328 ? S 14:27 0:00 /usr/local/sbin/zabbix_server: proxy poller #1 [exchanged data with 0 proxies in 0.000036 sec, idle 5 sec] zabbix 15623 0.0 0.0 389256 2152 ? S 14:27 0:00 /usr/local/sbin/zabbix_server: self-monitoring [processed data in 0.000040 sec, idle 1 sec] zabbix 15624 0.0 0.0 389256 2368 ? S 14:27 0:00 /usr/local/sbin/zabbix_server: task manager [processed 0 task(s) in 0.000318 sec, idle 5 sec] zabbix 15625 0.2 0.2 489020 8452 ? S 14:27 0:00 /usr/local/sbin/zabbix_server: poller #1 [got 0 values in 0.000028 sec, idle 1 sec] zabbix 15626 0.3 0.2 489020 10024 ? S 14:27 0:00 /usr/local/sbin/zabbix_server: poller #2 [got 0 values in 0.000015 sec, idle 1 sec] zabbix 15627 0.3 0.2 489020 10276 ? S 14:27 0:00 /usr/local/sbin/zabbix_server: poller #3 [got 0 values in 0.000031 sec, getting values] zabbix 15628 0.2 0.2 489020 9012 ? S 14:27 0:00 /usr/local/sbin/zabbix_server: poller #4 [got 0 values in 0.000030 sec, getting values] zabbix 15629 0.4 0.2 492064 12404 ? S 14:27 0:00 /usr/local/sbin/zabbix_server: poller #5 [got 2 values in 0.017872 sec, getting values] zabbix 15630 0.7 0.2 491092 12240 ? S 14:27 0:00 /usr/local/sbin/zabbix_server: poller #6 [got 1 values in 0.013115 sec, getting values] zabbix 15631 0.3 0.2 491096 11772 ? S 14:27 0:00 /usr/local/sbin/zabbix_server: poller #7 [got 4 values in 0.015075 sec, getting values] zabbix 15632 0.4 0.2 489952 10320 ? S 14:27 0:00 /usr/local/sbin/zabbix_server: poller #8 [got 0 values in 0.000027 sec, idle 1 sec] zabbix 15633 0.4 0.3 490004 13404 ? S 14:27 0:00 /usr/local/sbin/zabbix_server: poller #9 [got 12 values in 0.229332 sec, getting values] zabbix 15634 0.1 0.2 489020 8916 ? S 14:27 0:00 /usr/local/sbin/zabbix_server: poller #10 [got 0 values in 0.000026 sec, idle 1 sec] zabbix 15635 0.5 0.6 489196 25396 ? S 14:27 0:00 /usr/local/sbin/zabbix_server: poller #11 [got 9 values in 0.022920 sec, getting values] zabbix 15636 0.5 0.2 489020 9844 ? S 14:27 0:00 /usr/local/sbin/zabbix_server: poller #12 [got 0 values in 0.000049 sec, idle 1 sec] zabbix 15637 0.7 0.6 489876 25452 ? S 14:27 0:00 /usr/local/sbin/zabbix_server: poller #13 [got 3 values in 0.004794 sec, getting values] zabbix 15638 0.3 0.2 489052 9548 ? S 14:27 0:00 /usr/local/sbin/zabbix_server: poller #14 [got 0 values in 0.000027 sec, getting values] zabbix 15639 0.1 0.2 489020 9400 ? S 14:27 0:00 /usr/local/sbin/zabbix_server: poller #15 [got 0 values in 0.000047 sec, getting values] zabbix 15640 0.4 0.2 489964 11504 ? S 14:27 0:00 /usr/local/sbin/zabbix_server: poller #16 [got 0 values in 0.000033 sec, getting values] zabbix 15641 0.4 0.2 490000 12032 ? S 14:27 0:00 /usr/local/sbin/zabbix_server: poller #17 [got 8 values in 0.016944 sec, idle 1 sec] zabbix 15642 0.3 0.1 489960 8340 ? S 14:27 0:00 /usr/local/sbin/zabbix_server: poller #18 [got 0 values in 0.000036 sec, idle 1 sec] zabbix 15643 0.6 0.2 490052 11472 ? S 14:27 0:00 /usr/local/sbin/zabbix_server: poller #19 [got 1 values in 0.003761 sec, getting values] zabbix 15644 0.6 0.2 489068 12572 ? S 14:27 0:00 /usr/local/sbin/zabbix_server: poller #20 [got 1 values in 0.088769 sec, getting values] zabbix 15645 0.6 0.2 491092 11784 ? S 14:27 0:00 /usr/local/sbin/zabbix_server: poller #21 [got 13 values in 0.031699 sec, getting values] zabbix 15646 0.5 0.3 492076 12880 ? S 14:27 0:00 /usr/local/sbin/zabbix_server: poller #22 [got 0 values in 0.000031 sec, getting values] zabbix 15647 0.3 0.2 489020 8816 ? S 14:27 0:00 /usr/local/sbin/zabbix_server: poller #23 [got 0 values in 0.000017 sec, idle 1 sec] zabbix 15648 0.4 0.2 489016 9448 ? S 14:27 0:00 /usr/local/sbin/zabbix_server: poller #24 [got 0 values in 0.000027 sec, getting values] zabbix 15649 0.4 0.2 489016 8756 ? S 14:27 0:00 /usr/local/sbin/zabbix_server: poller #25 [got 0 values in 0.000017 sec, idle 1 sec] zabbix 15650 0.0 0.0 488196 3892 ? S 14:27 0:00 /usr/local/sbin/zabbix_server: unreachable poller #1 [got 0 values in 0.000046 sec, idle 5 sec] zabbix 15651 0.0 0.0 488196 3892 ? S 14:27 0:00 /usr/local/sbin/zabbix_server: unreachable poller #2 [got 0 values in 0.000028 sec, idle 5 sec] zabbix 15652 0.0 0.0 488196 3892 ? S 14:27 0:00 /usr/local/sbin/zabbix_server: unreachable poller #3 [got 0 values in 0.000031 sec, idle 5 sec] zabbix 15653 0.0 0.0 488196 3892 ? S 14:27 0:00 /usr/local/sbin/zabbix_server: unreachable poller #4 [got 0 values in 0.000035 sec, idle 5 sec] zabbix 15654 0.0 0.0 488196 3892 ? S 14:27 0:00 /usr/local/sbin/zabbix_server: unreachable poller #5 [got 0 values in 0.000041 sec, idle 5 sec] zabbix 15655 0.0 0.0 488196 3892 ? S 14:27 0:00 /usr/local/sbin/zabbix_server: unreachable poller #6 [got 0 values in 0.000047 sec, idle 5 sec] zabbix 15656 0.0 0.1 488196 4412 ? S 14:27 0:00 /usr/local/sbin/zabbix_server: unreachable poller #7 [got 0 values in 0.000040 sec, idle 5 sec] zabbix 15657 0.0 0.0 488196 3892 ? S 14:27 0:00 /usr/local/sbin/zabbix_server: unreachable poller #8 [got 0 values in 0.000046 sec, idle 5 sec] zabbix 15658 0.0 0.1 488400 4416 ? S 14:27 0:00 /usr/local/sbin/zabbix_server: unreachable poller #9 [got 0 values in 0.000039 sec, getting values] zabbix 15659 0.0 0.0 488196 3892 ? S 14:27 0:00 /usr/local/sbin/zabbix_server: unreachable poller #10 [got 0 values in 0.000025 sec, idle 5 sec] zabbix 15660 0.0 0.0 488196 3892 ? S 14:27 0:00 /usr/local/sbin/zabbix_server: unreachable poller #11 [got 0 values in 0.000037 sec, idle 5 sec] zabbix 15661 0.0 0.0 488196 3892 ? S 14:27 0:00 /usr/local/sbin/zabbix_server: unreachable poller #12 [got 0 values in 0.000040 sec, idle 5 sec] zabbix 15662 0.0 0.0 488196 3892 ? S 14:27 0:00 /usr/local/sbin/zabbix_server: unreachable poller #13 [got 0 values in 0.000037 sec, idle 5 sec] zabbix 15663 0.0 0.0 488196 3892 ? S 14:27 0:00 /usr/local/sbin/zabbix_server: unreachable poller #14 [got 0 values in 0.000066 sec, idle 5 sec] zabbix 15664 0.0 0.1 488400 4940 ? S 14:27 0:00 /usr/local/sbin/zabbix_server: unreachable poller #15 [got 0 values in 0.000042 sec, getting values] zabbix 15665 0.2 0.4 397592 20796 ? S 14:27 0:00 /usr/local/sbin/zabbix_server: trapper #1 [processed data in 0.000284 sec, waiting for connection] zabbix 15666 0.1 0.3 392360 16344 ? S 14:27 0:00 /usr/local/sbin/zabbix_server: trapper #2 [processed data in 0.000138 sec, waiting for connection] zabbix 15667 0.2 0.4 397592 17504 ? S 14:27 0:00 /usr/local/sbin/zabbix_server: trapper #3 [processed data in 0.001448 sec, waiting for connection] zabbix 15668 0.3 0.5 397596 21104 ? S 14:27 0:00 /usr/local/sbin/zabbix_server: trapper #4 [processed data in 0.001267 sec, waiting for connection] zabbix 15669 0.2 0.3 397592 15684 ? S 14:27 0:00 /usr/local/sbin/zabbix_server: trapper #5 [processed data in 0.000185 sec, waiting for connection] zabbix 15670 0.2 0.4 395484 17896 ? S 14:27 0:00 /usr/local/sbin/zabbix_server: trapper #6 [processed data in 0.000262 sec, waiting for connection] zabbix 15671 0.1 0.3 395520 13704 ? S 14:27 0:00 /usr/local/sbin/zabbix_server: trapper #7 [processed data in 0.000162 sec, waiting for connection] zabbix 15672 0.2 0.8 394472 37016 ? S 14:27 0:00 /usr/local/sbin/zabbix_server: trapper #8 [processed data in 0.000122 sec, waiting for connection] zabbix 15673 0.3 0.4 394488 20068 ? S 14:27 0:00 /usr/local/sbin/zabbix_server: trapper #9 [processed data in 0.000161 sec, waiting for connection] zabbix 15674 0.1 0.4 397592 17500 ? S 14:27 0:00 /usr/local/sbin/zabbix_server: trapper #10 [processed data in 0.000157 sec, waiting for connection] zabbix 15675 0.2 0.4 392532 18120 ? S 14:27 0:00 /usr/local/sbin/zabbix_server: trapper #11 [processed data in 0.000112 sec, waiting for connection] zabbix 15676 0.1 0.3 394500 14508 ? S 14:27 0:00 /usr/local/sbin/zabbix_server: trapper #12 [processed data in 0.000838 sec, waiting for connection] zabbix 15677 0.3 0.8 394472 36960 ? S 14:27 0:00 /usr/local/sbin/zabbix_server: trapper #13 [processed data in 0.000176 sec, waiting for connection] zabbix 15678 0.3 0.5 397616 24492 ? S 14:27 0:00 /usr/local/sbin/zabbix_server: trapper #14 [processed data in 0.001595 sec, waiting for connection] zabbix 15679 0.1 0.3 395480 15772 ? S 14:27 0:00 /usr/local/sbin/zabbix_server: trapper #15 [processed data in 0.000174 sec, waiting for connection] zabbix 15680 0.0 0.1 390692 7428 ? S 14:27 0:00 /usr/local/sbin/zabbix_server: icmp pinger #1 [got 0 values in 0.000041 sec, idle 1 sec] zabbix 15681 0.0 0.1 390692 5184 ? S 14:27 0:00 /usr/local/sbin/zabbix_server: icmp pinger #2 [got 0 values in 0.000032 sec, idle 1 sec] zabbix 15682 0.0 0.1 390692 5552 ? S 14:27 0:00 /usr/local/sbin/zabbix_server: icmp pinger #3 [got 0 values in 0.000025 sec, idle 1 sec] zabbix 15683 0.0 0.1 390696 4776 ? S 14:27 0:00 /usr/local/sbin/zabbix_server: icmp pinger #4 [pinging hosts] zabbix 15684 0.0 0.1 390696 5672 ? S 14:27 0:00 /usr/local/sbin/zabbix_server: icmp pinger #5 [pinging hosts] zabbix 15685 0.0 0.0 390692 3680 ? S 14:27 0:00 /usr/local/sbin/zabbix_server: icmp pinger #6 [got 0 values in 0.000029 sec, idle 1 sec] zabbix 15686 0.0 0.1 390696 5904 ? S 14:27 0:00 /usr/local/sbin/zabbix_server: icmp pinger #7 [pinging hosts] zabbix 15687 0.0 0.1 390692 4620 ? S 14:27 0:00 /usr/local/sbin/zabbix_server: icmp pinger #8 [got 0 values in 0.000024 sec, idle 1 sec] zabbix 15688 0.0 0.1 390692 5864 ? S 14:27 0:00 /usr/local/sbin/zabbix_server: icmp pinger #9 [got 0 values in 0.000023 sec, idle 1 sec] zabbix 15689 0.0 0.1 390692 6724 ? S 14:27 0:00 /usr/local/sbin/zabbix_server: icmp pinger #10 [got 0 values in 0.000026 sec, idle 1 sec] zabbix 15690 0.0 0.1 390696 5424 ? S 14:27 0:00 /usr/local/sbin/zabbix_server: icmp pinger #11 [pinging hosts] zabbix 15691 0.0 0.1 390696 6672 ? S 14:27 0:00 /usr/local/sbin/zabbix_server: icmp pinger #12 [pinging hosts] zabbix 15692 0.0 0.1 390696 4660 ? S 14:27 0:00 /usr/local/sbin/zabbix_server: icmp pinger #13 [pinging hosts] zabbix 15693 0.0 0.1 390692 5436 ? S 14:27 0:00 /usr/local/sbin/zabbix_server: icmp pinger #14 [got 0 values in 0.000028 sec, idle 1 sec] zabbix 15694 0.0 0.1 390692 5008 ? S 14:27 0:00 /usr/local/sbin/zabbix_server: icmp pinger #15 [got 0 values in 0.000030 sec, idle 1 sec] zabbix 15695 0.0 0.0 389256 2564 ? S 14:27 0:00 /usr/local/sbin/zabbix_server: alert manager #1 [sent 0, failed 0 alerts, idle 5.008666 sec during 5.008792 sec] zabbix 15696 0.7 0.5 390048 22840 ? S 14:27 0:00 /usr/local/sbin/zabbix_server: preprocessing manager #1 [queued 0, processed 777 values, idle 4.994421 sec during 5.006001 sec] zabbix 15697 0.0 0.0 389256 2260 ? S 14:27 0:00 /usr/local/sbin/zabbix_server: preprocessing worker #1 started zabbix 15698 0.0 0.0 389256 2260 ? S 14:27 0:00 /usr/local/sbin/zabbix_server: preprocessing worker #2 started zabbix 15699 0.0 0.0 389256 2260 ? S 14:27 0:00 /usr/local/sbin/zabbix_server: preprocessing worker #3 started zabbix 15700 0.0 0.0 389256 2260 ? S 14:27 0:00 /usr/local/sbin/zabbix_server: preprocessing worker #4 started zabbix 15701 0.0 0.0 389256 2260 ? S 14:27 0:00 /usr/local/sbin/zabbix_server: preprocessing worker #5 started zabbix 15702 0.0 0.0 389256 2256 ? S 14:27 0:00 /usr/local/sbin/zabbix_server: preprocessing worker #6 started zabbix 16344 0.0 0.0 108220 1180 ? S 14:28 0:00 sh -c /usr/sbin/fping -C10 -p200 -t500 2>&1 </tmp/zabbix_server_15683.pinger zabbix 16360 0.0 0.0 108220 1176 ? S 14:28 0:00 sh -c /usr/sbin/fping -C10 -p200 -t500 2>&1 </tmp/zabbix_server_15686.pinger zabbix 16362 0.0 0.0 108220 1172 ? S 14:28 0:00 sh -c /usr/sbin/fping -C5 -p200 -t500 2>&1 </tmp/zabbix_server_15690.pinger zabbix 16364 0.0 0.0 108220 1176 ? S 14:28 0:00 sh -c /usr/sbin/fping -C5 -p200 -t500 2>&1 </tmp/zabbix_server_15691.pinger zabbix 16366 0.0 0.0 108220 1176 ? S 14:28 0:00 sh -c /usr/sbin/fping -C3 -p200 -t500 2>&1 </tmp/zabbix_server_15684.pinger zabbix 16376 0.0 0.0 108220 1176 ? S 14:28 0:00 sh -c /usr/sbin/fping -C5 -p200 -t500 2>&1 </tmp/zabbix_server_15692.pinger
This is the configuration file:
# This is a configuration file for Zabbix server daemon # To get more information about Zabbix, visit http://www.zabbix.com ############ GENERAL PARAMETERS ################# ### Option: ListenPort # Listen port for trapper. # # Mandatory: no # Range: 1024-32767 # Default: # ListenPort=10051 ### Option: SourceIP # Source IP address for outgoing connections. # # Mandatory: no # Default: # SourceIP= SourceIP= ##SNIP## ### Option: LogType # Specifies where log messages are written to: # system - syslog # file - file specified with LogFile parameter # console - standard output # # Mandatory: no # Default: # LogType=file ### Option: LogFile # Log file name for LogType 'file' parameter. # # Mandatory: no # Default: # LogFile= LogFile=/tmp/zabbix_server.log ### Option: LogFileSize # Maximum size of log file in MB. # 0 - disable automatic log rotation. # # Mandatory: no # Range: 0-1024 # Default: # LogFileSize=1 ### Option: DebugLevel # Specifies debug level: # 0 - basic information about starting and stopping of Zabbix processes # 1 - critical information # 2 - error information # 3 - warnings # 4 - for debugging (produces lots of information) # 5 - extended debugging (produces even more information) # # Mandatory: no # Range: 0-5 # Default: # DebugLevel=3 ### Option: PidFile # Name of PID file. # # Mandatory: no # Default: # PidFile=/tmp/zabbix_server.pid PidFile=/tmp/zabbix_server.pid ### Option: SocketDir # IPC socket directory. # Directory to store IPC sockets used by internal Zabbix services. # # Mandatory: no # Default: # SocketDir=/tmp SocketDir=/opt/zabbix ### Option: DBHost # Database host name. # If set to localhost, socket is used for MySQL. # If set to empty string, socket is used for PostgreSQL. # # Mandatory: no # Default: # DBHost=localhost ### Option: DBName # Database name. # For SQLite3 path to database file must be provided. DBUser and DBPassword are ignored. # # Mandatory: yes # Default: # DBName= DBName=zabbix_db_v2 ### Option: DBSchema # Schema name. Used for IBM DB2 and PostgreSQL. # # Mandatory: no # Default: # DBSchema= ### Option: DBUser # Database user. Ignored for SQLite. # # Mandatory: no # Default: # DBUser= DBUser=zabbix ### Option: DBPassword # Database password. Ignored for SQLite. # Comment this line if no password is used. # # Mandatory: no # Default: # DBPassword= DBPassword=nw.zabbix ### Option: DBSocket # Path to MySQL socket. # # Mandatory: no # Default: # DBSocket=/tmp/mysql.sock DBSocket=/var/lib/mysql/mysql.sock ### Option: DBPort # Database port when not using local socket. Ignored for SQLite. # # Mandatory: no # Range: 1024-65535 # Default (for MySQL): # DBPort=3306 ############ ADVANCED PARAMETERS ################ ### Option: StartPollers # Number of pre-forked instances of pollers. # # Mandatory: no # Range: 0-1000 # Default: # StartPollers=5 StartPollers=25 ### Option: StartIPMIPollers # Number of pre-forked instances of IPMI pollers. # The IPMI manager process is automatically started when at least one IPMI poller is started. # # Mandatory: no # Range: 0-1000 # Default: # StartIPMIPollers=0 ### Option: StartPreprocessors # Number of pre-forked instances of preprocessing workers. # The preprocessing manager process is automatically started when preprocessor worker is started. # # Mandatory: no # Range: 1-1000 # Default: # StartPreprocessors=3 StartPreprocessors=6 ### Option: StartPollersUnreachable # Number of pre-forked instances of pollers for unreachable hosts (including IPMI and Java). # At least one poller for unreachable hosts must be running if regular, IPMI or Java pollers # are started. # # Mandatory: no # Range: 0-1000 # Default: # StartPollersUnreachable=1 StartPollersUnreachable=15 ### Option: StartTrappers # Number of pre-forked instances of trappers. # Trappers accept incoming connections from Zabbix sender, active agents and active proxies. # At least one trapper process must be running to display server availability and view queue # in the frontend. # # Mandatory: no # Range: 0-1000 # Default: # StartTrappers=5 StartTrappers=15 ### Option: StartPingers # Number of pre-forked instances of ICMP pingers. # # Mandatory: no # Range: 0-1000 # Default: # StartPingers=1 StartPingers=15 ### Option: StartDiscoverers # Number of pre-forked instances of discoverers. # # Mandatory: no # Range: 0-250 # Default: # StartDiscoverers=1 ### Option: StartHTTPPollers # Number of pre-forked instances of HTTP pollers. # # Mandatory: no # Range: 0-1000 # Default: # StartHTTPPollers=1 StartHTTPPollers=2 ### Option: StartTimers # Number of pre-forked instances of timers. # Timers process time-based trigger functions and maintenance periods. # Only the first timer process handles the maintenance periods. # # Mandatory: no # Range: 1-1000 # Default: # StartTimers=1 ### Option: StartEscalators # Number of pre-forked instances of escalators. # # Mandatory: no # Range: 0-100 # Default: # StartEscalators=1 ### Option: StartAlerters # Number of pre-forked instances of alerters. # Alerters send the notifications created by action operations. # # Mandatory: no # Range: 0-100 # Default: # StartAlerters=3 ### Option: JavaGateway # IP address (or hostname) of Zabbix Java gateway. # Only required if Java pollers are started. # # Mandatory: no # Default: # JavaGateway= ### Option: JavaGatewayPort # Port that Zabbix Java gateway listens on. # # Mandatory: no # Range: 1024-32767 # Default: # JavaGatewayPort=10052 ### Option: StartJavaPollers # Number of pre-forked instances of Java pollers. # # Mandatory: no # Range: 0-1000 # Default: # StartJavaPollers=0 ### Option: StartVMwareCollectors # Number of pre-forked vmware collector instances. # # Mandatory: no # Range: 0-250 # Default: # StartVMwareCollectors=0 ### Option: VMwareFrequency # How often Zabbix will connect to VMware service to obtain a new data. # # Mandatory: no # Range: 10-86400 # Default: # VMwareFrequency=60 ### Option: VMwarePerfFrequency # How often Zabbix will connect to VMware service to obtain performance data. # # Mandatory: no # Range: 10-86400 # Default: # VMwarePerfFrequency=60 ### Option: VMwareCacheSize # Size of VMware cache, in bytes. # Shared memory size for storing VMware data. # Only used if VMware collectors are started. # # Mandatory: no # Range: 256K-2G # Default: # VMwareCacheSize=8M ### Option: VMwareTimeout # Specifies how many seconds vmware collector waits for response from VMware service. # # Mandatory: no # Range: 1-300 # Default: # VMwareTimeout=10 ### Option: SNMPTrapperFile # Temporary file used for passing data from SNMP trap daemon to the server. # Must be the same as in zabbix_trap_receiver.pl or SNMPTT configuration file. # # Mandatory: no # Default: # SNMPTrapperFile=/tmp/zabbix_traps.tmp SNMPTrapperFile=/var/log/snmptt/snmptt.log ### Option: StartSNMPTrapper # If 1, SNMP trapper process is started. # # Mandatory: no # Range: 0-1 # Default: # StartSNMPTrapper=0 StartSNMPTrapper=1 ### Option: ListenIP # List of comma delimited IP addresses that the trapper should listen on. # Trapper will listen on all network interfaces if this parameter is missing. # # Mandatory: no # Default: # ListenIP=0.0.0.0 # ListenIP=127.0.0.1 ListenIP=127.0.0.1,##SNIP## ### Option: HousekeepingFrequency # How often Zabbix will perform housekeeping procedure (in hours). # Housekeeping is removing outdated information from the database. # To prevent Housekeeper from being overloaded, no more than 4 times HousekeepingFrequency # hours of outdated information are deleted in one housekeeping cycle, for each item. # To lower load on server startup housekeeping is postponed for 30 minutes after server start. # With HousekeepingFrequency=0 the housekeeper can be only executed using the runtime control option. # In this case the period of outdated information deleted in one housekeeping cycle is 4 times the # period since the last housekeeping cycle, but not less than 4 hours and not greater than 4 days. # # Mandatory: no # Range: 0-24 # Default: # HousekeepingFrequency=1 ### Option: MaxHousekeeperDelete # The table "housekeeper" contains "tasks" for housekeeping procedure in the format: # [housekeeperid], [tablename], [field], [value]. # No more than 'MaxHousekeeperDelete' rows (corresponding to [tablename], [field], [value]) # will be deleted per one task in one housekeeping cycle. # SQLite3 does not use this parameter, deletes all corresponding rows without a limit. # If set to 0 then no limit is used at all. In this case you must know what you are doing! # # Mandatory: no # Range: 0-1000000 # Default: # MaxHousekeeperDelete=5000 MaxHousekeeperDelete=15000 ### Option: CacheSize # Size of configuration cache, in bytes. # Shared memory size for storing host, item and trigger data. # # Mandatory: no # Range: 128K-8G # Default: # CacheSize=8M CacheSize=64M ### Option: CacheUpdateFrequency # How often Zabbix will perform update of configuration cache, in seconds. # # Mandatory: no # Range: 1-3600 # Default: # CacheUpdateFrequency=60 ### Option: StartDBSyncers # Number of pre-forked instances of DB Syncers. # # Mandatory: no # Range: 1-100 # Default: # StartDBSyncers=4 ### Option: HistoryCacheSize # Size of history cache, in bytes. # Shared memory size for storing history data. # # Mandatory: no # Range: 128K-2G # Default: # HistoryCacheSize=16M ### Option: HistoryIndexCacheSize # Size of history index cache, in bytes. # Shared memory size for indexing history cache. # # Mandatory: no # Range: 128K-2G # Default: # HistoryIndexCacheSize=4M ### Option: TrendCacheSize # Size of trend cache, in bytes. # Shared memory size for storing trends data. # # Mandatory: no # Range: 128K-2G # Default: # TrendCacheSize=4M TrendCacheSize=32M ### Option: ValueCacheSize # Size of history value cache, in bytes. # Shared memory size for caching item history data requests. # Setting to 0 disables value cache. # # Mandatory: no # Range: 0,128K-64G # Default: # ValueCacheSize=8M ValueCacheSize=64M ### Option: Timeout # Specifies how long we wait for agent, SNMP device or external check (in seconds). # # Mandatory: no # Range: 1-30 # Default: # Timeout=3 Timeout=30 ### Option: TrapperTimeout # Specifies how many seconds trapper may spend processing new data. # # Mandatory: no # Range: 1-300 # Default: # TrapperTimeout=300 ### Option: UnreachablePeriod # After how many seconds of unreachability treat a host as unavailable. # # Mandatory: no # Range: 1-3600 # Default: # UnreachablePeriod=45 ### Option: UnavailableDelay # How often host is checked for availability during the unavailability period, in seconds. # # Mandatory: no # Range: 1-3600 # Default: # UnavailableDelay=60 ### Option: UnreachableDelay # How often host is checked for availability during the unreachability period, in seconds. # # Mandatory: no # Range: 1-3600 # Default: # UnreachableDelay=15 ### Option: AlertScriptsPath # Full path to location of custom alert scripts. # Default depends on compilation options. # # Mandatory: no # Default: # AlertScriptsPath=${datadir}/zabbix/alertscripts ### Option: ExternalScripts # Full path to location of external scripts. # Default depends on compilation options. # # Mandatory: no # Default: # ExternalScripts=${datadir}/zabbix/externalscripts ### Option: FpingLocation # Location of fping. # Make sure that fping binary has root ownership and SUID flag set. # # Mandatory: no # Default: # FpingLocation=/usr/sbin/fping ### Option: Fping6Location # Location of fping6. # Make sure that fping6 binary has root ownership and SUID flag set. # Make empty if your fping utility is capable to process IPv6 addresses. # # Mandatory: no # Default: # Fping6Location=/usr/sbin/fping6 ### Option: SSHKeyLocation # Location of public and private keys for SSH checks and actions. # # Mandatory: no # Default: # SSHKeyLocation= ### Option: LogSlowQueries # How long a database query may take before being logged (in milliseconds). # Only works if DebugLevel set to 3, 4 or 5. # 0 - don't log slow queries. # # Mandatory: no # Range: 1-3600000 # Default: # LogSlowQueries=0 LogSlowQueries=3000 ### Option: TmpDir # Temporary directory. # # Mandatory: no # Default: # TmpDir=/tmp ### Option: StartProxyPollers # Number of pre-forked instances of pollers for passive proxies. # # Mandatory: no # Range: 0-250 # Default: # StartProxyPollers=1 ### Option: ProxyConfigFrequency # How often Zabbix Server sends configuration data to a Zabbix Proxy in seconds. # This parameter is used only for proxies in the passive mode. # # Mandatory: no # Range: 1-3600*24*7 # Default: # ProxyConfigFrequency=3600 ### Option: ProxyDataFrequency # How often Zabbix Server requests history data from a Zabbix Proxy in seconds. # This parameter is used only for proxies in the passive mode. # # Mandatory: no # Range: 1-3600 # Default: # ProxyDataFrequency=1 ### Option: AllowRoot # Allow the server to run as 'root'. If disabled and the server is started by 'root', the server # will try to switch to the user specified by the User configuration option instead. # Has no effect if started under a regular user. # 0 - do not allow # 1 - allow # # Mandatory: no # Default: # AllowRoot=0 ### Option: User # Drop privileges to a specific, existing user on the system. # Only has effect if run as 'root' and AllowRoot is disabled. # # Mandatory: no # Default: # User=zabbix ### Option: Include # You may include individual files or all files in a directory in the configuration file. # Installing Zabbix will create include directory in /usr/local/etc, unless modified during the compile time. # # Mandatory: no # Default: # Include= # Include=/usr/local/etc/zabbix_server.general.conf # Include=/usr/local/etc/zabbix_server.conf.d/ # Include=/usr/local/etc/zabbix_server.conf.d/*.conf ### Option: SSLCertLocation # Location of SSL client certificates. # This parameter is used only in web monitoring. # # Mandatory: no # Default: # SSLCertLocation=${datadir}/zabbix/ssl/certs ### Option: SSLKeyLocation # Location of private keys for SSL client certificates. # This parameter is used only in web monitoring. # # Mandatory: no # Default: # SSLKeyLocation=${datadir}/zabbix/ssl/keys ### Option: SSLCALocation # Override the location of certificate authority (CA) files for SSL server certificate verification. # If not set, system-wide directory will be used. # This parameter is used only in web monitoring and SMTP authentication. # # Mandatory: no # Default: # SSLCALocation= ####### LOADABLE MODULES ####### ### Option: LoadModulePath # Full path to location of server modules. # Default depends on compilation options. # # Mandatory: no # Default: # LoadModulePath=${libdir}/modules ### Option: LoadModule # Module to load at server startup. Modules are used to extend functionality of the server. # Format: LoadModule=<module.so> # The modules must be located in directory specified by LoadModulePath. # It is allowed to include multiple LoadModule parameters. # # Mandatory: no # Default: # LoadModule= ####### TLS-RELATED PARAMETERS ####### ### Option: TLSCAFile # Full pathname of a file containing the top-level CA(s) certificates for # peer certificate verification. # # Mandatory: no # Default: # TLSCAFile= ### Option: TLSCRLFile # Full pathname of a file containing revoked certificates. # # Mandatory: no # Default: # TLSCRLFile= ### Option: TLSCertFile # Full pathname of a file containing the server certificate or certificate chain. # # Mandatory: no # Default: # TLSCertFile= ### Option: TLSKeyFile # Full pathname of a file containing the server private key. # # Mandatory: no # Default: # TLSKeyFile=
I can't find any resolution from https://support.zabbix.com/browse/ZBX-14660
1- The SeLinux on this machine is set to Disabled.
2- The error is different (Cannot connect to service "preprocessing": No such file or directory)
3- I've posted other configurations to help troubleshooting
4- There is a twin machine running fine with the same configurations
5- I read the footnote in this page The open file limit was increased from 1024 to 102400
The problem still exists
Any hint is very appreciated.
Thank you
Configuration file seems OK, nothing extreme there.
Could you provide a log file?
Yes, i have snipped the ip and the host names.
I know that there are many missing hosts in the server, and therefore a lot of row like "cannot send list of active cheks..."
Here it is:
15674:20190102:143615.372 cannot send list of active checks to "xxx.xxx.xxx.xxx": host [host.name.snipped] not found 15666:20190102:143617.271 cannot send list of active checks to "xxx.xxx.xxx.xxx": host [host.name.snipped] not found 15666:20190102:143617.554 cannot send list of active checks to "xxx.xxx.xxx.xxx": host [host.name.snipped] not found 15666:20190102:143618.286 cannot send list of active checks to "xxx.xxx.xxx.xxx": host [host.name.snipped] not found 15675:20190102:143619.337 cannot send list of active checks to "xxx.xxx.xxx.xxx": host [host.name.snipped] not found 15675:20190102:143623.781 cannot send list of active checks to "xxx.xxx.xxx.xxx": host [host.name.snipped] not found 15675:20190102:143624.663 cannot send list of active checks to "xxx.xxx.xxx.xxx": host [host.name.snipped] not found 15673:20190102:143633.885 cannot send list of active checks to "xxx.xxx.xxx.xxx": host [host.name.snipped] not found 15619:20190102:143714.372 error reason for "host.name.snipped:mysql.status[Slow_queries]" changed: Value "Slow_queries 146229" of type "string" is not suitable for value type "Numeric (unsigned)" 15617:20190102:143715.375 error reason for "host.name.snipped:mysql.status[Uptime]" changed: Value "Uptime 2590979" of type "string" is not suitable for value type "Numeric (unsigned)" 15679:20190102:143717.080 cannot send list of active checks to "xxx.xxx.xxx.xxx": host [host.name.snipped] not found 15676:20190102:143746.785 cannot send list of active checks to "xxx.xxx.xxx.xxx": host [host.name.snipped] not found 15673:20190102:143749.704 cannot send list of active checks to "xxx.xxx.xxx.xxx": host [mx1.] not found 15669:20190102:143750.509 cannot send list of active checks to "xxx.xxx.xxx.xxx": host [host.name.snipped] not found 15669:20190102:143750.634 cannot send list of active checks to "xxx.xxx.xxx.xxx": host [host.name.snipped] not found 15670:20190102:143751.665 cannot send list of active checks to "xxx.xxx.xxx.xxx": host [host.name.snipped] not found 15670:20190102:143751.860 cannot send list of active checks to "xxx.xxx.xxx.xxx": host [host.name.snipped] not monitored 15670:20190102:143752.347 cannot send list of active checks to "xxx.xxx.xxx.xxx": host [host.name.snipped] not monitored 15670:20190102:143752.550 cannot send list of active checks to "xxx.xxx.xxx.xxx": host [host.name.snipped] not monitored 15676:20190102:143752.962 cannot send list of active checks to "xxx.xxx.xxx.xxx": host [host.name.snipped] not monitored 15670:20190102:143753.033 cannot send list of active checks to "xxx.xxx.xxx.xxx": host [host.name.snipped] not found 15676:20190102:143753.162 cannot send list of active checks to "xxx.xxx.xxx.xxx": host [host.name.snipped] not found 15674:20190102:143753.258 cannot send list of active checks to "xxx.xxx.xxx.xxx": host [host.name.snipped] not found 15670:20190102:143753.431 cannot send list of active checks to "xxx.xxx.xxx.xxx": host [host.name.snipped] not monitored 15676:20190102:143753.431 cannot send list of active checks to "xxx.xxx.xxx.xxx": host [host.name.snipped] not monitored 15676:20190102:143753.671 cannot send list of active checks to "xxx.xxx.xxx.xxx": host [host.name.snipped] not monitored 15670:20190102:143753.672 cannot send list of active checks to "xxx.xxx.xxx.xxx": host [host.name.snipped] not found 15670:20190102:143754.037 cannot send list of active checks to "xxx.xxx.xxx.xxx": host [host.name.snipped] not found 15670:20190102:143754.061 cannot send list of active checks to "xxx.xxx.xxx.xxx": host [host.name.snipped] not monitored 15676:20190102:143754.088 cannot send list of active checks to "xxx.xxx.xxx.xxx": host [host.name.snipped] not monitored 15674:20190102:143754.090 cannot send list of active checks to "xxx.xxx.xxx.xxx": host [host.name.snipped] not found 15674:20190102:143754.254 cannot send list of active checks to "xxx.xxx.xxx.xxx": host [host.name.snipped] not found 15674:20190102:143754.300 cannot send list of active checks to "xxx.xxx.xxx.xxx": host [host.name.snipped] not monitored 15670:20190102:143754.935 cannot send list of active checks to "xxx.xxx.xxx.xxx": host [host.name.snipped] not found 15670:20190102:143754.944 cannot send list of active checks to "xxx.xxx.xxx.xxx": host [host.name.snipped] not found 15670:20190102:143755.113 cannot send list of active checks to "xxx.xxx.xxx.xxx": host [host.name.snipped] not monitored 15676:20190102:143755.460 cannot send list of active checks to "xxx.xxx.xxx.xxx": host [host.name.snipped] not found 15674:20190102:143755.746 cannot send list of active checks to "xxx.xxx.xxx.xxx": host [host.name.snipped] not monitored 15670:20190102:143756.364 cannot send list of active checks to "xxx.xxx.xxx.xxx": host [host.name.snipped] not monitored 15665:20190102:143756.511 cannot send list of active checks to "xxx.xxx.xxx.xxx": host [host.name.snipped] not found 15665:20190102:143756.593 cannot send list of active checks to "xxx.xxx.xxx.xxx": host [host.name.snipped] not monitored 15670:20190102:143756.641 cannot send list of active checks to "xxx.xxx.xxx.xxx": host [host.name.snipped] not monitored 15674:20190102:143756.848 cannot send list of active checks to "xxx.xxx.xxx.xxx": host [host.name.snipped] not monitored 15676:20190102:143756.997 cannot send list of active checks to "xxx.xxx.xxx.xxx": host [host.name.snipped] not found 15670:20190102:143757.003 cannot send list of active checks to "xxx.xxx.xxx.xxx": host [host.name.snipped] not found 15670:20190102:143757.199 cannot send list of active checks to "xxx.xxx.xxx.xxx": host [host.name.snipped] not monitored 15676:20190102:143758.341 cannot send list of active checks to "xxx.xxx.xxx.xxx": host [host.name.snipped] not found 15676:20190102:143758.608 cannot send list of active checks to "xxx.xxx.xxx.xxx": host [host.name.snipped] not found 15670:20190102:143759.599 cannot send list of active checks to "xxx.xxx.xxx.xxx": host [host.name.snipped] not found 15618:20190102:143801.674 item "host.name.snipped:rabbitmq.total_connections" became supported 15670:20190102:143802.544 cannot send list of active checks to "xxx.xxx.xxx.xxx": host [host.name.snipped] not found 15667:20190102:143808.026 cannot send list of active checks to "xxx.xxx.xxx.xxx": host [host.name.snipped] not monitored 15675:20190102:143808.290 cannot send list of active checks to "xxx.xxx.xxx.xxx": host [host.name.snipped] not found 15675:20190102:143809.109 cannot send list of active checks to "xxx.xxx.xxx.xxx": host [host.name.snipped] not found 15667:20190102:143810.007 cannot send list of active checks to "xxx.xxx.xxx.xxx": host [host.name.snipped] not monitored 15675:20190102:143812.208 cannot send list of active checks to "xxx.xxx.xxx.xxx": host [host.name.snipped] not monitored 15671:20190102:143813.421 cannot send list of active checks to "xxx.xxx.xxx.xxx": host [host.name.snipped] not monitored 15619:20190102:143813.736 error reason for "host.name.snipped:mysql.status[Slow_queries]" changed: Value "Slow_queries 146994" of type "string" is not suitable for value type "Numeric (unsigned)" 15616:20190102:143814.739 error reason for "host.name.snipped:mysql.status[Uptime]" changed: Value "Uptime 2591039" of type "string" is not suitable for value type "Numeric (unsigned)" 15671:20190102:143814.762 cannot send list of active checks to "xxx.xxx.xxx.xxx": host [host.name.snipped] not found 15671:20190102:143814.853 cannot send list of active checks to "xxx.xxx.xxx.xxx": host [host.name.snipped] not found 15674:20190102:143815.396 cannot send list of active checks to "xxx.xxx.xxx.xxx": host [host.name.snipped] not found 15677:20190102:143817.368 cannot send list of active checks to "xxx.xxx.xxx.xxx": host [host.name.snipped] not found 15677:20190102:143817.570 cannot send list of active checks to "xxx.xxx.xxx.xxx": host [host.name.snipped] not found 15670:20190102:143818.385 cannot send list of active checks to "xxx.xxx.xxx.xxx": host [host.name.snipped] not found 15670:20190102:143819.354 cannot send list of active checks to "xxx.xxx.xxx.xxx": host [host.name.snipped] not found 15676:20190102:143823.974 cannot send list of active checks to "xxx.xxx.xxx.xxx": host [host.name.snipped] not found 15676:20190102:143824.679 cannot send list of active checks to "xxx.xxx.xxx.xxx": host [host.name.snipped] not found 15678:20190102:143833.854 cannot send list of active checks to "xxx.xxx.xxx.xxx": host [host.name.snipped] not found 15658:20190102:143844.489 resuming SNMP agent checks on host "host.name.snipped": connection restored 15617:20190102:143913.159 error reason for "host.name.snipped:mysql.status[Slow_queries]" changed: Value "Slow_queries 147633" of type "string" is not suitable for value type "Numeric (unsigned)" 15616:20190102:143914.224 error reason for "host.name.snipped:mysql.status[Uptime]" changed: Value "Uptime 2591099" of type "string" is not suitable for value type "Numeric (unsigned)" 15666:20190102:143917.103 cannot send list of active checks to "xxx.xxx.xxx.xxx": host [host.name.snipped] not found 15658:20190102:143945.051 cannot connect to preprocessing service: Cannot connect to service "preprocessing": [2] No such file or directory. 15604:20190102:143945.056 One child process died (PID:15658,exitcode/signal:1). Exiting ... 15604:20190102:143947.063 syncing history data... 15604:20190102:143947.079 syncing history data done 15604:20190102:143947.079 syncing trend data... 15604:20190102:144000.504 syncing trend data done
Thank you, but that's not enough. I would like to see which process "cannot connect to service", for that I need to see the piece after startup when each process reports that it has started. So I don't actually need the whole log file, I just want messages by the preprocessing manager and the faulty preprocessing worker.
Sorry, here is the startup piece:
21799:20190102:144049.347 Starting Zabbix Server. Zabbix 3.4.15 (revision 86739). 21799:20190102:144049.347 ****** Enabled features ****** 21799:20190102:144049.347 SNMP monitoring: YES 21799:20190102:144049.347 IPMI monitoring: YES 21799:20190102:144049.347 Web monitoring: YES 21799:20190102:144049.347 VMware monitoring: YES 21799:20190102:144049.347 SMTP authentication: NO 21799:20190102:144049.347 Jabber notifications: YES 21799:20190102:144049.347 Ez Texting notifications: YES 21799:20190102:144049.347 ODBC: YES 21799:20190102:144049.347 SSH2 support: YES 21799:20190102:144049.347 IPv6 support: YES 21799:20190102:144049.347 TLS support: NO 21799:20190102:144049.347 ****************************** 21799:20190102:144049.347 using configuration file: /usr/local/etc/zabbix_server.conf 21799:20190102:144049.355 current database version (mandatory/optional): 03040000/03040007 21799:20190102:144049.355 required mandatory version: 03040000 21799:20190102:144052.209 server #0 started [main process] 21807:20190102:144052.210 server #1 started [configuration syncer #1] 21808:20190102:144052.210 server #2 started [alerter #1] 21809:20190102:144052.211 server #3 started [alerter #2] 21810:20190102:144052.212 server #4 started [alerter #3] 21811:20190102:144052.212 server #5 started [housekeeper #1] 21812:20190102:144052.213 server #6 started [timer #1] 21813:20190102:144052.214 server #7 started [http poller #1] 21814:20190102:144052.215 server #8 started [http poller #2] 21815:20190102:144052.215 server #9 started [discoverer #1] 21817:20190102:144052.219 server #11 started [history syncer #2] 21816:20190102:144052.219 server #10 started [history syncer #1] 21820:20190102:144052.220 server #14 started [escalator #1] 21818:20190102:144052.220 server #12 started [history syncer #3] 21819:20190102:144052.220 server #13 started [history syncer #4] 21821:20190102:144052.221 server #15 started [snmp trapper #1] 21822:20190102:144052.221 server #16 started [proxy poller #1] 21823:20190102:144052.223 server #17 started [self-monitoring #1] 21824:20190102:144052.223 server #18 started [task manager #1] 21825:20190102:144052.223 server #19 started [poller #1] 21826:20190102:144052.224 server #20 started [poller #2] 21827:20190102:144052.226 server #21 started [poller #3] 21828:20190102:144052.228 server #22 started [poller #4] 21832:20190102:144052.229 server #26 started [poller #8] 21831:20190102:144052.231 server #25 started [poller #7] 21829:20190102:144052.235 server #23 started [poller #5] 21833:20190102:144052.239 server #27 started [poller #9] 21830:20190102:144052.243 server #24 started [poller #6] 21837:20190102:144052.252 server #31 started [poller #13] 21838:20190102:144052.255 server #32 started [poller #14] 21836:20190102:144052.259 server #30 started [poller #12] 21834:20190102:144052.264 server #28 started [poller #10] 21835:20190102:144052.270 server #29 started [poller #11] 21842:20190102:144052.280 server #36 started [poller #18] 21841:20190102:144052.280 server #35 started [poller #17] 21839:20190102:144052.284 server #33 started [poller #15] 21840:20190102:144052.287 server #34 started [poller #16] 21843:20190102:144052.304 server #37 started [poller #19] 21844:20190102:144052.318 server #38 started [poller #20] 21845:20190102:144052.325 server #39 started [poller #21] 21846:20190102:144052.329 server #40 started [poller #22] 21850:20190102:144052.334 server #44 started [unreachable poller #1] 21849:20190102:144052.339 server #43 started [poller #25] 21848:20190102:144052.342 server #42 started [poller #24] 21847:20190102:144052.346 server #41 started [poller #23] 21851:20190102:144052.364 server #45 started [unreachable poller #2] 21852:20190102:144052.368 server #46 started [unreachable poller #3] 21854:20190102:144052.377 server #48 started [unreachable poller #5] 21853:20190102:144052.382 server #47 started [unreachable poller #4] 21858:20190102:144052.396 server #52 started [unreachable poller #9] 21855:20190102:144052.398 server #49 started [unreachable poller #6] 21857:20190102:144052.400 server #51 started [unreachable poller #8] 21856:20190102:144052.405 server #50 started [unreachable poller #7] 21862:20190102:144052.425 server #56 started [unreachable poller #13] 21859:20190102:144052.432 server #53 started [unreachable poller #10] 21861:20190102:144052.436 server #55 started [unreachable poller #12] 21860:20190102:144052.440 server #54 started [unreachable poller #11] 21864:20190102:144052.461 server #58 started [unreachable poller #15] 21865:20190102:144052.466 server #59 started [trapper #1] 21866:20190102:144052.466 server #60 started [trapper #2] 21863:20190102:144052.468 server #57 started [unreachable poller #14] 21868:20190102:144052.500 server #62 started [trapper #4] 21867:20190102:144052.500 server #61 started [trapper #3] 21869:20190102:144052.508 server #63 started [trapper #5] 21870:20190102:144052.511 server #64 started [trapper #6] 21871:20190102:144052.558 server #65 started [trapper #7] 21872:20190102:144052.587 server #66 started [trapper #8] 21873:20190102:144052.599 server #67 started [trapper #9] 21874:20190102:144052.648 server #68 started [trapper #10] 21875:20190102:144052.674 server #69 started [trapper #11] 21878:20190102:144052.676 server #72 started [trapper #14] 21877:20190102:144052.679 server #71 started [trapper #13] 21876:20190102:144052.682 server #70 started [trapper #12] 21879:20190102:144052.711 server #73 started [trapper #15] 21883:20190102:144052.711 server #75 started [icmp pinger #2] 21882:20190102:144052.721 server #74 started [icmp pinger #1] 21888:20190102:144052.752 server #76 started [icmp pinger #3] 21896:20190102:144052.767 server #78 started [icmp pinger #5] 21889:20190102:144052.769 server #77 started [icmp pinger #4] 21904:20190102:144052.790 server #79 started [icmp pinger #6] 21912:20190102:144052.795 server #81 started [icmp pinger #8] 21910:20190102:144052.798 server #80 started [icmp pinger #7] 21914:20190102:144052.798 server #83 started [icmp pinger #10] 21913:20190102:144052.802 server #82 started [icmp pinger #9] 21915:20190102:144052.802 server #84 started [icmp pinger #11] 21916:20190102:144052.806 server #85 started [icmp pinger #12] 21919:20190102:144052.806 server #86 started [icmp pinger #13] 21922:20190102:144052.812 server #88 started [icmp pinger #15] 21920:20190102:144052.813 server #87 started [icmp pinger #14] 21924:20190102:144052.816 server #89 started [alert manager #1] 21926:20190102:144052.819 server #90 started [preprocessing manager #1] 21928:20190102:144052.822 server #91 started [preprocessing worker #1] 21929:20190102:144052.830 server #92 started [preprocessing worker #2] 21930:20190102:144052.831 server #93 started [preprocessing worker #3] 21931:20190102:144052.831 server #94 started [preprocessing worker #4] 21932:20190102:144052.835 server #95 started [preprocessing worker #5] 21933:20190102:144052.837 server #96 started [preprocessing worker #6]
Seems to be a different one - there is no PID 15658.
UPD: Maybe a coincidence, but there is such PID in previously posted list of processes:
zabbix 15658 0.0 0.1 488400 4416 ? S 14:27 0:00 /usr/local/sbin/zabbix_server: unreachable poller #9 [got 0 values in 0.000039 sec, getting values]
Could it be that some other user/process removes .sock files?
Yes, i think is a coincidence. I will start the server, wait for the problem and report back here
This is the startup piece:
11120:20190103:165501.711 Starting Zabbix Server. Zabbix 3.4.15 (revision 86739). 11120:20190103:165501.711 ****** Enabled features ****** 11120:20190103:165501.711 SNMP monitoring: YES 11120:20190103:165501.711 IPMI monitoring: YES 11120:20190103:165501.711 Web monitoring: YES 11120:20190103:165501.711 VMware monitoring: YES 11120:20190103:165501.711 SMTP authentication: NO 11120:20190103:165501.711 Jabber notifications: YES 11120:20190103:165501.711 Ez Texting notifications: YES 11120:20190103:165501.711 ODBC: YES 11120:20190103:165501.712 SSH2 support: YES 11120:20190103:165501.712 IPv6 support: YES 11120:20190103:165501.712 TLS support: NO 11120:20190103:165501.712 ****************************** 11120:20190103:165501.712 using configuration file: /usr/local/etc/zabbix_server.conf 11120:20190103:165501.725 current database version (mandatory/optional): 03040000/03040007 11120:20190103:165501.725 required mandatory version: 03040000 11120:20190103:165504.774 server #0 started [main process] 11171:20190103:165504.775 server #1 started [configuration syncer #1] 11172:20190103:165504.776 server #2 started [alerter #1] 11173:20190103:165504.777 server #3 started [alerter #2] 11174:20190103:165504.777 server #4 started [alerter #3] 11175:20190103:165504.778 server #5 started [housekeeper #1] 11176:20190103:165504.779 server #6 started [timer #1] 11177:20190103:165504.779 server #7 started [http poller #1] 11178:20190103:165504.780 server #8 started [http poller #2] 11181:20190103:165504.783 server #11 started [history syncer #2] 11183:20190103:165504.785 server #13 started [history syncer #4] 11184:20190103:165504.785 server #14 started [escalator #1] 11182:20190103:165504.786 server #12 started [history syncer #3] 11185:20190103:165504.791 server #15 started [snmp trapper #1] 11180:20190103:165504.791 server #10 started [history syncer #1] 11191:20190103:165504.792 server #21 started [poller #3] 11190:20190103:165504.792 server #20 started [poller #2] 11186:20190103:165504.793 server #16 started [proxy poller #1] 11187:20190103:165504.794 server #17 started [self-monitoring #1] 11188:20190103:165504.794 server #18 started [task manager #1] 11179:20190103:165504.795 server #9 started [discoverer #1] 11189:20190103:165504.795 server #19 started [poller #1] 11192:20190103:165504.795 server #22 started [poller #4] 11193:20190103:165504.797 server #23 started [poller #5] 11194:20190103:165504.797 server #24 started [poller #6] 11195:20190103:165504.797 server #25 started [poller #7] 11196:20190103:165504.800 server #26 started [poller #8] 11197:20190103:165504.800 server #27 started [poller #9] 11200:20190103:165504.802 server #30 started [poller #12] 11198:20190103:165504.802 server #28 started [poller #10] 11201:20190103:165504.804 server #31 started [poller #13] 11199:20190103:165504.805 server #29 started [poller #11] 11203:20190103:165504.805 server #33 started [poller #15] 11202:20190103:165504.807 server #32 started [poller #14] 11207:20190103:165504.809 server #37 started [poller #19] 11208:20190103:165504.810 server #38 started [poller #20] 11204:20190103:165504.814 server #34 started [poller #16] 11206:20190103:165504.817 server #36 started [poller #18] 11205:20190103:165504.821 server #35 started [poller #17] 11215:20190103:165504.824 server #45 started [unreachable poller #2] 11210:20190103:165504.828 server #40 started [poller #22] 11217:20190103:165504.837 server #47 started [unreachable poller #4] 11216:20190103:165504.841 server #46 started [unreachable poller #3] 11212:20190103:165504.841 server #42 started [poller #24] 11211:20190103:165504.845 server #41 started [poller #23] 11218:20190103:165504.848 server #48 started [unreachable poller #5] 11209:20190103:165504.849 server #39 started [poller #21] 11213:20190103:165504.853 server #43 started [poller #25] 11219:20190103:165504.854 server #49 started [unreachable poller #6] 11220:20190103:165504.857 server #50 started [unreachable poller #7] 11214:20190103:165504.861 server #44 started [unreachable poller #1] 11221:20190103:165504.864 server #51 started [unreachable poller #8] 11222:20190103:165504.868 server #52 started [unreachable poller #9] 11223:20190103:165504.874 server #53 started [unreachable poller #10] 11224:20190103:165504.877 server #54 started [unreachable poller #11] 11226:20190103:165504.879 server #56 started [unreachable poller #13] 11225:20190103:165504.883 server #55 started [unreachable poller #12] 11228:20190103:165504.885 server #58 started [unreachable poller #15] 11230:20190103:165504.888 server #60 started [trapper #2] 11229:20190103:165504.889 server #59 started [trapper #1] 11227:20190103:165504.890 server #57 started [unreachable poller #14] 11232:20190103:165504.936 server #62 started [trapper #4] 11231:20190103:165504.948 server #61 started [trapper #3] 11233:20190103:165504.971 server #63 started [trapper #5] 11234:20190103:165504.989 server #64 started [trapper #6] 11235:20190103:165505.027 server #65 started [trapper #7] 11237:20190103:165505.052 server #67 started [trapper #9] 11236:20190103:165505.065 server #66 started [trapper #8] 11238:20190103:165505.077 server #68 started [trapper #10] 11239:20190103:165505.106 server #69 started [trapper #11] 11240:20190103:165505.111 server #70 started [trapper #12] 11241:20190103:165505.123 server #71 started [trapper #13] 11242:20190103:165505.147 server #72 started [trapper #14] 11243:20190103:165505.167 server #73 started [trapper #15] 11244:20190103:165505.174 server #74 started [icmp pinger #1] 11245:20190103:165505.209 server #75 started [icmp pinger #2] 11247:20190103:165505.229 server #76 started [icmp pinger #3] 11250:20190103:165505.230 server #77 started [icmp pinger #4] 11252:20190103:165505.245 server #78 started [icmp pinger #5] 11253:20190103:165505.251 server #79 started [icmp pinger #6] 11256:20190103:165505.253 server #80 started [icmp pinger #7] 11259:20190103:165505.254 server #81 started [icmp pinger #8] 11260:20190103:165505.256 server #82 started [icmp pinger #9] 11264:20190103:165505.274 server #84 started [icmp pinger #11] 11262:20190103:165505.275 server #83 started [icmp pinger #10] 11265:20190103:165505.276 server #85 started [icmp pinger #12] 11266:20190103:165505.277 server #86 started [icmp pinger #13] 11267:20190103:165505.279 server #87 started [icmp pinger #14] 11269:20190103:165505.288 server #88 started [icmp pinger #15] 11271:20190103:165505.297 server #89 started [alert manager #1] 11272:20190103:165505.303 server #90 started [preprocessing manager #1] 11274:20190103:165506.005 server #92 started [preprocessing worker #2] 11275:20190103:165506.005 server #93 started [preprocessing worker #3] 11276:20190103:165506.013 server #94 started [preprocessing worker #4] 11277:20190103:165506.013 server #95 started [preprocessing worker #5] 11278:20190103:165506.013 server #96 started [preprocessing worker #6] 11273:20190103:165506.104 server #91 started [preprocessing worker #1]
and this is the ending piece:
11218:20190103:170945.043 cannot connect to preprocessing service: Cannot connect to service "preprocessing": [2] No such file or directory.
11120:20190103:170945.048 One child process died (PID:11218,exitcode/signal:1). Exiting ...
11120:20190103:170947.054 syncing history data...
11120:20190103:170947.121 syncing history data done
11120:20190103:170947.121 syncing trend data.
Hm, unreachable poller again... Are there any messages by this process in between? Would be interesting to increase logging level for unreachable pollers. Execute the follofing command after start of Zabbix server:
$ zabbix_server -c /path/to/config -R log_level_increase="unreachable poller"
I have tried to start and execute the command you requested. The PID died in this scenario is the number 24337
This is the startup piece:
24279:20190104:093555.782 Starting Zabbix Server. Zabbix 3.4.15 (revision 86739). 24279:20190104:093555.782 ****** Enabled features ****** 24279:20190104:093555.782 SNMP monitoring: YES 24279:20190104:093555.782 IPMI monitoring: YES 24279:20190104:093555.782 Web monitoring: YES 24279:20190104:093555.782 VMware monitoring: YES 24279:20190104:093555.782 SMTP authentication: NO 24279:20190104:093555.782 Jabber notifications: YES 24279:20190104:093555.782 Ez Texting notifications: YES 24279:20190104:093555.782 ODBC: YES 24279:20190104:093555.782 SSH2 support: YES 24279:20190104:093555.782 IPv6 support: YES 24279:20190104:093555.782 TLS support: NO 24279:20190104:093555.782 ****************************** 24279:20190104:093555.782 using configuration file: /usr/local/etc/zabbix_server.conf 24279:20190104:093555.788 current database version (mandatory/optional): 03040000/03040007 24279:20190104:093555.788 required mandatory version: 03040000 24279:20190104:093558.294 server #0 started [main process] 24282:20190104:093558.295 server #1 started [configuration syncer #1] 24283:20190104:093558.295 server #2 started [alerter #1] 24284:20190104:093558.296 server #3 started [alerter #2] 24285:20190104:093558.297 server #4 started [alerter #3] 24286:20190104:093558.298 server #5 started [housekeeper #1] 24287:20190104:093558.298 server #6 started [timer #1] 24290:20190104:093558.301 server #9 started [discoverer #1] 24292:20190104:093558.302 server #11 started [history syncer #2] 24293:20190104:093558.303 server #12 started [history syncer #3] 24288:20190104:093558.304 server #7 started [http poller #1] 24289:20190104:093558.304 server #8 started [http poller #2] 24294:20190104:093558.304 server #13 started [history syncer #4] 24291:20190104:093558.305 server #10 started [history syncer #1] 24295:20190104:093558.305 server #14 started [escalator #1] 24296:20190104:093558.306 server #15 started [snmp trapper #1] 24297:20190104:093558.306 server #16 started [proxy poller #1] 24300:20190104:093558.308 server #19 started [poller #1] 24298:20190104:093558.309 server #17 started [self-monitoring #1] 24301:20190104:093558.310 server #20 started [poller #2] 24299:20190104:093558.321 server #18 started [task manager #1] 24303:20190104:093558.322 server #22 started [poller #4] 24302:20190104:093558.331 server #21 started [poller #3] 24310:20190104:093558.331 server #29 started [poller #11] 24307:20190104:093558.334 server #26 started [poller #8] 24308:20190104:093558.335 server #27 started [poller #9] 24306:20190104:093558.340 server #25 started [poller #7] 24305:20190104:093558.345 server #24 started [poller #6] 24304:20190104:093558.348 server #23 started [poller #5] 24311:20190104:093558.366 server #30 started [poller #12] 24314:20190104:093558.389 server #33 started [poller #15] 24309:20190104:093558.390 server #28 started [poller #10] 24315:20190104:093558.396 server #34 started [poller #16] 24316:20190104:093558.397 server #35 started [poller #17] 24313:20190104:093558.405 server #32 started [poller #14] 24319:20190104:093558.409 server #38 started [poller #20] 24320:20190104:093558.411 server #39 started [poller #21] 24318:20190104:093558.418 server #37 started [poller #19] 24317:20190104:093558.420 server #36 started [poller #18] 24312:20190104:093558.422 server #31 started [poller #13] 24321:20190104:093558.427 server #40 started [poller #22] 24322:20190104:093558.428 server #41 started [poller #23] 24323:20190104:093558.432 server #42 started [poller #24] 24324:20190104:093558.436 server #43 started [poller #25] 24325:20190104:093558.437 server #44 started [unreachable poller #1] 24326:20190104:093558.441 server #45 started [unreachable poller #2] 24331:20190104:093558.443 server #50 started [unreachable poller #7] 24329:20190104:093558.447 server #48 started [unreachable poller #5] 24328:20190104:093558.448 server #47 started [unreachable poller #4] 24332:20190104:093558.451 server #51 started [unreachable poller #8] 24327:20190104:093558.455 server #46 started [unreachable poller #3] 24334:20190104:093558.458 server #53 started [unreachable poller #10] 24335:20190104:093558.464 server #54 started [unreachable poller #11] 24336:20190104:093558.470 server #55 started [unreachable poller #12] 24338:20190104:093558.474 server #57 started [unreachable poller #14] 24339:20190104:093558.476 server #58 started [unreachable poller #15] 24330:20190104:093558.476 server #49 started [unreachable poller #6] 24333:20190104:093558.480 server #52 started [unreachable poller #9] 24337:20190104:093558.481 server #56 started [unreachable poller #13] 24340:20190104:093558.485 server #59 started [trapper #1] 24344:20190104:093558.486 server #63 started [trapper #5] 24343:20190104:093558.486 server #62 started [trapper #4] 24342:20190104:093558.486 server #61 started [trapper #3] 24341:20190104:093558.486 server #60 started [trapper #2] 24345:20190104:093558.487 server #64 started [trapper #6] 24349:20190104:093558.487 server #68 started [trapper #10] 24348:20190104:093558.487 server #67 started [trapper #9] 24347:20190104:093558.487 server #66 started [trapper #8] 24346:20190104:093558.487 server #65 started [trapper #7] 24353:20190104:093558.488 server #72 started [trapper #14] 24352:20190104:093558.488 server #71 started [trapper #13] 24351:20190104:093558.488 server #70 started [trapper #12] 24350:20190104:093558.488 server #69 started [trapper #11] 24354:20190104:093558.488 server #73 started [trapper #15] 24357:20190104:093558.488 server #76 started [icmp pinger #3] 24356:20190104:093558.489 server #75 started [icmp pinger #2] 24355:20190104:093558.489 server #74 started [icmp pinger #1] 24358:20190104:093558.489 server #77 started [icmp pinger #4] 24363:20190104:093558.489 server #82 started [icmp pinger #9] 24362:20190104:093558.489 server #81 started [icmp pinger #8] 24361:20190104:093558.489 server #80 started [icmp pinger #7] 24360:20190104:093558.489 server #79 started [icmp pinger #6] 24359:20190104:093558.489 server #78 started [icmp pinger #5] 24367:20190104:093558.490 server #86 started [icmp pinger #13] 24366:20190104:093558.490 server #85 started [icmp pinger #12] 24368:20190104:093558.490 server #87 started [icmp pinger #14] 24365:20190104:093558.490 server #84 started [icmp pinger #11] 24364:20190104:093558.490 server #83 started [icmp pinger #10] 24370:20190104:093558.490 server #89 started [alert manager #1] 24371:20190104:093558.491 server #90 started [preprocessing manager #1] 24369:20190104:093558.491 server #88 started [icmp pinger #15] 24372:20190104:093558.573 server #91 started [preprocessing worker #1] 24375:20190104:093558.577 server #94 started [preprocessing worker #4] 24374:20190104:093558.577 server #93 started [preprocessing worker #3] 24376:20190104:093558.577 server #95 started [preprocessing worker #5] 24377:20190104:093558.577 server #96 started [preprocessing worker #6] 24373:20190104:093558.581 server #92 started [preprocessing worker #2]
Then, I have increased the unreachable poller verbosity, and this is the server log, with a "grep 24337"
24337:20190104:094802.396 __zbx_zbx_setproctitle() title:'unreachable poller #13 [got 0 values in 0.000246 sec, idle 1 sec]' 24337:20190104:094803.396 __zbx_zbx_setproctitle() title:'unreachable poller #13 [got 0 values in 0.000246 sec, getting values]' 24337:20190104:094803.397 In get_values() 24337:20190104:094803.397 In DCconfig_get_poller_items() poller_type:1 24337:20190104:094803.397 End of DCconfig_get_poller_items():0 24337:20190104:094803.397 In DCconfig_get_poller_nextcheck() poller_type:1 24337:20190104:094803.397 End of DCconfig_get_poller_nextcheck():1546591685 24337:20190104:094803.397 End of get_values():0 24337:20190104:094803.397 __zbx_zbx_setproctitle() title:'unreachable poller #13 [got 0 values in 0.000315 sec, idle 2 sec]' 24337:20190104:094805.397 __zbx_zbx_setproctitle() title:'unreachable poller #13 [got 0 values in 0.000315 sec, getting values]' 24337:20190104:094805.397 In get_values() 24337:20190104:094805.397 In DCconfig_get_poller_items() poller_type:1 24337:20190104:094805.397 End of DCconfig_get_poller_items():0 24337:20190104:094805.397 In DCconfig_get_poller_nextcheck() poller_type:1 24337:20190104:094805.398 End of DCconfig_get_poller_nextcheck():1546591686 24337:20190104:094805.398 End of get_values():0 24337:20190104:094805.398 __zbx_zbx_setproctitle() title:'unreachable poller #13 [got 0 values in 0.000816 sec, idle 1 sec]' 24337:20190104:094806.398 __zbx_zbx_setproctitle() title:'unreachable poller #13 [got 0 values in 0.000816 sec, getting values]' 24337:20190104:094806.398 In get_values() 24337:20190104:094806.398 In DCconfig_get_poller_items() poller_type:1 24337:20190104:094806.398 End of DCconfig_get_poller_items():0 24337:20190104:094806.398 In DCconfig_get_poller_nextcheck() poller_type:1 24337:20190104:094806.399 End of DCconfig_get_poller_nextcheck():1546591689 24337:20190104:094806.399 End of get_values():0 24337:20190104:094806.399 __zbx_zbx_setproctitle() title:'unreachable poller #13 [got 0 values in 0.000875 sec, idle 3 sec]' 24337:20190104:094809.400 __zbx_zbx_setproctitle() title:'unreachable poller #13 [got 0 values in 0.000875 sec, getting values]' 24337:20190104:094809.401 In get_values() 24337:20190104:094809.401 In DCconfig_get_poller_items() poller_type:1 24337:20190104:094809.401 End of DCconfig_get_poller_items():0 24337:20190104:094809.401 In DCconfig_get_poller_nextcheck() poller_type:1 24337:20190104:094809.402 End of DCconfig_get_poller_nextcheck():1546591692 24337:20190104:094809.403 End of get_values():0 24337:20190104:094809.403 __zbx_zbx_setproctitle() title:'unreachable poller #13 [got 0 values in 0.003354 sec, idle 3 sec]' 24337:20190104:094812.404 __zbx_zbx_setproctitle() title:'unreachable poller #13 [got 0 values in 0.003354 sec, getting values]' 24337:20190104:094812.404 In get_values() 24337:20190104:094812.404 In DCconfig_get_poller_items() poller_type:1 24337:20190104:094812.404 End of DCconfig_get_poller_items():0 24337:20190104:094812.404 In DCconfig_get_poller_nextcheck() poller_type:1 24337:20190104:094812.404 End of DCconfig_get_poller_nextcheck():1546591693 24337:20190104:094812.404 End of get_values():0 24337:20190104:094812.405 __zbx_zbx_setproctitle() title:'unreachable poller #13 [got 0 values in 0.000798 sec, idle 1 sec]' 24337:20190104:094813.406 __zbx_zbx_setproctitle() title:'unreachable poller #13 [got 0 values in 0.000798 sec, getting values]' 24337:20190104:094813.406 In get_values() 24337:20190104:094813.406 In DCconfig_get_poller_items() poller_type:1 24337:20190104:094813.406 End of DCconfig_get_poller_items():0 24337:20190104:094813.406 In DCconfig_get_poller_nextcheck() poller_type:1 24337:20190104:094813.407 End of DCconfig_get_poller_nextcheck():1546591694 24337:20190104:094813.407 End of get_values():0 24337:20190104:094813.408 __zbx_zbx_setproctitle() title:'unreachable poller #13 [got 0 values in 0.002053 sec, idle 1 sec]' 24337:20190104:094814.408 __zbx_zbx_setproctitle() title:'unreachable poller #13 [got 0 values in 0.002053 sec, getting values]' 24337:20190104:094814.408 In get_values() 24337:20190104:094814.409 In DCconfig_get_poller_items() poller_type:1 24337:20190104:094814.409 End of DCconfig_get_poller_items():0 24337:20190104:094814.409 In DCconfig_get_poller_nextcheck() poller_type:1 24337:20190104:094814.409 End of DCconfig_get_poller_nextcheck():1546591695 24337:20190104:094814.409 End of get_values():0 24337:20190104:094814.409 __zbx_zbx_setproctitle() title:'unreachable poller #13 [got 0 values in 0.000348 sec, idle 1 sec]' 24337:20190104:094815.409 __zbx_zbx_setproctitle() title:'unreachable poller #13 [got 0 values in 0.000348 sec, getting values]' 24337:20190104:094815.409 In get_values() 24337:20190104:094815.409 In DCconfig_get_poller_items() poller_type:1 24337:20190104:094815.409 End of DCconfig_get_poller_items():0 24337:20190104:094815.409 In DCconfig_get_poller_nextcheck() poller_type:1 24337:20190104:094815.409 End of DCconfig_get_poller_nextcheck():1546591706 24337:20190104:094815.409 End of get_values():0 24337:20190104:094815.409 __zbx_zbx_setproctitle() title:'unreachable poller #13 [got 0 values in 0.000316 sec, idle 5 sec]' 24337:20190104:094820.409 __zbx_zbx_setproctitle() title:'unreachable poller #13 [got 0 values in 0.000316 sec, getting values]' 24337:20190104:094820.410 In get_values() 24337:20190104:094820.410 In DCconfig_get_poller_items() poller_type:1 24337:20190104:094820.410 End of DCconfig_get_poller_items():0 24337:20190104:094820.410 In DCconfig_get_poller_nextcheck() poller_type:1 24337:20190104:094820.410 End of DCconfig_get_poller_nextcheck():1546591706 24337:20190104:094820.410 End of get_values():0 24337:20190104:094820.410 __zbx_zbx_setproctitle() title:'unreachable poller #13 [got 0 values in 0.000301 sec, idle 5 sec]' 24337:20190104:094825.410 __zbx_zbx_setproctitle() title:'unreachable poller #13 [got 0 values in 0.000301 sec, getting values]' 24337:20190104:094825.410 In get_values() 24337:20190104:094825.410 In DCconfig_get_poller_items() poller_type:1 24337:20190104:094825.410 End of DCconfig_get_poller_items():0 24337:20190104:094825.410 In DCconfig_get_poller_nextcheck() poller_type:1 24337:20190104:094825.410 End of DCconfig_get_poller_nextcheck():1546591706 24337:20190104:094825.410 End of get_values():0 24337:20190104:094825.410 __zbx_zbx_setproctitle() title:'unreachable poller #13 [got 0 values in 0.000297 sec, idle 1 sec]' 24337:20190104:094826.411 __zbx_zbx_setproctitle() title:'unreachable poller #13 [got 0 values in 0.000297 sec, getting values]' 24337:20190104:094826.411 In get_values() 24337:20190104:094826.411 In DCconfig_get_poller_items() poller_type:1 24337:20190104:094826.411 End of DCconfig_get_poller_items():0 24337:20190104:094826.411 In DCconfig_get_poller_nextcheck() poller_type:1 24337:20190104:094826.411 End of DCconfig_get_poller_nextcheck():1546591707 24337:20190104:094826.411 End of get_values():0 24337:20190104:094826.411 __zbx_zbx_setproctitle() title:'unreachable poller #13 [got 0 values in 0.000448 sec, idle 1 sec]' 24337:20190104:094827.411 __zbx_zbx_setproctitle() title:'unreachable poller #13 [got 0 values in 0.000448 sec, getting values]' 24337:20190104:094827.411 In get_values() 24337:20190104:094827.411 In DCconfig_get_poller_items() poller_type:1 24337:20190104:094827.411 End of DCconfig_get_poller_items():0 24337:20190104:094827.411 In DCconfig_get_poller_nextcheck() poller_type:1 24337:20190104:094827.411 End of DCconfig_get_poller_nextcheck():1546591708 24337:20190104:094827.411 End of get_values():0 24337:20190104:094827.411 __zbx_zbx_setproctitle() title:'unreachable poller #13 [got 0 values in 0.000268 sec, idle 1 sec]' 24337:20190104:094828.412 __zbx_zbx_setproctitle() title:'unreachable poller #13 [got 0 values in 0.000268 sec, getting values]' 24337:20190104:094828.412 In get_values() 24337:20190104:094828.412 In DCconfig_get_poller_items() poller_type:1 24337:20190104:094828.413 End of DCconfig_get_poller_items():0 24337:20190104:094828.413 In DCconfig_get_poller_nextcheck() poller_type:1 24337:20190104:094828.413 End of DCconfig_get_poller_nextcheck():1546591724 24337:20190104:094828.413 End of get_values():0 24337:20190104:094828.413 __zbx_zbx_setproctitle() title:'unreachable poller #13 [got 0 values in 0.001606 sec, idle 5 sec]' 24337:20190104:094833.414 __zbx_zbx_setproctitle() title:'unreachable poller #13 [got 0 values in 0.001606 sec, getting values]' 24337:20190104:094833.414 In get_values() 24337:20190104:094833.414 In DCconfig_get_poller_items() poller_type:1 24337:20190104:094833.414 End of DCconfig_get_poller_items():0 24337:20190104:094833.414 In DCconfig_get_poller_nextcheck() poller_type:1 24337:20190104:094833.414 End of DCconfig_get_poller_nextcheck():1546591724 24337:20190104:094833.414 End of get_values():0 24337:20190104:094833.414 __zbx_zbx_setproctitle() title:'unreachable poller #13 [got 0 values in 0.000368 sec, idle 5 sec]' 24337:20190104:094838.414 __zbx_zbx_setproctitle() title:'unreachable poller #13 [got 0 values in 0.000368 sec, getting values]' 24337:20190104:094838.415 In get_values() 24337:20190104:094838.415 In DCconfig_get_poller_items() poller_type:1 24337:20190104:094838.415 End of DCconfig_get_poller_items():0 24337:20190104:094838.415 In DCconfig_get_poller_nextcheck() poller_type:1 24337:20190104:094838.415 End of DCconfig_get_poller_nextcheck():1546591724 24337:20190104:094838.415 End of get_values():0 24337:20190104:094838.415 __zbx_zbx_setproctitle() title:'unreachable poller #13 [got 0 values in 0.000557 sec, idle 5 sec]' 24337:20190104:094843.415 __zbx_zbx_setproctitle() title:'unreachable poller #13 [got 0 values in 0.000557 sec, getting values]' 24337:20190104:094843.415 In get_values() 24337:20190104:094843.415 In DCconfig_get_poller_items() poller_type:1 24337:20190104:094843.416 End of DCconfig_get_poller_items():0 24337:20190104:094843.416 In DCconfig_get_poller_nextcheck() poller_type:1 24337:20190104:094843.416 End of DCconfig_get_poller_nextcheck():1546591724 24337:20190104:094843.416 End of get_values():0 24337:20190104:094843.416 __zbx_zbx_setproctitle() title:'unreachable poller #13 [got 0 values in 0.000340 sec, idle 1 sec]' 24337:20190104:094844.416 __zbx_zbx_setproctitle() title:'unreachable poller #13 [got 0 values in 0.000340 sec, getting values]' 24337:20190104:094844.416 In get_values() 24337:20190104:094844.416 In DCconfig_get_poller_items() poller_type:1 24337:20190104:094844.416 End of DCconfig_get_poller_items():1 24337:20190104:094844.416 In substitute_key_macros() data:'prtMarkerSuppliesLevel.5' 24337:20190104:094844.416 End of substitute_key_macros():SUCCEED data:'prtMarkerSuppliesLevel.5' 24337:20190104:094844.416 In substitute_simple_macros() data:'161' 24337:20190104:094844.416 In substitute_simple_macros() data:'public' 24337:20190104:094844.416 In substitute_key_macros() data:'1.3.6.1.2.1.43.11.1.1.9.1.5' 24337:20190104:094844.416 End of substitute_key_macros():SUCCEED data:'1.3.6.1.2.1.43.11.1.1.9.1.5' 24337:20190104:094844.416 In get_values_snmp() host:'Kyocera KM-2560 - Dev FE-BE' addr:'xxx.xxx.xxx.xxx' num:1 24337:20190104:094844.416 In zbx_snmp_open_session() 24337:20190104:094844.416 SNMP [public@xxx.xxx.xxx.xxx:161] 24337:20190104:094844.416 End of zbx_snmp_open_session() 24337:20190104:094844.416 In zbx_snmp_process_standard() 24337:20190104:094844.416 In zbx_snmp_translate() oid:'1.3.6.1.2.1.43.11.1.1.9.1.5' 24337:20190104:094844.417 End of zbx_snmp_translate() oid_translated:'1.3.6.1.2.1.43.11.1.1.9.1.5' 24337:20190104:094844.417 In zbx_snmp_get_values() num:1 level:0 24337:20190104:094844.424 zbx_snmp_get_values() snmp_synch_response() status:0 s_snmp_errno:0 errstat:2 mapping_num:1 24337:20190104:094844.424 zbx_snmp_get_values() snmp_synch_response() errindex:1 oid:'1.3.6.1.2.1.43.11.1.1.9.1.5' 24337:20190104:094844.424 End of zbx_snmp_get_values():SUCCEED 24337:20190104:094844.425 End of zbx_snmp_process_standard():SUCCEED 24337:20190104:094844.425 In zbx_snmp_close_session() 24337:20190104:094844.425 End of zbx_snmp_close_session() 24337:20190104:094844.426 End of get_values_snmp() 24337:20190104:094844.426 In zbx_activate_item_host() hostid:12462 itemid:137315 type:1 24337:20190104:094844.427 query [txnlev:1] [begin;] 24337:20190104:094844.428 query [txnlev:1] [update hosts set snmp_errors_from=0,snmp_disable_until=0 where hostid=12462] 24337:20190104:094844.428 query [txnlev:1] [commit;] 24337:20190104:094844.431 resuming SNMP agent checks on host "Kyocera KM-2560 - Dev FE-BE": connection restored 24337:20190104:094844.431 End of zbx_activate_item_host() 24337:20190104:094844.431 In zbx_preprocess_item_value() 24337:20190104:094844.431 End of zbx_preprocess_item_value() 24337:20190104:094844.431 In zbx_ipc_socket_open() 24337:20190104:094945.092 End of zbx_ipc_socket_open():FAIL 24337:20190104:094945.092 cannot connect to preprocessing service: Cannot connect to service "preprocessing": [2] No such file or directory. 24279:20190104:094945.096 One child process died (PID:24337,exitcode/signal:1). Exiting ...
Sorry for a delayed response. 
Thank you! That is exactly what I wanted!
We clearly see that this unreachable poller was working fine while it had no items to check
End of DCconfig_get_poller_items():0
but as soon as it got the first value and needed to hand over it to preprocessing something bad happened.
You say that
The files zabbix_server_alerter.sock and zabbix_server_preprocessing.sock are there for 10-15 minutes, then they disappear
Could it be that some other process removes them?
I've changed the socketdir: from /tmp to /opt/zabbix but the problem remains.
Maybe that process looks at Zabbix configuration file? Try to change location of config file and SocketDir at once.
There is no indication that this is a problem in Zabbix.
Zabbix itself doesn't remove .sock files while it is working, so looks like the problem is caused by third party software.