FreeBSD 10.1

Looks like a poller process exits. Can you set DebugLevel=4 in server config file and try again ? Are you using encryption (e.g. certificate is configured for server) ?

Encryption is not used or configured.

Thanks for log file! What is "StartPollers" and" StartPollersUnreachable" parameter values in server config file ?

Everything related to Poller and trappers commented out...

############ ADVANCED PARAMETERS ################

### Option: StartPollers
#	Number of pre-forked instances of pollers.
#
# Mandatory: no
# Range: 0-1000
# Default:
# StartPollers=5

### Option: StartIPMIPollers
#	Number of pre-forked instances of IPMI pollers.
#
# Mandatory: no
# Range: 0-1000
# Default:
# StartIPMIPollers=0

### Option: StartPollersUnreachable
#	Number of pre-forked instances of pollers for unreachable hosts (including IPMI and Java).
#	At least one poller for unreachable hosts must be running if regular, IPMI or Java pollers
#	are started.
#
# Mandatory: no
# Range: 0-1000
# Default:
# StartPollersUnreachable=1

### Option: StartTrappers
#	Number of pre-forked instances of trappers.
#	Trappers accept incoming connections from Zabbix sender, active agents and active proxies.
#	At least one trapper process must be running to display server availability and view queue
#	in the frontend.
#
# Mandatory: no
# Range: 0-1000
# Default:
# StartTrappers=5

### Option: StartPingers
#	Number of pre-forked instances of ICMP pingers.
#
# Mandatory: no
# Range: 0-1000
# Default:
# StartPingers=1

The suspicious part is "4" in the following line:

   872:20160412:171816.188 One child process died (PID:875,exitcode/signal:4). Exiting ...

If that is meant to indicate a signal, then that signal is SIGILL (illegal instruction).

What architecture are you running on? How did you install Zabbix - from packages or from sources? Did you compile Zabbix on the same machine? What ./configure command was used to build Zabbix?

FreeBSD 10.1 i386.

Installation is made as follows:
1. Portsnap fetch extract update
2. pkg delete zabbix-server 2.2
3. /usr/ports/net-mgmt/zabbix3-server make all install clean

Next corrected configuration files and installed zabbix3-frontend

Observation. On both occasions the following processes had not reported that they have started: #3-7 (5 pollers), #8 (poller unreachable), #14 (pinger), #19 (discoverer). Processes that had not reported "Got signal..." are 5 pollers, poller unreachable, pinger and main process. Note that discoverer reported getting signal!

Pollers, pollers unreachable and discoverers are the processes performing init_snmp(). And in discoverer code init_snmp() comes before logging that process has started. Hypothesis is, pollers are busy logging at first and pinger is queueing behind them. They all blocked user signals before acquiring log access, that's why there are no "Got signal..." messages from them. Discoverer gets to init_snmp() first and while it is there the first poller who gets logging done performs init_snmp() too and here something bad happens. We don't get to see his message possibly because it dies immediately.

Unfortunately, we can't dig deeper in this direction without recompiling Zabbix server because we have:

#	define SNMP_NO_DEBUGGING		/* disabling debugging messages from Net-SNMP library */

Dear tbiryukov, please check your NetSNMP library.

I have exactly the same issue:
FreeBSD 10.1-RELEASE-p5 i386
Zabbix 3.0.1_1
net-snmp-5.7.3_11
Zabbix and net-snmp installed from packages.

The only possibility to start server is to set:
StartPollers=0
StartPollersUnreachable=0
StartPingers=0

Zabbix version 2.4.8 works fine on the same system. Also I compiled Zabbix 3 without net-snmp - still no success.

zabbix_proxy (Zabbix) 3.0.2
Revision 59540 20 April 2016, compilation time: May 14 2016 14:49:04

FreeBSD 10.1-RELEASE FreeBSD 10.1-RELEASE #0 r274401: Tue Nov 11 22:51:51 UTC 2014 i386

  4282:20160514:160942.259 End of DCsync_configuration()
  4282:20160514:160942.260 proxy #0 started [main process]
  4285:20160514:160942.263 proxy #3 started [trapper #1]
  4286:20160514:160942.264 proxy #4 started [trapper #2]
  4287:20160514:160942.264 proxy #5 started [trapper #3]
  4288:20160514:160942.264 proxy #6 started [trapper #4]
  4289:20160514:160942.264 proxy #7 started [trapper #5]
  4282:20160514:160942.265 One child process died (PID:4283,exitcode/signal:4). Exiting ...
  4285:20160514:160942.265 In zbx_tls_init_child()
  4286:20160514:160942.265 In zbx_tls_init_child()
  4287:20160514:160942.268 In zbx_tls_init_child()
  4288:20160514:160942.270 In zbx_tls_init_child()
  4289:20160514:160942.271 In zbx_tls_init_child()
  4282:20160514:160942.274 zbx_on_exit() called
...
...
...
4285:20160514:160942.279 Got signal [signal:15(SIGTERM),sender_pid:4282,sender_uid:122,reason:65537]. Exiting ...
  4286:20160514:160942.279 Got signal [signal:15(SIGTERM),sender_pid:4282,sender_uid:122,reason:65537]. Exiting ...
  4288:20160514:160942.279 Got signal [signal:15(SIGTERM),sender_pid:4282,sender_uid:122,reason:65537]. Exiting ...
  4287:20160514:160942.280 Got signal [signal:15(SIGTERM),sender_pid:4282,sender_uid:122,reason:65537]. Exiting ...
...
...
...
4282:20160514:160944.306 Zabbix Proxy stopped. Zabbix 3.0.2 (revision 59540).
zabbix_proxy [4289]: [file:'log.c',line:294] lock failed: [22] Invalid argument

UPD:
Looks like "zabbix proxy 3.0.3rc1" builded from sources without net-snmp works for me:
zabbix_proxy (Zabbix) 3.0.3rc1
Revision 60079 12 May 2016, compilation time: May 14 2016 16:38:50

Dear sapzxc,

for completeness of the picture could you also try version 3.0.3rc1 built from sources with net-snmp and version 3.0.2 built from sources without net-snmp.

Could you also attach a more detailed log file so that we can identify process "PID:4283" and see which processes reported their start and getting SIGTERM from main process?

Unfortunately, in freebsd port, there no option "snmp" ( http://www.screencast.com/t/OpoR7MuXu ). Also when I try build 3.0.3rc1 with snmp, I got this error:

./../poller/checks_snmp.h:30:11: fatal error: 'net-snmp/net-snmp-config.h' file not found
#       include <net-snmp/net-snmp-config.h>

But actually port "net-snmp-5.7.3_7" installed in the system.

I do not using snmp (as I imaging), so at the moment rc1 build works fine for me.

Updating to FreeBSD 10.3-p4 resolved the problem. Even Zabbix 3.0.1_2 from pkg repository works.

The root cause may be in different versions of OpenSSL used by NetSNMP and Zabbix itself. Related issue is ZBX-11017.

Can anyone provide objdump or ldd output of crashing Zabbix binary?

No updates for more than a year. Closing as Cannot Reproduce because there is currently not enough information to continue investigation.