Major While fixing another issue, I discovered a bug that cause the zabbix server to crash. In particular, the crash happen while establishing an snmp session in the zbx_snmp_open_session() function located in src/zabbix_server/poller/checks_snmp.c:630. Here the code snippet:
#ifdef HAVE_NETSNMP_SESSION_LOCALNAME if (NULL != CONFIG_SOURCE_IP) { /* In some cases specifying just local host (without local port) is not enough. We do */ /* not care about the port number though so we let the OS select one by specifying 0. */ /* See marc.info/?l=net-snmp-bugs&m=115624676507760 for details. */ static char localname[64]; zbx_snprintf(localname, sizeof(localname), "%s:0", CONFIG_SOURCE_IP); session.localname = localname; } #endif SOCK_STARTUP; if (NULL == (ss = snmp_open(&session))) { SOCK_CLEANUP; zbx_strlcpy(error, "Cannot open SNMP session", max_error_len); } end: zabbix_log(LOG_LEVEL_DEBUG, "End of %s()", __function_name); return ss;
and here a core dump of the crashed process:
#0 thrkill () at -:3
#1 0x00000c4b902b435d in _libc_abort () at /usr/src/lib/libc/stdlib/abort.c:51
#2 0x00000c4934dc0e28 in log_fatal_signal (sig=11, siginfo=0x7f7fffe7ae80, context=0x7f7fffe7ad90) at sighandler.c:33
#3 0x00000c4934dc09de in fatal_signal_handler (sig=11, siginfo=0x7f7fffe7ae80, context=0x7f7fffe7ad90) at sighandler.c:60
#4 <signal handler called>
#5 strlen () at /usr/src/lib/libc/arch/amd64/string/strlen.S:124
#6 0x00000c4b90310f62 in _libc_strdup (str=0xc4c1d74ee00 <Address 0xc4c1d74ee00 out of bounds>) at /usr/src/lib/libc/string/strdup.c:44
#7 0x00000c4b7322f16c in netsnmp_ds_set_string () from /usr/local/lib/libnetsnmp.so.14.0
#8 0x00000c4b731ffeac in snmp_sess_open () from /usr/local/lib/libnetsnmp.so.14.0
#9 0x00000c4b731ffde9 in snmp_open () from /usr/local/lib/libnetsnmp.so.14.0
#10 0x00000c4934d1ea54 in zbx_snmp_open_session (item=0x7f7fffe7ee30, error=0x7f7fffe7b840 "", max_error_len=2048) at checks_snmp.c:630
#11 0x00000c4934d1dbaf in get_values_snmp (items=0x7f7fffe7ee30, results=0x7f7fffe7ce30, errcodes=0x7f7fffe7cc30, num=1) at checks_snmp.c:2097
#12 0x00000c4934d26dbd in get_values (poller_type=0 '\0', nextcheck=0x7f7fffff86ac) at poller.c:571
#13 0x00000c4934d25717 in poller_thread (args=0x7f7fffff8748) at poller.c:778
#14 0x00000c4934dd0f91 in zbx_thread_start (handler=0xc4934d25590 <poller_thread>, thread_args=0x7f7fffff8748) at threads.c:128
#15 0x00000c4934d0f54c in MAIN_ZABBIX_ENTRY (flags=2) at server.c:1040
#16 0x00000c4934dbf854 in daemon_start (allow_root=0, user=0x0, flags=2) at daemon.c:392
#17 0x00000c4934d0eca2 in main (argc=3, argv=0x7f7fffff96d8) at server.c:819
The server crash while trying to strdup() a string, that can be pointed to be the session.localname pointer set to a statically allocated buffer (localname).
The crash happen after ~30s the server is running and it seems to happen while performing a strlen() of the input buffer. Here the strdup implementation of the OpenBSD's libc:
char * strdup(const char *str) { size_t siz; char *copy; siz = strlen(str) + 1; if ((copy = malloc(siz)) == NULL) return(NULL); (void)memcpy(copy, str, siz); return(copy); }
As a verification I tried to force the session.localname pointer to NULL and the crash does not happen anymore.
Another check I did was converting the localname buffer to not be static, but the crash still happen.
It seems to be a problem in zabbix, but I was not able to understand it completely yet. It seems to be that the localname buffer array or the session.localname pointer are corrupted in some ways, causing the access to an invalid pointer and consequently a segmentation fault.
