[ZBX-12726] zabbix server crash while establishing snmp session Created: 2017 Sep 14 Updated: 2024 Apr 10 Resolved: 2017 Sep 21 |
|
Status: | Closed |
Project: | ZABBIX BUGS AND ISSUES |
Component/s: | Server (S) |
Affects Version/s: | 2.2.19, 3.0.10, 3.2.7, 3.4.1, 4.0.0alpha1 |
Fix Version/s: | None |
Type: | Problem report | Priority: | Major |
Reporter: | Andrea Biscuola (Inactive) | Assignee: | Unassigned |
Resolution: | Won't fix | Votes: | 0 |
Labels: | Sourceip, crash, segfault, server, snmp | ||
Remaining Estimate: | Not Specified | ||
Time Spent: | Not Specified | ||
Original Estimate: | Not Specified | ||
Environment: |
OpenBSD 6.1-CURRENT |
Team: | Team A |
Team: | Team A |
Sprint: | Sprint 17 |
Story Points: | 1 |
Description |
While fixing another issue, I discovered a bug that cause the zabbix server to crash. In particular, the crash happen while establishing an snmp session in the zbx_snmp_open_session() function located in src/zabbix_server/poller/checks_snmp.c:630. Here the code snippet: #ifdef HAVE_NETSNMP_SESSION_LOCALNAME if (NULL != CONFIG_SOURCE_IP) { /* In some cases specifying just local host (without local port) is not enough. We do */ /* not care about the port number though so we let the OS select one by specifying 0. */ /* See marc.info/?l=net-snmp-bugs&m=115624676507760 for details. */ static char localname[64]; zbx_snprintf(localname, sizeof(localname), "%s:0", CONFIG_SOURCE_IP); session.localname = localname; } #endif SOCK_STARTUP; if (NULL == (ss = snmp_open(&session))) { SOCK_CLEANUP; zbx_strlcpy(error, "Cannot open SNMP session", max_error_len); } end: zabbix_log(LOG_LEVEL_DEBUG, "End of %s()", __function_name); return ss; and here a core dump of the crashed process:
#0 thrkill () at -:3
#1 0x00000c4b902b435d in _libc_abort () at /usr/src/lib/libc/stdlib/abort.c:51
#2 0x00000c4934dc0e28 in log_fatal_signal (sig=11, siginfo=0x7f7fffe7ae80, context=0x7f7fffe7ad90) at sighandler.c:33
#3 0x00000c4934dc09de in fatal_signal_handler (sig=11, siginfo=0x7f7fffe7ae80, context=0x7f7fffe7ad90) at sighandler.c:60
#4 <signal handler called>
#5 strlen () at /usr/src/lib/libc/arch/amd64/string/strlen.S:124
#6 0x00000c4b90310f62 in _libc_strdup (str=0xc4c1d74ee00 <Address 0xc4c1d74ee00 out of bounds>) at /usr/src/lib/libc/string/strdup.c:44
#7 0x00000c4b7322f16c in netsnmp_ds_set_string () from /usr/local/lib/libnetsnmp.so.14.0
#8 0x00000c4b731ffeac in snmp_sess_open () from /usr/local/lib/libnetsnmp.so.14.0
#9 0x00000c4b731ffde9 in snmp_open () from /usr/local/lib/libnetsnmp.so.14.0
#10 0x00000c4934d1ea54 in zbx_snmp_open_session (item=0x7f7fffe7ee30, error=0x7f7fffe7b840 "", max_error_len=2048) at checks_snmp.c:630
#11 0x00000c4934d1dbaf in get_values_snmp (items=0x7f7fffe7ee30, results=0x7f7fffe7ce30, errcodes=0x7f7fffe7cc30, num=1) at checks_snmp.c:2097
#12 0x00000c4934d26dbd in get_values (poller_type=0 '\0', nextcheck=0x7f7fffff86ac) at poller.c:571
#13 0x00000c4934d25717 in poller_thread (args=0x7f7fffff8748) at poller.c:778
#14 0x00000c4934dd0f91 in zbx_thread_start (handler=0xc4934d25590 <poller_thread>, thread_args=0x7f7fffff8748) at threads.c:128
#15 0x00000c4934d0f54c in MAIN_ZABBIX_ENTRY (flags=2) at server.c:1040
#16 0x00000c4934dbf854 in daemon_start (allow_root=0, user=0x0, flags=2) at daemon.c:392
#17 0x00000c4934d0eca2 in main (argc=3, argv=0x7f7fffff96d8) at server.c:819
The server crash while trying to strdup() a string, that can be pointed to be the session.localname pointer set to a statically allocated buffer (localname). char * strdup(const char *str) { size_t siz; char *copy; siz = strlen(str) + 1; if ((copy = malloc(siz)) == NULL) return(NULL); (void)memcpy(copy, str, siz); return(copy); } As a verification I tried to force the session.localname pointer to NULL and the crash does not happen anymore. It seems to be a problem in zabbix, but I was not able to understand it completely yet. It seems to be that the localname buffer array or the session.localname pointer are corrupted in some ways, causing the access to an invalid pointer and consequently a segmentation fault. |
Comments |
Comment by Andrea Biscuola (Inactive) [ 2017 Sep 14 ] |
I'm also using net-snmp 5.7.3 on OpenBSD 6.1-CURRENT. As I mentioned, it's also possible that we are looking at a net-snmp subtle bug that happen only on the specific platform. |
Comment by Vladislavs Sokurenko [ 2017 Sep 14 ] |
Reproduced ==25890== Invalid read of size 1 ==25890== at 0x4C31B82: strlen (vg_replace_strmem.c:458) ==25890== by 0x6D02CDD: strdup (in /usr/lib64/libc-2.25.so) ==25890== by 0x54532ED: netsnmp_ds_set_string (in /usr/lib64/libnetsnmp.so.30.0.3) ==25890== by 0x5423E4F: snmp_sess_open (in /usr/lib64/libnetsnmp.so.30.0.3) ==25890== by 0x5424078: snmp_open (in /usr/lib64/libnetsnmp.so.30.0.3) ==25890== by 0x42C375: zbx_snmp_open_session (checks_snmp.c:630) ==25890== by 0x42C375: get_values_snmp (checks_snmp.c:2097) ==25890== by 0x42CF8A: get_value_snmp (checks_snmp.c:2071) ==25890== by 0x426DE3: discover_service (discoverer.c:270) ==25890== by 0x426DE3: process_check (discoverer.c:360) ==25890== by 0x426DE3: process_checks (discoverer.c:455) ==25890== by 0x427994: process_rule (discoverer.c:539) ==25890== by 0x427994: process_discovery (discoverer.c:742) ==25890== by 0x427994: discoverer_thread (discoverer.c:838) ==25890== by 0x4913D9: zbx_thread_start (threads.c:128) ==25890== by 0x421A51: MAIN_ZABBIX_ENTRY (server.c:1067) ==25890== by 0x488012: daemon_start (daemon.c:392) ==25890== Address 0x15e6f6e0 is 0 bytes inside a block of size 12 free'd ==25890== at 0x4C2FD18: free (vg_replace_malloc.c:530) ==25890== by 0x54532D8: netsnmp_ds_set_string (in /usr/lib64/libnetsnmp.so.30.0.3) ==25890== by 0x5423E0D: snmp_sess_open (in /usr/lib64/libnetsnmp.so.30.0.3) ==25890== by 0x5424078: snmp_open (in /usr/lib64/libnetsnmp.so.30.0.3) ==25890== by 0x42C375: zbx_snmp_open_session (checks_snmp.c:630) ==25890== by 0x42C375: get_values_snmp (checks_snmp.c:2097) ==25890== by 0x42CF8A: get_value_snmp (checks_snmp.c:2071) ==25890== by 0x426DE3: discover_service (discoverer.c:270) ==25890== by 0x426DE3: process_check (discoverer.c:360) ==25890== by 0x426DE3: process_checks (discoverer.c:455) ==25890== by 0x427994: process_rule (discoverer.c:539) ==25890== by 0x427994: process_discovery (discoverer.c:742) ==25890== by 0x427994: discoverer_thread (discoverer.c:838) ==25890== by 0x4913D9: zbx_thread_start (threads.c:128) ==25890== by 0x421A51: MAIN_ZABBIX_ENTRY (server.c:1067) ==25890== by 0x488012: daemon_start (daemon.c:392) ==25890== by 0x419333: main (server.c:819) ==25890== Block was alloc'd at ==25890== at 0x4C2EB6B: malloc (vg_replace_malloc.c:299) ==25890== by 0x6D02CE9: strdup (in /usr/lib64/libc-2.25.so) ==25890== by 0x54532ED: netsnmp_ds_set_string (in /usr/lib64/libnetsnmp.so.30.0.3) ==25890== by 0x5423E0D: snmp_sess_open (in /usr/lib64/libnetsnmp.so.30.0.3) ==25890== by 0x5424078: snmp_open (in /usr/lib64/libnetsnmp.so.30.0.3) ==25890== by 0x42C375: zbx_snmp_open_session (checks_snmp.c:630) ==25890== by 0x42C375: get_values_snmp (checks_snmp.c:2097) ==25890== by 0x42CF8A: get_value_snmp (checks_snmp.c:2071) ==25890== by 0x426DE3: discover_service (discoverer.c:270) ==25890== by 0x426DE3: process_check (discoverer.c:360) ==25890== by 0x426DE3: process_checks (discoverer.c:455) ==25890== by 0x427994: process_rule (discoverer.c:539) ==25890== by 0x427994: process_discovery (discoverer.c:742) ==25890== by 0x427994: discoverer_thread (discoverer.c:838) ==25890== by 0x4913D9: zbx_thread_start (threads.c:128) ==25890== by 0x421A51: MAIN_ZABBIX_ENTRY (server.c:1067) ==25890== by 0x488012: daemon_start (daemon.c:392) ==25890== ==25890== Invalid read of size 1 ==25890== at 0x4C31B94: strlen (vg_replace_strmem.c:458) ==25890== by 0x6D02CDD: strdup (in /usr/lib64/libc-2.25.so) ==25890== by 0x54532ED: netsnmp_ds_set_string (in /usr/lib64/libnetsnmp.so.30.0.3) ==25890== by 0x5423E4F: snmp_sess_open (in /usr/lib64/libnetsnmp.so.30.0.3) ==25890== by 0x5424078: snmp_open (in /usr/lib64/libnetsnmp.so.30.0.3) ==25890== by 0x42C375: zbx_snmp_open_session (checks_snmp.c:630) ==25890== by 0x42C375: get_values_snmp (checks_snmp.c:2097) ==25890== by 0x42CF8A: get_value_snmp (checks_snmp.c:2071) ==25890== by 0x426DE3: discover_service (discoverer.c:270) ==25890== by 0x426DE3: process_check (discoverer.c:360) ==25890== by 0x426DE3: process_checks (discoverer.c:455) ==25890== by 0x427994: process_rule (discoverer.c:539) ==25890== by 0x427994: process_discovery (discoverer.c:742) ==25890== by 0x427994: discoverer_thread (discoverer.c:838) ==25890== by 0x4913D9: zbx_thread_start (threads.c:128) ==25890== by 0x421A51: MAIN_ZABBIX_ENTRY (server.c:1067) ==25890== by 0x488012: daemon_start (daemon.c:392) ==25890== Address 0x15e6f6e1 is 1 bytes inside a block of size 12 free'd ==25890== at 0x4C2FD18: free (vg_replace_malloc.c:530) ==25890== by 0x54532D8: netsnmp_ds_set_string (in /usr/lib64/libnetsnmp.so.30.0.3) ==25890== by 0x5423E0D: snmp_sess_open (in /usr/lib64/libnetsnmp.so.30.0.3) ==25890== by 0x5424078: snmp_open (in /usr/lib64/libnetsnmp.so.30.0.3) ==25890== by 0x42C375: zbx_snmp_open_session (checks_snmp.c:630) ==25890== by 0x42C375: get_values_snmp (checks_snmp.c:2097) ==25890== by 0x42CF8A: get_value_snmp (checks_snmp.c:2071) ==25890== by 0x426DE3: discover_service (discoverer.c:270) ==25890== by 0x426DE3: process_check (discoverer.c:360) ==25890== by 0x426DE3: process_checks (discoverer.c:455) ==25890== by 0x427994: process_rule (discoverer.c:539) ==25890== by 0x427994: process_discovery (discoverer.c:742) ==25890== by 0x427994: discoverer_thread (discoverer.c:838) ==25890== by 0x4913D9: zbx_thread_start (threads.c:128) ==25890== by 0x421A51: MAIN_ZABBIX_ENTRY (server.c:1067) ==25890== by 0x488012: daemon_start (daemon.c:392) ==25890== by 0x419333: main (server.c:819) ==25890== Block was alloc'd at ==25890== at 0x4C2EB6B: malloc (vg_replace_malloc.c:299) ==25890== by 0x6D02CE9: strdup (in /usr/lib64/libc-2.25.so) ==25890== by 0x54532ED: netsnmp_ds_set_string (in /usr/lib64/libnetsnmp.so.30.0.3) ==25890== by 0x5423E0D: snmp_sess_open (in /usr/lib64/libnetsnmp.so.30.0.3) ==25890== by 0x5424078: snmp_open (in /usr/lib64/libnetsnmp.so.30.0.3) ==25890== by 0x42C375: zbx_snmp_open_session (checks_snmp.c:630) ==25890== by 0x42C375: get_values_snmp (checks_snmp.c:2097) ==25890== by 0x42CF8A: get_value_snmp (checks_snmp.c:2071) ==25890== by 0x426DE3: discover_service (discoverer.c:270) ==25890== by 0x426DE3: process_check (discoverer.c:360) ==25890== by 0x426DE3: process_checks (discoverer.c:455) ==25890== by 0x427994: process_rule (discoverer.c:539) ==25890== by 0x427994: process_discovery (discoverer.c:742) ==25890== by 0x427994: discoverer_thread (discoverer.c:838) ==25890== by 0x4913D9: zbx_thread_start (threads.c:128) ==25890== by 0x421A51: MAIN_ZABBIX_ENTRY (server.c:1067) ==25890== by 0x488012: daemon_start (daemon.c:392) ==25890== ==25890== Invalid read of size 8 ==25890== at 0x4C32FF0: memcpy@GLIBC_2.2.5 (vg_replace_strmem.c:1021) ==25890== by 0x54532ED: netsnmp_ds_set_string (in /usr/lib64/libnetsnmp.so.30.0.3) ==25890== by 0x5423E4F: snmp_sess_open (in /usr/lib64/libnetsnmp.so.30.0.3) ==25890== by 0x5424078: snmp_open (in /usr/lib64/libnetsnmp.so.30.0.3) ==25890== by 0x42C375: zbx_snmp_open_session (checks_snmp.c:630) ==25890== by 0x42C375: get_values_snmp (checks_snmp.c:2097) ==25890== by 0x42CF8A: get_value_snmp (checks_snmp.c:2071) ==25890== by 0x426DE3: discover_service (discoverer.c:270) ==25890== by 0x426DE3: process_check (discoverer.c:360) ==25890== by 0x426DE3: process_checks (discoverer.c:455) ==25890== by 0x427994: process_rule (discoverer.c:539) ==25890== by 0x427994: process_discovery (discoverer.c:742) ==25890== by 0x427994: discoverer_thread (discoverer.c:838) ==25890== by 0x4913D9: zbx_thread_start (threads.c:128) ==25890== by 0x421A51: MAIN_ZABBIX_ENTRY (server.c:1067) ==25890== by 0x488012: daemon_start (daemon.c:392) ==25890== by 0x419333: main (server.c:819) ==25890== Address 0x15e6f6e0 is 0 bytes inside a block of size 12 free'd ==25890== at 0x4C2FD18: free (vg_replace_malloc.c:530) ==25890== by 0x54532D8: netsnmp_ds_set_string (in /usr/lib64/libnetsnmp.so.30.0.3) ==25890== by 0x5423E0D: snmp_sess_open (in /usr/lib64/libnetsnmp.so.30.0.3) ==25890== by 0x5424078: snmp_open (in /usr/lib64/libnetsnmp.so.30.0.3) ==25890== by 0x42C375: zbx_snmp_open_session (checks_snmp.c:630) ==25890== by 0x42C375: get_values_snmp (checks_snmp.c:2097) ==25890== by 0x42CF8A: get_value_snmp (checks_snmp.c:2071) ==25890== by 0x426DE3: discover_service (discoverer.c:270) ==25890== by 0x426DE3: process_check (discoverer.c:360) ==25890== by 0x426DE3: process_checks (discoverer.c:455) ==25890== by 0x427994: process_rule (discoverer.c:539) ==25890== by 0x427994: process_discovery (discoverer.c:742) ==25890== by 0x427994: discoverer_thread (discoverer.c:838) ==25890== by 0x4913D9: zbx_thread_start (threads.c:128) ==25890== by 0x421A51: MAIN_ZABBIX_ENTRY (server.c:1067) ==25890== by 0x488012: daemon_start (daemon.c:392) ==25890== by 0x419333: main (server.c:819) ==25890== Block was alloc'd at ==25890== at 0x4C2EB6B: malloc (vg_replace_malloc.c:299) ==25890== by 0x6D02CE9: strdup (in /usr/lib64/libc-2.25.so) ==25890== by 0x54532ED: netsnmp_ds_set_string (in /usr/lib64/libnetsnmp.so.30.0.3) ==25890== by 0x5423E0D: snmp_sess_open (in /usr/lib64/libnetsnmp.so.30.0.3) ==25890== by 0x5424078: snmp_open (in /usr/lib64/libnetsnmp.so.30.0.3) ==25890== by 0x42C375: zbx_snmp_open_session (checks_snmp.c:630) ==25890== by 0x42C375: get_values_snmp (checks_snmp.c:2097) ==25890== by 0x42CF8A: get_value_snmp (checks_snmp.c:2071) ==25890== by 0x426DE3: discover_service (discoverer.c:270) ==25890== by 0x426DE3: process_check (discoverer.c:360) ==25890== by 0x426DE3: process_checks (discoverer.c:455) ==25890== by 0x427994: process_rule (discoverer.c:539) ==25890== by 0x427994: process_discovery (discoverer.c:742) ==25890== by 0x427994: discoverer_thread (discoverer.c:838) ==25890== by 0x4913D9: zbx_thread_start (threads.c:128) ==25890== by 0x421A51: MAIN_ZABBIX_ENTRY (server.c:1067) ==25890== by 0x488012: daemon_start (daemon.c:392) ==25890== ==25890== Invalid read of size 2 ==25890== at 0x4C32F60: memcpy@GLIBC_2.2.5 (vg_replace_strmem.c:1021) ==25890== by 0x54532ED: netsnmp_ds_set_string (in /usr/lib64/libnetsnmp.so.30.0.3) ==25890== by 0x5423E4F: snmp_sess_open (in /usr/lib64/libnetsnmp.so.30.0.3) ==25890== by 0x5424078: snmp_open (in /usr/lib64/libnetsnmp.so.30.0.3) ==25890== by 0x42C375: zbx_snmp_open_session (checks_snmp.c:630) ==25890== by 0x42C375: get_values_snmp (checks_snmp.c:2097) ==25890== by 0x42CF8A: get_value_snmp (checks_snmp.c:2071) ==25890== by 0x426DE3: discover_service (discoverer.c:270) ==25890== by 0x426DE3: process_check (discoverer.c:360) ==25890== by 0x426DE3: process_checks (discoverer.c:455) ==25890== by 0x427994: process_rule (discoverer.c:539) ==25890== by 0x427994: process_discovery (discoverer.c:742) ==25890== by 0x427994: discoverer_thread (discoverer.c:838) ==25890== by 0x4913D9: zbx_thread_start (threads.c:128) ==25890== by 0x421A51: MAIN_ZABBIX_ENTRY (server.c:1067) ==25890== by 0x488012: daemon_start (daemon.c:392) ==25890== by 0x419333: main (server.c:819) ==25890== Address 0x15e6f6e8 is 8 bytes inside a block of size 12 free'd ==25890== at 0x4C2FD18: free (vg_replace_malloc.c:530) ==25890== by 0x54532D8: netsnmp_ds_set_string (in /usr/lib64/libnetsnmp.so.30.0.3) ==25890== by 0x5423E0D: snmp_sess_open (in /usr/lib64/libnetsnmp.so.30.0.3) ==25890== by 0x5424078: snmp_open (in /usr/lib64/libnetsnmp.so.30.0.3) ==25890== by 0x42C375: zbx_snmp_open_session (checks_snmp.c:630) ==25890== by 0x42C375: get_values_snmp (checks_snmp.c:2097) ==25890== by 0x42CF8A: get_value_snmp (checks_snmp.c:2071) ==25890== by 0x426DE3: discover_service (discoverer.c:270) ==25890== by 0x426DE3: process_check (discoverer.c:360) ==25890== by 0x426DE3: process_checks (discoverer.c:455) ==25890== by 0x427994: process_rule (discoverer.c:539) ==25890== by 0x427994: process_discovery (discoverer.c:742) ==25890== by 0x427994: discoverer_thread (discoverer.c:838) ==25890== by 0x4913D9: zbx_thread_start (threads.c:128) ==25890== by 0x421A51: MAIN_ZABBIX_ENTRY (server.c:1067) ==25890== by 0x488012: daemon_start (daemon.c:392) ==25890== by 0x419333: main (server.c:819) ==25890== Block was alloc'd at ==25890== at 0x4C2EB6B: malloc (vg_replace_malloc.c:299) ==25890== by 0x6D02CE9: strdup (in /usr/lib64/libc-2.25.so) ==25890== by 0x54532ED: netsnmp_ds_set_string (in /usr/lib64/libnetsnmp.so.30.0.3) ==25890== by 0x5423E0D: snmp_sess_open (in /usr/lib64/libnetsnmp.so.30.0.3) ==25890== by 0x5424078: snmp_open (in /usr/lib64/libnetsnmp.so.30.0.3) ==25890== by 0x42C375: zbx_snmp_open_session (checks_snmp.c:630) ==25890== by 0x42C375: get_values_snmp (checks_snmp.c:2097) ==25890== by 0x42CF8A: get_value_snmp (checks_snmp.c:2071) ==25890== by 0x426DE3: discover_service (discoverer.c:270) ==25890== by 0x426DE3: process_check (discoverer.c:360) ==25890== by 0x426DE3: process_checks (discoverer.c:455) ==25890== by 0x427994: process_rule (discoverer.c:539) ==25890== by 0x427994: process_discovery (discoverer.c:742) ==25890== by 0x427994: discoverer_thread (discoverer.c:838) ==25890== by 0x4913D9: zbx_thread_start (threads.c:128) ==25890== by 0x421A51: MAIN_ZABBIX_ENTRY (server.c:1067) ==25890== by 0x488012: daemon_start (daemon.c:392) Only reproducible if SourceIP=127.0.0.1 is set |
Comment by Andrea Biscuola (Inactive) [ 2017 Sep 19 ] |
Before moving on, I would like to make some other tests. If wiper is right I would like to be able to properly identify the root cause of it and report the bug. |
Comment by Andrea Biscuola (Inactive) [ 2017 Sep 19 ] |
wiper vso Did a last check and it's a use-after-free problem. Going to report it to the net-snmp developers |
Comment by Andrea Biscuola (Inactive) [ 2017 Sep 19 ] |
Documented the known issue for 2.2, 3.0, 3.2 and 3.4. Can be checked from here: https://www.zabbix.com/documentation/2.2/manual/installation/known_issues. In particular, added a line for the SNMP monitoring reporting the problem and the workaround for it |
Comment by Andrea Biscuola (Inactive) [ 2017 Sep 19 ] |
Bug opened on the net-snmp project: |
Comment by Andrea Biscuola (Inactive) [ 2017 Oct 02 ] |
Update: A local fix was committed by the maintainer of the net-snmp package on OpenBSD Stuart Henderson: https://github.com/openbsd/ports/commit/374dee5d10ee0faecabacf9a55b55e08f648ec2f This mean that for OpenBSD 6.2, the issue will be solved until the upstream net-snmp project will fix the issue for all the other platforms. Once OpenBSD 6.2 will be released, we could amend the documentation for specifying that the problem happen only for OpenBSD <= 6.1 |