[ZBX-15611] Possible crash when all preprocessing steps are removed from item Created: 2019 Feb 07  Updated: 2024 Apr 10  Resolved: 2019 Feb 08

Status: Closed
Project: ZABBIX BUGS AND ISSUES
Component/s: Server (S)
Affects Version/s: 4.0.4, 4.2.0alpha3
Fix Version/s: 4.0.5rc1, 4.2.0beta1, 4.2 (plan)

Type: Problem report Priority: Critical
Reporter: Vladislavs Sokurenko Assignee: Vladislavs Sokurenko
Resolution: Fixed Votes: 0
Labels: crash
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Attachments: File ZBX-15611.diff    
Issue Links:
Causes
caused by ZBX-12802 Optimize memory usage of L2 item conf... Closed
Duplicate
Team: Team A
Sprint: Sprint 49 (Feb 2019)
Story Points: 0.5

 Description   

Steps to reproduce:

  1. Create item with 3 preprocessing steps and reload cache
  2. Remove all preprocessing steps and reload cache

Use following patch in order to set specific garbage value and confirm that crash is possible

Index: src/libs/zbxdbcache/dbconfig.c
===================================================================
--- src/libs/zbxdbcache/dbconfig.c    (revision 89538)
+++ src/libs/zbxdbcache/dbconfig.c    (working copy)
@@ -4359,6 +4359,7 @@
                 if (0 == preprocitem->preproc_ops.values_num)
                 {
                     zbx_vector_ptr_destroy(&preprocitem->preproc_ops);
+                    preprocitem->preproc_ops = 2; /* set freed memory to garbage */
                     zbx_hashset_remove_direct(&config->preprocitems, preprocitem);
                 }
                 else

Result:

vector.c:28:1: runtime error: null pointer passed as argument 1, which is declared to never be null
dbconfig.c:4264:23: runtime error: load of null pointer of type 'struct zbx_dc_preproc_op_t *'
==163110== Invalid read of size 8
==163110==    at 0x5AA48A: dc_compare_preprocops_by_step (dbconfig.c:4264)
==163110==    by 0x63ADF84: msort_with_tmp.part.0 (in /usr/lib64/libc-2.28.so)
==163110==    by 0x63AE1E5: qsort_r (in /usr/lib64/libc-2.28.so)
==163110==    by 0x65FE8B: zbx_vector_ptr_sort (vector.c:28)
==163110==    by 0x5AB2A1: DCsync_item_preproc (dbconfig.c:4382)
==163110==    by 0x5AD9A3: DCsync_configuration (dbconfig.c:4861)
==163110==    by 0x432C43: dbconfig_thread (dbconfig.c:93)
==163110==    by 0x66898C: zbx_thread_start (threads.c:132)
==163110==    by 0x41F1BF: MAIN_ZABBIX_ENTRY (server.c:1113)
==163110==    by 0x637F41: daemon_start (daemon.c:392)
==163110==    by 0x41E5BD: main (server.c:867)
==163110==  Address 0x0 is not stack'd, malloc'd or (recently) free'd
==163110== 
163110:20190207:093344.452 Got signal [signal:11(SIGSEGV),reason:1,refaddr:(nil)]. Crashing ...
163110:20190207:093344.453 ====== Fatal information: ======
163110:20190207:093344.454 Program counter: 0x5aa48a
163110:20190207:093344.456 === Registers: ===
163110:20190207:093344.458 r8      =                1 =                    1 =                    1
163110:20190207:093344.461 r9      =       1ffeffc900 =         137422162176 =         137422162176
163110:20190207:093344.461 r10     =                0 =                    0 =                    0
163110:20190207:093344.461 r11     =                2 =                    2 =                    2
163110:20190207:093344.462 r12     =                1 =                    1 =                    1
163110:20190207:093344.462 r13     =                0 =                    0 =                    0
163110:20190207:093344.462 r14     =                8 =                    8 =                    8
163110:20190207:093344.462 r15     =       1ffeffcfd0 =         137422163920 =         137422163920
163110:20190207:093344.462 rdi     =       1ffeffcce1 =         137422163169 =         137422163169
163110:20190207:093344.463 rsi     =       1ffeffcf10 =         137422163728 =         137422163728
163110:20190207:093344.463 rbp     =       1ffeffcf50 =         137422163792 =         137422163792
163110:20190207:093344.463 rbx     =                8 =                    8 =                    8
163110:20190207:093344.463 rdx     =               6d =                  109 =                  109
163110:20190207:093344.464 rax     =                0 =                    0 =                    0
163110:20190207:093344.464 rcx     =          5869b4b =             92707659 =             92707659
163110:20190207:093344.464 rsp     =       1ffeffcf20 =         137422163744 =         137422163744
163110:20190207:093344.464 rip     =           5aa48a =              5940362 =              5940362
163110:20190207:093344.465 efl     =                4 =                    4 =                    4
163110:20190207:093344.465 csgsfs  =                0 =                    0 =                    0
163110:20190207:093344.465 err     =                4 =                    4 =                    4
163110:20190207:093344.465 trapno  =                e =                   14 =                   14
163110:20190207:093344.466 oldmask =                0 =                    0 =                    0
163110:20190207:093344.466 cr2     =                0 =                    0 =                    0
163110:20190207:093344.466 === Backtrace: ===
163110:20190207:093344.501 16: ./sbin/zabbix_server: configuration syncer [synced configuration in 1549524824.240268 sec, syncing configuration](zbx_backtrace+0x53) [0x6388ed]
163110:20190207:093344.501 15: ./sbin/zabbix_server: configuration syncer [synced configuration in 1549524824.240268 sec, syncing configuration](zbx_log_fatal_info+0x367) [0x638fc2]
163110:20190207:093344.501 14: ./sbin/zabbix_server: configuration syncer [synced configuration in 1549524824.240268 sec, syncing configuration]() [0x639480]
163110:20190207:093344.502 13: /lib64/libpthread.so.0(+0x13030) [0x4ca0030]
163110:20190207:093344.502 12: ./sbin/zabbix_server: configuration syncer [synced configuration in 1549524824.240268 sec, syncing configuration]() [0x5aa48a]
163110:20190207:093344.502 11: /lib64/libc.so.6(+0x39f85) [0x63adf85]
163110:20190207:093344.503 10: /lib64/libc.so.6(qsort_r+0x246) [0x63ae1e6]
163110:20190207:093344.503 9: ./sbin/zabbix_server: configuration syncer [synced configuration in 1549524824.240268 sec, syncing configuration](zbx_vector_ptr_sort+0x123) [0x65fe8c]
163110:20190207:093344.503 8: ./sbin/zabbix_server: configuration syncer [synced configuration in 1549524824.240268 sec, syncing configuration]() [0x5ab2a2]
163110:20190207:093344.504 7: ./sbin/zabbix_server: configuration syncer [synced configuration in 1549524824.240268 sec, syncing configuration](DCsync_configuration+0xa69) [0x5ad9a4]
163110:20190207:093344.504 6: ./sbin/zabbix_server: configuration syncer [synced configuration in 1549524824.240268 sec, syncing configuration](dbconfig_thread+0x203) [0x432c44]
163110:20190207:093344.504 5: ./sbin/zabbix_server: configuration syncer [synced configuration in 1549524824.240268 sec, syncing configuration](zbx_thread_start+0x6c) [0x66898d]
163110:20190207:093344.505 4: ./sbin/zabbix_server: configuration syncer [synced configuration in 1549524824.240268 sec, syncing configuration](MAIN_ZABBIX_ENTRY+0xbe7) [0x41f1c0]
163110:20190207:093344.505 3: ./sbin/zabbix_server: configuration syncer [synced configuration in 1549524824.240268 sec, syncing configuration](daemon_start+0x531) [0x637f42]
163110:20190207:093344.505 2: ./sbin/zabbix_server: configuration syncer [synced configuration in 1549524824.240268 sec, syncing configuration](main+0x3d2) [0x41e5be]
163110:20190207:093344.505 1: /lib64/libc.so.6(__libc_start_main+0xf3) [0x6398413]
163110:20190207:093344.505 0: ./sbin/zabbix_server: configuration syncer [synced configuration in 1549524824.240268 sec, syncing configuration](_start+0x2e) [0x41c97e]


 Comments   
Comment by Vladislavs Sokurenko [ 2019 Feb 07 ]

Also possible crash when there are duplicate groups (same name but different id) but need to check with a customer if that's the case

==31520== Invalid read of size 1
==31520==    at 0x483CCE4: strcmp (vg_replace_strmem.c:849)
==31520==    by 0x588CF84: msort_with_tmp.part.0 (in /usr/lib64/libc-2.28.so)
==31520==    by 0x588CCF6: msort_with_tmp.part.0 (in /usr/lib64/libc-2.28.so)
==31520==    by 0x588CCF6: msort_with_tmp.part.0 (in /usr/lib64/libc-2.28.so)
==31520==    by 0x588CCF6: msort_with_tmp.part.0 (in /usr/lib64/libc-2.28.so)
==31520==    by 0x588D1E5: qsort_r (in /usr/lib64/libc-2.28.so)
==31520==    by 0x48C8DF: dc_hostgroups_update_cache (dbconfig.c:4635)
==31520==    by 0x48C8DF: DCsync_configuration (dbconfig.c:4822)
==31520==    by 0x421A35: dbconfig_thread (dbconfig.c:93)
==31520==    by 0x4B4315: zbx_thread_start (threads.c:132)
==31520==    by 0x41CA7D: MAIN_ZABBIX_ENTRY (server.c:1113)
==31520==    by 0x4A9EA0: daemon_start (daemon.c:392)
==31520==    by 0x41BBBB: main (server.c:867)
==31520==  Address 0x0 is not stack'd, malloc'd or (recently) free'd
==31520== 
 31520:20190207:143930.158 Got signal [signal:11(SIGSEGV),reason:1,refaddr:(nil)]. Crashing ...
 31520:20190207:143930.158 ====== Fatal information: ======
 31520:20190207:143930.159 Program counter: 0x483cce4
 31520:20190207:143930.160 === Registers: ===
 31520:20190207:143930.161 r8      =       1ffeffcd68 =         137422163304 =         137422163304
 31520:20190207:143930.162 r9      =                0 =                    0 =                    0
 31520:20190207:143930.163 r10     = ffffffe008d61390 = 18446743936418845584 =        -137290706032
 31520:20190207:143930.163 r11     =       1ffeffcd68 =         137422163304 =         137422163304
 31520:20190207:143930.163 r12     =                1 =                    1 =                    1
 31520:20190207:143930.164 r13     =          7d5e100 =            131457280 =            131457280
 31520:20190207:143930.164 r14     =                8 =                    8 =                    8
 31520:20190207:143930.164 r15     =       1ffeffcd60 =         137422163296 =         137422163296
 31520:20190207:143930.164 rdi     =                0 =                    0 =                    0
 31520:20190207:143930.164 rsi     =          7d5e704 =            131458820 =            131458820
 31520:20190207:143930.164 rbp     =                1 =                    1 =                    1
 31520:20190207:143930.165 rbx     =          7d5e108 =            131457288 =            131457288
 31520:20190207:143930.165 rdx     =                0 =                    0 =                    0
 31520:20190207:143930.165 rax     =          7d5ede8 =            131460584 =            131460584
 31520:20190207:143930.165 rcx     =                0 =                    0 =                    0
 31520:20190207:143930.165 rsp     =       1ffeffcb98 =         137422162840 =         137422162840
 31520:20190207:143930.165 rip     =          483cce4 =             75746532 =             75746532
 31520:20190207:143930.166 efl     =                0 =                    0 =                    0
 31520:20190207:143930.166 csgsfs  =                0 =                    0 =                    0
 31520:20190207:143930.166 err     =                4 =                    4 =                    4
 31520:20190207:143930.166 trapno  =                e =                   14 =                   14
 31520:20190207:143930.166 oldmask =                0 =                    0 =                    0
 31520:20190207:143930.167 cr2     =                0 =                    0 =                    0
 31520:20190207:143930.167 === Backtrace: ===
 31520:20190207:143930.199 17: ./sbin/zabbix_server: configuration syncer [synced configuration in 1549543170.094468 sec, syncing configuration](zbx_backtrace+0x48) [0x4aa468]
 31520:20190207:143930.199 16: ./sbin/zabbix_server: configuration syncer [synced configuration in 1549543170.094468 sec, syncing configuration](zbx_log_fatal_info+0x147) [0x4aa687]
 31520:20190207:143930.199 15: ./sbin/zabbix_server: configuration syncer [synced configuration in 1549543170.094468 sec, syncing configuration]() [0x4aa8f9]
 31520:20190207:143930.200 14: /lib64/libpthread.so.0(+0x13030) [0x4ca0030]
 31520:20190207:143930.200 13: /usr/lib64/valgrind/vgpreload_memcheck-amd64-linux.so(_vgr20160ZU_libcZdsoZa_strcmp+0x4) [0x483cce4]
 31520:20190207:143930.200 12: /lib64/libc.so.6(+0x39f85) [0x588cf85]
 31520:20190207:143930.200 11: /lib64/libc.so.6(+0x39cf7) [0x588ccf7]
 31520:20190207:143930.200 10: /lib64/libc.so.6(+0x39cf7) [0x588ccf7]
 31520:20190207:143930.200 9: /lib64/libc.so.6(+0x39cf7) [0x588ccf7]
 31520:20190207:143930.200 8: /lib64/libc.so.6(qsort_r+0x246) [0x588d1e6]
 31520:20190207:143930.201 7: ./sbin/zabbix_server: configuration syncer [synced configuration in 1549543170.094468 sec, syncing configuration](DCsync_configuration+0x2550) [0x48c8e0]
 31520:20190207:143930.201 6: ./sbin/zabbix_server: configuration syncer [synced configuration in 1549543170.094468 sec, syncing configuration](dbconfig_thread+0x106) [0x421a36]
 31520:20190207:143930.201 5: ./sbin/zabbix_server: configuration syncer [synced configuration in 1549543170.094468 sec, syncing configuration](zbx_thread_start+0x36) [0x4b4316]
 31520:20190207:143930.201 4: ./sbin/zabbix_server: configuration syncer [synced configuration in 1549543170.094468 sec, syncing configuration](MAIN_ZABBIX_ENTRY+0x6ee) [0x41ca7e]
 31520:20190207:143930.201 3: ./sbin/zabbix_server: configuration syncer [synced configuration in 1549543170.094468 sec, syncing configuration](daemon_start+0x1d1) [0x4a9ea1]
 31520:20190207:143930.201 2: ./sbin/zabbix_server: configuration syncer [synced configuration in 1549543170.094468 sec, syncing configuration](main+0x33c) [0x41bbbc]
 31520:20190207:143930.202 1: /lib64/libc.so.6(__libc_start_main+0xf3) [0x5877413]
 31520:20190207:143930.202 0: ./sbin/zabbix_server: configuration syncer [synced configuration in 1549543170.094468 sec, syncing configuration](_start+0x2e) [0x41bf2e]

Index: src/libs/zbxdbcache/dbconfig.c
===================================================================
--- src/libs/zbxdbcache/dbconfig.c	(revision 89538)
+++ src/libs/zbxdbcache/dbconfig.c	(working copy)
@@ -4150,7 +4150,7 @@
 		if (NULL == (group = (zbx_dc_hostgroup_t *)zbx_hashset_search(&config->hostgroups, &rowid)))
 			continue;
 
-		if (FAIL != (index = zbx_vector_ptr_search(&config->hostgroups_name, group, dc_compare_hgroups)))
+		if (FAIL != (index = zbx_vector_ptr_search(&config->hostgroups_name, group, ZBX_DEFAULT_UINT64_COMPARE_FUNC)))
 			zbx_vector_ptr_remove_noorder(&config->hostgroups_name, index);
 
 		if (ZBX_DC_HOSTGROUP_FLAGS_NONE != group->flags)

Can be checked with

select name from hstgrp group by name having COUNT(name) > 1;
Comment by Vladislavs Sokurenko [ 2019 Feb 08 ]

Fixed in:

  • pre-4.0.5rc1 r89627
  • pre-4.2.0alpha4 (trunk) r89628
Generated at Thu Apr 25 07:25:16 EEST 2024 using Jira 9.12.4#9120004-sha1:625303b708afdb767e17cb2838290c41888e9ff0.