[ZBX-10061] Zabbix server crash Created: 2015 Nov 09  Updated: 2017 May 30  Resolved: 2015 Nov 13

Status: Closed
Project: ZABBIX BUGS AND ISSUES
Component/s: Server (S)
Affects Version/s: 3.0.0alpha4
Fix Version/s: 3.0.0alpha5

Type: Incident report Priority: Blocker
Reporter: Alexander Vladishev Assignee: Unassigned
Resolution: Fixed Votes: 0
Labels: crash
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Attachments: File ZBX-10061.patch     File zabbix_server.log     File zabbix_server.log     File zabbix_server.objdump.gz    

 Description    
 29767:20151109:082408.702 server #4 started [poller #2]
...
 29767:20151109:132254.108 Got signal [signal:11(SIGSEGV),reason:1,refaddr:(nil)]. Crashing ...
 29767:20151109:132254.108 ====== Fatal information: ======
 29767:20151109:132254.108 Program counter: 0x44cfc6
 29767:20151109:132254.108 === Registers: ===
 29767:20151109:132254.108 r8      =           1ffe4d =              2096717 =              2096717
 29767:20151109:132254.108 r9      =     7fd2f56c34f0 =      140544037369072 =      140544037369072
 29767:20151109:132254.108 r10     = 7070757320746f4e =  8102104866965712718 =  8102104866965712718
 29767:20151109:132254.109 r11     = 796220646574726f =  8746589041877873263 =  8746589041877873263
 29767:20151109:132254.109 r12     =     7fd2f79b7fa8 =      140544074022824 =      140544074022824
 29767:20151109:132254.109 r13     =                d =                   13 =                   13
 29767:20151109:132254.109 r14     =            4dd60 =               318816 =               318816
 29767:20151109:132254.109 r15     =     7fd2f796a170 =      140544073703792 =      140544073703792
 29767:20151109:132254.109 rdi     =                0 =                    0 =                    0
 29767:20151109:132254.109 rsi     =                0 =                    0 =                    0
 29767:20151109:132254.109 rbp     =             114c =                 4428 =                 4428
 29767:20151109:132254.109 rbx     =                0 =                    0 =                    0
 29767:20151109:132254.109 rdx     =   746e6567412078 =    32772479101509752 =    32772479101509752
 29767:20151109:132254.109 rax     =                0 =                    0 =                    0
 29767:20151109:132254.109 rcx     = ffffffffffffffff = 18446744073709551615 =                   -1
 29767:20151109:132254.109 rsp     =     7fff93592d10 =      140735665483024 =      140735665483024
 29767:20151109:132254.109 rip     =           44cfc6 =              4509638 =              4509638
 29767:20151109:132254.109 efl     =            10202 =                66050 =                66050
 29767:20151109:132254.109 csgsfs  =               33 =                   51 =                   51
 29767:20151109:132254.109 err     =                4 =                    4 =                    4
 29767:20151109:132254.109 trapno  =                e =                   14 =                   14
 29767:20151109:132254.109 oldmask =                0 =                    0 =                    0
 29767:20151109:132254.109 cr2     =                0 =                    0 =                    0
 29767:20151109:132254.109 === Backtrace: ===
 29767:20151109:132254.132 13: /home/monitoring/zabbix-bin/sbin/zabbix_server: poller #2 [got 23 values in 0.058998 sec, getting values](print_fatal_info+0x19d) [0x4699bd]
 29767:20151109:132254.132 12: /home/monitoring/zabbix-bin/sbin/zabbix_server: poller #2 [got 23 values in 0.058998 sec, getting values]() [0x469d0e]
 29767:20151109:132254.132 11: /lib/x86_64-linux-gnu/libc.so.6(+0x36d40) [0x7fd2f55afd40]
 29767:20151109:132254.132 10: /home/monitoring/zabbix-bin/sbin/zabbix_server: poller #2 [got 23 values in 0.058998 sec, getting values]() [0x44cfc6]
 29767:20151109:132254.132 9: /home/monitoring/zabbix-bin/sbin/zabbix_server: poller #2 [got 23 values in 0.058998 sec, getting values]() [0x44d2bd]
 29767:20151109:132254.132 8: /home/monitoring/zabbix-bin/sbin/zabbix_server: poller #2 [got 23 values in 0.058998 sec, getting values](dc_flush_history+0x67) [0x44fd52]
 29767:20151109:132254.132 7: /home/monitoring/zabbix-bin/sbin/zabbix_server: poller #2 [got 23 values in 0.058998 sec, getting values]() [0x4248d2]
 29767:20151109:132254.132 6: /home/monitoring/zabbix-bin/sbin/zabbix_server: poller #2 [got 23 values in 0.058998 sec, getting values](poller_thread+0x139) [0x424ae0]
 29767:20151109:132254.132 5: /home/monitoring/zabbix-bin/sbin/zabbix_server: poller #2 [got 23 values in 0.058998 sec, getting values](zbx_thread_start+0x1f) [0x46a50a]
 29767:20151109:132254.132 4: /home/monitoring/zabbix-bin/sbin/zabbix_server: poller #2 [got 23 values in 0.058998 sec, getting values](MAIN_ZABBIX_ENTRY+0x49f) [0x417d62]
 29767:20151109:132254.132 3: /home/monitoring/zabbix-bin/sbin/zabbix_server: poller #2 [got 23 values in 0.058998 sec, getting values](daemon_start+0x288) [0x46929f]
 29767:20151109:132254.132 2: /home/monitoring/zabbix-bin/sbin/zabbix_server: poller #2 [got 23 values in 0.058998 sec, getting values](main+0x58f) [0x417762]
 29767:20151109:132254.132 1: /lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0xf5) [0x7fd2f559aec5]
 29767:20151109:132254.132 0: /home/monitoring/zabbix-bin/sbin/zabbix_server: poller #2 [got 23 values in 0.058998 sec, getting values]() [0x412809]


 Comments   
Comment by Alexander Vladishev [ 2015 Nov 10 ]

Same issue with Zabbix proxy:

 26511:20151109:130746.967 === Backtrace: ===
 26511:20151109:130746.977 12: /usr/local/sbin/zabbix_proxy: poller #3 [got 4 values in 0.004882 sec, getting values](print_fatal_info+0x242) [0x80a6cd2]
 26511:20151109:130746.978 11: /usr/local/sbin/zabbix_proxy: poller #3 [got 4 values in 0.004882 sec, getting values]() [0x80a742d]
 26511:20151109:130746.978 10: [0xf7763410]
 26511:20151109:130746.978 9: /usr/local/sbin/zabbix_proxy: poller #3 [got 4 values in 0.004882 sec, getting values]() [0x8085c00]
 26511:20151109:130746.978 8: /usr/local/sbin/zabbix_proxy: poller #3 [got 4 values in 0.004882 sec, getting values](dc_flush_history+0x33b) [0x8085fbb]
 26511:20151109:130746.978 7: /usr/local/sbin/zabbix_proxy: poller #3 [got 4 values in 0.004882 sec, getting values]() [0x80603d8]
 26511:20151109:130746.978 6: /usr/local/sbin/zabbix_proxy: poller #3 [got 4 values in 0.004882 sec, getting values](poller_thread+0xf3) [0x80614a3]
 26511:20151109:130746.978 5: /usr/local/sbin/zabbix_proxy: poller #3 [got 4 values in 0.004882 sec, getting values](zbx_thread_start+0x49) [0x80a7a09]
 26511:20151109:130746.978 4: /usr/local/sbin/zabbix_proxy: poller #3 [got 4 values in 0.004882 sec, getting values](MAIN_ZABBIX_ENTRY+0x373) [0x8057793]
 26511:20151109:130746.978 3: /usr/local/sbin/zabbix_proxy: poller #3 [got 4 values in 0.004882 sec, getting values](daemon_start+0x1b0) [0x80a5e60]
 26511:20151109:130746.978 2: /usr/local/sbin/zabbix_proxy: poller #3 [got 4 values in 0.004882 sec, getting values](main+0x592) [0x8058072]
 26511:20151109:130746.978 1: /lib/libc.so.6(__libc_start_main+0xe6) [0xf7500c96]
 26511:20151109:130746.978 0: /usr/local/sbin/zabbix_proxy: poller #3 [got 4 values in 0.004882 sec, getting values]() [0x8056b21]
Comment by Alexander Vladishev [ 2015 Nov 11 ]

Attached log file with debug information.

Comment by Glebs Ivanovskis (Inactive) [ 2015 Nov 11 ]

This is related to ZBXNEXT-444. DCadd_history_log() inserts NULL into history cache in case of meta update packets and consequent attempt to DCmove_text() this NULL during DCvacuum_text() causes crash on strlen(NULL).

Comment by Glebs Ivanovskis (Inactive) [ 2015 Nov 12 ]

Fixed in development branch svn://svn.zabbix.com/branches/dev/ZBX-10061 revision 56690.

Patch attached.

Comment by Glebs Ivanovskis (Inactive) [ 2015 Nov 12 ]

Fixed in pre-3.0.0alpha5 (trunk) r56705.

Generated at Fri Apr 26 23:47:07 EEST 2024 using Jira 9.12.4#9120004-sha1:625303b708afdb767e17cb2838290c41888e9ff0.