-
Incident report
-
Resolution: Duplicate
-
Blocker
-
None
-
2.2.0
-
Zabbix Server : Red Hat Enterprise Linux Server release 6.3 (Santiago) Linux S66CJ198 2.6.32-279.el6.x86_64 #1 SMP Wed Jun 13 18:24:36 EDT 2012 x86_64 x86_64 x86_64 GNU/Linux
Database : Oracle 11.2.0.3
Zabbix Server process kept crashing(even on restart) with the following error recorded in its logs.
========================== Begin Log======================================
27686:20140206:115718.435 Got signal [signal:11(SIGSEGV),reason:1,refaddr:(nil)]. Crashing ...
27686:20140206:115718.435 ====== Fatal information: ======
27686:20140206:115718.435 Program counter: 0x461e9a
27686:20140206:115718.435 === Registers: ===
27686:20140206:115718.435 r8 = 334e98fee8 = 220361981672 = 220361981672
27686:20140206:115718.435 r9 = 1 = 1 = 1
27686:20140206:115718.435 r10 = 39 = 57 = 57
27686:20140206:115718.435 r11 = 202 = 514 = 514
27686:20140206:115718.435 r12 = 4a5dad = 4873645 = 4873645
27686:20140206:115718.435 r13 = 800 = 2048 = 2048
27686:20140206:115718.435 r14 = 7fff38f72ad0 = 140734149110480 = 140734149110480
27686:20140206:115718.435 r15 = 7fff38f72ad8 = 140734149110488 = 140734149110488
27686:20140206:115718.436 rdi = 0 = 0 = 0
27686:20140206:115718.436 rsi = 4a5dad = 4873645 = 4873645
27686:20140206:115718.436 rbp = 0 = 0 = 0
27686:20140206:115718.436 rbx = 0 = 0 = 0
27686:20140206:115718.436 rdx = 0 = 0 = 0
27686:20140206:115718.436 rax = 1c96c20 = 29977632 = 29977632
27686:20140206:115718.436 rcx = 8000 = 32768 = 32768
27686:20140206:115718.436 rsp = 7fff38f72a60 = 140734149110368 = 140734149110368
27686:20140206:115718.436 rip = 461e9a = 4595354 = 4595354
27686:20140206:115718.436 efl = 10246 = 66118 = 66118
27686:20140206:115718.436 csgsfs = 2148000000000033 = 2398166801574789171 = 2398166801574789171
27686:20140206:115718.436 err = 4 = 4 = 4
27686:20140206:115718.436 trapno = e = 14 = 14
27686:20140206:115718.436 oldmask = 0 = 0 = 0
27686:20140206:115718.437 cr2 = 0 = 0 = 0
27686:20140206:115718.437 === Backtrace: ===
27686:20140206:115718.438 10: ./zabbix_server: alerter [sending alerts](print_fatal_info+0xa6) [0x45a7e6]
27686:20140206:115718.438 9: ./zabbix_server: alerter [sending alerts]() [0x45ae96]
27686:20140206:115718.438 8: /lib64/libc.so.6() [0x334e6329a0]
27686:20140206:115718.438 7: ./zabbix_server: alerter [sending alerts](zbx_get_escape_string_len+0xa) [0x461e9a]
27686:20140206:115718.438 6: ./zabbix_server: alerter [sending alerts](zbx_dyn_escape_string+0x13) [0x462a03]
27686:20140206:115718.438 5: ./zabbix_server: alerter [sending alerts](execute_action+0x127) [0x415707]
27686:20140206:115718.438 4: ./zabbix_server: alerter [sending alerts](main_alerter_loop+0x211) [0x415b31]
27686:20140206:115718.438 3: ./zabbix_server: alerter [sending alerts](MAIN_ZABBIX_ENTRY+0x820) [0x414e40]
27686:20140206:115718.438 2: ./zabbix_server: alerter [sending alerts](daemon_start+0x1a7) [0x459ec7]
27686:20140206:115718.438 1: /lib64/libc.so.6(__libc_start_main+0xfd) [0x334e61ed1d]
27686:20140206:115718.439 0: ./zabbix_server: alerter [sending alerts]() [0x40fee9]
==============================End Log=================================================
After doing some digging, I found out that a push notification action that was configured recently had the "Message" portion left as null. When an alert was triggered for this action zabbix_server promptly crashed. I tried to restart the zabbix_server process and it still kept crashing.
Workaround
=========
I connected to the Zabbix database and patched the message column of the 3 problematic rows in the "alerts" table to a not-null value. I was able to startup Zabbix server without issues after this.
- duplicates
-
ZBX-7400 crash in dbsyncer after upgrade to 2.2.0
- Closed