[ZBX-15023] Crashing process zabbix_server Created: 2018 Oct 17  Updated: 2018 Oct 26  Resolved: 2018 Oct 23

Status: Closed
Project: ZABBIX BUGS AND ISSUES
Component/s: None
Affects Version/s: 4.0.0
Fix Version/s: None

Type: Incident report Priority: Critical
Reporter: Craig Hopkins Assignee: Unassigned
Resolution: Duplicate Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Attachments: Text File log_dump.txt    
Issue Links:
Duplicate
duplicates ZBX-14908 Crashing escalator process zabbix_ser... Closed

 Description   

We updated from 3.4.14 to 4.0.0 this morning. It ran for a few hours and now

27455:20181017:131007.906 === Backtrace: ===
27455:20181017:131007.906 17: /usr/sbin/zabbix_server: escalator #1 [processed 0 escalations in 0.000000 sec, processing escalations](zbx_log_fatal_info+0x142) [0x55dc98268930]
27455:20181017:131007.906 16: /usr/sbin/zabbix_server: escalator #1 [processed 0 escalations in 0.000000 sec, processing escalations](+0xf0d50) [0x55dc98268d50]
27455:20181017:131007.906 15: /lib/x86_64-linux-gnu/libpthread.so.0(+0x110c0) [0x7f7bd0e150c0]
27455:20181017:131007.906 14: /lib/x86_64-linux-gnu/libc.so.6(strlen+0x26) [0x7f7bcdcb9676]
27455:20181017:131007.906 13: /lib/x86_64-linux-gnu/libc.so.6(__strdup+0xe) [0x7f7bcdcb93ae]
27455:20181017:131007.906 12: /usr/sbin/zabbix_server: escalator #1 [processed 0 escalations in 0.000000 sec, processing escalations](zbx_strdup2+0x4f) [0x55dc9827d65b]
27455:20181017:131007.906 11: /usr/sbin/zabbix_server: escalator #1 [processed 0 escalations in 0.000000 sec, processing escalations](substitute_simple_macros+0x1f0d) [0x55dc98220f92]
27455:20181017:131007.906 10: /usr/sbin/zabbix_server: escalator #1 [processed 0 escalations in 0.000000 sec, processing escalations](+0x704d1) [0x55dc981e84d1]
27455:20181017:131007.906 9: /usr/sbin/zabbix_server: escalator #1 [processed 0 escalations in 0.000000 sec, processing escalations](+0x73ad6) [0x55dc981ebad6]
27455:20181017:131007.906 8: /usr/sbin/zabbix_server: escalator #1 [processed 0 escalations in 0.000000 sec, processing escalations](+0x743c9) [0x55dc981ec3c9]
27455:20181017:131007.906 7: /usr/sbin/zabbix_server: escalator #1 [processed 0 escalations in 0.000000 sec, processing escalations](+0x75059) [0x55dc981ed059]
27455:20181017:131007.906 6: /usr/sbin/zabbix_server: escalator #1 [processed 0 escalations in 0.000000 sec, processing escalations](escalator_thread+0x183) [0x55dc981ed241]
27455:20181017:131007.906 5: /usr/sbin/zabbix_server: escalator #1 [processed 0 escalations in 0.000000 sec, processing escalations](zbx_thread_start+0x32) [0x55dc98275e5c]
27455:20181017:131007.906 4: /usr/sbin/zabbix_server: escalator #1 [processed 0 escalations in 0.000000 sec, processing escalations](MAIN_ZABBIX_ENTRY+0xc0e) [0x55dc981aff3c]
27455:20181017:131007.906 3: /usr/sbin/zabbix_server: escalator #1 [processed 0 escalations in 0.000000 sec, processing escalations](daemon_start+0x315) [0x55dc98268045]
27455:20181017:131007.906 2: /usr/sbin/zabbix_server: escalator #1 [processed 0 escalations in 0.000000 sec, processing escalations](main+0x2f6) [0x55dc981af32c]
27455:20181017:131007.906 1: /lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0xf1) [0x7f7bcdc592e1]
27455:20181017:131007.906 0: /usr/sbin/zabbix_server: escalator #1 [processed 0 escalations in 0.000000 sec, processing escalations](_start+0x2a) [0x55dc981ae51a]

I just end up with a lot of defunct processes that I have to kill -9 to remove. When I try to restart it, it will start up, then the pollers start, and then it faults and goes defunct again

 Comments   
Comment by Craig Hopkins [ 2018 Oct 17 ] 
# lsb_release -a
No LSB modules are available.
Distributor ID: Debian
Description: Debian GNU/Linux 9.5 (stretch)
Release: 9.5
Codename: stretch
Comment by Vladislavs Sokurenko [ 2018 Oct 17 ]

Can you please be so kind and provide more information about defunct processes ?

Comment by Craig Hopkins [ 2018 Oct 17 ] 
zabbix 30356 0.0 0.0 1863312 10864 ? S 13:50 0:00 \_ /usr/sbin/zabbix_server -c /etc/zabbix/zabbix_server.conf
zabbix 30357 0.0 0.0 0 0 ? Z 13:50 0:00 \_ [zabbix_server] <defunct>
zabbix 30358 0.0 0.0 0 0 ? Z 13:50 0:00 \_ [zabbix_server] <defunct>
zabbix 30359 0.0 0.0 0 0 ? Z 13:50 0:00 \_ [zabbix_server] <defunct>
zabbix 30360 0.0 0.0 0 0 ? Z 13:50 0:00 \_ [zabbix_server] <defunct>
zabbix 30361 0.0 0.0 0 0 ? Z 13:50 0:00 \_ [zabbix_server] <defunct>
zabbix 30362 0.0 0.0 0 0 ? Z 13:50 0:00 \_ [zabbix_server] <defunct>
zabbix 30363 0.0 0.0 0 0 ? Z 13:50 0:00 \_ [zabbix_server] <defunct>
zabbix 30364 0.0 0.0 0 0 ? Z 13:50 0:00 \_ [zabbix_server] <defunct>
zabbix 30365 0.0 0.0 0 0 ? Z 13:50 0:00 \_ [zabbix_server] <defunct>
zabbix 30366 0.0 0.0 0 0 ? Z 13:50 0:00 \_ [zabbix_server] <defunct>
zabbix 30367 0.0 0.0 0 0 ? Z 13:50 0:00 \_ [zabbix_server] <defunct>
zabbix 30368 0.0 0.0 0 0 ? Z 13:50 0:00 \_ [zabbix_server] <defunct>
zabbix 30369 0.0 0.0 0 0 ? Z 13:50 0:00 \_ [zabbix_server] <defunct>
zabbix 30370 0.0 0.0 0 0 ? Z 13:50 0:00 \_ [zabbix_server] <defunct>
zabbix 30371 0.0 0.0 0 0 ? Z 13:50 0:00 \_ [zabbix_server] <defunct>
zabbix 30372 0.0 0.0 0 0 ? Z 13:50 0:00 \_ [zabbix_server] <defunct>
zabbix 30373 0.0 0.0 0 0 ? Z 13:50 0:00 \_ [zabbix_server] <defunct>
zabbix 30374 0.0 0.0 0 0 ? Z 13:50 0:00 \_ [zabbix_server] <defunct>
zabbix 30375 0.0 0.0 0 0 ? Z 13:50 0:00 \_ [zabbix_server] <defunct>
zabbix 30376 0.0 0.0 0 0 ? Z 13:50 0:00 \_ [zabbix_server] <defunct>
zabbix 30377 0.0 0.0 0 0 ? Z 13:50 0:00 \_ [zabbix_server] <defunct>
zabbix 30378 0.0 0.0 1863048 10840 ? S 13:50 0:00 \_ /usr/sbin/zabbix_server -c /etc/zabbix/zabbix_server.conf
zabbix 30379 0.0 0.0 0 0 ? Z 13:50 0:00 \_ [zabbix_server] <defunct>
zabbix 30380 0.0 0.0 0 0 ? Z 13:50 0:00 \_ [zabbix_server] <defunct>
zabbix 30381 0.0 0.0 0 0 ? Z 13:50 0:00 \_ [zabbix_server] <defunct>
zabbix 30382 0.0 0.0 0 0 ? Z 13:50 0:00 \_ [zabbix_server] <defunct>
zabbix 30383 0.0 0.0 0 0 ? Z 13:50 0:00 \_ [zabbix_server] <defunct>
zabbix 30384 0.0 0.0 0 0 ? Z 13:50 0:00 \_ [zabbix_server] <defunct>
zabbix 30385 0.0 0.0 0 0 ? Z 13:50 0:00 \_ [zabbix_server] <defunct>
zabbix 30386 0.0 0.0 0 0 ? Z 13:50 0:00 \_ [zabbix_server] <defunct>
zabbix 30387 0.0 0.0 0 0 ? Z 13:50 0:00 \_ [zabbix_server] <defunct>
zabbix 30388 0.0 0.0 0 0 ? Z 13:50 0:00 \_ [zabbix_server] <defunct>
zabbix 30389 0.0 0.0 0 0 ? Z 13:50 0:00 \_ [zabbix_server] <defunct>
zabbix 30390 0.0 0.0 0 0 ? Z 13:50 0:00 \_ [zabbix_server] <defunct>
zabbix 30391 0.0 0.0 1863048 10564 ? S 13:50 0:00 \_ /usr/sbin/zabbix_server -c /etc/zabbix/zabbix_server.conf

 

Comment by Craig Hopkins [ 2018 Oct 17 ]

log_dump.txt

Comment by Craig Hopkins [ 2018 Oct 17 ]

Slightly further up the log, in case it helps:

 29125:20181017:133022.015 read line [rt4.mynetwork.com    : [0], 84 bytes, 5.21 ms (5.21 avg, 0% loss)]
 28641:20181017:133022.015 End of get_trigger_permission():rw
 28641:20181017:133022.015 In substitute_simple_macros() data:'{TRIGGER.STATE}: {TRIGGER.NAME}'
 28641:20181017:133022.015 Got signal [signal:11(SIGSEGV),reason:1,refaddr:(nil)]. Crashing ...
 28641:20181017:133022.015 ====== Fatal information: ======
 28641:20181017:133022.015 Program counter: 0x7f63206fd676
 28641:20181017:133022.015 === Registers: ===
 28641:20181017:133022.015 r8      =                0 =                    0 =                    0
 28641:20181017:133022.015 r9      =                a =                   10 =                   10
 28641:20181017:133022.015 r10     = fffffffffffff96a = 18446744073709549930 =                -1686
 28641:20181017:133022.015 r11     =                8 =                    8 =                    8
 28641:20181017:133022.015 r12     =                1 =                    1 =                    1
 28641:20181017:133022.015 r13     =     5631dd66e045 =       94772167893061 =       94772167893061
 28641:20181017:133022.015 r14     =                0 =                    0 =                    0
 28641:20181017:133022.015 r15     =                0 =                    0 =                    0
 28641:20181017:133022.015 rdi     =                0 =                    0 =                    0
 28641:20181017:133022.015 rsi     =              cb6 =                 3254 =                 3254
 28641:20181017:133022.015 rbp     =                0 =                    0 =                    0
 28641:20181017:133022.015 rbx     =                1 =                    1 =                    1
 28641:20181017:133022.015 rdx     =                0 =                    0 =                    0
 28641:20181017:133022.015 rax     =                0 =                    0 =                    0
 28641:20181017:133022.015 rcx     =                0 =                    0 =                    0
 28641:20181017:133022.015 rsp     =     7ffed85a7f78 =      140732528230264 =      140732528230264
 28641:20181017:133022.015 rip     =     7f63206fd676 =      140063722690166 =      140063722690166
 28641:20181017:133022.015 efl     =            10293 =                66195 =                66195
 28641:20181017:133022.015 csgsfs  =   2b000000000033 =    12103423998558259 =    12103423998558259
 28641:20181017:133022.015 err     =                4 =                    4 =                    4
 28641:20181017:133022.015 trapno  =                e =                   14 =                   14
 28641:20181017:133022.015 oldmask =                0 =                    0 =                    0
 28641:20181017:133022.015 cr2     =                0 =                    0 =                    0
Comment by Vladislavs Sokurenko [ 2018 Oct 17 ]

Thank you for your report, crash is fixed under ZBX-14908 however defunct/zombies situation should also be fixed and I don’t have a ticket for that yet.

Comment by Craig Hopkins [ 2018 Oct 17 ]

Okay, based on 14908, we've disabled "report unknown triggers" in the internal group, and zabbix is staying up now. We'll await a release in the stable repo to resolve this

 

Comment by Vladislavs Sokurenko [ 2018 Oct 23 ]

I have created ZBX-15048 for issue with zombie processes, closing this as a duplicate of ZBX-14908, thank you for your report.

Comment by Vladislavs Sokurenko [ 2018 Oct 25 ]

Could you please share if you are using OpenSSL 1.1.0 ?

Comment by Craig Hopkins [ 2018 Oct 26 ]

The system is always up to date to debian's releases, so currently

openssl                           1.1.0f-3+deb9u2

 

Generated at Thu Apr 10 18:02:15 EEST 2025 using Jira 9.12.4#9120004-sha1:625303b708afdb767e17cb2838290c41888e9ff0.