[ZBX-16970] Random crash server Created: 2019 Nov 21  Updated: 2019 Nov 22  Resolved: 2019 Nov 21

Status: Closed
Project: ZABBIX BUGS AND ISSUES
Component/s: None
Affects Version/s: None
Fix Version/s: None

Type: Incident report Priority: Critical
Reporter: Outscale Monitoring Team Assignee: Zabbix Support Team
Resolution: Commercial support required Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Attachments: PNG File image-2019-11-21-13-37-59-934.png     Text File logs-ZabbixServer.txt    

 Description   

Hi,

Since our migration to 4.2.4, our Zabbix Server crash randomly. Do you have any explanations?

You will find our logs file attached.

Thanks you by advance.

 

 Comments   
Comment by Outscale Monitoring Team [ 2019 Nov 21 ]

Hello again, 

Seems linked to the escalator process  as we have already met this issue 3 times since September.

For now it happened in the middle of the night with no actions at all , and also while shortly after adding new actions ( triggers and autoreg ).

Disabling the action and cleaning the escalations table permitted us to restart the server.

We never had any issue of this kind before upgrading from 3.4.8 to 4.2.4

DB is mysql, Server in 4.2.4 , Server with 7k hosts, 22 k nvps.

Not so sure if we should upgrade and if it would help in anything at this point but we keep our investigations.

 

Comment by Vladislavs Sokurenko [ 2019 Nov 21 ]

Could you please show output of show create table actions and show create table operations ?

Comment by Outscale Monitoring Team [ 2019 Nov 21 ]

Thanks for your answer Vladislavs.

FYI, i just restarted my server by truncating escalations again.

 

Here is the show create table actions : 

 

mysql> show create table actions;
--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+

Table Create Table

--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+

actions CREATE TABLE `actions` (
`actionid` bigint(20) unsigned NOT NULL,
`name` varchar(255) NOT NULL DEFAULT '',
`eventsource` int(11) NOT NULL DEFAULT '0',
`evaltype` int(11) NOT NULL DEFAULT '0',
`status` int(11) NOT NULL DEFAULT '0',
`esc_period` varchar(255) NOT NULL DEFAULT '1h',
`def_shortdata` varchar(255) NOT NULL DEFAULT '',
`def_longdata` text NOT NULL,
`r_shortdata` varchar(255) NOT NULL DEFAULT '',
`r_longdata` text NOT NULL,
`formula` varchar(255) NOT NULL DEFAULT '',
`pause_suppressed` int(11) DEFAULT NULL,
`ack_shortdata` varchar(255) NOT NULL DEFAULT '',
`ack_longdata` text NOT NULL,
PRIMARY KEY (`actionid`),
UNIQUE KEY `actions_2` (`name`),
KEY `actions_1` (`eventsource`,`status`)
) ENGINE=InnoDB DEFAULT CHARSET=latin1

--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
1 row in set (0.00 sec)

 

mysql> show create table operations;
------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+

Table Create Table

------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+

operations CREATE TABLE `operations` (
`operationid` bigint(20) unsigned NOT NULL,
`actionid` bigint(20) unsigned NOT NULL,
`operationtype` int(11) NOT NULL DEFAULT '0',
`esc_period` varchar(255) NOT NULL DEFAULT '0',
`esc_step_from` int(11) NOT NULL DEFAULT '1',
`esc_step_to` int(11) NOT NULL DEFAULT '1',
`evaltype` int(11) NOT NULL DEFAULT '0',
`recovery` int(11) NOT NULL DEFAULT '0',
PRIMARY KEY (`operationid`),
KEY `operations_1` (`actionid`),
CONSTRAINT `c_operations_1` FOREIGN KEY (`actionid`) REFERENCES `actions` (`actionid`) ON DELETE CASCADE
) ENGINE=InnoDB DEFAULT CHARSET=latin1

------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
1 row in set (0.00 sec)

 

 

Comment by Outscale Monitoring Team [ 2019 Nov 21 ]

Escalations table looking like that just before being truncated.

 

mysql> select * from escalations;
-------------------------------------------------------------------------------------------+

escalationid actionid triggerid eventid r_eventid nextcheck esc_step status itemid acknowledgeid

-------------------------------------------------------------------------------------------+

67602 228 2200291 44012762 NULL 1574339065 397 0 NULL NULL
70252 65 562487 44236030 NULL 1574339990 18 0 NULL NULL
70254 228 2855016 44236158 NULL 1574339065 21 0 NULL NULL
70258 281 NULL 44238302 NULL 0 0 0 NULL NULL
70260 138 708566 44240255 NULL 1574339065 5 0 NULL NULL
70261 137 708566 44240255 NULL 1574339065 5 0 NULL NULL
70262 65 337552 44240532 NULL 1574339990 7 0 NULL NULL

-------------------------------------------------------------------------------------------+
7 rows in set (0.01 sec)

Comment by Vladislavs Sokurenko [ 2019 Nov 21 ]

backtrace for easier searching:

582:20191121:050546.428 Got signal [signal:11(SIGSEGV),reason:1,refaddr:(nil)]. Crashing ...
582:20191121:050546.428 ====== Fatal information: ======
582:20191121:050546.428 Program counter: 0x7f6873f9bd67
582:20191121:050546.428 === Registers: ===
582:20191121:050546.428 r8 = 7f687431f060 = 140086602756192 = 140086602756192
582:20191121:050546.428 r9 = 1 = 1 = 1
582:20191121:050546.428 r10 = 2 = 2 = 2
582:20191121:050546.429 r11 = 0 = 0 = 0
582:20191121:050546.429 r12 = 7f6877be5565 = 140086662288741 = 140086662288741
582:20191121:050546.429 r13 = 0 = 0 = 0
582:20191121:050546.429 r14 = 0 = 0 = 0
582:20191121:050546.429 r15 = 0 = 0 = 0
582:20191121:050546.429 rdi = 0 = 0 = 0
582:20191121:050546.429 rsi = 0 = 0 = 0
582:20191121:050546.429 rbp = 0 = 0 = 0
582:20191121:050546.429 rbx = 9 = 9 = 9
582:20191121:050546.429 rdx = a = 10 = 10
582:20191121:050546.429 rax = 7f687431c260 = 140086602744416 = 140086602744416
582:20191121:050546.429 rcx = 0 = 0 = 0
582:20191121:050546.429 rsp = 7ffe6e0a5a70 = 140730744593008 = 140730744593008
582:20191121:050546.429 rip = 7f6873f9bd67 = 140086599073127 = 140086599073127
582:20191121:050546.429 efl = 10283 = 66179 = 66179
582:20191121:050546.429 csgsfs = 33 = 51 = 51
582:20191121:050546.429 err = 4 = 4 = 4
582:20191121:050546.429 trapno = e = 14 = 14
582:20191121:050546.429 oldmask = 0 = 0 = 0
582:20191121:050546.429 cr2 = 0 = 0 = 0
582:20191121:050546.429 === Backtrace: ===
597:20191121:050546.429 server #64 started poller #6
582:20191121:050546.429 15: /usr/sbin/zabbix_server: escalator #9 [processed 0 escalations in 0.000000 sec, processing escalations](zbx_backtrace+0x42) [0x7f6877b29f1a]
582:20191121:050546.429 14: /usr/sbin/zabbix_server: escalator #9 [processed 0 escalations in 0.000000 sec, processing escalations](zbx_log_fatal_info+0x151) [0x7f6877b2a198]
582:20191121:050546.429 13: /usr/sbin/zabbix_server: escalator #9 [processed 0 escalations in 0.000000 sec, processing escalations](+0x19b532) [0x7f6877b2a532]
582:20191121:050546.429 12: /lib64/libpthread.so.0(+0xf5e0) [0x7f687705e5e0]
582:20191121:050546.429 11: /lib64/libc.so.6(+0x39d67) [0x7f6873f9bd67]
582:20191121:050546.429 10: /lib64/libc.so.6(atoi+0x10) [0x7f6873f98770]
582:20191121:050546.429 9: /usr/sbin/zabbix_server: escalator #9 [processed 0 escalations in 0.000000 sec, processing escalations](get_db_actions_info+0x2f7) [0x7f6877ab4363]
582:20191121:050546.429 8: /usr/sbin/zabbix_server: escalator #9 [processed 0 escalations in 0.000000 sec, processing escalations](+0x80551) [0x7f6877a0f551]
582:20191121:050546.429 7: /usr/sbin/zabbix_server: escalator #9 [processed 0 escalations in 0.000000 sec, processing escalations](+0x8155c) [0x7f6877a1055c]
582:20191121:050546.429 6: /usr/sbin/zabbix_server: escalator #9 [processed 0 escalations in 0.000000 sec, processing escalations](escalator_thread+0x1f3) [0x7f6877a107c2]
582:20191121:050546.429 5: /usr/sbin/zabbix_server: escalator #9 [processed 0 escalations in 0.000000 sec, processing escalations](zbx_thread_start+0x37) [0x7f6877b37888]
582:20191121:050546.429 4: /usr/sbin/zabbix_server: escalator #9 [processed 0 escalations in 0.000000 sec, processing escalations](MAIN_ZABBIX_ENTRY+0xcae) [0x7f68779cf762]
582:20191121:050546.429 3: /usr/sbin/zabbix_server: escalator #9 [processed 0 escalations in 0.000000 sec, processing escalations](daemon_start+0x2f6) [0x7f6877b2975e]
582:20191121:050546.429 2: /usr/sbin/zabbix_server: escalator #9 [processed 0 escalations in 0.000000 sec, processing escalations](main+0x312) [0x7f68779ceab2]
582:20191121:050546.429 1: /lib64/libc.so.6(__libc_start_main+0xf5) [0x7f6873f83c05]
582:20191121:050546.429 0: /usr/sbin/zabbix_server: escalator #9 [processed 0 escalations in 0.000000 sec, processing escalations](+0x3eb79) [0x7f68779cdb79]
Comment by Vladislavs Sokurenko [ 2019 Nov 21 ]

This is incorrect

`pause_suppressed` int(11) DEFAULT NULL,

Must be

`pause_suppressed` int(11) NOT NULL DEFAULT '1',

No indication of a bug, closing.

Comment by Outscale Monitoring Team [ 2019 Nov 21 ]

Excellent ! We will fix our actions and give you a feedback. Thanks a lot Vladislavs.

Generated at Wed Jan 15 02:51:42 EET 2025 using Jira 9.12.4#9120004-sha1:625303b708afdb767e17cb2838290c41888e9ff0.