[ZBX-13347] Zabbix Server crashes: Got signal [signal:11(SIGSEGV),reason:1,refaddr:0x24ab0] Created: 2018 Jan 17  Updated: 2020 Dec 22  Resolved: 2019 Apr 03

Status: Closed
Project: ZABBIX BUGS AND ISSUES
Component/s: Server (S)
Affects Version/s: None
Fix Version/s: 3.0.27rc1, 4.0.7rc1, 4.2.1rc1, 4.4.0alpha1, 4.4 (plan)

Type: Problem report Priority: Critical
Reporter: Pascal Uhlmann Assignee: Vladislavs Sokurenko
Resolution: Fixed Votes: 1
Labels: crash, segfault, server
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified
Environment:

Ubuntu 16.04


Attachments: Zip Archive 79252.objdump.zip     Zip Archive 79252.p2.objdump.zip     Zip Archive 80550.objdump.zip     File ZBX-13347-3.0.diff     File ZBX-13347-3.4.15.diff     Text File crash-3.4.8-79252-p2-2.txt     Text File crash-3.4.8-79252-p2.txt     Text File crash-3.4.8.txt     Text File crash-3.4.9-80550.txt     File last_item_trigger_log34.patch     File last_trigger_log30.patch     File last_trigger_log34.patch     File valgrind.log     Text File zabbix_server.log     File zabbix_server.objdump.gz     XML File zbx_export_hosts.xml    
Issue Links:
Causes
Duplicate
is duplicated by ZBX-14477 Sigsegv error with Zabbix 3.4.10 Closed
is duplicated by ZBX-13692 Zabbix crash on history syncer Closed
Team: Team A
Sprint: Sprint 27, Sprint 28, Sprint 29, Sprint 30, Sprint 31, Sprint 32, Sprint 33, Sprint 34, Sprint 35, Sprint 36, Sprint 37, Sprint 38, Sprint 39, Sprint 40, Sprint 41, Sprint 42, Sprint 43, Sprint 44, Sprint 50 (Mar 2019), Sprint 51 (Apr 2019)
Story Points: 0.5

 Description   

Zabbix server unexpectedly crashes. Afterwards it can be restarted without any problems. I've attached the relevant lines of the zabbix_server.log.

We're using version 3.0.11. As I couldn't select this version I selected the oldest available. Maybe you can correct this.
Part of log file:

 26557:20180116:230552.028 Got signal [signal:11(SIGSEGV),reason:1,refaddr:0x24ab0]. Crashing ...
 ....
 26557:20180116:230552.029 === Backtrace: ===
 26557:20180116:230552.096 16: /usr/sbin/zabbix_server: history syncer #1 [synced 0 items in 0.000076 sec, syncing history](print_fatal_info+0x15b) [0x555a73736177]
 26557:20180116:230552.096 15: /usr/sbin/zabbix_server: history syncer #1 [synced 0 items in 0.000076 sec, syncing history](+0xd05b4) [0x555a737365b4]
 26557:20180116:230552.096 14: /lib/x86_64-linux-gnu/libc.so.6(+0x354b0) [0x7fe819c124b0]
 26557:20180116:230552.096 13: /lib/x86_64-linux-gnu/libc.so.6(strlen+0x26) [0x7fe819c68746]
 26557:20180116:230552.096 12: /lib/x86_64-linux-gnu/libc.so.6(__strdup+0xe) [0x7fe819c6847e]
 26557:20180116:230552.096 11: /usr/sbin/zabbix_server: history syncer #1 [synced 0 items in 0.000076 sec, syncing history](zbx_strdup2+0x4f) [0x555a7373f021]
 26557:20180116:230552.096 10: /usr/sbin/zabbix_server: history syncer #1 [synced 0 items in 0.000076 sec, syncing history](+0xad114) [0x555a73713114]
 26557:20180116:230552.096 9: /usr/sbin/zabbix_server: history syncer #1 [synced 0 items in 0.000076 sec, syncing history](DCconfig_get_triggers_by_itemids+0x10a) [0x555a73713f42]
 26557:20180116:230552.096 8: /usr/sbin/zabbix_server: history syncer #1 [synced 0 items in 0.000076 sec, syncing history](DCflush_nextchecks+0x29a) [0x555a73704e25]
 26557:20180116:230552.096 7: /usr/sbin/zabbix_server: history syncer #1 [synced 0 items in 0.000076 sec, syncing history](DCsync_history+0x3e8) [0x555a73701d3f]
 26557:20180116:230552.096 6: /usr/sbin/zabbix_server: history syncer #1 [synced 0 items in 0.000076 sec, syncing history](dbsyncer_thread+0x16f) [0x555a7369abd3]
 26557:20180116:230552.096 5: /usr/sbin/zabbix_server: history syncer #1 [synced 0 items in 0.000076 sec, syncing history](zbx_thread_start+0x37) [0x555a737373b4]
 26557:20180116:230552.096 4: /usr/sbin/zabbix_server: history syncer #1 [synced 0 items in 0.000076 sec, syncing history](MAIN_ZABBIX_ENTRY+0x7d7) [0x555a736998c2]
 26557:20180116:230552.096 3: /usr/sbin/zabbix_server: history syncer #1 [synced 0 items in 0.000076 sec, syncing history](daemon_start+0x329) [0x555a73735541]
 26557:20180116:230552.096 2: /usr/sbin/zabbix_server: history syncer #1 [synced 0 items in 0.000076 sec, syncing history](main+0x2a8) [0x555a736990d5]
 26557:20180116:230552.096 1: /lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0xf0) [0x7fe819bfd830]
 26557:20180116:230552.096 0: /usr/sbin/zabbix_server: history syncer #1 [synced 0 items in 0.000076 sec, syncing history](_start+0x29) [0x555a73691709]


 Comments   
Comment by Aleksejs Petrovs [ 2018 Jan 17 ]

Hello Pascal,

Could you please attach the output of the objdump -DSswx zabbix_server command?

Regards,
Aleksejs!

Comment by Vladislavs Sokurenko [ 2018 Jan 17 ]

Could you please be so kind and show output of:

show create table triggers;
Comment by Pascal Uhlmann [ 2018 Jan 17 ]

objdump -DSswx /usr/sbin/zabbix_server | gzip > zabbix_server.objdump.gz

Comment by Pascal Uhlmann [ 2018 Jan 17 ]
CREATE TABLE `triggers` (
  `triggerid` bigint(20) unsigned NOT NULL,
  `expression` varchar(2048) NOT NULL DEFAULT '',
  `description` varchar(255) NOT NULL DEFAULT '',
  `url` varchar(255) NOT NULL DEFAULT '',
  `status` int(11) NOT NULL DEFAULT '0',
  `value` int(11) NOT NULL DEFAULT '0',
  `priority` int(11) NOT NULL DEFAULT '0',
  `lastchange` int(11) NOT NULL DEFAULT '0',
  `comments` text NOT NULL,
  `error` varchar(128) NOT NULL DEFAULT '',
  `templateid` bigint(20) unsigned DEFAULT NULL,
  `type` int(11) NOT NULL DEFAULT '0',
  `state` int(11) NOT NULL DEFAULT '0',
  `flags` int(11) NOT NULL DEFAULT '0',
  PRIMARY KEY (`triggerid`),
  KEY `triggers_1` (`status`),
  KEY `c_triggers_1` (`templateid`),
  KEY `triggers_3` (`templateid`),
  KEY `triggers_2` (`value`,`lastchange`),
  CONSTRAINT `c_triggers_1` FOREIGN KEY (`templateid`) REFERENCES `triggers` (`triggerid`) ON DELETE CASCADE
) ENGINE=InnoDB DEFAULT CHARSET=utf8
Comment by Glebs Ivanovskis (Inactive) [ 2018 Jan 21 ]

Piece of backtrace:

 26557:20180116:230552.096 11: /usr/sbin/zabbix_server: history syncer #1 [synced 0 items in 0.000076 sec, syncing history](zbx_strdup2+0x4f) [0x555a7373f021]
 26557:20180116:230552.096 10: /usr/sbin/zabbix_server: history syncer #1 [synced 0 items in 0.000076 sec, syncing history](+0xad114) [0x555a73713114]

We know that zbx_strdup2() was called by instruction just before 0xad114.

Piece of objdump:

   ad103:	be f5 12 00 00       	mov    $0x12f5,%esi
   ad108:	48 8d 3d 59 1e 0b 00 	lea    0xb1e59(%rip),%rdi        # 15ef68 <syslog_app_name@@Base+0x17a18>
   ad10f:	e8 be be 02 00       	callq  d8fd2 <zbx_strdup2@@Base>	<--- path to crashing
   ad114:	48 89 c2             	mov    %rax,%rdx

Just before 0xad114 and callq d8fd2 <zbx_strdup2@@Base> there is passing of __FILE__ and __LINE__ to zbx_strdup2() by zbx_strdup() macro. In this case it's line 12f5‍16 = 4853‍10 of dbconfig.c:

 15ef60 6d3a2564 00000000 6462636f 6e666967  m:%d....dbconfig
 15ef70 2e63007a 61626269 785b0068 6f737400  .c.zabbix[.host.

Piece of code:

static void	DCget_trigger(DC_TRIGGER *dst_trigger, ZBX_DC_TRIGGER *src_trigger, unsigned char expand)
{
	dst_trigger->triggerid = src_trigger->triggerid;
	dst_trigger->description = zbx_strdup(NULL, src_trigger->description);	/* <--- line 4853 */
	dst_trigger->expression_orig = zbx_strdup(NULL, src_trigger->expression);
	dst_trigger->error = zbx_strdup(NULL, src_trigger->error);
	dst_trigger->new_error = NULL;
	dst_trigger->timespec.sec = 0;
	dst_trigger->timespec.ns = 0;
	dst_trigger->priority = src_trigger->priority;
	dst_trigger->type = src_trigger->type;
	dst_trigger->value = src_trigger->value;
	dst_trigger->state = src_trigger->state;
	dst_trigger->new_value = TRIGGER_VALUE_UNKNOWN;
	dst_trigger->lastchange = src_trigger->lastchange;
	dst_trigger->topoindex = src_trigger->topoindex;
	dst_trigger->status = src_trigger->status;

	if (ZBX_EXPAND_MACROS == expand)
	{
		dst_trigger->expression = dc_cache_expanded_expression(src_trigger->expression,
				&src_trigger->expression_ex, &dst_trigger->new_error);
	}
	else
		dst_trigger->expression = NULL;
}

When DCget_trigger() is called from DCconfig_get_triggers_by_itemids() src_trigger cannot be NULL and 0x24ab0 is too big for an offset in ZBX_DC_TRIGGER structure. Maybe src_trigger->description is not initialized?

Comment by Glebs Ivanovskis (Inactive) [ 2018 Jan 22 ]

Dear pu, could you check that no other process is accessing Zabbix servers shared memory? Please check processes running as the same user as Zabbix server (including other Zabbix components) and also processes running as privileged user.

I can't see any obvious place where Zabbix could leave description uninitialized so let's rule out external reasons of corruption before digging further. External process was involved in ZBX-11747.

Comment by Pascal Uhlmann [ 2018 Jan 22 ]

Apart from Zabbix server there aren't any processes running as user "zabbix". Even Zabbix agent is running as another user.

Comment by Glebs Ivanovskis (Inactive) [ 2018 Jan 22 ]

Can you show what is happening before the crash? Do you see messages like item "host:item" became not supported: or error reason for "host:item" changed:?

If you have more examples of crashes in the log file I will be glad to see Fatal information: + Registers: + Backtrace: from them.

Comment by Pascal Uhlmann [ 2018 Jan 23 ]

There are really such messages immediately before the crash messages:

26563:20180116:230550.010 error reason for "<HOST_A>:perf_counter[\8062(_Total)\8114]" changed: Invalid performance counter format.
26563:20180116:230550.010 error reason for "<HOST_A>:perf_counter[\8176(_Total)\8326]" changed: Invalid performance counter format.
26563:20180116:230550.010 error reason for "<HOST_A>:perf_counter[\8176(_Total)\8328]" changed: Invalid performance counter format.
26563:20180116:230550.010 error reason for "<HOST_A>:perf_counter[\8792\8796]" changed: Invalid performance counter format.
26563:20180116:230550.010 item "<HOST_B>:vfs.block.diskstat[sdb,10]" became not supported: Received value [] is not suitable for value type [Numeric (float)]
26563:20180116:230550.010 item "<HOST_B>:vfs.block.diskstat[sdb,1]" became not supported: Received value [] is not suitable for value type [Numeric (float)]
26563:20180116:230550.010 item "<HOST_B>:vfs.block.diskstat[sdb,3]" became not supported: Received value [] is not suitable for value type [Numeric (float)]
26563:20180116:230550.010 item "<HOST_B>:vfs.block.diskstat[sdb,5]" became not supported: Received value [] is not suitable for value type [Numeric (float)]
26563:20180116:230550.010 item "<HOST_B>:vfs.block.diskstat[sdb,7]" became not supported: Received value [] is not suitable for value type [Numeric (float)]
26563:20180116:230550.055 item "<HOST_B>:vfs.block.diskstat[sdb,9]" became not supported: Received value [] is not suitable for value type [Numeric (unsigned)] and data type [Decimal]

Unfortunately I haven't logs from the previous crashes anymore.

Comment by Glebs Ivanovskis (Inactive) [ 2018 Feb 06 ]

Looks like I have found a scenario that could have lead to such crash, but it is quite rare one with at least three processes involved. Dear pu, is your Zabbix crashing frequently?

Comment by Pascal Uhlmann [ 2018 Feb 06 ]

No, if I remember right it crashed two or three times. Since opening this bug report it didn't crash again.

Comment by Glebs Ivanovskis (Inactive) [ 2018 Feb 07 ]

I was trying to reproduce the crash using scenario I've found, but with no luck. Looks I was wrong. And I could not find any more leads. Since you are the only one who has experienced this crash and it does not happen too often, I'm going to stop progress of this ticket until Zabbix crashes again and we have a bigger piece of log file to understand what's happening. If you are enthusiastic enough to recompile your Zabbix from sources I can prepare a patch for you with some additional debug logging which may be useful in further investigation.

Comment by Glebs Ivanovskis (Inactive) [ 2018 Feb 20 ]

A similar crash in 3.4.6:

  1720:20180219:164526.419 Got signal [signal:11(SIGSEGV),reason:1,refaddr:0x1]. Crashing ...
  1720:20180219:164526.419 ====== Fatal information: ======
  1720:20180219:164526.419 Program counter: 0x7f01cb622dfa
  1720:20180219:164526.419 === Registers: ===
  1720:20180219:164526.419 r8      =                0 =                    0 =                    0
  1720:20180219:164526.420 r9      =     7f01cb95c7b8 =      139645687285688 =      139645687285688
  1720:20180219:164526.420 r10     =       fc06922730 =        1082442000176 =        1082442000176
  1720:20180219:164526.420 r11     =              200 =                  512 =                  512
  1720:20180219:164526.420 r12     =     7f01ceee89e6 =      139645743434214 =      139645743434214
  1720:20180219:164526.420 r13     =               22 =                   34 =                   34
  1720:20180219:164526.420 r14     =                0 =                    0 =                    0
  1720:20180219:164526.420 r15     =                0 =                    0 =                    0
  1720:20180219:164526.420 rdi     =                1 =                    1 =                    1
  1720:20180219:164526.420 rsi     =             1744 =                 5956 =                 5956
  1720:20180219:164526.420 rbp     =     7ffdd5ba7e40 =      140728189222464 =      140728189222464
  1720:20180219:164526.420 rbx     =                1 =                    1 =                    1
  1720:20180219:164526.420 rdx     =                0 =                    0 =                    0
  1720:20180219:164526.420 rax     =                1 =                    1 =                    1
  1720:20180219:164526.420 rcx     =                1 =                    1 =                    1
  1720:20180219:164526.420 rsp     =     7ffdd5ba7de8 =      140728189222376 =      140728189222376
  1720:20180219:164526.420 rip     =     7f01cb622dfa =      139645683903994 =      139645683903994
  1720:20180219:164526.420 efl     =            10293 =                66195 =                66195
  1720:20180219:164526.420 csgsfs  =               33 =                   51 =                   51
  1720:20180219:164526.420 err     =                4 =                    4 =                    4
  1720:20180219:164526.420 trapno  =                e =                   14 =                   14
  1720:20180219:164526.420 oldmask =                0 =                    0 =                    0
  1720:20180219:164526.420 cr2     =                1 =                    1 =                    1
  1720:20180219:164526.420 === Backtrace: ===
  1720:20180219:164526.422 16: /usr/sbin/zabbix_server: history syncer #34 [synced 2023 items in 1.097806 sec, syncing hist(zbx_log_fatal_info+0x176) [0x7f01cee2f861]
  1720:20180219:164526.422 15: /usr/sbin/zabbix_server: history syncer #34 [synced 2023 items in 1.097806 sec, syncing hist(+0xf7ce3) [0x7f01cee2fce3]
  1720:20180219:164526.422 14: /lib/x86_64-linux-gnu/libpthread.so.0(+0x10330) [0x7f01ce3cc330]
  1720:20180219:164526.422 13: /lib/x86_64-linux-gnu/libc.so.6(strlen+0x2a) [0x7f01cb622dfa]
  1720:20180219:164526.422 12: /lib/x86_64-linux-gnu/libc.so.6(__strdup+0xe) [0x7f01cb622b3e]
  1720:20180219:164526.422 11: /usr/sbin/zabbix_server: history syncer #34 [synced 2023 items in 1.097806 sec, syncing hist(zbx_strdup2+0x4f) [0x7f01cee4460a]
  1720:20180219:164526.422 10: /usr/sbin/zabbix_server: history syncer #34 [synced 2023 items in 1.097806 sec, syncing hist(+0xd7e81) [0x7f01cee0fe81]	<--- DCget_trigger()
  1720:20180219:164526.422 9: /usr/sbin/zabbix_server: history syncer #34 [synced 2023 items in 1.097806 sec, syncing hist(DCconfig_get_triggers_by_itemids+0xeb) [0x7f01cee11550]
  1720:20180219:164526.422 8: /usr/sbin/zabbix_server: history syncer #34 [synced 2023 items in 1.097806 sec, syncing hist(+0xc176a) [0x7f01cedf976a]	<--- DCmass_update_triggers()
  1720:20180219:164526.422 7: /usr/sbin/zabbix_server: history syncer #34 [synced 2023 items in 1.097806 sec, syncing hist(DCsync_history+0x6d0) [0x7f01cedfc7ee]
  1720:20180219:164526.422 6: /usr/sbin/zabbix_server: history syncer #34 [synced 2023 items in 1.097806 sec, syncing hist(dbsyncer_thread+0x185) [0x7f01ced806fe]
  1720:20180219:164526.422 5: /usr/sbin/zabbix_server: history syncer #34 [synced 2023 items in 1.097806 sec, syncing hist(zbx_thread_start+0x37) [0x7f01cee3ce40]
  1720:20180219:164526.422 4: /usr/sbin/zabbix_server: history syncer #34 [synced 2023 items in 1.097806 sec, syncing hist(MAIN_ZABBIX_ENTRY+0xacb) [0x7f01ced79307]
  1720:20180219:164526.422 3: /usr/sbin/zabbix_server: history syncer #34 [synced 2023 items in 1.097806 sec, syncing hist(daemon_start+0x325) [0x7f01cee2ef2a]
  1720:20180219:164526.422 2: /usr/sbin/zabbix_server: history syncer #34 [synced 2023 items in 1.097806 sec, syncing hist(main+0x317) [0x7f01ced7883a]
  1720:20180219:164526.422 1: /lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0xf5) [0x7f01cb5bbf45]
  1720:20180219:164526.422 0: /usr/sbin/zabbix_server: history syncer #34 [synced 2023 items in 1.097806 sec, syncing hist(+0x34499) [0x7f01ced6c499]
Comment by Michael Veksler [ 2018 Mar 14 ]

Added memory dump of last used trigger in DCget_trigger() in case of server crash
svn://svn.zabbix.com/branches/dev/ZBX-13347-30 in r78644.
svn://svn.zabbix.com/branches/dev/ZBX-13347-34 in r78667.

  816:20180314:170558.728 === Backtrace: ===
   816:20180314:170558.728 17: ./dist/sbin/zabbix_server(print_fatal_info+0xbd) [0x46837d]
   816:20180314:170558.728 16: ./dist/sbin/zabbix_server() [0x4687b2]
   816:20180314:170558.728 15: /lib/x86_64-linux-gnu/libpthread.so.0(+0x11390) [0x7f9a2b25c390]
   816:20180314:170558.728 14: /lib/x86_64-linux-gnu/libc.so.6(strlen+0x26) [0x7f9a2a5dc746]
   816:20180314:170558.728 13: /lib/x86_64-linux-gnu/libc.so.6(__strdup+0xe) [0x7f9a2a5dc47e]
   816:20180314:170558.728 12: ./dist/sbin/zabbix_server(zbx_strdup2+0x40) [0x46c860]
   816:20180314:170558.728 11: ./dist/sbin/zabbix_server() [0x44bfcc]
   816:20180314:170558.728 10: ./dist/sbin/zabbix_server(DCconfig_get_triggers_by_itemids+0x1df) [0x45153f]
   816:20180314:170558.728 9: ./dist/sbin/zabbix_server(DCsync_history+0x1d3c) [0x447a5c]
   816:20180314:170558.728 8: ./dist/sbin/zabbix_server(free_database_cache+0x48) [0x449248]
   816:20180314:170558.728 7: ./dist/sbin/zabbix_server(zbx_on_exit+0xf7) [0x419dc7]
   816:20180314:170558.728 6: /lib/x86_64-linux-gnu/libpthread.so.0(+0x11390) [0x7f9a2b25c390]
   816:20180314:170558.728 5: /lib/x86_64-linux-gnu/libpthread.so.0(__wait+0x20) [0x7f9a2b25be90]
   816:20180314:170558.728 4: ./dist/sbin/zabbix_server(MAIN_ZABBIX_ENTRY+0x3a9) [0x41a1d9]
   816:20180314:170558.728 3: ./dist/sbin/zabbix_server(daemon_start+0x1c3) [0x467c03]
   816:20180314:170558.728 2: ./dist/sbin/zabbix_server(main+0x3fd) [0x4151fd]
   816:20180314:170558.728 1: /lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0xf0) [0x7f9a2a571830]
   816:20180314:170558.728 0: ./dist/sbin/zabbix_server(_start+0x29) [0x415469]
   816:20180314:170558.728 === Trigger dump: ===
   816:20180314:170558.728 trigger dump(56B): b2 34 00 00 00 00 00 00 01 00 00 00 00 00 00 00 d4 49 b4 2b 9a 7f 00 00 00 00 00 00 00 00 00 00 ec 06 b4 2b 9a 7f 00 00 68 23 70 5a 01 03 00 00 00 01 00 00 00 00 00 00 
   816:20180314:170558.728 triggerid:13490 description:'0x1' type:0 status:0 priority:3
   816:20180314:170558.728   expression:'0x7f9a2bb449d4' cached:'0x4cbc9f'
   816:20180314:170558.728   value:0 state:0 error:'0x7f9a2bb406ec' lastchange:1517298536
   816:20180314:170558.728   topoindex:1 functional:0 locked:1
   816:20180314:170558.728 === Memory map: ===

Please, apply the patch from attachments and provide the additional information.

Comment by Glebs Ivanovskis (Inactive) [ 2018 Mar 26 ]

Dear pu, were you able to patch your Zabbix server?

Comment by Glebs Ivanovskis (Inactive) [ 2018 Apr 04 ]

Similar crash in 3.4.7, see ZBX-13692 for a log file and full objdump. Most interesting parts here:

Program counter, registers and backtrace:

 21807:20180403:145228.205 Got signal [signal:11(SIGSEGV),reason:128,refaddr:(nil)]. Crashing ...
 21807:20180403:145228.205 ====== Fatal information: ======
 21807:20180403:145228.205 Program counter: 0x7f4d2a945dfa
 21807:20180403:145228.205 === Registers: ===
 21807:20180403:145228.205 r8      =     7ffc54df6160 =      140721732411744 =      140721732411744
 21807:20180403:145228.205 r9      =     7ffc54df6150 =      140721732411728 =      140721732411728
 21807:20180403:145228.205 r10     =     7ffc54df6140 =      140721732411712 =      140721732411712
 21807:20180403:145228.205 r11     =              200 =                  512 =                  512
 21807:20180403:145228.205 r12     =           529f30 =              5414704 =              5414704
 21807:20180403:145228.205 r13     =             1746 =                 5958 =                 5958
 21807:20180403:145228.205 r14     =     7f4cef905300 =      139968413455104 =      139968413455104
 21807:20180403:145228.205 r15     =                8 =                    8 =                    8
 21807:20180403:145228.205 rdi     = 8000000000000018 =  9223372036854775832 = -9223372036854775784
 21807:20180403:145228.205 rsi     =             1746 =                 5958 =                 5958
 21807:20180403:145228.205 rbp     = 8000000000000018 =  9223372036854775832 = -9223372036854775784
 21807:20180403:145228.205 rbx     = 8000000000000018 =  9223372036854775832 = -9223372036854775784
 21807:20180403:145228.205 rdx     =                0 =                    0 =                    0
 21807:20180403:145228.205 rax     = 8000000000000018 =  9223372036854775832 = -9223372036854775784
 21807:20180403:145228.205 rcx     =               18 =                   24 =                   24
 21807:20180403:145228.205 rsp     =     7ffc5585e318 =      140721743323928 =      140721743323928
 21807:20180403:145228.205 rip     =     7f4d2a945dfa =      139969403575802 =      139969403575802
 21807:20180403:145228.206 efl     =            10293 =                66195 =                66195
 21807:20180403:145228.206 csgsfs  =               33 =                   51 =                   51
 21807:20180403:145228.206 err     =                0 =                    0 =                    0
 21807:20180403:145228.206 trapno  =                d =                   13 =                   13
 21807:20180403:145228.206 oldmask =                0 =                    0 =                    0
 21807:20180403:145228.206 cr2     =                0 =                    0 =                    0
 21807:20180403:145228.206 === Backtrace: ===
 21807:20180403:145228.246 15: /usr/sbin/zabbix_server: history syncer #15 [synced 0 items in 0.000010 sec, syncing history](zbx_log_fatal_info+0x1ae) [0x4a102e]
 21807:20180403:145228.246 14: /usr/sbin/zabbix_server: history syncer #15 [synced 0 items in 0.000010 sec, syncing history]() [0x4a12b6]
 21807:20180403:145228.246 13: /lib/x86_64-linux-gnu/libpthread.so.0(+0x10330) [0x7f4d2d54f330]
 21807:20180403:145228.246 12: /lib/x86_64-linux-gnu/libc.so.6(strlen+0x2a) [0x7f4d2a945dfa]
 21807:20180403:145228.246 11: /lib/x86_64-linux-gnu/libc.so.6(__strdup+0xe) [0x7f4d2a945b3e]
 21807:20180403:145228.246 10: /usr/sbin/zabbix_server: history syncer #15 [synced 0 items in 0.000010 sec, syncing history](zbx_strdup2+0x3f) [0x4ae4cf]
 21807:20180403:145228.246 9: /usr/sbin/zabbix_server: history syncer #15 [synced 0 items in 0.000010 sec, syncing history]() [0x48211d]
 21807:20180403:145228.246 8: /usr/sbin/zabbix_server: history syncer #15 [synced 0 items in 0.000010 sec, syncing history](DCconfig_get_triggers_by_itemids+0x150) [0x48cc50]
 21807:20180403:145228.246 7: /usr/sbin/zabbix_server: history syncer #15 [synced 0 items in 0.000010 sec, syncing history](DCsync_history+0x1f6f) [0x47dacf]
 21807:20180403:145228.246 6: /usr/sbin/zabbix_server: history syncer #15 [synced 0 items in 0.000010 sec, syncing history](dbsyncer_thread+0xcf) [0x42a6ef]
 21807:20180403:145228.246 5: /usr/sbin/zabbix_server: history syncer #15 [synced 0 items in 0.000010 sec, syncing history](zbx_thread_start+0x45) [0x4aa015]
 21807:20180403:145228.246 4: /usr/sbin/zabbix_server: history syncer #15 [synced 0 items in 0.000010 sec, syncing history](MAIN_ZABBIX_ENTRY+0x76d) [0x425c1d]
 21807:20180403:145228.246 3: /usr/sbin/zabbix_server: history syncer #15 [synced 0 items in 0.000010 sec, syncing history](daemon_start+0x1a9) [0x4a0999]
 21807:20180403:145228.246 2: /usr/sbin/zabbix_server: history syncer #15 [synced 0 items in 0.000010 sec, syncing history](main+0x337) [0x41d4b7]
 21807:20180403:145228.246 1: /lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0xf5) [0x7f4d2a8def45]
 21807:20180403:145228.246 0: /usr/sbin/zabbix_server: history syncer #15 [synced 0 items in 0.000010 sec, syncing history]() [0x41d7ba]

Piece of objdump:

  48210b:	be 46 17 00 00       	mov    $0x1746,%esi
  482110:	48 89 07             	mov    %rax,(%rdi)
  482113:	bf 30 9f 52 00       	mov    $0x529f30,%edi	<--- "dbconfig.c"
  482118:	e8 73 c3 02 00       	callq  4ae490 <zbx_strdup2>	<--- crash here
  48211d:	48 8b 4b 10          	mov    0x10(%rbx),%rcx

1746‍16 = 5958‍10 and here is the piece of code around line 5958 in dbconfig.c:

static void	DCget_trigger(DC_TRIGGER *dst_trigger, ZBX_DC_TRIGGER *src_trigger)
{
	int	i;

	dst_trigger->triggerid = src_trigger->triggerid;
	dst_trigger->description = zbx_strdup(NULL, src_trigger->description);	/* <--- line 5958 */
	dst_trigger->expression_orig = zbx_strdup(NULL, src_trigger->expression);
Comment by Glebs Ivanovskis (Inactive) [ 2018 Apr 05 ]

Dear Paskal, good new are that this crash happens on very rare occasions. Bad news are that we do not have a clue why it is happening. You can help us get a better understanding by applying last_trigger_log34.patch, it just adds some additional information to the log in case of crash and should not affect performance in any way.

Comment by Dmitry Verkhoturov [ 2018 Apr 09 ]

Patched 3.4.8 crashed, files in attachment ( crash-3.4.8.txt, 79252.objdump.zip)

Comment by Michael Veksler [ 2018 Apr 10 ]

Added memory dump of last used item and trigger in case of server crash.
svn://svn.zabbix.com/branches/dev/ZBX-13347-34 in r79535.

Please, apply the patch last_item_trigger_log34.patch from the attachments and provide the additional information.

Comment by Dmitry Verkhoturov [ 2018 Apr 12 ]

Newly patched 3.4.8 crashed, files in attachment ( crash-3.4.8-79252-p2.txt, 79252.p2.objdump.zip).
UPD: new crash today, crash-3.4.8-79252-p2-2.txt.

Comment by Kevin Daudt [ 2018 Jun 11 ]

Still seeing crashes with 3.4.9 (unpatched).

Comment by Vladislavs Sokurenko [ 2018 Jun 11 ]

Could you please provide backtrace kdaudt, thanks !

Comment by Kevin Daudt [ 2018 Jun 11 ]

vso Sorry, they have apparently been overwritten (due to Zabbix own log rotation). If I see it happen again, I will attach it.

Comment by Dmitry Verkhoturov [ 2018 Jun 14 ]

Patched 3.4.9 crashed, files in attachment ( crash-3.4.9-80550.txt, 80550.objdump.zip).

vso thanks, does item with itemid:9259138 have any triggers associated with it ?

Comment by Vladislavs Sokurenko [ 2018 Jun 15 ]

I think I have reproduced the issue, see valgrind.log log attached (it shows the same place as in crash but might be false positive)
Steps:
1. Stop Zabbix server and create 2 items trap1 and trap2
2. Create trigger:

{Zabbix server:trap1.last()}=0 or {Zabbix server:trap2.last()}=0

3. Start Zabbix server.
4. Remove trap2 item
5. Create 2 same triggers, different name should be used

{Zabbix server:trap1.last()}=0
{Zabbix server:trap1.last()}=0

6. Manually reload cache and send value to trap1 item

Comment by Vladislavs Sokurenko [ 2018 Jul 12 ]

Do you have any community patches applied or is it clean binary ? You could also increase log level of configuration syncer to trace and see if something fishy is going when triggers are dumped.
zabbix_server -R log_level_increase="configuration syncer"

Comment by Dmitry Verkhoturov [ 2018 Jul 12 ]

I thought you could reproduce it now. Would you be so kind to drop me an email to [email protected] so I could share additional data and debug logs to you?

Comment by Vladislavs Sokurenko [ 2018 Jul 12 ]

Unfortunately it looks like false positive by valgrind, it complains but does not crash, there must be another way to reproduce the issue.

Comment by Vladislavs Sokurenko [ 2018 Oct 23 ]

I am sorry but it was not possible to reproduce the issue. As I understood there is some special backup procedure that copy one Zabbix server database to other Zabbix server database while both are running, then there is something unexpected when cache reload happens.

Comment by Vladislavs Sokurenko [ 2019 Jan 17 ]

Do you use nested groups ?

Comment by Pascal Uhlmann [ 2019 Jan 18 ]

No, because the crash occured with Zabbix 3.0.11 which doesn't support nested groups.

Comment by Vladislavs Sokurenko [ 2019 Mar 29 ]

Issue can be reproduced with the following steps

1. create low level discovery rule with 2 item prototypes and one trigger prototype pointing to one item prototype
2. do discovery and reload cache
3. update trigger prototype to second item prototype
4. see in Zabbix server log:

update functions set itemid=28638 where functionid=18176;

5. reload cache
see that 2 items have one trigger while there are only one function:

118318:20190329:172531.421 itemid:28638 hostid:10084 key:'trap1["b"]'
118318:20190329:172531.421   type:2 value_type:4
118318:20190329:172531.421   interfaceid:0 port:''
118318:20190329:172531.421   state:0 error:''
118318:20190329:172531.421   flags:4 status:0
118318:20190329:172531.421   valuemapid:0
118318:20190329:172531.422   lastlogsize:0 mtime:0
118318:20190329:172531.422   delay:'0' nextcheck:0 lastclock:0
118318:20190329:172531.422   data_expected_from:1553873092
118318:20190329:172531.422   history:1
118318:20190329:172531.422   poller_type:255 location:0
118318:20190329:172531.423   inventory_link:0
118318:20190329:172531.423   priority:1 schedulable:1
118318:20190329:172531.423   triggers:
118318:20190329:172531.423     triggerid:15927
118318:20190329:172531.423 itemid:28639 hostid:10084 key:'trap2["b"]'
118318:20190329:172531.424   type:2 value_type:4
118318:20190329:172531.424   interfaceid:0 port:''
118318:20190329:172531.424   state:0 error:''
118318:20190329:172531.424   flags:4 status:0
118318:20190329:172531.425   valuemapid:0
118318:20190329:172531.425   lastlogsize:0 mtime:0
118318:20190329:172531.425   delay:'0' nextcheck:0 lastclock:0
118318:20190329:172531.425   data_expected_from:1553873092
118318:20190329:172531.425   history:1
118318:20190329:172531.426   poller_type:255 location:0
118318:20190329:172531.426   inventory_link:0
118318:20190329:172531.426   priority:1 schedulable:1
118318:20190329:172531.426   triggers:
118318:20190329:172531.427     triggerid:15927
118318:20190329:172531.427 End of DCdump_items()
118318:20190329:172531.428 In DCdump_triggers()
118318:20190329:172531.428 triggerid:15927 description:'trap1_trig' type:0 status:0 priority:0
118318:20190329:172531.428   expression:'{18176}=1' recovery_expression:''
118318:20190329:172531.428   value:0 state:0 error:'' lastchange:0
118318:20190329:172531.429   correlation_tag:'' recovery_mode:'0' correlation_mode:'0'
118318:20190329:172531.429   topoindex:1 functional:0 locked:0
118318:20190329:172531.429 End of DCdump_triggers()
118318:20190329:172531.429 In DCdump_trigdeps()
118318:20190329:172531.429 End of DCdump_trigdeps()
118318:20190329:172531.430 In DCdump_functions()
118318:20190329:172531.430 functionid:18176 triggerid:15927 itemid:28638 function:'last' parameter:'' timer:0

For example what it should be and is after restart

118609:20190329:172809.421 In DCdump_items()
118609:20190329:172809.425 itemid:28620 hostid:10084 key:'lld_rule'
118609:20190329:172809.427   type:2 value_type:4
118609:20190329:172809.428   interfaceid:0 port:''
118609:20190329:172809.429   state:0 error:''
118609:20190329:172809.429   flags:1 status:0
118609:20190329:172809.430   valuemapid:0
118609:20190329:172809.430   lastlogsize:0 mtime:0
118609:20190329:172809.431   delay:'0' nextcheck:0 lastclock:0
118609:20190329:172809.432   data_expected_from:1553873289
118609:20190329:172809.432   history:1
118609:20190329:172809.433   poller_type:255 location:0
118609:20190329:172809.433   inventory_link:0
118609:20190329:172809.434   priority:1 schedulable:1
118609:20190329:172809.436 itemid:28636 hostid:10084 key:'trap1["bz"]'
118609:20190329:172809.436   type:2 value_type:4
118609:20190329:172809.436   interfaceid:0 port:''
118609:20190329:172809.437   state:0 error:''
118609:20190329:172809.437   flags:4 status:0
118609:20190329:172809.437   valuemapid:0
118609:20190329:172809.437   lastlogsize:0 mtime:0
118609:20190329:172809.437   delay:'0' nextcheck:0 lastclock:0
118609:20190329:172809.437   data_expected_from:1553873289
118609:20190329:172809.438   history:1
118609:20190329:172809.438   poller_type:255 location:0
118609:20190329:172809.438   inventory_link:0
118609:20190329:172809.438   priority:1 schedulable:1
118609:20190329:172809.438 itemid:28637 hostid:10084 key:'trap2["bz"]'
118609:20190329:172809.439   type:2 value_type:4
118609:20190329:172809.439   interfaceid:0 port:''
118609:20190329:172809.439   state:0 error:''
118609:20190329:172809.439   flags:4 status:0
118609:20190329:172809.439   valuemapid:0
118609:20190329:172809.440   lastlogsize:0 mtime:0
118609:20190329:172809.440   delay:'0' nextcheck:0 lastclock:0
118609:20190329:172809.440   data_expected_from:1553873289
118609:20190329:172809.440   history:1
118609:20190329:172809.440   poller_type:255 location:0
118609:20190329:172809.441   inventory_link:0
118609:20190329:172809.441   priority:1 schedulable:1
118609:20190329:172809.441 itemid:28638 hostid:10084 key:'trap1["b"]'
118609:20190329:172809.441   type:2 value_type:4
118609:20190329:172809.442   interfaceid:0 port:''
118609:20190329:172809.442   state:0 error:''
118609:20190329:172809.442   flags:4 status:0
118609:20190329:172809.442   valuemapid:0
118609:20190329:172809.442   lastlogsize:0 mtime:0
118609:20190329:172809.442   delay:'0' nextcheck:0 lastclock:0
118609:20190329:172809.443   data_expected_from:1553873289
118609:20190329:172809.443   history:1
118609:20190329:172809.443   poller_type:255 location:0
118609:20190329:172809.443   inventory_link:0
118609:20190329:172809.443   priority:1 schedulable:1
118609:20190329:172809.444   triggers:
118609:20190329:172809.445     triggerid:15927
118609:20190329:172809.445 itemid:28639 hostid:10084 key:'trap2["b"]'
118609:20190329:172809.445   type:2 value_type:4
118609:20190329:172809.446   interfaceid:0 port:''
118609:20190329:172809.446   state:0 error:''
118609:20190329:172809.446   flags:4 status:0
118609:20190329:172809.446   valuemapid:0
118609:20190329:172809.446   lastlogsize:0 mtime:0
118609:20190329:172809.446   delay:'0' nextcheck:0 lastclock:0
118609:20190329:172809.447   data_expected_from:1553873289
118609:20190329:172809.447   history:1
118609:20190329:172809.447   poller_type:255 location:0
118609:20190329:172809.447   inventory_link:0
118609:20190329:172809.447   priority:1 schedulable:1
118609:20190329:172809.448 End of DCdump_items()
118609:20190329:172809.455 In DCdump_triggers()
118609:20190329:172809.457 triggerid:15927 description:'trap1_trig' type:0 status:0 priority:0
118609:20190329:172809.458   expression:'{18176}=1' recovery_expression:''
118609:20190329:172809.459   value:0 state:0 error:'' lastchange:0
118609:20190329:172809.460   correlation_tag:'' recovery_mode:'0' correlation_mode:'0'
118609:20190329:172809.461   topoindex:1 functional:0 locked:0
118609:20190329:172809.462 End of DCdump_triggers()
118609:20190329:172809.462 In DCdump_trigdeps()
118609:20190329:172809.463 End of DCdump_trigdeps()
118609:20190329:172809.464 In DCdump_functions()
118609:20190329:172809.466 functionid:18176 triggerid:15927 itemid:28638 function:'last' parameter:'' timer:0
118609:20190329:172809.467 End of DCdump_functions()

6. notice that 2 different items have same trigger while functions do not reflect it
7. delete discovered trigger and reload cache.
See no functions and no triggers actually exist but there is trigger on item:

118865:20190329:173246.314 itemid:28620 hostid:10084 key:'lld_rule'
118865:20190329:173246.314   type:2 value_type:4
118865:20190329:173246.314   interfaceid:0 port:''
118865:20190329:173246.314   state:0 error:''
118865:20190329:173246.315   flags:1 status:0
118865:20190329:173246.315   valuemapid:0
118865:20190329:173246.315   lastlogsize:0 mtime:0
118865:20190329:173246.315   delay:'0' nextcheck:0 lastclock:0
118865:20190329:173246.315   data_expected_from:1553873515
118865:20190329:173246.316   history:1
118865:20190329:173246.316   poller_type:255 location:0
118865:20190329:173246.316   inventory_link:0
118865:20190329:173246.316   priority:1 schedulable:1
118865:20190329:173246.316 itemid:28636 hostid:10084 key:'trap1["bz"]'
118865:20190329:173246.317   type:2 value_type:4
118865:20190329:173246.317   interfaceid:0 port:''
118865:20190329:173246.317   state:0 error:''
118865:20190329:173246.317   flags:4 status:0
118865:20190329:173246.317   valuemapid:0
118865:20190329:173246.318   lastlogsize:0 mtime:0
118865:20190329:173246.318   delay:'0' nextcheck:0 lastclock:0
118865:20190329:173246.318   data_expected_from:1553873515
118865:20190329:173246.318   history:1
118865:20190329:173246.318   poller_type:255 location:0
118865:20190329:173246.319   inventory_link:0
118865:20190329:173246.319   priority:1 schedulable:1
118865:20190329:173246.319 itemid:28637 hostid:10084 key:'trap2["bz"]'
118865:20190329:173246.319   type:2 value_type:4
118865:20190329:173246.320   interfaceid:0 port:''
118865:20190329:173246.320   state:0 error:''
118865:20190329:173246.320   flags:4 status:0
118865:20190329:173246.320   valuemapid:0
118865:20190329:173246.320   lastlogsize:0 mtime:0
118865:20190329:173246.321   delay:'0' nextcheck:0 lastclock:0
118865:20190329:173246.321   data_expected_from:1553873515
118865:20190329:173246.321   history:1
118865:20190329:173246.321   poller_type:255 location:0
118865:20190329:173246.321   inventory_link:0
118865:20190329:173246.322   priority:1 schedulable:1
118865:20190329:173246.322 itemid:28638 hostid:10084 key:'trap1["b"]'
118865:20190329:173246.322   type:2 value_type:4
118865:20190329:173246.322   interfaceid:0 port:''
118865:20190329:173246.322   state:0 error:''
118865:20190329:173246.323   flags:4 status:0
118865:20190329:173246.323   valuemapid:0
118865:20190329:173246.323   lastlogsize:0 mtime:0
118865:20190329:173246.323   delay:'0' nextcheck:0 lastclock:0
118865:20190329:173246.323   data_expected_from:1553873515
118865:20190329:173246.324   history:1
118865:20190329:173246.324   poller_type:255 location:0
118865:20190329:173246.324   inventory_link:0
118865:20190329:173246.324   priority:1 schedulable:1
118865:20190329:173246.324   triggers:
118865:20190329:173246.325     triggerid:15927
118865:20190329:173246.325 itemid:28639 hostid:10084 key:'trap2["b"]'
118865:20190329:173246.325   type:2 value_type:4
118865:20190329:173246.325   interfaceid:0 port:''
118865:20190329:173246.326   state:0 error:''
118865:20190329:173246.326   flags:4 status:0
118865:20190329:173246.326   valuemapid:0
118865:20190329:173246.326   lastlogsize:0 mtime:0
118865:20190329:173246.326   delay:'0' nextcheck:0 lastclock:0
118865:20190329:173246.327   data_expected_from:1553873515
118865:20190329:173246.327   history:1
118865:20190329:173246.327   poller_type:255 location:0
118865:20190329:173246.327   inventory_link:0
118865:20190329:173246.328   priority:1 schedulable:1
118865:20190329:173246.328 End of DCdump_items()

send value and see valgrind complain

==118874== Conditional jump or move depends on uninitialised value(s)
==118874==    at 0x81AA96: zbx_default_uint64_compare_func (algodefs.c:283)
==118874==    by 0x63B57F8: bsearch (in /usr/lib64/libc-2.28.so)
==118874==    by 0x83A059: zbx_vector_uint64_bsearch (vector.c:26)
==118874==    by 0x736F4A: zbx_determine_items_in_expressions (expression.c:4557)
==118874==    by 0x74BFC9: recalculate_triggers (dbcache.c:1552)
==118874==    by 0x755A59: sync_server_history (dbcache.c:2952)
==118874==    by 0x7567A2: zbx_sync_history_cache (dbcache.c:3160)
==118874==    by 0x4390B8: dbsyncer_thread (dbsyncer.c:86)
==118874==    by 0x847771: zbx_thread_start (threads.c:132)
==118874==    by 0x4249A2: MAIN_ZABBIX_ENTRY (server.c:1177)
==118874==    by 0x816D26: daemon_start (daemon.c:392)
==118874==    by 0x42392E: main (server.c:895)
==118874==  Uninitialised value was created by a heap allocation
==118874==    at 0x483880B: malloc (vg_replace_malloc.c:309)
==118874==    by 0x85411B: zbx_malloc2 (misc.c:518)
==118874==    by 0x736D5A: zbx_determine_items_in_expressions (expression.c:4546)
==118874==    by 0x74BFC9: recalculate_triggers (dbcache.c:1552)
==118874==    by 0x755A59: sync_server_history (dbcache.c:2952)
==118874==    by 0x7567A2: zbx_sync_history_cache (dbcache.c:3160)
==118874==    by 0x4390B8: dbsyncer_thread (dbsyncer.c:86)
==118874==    by 0x847771: zbx_thread_start (threads.c:132)
==118874==    by 0x4249A2: MAIN_ZABBIX_ENTRY (server.c:1177)
==118874==    by 0x816D26: daemon_start (daemon.c:392)
==118874==    by 0x42392E: main (server.c:895)
==118874==

Or the other time if I memset freed memory to 0xff but leave trigger enabled state in freed memory

==122287== Invalid read of size 1
==122287==    at 0x483BC26: strlen (vg_replace_strmem.c:461)
==122287==    by 0x6406661: strdup (in /usr/lib64/libc-2.28.so)
==122287==    by 0x8543B3: zbx_strdup2 (misc.c:569)
==122287==    by 0x79BC14: DCget_trigger (dbconfig.c:6707)
==122287==    by 0x7A0FA6: DCconfig_get_triggers_by_itemids (dbconfig.c:7331)
==122287==    by 0x74BFAA: recalculate_triggers (dbcache.c:1551)
==122287==    by 0x755A59: sync_server_history (dbcache.c:2952)
==122287==    by 0x7567A2: zbx_sync_history_cache (dbcache.c:3160)
==122287==    by 0x4390B8: dbsyncer_thread (dbsyncer.c:86)
==122287==    by 0x847843: zbx_thread_start (threads.c:132)
==122287==    by 0x4249A2: MAIN_ZABBIX_ENTRY (server.c:1177)
==122287==    by 0x816DF8: daemon_start (daemon.c:392)
==122287==  Address 0xffffffffffffffff is not stack'd, malloc'd or (recently) free'd
==122287== 
122287:20190329:175438.974 Got signal [signal:11(SIGSEGV),reason:1,refaddr:0xffffffffffffffff]. Crashing ...
122287:20190329:175438.974 ====== Fatal information: ======
122287:20190329:175438.975 Program counter: 0x483bc26
122287:20190329:175438.975 === Registers: ===
122287:20190329:175438.976 r8      =                0 =                    0 =                    0
122287:20190329:175438.978 r9      =          6c76a80 =            113732224 =            113732224
122287:20190329:175438.978 r10     =                8 =                    8 =                    8
122287:20190329:175438.979 r11     = ffffffffffffffff = 18446744073709551615 =                   -1
122287:20190329:175438.979 r12     =          888c280 =            143180416 =            143180416
122287:20190329:175438.979 r13     =       1ffefff2d0 =         137422172880 =         137422172880
122287:20190329:175438.980 r14     =                0 =                    0 =                    0
122287:20190329:175438.980 r15     =                0 =                    0 =                    0
122287:20190329:175438.980 rdi     = ffffffffffffffff = 18446744073709551615 =                   -1
122287:20190329:175438.980 rsi     =             1a33 =                 6707 =                 6707
122287:20190329:175438.981 rbp     = ffffffffffffffff = 18446744073709551615 =                   -1
122287:20190329:175438.981 rbx     = ffffffffffffffff = 18446744073709551615 =                   -1
122287:20190329:175438.981 rdx     =                0 =                    0 =                    0
122287:20190329:175438.982 rax     =                0 =                    0 =                    0
122287:20190329:175438.982 rcx     = ffffffffffffffff = 18446744073709551615 =                   -1
122287:20190329:175438.982 rsp     =       1ffeff21f8 =         137422119416 =         137422119416
122287:20190329:175438.982 rip     =          483bc26 =             75742246 =             75742246
122287:20190329:175438.983 efl     =               44 =                   68 =                   68
122287:20190329:175438.983 csgsfs  =                0 =                    0 =                    0
122287:20190329:175438.983 err     =                5 =                    5 =                    5
122287:20190329:175438.984 trapno  =                e =                   14 =                   14
122287:20190329:175438.984 oldmask =                0 =                    0 =                    0
122287:20190329:175438.985 cr2     = ffffffffffffffff = 18446744073709551615 =                   -1
122287:20190329:175438.985 === Backtrace: ===
122287:20190329:175439.017 18: ./sbin/zabbix_server: history syncer #1 [processed 0 values, 0 triggers in 0.000644 sec, syncing history](zbx_backtrace+0x53) [0x8177a4]
122287:20190329:175439.017 17: ./sbin/zabbix_server: history syncer #1 [processed 0 values, 0 triggers in 0.000644 sec, syncing history](zbx_log_fatal_info+0x367) [0x817e79]
122287:20190329:175439.018 16: ./sbin/zabbix_server: history syncer #1 [processed 0 values, 0 triggers in 0.000644 sec, syncing history]() [0x818337]
122287:20190329:175439.018 15: /lib64/libpthread.so.0(+0x13030) [0x4ca0030]
122287:20190329:175439.018 14: /usr/lib64/valgrind/vgpreload_memcheck-amd64-linux.so(_vgr20070ZU_libcZdsoZa_strlen+0x6) [0x483bc26]
122287:20190329:175439.018 13: /lib64/libc.so.6(__strdup+0x12) [0x6406662]
122287:20190329:175439.018 12: ./sbin/zabbix_server: history syncer #1 [processed 0 values, 0 triggers in 0.000644 sec, syncing history](zbx_strdup2+0x72) [0x8543b4]
122287:20190329:175439.019 11: ./sbin/zabbix_server: history syncer #1 [processed 0 values, 0 triggers in 0.000644 sec, syncing history]() [0x79bc15]
122287:20190329:175439.019 10: ./sbin/zabbix_server: history syncer #1 [processed 0 values, 0 triggers in 0.000644 sec, syncing history](DCconfig_get_triggers_by_itemids+0x229) [0x7a0fa7]
122287:20190329:175439.019 9: ./sbin/zabbix_server: history syncer #1 [processed 0 values, 0 triggers in 0.000644 sec, syncing history]() [0x74bfab]
122287:20190329:175439.019 8: ./sbin/zabbix_server: history syncer #1 [processed 0 values, 0 triggers in 0.000644 sec, syncing history]() [0x755a5a]
122287:20190329:175439.020 7: ./sbin/zabbix_server: history syncer #1 [processed 0 values, 0 triggers in 0.000644 sec, syncing history](zbx_sync_history_cache+0x11d) [0x7567a3]
122287:20190329:175439.020 6: ./sbin/zabbix_server: history syncer #1 [processed 0 values, 0 triggers in 0.000644 sec, syncing history](dbsyncer_thread+0x258) [0x4390b9]
122287:20190329:175439.020 5: ./sbin/zabbix_server: history syncer #1 [processed 0 values, 0 triggers in 0.000644 sec, syncing history](zbx_thread_start+0x6c) [0x847844]
122287:20190329:175439.020 4: ./sbin/zabbix_server: history syncer #1 [processed 0 values, 0 triggers in 0.000644 sec, syncing history](MAIN_ZABBIX_ENTRY+0x1059) [0x4249a3]
122287:20190329:175439.020 3: ./sbin/zabbix_server: history syncer #1 [processed 0 values, 0 triggers in 0.000644 sec, syncing history](daemon_start+0x531) [0x816df9]
122287:20190329:175439.021 2: ./sbin/zabbix_server: history syncer #1 [processed 0 values, 0 triggers in 0.000644 sec, syncing history](main+0x3d2) [0x42392f]
122287:20190329:175439.021 1: /lib64/libc.so.6(__libc_start_main+0xf3) [0x63a0413]
122287:20190329:175439.021 0: ./sbin/zabbix_server: history syncer #1 [processed 0 values, 0 triggers in 0.000644 sec, syncing history](_start+0x2e) [0x421ade]
Comment by Vladislavs Sokurenko [ 2019 Apr 01 ]

Fixed in development branch:
svn://svn.zabbix.com/branches/dev/ZBX-13347

Comment by Vladislavs Sokurenko [ 2019 Apr 03 ]

Fixed in:

  • pre-3.0.27rc1 r91907
  • pre-4.0.7rc1 r91908
  • pre-4.2.1rc1 r91909
  • pre-4.4.0alpha1 r91910
Generated at Thu Mar 28 15:02:15 EET 2024 using Jira 9.12.4#9120004-sha1:625303b708afdb767e17cb2838290c41888e9ff0.