[ZBX-13347] Zabbix Server crashes: Got signal [signal:11(SIGSEGV),reason:1,refaddr:0x24ab0] Created: 2018 Jan 17 Updated: 2020 Dec 22 Resolved: 2019 Apr 03 |
|
Status: | Closed |
Project: | ZABBIX BUGS AND ISSUES |
Component/s: | Server (S) |
Affects Version/s: | None |
Fix Version/s: | 3.0.27rc1, 4.0.7rc1, 4.2.1rc1, 4.4.0alpha1, 4.4 (plan) |
Type: | Problem report | Priority: | Critical |
Reporter: | Pascal Uhlmann | Assignee: | Vladislavs Sokurenko |
Resolution: | Fixed | Votes: | 1 |
Labels: | crash, segfault, server | ||
Remaining Estimate: | Not Specified | ||
Time Spent: | Not Specified | ||
Original Estimate: | Not Specified | ||
Environment: |
Ubuntu 16.04 |
Attachments: | 79252.objdump.zip 79252.p2.objdump.zip 80550.objdump.zip ZBX-13347-3.0.diff ZBX-13347-3.4.15.diff crash-3.4.8-79252-p2-2.txt crash-3.4.8-79252-p2.txt crash-3.4.8.txt crash-3.4.9-80550.txt last_item_trigger_log34.patch last_trigger_log30.patch last_trigger_log34.patch valgrind.log zabbix_server.log zabbix_server.objdump.gz zbx_export_hosts.xml | ||||||||||||||||
Issue Links: |
|
||||||||||||||||
Team: | Team A | ||||||||||||||||
Sprint: | Sprint 27, Sprint 28, Sprint 29, Sprint 30, Sprint 31, Sprint 32, Sprint 33, Sprint 34, Sprint 35, Sprint 36, Sprint 37, Sprint 38, Sprint 39, Sprint 40, Sprint 41, Sprint 42, Sprint 43, Sprint 44, Sprint 50 (Mar 2019), Sprint 51 (Apr 2019) | ||||||||||||||||
Story Points: | 0.5 |
Description |
Zabbix server unexpectedly crashes. Afterwards it can be restarted without any problems. I've attached the relevant lines of the zabbix_server.log. We're using version 3.0.11. As I couldn't select this version I selected the oldest available. Maybe you can correct this. 26557:20180116:230552.028 Got signal [signal:11(SIGSEGV),reason:1,refaddr:0x24ab0]. Crashing ... .... 26557:20180116:230552.029 === Backtrace: === 26557:20180116:230552.096 16: /usr/sbin/zabbix_server: history syncer #1 [synced 0 items in 0.000076 sec, syncing history](print_fatal_info+0x15b) [0x555a73736177] 26557:20180116:230552.096 15: /usr/sbin/zabbix_server: history syncer #1 [synced 0 items in 0.000076 sec, syncing history](+0xd05b4) [0x555a737365b4] 26557:20180116:230552.096 14: /lib/x86_64-linux-gnu/libc.so.6(+0x354b0) [0x7fe819c124b0] 26557:20180116:230552.096 13: /lib/x86_64-linux-gnu/libc.so.6(strlen+0x26) [0x7fe819c68746] 26557:20180116:230552.096 12: /lib/x86_64-linux-gnu/libc.so.6(__strdup+0xe) [0x7fe819c6847e] 26557:20180116:230552.096 11: /usr/sbin/zabbix_server: history syncer #1 [synced 0 items in 0.000076 sec, syncing history](zbx_strdup2+0x4f) [0x555a7373f021] 26557:20180116:230552.096 10: /usr/sbin/zabbix_server: history syncer #1 [synced 0 items in 0.000076 sec, syncing history](+0xad114) [0x555a73713114] 26557:20180116:230552.096 9: /usr/sbin/zabbix_server: history syncer #1 [synced 0 items in 0.000076 sec, syncing history](DCconfig_get_triggers_by_itemids+0x10a) [0x555a73713f42] 26557:20180116:230552.096 8: /usr/sbin/zabbix_server: history syncer #1 [synced 0 items in 0.000076 sec, syncing history](DCflush_nextchecks+0x29a) [0x555a73704e25] 26557:20180116:230552.096 7: /usr/sbin/zabbix_server: history syncer #1 [synced 0 items in 0.000076 sec, syncing history](DCsync_history+0x3e8) [0x555a73701d3f] 26557:20180116:230552.096 6: /usr/sbin/zabbix_server: history syncer #1 [synced 0 items in 0.000076 sec, syncing history](dbsyncer_thread+0x16f) [0x555a7369abd3] 26557:20180116:230552.096 5: /usr/sbin/zabbix_server: history syncer #1 [synced 0 items in 0.000076 sec, syncing history](zbx_thread_start+0x37) [0x555a737373b4] 26557:20180116:230552.096 4: /usr/sbin/zabbix_server: history syncer #1 [synced 0 items in 0.000076 sec, syncing history](MAIN_ZABBIX_ENTRY+0x7d7) [0x555a736998c2] 26557:20180116:230552.096 3: /usr/sbin/zabbix_server: history syncer #1 [synced 0 items in 0.000076 sec, syncing history](daemon_start+0x329) [0x555a73735541] 26557:20180116:230552.096 2: /usr/sbin/zabbix_server: history syncer #1 [synced 0 items in 0.000076 sec, syncing history](main+0x2a8) [0x555a736990d5] 26557:20180116:230552.096 1: /lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0xf0) [0x7fe819bfd830] 26557:20180116:230552.096 0: /usr/sbin/zabbix_server: history syncer #1 [synced 0 items in 0.000076 sec, syncing history](_start+0x29) [0x555a73691709] |
Comments |
Comment by Aleksejs Petrovs [ 2018 Jan 17 ] |
Hello Pascal, Could you please attach the output of the objdump -DSswx zabbix_server command? Regards, |
Comment by Vladislavs Sokurenko [ 2018 Jan 17 ] |
Could you please be so kind and show output of: show create table triggers; |
Comment by Pascal Uhlmann [ 2018 Jan 17 ] |
objdump -DSswx /usr/sbin/zabbix_server | gzip > zabbix_server.objdump.gz |
Comment by Pascal Uhlmann [ 2018 Jan 17 ] |
CREATE TABLE `triggers` ( `triggerid` bigint(20) unsigned NOT NULL, `expression` varchar(2048) NOT NULL DEFAULT '', `description` varchar(255) NOT NULL DEFAULT '', `url` varchar(255) NOT NULL DEFAULT '', `status` int(11) NOT NULL DEFAULT '0', `value` int(11) NOT NULL DEFAULT '0', `priority` int(11) NOT NULL DEFAULT '0', `lastchange` int(11) NOT NULL DEFAULT '0', `comments` text NOT NULL, `error` varchar(128) NOT NULL DEFAULT '', `templateid` bigint(20) unsigned DEFAULT NULL, `type` int(11) NOT NULL DEFAULT '0', `state` int(11) NOT NULL DEFAULT '0', `flags` int(11) NOT NULL DEFAULT '0', PRIMARY KEY (`triggerid`), KEY `triggers_1` (`status`), KEY `c_triggers_1` (`templateid`), KEY `triggers_3` (`templateid`), KEY `triggers_2` (`value`,`lastchange`), CONSTRAINT `c_triggers_1` FOREIGN KEY (`templateid`) REFERENCES `triggers` (`triggerid`) ON DELETE CASCADE ) ENGINE=InnoDB DEFAULT CHARSET=utf8 |
Comment by Glebs Ivanovskis (Inactive) [ 2018 Jan 21 ] |
Piece of backtrace: 26557:20180116:230552.096 11: /usr/sbin/zabbix_server: history syncer #1 [synced 0 items in 0.000076 sec, syncing history](zbx_strdup2+0x4f) [0x555a7373f021] 26557:20180116:230552.096 10: /usr/sbin/zabbix_server: history syncer #1 [synced 0 items in 0.000076 sec, syncing history](+0xad114) [0x555a73713114] We know that zbx_strdup2() was called by instruction just before 0xad114. Piece of objdump: ad103: be f5 12 00 00 mov $0x12f5,%esi ad108: 48 8d 3d 59 1e 0b 00 lea 0xb1e59(%rip),%rdi # 15ef68 <syslog_app_name@@Base+0x17a18> ad10f: e8 be be 02 00 callq d8fd2 <zbx_strdup2@@Base> <--- path to crashing ad114: 48 89 c2 mov %rax,%rdx Just before 0xad114 and callq d8fd2 <zbx_strdup2@@Base> there is passing of __FILE__ and __LINE__ to zbx_strdup2() by zbx_strdup() macro. In this case it's line 12f516 = 485310 of dbconfig.c: 15ef60 6d3a2564 00000000 6462636f 6e666967 m:%d....dbconfig 15ef70 2e63007a 61626269 785b0068 6f737400 .c.zabbix[.host. Piece of code: static void DCget_trigger(DC_TRIGGER *dst_trigger, ZBX_DC_TRIGGER *src_trigger, unsigned char expand) { dst_trigger->triggerid = src_trigger->triggerid; dst_trigger->description = zbx_strdup(NULL, src_trigger->description); /* <--- line 4853 */ dst_trigger->expression_orig = zbx_strdup(NULL, src_trigger->expression); dst_trigger->error = zbx_strdup(NULL, src_trigger->error); dst_trigger->new_error = NULL; dst_trigger->timespec.sec = 0; dst_trigger->timespec.ns = 0; dst_trigger->priority = src_trigger->priority; dst_trigger->type = src_trigger->type; dst_trigger->value = src_trigger->value; dst_trigger->state = src_trigger->state; dst_trigger->new_value = TRIGGER_VALUE_UNKNOWN; dst_trigger->lastchange = src_trigger->lastchange; dst_trigger->topoindex = src_trigger->topoindex; dst_trigger->status = src_trigger->status; if (ZBX_EXPAND_MACROS == expand) { dst_trigger->expression = dc_cache_expanded_expression(src_trigger->expression, &src_trigger->expression_ex, &dst_trigger->new_error); } else dst_trigger->expression = NULL; } When DCget_trigger() is called from DCconfig_get_triggers_by_itemids() src_trigger cannot be NULL and 0x24ab0 is too big for an offset in ZBX_DC_TRIGGER structure. Maybe src_trigger->description is not initialized? |
Comment by Glebs Ivanovskis (Inactive) [ 2018 Jan 22 ] |
Dear pu, could you check that no other process is accessing Zabbix servers shared memory? Please check processes running as the same user as Zabbix server (including other Zabbix components) and also processes running as privileged user. I can't see any obvious place where Zabbix could leave description uninitialized so let's rule out external reasons of corruption before digging further. External process was involved in |
Comment by Pascal Uhlmann [ 2018 Jan 22 ] |
Apart from Zabbix server there aren't any processes running as user "zabbix". Even Zabbix agent is running as another user. |
Comment by Glebs Ivanovskis (Inactive) [ 2018 Jan 22 ] |
Can you show what is happening before the crash? Do you see messages like item "host:item" became not supported: or error reason for "host:item" changed:? If you have more examples of crashes in the log file I will be glad to see Fatal information: + Registers: + Backtrace: from them. |
Comment by Pascal Uhlmann [ 2018 Jan 23 ] |
There are really such messages immediately before the crash messages: 26563:20180116:230550.010 error reason for "<HOST_A>:perf_counter[\8062(_Total)\8114]" changed: Invalid performance counter format. 26563:20180116:230550.010 error reason for "<HOST_A>:perf_counter[\8176(_Total)\8326]" changed: Invalid performance counter format. 26563:20180116:230550.010 error reason for "<HOST_A>:perf_counter[\8176(_Total)\8328]" changed: Invalid performance counter format. 26563:20180116:230550.010 error reason for "<HOST_A>:perf_counter[\8792\8796]" changed: Invalid performance counter format. 26563:20180116:230550.010 item "<HOST_B>:vfs.block.diskstat[sdb,10]" became not supported: Received value [] is not suitable for value type [Numeric (float)] 26563:20180116:230550.010 item "<HOST_B>:vfs.block.diskstat[sdb,1]" became not supported: Received value [] is not suitable for value type [Numeric (float)] 26563:20180116:230550.010 item "<HOST_B>:vfs.block.diskstat[sdb,3]" became not supported: Received value [] is not suitable for value type [Numeric (float)] 26563:20180116:230550.010 item "<HOST_B>:vfs.block.diskstat[sdb,5]" became not supported: Received value [] is not suitable for value type [Numeric (float)] 26563:20180116:230550.010 item "<HOST_B>:vfs.block.diskstat[sdb,7]" became not supported: Received value [] is not suitable for value type [Numeric (float)] 26563:20180116:230550.055 item "<HOST_B>:vfs.block.diskstat[sdb,9]" became not supported: Received value [] is not suitable for value type [Numeric (unsigned)] and data type [Decimal] Unfortunately I haven't logs from the previous crashes anymore. |
Comment by Glebs Ivanovskis (Inactive) [ 2018 Feb 06 ] |
Looks like I have found a scenario that could have lead to such crash, but it is quite rare one with at least three processes involved. Dear pu, is your Zabbix crashing frequently? |
Comment by Pascal Uhlmann [ 2018 Feb 06 ] |
No, if I remember right it crashed two or three times. Since opening this bug report it didn't crash again. |
Comment by Glebs Ivanovskis (Inactive) [ 2018 Feb 07 ] |
I was trying to reproduce the crash using scenario I've found, but with no luck. Looks I was wrong. And I could not find any more leads. Since you are the only one who has experienced this crash and it does not happen too often, I'm going to stop progress of this ticket until Zabbix crashes again and we have a bigger piece of log file to understand what's happening. If you are enthusiastic enough to recompile your Zabbix from sources I can prepare a patch for you with some additional debug logging which may be useful in further investigation. |
Comment by Glebs Ivanovskis (Inactive) [ 2018 Feb 20 ] |
A similar crash in 3.4.6: 1720:20180219:164526.419 Got signal [signal:11(SIGSEGV),reason:1,refaddr:0x1]. Crashing ... 1720:20180219:164526.419 ====== Fatal information: ====== 1720:20180219:164526.419 Program counter: 0x7f01cb622dfa 1720:20180219:164526.419 === Registers: === 1720:20180219:164526.419 r8 = 0 = 0 = 0 1720:20180219:164526.420 r9 = 7f01cb95c7b8 = 139645687285688 = 139645687285688 1720:20180219:164526.420 r10 = fc06922730 = 1082442000176 = 1082442000176 1720:20180219:164526.420 r11 = 200 = 512 = 512 1720:20180219:164526.420 r12 = 7f01ceee89e6 = 139645743434214 = 139645743434214 1720:20180219:164526.420 r13 = 22 = 34 = 34 1720:20180219:164526.420 r14 = 0 = 0 = 0 1720:20180219:164526.420 r15 = 0 = 0 = 0 1720:20180219:164526.420 rdi = 1 = 1 = 1 1720:20180219:164526.420 rsi = 1744 = 5956 = 5956 1720:20180219:164526.420 rbp = 7ffdd5ba7e40 = 140728189222464 = 140728189222464 1720:20180219:164526.420 rbx = 1 = 1 = 1 1720:20180219:164526.420 rdx = 0 = 0 = 0 1720:20180219:164526.420 rax = 1 = 1 = 1 1720:20180219:164526.420 rcx = 1 = 1 = 1 1720:20180219:164526.420 rsp = 7ffdd5ba7de8 = 140728189222376 = 140728189222376 1720:20180219:164526.420 rip = 7f01cb622dfa = 139645683903994 = 139645683903994 1720:20180219:164526.420 efl = 10293 = 66195 = 66195 1720:20180219:164526.420 csgsfs = 33 = 51 = 51 1720:20180219:164526.420 err = 4 = 4 = 4 1720:20180219:164526.420 trapno = e = 14 = 14 1720:20180219:164526.420 oldmask = 0 = 0 = 0 1720:20180219:164526.420 cr2 = 1 = 1 = 1 1720:20180219:164526.420 === Backtrace: === 1720:20180219:164526.422 16: /usr/sbin/zabbix_server: history syncer #34 [synced 2023 items in 1.097806 sec, syncing hist(zbx_log_fatal_info+0x176) [0x7f01cee2f861] 1720:20180219:164526.422 15: /usr/sbin/zabbix_server: history syncer #34 [synced 2023 items in 1.097806 sec, syncing hist(+0xf7ce3) [0x7f01cee2fce3] 1720:20180219:164526.422 14: /lib/x86_64-linux-gnu/libpthread.so.0(+0x10330) [0x7f01ce3cc330] 1720:20180219:164526.422 13: /lib/x86_64-linux-gnu/libc.so.6(strlen+0x2a) [0x7f01cb622dfa] 1720:20180219:164526.422 12: /lib/x86_64-linux-gnu/libc.so.6(__strdup+0xe) [0x7f01cb622b3e] 1720:20180219:164526.422 11: /usr/sbin/zabbix_server: history syncer #34 [synced 2023 items in 1.097806 sec, syncing hist(zbx_strdup2+0x4f) [0x7f01cee4460a] 1720:20180219:164526.422 10: /usr/sbin/zabbix_server: history syncer #34 [synced 2023 items in 1.097806 sec, syncing hist(+0xd7e81) [0x7f01cee0fe81] <--- DCget_trigger() 1720:20180219:164526.422 9: /usr/sbin/zabbix_server: history syncer #34 [synced 2023 items in 1.097806 sec, syncing hist(DCconfig_get_triggers_by_itemids+0xeb) [0x7f01cee11550] 1720:20180219:164526.422 8: /usr/sbin/zabbix_server: history syncer #34 [synced 2023 items in 1.097806 sec, syncing hist(+0xc176a) [0x7f01cedf976a] <--- DCmass_update_triggers() 1720:20180219:164526.422 7: /usr/sbin/zabbix_server: history syncer #34 [synced 2023 items in 1.097806 sec, syncing hist(DCsync_history+0x6d0) [0x7f01cedfc7ee] 1720:20180219:164526.422 6: /usr/sbin/zabbix_server: history syncer #34 [synced 2023 items in 1.097806 sec, syncing hist(dbsyncer_thread+0x185) [0x7f01ced806fe] 1720:20180219:164526.422 5: /usr/sbin/zabbix_server: history syncer #34 [synced 2023 items in 1.097806 sec, syncing hist(zbx_thread_start+0x37) [0x7f01cee3ce40] 1720:20180219:164526.422 4: /usr/sbin/zabbix_server: history syncer #34 [synced 2023 items in 1.097806 sec, syncing hist(MAIN_ZABBIX_ENTRY+0xacb) [0x7f01ced79307] 1720:20180219:164526.422 3: /usr/sbin/zabbix_server: history syncer #34 [synced 2023 items in 1.097806 sec, syncing hist(daemon_start+0x325) [0x7f01cee2ef2a] 1720:20180219:164526.422 2: /usr/sbin/zabbix_server: history syncer #34 [synced 2023 items in 1.097806 sec, syncing hist(main+0x317) [0x7f01ced7883a] 1720:20180219:164526.422 1: /lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0xf5) [0x7f01cb5bbf45] 1720:20180219:164526.422 0: /usr/sbin/zabbix_server: history syncer #34 [synced 2023 items in 1.097806 sec, syncing hist(+0x34499) [0x7f01ced6c499] |
Comment by Michael Veksler [ 2018 Mar 14 ] |
Added memory dump of last used trigger in DCget_trigger() in case of server crash 816:20180314:170558.728 === Backtrace: === 816:20180314:170558.728 17: ./dist/sbin/zabbix_server(print_fatal_info+0xbd) [0x46837d] 816:20180314:170558.728 16: ./dist/sbin/zabbix_server() [0x4687b2] 816:20180314:170558.728 15: /lib/x86_64-linux-gnu/libpthread.so.0(+0x11390) [0x7f9a2b25c390] 816:20180314:170558.728 14: /lib/x86_64-linux-gnu/libc.so.6(strlen+0x26) [0x7f9a2a5dc746] 816:20180314:170558.728 13: /lib/x86_64-linux-gnu/libc.so.6(__strdup+0xe) [0x7f9a2a5dc47e] 816:20180314:170558.728 12: ./dist/sbin/zabbix_server(zbx_strdup2+0x40) [0x46c860] 816:20180314:170558.728 11: ./dist/sbin/zabbix_server() [0x44bfcc] 816:20180314:170558.728 10: ./dist/sbin/zabbix_server(DCconfig_get_triggers_by_itemids+0x1df) [0x45153f] 816:20180314:170558.728 9: ./dist/sbin/zabbix_server(DCsync_history+0x1d3c) [0x447a5c] 816:20180314:170558.728 8: ./dist/sbin/zabbix_server(free_database_cache+0x48) [0x449248] 816:20180314:170558.728 7: ./dist/sbin/zabbix_server(zbx_on_exit+0xf7) [0x419dc7] 816:20180314:170558.728 6: /lib/x86_64-linux-gnu/libpthread.so.0(+0x11390) [0x7f9a2b25c390] 816:20180314:170558.728 5: /lib/x86_64-linux-gnu/libpthread.so.0(__wait+0x20) [0x7f9a2b25be90] 816:20180314:170558.728 4: ./dist/sbin/zabbix_server(MAIN_ZABBIX_ENTRY+0x3a9) [0x41a1d9] 816:20180314:170558.728 3: ./dist/sbin/zabbix_server(daemon_start+0x1c3) [0x467c03] 816:20180314:170558.728 2: ./dist/sbin/zabbix_server(main+0x3fd) [0x4151fd] 816:20180314:170558.728 1: /lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0xf0) [0x7f9a2a571830] 816:20180314:170558.728 0: ./dist/sbin/zabbix_server(_start+0x29) [0x415469] 816:20180314:170558.728 === Trigger dump: === 816:20180314:170558.728 trigger dump(56B): b2 34 00 00 00 00 00 00 01 00 00 00 00 00 00 00 d4 49 b4 2b 9a 7f 00 00 00 00 00 00 00 00 00 00 ec 06 b4 2b 9a 7f 00 00 68 23 70 5a 01 03 00 00 00 01 00 00 00 00 00 00 816:20180314:170558.728 triggerid:13490 description:'0x1' type:0 status:0 priority:3 816:20180314:170558.728 expression:'0x7f9a2bb449d4' cached:'0x4cbc9f' 816:20180314:170558.728 value:0 state:0 error:'0x7f9a2bb406ec' lastchange:1517298536 816:20180314:170558.728 topoindex:1 functional:0 locked:1 816:20180314:170558.728 === Memory map: === Please, apply the patch from attachments and provide the additional information. |
Comment by Glebs Ivanovskis (Inactive) [ 2018 Mar 26 ] |
Dear pu, were you able to patch your Zabbix server? |
Comment by Glebs Ivanovskis (Inactive) [ 2018 Apr 04 ] |
Similar crash in 3.4.7, see Program counter, registers and backtrace: 21807:20180403:145228.205 Got signal [signal:11(SIGSEGV),reason:128,refaddr:(nil)]. Crashing ... 21807:20180403:145228.205 ====== Fatal information: ====== 21807:20180403:145228.205 Program counter: 0x7f4d2a945dfa 21807:20180403:145228.205 === Registers: === 21807:20180403:145228.205 r8 = 7ffc54df6160 = 140721732411744 = 140721732411744 21807:20180403:145228.205 r9 = 7ffc54df6150 = 140721732411728 = 140721732411728 21807:20180403:145228.205 r10 = 7ffc54df6140 = 140721732411712 = 140721732411712 21807:20180403:145228.205 r11 = 200 = 512 = 512 21807:20180403:145228.205 r12 = 529f30 = 5414704 = 5414704 21807:20180403:145228.205 r13 = 1746 = 5958 = 5958 21807:20180403:145228.205 r14 = 7f4cef905300 = 139968413455104 = 139968413455104 21807:20180403:145228.205 r15 = 8 = 8 = 8 21807:20180403:145228.205 rdi = 8000000000000018 = 9223372036854775832 = -9223372036854775784 21807:20180403:145228.205 rsi = 1746 = 5958 = 5958 21807:20180403:145228.205 rbp = 8000000000000018 = 9223372036854775832 = -9223372036854775784 21807:20180403:145228.205 rbx = 8000000000000018 = 9223372036854775832 = -9223372036854775784 21807:20180403:145228.205 rdx = 0 = 0 = 0 21807:20180403:145228.205 rax = 8000000000000018 = 9223372036854775832 = -9223372036854775784 21807:20180403:145228.205 rcx = 18 = 24 = 24 21807:20180403:145228.205 rsp = 7ffc5585e318 = 140721743323928 = 140721743323928 21807:20180403:145228.205 rip = 7f4d2a945dfa = 139969403575802 = 139969403575802 21807:20180403:145228.206 efl = 10293 = 66195 = 66195 21807:20180403:145228.206 csgsfs = 33 = 51 = 51 21807:20180403:145228.206 err = 0 = 0 = 0 21807:20180403:145228.206 trapno = d = 13 = 13 21807:20180403:145228.206 oldmask = 0 = 0 = 0 21807:20180403:145228.206 cr2 = 0 = 0 = 0 21807:20180403:145228.206 === Backtrace: === 21807:20180403:145228.246 15: /usr/sbin/zabbix_server: history syncer #15 [synced 0 items in 0.000010 sec, syncing history](zbx_log_fatal_info+0x1ae) [0x4a102e] 21807:20180403:145228.246 14: /usr/sbin/zabbix_server: history syncer #15 [synced 0 items in 0.000010 sec, syncing history]() [0x4a12b6] 21807:20180403:145228.246 13: /lib/x86_64-linux-gnu/libpthread.so.0(+0x10330) [0x7f4d2d54f330] 21807:20180403:145228.246 12: /lib/x86_64-linux-gnu/libc.so.6(strlen+0x2a) [0x7f4d2a945dfa] 21807:20180403:145228.246 11: /lib/x86_64-linux-gnu/libc.so.6(__strdup+0xe) [0x7f4d2a945b3e] 21807:20180403:145228.246 10: /usr/sbin/zabbix_server: history syncer #15 [synced 0 items in 0.000010 sec, syncing history](zbx_strdup2+0x3f) [0x4ae4cf] 21807:20180403:145228.246 9: /usr/sbin/zabbix_server: history syncer #15 [synced 0 items in 0.000010 sec, syncing history]() [0x48211d] 21807:20180403:145228.246 8: /usr/sbin/zabbix_server: history syncer #15 [synced 0 items in 0.000010 sec, syncing history](DCconfig_get_triggers_by_itemids+0x150) [0x48cc50] 21807:20180403:145228.246 7: /usr/sbin/zabbix_server: history syncer #15 [synced 0 items in 0.000010 sec, syncing history](DCsync_history+0x1f6f) [0x47dacf] 21807:20180403:145228.246 6: /usr/sbin/zabbix_server: history syncer #15 [synced 0 items in 0.000010 sec, syncing history](dbsyncer_thread+0xcf) [0x42a6ef] 21807:20180403:145228.246 5: /usr/sbin/zabbix_server: history syncer #15 [synced 0 items in 0.000010 sec, syncing history](zbx_thread_start+0x45) [0x4aa015] 21807:20180403:145228.246 4: /usr/sbin/zabbix_server: history syncer #15 [synced 0 items in 0.000010 sec, syncing history](MAIN_ZABBIX_ENTRY+0x76d) [0x425c1d] 21807:20180403:145228.246 3: /usr/sbin/zabbix_server: history syncer #15 [synced 0 items in 0.000010 sec, syncing history](daemon_start+0x1a9) [0x4a0999] 21807:20180403:145228.246 2: /usr/sbin/zabbix_server: history syncer #15 [synced 0 items in 0.000010 sec, syncing history](main+0x337) [0x41d4b7] 21807:20180403:145228.246 1: /lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0xf5) [0x7f4d2a8def45] 21807:20180403:145228.246 0: /usr/sbin/zabbix_server: history syncer #15 [synced 0 items in 0.000010 sec, syncing history]() [0x41d7ba] Piece of objdump: 48210b: be 46 17 00 00 mov $0x1746,%esi 482110: 48 89 07 mov %rax,(%rdi) 482113: bf 30 9f 52 00 mov $0x529f30,%edi <--- "dbconfig.c" 482118: e8 73 c3 02 00 callq 4ae490 <zbx_strdup2> <--- crash here 48211d: 48 8b 4b 10 mov 0x10(%rbx),%rcx 174616 = 595810 and here is the piece of code around line 5958 in dbconfig.c: static void DCget_trigger(DC_TRIGGER *dst_trigger, ZBX_DC_TRIGGER *src_trigger) { int i; dst_trigger->triggerid = src_trigger->triggerid; dst_trigger->description = zbx_strdup(NULL, src_trigger->description); /* <--- line 5958 */ dst_trigger->expression_orig = zbx_strdup(NULL, src_trigger->expression); |
Comment by Glebs Ivanovskis (Inactive) [ 2018 Apr 05 ] |
Dear Paskal, good new are that this crash happens on very rare occasions. Bad news are that we do not have a clue why it is happening. You can help us get a better understanding by applying last_trigger_log34.patch, it just adds some additional information to the log in case of crash and should not affect performance in any way. |
Comment by Dmitry Verkhoturov [ 2018 Apr 09 ] |
Patched 3.4.8 crashed, files in attachment ( crash-3.4.8.txt, 79252.objdump.zip) |
Comment by Michael Veksler [ 2018 Apr 10 ] |
Added memory dump of last used item and trigger in case of server crash. Please, apply the patch last_item_trigger_log34.patch from the attachments and provide the additional information. |
Comment by Dmitry Verkhoturov [ 2018 Apr 12 ] |
Newly patched 3.4.8 crashed, files in attachment ( crash-3.4.8-79252-p2.txt, 79252.p2.objdump.zip). |
Comment by Kevin Daudt [ 2018 Jun 11 ] |
Still seeing crashes with 3.4.9 (unpatched). |
Comment by Vladislavs Sokurenko [ 2018 Jun 11 ] |
Could you please provide backtrace kdaudt, thanks ! |
Comment by Kevin Daudt [ 2018 Jun 11 ] |
vso Sorry, they have apparently been overwritten (due to Zabbix own log rotation). If I see it happen again, I will attach it. |
Comment by Dmitry Verkhoturov [ 2018 Jun 14 ] |
Patched 3.4.9 crashed, files in attachment ( crash-3.4.9-80550.txt, 80550.objdump.zip). vso thanks, does item with itemid:9259138 have any triggers associated with it ? |
Comment by Vladislavs Sokurenko [ 2018 Jun 15 ] |
I think I have reproduced the issue, see valgrind.log log attached (it shows the same place as in crash but might be false positive) {Zabbix server:trap1.last()}=0 or {Zabbix server:trap2.last()}=0 3. Start Zabbix server. {Zabbix server:trap1.last()}=0 {Zabbix server:trap1.last()}=0 6. Manually reload cache and send value to trap1 item |
Comment by Vladislavs Sokurenko [ 2018 Jul 12 ] |
Do you have any community patches applied or is it clean binary ? You could also increase log level of configuration syncer to trace and see if something fishy is going when triggers are dumped. |
Comment by Dmitry Verkhoturov [ 2018 Jul 12 ] |
I thought you could reproduce it now. Would you be so kind to drop me an email to [email protected] so I could share additional data and debug logs to you? |
Comment by Vladislavs Sokurenko [ 2018 Jul 12 ] |
Unfortunately it looks like false positive by valgrind, it complains but does not crash, there must be another way to reproduce the issue. |
Comment by Vladislavs Sokurenko [ 2018 Oct 23 ] |
I am sorry but it was not possible to reproduce the issue. As I understood there is some special backup procedure that copy one Zabbix server database to other Zabbix server database while both are running, then there is something unexpected when cache reload happens. |
Comment by Vladislavs Sokurenko [ 2019 Jan 17 ] |
Do you use nested groups ? |
Comment by Pascal Uhlmann [ 2019 Jan 18 ] |
No, because the crash occured with Zabbix 3.0.11 which doesn't support nested groups. |
Comment by Vladislavs Sokurenko [ 2019 Mar 29 ] |
Issue can be reproduced with the following steps 1. create low level discovery rule with 2 item prototypes and one trigger prototype pointing to one item prototype update functions set itemid=28638 where functionid=18176; 5. reload cache 118318:20190329:172531.421 itemid:28638 hostid:10084 key:'trap1["b"]' 118318:20190329:172531.421 type:2 value_type:4 118318:20190329:172531.421 interfaceid:0 port:'' 118318:20190329:172531.421 state:0 error:'' 118318:20190329:172531.421 flags:4 status:0 118318:20190329:172531.421 valuemapid:0 118318:20190329:172531.422 lastlogsize:0 mtime:0 118318:20190329:172531.422 delay:'0' nextcheck:0 lastclock:0 118318:20190329:172531.422 data_expected_from:1553873092 118318:20190329:172531.422 history:1 118318:20190329:172531.422 poller_type:255 location:0 118318:20190329:172531.423 inventory_link:0 118318:20190329:172531.423 priority:1 schedulable:1 118318:20190329:172531.423 triggers: 118318:20190329:172531.423 triggerid:15927 118318:20190329:172531.423 itemid:28639 hostid:10084 key:'trap2["b"]' 118318:20190329:172531.424 type:2 value_type:4 118318:20190329:172531.424 interfaceid:0 port:'' 118318:20190329:172531.424 state:0 error:'' 118318:20190329:172531.424 flags:4 status:0 118318:20190329:172531.425 valuemapid:0 118318:20190329:172531.425 lastlogsize:0 mtime:0 118318:20190329:172531.425 delay:'0' nextcheck:0 lastclock:0 118318:20190329:172531.425 data_expected_from:1553873092 118318:20190329:172531.425 history:1 118318:20190329:172531.426 poller_type:255 location:0 118318:20190329:172531.426 inventory_link:0 118318:20190329:172531.426 priority:1 schedulable:1 118318:20190329:172531.426 triggers: 118318:20190329:172531.427 triggerid:15927 118318:20190329:172531.427 End of DCdump_items() 118318:20190329:172531.428 In DCdump_triggers() 118318:20190329:172531.428 triggerid:15927 description:'trap1_trig' type:0 status:0 priority:0 118318:20190329:172531.428 expression:'{18176}=1' recovery_expression:'' 118318:20190329:172531.428 value:0 state:0 error:'' lastchange:0 118318:20190329:172531.429 correlation_tag:'' recovery_mode:'0' correlation_mode:'0' 118318:20190329:172531.429 topoindex:1 functional:0 locked:0 118318:20190329:172531.429 End of DCdump_triggers() 118318:20190329:172531.429 In DCdump_trigdeps() 118318:20190329:172531.429 End of DCdump_trigdeps() 118318:20190329:172531.430 In DCdump_functions() 118318:20190329:172531.430 functionid:18176 triggerid:15927 itemid:28638 function:'last' parameter:'' timer:0 For example what it should be and is after restart 118609:20190329:172809.421 In DCdump_items() 118609:20190329:172809.425 itemid:28620 hostid:10084 key:'lld_rule' 118609:20190329:172809.427 type:2 value_type:4 118609:20190329:172809.428 interfaceid:0 port:'' 118609:20190329:172809.429 state:0 error:'' 118609:20190329:172809.429 flags:1 status:0 118609:20190329:172809.430 valuemapid:0 118609:20190329:172809.430 lastlogsize:0 mtime:0 118609:20190329:172809.431 delay:'0' nextcheck:0 lastclock:0 118609:20190329:172809.432 data_expected_from:1553873289 118609:20190329:172809.432 history:1 118609:20190329:172809.433 poller_type:255 location:0 118609:20190329:172809.433 inventory_link:0 118609:20190329:172809.434 priority:1 schedulable:1 118609:20190329:172809.436 itemid:28636 hostid:10084 key:'trap1["bz"]' 118609:20190329:172809.436 type:2 value_type:4 118609:20190329:172809.436 interfaceid:0 port:'' 118609:20190329:172809.437 state:0 error:'' 118609:20190329:172809.437 flags:4 status:0 118609:20190329:172809.437 valuemapid:0 118609:20190329:172809.437 lastlogsize:0 mtime:0 118609:20190329:172809.437 delay:'0' nextcheck:0 lastclock:0 118609:20190329:172809.437 data_expected_from:1553873289 118609:20190329:172809.438 history:1 118609:20190329:172809.438 poller_type:255 location:0 118609:20190329:172809.438 inventory_link:0 118609:20190329:172809.438 priority:1 schedulable:1 118609:20190329:172809.438 itemid:28637 hostid:10084 key:'trap2["bz"]' 118609:20190329:172809.439 type:2 value_type:4 118609:20190329:172809.439 interfaceid:0 port:'' 118609:20190329:172809.439 state:0 error:'' 118609:20190329:172809.439 flags:4 status:0 118609:20190329:172809.439 valuemapid:0 118609:20190329:172809.440 lastlogsize:0 mtime:0 118609:20190329:172809.440 delay:'0' nextcheck:0 lastclock:0 118609:20190329:172809.440 data_expected_from:1553873289 118609:20190329:172809.440 history:1 118609:20190329:172809.440 poller_type:255 location:0 118609:20190329:172809.441 inventory_link:0 118609:20190329:172809.441 priority:1 schedulable:1 118609:20190329:172809.441 itemid:28638 hostid:10084 key:'trap1["b"]' 118609:20190329:172809.441 type:2 value_type:4 118609:20190329:172809.442 interfaceid:0 port:'' 118609:20190329:172809.442 state:0 error:'' 118609:20190329:172809.442 flags:4 status:0 118609:20190329:172809.442 valuemapid:0 118609:20190329:172809.442 lastlogsize:0 mtime:0 118609:20190329:172809.442 delay:'0' nextcheck:0 lastclock:0 118609:20190329:172809.443 data_expected_from:1553873289 118609:20190329:172809.443 history:1 118609:20190329:172809.443 poller_type:255 location:0 118609:20190329:172809.443 inventory_link:0 118609:20190329:172809.443 priority:1 schedulable:1 118609:20190329:172809.444 triggers: 118609:20190329:172809.445 triggerid:15927 118609:20190329:172809.445 itemid:28639 hostid:10084 key:'trap2["b"]' 118609:20190329:172809.445 type:2 value_type:4 118609:20190329:172809.446 interfaceid:0 port:'' 118609:20190329:172809.446 state:0 error:'' 118609:20190329:172809.446 flags:4 status:0 118609:20190329:172809.446 valuemapid:0 118609:20190329:172809.446 lastlogsize:0 mtime:0 118609:20190329:172809.446 delay:'0' nextcheck:0 lastclock:0 118609:20190329:172809.447 data_expected_from:1553873289 118609:20190329:172809.447 history:1 118609:20190329:172809.447 poller_type:255 location:0 118609:20190329:172809.447 inventory_link:0 118609:20190329:172809.447 priority:1 schedulable:1 118609:20190329:172809.448 End of DCdump_items() 118609:20190329:172809.455 In DCdump_triggers() 118609:20190329:172809.457 triggerid:15927 description:'trap1_trig' type:0 status:0 priority:0 118609:20190329:172809.458 expression:'{18176}=1' recovery_expression:'' 118609:20190329:172809.459 value:0 state:0 error:'' lastchange:0 118609:20190329:172809.460 correlation_tag:'' recovery_mode:'0' correlation_mode:'0' 118609:20190329:172809.461 topoindex:1 functional:0 locked:0 118609:20190329:172809.462 End of DCdump_triggers() 118609:20190329:172809.462 In DCdump_trigdeps() 118609:20190329:172809.463 End of DCdump_trigdeps() 118609:20190329:172809.464 In DCdump_functions() 118609:20190329:172809.466 functionid:18176 triggerid:15927 itemid:28638 function:'last' parameter:'' timer:0 118609:20190329:172809.467 End of DCdump_functions() 6. notice that 2 different items have same trigger while functions do not reflect it 118865:20190329:173246.314 itemid:28620 hostid:10084 key:'lld_rule' 118865:20190329:173246.314 type:2 value_type:4 118865:20190329:173246.314 interfaceid:0 port:'' 118865:20190329:173246.314 state:0 error:'' 118865:20190329:173246.315 flags:1 status:0 118865:20190329:173246.315 valuemapid:0 118865:20190329:173246.315 lastlogsize:0 mtime:0 118865:20190329:173246.315 delay:'0' nextcheck:0 lastclock:0 118865:20190329:173246.315 data_expected_from:1553873515 118865:20190329:173246.316 history:1 118865:20190329:173246.316 poller_type:255 location:0 118865:20190329:173246.316 inventory_link:0 118865:20190329:173246.316 priority:1 schedulable:1 118865:20190329:173246.316 itemid:28636 hostid:10084 key:'trap1["bz"]' 118865:20190329:173246.317 type:2 value_type:4 118865:20190329:173246.317 interfaceid:0 port:'' 118865:20190329:173246.317 state:0 error:'' 118865:20190329:173246.317 flags:4 status:0 118865:20190329:173246.317 valuemapid:0 118865:20190329:173246.318 lastlogsize:0 mtime:0 118865:20190329:173246.318 delay:'0' nextcheck:0 lastclock:0 118865:20190329:173246.318 data_expected_from:1553873515 118865:20190329:173246.318 history:1 118865:20190329:173246.318 poller_type:255 location:0 118865:20190329:173246.319 inventory_link:0 118865:20190329:173246.319 priority:1 schedulable:1 118865:20190329:173246.319 itemid:28637 hostid:10084 key:'trap2["bz"]' 118865:20190329:173246.319 type:2 value_type:4 118865:20190329:173246.320 interfaceid:0 port:'' 118865:20190329:173246.320 state:0 error:'' 118865:20190329:173246.320 flags:4 status:0 118865:20190329:173246.320 valuemapid:0 118865:20190329:173246.320 lastlogsize:0 mtime:0 118865:20190329:173246.321 delay:'0' nextcheck:0 lastclock:0 118865:20190329:173246.321 data_expected_from:1553873515 118865:20190329:173246.321 history:1 118865:20190329:173246.321 poller_type:255 location:0 118865:20190329:173246.321 inventory_link:0 118865:20190329:173246.322 priority:1 schedulable:1 118865:20190329:173246.322 itemid:28638 hostid:10084 key:'trap1["b"]' 118865:20190329:173246.322 type:2 value_type:4 118865:20190329:173246.322 interfaceid:0 port:'' 118865:20190329:173246.322 state:0 error:'' 118865:20190329:173246.323 flags:4 status:0 118865:20190329:173246.323 valuemapid:0 118865:20190329:173246.323 lastlogsize:0 mtime:0 118865:20190329:173246.323 delay:'0' nextcheck:0 lastclock:0 118865:20190329:173246.323 data_expected_from:1553873515 118865:20190329:173246.324 history:1 118865:20190329:173246.324 poller_type:255 location:0 118865:20190329:173246.324 inventory_link:0 118865:20190329:173246.324 priority:1 schedulable:1 118865:20190329:173246.324 triggers: 118865:20190329:173246.325 triggerid:15927 118865:20190329:173246.325 itemid:28639 hostid:10084 key:'trap2["b"]' 118865:20190329:173246.325 type:2 value_type:4 118865:20190329:173246.325 interfaceid:0 port:'' 118865:20190329:173246.326 state:0 error:'' 118865:20190329:173246.326 flags:4 status:0 118865:20190329:173246.326 valuemapid:0 118865:20190329:173246.326 lastlogsize:0 mtime:0 118865:20190329:173246.326 delay:'0' nextcheck:0 lastclock:0 118865:20190329:173246.327 data_expected_from:1553873515 118865:20190329:173246.327 history:1 118865:20190329:173246.327 poller_type:255 location:0 118865:20190329:173246.327 inventory_link:0 118865:20190329:173246.328 priority:1 schedulable:1 118865:20190329:173246.328 End of DCdump_items() send value and see valgrind complain ==118874== Conditional jump or move depends on uninitialised value(s) ==118874== at 0x81AA96: zbx_default_uint64_compare_func (algodefs.c:283) ==118874== by 0x63B57F8: bsearch (in /usr/lib64/libc-2.28.so) ==118874== by 0x83A059: zbx_vector_uint64_bsearch (vector.c:26) ==118874== by 0x736F4A: zbx_determine_items_in_expressions (expression.c:4557) ==118874== by 0x74BFC9: recalculate_triggers (dbcache.c:1552) ==118874== by 0x755A59: sync_server_history (dbcache.c:2952) ==118874== by 0x7567A2: zbx_sync_history_cache (dbcache.c:3160) ==118874== by 0x4390B8: dbsyncer_thread (dbsyncer.c:86) ==118874== by 0x847771: zbx_thread_start (threads.c:132) ==118874== by 0x4249A2: MAIN_ZABBIX_ENTRY (server.c:1177) ==118874== by 0x816D26: daemon_start (daemon.c:392) ==118874== by 0x42392E: main (server.c:895) ==118874== Uninitialised value was created by a heap allocation ==118874== at 0x483880B: malloc (vg_replace_malloc.c:309) ==118874== by 0x85411B: zbx_malloc2 (misc.c:518) ==118874== by 0x736D5A: zbx_determine_items_in_expressions (expression.c:4546) ==118874== by 0x74BFC9: recalculate_triggers (dbcache.c:1552) ==118874== by 0x755A59: sync_server_history (dbcache.c:2952) ==118874== by 0x7567A2: zbx_sync_history_cache (dbcache.c:3160) ==118874== by 0x4390B8: dbsyncer_thread (dbsyncer.c:86) ==118874== by 0x847771: zbx_thread_start (threads.c:132) ==118874== by 0x4249A2: MAIN_ZABBIX_ENTRY (server.c:1177) ==118874== by 0x816D26: daemon_start (daemon.c:392) ==118874== by 0x42392E: main (server.c:895) ==118874== Or the other time if I memset freed memory to 0xff but leave trigger enabled state in freed memory ==122287== Invalid read of size 1 ==122287== at 0x483BC26: strlen (vg_replace_strmem.c:461) ==122287== by 0x6406661: strdup (in /usr/lib64/libc-2.28.so) ==122287== by 0x8543B3: zbx_strdup2 (misc.c:569) ==122287== by 0x79BC14: DCget_trigger (dbconfig.c:6707) ==122287== by 0x7A0FA6: DCconfig_get_triggers_by_itemids (dbconfig.c:7331) ==122287== by 0x74BFAA: recalculate_triggers (dbcache.c:1551) ==122287== by 0x755A59: sync_server_history (dbcache.c:2952) ==122287== by 0x7567A2: zbx_sync_history_cache (dbcache.c:3160) ==122287== by 0x4390B8: dbsyncer_thread (dbsyncer.c:86) ==122287== by 0x847843: zbx_thread_start (threads.c:132) ==122287== by 0x4249A2: MAIN_ZABBIX_ENTRY (server.c:1177) ==122287== by 0x816DF8: daemon_start (daemon.c:392) ==122287== Address 0xffffffffffffffff is not stack'd, malloc'd or (recently) free'd ==122287== 122287:20190329:175438.974 Got signal [signal:11(SIGSEGV),reason:1,refaddr:0xffffffffffffffff]. Crashing ... 122287:20190329:175438.974 ====== Fatal information: ====== 122287:20190329:175438.975 Program counter: 0x483bc26 122287:20190329:175438.975 === Registers: === 122287:20190329:175438.976 r8 = 0 = 0 = 0 122287:20190329:175438.978 r9 = 6c76a80 = 113732224 = 113732224 122287:20190329:175438.978 r10 = 8 = 8 = 8 122287:20190329:175438.979 r11 = ffffffffffffffff = 18446744073709551615 = -1 122287:20190329:175438.979 r12 = 888c280 = 143180416 = 143180416 122287:20190329:175438.979 r13 = 1ffefff2d0 = 137422172880 = 137422172880 122287:20190329:175438.980 r14 = 0 = 0 = 0 122287:20190329:175438.980 r15 = 0 = 0 = 0 122287:20190329:175438.980 rdi = ffffffffffffffff = 18446744073709551615 = -1 122287:20190329:175438.980 rsi = 1a33 = 6707 = 6707 122287:20190329:175438.981 rbp = ffffffffffffffff = 18446744073709551615 = -1 122287:20190329:175438.981 rbx = ffffffffffffffff = 18446744073709551615 = -1 122287:20190329:175438.981 rdx = 0 = 0 = 0 122287:20190329:175438.982 rax = 0 = 0 = 0 122287:20190329:175438.982 rcx = ffffffffffffffff = 18446744073709551615 = -1 122287:20190329:175438.982 rsp = 1ffeff21f8 = 137422119416 = 137422119416 122287:20190329:175438.982 rip = 483bc26 = 75742246 = 75742246 122287:20190329:175438.983 efl = 44 = 68 = 68 122287:20190329:175438.983 csgsfs = 0 = 0 = 0 122287:20190329:175438.983 err = 5 = 5 = 5 122287:20190329:175438.984 trapno = e = 14 = 14 122287:20190329:175438.984 oldmask = 0 = 0 = 0 122287:20190329:175438.985 cr2 = ffffffffffffffff = 18446744073709551615 = -1 122287:20190329:175438.985 === Backtrace: === 122287:20190329:175439.017 18: ./sbin/zabbix_server: history syncer #1 [processed 0 values, 0 triggers in 0.000644 sec, syncing history](zbx_backtrace+0x53) [0x8177a4] 122287:20190329:175439.017 17: ./sbin/zabbix_server: history syncer #1 [processed 0 values, 0 triggers in 0.000644 sec, syncing history](zbx_log_fatal_info+0x367) [0x817e79] 122287:20190329:175439.018 16: ./sbin/zabbix_server: history syncer #1 [processed 0 values, 0 triggers in 0.000644 sec, syncing history]() [0x818337] 122287:20190329:175439.018 15: /lib64/libpthread.so.0(+0x13030) [0x4ca0030] 122287:20190329:175439.018 14: /usr/lib64/valgrind/vgpreload_memcheck-amd64-linux.so(_vgr20070ZU_libcZdsoZa_strlen+0x6) [0x483bc26] 122287:20190329:175439.018 13: /lib64/libc.so.6(__strdup+0x12) [0x6406662] 122287:20190329:175439.018 12: ./sbin/zabbix_server: history syncer #1 [processed 0 values, 0 triggers in 0.000644 sec, syncing history](zbx_strdup2+0x72) [0x8543b4] 122287:20190329:175439.019 11: ./sbin/zabbix_server: history syncer #1 [processed 0 values, 0 triggers in 0.000644 sec, syncing history]() [0x79bc15] 122287:20190329:175439.019 10: ./sbin/zabbix_server: history syncer #1 [processed 0 values, 0 triggers in 0.000644 sec, syncing history](DCconfig_get_triggers_by_itemids+0x229) [0x7a0fa7] 122287:20190329:175439.019 9: ./sbin/zabbix_server: history syncer #1 [processed 0 values, 0 triggers in 0.000644 sec, syncing history]() [0x74bfab] 122287:20190329:175439.019 8: ./sbin/zabbix_server: history syncer #1 [processed 0 values, 0 triggers in 0.000644 sec, syncing history]() [0x755a5a] 122287:20190329:175439.020 7: ./sbin/zabbix_server: history syncer #1 [processed 0 values, 0 triggers in 0.000644 sec, syncing history](zbx_sync_history_cache+0x11d) [0x7567a3] 122287:20190329:175439.020 6: ./sbin/zabbix_server: history syncer #1 [processed 0 values, 0 triggers in 0.000644 sec, syncing history](dbsyncer_thread+0x258) [0x4390b9] 122287:20190329:175439.020 5: ./sbin/zabbix_server: history syncer #1 [processed 0 values, 0 triggers in 0.000644 sec, syncing history](zbx_thread_start+0x6c) [0x847844] 122287:20190329:175439.020 4: ./sbin/zabbix_server: history syncer #1 [processed 0 values, 0 triggers in 0.000644 sec, syncing history](MAIN_ZABBIX_ENTRY+0x1059) [0x4249a3] 122287:20190329:175439.020 3: ./sbin/zabbix_server: history syncer #1 [processed 0 values, 0 triggers in 0.000644 sec, syncing history](daemon_start+0x531) [0x816df9] 122287:20190329:175439.021 2: ./sbin/zabbix_server: history syncer #1 [processed 0 values, 0 triggers in 0.000644 sec, syncing history](main+0x3d2) [0x42392f] 122287:20190329:175439.021 1: /lib64/libc.so.6(__libc_start_main+0xf3) [0x63a0413] 122287:20190329:175439.021 0: ./sbin/zabbix_server: history syncer #1 [processed 0 values, 0 triggers in 0.000644 sec, syncing history](_start+0x2e) [0x421ade] |
Comment by Vladislavs Sokurenko [ 2019 Apr 01 ] |
Fixed in development branch: |
Comment by Vladislavs Sokurenko [ 2019 Apr 03 ] |
Fixed in:
|