ZABBIX BUGS AND ISSUES

Calculated items become unsupported (was: zabbix_server crashed)

Details

  • Type: Bug Bug
  • Status: Closed Closed
  • Priority: Critical Critical
  • Resolution: Fixed
  • Affects Version/s: 1.8.7
  • Fix Version/s: 1.8.8, 1.9.7 (beta)
  • Component/s: Server (S)
  • Labels:
    None
  • Zabbix ID:
    NA

Description

zabbix_server crashed a few times during the day:

30696:20110915:170253.244 Got signal [signal:11(SIGSEGV),reason:1,refaddr:(nil)]. Crashing ...
 30696:20110915:170253.248 ====== Fatal information: ======
 30696:20110915:170253.248 Program counter: 0x7f3702b7e7c1
 30696:20110915:170253.248 === Registers: ===
 30696:20110915:170253.248 r8 = 3 = 3 = 3
 30696:20110915:170253.248 r9 = 1 = 1 = 1
 30696:20110915:170253.248 r10 = 7f3702b8182e = 139874245548078 = 139874245548078
 30696:20110915:170253.248 r11 = 7f3702b8182e = 139874245548078 = 139874245548078
 30696:20110915:170253.248 r12 = 81f = 2079 = 2079
 30696:20110915:170253.248 r13 = 0 = 0 = 0
 30696:20110915:170253.248 r14 = 26c2618 = 40642072 = 40642072
 30696:20110915:170253.248 r15 = 26a3230 = 40514096 = 40514096
 30696:20110915:170253.248 rdi = 0 = 0 = 0
 30696:20110915:170253.248 rsi = 81f = 2079 = 2079
 30696:20110915:170253.249 rbp = 4ac67c = 4900476 = 4900476
 30696:20110915:170253.249 rbx = 0 = 0 = 0
 30696:20110915:170253.249 rdx = 0 = 0 = 0
 30696:20110915:170253.249 rax = 0 = 0 = 0
 30696:20110915:170253.249 rcx = 0 = 0 = 0
 30696:20110915:170253.249 rsp = 7fff5209c288 = 140734569759368 = 140734569759368
 30696:20110915:170253.249 rip = 7f3702b7e7c1 = 139874245535681 = 139874245535681
 30696:20110915:170253.249 efl = 10283 = 66179 = 66179
 30696:20110915:170253.249 csgsfs = 33 = 51 = 51
 30696:20110915:170253.249 err = 4 = 4 = 4
 30696:20110915:170253.249 trapno = e = 14 = 14
 30696:20110915:170253.249 oldmask = 0 = 0 = 0
 30696:20110915:170253.249 cr2 = 0 = 0 = 0
 30696:20110915:170253.249 === Backtrace: ===
 30696:20110915:170253.249 14: /usr/local/sbin/zabbix_server(print_fatal_info+0xa2) [0x4492e8]
 30696:20110915:170253.249 13: /usr/local/sbin/zabbix_server() [0x4487f5]
 30696:20110915:170253.250 12: /lib/libc.so.6(+0x34810) [0x7f3702b36810]
 30696:20110915:170253.250 11: /lib/libc.so.6(strlen+0x11) [0x7f3702b7e7c1]
 30696:20110915:170253.250 10: /lib/libc.so.6(__strdup+0x16) [0x7f3702b7e476]
 30696:20110915:170253.250 9: /usr/local/sbin/zabbix_server(zbx_strdup2+0x30) [0x44bb1d]
 30696:20110915:170253.250 8: /usr/local/sbin/zabbix_server(DBget_history+0x28f) [0x46242b]
 30696:20110915:170253.250 7: /usr/local/sbin/zabbix_server(evaluate_function+0x422) [0x4548bb]
 30696:20110915:170253.250 6: /usr/local/sbin/zabbix_server(get_value_calculated+0x756) [0x42d7a6]
 30696:20110915:170253.250 5: /usr/local/sbin/zabbix_server() [0x429a25]
 30696:20110915:170253.250 4: /usr/local/sbin/zabbix_server(main_poller_loop+0x85) [0x42a745]
 30696:20110915:170253.250 3: /usr/local/sbin/zabbix_server(MAIN_ZABBIX_ENTRY+0x501) [0x423323]
 30696:20110915:170253.250 2: /usr/local/sbin/zabbix_server(daemon_start+0x2d0) [0x448d81]
 30696:20110915:170253.250 1: /lib/libc.so.6(__libc_start_main+0xed) [0x7f3702b2317d]
 30696:20110915:170253.250 0: /usr/local/sbin/zabbix_server() [0x420481]
 30696:20110915:170253.250 === Memory map: ===
 30696:20110915:170253.250 00400000-006c1000 r-xp 00000000 93:00 527920 /usr/local/sbin/zabbix_server
 30696:20110915:170253.250 008c1000-00965000 rw-p 002c1000 93:00 527920 /usr/local/sbin/zabbix_server
 30696:20110915:170253.250 00965000-00970000 rw-p 00000000 00:00 0
 30696:20110915:170253.250 025f0000-026ff000 rw-p 00000000 00:00 0 [heap]
 30696:20110915:170253.250 7f36fbacc000-7f36fcdfb000 r--p 00000000 93:00 533194 /usr/lib/locale/locale-archive
 30696:20110915:170253.250 7f36fcdfb000-7f36fce03000 rw-s 00000000 00:04 7929865 /SYSV530000ab (deleted)
 30696:20110915:170253.250 7f36fce03000-7f36fd2d0000 rw-s 00000000 00:04 7897094 /SYSV730000ab (deleted)
 30696:20110915:170253.250 7f36fd2d0000-7f36fee04000 rw-s 00000000 00:04 7864325 /SYSV670000ab (deleted)
 30696:20110915:170253.250 7f36fee04000-7f36ff605000 rw-s 00000000 00:04 7831556 /SYSV740000ab (deleted)
 30696:20110915:170253.250 7f36ff605000-7f3700606000 rw-s 00000000 00:04 7798787 /SYSV780000ab (deleted)
 30696:20110915:170253.250 7f3700606000-7f3700e0f000 rw-s 00000000 00:04 7766018 /SYSV680000ab (deleted)
 30696:20110915:170253.250 7f3700e0f000-7f3700e24000 r-xp 00000000 93:00 532432 /usr/lib/libgcc_s.so.1
 30696:20110915:170253.250 7f3700e24000-7f3701024000 ---p 00015000 93:00 532432 /usr/lib/libgcc_s.so.1
 30696:20110915:170253.250 7f3701024000-7f3701025000 rw-p 00015000 93:00 532432 /usr/lib/libgcc_s.so.1
 30696:20110915:170253.250 7f3701025000-7f3701026000 ---p 00000000 00:00 0
 30696:20110915:170253.250 7f3701026000-7f3701826000 rw-p 00000000 00:00 0
 30696:20110915:170253.250 7f3701826000-7f3701832000 r-xp 00000000 93:00 530563 /lib/libnss_files-2.14.so
 30696:20110915:170253.250 7f3701832000-7f3701a31000 ---p 0000c000 93:00 530563 /lib/libnss_files-2.14.so
 30696:20110915:170253.250 7f3701a31000-7f3701a32000 r--p 0000b000 93:00 530563 /lib/libnss_files-2.14.so
 30696:20110915:170253.250 7f3701a32000-7f3701a33000 rw-p 0000c000 93:00 530563 /lib/libnss_files-2.14.so
 30696:20110915:170253.251 7f3701a33000-7f3701a38000 r-xp 00000000 93:00 527558 /usr/lib/libgdbm.so.3.0.0
 30696:20110915:170253.251 7f3701a38000-7f3701c37000 ---p 00005000 93:00 527558 /usr/lib/libgdbm.so.3.0.0
 30696:20110915:170253.251 7f3701c37000-7f3701c38000 rw-p 00004000 93:00 527558 /usr/lib/libgdbm.so.3.0.0
 30696:20110915:170253.251 7f3701c38000-7f3701c3f000 r-xp 00000000 93:00 530031 /usr/local/lib/libOpenIPMIutils.so.0.0.1
 30696:20110915:170253.251 7f3701c3f000-7f3701e3f000 ---p 00007000 93:00 530031 /usr/local/lib/libOpenIPMIutils.so.0.0.1
 30696:20110915:170253.251 7f3701e3f000-7f3701e40000 rw-p 00007000 93:00 530031 /usr/local/lib/libOpenIPMIutils.so.0.0.1
 30696:20110915:170253.251 7f3701e40000-7f3701e95000 r-xp 00000000 93:00 527661 /usr/lib/libssl.so.1.0.0
 30696:20110915:170253.251 7f3701e95000-7f3702094000 ---p 00055000 93:00 527661 /usr/lib/libssl.so.1.0.0
 30696:20110915:170253.251 7f3702094000-7f370209c000 rw-p 00054000 93:00 527661 /usr/lib/libssl.so.1.0.0
 30696:20110915:170253.251 7f370209c000-7f37020a3000 r-xp 00000000 93:00 530571 /lib/librt-2.14.so
 30696:20110915:170253.251 7f37020a3000-7f37022a2000 ---p 00007000 93:00 530571 /lib/librt-2.14.so
 30696:20110915:170253.251 7f37022a2000-7f37022a3000 r--p 00006000 93:00 530571 /lib/librt-2.14.so
 30696:20110915:170253.251 7f37022a3000-7f37022a4000 rw-p 00007000 93:00 530571 /lib/librt-2.14.so
 30696:20110915:170253.251 7f37022a4000-7f37022c8000 r-xp 00000000 93:00 532585 /usr/lib/libssh2.so.1.0.1
 30696:20110915:170253.251 7f37022c8000-7f37024c8000 ---p 00024000 93:00 532585 /usr/lib/libssh2.so.1.0.1
 30696:20110915:170253.251 7f37024c8000-7f37024c9000 rw-p 00024000 93:00 532585 /usr/lib/libssh2.so.1.0.1
 30696:20110915:170253.251 7f37024c9000-7f37024e0000 r-xp 00000000 93:00 530568 /lib/libpthread-2.14.so
 30696:20110915:170253.251 7f37024e0000-7f37026e0000 ---p 00017000 93:00 530568 /lib/libpthread-2.14.so
 30696:20110915:170253.251 7f37026e0000-7f37026e1000 r--p 00017000 93:00 530568 /lib/libpthread-2.14.so
 30696:20110915:170253.251 7f37026e1000-7f37026e2000 rw-p 00018000 93:00 530568 /lib/libpthread-2.14.so
 30696:20110915:170253.251 7f37026e2000-7f37026e6000 rw-p 00000000 00:00 0
 30696:20110915:170253.251 7f37026e6000-7f37026fe000 r-xp 00000000 93:00 527735 /usr/lib/libz.so.1.2.5
 30696:20110915:170253.251 7f37026fe000-7f37028fd000 ---p 00018000 93:00 527735 /usr/lib/libz.so.1.2.5
 30696:20110915:170253.251 7f37028fd000-7f37028fe000 rw-p 00017000 93:00 527735 /usr/lib/libz.so.1.2.5
 30696:20110915:170253.251 7f37028fe000-7f3702900000 r-xp 00000000 93:00 530548 /lib/libdl-2.14.so
 30696:20110915:170253.251 7f3702900000-7f3702b00000 ---p 00002000 93:00 530548 /lib/libdl-2.14.so
 30696:20110915:170253.251 7f3702b00000-7f3702b01000 r--p 00002000 93:00 530548 /lib/libdl-2.14.so
 30696:20110915:170253.251 7f3702b01000-7f3702b02000 rw-p 00003000 93:00 530548 /lib/libdl-2.14.so
 30696:20110915:170253.251 7f3702b02000-7f3702c59000 r-xp 00000000 93:00 525516 /lib/libc-2.14.so
 30696:20110915:170253.251 7f3702c59000-7f3702e59000 ---p 00157000 93:00 525516 /lib/libc-2.14.so
 30696:20110915:170253.251 7f3702e59000-7f3702e5d000 r--p 00157000 93:00 525516 /lib/libc-2.14.so
 30696:20110915:170253.251 7f3702e5d000-7f3702e5e000 rw-p 0015b000 93:00 525516 /lib/libc-2.14.so
 30696:20110915:170253.251 7f3702e5e000-7f3702e63000 rw-p 00000000 00:00 0
 30696:20110915:170253.251 7f3702e63000-7f3702e76000 r-xp 00000000 93:00 530570 /lib/libresolv-2.14.so
 30696:20110915:170253.251 7f3702e76000-7f3703076000 ---p 00013000 93:00 530570 /lib/libresolv-2.14.so
 30696:20110915:170253.251 7f3703076000-7f3703077000 r--p 00013000 93:00 530570 /lib/libresolv-2.14.so
 30696:20110915:170253.252 7f3703077000-7f3703078000 rw-p 00014000 93:00 530570 /lib/libresolv-2.14.so
 30696:20110915:170253.252 7f3703078000-7f370307a000 rw-p 00000000 00:00 0
 30696:20110915:170253.252 7f370307a000-7f37030fb000 r-xp 00000000 93:00 530556 /lib/libm-2.14.so
 30696:20110915:170253.252 7f37030fb000-7f37032fa000 ---p 00081000 93:00 530556 /lib/libm-2.14.so
 30696:20110915:170253.252 7f37032fa000-7f37032fb000 r--p 00080000 93:00 530556 /lib/libm-2.14.so
 30696:20110915:170253.252 7f37032fb000-7f37032fc000 rw-p 00081000 93:00 530556 /lib/libm-2.14.so
 30696:20110915:170253.252 7f37032fc000-7f3703301000 r-xp 00000000 93:00 530023 /usr/local/lib/libOpenIPMIposix.so.0.0.1
 30696:20110915:170253.252 7f3703301000-7f3703501000 ---p 00005000 93:00 530023 /usr/local/lib/libOpenIPMIposix.so.0.0.1
 30696:20110915:170253.252 7f3703501000-7f3703502000 rw-p 00005000 93:00 530023 /usr/local/lib/libOpenIPMIposix.so.0.0.1
 30696:20110915:170253.252 7f3703502000-7f37035ed000 r-xp 00000000 93:00 530017 /usr/local/lib/libOpenIPMI.so.0.0.5
 30696:20110915:170253.252 7f37035ed000-7f37037ec000 ---p 000eb000 93:00 530017 /usr/local/lib/libOpenIPMI.so.0.0.5
 30696:20110915:170253.252 7f37037ec000-7f3703808000 rw-p 000ea000 93:00 530017 /usr/local/lib/libOpenIPMI.so.0.0.5
 30696:20110915:170253.252 7f3703808000-7f370380c000 rw-p 00000000 00:00 0
 30696:20110915:170253.252 7f370380c000-7f37039a2000 r-xp 00000000 93:00 527529 /usr/lib/libcrypto.so.1.0.0
 30696:20110915:170253.252 7f37039a2000-7f3703ba1000 ---p 00196000 93:00 527529 /usr/lib/libcrypto.so.1.0.0
 30696:20110915:170253.252 7f3703ba1000-7f3703bc4000 rw-p 00195000 93:00 527529 /usr/lib/libcrypto.so.1.0.0
 30696:20110915:170253.252 7f3703bc4000-7f3703bc8000 rw-p 00000000 00:00 0
 30696:20110915:170253.252 7f3703bc8000-7f3703c6f000 r-xp 00000000 93:00 532558 /usr/lib/libnetsnmp.so.25.0.1
 30696:20110915:170253.252 7f3703c6f000-7f3703e6e000 ---p 000a7000 93:00 532558 /usr/lib/libnetsnmp.so.25.0.1
 30696:20110915:170253.252 7f3703e6e000-7f3703e72000 rw-p 000a6000 93:00 532558 /usr/lib/libnetsnmp.so.25.0.1
 30696:20110915:170253.252 7f3703e72000-7f3703ea8000 rw-p 00000000 00:00 0
 30696:20110915:170253.252 7f3703ea8000-7f3703f02000 r-xp 00000000 93:00 532305 /usr/lib/libcurl.so.4.2.0
 30696:20110915:170253.252 7f3703f02000-7f3704102000 ---p 0005a000 93:00 532305 /usr/lib/libcurl.so.4.2.0
 30696:20110915:170253.252 7f3704102000-7f3704105000 rw-p 0005a000 93:00 532305 /usr/lib/libcurl.so.4.2.0
 30696:20110915:170253.252 7f3704105000-7f3704124000 r-xp 00000000 93:00 525453 /lib/ld-2.14.so
 30696:20110915:170253.252 7f370430a000-7f3704313000 rw-p 00000000 00:00 0
 30696:20110915:170253.252 7f3704320000-7f3704323000 rw-p 00000000 00:00 0
 30696:20110915:170253.252 7f3704323000-7f3704324000 r--p 0001e000 93:00 525453 /lib/ld-2.14.so
 30696:20110915:170253.252 7f3704324000-7f3704325000 rw-p 0001f000 93:00 525453 /lib/ld-2.14.so
 30696:20110915:170253.252 7f3704325000-7f3704326000 rw-p 00000000 00:00 0
 30696:20110915:170253.252 7fff52086000-7fff520f0000 rw-p 00000000 00:00 0 [stack]
 30696:20110915:170253.252 7fff521bb000-7fff521bc000 r-xp 00000000 00:00 0 [vdso]
 30696:20110915:170253.252 ffffffffff600000-ffffffffff601000 r-xp 00000000 00:00 0 [vsyscall]
 30696:20110915:170253.252 ================================
 30696:20110915:170253.252 Please consider attaching a disassembly listing to your bug report.
 30696:20110915:170253.252 This listing can be produced with, e.g., objdump -DSswx zabbix_server.
 30696:20110915:170253.252 ================================
 30671:20110915:170253.254 One child process died (PID:30696,exitcode/signal:255). Exiting ...
 30671:20110915:170255.255 Syncing history data...
 30671:20110915:170255.594 Syncing history data... done.
 30671:20110915:170255.594 Syncing trends data...
 30671:20110915:170257.168 Syncing trends data... done.
 30671:20110915:170257.168 Zabbix Server stopped. Zabbix 1.8.7 (revision 21394).

Activity

Hide
richlv added a comment -

this was most likely fixed as ZBX-4099
could you please try 1.8.8rc2 ?

Show
richlv added a comment - this was most likely fixed as ZBX-4099 could you please try 1.8.8rc2 ?
Hide
Robert Jerzak added a comment -

I just compiled 1.8.8rc2 and so far so good - no crashes.

After that I have problem with calculated items:

2277:20110915:174822.334 Item [h069:load.per_core] became not supported: Invalid first parameter in function [last("system.cpu.load[,avg1]")]

Item "load.per_core" formula looks like:
(last("system.cpu.load[,avg1]")/last("system.cpu.num"))*100

2277:20110915:174924.614 Item [h154:swap.used] became not supported: Invalid first parameter in function [last("system.swap.size[,total]")]

Item "swap.used" formula looks like:
last("system.swap.size[,total]")-last("system.swap.size[,free]")

2278:20110915:174822.327 Item [d042:disk.writes.ms] became not supported: Invalid first parameter in function [last("disk.writes.time")]

Item "disk.writes.ms" formula looks like:
(last("disk.writes.time")+1)/(last("disk.writes")+1)

According to documentation these items looks correct and works fine with previous version 1.8.7.

Is there any change in calculated item formula that makes incompatibility?

Show
Robert Jerzak added a comment - I just compiled 1.8.8rc2 and so far so good - no crashes. After that I have problem with calculated items: 2277:20110915:174822.334 Item [h069:load.per_core] became not supported: Invalid first parameter in function [last("system.cpu.load[,avg1]")] Item "load.per_core" formula looks like: (last("system.cpu.load[,avg1]")/last("system.cpu.num"))*100 2277:20110915:174924.614 Item [h154:swap.used] became not supported: Invalid first parameter in function [last("system.swap.size[,total]")] Item "swap.used" formula looks like: last("system.swap.size[,total]")-last("system.swap.size[,free]") 2278:20110915:174822.327 Item [d042:disk.writes.ms] became not supported: Invalid first parameter in function [last("disk.writes.time")] Item "disk.writes.ms" formula looks like: (last("disk.writes.time")+1)/(last("disk.writes")+1) According to documentation these items looks correct and works fine with previous version 1.8.7. Is there any change in calculated item formula that makes incompatibility?
Hide
richlv added a comment -

most likely they are missing data and can not be calculated right now. they are expected to populate data a bit later then

Show
richlv added a comment - most likely they are missing data and can not be calculated right now. they are expected to populate data a bit later then
Hide
Robert Jerzak added a comment -

It seems that all calculated item in my system became 'not supported' after update do version 1.8.8rc2 from crashing 1.8.7.

Show
Robert Jerzak added a comment - It seems that all calculated item in my system became 'not supported' after update do version 1.8.8rc2 from crashing 1.8.7.
Hide
Igor Danoshaites added a comment -

Robert,

Can you please try to activate your calculated items again ?

This could be done in the configuration screen of your item setting Status=Active

For us seems that this issue is similar with ZBX-4099 issue, where we fixed server crash when calculating trigger functions min(), max(), avg() or delta() and also when calculating aggregate and calculate items which are using the same trigger functions.

Can you please confirm that now this issue is fixed for you ?

Show
Igor Danoshaites added a comment - Robert, Can you please try to activate your calculated items again ? This could be done in the configuration screen of your item setting Status=Active For us seems that this issue is similar with ZBX-4099 issue, where we fixed server crash when calculating trigger functions min(), max(), avg() or delta() and also when calculating aggregate and calculate items which are using the same trigger functions. Can you please confirm that now this issue is fixed for you ?
Hide
Robert Jerzak added a comment -

Yes, I activated a few of calculated items again but they 'became not supported'.

So, I suppose crashing zabbix_server issue (1.8.7) is fixed but problem with processing calculate items is another issue.

Show
Robert Jerzak added a comment - Yes, I activated a few of calculated items again but they 'became not supported'. So, I suppose crashing zabbix_server issue (1.8.7) is fixed but problem with processing calculate items is another issue.
Alexander Vladishev made changes -
Field Original Value New Value
Assignee Alexander Vladishev [ sasha ]
Hide
Alexander Vladishev added a comment -

Confirmed incorrect processing of quoted function parameters in calculates items.

Show
Alexander Vladishev added a comment - Confirmed incorrect processing of quoted function parameters in calculates items.
Alexander Vladishev made changes -
Status Open [ 1 ] Confirmed [ 10000 ]
Alexander Vladishev made changes -
Status Confirmed [ 10000 ] In Progress [ 3 ]
Hide
Alexander Vladishev added a comment -

Available in the development branch svn://svn.zabbix.com/branches/dev/ZBX-4150

Show
Alexander Vladishev added a comment - Available in the development branch svn://svn.zabbix.com/branches/dev/ZBX-4150
Alexander Vladishev made changes -
Status In Progress [ 3 ] Resolved [ 5 ]
Assignee Alexander Vladishev [ sasha ] Rudolfs Kreicbergs [ stoney ]
Fix Version/s 1.8.8 [ 10401 ]
Fix Version/s 1.9.7 (trunk) [ 10502 ]
Resolution Fixed [ 1 ]
Hide
richlv added a comment - - edited

will this result in 1.8.8rc3 ? if it's a regression after 1.8.7, it might be a good idea

<rudolfs> yes, that is the current plan

<richlv> thanks

Show
richlv added a comment - - edited will this result in 1.8.8rc3 ? if it's a regression after 1.8.7, it might be a good idea <rudolfs> yes, that is the current plan <richlv> thanks
Rudolfs Kreicbergs made changes -
Status Resolved [ 5 ] Tested [ 10002 ]
Assignee Rudolfs Kreicbergs [ stoney ] Alexander Vladishev [ sasha ]
Alexander Vladishev made changes -
Summary zabbix_server crashed Calculated items become unsupported (was: zabbix_server crashed)
Hide
Alexander Vladishev added a comment -

Fixed in version pre1.8.8, r21808.

Show
Alexander Vladishev added a comment - Fixed in version pre1.8.8, r21808.
Alexander Vladishev made changes -
Status Tested [ 10002 ] Closed [ 6 ]
Assignee Alexander Vladishev [ sasha ]

People

Vote (0)
Watch (3)

Dates

  • Created:
    Updated:
    Resolved: