-
Problem report
-
Resolution: Cannot Reproduce
-
Critical
-
None
-
5.0.9
-
None
Zabbix server crashed. I'm attaching the log. The entry which may point us to the root cause of the crash is this: ERROR file:dbconfig.c,line:10332 Something impossible has just happened.
zabbix_server [1137727]: ERROR [file:dbconfig.c,line:10332] Something impossible has just happened. 1137727:20210331:021248.388 === Backtrace: === 1137727:20210331:021248.389 12: /usr/local/sbin/zabbix_server: poller #331 [got 2 values in 3.499265 sec, getting values](zbx_backtrace+0x35) [0x53a9e5] 1137727:20210331:021248.389 11: /usr/local/sbin/zabbix_server: poller #331 [got 2 values in 3.499265 sec, getting values]() [0x50dddd] 1137727:20210331:021248.389 10: /usr/local/sbin/zabbix_server: poller #331 [got 2 values in 3.499265 sec, getting values](DCget_host_count+0x10) [0x520540] 1137727:20210331:021248.389 9: /usr/local/sbin/zabbix_server: poller #331 [got 2 values in 3.499265 sec, getting values](get_value_internal+0x40d) [0x44503d] 1137727:20210331:021248.389 8: /usr/local/sbin/zabbix_server: poller #331 [got 2 values in 3.499265 sec, getting values](zbx_check_items+0x1b3) [0x440ed3] 1137727:20210331:021248.389 7: /usr/local/sbin/zabbix_server: poller #331 [got 2 values in 3.499265 sec, getting values]() [0x4413b9] 1137727:20210331:021248.389 6: /usr/local/sbin/zabbix_server: poller #331 [got 2 values in 3.499265 sec, getting values](poller_thread+0x104) [0x4418c4] 1137727:20210331:021248.389 5: /usr/local/sbin/zabbix_server: poller #331 [got 2 values in 3.499265 sec, getting values](zbx_thread_start+0x27) [0x544ff7] 1137727:20210331:021248.389 4: /usr/local/sbin/zabbix_server: poller #331 [got 2 values in 3.499265 sec, getting values](MAIN_ZABBIX_ENTRY+0x90c) [0x42d71c] 1137727:20210331:021248.389 3: /usr/local/sbin/zabbix_server: poller #331 [got 2 values in 3.499265 sec, getting values](daemon_start+0x1b1) [0x53a7e1] 1137727:20210331:021248.389 2: /usr/local/sbin/zabbix_server: poller #331 [got 2 values in 3.499265 sec, getting values](main+0x523) [0x425d13] 1137727:20210331:021248.389 1: /lib64/libc.so.6(__libc_start_main+0xf3) [0x7fd7e374b803] 1137727:20210331:021248.389 0: /usr/local/sbin/zabbix_server: poller #331 [got 2 values in 3.499265 sec, getting values](_start+0x2e) [0x42c8ee]
After which the actual crash happened:
1138369:20210331:021310.504 Got signal [signal:11(SIGSEGV),reason:1,refaddr:0x6063be41]. Crashing ...
1138369:20210331:021310.504 ====== Fatal information: ======
1138369:20210331:021310.505 Program counter: 0x7fd7e3885895
1138369:20210331:021310.505 === Registers: ===
1138369:20210331:021310.506 r8 = 200000 = 2097152 = 2097152
1138369:20210331:021310.506 r9 = 23ee740 = 37676864 = 37676864
1138369:20210331:021310.506 r10 = 2710 = 10000 = 10000
1138369:20210331:021310.507 r11 = 90 = 144 = 144
1138369:20210331:021310.507 r12 = 5db78d = 6141837 = 6141837
1138369:20210331:021310.508 r13 = 1c65 = 7269 = 7269
1138369:20210331:021310.508 r14 = 7fd67b3a82a8 = 140559167161000 = 140559167161000
1138369:20210331:021310.508 r15 = 3d87620 = 64517664 = 64517664
1138369:20210331:021310.508 rdi = 6063be41 = 1617149505 = 1617149505
1138369:20210331:021310.508 rsi = 1c65 = 7269 = 7269
1138369:20210331:021310.508 rbp = 6063be41 = 1617149505 = 1617149505
1138369:20210331:021310.508 rbx = 9 = 9 = 9
1138369:20210331:021310.508 rdx = 6063be41 = 1617149505 = 1617149505
1138369:20210331:021310.508 rax = d0 = 208 = 208
1138369:20210331:021310.508 rcx = 1 = 1 = 1
1138369:20210331:021310.508 rsp = 7ffd30c93118 = 140725421945112 = 140725421945112
1138369:20210331:021310.508 rip = 7fd7e3885895 = 140565212059797 = 140565212059797
1138369:20210331:021310.508 efl = 10287 = 66183 = 66183
1138369:20210331:021310.508 csgsfs = 2b000000000033 = 12103423998558259 = 12103423998558259
1138369:20210331:021310.508 err = 4 = 4 = 4
1138369:20210331:021310.508 trapno = e = 14 = 14
1138369:20210331:021310.508 oldmask = 0 = 0 = 0
1138369:20210331:021310.509 cr2 = 6063be41 = 1617149505 = 1617149505
1138369:20210331:021310.509 === Backtrace: ===
1138369:20210331:021310.509 14: /usr/local/sbin/zabbix_server: preprocessing manager #1 [queued 0, processed 5547 values, idle 5.291521 sec during 5.331242 sec](zbx_backtrace+0x35) [0x53a9e5]
1138369:20210331:021310.509 13: /usr/local/sbin/zabbix_server: preprocessing manager #1 [queued 0, processed 5547 values, idle 5.291521 sec during 5.331242 sec](zbx_log_fatal_info+0x29d) [0x53ad7d]
1138369:20210331:021310.509 12: /usr/local/sbin/zabbix_server: preprocessing manager #1 [queued 0, processed 5547 values, idle 5.291521 sec during 5.331242 sec]() [0x53b306]
1138369:20210331:021310.509 11: /lib64/libpthread.so.0(+0x12b30) [0x7fd7e4e50b30]
1138369:20210331:021310.509 10: /lib64/libc.so.6(+0x15d895) [0x7fd7e3885895]
1138369:20210331:021310.509 9: /lib64/libc.so.6(__strdup+0x12) [0x7fd7e37b1f92]
1138369:20210331:021310.509 8: /usr/local/sbin/zabbix_server: preprocessing manager #1 [queued 0, processed 5547 values, idle 5.291521 sec during 5.331242 sec](zbx_strdup2+0x33) [0x54ac33]
1138369:20210331:021310.509 7: /usr/local/sbin/zabbix_server: preprocessing manager #1 [queued 0, processed 5547 values, idle 5.291521 sec during 5.331242 sec](DCconfig_get_preprocessable_items+0x11e) [0x51ce0e]
1138369:20210331:021310.509 6: /usr/local/sbin/zabbix_server: preprocessing manager #1 [queued 0, processed 5547 values, idle 5.291521 sec during 5.331242 sec](preprocessing_manager_thread+0x665) [0x476645]
1138369:20210331:021310.509 5: /usr/local/sbin/zabbix_server: preprocessing manager #1 [queued 0, processed 5547 values, idle 5.291521 sec during 5.331242 sec](zbx_thread_start+0x27) [0x544ff7]
1138369:20210331:021310.509 4: /usr/local/sbin/zabbix_server: preprocessing manager #1 [queued 0, processed 5547 values, idle 5.291521 sec during 5.331242 sec](MAIN_ZABBIX_ENTRY+0x6ac) [0x42d4bc]
1138369:20210331:021310.509 3: /usr/local/sbin/zabbix_server: preprocessing manager #1 [queued 0, processed 5547 values, idle 5.291521 sec during 5.331242 sec](daemon_start+0x1b1) [0x53a7e1]
1138369:20210331:021310.509 2: /usr/local/sbin/zabbix_server: preprocessing manager #1 [queued 0, processed 5547 values, idle 5.291521 sec during 5.331242 sec](main+0x523) [0x425d13]
1138369:20210331:021310.509 1: /lib64/libc.so.6(__libc_start_main+0xf3) [0x7fd7e374b803]
1138369:20210331:021310.509 0: /usr/local/sbin/zabbix_server: preprocessing manager #1 [queued 0, processed 5547 values, idle 5.291521 sec during 5.331242 sec](_start+0x2e) [0x42c8ee]
1138369:20210331:021310.509 === Memory map: ===
1138369:20210331:021310.509 00400000-006cc000 r-xp 00000000 fd:02 14854 /usr/local/sbin/zabbix_server
1138369:20210331:021310.509 008cb000-008cc000 r--p 002cb000 fd:02 14854 /usr/local/sbin/zabbix_server
1138369:20210331:021310.509 008cc000-008d8000 rw-p 002cc000 fd:02 14854 /usr/local/sbin/zabbix_server
1138369:20210331:021310.509 008d8000-008e1000 rw-p 00000000 00:00 0
1138369:20210331:021310.509 01970000-01991000 rw-p 00000000 00:00 0 [heap]
1138369:20210331:021310.509 01991000-01b2d000 rw-p 00000000 00:00 0 [heap]
1138369:20210331:021310.509 01b2d000-0417d000 rw-p 00000000 00:00 0 [heap]
1138369:20210331:021310.509 7fd615aeb000-7fd615b02000 r-xp 00000000 fd:02 142635 /usr/lib64/libgcc_s-8-20191121.so.1
1138369:20210331:021310.509 7fd615b02000-7fd615d01000 ---p 00017000 fd:02 142635 /usr/lib64/libgcc_s-8-20191121.so.1
1138369:20210331:021310.509 7fd615d01000-7fd615d02000 r--p 00016000 fd:02 142635 /usr/lib64/libgcc_s-8-20191121.so.1
1138369:20210331:021310.509 7fd615d02000-7fd615d03000 rw-p 00017000 fd:02 142635 /usr/lib64/libgcc_s-8-20191121.so.1
1138369:20210331:021310.509 7fd615d03000-7fd615ec8000 rw-p 00000000 00:00 0
1138369:20210331:021310.509 7fd615ff6000-7fd615ffb000 r-xp 00000000 fd:02 137513 /usr/lib64/libnss_dns-2.28.so
1138369:20210331:021310.509 7fd615ffb000-7fd6161fb000 ---p 00005000 fd:02 137513 /usr/lib64/libnss_dns-2.28.so
1138369:20210331:021310.509 7fd6161fb000-7fd6161fc000 r--p 00005000 fd:02 137513 /usr/lib64/libnss_dns-2.28.so
1138369:20210331:021310.509 7fd6161fc000-7fd6161fd000 rw-p 00006000 fd:02 137513 /usr/lib64/libnss_dns-2.28.so
1138369:20210331:021310.509 7fd6161fd000-7fd616208000 r-xp 00000000 fd:02 137525 /usr/lib64/libnuma.so.1.0.0
1138369:20210331:021310.509 7fd616208000-7fd616407000 ---p 0000b000 fd:02 137525 /usr/lib64/libnuma.so.1.0.0
1138369:20210331:021310.509 7fd616407000-7fd616408000 r--p 0000a000 fd:02 137525 /usr/lib64/libnuma.so.1.0.0
1138369:20210331:021310.509 7fd616408000-7fd616409000 rw-p 0000b000 fd:02 137525 /usr/lib64/libnuma.so.1.0.0
1138369:20210331:021310.509 7fd616409000-7fd616414000 r-xp 00000000 fd:02 141817 /usr/lib64/libnss_files-2.28.so
1138369:20210331:021310.509 7fd616414000-7fd616613000 ---p 0000b000 fd:02 141817
1138369:20210331:021310.510 7fd7e4787000-7fd7e4795000 r-xp 00000000 fd:02 137472 /usr/lib64/liblber-2.4.so.2.10.9
1138369:20210331:021310.510 7fd7e4795000-7fd7e4995000 ---p 0000e000 fd:02 137472 /usr/lib64/liblber-2.4.so.2.10.9
1138369:20210331:021310.510 7fd7e4995000-7fd7e4996000 r--p 0000e000 fd:02 137472 /usr/lib64/liblber-2.4.so.2.10.9
1138369:20210331:021310.510 7fd7e4996000-7fd7e4997000 rw-p 0000f000 fd:02 137472 /usr/lib64/liblber-2.4.so.2.10.9
1138369:20210331:021310.510 7fd7e4997000-7fd7e49e3000 r-xp 00000000 fd:02 141898 /usr/lib64/libldap-2.4.so.2.10.9
1138369:20210331:021310.510 7fd7e49e3000-7fd7e4be2000 ---p 0004c000 fd:02 141898 /usr/lib64/libldap-2.4.so.2.10.9
1138369:20210331:021310.510 7fd7e4be2000-7fd7e4be4000 r--p 0004b000 fd:02 141898 /usr/lib64/libldap-2.4.so.2.10.9
1138369:20210331:021310.510 7fd7e4be4000-7fd7e4be5000 rw-p 0004d000 fd:02 141898 /usr/lib64/libldap-2.4.so.2.10.9
1138369:20210331:021310.510 7fd7e4be5000-7fd7e4c3b000 r-xp 00000000 fd:02 137539 /usr/lib64/libevent-2.1.so.6.0.2
1138369:20210331:021310.510 7fd7e4c3b000-7fd7e4e3b000 ---p 00056000 fd:02 137539 /usr/lib64/libevent-2.1.so.6.0.2
1138369:20210331:021310.510 7fd7e4e3b000-7fd7e4e3d000 r--p 00056000 fd:02 137539 /usr/lib64/libevent-2.1.so.6.0.2
1138369:20210331:021310.510 7fd7e4e3d000-7fd7e4e3e000 rw-p 00058000 fd:02 137539 /usr/lib64/libevent-2.1.so.6.0.2
1138369:20210331:021310.510 7fd7e4e3e000-7fd7e4e59000 r-xp 00000000 fd:02 137473 /usr/lib64/libpthread-2.28.so
1138369:20210331:021310.510 7fd7e4e59000-7fd7e5058000 ---p 0001b000 fd:02 137473 /usr/lib64/libpthread-2.28.so
1138369:20210331:021310.510 7fd7e5058000-7fd7e5059000 r--p 0001a000 fd:02 137473 /usr/lib64/libpthread-2.28.so
1138369:20210331:021310.510 7fd7e5059000-7fd7e505a000 rw-p 0001b000 fd:02 137473 /usr/lib64/libpthread-2.28.so
1138369:20210331:021310.510 7fd7e505a000-7fd7e505e000 rw-p 00000000 00:00 0
1138369:20210331:021310.510 7fd7e505e000-7fd7e5074000 r-xp 00000000 fd:02 141937 /usr/lib64/libz.so.1.2.11
1138369:20210331:021310.510 7fd7e5074000-7fd7e5273000 ---p 00016000 fd:02 141937 /usr/lib64/libz.so.1.2.11
1138369:20210331:021310.510 7fd7e5273000-7fd7e5274000 r--p 00015000 fd:02 141937 /usr/lib64/libz.so.1.2.11
1138369:20210331:021310.510 7fd7e5274000-7fd7e5275000 rw-p 00000000 00:00 0
1138369:20210331:021310.510 7fd7e5275000-7fd7e52b2000 r-xp 00000000 fd:02 163087 /usr/lib64/libssh2.so.1.0.1
1138369:20210331:021310.510 7fd7e52b2000-7fd7e54b1000 ---p 0003d000 fd:02 163087 /usr/lib64/libssh2.so.1.0.1
1138369:20210331:021310.510 7fd7e54b1000-7fd7e54b3000 r--p 0003c000 fd:02 163087 /usr/lib64/libssh2.so.1.0.1
1138369:20210331:021310.510 7fd7e54b3000-7fd7e54b4000 rw-p 0003e000 fd:02 163087 /usr/lib64/libssh2.so.1.0.1
1138369:20210331:021310.510 7fd7e54b4000-7fd7e5593000 r-xp 00000000 fd:02 163018 /usr/lib64/libnetsnmp.so.35.0.0
1138369:20210331:021310.510 7fd7e5593000-7fd7e5793000 ---p 000df000 fd:02 163018 /usr/lib64/libnetsnmp.so.35.0.0
1138369:20210331:021310.510 7fd7e5793000-7fd7e5796000 r--p 000df000 fd:02 163018 /usr/lib64/libnetsnmp.so.35.0.0
1138369:20210331:021310.510 7fd7e5796000-7fd7e5799000 rw-p 000e2000 fd:02 163018 /usr/lib64/libnetsnmp.so.35.0.0
1138369:20210331:021310.510 7fd7e5799000-7fd7e585f000 rw-p 00000000 00:00 0
1138369:20210331:021310.510 7fd7e585f000-7fd7e58c4000 r-xp 00000000 fd:02 168432 /usr/lib64/libodbc.so.2.0.0
1138369:20210331:021310.510 7fd7e58c4000-7fd7e5ac4000 ---p 00065000 fd:02 168432 /usr/lib64/libodbc.so.2.0.0
1138369:20210331:021310.510 7fd7e5ac4000-7fd7e5ac5000 r--p 00065000 fd:02 168432 /usr/lib64/libodbc.so.2.0.0
1138369:20210331:021310.510 7fd7e5ac5000-7fd7e5acc000 rw-p 00066000 fd:02 168432 /usr/lib64/libodbc.so.2.0.0
1138369:20210331:021310.510 7fd7e5acc000-7fd7e5ad0000 rw-p 00000000 00:00 0
1138369:20210331:021310.510 7fd7e5ad0000-7fd7e5c2d000 r-xp 00000000 fd:02 142072 /usr/lib64/libxml2.so.2.9.7
1138369:20210331:021310.510 7fd7e5c2d000-7fd7e5e2c000 ---p 0015d000 fd:02 142072 /usr/lib64/libxml2.so.2.9.7
1138369:20210331:021310.510 7fd7e5e2c000-7fd7e5e35000 r--p 0015c000 fd:02 142072 /usr/lib64/libxml2.so.2.9.7
1138369:20210331:021310.510 7fd7e5e35000-7fd7e5e36000 rw-p 00165000 fd:02 142072 /usr/lib64/libxml2.so.2.9.7
1138369:20210331:021310.510 7fd7e5e36000-7fd7e5e38000 rw-p 00000000 00:00 0
1138369:20210331:021310.510 7fd7e5e38000-7fd7e636c000 r-xp 00000000 fd:02 44440 /usr/lib/oracle/21/client64/lib/libnnz21.so
1138369:20210331:021310.510 7fd7e636c000-7fd7e656b000 ---p 00534000 fd:02 44440 /usr/lib/oracle/21/client64/lib/libnnz21.so
1138369:20210331:021310.510 7fd7e656b000-7fd7e65a3000 r--p 00533000 fd:02 44440 /usr/lib/oracle/21/client64/lib/libnnz21.so
1138369:20210331:021310.510 7fd7e65a3000-7fd7e65e0000 rw-p 0056b000 fd:02 44440 /usr/lib/oracle/21/client64/lib/libnnz21.so
1138369:20210331:021310.510 7fd7e65e0000-7fd7e65e2000 rw-p 00000000 00:00 0
1138369:20210331:021310.510 7fd7e65e2000-7fd7ea572000 r-xp 00000000 fd:02 44437 /usr/lib/oracle/21/client64/lib/libclntsh.so.21.1
1138369:20210331:021310.510 7fd7ea572000-7fd7ea771000 ---p 03f90000 fd:02 44437 /usr/lib/oracle/21/client64/lib/libclntsh.so.21.1
1138369:20210331:021310.510 7fd7ea771000-7fd7ea8f3000 r--p 03f8f000 fd:02 44437 /usr/lib/oracle/21/client64/lib/libclntsh.so.21.1
1138369:20210331:021310.510 7fd7ea8f3000-7fd7ea936000 rw-p 04111000 fd:02 44437 /usr/lib/oracle/21/client64/lib/libclntsh.so.21.1
1138369:20210331:021310.510 7fd7ea936000-7fd7ea963000 rw-p 00000000 00:00 0
1138369:20210331:021310.510 7fd7ea963000-7fd7ea98c000 r-xp 00000000 fd:02 147350 /usr/lib64/ld-2.28.so
1138369:20210331:021310.510 7fd7eaac4000-7fd7eab08000 rw-p 00000000 00:00 0
1138369:20210331:021310.510 7fd7eab08000-7fd7eab5d000 rw-s 00000000 00:05 32783 /SYSV00000000 (deleted)
1138369:20210331:021310.510 7fd7eab5d000-7fd7eab7d000 rw-p 00000000 00:00 0
1138369:20210331:021310.510 7fd7eab88000-7fd7eab89000 rw-s 00000000 00:05 32778 /SYSV00000000 (deleted)
1138369:20210331:021310.510 7fd7eab89000-7fd7eab8b000 rw-p 00000000 00:00 0
1138369:20210331:021310.510 7fd7eab8b000-7fd7eab8c000 r--p 00028000 fd:02 147350 /usr/lib64/ld-2.28.so
1138369:20210331:021310.510 7fd7eab8c000-7fd7eab8d000 rw-p 00029000 fd:02 147350 /usr/lib64/ld-2.28.so
1138369:20210331:021310.510 7fd7eab8d000-7fd7eab8e000 rw-p 00000000 00:00 0
1138369:20210331:021310.510 7ffd30c6b000-7ffd30c96000 rw-p 00000000 00:00 0 [stack]
1138369:20210331:021310.510 7ffd30d06000-7ffd30d09000 r--p 00000000 00:00 0 [vvar]
1138369:20210331:021310.510 7ffd30d09000-7ffd30d0b000 r-xp 00000000 00:00 0 [vdso]
1138369:20210331:021310.510 ffffffffff600000-ffffffffff601000 r-xp 00000000 00:00 0 [vsyscall]
1138369:20210331:021310.510 ================================
1138369:20210331:021310.510 Please consider attaching a disassembly listing to your bug report.
1138369:20210331:021310.510 This listing can be produced with, e.g., objdump -DSswx zabbix_server.
1138369:20210331:021310.510 ================================
1138415:20210331:021310.530 cannot read preprocessing service request
1138412:20210331:021310.530 cannot read preprocessing service request
1138390:20210331:021310.530 cannot read preprocessing service request
1138383:20210331:021310.538 cannot read preprocessing service request
1138376:20210331:021310.538 cannot read preprocessing service request
1137285:20210331:021310.539 One child process died (PID:1138369,exitcode/signal:1). Exiting ...
zabbix_server [1137285]: Error waiting for process with PID 1138369: [10] No child processes
1137285:20210331:021311.936 syncing history data...
1137285:20210331:021312.021 syncing history data... 100.000000%
1137285:20210331:021312.021 syncing history data done
1137285:20210331:021312.021 syncing trend data...
1137285:20210331:021321.709 syncing trend data done
1137285:20210331:021321.827 Zabbix Server stopped. Zabbix 5.0.9 (revision 4d07aaafe2).
The DB is Oracle 19c, schema of 5.0.2 used