Uploaded image for project: 'ZABBIX BUGS AND ISSUES'
  1. ZABBIX BUGS AND ISSUES
  2. ZBX-19193

ERROR file:dbconfig.c,line:10332 Something impossible has just happened

    XMLWordPrintable

Details

    • Problem report
    • Status: Closed
    • Critical
    • Resolution: Cannot Reproduce
    • 5.0.9
    • None
    • Server (S)
    • None

    Description

      Zabbix server crashed. I'm attaching the log. The entry which may point us to the root cause of the crash is this: ERROR file:dbconfig.c,line:10332 Something impossible has just happened. 

      zabbix_server [1137727]: ERROR [file:dbconfig.c,line:10332] Something impossible has just happened.
      1137727:20210331:021248.388 === Backtrace: ===
      1137727:20210331:021248.389 12: /usr/local/sbin/zabbix_server: poller #331 [got 2 values in 3.499265 sec, getting values](zbx_backtrace+0x35) [0x53a9e5]
      1137727:20210331:021248.389 11: /usr/local/sbin/zabbix_server: poller #331 [got 2 values in 3.499265 sec, getting values]() [0x50dddd]
      1137727:20210331:021248.389 10: /usr/local/sbin/zabbix_server: poller #331 [got 2 values in 3.499265 sec, getting values](DCget_host_count+0x10) [0x520540]
      1137727:20210331:021248.389 9: /usr/local/sbin/zabbix_server: poller #331 [got 2 values in 3.499265 sec, getting values](get_value_internal+0x40d) [0x44503d]
      1137727:20210331:021248.389 8: /usr/local/sbin/zabbix_server: poller #331 [got 2 values in 3.499265 sec, getting values](zbx_check_items+0x1b3) [0x440ed3]
      1137727:20210331:021248.389 7: /usr/local/sbin/zabbix_server: poller #331 [got 2 values in 3.499265 sec, getting values]() [0x4413b9]
      1137727:20210331:021248.389 6: /usr/local/sbin/zabbix_server: poller #331 [got 2 values in 3.499265 sec, getting values](poller_thread+0x104) [0x4418c4]
      1137727:20210331:021248.389 5: /usr/local/sbin/zabbix_server: poller #331 [got 2 values in 3.499265 sec, getting values](zbx_thread_start+0x27) [0x544ff7]
      1137727:20210331:021248.389 4: /usr/local/sbin/zabbix_server: poller #331 [got 2 values in 3.499265 sec, getting values](MAIN_ZABBIX_ENTRY+0x90c) [0x42d71c]
      1137727:20210331:021248.389 3: /usr/local/sbin/zabbix_server: poller #331 [got 2 values in 3.499265 sec, getting values](daemon_start+0x1b1) [0x53a7e1]
      1137727:20210331:021248.389 2: /usr/local/sbin/zabbix_server: poller #331 [got 2 values in 3.499265 sec, getting values](main+0x523) [0x425d13]
      1137727:20210331:021248.389 1: /lib64/libc.so.6(__libc_start_main+0xf3) [0x7fd7e374b803]
      1137727:20210331:021248.389 0: /usr/local/sbin/zabbix_server: poller #331 [got 2 values in 3.499265 sec, getting values](_start+0x2e) [0x42c8ee]
      

      After which the actual crash happened:

      1138369:20210331:021310.504 Got signal [signal:11(SIGSEGV),reason:1,refaddr:0x6063be41]. Crashing ...
      1138369:20210331:021310.504 ====== Fatal information: ======
      1138369:20210331:021310.505 Program counter: 0x7fd7e3885895
      1138369:20210331:021310.505 === Registers: ===
      1138369:20210331:021310.506 r8      =           200000 =              2097152 =              2097152
      1138369:20210331:021310.506 r9      =          23ee740 =             37676864 =             37676864
      1138369:20210331:021310.506 r10     =             2710 =                10000 =                10000
      1138369:20210331:021310.507 r11     =               90 =                  144 =                  144
      1138369:20210331:021310.507 r12     =           5db78d =              6141837 =              6141837
      1138369:20210331:021310.508 r13     =             1c65 =                 7269 =                 7269
      1138369:20210331:021310.508 r14     =     7fd67b3a82a8 =      140559167161000 =      140559167161000
      1138369:20210331:021310.508 r15     =          3d87620 =             64517664 =             64517664
      1138369:20210331:021310.508 rdi     =         6063be41 =           1617149505 =           1617149505
      1138369:20210331:021310.508 rsi     =             1c65 =                 7269 =                 7269
      1138369:20210331:021310.508 rbp     =         6063be41 =           1617149505 =           1617149505
      1138369:20210331:021310.508 rbx     =                9 =                    9 =                    9
      1138369:20210331:021310.508 rdx     =         6063be41 =           1617149505 =           1617149505
      1138369:20210331:021310.508 rax     =               d0 =                  208 =                  208
      1138369:20210331:021310.508 rcx     =                1 =                    1 =                    1
      1138369:20210331:021310.508 rsp     =     7ffd30c93118 =      140725421945112 =      140725421945112
      1138369:20210331:021310.508 rip     =     7fd7e3885895 =      140565212059797 =      140565212059797
      1138369:20210331:021310.508 efl     =            10287 =                66183 =                66183
      1138369:20210331:021310.508 csgsfs  =   2b000000000033 =    12103423998558259 =    12103423998558259
      1138369:20210331:021310.508 err     =                4 =                    4 =                    4
      1138369:20210331:021310.508 trapno  =                e =                   14 =                   14
      1138369:20210331:021310.508 oldmask =                0 =                    0 =                    0
      1138369:20210331:021310.509 cr2     =         6063be41 =           1617149505 =           1617149505
      1138369:20210331:021310.509 === Backtrace: ===
      1138369:20210331:021310.509 14: /usr/local/sbin/zabbix_server: preprocessing manager #1 [queued 0, processed 5547 values, idle 5.291521 sec during 5.331242 sec](zbx_backtrace+0x35) [0x53a9e5]
      1138369:20210331:021310.509 13: /usr/local/sbin/zabbix_server: preprocessing manager #1 [queued 0, processed 5547 values, idle 5.291521 sec during 5.331242 sec](zbx_log_fatal_info+0x29d) [0x53ad7d]
      1138369:20210331:021310.509 12: /usr/local/sbin/zabbix_server: preprocessing manager #1 [queued 0, processed 5547 values, idle 5.291521 sec during 5.331242 sec]() [0x53b306]
      1138369:20210331:021310.509 11: /lib64/libpthread.so.0(+0x12b30) [0x7fd7e4e50b30]
      1138369:20210331:021310.509 10: /lib64/libc.so.6(+0x15d895) [0x7fd7e3885895]
      1138369:20210331:021310.509 9: /lib64/libc.so.6(__strdup+0x12) [0x7fd7e37b1f92]
      1138369:20210331:021310.509 8: /usr/local/sbin/zabbix_server: preprocessing manager #1 [queued 0, processed 5547 values, idle 5.291521 sec during 5.331242 sec](zbx_strdup2+0x33) [0x54ac33]
      1138369:20210331:021310.509 7: /usr/local/sbin/zabbix_server: preprocessing manager #1 [queued 0, processed 5547 values, idle 5.291521 sec during 5.331242 sec](DCconfig_get_preprocessable_items+0x11e) [0x51ce0e]
      1138369:20210331:021310.509 6: /usr/local/sbin/zabbix_server: preprocessing manager #1 [queued 0, processed 5547 values, idle 5.291521 sec during 5.331242 sec](preprocessing_manager_thread+0x665) [0x476645]
      1138369:20210331:021310.509 5: /usr/local/sbin/zabbix_server: preprocessing manager #1 [queued 0, processed 5547 values, idle 5.291521 sec during 5.331242 sec](zbx_thread_start+0x27) [0x544ff7]
      1138369:20210331:021310.509 4: /usr/local/sbin/zabbix_server: preprocessing manager #1 [queued 0, processed 5547 values, idle 5.291521 sec during 5.331242 sec](MAIN_ZABBIX_ENTRY+0x6ac) [0x42d4bc]
      1138369:20210331:021310.509 3: /usr/local/sbin/zabbix_server: preprocessing manager #1 [queued 0, processed 5547 values, idle 5.291521 sec during 5.331242 sec](daemon_start+0x1b1) [0x53a7e1]
      1138369:20210331:021310.509 2: /usr/local/sbin/zabbix_server: preprocessing manager #1 [queued 0, processed 5547 values, idle 5.291521 sec during 5.331242 sec](main+0x523) [0x425d13]
      1138369:20210331:021310.509 1: /lib64/libc.so.6(__libc_start_main+0xf3) [0x7fd7e374b803]
      1138369:20210331:021310.509 0: /usr/local/sbin/zabbix_server: preprocessing manager #1 [queued 0, processed 5547 values, idle 5.291521 sec during 5.331242 sec](_start+0x2e) [0x42c8ee]
      1138369:20210331:021310.509 === Memory map: ===
      1138369:20210331:021310.509 00400000-006cc000 r-xp 00000000 fd:02 14854                              /usr/local/sbin/zabbix_server
      1138369:20210331:021310.509 008cb000-008cc000 r--p 002cb000 fd:02 14854                              /usr/local/sbin/zabbix_server
      1138369:20210331:021310.509 008cc000-008d8000 rw-p 002cc000 fd:02 14854                              /usr/local/sbin/zabbix_server
      1138369:20210331:021310.509 008d8000-008e1000 rw-p 00000000 00:00 0 
      1138369:20210331:021310.509 01970000-01991000 rw-p 00000000 00:00 0                                  [heap]
      1138369:20210331:021310.509 01991000-01b2d000 rw-p 00000000 00:00 0                                  [heap]
      1138369:20210331:021310.509 01b2d000-0417d000 rw-p 00000000 00:00 0                                  [heap]
      1138369:20210331:021310.509 7fd615aeb000-7fd615b02000 r-xp 00000000 fd:02 142635                     /usr/lib64/libgcc_s-8-20191121.so.1
      1138369:20210331:021310.509 7fd615b02000-7fd615d01000 ---p 00017000 fd:02 142635                     /usr/lib64/libgcc_s-8-20191121.so.1
      1138369:20210331:021310.509 7fd615d01000-7fd615d02000 r--p 00016000 fd:02 142635                     /usr/lib64/libgcc_s-8-20191121.so.1
      1138369:20210331:021310.509 7fd615d02000-7fd615d03000 rw-p 00017000 fd:02 142635                     /usr/lib64/libgcc_s-8-20191121.so.1
      1138369:20210331:021310.509 7fd615d03000-7fd615ec8000 rw-p 00000000 00:00 0 
      1138369:20210331:021310.509 7fd615ff6000-7fd615ffb000 r-xp 00000000 fd:02 137513                     /usr/lib64/libnss_dns-2.28.so
      1138369:20210331:021310.509 7fd615ffb000-7fd6161fb000 ---p 00005000 fd:02 137513                     /usr/lib64/libnss_dns-2.28.so
      1138369:20210331:021310.509 7fd6161fb000-7fd6161fc000 r--p 00005000 fd:02 137513                     /usr/lib64/libnss_dns-2.28.so
      1138369:20210331:021310.509 7fd6161fc000-7fd6161fd000 rw-p 00006000 fd:02 137513                     /usr/lib64/libnss_dns-2.28.so
      1138369:20210331:021310.509 7fd6161fd000-7fd616208000 r-xp 00000000 fd:02 137525                     /usr/lib64/libnuma.so.1.0.0
      1138369:20210331:021310.509 7fd616208000-7fd616407000 ---p 0000b000 fd:02 137525                     /usr/lib64/libnuma.so.1.0.0
      1138369:20210331:021310.509 7fd616407000-7fd616408000 r--p 0000a000 fd:02 137525                     /usr/lib64/libnuma.so.1.0.0
      1138369:20210331:021310.509 7fd616408000-7fd616409000 rw-p 0000b000 fd:02 137525                     /usr/lib64/libnuma.so.1.0.0
      1138369:20210331:021310.509 7fd616409000-7fd616414000 r-xp 00000000 fd:02 141817                     /usr/lib64/libnss_files-2.28.so
      1138369:20210331:021310.509 7fd616414000-7fd616613000 ---p 0000b000 fd:02 141817                     
      1138369:20210331:021310.510 7fd7e4787000-7fd7e4795000 r-xp 00000000 fd:02 137472                     /usr/lib64/liblber-2.4.so.2.10.9
      1138369:20210331:021310.510 7fd7e4795000-7fd7e4995000 ---p 0000e000 fd:02 137472                     /usr/lib64/liblber-2.4.so.2.10.9
      1138369:20210331:021310.510 7fd7e4995000-7fd7e4996000 r--p 0000e000 fd:02 137472                     /usr/lib64/liblber-2.4.so.2.10.9
      1138369:20210331:021310.510 7fd7e4996000-7fd7e4997000 rw-p 0000f000 fd:02 137472                     /usr/lib64/liblber-2.4.so.2.10.9
      1138369:20210331:021310.510 7fd7e4997000-7fd7e49e3000 r-xp 00000000 fd:02 141898                     /usr/lib64/libldap-2.4.so.2.10.9
      1138369:20210331:021310.510 7fd7e49e3000-7fd7e4be2000 ---p 0004c000 fd:02 141898                     /usr/lib64/libldap-2.4.so.2.10.9
      1138369:20210331:021310.510 7fd7e4be2000-7fd7e4be4000 r--p 0004b000 fd:02 141898                     /usr/lib64/libldap-2.4.so.2.10.9
      1138369:20210331:021310.510 7fd7e4be4000-7fd7e4be5000 rw-p 0004d000 fd:02 141898                     /usr/lib64/libldap-2.4.so.2.10.9
      1138369:20210331:021310.510 7fd7e4be5000-7fd7e4c3b000 r-xp 00000000 fd:02 137539                     /usr/lib64/libevent-2.1.so.6.0.2
      1138369:20210331:021310.510 7fd7e4c3b000-7fd7e4e3b000 ---p 00056000 fd:02 137539                     /usr/lib64/libevent-2.1.so.6.0.2
      1138369:20210331:021310.510 7fd7e4e3b000-7fd7e4e3d000 r--p 00056000 fd:02 137539                     /usr/lib64/libevent-2.1.so.6.0.2
      1138369:20210331:021310.510 7fd7e4e3d000-7fd7e4e3e000 rw-p 00058000 fd:02 137539                     /usr/lib64/libevent-2.1.so.6.0.2
      1138369:20210331:021310.510 7fd7e4e3e000-7fd7e4e59000 r-xp 00000000 fd:02 137473                     /usr/lib64/libpthread-2.28.so
      1138369:20210331:021310.510 7fd7e4e59000-7fd7e5058000 ---p 0001b000 fd:02 137473                     /usr/lib64/libpthread-2.28.so
      1138369:20210331:021310.510 7fd7e5058000-7fd7e5059000 r--p 0001a000 fd:02 137473                     /usr/lib64/libpthread-2.28.so
      1138369:20210331:021310.510 7fd7e5059000-7fd7e505a000 rw-p 0001b000 fd:02 137473                     /usr/lib64/libpthread-2.28.so
      1138369:20210331:021310.510 7fd7e505a000-7fd7e505e000 rw-p 00000000 00:00 0 
      1138369:20210331:021310.510 7fd7e505e000-7fd7e5074000 r-xp 00000000 fd:02 141937                     /usr/lib64/libz.so.1.2.11
      1138369:20210331:021310.510 7fd7e5074000-7fd7e5273000 ---p 00016000 fd:02 141937                     /usr/lib64/libz.so.1.2.11
      1138369:20210331:021310.510 7fd7e5273000-7fd7e5274000 r--p 00015000 fd:02 141937                     /usr/lib64/libz.so.1.2.11
      1138369:20210331:021310.510 7fd7e5274000-7fd7e5275000 rw-p 00000000 00:00 0 
      1138369:20210331:021310.510 7fd7e5275000-7fd7e52b2000 r-xp 00000000 fd:02 163087                     /usr/lib64/libssh2.so.1.0.1
      1138369:20210331:021310.510 7fd7e52b2000-7fd7e54b1000 ---p 0003d000 fd:02 163087                     /usr/lib64/libssh2.so.1.0.1
      1138369:20210331:021310.510 7fd7e54b1000-7fd7e54b3000 r--p 0003c000 fd:02 163087                     /usr/lib64/libssh2.so.1.0.1
      1138369:20210331:021310.510 7fd7e54b3000-7fd7e54b4000 rw-p 0003e000 fd:02 163087                     /usr/lib64/libssh2.so.1.0.1
      1138369:20210331:021310.510 7fd7e54b4000-7fd7e5593000 r-xp 00000000 fd:02 163018                     /usr/lib64/libnetsnmp.so.35.0.0
      1138369:20210331:021310.510 7fd7e5593000-7fd7e5793000 ---p 000df000 fd:02 163018                     /usr/lib64/libnetsnmp.so.35.0.0
      1138369:20210331:021310.510 7fd7e5793000-7fd7e5796000 r--p 000df000 fd:02 163018                     /usr/lib64/libnetsnmp.so.35.0.0
      1138369:20210331:021310.510 7fd7e5796000-7fd7e5799000 rw-p 000e2000 fd:02 163018                     /usr/lib64/libnetsnmp.so.35.0.0
      1138369:20210331:021310.510 7fd7e5799000-7fd7e585f000 rw-p 00000000 00:00 0 
      1138369:20210331:021310.510 7fd7e585f000-7fd7e58c4000 r-xp 00000000 fd:02 168432                     /usr/lib64/libodbc.so.2.0.0
      1138369:20210331:021310.510 7fd7e58c4000-7fd7e5ac4000 ---p 00065000 fd:02 168432                     /usr/lib64/libodbc.so.2.0.0
      1138369:20210331:021310.510 7fd7e5ac4000-7fd7e5ac5000 r--p 00065000 fd:02 168432                     /usr/lib64/libodbc.so.2.0.0
      1138369:20210331:021310.510 7fd7e5ac5000-7fd7e5acc000 rw-p 00066000 fd:02 168432                     /usr/lib64/libodbc.so.2.0.0
      1138369:20210331:021310.510 7fd7e5acc000-7fd7e5ad0000 rw-p 00000000 00:00 0 
      1138369:20210331:021310.510 7fd7e5ad0000-7fd7e5c2d000 r-xp 00000000 fd:02 142072                     /usr/lib64/libxml2.so.2.9.7
      1138369:20210331:021310.510 7fd7e5c2d000-7fd7e5e2c000 ---p 0015d000 fd:02 142072                     /usr/lib64/libxml2.so.2.9.7
      1138369:20210331:021310.510 7fd7e5e2c000-7fd7e5e35000 r--p 0015c000 fd:02 142072                     /usr/lib64/libxml2.so.2.9.7
      1138369:20210331:021310.510 7fd7e5e35000-7fd7e5e36000 rw-p 00165000 fd:02 142072                     /usr/lib64/libxml2.so.2.9.7
      1138369:20210331:021310.510 7fd7e5e36000-7fd7e5e38000 rw-p 00000000 00:00 0 
      1138369:20210331:021310.510 7fd7e5e38000-7fd7e636c000 r-xp 00000000 fd:02 44440                      /usr/lib/oracle/21/client64/lib/libnnz21.so
      1138369:20210331:021310.510 7fd7e636c000-7fd7e656b000 ---p 00534000 fd:02 44440                      /usr/lib/oracle/21/client64/lib/libnnz21.so
      1138369:20210331:021310.510 7fd7e656b000-7fd7e65a3000 r--p 00533000 fd:02 44440                      /usr/lib/oracle/21/client64/lib/libnnz21.so
      1138369:20210331:021310.510 7fd7e65a3000-7fd7e65e0000 rw-p 0056b000 fd:02 44440                      /usr/lib/oracle/21/client64/lib/libnnz21.so
      1138369:20210331:021310.510 7fd7e65e0000-7fd7e65e2000 rw-p 00000000 00:00 0 
      1138369:20210331:021310.510 7fd7e65e2000-7fd7ea572000 r-xp 00000000 fd:02 44437                      /usr/lib/oracle/21/client64/lib/libclntsh.so.21.1
      1138369:20210331:021310.510 7fd7ea572000-7fd7ea771000 ---p 03f90000 fd:02 44437                      /usr/lib/oracle/21/client64/lib/libclntsh.so.21.1
      1138369:20210331:021310.510 7fd7ea771000-7fd7ea8f3000 r--p 03f8f000 fd:02 44437                      /usr/lib/oracle/21/client64/lib/libclntsh.so.21.1
      1138369:20210331:021310.510 7fd7ea8f3000-7fd7ea936000 rw-p 04111000 fd:02 44437                      /usr/lib/oracle/21/client64/lib/libclntsh.so.21.1
      1138369:20210331:021310.510 7fd7ea936000-7fd7ea963000 rw-p 00000000 00:00 0 
      1138369:20210331:021310.510 7fd7ea963000-7fd7ea98c000 r-xp 00000000 fd:02 147350                     /usr/lib64/ld-2.28.so
      1138369:20210331:021310.510 7fd7eaac4000-7fd7eab08000 rw-p 00000000 00:00 0 
      1138369:20210331:021310.510 7fd7eab08000-7fd7eab5d000 rw-s 00000000 00:05 32783                      /SYSV00000000 (deleted)
      1138369:20210331:021310.510 7fd7eab5d000-7fd7eab7d000 rw-p 00000000 00:00 0 
      1138369:20210331:021310.510 7fd7eab88000-7fd7eab89000 rw-s 00000000 00:05 32778                      /SYSV00000000 (deleted)
      1138369:20210331:021310.510 7fd7eab89000-7fd7eab8b000 rw-p 00000000 00:00 0 
      1138369:20210331:021310.510 7fd7eab8b000-7fd7eab8c000 r--p 00028000 fd:02 147350                     /usr/lib64/ld-2.28.so
      1138369:20210331:021310.510 7fd7eab8c000-7fd7eab8d000 rw-p 00029000 fd:02 147350                     /usr/lib64/ld-2.28.so
      1138369:20210331:021310.510 7fd7eab8d000-7fd7eab8e000 rw-p 00000000 00:00 0 
      1138369:20210331:021310.510 7ffd30c6b000-7ffd30c96000 rw-p 00000000 00:00 0                          [stack]
      1138369:20210331:021310.510 7ffd30d06000-7ffd30d09000 r--p 00000000 00:00 0                          [vvar]
      1138369:20210331:021310.510 7ffd30d09000-7ffd30d0b000 r-xp 00000000 00:00 0                          [vdso]
      1138369:20210331:021310.510 ffffffffff600000-ffffffffff601000 r-xp 00000000 00:00 0                  [vsyscall]
      1138369:20210331:021310.510 ================================
      1138369:20210331:021310.510 Please consider attaching a disassembly listing to your bug report.
      1138369:20210331:021310.510 This listing can be produced with, e.g., objdump -DSswx zabbix_server.
      1138369:20210331:021310.510 ================================
      1138415:20210331:021310.530 cannot read preprocessing service request
      1138412:20210331:021310.530 cannot read preprocessing service request
      1138390:20210331:021310.530 cannot read preprocessing service request
      1138383:20210331:021310.538 cannot read preprocessing service request
      1138376:20210331:021310.538 cannot read preprocessing service request
      1137285:20210331:021310.539 One child process died (PID:1138369,exitcode/signal:1). Exiting ...
      zabbix_server [1137285]: Error waiting for process with PID 1138369: [10] No child processes
      1137285:20210331:021311.936 syncing history data...
      1137285:20210331:021312.021 syncing history data... 100.000000%
      1137285:20210331:021312.021 syncing history data done
      1137285:20210331:021312.021 syncing trend data...
      1137285:20210331:021321.709 syncing trend data done
      1137285:20210331:021321.827 Zabbix Server stopped. Zabbix 5.0.9 (revision 4d07aaafe2).
      

      The DB is Oracle 19c, schema of 5.0.2 used

      Attachments

        Activity

          People

            zabbix.dev Zabbix Development Team
            agavrilovs Aleksandrs Petrovs-Gavrilovs
            Votes:
            0 Vote for this issue
            Watchers:
            7 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved: