Uploaded image for project: 'ZABBIX BUGS AND ISSUES'
  1. ZABBIX BUGS AND ISSUES
  2. ZBX-16434

Zabbix server 3.4 (CentOS 7.6) segfaulting after several minutes

XMLWordPrintable

    • Icon: Problem report Problem report
    • Resolution: Cannot Reproduce
    • Icon: Critical Critical
    • None
    • 3.0.28
    • Server (S)
    • None
    • CentOS Linux release 7.6.1810 (Core)
      zabbix-server-pgsql-3.4.15-1.el7.x86_64

      System was part of a 2-node cluster with pacemaker but had not been active in some time.  CentOS patches are up to date including kernel and glibc.  It's possible the active node (now failed) was modified before it failed.  Old server starts but dies with SEGV after a minute or two. Attaching objdump and debugging server log.

       

      23382:20190729:142853.253 Got signal [signal:11(SIGSEGV),reason:1,refaddr:(nil)]. Crashing ...
       23382:20190729:142853.253 ====== Fatal information: ======
       23382:20190729:142853.253 Program counter: 0x7f1410b84531
       23382:20190729:142853.253 === Registers: ===
       23382:20190729:142853.253 r8 = 0 = 0 = 0
       23382:20190729:142853.253 r9 = 1 = 1 = 1
       23382:20190729:142853.253 r10 = 2 = 2 = 2
       23382:20190729:142853.253 r11 = 0 = 0 = 0
       23382:20190729:142853.254 r12 = 55e04da2f41a = 94421863560218 = 94421863560218
       23382:20190729:142853.254 r13 = f = 15 = 15
       23382:20190729:142853.254 r14 = 0 = 0 = 0
       23382:20190729:142853.254 r15 = 0 = 0 = 0
       23382:20190729:142853.254 rdi = 0 = 0 = 0
       23382:20190729:142853.254 rsi = e1 = 225 = 225
       23382:20190729:142853.254 rbp = 7ffccb51b2e0 = 140723719615200 = 140723719615200
       23382:20190729:142853.254 rbx = 0 = 0 = 0
       23382:20190729:142853.254 rdx = 0 = 0 = 0
       23382:20190729:142853.254 rax = 0 = 0 = 0
       23382:20190729:142853.254 rcx = 0 = 0 = 0
       23382:20190729:142853.254 rsp = 7ffccb51b288 = 140723719615112 = 140723719615112
       23382:20190729:142853.254 rip = 7f1410b84531 = 139724156585265 = 139724156585265
       23382:20190729:142853.254 efl = 10283 = 66179 = 66179
       23382:20190729:142853.254 csgsfs = 33 = 51 = 51
       23382:20190729:142853.254 err = 4 = 4 = 4
       23382:20190729:142853.254 trapno = e = 14 = 14
       23382:20190729:142853.254 oldmask = 0 = 0 = 0
       23382:20190729:142853.254 cr2 = 0 = 0 = 0
       23382:20190729:142853.254 === Backtrace: ===
       23382:20190729:142853.255 27: /usr/sbin/zabbix_server: trapper #15 [processing data](zbx_backtrace+0x35) [0x55e04d974699]
       23382:20190729:142853.255 26: /usr/sbin/zabbix_server: trapper #15 [processing data](zbx_log_fatal_info+0x113) [0x55e04d9748ae]
       23382:20190729:142853.255 25: /usr/sbin/zabbix_server: trapper #15 [processing data](+0xe9bc1) [0x55e04d974bc1]
       23382:20190729:142853.255 24: /lib64/libc.so.6(+0x36340) [0x7f1410b2e340]
       23382:20190729:142853.255 23: /lib64/libc.so.6(+0x8c531) [0x7f1410b84531]
       23382:20190729:142853.255 22: /lib64/libc.so.6(__strdup+0xe) [0x7f1410b8423e]
       23382:20190729:142853.255 21: /usr/sbin/zabbix_server: trapper #15 [processing data](zbx_strdup2+0x4f) [0x55e04d9889be]
       23382:20190729:142853.255 20: /usr/sbin/zabbix_server: trapper #15 [processing data](+0x172b17) [0x55e04d9fdb17]
       23382:20190729:142853.255 19: /usr/sbin/zabbix_server: trapper #15 [processing data](DBcopy_template_items+0x6e) [0x55e04da00ffd]
       23382:20190729:142853.255 18: /usr/sbin/zabbix_server: trapper #15 [processing data](DBcopy_template_elements+0x32a) [0x55e04d9caf27]
       23382:20190729:142853.255 17: /usr/sbin/zabbix_server: trapper #15 [processing data](op_template_add+0xae) [0x55e04d8c5131]
       23382:20190729:142853.256 16: /usr/sbin/zabbix_server: trapper #15 [processing data](+0x3736b) [0x55e04d8c236b]
       23382:20190729:142853.256 15: /usr/sbin/zabbix_server: trapper #15 [processing data](process_actions+0x243) [0x55e04d8c29b2]
       23382:20190729:142853.256 14: /usr/sbin/zabbix_server: trapper #15 [processing data](+0x3da76) [0x55e04d8c8a76]
       23382:20190729:142853.256 13: /usr/sbin/zabbix_server: trapper #15 [processing data](zbx_process_events+0x229) [0x55e04d8c9cad]
       23382:20190729:142853.256 12: /usr/sbin/zabbix_server: trapper #15 [processing data](DBregister_host_flush+0x465) [0x55e04d9cfac9]
       23382:20190729:142853.256 11: /usr/sbin/zabbix_server: trapper #15 [processing data](DBregister_host+0x69) [0x55e04d9cefe7]
       23382:20190729:142853.256 10: /usr/sbin/zabbix_server: trapper #15 [processing data](+0x6ebbb) [0x55e04d8f9bbb]
       23382:20190729:142853.256 9: /usr/sbin/zabbix_server: trapper #15 [processing data](send_list_of_active_checks_json+0x296) [0x55e04d8fa73a]
       23382:20190729:142853.256 8: /usr/sbin/zabbix_server: trapper #15 [processing data](+0x6bc4e) [0x55e04d8f6c4e]
       23382:20190729:142853.256 7: /usr/sbin/zabbix_server: trapper #15 [processing data](+0x6c18a) [0x55e04d8f718a]
       23382:20190729:142853.256 6: /usr/sbin/zabbix_server: trapper #15 [processing data](trapper_thread+0x237) [0x55e04d8f73c3]
       23382:20190729:142853.256 5: /usr/sbin/zabbix_server: trapper #15 [processing data](zbx_thread_start+0x37) [0x55e04d981c2d]
       23382:20190729:142853.256 4: /usr/sbin/zabbix_server: trapper #15 [processing data](MAIN_ZABBIX_ENTRY+0x911) [0x55e04d8cb69e]
       23382:20190729:142853.256 3: /usr/sbin/zabbix_server: trapper #15 [processing data](daemon_start+0x325) [0x55e04d973ee6]
       23382:20190729:142853.256 2: /usr/sbin/zabbix_server: trapper #15 [processing data](main+0x312) [0x55e04d8cad8b]
       23382:20190729:142853.256 1: /lib64/libc.so.6(__libc_start_main+0xf5) [0x7f1410b1a495]
       23382:20190729:142853.256 0: /usr/sbin/zabbix_server: trapper #15 [processing data](+0x344a9) [0x55e04d8bf4a9]
       23382:20190729:142853.256 === Memory map: ===
       23382:20190729:142853.256 55e04d88b000-55e04da96000 r-xp 00000000 ca:01 86707 /usr/sbin/zabbix_server_pgsql
       23382:20190729:142853.257 55e04dc95000-55e04dcfd000 r--p 0020a000 ca:01 86707 /usr/sbin/zabbix_server_pgsql
       23382:20190729:142853.257 55e04dcfd000-55e04dd06000 rw-p 00272000 ca:01 86707 /usr/sbin/zabbix_server_pgsql
       23382:20190729:142853.257 55e04dd06000-55e04dd0f000 rw-p 00000000 00:00 0 
       23382:20190729:142853.257 55e04f1b5000-55e04f1d6000 rw-p 00000000 00:00 0 [heap]
       23382:20190729:142853.257 55e04f1d6000-55e04f385000 rw-p 00000000 00:00 0 [heap]
       23382:20190729:142853.257 7f13f5d2c000-7f13f5d3b000 r-xp 00000000 ca:01 91754 /usr/lib64/libbz2.so.1.0.6
       23382:20190729:142853.257 7f13f5d3b000-7f13f5f3a000 ---p 0000f000 ca:01 91754 /usr/lib64/libbz2.so.1.0.6
       23382:20190729:142853.257 7f13f5f3a000-7f13f5f3b000 r--p 0000e000 ca:01 91754 /usr/lib64/libbz2.so.1.0.6
       23382:20190729:142853.257 7f13f5f3b000-7f13f5f3c000 rw-p 0000f000 ca:01 91754 /usr/lib64/libbz2.so.1.0.6
       23382:20190729:142853.257 7f13f5f3c000-7f13f5f53000 r-xp 00000000 ca:01 18011 /usr/lib64/libelf-0.172.so
       23382:20190729:142853.257 7f13f5f53000-7f13f6152000 ---p 00017000 ca:01 18011 /usr/lib64/libelf-0.172.so
       23382:20190729:142853.257 7f13f6152000-7f13f6153000 r--p 00016000 ca:01 18011 /usr/lib64/libelf-0.172.so
       23382:20190729:142853.257 7f13f6153000-7f13f6154000 rw-p 00017000 ca:01 18011 /usr/lib64/libelf-0.172.so
       23382:20190729:142853.257 7f13f6154000-7f13f6158000 r-xp 00000000 ca:01 18050 /usr/lib64/libattr.so.1.1.0
       23382:20190729:142853.257 7f13f6158000-7f13f6357000 ---p 00004000 ca:01 18050 /usr/lib64/libattr.so.1.1.0
       23382:20190729:142853.257 7f13f6357000-7f13f6358000 r--p 00003000 ca:01 18050 /usr/lib64/libattr.so.1.1.0
       23382:20190729:142853.257 7f13f6358000-7f13f6359000 rw-p 00004000 ca:01 18050 /usr/lib64/libattr.so.1.1.0
       23382:20190729:142853.257 7f13f6359000-7f13f636e000 r-xp 00000000 ca:01 727 /usr/lib64/libgcc_s-4.8.5-20150702.so.1
       23382:20190729:142853.257 7f13f636e000-7f13f656d000 ---p 00015000 ca:01 727 /usr/lib64/libgcc_s-4.8.5-20150702.so.1
       23382:20190729:142853.257 7f13f656d000-7f13f656e000 r--p 00014000 ca:01 727 /usr/lib64/libgcc_s-4.8.5-20150702.so.1
       23382:20190729:142853.257 7f13f656e000-7f13f656f000 rw-p 00015000 ca:01 727 /usr/lib64/libgcc_s-4.8.5-20150702.so.1
       23382:20190729:142853.257 7f13f656f000-7f13f65bb000 r-xp 00000000 ca:01 7222 /usr/lib64/libdw-0.172.so
       23382:20190729:142853.257 7f13f65bb000-7f13f67bb000 ---p 0004c000 ca:01 7222 /usr/lib64/libdw-0.172.so
       23382:20190729:142853.257 7f13f67bb000-7f13f67bd000 r--p 0004c000 ca:01 7222 /usr/lib64/libdw-0.172.so
       23382:20190729:142853.257 7f13f67bd000-7f13f67be000 rw-p 0004e000 ca:01 7222 /usr/lib64/libdw-0.172.so
       23382:20190729:142853.257 7f13f67be000-7f13f67c2000 r-xp 00000000 ca:01 88258 /usr/lib64/libcap.so.2.22
       23382:20190729:142853.257 7f13f67c2000-7f13f69c1000 ---p 00004000 ca:01 88258 /usr/lib64/libcap.so.2.22
       23382:20190729:142853.258 7f13f69c1000-7f13f69c2000 r--p 00003000 ca:01 88258 /usr/lib64/libcap.so.2.22
       23382:20190729:142853.258 7f13f69c2000-7f13f69c3000 rw-p 00004000 ca:01 88258 /usr/lib64/libcap.so.2.22
       23382:20190729:142853.258 7f13f69c3000-7f13f69d5000 r-xp 00000000 ca:01 181 /usr/lib64/libnss_myhostname.so.2
       23382:20190729:142853.258 7f13f69d5000-7f13f6bd4000 ---p 00012000 ca:01 181 /usr/lib64/libnss_myhostname.so.2
       23382:20190729:142853.258 7f13f6bd4000-7f13f6bd7000 r--p 00011000 ca:01 181 /usr/lib64/libnss_myhostname.so.2
       23382:20190729:142853.258 7f13f6bd7000-7f13f6bd8000 rw-p 00014000 ca:01 181 /usr/lib64/libnss_myhostname.so.2
       23382:20190729:142853.258 7f13f6bd8000-7f13f6bdd000 r-xp 00000000 ca:01 22616 /usr/lib64/libnss_dns-2.17.so
       23382:20190729:142853.258 7f13f6bdd000-7f13f6ddd000 ---p 00005000 ca:01 22616 /usr/lib64/libnss_dns-2.17.so
       23382:20190729:142853.258 7f13f6ddd000-7f13f6dde000 r--p 00005000 ca:01 22616 /usr/lib64/libnss_dns-2.17.so
       23382:20190729:142853.258 7f13f6dde000-7f13f6ddf000 rw-p 00006000 ca:01 22616 /usr/lib64/libnss_dns-2.17.so
       23382:20190729:142853.258 7f13f6ddf000-7f13f75df000 rw-s 00000000 00:04 5767175 /SYSV00000000 (deleted)
       23382:20190729:142853.258 7f13f75df000-7f13f8913000 rw-s 00000000 00:04 5701637 /SYSV00000000 (deleted)
       23382:20190729:142853.258 7f13f8913000-7f13ff5e0000 rw-s 00000000 00:04 5668868 /SYSV00000000 (deleted)
       23382:20190729:142853.258 7f13ff5e0000-7f14035e0000 rw-s 00000000 00:04 5636099 /SYSV00000000 (deleted)
       23382:20190729:142853.258 7f14035e0000-7f14039e0000 rw-s 00000000 00:04 5603330 /SYSV00000000 (deleted)
       23382:20190729:142853.258 7f14039e0000-7f140b9e0000 rw-s 00000000 00:04 5570561 /SYSV00000000 (deleted)
       23382:20190729:142853.258 7f140b9e0000-7f140b9ec000 r-xp 00000000 ca:01 22619 /usr/lib64/libnss_files-2.17.so
       23382:20190729:142853.258 7f140b9ec000-7f140bbeb000 ---p 0000c000 ca:01 22619 /usr/lib64/libnss_files-2.17.so
       23382:20190729:142853.258 7f140bbeb000-7f140bbec000 r--p 0000b000 ca:01 22619 /usr/lib64/libnss_files-2.17.so
       23382:20190729:142853.258 7f140bbec000-7f140bbed000 rw-p 0000c000 ca:01 22619 /usr/lib64/libnss_files-2.17.so
       23382:20190729:142853.258 7f140bbed000-7f140bbf3000 rw-p 00000000 00:00 0 
       23382:20190729:142853.258 7f140bbf3000-7f140bbf5000 r-xp 00000000 ca:01 74802 /usr/lib64/libfreebl3.so
       23382:20190729:142853.258 7f140bbf5000-7f140bdf4000 ---p 00002000 ca:01 74802 /usr/lib64/libfreebl3.so
       23382:20190729:142853.258 7f140bdf4000-7f140bdf5000 r--p 00001000 ca:01 74802 /usr/lib64/libfreebl3.so
       23382:20190729:142853.258 7f140bdf5000-7f140bdf6000 rw-p 00002000 ca:01 74802 /usr/lib64/libfreebl3.so
       23382:20190729:142853.259 7f140bdf6000-7f140bdfd000 r-xp 00000000 ca:01 91780 /usr/lib64/libffi.so.6.0.1
       23382:20190729:142853.259 7f140bdfd000-7f140bffc000 ---p 00007000 ca:01 91780 /usr/lib64/libffi.so.6.0.1
       23382:20190729:142853.259 7f140bffc000-7f140bffd000 r--p 00006000 ca:01 91780 /usr/lib64/libffi.so.6.0.1
       23382:20190729:142853.259 7f140bffd000-7f140bffe000 rw-p 00007000 ca:01 91780 /usr/lib64/libffi.so.6.0.1
       23382:20190729:142853.259 7f140bffe000-7f140c022000 r-xp 00000000 ca:01 17884 /usr/lib64/libselinux.so.1
       23382:20190729:142853.259 7f140c022000-7f140c221000 ---p 00024000 ca:01 17884 /usr/lib64/libselinux.so.1
       23382:20190729:142853.259 7f140c221000-7f140c222000 r--p 00023000 ca:01 17884 /usr/lib64/libselinux.so.1
       23382:20190729:142853.259 7f140c222000-7f140c223000 rw-p 00024000 ca:01 17884 /usr/lib64/libselinux.so.1
       23382:20190729:142853.259 7f140c223000-7f140c225000 rw-p 00000000 00:00 0 
       23382:20190729:142853.259 7f140c225000-7f140c22c000 r-xp 00000000 ca:01 40202 /usr/lib64/librt-2.17.so
       23382:20190729:142853.259 7f140c22c000-7f140c42b000 ---p 00007000 ca:01 40202 /usr/lib64/librt-2.17.so
       23382:20190729:142853.259 7f140c42b000-7f140c42c000 r--p 00006000 ca:01 40202 /usr/lib64/librt-2.17.so
       23382:20190729:142853.259 7f140c42c000-7f140c42d000 rw-p 00007000 ca:01 40202 /usr/lib64/librt-2.17.so
       23382:20190729:142853.259 7f140c42d000-7f140c435000 r-xp 00000000 ca:01 22600 /usr/lib64/libcrypt-2.17.so
       23382:20190729:142853.259 7f140c435000-7f140c634000 ---p 00008000 ca:01 22600 /usr/lib64/libcrypt-2.17.so
       23382:20190729:142853.259 7f140c634000-7f140c635000 r--p 00007000 ca:01 22600 /usr/lib64/libcrypt-2.17.so
       23382:20190729:142853.259 7f140c635000-7f140c636000 rw-p 00008000 ca:01 22600 /usr/lib64/libcrypt-2.17.so
       23382:20190729:142853.259 7f140c636000-7f140c664000 rw-p 00000000 00:00 0 
       23382:20190729:142853.259 7f140c664000-7f140c6da000 r-xp 00000000 ca:01 105317 /usr/lib64/libgmp.so.10.2.0
       23382:20190729:142853.259 7f140c6da000-7f140c8d9000 ---p 00076000 ca:01 105317 /usr/lib64/libgmp.so.10.2.0
       23382:20190729:142853.259 7f140c8d9000-7f140c8db000 r--p 00075000 ca:01 105317 /usr/lib64/libgmp.so.10.2.0
       23382:20190729:142853.259 7f140c8db000-7f140c8dc000 rw-p 00077000 ca:01 105317 /usr/lib64/libgmp.so.10.2.0
       23382:20190729:142853.259 7f140c8dc000-7f140c902000 r-xp 00000000 ca:01 169756 /usr/lib64/libhogweed.so.2.5
       23382:20190729:142853.259 7f140c902000-7f140cb01000 ---p 00026000 ca:01 169756 /usr/lib64/libhogweed.so.2.5
       23382:20190729:142853.259 7f140cb01000-7f140cb02000 r--p 00025000 ca:01 169756 /usr/lib64/libhogweed.so.2.5
       23382:20190729:142853.259 7f140cb02000-7f140cb03000 rw-p 00026000 ca:01 169756 /usr/lib64/libhogweed.so.2.5
       23382:20190729:142853.259 7f140cb03000-7f140cb32000 r-xp 00000000 ca:01 169758 /usr/lib64/libnettle.so.4.7
       23382:20190729:142853.259 7f140cb32000-7f140cd32000 ---p 0002f000 ca:01 169758 /usr/lib64/libnettle.so.4.7
       23382:20190729:142853.259 7f140cd32000-7f140cd33000 r--p 0002f000 ca:01 169758 /usr/lib64/libnettle.so.4.7
       23382:20190729:142853.259 7f140cd33000-7f140cd34000 rw-p 00030000 ca:01 169758 /usr/lib64/libnettle.so.4.7
       23382:20190729:142853.260 7f140cd34000-7f140cd45000 r-xp 00000000 ca:01 75607 /usr/lib64/libtasn1.so.6.5.3
       23382:20190729:142853.260 7f140cd45000-7f140cf45000 ---p 00011000 ca:01 75607 /usr/lib64/libtasn1.so.6.5.3
       23382:20190729:142853.260 7f140cf45000-7f140cf46000 r--p 00011000 ca:01 75607 /usr/lib64/libtasn1.so.6.5.3
       23382:20190729:142853.260 7f140cf46000-7f140cf47000 rw-p 00012000 ca:01 75607 /usr/lib64/libtasn1.so.6.5.3
       23382:20190729:142853.260 7f140cf47000-7f140d061000 r-xp 00000000 ca:01 105145 /usr/lib64/libp11-kit.so.0.3.0
       23382:20190729:142853.260 7f140d061000-7f140d261000 ---p 0011a000 ca:01 105145 /usr/lib64/libp11-kit.so.0.3.0
       23382:20190729:142853.260 7f140d261000-7f140d26b000 r--p 0011a000 ca:01 105145 /usr/lib64/libp11-kit.so.0.3.0
       23382:20190729:142853.260 7f140d26b000-7f140d275000 rw-p 00124000 ca:01 105145 /usr/lib64/libp11-kit.so.0.3.0
       23382:20190729:142853.260 7f140d275000-7f140d276000 rw-p 00000000 00:00 0 
       23382:20190729:142853.260 7f140d276000-7f140d279000 r-xp 00000000 ca:01 105156 /usr/lib64/libkeyutils.so.1.5
       23382:20190729:142853.260 7f140d279000-7f140d478000 ---p 00003000 ca:01 105156 /usr/lib64/libkeyutils.so.1.5
       23382:20190729:142853.260 7f140d478000-7f140d479000 r--p 00002000 ca:01 105156 /usr/lib64/libkeyutils.so.1.5
       23382:20190729:142853.260 7f140d479000-7f140d47a000 rw-p 00003000 ca:01 105156 /usr/lib64/libkeyutils.so.1.5
       23382:20190729:142853.260 7f140d47a000-7f140d488000 r-xp 00000000 ca:01 105243 /usr/lib64/libkrb5support.so.0.1
       23382:20190729:142853.260 7f140d488000-7f140d688000 ---p 0000e000 ca:01 105243 /usr/lib64/libkrb5support.so.0.1
       23382:20190729:142853.260 7f140d688000-7f140d689000 r--p 0000e000 ca:01 105243 /usr/lib64/libkrb5support.so.0.1
      
      \{ . . . }
      
      
       23382:20190729:142853.269 ================================
       23382:20190729:142853.269 Please consider attaching a disassembly listing to your bug report.
       23382:20190729:142853.269 This listing can be produced with, e.g., objdump -DSswx zabbix_server.
       23382:20190729:142853.269 ================================
       23059:20190729:142853.270 One child process died (PID:23382,exitcode/signal:1). Exiting ...
       23059:20190729:142855.289 syncing history data...
       23059:20190729:142855.491 syncing history data done
       23059:20190729:142855.491 syncing trend data...
       23059:20190729:142911.795 syncing trend data done
       23059:20190729:142911.796 Zabbix Server stopped. Zabbix 3.4.15 (revision 86739).
      

       

       

            neogan Andrei Gushchin (Inactive)
            cloudops Cloud Ops
            Votes:
            0 Vote for this issue
            Watchers:
            5 Start watching this issue

              Created:
              Updated:
              Resolved: