[ZBX-8621] Got signal [signal:11(SIGSEGV),reason:128,refaddr:(nil)]. Crashing Created: 2014 Aug 15  Updated: 2017 May 30  Resolved: 2014 Sep 25

Status: Closed
Project: ZABBIX BUGS AND ISSUES
Component/s: Proxy (P)
Affects Version/s: 2.2.5
Fix Version/s: 2.2.7rc1, 2.4.2rc1, 2.5.0

Type: Incident report Priority: Blocker
Reporter: Sergey Assignee: Unassigned
Resolution: Fixed Votes: 0
Labels: crash
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified
Environment:
  • Oracle linux 6.5;
  • 3.8.13-35.3.5.el6uek.x86_64 #2 SMP Fri Aug 8 21:58:11 PDT 2014 x86_64 x86_64 x86_64 GNU/Linux;
  • php 5.5;
  • MariaDB 10.0.13;

Attachments: File crash_log.patch     File validation.patch     File validation.patch     File zabbix_proxy.objdump.gz    
Issue Links:
Duplicate
is duplicated by ZBX-8941 server crashes if snmp error index in... Closed

 Description   

Before upgrading to zabbix 2.2.5, we used 2.2.2 version and proxy was very stable and reliable.
After upgrade: several times a day, zabbix-proxy crashes with following notification in its logfile:

4675:20140815:000025.140 Got signal [signal:11(SIGSEGV),reason:128,refaddr:(nil)]. Crashing ...
  4675:20140815:000025.140 ====== Fatal information: ======
  4675:20140815:000025.140 Program counter: 0x4175fa
  4675:20140815:000025.140 === Registers: ===
  4675:20140815:000025.140 r8      =                0 =                    0 =                    0
  4675:20140815:000025.140 r9      =                0 =                    0 =                    0
  4675:20140815:000025.140 r10     =       341118fed0 =         223625150160 =         223625150160
  4675:20140815:000025.140 r11     =       3410e84350 =         223621956432 =         223621956432
  4675:20140815:000025.140 r12     =         53ed234b =           1408049995 =           1408049995
  4675:20140815:000025.140 r13     =          1f33110 =             32715024 =             32715024
  4675:20140815:000025.140 r14     =     7fff8fde9620 =      140735607117344 =      140735607117344
  4675:20140815:000025.140 r15     =     7fff8fe61800 =      140735607609344 =      140735607609344
  4675:20140815:000025.140 rdi     =          1f33110 =             32715024 =             32715024
  4675:20140815:000025.140 rsi     =           4943ab =              4801451 =              4801451
  4675:20140815:000025.141 rbp     =     7fff8fde9850 =      140735607117904 =      140735607117904
  4675:20140815:000025.141 rbx     =                0 =                    0 =                    0
  4675:20140815:000025.141 rdx     =        29f691a58 =          11264399960 =          11264399960
  4675:20140815:000025.141 rax     =     84746f76ef88 =      145635621138312 =      145635621138312
  4675:20140815:000025.141 rcx     =     8011ebc5b068 =      140814458400872 =      140814458400872
  4675:20140815:000025.141 rsp     =     7fff8fde9170 =      140735607116144 =      140735607116144
  4675:20140815:000025.141 rip     =           4175fa =              4290042 =              4290042
  4675:20140815:000025.141 efl     =            10202 =                66050 =                66050
  4675:20140815:000025.141 csgsfs  =               33 =                   51 =                   51
  4675:20140815:000025.141 err     =                0 =                    0 =                    0
  4675:20140815:000025.141 trapno  =                d =                   13 =                   13
  4675:20140815:000025.141 oldmask =                0 =                    0 =                    0
  4675:20140815:000025.141 cr2     =                0 =                    0 =                    0
  4675:20140815:000025.141 === Backtrace: ===
  4675:20140815:000025.141 13: /usr/local/sbin/zabbix_proxy: poller #15 [got 3 values in 0.157237 sec, getting values](print_fatal_info+0xad) [0x44d39d]
  4675:20140815:000025.141 12: /usr/local/sbin/zabbix_proxy: poller #15 [got 3 values in 0.157237 sec, getting values]() [0x44da7d]
  4675:20140815:000025.141 11: /lib64/libc.so.6() [0x3410e329a0]
  4675:20140815:000025.141 10: /usr/local/sbin/zabbix_proxy: poller #15 [got 3 values in 0.157237 sec, getting values]() [0x4175fa]
  4675:20140815:000025.141 9: /usr/local/sbin/zabbix_proxy: poller #15 [got 3 values in 0.157237 sec, getting values]() [0x4176ce]
  4675:20140815:000025.141 8: /usr/local/sbin/zabbix_proxy: poller #15 [got 3 values in 0.157237 sec, getting values]() [0x4179d2]
  4675:20140815:000025.141 7: /usr/local/sbin/zabbix_proxy: poller #15 [got 3 values in 0.157237 sec, getting values](get_values_snmp+0x68c) [0x41963c]
  4675:20140815:000025.141 6: /usr/local/sbin/zabbix_proxy: poller #15 [got 3 values in 0.157237 sec, getting values]() [0x41d083]
  4675:20140815:000025.141 5: /usr/local/sbin/zabbix_proxy: poller #15 [got 3 values in 0.157237 sec, getting values](main_poller_loop+0x82) [0x41da02]
  4675:20140815:000025.141 4: /usr/local/sbin/zabbix_proxy: poller #15 [got 3 values in 0.157237 sec, getting values](MAIN_ZABBIX_ENTRY+0x5d7) [0x410e67]
  4675:20140815:000025.141 3: /usr/local/sbin/zabbix_proxy: poller #15 [got 3 values in 0.157237 sec, getting values](daemon_start+0x1a3) [0x44ca93]
  4675:20140815:000025.141 2: /usr/local/sbin/zabbix_proxy: poller #15 [got 3 values in 0.157237 sec, getting values](main+0x2a8) [0x411498]
  4675:20140815:000025.141 1: /lib64/libc.so.6(__libc_start_main+0xfd) [0x3410e1ed1d]
  4675:20140815:000025.141 0: /usr/local/sbin/zabbix_proxy: poller #15 [got 3 values in 0.157237 sec, getting values]() [0x4101f9]
  4675:20140815:000025.141 === Memory map: ===
  4675:20140815:000025.141 00400000-00528000 r-xp 00000000 fc:01 791949                             /usr/local/sbin/zabbix_proxy
  4675:20140815:000025.141 00727000-0072b000 rw-p 00127000 fc:01 791949                             /usr/local/sbin/zabbix_proxy
  4675:20140815:000025.141 0072b000-00731000 rw-p 00000000 00:00 0
  4675:20140815:000025.142 01e23000-01e44000 rw-p 00000000 00:00 0                                  [heap]
  4675:20140815:000025.142 01e44000-01e80000 rw-p 00000000 00:00 0                                  [heap]
  4675:20140815:000025.142 01e80000-01f8f000 rw-p 00000000 00:00 0                                  [heap]
  4675:20140815:000025.142 3410600000-3410620000 r-xp 00000000 fc:01 524734                         /lib64/ld-2.12.so
  4675:20140815:000025.142 341081f000-3410820000 r--p 0001f000 fc:01 524734                         /lib64/ld-2.12.so
  4675:20140815:000025.142 3410820000-3410821000 rw-p 00020000 fc:01 524734                         /lib64/ld-2.12.so
  4675:20140815:000025.142 3410821000-3410822000 rw-p 00000000 00:00 0
  4675:20140815:000025.142 3410e00000-3410f8b000 r-xp 00000000 fc:01 524735                         /lib64/libc-2.12.so
  4675:20140815:000025.142 3410f8b000-341118a000 ---p 0018b000 fc:01 524735                         /lib64/libc-2.12.so
  4675:20140815:000025.142 341118a000-341118e000 r--p 0018a000 fc:01 524735                         /lib64/libc-2.12.so
  4675:20140815:000025.142 341118e000-341118f000 rw-p 0018e000 fc:01 524735                         /lib64/libc-2.12.so
  4675:20140815:000025.142 341118f000-3411194000 rw-p 00000000 00:00 0
  4675:20140815:000025.142 3411200000-3411217000 r-xp 00000000 fc:01 524740                         /lib64/libpthread-2.12.so
  4675:20140815:000025.142 3411217000-3411417000 ---p 00017000 fc:01 524740                         /lib64/libpthread-2.12.so
  4675:20140815:000025.142 3411417000-3411418000 r--p 00017000 fc:01 524740                         /lib64/libpthread-2.12.so
  4675:20140815:000025.142 3411418000-3411419000 rw-p 00018000 fc:01 524740                         /lib64/libpthread-2.12.so
  4675:20140815:000025.142 3411419000-341141d000 rw-p 00000000 00:00 0
  4675:20140815:000025.142 3411600000-3411602000 r-xp 00000000 fc:01 524742                         /lib64/libdl-2.12.so
  4675:20140815:000025.142 3411602000-3411802000 ---p 00002000 fc:01 524742                         /lib64/libdl-2.12.so
  4675:20140815:000025.142 3411802000-3411803000 r--p 00002000 fc:01 524742                         /lib64/libdl-2.12.so
  4675:20140815:000025.142 3411803000-3411804000 rw-p 00003000 fc:01 524742                         /lib64/libdl-2.12.so
  4675:20140815:000025.142 3411a00000-3411a15000 r-xp 00000000 fc:01 524737                         /lib64/libz.so.1.2.3
  4675:20140815:000025.142 3411a15000-3411c14000 ---p 00015000 fc:01 524737                         /lib64/libz.so.1.2.3
  4675:20140815:000025.142 3411c14000-3411c15000 r--p 00014000 fc:01 524737                         /lib64/libz.so.1.2.3
  4675:20140815:000025.142 3411c15000-3411c16000 rw-p 00015000 fc:01 524737                         /lib64/libz.so.1.2.3
  4675:20140815:000025.142 3411e00000-3411e07000 r-xp 00000000 fc:01 524747                         /lib64/librt-2.12.so
  4675:20140815:000025.142 3411e07000-3412006000 ---p 00007000 fc:01 524747                         /lib64/librt-2.12.so
  4675:20140815:000025.142 3412006000-3412007000 r--p 00006000 fc:01 524747                         /lib64/librt-2.12.so
  4675:20140815:000025.142 3412007000-3412008000 rw-p 00007000 fc:01 524747                         /lib64/librt-2.12.so
  4675:20140815:000025.142 3412200000-3412283000 r-xp 00000000 fc:01 524736                         /lib64/libm-2.12.so
  4675:20140815:000025.142 3412283000-3412482000 ---p 00083000 fc:01 524736                         /lib64/libm-2.12.so
  4675:20140815:000025.142 3412482000-3412483000 r--p 00082000 fc:01 524736                         /lib64/libm-2.12.so
  4675:20140815:000025.142 3412483000-3412484000 rw-p 00083000 fc:01 524736                         /lib64/libm-2.12.so
  4675:20140815:000025.142 3412600000-34126a0000 r-xp 00000000 fc:01 793594                         /usr/lib64/libnetsnmp.so.20.0.0
  4675:20140815:000025.142 34126a0000-341289f000 ---p 000a0000 fc:01 793594                         /usr/lib64/libnetsnmp.so.20.0.0
  4675:20140815:000025.142 341289f000-34128a2000 r--p 0009f000 fc:01 793594                         /usr/lib64/libnetsnmp.so.20.0.0
  4675:20140815:000025.142 34128a2000-34128a4000 rw-p 000a2000 fc:01 793594                         /usr/lib64/libnetsnmp.so.20.0.0
  4675:20140815:000025.142 34128a4000-34128da000 rw-p 00000000 00:00 0
  4675:20140815:000025.142 3412a00000-3412a16000 r-xp 00000000 fc:01 524750                         /lib64/libresolv-2.12.so
  4675:20140815:000025.142 3412a16000-3412c16000 ---p 00016000 fc:01 524750                         /lib64/libresolv-2.12.so
  4675:20140815:000025.142 3412c16000-3412c17000 r--p 00016000 fc:01 524750                         /lib64/libresolv-2.12.so
  4675:20140815:000025.142 3412c17000-3412c18000 rw-p 00017000 fc:01 524750                         /lib64/libresolv-2.12.so
  4675:20140815:000025.142 3412c18000-3412c1a000 rw-p 00000000 00:00 0
  4675:20140815:000025.142 3412e00000-3412ee8000 r-xp 00000000 fc:01 812365                         /usr/lib64/libOpenIPMI.so.0.0.5
  4675:20140815:000025.142 3412ee8000-34130e8000 ---p 000e8000 fc:01 812365                         /usr/lib64/libOpenIPMI.so.0.0.5
  4675:20140815:000025.142 34130e8000-3413104000 rw-p 000e8000 fc:01 812365                         /usr/lib64/libOpenIPMI.so.0.0.5
  4675:20140815:000025.142 3413104000-3413108000 rw-p 00000000 00:00 0
  4675:20140815:000025.142 3413200000-341321d000 r-xp 00000000 fc:01 524423                         /lib64/libtinfo.so.5.7
  4675:20140815:000025.143 341321d000-341341d000 ---p 0001d000 fc:01 524423                         /lib64/libtinfo.so.5.7
  4675:20140815:000025.143 341341d000-3413421000 rw-p 0001d000 fc:01 524423                         /lib64/libtinfo.so.5.7
  4675:20140815:000025.143 3413600000-3413605000 r-xp 00000000 fc:01 790939                         /usr/lib64/libOpenIPMIposix.so.0.0.1
  4675:20140815:000025.143 3413605000-3413804000 ---p 00005000 fc:01 790939                         /usr/lib64/libOpenIPMIposix.so.0.0.1
  4675:20140815:000025.143 3413804000-3413805000 rw-p 00004000 fc:01 790939                         /usr/lib64/libOpenIPMIposix.so.0.0.1
  4675:20140815:000025.143 3413a00000-3413a06000 r-xp 00000000 fc:01 812353                         /usr/lib64/libgdbm.so.2.0.0
  4675:20140815:000025.143 3413a06000-3413c05000 ---p 00006000 fc:01 812353                         /usr/lib64/libgdbm.so.2.0.0
  4675:20140815:000025.143 3413c05000-3413c06000 rw-p 00005000 fc:01 812353                         /usr/lib64/libgdbm.so.2.0.0
  4675:20140815:000025.143 3414200000-34143b5000 r-xp 00000000 fc:01 800426                         /usr/lib64/libcrypto.so.1.0.1e
  4675:20140815:000025.143 34143b5000-34145b5000 ---p 001b5000 fc:01 800426                         /usr/lib64/libcrypto.so.1.0.1e
  4675:20140815:000025.143 34145b5000-34145d0000 r--p 001b5000 fc:01 800426                         /usr/lib64/libcrypto.so.1.0.1e
  4675:20140815:000025.143 34145d0000-34145dc000 rw-p 001d0000 fc:01 800426                         /usr/lib64/libcrypto.so.1.0.1e
  4675:20140815:000025.143 34145dc000-34145e0000 rw-p 00000000 00:00 0
  4675:20140815:000025.143 3414600000-3414603000 r-xp 00000000 fc:01 524754                         /lib64/libcom_err.so.2.1
  4675:20140815:000025.143 3414603000-3414802000 ---p 00003000 fc:01 524754                         /lib64/libcom_err.so.2.1
  4675:20140815:000025.143 3414802000-3414803000 r--p 00002000 fc:01 524754                         /lib64/libcom_err.so.2.1
  4675:20140815:000025.143 3414803000-3414804000 rw-p 00003000 fc:01 524754                         /lib64/libcom_err.so.2.1
  4675:20140815:000025.143 3415200000-3415222000 r-xp 00000000 fc:01 524768                         /lib64/libncurses.so.5.7
  4675:20140815:000025.143 3415222000-3415421000 ---p 00022000 fc:01 524768                         /lib64/libncurses.so.5.7
  4675:20140815:000025.143 3415421000-3415422000 rw-p 00021000 fc:01 524768                         /lib64/libncurses.so.5.7
  4675:20140815:000025.143 3415e00000-3415e29000 r-xp 00000000 fc:01 524753                         /lib64/libk5crypto.so.3.1
  4675:20140815:000025.143 3415e29000-3416029000 ---p 00029000 fc:01 524753                         /lib64/libk5crypto.so.3.1
  4675:20140815:000025.143 3416029000-341602a000 r--p 00029000 fc:01 524753                         /lib64/libk5crypto.so.3.1
  4675:20140815:000025.143 341602a000-341602b000 rw-p 0002a000 fc:01 524753                         /lib64/libk5crypto.so.3.1
  4675:20140815:000025.143 341602b000-341602c000 rw-p 00000000 00:00 0
  4675:20140815:000025.143 3416200000-341620a000 r-xp 00000000 fc:01 524752                         /lib64/libkrb5support.so.0.1
  4675:20140815:000025.143 341620a000-3416409000 ---p 0000a000 fc:01 524752                         /lib64/libkrb5support.so.0.1
  4675:20140815:000025.143 3416409000-341640a000 r--p 00009000 fc:01 524752                         /lib64/libkrb5support.so.0.1
  4675:20140815:000025.143 341640a000-341640b000 rw-p 0000a000 fc:01 524752                         /lib64/libkrb5support.so.0.1
  4675:20140815:000025.143 3416600000-34166db000 r-xp 00000000 fc:01 524755                         /lib64/libkrb5.so.3.3
  4675:20140815:000025.143 34166db000-34168da000 ---p 000db000 fc:01 524755                         /lib64/libkrb5.so.3.3
  4675:20140815:000025.143 34168da000-34168e4000 r--p 000da000 fc:01 524755                         /lib64/libkrb5.so.3.3
  4675:20140815:000025.143 34168e4000-34168e6000 rw-p 000e4000 fc:01 524755                         /lib64/libkrb5.so.3.3
  4675:20140815:000025.143 3416e00000-3416e02000 r-xp 00000000 fc:01 524749                         /lib64/libkeyutils.so.1.3
  4675:20140815:000025.143 3416e02000-3417001000 ---p 00002000 fc:01 524749                         /lib64/libkeyutils.so.1.3
  4675:20140815:000025.143 3417001000-3417002000 r--p 00001000 fc:01 524749                         /lib64/libkeyutils.so.1.3
  4675:20140815:000025.143 3417002000-3417003000 rw-p 00002000 fc:01 524749                         /lib64/libkeyutils.so.1.3
  4675:20140815:000025.143 3417a00000-3417a41000 r-xp 00000000 fc:01 524756                         /lib64/libgssapi_krb5.so.2.2
  4675:20140815:000025.143 3417a41000-3417c41000 ---p 00041000 fc:01 524756                         /lib64/libgssapi_krb5.so.2.2
  4675:20140815:000025.143 3417c41000-3417c42000 r--p 00041000 fc:01 524756                         /lib64/libgssapi_krb5.so.2.2
  4675:20140815:000025.143 3417c42000-3417c44000 rw-p 00042000 fc:01 524756                         /lib64/libgssapi_krb5.so.2.2
  4675:20140815:000025.143 3417e00000-3417f48000 r-xp 00000000 fc:01 815151                         /usr/lib64/libxml2.so.2.7.6
  4675:20140815:000025.143 3417f48000-3418148000 ---p 00148000 fc:01 815151                         /usr/lib64/libxml2.so.2.7.6
  4675:20140815:000025.143 3418148000-3418151000 rw-p 00148000 fc:01 815151                         /usr/lib64/libxml2.so.2.7.6
  4675:20140815:000025.143 3418151000-3418153000 rw-p 00000000 00:00 0
  4675:20140815:000025.143 3418200000-3418261000 r-xp 00000000 fc:01 815307                         /usr/lib64/libssl.so.1.0.1e
  4675:20140815:000025.143 3418261000-3418461000 ---p 00061000 fc:01 815307                         /usr/lib64/libssl.so.1.0.1e
  4675:20140815:000025.144 3418461000-3418465000 r--p 00061000 fc:01 815307                         /usr/lib64/libssl.so.1.0.1e
  4675:20140815:000025.144 3418465000-341846c000 rw-p 00065000 fc:01 815307                         /usr/lib64/libssl.so.1.0.1e
  4675:20140815:000025.144 3419200000-3419227000 r-xp 00000000 fc:01 815318                         /usr/lib64/libssh2.so.1.0.1
  4675:20140815:000025.144 3419227000-3419426000 ---p 00027000 fc:01 815318                         /usr/lib64/libssh2.so.1.0.1
  4675:20140815:000025.144 3419426000-3419427000 rw-p 00026000 fc:01 815318                         /usr/lib64/libssh2.so.1.0.1
  4675:20140815:000025.144 3419427000-3419428000 rw-p 00000000 00:00 0
  4675:20140815:000025.144 7f6997820000-7f6997827000 r--s 00000000 fc:01 790187                     /usr/lib64/gconv/gconv-modules.cache
  4675:20140815:000025.144 7f6997827000-7f6997828000 rw-p 00000000 00:00 0
  4675:20140815:000025.144 7f6997828000-7f699d6b9000 r--p 00000000 fc:01 809926                     /usr/lib/locale/locale-archive
  4675:20140815:000025.144 7f699d6b9000-7f699e9ed000 rw-s 00000000 00:04 294916                     /SYSV73011769 (deleted)
  4675:20140815:000025.144 7f699e9ed000-7f69a56ba000 rw-s 00000000 00:04 262147                     /SYSV67011769 (deleted)
  4675:20140815:000025.144 7f69a56ba000-7f69a66ba000 rw-s 00000000 00:04 229378                     /SYSV78011769 (deleted)
  4675:20140815:000025.144 7f69a66ba000-7f69a6eba000 rw-s 00000000 00:04 196609                     /SYSV68011769 (deleted)
  4675:20140815:000025.144 7f69a6eba000-7f69a6ec6000 r-xp 00000000 fc:01 524319                     /lib64/libnss_files-2.12.so
  4675:20140815:000025.144 7f69a6ec6000-7f69a70c6000 ---p 0000c000 fc:01 524319                     /lib64/libnss_files-2.12.so
  4675:20140815:000025.144 7f69a70c6000-7f69a70c7000 r--p 0000c000 fc:01 524319                     /lib64/libnss_files-2.12.so
  4675:20140815:000025.144 7f69a70c7000-7f69a70c8000 rw-p 0000d000 fc:01 524319                     /lib64/libnss_files-2.12.so
  4675:20140815:000025.144 7f69a70c8000-7f69a70cd000 rw-p 00000000 00:00 0
  4675:20140815:000025.144 7f69a70cd000-7f69a70ea000 r-xp 00000000 fc:01 524751                     /lib64/libselinux.so.1
  4675:20140815:000025.144 7f69a70ea000-7f69a72e9000 ---p 0001d000 fc:01 524751                     /lib64/libselinux.so.1
  4675:20140815:000025.144 7f69a72e9000-7f69a72ea000 r--p 0001c000 fc:01 524751                     /lib64/libselinux.so.1
  4675:20140815:000025.144 7f69a72ea000-7f69a72eb000 rw-p 0001d000 fc:01 524751                     /lib64/libselinux.so.1
  4675:20140815:000025.144 7f69a72eb000-7f69a72ee000 rw-p 00000000 00:00 0
  4675:20140815:000025.144 7f69a72ee000-7f69a7304000 r-xp 00000000 fc:01 524739                     /lib64/libgcc_s-4.4.7-20120601.so.1
  4675:20140815:000025.144 7f69a7304000-7f69a7503000 ---p 00016000 fc:01 524739                     /lib64/libgcc_s-4.4.7-20120601.so.1
  4675:20140815:000025.144 7f69a7503000-7f69a7504000 rw-p 00015000 fc:01 524739                     /lib64/libgcc_s-4.4.7-20120601.so.1
  4675:20140815:000025.144 7f69a7504000-7f69a7506000 rw-p 00000000 00:00 0
  4675:20140815:000025.144 7f69a7506000-7f69a750d000 r-xp 00000000 fc:01 815143                     /usr/lib64/libOpenIPMIutils.so.0.0.1
  4675:20140815:000025.144 7f69a750d000-7f69a770d000 ---p 00007000 fc:01 815143                     /usr/lib64/libOpenIPMIutils.so.0.0.1
  4675:20140815:000025.144 7f69a770d000-7f69a770e000 rw-p 00007000 fc:01 815143                     /usr/lib64/libOpenIPMIutils.so.0.0.1
  4675:20140815:000025.144 7f69a770e000-7f69a770f000 rw-p 00000000 00:00 0
  4675:20140815:000025.144 7f69a770f000-7f69a77f7000 r-xp 00000000 fc:01 792072                     /usr/lib64/libstdc++.so.6.0.13
  4675:20140815:000025.144 7f69a77f7000-7f69a79f7000 ---p 000e8000 fc:01 792072                     /usr/lib64/libstdc++.so.6.0.13
  4675:20140815:000025.144 7f69a79f7000-7f69a79fe000 r--p 000e8000 fc:01 792072                     /usr/lib64/libstdc++.so.6.0.13
  4675:20140815:000025.144 7f69a79fe000-7f69a7a00000 rw-p 000ef000 fc:01 792072                     /usr/lib64/libstdc++.so.6.0.13
  4675:20140815:000025.144 7f69a7a00000-7f69a7a1a000 rw-p 00000000 00:00 0
  4675:20140815:000025.144 7f69a7a1a000-7f69a7d0a000 r-xp 00000000 fc:01 809955                     /usr/lib64/libmysqlclient.so.18.0.0
  4675:20140815:000025.144 7f69a7d0a000-7f69a7f0a000 ---p 002f0000 fc:01 809955                     /usr/lib64/libmysqlclient.so.18.0.0
  4675:20140815:000025.144 7f69a7f0a000-7f69a7f77000 rw-p 002f0000 fc:01 809955                     /usr/lib64/libmysqlclient.so.18.0.0
  4675:20140815:000025.144 7f69a7f77000-7f69a7f7c000 rw-p 00000000 00:00 0
  4675:20140815:000025.144 7f69a7f7c000-7f69a7f86000 rw-s 00000000 00:04 327685                     /SYSV53011769 (deleted)
  4675:20140815:000025.144 7f69a7f86000-7f69a7f87000 rw-p 00000000 00:00 0
  4675:20140815:000025.144 7f69a7f87000-7f69a7f88000 rw-p 00000000 00:00 0
  4675:20140815:000025.144 7fff8fdd7000-7fff8fe64000 rw-p 00000000 00:00 0                          [stack]
  4675:20140815:000025.144 7fff8fffe000-7fff90000000 r-xp 00000000 00:00 0                          [vdso]
  4675:20140815:000025.144 ffffffffff600000-ffffffffff601000 r-xp 00000000 00:00 0                  [vsyscall]
  4675:20140815:000025.144 ================================
  4675:20140815:000025.145 Please consider attaching a disassembly listing to your bug report.
  4675:20140815:000025.145 This listing can be produced with, e.g., objdump -DSswx zabbix_proxy.
  4675:20140815:000025.145 ================================
  4656:20140815:000025.147 One child process died (PID:4675,exitcode/signal:255). Exiting ...
  4656:20140815:000027.148 syncing history data...
  4656:20140815:000027.153 syncing history data done
  4656:20140815:000027.153 Zabbix Proxy stopped. Zabbix 2.2.5 (revision 47411).


 Comments   
Comment by Sergey [ 2014 Aug 15 ]

Additionally. We had a similar problem with a proxy version 2.2.5, which worked under the FreeBSD 8.

Comment by Aleksandrs Saveljevs [ 2014 Aug 15 ]

Could you please attach the disassembly of zabbix_proxy? As the log file suggests, it can be obtained using "objdump -DSswx zabbix_proxy". If you have DebugLevel=4 log, it might be helpful, too.

Comment by Sergey [ 2014 Aug 15 ]

Unfortunately we can't on debug level 4 now

Comment by Arturs Galapovs (Inactive) [ 2014 Aug 22 ]

Sergey,
we did not manage to reproduce this problem on our side. Could you please patch 2.2.5 with crash_log.patch and run produced custom proxy? You can find patch file in attachments.
You can patch with:

patch -p0 < crash_log.patch

command in Zabbix sources root directory.

These changes will log additional information only before actual crash, so you should not expect increased load on proxy side. This will help us to get closer to the actual problem.

Additionally could you please also increase LogFileSize (100 will be perfect) option in Zabbix proxy configuration file - additional logging could produce significant output into log file depending on your monitoring configurations. All custom log will be marked with --ZBX-8621-- and --ZBX-8621-end--

Thanks in advance!

Comment by Sergey [ 2014 Aug 22 ]

Done and waiting for a crash.

Comment by Arturs Galapovs (Inactive) [ 2014 Aug 26 ]

Hi Sergey,
Just a small note: proxy will not crash - it will log and exit normally if the problem is spotted. Just in case you are using some problem monitoring technics based on actual crash.

Comment by Sergey [ 2014 Aug 26 ]

Hi, Arturs
To date, no crashes are found, probably because there was a rollback zabbix server to 2.2.2 version. Rollback has been associated with the Incorrect time calculation in queue.

Comment by Arturs Galapovs (Inactive) [ 2014 Aug 26 ]

Sergey,
You are correct - your particular crash is associated with feature (bulk snmp request) that is present in 2.2.5 and is absent in 2.2.2.

Comment by Sergey [ 2014 Sep 08 ]

Hi. The crash happened again. Version of zabbix server - 2.2.2 and version of proxy srver - 2.2.5.
Add some part of the log file after crash:

 20503:20140906:034006.554 -----------ZBX-8621-----------
 20503:20140906:034006.554 zbx_snmp_get_values() parameters: ss=0x1157f00, items=0x7fff830cb390, oids=0x7fff830c2740, results=0x7fff83137f90, errcodes=0x7fff83139f90, query_and_ignore_type=(nil), num=3, level
 20503:20140906:034006.554 internal variables:
 i=4, j=1409963991, mapping_num=3
 20503:20140906:034006.554 mapping[]:[0]0       [1]1    [2]2    [3]49   [4]1409963991   [5]0    [6]494994       [7]0    [8]17421184     [9]0    [10]-2096358096 [11]32767       [12]-2096357872 [13]32767
 20503:20140906:034006.554 ---pdu response:---
 20503:20140906:034006.554 pdu:
 version=1, command=162, reqid=329481303, msgid=0, transid=199145, sessid=0, errstat=0, errindex=0, time=0, flags=256, securityModel=2, securityLevel=1, msgParseModel=0, transport_data=0x11426b0, transport_da
 20503:20140906:034006.554 pdu variables:
 20503:20140906:034006.554 this variable=0x115b6c0, next_variable=0x115bb40, name=18200304, name_length=11, type=2, val.data=0x115baf0, val_len=8, name_loc=0x115b6f0, buf=0x115baf0, data=(nil), index=0
 20503:20140906:034006.554 this variable=0x115bb40, next_variable=0x115bfc0, name=18201456, name_length=11, type=2, val.data=0x115bf70, val_len=8, name_loc=0x115bb70, buf=0x115bf70, data=(nil), index=0
 20503:20140906:034006.554 this variable=0x115bfc0, next_variable=0x116ea80, name=18202608, name_length=11, type=2, val.data=0x115c3f0, val_len=8, name_loc=0x115bff0, buf=0x115c3f0, data=(nil), index=0
 20503:20140906:034006.554 this variable=0x116ea80, next_variable=0x116ef00, name=18279088, name_length=11, type=2, val.data=0x116eeb0, val_len=8, name_loc=0x116eab0, buf=0x116eeb0, data=(nil), index=0
 20503:20140906:034006.554 this variable=0x116ef00, next_variable=0x116f380, name=18280240, name_length=11, type=2, val.data=0x116f330, val_len=8, name_loc=0x116ef30, buf=0x116f330, data=(nil), index=0
 20503:20140906:034006.554 this variable=0x116f380, next_variable=(nil), name=18281392, name_length=11, type=2, val.data=0x116f7b0, val_len=8, name_loc=0x116f3b0, buf=0x116f7b0, data=(nil), index=0
 20503:20140906:034006.554 -----------ZBX-8621-end-----------
Comment by Aleksandrs Saveljevs [ 2014 Sep 18 ]

Sergey, thank you for the additional information.

It seems that the scenario is approximately as follows. Proxy sent a request with 6 variables. The request timed out, so the proxy retried with 3 variables (as per https://www.zabbix.com/documentation/2.2/manual/config/items/itemtypes/snmp#internal_workings_of_bulk_processing). It then received a response with 6 variables, and since it only expected 3 variables, went out of bounds and crashed.

Now, the question is why it received a response with 6 variables and how could it mistake that for a response to the request with 3 variables, considering that each SNMP request has an ID.

We should of course add a check to server and proxy to validate that the response indeed contains the variables that we expect, but it would be nice to see the real traffic that leads to that.

Sergey, could you please do some tcpdump (unfortunately, the attached logs do not say which device you should sniff traffic to) and attach the *.pcap file with data for one minute before the crash?

Also, which version of Net-SNMP library are you using?

Comment by Sergey [ 2014 Sep 18 ]

Hi, Alexandrs. we are using net-snmp-libs.x86_64 (1: 5.5-49.0.1.el6_5.1) library. the same configuration used on other proxy without any crashes. Unfortunately, I can't capture traffic because this is so aperiodic event and we've a lot of data that pass through proxy. Can I help other way?

Comment by Aleksandrs Saveljevs [ 2014 Sep 18 ]

The debug dump that you posted three comments earlier is cut to a certain width, so some lines are shorter than they actual were in the real log file. However, in the real log file in one of these lines you should see the IP address of the SNMP device that misbehaves. This is probably only one device that acts in a strange way and, if you capture traffic for that device only, it should be a relatively little amount of traffic. Do you still have that original debug dump? If you do, could you please tell us the model of this device?

Comment by Sergey [ 2014 Sep 19 ]

Full version of the dump. Possible IP device displays at 4 level debug (we are using 3)? I don't find IP either before or after that.

20503:20140906:034006.554 ----------ZBX-8621----------
20503:20140906:034006.554 zbx_snmp_get_values() parameters: ss=0x1157f00, items=0x7fff830cb390, oids=0x7fff830c2740, results=0x7fff83137f90, errcodes=0x7fff83139f90, query_and_ignore_type=(nil),
num=3, level=1, error=0x7fff830ca7c0 max_error_len=2048, max_succeed=0x7fff830cb26c, min_fail=0x7fff830cb268
20503:20140906:034006.554 internal variables:
i=4, j=1409963991, mapping_num=3
20503:20140906:034006.554 mapping[]:[0]0 [1]1 [2]2 [3]49 [4]1409963991 [5]0 [6]494994 [7]0 [8]17421184 [9]0 [10]-2096358096 [11]32767
[12]-2096357872 [13]32767 [14]-2096358096 [15]32767 [16]-2096357872 [17]32767 [18]308558523 [19]52 [20]-2096358096 [21]32767 [22]308381748
[23]52 [24]17961312 [25]0 [26]-2096357688 [27]0 [28]0 [29]0 [30]0 [31]0 [32]64 [33]0 [34]0 [35]0 [36]0 [37]0 [38]18187808 [39]0 [40]0 [41]0
[42]0 [43]0 [44]0 [45]0 [46]18189248 [47]0 [48]18185984 [49]0 [50]17961312 [51]0 [52]16 [53]0 [54]308552876 [55]52 [56]1409964006 [57]0 [58]510039
[59]0 [60]17961312 [61]0 [62]-2096357712 [63]32767 [64]18185984 [65]0 [66]-2096357688 [67]32767 [68]-2096357684 [69]32767 [70]308553176 [71]52
[72]-2096357872 [73]32767 [74]308383623 [75]52 [76]66 [77]32767 [78]0 [79]0 [80]0 [81]0 [82]-2096355688 [83]32767 [84]0 [85]0 [86]0
[87]0 [88]0 [89]0 [90]0 [91]0 [92]4814527 [93]0 [94]2 [95]0 [96]-1 [97]-1 [98]6 [99]0 [100]0 [101]0 [102]0 [103]0 [104]0 [105]0 [106]0 [107]0
[108]0 [109]0 [110]0 [111]0 [112]0 [113]0 [114]0 [115]0 [116]0 [117]2 [118]329481302 [119]0 [120]0 [121]0 [122]18195328 [123]0 [124]0 [125]0 [126]0 [127]0

20503:20140906:034006.554 --pdu response:--
20503:20140906:034006.554 pdu:
version=1, command=162, reqid=329481303, msgid=0, transid=199145, sessid=0, errstat=0, errindex=0, time=0, flags=256, securityModel=2, securityLevel=1, msgParseModel=0,
transport_data=0x11426b0, transport_data_length=24, tDomain=0x34128a34c0, tDomainLen=7, variables=0x115b6c0, community=0x115b6a0, community_len=8, enterprise=(nil), enterprise_length=0,
trap_type=0, specific_type=0, agent_addr=0.0.0.0, contextEngineID=(nil), contextEngineIDLen=0, contextName=(nil), contextNameLen=0, securityEngineID=(nil), securityEngineIDLen=0,
securityName=(nil), securityNameLen=0, priority=0, range_subid=0, securityStateRef=(nil)
20503:20140906:034006.554 pdu variables:
20503:20140906:034006.554 this variable=0x115b6c0, next_variable=0x115bb40, name=18200304, name_length=11, type=2, val.data=0x115baf0, val_len=8, name_loc=0x115b6f0, buf=0x115baf0, data=(nil), index=0
20503:20140906:034006.554 this variable=0x115bb40, next_variable=0x115bfc0, name=18201456, name_length=11, type=2, val.data=0x115bf70, val_len=8, name_loc=0x115bb70, buf=0x115bf70, data=(nil), index=0
20503:20140906:034006.554 this variable=0x115bfc0, next_variable=0x116ea80, name=18202608, name_length=11, type=2, val.data=0x115c3f0, val_len=8, name_loc=0x115bff0, buf=0x115c3f0, data=(nil), index=0
20503:20140906:034006.554 this variable=0x116ea80, next_variable=0x116ef00, name=18279088, name_length=11, type=2, val.data=0x116eeb0, val_len=8, name_loc=0x116eab0, buf=0x116eeb0, data=(nil), index=0
20503:20140906:034006.554 this variable=0x116ef00, next_variable=0x116f380, name=18280240, name_length=11, type=2, val.data=0x116f330, val_len=8, name_loc=0x116ef30, buf=0x116f330, data=(nil), index=0
20503:20140906:034006.554 this variable=0x116f380, next_variable=(nil), name=18281392, name_length=11, type=2, val.data=0x116f7b0, val_len=8, name_loc=0x116f3b0, buf=0x116f7b0, data=(nil), index=0
20503:20140906:034006.554 ----------ZBX-8621-end----------

Comment by Aleksandrs Saveljevs [ 2014 Sep 25 ]

Ah, I expected to find the IP in "agent_addr=0.0.0.0", but it just says "0.0.0.0" instead.

Comment by Aleksandrs Saveljevs [ 2014 Sep 25 ]

Development branch svn://svn.zabbix.com/branches/dev/ZBX-8621 adds validation to the SNMP response. Previously, we assumed it was correct, but now we actually check that the variable bindings in the response are the same as in the request.

Comment by Aleksandrs Saveljevs [ 2014 Sep 25 ]

Sergey, I have attached the patch to this issue that introduces SNMP response validation for Zabbix 2.2.5 as "validation.patch". If you apply this patch, then on receiving a bad response the proxy will log the following line with DebugLevel=3 instead of crashing:

SNMP response from host "%s" contains too many variable bindings

This will give you the hostname that sends bad responses. If you could tcpdump SNMP traffic to this host and attach the bad traffic fragment here, that would be wonderful.

There is no need to run the patch that Arturs has provided previously.

Comment by Sergey [ 2014 Sep 25 ]

Hi, I'm successfully patched it and After few minutes we have got a crash.

Dump after crash:

  9104:20140925:162702.236 SNMP response from host "CS-121" does not contain variable bindings in the requested order
*** glibc detected *** /usr/local/sbin/zabbix_proxy: poller #29 [got 46 values in 1.495208 sec, getting values]: free(): invalid pointer: 0x0000000000496d78 ***
======= Backtrace: =========
/lib64/libc.so.6[0x3410e76166]
/usr/local/sbin/zabbix_proxy: poller #29 [got 46 values in 1.495208 sec, getting values](free_result+0x7e)[0x42cb1e]
/usr/local/sbin/zabbix_proxy: poller #29 [got 46 values in 1.495208 sec, getting values][0x41ceb2]
/usr/local/sbin/zabbix_proxy: poller #29 [got 46 values in 1.495208 sec, getting values](main_poller_loop+0x82)[0x41dc02]
/usr/local/sbin/zabbix_proxy: poller #29 [got 46 values in 1.495208 sec, getting values](MAIN_ZABBIX_ENTRY+0x5d7)[0x410e67]
/usr/local/sbin/zabbix_proxy: poller #29 [got 46 values in 1.495208 sec, getting values](daemon_start+0x1a3)[0x44cc93]
/usr/local/sbin/zabbix_proxy: poller #29 [got 46 values in 1.495208 sec, getting values](main+0x2a8)[0x411498]
/lib64/libc.so.6(__libc_start_main+0xfd)[0x3410e1ed1d]
/usr/local/sbin/zabbix_proxy: poller #29 [got 46 values in 1.495208 sec, getting values][0x4101f9]
======= Memory map: ========
00400000-00528000 r-xp 00000000 fc:01 793868                             /usr/local/sbin/zabbix_proxy
00727000-0072b000 rw-p 00127000 fc:01 793868                             /usr/local/sbin/zabbix_proxy
0072b000-00731000 rw-p 00000000 00:00 0
00866000-00887000 rw-p 00000000 00:00 0                                  [heap]
00887000-008c3000 rw-p 00000000 00:00 0                                  [heap]
008c3000-00990000 rw-p 00000000 00:00 0                                  [heap]
3410600000-3410620000 r-xp 00000000 fc:01 524734                         /lib64/ld-2.12.so
341081f000-3410820000 r--p 0001f000 fc:01 524734                         /lib64/ld-2.12.so
3410820000-3410821000 rw-p 00020000 fc:01 524734                         /lib64/ld-2.12.so
3410821000-3410822000 rw-p 00000000 00:00 0
3410e00000-3410f8b000 r-xp 00000000 fc:01 524735                         /lib64/libc-2.12.so
3410f8b000-341118a000 ---p 0018b000 fc:01 524735                         /lib64/libc-2.12.so
341118a000-341118e000 r--p 0018a000 fc:01 524735                         /lib64/libc-2.12.so
341118e000-341118f000 rw-p 0018e000 fc:01 524735                         /lib64/libc-2.12.so
341118f000-3411194000 rw-p 00000000 00:00 0
3411200000-3411217000 r-xp 00000000 fc:01 524740                         /lib64/libpthread-2.12.so
3411217000-3411417000 ---p 00017000 fc:01 524740                         /lib64/libpthread-2.12.so
3411417000-3411418000 r--p 00017000 fc:01 524740                         /lib64/libpthread-2.12.so
3411418000-3411419000 rw-p 00018000 fc:01 524740                         /lib64/libpthread-2.12.so
3411419000-341141d000 rw-p 00000000 00:00 0
3411600000-3411602000 r-xp 00000000 fc:01 524742                         /lib64/libdl-2.12.so
3411602000-3411802000 ---p 00002000 fc:01 524742                         /lib64/libdl-2.12.so
3411802000-3411803000 r--p 00002000 fc:01 524742                         /lib64/libdl-2.12.so
3411803000-3411804000 rw-p 00003000 fc:01 524742                         /lib64/libdl-2.12.so
3411a00000-3411a15000 r-xp 00000000 fc:01 524737                         /lib64/libz.so.1.2.3
3411a15000-3411c14000 ---p 00015000 fc:01 524737                         /lib64/libz.so.1.2.3
3411c14000-3411c15000 r--p 00014000 fc:01 524737                         /lib64/libz.so.1.2.3
3411c15000-3411c16000 rw-p 00015000 fc:01 524737                         /lib64/libz.so.1.2.3
3411e00000-3411e07000 r-xp 00000000 fc:01 524747                         /lib64/librt-2.12.so
3411e07000-3412006000 ---p 00007000 fc:01 524747                         /lib64/librt-2.12.so
3412006000-3412007000 r--p 00006000 fc:01 524747                         /lib64/librt-2.12.so
3412007000-3412008000 rw-p 00007000 fc:01 524747                         /lib64/librt-2.12.so
3412200000-3412283000 r-xp 00000000 fc:01 524736                         /lib64/libm-2.12.so
3412283000-3412482000 ---p 00083000 fc:01 524736                         /lib64/libm-2.12.so
3412482000-3412483000 r--p 00082000 fc:01 524736                         /lib64/libm-2.12.so
3412483000-3412484000 rw-p 00083000 fc:01 524736                         /lib64/libm-2.12.so
3412600000-34126a0000 r-xp 00000000 fc:01 793594                         /usr/lib64/libnetsnmp.so.20.0.0
34126a0000-341289f000 ---p 000a0000 fc:01 793594                         /usr/lib64/libnetsnmp.so.20.0.0
341289f000-34128a2000 r--p 0009f000 fc:01 793594                         /usr/lib64/libnetsnmp.so.20.0.0
34128a2000-34128a4000 rw-p 000a2000 fc:01 793594                         /usr/lib64/libnetsnmp.so.20.0.0
34128a4000-34128da000 rw-p 00000000 00:00 0
3412a00000-3412a16000 r-xp 00000000 fc:01 524750                         /lib64/libresolv-2.12.so
3412a16000-3412c16000 ---p 00016000 fc:01 524750                         /lib64/libresolv-2.12.so
3412c16000-3412c17000 r--p 00016000 fc:01 524750                         /lib64/libresolv-2.12.so
3412c17000-3412c18000 rw-p 00017000 fc:01 524750                         /lib64/libresolv-2.12.so
3412c18000-3412c1a000 rw-p 00000000 00:00 0
3412e00000-3412ee8000 r-xp 00000000 fc:01 812365                         /usr/lib64/libOpenIPMI.so.0.0.5
3412ee8000-34130e8000 ---p 000e8000 fc:01 812365                         /usr/lib64/libOpenIPMI.so.0.0.5
34130e8000-3413104000 rw-p 000e8000 fc:01 812365                         /usr/lib64/libOpenIPMI.so.0.0.5
3413104000-3413108000 rw-p 00000000 00:00 0
3413200000-341321d000 r-xp 00000000 fc:01 524423                         /lib64/libtinfo.so.5.7
341321d000-341341d000 ---p 0001d000 fc:01 524423                         /lib64/libtinfo.so.5.7
341341d000-3413421000 rw-p 0001d000 fc:01 524423                         /lib64/libtinfo.so.5.7
3413600000-3413605000 r-xp 00000000 fc:01 790939                         /usr/lib64/libOpenIPMIposix.so.0.0.1
3413605000-3413804000 ---p 00005000 fc:01 790939                         /usr/lib64/libOpenIPMIposix.so.0.0.1
3413804000-3413805000 rw-p 00004000 fc:01 790939                         /usr/lib64/libOpenIPMIposix.so.0.0.1
3413a00000-3413a06000 r-xp 00000000 fc:01 812353                         /usr/lib64/libgdbm.so.2.0.0
3413a06000-3413c05000 ---p 00006000 fc:01 812353                         /usr/lib64/libgdbm.so.2.0.0
3413c05000-3413c06000 rw-p 00005000 fc:01 812353                         /usr/lib64/libgdbm.so.2.0.0
3414200000-34143b5000 r-xp 00000000 fc:01 800426                         /usr/lib64/libcrypto.so.1.0.1e
34143b5000-34145b5000 ---p 001b5000 fc:01 800426                         /usr/lib64/libcrypto.so.1.0.1e
34145b5000-34145d0000 r--p 001b5000 fc:01 800426                         /usr/lib64/libcrypto.so.1.0.1e
34145d0000-34145dc000 rw-p 001d0000 fc:01 800426                         /usr/lib64/libcrypto.so.1.0.1e
34145dc000-34145e0000 rw-p 00000000 00:00 0
3414600000-3414603000 r-xp 00000000 fc:01 524754                         /lib64/libcom_err.so.2.1
3414603000-3414802000 ---p 00003000 fc:01 524754                         /lib64/libcom_err.so.2.1
3414802000-3414803000 r--p 00002000 fc:01 524754                         /lib64/libcom_err.so.2.1
3414803000-3414804000 rw-p 00003000 fc:01 524754                         /lib64/libcom_err.so.2.1
3415200000-3415222000 r-xp 00000000 fc:01 524768                         /lib64/libncurses.so.5.7
3415222000-3415421000 ---p 00022000 fc:01 524768                         /lib64/libncurses.so.5.7
3415421000-3415422000 rw-p 00021000 fc:01 524768                         /lib64/libncurses.so.5.7
3415e00000-3415e29000 r-xp 00000000 fc:01 524753                         /lib64/libk5crypto.so.3.1
3415e29000-3416029000 ---p 00029000 fc:01 524753                         /lib64/libk5crypto.so.3.1
3416029000-341602a000 r--p 00029000 fc:01 524753                         /lib64/libk5crypto.so.3.1
341602a000-341602b000 rw-p 0002a000 fc:01 524753                         /lib64/libk5crypto.so.3.1
341602b000-341602c000 rw-p 00000000 00:00 0
3416200000-341620a000 r-xp 00000000 fc:01 524752                         /lib64/libkrb5support.so.0.1
341620a000-3416409000 ---p 0000a000 fc:01 524752                         /lib64/libkrb5support.so.0.1
3416409000-341640a000 r--p 00009000 fc:01 524752                         /lib64/libkrb5support.so.0.1
341640a000-341640b000 rw-p 0000a000 fc:01 524752                         /lib64/libkrb5support.so.0.1
3416600000-34166db000 r-xp 00000000 fc:01 524755                         /lib64/libkrb5.so.3.3
34166db000-34168da000 ---p 000db000 fc:01 524755                         /lib64/libkrb5.so.3.3
34168da000-34168e4000 r--p 000da000 fc:01 524755                         /lib64/libkrb5.so.3.3
34168e4000-34168e6000 rw-p 000e4000 fc:01 524755                         /lib64/libkrb5.so.3.3
3416e00000-3416e02000 r-xp 00000000 fc:01 524749                         /lib64/libkeyutils.so.1.3
3416e02000-3417001000 ---p 00002000 fc:01 524749                         /lib64/libkeyutils.so.1.3
3417001000-3417002000 r--p 00001000 fc:01 524749                         /lib64/libkeyutils.so.1.3
3417002000-3417003000 rw-p 00002000 fc:01 524749                         /lib64/libkeyutils.so.1.3
3417a00000-3417a41000 r-xp 00000000 fc:01 524756                         /lib64/libgssapi_krb5.so.2.2
3417a41000-3417c41000 ---p 00041000 fc:01 524756                         /lib64/libgssapi_krb5.so.2.2
3417c41000-3417c42000 r--p 00041000 fc:01 524756                         /lib64/libgssapi_krb5.so.2.2
3417c42000-3417c44000 rw-p 00042000 fc:01 524756                         /lib64/libgssapi_krb5.so.2.2
3417e00000-3417f48000 r-xp 00000000 fc:01 815151                         /usr/lib64/libxml2.so.2.7.6
3417f48000-3418148000 ---p 00148000 fc:01 815151                         /usr/lib64/libxml2.so.2.7.6
3418148000-3418151000 rw-p 00148000 fc:01 815151                         /usr/lib64/libxml2.so.2.7.6
3418151000-3418153000 rw-p 00000000 00:00 0
3418200000-3418261000 r-xp 00000000 fc:01 815307                         /usr/lib64/libssl.so.1.0.1e
3418261000-3418461000 ---p 00061000 fc:01 815307                         /usr/lib64/libssl.so.1.0.1e
3418461000-3418465000 r--p 00061000 fc:01 815307                         /usr/lib64/libssl.so.1.0.1e
3418465000-341846c000 rw-p 00065000 fc:01 815307                         /usr/lib64/libssl.so.1.0.1e
3419200000-3419227000 r-xp 00000000 fc:01 815318                         /usr/lib64/libssh2.so.1.0.1
3419227000-3419426000 ---p 00027000 fc:01 815318                         /usr/lib64/libssh2.so.1.0.1
3419426000-3419427000 rw-p 00026000 fc:01 815318                         /usr/lib64/libssh2.so.1.0.1
3419427000-3419428000 rw-p 00000000 00:00 0
7f15178ba000-7f15178bb000 rw-p 00000000 00:00 0
7f15178bb000-7f151d74c000 r--p 00000000 fc:01 809926                     /usr/lib/locale/locale-archive
7f151d74c000-7f151ea80000 rw-s 00000000 00:04 4915204                    /SYSV73011769 (deleted)
7f151ea80000-7f152574d000 rw-s 00000000 00:04 4882435                    /SYSV67011769 (deleted)
7f152574d000-7f152674d000 rw-s 00000000 00:04 4849666                    /SYSV78011769 (deleted)
7f152674d000-7f1526f4d000 rw-s 00000000 00:04 4816897                    /SYSV68011769 (deleted)
7f1526f4d000-7f1526f59000 r-xp 00000000 fc:01 524319                     /lib64/libnss_files-2.12.so
7f1526f59000-7f1527159000 ---p 0000c000 fc:01 524319                     /lib64/libnss_files-2.12.so
7f1527159000-7f152715a000 r--p 0000c000 fc:01 524319                     /lib64/libnss_files-2.12.so
7f152715a000-7f152715b000 rw-p 0000d000 fc:01 524319                     /lib64/libnss_files-2.12.so
7f152715b000-7f1527160000 rw-p 00000000 00:00 0
7f1527160000-7f152717d000 r-xp 00000000 fc:01 524751                     /lib64/libselinux.so.1
7f152717d000-7f152737c000 ---p 0001d000 fc:01 524751                     /lib64/libselinux.so.1
7f152737c000-7f152737d000 r--p 0001c000 fc:01 524751                     /lib64/libselinux.so.1
7f152737d000-7f152737e000 rw-p 0001d000 fc:01 524751                     /lib64/libselinux.so.1
7f152737e000-7f1527381000 rw-p 00000000 00:00 0
7f1527381000-7f1527397000 r-xp 00000000 fc:01 524739                     /lib64/libgcc_s-4.4.7-20120601.so.1
7f1527397000-7f1527596000 ---p 00016000 fc:01 524739                     /lib64/libgcc_s-4.4.7-20120601.so.1
7f1527596000-7f1527597000 rw-p 00015000 fc:01 524739                     /lib64/libgcc_s-4.4.7-20120601.so.1
7f1527597000-7f1527599000 rw-p 00000000 00:00 0
7f1527599000-7f15275a0000 r-xp 00000000 fc:01 815143                     /usr/lib64/libOpenIPMIutils.so.0.0.1
7f15275a0000-7f15277a0000 ---p 00007000 fc:01 815143                     /usr/lib64/libOpenIPMIutils.so.0.0.1
7f15277a0000-7f15277a1000 rw-p 00007000 fc:01 815143                     /usr/lib64/libOpenIPMIutils.so.0.0.1
7f15277a1000-7f15277a2000 rw-p 00000000 00:00 0
7f15277a2000-7f152788a000 r-xp 00000000 fc:01 792072                     /usr/lib64/libstdc++.so.6.0.13
7f152788a000-7f1527a8a000 ---p 000e8000 fc:01 792072                     /usr/lib64/libstdc++.so.6.0.13
7f1527a8a000-7f1527a91000 r--p 000e8000 fc:01 792072                     /usr/lib64/libstdc++.so.6.0.13
7f1527a91000-7f1527a93000 rw-p 000ef000 fc:01 792072                     /usr/lib64/libstdc++.so.6.0.13
7f1527a93000-7f1527aad000 rw-p 00000000 00:00 0
7f1527aad000-7f1527d9d000 r-xp 00000000 fc:01 809955                     /usr/lib64/libmysqlclient.so.18.0.0
7f1527d9d000-7f1527f9d000 ---p 002f0000 fc:01 809955                     /usr/lib64/libmysqlclient.so.18.0.0
7f1527f9d000-7f152800a000 rw-p 002f0000 fc:01 809955                     /usr/lib64/libmysqlclient.so.18.0.0
7f152800a000-7f152800f000 rw-p 00000000 00:00 0
7f152800f000-7f1528019000 rw-s 00000000 00:04 4947973                    /SYSV53011769 (deleted)
7f1528019000-7f152801a000 rw-p 00000000 00:00 0
7f152801a000-7f152801b000 rw-p 00000000 00:00 0
7fffdf590000-7fffdf63c000 rw-p 00000000 00:00 0                          [stack]
7fffdf6ea000-7fffdf6ec000 r-xp 00000000 00:00 0                          [vdso]
ffffffffff600000-ffffffffff601000 r-xp 00000000 00:00 0                  [vsyscall]
  9068:20140925:162702.238 One child process died (PID:9104,exitcode/signal:6). Exiting ...
Comment by Sergey [ 2014 Sep 25 ]

if I am right equipment causing crashes named CS-121.
Next 2 crashes:

  5424:20140925:161702.096 SNMP response from host "CS-121" does not contain variable bindings in the requested order
*** glibc detected *** /usr/local/sbin/zabbix_proxy: poller #22 [got 0 values in 0.000006 sec, getting values]: free(): invalid pointer: 0x0000000000496d78 ***
======= Backtrace: =========
/lib64/libc.so.6[0x3410e76166]
/usr/local/sbin/zabbix_proxy: poller #22 [got 0 values in 0.000006 sec, getting values](free_result+0x7e)[0x42cb1e]
/usr/local/sbin/zabbix_proxy: poller #22 [got 0 values in 0.000006 sec, getting values][0x41ceb2]
/usr/local/sbin/zabbix_proxy: poller #22 [got 0 values in 0.000006 sec, getting values](main_poller_loop+0x82)[0x41dc02]
/usr/local/sbin/zabbix_proxy: poller #22 [got 0 values in 0.000006 sec, getting values](MAIN_ZABBIX_ENTRY+0x5d7)[0x410e67]
/usr/local/sbin/zabbix_proxy: poller #22 [got 0 values in 0.000006 sec, getting values](daemon_start+0x1a3)[0x44cc93]
/usr/local/sbin/zabbix_proxy: poller #22 [got 0 values in 0.000006 sec, getting values](main+0x2a8)[0x411498]
/lib64/libc.so.6(__libc_start_main+0xfd)[0x3410e1ed1d]
/usr/local/sbin/zabbix_proxy: poller #22 [got 0 values in 0.000006 sec, getting values][0x4101f9]

  7295:20140925:162202.090 SNMP response from host "CS-121" does not contain variable bindings in the requested order
*** glibc detected *** /usr/local/sbin/zabbix_proxy: poller #39 [got 0 values in 0.000006 sec, getting values]: free(): invalid pointer: 0x0000000000496d78 ***
======= Backtrace: =========
/lib64/libc.so.6[0x3410e76166]
/usr/local/sbin/zabbix_proxy: poller #39 [got 0 values in 0.000006 sec, getting values](free_result+0x7e)[0x42cb1e]
/usr/local/sbin/zabbix_proxy: poller #39 [got 0 values in 0.000006 sec, getting values][0x41ceb2]
/usr/local/sbin/zabbix_proxy: poller #39 [got 0 values in 0.000006 sec, getting values](main_poller_loop+0x82)[0x41dc02]
/usr/local/sbin/zabbix_proxy: poller #39 [got 0 values in 0.000006 sec, getting values](MAIN_ZABBIX_ENTRY+0x5d7)[0x410e67]
/usr/local/sbin/zabbix_proxy: poller #39 [got 0 values in 0.000006 sec, getting values](daemon_start+0x1a3)[0x44cc93]
/usr/local/sbin/zabbix_proxy: poller #39 [got 0 values in 0.000006 sec, getting values](main+0x2a8)[0x411498]
/lib64/libc.so.6(__libc_start_main+0xfd)[0x3410e1ed1d]
/usr/local/sbin/zabbix_proxy: poller #39 [got 0 values in 0.000006 sec, getting values][0x4101f9]
Comment by Aleksandrs Saveljevs [ 2014 Sep 25 ]

Sergey, thank you very much for cooperation and sorry for the crashes. I have fixed "validation.patch", please use the new version instead.

Meanwhile, if you could do some tcpdump of the offending devices that the patched proxy reports, that would be great.

Comment by Sergey [ 2014 Oct 06 ]

Hi, Alexandrs. During this time we did not have any crashes. So, now I didn't have any addition information for you. Thank you too.

Comment by Andris Zeila [ 2014 Oct 07 ]

Successfully tested

Comment by Aleksandrs Saveljevs [ 2014 Oct 08 ]

Fixed in pre-2.2.7 r49660, pre-2.4.2 r49661, and pre-2.5.0 (trunk) r49663.

Comment by Aleksandrs Saveljevs [ 2014 Oct 08 ]

(1) Documented at https://www.zabbix.com/documentation/2.2/manual/introduction/whatsnew227#daemon_improvements .

<richlv> great level of detail, but we should not mention 2.4 in 2.2 docs, and we should mention this in the general snmp monitoring pages for 2.4 and 3.0 as a suggestion to disable bulk operations per interface, and as a suggestion to disable them globally for 2.2

asaveljevs Updated:

RESOLVED.

<richlv> added a couple of links to the config param pages for 2.2 to make it clear what configuration is referenced here. that option should be first added, though...

wiper should we list all possible SNMP validation error messages?

<richlv> well, we had a documentation page like this :
https://www.zabbix.com/documentation/1.8/manual/troubleshooting/error_and_warning_messages
unfortunately, we messed up with it. if we could re-create it for recent version + add snmp messages, that would be great

asaveljevs Recreating such a page is probably out of scope of this development. CLOSED.

<richlv> for the record, reported as ZBX-8939 - this subissue still CLOSED

Comment by Aleksandrs Saveljevs [ 2014 Oct 09 ]

(2) Currently, it is implemented in such a way that we look at the received variable bindings one by one. As long as variable bindings match the request, we set item values. As soon as we encounter a bad variable binding, we set the remaining items to not supported. However, it would be better if we would reject the whole response and make all items go unsupported, because if a device tends to misbehave, we cannot be sure that the initial variable bindings are trustworthy.

asaveljevs RESOLVED in r49699.

wiper CLOSED

Comment by Andris Zeila [ 2014 Oct 09 ]

Successfully tested

Comment by Aleksandrs Saveljevs [ 2014 Oct 09 ]

Fixed in pre-2.2.7 r49710, pre-2.4.2 r49712, and pre-2.5.0 (trunk) r49713.

Generated at Wed Apr 24 18:05:17 EEST 2024 using Jira 9.12.4#9120004-sha1:625303b708afdb767e17cb2838290c41888e9ff0.