[ZBX-13730] High CPU / Faliure to Match REGEX - ZABBIX SUPPORT

XML

Word

Printable

Details

Type: Problem report
Status: Closed
Priority: Critical
Resolution: Fixed
Affects Version/s: 3.4.8
Fix Version/s: 3.4.13rc1, 4.0.0beta1, 4.0 (plan)
Component/s: Agent (G), Proxy (P), Server (S)
Labels:
- cpu
- pcre
- regex
Environment:

Hide
uname -a
Linux zabbix 3.10.0-514.26.2.el7.x86_64 #1 SMP Tue Jul 4 15:04:05 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux

# ldd /usr/sbin/zabbix_server
        linux-vdso.so.1 => (0x00007ffde0ba3000)
        libmysqlclient.so.18 => /usr/lib64/mysql/libmysqlclient.so.18 (0x00007fe4b4ce6000)
        libpthread.so.0 => /lib64/libpthread.so.0 (0x00007fe4b4aca000)
        libz.so.1 => /lib64/libz.so.1 (0x00007fe4b48b3000)
        libm.so.6 => /lib64/libm.so.6 (0x00007fe4b45b1000)
        libssl.so.10 => /lib64/libssl.so.10 (0x00007fe4b4343000)
        libcrypto.so.10 => /lib64/libcrypto.so.10 (0x00007fe4b3f58000)
        libdl.so.2 => /lib64/libdl.so.2 (0x00007fe4b3d54000)
        libiksemel.so.3 => /lib64/libiksemel.so.3 (0x00007fe4b3b46000)
        libxml2.so.2 => /lib64/libxml2.so.2 (0x00007fe4b37db000)
        libodbc.so.2 => /lib64/libodbc.so.2 (0x00007fe4b3573000)
        libnetsnmp.so.31 => /lib64/libnetsnmp.so.31 (0x00007fe4b3270000)
        libssh2.so.1 => /lib64/libssh2.so.1 (0x00007fe4b3045000)
        libOpenIPMI.so.0 => /lib64/libOpenIPMI.so.0 (0x00007fe4b2d38000)
        libOpenIPMIposix.so.0 => /lib64/libOpenIPMIposix.so.0 (0x00007fe4b2b32000)
        libevent-2.0.so.5 => /lib64/libevent-2.0.so.5 (0x00007fe4b28e9000)
        libldap-2.4.so.2 => /lib64/libldap-2.4.so.2 (0x00007fe4b2696000)
        liblber-2.4.so.2 => /lib64/liblber-2.4.so.2 (0x00007fe4b2487000)
        libcurl.so.4 => /lib64/libcurl.so.4 (0x00007fe4b221d000)
        libresolv.so.2 => /lib64/libresolv.so.2 (0x00007fe4b2003000)
        libpcreposix.so.0 => /lib64/libpcreposix.so.0 (0x00007fe4b1e00000)
        libpcre.so.1 => /lib64/libpcre.so.1 (0x00007fe4b1b9e000)
        libgcc_s.so.1 => /lib64/libgcc_s.so.1 (0x00007fe4b1988000)
        libc.so.6 => /lib64/libc.so.6 (0x00007fe4b15c7000)
        libstdc++.so.6 => /lib64/libstdc++.so.6 (0x00007fe4b12bd000)
        /lib64/ld-linux-x86-64.so.2 (0x00007fe4b5747000)
        libgssapi_krb5.so.2 => /lib64/libgssapi_krb5.so.2 (0x00007fe4b106f000)
        libkrb5.so.3 => /lib64/libkrb5.so.3 (0x00007fe4b0d88000)
        libcom_err.so.2 => /lib64/libcom_err.so.2 (0x00007fe4b0b83000)
        libk5crypto.so.3 => /lib64/libk5crypto.so.3 (0x00007fe4b0951000)
        libgnutls.so.28 => /lib64/libgnutls.so.28 (0x00007fe4b0619000)
        libgcrypt.so.11 => /lib64/libgcrypt.so.11 (0x00007fe4b0397000)
        libgpg-error.so.0 => /lib64/libgpg-error.so.0 (0x00007fe4b0192000)
        liblzma.so.5 => /lib64/liblzma.so.5 (0x00007fe4aff6c000)
        libltdl.so.7 => /lib64/libltdl.so.7 (0x00007fe4afd61000)
        libOpenIPMIutils.so.0 => /lib64/libOpenIPMIutils.so.0 (0x00007fe4afb58000)
        libgdbm.so.4 => /lib64/libgdbm.so.4 (0x00007fe4af94f000)
        libsasl2.so.3 => /lib64/libsasl2.so.3 (0x00007fe4af731000)
        libssl3.so => /lib64/libssl3.so (0x00007fe4af4e5000)
        libsmime3.so => /lib64/libsmime3.so (0x00007fe4af2be000)
        libnss3.so => /lib64/libnss3.so (0x00007fe4aef93000)
        libnssutil3.so => /lib64/libnssutil3.so (0x00007fe4aed66000)
        libplds4.so => /lib64/libplds4.so (0x00007fe4aeb62000)
        libplc4.so => /lib64/libplc4.so (0x00007fe4ae95c000)
        libnspr4.so => /lib64/libnspr4.so (0x00007fe4ae71e000)
        libidn.so.11 => /lib64/libidn.so.11 (0x00007fe4ae4eb000)
        libkrb5support.so.0 => /lib64/libkrb5support.so.0 (0x00007fe4ae2db000)
        libkeyutils.so.1 => /lib64/libkeyutils.so.1 (0x00007fe4ae0d7000)
        libp11-kit.so.0 => /lib64/libp11-kit.so.0 (0x00007fe4ade90000)
        libtspi.so.1 => /lib64/libtspi.so.1 (0x00007fe4adc1f000)
        libtasn1.so.6 => /lib64/libtasn1.so.6 (0x00007fe4ada0b000)
        libnettle.so.4 => /lib64/libnettle.so.4 (0x00007fe4ad7d9000)
        libhogweed.so.2 => /lib64/libhogweed.so.2 (0x00007fe4ad5b2000)
        libgmp.so.10 => /lib64/libgmp.so.10 (0x00007fe4ad33b000)
        libcrypt.so.1 => /lib64/libcrypt.so.1 (0x00007fe4ad103000)
        librt.so.1 => /lib64/librt.so.1 (0x00007fe4acefb000)
        libselinux.so.1 => /lib64/libselinux.so.1 (0x00007fe4accd3000)
        libffi.so.6 => /lib64/libffi.so.6 (0x00007fe4acacb000)
        libfreebl3.so => /lib64/libfreebl3.so (0x00007fe4ac8c7000)

Show
uname -a Linux zabbix 3.10.0-514.26.2.el7.x86_64 #1 SMP Tue Jul 4 15:04:05 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux # ldd /usr/sbin/zabbix_server         linux-vdso.so.1 => (0x00007ffde0ba3000)         libmysqlclient.so.18 => /usr/lib64/mysql/libmysqlclient.so.18 (0x00007fe4b4ce6000)         libpthread.so.0 => /lib64/libpthread.so.0 (0x00007fe4b4aca000)         libz.so.1 => /lib64/libz.so.1 (0x00007fe4b48b3000)         libm.so.6 => /lib64/libm.so.6 (0x00007fe4b45b1000)         libssl.so.10 => /lib64/libssl.so.10 (0x00007fe4b4343000)         libcrypto.so.10 => /lib64/libcrypto.so.10 (0x00007fe4b3f58000)         libdl.so.2 => /lib64/libdl.so.2 (0x00007fe4b3d54000)         libiksemel.so.3 => /lib64/libiksemel.so.3 (0x00007fe4b3b46000)         libxml2.so.2 => /lib64/libxml2.so.2 (0x00007fe4b37db000)         libodbc.so.2 => /lib64/libodbc.so.2 (0x00007fe4b3573000)         libnetsnmp.so.31 => /lib64/libnetsnmp.so.31 (0x00007fe4b3270000)         libssh2.so.1 => /lib64/libssh2.so.1 (0x00007fe4b3045000)         libOpenIPMI.so.0 => /lib64/libOpenIPMI.so.0 (0x00007fe4b2d38000)         libOpenIPMIposix.so.0 => /lib64/libOpenIPMIposix.so.0 (0x00007fe4b2b32000)         libevent-2.0.so.5 => /lib64/libevent-2.0.so.5 (0x00007fe4b28e9000)         libldap-2.4.so.2 => /lib64/libldap-2.4.so.2 (0x00007fe4b2696000)         liblber-2.4.so.2 => /lib64/liblber-2.4.so.2 (0x00007fe4b2487000)         libcurl.so.4 => /lib64/libcurl.so.4 (0x00007fe4b221d000)         libresolv.so.2 => /lib64/libresolv.so.2 (0x00007fe4b2003000)         libpcreposix.so.0 => /lib64/libpcreposix.so.0 (0x00007fe4b1e00000)         libpcre.so.1 => /lib64/libpcre.so.1 (0x00007fe4b1b9e000)         libgcc_s.so.1 => /lib64/libgcc_s.so.1 (0x00007fe4b1988000)         libc.so.6 => /lib64/libc.so.6 (0x00007fe4b15c7000)         libstdc++.so.6 => /lib64/libstdc++.so.6 (0x00007fe4b12bd000)         /lib64/ld-linux-x86-64.so.2 (0x00007fe4b5747000)         libgssapi_krb5.so.2 => /lib64/libgssapi_krb5.so.2 (0x00007fe4b106f000)         libkrb5.so.3 => /lib64/libkrb5.so.3 (0x00007fe4b0d88000)         libcom_err.so.2 => /lib64/libcom_err.so.2 (0x00007fe4b0b83000)         libk5crypto.so.3 => /lib64/libk5crypto.so.3 (0x00007fe4b0951000)         libgnutls.so.28 => /lib64/libgnutls.so.28 (0x00007fe4b0619000)         libgcrypt.so.11 => /lib64/libgcrypt.so.11 (0x00007fe4b0397000)         libgpg-error.so.0 => /lib64/libgpg-error.so.0 (0x00007fe4b0192000)         liblzma.so.5 => /lib64/liblzma.so.5 (0x00007fe4aff6c000)         libltdl.so.7 => /lib64/libltdl.so.7 (0x00007fe4afd61000)         libOpenIPMIutils.so.0 => /lib64/libOpenIPMIutils.so.0 (0x00007fe4afb58000)         libgdbm.so.4 => /lib64/libgdbm.so.4 (0x00007fe4af94f000)         libsasl2.so.3 => /lib64/libsasl2.so.3 (0x00007fe4af731000)         libssl3.so => /lib64/libssl3.so (0x00007fe4af4e5000)         libsmime3.so => /lib64/libsmime3.so (0x00007fe4af2be000)         libnss3.so => /lib64/libnss3.so (0x00007fe4aef93000)         libnssutil3.so => /lib64/libnssutil3.so (0x00007fe4aed66000)         libplds4.so => /lib64/libplds4.so (0x00007fe4aeb62000)         libplc4.so => /lib64/libplc4.so (0x00007fe4ae95c000)         libnspr4.so => /lib64/libnspr4.so (0x00007fe4ae71e000)         libidn.so.11 => /lib64/libidn.so.11 (0x00007fe4ae4eb000)         libkrb5support.so.0 => /lib64/libkrb5support.so.0 (0x00007fe4ae2db000)         libkeyutils.so.1 => /lib64/libkeyutils.so.1 (0x00007fe4ae0d7000)         libp11-kit.so.0 => /lib64/libp11-kit.so.0 (0x00007fe4ade90000)         libtspi.so.1 => /lib64/libtspi.so.1 (0x00007fe4adc1f000)         libtasn1.so.6 => /lib64/libtasn1.so.6 (0x00007fe4ada0b000)         libnettle.so.4 => /lib64/libnettle.so.4 (0x00007fe4ad7d9000)         libhogweed.so.2 => /lib64/libhogweed.so.2 (0x00007fe4ad5b2000)         libgmp.so.10 => /lib64/libgmp.so.10 (0x00007fe4ad33b000)         libcrypt.so.1 => /lib64/libcrypt.so.1 (0x00007fe4ad103000)         librt.so.1 => /lib64/librt.so.1 (0x00007fe4acefb000)         libselinux.so.1 => /lib64/libselinux.so.1 (0x00007fe4accd3000)         libffi.so.6 => /lib64/libffi.so.6 (0x00007fe4acacb000)         libfreebl3.so => /lib64/libfreebl3.so (0x00007fe4ac8c7000)

Team:
Team C
Sprint:
Sprint 31, Sprint 32, Sprint 33, Sprint 34, Sprint 35, Sprint 36, Sprint 37, Sprint 38, Sprint 39, Sprint 40
Story Points:
2

Description

Steps to reproduce:
Create a hosttem with the following Macros:
{$IP_NEIGHBOR1} value 201.16.1.34
{$IP_NEIGHBOR2} value 201.16.1.35

On this host create a item with the keys bellow:
Item1

snmptrap["(.|[[:space:]])*OSPF-TRAP-MIB::ospfNbrStateChange(.|[[:space:]])*OSPF-MIB::ospfNbrState.{$IP_NEIGHBOR1}.0(.|[[:space:]])*"]

Item2

snmptrap["(.|[[:space:]])*OSPF-TRAP-MIB::ospfNbrStateChange(.|[[:space:]])*OSPF-MIB::ospfNbrState.{$IP_NEIGHBOR2}.0(.|[[:space:]])*"]

Send a trap like this:

17:41:30 2018/04/10 ZBXTRAP 201.16.0.20
PDU INFO:
  notificationtype               TRAP
  version                        1
  receivedfrom                   UDP: [201.16.0.20]:59065->[201.16.8.5]:162
  errorstatus                    0
  messageid                      0
  community                      brfmgmt
  transactionid                  3358
  errorindex                     0
  requestid                      1915218045
VARBINDS:
  DISMAN-EVENT-MIB::sysUpTimeInstance type=67 value=Timeticks: (227107105) 26 days, 6:51:11.05
  SNMPv2-MIB::snmpTrapOID.0      type=6  value=OID: OSPF-TRAP-MIB::ospfNbrStateChange
  OSPF-MIB::ospfRouterId.0       type=64 value=IpAddress: 201.16.0.20
  OSPF-MIB::ospfNbrIpAddr.201.16.1.34.0 type=64 value=IpAddress: 201.16.1.34
  OSPF-MIB::ospfNbrAddressLessIndex.201.16.1.34.0 type=2  value=INTEGER: 0
  OSPF-MIB::ospfNbrRtrId.201.16.1.34.0 type=64 value=IpAddress: 201.16.0.6
  OSPF-MIB::ospfNbrState.201.16.1.34.0 type=2  value=INTEGER: 8
  SNMPv2-MIB::snmpTrapEnterprise.0 type=6  value=OID: JUNIPER-CHASSIS-DEFINES-MIB::jnxProductNameMX480

Then the zabbix main trapper process shall spike to 100% cpu utilization.

Attachments

- Sort By Name
- Sort By Date
- Ascending
- Descending
- Thumbnails
- List
- Download All

global_regexp_exists.patch
4 kB
2018 Jun 01 13:56
Selection_530.png
36 kB
2018 Jul 03 18:08
Selection_533.png
49 kB
2018 Jul 11 18:14
strace - zabbix trapper.txt
91 kB
2018 Apr 11 00:00
zabbix-3.4.13rc1-compat_pcre.patch
2 kB
2018 Aug 22 09:29

Issue Links

causes

ZBX-15010 Regex in "log" item causes "cannot compile regular expression" error

Closed

ZBX-15107 Incorrect error handling in compile_filename_regexp() if file name regexp does not compile in logrt[], logrt.count[] items

Closed

ZBX-15318 Invalid regular expression in snmptrap[] Item

Closed

ZBX-15178 If "output template" references captured groups which do not exist in regexp then item value can be unpredictable

Closed

Activity

People

Assignee:: Jurijs Klopovskis

Reporter:: Klaus Schneider

Votes:: 0 Vote for this issue

Watchers:: 11 Start watching this issue

Dates

Created:: 2018 Apr 11 00:00

Updated:: 2019 Jan 23 18:00

Resolved:: 2018 Aug 10 10:02