-
Incident report
-
Resolution: Duplicate
-
Critical
-
None
-
3.2.2
-
Proxy server runs on:
RHEL 2.6.32-642.6.2.el6.x86_64 (Virtual server under vCenter)
2 CPU, 6GB RAM
MySQL DB Community server 5.7.17 running on same server
-------------------------------------------------------
Compiled from source with following options:
./configure \
--prefix=/opt/zabbix_proxy \
--enable-proxy \
--with-ldap \
--with-mysql \
--with-libxml2 \
--with-libcurl \
--with-ssh2 \
--with-openssl \
--with-net-snmp
-------------------------------------------------------
Configuration File
----------------------
DBHost=localhost
DBName=zabbix
DBUser=zabbix
DBPassword=******
Hostname=zbx_proxy
LogFile=/var/log/zabbix_proxy/zabbix_proxy.log
PidFile=/var/run/zabbix_proxy/zabbix_proxy.pid
Server=zbx_server
ConfigFrequency=900
CacheSize=256M
HistoryCacheSize=128M
StartDiscoverers=50
StartPollers=50
StartPingers=5
StartVMwareCollectors=50
VMwareFrequency=600
VMwarePerfFrequency=60
VMwareCacheSize=256M
VMwareTimeout=60
ProxyOfflineBuffer=336
HeartbeatFrequency=90
DataSenderFrequency=5
TLSConnect=psk
TLSPSKFile=/opt/zabbix_proxy/etc/zabbix_proxy.psk
TLSPSKIdentity=PSK-zbx_proxy
-------------------------------------------------------
ldd output for zabbix_proxy
ldd /opt/zabbix_proxy/sbin/zabbix_proxy
linux-vdso.so.1 => (0x00007ffdc4d82000)
libmysqlclient.so.20 => /usr/lib64/mysql/libmysqlclient.so.20 (0x00007f8a6753a000)
libxml2.so.2 => /usr/lib64/libxml2.so.2 (0x0000003def800000)
libnetsnmp.so.20 => /usr/lib64/libnetsnmp.so.20 (0x0000003b15600000)
libssh2.so.1 => /usr/lib64/libssh2.so.1 (0x0000003b15a00000)
libssl.so.10 => /usr/lib64/libssl.so.10 (0x00007f8a672cb000)
libcrypto.so.10 => /usr/lib64/libcrypto.so.10 (0x0000003b15200000)
libldap-2.4.so.2 => /lib64/libldap-2.4.so.2 (0x000000394fa00000)
liblber-2.4.so.2 => /lib64/liblber-2.4.so.2 (0x00000032d1a00000)
libcurl.so.4 => /usr/lib64/libcurl.so.4 (0x0000003b15e00000)
libm.so.6 => /lib64/libm.so.6 (0x00000032d0200000)
libdl.so.2 => /lib64/libdl.so.2 (0x00000032cfa00000)
librt.so.1 => /lib64/librt.so.1 (0x00000032cfe00000)
libresolv.so.2 => /lib64/libresolv.so.2 (0x00000032d1200000)
libc.so.6 => /lib64/libc.so.6 (0x00000032cf200000)
libpthread.so.0 => /lib64/libpthread.so.0 (0x00000032cf600000)
libstdc++.so.6 => /usr/lib64/libstdc++.so.6 (0x00000032d2600000)
libgcc_s.so.1 => /lib64/libgcc_s.so.1 (0x00000032d1600000)
libz.so.1 => /lib64/libz.so.1 (0x00000032d0600000)
libgssapi_krb5.so.2 => /lib64/libgssapi_krb5.so.2 (0x00000032dc200000)
libkrb5.so.3 => /lib64/libkrb5.so.3 (0x00000032d9e00000)
libcom_err.so.2 => /lib64/libcom_err.so.2 (0x00000032d8600000)
libk5crypto.so.3 => /lib64/libk5crypto.so.3 (0x00000032daa00000)
libsasl2.so.2 => /usr/lib64/libsasl2.so.2 (0x000000394f600000)
libssl3.so => /usr/lib64/libssl3.so (0x00000032d8a00000)
libsmime3.so => /usr/lib64/libsmime3.so (0x00000032d7e00000)
libnss3.so => /usr/lib64/libnss3.so (0x00000032d3a00000)
libnssutil3.so => /usr/lib64/libnssutil3.so (0x00000032d3e00000)
libplds4.so => /lib64/libplds4.so (0x00000032d4a00000)
libplc4.so => /lib64/libplc4.so (0x00000032d4600000)
libnspr4.so => /lib64/libnspr4.so (0x00000032d3600000)
libidn.so.11 => /lib64/libidn.so.11 (0x00000032d1e00000)
/lib64/ld-linux-x86-64.so.2 (0x00000032cee00000)
libkrb5support.so.0 => /lib64/libkrb5support.so.0 (0x00000032d9a00000)
libkeyutils.so.1 => /lib64/libkeyutils.so.1 (0x00000032d9200000)
libcrypt.so.1 => /lib64/libcrypt.so.1 (0x000000394ee00000)
libselinux.so.1 => /lib64/libselinux.so.1 (0x00000032d0a00000)
libfreebl3.so => /lib64/libfreebl3.so (0x000000394f200000)
Master Zabbix server is on Physical DL 380 with an Oracle DB on an external DB server.Proxy server runs on: RHEL 2.6.32-642.6.2.el6.x86_64 (Virtual server under vCenter) 2 CPU, 6GB RAM MySQL DB Community server 5.7.17 running on same server ------------------------------------------------------- Compiled from source with following options: ./configure \ --prefix=/opt/zabbix_proxy \ --enable-proxy \ --with-ldap \ --with-mysql \ --with-libxml2 \ --with-libcurl \ --with-ssh2 \ --with-openssl \ --with-net-snmp ------------------------------------------------------- Configuration File ---------------------- DBHost=localhost DBName=zabbix DBUser=zabbix DBPassword=****** Hostname=zbx_proxy LogFile=/var/log/zabbix_proxy/zabbix_proxy.log PidFile=/var/run/zabbix_proxy/zabbix_proxy.pid Server=zbx_server ConfigFrequency=900 CacheSize=256M HistoryCacheSize=128M StartDiscoverers=50 StartPollers=50 StartPingers=5 StartVMwareCollectors=50 VMwareFrequency=600 VMwarePerfFrequency=60 VMwareCacheSize=256M VMwareTimeout=60 ProxyOfflineBuffer=336 HeartbeatFrequency=90 DataSenderFrequency=5 TLSConnect=psk TLSPSKFile=/opt/zabbix_proxy/etc/zabbix_proxy.psk TLSPSKIdentity=PSK-zbx_proxy ------------------------------------------------------- ldd output for zabbix_proxy ldd /opt/zabbix_proxy/sbin/zabbix_proxy linux-vdso.so.1 => (0x00007ffdc4d82000) libmysqlclient.so.20 => /usr/lib64/mysql/libmysqlclient.so.20 (0x00007f8a6753a000) libxml2.so.2 => /usr/lib64/libxml2.so.2 (0x0000003def800000) libnetsnmp.so.20 => /usr/lib64/libnetsnmp.so.20 (0x0000003b15600000) libssh2.so.1 => /usr/lib64/libssh2.so.1 (0x0000003b15a00000) libssl.so.10 => /usr/lib64/libssl.so.10 (0x00007f8a672cb000) libcrypto.so.10 => /usr/lib64/libcrypto.so.10 (0x0000003b15200000) libldap-2.4.so.2 => /lib64/libldap-2.4.so.2 (0x000000394fa00000) liblber-2.4.so.2 => /lib64/liblber-2.4.so.2 (0x00000032d1a00000) libcurl.so.4 => /usr/lib64/libcurl.so.4 (0x0000003b15e00000) libm.so.6 => /lib64/libm.so.6 (0x00000032d0200000) libdl.so.2 => /lib64/libdl.so.2 (0x00000032cfa00000) librt.so.1 => /lib64/librt.so.1 (0x00000032cfe00000) libresolv.so.2 => /lib64/libresolv.so.2 (0x00000032d1200000) libc.so.6 => /lib64/libc.so.6 (0x00000032cf200000) libpthread.so.0 => /lib64/libpthread.so.0 (0x00000032cf600000) libstdc++.so.6 => /usr/lib64/libstdc++.so.6 (0x00000032d2600000) libgcc_s.so.1 => /lib64/libgcc_s.so.1 (0x00000032d1600000) libz.so.1 => /lib64/libz.so.1 (0x00000032d0600000) libgssapi_krb5.so.2 => /lib64/libgssapi_krb5.so.2 (0x00000032dc200000) libkrb5.so.3 => /lib64/libkrb5.so.3 (0x00000032d9e00000) libcom_err.so.2 => /lib64/libcom_err.so.2 (0x00000032d8600000) libk5crypto.so.3 => /lib64/libk5crypto.so.3 (0x00000032daa00000) libsasl2.so.2 => /usr/lib64/libsasl2.so.2 (0x000000394f600000) libssl3.so => /usr/lib64/libssl3.so (0x00000032d8a00000) libsmime3.so => /usr/lib64/libsmime3.so (0x00000032d7e00000) libnss3.so => /usr/lib64/libnss3.so (0x00000032d3a00000) libnssutil3.so => /usr/lib64/libnssutil3.so (0x00000032d3e00000) libplds4.so => /lib64/libplds4.so (0x00000032d4a00000) libplc4.so => /lib64/libplc4.so (0x00000032d4600000) libnspr4.so => /lib64/libnspr4.so (0x00000032d3600000) libidn.so.11 => /lib64/libidn.so.11 (0x00000032d1e00000) /lib64/ld-linux-x86-64.so.2 (0x00000032cee00000) libkrb5support.so.0 => /lib64/libkrb5support.so.0 (0x00000032d9a00000) libkeyutils.so.1 => /lib64/libkeyutils.so.1 (0x00000032d9200000) libcrypt.so.1 => /lib64/libcrypt.so.1 (0x000000394ee00000) libselinux.so.1 => /lib64/libselinux.so.1 (0x00000032d0a00000) libfreebl3.so => /lib64/libfreebl3.so (0x000000394f200000) Master Zabbix server is on Physical DL 380 with an Oracle DB on an external DB server.
Server, proxy and agent compiled from sources as I've always done.
All versions from 1.8 to 3.0.4 have compiled and run without any problems until now.
Compiled configuration completes without any problems.
Compile completes without any problems.
I create an RPM from compile as we have 12 proxies and two servers and an RPM makes distribution easier.
Main zabbix servers use an external Oracle DB each.
All proxies use local MySQL DB on each proxy.
Proxies are used to interface with VMware environment
I was able to upgrade our Dev/test zabbix server and proxy from 3.0.4 to 3.2.2 without any problems.
Our first production zabbix server upgraded from 3.0.4 to 3.2.2 without any problems.
The first production proxy crashes after upgrade from 3.0.4 to 3.2.2.
I have stopped any further upgrades to proxies until this problem can be resolved.
This proxy has 34 servers and 1112 items connecting to it.
And has run versions 2.2.2 to 3.0.4 without any problems till now.
The first zabbix server has five proxies connecting to it and
~1030 hosts with ~40,000 items.
I have not upgraded our second zabbix server.
Some basic troubleshooting I've been able to perform shows that if I set either
StartVMwareCollectors to zero
or
StartVMwareCollectors to zero
the proxy does not crash. If both parameters are set to a non-zero value the proxy crashes.
Please see attached log and disassembly.
Any help, insight, comments would be appreciated.
- duplicates
-
-
- Closed
-