Hello Leonardo,
Could you attach:
- current zabbix_server.conf and log file containing full upgrade part, logs before and after an error you mentioned above
- output from the following commands:
ldd zabbix_server sestatus;
Do you have this problem currently?
Regards,
Edgar
Unfortunately I lost the upgrade log... but I check old logs when was running 3.4 and this problem didn't happen before, and yes it is happen now once an hour (my discovery interval I believe)
# ldd /usr/sbin/zabbix_server linux-vdso.so.1 => (0x00007ffc5bd9e000) libpq.so.5 => /lib64/libpq.so.5 (0x00007f9191084000) libiksemel.so.3 => /lib64/libiksemel.so.3 (0x00007f9190e76000) libxml2.so.2 => /lib64/libxml2.so.2 (0x00007f9190b0b000) libodbc.so.2 => /lib64/libodbc.so.2 (0x00007f91908a3000) libnetsnmp.so.31 => /lib64/libnetsnmp.so.31 (0x00007f91905a1000) libssh2.so.1 => /lib64/libssh2.so.1 (0x00007f9190376000) libOpenIPMI.so.0 => /lib64/libOpenIPMI.so.0 (0x00007f9190069000) libOpenIPMIposix.so.0 => /lib64/libOpenIPMIposix.so.0 (0x00007f918fe63000) libz.so.1 => /lib64/libz.so.1 (0x00007f918fc4c000) libpthread.so.0 => /lib64/libpthread.so.0 (0x00007f918fa30000) libevent-2.0.so.5 => /lib64/libevent-2.0.so.5 (0x00007f918f7e8000) libssl.so.10 => /lib64/libssl.so.10 (0x00007f918f579000) libcrypto.so.10 => /lib64/libcrypto.so.10 (0x00007f918f18f000) libldap-2.4.so.2 => /lib64/libldap-2.4.so.2 (0x00007f918ef3c000) liblber-2.4.so.2 => /lib64/liblber-2.4.so.2 (0x00007f918ed2c000) libcurl.so.4 => /lib64/libcurl.so.4 (0x00007f918eac3000) libm.so.6 => /lib64/libm.so.6 (0x00007f918e7c1000) libdl.so.2 => /lib64/libdl.so.2 (0x00007f918e5bc000) libresolv.so.2 => /lib64/libresolv.so.2 (0x00007f918e3a2000) libpcre.so.1 => /lib64/libpcre.so.1 (0x00007f918e141000) libc.so.6 => /lib64/libc.so.6 (0x00007f918dd7f000) libkrb5.so.3 => /lib64/libkrb5.so.3 (0x00007f918da98000) libcom_err.so.2 => /lib64/libcom_err.so.2 (0x00007f918d894000) libgssapi_krb5.so.2 => /lib64/libgssapi_krb5.so.2 (0x00007f918d645000) libldap_r-2.4.so.2 => /lib64/libldap_r-2.4.so.2 (0x00007f918d3e9000) libgnutls.so.28 => /lib64/libgnutls.so.28 (0x00007f918d0b1000) libgcrypt.so.11 => /lib64/libgcrypt.so.11 (0x00007f918ce2f000) libgpg-error.so.0 => /lib64/libgpg-error.so.0 (0x00007f918cc2a000) liblzma.so.5 => /lib64/liblzma.so.5 (0x00007f918ca04000) libltdl.so.7 => /lib64/libltdl.so.7 (0x00007f918c7f9000) libOpenIPMIutils.so.0 => /lib64/libOpenIPMIutils.so.0 (0x00007f918c5f0000) libgdbm.so.4 => /lib64/libgdbm.so.4 (0x00007f918c3e7000) /lib64/ld-linux-x86-64.so.2 (0x00007f919177a000) libk5crypto.so.3 => /lib64/libk5crypto.so.3 (0x00007f918c1b4000) libsasl2.so.3 => /lib64/libsasl2.so.3 (0x00007f918bf97000) libssl3.so => /lib64/libssl3.so (0x00007f918bd54000) libsmime3.so => /lib64/libsmime3.so (0x00007f918bb2c000) libnss3.so => /lib64/libnss3.so (0x00007f918b806000) libnssutil3.so => /lib64/libnssutil3.so (0x00007f918b5da000) libplds4.so => /lib64/libplds4.so (0x00007f918b3d5000) libplc4.so => /lib64/libplc4.so (0x00007f918b1d0000) libnspr4.so => /lib64/libnspr4.so (0x00007f918af92000) libidn.so.11 => /lib64/libidn.so.11 (0x00007f918ad5e000) libkrb5support.so.0 => /lib64/libkrb5support.so.0 (0x00007f918ab4f000) libkeyutils.so.1 => /lib64/libkeyutils.so.1 (0x00007f918a94a000) libp11-kit.so.0 => /lib64/libp11-kit.so.0 (0x00007f918a704000) libtspi.so.1 => /lib64/libtspi.so.1 (0x00007f918a492000) libtasn1.so.6 => /lib64/libtasn1.so.6 (0x00007f918a27e000) libnettle.so.4 => /lib64/libnettle.so.4 (0x00007f918a04d000) libhogweed.so.2 => /lib64/libhogweed.so.2 (0x00007f9189e25000) libgmp.so.10 => /lib64/libgmp.so.10 (0x00007f9189bae000) libcrypt.so.1 => /lib64/libcrypt.so.1 (0x00007f9189976000) librt.so.1 => /lib64/librt.so.1 (0x00007f918976e000) libselinux.so.1 => /lib64/libselinux.so.1 (0x00007f9189546000) libffi.so.6 => /lib64/libffi.so.6 (0x00007f918933e000) libfreebl3.so => /lib64/libfreebl3.so (0x00007f918913a000) # sestatus SELinux status: disabled # grep -v "^#" /etc/zabbix/zabbix_server.conf | grep "^." LogFile=/var/log/zabbix/zabbix_server.log LogFileSize=0 DebugLevel=3 PidFile=/var/run/zabbix/zabbix_server.pid DBHost=******** DBName=**** DBUser=******* DBPassword=****** DBPort=5432 StartPollers=50 StartPollersUnreachable=15 StartPingers=7 StartDiscoverers=35 SNMPTrapperFile=/var/log/snmptrap/snmptrap.log CacheSize=512M TrendCacheSize=16M ValueCacheSize=256M Timeout=3 AlertScriptsPath=/usr/lib/zabbix/alertscripts ExternalScripts=/usr/lib/zabbix/externalscripts LogSlowQueries=3000
The error is different from ZBX-13379 ,but could you please execute the following query to be sure you do not have any items stuck in a strange status ?
select count(*) from items i,hosts h where i.hostid=h.hostid and h.status in (0,1) and i.flags<>2 and i.status not in (0,1);
0 is the result...
Same here. Also occurred after upgrade from 3.2. Current version 4.0.3
zabbix_server [10567]: ERROR [file:lld_item.c,line:3547] Something impossible has just happened. 10567:20190125:123835.397 === Backtrace: === 10567:20190125:123835.398 13: /usr/sbin/zabbix_server: poller #1 [got 1 values in 0.001649 sec, getting values](zbx_backtrace+0x35) [0x5559784739ad] 10567:20190125:123835.398 12: /usr/sbin/zabbix_server: poller #1 [got 1 values in 0.001649 sec, getting values](+0x15f3ab) [0x5559784e03ab] 10567:20190125:123835.398 11: /usr/sbin/zabbix_server: poller #1 [got 1 values in 0.001649 sec, getting values](+0x15f53d) [0x5559784e053d] 10567:20190125:123835.398 10: /usr/sbin/zabbix_server: poller #1 [got 1 values in 0.001649 sec, getting values](lld_update_items+0x395) [0x5559784e3d4c] 10567:20190125:123835.398 9: /usr/sbin/zabbix_server: poller #1 [got 1 values in 0.001649 sec, getting values](lld_process_discovery_rule+0x471) [0x5559784d529d] 10567:20190125:123835.398 8: /usr/sbin/zabbix_server: poller #1 [got 1 values in 0.001649 sec, getting values](zbx_preprocess_item_value+0x94) [0x55597840f6eb] 10567:20190125:123835.398 7: /usr/sbin/zabbix_server: poller #1 [got 1 values in 0.001649 sec, getting values](+0x4d8f0) [0x5559783ce8f0] 10567:20190125:123835.398 6: /usr/sbin/zabbix_server: poller #1 [got 1 values in 0.001649 sec, getting values](poller_thread+0x1b7) [0x5559783cf761] 10567:20190125:123835.398 5: /usr/sbin/zabbix_server: poller #1 [got 1 values in 0.001649 sec, getting values](zbx_thread_start+0x37) [0x555978480fe0] 10567:20190125:123835.398 4: /usr/sbin/zabbix_server: poller #1 [got 1 values in 0.001649 sec, getting values](MAIN_ZABBIX_ENTRY+0x9b3) [0x5559783ba548] 10567:20190125:123835.398 3: /usr/sbin/zabbix_server: poller #1 [got 1 values in 0.001649 sec, getting values](daemon_start+0x31b) [0x5559784731fe] 10567:20190125:123835.398 2: /usr/sbin/zabbix_server: poller #1 [got 1 values in 0.001649 sec, getting values](main+0x312) [0x5559783b9b93] 10567:20190125:123835.398 1: /lib64/libc.so.6(__libc_start_main+0xf5) [0x7fc0b2eab3d5] 10567:20190125:123835.398 0: /usr/sbin/zabbix_server: poller #1 [got 1 values in 0.001649 sec, getting values](+0x37d69) [0x5559783b8d69]
Are you using application discovery ?
just ping and snmp discovery
Hi.
I have a similar problem.
Environment: CentOS 7.2; Database ORACLE.
Upgrade 3.2 to 4.0.3.
zabbix_server [11489]: ERROR [file:lld_item.c,line:3547] Something impossible has just happened. 11489:20190219:203649.368 === Backtrace: === 11489:20190219:203649.368 13: /opt/zabbix/zabbix_server4.0.3/sbin/zabbix_server:* trapper* #7 [processing data](zbx_backtrace+0x2b) [0x49b5fb] 11489:20190219:203649.368 12: /opt/zabbix/zabbix_server4.0.3/sbin/zabbix_server: trapper #7 [processing data](lld_update_items+0x3039) [0x4ea609] 11489:20190219:203649.368 11: /opt/zabbix/zabbix_server4.0.3/sbin/zabbix_server: trapper #7 [processing data](lld_process_discovery_rule+0xc5b) [0x4e09cb] 11489:20190219:203649.368 10: /opt/zabbix/zabbix_server4.0.3/sbin/zabbix_server: trapper #7 [processing data](zbx_preprocess_item_value+0x73) [0x459733] 11489:20190219:203649.368 9: /opt/zabbix/zabbix_server4.0.3/sbin/zabbix_server: trapper #7 [processing data](process_history_data+0x15f) [0x4dd61f] 11489:20190219:203649.368 8: /opt/zabbix/zabbix_server4.0.3/sbin/zabbix_server: trapper #7 [processing data]() [0x4ddea2] 11489:20190219:203649.368 7: /opt/zabbix/zabbix_server4.0.3/sbin/zabbix_server: trapper #7 [processing data]() [0x43df06] 11489:20190219:203649.368 6: /opt/zabbix/zabbix_server4.0.3/sbin/zabbix_server: trapper #7 [processing data](trapper_thread+0x18d) [0x43ec8d] 11489:20190219:203649.368 5: /opt/zabbix/zabbix_server4.0.3/sbin/zabbix_server: trapper #7 [processing data](zbx_thread_start+0x3e) [0x4a459e] 11489:20190219:203649.368 4: /opt/zabbix/zabbix_server4.0.3/sbin/zabbix_server: trapper #7 [processing data](MAIN_ZABBIX_ENTRY+0x72f) [0x420b5f] 11489:20190219:203649.368 3: /opt/zabbix/zabbix_server4.0.3/sbin/zabbix_server: trapper #7 [processing data](daemon_start+0x1a5) [0x49b115] 11489:20190219:203649.368 2: /opt/zabbix/zabbix_server4.0.3/sbin/zabbix_server: trapper #7 [processing data](main+0x3d7) [0x41fc87] 11489:20190219:203649.368 1: /usr/lib64/libc.so.6(__libc_start_main+0xf5) [0x7f4cc2dc0b15] 11489:20190219:203649.368 0: /opt/zabbix/zabbix_server4.0.3/sbin/zabbix_server: trapper #7 [processing data]() [0x41ffad]
command: ldd zabbix_server
linux-vdso.so.1 => (0x00007ffcb13fe000)
libclntsh.so.11.1 => /usr/lib/oracle/11.2/client64/lib/libclntsh.so.11.1 (0x00007fb545636000)
libnnz11.so => /usr/lib/oracle/11.2/client64/lib/libnnz11.so (0x00007fb545268000)
libxml2.so.2 => /usr/lib64/libxml2.so.2 (0x00007fb544eff000)
libnetsnmp.so.31 => /usr/lib64/libnetsnmp.so.31 (0x00007fb544bfd000)
libssh2.so.1 => /usr/lib64/libssh2.so.1 (0x00007fb5449d2000)
libz.so.1 => /usr/lib64/libz.so.1 (0x00007fb5447bc000)
libpthread.so.0 => /usr/lib64/libpthread.so.0 (0x00007fb5445a0000)
libevent-2.0.so.5 => /usr/lib64/libevent-2.0.so.5 (0x00007fb544357000)
libldap-2.4.so.2 => /usr/lib64/libldap-2.4.so.2 (0x00007fb544104000)
liblber-2.4.so.2 => /usr/lib64/liblber-2.4.so.2 (0x00007fb543ef5000)
libcurl.so.4 => /usr/lib64/libcurl.so.4 (0x00007fb543c8c000)
libm.so.6 => /usr/lib64/libm.so.6 (0x00007fb54398a000)
libdl.so.2 => /usr/lib64/libdl.so.2 (0x00007fb543786000)
libresolv.so.2 => /usr/lib64/libresolv.so.2 (0x00007fb54356b000)
libpcre.so.1 => /usr/lib64/libpcre.so.1 (0x00007fb54330a000)
libc.so.6 => /usr/lib64/libc.so.6 (0x00007fb542f49000)
libnsl.so.1 => /usr/lib64/libnsl.so.1 (0x00007fb542d2f000)
libaio.so.1 => /usr/lib64/libaio.so.1 (0x00007fb542b2d000)
liblzma.so.5 => /usr/lib64/liblzma.so.5 (0x00007fb542908000)
libssl.so.10 => /usr/lib64/libssl.so.10 (0x00007fb54269a000)
libcrypto.so.10 => /usr/lib64/libcrypto.so.10 (0x00007fb5422b3000)
/lib64/ld-linux-x86-64.so.2 (0x00007fb547fa6000)
libsasl2.so.3 => /usr/lib64/libsasl2.so.3 (0x00007fb542096000)
libssl3.so => /usr/lib64/libssl3.so (0x00007fb541e53000)
libsmime3.so => /usr/lib64/libsmime3.so (0x00007fb541c2c000)
libnss3.so => /usr/lib64/libnss3.so (0x00007fb541906000)
libnssutil3.so => /usr/lib64/libnssutil3.so (0x00007fb5416d9000)
libplds4.so => /usr/lib64/libplds4.so (0x00007fb5414d5000)
libplc4.so => /usr/lib64/libplc4.so (0x00007fb5412d0000)
libnspr4.so => /usr/lib64/libnspr4.so (0x00007fb541091000)
libidn.so.11 => /usr/lib64/libidn.so.11 (0x00007fb540e5e000)
libgssapi_krb5.so.2 => /usr/lib64/libgssapi_krb5.so.2 (0x00007fb540c12000)
libkrb5.so.3 => /usr/lib64/libkrb5.so.3 (0x00007fb54092c000)
libk5crypto.so.3 => /usr/lib64/libk5crypto.so.3 (0x00007fb5406fa000)
libcom_err.so.2 => /usr/lib64/libcom_err.so.2 (0x00007fb5404f6000)
libcrypt.so.1 => /usr/lib64/libcrypt.so.1 (0x00007fb5402be000)
librt.so.1 => /usr/lib64/librt.so.1 (0x00007fb5400b6000)
libkrb5support.so.0 => /usr/lib64/libkrb5support.so.0 (0x00007fb53fea6000)
libkeyutils.so.1 => /usr/lib64/libkeyutils.so.1 (0x00007fb53fca2000)
libfreebl3.so => /usr/lib64/libfreebl3.so (0x00007fb53fa9e000)
libselinux.so.1 => /usr/lib64/libselinux.so.1 (0x00007fb53f879000)
command: sestatus
SELinux status: disabled
Befor error in the same time in zabbix_server.log i have
[Z3005] query failed: [-1] ORA-04030: out of process memory when trying to allocate 254488 bytes (QERGH hash-agg,kllcqas:kllsltba) [Z3005] query failed: [-1] ORA-04030: out of process memory when trying to allocate 123416 bytes (QERHJ hash-joi,kllcqas:kllsltba)
And in the same time one zabbix_proxy item:
Maximum percentage of time for sending data to the Zabbix server work on 100%
Closed as a duplicate. Please move all discussions to ZBX-15299.
If anyone experience the issue, please double check that schema is correct:
zabbix=> \d+ items_applications;
Table "public.items_applications"
Column | Type | Collation | Nullable | Default | Storage | Stats target | Description
---------------+--------+-----------+----------+---------+---------+--------------+-------------
itemappid | bigint | | not null | | plain | |
applicationid | bigint | | not null | | plain | |
itemid | bigint | | not null | | plain | |
Indexes:
"items_applications_pkey" PRIMARY KEY, btree (itemappid)
"items_applications_1" UNIQUE, btree (applicationid, itemid)
"items_applications_2" btree (itemid)
Foreign-key constraints:
"c_items_applications_1" FOREIGN KEY (applicationid) REFERENCES applications(applicationid) ON DELETE CASCADE
"c_items_applications_2" FOREIGN KEY (itemid) REFERENCES items(itemid) ON DELETE CASCADE
Now I'm running 4.0.10... but the problem still happening.
zabbix=# \d+ items_applications; Table "public.items_applications" Column | Type | Modifiers | Storage | Stats target | Description ---------------+--------+-----------+---------+--------------+------------- itemappid | bigint | not null | plain | | applicationid | bigint | not null | plain | | itemid | bigint | not null | plain | | Indexes: "items_applications_pkey" PRIMARY KEY, btree (itemappid) "items_applications_1" UNIQUE, btree (applicationid, itemid) "items_applications_2" btree (itemid) Has OIDs: no
Your table does not contain foreign keys constraints. You need to add them. Otherwise, the program may work unpredictably
ALTER TABLE ONLY items_applications ADD CONSTRAINT c_items_applications_1 FOREIGN KEY (applicationid) REFERENCES applications (applicationid) ON DELETE CASCADE; ALTER TABLE ONLY items_applications ADD CONSTRAINT c_items_applications_2 FOREIGN KEY (itemid) REFERENCES items (itemid) ON DELETE CASCADE;
Other tables also must be checked.
done!
zabbix=# \d+ items_applications; Table "public.items_applications" Column | Type | Modifiers | Storage | Stats target | Description ---------------+--------+-----------+---------+--------------+------------- itemappid | bigint | not null | plain | | applicationid | bigint | not null | plain | | itemid | bigint | not null | plain | | Indexes: "items_applications_pkey" PRIMARY KEY, btree (itemappid) "items_applications_1" UNIQUE, btree (applicationid, itemid) "items_applications_2" btree (itemid) Foreign-key constraints: "c_items_applications_1" FOREIGN KEY (applicationid) REFERENCES applications(applicationid) ON DELETE CASCADE "c_items_applications_2" FOREIGN KEY (itemid) REFERENCES items(itemid) ON DELETE CASCADE Has OIDs: no
I'll follow the logs one day or so 
Thank you so much!
looks like it fixed the problem indeed... I clean up my database and was able to active all Constraints again... there was some cases where the ID was 0 and I had to update to NULL, e.g: hostid on screen table, I don't know if I messed up it in a moment before, or the upgrade process didn't do this update.
Again thank you so much about this!
ZBX-17116 is similar