• CentOS 7 Linux 3.10.0-327.18.2.el7.x86_64 (Operating System)
• zabbix-proxy-pgsql.x86_64 3.0.3-1.el7 (Zabbix Proxy)
• postgresql95-server.x86_64 9.5.3-2PGDG.rhel7 (Zabbix Proxy database)

Backtrace for easier searching:

  6763:20160621:145857.091 === Backtrace: ===
  6763:20160621:145857.091 20: /usr/sbin/zabbix_proxy: poller #37 [got 1 values in 0.004789 sec, getting values](print_fatal_info+0xae) [0x473b5e]
  6763:20160621:145857.091 19: /usr/sbin/zabbix_proxy: poller #37 [got 1 values in 0.004789 sec, getting values]() [0x473e37]
  6763:20160621:145857.091 18: /usr/lib64/libc.so.6(+0x35670) [0x7ff280d25670]
  6763:20160621:145857.092 17: /usr/lib64/libclntsh.so.11.1(kpuhhaloc+0x3ab) [0x7ff26eb4d911]
  6763:20160621:145857.092 16: /usr/lib64/libclntsh.so.11.1(kpughndl0+0x224) [0x7ff26eba074c]
  6763:20160621:145857.092 15: /usr/lib64/libclntsh.so.11.1(OCIHandleAlloc+0x19) [0x7ff26eb7360f]
  6763:20160621:145857.092 14: /usr/lib/oracle/11.2/client64/lib/libsqora.so.11.1(bcoSQLAllocEnv+0x11e) [0x7ff283792eb6]
  6763:20160621:145857.092 13: /usr/lib/oracle/11.2/client64/lib/libsqora.so.11.1(+0x7fa57) [0x7ff2837eaa57]
  6763:20160621:145857.092 12: /usr/lib/oracle/11.2/client64/lib/libsqora.so.11.1(SQLAllocHandle+0x31) [0x7ff2837eac31]
  6763:20160621:145857.092 11: /usr/lib64/libodbc.so.2(+0xd8dc) [0x7ff282f3d8dc]
  6763:20160621:145857.092 10: /usr/lib64/libodbc.so.2(SQLConnect+0x1a7) [0x7ff282f3faf7]
  6763:20160621:145857.092 9: /usr/sbin/zabbix_proxy: poller #37 [got 1 values in 0.004789 sec, getting values](odbc_DBconnect+0x121) [0x496ca1]
  6763:20160621:145857.092 8: /usr/sbin/zabbix_proxy: poller #37 [got 1 values in 0.004789 sec, getting values](get_value_db+0x130) [0x42f210]
  6763:20160621:145857.092 7: /usr/sbin/zabbix_proxy: poller #37 [got 1 values in 0.004789 sec, getting values]() [0x426e3a]
  6763:20160621:145857.092 6: /usr/sbin/zabbix_proxy: poller #37 [got 1 values in 0.004789 sec, getting values](poller_thread+0x100) [0x427000]
  6763:20160621:145857.092 5: /usr/sbin/zabbix_proxy: poller #37 [got 1 values in 0.004789 sec, getting values](zbx_thread_start+0x45) [0x474835]
  6763:20160621:145857.092 4: /usr/sbin/zabbix_proxy: poller #37 [got 1 values in 0.004789 sec, getting values](MAIN_ZABBIX_ENTRY+0x585) [0x4185c5]
  6763:20160621:145857.092 3: /usr/sbin/zabbix_proxy: poller #37 [got 1 values in 0.004789 sec, getting values](daemon_start+0x1c0) [0x473400]
  6763:20160621:145857.092 2: /usr/sbin/zabbix_proxy: poller #37 [got 1 values in 0.004789 sec, getting values](main+0x381) [0x417371]
  6763:20160621:145857.092 1: /usr/lib64/libc.so.6(__libc_start_main+0xf5) [0x7ff280d11b15]
  6763:20160621:145857.092 0: /usr/sbin/zabbix_proxy: poller #37 [got 1 values in 0.004789 sec, getting values]() [0x4177ad]

Related issues: ZBX-9009 and ZBX-10897.

It may be crashing because the proxy has not managed to sync configuration with the server yet and so it still thinks that ODBC items are enabled. You could probably work around this by either (1) a direct SQL query on the proxy database or (2) disabling pollers temporarily, waiting until it syncs with server, and enabling the pollers again.

Could you please share your odbc.ini and odbcinst.ini files? Are you using any interesting settings for Oracle's ODBC?

I just uploaded the files. No, I'm not using anything unusual, at least that I can think of. My other proxy works fine with the same ODBC items (I created a template). if you want, I can upload the template for you to check the selects.

The Unix ODBC version on the crashing proxy is:

• unixODBC.x86_64 2.3.1-11.el7

The other proxy (not crashing)

• CentOS 6 Linux 2.6.32-573.el6.x86_64 (Operating System)
• zabbix-proxy-pgsql.x86_64 3.0.3-1.el6 (Zabbix Proxy)
• postgresql95-server.x86_64 9.5.3-2PGDG.rhel6 (Zabbix Proxy Dataase)
• unixODBC.x86_64 2.2.14-14.el6 (Unix ODBC version)

The oracle client version on both proxies is the 11.2.0.4.0-1

"... (2) disabling pollers temporarily, waiting until it syncs with server, and enabling the pollers again."

I did that and the proxy stopped crashing, thank you for the suggestion.
Now I just need to know why it crashes in the first place, cause database monitoring is a must for my environment.

As a workaround for ODBC checks I would suggest a separate proxy on a different DB engine (SQLite perhaps?). There are problems when Zabbix performs ODBC checks for a database with the same engine as Zabbix own DB. It may or may not work depending on DB client library, UnixODBC and DB connector library versions.

Note that in this case, unlike ZBX-10897, Zabbix backend is PostgreSQL.

Hm... Sorry, didn't notice that. It is very strange.

Note that it crashes many-many times with the same message:

Got signal [signal:11(SIGSEGV),reason:1,refaddr:0x68]. Crashing ...

Always 0x68... It does not look like dereference of uninitialized pointer, more like dereferencing of a non-pointer object. Could it be a binary compatibility issue between UnixODBC and Oracle?

Actually, the message suggests that this is a duplicate of ZBX-9009.

It could be a NULL-pointer dereference. For instance, suppose there is a large struct X with many fields. If we access x->field, where x is NULL and fields's offset in the structure is 0x68, then such a segfault is possible.

For the record:
A combination of CentOS 5 and Oracle Instant-Client 11.2 leads to a crashing Zabbix proxy as well. Well, actually it did not crash but shuts itself down.
This is most likely due to an incompatibility of the Instant-Client with the quite old unixODBC version distributed with CentOS 5. Haven't investigated this though.

[...]
zabbix_proxy: poller #3 [got 0 values in 0.000003 sec, getting values]: symbol lookup error: /usr/lib/oracle/11.2/client/lib/libsqora.so.11.1: undefined symbol: SQLGetP
rivateProfileStringW
  2107:20160623:145931.434 One child process died (PID:2114,exitcode/signal:127). Exiting ...
  2107:20160623:145933.444 syncing history data...
  2107:20160623:145933.445 syncing history data done
  2107:20160623:145933.446 Zabbix Proxy stopped. Zabbix 2.2.7 (revision 50148).

Using a 10.2 Instant-Client instead appears to be stable.

Maybe worth to try here different combinations of involved components too.

"Maybe worth to try here different combinations of involved components too."

I installed zabbix proxy from repo.zabbix.com using yum. So unixODBC is installed as a dependency, and when I try to uninstall it, in order to install a different version, zabbix proxy will uninstall itself cause of missing dependencies. What do you recommend to work around this? Because as my other proxy is working just fine, I also believe some incompatibilities between versions may be causing this issue.

Also thank you for your help so far.

Well, while changing the used unixODBC version (implies custom compilation of Zabbix proxy or changing the OS level) one could also think of using different versions of Oracle libraries.
Anyway, I haven't followed this issue in detail and can't judge whether changing involved components is a valid suggestion in the end.

Dear diego_souza, do you have just one entry in odbcinst.ini or is it just the problematic one? Is there any chance you have two Oracle ODBC drivers configured?

No, not a chance. I have just one Oracle ODBC driver configured. The entry is in the file I uploaded. I even commented the mysql and postgresql entries, but no change came of that.

Guys, this has to have to do (is that correct?? not native English speaker) with compatibility issues between different version of packages. I installed another proxy in another VM to monitor the same host. I used CentOS 6.8, installed everything in the same exact way as I installed the crashing proxy (CentOS 7), copied the configuration files from the crashing proxy to the new, and changed the host from being monitored by the crashing proxy to being monitored by the new proxy.

And it's working!

As it's a CentOS 6.8, the only difference is that unixODBC comes in version 2.2.14, instead of version 2.3.1. The thing is: in both proxies I can connect and query the databases using "isql -v DSN" without problems.

Maybe zabbix doesn't like unixODBC 2.3.1??

As Marc suggested, I'll try to install a different version of oracle instant client in the crashing proxy. I'll post the results when I do it.

Glad to hear that you've found a workaround! And thank you for keeping experimenting!

We have a bunch of issues with unixODBC, most of them boil down to the fact that Zabbix and ODBC driver link to incompatible versions of same DB client library which cannot be both loaded in runtime. But yours is different, your backend DB and monitored DB have different engines. One of two: either Zabbix does not like unixODBC 2.3.x or a particular ODBC driver.

There were significant changes in unixODBC between 2.2.14 and 2.3.x, they even changed library so version. They also split the project into three and migrated from CVS to SVN at the same which makes investigation even harder. Their ChangeLog is scarce, but there are interesting things in it:

• 28.Nov.2011 2.3.1 Released
  • ...
  • Major change is to change the library version number from 1 to 2 to signal the SQLLEN change for 64 land. Should have been done for 2.3.0, but better late than never. So if after installing you have apps that can't find libodbc.so, its likely they are linked to libodbc.so.1, so just create a symlink from libodbc.so.2
    • ...
    • Change type definition of a integer in SQLConnect.c, just to avoid confusion
    • ...
    • Change lib version to 2 to reflect SQLLEN changes in v2.3
• 20.April.2010 2.3.0 Released
  • New release, new major version. This is now the reduced unixODBC, after the GUI and additional driver parts have been split off to their own project.
  • The new minor number is a indication of the change to the new default SQLLEN size for 64 bit platfoems so projects using unixODBC can have a hope of distinguising between new and old.
    • ...
    • Create SVN repository at sourceforge
    • ...
    • Change minor version number because of the SQLLEN change

Also, header diff reveals interesting things, search for #if (SIZEOF_LONG == 8).

• 2.2.4

  /*
  	 * I (Nick) have made these changes, to cope with the new 3.52 MS
  	 * changes for 64 bit ODBC, but looking at MS's spec they havn't
  	 * finished it themself. For example, SQLBindCol now expects the
  	 * indicator variable to be a SQLLEN which then is a pointer to
  	 * a 64 bit value. However the online book that comes with the
  	 * headers, then goes on to describe the indicator_ptr in the
  	 * descriptor record (which is set by SQLBindCol) as a pointer
  	 * to a SQLINTEGER (32 bit). So I don't think its ready for the 		
  	 * big time yet. Thats not to mention all the ODBC apps on 64 bit
  	 * platforms that this would break...
  	 *
  	 * I have just discovered that on win64 sizeof(long) == 4, so its
  	 * all smoke and mirrors...
  	 *
  	 */
  

• 2.3.0

  /*
  	 * Hopefully by now it should be safe to assume most drivers know about SQLLEN now	
  	 * and the defaukt is now sizeof( SQLLEN ) = 8 on 64 bit platforms	
  	 *	
  	 */
  

Probably, playing around with BUILD_LEGACY_64_BIT_MODE can make 2.3.x work with Zabbix...

Update: slencheck is a utility which attempts to check whether an ODBC driver was built with 32-bit or 64-bit SQLLEN types, try it.

ZBXNEXT-1714 might solve all this headache - at some point.

Glebs, for what I can see, "slencheck" is only available from 2.3.2 forward, and my version is 2.3.1. Since unixODBC is installed as a dependency of the zabbix-proxy-pgsql package, how can I update or downgrade without causing problems to zabbix? Can I remove it with "rpm -e --nodeps" and install the latest or older version from source, without breaking zabbix proxy installation?

I'm not an expert on administrating things Perhaps you can install unixODBC 2.3.2 and your ODBC driver on a separate system without touching Zabbix?

Currently I am running some tests with MySQL Connector ODBC, I'm going to do test Oracle drivers on Monday.

OK, thanks for the testing. I'll figure this out and post the results as soon as I do it.

See officially recommended unixODBC versions for Oracle ODBC drivers.

Installed the Oracle Client 12.1 as suggested by Marc and by the link above, but to no avail. The proxy keeps crashing.

Glebs,

I managed to install the unixODBC 2.3.4 from source, with the default configure options, in the crashing proxy, but the crashes continue to happen.

that is the result of the "slencheck" utility:

slencheck bi2
slencheck: sizeof(SQLLEN) == 8
slencheck: driver manager and driver matched

Do you know what that means? And what do you mean with "playing with BUILD_LEGACY_64_BIT_MODE"? I noticed that this option is present in a few configure files.

Glad if you can help me.

What is current configuration of the crashing proxy (unixODBC version, Oracle driver version, Oracle client version)? As far as I understand proxy is still from repositories. Maybe it's worth recompiling it with unixODBC 2.3.4? slencheck output gives hope that everything should work. If it's still crashing, could you show a more recent backtrace?

• Oracle client and odbc driver version is now 12.1.0.2.0-1
• unixODBC version is 2.3.4
• Zabbix Proxy still the same.

What is the difference if I recompile it? I ran this:

yum deplist zabbix-proxy-pgsql | grep -i odbc
dependency: libodbc.so.2()(64bit)
provider: unixODBC.x86_64 2.3.1-11.el7

from the output of this command, this lib is the only dependency that zabbix needs from the unixODBC package. In the compilation process, does zabbix links to this lib in a way that it doesn't matter if I just change the version of the lib? I really don't know this, so glad if you could answer.

 31393:20160704:175820.738 === Backtrace: ===
 31393:20160704:175820.739 20: /usr/sbin/zabbix_proxy: poller #14 [got 1 values in 0.009260 sec, getting values](print_fatal_info+0xae) [0x473b5e]
 31393:20160704:175820.739 19: /usr/sbin/zabbix_proxy: poller #14 [got 1 values in 0.009260 sec, getting values]() [0x473e37]
 31393:20160704:175820.739 18: /usr/lib64/libc.so.6(+0x35670) [0x7f7a9e800670]
 31393:20160704:175820.739 17: /usr/lib64/libclntsh.so.12.1(kpuhhaloc+0x38b) [0x7f7a8bfe5bfb]
 31393:20160704:175820.739 16: /usr/lib64/libclntsh.so.12.1(kpughndl0+0x2fe) [0x7f7a8c041cae]
 31393:20160704:175820.739 15: /usr/lib64/libclntsh.so.12.1(OCIHandleAlloc+0x1b) [0x7f7a8c01321b]
 31393:20160704:175820.739 14: /usr/lib/oracle/12.1/client64/lib/libsqora.so.12.1(bcoSQLAllocEnv+0x136) [0x7f7a8edb1e46]
 31393:20160704:175820.739 13: /usr/lib/oracle/12.1/client64/lib/libsqora.so.12.1(+0x1a5220) [0x7f7a8ee61220]
 31393:20160704:175820.739 12: /usr/lib/oracle/12.1/client64/lib/libsqora.so.12.1(SQLAllocHandle+0x43) [0x7f7a8ee610e3]
 31393:20160704:175820.739 11: /usr/lib64/libodbc.so.2(+0xd98c) [0x7f7aa0a1898c]
 31393:20160704:175820.739 10: /usr/lib64/libodbc.so.2(SQLConnect+0x195) [0x7f7aa0a1aa85]
 31393:20160704:175820.739 9: /usr/sbin/zabbix_proxy: poller #14 [got 1 values in 0.009260 sec, getting values](odbc_DBconnect+0x121) [0x496ca1]
 31393:20160704:175820.739 8: /usr/sbin/zabbix_proxy: poller #14 [got 1 values in 0.009260 sec, getting values](get_value_db+0x130) [0x42f210]
 31393:20160704:175820.739 7: /usr/sbin/zabbix_proxy: poller #14 [got 1 values in 0.009260 sec, getting values]() [0x426e3a]
 31393:20160704:175820.739 6: /usr/sbin/zabbix_proxy: poller #14 [got 1 values in 0.009260 sec, getting values](poller_thread+0x100) [0x427000]
 31393:20160704:175820.739 5: /usr/sbin/zabbix_proxy: poller #14 [got 1 values in 0.009260 sec, getting values](zbx_thread_start+0x45) [0x474835]
 31393:20160704:175820.739 4: /usr/sbin/zabbix_proxy: poller #14 [got 1 values in 0.009260 sec, getting values](MAIN_ZABBIX_ENTRY+0x585) [0x4185c5]
 31393:20160704:175820.739 3: /usr/sbin/zabbix_proxy: poller #14 [got 1 values in 0.009260 sec, getting values](daemon_start+0x1c0) [0x473400]
 31393:20160704:175820.739 2: /usr/sbin/zabbix_proxy: poller #14 [got 1 values in 0.009260 sec, getting values](main+0x381) [0x417371]
 31393:20160704:175820.739 1: /usr/lib64/libc.so.6(__libc_start_main+0xf5) [0x7f7a9e7ecb15]
 31393:20160704:175820.739 0: /usr/sbin/zabbix_proxy: poller #14 [got 1 values in 0.009260 sec, getting values]() [0x4177ad]

In compiling and linking process Zabbix includes library headers and links to a particular version of the library. What does ldd zabbix_proxy | grep libodbc show?

Headers saw some changes between 2.2.14 and 2.3.x (involving SQLLEN).

Backtrace didn't change significantly. Is it still crashing with refaddr:0x68?

$ ldd /usr/sbin/zabbix_proxy | grep libodbc
	libodbc.so.2 => /lib64/libodbc.so.2 (0x00007f681056f000)

/lib64/libodbc.so.2 is a link to the lib installed by the unixODBC package.

and yes, it's still crashing with refaddr:0x68

So, zabbix_proxy still loads unixODBC 2.3.1 then?

Well, I've run out of ideas. Further investigation will need adding more debug logging which will require recompilation of Zabbix anyway...

no, when I uninstalled 2.3.1, it removed the lib and the link. When I installed the 2.3.4, I recreated the link to the new lib.

So...

here's the thing:

• I uninstalled the unixODBC 2.3.4, and installed the 2.2.14.
• Uninstalled the Oracle client and odbc driver 12.1 and installed the 11.2 again.
• Recreated the links to all the dependencies and libs needed, pointing to the new path, and the outcome is:

 24771:20160705:114321.427 === Backtrace: ===
 24771:20160705:114321.427 20: /usr/sbin/zabbix_proxy: poller #2 [got 22 values in 0.058808 sec, getting values](print_fatal_info+0xae) [0x473b5e]
 24771:20160705:114321.427 19: /usr/sbin/zabbix_proxy: poller #2 [got 22 values in 0.058808 sec, getting values]() [0x473e37]
 24771:20160705:114321.427 18: /usr/lib64/libc.so.6(+0x35670) [0x7f9e53b3f670]
 24771:20160705:114321.427 17: /usr/lib64/libclntsh.so.11.1(kpuhhaloc+0x3ab) [0x7f9e41967911]
 24771:20160705:114321.427 16: /usr/lib64/libclntsh.so.11.1(kpughndl0+0x224) [0x7f9e419ba74c]
 24771:20160705:114321.427 15: /usr/lib64/libclntsh.so.11.1(OCIHandleAlloc+0x19) [0x7f9e4198d60f]
 24771:20160705:114321.428 14: /usr/lib/oracle/11.2/client64/lib/libsqora.so.11.1(bcoSQLAllocEnv+0x11e) [0x7f9e565abeb6]
 24771:20160705:114321.428 13: /usr/lib/oracle/11.2/client64/lib/libsqora.so.11.1(+0x7fa57) [0x7f9e56603a57]
 24771:20160705:114321.428 12: /usr/lib/oracle/11.2/client64/lib/libsqora.so.11.1(SQLAllocHandle+0x31) [0x7f9e56603c31]
 24771:20160705:114321.428 11: /usr/lib64/libodbc.so.2(+0xea94) [0x7f9e55d58a94]
 24771:20160705:114321.428 10: /usr/lib64/libodbc.so.2(SQLConnect+0x167) [0x7f9e55d59877]
 24771:20160705:114321.428 9: /usr/sbin/zabbix_proxy: poller #2 [got 22 values in 0.058808 sec, getting values](odbc_DBconnect+0x121) [0x496ca1]
 24771:20160705:114321.428 8: /usr/sbin/zabbix_proxy: poller #2 [got 22 values in 0.058808 sec, getting values](get_value_db+0x130) [0x42f210]
 24771:20160705:114321.428 7: /usr/sbin/zabbix_proxy: poller #2 [got 22 values in 0.058808 sec, getting values]() [0x426e3a]
 24771:20160705:114321.428 6: /usr/sbin/zabbix_proxy: poller #2 [got 22 values in 0.058808 sec, getting values](poller_thread+0x100) [0x427000]
 24771:20160705:114321.428 5: /usr/sbin/zabbix_proxy: poller #2 [got 22 values in 0.058808 sec, getting values](zbx_thread_start+0x45) [0x474835]
 24771:20160705:114321.428 4: /usr/sbin/zabbix_proxy: poller #2 [got 22 values in 0.058808 sec, getting values](MAIN_ZABBIX_ENTRY+0x585) [0x4185c5]
 24771:20160705:114321.428 3: /usr/sbin/zabbix_proxy: poller #2 [got 22 values in 0.058808 sec, getting values](daemon_start+0x1c0) [0x473400]
 24771:20160705:114321.428 2: /usr/sbin/zabbix_proxy: poller #2 [got 22 values in 0.058808 sec, getting values](main+0x381) [0x417371]
 24771:20160705:114321.428 1: /usr/lib64/libc.so.6(__libc_start_main+0xf5) [0x7f9e53b2bb15]
 24771:20160705:114321.428 0: /usr/sbin/zabbix_proxy: poller #2 [got 22 values in 0.058808 sec, getting values]() [0x4177ad]

So now I have a CentOS 7 crashing proxy with the same versions of packages and libs that I have on the CentOS 6 proxy that is working OK. I got lost here. I really don't know what is causing that. I'll try to get some more help to debug this issue and post here anything that I find. Let me know if you have any new ideas.

First things first:

I FOUND THE SOLUTION!!!!!!

The explanation:

Turns out the issue had nothing to do with unixODBC version, Oracle Client version or any other package. The problem was one change made from sysVinit (CentOS 6) to systemd (CentOS 7).

Now the details:

I have to create a file /etc/sysconfig/zabbix-proxy with the environment variables pointing to the oracle driver and libs:

ORACLE_HOME=/usr/lib/oracle/11.2/client64
LD_LIBRARY_PATH=$ORACLE_HOME/lib:/usr/lib64
TNS_ADMIN=$ORACLE_HOME/network/admin
PATH=$PATH:$ORACLE_HOME/bin

export ORACLE_HOME
export LD_LIBRARY_PATH
export TNS_ADMIN
export PATH

This works on CentOS 6, because of the way zabbix-proxy service reads this file on initialization, which is the following:

$ cat /etc/init.d/zabbix-proxy | grep sysconfig
if [ -f /etc/sysconfig/zabbix-proxy ]; then
    . /etc/sysconfig/zabbix-proxy

it "sources" the file, so bash interprets it and expand the variables.

Now, when systemd service starts, it reads the file as text. I took a long time to find this info on the systemd manuals. the file is like that:

$ cat /usr/lib/systemd/system/zabbix-proxy.service | grep sysconfig
EnvironmentFile=-/etc/sysconfig/zabbix-proxy

So if I put the same content that works on CentOS 6, this time the bash won't interpret the file, and the env var will not be expanded, causing the Proxy to crash cause it doesn't understand what the hell is $ORACLE_HOME or $PATH. I got it working by replacing the content of /etc/sysconfig/zabbix-proxy for this:

ORACLE_HOME=/usr/lib/oracle/12.1/client64
LD_LIBRARY_PATH=/usr/lib/oracle/12.1/client64/lib:/usr/lib64
TNS_ADMIN=/usr/lib/oracle/12.1/client64/network/admin
PATH=/usr/lib/oracle/12.1/client64/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin

and it works with unixODBC 2.2.14, 2.3.1, 2.3.4, Oracle Client 11.2, 12.1...

I'm currently monitoring Oracle via ODBC in the (former) crashing proxy without issues.

Now, I believe it's worth documenting this info at zabbix official documentation. Maybe add an error message to point at this specific issue?!?!

Anyway, thank you all for your help!!!

Great! I am very glad that the issue got resolved eventually. I am also glad that there was nothing to fix in Zabbix this time.

Since the root cause of the problem was in CentOS subtleties and the key to the solution was in proper Oracle configuration I don't think this topic belongs to our documentation, otherwise it will become littered with information not directly related to Zabbix. zabbix.org seems to be a more appropriate place. There are a lot of articles like "Install Zabbix on a specific OS", "Zabbix and specific DB", "Monitoring this and that with Zabbix". I encourage you to share your findings there. Also, this way you will be able to receive credits from the grateful community. And we will put a link to your article here.

I am closing the issue as Won't Fix if you don't mind.