[ZBXNEXT-2485] Single out ODBC pollers in special poller type Created: 2014 Sep 29  Updated: 2022 Jan 25  Resolved: 2022 Jan 25

Status: Closed
Project: ZABBIX FEATURE REQUESTS
Component/s: Documentation (D), Proxy (P), Server (S)
Affects Version/s: 2.2.6, 2.4.0
Fix Version/s: 6.0.0beta2, 6.0 (plan)

Type: Change Request Priority: Major
Reporter: Vadim Nesterov Assignee: Dmitrijs Goloscapovs
Resolution: Fixed Votes: 30
Labels: database, odbc, oracle, proxy, server
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified
Environment:

UnixODBC


Issue Links:
Duplicate
Sprint: Sprint 84 (Jan 2022)
Story Points: 1

 Description   

The problem is: ODBC checks works in usual poller process, so if we have 50 pollers and a lot of ODBC checks, 50 connections to database can be created. DBAs are very unhappy about this situation.

To solve:
1. single out odbc pollers in separate type of poller processes such as snmp, http, ipmi pollers.
2. Add to config number of ODBC pollers, timeout connection setting for a DSN

The connection to database by default must be open if process is running or
timeout is not expired, because there are can be very rare db checks, it is cheaper to open a new connection, than hold it.

Connection must supported on DSN string, one DSN = one opened connection pointer.

For example Oracle ODBC driver for Linux doesn't support shared connections, ODBC Pooling = yes and CPTimeout, zabbix crashes with these settings.

So the connection persistence must be realized on process level.

db.select[select_name,dsn,persistence=yes]

persistence=yes - by default this parameter is yes, and poller uses opened connection to db. If db.select[,,no] - poller must open new connection and close it after query.

It is needed rarely, but for special session queries.



 Comments   
Comment by Aleksandrs Saveljevs [ 2014 Sep 29 ]

A slightly related issue might be ZBXNEXT-2200, which talks about having a configuration parameter which specifies how many pollers may access a given host at any given time.

Comment by Marc [ 2014 Sep 29 ]

I can also think of pooling SQL queries to be issued over a single TCP connection.

Btw, for now, when it comes to serious database monitoring, which means doing significantly more than a couple of queries, then DBforBIX could be your (and probably your DBAs) friend.
You loose some flexibility and comfort since it works autonomous and makes use of Zabbix trapper protocol. On the other hand it allows to avoid putting credentials into Zabbix or on Zabbix server/proxy host which then could, under certain circumstances, be exposed pretty easy (ZBX-8464, ZBX-4916, ...)
Depending on implementation ZBXNEXT-1660 might help but is not expected to happen sooner or later

Comment by Vadim Nesterov [ 2014 Sep 29 ]

Marc, thanks but DBforBIX is terrible, yes as you mention for its trapper type, and the main thing - it is difficult to do LLD, we have hundreds of tables and jobs, and I don't want to generate such config for DBforBIX.
ODBC persistence should be realized right in zabbix, I think it is good way. What about securing connection - it is not problem you may do this in different way: vpn, ssh and so on.

Comment by Marc [ 2014 Sep 29 ]

I see. My concerns regarding security are not about the connectivity, since there are as already mentioned by you ways to work around that.
Anyhow, I totally agree that improving Zabbix its database monitoring capabilities is the way to go The recently added ODBC discovery function is a first step.

Since I don't believe there will be session pooling for any connection type in the future (everything is about trapping or polling but not about persistent connections), Zabbix can't reach efficiency of applications like DBforBIX in that domain.

Although that might possibly not be necessary, even when planning to monitor thousands of database (instances).
Pooling of queries and querying them in a bulk should do. At least I hope so, otherwise we're still bound to 3rd party products like DBforBIX or Oracle Enterprise Manager, what is not my preference

Comment by Vadim Nesterov [ 2014 Sep 29 ]

Let's wait for developers thoughts

Comment by Vadim Nesterov [ 2014 Nov 12 ]

Any comment from developers? Can be this feature planned to 2.5 version?

Comment by Marc [ 2015 Aug 13 ]

ZBXNEXT-2902 asks for persistent sessions in context of loadable modules support of alerter process.

@nucleusv
It seams not to be on the road map yet but there is always the fast lane over (co-)sponsoring development.

Comment by Marc [ 2016 Mar 10 ]

ZBXNEXT-3188 is similar but more general minded.

Comment by Vadim Nesterov [ 2016 Mar 15 ]

@Marc, ZBX's which you've provided I think they it is another story...

Comment by Aleksandrs Saveljevs [ 2016 Apr 12 ]

Related request: ZBXNEXT-3239.

Comment by Glebs Ivanovskis (Inactive) [ 2016 Jul 01 ]

Dear nucleusv, can you suggest some more reading on this statement?

For example Oracle ODBC driver for Linux doesn't support shared connections, ODBC Pooling = yes and CPTimeout, zabbix crashes with these settings.

Comment by Vadim Nesterov [ 2016 Jul 02 ]

Glebs, what do you mean some more reading ?

Comment by Glebs Ivanovskis (Inactive) [ 2016 Jul 04 ]

I mean... Is there a related piece of Oracle documentation or some expert's blog/article/paper/presentation covering this topic (this one I've already seen )? Please point me to it. Or is it, perhaps, a well known fact among Oracle administrators?

Comment by Anton Alekseev [ 2016 Sep 05 ]

glebs.ivanovskis, maybe I can help on this matter.

As of release 2.0.0 of unixODBC the driver supports connection pooling.
It works on per-process basis by leaving connection open so application process may reuse the connection on the next try.
Details can be found at http://www.unixodbc.org/doc/conn_pool.html

There are couple of problems on using ODBC Pooling with Zabbix:
1. As it has already been said, it works on per-process basis. So if Zabbix server once asked ODBC for connection to DB #1 via poller #1 and the next time it asks ODBC for connection to DB #1 via poller #2, unixODBC will open new connection, since pollers #1 and #2 are different processes. So at the worst case we will have (count of zabbix_server pollers) connections open on every monitored DB. It makes (count of monitored DB) x (count of zabbix_server pollers) open connections, which are a lot!
2. As Vadim Nesterov has said, Oracle ODBC driver doesn't support pooling.

Comment by Glebs Ivanovskis (Inactive) [ 2016 Sep 05 ]

akint, thank you for response! Unfortunately, very little new information for me. What I can't understand is if pooling is done on UnixODBC side why Oracle driver needs to support pooling too? Or do you mean that combination "Zabbix + Oracle driver + pooling" leads to crashes?

Comment by Vadim Nesterov [ 2016 Sep 05 ]

@Anton Alekseev,

> So at the worst case we will have (count of zabbix_server pollers) connections open on every monitored DB. It makes (count of monitored DB) x (count of zabbix_server pollers) open connections, which are a lot!

I don't suggest to go this way. I suggest to make a special process type -> DB poller (number of such processes are set up in config), like snmp pollers and etc.

> Or do you mean that combination "Zabbix + Oracle driver + pooling" leads to crashes?
When I worked with this this issue, I had Oracle 11 drivers, they didn't support pooling and it was zabbix crashing.

Comment by Anton Alekseev [ 2016 Sep 06 ]

glebs.ivanovskis, when you enable ODBC Pooling for Oracle driver by setting non-zero CPTimeout, any application, which uses this driver, crashes with segmentation fault.
I have tested it with the every combination of:

  • drivers: Oracle 11 and 12
  • unixODBC: 2.2.14 (default in EL6) and 2.3.4 (most recent, compiled from source).
  • applications: zabbix_server and simple Python script using pyodbc.

nucleusv I'm agree on your proposal, I have just tried to point out that using unixODBC Pooling would be inefficient even if it worked. =)

Comment by Fabian van der Hoeven [ 2018 Jan 16 ]

I would love something like connection pooling in the zabbix-server. I've written a sort of proxy (between zabbix-server/proxy and zabbix-agent) to catch DB requests and provide connection pooling. It forwards every other request to the normal zabbix-agent.
(if you're interested: https://github.com/Fabian1976/mios-agent)
I haven't updated or used it in a while, so I don't know if it still functions in Zabbix 3.2 or higher.

Comment by Marc [ 2018 Jan 16 ]

Since having made good experience by installing PgBouncer alongside the Zabbix frontend to pool its database connections, I'm seriously thinking of using PgBouncer alongside Zabbix proxies for pooling ODBC connections too.

Possibly, using a 3rd party pooling application could be a option too, if such kind of pooling software exists for all supported RDBMS platforms, of course. This may come at the expense of having another configuration to maintain. On the other hand, the odbci.ini has to be configured anyway. Last but not least, tailored pooling software promises to provide the best implementation for the respective RDBMS.

Comment by Dmitrijs Goloscapovs [ 2022 Jan 07 ]

Available in versions:
  - pre-6.0.0beta2 - 56f98dcb04, 023f240bae3

Documentation updated:

Comment by Alexei Vladishev [ 2022 Jan 11 ]

This functionality is coming in Zabbix 6.0 LTS in a few weeks. Meanwhile you may preview it in the latest beta or RC releases.

Generated at Fri Apr 19 01:41:58 EEST 2024 using Jira 9.12.4#9120004-sha1:625303b708afdb767e17cb2838290c41888e9ff0.