Hello!

Has the same issue occuring on older Zabbix versions or you face it on 5.4.6 only?

Hi, thanks alot 

actually, it's the first time we use the Zabbix proxy so yes. before we only used the Zabbix server. 

Hi , 
any update about the issue? 

Hello!
Could you provide debug level logs for server trapper processes and proxy data sender process?

please find attached logs of proxy with debug level 5 
265:20211111:140129.813 proxy_get_history_data() 2 record(s) missing. Waiting 0.100000 sec, retrying.
   265:20211111:140129.916 proxy_get_history_data() 2 record(s) missing. No more retries.
   265:20211111:140129.916 proxy_get_history_data() 2 record(s) missing. Waiting 0.100000 sec, retrying.
   265:20211111:140130.019 proxy_get_history_data() 2 record(s) missing. No more retries.
   265:20211111:140130.019 proxy_get_history_data() 2 record(s) missing. Waiting 0.100000 sec, retrying.
   265:20211111:140130.122 proxy_get_history_data() 2 record(s) missing. No more retries.
   265:20211111:140130.122 proxy_get_history_data() 2 record(s) missing. Waiting 0.100000 sec, retrying.
   265:20211111:140130.224 proxy_get_history_data() 2 record(s) missing. No more retries.
   265:20211111:140130.224 proxy_get_history_data() 2 record(s) missing. Waiting 0.100000 sec, retrying.
   265:20211111:140130.327 proxy_get_history_data() 2 record(s) missing. No more retries.
   265:20211111:140130.327 proxy_get_history_data() 2 record(s) missing. Waiting 0.100000 sec, retrying.
   265:20211111:140130.430 proxy_get_history_data() 2 record(s) missing. No more retries.
   265:20211111:140130.430 proxy_get_history_data() 2 record(s) missing. Waiting 0.100000 sec, retrying.
   265:20211111:140130.532 proxy_get_history_data() 2 record(s) missing. No more retries.
   265:20211111:140130.532 proxy_get_history_data() 2 record(s) missing. Waiting 0.100000 sec, retrying.
   265:20211111:140130.635 proxy_get_history_data() 2 record(s) missing. No more retries.
   265:20211111:140130.660 proxy_get_history_data() 2 record(s) missing. Waiting 0.100000 sec, retrying.
   265:20211111:140130.762 proxy_get_history_data() 2 record(s) missing. No more retries.
   265:20211111:140130.762 proxy_get_history_data() 2 record(s) missing. Waiting 0.100000 sec, retrying.
those logs that i keep seeing and i think its related 

once i run single instance of mariadb it works fine 

any update on this issue ? 

Tried to reproduce on Galera 3-nodes Master-Master Cluster - no issues in 6.0/6.2

Zabbix 5.4 is an unsupported version. If the problem is actually on 6.0/6.2 - please, notify us with the exact reproducing  steps

I hit exactly this issue with 6.2.6, against a 3-node MariaDB 10.5.18 Galera cluster, using an identical config for my servers. The servers run fine, but the proxies deliver a steady 10 items / second, which causes the proxy queues to rapidly fill up. After confirming that the bottleneck wasn't in the network or the databases, increasing the logging level for the data sender process revealed logs identical to the above - 10 sets of:

2 record(s) missing. No more retries.

2 record(s) missing. Waiting 0.100000 sec, retrying.

After which it appears to send 10 records. I've replaced the databases with single instances now and the issue has disappeared, but it was consistently recreateable until I dropped Galera. If others can't recreate it, I'll spin up some more test instances and get more complete logs.

Please, provide the details about your Galera cluster details as well - configuration, what have you using for requests re-direction in case of failover

Greetings!

We see the same issue with a MariaDB Galera setup.

The proxy is pulling data from its agents (5k+ items), but never sending it to the server.

Zabbix-Proxy: 6.4.13-1

MariaDB Server: 10.11.7

Even if there is just one node in the cluster - we hit this error: (at DebugLevel=4)

> journalctl -u zabbix-proxy.service -f | grep proxy_get_history_data

> proxy_get_history_data() 1 record(s) missing. No more retries

We traced its source to this area in the code: https://github.com/zabbix/zabbix/tree/master/src/libs/zbxproxybuffer

I think it ocurrs at pb_history.c => pb_history_get_rows_db

It feels like the proxy is stuck in an endless retry loop. Maybe because of some insert-delay caused by Galera? (as far as I understand it: proxy writes data to db and checks that all records that were written, do actually exist - before sending them to the server)

See also the comment inside the code:

> At least one record is missing. It can happen if some DB syncer process has started but not yet committed a transaction or a rollback occurred in a DB syncer.

For now we rolled-back the proxy to a standalone MariaDB instance as the data of the proxy is not (that) important.