[ZBX-10727] Proxy problem - Message from server is shorter than expected - Message ignored. Created: 2016 Apr 29 Updated: 2017 Jun 29 Resolved: 2016 May 06 |
|
Status: | Closed |
Project: | ZABBIX BUGS AND ISSUES |
Component/s: | Proxy (P), Server (S) |
Affects Version/s: | 3.0.2 |
Fix Version/s: | 2.0.18rc1, 2.2.13rc1, 3.0.3rc1, 3.2.0alpha1 |
Type: | Incident report | Priority: | Critical |
Reporter: | Bence Balog | Assignee: | Unassigned |
Resolution: | Fixed | Votes: | 0 |
Labels: | timeout | ||
Remaining Estimate: | Not Specified | ||
Time Spent: | Not Specified | ||
Original Estimate: | Not Specified | ||
Environment: |
Zabbix server and proxy 3.0.2 |
Issue Links: |
|
Description |
After upgrading our 2.4.7 environment to 3.0.2 two proxies are not receiving correct data from the server, therefor no data received from those proxies. Error messages in the log (zabbix-proxy): From the server logs Heartbeat messages are OK, from the GUI the "last seen" value is a few seconds. There are other proxies on the local network and some remote ones via ipsec vpn which are working properly. Before the upgrade everything was fine. |
Comments |
Comment by Aleksandrs Saveljevs [ 2016 May 02 ] |
How much time passes between "sending ..." and "cannot send ..." messages? |
Comment by Aleksandrs Saveljevs [ 2016 May 02 ] |
Do you use TLS between Zabbix server and these proxies? If so, which encryption libraries do you use? |
Comment by Bence Balog [ 2016 May 02 ] |
Thanks for your reply. 3 seconds between sending and timeout: No, we are not using TLS, all TLS settings are on default (unencrypted) |
Comment by Aleksandrs Saveljevs [ 2016 May 02 ] |
There seems to be a difference between how Zabbix server handles passive and active proxies. With passive proxies (as can be seen in src/zabbix_server/proxypoller/proxypoller.c), it uses a timeout of CONFIG_TRAPPER_TIMEOUT (300 seconds) to send configuration data. With active proxies (as can be seen in src/zabbix_server/trapper/proxyconfig.c), it only uses a timeout of CONFIG_TIMEOUT (3 seconds by default). However, this issue seems to have existed forever, so it is currently unclear why you are experiencing this problem after upgrade to 3.0. |
Comment by Bence Balog [ 2016 May 02 ] |
OK, but why does the proxy expecting larger data package and drops the received one? |
Comment by Aleksandrs Saveljevs [ 2016 May 02 ] |
One reasonable guess would be that, since ZBX_TCP_WRITE() times out, Zabbix server fails to send all data (it sends the length of the data in the message header). Therefore, proxy drops whatever it has received, because the data is incomplete. |
Comment by Bence Balog [ 2016 May 02 ] |
How can I increase the value of CONFIG_TIMEOUT ? |
Comment by Aleksandrs Saveljevs [ 2016 May 02 ] |
Just increase Timeout parameter in your configuration file. |
Comment by Bence Balog [ 2016 May 02 ] |
Thanks a lot. Transmitting the data package lasted 8 seconds. Now it is OK. |
Comment by Aleksandrs Saveljevs [ 2016 May 02 ] |
Fixed in development branch svn://svn.zabbix.com/branches/dev/ZBX-10727 by replacing CONFIG_TIMEOUT with CONFIG_TRAPPER_TIMEOUT in send_proxyconfig() function. |
Comment by Andris Zeila [ 2016 May 05 ] |
Successfully tested |
Comment by Aleksandrs Saveljevs [ 2016 May 05 ] |
Fixed in pre-2.0.18rc1 r59893, pre-2.2.13rc1 r59894, pre-3.0.3rc1 r59896, pre-3.1.0 (trunk) r59897. |
Comment by Glebs Ivanovskis (Inactive) [ 2017 May 29 ] |
The reason why it started to fail with such message after upgrade to 3.0 is |
Comment by AndreaConsadori [ 2017 Jun 29 ] |
this issue it seems to be back on debian 8 with zabbix 9342:20170629:140129.903 sending configuration data to proxy "ProxyIP" at "172.30.7.98", datalen 461682 zabbix_server (Zabbix) 3.2.6 zabbix_proxy (Zabbix) 3.2.6 Linux ipproxy 3.16.0-4-amd64 #1 SMP Debian 3.16.43-2 (2017-04-30) x86_64 GNU/Linux |
Comment by Glebs Ivanovskis (Inactive) [ 2017 Jun 29 ] |
Well, I see no indication of a bug. Server was trying to send data to proxy for 5 minutes, but timed out. Proxy received incomplete message (length stated in header is larger than what proxy managed to read from socket) and ignored this message (letting you know what happened). |