[ZBX-10727] Proxy problem - Message from server is shorter than expected - Message ignored. Created: 2016 Apr 29  Updated: 2017 Jun 29  Resolved: 2016 May 06

Status: Closed
Project: ZABBIX BUGS AND ISSUES
Component/s: Proxy (P), Server (S)
Affects Version/s: 3.0.2
Fix Version/s: 2.0.18rc1, 2.2.13rc1, 3.0.3rc1, 3.2.0alpha1

Type: Incident report Priority: Critical
Reporter: Bence Balog Assignee: Unassigned
Resolution: Fixed Votes: 0
Labels: timeout
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified
Environment:

Zabbix server and proxy 3.0.2
OS: Debian 8


Issue Links:
Duplicate
is duplicated by ZBX-10781 Communication not working between ser... Closed

 Description   

After upgrading our 2.4.7 environment to 3.0.2 two proxies are not receiving correct data from the server, therefor no data received from those proxies.

Error messages in the log (zabbix-proxy):
Message from 10.243.130.12 is shorter than expected 1859411 bytes. Message ignored.
End of get_data_from_server():FAIL
cannot obtain configuration data from server at "10.243.130.12":

From the server logs
sending configuration data to proxy "CZ_Zabbix_proxy_01" at "172.22.2.251", datalen 1859411
cannot send configuration data to proxy "CZ_Zabbix_proxy_01" at "172.22.2.251": ZBX_TCP_WRITE() timed out

Heartbeat messages are OK, from the GUI the "last seen" value is a few seconds.

There are other proxies on the local network and some remote ones via ipsec vpn which are working properly.

Before the upgrade everything was fine.



 Comments   
Comment by Aleksandrs Saveljevs [ 2016 May 02 ]

How much time passes between "sending ..." and "cannot send ..." messages?

Comment by Aleksandrs Saveljevs [ 2016 May 02 ]

Do you use TLS between Zabbix server and these proxies? If so, which encryption libraries do you use?

Comment by Bence Balog [ 2016 May 02 ]

Thanks for your reply.

3 seconds between sending and timeout:
2493:20160502:113436.526 sending configuration data to proxy "CZ_Zabbix_proxy_01" at "172.22.2.251", datalen 1859411
2493:20160502:113439.526 cannot send configuration data to proxy "CZ_Zabbix_proxy_01" at "172.22.2.251": ZBX_TCP_WRITE() timed out

No, we are not using TLS, all TLS settings are on default (unencrypted)

Comment by Aleksandrs Saveljevs [ 2016 May 02 ]

There seems to be a difference between how Zabbix server handles passive and active proxies. With passive proxies (as can be seen in src/zabbix_server/proxypoller/proxypoller.c), it uses a timeout of CONFIG_TRAPPER_TIMEOUT (300 seconds) to send configuration data. With active proxies (as can be seen in src/zabbix_server/trapper/proxyconfig.c), it only uses a timeout of CONFIG_TIMEOUT (3 seconds by default). However, this issue seems to have existed forever, so it is currently unclear why you are experiencing this problem after upgrade to 3.0.

Comment by Bence Balog [ 2016 May 02 ]

OK, but why does the proxy expecting larger data package and drops the received one?

Comment by Aleksandrs Saveljevs [ 2016 May 02 ]

One reasonable guess would be that, since ZBX_TCP_WRITE() times out, Zabbix server fails to send all data (it sends the length of the data in the message header). Therefore, proxy drops whatever it has received, because the data is incomplete.

Comment by Bence Balog [ 2016 May 02 ]

How can I increase the value of CONFIG_TIMEOUT ?

Comment by Aleksandrs Saveljevs [ 2016 May 02 ]

Just increase Timeout parameter in your configuration file.

Comment by Bence Balog [ 2016 May 02 ]

Thanks a lot. Transmitting the data package lasted 8 seconds.
I changed the value to 10 sec.

Now it is OK.

Comment by Aleksandrs Saveljevs [ 2016 May 02 ]

Fixed in development branch svn://svn.zabbix.com/branches/dev/ZBX-10727 by replacing CONFIG_TIMEOUT with CONFIG_TRAPPER_TIMEOUT in send_proxyconfig() function.

Comment by Andris Zeila [ 2016 May 05 ]

Successfully tested

Comment by Aleksandrs Saveljevs [ 2016 May 05 ]

Fixed in pre-2.0.18rc1 r59893, pre-2.2.13rc1 r59894, pre-3.0.3rc1 r59896, pre-3.1.0 (trunk) r59897.

Comment by Glebs Ivanovskis (Inactive) [ 2017 May 29 ]

The reason why it started to fail with such message after upgrade to 3.0 is ZBX-8914 which improved/introduced validation of received message length against what was promised in the header. Although incomplete data would cause error anyway, "invalid JSON..." or something like that.

Comment by AndreaConsadori [ 2017 Jun 29 ]

this issue it seems to be back on debian 8 with zabbix

9342:20170629:140129.903 sending configuration data to proxy "ProxyIP" at "172.30.7.98", datalen 461682
9342:20170629:140629.903 cannot send configuration data to proxy "ProxyIP" at "172.30.7.98": ZBX_TCP_WRITE() timed out

zabbix_server (Zabbix) 3.2.6
Revision 67849 4 May 2017, compilation time: May 6 2017 15:48:59

zabbix_proxy (Zabbix) 3.2.6
Revision 67849 4 May 2017, compilation time: May 6 2017 15:59:40

Linux ipproxy 3.16.0-4-amd64 #1 SMP Debian 3.16.43-2 (2017-04-30) x86_64 GNU/Linux

Comment by Glebs Ivanovskis (Inactive) [ 2017 Jun 29 ]

Well, I see no indication of a bug. Server was trying to send data to proxy for 5 minutes, but timed out. Proxy received incomplete message (length stated in header is larger than what proxy managed to read from socket) and ignored this message (letting you know what happened).

Generated at Wed Apr 24 21:07:13 EEST 2024 using Jira 9.12.4#9120004-sha1:625303b708afdb767e17cb2838290c41888e9ff0.