-
Incident report
-
Resolution: Won't fix
-
Major
-
None
-
3.2.8
-
Proxy running on RHEL 6, also happens directly with server on RHEL 7, both VMware virtual machines with latest updates.
Reasonably sized (< 30 kB) LLD objects are sometimes only partially received/parsed by the proxy/server trapper, which leads to the error "cannot parse as a valid JSON object: unexpected end of string data". The data is truncated between frames of the same TCP stream. Retrying in a loop, eventually the same send will succeed.
Zabbix logs (at level 4) show the truncated data, while traffic dumps show the whole data/stream received/acknowledged. A failed transmission is followed by a TCP reset.
Sending a lot of normal items in one send() may also fail, this is much more infrequent, but logically could be due to the same underlying issue.
I've written the sender myself, in Perl, it uses a single send() call. Socket options (TCP_NODELAY) do not seem to make a difference and neither does disabling TSO/GRO/GSO offloading on the proxy, which is done anyway for the duration of capturing traffic to get dumps with real frames.
I can send both a proxy log snippet and the corresponding capture file as-is to the Zabbix team, but they may not be attached here.