Uploaded image for project: 'ZABBIX BUGS AND ISSUES'
  1. ZABBIX BUGS AND ISSUES
  2. ZBX-12836

Trapper intermittently fails to assemble received data (TCP stream)

    XMLWordPrintable

    Details

    • Type: Incident report
    • Status: Closed
    • Priority: Major
    • Resolution: Won't fix
    • Affects Version/s: 3.2.8
    • Fix Version/s: None
    • Component/s: Proxy (P), Server (S)
    • Labels:
    • Environment:
      Proxy running on RHEL 6, also happens directly with server on RHEL 7, both VMware virtual machines with latest updates.

      Description

      Reasonably sized (< 30 kB) LLD objects are sometimes only partially received/parsed by the proxy/server trapper, which leads to the error "cannot parse as a valid JSON object: unexpected end of string data". The data is truncated between frames of the same TCP stream. Retrying in a loop, eventually the same send will succeed.

      Zabbix logs (at level 4) show the truncated data, while traffic dumps show the whole data/stream received/acknowledged. A failed transmission is followed by a TCP reset.

      Sending a lot of normal items in one send() may also fail, this is much more infrequent, but logically could be due to the same underlying issue.

      I've written the sender myself, in Perl, it uses a single send() call. Socket options (TCP_NODELAY) do not seem to make a difference and neither does disabling TSO/GRO/GSO offloading on the proxy, which is done anyway for the duration of capturing traffic to get dumps with real frames.

      I can send both a proxy log snippet and the corresponding capture file as-is to the Zabbix team, but they may not be attached here.

        Attachments

          Activity

            People

            Assignee:
            Unassigned Unassigned
            Reporter:
            jannek Janne Korkkula
            Votes:
            0 Vote for this issue
            Watchers:
            4 Start watching this issue

              Dates

              Created:
              Updated:
              Resolved: