ZABBIX BUGS AND ISSUES
  1. ZABBIX BUGS AND ISSUES
  2. ZBX-10530

zbx_tcp_accept() does read() without receive timeout

    Details

      Description

      I updated to Zabbix 3.0.1 (from 2.4.7) and after a random while Zabbix mostly stops working. All internal/simple checks work but no incoming data (i.e. trapper items).

      Netstat shows that trapper TCP connections never timeout. Strace shows that recvfrom(9<TCP:[server:10051->agent:55098]>, "Z", 1, MSG_PEEK, NULL, NULL) call never finishes (until manual SIGALRM or similar).

      trapper_thread() does basically zbx_tcp_accept() followed by zbx_tcp_recv_to()... But only the latter calls zbx_socket_timeout_set().

      Due to adding TLS support zbx_tcp_accept() does MSG_PEEK for the first byte and it is done before socket timeouts are set so it will wait infinitely.

      In my case we have lots of Zabbix agents on devices connected over mobile broadband networks which tend to have not so good connections. I get a new stuck connection maybe once or twice per hour so this leads to Zabbix server "crashing" once or twice per day.

        Issue Links

          Activity

          Hide
          Anssi Kolehmainen added a comment -

          Trivial fix for this issue. Move zbx_socket_timeout_set() a few lines earlier before the blocking read() call. No hung sockets after 4 hours and otherwise seems to be working fine.

          Show
          Anssi Kolehmainen added a comment - Trivial fix for this issue. Move zbx_socket_timeout_set() a few lines earlier before the blocking read() call. No hung sockets after 4 hours and otherwise seems to be working fine.
          Hide
          richlv added a comment -

          ZBX-10469 might be the same problem

          Show
          richlv added a comment - ZBX-10469 might be the same problem
          Hide
          orogor added a comment -

          got the same issue on proxy

          Show
          orogor added a comment - got the same issue on proxy
          Hide
          Aleksandrs Saveljevs added a comment -

          Fixed in development branch svn://svn.zabbix.com/branches/dev/ZBX-10530 .

          Show
          Aleksandrs Saveljevs added a comment - Fixed in development branch svn://svn.zabbix.com/branches/dev/ZBX-10530 .
          Hide
          Andris Zeila added a comment - - edited

          Successfully tested.

          (1) One note, maybe instead of "from %s: recv() peek failed: %s" error message we should give less technical one - "reading first byte from connection failed" or something like. On the other hand - I'm not sure if this error message can reach frontend.

          Aleksandrs Saveljevs Added your message in r59185. RESOLVED.

          Andris Zeila Looks good. CLOSED.

          Show
          Andris Zeila added a comment - - edited Successfully tested. (1) One note, maybe instead of "from %s: recv() peek failed: %s" error message we should give less technical one - "reading first byte from connection failed" or something like. On the other hand - I'm not sure if this error message can reach frontend. Aleksandrs Saveljevs Added your message in r59185. RESOLVED. Andris Zeila Looks good. CLOSED.
          Hide
          richlv added a comment -

          user-friendlier messages is always a good idea - whether they end up in the frontend, server log or elsewhere

          Show
          richlv added a comment - user-friendlier messages is always a good idea - whether they end up in the frontend, server log or elsewhere
          Hide
          Aleksandrs Saveljevs added a comment -

          Fixed in pre-3.0.2rc1 r59193 and pre-3.1.0 (trunk) r59194.

          Show
          Aleksandrs Saveljevs added a comment - Fixed in pre-3.0.2rc1 r59193 and pre-3.1.0 (trunk) r59194.
          Hide
          Diego e Silva de Souza added a comment - - edited

          How and where do I apply this patch?

          Can I correct this issue in a zabbix proxy installed via rpm package?

          Show
          Diego e Silva de Souza added a comment - - edited How and where do I apply this patch? Can I correct this issue in a zabbix proxy installed via rpm package?
          Hide
          orogor added a comment -

          most likely, just update proxies
          Affects Version/s:
          3.0.1
          Fix Version/s:
          3.0.2rc1, 3.1.0 (trunk)

          Show
          orogor added a comment - most likely, just update proxies Affects Version/s: 3.0.1 Fix Version/s: 3.0.2rc1, 3.1.0 (trunk)

            People

            • Assignee:
              Unassigned
              Reporter:
              Anssi Kolehmainen
            • Votes:
              3 Vote for this issue
              Watchers:
              12 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: