[ZBX-18880] ZABBIX agent repeatedly timed out Created: 2021 Jan 14 Updated: 2021 Apr 23 Resolved: 2021 Apr 23 |
|
Status: | Closed |
Project: | ZABBIX BUGS AND ISSUES |
Component/s: | Agent (G) |
Affects Version/s: | 5.0.7, 5.2.3 |
Fix Version/s: | None |
Type: | Problem report | Priority: | Trivial |
Reporter: | rrr | Assignee: | Aigars Kadikis |
Resolution: | Cannot Reproduce | Votes: | 0 |
Labels: | None | ||
Remaining Estimate: | Not Specified | ||
Time Spent: | Not Specified | ||
Original Estimate: | Not Specified | ||
Environment: |
Debian Linux 10.7 with ZABBIX Server 5.2.3 |
Description |
Every ZABBIX agentd that establishes a direct connection to the ZABBIX server, repeatedly timed out: Agentd Logfile: ... 556:20210113:190753.025 active check data upload to [127.0.0.1:10051] started to fail ([recv] ZBX_TCP_READ() timed out) 556:20210113:190755.366 active check data upload to [127.0.0.1:10051] is working again 556:20210113:190902.485 active check data upload to [127.0.0.1:10051] started to fail ([recv] ZBX_TCP_READ() timed out) 556:20210113:190906.486 active check data upload to [127.0.0.1:10051] is working again 556:20210113:191656.436 active check data upload to [127.0.0.1:10051] started to fail ([recv] ZBX_TCP_READ() timed out) 556:20210113:191700.437 active check data upload to [127.0.0.1:10051] is working again 556:20210113:191805.561 active check data upload to [127.0.0.1:10051] started to fail ([recv] ZBX_TCP_READ() timed out) ... The ZABBIX agentd which is locally installed on the ZABBIX server is also affected. Network problems can thus be excluded. If I connect those Agents to an ZABBIX proxy, the error messages do not appear. Affected Versions: ZABBIX Server/Proxy 5.2.3, ZABBIX Agentd 4.4.10, 5.0.7 & 5.2.3 Agentd Logfile (with LogLevel=4): ... 30552:20210113:231238.210 zbx_setproctitle() title:'active checks #1 [idle 1 sec]' 30548:20210113:231238.979 zbx_setproctitle() title:'collector [processing data]' 30548:20210113:231238.979 In update_cpustats() 30548:20210113:231238.979 End of update_cpustats() 30548:20210113:231238.979 zbx_setproctitle() title:'collector [idle 1 sec]' 30552:20210113:231239.210 In send_buffer() host:'127.0.0.1' port:10051 entries:0/100 30552:20210113:231239.210 End of send_buffer():SUCCEED 30552:20210113:231239.211 zbx_setproctitle() title:'active checks #1 [processing active checks]' 30552:20210113:231239.211 In process_active_checks() server:'127.0.0.1' port:10051 30552:20210113:231239.211 for key [agent.ping] received value [1] 30552:20210113:231239.211 In process_value() key:'zabbix.example.org:agent.ping' lastlogsize:null value:'1' 30552:20210113:231239.211 buffer: new element 0 30552:20210113:231239.211 End of process_value():SUCCEED 30552:20210113:231239.211 In need_meta_update() key:agent.ping 30552:20210113:231239.211 End of need_meta_update():FAIL 30552:20210113:231239.211 In send_buffer() host:'127.0.0.1' port:10051 entries:1/100 30552:20210113:231239.211 JSON before sending [{"request":"agent data","session":"97d0d657e50796076bdd54ecea567b08","data":[ {"host":"zabbix.example.org","key":"agent.ping","value":"1","id":2158,"clock":1610575959,"ns":211146130} ],"clock":1610575959,"ns":211418113}] 30548:20210113:231239.980 zbx_setproctitle() title:'collector [processing data]' 30548:20210113:231239.980 In update_cpustats() 30548:20210113:231239.980 End of update_cpustats() 30548:20210113:231239.980 zbx_setproctitle() title:'collector [idle 1 sec]' 30548:20210113:231240.980 zbx_setproctitle() title:'collector [processing data]' 30548:20210113:231240.980 In update_cpustats() 30548:20210113:231240.980 End of update_cpustats() 30548:20210113:231240.980 zbx_setproctitle() title:'collector [idle 1 sec]' 30548:20210113:231241.980 zbx_setproctitle() title:'collector [processing data]' 30548:20210113:231241.980 In update_cpustats() 30548:20210113:231241.980 End of update_cpustats() 30548:20210113:231241.980 zbx_setproctitle() title:'collector [idle 1 sec]' 30552:20210113:231242.211 active check data upload to [127.0.0.1:10051] started to fail ([recv] ZBX_TCP_READ() timed out) 30552:20210113:231242.211 send value error: [recv] ZBX_TCP_READ() timed out 30552:20210113:231242.211 End of send_buffer():FAIL 30552:20210113:231242.211 End of process_active_checks() 30552:20210113:231242.211 In get_min_nextcheck() 30552:20210113:231242.211 End of get_min_nextcheck():1610575969 30552:20210113:231242.211 In send_buffer() host:'127.0.0.1' port:10051 entries:1/100 30552:20210113:231242.211 JSON before sending [{"request":"agent data","session":"97d0d657e50796076bdd54ecea567b08","data":[ {"host":"zabbix.example.org","key":"agent.ping","value":"1","id":2158,"clock":1610575959,"ns":211146130} ],"clock":1610575962,"ns":211738479}] 30548:20210113:231242.981 zbx_setproctitle() title:'collector [processing data]' 30548:20210113:231242.981 In update_cpustats() 30548:20210113:231242.981 End of update_cpustats() 30548:20210113:231242.981 zbx_setproctitle() title:'collector [idle 1 sec]' 30548:20210113:231243.981 zbx_setproctitle() title:'collector [processing data]' 30548:20210113:231243.981 In update_cpustats() 30548:20210113:231243.981 End of update_cpustats() 30548:20210113:231243.981 zbx_setproctitle() title:'collector [idle 1 sec]' 30548:20210113:231244.981 zbx_setproctitle() title:'collector [processing data]' 30548:20210113:231244.981 In update_cpustats() 30548:20210113:231244.981 End of update_cpustats() 30548:20210113:231244.981 zbx_setproctitle() title:'collector [idle 1 sec]' 30552:20210113:231245.211 send value error: [recv] ZBX_TCP_READ() timed out 30552:20210113:231245.211 End of send_buffer():FAIL 30552:20210113:231245.211 zbx_setproctitle() title:'active checks #1 [idle 1 sec]' 30548:20210113:231245.982 zbx_setproctitle() title:'collector [processing data]' 30548:20210113:231245.982 In update_cpustats() 30548:20210113:231245.982 End of update_cpustats() 30548:20210113:231245.982 zbx_setproctitle() title:'collector [idle 1 sec]' 30552:20210113:231246.212 In send_buffer() host:'127.0.0.1' port:10051 entries:1/100 30552:20210113:231246.212 JSON before sending [{"request":"agent data","session":"97d0d657e50796076bdd54ecea567b08","data":[ {"host":"zabbix.example.org","key":"agent.ping","value":"1","id":2158,"clock":1610575959,"ns":211146130} ],"clock":1610575966,"ns":212279051}] 30552:20210113:231246.325 JSON back [ {"response":"success","info":"processed: 0; failed: 1; total: 1; seconds spent: 0.000015"} ] 30552:20210113:231246.325 In check_response() response:' {"response":"success","info":"processed: 0; failed: 1; total: 1; seconds spent: 0.000015"} ' 30552:20210113:231246.325 info from server: 'processed: 0; failed: 1; total: 1; seconds spent: 0.000015' 30552:20210113:231246.325 End of check_response():SUCCEED 30552:20210113:231246.325 OK 30552:20210113:231246.325 active check data upload to [127.0.0.1:10051] is working again 30552:20210113:231246.325 End of send_buffer():SUCCEED 30552:20210113:231246.325 zbx_setproctitle() title:'active checks #1 [idle 1 sec]' 30548:20210113:231246.982 zbx_setproctitle() title:'collector [processing data]' 30548:20210113:231246.982 In update_cpustats() 30548:20210113:231246.982 End of update_cpustats() 30548:20210113:231246.982 zbx_setproctitle() title:'collector [idle 1 sec]' 30552:20210113:231247.325 In send_buffer() host:'127.0.0.1' port:10051 entries:0/100 30552:20210113:231247.325 End of send_buffer():SUCCEED 30552:20210113:231247.325 zbx_setproctitle() title:'active checks #1 [idle 1 sec]' 30548:20210113:231247.982 zbx_setproctitle() title:'collector [processing data]' 30548:20210113:231247.982 In update_cpustats() 30548:20210113:231247.982 End of update_cpustats() 30548:20210113:231247.983 zbx_setproctitle() title:'collector [idle 1 sec]' 30552:20210113:231248.325 In send_buffer() host:'127.0.0.1' port:10051 entries:0/100 30552:20210113:231248.325 End of send_buffer():SUCCEED 30552:20210113:231248.325 zbx_setproctitle() title:'active checks #1 [idle 1 sec]' 30548:20210113:231248.983 zbx_setproctitle() title:'collector [processing data]' 30548:20210113:231248.983 In update_cpustats() ... |
Comments |
Comment by Aigars Kadikis [ 2021 Feb 03 ] |
Are there any free trapper processes on Zabbix core server? This will show all the trapper processes:
watch -n.1 'ps auxww | grep "[z]abbix_server: trapper #"'
This will show only free ones:
watch -n.1 'ps auxww | grep "[z]abbix_server: trapper #.*waiting for connection"'
|
Comment by Aigars Kadikis [ 2021 Apr 23 ] |
Closing due to long time no response. |