[ZBX-21704] Docker zabbix fping can't allocate memmory Created: 2022 Sep 28 Updated: 2023 Jun 02 Resolved: 2022 Oct 03 |
|
| Status: | Closed |
| Project: | ZABBIX BUGS AND ISSUES |
| Component/s: | None |
| Affects Version/s: | None |
| Fix Version/s: | None |
| Type: | Incident report | Priority: | Trivial |
| Reporter: | Bozhan Boyadzhiev | Assignee: | Zabbix Support Team |
| Resolution: | Incomplete | Votes: | 0 |
| Labels: | None | ||
| Remaining Estimate: | Not Specified | ||
| Time Spent: | Not Specified | ||
| Original Estimate: | Not Specified | ||
| Issue Links: |
|
||||
| Description |
|
Steps to reproduce:
docker run --name zabbix-server-mysql -t \ --add-host mysql-server:172.20.240.1 \ -e DB_SERVER_HOST="mysql-server" \ -e MYSQL_DATABASE="zabbix" \ -e MYSQL_USER="zabbixuser" \ -e MYSQL_PASSWORD="BIG_BAD_PASS" \ -e ZBX_JAVAGATEWAY="zabbix-java-gateway" \ -e ZBX_CACHESIZE=1024M \ -e ZBX_TRAPPERTIMEOUT=300 \ -e ZBX_TIMEOUT=30 \ -v /dev/log:/dev/log -v /var/run/systemd/journal/socket:/var/run/systemd/journal/socket \ --network=zabbix-net \ -p 10051:10051 \ --restart unless-stopped \ -d dockerhub.artifactory.pyrsoftware.ca/zabbix/zabbix-server-mysql:ubuntu-6.2.3
215:20220928:144844.092 /usr/bin/fping -C3 -i0 2>&1 </tmp/zabbix_server_215.pinger;: [12] Cannot allocate memory 215:20220928:144845.095 /usr/bin/fping -C3 -i0 2>&1 </tmp/zabbix_server_215.pinger;: [12] Cannot allocate memory 215:20220928:144846.099 /usr/bin/fping -C3 -i0 2>&1 </tmp/zabbix_server_215.pinger;: [12] Cannot allocate memory 215:20220928:144847.102 /usr/bin/fping -C3 -i0 2>&1 </tmp/zabbix_server_215.pinger;: [12] Cannot allocate memory 215:20220928:144848.105 /usr/bin/fping -C3 -i0 2>&1 </tmp/zabbix_server_215.pinger;: [12] Cannot allocate memory 215:20220928:144849.106 /usr/bin/fping -C3 -i0 2>&1 </tmp/zabbix_server_215.pinger;: [12] Cannot allocate memory 215:20220928:144850.109 /usr/bin/fping -C3 -i0 2>&1 </tmp/zabbix_server_215.pinger;: [12] Cannot allocate memory 215:20220928:144851.112 /usr/bin/fping -C3 -i0 2>&1 </tmp/zabbix_server_215.pinger;: [12] Cannot allocate memory 215:20220928:144852.116 /usr/bin/fping -C3 -i0 2>&1 </tmp/zabbix_server_215.pinger;: [12] Cannot allocate memory 215:20220928:144853.119 /usr/bin/fping -C3 -i0 2>&1 </tmp/zabbix_server_215.pinger;: [12] Cannot allocate memory 215:20220928:144854.122 /usr/bin/fping -C3 -i0 2>&1 </tmp/zabbix_server_215.pinger;: [12] Cannot allocate memory 215:20220928:144855.125 /usr/bin/fping -C3 -i0 2>&1 </tmp/zabbix_server_215.pinger;: [12] Cannot allocate memory 215:20220928:144856.129 /usr/bin/fping -C3 -i0 2>&1 </tmp/zabbix_server_215.pinger;: [12] Cannot allocate memory 215:20220928:144857.132 /usr/bin/fping -C3 -i0 2>&1 </tmp/zabbix_server_215.pinger;: [12] Cannot allocate memory 215:20220928:144858.135 /usr/bin/fping -C3 -i0 2>&1 </tmp/zabbix_server_215.pinger;: [12] Cannot allocate memory There is plenty free memory : # free -m total used free shared buff/cache available Mem: 32011 1588 28859 80 1564 29998 Swap: 2047 0 2047 |
| Comments |
| Comment by Alexey Pustovalov [ 2022 Sep 28 ] |
|
are you able to use fping directly inside container? |
| Comment by Bozhan Boyadzhiev [ 2022 Sep 29 ] |
|
Yes i can zabbix@48ce8eb20561:/var/lib/zabbix$ cat /etc/hosts 127.0.0.1 localhost ::1 localhost ip6-localhost ip6-loopback fe00::0 ip6-localnet ff00::0 ip6-mcastprefix ff02::1 ip6-allnodes ff02::2 ip6-allrouters 172.20.240.1 mysql-server 172.20.240.3 48ce8eb20561 zabbix@48ce8eb20561:/var/lib/zabbix$ fping 172.20.240.1 172.20.240.1 is alive zabbix@48ce8eb20561:/var/lib/zabbix$ fping 172.20.240.3 172.20.240.3 is alive |
| Comment by Jose Roas [ 2022 Sep 29 ] |
|
Hello please running this command and upload the output docker stats |
| Comment by Bozhan Boyadzhiev [ 2022 Sep 29 ] |
|
I checked my other installation with version 6.0.2 and there i have such file /tmp/zabbix_server_xxx.pinger; but i don't have it on container with version 6.2.3 |
| Comment by Bozhan Boyadzhiev [ 2022 Sep 29 ] |
|
I don't have docker stats, because i am redoing all export/import of data, but percents of usage was lower than 2 % per container for all values |
| Comment by Alexey Pustovalov [ 2022 Sep 29 ] |
|
please create the file manually /tmp/zabbix_server_test.pinger: <IP> <IP2> <IP3> then execute: /usr/bin/fping -C3 -i0 2>&1 < /tmp/zabbix_server_test.pinger successful? |
| Comment by Bozhan Boyadzhiev [ 2022 Sep 29 ] |
|
Sorry i can't do that tests anymore . please close the case as it seems no body else has the issue. Thanks. |
| Comment by Bozhan Boyadzhiev [ 2023 Mar 14 ] |
|
I was able to do some more tests. I tried several zabbix versions and issue starts with 6.0.5 , i don't see memory allocation problem with 6.0.4 , I started the container with DEBUG tag =4 and got this: 266:20230314:185458.194 End of zbx_preprocess_item_value()
266:20230314:185458.194 In zbx_ipc_socket_write()
266:20230314:185458.194 End of zbx_ipc_socket_write():SUCCEED
266:20230314:185458.194 End of process_history_data() processed:1
266:20230314:185458.194 End of process_client_history_data():SUCCEED
266:20230314:185458.194 In zbx_send_response_ext()
266:20230314:185458.194 zbx_send_response_ext() '{"response":"success","info":"processed: 1; failed: 0; total: 1; seconds spent: 0.001074"}'
266:20230314:185458.194 End of zbx_send_response_ext():SUCCEED
266:20230314:185458.194 End of recv_agenthistory()
266:20230314:185458.194 zbx_setproctitle() title:'trapper #4 [processed data in 0.001610 sec, waiting for connection]'
227:20230314:185458.194 End of zbx_ipc_service_recv():1
227:20230314:185458.194 In preprocessor_add_request()
227:20230314:185458.194 In preprocessor_sync_configuration()
227:20230314:185458.194 In DCconfig_get_preprocessable_items()
227:20230314:185458.194 End of DCconfig_get_preprocessable_items() items:8730
227:20230314:185458.194 End of preprocessor_sync_configuration() item config size: 8730, history cache size: 1148
227:20230314:185458.194 In preprocessor_enqueue() itemid: 269358
227:20230314:185458.194 In preprocessor_enqueue_dependent() itemid: 269358
227:20230314:185458.194 End of preprocessor_enqueue_dependent()
227:20230314:185458.194 End of preprocessor_enqueue()
227:20230314:185458.194 In preprocessor_assign_tasks()
227:20230314:185458.194 In preprocessor_get_next_task()
227:20230314:185458.194 End of preprocessor_get_next_task()
227:20230314:185458.194 End of preprocessor_assign_tasks()
227:20230314:185458.194 End of preprocessor_add_request()
227:20230314:185458.194 In zbx_ipc_service_recv() timeout:1.000
270:20230314:185458.194 /tmp/zabbix_server_270.pinger
270:20230314:185458.194 10.30.90.64
270:20230314:185458.194 10.30.90.18
270:20230314:185458.194 10.30.27.253
270:20230314:185458.194 10.30.151.102
270:20230314:185458.194 10.11.30.252
270:20230314:185458.194 10.30.46.8
270:20230314:185458.194 10.30.53.52
270:20230314:185458.194 10.30.141.21
270:20230314:185458.194 10.30.154.91
270:20230314:185458.194 10.12.37.165
270:20230314:185458.194 10.11.12.188
270:20230314:185458.194 10.11.62.130
270:20230314:185458.194 10.30.164.19
270:20230314:185458.194 10.123.121.180
270:20230314:185458.194 10.30.20.145
270:20230314:185458.194 10.11.159.89
270:20230314:185458.194 10.11.62.152
270:20230314:185458.194 10.11.16.197
270:20230314:185458.194 10.30.165.21
270:20230314:185458.194 10.30.148.36
270:20230314:185458.194 10.30.83.3
270:20230314:185458.194 10.30.49.2
270:20230314:185458.194 10.30.33.212
270:20230314:185458.194 10.30.24.138
270:20230314:185458.194 10.11.10.11
270:20230314:185458.194 10.30.148.222
270:20230314:185458.194 10.30.86.200
270:20230314:185458.194 10.30.142.130
270:20230314:185458.194 10.30.133.63
270:20230314:185458.194 /usr/bin/fping -C1 -t200 -i0 2>&1 </tmp/zabbix_server_270.pinger;
270:20230314:185458.195 /usr/bin/fping -C1 -t200 -i0 2>&1 </tmp/zabbix_server_270.pinger;: [12] Cannot allocate memory
270:20230314:185458.195 End of zbx_ping():NOTSUPPORTED
270:20230314:185458.195 In process_values()
270:20230314:185458.195 host [10.30.90.64] /usr/bin/fping -C1 -t200 -i0 2>&1 </tmp/zabbix_server_270.pinger;: [12] Cannot allocate memory
270:20230314:185458.195 In process_value()
270:20230314:185458.195 In zbx_preprocess_item_value()
270:20230314:185458.195 End of zbx_preprocess_item_value()
270:20230314:185458.195 End of process_value()
270:20230314:185458.195 host [10.30.90.18] /usr/bin/fping -C1 -t200 -i0 2>&1 </tmp/zabbix_server_270.pinger;: [12] Cannot allocate memory
270:20230314:185458.195 In process_value()
270:20230314:185458.195 In zbx_preprocess_item_value()
270:20230314:185458.195 End of zbx_preprocess_item_value()
270:20230314:185458.195 End of process_value()
270:20230314:185458.195 host [10.30.27.253] /usr/bin/fping -C1 -t200 -i0 2>&1 </tmp/zabbix_server_270.pinger;: [12] Cannot allocate memory
270:20230314:185458.195 In process_value()
|
| Comment by Bozhan Boyadzhiev [ 2023 Mar 14 ] |
|
zabbix@d4a046f29f2f:/tmp$ /usr/bin/fping -C1 -t200 -i0 2>&1 </tmp/test123 10.30.53.88 : 3.57 |
| Comment by Bozhan Boyadzhiev [ 2023 Mar 14 ] |
|
It seem there isn't such issue with centos based image |