[ZBX-21388] MongoDB plugin starts to timeout after some time Created: 2022 Jul 26 Updated: 2024 Apr 10 Resolved: 2022 Sep 29 |
|
Status: | Closed |
Project: | ZABBIX BUGS AND ISSUES |
Component/s: | Agent2 plugin (G) |
Affects Version/s: | 6.2.0 |
Fix Version/s: | 6.0.10rc1, 6.2.4rc1, 6.4.0beta1, 6.4 (plan) |
Type: | Problem report | Priority: | Trivial |
Reporter: | Nathan | Assignee: | Eriks Sneiders |
Resolution: | Fixed | Votes: | 6 |
Labels: | None | ||
Remaining Estimate: | Not Specified | ||
Time Spent: | Not Specified | ||
Original Estimate: | Not Specified |
Attachments: |
![]() ![]() ![]() ![]() |
||||||||||||||||
Issue Links: |
|
||||||||||||||||
Team: | |||||||||||||||||
Sprint: | Sprint 92 (Sep 2022) | ||||||||||||||||
Story Points: | 1.5 |
Description |
Summary:
My environment:
(Faster) Reproduction: while [ 1 ]; do zabbix_get -s 127.0.0.1 -p 10050 -k 'mongodb.server.status["tcp://localhost:27017","XXX","XXX"]' || break; done
Details: $ zabbix_get -s 127.0.0.1 -p 10050 -k 'mongodb.server.status["tcp://localhost:27017","XXX","XXX"]' ZBX_NOTSUPPORTED: timeout occurred $ zabbix_get -s 127.0.0.1 -p 10050 -k 'mongodb.server.status["tcp://another.working.mongodb:27017","YYY","YYY"]' ZBX_NOTSUPPORTED: timeout occurred $ zabbix_get -s 127.0.0.1 -p 10050 -k 'agent.hostname' my-hostname Note: this only concerns the metrics of MongoDB plugin, the others continue to work. Problem, when I set "DebugLevel=5" I can't reproduce the bug anymore (maybe it's a concurrency bug and logging takes enough time to reduce the probability of it occurring?). |
Comments |
Comment by Nathan [ 2022 Jul 26 ] |
Maybe this is related to https://support.zabbix.com/browse/ZBX-21272 ? It don't mention that their problem only occurs after a long period of time, so I don't think so. |
Comment by Alessandro Lombardi [ 2022 Jul 29 ] |
I have created the other ticket but the issue in my case starts after few minutes from the restart of the agent, the check works only for few seconds/minutes and after that there is only the timeout. |
Comment by Kalchenko Oleksandr [ 2022 Aug 05 ] |
same issue here It works fine for some minutes after agent restart and then "timeout occured" for all of items. |
Comment by Kirill [ 2022 Aug 22 ] |
I have the same problem with Mongo and PHP-FPM also So I don't think it is only a MongoDB plugin problem. |
Comment by Metin ERDEM [ 2022 Aug 24 ] |
is there a solution? i have the some problems |
Comment by Alessandro Lombardi [ 2022 Aug 25 ] |
I have the same issue on zabbix 6.0.7 with mongodb 5.0.8 |
Comment by Fabio Papa [ 2022 Aug 25 ] |
Hi, having the same issue, using zabbix agent2 6.2.1 with mongodb 4.4.4. I managed to get a level 5 log where the connection initially works but then it stops. The timeout value has been edited in the plugin configuration to the maximum (30). It does not seem to me that the plugin is honoring that, though, looking at the timestamps. It also seems that once a timeout occurs, no other operation is ever succesfull, no matter what. Restarting the agent makes it work again for a bit. |
Comment by Muhammet [ 2022 Aug 25 ] |
i have the same issue on zabbix 6.2.1 |
Comment by Sébastien Blaisot [ 2022 Aug 26 ] |
I can confirm I've seen the same problem here with mongodb 4.2.22 and zabbix 6.2.1 (server and agent) . |
Comment by Sébastien Blaisot [ 2022 Aug 26 ] |
Could it be that mongodb plugin is not properly closing connection to mongoDB and after a while has saturated all available mongodb connections for a given source and cannot connect anymore (with connections stuck waiting for available slots) until a restart that free the stalled connections? |
Comment by Sébastien Blaisot [ 2022 Aug 26 ] |
another hypothesis is that maybe the plugin keeps connections to mongo open and that mongodb closes inactive connections after some time so the plugin can not use them anymore |
Comment by Sébastien Blaisot [ 2022 Aug 26 ] |
I reproduced with loglevel = 5. First connection failing show the following log : ``` 2022/08/26 13:45:27.284335 connection established using TLSv1.3 TLS_CHACHA20_POLY1305_SHA256 |
Comment by Sébastien Blaisot [ 2022 Aug 31 ] |
Setting Plugins.MongoDB.Sessions.local.Uri=tcp://localhost:27017 and using mongodb.server.status["local"] doesn't change anything. I'm still able to reproduce from zabbix server in a few seconds (to a few minutes maximum) with while :; do zabbix_get -s agent_ip -k mongodb.server.status["local"] --tls-psk-file /path/to/psk_file --tls-psk-identity "My PSK" --tls-connect psk || break; done |
Comment by Sébastien Blaisot [ 2022 Sep 19 ] |
I have compiled mongodb plugin from branch
Steps followed : |
Comment by Sébastien Blaisot [ 2022 Sep 20 ] |
unfortunately after about 10h I hit this bug again with mongodb plugin branch |
Comment by Eriks Sneiders [ 2022 Sep 20 ] |
Hi, sblaisot this fix is implemented on both ends the MongoDB plugin and Zabbix agent 2, if you want to try it out on your own end, before it's implemented in the Zabbix release, you should also use a Zabbix agent 2 from the correct branch. This will also add additional logging, and a more detailed error of the issue should appear in Zabbix agent 2 log, should the issue persist. |
Comment by Sébastien Blaisot [ 2022 Sep 21 ] |
Thanks Eriks, I recompiled agent2 also and now it has been working stable for the last 24 hours. I will let you know if I encounter thsi issue again. |
Comment by Eriks Sneiders [ 2022 Sep 26 ] |
Fixed in
|
Comment by Jurijs Klopovskis [ 2022 Sep 27 ] |
Relased zabbix-agent2-plugin-mongodb-1.1.0 Released plugins packages. |
Comment by Alessandro Lombardi [ 2022 Sep 29 ] |
The new plugin version with the new zabbix agent2 6.0.10rc1 seems working fine, thanks. Eriks is the new plugin for mongodb available for windows? I have not find any documentation about this. I check the code and seems not complete for windows but can be compiled only for Linux servers. |
Comment by Eriks Sneiders [ 2022 Sep 30 ] |
Hi, lombardi.aless! I am happy to hear that the plugin is working fine. As for Windows, although at the moment we do not have any msi installers for plugins, the plugin can be compiled from source, and will work on windows. |
Comment by Alessandro Lombardi [ 2022 Sep 30 ] |
Thanks Eriks, I have compiled the plugin and is working also on windows. |