[ZBX-22817] vfs.fs.get triggers autofs automounting Created: 2023 May 18 Updated: 2025 May 09 |
|
| Status: | Confirmed |
| Project: | ZABBIX BUGS AND ISSUES |
| Component/s: | Agent2 plugin (G) |
| Affects Version/s: | 6.4.2 |
| Fix Version/s: | None |
| Type: | Problem report | Priority: | Trivial |
| Reporter: | Jason | Assignee: | Stanislavs Jurgensons (Inactive) |
| Resolution: | Unresolved | Votes: | 6 |
| Labels: | None | ||
| Remaining Estimate: | Not Specified | ||
| Time Spent: | Not Specified | ||
| Original Estimate: | Not Specified | ||
| Issue Links: |
|
||||
| Description |
|
Steps to reproduce:
Result:
systemd[1]: some-mount-point.automount: Got automount request for /some/mount/point, triggered by 1058985 (zabbix_agent2)
Expected:
|
| Comments |
| Comment by Jason [ 2023 Jul 08 ] |
|
I've noticed this appears to only happen with agent2, and not the non-go version (1 I assume?)
|
| Comment by Jason [ 2023 Oct 09 ] |
|
This is still a problem, any updates on a fix? Even a workaround would be good; some way to not use vfs.fs.get but still have FS checks? |
| Comment by fbo [ 2023 Oct 13 ] |
|
Same issue with zabbix-agent-6.4.7 and systemd automount I use systemd automount :
// 192.168.1.5:/backup /backup nfs x-systemd.automount,x-systemd.idle-timeout=60,rw,sync,noauto,_netdev 0 0
From /var/log/messages : // Oct 13 07:20:34 192.168.1.5 systemd: Got automount request for /backup, triggered by 10603 (zabbix_agentd) Oct 13 07:20:34 192.168.1.6 systemd: Mounting /backup... Oct 13 07:20:34 192.168.1.6 systemd: Mounted /backup. Oct 13 07:25:34 192.168.1.6 systemd: Unmounting /backup... Oct 13 07:25:34 192.168.1.6 systemd: Unmounted /backup. Oct 13 07:25:34 192.168.1.6 systemd: Got automount request for /backup, triggered by 10957 (zabbix_agentd) Oct 13 07:25:34 192.168.1.6 systemd: Mounting /backup... Oct 13 07:25:34 192.168.1.6 systemd: Mounted /backup. And from systemctl : // systemctl status backup.automount ● backup.automount Loaded: loaded (/etc/fstab; bad; vendor preset: disabled) Active: active (running) since Tue 2023-10-10 11:59:18 CEST; 2 days ago Where: /backup Docs: man:fstab(5) man:systemd-fstab-generator(8)Oct 13 02:57:34 192.168.1.5 systemd[1]: Got automount request for /backup, triggered by 23176 (zabbix_agentd) Oct 13 03:17:34 192.168.1.6 systemd[1]: Got automount request for /backup, triggered by 24549 (zabbix_agentd) Oct 13 03:22:34 192.168.1.6 systemd[1]: Got automount request for /backup, triggered by 24897 (zabbix_agentd) Oct 13 04:40:34 192.168.1.6 systemd[1]: Got automount request for /backup, triggered by 31319 (zabbix_agentd) Oct 13 04:50:34 192.168.1.6 systemd[1]: Got automount request for /backup, triggered by 32010 (zabbix_agentd) Oct 13 06:25:34 192.168.1.6 systemd[1]: Got automount request for /backup, triggered by 6797 (zabbix_agentd) Oct 13 06:55:34 192.168.1.6 systemd[1]: Got automount request for /backup, triggered by 8867 (zabbix_agentd) Oct 13 07:10:34 192.168.1.6 systemd[1]: Got automount request for /backup, triggered by 9919 (zabbix_agentd) Oct 13 07:20:34 192.168.1.6 systemd[1]: Got automount request for /backup, triggered by 10603 (zabbix_agentd) Oct 13 07:25:34 192.168.1.6 systemd[1]: Got automount request for /backup, triggered by 10957 (zabbix_agentd)
|
| Comment by Edgars Melveris [ 2023 Oct 16 ] |
|
Please increase the log level of the agent and attach it here after the problem appears. |
| Comment by fbo [ 2023 Oct 16 ] |
19863:20231016:135833.653 In update_cpustats()
19863:20231016:135833.653 End of update_cpustats()
19863:20231016:135833.653 zbx_setproctitle() title:'collector [idle 1 sec]'
19865:20231016:135834.256 zbx_setproctitle() title:'listener #2 [processing request]'
19865:20231016:135834.256 Requested [vfs.fs.get]
19865:20231016:135834.256 In zbx_execute_threaded_metric() key:'vfs.fs.get'
19931:20231016:135834.256 executing in data process for key:'vfs.fs.get'
19865:20231016:135834.296 End of zbx_execute_threaded_metric():SYSINFO_SUCCEED ''
19865:20231016:135834.296 Sending back [[{"fsname":"/sys","fstype":"sysfs","bytes":{"total":0,"free":0,"used":0,"pfree":0.000000,"pused":0.000000},"inodes":{"total":0,"free":0,"used":0,"pfree":100.000000,"pused":0.000000},"options":"rw,seclabel,nosuid,nodev,noexec,relatime"},{"fsname":"/proc","fstype":"proc","bytes":{"total":0,"free":0,"used":0,"pfree":0.000000,"pused":0.000000},"inodes":{"total":0,"free":0,"used":0,"pfree":100.000000,"pused":0.000000},"options":"rw,nosuid,nodev,noexec,relatime"},{"fsname":"/dev","fstype":"devtmpfs","bytes":{"total":1974575104,"free":1974575104,"used":0,"pfree":100.000000,"pused":0.000000},"inodes":{"total":482074,"free":481656,"used":418,"pfree":99.913291,"pused":0.086709},"options":"rw,seclabel,nosuid,size=1928296k,nr_inodes=482074,mode=755"},{"fsname":"/sys/kernel/security","fstype":"securityfs","bytes":{"total":0,"free":0,"used":0,"pfree":0.000000,"pused":0.000000},"inodes":{"total":0,"free":0,"used":0,"pfree":100.000000,"pused":0.000000},"options":"rw,nosuid,nodev,noexec,relatime"},{"fsname":"/dev/shm","fstype":"tmpfs","bytes":{"total":1986715648,"free":1986715648,"used":0,"pfree":100.000000,"pused":0.000000},"inodes":{"total":485038,"free":485037,"used":1,"pfree":99.999794,"pused":0.000206},"options":"rw,seclabel,nosuid,nodev"},{"fsname":"/dev/pts","fstype":"devpts","bytes":{"total":0,"free":0,"used":0,"pfree":0.000000,"pused":0.000000},"inodes":{"total":0,"free":0,"used":0,"pfree":100.000000,"pused":0.000000},"options":"rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000"},{"fsname":"/run","fstype":"tmpfs","bytes":{"total":1986715648,"free":1919344640,"used":67371008,"pfree":96.608925,"pused":3.391075},"inodes":{"total":485038,"free":484364,"used":674,"pfree":99.861042,"pused":0.138958},"options":"rw,seclabel,nosuid,nodev,mode=755"},{"fsname":"/sys/fs/cgroup","fstype":"tmpfs","bytes":{"total":1986715648,"free":1986715648,"used":0,"pfree":100.000000,"pused":0.000000},"inodes":{"total":485038,"free":485022,"used":16,"pfree":99.996701,"pused":0.003299},"options":"ro,seclabel,nosuid,nodev,noexec,mode=755"},{"fsname":"/sys/fs/cgroup/systemd","fstype":"cgroup","bytes":{"total":0,"free":0,"used":0,"pfree":0.000000,"pused":0.000000},"inodes":{"total":0,"free":0,"used":0,"pfree":100.000000,"pused":0.000000},"options":"rw,seclabel,nosuid,nodev,noexec,relatime,xattr,release_agent=/usr/lib/systemd/systemd-cgroups-agent,name=systemd"},{"fsname":"/sys/fs/pstore","fstype":"pstore","bytes":{"total":0,"free":0,"used":0,"pfree":0.000000,"pused":0.000000},"inodes":{"total":0,"free":0,"used":0,"pfree":100.000000,"pused":0.000000},"options":"rw,nosuid,nodev,noexec,relatime"},{"fsname":"/sys/fs/cgroup/net_cls,net_prio","fstype":"cgroup","bytes":{"total":0,"free":0,"used":0,"pfree":0.000000,"pused":0.000000},"inodes":{"total":0,"free":0,"used":0,"pfree":100.000000,"pused":0.000000},"options":"rw,seclabel,nosuid,nodev,noexec,relatime,net_prio,net_cls"},{"fsname":"/sys/fs/cgroup/blkio","fstype":"cgroup","bytes":{"total":0,"free":0,"used":0,"pfree":0.000000,"pused":0.000000},"inodes":{"total":0,"free":0,"used":0,"pfree":100.000000,"pused":0.000000},"options":"rw,seclabel,nosuid,nodev,noexec,relatime,blkio"},{"fsname":"/sys/fs/cgroup/hugetlb","fstype":"cgroup","bytes":{"total":0,"free":0,"used":0,"pfree":0.000000,"pused":0.000000},"inodes":{"total":0,"free":0,"used":0,"pfree":100.000000,"pused":0.000000},"options":"rw,seclabel,nosuid,nodev,noexec,relatime,hugetlb"},{"fsname":"/sys/fs/cgroup/freezer","fstype":"cgroup","bytes":{"total":0,"free":0,"used":0,"pfree":0.000000,"pused":0.000000},"inodes":{"total":0,"free":0,"used":0,"pfree":100.000000,"pused":0.000000},"options":"rw,seclabel,nosuid,nodev,noexec,relatime,freezer"},{"fsname":"/sys/fs/cgroup/cpu,cpuacct","fstype":"cgroup","bytes":{"total":0,"free":0,"used":0,"pfree":0.000000,"pused":0.000000},"inodes":{"total":0,"free":0,"used":0,"pfree":100.000000,"pused":0.000000},"options":"rw,seclabel,nosuid,nodev,noexec,relatime,cpuacct,cpu"},{"fsname":"/sys/fs/cgroup/devices","fstype":"cgroup","bytes":{"total":0,"free":0,"used":0,"pfree":0.000000,"pused":0.000000},"inodes":{"total":0,"free":0,"used":0,"pfree":100.000000,"pused":0.000000},"options":"rw,seclabel,nosuid,nodev,noexec,relatime,devices"},{"fsname":"/sys/fs/cgroup/pids","fstype":"cgroup","bytes":{"total":0,"free":0,"used":0,"pfree":0.000000,"pused":0.000000},"inodes":{"total":0,"free":0,"used":0,"pfree":100.000000,"pused":0.000000},"options":"rw,seclabel,nosuid,nodev,noexec,relatime,pids"},{"fsname":"/sys/fs/cgroup/memory","fstype":"cgroup","bytes":{"total":0,"free":0,"used":0,"pfree":0.000000,"pused":0.000000},"inodes":{"total":0,"free":0,"used":0,"pfree":100.000000,"pused":0.000000},"options":"rw,seclabel,nosuid,nodev,noexec,relatime,memory"},{"fsname":"/sys/fs/cgroup/cpuset","fstype":"cgroup","bytes":{"total":0,"free":0,"used":0,"pfree":0.000000,"pused":0.000000},"inodes":{"total":0,"free":0,"used":0,"pfree":100.000000,"pused":0.000000},"options":"rw,seclabel,nosuid,nodev,noexec,relatime,cpuset"},{"fsname":"/sys/fs/cgroup/perf_event","fstype":"cgroup","bytes":{"total":0,"free":0,"used":0,"pfree":0.000000,"pused":0.000000},"inodes":{"total":0,"free":0,"used":0,"pfree":100.000000,"pused":0.000000},"options":"rw,seclabel,nosuid,nodev,noexec,relatime,perf_event"},{"fsname":"/sys/kernel/config","fstype":"configfs","bytes":{"total":0,"free":0,"used":0,"pfree":0.000000,"pused":0.000000},"inodes":{"total":0,"free":0,"used":0,"pfree":100.000000,"pused":0.000000},"options":"rw,relatime"},{"fsname":"/","fstype":"xfs","bytes":{"total":4284481536,"free":1456164864,"used":2828316672,"pfree":33.986956,"pused":66.013044},"inodes":{"total":2097152,"free":2035732,"used":61420,"pfree":97.071266,"pused":2.928734},"options":"rw,seclabel,relatime,attr2,inode64,noquota"},{"fsname":"/sys/fs/selinux","fstype":"selinuxfs","bytes":{"total":0,"free":0,"used":0,"pfree":0.000000,"pused":0.000000},"inodes":{"total":0,"free":0,"used":0,"pfree":100.000000,"pused":0.000000},"options":"rw,relatime"},{"fsname":"/proc/sys/fs/binfmt_misc","fstype":"autofs","bytes":{"total":0,"free":0,"used":0,"pfree":0.000000,"pused":0.000000},"inodes":{"total":0,"free":0,"used":0,"pfree":100.000000,"pused":0.000000},"options":"rw,relatime,fd=31,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=12719"},{"fsname":"/backup","fstype":"autofs","bytes":{"total":10995116277760,"free":6773774745600,"used":4221341532160,"pfree":61.607122,"pused":38.392878},"inodes":{"total":13230283060,"free":13230027851,"used":255209,"pfree":99.998071,"pused":0.001929},"options":"rw,relatime,fd=37,pgrp=1,timeout=60,minproto=5,maxproto=5,direct,pipe_ino=12818"},{"fsname":"/dev/mqueue","fstype":"mqueue","bytes":{"total":0,"free":0,"used":0,"pfree":0.000000,"pused":0.000000},"inodes":{"total":0,"free":0,"used":0,"pfree":100.000000,"pused":0.000000},"options":"rw,seclabel,relatime"},{"fsname":"/dev/hugepages","fstype":"hugetlbfs","bytes":{"total":0,"free":0,"used":0,"pfree":0.000000,"pused":0.000000},"inodes":{"total":0,"free":0,"used":0,"pfree":100.000000,"pused":0.000000},"options":"rw,seclabel,relatime"},{"fsname":"/sys/kernel/debug","fstype":"debugfs","bytes":{"total":0,"free":0,"used":0,"pfree":0.000000,"pused":0.000000},"inodes":{"total":0,"free":0,"used":0,"pfree":100.000000,"pused":0.000000},"options":"rw,relatime"},{"fsname":"/sys/fs/fuse/connections","fstype":"fusectl","bytes":{"total":0,"free":0,"used":0,"pfree":0.000000,"pused":0.000000},"inodes":{"total":0,"free":0,"used":0,"pfree":100.000000,"pused":0.000000},"options":"rw,relatime"},{"fsname":"/boot","fstype":"xfs","bytes":{"total":1063256064,"free":820473856,"used":242782208,"pfree":77.166158,"pused":22.833842},"inodes":{"total":524288,"free":523948,"used":340,"pfree":99.935150,"pused":0.064850},"options":"rw,seclabel,relatime,attr2,inode64,noquota"},{"fsname":"/home","fstype":"xfs","bytes":{"total":1063256064,"free":1028100096,"used":35155968,"pfree":96.693556,"pused":3.306444},"inodes":{"total":524288,"free":524024,"used":264,"pfree":99.949646,"pused":0.050354},"options":"rw,seclabel,relatime,attr2,inode64,noquota"},{"fsname":"/var","fstype":"xfs","bytes":{"total":5358223360,"free":2801045504,"used":2557177856,"pfree":52.275639,"pused":47.724361},"inodes":{"total":2621440,"free":2614855,"used":6585,"pfree":99.748802,"pused":0.251198},"options":"rw,seclabel,relatime,attr2,inode64,noquota"},{"fsname":"/opt/data","fstype":"xfs","bytes":{"total":21464350720,"free":7818596352,"used":13645754368,"pfree":36.425963,"pused":63.574037},"inodes":{"total":10485760,"free":10485423,"used":337,"pfree":99.996786,"pused":0.003214},"options":"rw,seclabel,relatime,attr2,inode64,noquota"},{"fsname":"/opt/admin","fstype":"xfs","bytes":{"total":264933376,"free":251285504,"used":13647872,"pfree":94.848564,"pused":5.151436},"inodes":{"total":131072,"free":131069,"used":3,"pfree":99.997711,"pused":0.002289},"options":"rw,seclabel,relatime,attr2,inode64,noquota"},{"fsname":"/opt/app","fstype":"xfs","bytes":{"total":3210739712,"free":2614648832,"used":596090880,"pfree":81.434469,"pused":18.565531},"inodes":{"total":1572864,"free":1570644,"used":2220,"pfree":99.858856,"pused":0.141144},"options":"rw,seclabel,relatime,attr2,inode64,noquota"},{"fsname":"/opt/app/logs","fstype":"xfs","bytes":{"total":5358223360,"free":2842509312,"used":2515714048,"pfree":53.049474,"pused":46.950526},"inodes":{"total":2621440,"free":2621402,"used":38,"pfree":99.998550,"pused":0.001450},"options":"rw,seclabel,relatime,attr2,inode64,noquota"},{"fsname":"/tmp","fstype":"xfs","bytes":{"total":2136997888,"free":2102870016,"used":34127872,"pfree":98.402999,"pused":1.597001},"inodes":{"total":1048576,"free":1048548,"used":28,"pfree":99.997330,"pused":0.002670},"options":"rw,seclabel,relatime,attr2,inode64,noquota"},{"fsname":"/var/lib/nfs/rpc_pipefs","fstype":"rpc_pipefs","bytes":{"total":0,"free":0,"used":0,"pfree":0.000000,"pused":0.000000},"inodes":{"total":0,"free":0,"used":0,"pfree":100.000000,"pused":0.000000},"options":"rw,relatime"},{"fsname":"/proc/sys/fs/binfmt_misc","fstype":"binfmt_misc","bytes":{"total":0,"free":0,"used":0,"pfree":0.000000,"pused":0.000000},"inodes":{"total":0,"free":0,"used":0,"pfree":100.000000,"pused":0.000000},"options":"rw,relatime"},{"fsname":"/run/user/5127","fstype":"tmpfs","bytes":{"total":397344768,"free":397344768,"used":0,"pfree":100.000000,"pused":0.000000},"inodes":{"total":485038,"free":485037,"used":1,"pfree":99.999794,"pused":0.000206},"options":"rw,seclabel,nosuid,nodev,relatime,size=388032k,mode=700,uid=5127,gid=5127"},{"fsname":"/backup","fstype":"nfs4","bytes":{"total":10995116277760,"free":6773774745600,"used":4221341532160,"pfree":61.607122,"pused":38.392878},"inodes":{"total":13230283060,"free":13230027851,"used":255209,"pfree":99.998071,"pused":0.001929},"options":"rw,sync,relatime,vers=4.1,rsize=1048576,wsize=1048576,namlen=255,hard,proto=tcp,timeo=600,retrans=2,sec=sys,clientaddr=xx.xx.xx.xx,local_lock=none,addr=xx.xx.xx.xx"}]]
19865:20231016:135834.296 zbx_setproctitle() title:'listener #2 [waiting for connection]'
19863:20231016:135834.653 zbx_setproctitle() title:'collector [processing data]'
19863:20231016:135834.653 In update_cpustats()
19863:20231016:135834.653 End of update_cpustats()
19863:20231016:135834.653 zbx_setproctitle() title:'collector [idle 1 sec]'
19863:20231016:135835.654 zbx_setproctitle() title:'collector [processing data]'
19863:20231016:135835.654 In update_cpustats()
19863:20231016:135835.654 End of update_cpustats()
19863:20231016:135835.654 zbx_setproctitle() title:'collector [idle 1 sec]'
19866:20231016:135836.384 zbx_setproctitle() title:'listener #3 [processing request]'
19866:20231016:135836.384 Requested [vfs.file.contents[/sys/block/sdc/stat]]
19866:20231016:135836.385 Sending back [ 1470370 55 167361663 3849607 14083831 191 509879254 11601279 0 8995772 15442978]
19866:20231016:135836.385 zbx_setproctitle() title:'listener #3 [waiting for connection]'
Oct 16 13:58:34 192.168.1.6 systemd[1]: Got automount request for /backup, triggered by 19931 (zabbix_agentd)
|
| Comment by Edgars Melveris [ 2023 Oct 16 ] |
|
Can you also add details about the OS and it's version and where is this FS mounted on? IE if it makes a difference where it's mounted? |
| Comment by fbo [ 2023 Oct 16 ] |
|
Last update done in September. uname -rvm Linux 3.10.0-1160.95.1.el7.x86_64 #1 SMP Fri Jun 23 08:44:55 EDT 2023 x86_64 cat /etc/redhat-release Red Hat Enterprise Linux Server release 7.9 (Maipo) No difference where it is mounted. I will have to double check but we don't have this problem on our old zabbix infra. (Zabbix agent 5.0.9 and Zabbix server 5.2.6) |
| Comment by Jason [ 2023 Oct 17 ] |
|
For me, I only see the issue when using agent2, I thought. I'm running anything from Debian 9-12 with the issue. It's doing it for CIFS and NFS. |
| Comment by Edgars Melveris [ 2023 Oct 20 ] |
|
Is the mount point unmounted when you get the log entry? I could reproduce this with both agents, but only if the mountpoint is unmounted at the time of the request. |
| Comment by fbo [ 2023 Oct 20 ] |
|
Yes mount point is unmounted. |
| Comment by Jason [ 2023 Oct 20 ] |
|
As the OP, this isn't my experience; systemd unmounts just fine, and then on the next zabbix poll it is auto mounted again. |
| Comment by Edgars Melveris [ 2023 Oct 20 ] |
|
As a workaround you can use the legacy approach for discovery: And then use legacy items to collect values for each FS individually: |
| Comment by Jason [ 2023 Oct 22 ] |
|
That would mean different history etc though, wouldn't it? Is the bug complex to fix? |
| Comment by Denis Pavlov [ 2024 Mar 25 ] |
|
found same issue with before vfs.fs.get agent call no sshfs mount in /proc/mounts on agent host vfs.fs.get call somehow triggers automount, I get remote FS mounted I guess it is never unmounted because server calls vfs.fs.get too frequently, systemd has no chance to see idle state. vfs.fs.get should not trigger mounts, and to best performance apply filters before getting data on agent # cat /proc/mounts ... extbackup@extbackup.****:/mnt/cloud_servers/*** /mnt/backup fuse.sshfs rw,relatime,user_id=0,group_id=0,allow_other 0 0
# cat /etc/redhat-release AlmaLinux release 8.9 (Midnight Oncilla) |
| Comment by Denis Pavlov [ 2024 Jul 27 ] |
|
vfs.fs.get is very evil now in terms of performance and triggering automounts Will it ever be fixed? |
| Comment by Denis Pavlov [ 2024 Dec 16 ] |
|
Anyone? |
| Comment by Jeff Schwartz [ 2025 Jan 28 ] |
|
I'm currently building a new Zabbix 7.0 environment to replace 6.0. I'm seeing this same behavior with zabbix-agent2-7.0.8/zabbix-server-mysql-7.0.8-release1.el9.x86_64. This is not present in our current production environment running zabbix-agent2-6.0.36/zabbix-server-mysql-6.0.32. Evil, indeed. |
| Comment by Jeff Schwartz [ 2025 Jan 28 ] |
|
Another workaround: /etc/sysconfig/autofs
BROWSE_MODE="no"
This removes the mount points from /proc/mounts. Would still like to see this fixed in the code though. |
| Comment by Jason [ 2025 Jan 29 ] |
|
My ticket kinda got highjacked a fair bit back in 2023, and a developer (zux) appears to have run with someone elses responses and not mine. I'm running Debian, not RedHat. BROWSE_MODE appears to be:
Annoyingly, systemd refers to a mount type of autofs which confuses things a little, because that typically refers to the classic automount (eg. /etc/auto.master etc) – see [autofs(5)|https://www.man7.org/linux/man-pages/man5/autofs.5.html.] Hey zabbix.dev can we get some attention on this long-standing bug please?
|
| Comment by Jason [ 2025 Feb 12 ] |
|
This is due to Essentially the move in the template from vfs.fs.discovery to vfs.fs.get. Bizarre how this wasn't tested before implementation. |
| Comment by Jason [ 2025 Feb 12 ] |
|
Meanwhile, I'm using a slightly annoying-but-works workaround; using Linux namespaces and systemd's InaccessiblePaths. Use systemctl edit to edit the unit for Zabbix agent (dropin mode), and put the following in it: [Service] InaccessiblePaths=/mnt (Or whatever paths you need). Finally, I'm now seeing auto unmounting again!
|