[ZBX-10884] log[] and logrt[] items repeatedly reread log file from the beginning if file system is 100% full and the log file is being appended Created: 2016 Jun 07  Updated: 2020 Aug 05

Status: Open
Project: ZABBIX BUGS AND ISSUES
Component/s: Agent (G)
Affects Version/s: 2.0.18, 3.2.0alpha1
Fix Version/s: None

Type: Incident report Priority: Trivial
Reporter: Andris Mednis Assignee: Unassigned
Resolution: Unresolved Votes: 0
Labels: log, logmonitoring, logrt
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified
Environment:

Debian testing GNU/Linux



 Description   

If a file system with log files is 100% full and new records are being written into a log file it results in a log file not changing its size anymore but constantly updating its modification time. Zabbix agent waits a little time in a hope that things will stabilize but in the end concludes that file has been overwritten and reanalyzes it from the start. All matching lines are sent to server again. This cycle repeats until situation is resolved.



 Comments   
Comment by Andris Mednis [ 2016 Jun 07 ]

Documented in "Known issues":
https://www.zabbix.com/documentation/2.0/manual/installation/known_issues#known_issues_for_2013_and_later
https://www.zabbix.com/documentation/2.2/manual/installation/known_issues#known_issues_for_224_and_later
https://www.zabbix.com/documentation/3.0/manual/installation/known_issues#log_file_monitoring
https://www.zabbix.com/documentation/3.2/manual/installation/known_issues#log_file_monitoring

Comment by Andris Mednis [ 2016 Jun 08 ]

One solution could be modifying the function is_same_file(). There is already a check for not changing file size and increasing 'mtime'. If this is the case then check free space in file system. As proposed by sandis.neilands the VFS_FS_SIZE(<mount_point>, free) function can be used for that. If there are 0 free bytes, then assume it is the same file, do not reread it. On MS Windows the function GetVolumePathName() could be used to get mount point from file name. On UNIX getting mount point from file name is more complicated, also symbolic links should be taken into account.

Comment by Ronald Rood [ 2020 Aug 05 ]

This could easily be prevented by taking a hash value of the first 100 or so bytes of the file. Most logs will have a timestamp in them .... so if the hash does not change, the file did not change. I was in the assumption this was already done ....

Generated at Fri Mar 29 15:19:24 EET 2024 using Jira 9.12.4#9120004-sha1:625303b708afdb767e17cb2838290c41888e9ff0.