[ZBXNEXT-1567] New item: Count standard files in directory (Z4) Created: 2013 Jan 12  Updated: 2024 Apr 10  Resolved: 2017 Dec 16

Status: Closed
Project: ZABBIX FEATURE REQUESTS
Component/s: Agent (G)
Affects Version/s: 2.0.4
Fix Version/s: 4.0.0alpha1, 4.0 (plan)

Type: New Feature Request Priority: Minor
Reporter: Marcel Hecko Assignee: Unassigned
Resolution: Fixed Votes: 9
Labels: count, directory, newitemkey, patch
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified
Environment:

gcc version 4.7.2 (Ubuntu/Linaro 4.7.2-2ubuntu1)


Attachments: PNG File 13.png     File vfs.dir.filecount.patch     PNG File win_dir_file.png     File zbxNext1567.c    
Issue Links:
Causes
causes ZBX-13264 Cannot compile Zabbix agent on OS/X Closed
Duplicate
Sub-task
part of ZBXNEXT-6875 Migrate Agent2 metrics to native GoLang Closed
Team: Team A
Team: Team A
Sprint: Sprint 20, Sprint 21, Sprint 22
Story Points: 2.5

 Description   

Attaching a patch for a new item: vfs.dir.filecount[<dirname>] which counts files in directory. These are only standard files, not special files like links or socket pointers.
Tested on Linux, should be ready to use in Windows, however not tested.



 Comments   
Comment by Neil [ 2013 Feb 16 ]

a great & needed idea!

Comment by Wintel Servers [ 2013 Mar 21 ]

Hi Guys, That is what I really looking for. Please how can I implement that on my environment.

Zabbix 2.04.

I appreciate your help. Thanks.

Comment by Volker Fröhlich [ 2013 May 11 ]

I think that will lead up to all kinds of follow-up requests, requesting to immitate the "find" command. It also has a certain timeout potential.

I can see that Windows machines probably need something else. As far as Linux is concerned, I'd rather go for a configuration management tool or package management.

Comment by richlv [ 2013 May 11 ]

loadable modules could helpalot. once they are documented

Comment by Peter Schmeisser [ 2014 Feb 14 ]

we use simple Userparameter like this:
UserParameter=our.dir.filecount[*],ls -f $1 2>&1 |awk ' /Permission denied/ || /not found/

{x=-1 ; exit}

{ x++ }

END

{ print x }

'

this use option "-f" for more performance
and send "-1" if there "permissions denied" (SunOS&Linux) or "not found" (HPUX)
We have tested with 1.5 Mill Files
Reaction time was 2s
and it is very deliberately not recursive
we only need to know the count of files in one directory

but i know, a c-module is better

Comment by Matthias Baur [ 2014 May 20 ]

@pedrums Problem with 'ls -f' is, that it also shows you the '.' and '..' links. I've tweaked it a bit and it now looks like this:

UserParameter=dir.filecount[*],ls $1 2>&1 |awk ' /Permission denied/ || /not found/ {x=-1 ; exit} { x++ } END { if (x) print x; else print 0 } '
Comment by Alper Karayilan [ 2014 Dec 31 ]

Here is a very nice and simple solution for Windows:

https://87.110.183.172/forum/showpost.php?p=150088&postcount=29

Item:

system.run[dir /A-d "C:\failed\logs\" | find /c "/"]

Example trigger:

{hostname:system.run[dir /A-d "C:\failed\logs\" | find /c "/"].str("File Not Found")}=0   

(fires if a file appears in the folder)

Comment by Marc [ 2015 Jan 01 ]

Attached file zbxNext1567.c is an implementation of marcel's patch as loadable module.

Comment by Daniel V [ 2016 Dec 19 ]

For Windows 10 or Windows Server 2016 the Example from alper.karayilan broke.

A little adjustment and it works again:

dir /A-d /B "D:\yourfolder" |find /v /c ""

Also be aware, that the Trigger checks for "File Not Found" which is fine as long the system language of the host is English.

Comment by Andris Zeila [ 2017 Nov 24 ]

(1) We have str2uint64() function that should be used when parsing size parameter.

valdis Implemented in r75081. RESOLVED

wiper CLOSED

Comment by Andris Zeila [ 2017 Nov 24 ]

(2) The following code in function etypes_to_mask() appears to be redundant:

	if (DET_DEV & ret)
		ret |= DET_DEV2;

	if (DET_ALL & ret)
		ret |= DET_MASK;

The DET_ALL and DET_DEV cases are already handled by etype_to_mask() function and cannot be returned.

Also it would be better to change DET_MASK and DET_FLAG defines to

#define DET_MASK	(DET_FILE | DET_DIR | DET_SYM | DET_SOCK | DET_BDEV | DET_CDEV | DET_FIFO)
#define DET_DEV2	(DET_BDEV | DET_CDEV)

And I think DET_MASK should be renamed to DET_ALL, for example types_incl = DET_ALL; would be more explanatory than types_incl = DET_MASK;.
While at it - we use hex format when defining flags.

valdis Nine user-visible strings get translated into macros with exactly corresponding names. If we translate "all" into DET_EVERYTHING and use DET_ALL with a different meaning, that will be less consistent and thus more confusing. DET_MASK might not be the perfect name, but very reasonable, because it is indeed used as a mask in the final expression.

valdis As a compromise, we now have DET_ALLMASK and the other improvements without compromises in r75081. RESOLVED

wiper CLOSED with minor fix in r75108

Comment by Andris Zeila [ 2017 Nov 24 ]

(3) Compiler warnings:

Making all in common
dir.c: In function ‘prepare_count_parameters’:
dir.c:324:59: warning: unused parameter ‘min_time’ [-Wunused-parameter]
   zbx_uint64_t *min_size, zbx_uint64_t *max_size, time_t *min_time, time_t *max_time)
                                                           ^~~~~~~~
dir.c:324:77: warning: unused parameter ‘max_time’ [-Wunused-parameter]
   zbx_uint64_t *min_size, zbx_uint64_t *max_size, time_t *min_time, time_t *max_time)
                                                                             ^~~~~~~~
dir.c: In function ‘vfs_dir_count’:
dir.c:906:17: warning: comparison between signed and unsigned integer expressions [-Wsign-compare]
       (min_size <= status.st_size && status.st_size <= max_size) &&
                 ^~
dir.c:906:53: warning: comparison between signed and unsigned integer expressions [-Wsign-compare]
       (min_size <= status.st_size && status.st_size <= max_size) &&
                                                     ^~

valdis RESOLVED in r75081.

wiper CLOSED

Comment by Andris Zeila [ 2017 Nov 24 ]

(4) My suggestion would be to use zbx_snprintf_alloc() in vfs_dir_count() function to make path and zbx_strdup() it in queue_directory() function:

  1. less memory allocations for non directory objects
  2. avoiding 'hidden' ownership transfer

wiper Similar pattern was already used in vfs_dir_size() function. Ideally we should fix in both places, but for now we will leave it.
WONTFIX

Comment by Andris Zeila [ 2017 Nov 24 ]

(5) We should not directly manipulate vector values_num property unless really needed. In this case (vfs_dir_count) use zbx_vector_ptr_remove (or zbx_vetor_ptr_remove_noorder - no difference when removing the last element) and replace list.values_num++; with

		zbx_free(item->path);
		zbx_free(item);

Good way would be to wrap it in a cleanup function (directory_item_free() for example) and reuse it when freeing the vector:

zbx_vector_ptr_clear_ext(&list, directory_item_free);
zbx_vector_ptr_destroy(&list);

Also it would be better to pick names that better reflects the stored data, for example list -> directory_items, item -> directory_item

wiper Similar pattern was already used in vfs_dir_size() function. Ideally we should fix in both places, but for now we will leave it.
WONTFIX

Comment by Andris Zeila [ 2017 Nov 27 ]

I noticed that comments (4), (5) are also relevant to vfs.dir.size key (which probably was taken as base template). So we either fix in both places or none. More on this later.

wiper It was decided to leave it.

Comment by Andris Zeila [ 2017 Nov 27 ]

(6) Coding style issues (taken from https://www.zabbix.org/wiki/C_coding_guidelines):
Hexadecimal constants should be written in lowercase:

	zbx_uint64_t		min_size = 0, max_size = 0x7fffffffffffffff;
	time_t			min_time = 0, max_time = 0x7fffffff;

If a constant is used in a bitwise expression, it should be written on the right:

(S_ISREG(status.st_mode) && 0 != (types & DET_FILE)) ||

If a numeric value or a pointer is tested for being non-zero, write 0 or NULL explicitly:

if (0 != (types & DET_DIR) && 0 != match)

valdis RESOLVED in r75081.

wiper CLOSED with a minor addition in r75116.
Regarding the case of hexadecimal constants - while it does not follow our coding style guide it does improve readability of longer constants. It might make a sense to adopt it.

Comment by Andris Zeila [ 2017 Nov 28 ]

(7) Memory leak in etypes_to_mask() function - etype returned by get_param_dyn() must be freed afterwards.

==24835== 5 bytes in 1 blocks are definitely lost in loss record 2 of 27
==24835==    at 0x4C2BBAF: malloc (vg_replace_malloc.c:299)
==24835==    by 0x14CE96: zbx_malloc2 (misc.c:491)
==24835==    by 0x15340D: get_param_dyn (str.c:1363)
==24835==    by 0x138C01: etypes_to_mask (dir.c:268)
==24835==    by 0x138D35: prepare_count_parameters (dir.c:300)
==24835==    by 0x1396BD: vfs_dir_count (dir.c:826)
==24835==    by 0x127972: zbx_execute_threaded_metric (sysinfo.c:1275)
==24835==    by 0x139B99: VFS_DIR_COUNT (dir.c:919)
==24835==    by 0x12641A: process (sysinfo.c:636)
==24835==    by 0x125A02: test_parameter (sysinfo.c:401)
==24835==    by 0x124AEC: main (zabbix_agentd.c:1206)

valdis RESOLVED in r75171

wiper CLOSED

Comment by Andris Zeila [ 2017 Nov 28 ]

(8) On windows vfs.dir.count appears to follow symbolic links and junctions.

valdis After r75171 – no more! Now it also avoids infinite recursion. RESOLVED

wiper CLOSED

Comment by Andris Zeila [ 2017 Nov 30 ]

Successfully tested

Comment by Valdis Kauķis (Inactive) [ 2017 Nov 30 ]

Specification is updated to include min_age and max_age parameters.

Comment by Andris Zeila [ 2017 Dec 01 ]

(11) Coding style in prepare_count_parameters():

Avoid using complicated expressions (more complicated than simple assignment of a single value) in variable declarations.

valdis RESOLVED in r75301

wiper CLOSED

Comment by Valdis Kauķis (Inactive) [ 2017 Dec 07 ]

Available in:

  • 4.0.0alpha1 r75517
Comment by Natalja Cernohajeva (Inactive) [ 2017 Dec 07 ]

All necessary documentation update has been coordinated with valdis and executed. Please review and let me know if anything else needs to be done:

Zabbix agent items;
WhatsNew.

Comment by Valdis Kauķis (Inactive) [ 2017 Dec 07 ]

(14) [D] In both documentation links, "Directory file count." should be adjusted to "Directory entry count.". Some of the entries that can be counted technically are not files, e.g. sub-directories.

natalja.cernohajeva Please, confirm the change.
RESOLVED

valdis CLOSED

Comment by Glebs Ivanovskis (Inactive) [ 2017 Dec 14 ]

(15) [D] Not documented in https://www.zabbix.com/documentation/4.0/manual/regular_expressions#regular_expression_support_by_location

vso please add to same place as vfs.dir.size[]

martins-v Thanks for noticing, RESOLVED

vso CLOSED

Comment by Judy [ 2018 Nov 27 ]

Bugs in vfs.dir.count[] on Windows Server 2012. Was using this to monitor the number of files in a folder which could have as many as a few hundred thousand.
Bug1: when the number of files went above circa 30k, the monitoring stopped working until the count went down again.
Bug2: the Zabbix agent often left open handles on random files which were not released until the agent was restarted. This prevented other parts of the system from working properly

Comment by Vladislavs Sokurenko [ 2018 Nov 27 ]

Thank you for your comment Schneider I have created a bug report ZBX-15225. Maybe there were some kind of errors in the log file ?
How big is Timeout variable on zabbix agent ? I think this could happen that thread is killed with handle open if it hangs. One solution would be that parent process must store active handles and close after killing a thread.

Generated at Sat Apr 20 15:47:07 EEST 2024 using Jira 9.12.4#9120004-sha1:625303b708afdb767e17cb2838290c41888e9ff0.