[ZBX-19576] False positive in network service discovery Created: 2021 Jun 17  Updated: 2021 Dec 28

Status: Need info
Project: ZABBIX BUGS AND ISSUES
Component/s: None
Affects Version/s: None
Fix Version/s: None

Type: Problem report Priority: Trivial
Reporter: msl0 Assignee: Zabbix Support Team
Resolution: Unresolved Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified
Environment:

Ubuntu 20.04, Zabbix Server 5.0.9


Attachments: PNG File image-2021-06-21-09-17-45-357.png     PNG File image-2021-06-21-09-19-18-111.png     PNG File image-2021-06-21-09-20-26-230.png     PNG File image-2021-06-22-09-16-06-425.png    

 Description   

Sometimes network discovery generates service discovered events even if service is continuous running on remote host.
It could be caused with single package lost, hard to say. Such situations generate unwanted discovery actions.

In my opinion it could be fixed by adding option to specify number of check attempt when service state is changing. For example if SSH discovery check failed, when previous time it was in UP state, discovery process should run next attempt before generate event

If there is currently another mechanism to avoid problem with generating unwanted discovery events in Zabbix, please let me know



 Comments   
Comment by Dmitry Krupornitsky [ 2021 Jun 18 ]

Hello, 

Please show your discovery settings and action for discovery events configuration.

Also please enclose info about false positives: what are they, any distinctive features and so on.

Thank you

Comment by msl0 [ 2021 Jun 21 ]

Hello Dmitry,

As I said before I try to discover service in subnet. Sometimes Zabbix generates new "service discovered events" even if service was not touched (is continuously running) and service was discovered in past. When I checked the service, it was still available.

Is there possibility to setup number of attempt when check fail?

 

Comment by Dmitry Krupornitsky [ 2021 Jun 21 ]

Okay, lets modify a bit your algorithm to make it less sensitive. 

In discovery conditions section:

  • Discovery status equals UP (Service Up means the host has been up the last few times Zabbix discovered it)
  • SSH check
  • Uptime/Downtime is greater/less than 1800s (modify it as you wish) - for uptime 5min or more

Don't you have duplicates in system.hostname for hosts in the network?

What are the settings for 'Network discovery data storage period'? (Administration->General->Housekeeping)? 

Unfortunately the network discovery is a pretty straightforward process done by discovery processes and you can't setup complex discovery rules for separate host if it fails a part of the test.

Comment by msl0 [ 2021 Jun 22 ]

AFAIK Service UP generates events every time when service is up. Provided condition is met when SSH is up more than 1800s.
After modifying the condition of my action, I can see that the action is performed every time the discovery process starts. So I get more alerts than earlier.

No I don't have duplicates in system.hostname. Some hosts have more than one IP address, so that's why I use system.hostname.

I use default settings for housekeeping.

 

I didn't mean "complex discovery rules for separate host" but just simple number of retry if discovery check fail (or change from UP to DOWN state) for any of hosts in scope of discovery rule. This can effectively reduce the sensitivity of the discovery process if its needed in some cases. If currently that option doesn't exists, maybe there is possibility to add that simple feature in next releases. What do you think?

Comment by Dmitry Krupornitsky [ 2021 Jul 08 ]

Hello, 

We need to check what comes to Zabbix and what makes it think we've a newly discovered host.

If you have MySQL then what I need to execute the following SQL on your DB CLI shortly after you have encountered alerts for discovery:

Select
    zabbix.dservices.dns As `Services.DNS`,
    zabbix.dservices.ip As `Services.IP`,
    From_UnixTime(zabbix.dservices.lastdown) As `Services.Lastdown`,
    From_UnixTime(zabbix.dservices.lastup) As `Services.Lastup`,
    zabbix.dservices.port As `Services.Port`,
    zabbix.dservices.status As `Services.Status`,
    zabbix.dservices.value As `Services.Value`,
    zabbix.dhosts.status As `Host.status`,
    From_UnixTime(zabbix.dhosts.lastup) As `Host.lastup`,
    From_UnixTime(zabbix.dhosts.lastdown) As `Host.lastdown`,
    zabbix.drules.name As NetwDiskRuleName
From
    zabbix.dservices Inner Join
    zabbix.dhosts On zabbix.dservices.dhostid = zabbix.dhosts.dhostid Inner Join
    zabbix.drules On zabbix.dhosts.druleid = zabbix.drules.druleid Inner Join
    zabbix.dservices dservices1 On dservices1.dhostid = zabbix.dhosts.dhostid
Order By
    `Services.IP`

It just displays the contents of 3 discovery-related tables: dservices, dhosts, drules. Please look for strange items in the result or enclose it here having removed all sensitive information. It may be a wise idea to have it run twice - when everything is OK and after a false alert.

Thank you.

Comment by msl0 [ 2021 Jul 16 ]

Hi,

I received alert from hosts
  - 192.168.0.208 on 16.07.2021 at 04:27
  - 192.168.0.223 on 16.07.2021 at 15:47

Device service name: SSH
Device service port: 22
Device service status: UP
Device service uptime: 5s

I get the most strange alerts from these 2 hosts (always only SSH service), but when I check using SSH client I can connect to SSH server. I removed sensitive information and changed subnet, In Service.Value is only name of Zabbix agent for service 10050 where Zabbix agent is installed. SQL output:

Services.DNS Services.IP Services.Lastdown Services.Lastup Services.Port Services.Status Host.status Host.lastup Host.lastdown NetwDiskRuleName
                                        192.168.0.208   1970-01-01 01:00:00   2021-07-06 14:49:07  0 0 0  2021-07-06 14:49:07   1970-01-01 01:00:00   192.168.0.0/24   
                                        192.168.0.208   1970-01-01 01:00:00   2021-07-06 14:49:07  0 0 0  2021-07-06 14:49:07   1970-01-01 01:00:00   192.168.0.0/24   
                                        192.168.0.208   2021-07-16 15:44:19   1970-01-01 01:00:00  22 1 0  2021-07-06 14:49:07   1970-01-01 01:00:00   192.168.0.0/24   
                                        192.168.0.208   2021-07-16 15:44:19   1970-01-01 01:00:00  22 1 0  2021-07-06 14:49:07   1970-01-01 01:00:00   192.168.0.0/24   
                                        192.168.0.223   1970-01-01 01:00:00   2021-05-05 20:22:12  0 0 0  2021-03-23 16:49:19   1970-01-01 01:00:00   192.168.0.0/24   
                                        192.168.0.223   1970-01-01 01:00:00   2021-05-05 20:22:12  0 0 0  2021-03-23 16:49:19   1970-01-01 01:00:00   192.168.0.0/24   
                                        192.168.0.223   1970-01-01 01:00:00   2021-05-05 20:22:12  0 0 0  2021-03-23 16:49:19   1970-01-01 01:00:00   192.168.0.0/24   
                                        192.168.0.223   1970-01-01 01:00:00   2021-03-23 16:49:19  80 0 0  2021-03-23 16:49:19   1970-01-01 01:00:00   192.168.0.0/24   
                                        192.168.0.223   1970-01-01 01:00:00   2021-03-23 16:49:19  80 0 0  2021-03-23 16:49:19   1970-01-01 01:00:00   192.168.0.0/24   
                                        192.168.0.223   1970-01-01 01:00:00   2021-03-23 16:49:19  80 0 0  2021-03-23 16:49:19   1970-01-01 01:00:00   192.168.0.0/24   
                                        192.168.0.223   1970-01-01 01:00:00   2021-07-16 15:47:33  22 0 0  2021-03-23 16:49:19   1970-01-01 01:00:00   192.168.0.0/24   
                                        192.168.0.223   1970-01-01 01:00:00   2021-07-16 15:47:33  22 0 0  2021-03-23 16:49:19   1970-01-01 01:00:00   192.168.0.0/24   
                                        192.168.0.223   1970-01-01 01:00:00   2021-07-16 15:47:33  22 0 0  2021-03-23 16:49:19   1970-01-01 01:00:00   192.168.0.0/24   

For example I don't see unwanted network discovery actions for host 192.168.0.98 and 192.168.0.9.
BTW I'm no sure why Services.Lastup is outdated for some services, but Zabbix Discovery page shows that these services are running above 1 month (192.168.0.9)
SQL output:

Services.DNS Services.IP Services.Lastdown Services.Lastup Services.Port Services.Status Host.status Host.lastup Host.lastdown NetwDiskRuleName
   192.168.0.98    1970-01-01 01:00:00   2021-07-12 13:33:54  0 0 0  2021-07-12 17:19:37   1970-01-01 01:00:00   192.168.0.0/24   
   192.168.0.98    1970-01-01 01:00:00   2021-07-12 13:33:54  0 0 0  2021-07-12 17:19:37   1970-01-01 01:00:00   192.168.0.0/24   
   192.168.0.98    1970-01-01 01:00:00   2021-07-12 13:33:54  0 0 0  2021-07-12 17:19:37   1970-01-01 01:00:00   192.168.0.0/24   
   192.168.0.98    1970-01-01 01:00:00   2021-07-12 13:33:54  0 0 0  2021-07-12 17:19:37   1970-01-01 01:00:00   192.168.0.0/24   
   192.168.0.98    1970-01-01 01:00:00   2021-07-12 13:33:54  22 0 0  2021-07-12 17:19:37   1970-01-01 01:00:00   192.168.0.0/24   
   192.168.0.98    1970-01-01 01:00:00   2021-07-12 13:33:54  22 0 0  2021-07-12 17:19:37   1970-01-01 01:00:00   192.168.0.0/24   
   192.168.0.98    1970-01-01 01:00:00   2021-07-12 13:33:54  22 0 0  2021-07-12 17:19:37   1970-01-01 01:00:00   192.168.0.0/24   
   192.168.0.98    1970-01-01 01:00:00   2021-07-12 13:33:54  22 0 0  2021-07-12 17:19:37   1970-01-01 01:00:00   192.168.0.0/24   
   192.168.0.98    1970-01-01 01:00:00   2021-07-12 17:19:37  10050 0 0  2021-07-12 17:19:37   1970-01-01 01:00:00   192.168.0.0/24   
   192.168.0.98    1970-01-01 01:00:00   2021-07-12 17:19:37  10050 0 0  2021-07-12 17:19:37   1970-01-01 01:00:00   192.168.0.0/24   
   192.168.0.98    1970-01-01 01:00:00   2021-07-12 17:19:37  10050 0 0  2021-07-12 17:19:37   1970-01-01 01:00:00   192.168.0.0/24   
   192.168.0.98    1970-01-01 01:00:00   2021-07-12 17:19:37  10050 0 0  2021-07-12 17:19:37   1970-01-01 01:00:00   192.168.0.0/24   
   192.168.0.98    1970-01-01 01:00:00   2021-07-15 20:34:06  443 0 0  2021-07-12 17:19:37   1970-01-01 01:00:00   192.168.0.0/24   
   192.168.0.98    1970-01-01 01:00:00   2021-07-15 20:34:06  443 0 0  2021-07-12 17:19:37   1970-01-01 01:00:00   192.168.0.0/24   
   192.168.0.98    1970-01-01 01:00:00   2021-07-15 20:34:06  443 0 0  2021-07-12 17:19:37   1970-01-01 01:00:00   192.168.0.0/24   
   192.168.0.98    1970-01-01 01:00:00   2021-07-15 20:34:06  443 0 0  2021-07-12 17:19:37   1970-01-01 01:00:00   192.168.0.0/24   
                                        192.168.0.9     1970-01-01 01:00:00   2021-05-13 10:21:39  0 0 0  2021-05-21 16:36:43   1970-01-01 01:00:00   192.168.0.0/24   
                                        192.168.0.9     1970-01-01 01:00:00   2021-05-13 10:21:39  0 0 0  2021-05-21 16:36:43   1970-01-01 01:00:00   192.168.0.0/24   
                                        192.168.0.9     1970-01-01 01:00:00   2021-05-13 10:21:39  0 0 0  2021-05-21 16:36:43   1970-01-01 01:00:00   192.168.0.0/24   
                                        192.168.0.9     1970-01-01 01:00:00   2021-05-13 10:21:39  22 0 0  2021-05-21 16:36:43   1970-01-01 01:00:00   192.168.0.0/24   
                                        192.168.0.9     1970-01-01 01:00:00   2021-05-13 10:21:39  22 0 0  2021-05-21 16:36:43   1970-01-01 01:00:00   192.168.0.0/24   
                                        192.168.0.9     1970-01-01 01:00:00   2021-05-13 10:21:39  22 0 0  2021-05-21 16:36:43   1970-01-01 01:00:00   192.168.0.0/24   
                                        192.168.0.9     1970-01-01 01:00:00   2021-05-21 16:36:43  10050 0 0  2021-05-21 16:36:43   1970-01-01 01:00:00   192.168.0.0/24   
                                        192.168.0.9     1970-01-01 01:00:00   2021-05-21 16:36:43  10050 0 0  2021-05-21 16:36:43   1970-01-01 01:00:00   192.168.0.0/24   
                                        192.168.0.9     1970-01-01 01:00:00   2021-05-21 16:36:43  10050 0 0  2021-05-21 16:36:43   1970-01-01 01:00:00   192.168.0.0/24   

It would be good to have option "execute x attempts before change service status" to make network discovery less sensitive.

If you need more information please let me know

 

Comment by Hitesh Sharma [ 2021 Dec 28 ]

Hello Everyone,

Recently I have come across a problem related to discovery alert notification described below:

Zabbix discovers the host and keep on sending same discovery notification after every 10 mins and in the discovery rule I have kept update interval as 10m. As per the documentation(update interval), it says how often Zabbix will execute the rule. It seems Zabbix is executing this rule every 10 mins and sending multiple discovery notifications. Ideally, it should send only once upon discovery.

Can anyone please help here to find the issue and suggest a solution? Thanks in advance.
 
Below is the notification received and discovery action.
 
Host
ip-XX-XX-XX-XX
Event time
2021.12.28 22:22:19
Details
Discovery rule: XXXXXX
Device IP: XXXXXX
Device DNS: XXXXX
Device status: UP
Device uptime: 3d 23h 55m 31s
Device service name: SNMPv2c agent
Device service port: 161
Device service status: UP
Device service uptime: 3d 23h 55m 31s
 

ACTION1 Proxy equals Zabbix proxy
Discovery rule equals ABCD
Received value contains Linux
Discovery status equals Up
Service type equals SNMPv2 agent
Send message to users: hitesh.sharma via Slack
Add to host groups: ABCD
Link to templates: Linux SNMP
Generated at Fri May 16 10:39:10 EEST 2025 using Jira 9.12.4#9120004-sha1:625303b708afdb767e17cb2838290c41888e9ff0.