The same thing for custom discovery would also be great. Let's say I have custom discovery scripts on tomcat appservers that are hosting multiple apps. Then I could discover all of them and assign different zabbix application to each app.

It is similarly with ZBXNEXT-619

Not really Julian. That is for 1.8 which doesn't even have these discovery functionalities.

(1) [S] Database patch is available in development branch svn://svn.zabbix.com/branches/dev/ZBXNEXT-1219

asaveljevs CLOSED

(2) [S] Added batch processing support to item low level discovery.

RESOLVED in r52360

asaveljevs CLOSED

[S] Server side low level application discovery available for testing (r52435).

(3) [IS] Specification currently says that the name of application prototype should be 64 characters long, but the name of a real application has 255 character limit. Application prototype name length should probably be 255 characters, too.

wiper RESOLVED in r52484

asaveljevs CLOSED

(4) As discussed yesterday, "application_prototype_template" table should be removed and "templateid" field moved into "application_prototype" table.

wiper RESOLVED in r52484

asaveljevs CLOSED

(5) [S] It should be checked whether application prototypes are handled correctly when linking a template to a host.

wiper RESOLVED in r52594

asaveljevs Create a discovery action that adds a host and links it to a simple template with one discovery rule, one item prototype and one application prototype.

Observe that it fails with the following SQL error:

 15814:20150610:165300.123 In DBcopy_template_item_application_prototypes()
 15814:20150610:165300.123 query [txnlev:1] [select ap.application_prototypeid,i.itemid from items i_ap,item_application_prototype iap left join application_prototype ap on ap.templateid=iap.application_prototypeid left join items i_t on i_t.itemid=iap.itemid left join items i on i.templateid=i_t.itemid where i.hostid=10090 and i_ap.itemid=ap.itemid and i_ap.hostid=10090 and i_t.hostid=10088]
 15814:20150610:165300.123 query [txnlev:1] [insert into item_application_prototype (application_prototypeid,itemid) values (44,126);
]
 15814:20150610:165300.123 [Z3005] query failed: [1062] Duplicate entry '0' for key 'PRIMARY' [insert into item_application_prototype (application_prototypeid,itemid) values (44,126);

asaveljevs REOPENED

wiper RESOLVED in r54060

asaveljevs It now fails with a different error in the same scenario:

  9735:20150617:141831.544 In DBcopy_template_item_application_prototypes()
  9735:20150617:141831.544 query [txnlev:1] [select ap.application_prototypeid,i.itemid from items i_ap,item_application_prototype iap left join application_prototype ap on ap.templateid=iap.application_prototypeid left join items i_t on i_t.itemid=iap.itemid left join items i on i.templateid=i_t.itemid where i.hostid=10096 and i_ap.itemid=ap.itemid and i_ap.hostid=10096 and i_t.hostid in (10085,10086)]
zabbix_server [9735]: ERROR [file:db.c,line:2547] Something impossible has just happened.
  9715:20150617:141831.546 One child process died (PID:9735,exitcode/signal:1). Exiting ...
  9715:20150617:141831.546 zbx_on_exit() called
  9719:20150617:141831.546 Got signal [signal:15(SIGTERM),sender_pid:9715,sender_uid:1000,reason:0]. Exiting ...

asaveljevs REOPENED

wiper RESOLVED in r54096

asaveljevs CLOSED

(6) Primary key for table "application_discovery" does not match the specification. In the specification, it says "applicationid". In the implementation, it is "applicationid,application_prototypeid". The implemented solution looks better, so probably it is the specification that should be updated.

wiper You are right, it should be applicationid,application_prototypeid as implemented. Different discovery rules can discover the same application an to track it we must add separate discovery record for each discovery, resulting in non-unique applicationid values.

asaveljevs CLOSED

(7) Please review minor code review fixes in r52741.

wiper Thanks! CLOSED

Server side reviewed and database patches tested up to r52741. Will continue with server-side testing after frontend side has been developed.

(8) [S] I kept receiving error "Composite private keys are not supported in this API version.", due to table item_application_prototype having two primary keys "application_prototypeid, itemid". In order for API to work with relations, we need to change it to "UNIQUE" and add new "PRIMARY" key.

wiper RESOLVED in r52853

asaveljevs It is suggested to change the name of "itemapplicationprototypeid" field, for example, by adding "_" characters. REOPENED.

wiper RESOLVED in r53238, server side tested, frontend should be OK too, but check it please.

iivs Frontend checked. Seems it still works.
CLOSED.

asaveljevs I wonder whether ZBX_FIELDNAME_LEN in include/dbschema.h should be updated from 26 to 28, too. REOPENED.

wiper RESOLVED in r53355

asaveljevs CLOSED

(9) [S] If server fails to create an item during lld processing it might still create associated application even if the application has no links to other items.

wiper RESOLVED in r53045

asaveljevs Looks good, but please see r53180 for stylistic fixes. RESOLVED.

wiper Thanks, CLOSED

(10) Specification does not describe whether application prototypes should be listed in the "Application" column for item prototypes.

iivs Specification updated.

RESOLVED

asaveljevs CLOSED

(11) [F] In an item prototype, it is possible to select an application that was already created by a discovery rule (either the same or a different one). However, when trying to save this selection, it gives the following error message:

Application with ID "468" is not available on "Zabbix server".

asaveljevs It should be clarified in the specification whether regular items and Web scenarios can be linked to discovered applications.

iivs Discovered applications should not be in Applications list for item prototypes as well as in web scenarios. Specification updated and RESOLVED in r53595

asaveljevs CLOSED

(12) [F] Suppose our item prototype is assigned to an application prototype. In "Application prototypes" list box, that application prototype is selected. Clear that selection and attempt to update the item. It will give the following error:

Field "application_prototypes" is mandatory.

With regular applications, the same scenario works well.

iivs RESOLVED in r53615

asaveljevs CLOSED

(13) [F] Suppose our item prototype is assigned to an application prototype. Discovery has already run, so there exists a created application. Go to that item prototype, unassign it from the application prototype, and save. Note that the created application became a regular application. It should have been removed.

asaveljevs The same seems to happen if we delete the item prototype that uses the application prototype.

iivs RESOLVED in r53650

asaveljevs CLOSED

(14) [S] Specification says the following:

Two (or more) discovery rules can result in discovering the same application name. In this case one application is created, items from both discovery rules are linked to this application and one application discovery record for each discovery rule is created.

It seems that only one record is created in "application_discovery" table - only for the first discovery rule that created the application.

wiper Was broken while fixing (9). RESOLVED in r53365
This fixes bug when matching application was being discovered by another discovery rule. However if one discovery rule has multiple application prototypes generating the same application name then only one application discovery record will be created.

asaveljevs CLOSED

(15) [S] Created items obey "Keep lost resources period (in days)" setting in the discovery rule. Unfortunately, created applications do not.

wiper Applications should be removed when there are no more discovered items assigned to them. So you are saying items were removed and the discovered application without items was left untouched ?

asaveljevs I meant the other way round. Suppose we keep lost resources for 30 days. Items, that are not discovered anymore, will stay there for 30 days, even if we send empty JSON. However, discovered applications are removed immediately in this case.

wiper So.... the specs were eventually changed to allow lost application tracking similarly to other resources. The application removal code was practically rewritten. RESOLVED

asaveljevs CLOSED

(16) [S] Another, more complicated test case, in continuation of (14).

Create an item prototype with the following application prototypes:

mysql> select * from application_prototype;
+-------------------------+--------+------------+---------------------+
| application_prototypeid | itemid | templateid | name                |
+-------------------------+--------+------------+---------------------+
|                      25 |  23827 |       NULL | APP_{#A}{#B}        |
|                      26 |  23827 |       NULL | APP_{#A}{#C}        |
|                      27 |  23827 |       NULL | APP_{#B}{#C}        |
+-------------------------+--------+------------+---------------------+

Send the two following JSONs:

$ zabbix_sender -z 192.168.1.2 -s macro -k trapper.discovery -o '{"data":[{"{#VALUE}":"value","{#A}":"1","{#B}":"2","{#C}":"3"}]}'
$ zabbix_sender -z 192.168.1.2 -s macro -k trapper.discovery -o '{"data":[{"{#VALUE}":"value","{#A}":"2","{#B}":"3","{#C}":"1"}]}'

Observe the following in the database tables:

mysql> select d.applicationid, a.name, d.application_prototypeid, p.name from application_discovery d, applications a, application_prototype p where d.applicationid=a.applicationid and d.application_prototypeid=p.application_prototypeid;                    
+---------------+--------+-------------------------+--------------+
| applicationid | name   | application_prototypeid | name         |
+---------------+--------+-------------------------+--------------+
|           506 | APP_21 |                      26 | APP_{#A}{#C} |
|           505 | APP_23 |                      27 | APP_{#B}{#C} |
|           507 | APP_31 |                      27 | APP_{#B}{#C} |
+---------------+--------+-------------------------+--------------+
3 rows in set (0.00 sec)

It claims that APP_23 was created from APP_{#B}{#C}, but this is false.

wiper It's more like continuation of (15). APP_23 was created by APP_{#B}{#C} during the first discovery. Because the 'empty' applications are removed only after discovery has been processed, the prototype APP_{#A}{#B} reused existing APP_23 instead of creating new one.

asaveljevs Yes, I understand the process, but won't this database inconsistency cause us problems down the road? As it is, the database structure is different, depending on the history of the application.

wiper RESOLVED

asaveljevs The database data remained the same in the described test case (APP_23 being discovered from APP_{#B}{#C}), but the application is now marked as non-discovered and also marked to be deleted at a certain time. This seems to be expected. CLOSED.

(17) [F] We dropped requirement that discovered applications without discovered items must be removed. The server was updated and if the frontend handled discovered application removal somewhere - it should be updated too.

iivs RESOLVED in r53650

asaveljevs CLOSED

(18) [F] For lost applications frontend should display '!' icon with apropriate tooltip (like it's done with other resources).

iivs The new design was already merged, but it seems like the icon display is not fully funcional yet. However, the code is added and if you hover a mouse on an application that is no longer discovered, you can see the tooltip.

RESOLVED in r53673

wiper If two discovery rules have discovered one application and then one of discovery rules stops discovering it, the application should not be marked as 'lost'.
REOPENED

iivs RESOLVED in r53685

asaveljevs CLOSED

(19) updated to trunk and resolved conflicts, please review r53556

wiper server side looks ok

CLOSED

(20) [S] In order to get data via API from application_discovery table, I need a unique key, but I get the same error as before Composite private keys are not supported in this API version. due to table having two keys.

wiper Database patch and server side code updated in r53684
RESOLVED

asaveljevs CLOSED

(21) Added translation strings:

• New application prototype
• Application prototypes
• Cannot add a discovered application "%1$s" to a web scenario.
• The application is not discovered anymore and will be deleted the next time discovery rule is processed.
• The application is not discovered anymore and will be deleted in %1$s (on %2$s at %3$s).
• Cannot set "templateid" field for application prototype in item prototype "%1$s".
• Duplicate "name" value "%1$s" for application prototype in item prototype "%2$s".
• Missing "name" field for application prototype in item prototype "%1$s".
• Cannot update discovered application "%1$s".
• Cannot delete discovered application "%1$s".
• Empty application prototype name in item prototype "%1$s".

oleg.egorov CLOSED

<richlv> r54623 also added :

• msgid "Cannot set \"%1$s\" for application \"%2$s\"."
• msgid "Cannot update \"%1$s\" for application \"%2$s\"."

and removed :

• msgid "Cannot set \"templateid\" for application \"%1$s\"."
• msgid "Cannot update \"templateid\" for application \"%1$s\"."

let's verify whether that's intentional and whether the added strings are ok

oleg.egorov Approved removed strings

Added strings:

• Cannot update "%1$s" for application "%2$s".
• Cannot set "%1$s" for application "%2$s".

CLOSED

(22) [S] Purely stylistically, the columns in schema.tmpl for new tables do not have the same alignment as other tables. This is probably due to the fact that their field names are very long, but see other tables (e.g., "items" table) for how we deal with overly long field names.

wiper RESOLVED in r54061

asaveljevs CLOSED

(23) [S] There seems to be a mistake in DBpatch_2050017(). The following specifies "templateid" field as ZBX_NOTNULL, whereas NULL's are actually allowed in the schema:

const ZBX_FIELD	field = {"templateid", NULL, "application_prototype", "application_prototypeid",
		0, ZBX_TYPE_ID, ZBX_NOTNULL, ZBX_FK_CASCADE_DELETE};

wiper RESOLVED in 54057

asaveljevs CLOSED

(24) [S] Please review fixes in r54006. The changes are mostly stylistic, except one place where a linear search is replaced with binary search.

wiper Thanks! I also added additional DBexecute_overflowed_sql() calls during lost application removal.
RESOLVED in r54058

asaveljevs CLOSED

(25) [F] Full clone operation does not seem to clone application prototypes.

iivs RESOLVED in r54050

asaveljevs CLOSED

(26) [S] Create an item prototype with two application prototypes:

Application_{#A}{#B}
Application_{#B}{#A}

After sending in {#A}=A and {#B}=B, two applications are correctly created: Application_AB and Application_BA.

Now, send {#A}=B and {#B}=A. Observe that applications have been marked as non-discovered and the discovered item is not linked to any application.

asaveljevs "Info" column in frontend contains the following errors:

Cannot create application: application with the same name "Application_BA" already exists.
Cannot create application: application with the same name "Application_AB" already exists.

asaveljevs After discussion, this seems to be expected. Items work the same way. WON'T FIX.

(27) [F] As a continuation to (25), if we full clone a host with discovered applications, these applications become regular applications on the cloned host.

iivs RESOLVED in r54049

asaveljevs CLOSED

(28) [S] Create a discovery rule with "Keep lost resources" of 0. Make it include one item prototype:

echo[{#VALUE}]

Create two application prototypes for this item prototype:

Application_{#A}{#B}
Application_{#B}{#A}

Send the following JSONs:

$ zabbix_sender -z 127.0.0.1 -s 'Zabbix server' -k 'trapper.discovery' -o '{"data":[]}'
$ zabbix_sender -z 127.0.0.1 -s 'Zabbix server' -k 'trapper.discovery' -o '{"data":[{"{#VALUE}":"value1","{#A}":"Z","{#B}":"Z"},{"{#VALUE}":"value2","{#A}":"Z","{#B}":"Z"},{"{#VALUE}":"value3","{#A}":"Z","{#B}":"Z"}]}'
$ zabbix_sender -z 127.0.0.1 -s 'Zabbix server' -k 'trapper.discovery' -o '{"data":[{"{#VALUE}":"value1","{#A}":"a1","{#B}":"b1"},{"{#VALUE}":"value2","{#A}":"a2","{#B}":"b2"},{"{#VALUE}":"value3","{#A}":"a3","{#B}":"b3"}]}'

Observe the following in zabbix_server.log:

zabbix_server [15174]: ERROR [file:lld_item.c,line:1672] Something impossible has just happened.
zabbix_server [15174]: ERROR [file:lld_item.c,line:1672] Something impossible has just happened.
zabbix_server [15174]: ERROR [file:lld_item.c,line:1672] Something impossible has just happened.

wiper RESOLVED in r54072

asaveljevs CLOSED

(29) The new LLD code uses hardcoded zbx_hash_modfnv() in data structure helper functions. It might be useful to introduce a new define, say, ZBX_DEFAULT_HASH_ALGO, in addition to the already existing ZBX_DEFAULT_STRING_HASH_ALGO and ZBX_DEFAULT_UINT64_HASH_ALGO.

wiper RESOLVED in r54073-r54076

asaveljevs CLOSED

(30) [S] Suppose we have a template that has both normal applications and applications prototypes. That template is linked to a host, on which discovery has already run. Therefore, the host has normal applications and discovered applications. Create an action that unlinks the template from the host. When this action is run, observe that normal applications remained on the host and discovered applications turned into normal applications.

Clearly, discovered applications should have been removed. As for normal applications, the expected behavior seems to be that they remain on the host only if they still have some items linked to them (but this should be clarified).

asaveljevs Frontend removes both normal and discovered applications when the template is unlinked and cleared.

wiper RESOLVED in r54117

asaveljevs The function comment for DBdelete_template_applications() says the following:

 *           This function does not remove applications discovered by         *
 *           application prototypes.                                          *
 *           Use DBdelete_template_discovered_applications() for that.        *

It does not seem to be valid anymore.

asaveljevs It is also proposed to rename "druleids" variable, because the first impression is that it refers to "drules" table "druleid" field, rather than item IDs.

asaveljevs Please also see r54292 for a bit of style.

asaveljevs Applications seem to be removed well when they have no items attached to them. However, in case an application was attached to an item from the same template, the item is removed, but the application is not. REOPENED.

wiper RESOLVED in r54552. Also renamed druleids to lld_ruleids. And after the changes the comment was correct.

asaveljevs Regarding DBdelete_template_discovered_applications():

• with all due respect to Eugene, he does not seem to be the author of this brand new function;
• we do zbx_vector_uint64_sort() on lld_ruleids(), but it is already sorted after DBselect_uint64();
• we do zbx_vector_uint64_uniq() on lld_ruleids(), but it is doubtful that this array can have duplicates.

REOPENED

wiper RESOLVED in r54595

asaveljevs CLOSED

(31) [F] Create two discovery rules (one with keep lost resources period of 30 and another of 3000 days) and let them discover the same application. Now, let them discover nothing.

For items, the message in the "Info" column will correctly be "The item is not discovered anymore and will be deleted in 29d 23h 52m (on 2015-07-18 at 11:50)." and "The item is not discovered anymore and will be deleted in 8y 2m 19d (on 2023-09-04 at 11:52)."

However, for applications the message will be "The application is not discovered anymore and will be deleted the next time discovery rule is processed.", even though "ts_delete" in the database is correct.

With one discovery rule it seems to work OK.

iivs RESOLVED in r54260

asaveljevs CLOSED

(32) [F] Applications that are created from templates should not be removed when the templates are unlinked if there are items assigned to it.

iivs As we discussed, the link between item and discovered application checking is no longer required. I removed automatic deletion of discovered applications when deleting discovered items. Now discovered applications will be deleted upon deleting an application prototype, when ever it occurs. So the only check that is required, is to see if discovered application is not discovered by other rules. If not, we can delete it.

RESOLVED in r54266

asaveljevs The problem does not seem to be fixed. Create a template with just one application. Link it to a host with one item. Link that item to that application. Unlink and clear that template. Observe that the application was also removed. REOPENED.

iivs As we discussed, this was actually and old problem.
RESOLVED in r54308

asaveljevs Now, all template applications remain on the host if at least one of them still has a linked item. This is due to the fact that "not exists (select" lists all application IDs, rather than a particular one. (Also note that there is an extra space on line 600.) REOPENED

iivs RESOLVED in r54330, r54331

asaveljevs Looks good. CLOSED

Is there any way I can be removed as reporter for this feature?

I haven't used zabbix for two years and I'm not planning to use it anymore and I'm getting several e-mails day about this.

attilla, in the upper right corner, "Watchers" - are you watching the issue ? if yes, remove that and maybe it helps.

A little more patience, Attilla. You have endured most of it - development is mostly done, only code review for the frontend is left.

(33) [F] Case sensitive issue.

See ZBXNEXT-1219-1.png

Quotes issue:
Try to create application prototype with name 'test'
They will be added, but no selected. Simple application works fine with quotes.

Empty application prototype:
Via API possible create empty application prototype

iivs RESOLVED in r54483

oleg.egorov CLOSED

(34) [F] Please review trivial code improvements in r54488

iivs Thanks!
CLOSED.

(35) [F] As discussed, please fix 0 and 00 issue (if exist application prototype with name "0", impossible create "00") using zbx_objectValues

iivs RESOLVED in r54549

oleg.egorov CLOSED

Frontend and server sides successfully tested

Released in pre-2.5.0 r54623

(37) [S] Crash in DBcopy_template_application_prototypes() when server is linking to template containing discovery rules without application prototypes.

wiper RESOLVED in r54686

asaveljevs CLOSED

(38) [S] During item-application link validation server logs THIS_SHOULD_NOT_HAPPEN if it can't find the application id in applications vector. However theoretically it can happen. Applications vector contains only discovered applications while the link might be to normal application. Consider this scenario:

1. have item prototype linked to normal application
2. discover item(s) based on this prototype
3. unlink item prototype from application
4. repeat (2)

wiper RESOLVED in r54671

asaveljevs CLOSED

(39) [S] The item prototype -> discovered item mapping built during item lld processing and used by trigger lld processing was broken. Because of that trigger prototypes were not discovered.

wiper RESOLVED in r54671

asaveljevs CLOSED

(40) [S] The item prototype -> discovered item mapping was populated from items index, which contained only already discovered items. Therefore triggers were discovered only on the second discovery attempt.

wiper RESOLVED in r54671

asaveljevs CLOSED

subissues 39 & 40 might fix ZBX-9760

wiper Most likely. Only one trigger being discovered sounds awfully like (39).

wiper Although it's strange how people can get discovery working because of (37).

(41) [S] Moved item_application_prototype allocation to be done after 'should not happen' check.
RESOLVED in r54996

asaveljevs CLOSED

(42) Suppose we have graph prototypes and send the following JSON:

$ zabbix_sender -z 127.0.0.1 -s 'ZBXNEXT-1219' -k 'trapper.discovery' -o '{"data":[{"{#VALUE}":"value1","{#A}":"a","{#B}":"b"},{"{#VALUE}":"value2","{#A}":"a","{#B}":"b"},{"{#VALUE}":"value3","{#A}":"a","{#B}":"b"}]}' -vv

The following errors appear in server log file:

 27143:20150817:140637.474 [Z3005] query failed: [1452] Cannot add or update a child row: a foreign key constraint fails (`issue_zbxnext_1219`.`functions`, CONSTRAINT `c_functions_1` FOREIGN KEY (`itemid`) REFERENCES `items` (`itemid`) ON DELETE CASCADE) [insert into functions (functionid,itemid,triggerid,function,parameter) values (13219,23701,13593,'last',''),(13220,23703,13593,'last',''),(13221,23702,13593,'last',''),(13222,null,13594,'last',''),(13223,null,13594,'last',''),(13224,null,13594,'last',''),(13225,null,13595,'last',''),(13226,null,13595,'last',''),(13227,null,13595,'last',''),(13228,23702,13596,'last',''),(13229,null,13597,'last',''),(13230,null,13598,'last',''),(13231,23703,13599,'last',''),(13232,null,13600,'last',''),(13233,null,13601,'last',''),(13234,23701,13602,'last',''),(13235,null,13603,'last',''),(13236,null,13604,'last','');
]
 27143:20150817:140637.478 [Z3005] query failed: [1048] Column 'itemid' cannot be null [insert into graphs_items (gitemid,graphid,itemid,drawtype,sortorder,color,yaxisside,calc_fnc,type) values (1850,574,null,0,0,'00C800',0,2,0);
]
 27143:20150817:140637.481 [Z3005] query failed: [1048] Column 'itemid' cannot be null [insert into graphs_items (gitemid,graphid,itemid,drawtype,sortorder,color,yaxisside,calc_fnc,type) values (1850,574,null,0,0,'00C800',0,2,0);
]
 27143:20150817:140637.484 [Z3005] query failed: [1048] Column 'itemid' cannot be null [insert into graphs_items (gitemid,graphid,itemid,drawtype,sortorder,color,yaxisside,calc_fnc,type) values (1850,574,null,0,0,'C80000',0,2,0);
]

Because XML export does not work in current trunk, it is not possible to attach the host's XML. Please ask me for a database dump instead.

wiper RESOLVED in 55001

asaveljevs CLOSED

(43) [S] Duplicated item key validation is broken.

If we have already discovered item in the database and on next discovery rule the same key is discovered by another lld row then the duplicated key is not detected and server will try to create another item with the same key.

wiper RESOLVED in r55002

asaveljevs CLOSED

Released in:

• pre-2.5.0 r55035

(43) Updated general documentation:

• https://www.zabbix.com/documentation/3.0/manual/introduction/whatsnew300#linking_to_applications_based_on_discovery_values
• https://www.zabbix.com/documentation/3.0/manual/discovery/low_level_discovery#discovery_of_file_systems (see 'item prototype')

Please review. RESOLVED.

wiper Added application discovery notes page to explain some application discovery nuances. Please review.

martins-v Thanks, looking good. I changed the structure slightly so that we could add notes related to other LLD topics under the same subsection.

sasha Thanks! CLOSED

(44) broken XML export

Moved to ZBX-9807

CLOSED