[ZBXNEXT-581] Ability to filter by multiple LLD macros Created: 2010 Dec 12  Updated: 2017 Jan 26  Due: 2014 Apr 17  Resolved: 2014 Feb 13

Status: Closed
Project: ZABBIX FEATURE REQUESTS
Component/s: Frontend (F), Installation (I), Server (S)
Affects Version/s: None
Fix Version/s: 2.3.0

Type: New Feature Request Priority: Major
Reporter: Oleksii Zagorskyi Assignee: Pavels Jelisejevs (Inactive)
Resolution: Fixed Votes: 65
Labels: flexibility, lld
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified
Environment:

latest trunk v1.9.x
some experience with LLD feature usage.


Issue Links:
Duplicate
is duplicated by ZBXNEXT-1573 exclude item by LLD Closed
is duplicated by ZBXNEXT-1449 Low Level Discovery Multiple Filters Closed
is duplicated by ZBXNEXT-1008 more advanced filtering for lld Closed
is duplicated by ZBX-7353 no filtering in lld rules gets export... Open
is duplicated by ZBX-5676 vfs.fs.discovery should not discover ... Closed

 Description   

Requested:
1. Ability to get more than one

{# SNMPVALUE} in one rule LLD for SNMP (note this has been split out to ZBXNEXT-1554)
2. Multiple macros filtering (useful not only for SNMP).

In detail:
I mean the ability to get for example IF-MIB::ifDescr and IF-MIB::ifAlias etc in one discovery rule and then use {# SNMPINDEX} macro and two or more values of {# SNMPVALUE}

macros in name of items, graphs, triggers prototypes for this rule.
I'm sure that we can find a lot of real examples, where it could be helpful to get more than one value to build nice names for the objects mentioned or for example for complex trigger expressions.

In addition, it would be very useful to be able to do filtering on several macros simultaneously, including

{# SNMPINDEX} and multiple {#SNMPVALUE} instead of filtering one macro {# SNMPINDEX}

or

{# SNMPVALUE}

.

I'm sure that the LLD engine should be flexible enough, because it provides no possibility of next manual editing the objects that have been created automatically.
I would not want that at the end the LLD would be less flexible and convenient than usual templates usage, but without these futures seems it's not possible.

spec placeholder at https://www.zabbix.org/wiki/Docs/specs/ZBXNEXT-581



 Comments   
Comment by fmrapid [ 2010 Dec 13 ]

All data enrichment that is possible using automated methods such LLD, scripted API need to store this item or host related data somewhere. Using the item description and host description can only be used to store a limited number of data.

In the current scheme of things, this requests is useful for getting more accurate and updatable information in the description field.

In the longer term, the item and host tables need to be extended(or linked to other tables) for storing additional fields that can then be used throughout Zabbix. See ZBXNEXT-577. These can be predefined fields and user defined custom fields.

This feature requests gets my vote, but the dev team needs to look at the big picture in how powerful LLD and API can be.

fmrapid

Comment by Oleksii Zagorskyi [ 2012 Sep 04 ]

Some interesting approach described here (in Russian) http://www.zabbix.com/forum/showthread.php?t=28870

Comment by Oleksii Zagorskyi [ 2013 Sep 08 ]

I've improved my issue description a bit, hopefully now it will look closer to English than before
This is a "preparation" to some ... promises :>
https://twitter.com/avladishev/status/376257646401056768

Comment by jabalv [ 2013 Nov 18 ]

I think would be good implement this feature together with ZBXNEXT-1606. Those features together will help make JAVA application monitoring more advanced.
Filtering by multiple macros is useful, but if there would be possibility run multiple discovery rules with same key on host and each time filter by multiple, different macros then for example from jmx.discovery key we can get out everything.

Comment by Alexander Vladishev [ 2013 Dec 14 ]

(1) compilation warning:

dbupgrade.c: In function ‘DBcheck_version’:
dbupgrade.c:702:3: warning: ‘required’ may be used uninitialized in this function [-Wuninitialized]

wiper RESOLVED in r40994

sasha CLOSED

Comment by Alexander Vladishev [ 2013 Dec 14 ]

(2) "total" variable is initialized in DBcheck_version()

wiper RESOLVED in r40994

sasha CLOSED

Comment by Alexander Vladishev [ 2013 Dec 14 ]

(3) Please review my changes in r40995.

wiper reviewed and CLOSED

Comment by Andris Zeila [ 2013 Dec 14 ]

The generic database upgrade changes are available in pre-2.3.0 r40997

Comment by Andris Zeila [ 2013 Dec 14 ]

ZBXNEXT-581 specific database patch available in development branch svn://svn.zabbix.com/branches/dev/ZBXNEXT-581

Comment by Alexander Vladishev [ 2013 Dec 16 ]

(4) errors occurred when upgrading a database:

 24739:20131216:100958.161 [Z3005] query failed: [0] PGRES_FATAL_ERROR:ERROR:  insert or update on table "item_condition" violates foreign key constraint "c_item_condition_2"
DETAIL:  Key (templateid)=(22444) is not present in table "item_condition".
 [insert into item_condition (item_conditionid,itemid,templateid,macro,value) values (23278,23278,22444,'{#IFNAME}','@Network interfaces for discovery')]

"templateid" field should be removed from the new DB patches. Specification is already updated.

wiper RESOLVED in r41031

sasha CLOSED

Comment by Alexander Vladishev [ 2013 Dec 16 ]

(5) Only discovery rules should be processed by DBpatch_2030006(). Please add " and flags=ZBX_FLAG_DISCOVERY_RULE" into the SQL condition.

wiper RESOLVED in r41032

sasha CLOSED

Comment by Alexander Vladishev [ 2013 Dec 16 ]

Database scheme changes was successfully tested!

Please review my fix in r41041:41042.

Comment by Andris Zeila [ 2013 Dec 18 ]

[S] Server side ready for testing in r41095

Comment by Alexander Vladishev [ 2014 Jan 06 ]

(6) please review my changes in r41292.

wiper thanks, reviewed and CLOSED

Comment by Alexander Vladishev [ 2014 Jan 06 ]

(7) unused macro ZBX_OPCODE_CLASS_OP_UNARY

wiper RESOLVED in r41296

sasha CLOSED

Comment by Alexander Vladishev [ 2014 Jan 06 ]

(8) translate_expression() strange code

if (NULL == *out)
{
        size = 256;
        *out = zbx_malloc(NULL, size);
}
  • "size" can be uninitialized if "*out" is not null

wiper RESOLVED in r41298

sasha REOPENED

Will be better to use zbx_strncpy_alloc() and zbx_chrcpy_alloc() in this function.

In this case "size" should be renamed into "out_alloc". And it should be the pointer.

wiper RESOLVED in r41318

sasha CLOSED

Comment by Alexander Vladishev [ 2014 Jan 06 ]

(9) translate_expression():4436,4534

  • should be used zbx_realloc() instead of realloc()

wiper RESOLVED in 41297

sasha CLOSED

Comment by Alexander Vladishev [ 2014 Jan 06 ]

(10) translate_expression() too many error messages. Тhere could be one message on all cases.

For example:
'invalid expression: error at position %d'

wiper RESOLVED in r41318

sasha CLOSED

Comment by Alexander Vladishev [ 2014 Jan 06 ]

(11) "start - expression" should be converted into (int)

zabbix_log(LOG_LEVEL_DEBUG, "expression '%s' error:"
                " unmatched right parentheses ')' at %d (%s)",
                expression, start - expression, start);

sasha CLOSED

Comment by Alexander Vladishev [ 2014 Jan 06 ]

(12) translate_expression(): value_len always equal op_len and can be removed from code

wiper RESOLVED in r41305

sasha CLOSED

Comment by Alexander Vladishev [ 2014 Jan 06 ]

(13) translate_expression(): what is WDN?

wiper debug output leftovers, RESOLVED in r41304

sasha CLOSED

Comment by Alexander Vladishev [ 2014 Jan 06 ]

(14) translate_expression(): is not working properly without delimiters

For example, expression (

{19}OR{22}OR{19}

OR

{19})AND({19}

OR

{22}) was converted into ({19}OR{22}

OR

{19}OR{19}

)&(

{19}

OR

{22}

)

wiper RESLOVED in r41304

sasha CLOSED

Comment by Alexander Vladishev [ 2014 Jan 06 ]

(15) expression_get_token() '_' - in our expressions such character shouldn't be

wiper resolved in r41302

sasha CLOSED

Comment by Alexander Vladishev [ 2014 Jan 07 ]

(16) operators can be in lower case only

wiper RESOLVED in r41318

sasha REOPENED expression_get_token() isalpha(*pend) should be replaced by "*pend >= 'a' && *pend <= 'z'"

wiper RESOLVED in r41330

sasha CLOSED

Comment by Alexander Vladishev [ 2014 Jan 07 ]

(17) Please review my changes in r41322

wiper thanks, CLOSED

Comment by Pavels Jelisejevs (Inactive) [ 2014 Jan 07 ]

(18) I've changed the default value of items.formula to "" and added two DB patches. Please review r41328.

wiper Successfully tested, CLOSED.

Comment by Pavels Jelisejevs (Inactive) [ 2014 Jan 13 ]

Frontend and API ready for testing.

RESOLVED.

Comment by Alexander Vladishev [ 2014 Jan 18 ]

(19) [F] cannot open host_discovery.php page

Apache error.log:

[Sat Jan 18 06:36:12 2014] [error] [client 127.0.0.1] PHP Fatal error:  Cannot create references to elements of a temporary array expression in frontends/php/host_discovery.php on line 195

... and according to our coding guidelines the "Superglobal variables should never be modified during script execution."

jelisejev The error does not appear on my PHP version, but you're right about modifying request. I've rewritten that part of the code in r41675, so the error should be gone as well.

sasha CLOSED

Comment by Alexander Vladishev [ 2014 Jan 18 ]

(20) [AF] debug output are present when create a new discovery rule.

array(33) { ["itemid"]=> NULL ["interfaceid"]=> string(1) "1" ["name"]=> string(27) "Network interface discovery" ["description"]=> string(0) "" ["key_"]=> string(16) "net.if.discovery" ["hostid"]=> string(5) "10105" ["delay"]=> string(2) "30" ["status"]=> string(1) "0" ["type"]=> string(1) "0" ["snmp_community"]=> NULL ["snmp_oid"]=> NULL ["trapper_hosts"]=> NULL ["port"]=> NULL ["snmpv3_contextname"]=> NULL ["snmpv3_securityname"]=> NULL ["snmpv3_securitylevel"]=> NULL ["snmpv3_authprotocol"]=> string(1) "0" ["snmpv3_authpassphrase"]=> NULL ["snmpv3_privprotocol"]=> string(1) "0" ["snmpv3_privpassphrase"]=> NULL ["delay_flex"]=> string(0) "" ["authtype"]=> NULL ["username"]=> NULL ["password"]=> NULL ["publickey"]=> NULL ["privatekey"]=> NULL ["params"]=> NULL ["ipmi_sensor"]=> NULL ["lifetime"]=> string(2) "30" ["filter"]=> array(2) { ["evaltype"]=> string(1) "0" ["conditions"]=> array(0) { } } ["flags"]=> int(1) ["value_type"]=> int(4) ["data_type"]=> int(0) }

jelisejev RESOLVED in r41670.

sasha CLOSED

Comment by Alexander Vladishev [ 2014 Jan 18 ]

The server side and the upgrade patch has successfully tested!

[IS] Please review small changes in r41637.

Comment by Pavels Jelisejevs (Inactive) [ 2014 Jan 20 ]

(21) I've removed some unused code in r41674.

sasha CLOSED

Comment by Ivo Kurzemnieks [ 2014 Jan 23 ]

(22) Update specification https://www.zabbix.org/wiki/Docs/specs/ZBXNEXT-581

  • ACTION_EVAL_TYPE... has changed to _CONDITION_EVAL_TYPE_...;
  • 'Conditions' tab has changed to 'Filters' tab;
  • table column items.formula default value ...|'1' -> ...|'empty'

jelisejev RESOLVED.

iivs CLOSED.

Comment by Ivo Kurzemnieks [ 2014 Jan 23 ]

(23) [AF] Please, update development branch from latest trunk to eliminate old API_OUTPUT_REFER in CDiscoveryRule.php: 82 and copyright notice years (2000-2011, 2000-2012 and 2001-2013) in many files. I've correct for newly added files. See r41857

jelisejev RESOLVED in r42377-42384.

iivs CLOSED.

Comment by Ivo Kurzemnieks [ 2014 Jan 23 ]

(24) [F] Problem in edit discovery form.
1. Adding new filter labels, they sort like so: A - Z, then AA - AZ, then BA - BZ, etc.
After incorrect save, letters sort like so: A, AA - AZ, B, BA - BZ, etc .

2. Deleting letters and adding new, newer letters don't continue from previous letter. For example there are labels A, B, C and D. I delete D, and add new letter. New letter is E and not D.

3. Similar situation when I have labels A, B, C and D and I delete A and B. Then save form, get error about something incorrect and then I add new labels. I will get labels C, D, C and D. Then if try to save form again, letters that repeat will be removed.

jelisejev 1. and 3. RESOLVED in r42459. 2. is not a bug, it's suppose to be that way. For instance, if a user types and a custom formula "A and B and C", removes C and adds another condition - we can't shouldn't just replace C with the new one, since they may be different.

iivs CLOSED.

Comment by Ivo Kurzemnieks [ 2014 Jan 23 ]

(25) [F] Another problem in edit discovery form.
IE8 and IE9 placeholders are not displayed on page load. Placeholders appear only after pressing "Add".

jelisejev RESOLVED in r42463.

iivs CLOSED.

Comment by Ivo Kurzemnieks [ 2014 Jan 24 ]

(26) [AF] Coding style and other small changes:

  • host_discovery.php: 99, 283 lines overflow
  • host_discovery.php: 231 change get_request to getRequest
  • host_discovery.php: 241 brackets before zbx_empty are not required. Change
    !(zbx_empty($condition['macro']))

    to

    !zbx_empty($condition['macro'])
  • common.macros.php: 51 line overflow
  • CDiscoveryRule.php: 1264-1265 operator || should be at beginning
  • forms.inc.php: 1126 change !isset($_REQUEST['form_refresh']) to !hasRequest('form_refresh'). And since 'form_refresh' repeats twice, probably that piece can be simplified to:
    if (!hasRequest('form_refresh') && ($item || $data['limited'])) { ... }
  • configuration.action.edit.php: 954-956 translations are uppercase. Please unify so they are the same as in lines 110-112
  • CDiscoveryRule.php: 841, 848 extra comma for last array element
  • CItemPrototype.php: 351-353 two if conditions can be united under one
  • main.js: 1251, 1260 change == operator to === and != to !== when checking typeof
  • configuration.host.discovery.edit.js.php: 29 add space before { bracket
  • configuration.host.discovery.edit.js.php: 39 would be good to add ; at the end

jelisejev RESOLVED in r42470. But I couldn't find this place in code: "CItemPrototype.php: 351-353 two if conditions can be united under one".

iivs CLOSED.

Comment by Ivo Kurzemnieks [ 2014 Jan 24 ]

(27) [A] CDiscoveryRule.php

  • line 1438 has request for 'formulaid', but table 'item_condition' does not have such column. And since we request all table columns, we can use API_OUTPUT_EXTEND there.
  • line 1448 has same if condition as line 1436
    if ($formulaRequested || $evalFormulaRequested || $conditionsRequested) 

    This could be unified under one if condition.

  • line 1461 has variable $rule['evaltype']. Assuming it was meant to be $filter['evaltype']
  • line 264 removes fields 'formula' and 'evaltype' if not requested in output, but in line 268 it will remove those fields either way. So code
    $result = $this->unsetExtraFields($result, array('hostid', 'formula', 'evaltype'), $options['output']);

    can be changed to:

    $result = $this->unsetExtraFields($result, array('hostid'), $options['output']);

jelisejev RESOLVED in r42476.

iivs CLOSED.

Comment by Ivo Kurzemnieks [ 2014 Jan 24 ]

(28) [F] Create a templated discovery rule. In edit form choose 'clone'. Notice that read only fields no longer change to editable.

jelisejev RESOLVED in r42500.

iivs CLOSED.

Comment by Ivo Kurzemnieks [ 2014 Jan 28 ]

(29) [A] In discoveryrule create and update methods "evaltype" and "formula" can be passed directly with incorrect values if "filter" is not set. However, if filter is set, values will be overridden. If filter is not set, params "evaltype" and "formula" must be unset.

jelisejev RESOLVED in r42502.

iivs CLOSED.

Comment by Ivo Kurzemnieks [ 2014 Jan 30 ]

(30) [AF] Create a discovery rule in frontend and add a filter macro there. Save it and open it for editing. Now let's say you forgot you opened it in browser and you open API and execute the following code. For example:

{
    "name": "test",
    "itemid": "23761",
    "filter": {
        "evaltype": "0",
        "conditions": []
    }
}

This should delete the filter. It's a succeeds. Now try to save again from previously opened tab. It says it succeeds, but we now have undefined indexes there. Because we no longer had those old condition ID's.

Undefined offset: 91 [host_discovery.php:300 → CAPIObject->update() → CAPIObject->__call() → czbxrpc::call() → czbxrpc::callAPI() → call_user_func() → CDiscoveryRule->update() → CDiscoveryRule->updateReal() → DB::replace() in C:\Development\ZBXNEXT-581-test\frontends\php\include\classes\db\DB.php:642]
Argument 2 passed to DB::recordModified() must be of the type array, null given, called in C:\Development\ZBXNEXT-581-test\frontends\php\include\classes\db\DB.php on line 642 and defined [host_discovery.php:300 → CAPIObject->update() → CAPIObject->__call() → czbxrpc::call() → czbxrpc::callAPI() → call_user_func() → CDiscoveryRule->update() → CDiscoveryRule->updateReal() → DB::replace() → DB::recordModified() in C:\Development\ZBXNEXT-581-test\frontends\php\include\classes\db\DB.php:689]
Invalid argument supplied for foreach() [host_discovery.php:300 → CAPIObject->update() → CAPIObject->__call() → czbxrpc::call() → czbxrpc::callAPI() → call_user_func() → CDiscoveryRule->update() → CDiscoveryRule->updateReal() → DB::replace() → DB::recordModified() in C:\Development\ZBXNEXT-581-test\frontends\php\include\classes\db\DB.php:690]

The filter is empty, although we had value there when we opened the form for editing. I think the way it should work it should save it as new, with new condition IDs.

jelisejev RESOLVED in r42620.

iivs Very nice solution.
CLOSED.

Comment by Eduards Samersovs (Inactive) [ 2014 Jan 31 ]

(31)

  • main.js:1193 Please call jQuery extend with first {} parameter $.extend({}, defaults, options), also "var" is not require but good to add ";" to the end of definition.
  • also be good to give a better name to look like $('.my-table').rowDynamic({})

iivs I assume it's main.js:1303

jelisejev RESOLVED in r42621.

iivs CLOSED.

Comment by Pavels Jelisejevs (Inactive) [ 2014 Feb 13 ]

(32) Macro validation works incorrectly: it accepts "{#TEST2}{#TEST2}" as a valid macro.

jelisejev RESOLVED in r42624.

iivs CLOSED.

Comment by Ivo Kurzemnieks [ 2014 Feb 17 ]

(33) See minor changes I've made in r42739

jelisejev Thanks, CLOSED.

Comment by Ivo Kurzemnieks [ 2014 Feb 17 ]

TESTED.

Please close (33) before merging.

Comment by Pavels Jelisejevs (Inactive) [ 2014 Feb 18 ]

Available in 2.3.0 r42770.

Comment by Pavels Jelisejevs (Inactive) [ 2014 Feb 18 ]

(34) API docs need to be updated.

jelisejev Updated:

iivs CLOSED.

Comment by Pavels Jelisejevs (Inactive) [ 2014 Feb 18 ]

(35) Documentation needs to be updated.

martins-v Updated:

RESOLVED.

jelisejev We should probably add more information about custom expressions in the discovery rule section. Or, may be, link to the description in the action documentation.

martins-v More details added, as well as a link to action condition section.

RESOLVED.

jelisejev Great, thanks! CLOSED.

Comment by richlv [ 2014 Feb 27 ]

(37) (related to (35), too)

regarding the calculation type dropdown :

two of these definitely have to be changed
(and i'll note that "And / or" does not look that great to me =) )

jelisejev Updated the spec, RESOLVED in r43088.

iivs CLOSED.

Comment by Pavels Jelisejevs (Inactive) [ 2014 Feb 28 ]

(38) Another small issue we've missed: action operation condition expression should be displayed without extra brackets.

jelisejev RESOLVED in r43090. I've changed the action operation condition formula to be displayed the same was as action condition formula.

iivs CLOSED.

Comment by Pavels Jelisejevs (Inactive) [ 2014 Feb 28 ]

RESOLVED in svn://svn.zabbix.com/branches/dev/ZBXNEXT-581.

Comment by Ivo Kurzemnieks [ 2014 Feb 28 ]

TESTED.

Comment by Pavels Jelisejevs (Inactive) [ 2014 Mar 03 ]

Fixed in 2.3.0 r43115.

Comment by richlv [ 2014 Mar 03 ]

(39) in api/classes/CDiscoveryRule.php :

'messageUnsupported' => _('Unsupported parameter "%2$s" the filter of discovery rule "%1$s".')

that does not seem to be a valid sentence

jelisejev Corrected directly in trunk r43193. CLOSED.

Comment by richlv [ 2014 Mar 03 ]

(40) looks like most of new/changed translation strings are missing in https://www.zabbix.org/wiki/Docs/specs/ZBXNEXT-581#Translation_strings

jelisejev I've updated the spec and made some corrections in r43961.

<richlv> some things were missing and slightly outdated again - i updated the lists, also noting obvious string changes. please verify that the current list is correct.
also, it seems that "Regexp" could have been changed to "regular expression" - if so, feel free to move it to the 'changed strings' section instead

jelisejev I've moved "Regexp" to the changed string section. Everything else looks correct. CLOSED.

Comment by richlv [ 2014 Mar 31 ]

(41) this issue removed 'filter' field in the items table, but didn't update data_test.sql - it broke some tests that way

jelisejev RESOLVED in svn://svn.zabbix.com/trunk r43963.

sasha CLOSED

Comment by Juris Miščenko (Inactive) [ 2014 Apr 01 ]

(42) (C code) The SQL query in DBcopy_template_items still selects the 'filter' field from the 'items' table. This field was removed in DBpatch_2030026 so this code should be synchronised with that change.

wiper Fixed in development branch svn://svn.zabbix.com/branches/dev/ZBXNEXT-581

sasha REOPENED

DBcopy_template_items() function should populate "item_condition" table.

wiper Fixed in development branch svn://svn.zabbix.com/branches/dev/ZBXNEXT-581

wiper Refactored DBcopy_template_items() function. RESOLVED

sasha Successfully tested! CLOSED

Comment by Andris Zeila [ 2014 Apr 17 ]

Released in pre-2.3.0 r44562

Generated at Wed Apr 24 14:50:59 EEST 2024 using Jira 9.12.4#9120004-sha1:625303b708afdb767e17cb2838290c41888e9ff0.