-
Type:
Problem report
-
Resolution: Unresolved
-
Priority:
Major
-
None
-
Affects Version/s: 7.4.14, 7.4.15, 8.0.0rc1
-
Component/s: Server (S)
-
None
-
Environment:Zabbix server 7.4.14, MariaDB 10.11.6 backend. Zabbix server 7.4.15 has the identical file (see description).
Hosts monitored via SNMP (H3C switches). LLD rule: net.if.discovery.
Graph prototype names: "Interface {#IFNAME}({#IFALIAS}): Network traffic" and "Interface {#IFNAME}({#IFALIAS}): Network optical power".Zabbix server 7.4.14, MariaDB 10.11.6 backend. Zabbix server 7.4.15 has the identical file (see description). Hosts monitored via SNMP (H3C switches). LLD rule: net.if.discovery. Graph prototype names: "Interface {#IFNAME}({#IFALIAS}): Network traffic" and "Interface {#IFNAME}({#IFALIAS}): Network optical power".
-
Support backlog
Problem
When a macro value used inside a graph prototype name changes, the LLD process creates a new graph (new graphid, same member items, same parent prototype) and marks the previously existing graph as lost - instead of renaming the existing graph in place.
This breaks graph history continuity: every change of an interface description on the device gives that interface's traffic graph a new graphid, and the old graph is abandoned and later removed by the housekeeper.
In Zabbix 7.0.30 the very same operation renames the existing graph in place and keeps the graphid. This is therefore a regression introduced by the 7.4 LLD rework.
Steps to reproduce
- Have a host with an SNMP interface LLD rule and a graph prototype named "Interface
{#IFNAME}
(
{#IFALIAS}): Network traffic", creating one graph per discovered interface.
# Note the graphid of the traffic graph of one interface, e.g. GE1/0/28 -> graphid A.
# Change that interface's description (ifAlias) on the device (e.g. from an empty value to "test").
# Force the LLD rule to run: task.create {"type":6,"request":{"itemid":"<discovery rule itemid>"}}.
# Compare:
* Expected (as in 7.0.30): graphid A is renamed to the new name and stays alive.
* Actual (7.4.x): a new graph (graphid B) is created with the new name, and graphid A is marked lost (graph_discovery.status=1, ts_delete set).
A second prototype on the same host, same LLD rule and same discovery run behaves correctly (renames in place). This shows the outcome depends on incidental item ordering, not on the macro change itself.
h3. Observed data (production instance, 7.4.14)
Interface GE1/0/28 has 7 instance items created by the LLD, ordered by itemid:
429478 net.if.in[ifHCInOctets.28] <- member of the TRAFFIC graph prototype 429531 net.if.out[ifHCOutOctets.28] <- member of the TRAFFIC graph prototype 429584 net.if.type[ifType.28] 429637 net.if.status[ifOperStatus.28] 429690 net.if.speed[ifHighSpeed.28] 429744 net.if.op-rx[ifrx.28] <- member of the OPTICAL graph prototype 429782 net.if.op-tx[iftx.28] <- member of the OPTICAL graph prototype TRAFFIC prototype (members = items 1,2 -> NOT at the tail of item_links) -> NEW GRAPH CREATED OPTICAL prototype (members = items 6,7 -> AT the tail of item_links) -> RENAMED IN PLACE
Result after changing the description (same host, same LLD run):
graphid 86181 "... Network optical power" -> renamed in place, still alive OK graphid 86127 "... Network traffic" -> marked LOST problem graphid 165121 "... Network traffic" (old) -> marked LOST problem graphid 165122 "... Network traffic" (new) -> CREATED (new graphid) problem
Server debug log (zabbix_server -R log_level_increase="lld worker") confirms the difference between the two paths:
optical path: lld_graphs_get / lld_gitems_get / lld_graphs_save traffic path: lld_graphs_get / lld_gitems_get / lld_graph_make <- a new graph is allocated
h3. Root cause
File src/zabbix_server/lld/lld_graph.c, function lld_graph_get() - identical in 7.4.14, 7.4.15 and master (md5 61ae9c48f6f3d5a450e503d29dfd27fd for 7.4.14 and 7.4.15):
static zbx_lld_graph_t *lld_graph_get(zbx_hashset_t *graph_index, const zbx_vector_lld_item_link_ptr_t *item_links, const char *name) { zbx_lld_item_graphs_t *ig = NULL; /* declared OUTSIDE the loop */ zbx_lld_graph_t *graph = NULL; for (int i = 0; i < item_links->values_num && NULL == graph; i++) { const zbx_lld_item_link_t *item_link = item_links->values[i]; if (NULL != (ig = (zbx_lld_item_graphs_t *)zbx_hashset_search(graph_index, &item_link->itemid))) { /* ig is OVERWRITTEN on every iteration */ for (int j = 0; j < ig->graphs.values_num; j++) { if (0 == strcmp(ig->graphs.values[j]->name, name)) { graph = ig->graphs.values[j]; break; } } } } if (NULL == ig) /* holds only the LAST iteration's result */ return NULL; /* -> caller treats it as "no existing graph" */ if (NULL == graph) graph = ig->graphs.values[0]; /* fallback, reachable only if last iteration hit */
The loop keeps iterating while the name does not match. Since the graph name has just changed (a macro value changed), the inner name comparison never matches, so the loop runs through all item_links. When it ends, ig holds the search result for the last item_link only.
* If the last item_link belongs to this graph -> ig != NULL -> the fallback reuses that graph -> correct in-place rename.
* If the last item_link does not belong to this graph (not present in the index under that itemid) -> ig == NULL -> return NULL -> the caller lld_graph_make() takes the else branch and allocates a new graph object (graphid = 0), which becomes a new graph record on save.
Which case occurs depends purely on the position of the prototype's member items inside the row's item_links vector, i.e. on incidental itemid ordering. This explains why one prototype renames correctly while another prototype on the same host, same rule and same discovery run creates a new graph every time.
h3. Reference: how 7.0.30 does it correctly
File src/libs/zbxdbhigh/lld_graph.c (7.0.30), functions lld_graph_by_item() and lld_graph_get():
static zbx_lld_graph_t *lld_graph_by_item(const zbx_vector_lld_graph_ptr_t *graphs, zbx_uint64_t itemid) { for (int i = 0; i < graphs->values_num; i++) { zbx_lld_graph_t *graph = graphs->values[i]; if (0 != (graph->flags & ZBX_FLAG_LLD_GRAPH_DISCOVERED)) continue; /* skip graphs already claimed in this run */ for (int j = 0; j < graph->gitems.values_num; j++) if (graph->gitems.values[j]->itemid == itemid) return graph; /* match purely by ITEM, never by name */ } return NULL; } static zbx_lld_graph_t *lld_graph_get(const zbx_vector_lld_graph_ptr_t *graphs, const zbx_vector_lld_item_link_ptr_t *item_links) { zbx_lld_graph_t *graph; for (int i = 0; i < item_links->values_num; i++) if (NULL != (graph = lld_graph_by_item(graphs, item_links->values[i]->itemid))) return graph; /* ANY hit wins; no cross-iteration state */ return NULL; }
and in lld_graph_make(), once a graph is found, the name is updated in place:
if (NULL != (graph = lld_graph_get(graphs, &lld_row->item_links))) { if (0 != strcmp(graph->name, buffer)) { graph->name_orig = graph->name; graph->name = buffer; /* rename, keep graphid */ } }
h3. Suggested fix
Minimal change, behaviour-equivalent to 7.0.30 ("any hit wins"), no new semantics:
zbx_lld_item_graphs_t *ig = NULL; + int found = 0; zbx_lld_graph_t *graph = NULL; for (int i = 0; i < item_links->values_num && NULL == graph; i++) { const zbx_lld_item_link_t *item_link = item_links->values[i]; if (NULL != (ig = (zbx_lld_item_graphs_t *)zbx_hashset_search(graph_index, &item_link->itemid))) { + found = 1; for (int j = 0; j < ig->graphs.values_num; j++) { if (0 == strcmp(ig->graphs.values[j]->name, name)) { graph = ig->graphs.values[j]; break; } } } } - if (NULL == ig) + if (0 == found) return NULL;
With this change ig always refers to the last group that actually hit, which is exactly the group the existing fallback line already assumes, so the fallback keeps its current meaning while the spurious NULL case disappears.
h3. Impact (measured on one production instance)
graph prototypes whose name contains {#IFALIAS}: 1526 (1367 template-level + 159 host-level)
instance graphs managed by them: 16591
graphs currently in LOST state with description: 170 (growing)
graphs currently in LOST state without desc: 2
Note on latest version
The function is byte-identical in 7.4.14, 7.4.15 and master (md5 of lld_graph.c: 61ae9c48f6f3d5a450e503d29dfd27fd for both 7.4.14 and 7.4.15), so the issue is not fixed in the latest released version. Verification was therefore done by source comparison rather than by re-testing on an upgraded instance.
- caused by
-
ZBXNEXT-1527 cascaded/nested lld
-
- Closed
-