-
Type:
Incident report
-
Resolution: Unresolved
-
Priority:
Trivial
-
None
-
Affects Version/s: None
-
Component/s: API (A), Frontend (F), Server (S)
-
None
Steps to reproduce
Clean installation, frontend and server 7.0.32. The server may be stopped. If it is running, perform the steps more than 30 minutes after server start-up (after the first housekeeper cycle). The issue is then visible during the following 7 days.
A shell file that performs both scenarios through the API is attached (repro.sh).
Scenario A - host group deletion
- Create host groups HG-X and HG-Y.
- Create user group UG-A with host group rights HG-X = Read-write. No rights on HG-Y.
- Create user user-a in UG-A only.
- Create host host-a in groups HG-X and HG-Y. The host gets a host group set S made of HG-X and HG-Y, and a permission row for the user group set of user-a on S is created (Read-write).
- Delete host group HG-X (hostgroup.delete).
- Edit user group UG-A and grant HG-Y = Read (usergroup.update with hostgroup_rights).
Scenario B - user group deletion (same defect on the user group set side)
- Create host group HG-Z and host host-b in HG-Z only.
- Create user groups UG-1 (rights: HG-Z = Read-write) and UG-2 (no rights).
- Create user user-b in UG-1 and UG-2. The user gets a user group set U made of UG-1 and UG-2, and a permission row for U on the host group set of host-b is created (Read-write).
- Delete user group UG-1 (usergroup.delete).
- Edit user group UG-2 and grant HG-Z = Read.
Result
Step 6 (A) / step 5 (B) fails, the transaction is rolled back and the user group cannot be saved. The frontend shows a database error: duplicate key value violates unique constraint permission_pkey, raised from CUserGroup::updatePermissions(). The full error text from production is attached (error.txt).
The same error is raised by hostgroup.propagate, because CHostGroup::inheritPermissions() calls usergroup.update.
After step 5 of scenario A, the host group set S still exists, is not linked to any host, has lost HG-X from its member list (table hgset_group), keeps the hash of the old membership and keeps the old permission row. A diagnostic query is attached (check.sql).
Expected
User group rights can be updated at any time after a host group or user group deletion.
There should be no rows in the permission table that do not correspond to the current rights of the current set members.
Root cause analysis
ZBX-27561(7.0.28) stopped removing unused group sets synchronously:- CHostGroup::deleteForce() and CTemplateGroup::delete(): the former deleteUnusedHgSets() removed every host group set containing the deleted group (cascading to hgset_group and permission). It was replaced by deleteUnusedHgSetGroups(), which only removes the hgset_group rows of the deleted groups. This is required because of the RESTRICT foreign key from hgset_group.groupid to hstgrp.
- CUserGroup::delete(): deleteUnusedUgSets() was replaced by deleteUnusedUgSetGroups(), which only removes the ugset_group rows of the deleted user groups.
- Server zbx_db_delete_groups() (src/libs/zbxdbwrap/host.c): host group sets are only unlinked from hosts and their hgset_group rows removed.
- Removal of the remaining hgset / ugset records and their permission rows is delegated to the server housekeeper, function housekeeping_group_sets().
- After hosts or users are moved to new sets, the old set is left without hosts or users, with fewer members than its hash represents, and with the permission rows calculated for the old membership. The rights of the deleted group are removed by the cascading foreign key on the rights table, the permission rows are not.
- CUserGroup::updateRights() calls getPermissions(), which picks all host group sets containing the changed groups through hgset_group (including sets without hosts) and calculates the previous permission from the rights table only, never reading the permission table. For the stale set the calculated previous value is "none" while a row already exists, so a new row is added and the primary key is violated. The same happens for stale user group sets picked through ugset_group in getUgSetPermissions().
- housekeeping_group_sets() (src/zabbix_server/housekeeper/housekeeper_server.c) runs at most once per week (SEC_PER_WEEK, kept in a static last_exec_time variable), regardless of HousekeepingFrequency. The runtime control option housekeeper_execute does not bypass this limit. Therefore the error window after every group deletion is up to 7 days.
- With a server older than 7.0.28 (for example frontend 7.0.32 with server 7.0.25, a supported combination within the same major version) the housekeeper cleanup does not exist at all and stale sets accumulate permanently.
ZBX-26766 (choosing the lowest hgsetid per hash) does not help: CUserGroup does not use the hash at all.
Production impact
Environment of the reporter: frontend 7.0.32, server 7.0.25, PostgreSQL. After a mass restructuring of host groups, host group sets were classified as follows:
| Linked to hosts | Has members | Hash matches members | Count |
|---|---|---|---|
| yes | yes | yes | 3061 |
| no | yes | yes | 895 |
| no | yes | no (mutilated) | 2492 |
| no | none (orphan) | no | 6 |
208 506 permission rows reference unused sets. Several user groups could not be saved.
No access leak was observed: stale sets have no hosts or users, so the stale permission rows are not used by access checks.
Workaround
Run the same statements as housekeeping_group_sets() manually in one transaction (attached as workaround.sql), or restart a 7.0.28+ server and wait for the first housekeeper cycle (about 30 minutes after start).
Suggested fix (any of)
- In CHostGroup::deleteForce(), CTemplateGroup::delete(), CUserGroup::delete() and server zbx_db_delete_groups(), also remove the permission rows of the affected sets that have no hosts or users. Such sets cannot be reused by hash anyway, because their hash contains a deleted group.
- In CUserGroup::getPermissions() and getUgSetPermissions(), ignore sets that are not linked to any host (host_hgset) or user (user_ugset).
- In CUserGroup::updatePermissions(), read the actual permission rows (or use an upsert) instead of relying on the value derived from the rights table.
Attachments
- repro.sh - reproduces both scenarios through the JSON-RPC API (curl + jq)
- check.sql - diagnostic queries (set classification, members, hosts/users and permission rows per set)
- workaround.sql - manual cleanup, same statements as housekeeping_group_sets()
- error.txt - full frontend error from production
- docker-compose.yml - disposable 7.0.32 test environment