Our 2.4 system has started getting stuck triggers, which become a PROBLEM normally but then do not revert to OK after the real issue goes away.

For example, we might have a trigger to alert > 50% CPU, so at 60% we get a PROBLEM, but when it goes down to 40%, the problem does not go away.

This only happens rarely, far less than 1% of the time - we have 65,603 active triggers and only a few have this issue, but it's random.

This especially happens to Trapper items, but can also affect Agent items.

This just started 2-3 months ago and is getting worse, though nothing really changed - we were on 2.4.1 for a year and after this started, we upgraded to 2.4.7 but no help.

So we modified the source to improve logging (as DEBUG logging immediately consumes all CPU and disk space), in various functions, and we see the DB is getting updated, but the actual DB is not, see below:

19748:20151229:155649.693 In process_trigger() triggerid:537951 value:0(0) new_value:1 
19748:20151229:155649.693 Trigger [537951]: 
value_changed: 1 
state_changed: 0 
multiple_problem: 0 
error_changed: 0
19748:20151229:155649.693 SQL VALUE:: (update triggers set lastchange=1451375807,value=1 where triggerid=537951) ::
19748:20151229:155649.694 End of process_trigger():SUCCEED

At next time, zabbix detect the status, and also execute "update" table "triggers" operation as below.

19749:20151229:160152.526 In process_trigger() triggerid:537951 value:1(0) new_value:0
19749:20151229:160152.526 Trigger [537951]: 
value_changed: 1 
state_changed: 0 
multiple_problem: 0 
error_changed: 0
19749:20151229:160152.526 SQL VALUE:: (update triggers set lastchange=1451376107,value=0 where triggerid=537951) ::
19749:20151229:160152.526 End of process_trigger():SUCCEED

But even though, zabbix executed the SQL, the database side still is not updated as below.

mysql> select triggerid,lastchange,value from triggers where triggerid=537951;
+-----------+------------+-------+
| triggerid | lastchange | value |
+-----------+------------+-------+
| 537951    | 1451375807 | 1     |
+-----------+------------+-------+