Actions get executed when they shouldn't on busy system

XMLWordPrintable

    • Type: Incident report
    • Resolution: Won't fix
    • Priority: Critical
    • None
    • Affects Version/s: 1.8.3
    • Component/s: Server (S)
    • None
    • Environment:
      Centos 5.4 64 bits

      We have a zabbix server monitoring about 292 hosts, 6481 items, 3546 triggers. Now every night about 50 servers are scheduled for reboot due to maintenance. We have configured maintenance periods to make sure that those servers don't wake us up in the middle of the night while they reboot. We still receive about 5% of actions that are happening in maintenance periods, sending emails to the technicien who is on watch.

      At first we thought it was a "maintenance period" problem, so I patched the code in "timer.c" to also put the host in a group called "Maintenance" and log when hosts were put in and out of this special group. I also updated all the actions to make sure that no action would be executed if the host was in the "Maintenance" group or if it was in "maintenance".

      I can confirm that the code in timer.c is correct since all the hosts are correctly put in and out of their maintenance periods as they are supposed to (and also in and out of the "Maintenance" group due to my patch. However we are still receiving the false positive (actions) on a daily basis from host that are both in maintenance status AND in the "Maintenance" group.

      I just can't figure why SOME of the action conditions seems to be evaluated correclty (of 3 "and" conditions, only the one that evaluate to true seems to be evaluated correctly, even if the two others evaluate to false and would not execute the action). I also saw the action sending the mail only to 3 out of 5 configured destination for the action...

      Recorrection of bug #ZBX-2305 in revision 11595 seems to have helped, but didn't resolve the problem. untergeek on the forum mentionned that somteime trigger dependencies were not being evaluated properly in cases where the first step of the Action was immediate and suggested I start to send notification at step 2, we didn't try this yet, mostly because it would introduce another 60 seconds delay in the event chain.

      I will post screenshots of this problem, in the meantime I would really like to have some documentation of where the actions and events are evaluated and executed in the code to be able to try and troubleshoot things with you.

      I sent a request to my boss to buy support from you guys, I should have the purchase order soon, but I would really like to see this problem resolved as soon as possible.

      Thanks

            Assignee:
            Unassigned
            Reporter:
            Marc-Andre Gosselin
            Votes:
            0 Vote for this issue
            Watchers:
            1 Start watching this issue

              Created:
              Updated:
              Resolved: