• Icon: Problem report Problem report
    • Resolution: Unresolved
    • Icon: Trivial Trivial
    • None
    • 6.4.18
    • Server (S)
    • None

      We use custom Zabbix media type to integrate it's events with external alerting system.
      This media type stores external alert's ID in Zabbix event's tag.
      Media type is executed within simple action with zero delay for the first step.

      From time to time Zabbix generates many events with zero duration (event start = event end). When Zabbix starts to process these events, it executes 2 actions:

      1. Create alert in external system (and receive it's ID).
      2. Close alert in external system by it's ID (received during previous step).

      As far as I can see, actions are being executed with ~3 seconds interval.
      But for some of these events close action fails because at the time of it's execution event tag with alert ID is missing. Action retries fails with the same error.

      It happens with <1% of events. But it happens.
      Setting concurrent sessions to 1 in media type settings doesn't solve the issue.
      So for me it seems to be some kind of a race condition between actions execution.

      Steps to reproduce:

      1. Use some media type that stores external system ID in Zabbix event tag and throw errors if this ID is missing for subsequent actions.
      2. Create typical action for above media type with Notify all involved on event resolve.
      3. Create Zabbix trapper item.
      4. Create last()=1 trigger for above item with multiple event generation.
      5. Generate bunch (200 for ex) of item values (0/1) with same timestamp and send it with zabbix_sender.
      6. Check action log for errors.

      Result:
      Errors in action log with missing external system ID
      Expected:
      No errors

        1. rc_test_action.PNG
          rc_test_action.PNG
          100 kB
        2. rc_test_host.yaml
          0.7 kB
        3. rc_test_media.yaml
          2 kB
        4. rc_test_sender.py
          0.4 kB
        5. zabbix_server.log.gz
          2.08 MB

            zabbix.support Zabbix Support Team
            akhudushin Aleksandr Khudushin
            Votes:
            0 Vote for this issue
            Watchers:
            2 Start watching this issue

              Created:
              Updated: