Thousands of Email notifications sent for hours

XMLWordPrintable

    • Type: Problem report
    • Resolution: Won't fix
    • Priority: Trivial
    • None
    • Affects Version/s: None
    • Component/s: None
    • None
    • Environment:
      Zabbix Server: 4.0.20, MySQL DB Version: 5.6.10, Operating System: Ubuntu 18.04

       

      Steps to reproduce:

      1. We have a trapper item prototype which gets data from a script. When item does not receive data it was set to delete with "Keep lost resources period" set to 0.
      2. This item prototype has an associated trigger prototype. 
      3. Starting at 3:55 PM CST there were 12788 emails sent to one user for 246 unique discovered items until 9:15 PM CST when i disabled the action to stop the emails. Refer attached snapshot of Alerter. There were thousands of OK emails as well as PROBLEM emails. 
      4. Even though our Action is configured to send email every 5 mins, I could see multiple emails being sent for same item within a few seconds interval. Refer attached snapshot of action log.
      5. Action log indicates multiple events were generated for same problem even though trigger has "PROBLEM event generation mode" set to single. 
      6. Looks like some dead loop occurred in escalation logic. Please help us fix this problem so that it won't reoccur again. 

      Result:
      See screenshots
      See log file 
      Expected:
      Emails should be sent every 5 mins. Email count should have quite less. Multiple problem events should not have been generated. 

        1. alerter process utilization.png
          alerter process utilization.png
          108 kB
        2. emails-1.png
          emails-1.png
          956 kB
        3. emails-2.png
          emails-2.png
          897 kB
        4. esc-cancelled.log
          22 kB
        5. item-prototype.PNG
          item-prototype.PNG
          35 kB
        6. trigger-prototype.PNG
          trigger-prototype.PNG
          44 kB
        7. action log.png
          action log.png
          105 kB
        8. number-of-items1.PNG
          number-of-items1.PNG
          60 kB
        9. number-of-items2.PNG
          number-of-items2.PNG
          59 kB
        10. emailcount.csv
          3 kB
        11. alerts.csv
          2.51 MB
        12. image-2020-07-04-12-46-37-443.png
          image-2020-07-04-12-46-37-443.png
          19 kB
        13. queryoutput.csv
          177 kB
        14. image-2020-07-07-19-16-11-413.png
          image-2020-07-07-19-16-11-413.png
          231 kB

            Assignee:
            Zabbix Support Team
            Reporter:
            Multazim Deshmukh
            Votes:
            0 Vote for this issue
            Watchers:
            4 Start watching this issue

              Created:
              Updated:
              Resolved: