[ZBX-9428] Low alerter performance on 2.4.2 vs 1.8.15 Created: 2015 Mar 24  Updated: 2017 May 30  Resolved: 2015 Jul 09

Status: Closed
Project: ZABBIX BUGS AND ISSUES
Component/s: Server (S)
Affects Version/s: 2.4.2
Fix Version/s: None

Type: Incident report Priority: Major
Reporter: Viktor Assignee: Unassigned
Resolution: Cannot Reproduce Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Attachments: Text File alert.log    

 Description   

Activation of 700+ triggers make alerter go to 100% cpu load and 5 minutes delay of messages sending. Alerter on zabbix 1.8.15 works fine on this load. Is it possible to improve alerter performance?



 Comments   
Comment by Marc [ 2015 Mar 26 ]

One option could be ZBXNEXT-2442.

Comment by Aleksandrs Saveljevs [ 2015 Mar 27 ]

What kind of notifications are you using? Does "5 minutes delay of messages sending" mean that at most 1 notification is sent in 5 minutes?

Comment by Viktor [ 2015 Mar 27 ]

We are using Email media type to 127.0.0.1.
It`s mean that for 5 minutes after this triggers is ON any email was not sent. Alerter even does not connect to postfix.

Comment by Aleksandrs Saveljevs [ 2015 Mar 27 ]

Would it be possible to provide a DebugLevel=4 log showing what alerter process is doing? Note that since Zabbix 2.4.0 it is possible to change debug level for separate processes at runtime without restarting the server (see http://blog.zabbix.com/zabbix-2-4-features-part-6-runtime-loglevel-changing/).

Comment by Viktor [ 2015 Mar 30 ]

I was not able to reproduse it today (i will try later), but I`ve noticed 30-seconds delay after enabling trigger and sending alert.
Trigger was enabled at 2015-03-27 18:37:41.
But at same time alert from another trigger was processed for sending. And only 30 seconds later (Mar 27 18:38:11) my message was send.
In log:
6682:20150327:183711.804 alerter sending alerts
6682:20150327:183711.804 alerter sent alerts: 0 success, 0 fail in 0.000661 sec, idle 30 sec
6682:20150327:183741.806 alerter sending alerts
6682:20150327:183741.851 alert ID 703623 was sent successfully <- this is my message

maybe this is the reason? is it normal 'idle 30 sec'?

Как вы относитесь к комментариям на русском?

Comment by richlv [ 2015 Mar 30 ]

30 seconds sound normal, that's the default frequency at which zabbix tries to send the messages - parameter SenderFrequency in the server config file

Comment by richlv [ 2015 Mar 30 ]

also please see https://www.zabbix.org/wiki/Docs/bug_reporting_guidelines#Reporting_an_issue

Comment by Viktor [ 2015 Mar 31 ]

I still trying to reproduce this case, and I've noticed that the speed of mail sending decreasing.
From 12:09:18 to 12:09:22 100 messages was sent (25msg/second)
From 12:09:23 speed is 1 msg/second
select count from alerts where status=0 and alerttype=0 and sendto like 'alertertes%'; return >500 items, but speed is too slow.
It`s look like only when user groups (with 3 user in group) is used in actions and 2 steps (second with 1min delay) are defined this case can be reproduced.

Comment by Viktor [ 2015 Apr 01 ]

Here is log file alert.log

Comment by Aleksandrs Saveljevs [ 2015 Apr 01 ]

According to "alert.log", the slowness is in send_email() function. If we grep the log by this function, we will see the following:

  ...
  1436:20150331:120922.418 In send_email() smtp_server:'127.0.0.1'
  1436:20150331:120922.457 End of send_email():SUCCEED

  1436:20150331:120922.458 In send_email() smtp_server:'127.0.0.1'
  1436:20150331:120922.497 End of send_email():SUCCEED

  1436:20150331:120922.497 In send_email() smtp_server:'127.0.0.1'
  1436:20150331:120922.537 End of send_email():SUCCEED

  1436:20150331:120922.537 In send_email() smtp_server:'127.0.0.1'
  1436:20150331:120923.577 End of send_email():SUCCEED

  1436:20150331:120923.577 In send_email() smtp_server:'127.0.0.1'
  1436:20150331:120924.619 End of send_email():SUCCEED

  1436:20150331:120924.620 In send_email() smtp_server:'127.0.0.1'
  1436:20150331:120925.657 End of send_email():SUCCEED
  ...

It can be seen that before 12:09:22, it takes around 40 ms to send an email. After 12:09:22, it takes around 1s 40 ms to send an email.

So my guess is that at some point the email server is starting to limit the rate at which Zabbix server can send emails. Could you please check that?

Generated at Wed Jul 16 09:49:12 EEST 2025 using Jira 9.12.4#9120004-sha1:625303b708afdb767e17cb2838290c41888e9ff0.