[ZBX-7983] Widget "System Status" is slow on installations with huge amount of PROBLEM triggers and Host Groups Created: 2014 Mar 24  Updated: 2017 Sep 16  Due: 2014 Apr 17  Resolved: 2014 Jun 02

Status: Closed
Project: ZABBIX BUGS AND ISSUES
Component/s: Frontend (F)
Affects Version/s: 2.2.2
Fix Version/s: 2.2.4rc1, 2.3.2

Type: Incident report Priority: Major
Reporter: Alexey Pustovalov Assignee: Ivo Kurzemnieks
Resolution: Fixed Votes: 0
Labels: dashboard, performance
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Attachments: PNG File groups.png    
Issue Links:
Duplicate
is duplicated by ZBX-6869 Getting last event in CTrigger.php is... Closed

 Description   
******************** Script profiler ********************
Total time: 39.737475
Total SQL time: 10.014377
SQL count: 5764 (selects: 5749 | executes: 15)
Peak memory usage: 461.25M
Memory limit: 1024M

Amount of Host Groups: 2110
Amount of PROBLEM triggers: 5709

The issue is in selectLastEvent option. There is very slow place.



 Comments   
Comment by Pavels Jelisejevs (Inactive) [ 2014 Mar 26 ]

A related issue - ZBX-6603.

Comment by Eduards Samersovs (Inactive) [ 2014 Apr 16 ]

Fixed in development branch svn://svn.zabbix.com/branches/dev/ZBX-7983

Comment by Ivo Kurzemnieks [ 2014 Apr 30 ]

(1) No translatable strings changed.

oleg.egorov CLOSED

Comment by Ivo Kurzemnieks [ 2014 Apr 30 ]

(2) Following request does not return 'lastEvent'

{
    "jsonrpc": "2.0",
    "method": "trigger.get",
    "id": 9,
    "auth": "ccf37ec70b8a8463fd9a23857ed76b3a",
    "params": {
        "output": [
            "triggerid",
            "description"
        ],
        "hostids": [
            "10084"
        ],
        "selectLastEvent": [
            "eventid",
            "clock"
        ]
    }
}

Problem is that now "objectid" needs to be requested now. It shouldn't be required in request.

iivs RESOLVED in r45057, r45084

oleg.egorov CLOSED

Comment by Ivo Kurzemnieks [ 2014 Apr 30 ]

(3) blocks.inc.php: 249 "clock" and "value" are not required, since they are read from $trigger['lastchange'] and $trigger['value'] on lines 298 and 299.

iivs RESOLVED in r45042

oleg.egorov CLOSED

Comment by Oleg Egorov (Inactive) [ 2014 May 10 ]

Function make_system_status execution time is 21 sec
And

	foreach ($groups as $group) {
		$groupRow = new CRow();

		if ($showAllNodes) {
			$groupRow->addItem($group['nodename']);
		}

		$name = new CLink($group['name'], 'tr_status.php?groupid='.$group['groupid'].'&hostid=0&show_triggers='.TRIGGERS_OPTION_ONLYTRUE);
		$groupRow->addItem($name);

		foreach ($group['tab_priority'] as $severity => $data) {
			if (!is_null($filter['severity']) && !isset($filter['severity'][$severity])) {
				continue;
			}

			$allTriggersNum = $data['count'];
			if ($allTriggersNum) {
				$allTriggersNum = new CSpan($allTriggersNum, 'pointer');
				$allTriggersNum->setHint(makeTriggersPopup($data['triggers'], $ackParams, $actions));
			}

			$unackTriggersNum = $data['count_unack'];
			if ($unackTriggersNum) {
				$unackTriggersNum = new CSpan($unackTriggersNum, 'pointer red bold');
				$unackTriggersNum->setHint(makeTriggersPopup($data['triggers_unack'], $ackParams, $actions));
			}

			switch ($filter['extAck']) {
				case EXTACK_OPTION_ALL:
					$groupRow->addItem(getSeverityCell($severity, $allTriggersNum, !$allTriggersNum));
					break;

				case EXTACK_OPTION_UNACK:
					$groupRow->addItem(getSeverityCell($severity, $unackTriggersNum, !$unackTriggersNum));
					break;

				case EXTACK_OPTION_BOTH:
					if ($unackTriggersNum) {
						$span = new CSpan(SPACE._('of').SPACE);
						$unackTriggersNum = new CSpan($unackTriggersNum);
					}
					else {
						$span = null;
						$unackTriggersNum = null;
					}

					$groupRow->addItem(getSeverityCell($severity, array($unackTriggersNum, $span, $allTriggersNum), !$allTriggersNum));
					break;
			}
		}
		$table->addRow($groupRow);
	}

Execution time is 10s...
10s to display 8 groups.

See groups.png

Updated

$allTriggersNum->setHint(makeTriggersPopup($data['triggers'], $ackParams, $actions));

This code have longer execution time

In makeTriggersPopup function possible not use get_node_name_by_elid for each trigger if zabbix installation is without nodes.

Comment by Ivo Kurzemnieks [ 2014 May 14 ]

(4) Refactored the code to improve "System status" widget performance even more:

  • removed duplicate node column for trigger pop-up;
  • added function to return all node names by passing an array instead of each object;

RESOLVED in r45494, r45650

oleg.egorov CLOSED

Comment by Ivo Kurzemnieks [ 2014 May 19 ]

(5) Highly experimental solution provides 50% faster loading and 50% less memory usage to fix map loading issue. See r45609

oleg.egorov Confirmed execution time in the large installation

Was:
Total time: 23.624351
Total SQL time: 15.13987
SQL count: 131 (selects: 63 | executes: 68)
Peak memory usage: 115.75M
Memory limit: 1024M

Now:
Total time: 14.412824
Total SQL time: 10.434596
SQL count: 138 (selects: 66 | executes: 72)
Peak memory usage: 103.75M
Memory limit: 1024M

CLOSED

Comment by Oleg Egorov (Inactive) [ 2014 May 19 ]

(6)Was:
Total time: 0.746042
Total SQL time: 0.104005
SQL count: 73 (selects: 37 | executes: 36)
Peak memory usage: 14.75M
Memory limit: 1024M

Now:
Total time: 0.778044
Total SQL time: 0.096006
SQL count: 81 (selects: 40 | executes: 41)
Peak memory usage: 78.5M
Memory limit: 1024M

DB o_22_test
maps.php?sysmapid=22

Map problem still exists, and if no idea how to fix this problem, please open new ZBX

iivs Moved to ZBX-8232.
CLOSED.

Comment by Oleg Egorov (Inactive) [ 2014 May 20 ]

System status widget

Was:
Speed: 0.93605399131775 s.
Memory: 10.68 MB
Max memory: 10.86 MB

Now:
Speed: 0.65403699874878 s.
Memory: 8.37 MB
Max memory: 8.56 MB

Good performance improvement for middle and huge Zabbix installations and no performance changes for little installations

Comment by Oleg Egorov (Inactive) [ 2014 May 20 ]

(7)

		$eventAcknowledges = API::Event()->get(array(
			'eventids' => $eventIds,
			'select_acknowledges' => API_OUTPUT_EXTEND,
			'preservekeys' => true
		));

'select_acknowledges' => API_OUTPUT_EXTEND

But there used only 'clock', 'message', 'alias', 'name', 'surname'

iivs I also added eventid, since it's used as key
RESOLVED in r45685

oleg.egorov CLOSED

Comment by Alexander Vladishev [ 2014 May 20 ]

(8) please review my changes in r45684.

oleg.egorov CLOSED

Comment by Ivo Kurzemnieks [ 2014 May 21 ]

(9) One more update in various frontend places. Now 'config' is passed as parameter. Before it was called in various places inside loops, causing PHP to slow down and eat more memory.

RESOLVED in r45727.

oleg.egorov More issues

  • Minor coding style problem in tr_status.php: 88 unnecessary line break
  • More about config using improvement.
    In maps.inc.php function getSelementsInfo possible use more times select_config()
    This problem was related to map performance (ZBX-8232)
  • Unnecessary config usage in events.php
    In file begin
    require_once dirname(__FILE__).'/include/page_header.php';
    

But after $config = select_config();
Please check other places

iivs There are more places to get rid of second call of $config in various ways. I managed to remove usage in few views and call it only in controller or even page header once.

RESOLVED in r45886

oleg.egorov

Configuration->Maps->Edit

Missing argument 2 for getSeverityCaption(), called in C:\xampp\htdocs\ZBX-7983\frontends\php\include\classes\class.cseverity.php on line 42 and defined [sysmaps.php:275 → CView->render() → include() → CSeverity->__construct() → getSeverityCaption() in C:\xampp\htdocs\ZBX-7983\frontends\php\include\triggers.inc.php:43]
Undefined variable: config [sysmaps.php:275 → CView->render() → include() → CSeverity->__construct() → getSeverityCaption() in C:\xampp\htdocs\ZBX-7983\frontends\php\include\triggers.inc.php:45]
Undefined variable: config [sysmaps.php:275 → CView->render() → include() → CSeverity->__construct() → getSeverityCaption() in C:\xampp\htdocs\ZBX-7983\frontends\php\include\triggers.inc.php:46]
Undefined variable: config [sysmaps.php:275 → CView->render() → include() → CSeverity->__construct() → getSeverityCaption() in C:\xampp\htdocs\ZBX-7983\frontends\php\include\triggers.inc.php:47]
Undefined variable: config [sysmaps.php:275 → CView->render() → include() → CSeverity->__construct() → getSeverityCaption() in C:\xampp\htdocs\ZBX-7983\frontends\php\include\triggers.inc.php:48]
Undefined variable: config [sysmaps.php:275 → CView->render() → include() → CSeverity->__construct() → getSeverityCaption() in C:\xampp\htdocs\ZBX-7983\frontends\php\include\triggers.inc.php:49]
Undefined variable: config [sysmaps.php:275 → CView->render() → include() → CSeverity->__construct() → getSeverityCaption() in C:\xampp\htdocs\ZBX-7983\frontends\php\include\triggers.inc.php:50]

Same problem in trigger edit form

And in trigger prototypes

Undefined index: config [trigger_prototypes.php:323 → CView->render() → include() in C:\xampp\htdocs\ZBX-7983\frontends\php\include\views\configuration.triggers.list.php:253]

REOPENED

iivs RESOLVED in r46110

iivs I updated to latest from /branches/2.2 there were many conflicts, please review r46112

oleg.egorov MOVED TO ZBX-8232

iivs After last discussion we decided to revert all changes about configuration now passed as parameter. It touched too many places infrontend, potentially breaking a lot of things and code didn't look that good. It is possible to optimize many places, but this needs a different and better solution. Code has been reverted and updated to latest /branches/2.2 please review once more r46198

oleg.egorov CLOSED

Comment by Marc [ 2014 May 26 ]

Haven't read this ticket in detail but might it address/improve/solve ZBX-6557 as well?

Comment by Ivo Kurzemnieks [ 2014 May 27 ]

Marc, no this will not affect fullscreen and normal screen behaviour.

Comment by Oleg Egorov (Inactive) [ 2014 Jun 04 ]

TESTED

Comment by Ivo Kurzemnieks [ 2014 Jun 05 ]
  • Optimized last event calculation in "System status" widget improving speed and less memory usage.
  • For 2.2 removed duplicating node column on trigger pop-up.

Fixed in pre-2.2.4rc1 r46225 and 2.3.2 (trunk) r46228, r46232

Comment by Martins Valkovskis [ 2014 Jun 06 ]

Thanks for the summary, documented in:

iivs Great! Thanks!

Comment by Oleksii Zagorskyi [ 2014 Jul 22 ]

Looks like current changed caused a regression, see ZBX-8424.

Generated at Wed Apr 24 05:35:30 EEST 2024 using Jira 9.12.4#9120004-sha1:625303b708afdb767e17cb2838290c41888e9ff0.