ZABBIX BUGS AND ISSUES
  1. ZABBIX BUGS AND ISSUES
  2. ZBX-7983

Widget "System Status" is slow on installations with huge amount of PROBLEM triggers and Host Groups

    Details

      Description

      ******************** Script profiler ********************
      Total time: 39.737475
      Total SQL time: 10.014377
      SQL count: 5764 (selects: 5749 | executes: 15)
      Peak memory usage: 461.25M
      Memory limit: 1024M

      Amount of Host Groups: 2110
      Amount of PROBLEM triggers: 5709

      The issue is in selectLastEvent option. There is very slow place.

        Issue Links

          Activity

          Hide
          Pavels Jelisejevs (Inactive) added a comment -

          A related issue - ZBX-6603.

          Show
          Pavels Jelisejevs (Inactive) added a comment - A related issue - ZBX-6603 .
          Hide
          Eduards Samersovs (Inactive) added a comment -

          Fixed in development branch svn://svn.zabbix.com/branches/dev/ZBX-7983

          Show
          Eduards Samersovs (Inactive) added a comment - Fixed in development branch svn://svn.zabbix.com/branches/dev/ZBX-7983
          Hide
          Ivo Kurzemnieks added a comment - - edited

          (1) No translatable strings changed.

          Oleg Egorov CLOSED

          Show
          Ivo Kurzemnieks added a comment - - edited (1) No translatable strings changed. Oleg Egorov CLOSED
          Hide
          Ivo Kurzemnieks added a comment - - edited

          (2) Following request does not return 'lastEvent'

          {
              "jsonrpc": "2.0",
              "method": "trigger.get",
              "id": 9,
              "auth": "ccf37ec70b8a8463fd9a23857ed76b3a",
              "params": {
                  "output": [
                      "triggerid",
                      "description"
                  ],
                  "hostids": [
                      "10084"
                  ],
                  "selectLastEvent": [
                      "eventid",
                      "clock"
                  ]
              }
          }
          

          Problem is that now "objectid" needs to be requested now. It shouldn't be required in request.

          Ivo Kurzemnieks RESOLVED in r45057, r45084

          Oleg Egorov CLOSED

          Show
          Ivo Kurzemnieks added a comment - - edited (2) Following request does not return 'lastEvent' { "jsonrpc": "2.0", "method": "trigger.get", "id": 9, "auth": "ccf37ec70b8a8463fd9a23857ed76b3a", "params": { "output": [ "triggerid", "description" ], "hostids": [ "10084" ], "selectLastEvent": [ "eventid", "clock" ] } } Problem is that now "objectid" needs to be requested now. It shouldn't be required in request. Ivo Kurzemnieks RESOLVED in r45057, r45084 Oleg Egorov CLOSED
          Hide
          Ivo Kurzemnieks added a comment - - edited

          (3) blocks.inc.php: 249 "clock" and "value" are not required, since they are read from $trigger['lastchange'] and $trigger['value'] on lines 298 and 299.

          Ivo Kurzemnieks RESOLVED in r45042

          Oleg Egorov CLOSED

          Show
          Ivo Kurzemnieks added a comment - - edited (3) blocks.inc.php: 249 "clock" and "value" are not required, since they are read from $trigger ['lastchange'] and $trigger ['value'] on lines 298 and 299. Ivo Kurzemnieks RESOLVED in r45042 Oleg Egorov CLOSED
          Hide
          Oleg Egorov added a comment - - edited

          Function make_system_status execution time is 21 sec
          And

          	foreach ($groups as $group) {
          		$groupRow = new CRow();
          
          		if ($showAllNodes) {
          			$groupRow->addItem($group['nodename']);
          		}
          
          		$name = new CLink($group['name'], 'tr_status.php?groupid='.$group['groupid'].'&hostid=0&show_triggers='.TRIGGERS_OPTION_ONLYTRUE);
          		$groupRow->addItem($name);
          
          		foreach ($group['tab_priority'] as $severity => $data) {
          			if (!is_null($filter['severity']) && !isset($filter['severity'][$severity])) {
          				continue;
          			}
          
          			$allTriggersNum = $data['count'];
          			if ($allTriggersNum) {
          				$allTriggersNum = new CSpan($allTriggersNum, 'pointer');
          				$allTriggersNum->setHint(makeTriggersPopup($data['triggers'], $ackParams, $actions));
          			}
          
          			$unackTriggersNum = $data['count_unack'];
          			if ($unackTriggersNum) {
          				$unackTriggersNum = new CSpan($unackTriggersNum, 'pointer red bold');
          				$unackTriggersNum->setHint(makeTriggersPopup($data['triggers_unack'], $ackParams, $actions));
          			}
          
          			switch ($filter['extAck']) {
          				case EXTACK_OPTION_ALL:
          					$groupRow->addItem(getSeverityCell($severity, $allTriggersNum, !$allTriggersNum));
          					break;
          
          				case EXTACK_OPTION_UNACK:
          					$groupRow->addItem(getSeverityCell($severity, $unackTriggersNum, !$unackTriggersNum));
          					break;
          
          				case EXTACK_OPTION_BOTH:
          					if ($unackTriggersNum) {
          						$span = new CSpan(SPACE._('of').SPACE);
          						$unackTriggersNum = new CSpan($unackTriggersNum);
          					}
          					else {
          						$span = null;
          						$unackTriggersNum = null;
          					}
          
          					$groupRow->addItem(getSeverityCell($severity, array($unackTriggersNum, $span, $allTriggersNum), !$allTriggersNum));
          					break;
          			}
          		}
          		$table->addRow($groupRow);
          	}
          

          Execution time is 10s...
          10s to display 8 groups.

          See groups.png

          Updated

          $allTriggersNum->setHint(makeTriggersPopup($data['triggers'], $ackParams, $actions));
          

          This code have longer execution time

          In makeTriggersPopup function possible not use get_node_name_by_elid for each trigger if zabbix installation is without nodes.

          Show
          Oleg Egorov added a comment - - edited Function make_system_status execution time is 21 sec And foreach ($groups as $group) { $groupRow = new CRow(); if ($showAllNodes) { $groupRow->addItem($group['nodename']); } $name = new CLink($group['name'], 'tr_status.php?groupid='.$group['groupid'].'&hostid=0&show_triggers='.TRIGGERS_OPTION_ONLYTRUE); $groupRow->addItem($name); foreach ($group['tab_priority'] as $severity => $data) { if (!is_null($filter['severity']) && !isset($filter['severity'][$severity])) { continue ; } $allTriggersNum = $data['count']; if ($allTriggersNum) { $allTriggersNum = new CSpan($allTriggersNum, 'pointer'); $allTriggersNum->setHint(makeTriggersPopup($data['triggers'], $ackParams, $actions)); } $unackTriggersNum = $data['count_unack']; if ($unackTriggersNum) { $unackTriggersNum = new CSpan($unackTriggersNum, 'pointer red bold'); $unackTriggersNum->setHint(makeTriggersPopup($data['triggers_unack'], $ackParams, $actions)); } switch ($filter['extAck']) { case EXTACK_OPTION_ALL: $groupRow->addItem(getSeverityCell($severity, $allTriggersNum, !$allTriggersNum)); break ; case EXTACK_OPTION_UNACK: $groupRow->addItem(getSeverityCell($severity, $unackTriggersNum, !$unackTriggersNum)); break ; case EXTACK_OPTION_BOTH: if ($unackTriggersNum) { $span = new CSpan(SPACE._('of').SPACE); $unackTriggersNum = new CSpan($unackTriggersNum); } else { $span = null ; $unackTriggersNum = null ; } $groupRow->addItem(getSeverityCell($severity, array($unackTriggersNum, $span, $allTriggersNum), !$allTriggersNum)); break ; } } $table->addRow($groupRow); } Execution time is 10s... 10s to display 8 groups. See groups.png Updated $allTriggersNum->setHint(makeTriggersPopup($data['triggers'], $ackParams, $actions)); This code have longer execution time In makeTriggersPopup function possible not use get_node_name_by_elid for each trigger if zabbix installation is without nodes.
          Hide
          Ivo Kurzemnieks added a comment - - edited

          (4) Refactored the code to improve "System status" widget performance even more:

          • removed duplicate node column for trigger pop-up;
          • added function to return all node names by passing an array instead of each object;

          RESOLVED in r45494, r45650

          Oleg Egorov CLOSED

          Show
          Ivo Kurzemnieks added a comment - - edited (4) Refactored the code to improve "System status" widget performance even more: removed duplicate node column for trigger pop-up; added function to return all node names by passing an array instead of each object; RESOLVED in r45494, r45650 Oleg Egorov CLOSED
          Hide
          Ivo Kurzemnieks added a comment - - edited

          (5) Highly experimental solution provides 50% faster loading and 50% less memory usage to fix map loading issue. See r45609

          Oleg Egorov Confirmed execution time in the large installation

          Was:
          Total time: 23.624351
          Total SQL time: 15.13987
          SQL count: 131 (selects: 63 | executes: 68)
          Peak memory usage: 115.75M
          Memory limit: 1024M

          Now:
          Total time: 14.412824
          Total SQL time: 10.434596
          SQL count: 138 (selects: 66 | executes: 72)
          Peak memory usage: 103.75M
          Memory limit: 1024M

          CLOSED

          Show
          Ivo Kurzemnieks added a comment - - edited (5) Highly experimental solution provides 50% faster loading and 50% less memory usage to fix map loading issue. See r45609 Oleg Egorov Confirmed execution time in the large installation Was : Total time: 23.624351 Total SQL time: 15.13987 SQL count: 131 (selects: 63 | executes: 68) Peak memory usage: 115.75M Memory limit: 1024M Now : Total time: 14.412824 Total SQL time: 10.434596 SQL count: 138 (selects: 66 | executes: 72) Peak memory usage: 103.75M Memory limit: 1024M CLOSED
          Hide
          Oleg Egorov added a comment - - edited

          (6)Was:
          Total time: 0.746042
          Total SQL time: 0.104005
          SQL count: 73 (selects: 37 | executes: 36)
          Peak memory usage: 14.75M
          Memory limit: 1024M

          Now:
          Total time: 0.778044
          Total SQL time: 0.096006
          SQL count: 81 (selects: 40 | executes: 41)
          Peak memory usage: 78.5M
          Memory limit: 1024M

          DB o_22_test
          maps.php?sysmapid=22

          Map problem still exists, and if no idea how to fix this problem, please open new ZBX

          Ivo Kurzemnieks Moved to ZBX-8232.
          CLOSED.

          Show
          Oleg Egorov added a comment - - edited (6) Was : Total time: 0.746042 Total SQL time: 0.104005 SQL count: 73 (selects: 37 | executes: 36) Peak memory usage: 14.75M Memory limit: 1024M Now : Total time: 0.778044 Total SQL time: 0.096006 SQL count: 81 (selects: 40 | executes: 41) Peak memory usage: 78.5M Memory limit: 1024M DB o_22_test maps.php?sysmapid=22 Map problem still exists, and if no idea how to fix this problem, please open new ZBX Ivo Kurzemnieks Moved to ZBX-8232 . CLOSED.
          Hide
          Oleg Egorov added a comment - - edited

          System status widget

          Was:
          Speed: 0.93605399131775 s.
          Memory: 10.68 MB
          Max memory: 10.86 MB

          Now:
          Speed: 0.65403699874878 s.
          Memory: 8.37 MB
          Max memory: 8.56 MB

          Good performance improvement for middle and huge Zabbix installations and no performance changes for little installations

          Show
          Oleg Egorov added a comment - - edited System status widget Was: Speed: 0.93605399131775 s. Memory: 10.68 MB Max memory: 10.86 MB Now: Speed: 0.65403699874878 s. Memory: 8.37 MB Max memory: 8.56 MB Good performance improvement for middle and huge Zabbix installations and no performance changes for little installations
          Hide
          Oleg Egorov added a comment - - edited

          (7)

          		$eventAcknowledges = API::Event()->get(array(
          			'eventids' => $eventIds,
          			'select_acknowledges' => API_OUTPUT_EXTEND,
          			'preservekeys' => true
          		));
          

          'select_acknowledges' => API_OUTPUT_EXTEND

          But there used only 'clock', 'message', 'alias', 'name', 'surname'

          Ivo Kurzemnieks I also added eventid, since it's used as key
          RESOLVED in r45685

          Oleg Egorov CLOSED

          Show
          Oleg Egorov added a comment - - edited (7) $eventAcknowledges = API::Event()->get(array( 'eventids' => $eventIds, 'select_acknowledges' => API_OUTPUT_EXTEND, 'preservekeys' => true )); 'select_acknowledges' => API_OUTPUT_EXTEND But there used only 'clock', 'message', 'alias', 'name', 'surname' Ivo Kurzemnieks I also added eventid, since it's used as key RESOLVED in r45685 Oleg Egorov CLOSED
          Hide
          Alexander Vladishev added a comment - - edited

          (8) please review my changes in r45684.

          Oleg Egorov CLOSED

          Show
          Alexander Vladishev added a comment - - edited (8) please review my changes in r45684. Oleg Egorov CLOSED
          Hide
          Ivo Kurzemnieks added a comment - - edited

          (9) One more update in various frontend places. Now 'config' is passed as parameter. Before it was called in various places inside loops, causing PHP to slow down and eat more memory.

          RESOLVED in r45727.

          Oleg Egorov More issues

          • Minor coding style problem in tr_status.php: 88 unnecessary line break
          • More about config using improvement.
            In maps.inc.php function getSelementsInfo possible use more times select_config()
            This problem was related to map performance (ZBX-8232)
          • Unnecessary config usage in events.php
            In file begin
            require_once dirname(__FILE__).'/include/page_header.php';
            

          But after $config = select_config();
          Please check other places

          Ivo Kurzemnieks There are more places to get rid of second call of $config in various ways. I managed to remove usage in few views and call it only in controller or even page header once.

          RESOLVED in r45886

          Oleg Egorov

          Configuration->Maps->Edit

          Missing argument 2 for getSeverityCaption(), called in C:\xampp\htdocs\ZBX-7983\frontends\php\include\classes\class.cseverity.php on line 42 and defined [sysmaps.php:275 → CView->render() → include() → CSeverity->__construct() → getSeverityCaption() in C:\xampp\htdocs\ZBX-7983\frontends\php\include\triggers.inc.php:43]
          Undefined variable: config [sysmaps.php:275 → CView->render() → include() → CSeverity->__construct() → getSeverityCaption() in C:\xampp\htdocs\ZBX-7983\frontends\php\include\triggers.inc.php:45]
          Undefined variable: config [sysmaps.php:275 → CView->render() → include() → CSeverity->__construct() → getSeverityCaption() in C:\xampp\htdocs\ZBX-7983\frontends\php\include\triggers.inc.php:46]
          Undefined variable: config [sysmaps.php:275 → CView->render() → include() → CSeverity->__construct() → getSeverityCaption() in C:\xampp\htdocs\ZBX-7983\frontends\php\include\triggers.inc.php:47]
          Undefined variable: config [sysmaps.php:275 → CView->render() → include() → CSeverity->__construct() → getSeverityCaption() in C:\xampp\htdocs\ZBX-7983\frontends\php\include\triggers.inc.php:48]
          Undefined variable: config [sysmaps.php:275 → CView->render() → include() → CSeverity->__construct() → getSeverityCaption() in C:\xampp\htdocs\ZBX-7983\frontends\php\include\triggers.inc.php:49]
          Undefined variable: config [sysmaps.php:275 → CView->render() → include() → CSeverity->__construct() → getSeverityCaption() in C:\xampp\htdocs\ZBX-7983\frontends\php\include\triggers.inc.php:50]

          Same problem in trigger edit form

          And in trigger prototypes

          Undefined index: config [trigger_prototypes.php:323 → CView->render() → include() in C:\xampp\htdocs\ZBX-7983\frontends\php\include\views\configuration.triggers.list.php:253]

          REOPENED

          Ivo Kurzemnieks RESOLVED in r46110

          Ivo Kurzemnieks I updated to latest from /branches/2.2 there were many conflicts, please review r46112

          Oleg Egorov MOVED TO ZBX-8232

          Ivo Kurzemnieks After last discussion we decided to revert all changes about configuration now passed as parameter. It touched too many places infrontend, potentially breaking a lot of things and code didn't look that good. It is possible to optimize many places, but this needs a different and better solution. Code has been reverted and updated to latest /branches/2.2 please review once more r46198

          Oleg Egorov CLOSED

          Show
          Ivo Kurzemnieks added a comment - - edited (9) One more update in various frontend places. Now 'config' is passed as parameter. Before it was called in various places inside loops, causing PHP to slow down and eat more memory. RESOLVED in r45727. Oleg Egorov More issues Minor coding style problem in tr_status.php: 88 unnecessary line break More about config using improvement. In maps.inc.php function getSelementsInfo possible use more times select_config() This problem was related to map performance ( ZBX-8232 ) Unnecessary config usage in events.php In file begin require_once dirname(__FILE__).'/include/page_header.php'; But after $config = select_config(); Please check other places Ivo Kurzemnieks There are more places to get rid of second call of $config in various ways. I managed to remove usage in few views and call it only in controller or even page header once. RESOLVED in r45886 Oleg Egorov Configuration->Maps->Edit Missing argument 2 for getSeverityCaption(), called in C:\xampp\htdocs\ ZBX-7983 \frontends\php\include\classes\class.cseverity.php on line 42 and defined [sysmaps.php:275 → CView->render() → include() → CSeverity->__construct() → getSeverityCaption() in C:\xampp\htdocs\ZBX-7983\frontends\php\include\triggers.inc.php:43] Undefined variable: config [sysmaps.php:275 → CView->render() → include() → CSeverity->__construct() → getSeverityCaption() in C:\xampp\htdocs\ZBX-7983\frontends\php\include\triggers.inc.php:45] Undefined variable: config [sysmaps.php:275 → CView->render() → include() → CSeverity->__construct() → getSeverityCaption() in C:\xampp\htdocs\ZBX-7983\frontends\php\include\triggers.inc.php:46] Undefined variable: config [sysmaps.php:275 → CView->render() → include() → CSeverity->__construct() → getSeverityCaption() in C:\xampp\htdocs\ZBX-7983\frontends\php\include\triggers.inc.php:47] Undefined variable: config [sysmaps.php:275 → CView->render() → include() → CSeverity->__construct() → getSeverityCaption() in C:\xampp\htdocs\ZBX-7983\frontends\php\include\triggers.inc.php:48] Undefined variable: config [sysmaps.php:275 → CView->render() → include() → CSeverity->__construct() → getSeverityCaption() in C:\xampp\htdocs\ZBX-7983\frontends\php\include\triggers.inc.php:49] Undefined variable: config [sysmaps.php:275 → CView->render() → include() → CSeverity->__construct() → getSeverityCaption() in C:\xampp\htdocs\ZBX-7983\frontends\php\include\triggers.inc.php:50] Same problem in trigger edit form And in trigger prototypes Undefined index: config [trigger_prototypes.php:323 → CView->render() → include() in C:\xampp\htdocs\ZBX-7983\frontends\php\include\views\configuration.triggers.list.php:253] REOPENED Ivo Kurzemnieks RESOLVED in r46110 Ivo Kurzemnieks I updated to latest from /branches/2.2 there were many conflicts, please review r46112 Oleg Egorov MOVED TO ZBX-8232 Ivo Kurzemnieks After last discussion we decided to revert all changes about configuration now passed as parameter. It touched too many places infrontend, potentially breaking a lot of things and code didn't look that good. It is possible to optimize many places, but this needs a different and better solution. Code has been reverted and updated to latest /branches/2.2 please review once more r46198 Oleg Egorov CLOSED
          Hide
          Marc added a comment -

          Haven't read this ticket in detail but might it address/improve/solve ZBX-6557 as well?

          Show
          Marc added a comment - Haven't read this ticket in detail but might it address/improve/solve ZBX-6557 as well?
          Hide
          Ivo Kurzemnieks added a comment -

          Marc, no this will not affect fullscreen and normal screen behaviour.

          Show
          Ivo Kurzemnieks added a comment - Marc, no this will not affect fullscreen and normal screen behaviour.
          Hide
          Oleg Egorov added a comment -

          TESTED

          Show
          Oleg Egorov added a comment - TESTED
          Hide
          Ivo Kurzemnieks added a comment - - edited
          • Optimized last event calculation in "System status" widget improving speed and less memory usage.
          • For 2.2 removed duplicating node column on trigger pop-up.

          Fixed in pre-2.2.4rc1 r46225 and 2.3.2 (trunk) r46228, r46232

          Show
          Ivo Kurzemnieks added a comment - - edited Optimized last event calculation in "System status" widget improving speed and less memory usage. For 2.2 removed duplicating node column on trigger pop-up. Fixed in pre-2.2.4rc1 r46225 and 2.3.2 (trunk) r46228, r46232
          Hide
          Martins Valkovskis added a comment - - edited
          Show
          Martins Valkovskis added a comment - - edited Thanks for the summary, documented in: https://www.zabbix.com/documentation/2.2/manual/introduction/whatsnew224#miscellaneous_improvements Ivo Kurzemnieks Great! Thanks!
          Hide
          Oleksiy Zagorskyi added a comment -

          Looks like current changed caused a regression, see ZBX-8424.

          Show
          Oleksiy Zagorskyi added a comment - Looks like current changed caused a regression, see ZBX-8424 .

            People

            • Assignee:
              Ivo Kurzemnieks
              Reporter:
              Alexey Pustovalov
            • Votes:
              0 Vote for this issue
              Watchers:
              1 Start watching this issue

              Dates

              • Due:
                Created:
                Updated:
                Resolved: