[ZBX-10388] Zabbix 3.0 System status dashboard field loads for minutes, browser consumes CPU Created: 2016 Feb 16  Updated: 2017 May 30  Resolved: 2016 Oct 19

Status: Closed
Project: ZABBIX BUGS AND ISSUES
Component/s: Frontend (F)
Affects Version/s: 3.0.0
Fix Version/s: 3.0.6rc1, 3.2.2rc1, 3.4.0alpha1

Type: Incident report Priority: Major
Reporter: Dmitry Verkhoturov Assignee: Unassigned
Resolution: Fixed Votes: 2
Labels: dashboard, performance
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified
Environment:

[dverhoturov@mon-test ~]$ uname -a
Linux hostname 3.13.0-74-generic #118-Ubuntu SMP Thu Dec 17 22:52:10 UTC 2015 x86_64 x86_64 x86_64 GNU/Linux
[dverhoturov@hostname ~]$ cat /etc/release
DISTRIB_ID=Ubuntu
DISTRIB_RELEASE=14.04
DISTRIB_CODENAME=trusty
DISTRIB_DESCRIPTION="Ubuntu 14.04.3 LTS"
NAME="Ubuntu"
VERSION="14.04.3 LTS, Trusty Tahr"
ID=ubuntu
ID_LIKE=debian
PRETTY_NAME="Ubuntu 14.04.3 LTS"
VERSION_ID="14.04"


Attachments: Text File ZBX-10388.patch     PNG File ZBX-10388.patch.png     File cut_popup_description.3.0.4.patch     File cut_popup_description.patch     File syssum_widget_js_events.patch     PNG File syssum_widget_js_events.patch.png     File syssum_widget_json_data.patch     PNG File zabbix-2.4_system-status_load_firefox_network_load_timing.png     PNG File zabbix-3.0_system-status_load_firefox_network_load.png     PNG File zabbix-3.0_system-status_load_firefox_network_load_timing.png     File zabbix-3.0_system-status_load_on_chrome.mov     File zabbix-3.0_system-status_loading_profile_for_firefox.json    

 Description   

Dashboard with ~19k problem triggers draws for around 10 minutes on my laptop consuming 100% of one CPU core all the time.
My browser: firefox
Amount of groups: 109
Amount of hosts: few thousands



 Comments   
Comment by Alexander Vladishev [ 2016 Feb 16 ]

I close the issue as duplicate of ZBX-9407.

Comment by Dmitry Verkhoturov [ 2016 Feb 16 ]

I believe it's not related with SQL, it happens after server gives all the data to the browser and browser start drawing this table. On my colleague's desktop this widget loads ~5 minutes with 19k problem triggers and ~2 minutes with 2k problem triggers (after cleanup).
2.4.6 \ 2.4.7 not affected for sure, it happened only with our 3.0.0 test installation.
I beg you to spend a few minutes for detail investigation before closing it again.

Comment by Dmitry Verkhoturov [ 2016 Mar 18 ]

sasha, it's a blocker for us, could you please take a look?

Comment by Dmitry Verkhoturov [ 2016 Apr 12 ]

I can't reproduce problem on setup without history loaded (only configurations + alarms history), please stall this ticket for now — I'll be able to tell if problem is real or not for sure in ~1 month from now after performing additional tests. Sorry for not checking enough careful first time.

Comment by Dmitry Verkhoturov [ 2016 May 11 ]

I've reproduced the problem on production-load box.
What I can observe — both Zabbix 2.4.7 and 3.0.2 waits for data from SQL (6 and 16 seconds accordingly), then 2.4.7 transfers 87KB of data to browser for render (pure html) when 3.0.2 transfers around 21MB in exact same conditions (html + javascript somewhy). Feel free to contact me if you have any problems locating the problem.

gunarspujats Have you turned off debugging mode for your usergroup?

Paskal I've never turned it on.

Comment by Gunars Pujats (Inactive) [ 2016 Jun 09 ]

System status widget patch to reduce response data amount.

Comment by Gunars Pujats (Inactive) [ 2016 Jun 09 ]

System status widget patch for sending response as json data and creating table using JS template.

Comment by Gunars Pujats (Inactive) [ 2016 Jun 09 ]

(1) No translation strings changed.

oleg.egorov CLOSED

Comment by Gunars Pujats (Inactive) [ 2016 Jun 09 ]

RESOLVED in development branch svn://svn.zabbix.com/branches/dev/ZBX-10388 r60589

Tested sending response as JSON data and generating table with JS template on client side in r60585. Patch attached.
Reducing response data amount and generating table with PHP worked faster on test environment, therefore this solution is committed, patch attached.

Comment by Dmitry Verkhoturov [ 2016 Jun 09 ]

gunarspujats, I've took all frontend/php files from svn://svn.zabbix.com/branches/dev/ZBX-10388 to my 3.0.2 setup and I can't get system.status widget working — it loads endlessly. Is there any possibility of performance degradation introduced?

Paskal, syssum_widget_json_data.patch is updated, new file was missing.
Version in dev branch should work approximately the same time on the server side, optimization is in reducing response data amount, on test environment it reduced from 120 MB to 40 MB, therefore data transmission and displaying should be faster.

Paskal, try please cut_popup_description.patch, it removes macro resolving for trigger description in system status widget popup table. Should improve performance significantly.
You can use it, if this regression isn't critical.

Comment by Aleksandrs Saveljevs [ 2016 Jul 18 ]

Somewhat related issue: ZBX-10992.

Comment by Dmitry Verkhoturov [ 2016 Jul 21 ]

3.0.4rc1: 3Mb where I saw 21Mb before.
cut_popup_description.patch for 3.0.4rc1 (in attachment): same 3Mb, no change at all.
It's not perfect but much better than original, thank you, gunarspujats!

Comment by Dmitry Verkhoturov [ 2016 Jul 25 ]

I'm sorry, but why this patch didn't make it in 3.0.4? It's working. sasha?

Comment by Dmitry Verkhoturov [ 2016 Sep 20 ]

Guys, could someone please write an update, what is going on here?

oleg.egorov Development and testing

Paskal Please write me personally if you'll need test platform, I am DmitryV on #zabbix irc channel.

oleg.egorov Thank you, Dmitry, can I ask you to test syssum_widget_js_events.patch

Paskal Oleg, please see syssum_widget_js_events.patch.png — it made things worse, I couldn't wait for it to finish.

Paskal Oleg, I've enabled gzip compression for apache and it helped a lot — data still need some time to get fetched from DB, but delivered to browser in no time after that. I guess we could either give up on testing it or I could provide you a sandbox where you can see the problem yourself and find a way to fix it (if that's not clear yet).

Comment by Oleg Egorov (Inactive) [ 2016 Oct 04 ]

(2) As discussed with sasha and gunarspujats, should be used universal solution, because now fixed only system status widget.

gunarspujats RESOLVED in r63140

oleg.egorov Coding style

$id = 'prefix'.uniqid();

It's the same as

$id = uniqid('prefix');

And is better to use.

This is last use, after fix you can merge

gunarspujats RESOLVED in r63187

oleg.egorov CLOSED

Comment by Oleg Egorov (Inactive) [ 2016 Oct 17 ]

Hi, Paskal, can we ask You to test ZBX-10388.patch?

Comment by Dmitry Verkhoturov [ 2016 Oct 17 ]

system.view response size decreased by half (compressed), so if 50 seconds-long request from DB is going to be investigated\fixed in ZBX-9407, it seems what something expected is accomplished.

oleg.egorov Thank you for your help

Comment by Oleg Egorov (Inactive) [ 2016 Oct 18 ]

TESTED

Comment by Gunars Pujats (Inactive) [ 2016 Oct 18 ]

Fixed in:

  • pre-3.0.6rc1 r63247
  • pre-3.2.2rc1 r63248
  • pre-3.3.0 r63249
Generated at Fri Apr 19 21:30:42 EEST 2024 using Jira 9.12.4#9120004-sha1:625303b708afdb767e17cb2838290c41888e9ff0.