[ZBXNEXT-2013] Problems with "Latest data" after 2.2.0 upgrade Created: 2013 Nov 12  Updated: 2017 May 31  Resolved: 2013 Dec 17

Status: Closed
Project: ZABBIX FEATURE REQUESTS
Component/s: Frontend (F)
Affects Version/s: 2.2.0
Fix Version/s: None

Type: Change Request Priority: Critical
Reporter: Urs Weiss Assignee: Unassigned
Resolution: Duplicate Votes: 5
Labels: database, frontend, performance
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified
Environment:

CentOS 6.4
MySQL 5.5.34-32.0 (Percona Server)
Nginx 1.0.15 / php-fpm 5.3.3 with eAccelerator


Attachments: Text File query.txt     JPEG File zabbix_frontend_bug.jpg    
Issue Links:
Duplicate
duplicates ZBX-7373 Very slow SQL query for getting last ... Closed

 Description   

Forum thread: https://www.zabbix.com/forum/showthread.php?t=43096

After upgrading to 2.2.0 i had very big problems when trying to show values within the "latest data" section. At the end MySQL DB crashes sooner or later because it hasn't enough RAM. This worked fine all the time, never had any problems with older versions. Even increasing RAM doesn't help anything.

An example:
I'm opening Latest data section. Group "one" is open, it has four hosts with around 35-40 checks each. Fine. Selecting another host of the group sometimes takes around 10 seconds to loaded. Was much less than one second before. Loaded one time, it seems to be fast again. At least for some time.

OK, lets switch to another group. Group "two", three devices, 16 items each. Takes 8 seconds to load, less than one before the upgrade... Most often it doesn't even switches to the new group, but jumps back to the one which was selected before.

Next group. Now it's getting interesting... Cisco switches, three of them, around 2600 checks all together (yes, 2600...).
RAM of the MySQL Process jumps from ~9% to ~50% (1.8g), CPU at 100%. RAM keeps growing slowly. CPU comes down after some minutes. No values shown at all...

Switching to another group and back to the Cisco switches. Same behaviour as before, RAM jumps from 55% to 87% (3.3g), system starts to swap. If i do this one more time now, MySQL crashes and restarts...

Worked amazingly fine with 2.0.9 and 2GB RAM before, and now, with 2.2.0 it's not. Even if i double the RAM. Very very strange...

Any help would be appreciated.
Urs



 Comments   
Comment by richlv [ 2013 Nov 12 ]

might be related to ZBXNEXT-1689 and partially to ZBX-869

Comment by Urs Weiss [ 2013 Nov 12 ]

Btw. I'm using MySQL partitions. Biggest history_uint partition (daily) is 336MB, trends_uint (monthly) 196MB. Inserting values doesn't makes much load. Load is between 0.02 and 0.1 if no frontend is opened.

Comment by Alessandro [ 2013 Nov 15 ]

PostgreSQL 9.1, partitioned, biggest daily partition - 422MB, trends - 122MB. same problem - latest data is not loading.

Comment by Oleksii Zagorskyi [ 2013 Nov 15 ]

I'd recommend to close current issue as duplicate and continue in ZBX-7373.
Then need to move useful comments there.

Comment by Urs Weiss [ 2013 Nov 15 ]

As already written on the forum:

Don't think it's the same problem.

It's slow even if i only have 40 items. And, every load of the latest data pages fills up the memory more and more until MySQL crashes. So, i can't use Zabbix anymore at the moment because of this...

Comment by richlv [ 2013 Nov 15 ]

it fills up memory on zabbix db side ? you are not running the browser on the same system, right ?
which process eats up memory ?

Comment by Alessandro [ 2013 Nov 16 ]

in my case there is no problem with memory, but the query is extremely long - I have a DB, located at 24 15K drives in RAID10, but even after half an hour he does not finish. before the upgrade, the same operation takes seconds.

query is attached.

Comment by Urs Weiss [ 2013 Nov 18 ]

Yes, the memory fills up on DB side. I've updated MySQL because i thought it may is a MySQL bug first. But even the latest version behaves like before.

The server has no GUI, so nothing except Zabbix is running on this machine.

Comment by Sergey [ 2013 Nov 18 ]

My server is virtual host VirtualBox on OS CentOS 6.4/3Gb RAM/HDD 50Gb
I have the same problem after upgrading to 2.0.9 from 2.2.0 repository
#rpm -ivh http://repo.zabbix.com/zabbix/2.2/rhel/6/x86_64/zabbix-release-2.2-1.el6.noarch.rpm
#yum update

Tell me the problem is localized or what you need for data modeling?
When the patch will be released?

Comment by Stephen Dayman [ 2013 Nov 19 ]

Same issue happening for me. I have a fairly small instance which only does web scenarios and the Latest Data page will not load (even after several minutes of waiting). The database is a physical server running shared instance of Oracle. Zabbix_server and the UI are on a separate cluster. There were 0 issues with 2.0.9. Please let me know if there is any information I can provide that could help.

Comment by Sergey [ 2013 Nov 19 ]

I solved the problem
TRUNCATE TABLE history;
TRUNCATE TABLE history_str;
TRUNCATE TABLE history_uint;
TRUNCATE TABLE history_log;

Created not exist index
CREATE INDEX `history_uint_1` ON `history_uint` (`itemid`, `clock`);

Comment by Kenneth Palmertree [ 2013 Nov 19 ]

That is only a temp fix. You will run into response time issues, when your history tables get larger with more rows.

Comment by Sergey [ 2013 Nov 20 ]

No, the problem was the lack of an index.

I compared the data schema clean install and upgrade 2.0.9-> 2.2.0

Created not exist index - is solved problem
CREATE INDEX `history_uint_1` ON `history_uint` (`itemid`, `clock`);

Comment by Stephen Dayman [ 2013 Nov 21 ]

On upgrading from 2.0.9 -> 2.2.0 this index was never missing. It is already present. I've also tried to rebuild the index which had no effect.

Comment by Kenneth Palmertree [ 2013 Nov 21 ]

Stephen, if i had to guess it might be timing out on the php side because it is taking longer now to pull latest data.

See if you can find this setting in your php.ini file and see if it matches when your timeout occurs:

; Maximum execution time of each script, in seconds
; http://www.php.net/manual/en/info.configuration.php#ini.max-execution-time
max_execution_time = 600

Comment by Stephen Dayman [ 2013 Nov 22 ]

Thanks for the suggestion.
This is set to 300 on my server. I'll increase to allow for extra time but my users will not like waiting 5+ minutes to get the data they need to investigate a suspected incident. This page loaded close to instantly in 2.0.9

Comment by Urs Weiss [ 2013 Nov 22 ]

Thats definitely a problem of the frontend and how they changed the "latest data" page. An increase from much less than 1 to 10 and sometimes much more seconds... no comment.

Anyway don't know why they have changed it at all. It has worked great. Maybe there is a reason. Would like to know it if so.

Comment by Alessandro [ 2013 Nov 25 ]

api/classes/managers/CHistoryManager.php, line 42:

' WHERE h.itemid='.zbx_dbstr($item['itemid']).
>>
' WHERE h.clock >'.zbx_dbstr(time()-60*60*12). ' and h.itemid='.zbx_dbstr($item['itemid']).

worked only in case of partitioned DB, separated by "clock" field, like:

CREATE TABLE partitions.history_uint_p2013_08_28 (
  CONSTRAINT history_uint_p2013_08_28_clock_check CHECK ((clock >= 1377633600) AND (clock < 1377720000))
) INHERITS (public.history_uint)
WITH OIDS;
Comment by Stefan [ 2013 Dec 02 ]

I've running the Zabbix Appliance and seen that the storage io performance is going up to around 80 percent if i'm viewing the latest data. They need around 40 seconds to load the page.
The loading time can be decreased by changing the innodb_buffer_pool_size in /etc/my.cfg.
My appliance using 1GB of memory and i've set up innodb_buffer_pool_size to 512M. First time opening the last data need some time but second time loading the data around 2 seconds.

Comment by Ricardo Lopes [ 2013 Dec 10 ]

Hi guys.

I'm having the same problem and I need your support. Most of the times I open last data or a screen, or anything else, MySQL is jumping to 100% usage and IO 100% also.
This was not happening in 2.0.9.

My specs / environment:

  • CentOS 6
  • Zabbix 2.2 scratch install with MYSQL
  • Partitioning applied to MySQL

I saw some interesting comments like:

  • this happens because one index is missing ?!?
  • this index is not missing if you are coming from one upgrade 2.0.9? (not my case... I did a scratch)

This bug is killing me and killing the server.
Is there a workaround?

Since this sever is new I did the suggested and truncated the tables:
mysql> TRUNCATE TABLE history;
Query OK, 0 rows affected (0.26 sec)

mysql> TRUNCATE TABLE history_str;
Query OK, 0 rows affected (0.15 sec)

mysql> TRUNCATE TABLE history_uint;
Query OK, 0 rows affected (0.20 sec)

mysql> TRUNCATE TABLE history_log;
Query OK, 0 rows affected (0.23 sec)

mysql> CREATE INDEX `history_uint_1` ON `history_uint` (`itemid`, `clock`);
ERROR 1061 (42000): Duplicate key name 'history_uint_1'

How can I solve this bug ?
Please help.

Comment by Ricardo Lopes [ 2013 Dec 10 ]

I don't understand. Is this a workaround? How to implemente this in a 2.2 scratch install with a MySQL db?

Comment by Dimitri Bellini [ 2013 Dec 10 ]

Hi Ricardo,
I think you have discovered the same problem of mine, please check this issue ZBX-6296 and ZBX-7373.
I hope in Zabbix team for a fix or workaround

Comment by Ricardo Lopes [ 2013 Dec 10 ]

Hi Dimitri!
I saw the links but it seems that the problem is not fixed in the last release 2.2.1 rc1 correct?
Could you point me out the workaround / solution (or an improvement) if you have?

Many thanks!!!!

Comment by Ricardo Lopes [ 2013 Dec 10 ]

BTW Dimitri... I've updated /usr/share/zabbix/api/classes/managers/CHistoryManager.php
With the workaround..... is this the solution?
Seems to be working fine now

Comment by Dimitri Bellini [ 2013 Dec 10 ]

HI Ricardo,
My problem is related to some big host with a huge amount of items and slow browsing "latest data" frontend page,I have tested the Zabbix 2.2.1 and the TRUNK but that version don't fix the problem.
I have also try workaround on this page but seems not working with my DB partitioning schema.
Regards,

Comment by Ugis [ 2013 Dec 12 ]

This is how it looks after increasing memory_limit in php.ini. Before that blank page at all.

Comment by Oleksii Zagorskyi [ 2013 Dec 17 ]

I believe this issue is duplicate of ZBX-7373 which taken into account by developers.
I'm closing this issue to not have a lot of duplicated comments where we can mix (and lost) useful ones.

Who has useful comments here, please copy-pate to the ZBX-7373

After 2.2.2 release (it supposedly will be better than 2.2.1) it could be re-estimated and new issue (with clear description) could be created.

Generated at Fri Apr 26 23:43:12 EEST 2024 using Jira 9.12.4#9120004-sha1:625303b708afdb767e17cb2838290c41888e9ff0.