[ZBX-14396] Template full clone exhaust memory (large system) Created: 2018 May 24  Updated: 2024 Apr 10  Resolved: 2018 Nov 08

Status: Closed
Project: ZABBIX BUGS AND ISSUES
Component/s: API (A)
Affects Version/s: 3.4.8
Fix Version/s: 3.4.15rc1, 4.0.2rc1, 4.2.0alpha1, 4.2 (plan)

Type: Problem report Priority: Major
Reporter: Maris Danne Assignee: Gregory Chalenko
Resolution: Fixed Votes: 0
Labels: fullcloning, template
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Issue Links:
Causes
caused by ZBXNEXT-3006 A cache to provide multiple metrics o... Closed
Duplicate
Team: Team D
Sprint: Sprint 34, Sprint 35, Sprint 36, Sprint 37, Sprint 38, Sprint 39, Sprint 40, Sprint 41, Sprint 42, Sprint 43, Sprint 44, Sprint 45, Sprint 46, Nov 2018
Story Points: 2

 Description   

Problem is the same as ZBX-12933

It's impossible create template full clone (if there is LLD).

The problem is that it eats up too much memory more than 2GB.

log:

PHP Fatal error:  Allowed memory size of 2147483648 bytes exhausted (tried to allocate 77 bytes) in /usr/share/zabbix/include/classes/api/services/CDiscoveryRule.php on line 1137

Workaround:

Changed (in /zabbix/include/classes/api/services/CDiscoveryRule.php:1137):

 

foreach ($newPrototypes as $i => $newPrototype) {
   unset($newPrototypes[$i]['templateid']);
}

 

To:

 

reset($newPrototypes);

while(current($newPrototypes)){
    unset($newPrototypes[key($newPrototypes)]['templateid']);
    next($newPrototypes);
}

While loop eats less memory, then foreach.

 

With this solution we achieved that full clone does not eat up memory (and stop crashing), but the cloning process still is taking very long time (you may need to increase execution time).

For us this is problem to clone templates in large system.

 Comments   
Comment by Alexander Vladishev [ 2018 Jul 17 ]

Main problem was introduced in ZBXNEXT-3006. Prior to this fix, the copyItemPrototypes() method returned an array with 'itemids' key. After - a boolean value. So, API::ItemPrototype()->get() method now returns all item prototypes, instead of copied item prototypes, i.e. without filtering by 'itemids'.

Comment by Alexander Vladishev [ 2018 Oct 15 ]

Performance tests:

Environment:

  • standard Zabbix database
  • + 1000 hosts with linked "Template OS Linux"
  • host "Zabbix server" with unlinked (not "unlink and clear") "Template OS Linux" and "Template App Zabbix Server" templates

Operation: Full clone of "Zabbix server":
Before the fix:

Total time: 0.918612
Total SQL time: 0.530619
SQL count: 1426 (selects: 867 | executes: 559)
Peak memory usage: 124M

After the fix:

Total time: 0.570373
Total SQL time: 0.460414
SQL count: 1426 (selects: 867 | executes: 559)
Peak memory usage: 22M
Comment by Gregory Chalenko [ 2018 Nov 08 ]

Fixed in:

  • 3.4.15rc1 r86112
  • 4.0.2rc1 r86618
  • 4.2.0alpha1 (trunk) r86619
Comment by richlv [ 2018 Nov 25 ]

(1) The changelog entry is "fixed memory usage during template full clone" - perhaps a more accurate version could be "fixed excessive memory usage during template full clone"

gcalenko Thank you. Changelog message fixed in r87329.

Generated at Wed Apr 24 18:44:45 EEST 2024 using Jira 9.12.4#9120004-sha1:625303b708afdb767e17cb2838290c41888e9ff0.