[ZBX-5863] LLD and 64kB size limit Created: 2012 Nov 16 Updated: 2018 Feb 09 Resolved: 2013 Jan 24 |
|
Status: | Closed |
Project: | ZABBIX BUGS AND ISSUES |
Component/s: | Agent (G) |
Affects Version/s: | 2.0.3 |
Fix Version/s: | 2.0.5rc1, 2.1.0 |
Type: | Incident report | Priority: | Major |
Reporter: | Ildefonso Camargo | Assignee: | Unassigned |
Resolution: | Fixed | Votes: | 3 |
Labels: | discovery, json, lld | ||
Remaining Estimate: | Not Specified | ||
Time Spent: | Not Specified | ||
Original Estimate: | Not Specified | ||
Environment: |
OS: Debian GNU/Linux Squeeze, 64 bits |
Issue Links: |
|
Description |
Greetings, I created a simple script for discovering DB objects, and it have been working just fine on several servers, recently I installed it on a server that has many DB objects: the discovered DB objects never showed up on Zabbix UI, and I saw nothing on the server's logs (nor agent's logs either), and on the UI I was getting "Value should be a JSON object", so, I tried to run zabbix_get : zabbix_get -s SERVER_IP -k postgresql.tables.discovery and I saw the output was truncated, so, I decided to see if it had a particular size: Code: zabbix_get -s SERVER_IP -k postgresql.tables.discovery|wc -c The total size is 82065 (from discovery script output) So, I was wondering: is there a limit on the JSON answer size? I found something similar, but for proxy, on bug Thanks in advance! Ildefonso Camargo |
Comments |
Comment by Oleksii Zagorskyi [ 2012 Nov 17 ] |
Here is some info about limitations http://www.zabbix.com/documentation/2.0/manual/installation/upgrade_notes_203, probably not related. |
Comment by Ildefonso Camargo [ 2012 Nov 17 ] |
Oleksiy, thanks for answering! Yes, unfortunately, it is unrelated.... the limit I'm observing is just 64kB . I'll try to dive into the code today, maybe it is something simple to fix, but I'm not familiar with Zabbix's code, so, it can take some time for me. Any help is welcome. Ildefonso. |
Comment by Ildefonso Camargo [ 2012 Nov 22 ] |
sigh... I can't see a way of changing original issue description, this is not on Server, this is on Agent. On this thread, Palmertree gave a solution: http://www.zabbix.com/forum/showthread.php?p=122179 changing include/common.h #define MAX_BUFFER_LEN 65536 to #define MAX_BUFFER_LEN 262144 And recompile Zabbix Agent, fixed the problem. Please, remember this JSON object is coming from a UserParameter script. I'm not sure if changing this value globally is the answer, as it could affect other components so, please, someone with more knowledge on the source code, can comment if is it safe to change this constant globally? Thanks in advance, Ildefonso Camargo |
Comment by Oleksii Zagorskyi [ 2012 Nov 22 ] |
Component has been changed from server to agent, as requested. |
Comment by Marc [ 2012 Dec 04 ] |
Just encounter the same issue today. I've tried to gather a lot of statistics from an IBM SAN Volume Controller 2145-CG8 (~50 Items/Virtual Disk, ~500 Virtual Disks per SVC) dynamically via LLD. That worked very well until I connected my implementation to one of our real SVCs. That results in a JSON file of ~900KB for a single SVC. Would be nice to get MAX_BUFFER_LEN configurable. |
Comment by Łukasz Jernaś [ 2012 Dec 13 ] |
Maybe reading/writing to the buffer should block until the buffer is free again? This would also fix our issue with the F5 load balancers. |
Comment by Alexei Vladishev [ 2013 Jan 23 ] |
Possibly related to |
Comment by Andris Zeila [ 2013 Jan 24 ] |
Fixed in development branch svn://svn.zabbix.com/branches/dev/ZBX-5863 The UserParameter (zbx_execute) output buffer is now dynamically allocated and limited to 512KB. The built-in LLD discovery services already used dynamically allocated buffer to assemble the resulting JSON data. |
Comment by richlv [ 2013 Jan 27 ] |
(1) (for documentation) - would this result in memory usage reduction when lld responses are less than the previous size limit ? wiper It might reduce peak memory usage during script execution by 60KB. I don't think this is a significant reduction. <richlv> added a basic note at https://www.zabbix.com/documentation/2.0/manual/introduction/whatsnew205#daemon_improvements - please, review wiper reviewed, looks good. |
Comment by dimir [ 2013 Jan 29 ] |
Successfully tested. Please review my changes in r33234. |
Comment by Andris Zeila [ 2013 Jan 30 ] |
Fixed in: |
Comment by Jori Hämäläinen [ 2018 Feb 08 ] |
> The UserParameter (zbx_execute) output buffer is now dynamically allocated and limited to 512KB. But this does not solve this and like: I am also in a situation where LLD returns more that 512kB of JSON structure. And I really cannot splice it. What is your recommendation? I am running 3.2.10.. |
Comment by Viktors Tjarve [ 2018 Feb 09 ] |
If JSON is sent directly to (not through proxy) zabbix_server using zabbix_sender then there is no limit for JSON size. |
Comment by Jori Hämäläinen [ 2018 Feb 09 ] |
My application is using Zabbix Proxy to make LLD discovery to Zabbix-Agentd on server which produces a large JSON structure. And on zabbix-agent log from Userparameter I see 3623:20180206:205304.315 command output exceeded limit of 512 KB And LLD is the only UserParameter I have.. ./zabbix_autodiscovery.pl | wc So I've 2582 lines of JSON answer with 855035 bytes. Every line is relevant with multiple {#TAGS}per line which I need to setup to monitoring. |
Comment by Viktors Tjarve [ 2018 Feb 09 ] |
Sorry, this tracker is for bug reports only. |