|
Because the item key openvz.ubc is already provided by the BulkUserParameter definition, there is of course no need to include the item key part in the Bulk User parameter's output:
# sudo tac /proc/user_beancounters | awk '$1=="uid"{for(i=4;i>=0;i--)h[i]=$(7-i);next}$6~/./{r=$(NF-5);for(i=0;i<5;i++)printf("[%s,%s] %d\n",r,h[i],$(NF-i))}'
[kmemsize,failcnt] 0
[kmemsize,limit] 1217787766
[kmemsize,barrier] 1217604266
--- SNIP --- SNAP --- SNIP --- SNAP ---
On the other hand could it be desired to provide different item keys by one Bulk parameter. So it could also be kept in the output but left out int the definition:
BulkUserParameter=1,sudo tac /proc/user_beancounters | awk ...
|
|
This is similar to what I suggested in this comment in issue ZBXNEXT-103, but you forgot to how will these items be defined in the Web UI?
I thought about it from the last time I replied to the issue above and here is an idea how I would implement this. I wouldn't even create a special zabbix_agent.conf parameter like BulkUserParameter, but just use the existing one UserParameter and only add a new type of item in the UI called Zabbix Agent Bulk. This item type would define the user parameter which will be called on the agent to return multiple items. This item type would have everything as regular items, but in it you would need to define item keys which will be returned. The only thing I'm not sure is if this bulk item should also return a value for itself, if it fails to execute, or something.
|
|
kustodian,
it may appear similar but it's different. This request is neither about providing additional context information alongside an item's value (status, criticity, etc.) nor is it about returning separate values or key=value for a single item. A Zabbix item is atomic and that's a very good design decision in my opinion - performance wise and in terms of processability too.
The motivation of this request is to significant improve performance and efficiency in cases where lots of valuable information can be provided by a single custom call.
From my experience when extending Zabbix one has rarely complete distinct custom items. In Most cases one accesses a single interface that provides plenty of valuable information one likely wants to obtain by dedicated items:
Database metrics -> pgsql, sqlite3, db_stat, sqlplus, mysql,...
Hardware controller metrics -> MegaCli64, ...
BIND name server -> rndc
NTP time server -> ntpq
Device mapper -> multipath, multipathd
...
To name only some of them.
All of these commands may provide lots of information for separate Zabbix items in one run. Instead of calling them again and again for each custom item, this ticket requests a way to call them only once and provide their output in a cache for potential Item calls in the future. In fact the proper formatted output gets parsed to fill the cache.
These item calls based on dynamic supported item keys (keys from BulkParameterCache) do not need to issue any fork or command, access a file or do any other kind of processing.
They just have to "echo" what is currently in memory for a specific key.
Because this request does not affect any design principle of Zabbix, there is also no need to touch the Web frontend nor the way things get handled. It's a change at the very beginning of the process chain.
|
|
I'm not sure I exactly understand how you suggestion works, but let me try to understand it. You would basically create an item in the UI which will call the openvz.ubc BulkUserParameter and that item will update the cache for all those items on the agent. Then you would create items in the UI which will have item keys like those which the BulkUserParameter updates in the agent cache, so if the server requests values for those items, the agent would send the data from the cache, without running any queries.
If that is the case, you are right, nothing would need to be changed the UI, but it would still be a little complicated to answer how are those items being updated, unless you know how the script works. Also what will the openvz.ubc item return to the server, since it has to return something, it cannot only update cache by echo-ing items/values?
|
|
Well, I've got several scenarios for a possible implementation in mind.
This is the one I currently favor for Bulk user parameter :
--- SNIP zabbix_agentd.conf ---
# BulkUserParameter=<key>,<interval>,<command>
BulkUserParameter=openvz.ubc,1,sudo tac /proc/user_beancounters | awk ...
--- SNAP zabbix_agentd.conf ---
Configuration of BulkUserParameter is considered on Zabbix agent (re)start only. The configured <key> becomes a supported item key.
The Zabbix agent's collector process executes the deposited <command> every <interval> minutes, parses its output and updates the corresponding cache entries.
The <command> its output could look like this:
[kmemsize,failcnt] 0
[kmemsize,limit] 1217787766
[kmemsize,barrier] 1217604266
--- SNIP --- SNAP --- SNIP --- SNAP ---
The corresponding cache representation could then look like this:
openvz.ubc[kmemsize,failcnt] 0
openvz.ubc[kmemsize,limit] 1217787766
openvz.ubc[kmemsize,barrier] 1217604266
--- SNIP --- SNAP --- SNIP --- SNAP ---
When the Zabbix agent gets a request for the item key openvz.ubc[kmemsize,limit], then it knows that openvz.ubc[*] is a Bulk user parameter, searches in the Bulk Parameter Cache for the related cache entry and returns the corresponding value. In this case 1217787766.
By this the actual custom command (Bulk User parameter) gets automatically executed once per minute and keeps current values for 100 different Items in memory (in this example).
Which item values finally get send/returned to the Zabbix server/proxy in which interval, is decided on Zabbix frontend, resp. by Zabbix server in dependency which items had been created. Another benefit is that one can decide/change on Zabbix frontend the interval for each such item.
The latter is by the way one of the reasons why I don't like to use zabbix_sender instead. When bulk sending item values by Zabbix sender, then all metrics get send to Zabbix server/proxy, regardless if they are configured or not. Further the update intervals are not individually changeable in Zabbix frontend but are the same to all items.
|
|
Similar idea but makes use of memcached:
https://github.com/crackmonkey/zbxcache
Edit:
Yet another example but makes use of a cache file:
https://bitbucket.org/sivann/runcached/
Special thanks to steki for dropping a comment that made me find that 
|
|
Hello everybody,
I think this issue is very important and concerns not only UserParameters, but all item types (simple checks, external checks, ssh checks, log monitoring, database monitoring etc). However, for SNMP checks it is implemented in BulkSNMP.
There are a lot of workarounds: temp files, per command or value cache etc. But they are all should be done manually and kept in mind by Zabbix users with a restriction of different item update intervals.
My suggestion is to split definitions: ITEMS and CHECKS. Items is a monitoring parameter, check is a collection method. Multiple items can be assigned to one check by string id or by order number in the check output with delimiters (depending on the check type).
I think this method is suitable for backward compatibility and less painful upgrade, because in current version there is one check for one item and we can easily convert all templates.
As for LLD checks it can also be done the same way. Every check can return one line (for usual checks) or several lines (for discovery rules and item prototypes). So one check for the whole LLD. For backward compatibility there can be additional check option - format: JSON or raw (with parsing rules).
Check output can be parsed by agent according to check settings and transferred in the "value" JSON sub-structure - no need to change protocol. Only additional check fields (such as multiline indicator, format and parsing rules) can be added to the server request (or response for active checks).
The only pitfall is item keys. They should be assigned to checks (not items), but what to do with trigger expressions... I see two ways:
1) during upgrade convert item keys to item names in trigger expressions and refuse item keys at all,
2) copy items keys to checks keys and make item keys to be in free format and used only in triggers.
In conclusion.
IMHO this architectural improvement must be done ASAP, because as later as more difficult.
|
|
Related issue: ZBXNEXT-1443
|
|
dimir,
Sorry, but no, it is not related to ZBXNEXT-1443. It is absolutely different issues.
|
|
I would disagree. I understand that this issue is bit different but to me the idea is the same: one check results in multiple values for other items. Item pre-processing is the first step to that, so to me these issues are related.
|
|
Please take care of server side testing - wiper
|
|
Server side is ready for testing in development branch svn://svn.zabbix.com/branches/dev/ZBXNEXT-3006
|
|
(3) [I] Monitoring of the new processes must be added to the default Zabbix templates (zabbix[process,<process>,avg,busy] items with corresponding triggers.
vjaceslavs Added monitoring to templates. RESOLVED in r69199
wiper CLOSED
|
|
(4) [S] preprocessor_link_delta_items can be called with unitialized item configuration if item is not "preprocessable".
vjaceslavs RESOLVED in r69198, r69241
wiper CLOSED
|
|
(5) [S] Naming improvements:
- using zbx_preprocessor_hold() (begin() ?), zbx_preprocessor_flush() instead of zbx_preprocessor_send_command() would easier to read.
- would better to avoid using generic file names (worker, manager) where possible. Use preproc_worker, preproc_manager or something like that
- all structures must be prefixed with zbx_, so that means using zbx_queue_t, zbx_queue_item_t, zbx_queue_iterator_t. Actually maybe it would be better to rename it to list (zbx_list_t, ...). We already have circural queue zbx_queue_ptr_t and the new queue is basically a singly linked list.
vjaceslavs RESOLVED in r69241
wiper CLOSED
|
|
(6) [S] item.dependent_items vector update is done by clearing and rebuilding the dependent item vector for all items.
This will not work with the configuration sync changes in trunk. Only the changed rows will be passed to DCsync_items(). There are few options:
- when adding/removing dependent item (or item's type changed to/from dependent) update its' master items dependent_items vector. This would probably require to generate itemid->master_itemid pair vector from configuration cache and synchronize it with itemid->master_itemid pairs selected from database (select itemid,master_itemid from items where master_itemid is not null).
- after syncing items (and before updating dependent items) iterate through config->items to reset dependent items.
The (1) is more optimal for partial configuration cache updates, but (2) is much easier to implement.
vjaceslavs RESOLVED in r69628
wiper CLOSED
|
|
(7) [S] Additional check was missing for LLD items that are not discovered anymore (items without lld_row should not be linked with parent as they will not be updated).
RESOLVED in r69272, r69273
wiper CLOSED
|
|
(10) [S] More naming improvements:
Preprocessor manager internal type names also must be prefixed with zbx_ (preprocessing_states, preprocessing_request_t, preprocessing_worker_t, delta_item_index_t, preprocessing_manager_t)
A 'dep' abbreviation is used when processing trigger dependencies. Vectors/arrays storing identifiers usually have ids suffix. And commonly the number of something has suffix _num. So dep_itemids in ZBX_DC_ITEM structure and dep_itemids_num, dep_itemids in DC_ITEM structure would be better match to existing names.
vjaceslavs RESOLVED in r69310
wiper CLOSED
|
|
(11) [S] Fixed dependent item copying when linking templates. Also minor code improvement, please review.
RESOLVED in r69343, r69349
vjaceslavs Thanks! CLOSED
|
|
(12) [S] Fixed possible memory leaks/corruption when setting the calculated value/error to agent result. Also cleaned up uninitalized memory errors reported by valgrind (might have been false positive though). Please review.
RESOLVED in r69755, r69757
vjaceslavs Thanks! Reviewed and CLOSED
|
|
(13) [S] Value type is used for type hinting during string to numeric conversions in trunk. While currently value type is not used by preprocessor it will be easier to merge if it was added to the manager->worker request package.
vjaceslavs RESOLVED in r69852 as a part or merge
wiper CLOSED
|
|
(14) [S] Another 'todo' when merging trunk - new preprocessing options were added to trunk. The location of item_preproc.c file was changed - you will probably need to merge the changes manually.
vjaceslavs RESOLVED in r69852 as a part or merge
wiper CLOSED
|
|
(15) [F] The dependent item graphs consists of dots without lines. I understand this is because delay period for dependent items is 0. For dependent items we will have to take the master's delay period (in worst case scenario we will need to travel dependent item chain to the 'root' item).
Moved to frontend task ZBXNEXT-3883 as (5).
CLOSED
|
|
Server side is tested, however due to the differences with trunk some things will have to be retested after merge.
|
|
(16) [S] Fixed few merge issues, please review r69865
vjaceslavs Thanks! Reviewed and added some minor fixes in r69869. RESOLVED
wiper CLOSED
|
|
(17) [S] In DCsync_items() the ids vector is used only to update dependent items in master items. My suggestion would be to use ptr vector where only new dependent items or dependent items with changed master items would be added.
vjaceslavs RESOLVED in r69868
wiper CLOSED
|
|
Server side (trunk merge) tested.
Frontend is under ZBXNEXT-3883
|
|
Hi Andris,
As i could see using SVN Trunk there is a new Item called "Dependent item", how can i test it? Is this feature planned for 3.4?
Thanks so much
|
|
Dimitri, it's marked for 3.4 in Fix Version/s (you should be able to see it above). However It doesn't seem to be merged in trunk yet.
|
|
Hi Andris,
Thanks, i have retested the feature but i did not understand how is working
|
|
Shortly - it allows to create items with another item as data source and using preprocessing options to extract required value. I'd suggest to wait for documentation.
|
|
I understood the concept but as you suggest i will wait documentation, thanks very much!
|
|
(23) [S] As preprocessing is done before values are stored into history cache the ZBX_FLAG_ITEM_FIELDS_PREPROC flag can be removed when getting items from configuration cache in dbcache.c:DCmass_update_items() function.
vjaceslavs Changed to ZBX_FLAG_ITEM_FIELDS_DEFAULT. RESOLVED in r70675
wiper CLOSED
|
|
(24) [S] At some point in the trunk we added step number to the preprocessing step failure message, something like Item preprocessing step #1 failed: <message>.
With preprocessing relocation to preprocessing manager and multiple merges this message seems to be lost. Now it simply says 'cannot apply multiplier "2" to value "x" of type "string": cannot convert value to numeric type'.
vjaceslavs RESOLVED in r71044
wiper CLOSED
|
|
(27) [I] Proxy does not compile
./configure --enable-proxy --with-mysql
make[3]: *** No rule to make target '../../src/zabbix_server/preprocessor/libpreprocessor.a', needed by 'zabbix_proxy'. Stop.
make[2]: *** [Makefile:673: install-recursive] Error 1
make[1]: *** [Makefile:471: install-recursive] Error 1
make: *** [Makefile:477: install-recursive] Error 1
wiper RESOLVED in r70796
vso CLOSED
|
|
Gents, please, please with sugar on top, re-consider sharing specs with the community from the very beginning.
This is just yet another example of missing the point, which could have been easily identified, if acting more open and transparent.
While the current implementation is possibly a nice feature, it has less to do with the demand behind ZBXNEXT-3006. Since the issue description/comments were obviously not clear enough, you can also take a look at the Eichhoernchen project to get a better idea of it.
The Eichhoernchen project does not yet provide a solution, but it properly addresses the origin demand.
Anyway, I appreciate your commitment!
|
|
As I understand the Eichhoernchen project it implements workaround for:
And this is a pity that with current Zabbix data flow architecture only one value of only one metric can be returned at a time.
This feature besides moving item preprocessing to separate processes, itroduces dependent items, allowing to use an item as data source for other items, so basically returning multiple metrics from an item.
|
|
As I understood the current implementation, a master item updates all dependent items at the time of the master items update.
If we keep the terminology, then this feature request was actually about a master item that may update dependent items with individual update intervals.
In the origin request the master item update interval would give the minimum usable update interval of dependent items. A smaller update interval of dependent items would be possible, but would of course not lead to a greater detail. But a larger update interval allows to lead to less item values and easier processing (rendering graphs, 3rd party API use cases, etc.)
This can make a huge difference when indeed going up to 999 items for thousands of hosts.
The Eichhoernchen project instead was even attempting to make this smart. Smart in the sense that the master data gets updated in dependency of dependent items. that's to say, update the master data only, if the the data does not exist yet, or a dependent items is about to be updated, where the item's update interval is smaller than the age of the master data.
The current implementation is rather similar to repetitive sending data for multiple items in a bulk by Zabbix sender, than to a cache that allows individual items, incl. individual item update intervals.
While I've to admit that the issue description didn't pointed out that individual update interval fact explicitly (to me it was an obvious fact of using a cache as described), it has been at least mentioned in the comments.
Having been able to follow the spec from the very beginning, could have helped to clarify that this is not addressing the actual need, resp. is a yet cool but different ZBXNEXT.
|
|
I think we can improve to make everybody happy, see ZBXNEXT-4016.
|
|
Hi Glebs,
I came back I supposed, from what said by Marc, that the main problem is clarification of what kind of implementation is on developing.
From what i read this kind of "feature" seem very good for example on API request, receive a big file with a lot of metrics (using the "Master Item"), extract what is need (i do not know how) and send to "child items". From my point of you seems to solve most of my needs but i did not know for other cases.
So the best thing is to provide (please, please, please) a better details of the implementations.
Thank very much
|
|
Released in:
|
|
+1
|
|
+1
|
|
Please also cast a vote!
|
|
seriously guys, "+1" comments are not helpful. they actually only clutter the issues, result in useless notifications and i'm considering some mass cleanup (read - deletion) of such comments :/
issue voting feature is in the upper right corner. it's also described in bug reporting guidelines at https://zabbix.org/wiki/Docs/bug_reporting_guidelines#Reporting_an_issue
|
|
similar to ZBX-6971
|
|
Hi guys, i find workaround (bad but works)
Make sleep 3600 on error, so you will see timeouts on dashboard & etc
|
|
How is this still a problem with modern versions of Zabbix?
It can't be that hard to report error on exit code != 0.
|
|
@Timofey, thanks for the 'workaround' btw. Setting 'sleep 120' before every exit at least makes Zabbix see the message as failed.
|
|
Can someone update the affected versions, update the priority and get this feature request assigned?
With 14 votes and 3.2.1 out it's about time to get this issue out of the way. Thanks
|
|
I would also argue that if a non-zero exit code is received, any stderr script output should be shown as the error message (perhaps truncated) in the frontend.
|
|
stefan.radman, if you have a support agreement with the zabbix company, you might want to raise this topic through your contact person there.
14 votes puts this feature request somewhere at place 120 of the highest voted requests
|
|
got your point. We have a support agreement and I might raise it there. Thanks
|
|
The timeout for alerter is hardcoded to 40 seconds (as I recall), so sleeping alert scripts on failure could cause alerter processed to be too busy (I saw that a few times on different installations because of simply slow alert script).
So that workaround is risky.
|
|
I raised an issue through support and it got assigned.
|
|
It would be also nice if SDTERR of alert script would be stored as error message (in alerts), similarly as we log errors for "Email" media type.
I use the this dummy script as alert media (and remmote command, below):
# cat /etc/zabbix/alertscripts/test.pl
#!/usr/bin/perl
#use warnings;
use strict;
print "done\n";
sdfg sf
And for consistency .... remote command (performed by escalator, not alerter) should be taken into account too, because they even do not log the STDERR to server log, which is not very nice.
So, to be consistent, wold be good to check exit code for remote command too, print STDERR to the log and store in alerts.
20689:20161113:162306.866 Conditions match our event. Execute operation.
20689:20161113:162306.866 In execute_commands()
20689:20161113:162306.866 query [txnlev:1] [select distinct h.hostid,h.host,o.type,o.scriptid,o.execute_on,o.port,o.authtype,o.username,o.password,o.publickey,o.privatekey,o.command,h.tls_connect,h.ipmi_authtype,h.ipmi_privilege,h.ipmi_username,h.ipmi_password from opcommand o,opcommand_grp og,hosts_groups hg,hosts h where o.operationid=og.operationid and og.groupid=hg.groupid and hg.hostid=h.hostid and o.operationid=17 and h.status=0 union select distinct h.hostid,h.host,o.type,o.scriptid,o.execute_on,o.port,o.authtype,o.username,o.password,o.publickey,o.privatekey,o.command,h.tls_connect,h.ipmi_authtype,h.ipmi_privilege,h.ipmi_username,h.ipmi_password from opcommand o,opcommand_hst oh,hosts h where o.operationid=oh.operationid and oh.hostid=h.hostid and o.operationid=17 and h.status=0 union select distinct 0,null,o.type,o.scriptid,o.execute_on,o.port,o.authtype,o.username,o.password,o.publickey,o.privatekey,o.command,1,0,2,null,null from opcommand o,opcommand_hst oh where o.operationid=oh.operationid and o.operationid=17 and oh.hostid is null]
20689:20161113:162306.868 In substitute_simple_macros() data:'/etc/zabbix/alertscripts/test.pl'
20689:20161113:162306.868 In zbx_execute_script()
20689:20161113:162306.868 In zbx_popen() command:'/etc/zabbix/alertscripts/test.pl'
20689:20161113:162306.870 End of zbx_popen():9
20827:20161113:162306.870 zbx_popen(): executing script
20689:20161113:162306.878 In zbx_waitpid()
20689:20161113:162306.878 zbx_waitpid() exited, status:255
20689:20161113:162306.878 End of zbx_waitpid():20827
20689:20161113:162306.878 End of zbx_execute_script():SUCCEED
20689:20161113:162306.878 In add_command_alert()
20689:20161113:162306.878 End of add_command_alert()
20689:20161113:162306.878 In DCget_nextid() table:'alerts' num:1
20689:20161113:162306.878 End of DCget_nextid() table:'alerts' [128:128]
20689:20161113:162306.878 query [txnlev:1] [insert into alerts (alertid,actionid,eventid,clock,message,status,error,esc_step,alerttype) values (128,3,1498,1479046986,':/etc/zabbix/alertscripts/test.pl',1,'',1,1);]
20689:20161113:162306.879 End of execute_commands()
...
20689:20161113:162306.879 Conditions match our event. Execute operation.
20689:20161113:162306.879 In execute_commands()
20689:20161113:162306.879 query [txnlev:1] [select distinct h.hostid,h.host,o.type,o.scriptid,o.execute_on,o.port,o.authtype,o.username,o.password,o.publickey,o.privatekey,o.command,h.tls_connect,h.ipmi_authtype,h.ipmi_privilege,h.ipmi_username,h.ipmi_password from opcommand o,opcommand_grp og,hosts_groups hg,hosts h where o.operationid=og.operationid and og.groupid=hg.groupid and hg.hostid=h.hostid and o.operationid=18 and h.status=0 union select distinct h.hostid,h.host,o.type,o.scriptid,o.execute_on,o.port,o.authtype,o.username,o.password,o.publickey,o.privatekey,o.command,h.tls_connect,h.ipmi_authtype,h.ipmi_privilege,h.ipmi_username,h.ipmi_password from opcommand o,opcommand_hst oh,hosts h where o.operationid=oh.operationid and oh.hostid=h.hostid and o.operationid=18 and h.status=0 union select distinct 0,null,o.type,o.scriptid,o.execute_on,o.port,o.authtype,o.username,o.password,o.publickey,o.privatekey,o.command,1,0,2,null,null from opcommand o,opcommand_hst oh where o.operationid=oh.operationid and o.operationid=18 and oh.hostid is null]
20689:20161113:162306.881 In substitute_simple_macros() data:'someeeeee'
20689:20161113:162306.881 In zbx_execute_script()
20689:20161113:162306.882 In zbx_popen() command:'someeeeee'
20689:20161113:162306.883 End of zbx_popen():9
20829:20161113:162306.883 zbx_popen(): executing script
20689:20161113:162306.885 In zbx_waitpid()
20689:20161113:162306.885 zbx_waitpid() exited, status:127
20689:20161113:162306.885 End of zbx_waitpid():20829
20689:20161113:162306.885 End of zbx_execute_script():SUCCEED
20689:20161113:162306.885 In add_command_alert()
20689:20161113:162306.885 End of add_command_alert()
20689:20161113:162306.885 In DCget_nextid() table:'alerts' num:1
20689:20161113:162306.885 End of DCget_nextid() table:'alerts' [129:129]
20689:20161113:162306.885 query [txnlev:1] [insert into alerts (alertid,actionid,eventid,clock,message,status,error,esc_step,alerttype) values (129,3,1498,1479046986,':someeeeee',1,'',1,1);]
20689:20161113:162306.886 End of execute_commands()
How does it look like currently in 3.0.5 frontend, falsely saying they were ok:
|
|
remote commands might be much harder - server does not wait for the command to be executed. it just sends the command to the agent and gets back '1' (see system.run[nowait,<command>])
|
|
Rich, for commands executed on agent side - that's correct and you are right. I don't consider such case here.
In previous comment I meant "remote commands" executed on zabbix server, as shown in the debug log above.
Doc https://www.zabbix.com/documentation/3.0/manual/config/notifications/action/operation/remote_command says:
On Zabbix server remote commands are executed with timeout as set in the TrapperTimeout parameter of zabbix_server.conf file
|
|
I hope it will be implemented in Zabbix 3.4, it's in the roadmap.
|
|
Can you please publish this roadmap, maybe here?
http://zabbix.org/mw/index.php?title=Docs/roadmap
|
|
Thinking about, for example, an alert script, used to sent SMSs through a sms-gateway in the Internet, where connections to the gateway randomly could be failed before timeout ...
Just a question - Zabbix uses ALERT_MAX_RETRIES=3 to send alerts.
Does it make any sense to do the same thing for custom alert scripts when it exits with error code (currently it applies only for timeout of script execution) ?
Also, thinking about the 40 seconds hardcoded timeout to be configurable ...
|
|
No, I don't think it makes sense to retry when the script returns an error code, with one possible exception being exit code 75:
$ grep -A3 TEMPFAIL /usr/include/sysexits.h
* EX_TEMPFAIL -- temporary failure, indicating something that
* is not really an error. In sendmail, this means
* that a mailer (e.g.) could not create a connection,
* and the request should be reattempted later.
--
#define EX_TEMPFAIL 75
#define EX_PROTOCOL 76
#define EX_NOPERM 77
#define EX_CONFIG 78
See also: https://tools.ietf.org/html/rfc3463#section-2
Maybe also code 11:
$ grep EAGAIN /usr/include/asm-generic/errno-base.h
#define EAGAIN 11
Configurable timeout (and possibly retries) would definitely be an asset.
|
|
Ready for testing in development branch svn://svn.zabbix.com/branches/dev/ZBXNEXT-1380
|
|
(1) [PSA] Memory leak.
output is only freed on SUCCEED, otherwise address is lost and memory is not freed.
if (SUCCEED == (rc = zbx_execute_script(&host, &script, NULL, &output, error, sizeof(error))))
{
zabbix_log(LOG_LEVEL_DEBUG, "%s output:\n%s", script.command, error);
zbx_free(output);
}
While it is also allocated on not SUCCEED.
if (SUCCEED != ret && NULL != result)
*result = zbx_strdup(*result, "");
vjaceslavs RESOLVED in r64863
vso CLOSED, please review small style fixes in r64868
|
|
(2) [PSA] Unused variable added.
output variable is allocated and freed, however it is not used.
if (SUCCEED == (rc = zbx_execute_script(&host, &script, NULL, &output, error, sizeof(error))))
{
zabbix_log(LOG_LEVEL_DEBUG, "%s output:\n%s", script.command, error);
zbx_free(output);
}
vjaceslavs RESOLVED in r64863
vso CLOSED
|
|
(3) Log is not formatted to zabbix style and misleading.
It says printing output while output variable is not printed, instead it prints error variable, and furthermore log is only executed on SUCCEED so there should be no error to print.
It will not be clear that command is printed when reading log, there is just command without description, also there are no apostrophe as in other places so not clear where it ends.
There are new line in the middle of log, there are no such logs in zabbix and this breaks consistency.
zabbix_log(LOG_LEVEL_DEBUG, "%s output:\n%s", script.command, error);
Example logs:
zabbix_log(LOG_LEVEL_DEBUG, "In %s() host:'%s' hostid:%d", __function_name, host, hostid);
zabbix_log(LOG_LEVEL_DEBUG, "get value from agent result: '%s'", s.buffer);
vjaceslavs There was an error in code (2) - should be "output" (not an "error"). And log "style" was taken from alerter.c (126) to make consistent output so can't agree with "there are no such logs in zabbix". Should I change existing log style in alerter as well?
vso CLOSED, (2) fixes it, it's fine for long data.
|
|
(4) When exec function return error, frontend still display success.
The exec() functions return only if an error has occurred, but we set success on error.
Please see below, the log say "execl() failed but EXIT_SUCCESS is returned."
Same is with setpgid in zbx_popen function.
zabbix_log(LOG_LEVEL_DEBUG, "%s(): executing script", __function_name);
execl("/bin/sh", "sh", "-c", command, NULL);
zabbix_log(LOG_LEVEL_WARNING, "execl() failed for [%s]: %s", command, zbx_strerror(errno));
exit(EXIT_SUCCESS);
vjaceslavs RESOLVED in r64870.
vso CLOSED, now process exited with code: 1 is displayed.
|
|
(5) If child did not terminate normally, still SUCCESS is returned.
The code below means if the child terminated normally and exit status is not success then fail.
else SUCCESS
But what if child did not terminate normally ? It shall also be failure.
else if (WIFEXITED(status) && EXIT_SUCCESS != WEXITSTATUS(status))
{
...
}
else
ret = SUCCEED;
vjaceslavs There are multiple options when this kind of "false positive" result can occur:
1. Process was terminated by a signal (no exit code to check).
2. Internal error (GetExitCodeProcess fails, etc.) where is no way to get status code.
So the question is: should we consider absence of exit code as an error code != 0. If we do so, then we should consider if we need to retry execution in this case as it can cause unpredictable results.
I suggest to add extra status code "UNKNOWN" for situations where script execution status is unknown (so we don't have "Done" in frontend).
vjaceslavs Even more, on unix-like systems, command is being executed from shell and even if signal occurs it is handled by shell and exit code is set to non-zero so this is a really specific case when shell crashes.
vjaceslavs It was decided to consider signaled state as a failure without altering the retry logic. RESOLVED in r65091
vso CLOSED
|
|
(6) Script on zabbix_agent gets killed before it finish execution, if it executes longer than "Timeout" which is 3 by default and max 30 seconds.
Previously:
Remote commands on Zabbix agent are executed without timeout by the key
Meaning that users can have very long running scripts without being afraid that they will get timed out and killed.
Now:
On Zabbix agent remote commands are executed with timeout as set in the Timeout parameter of zabbix_agent.conf file.
Expected:
When user has script that executes longer than Timeout variable from config file, when he upgrades to new version, he must still be able to execute scripts as before, which is without time limit.
Actual:
All users who have script running longer than Timeout variable from config file will get their script killed and see "Timeout while executing a shell script." in the frontend.
This is blocker.
I suggest that such use cases need to be clearly defined in specification and solution discussed and approved.
Why this is happening ?
Output has been added to code below.
if (SUCCEED == (rc = zbx_execute_script(&host, &script, NULL, &output, error, sizeof(error))))
zabbix_log(LOG_LEVEL_DEBUG, "%s output:\n%s", script.command, output);
And in function zbx_execute_script_on_agent() it is used to change "nowait" to "wait
item.key = zbx_dsprintf(item.key, "system.run[%s,%s]", param, NULL == result ? "nowait" : "wait");
vjaceslavs There are no other options if we need to get output for log. So the only options are to drop the log to allow users to execute long running scripts or to check result code, but limit execution time.
I am not sure if we can consider that there is no need for exit code check in this case. I agree that behavior changes for multiple cases, but it is not really a blocker as user can create background scripts for cases like that (if it is an intended behavior).
Documentation for scripts does not provide information about this kind of behavior (execute on agent = async without limits, execute on server = limited by a timeout) so as far as I understand, I would say that current solution unifies solutions for server and agent.
Similar situation is with system.run item that will now start to check exit code and can become unsupported because of that.
vso Here is link from documentation Remote commands on Zabbix agent are executed without timeout
All I mean that changes/behavior must be very well documented and decision made. Current consequences are that if user has script that needs to be running infinitely or more than 30 seconds, it suddenly will not be possible, it will get killed at max 30 seconds but by default in 3 seconds.
The reason why I think that current solution for agent is not acceptable is because it will suddenly kill user scripts that previously were not killed and limit functionality of scripts to 30 seconds on agent. (killing scripts in the middle of work could also break something)
Also performance must be evaluated, if agent now waits for commands to execute and server wait for agent response.
I don't see any problems in checking result code on custom alertscript that is executed on server though.
vjaceslavs I added extra description about such cases in specification. We could also implement async remote command execution to bypass this timeout limit but it is not an easy fix so it should be evaluated.
vso Thanks ! Lets wait for approval.
vjaceslavs It was decided to ignore return code of commands executed on agent and to introduce new action status - "RAN" that means "Executed, but not checked". RESOLVED in r65091
vso CLOSED, though exact messages that need to be displayed are to be confirmed, but it will be minor change.
|
|
(7) Custom script (remote command in action -> operation) can be executed multiple times on server if Target list contains multiple records of current host.
Steps to reproduce:
1. Create Action
2. Add operation
3. Set Operation type to "Remote command"
4. Set Type to custom script
5. Set Execute on to "Server"
6. Add host to the Target list multiple times (for example Current host + Host <HOSTNAME>)
Action operation will be executed twice.
vjaceslavs RESOLVED in r65091
vso REOPENED. Fix must be reverted, it's frontend bug and this fix is not the way it can be handled, I suggest to make separate bug report.
Why is it not good solution ?
There can only be one server, that's why there shall be no host checks.
But after new changes there is additional SQL query called:
sql [select distinct h.hostid,h.host,h.tls_connect from functions f,items i,hosts h where f.itemid=i.itemid and i.hostid=h.hostid and h.status=0 and f.triggerid=13558]
4341:20170116:153222.315 End of get_dynamic_hostid():SUCCEED
As well as bunch of other code for checking host duplicates etc..
All this is done to fill host parameter which in the end is not used by server anyway, and it can't be used because no command is executed on any host, it's executed on server.
case ZBX_SCRIPT_EXECUTE_ON_AGENT:
ret = zbx_execute_script_on_agent(host, script->command, result,
error, max_error_len);
break;
case ZBX_SCRIPT_EXECUTE_ON_SERVER:
ret = zbx_execute(script->command, result, error, max_error_len,
CONFIG_TRAPPER_TIMEOUT);
break;
vjaceslavs Documentation states that "A command on a host is executed only once, even if the host matches more than once (e.g. from several host groups; individually and from a host group)"
And yes, there can be only one server, but nevertheless there can be multiple hosts (host is being set for action operations even for Execute on = Server) and current solution is causing multiple script executions if host is specified multiple times:
How we treat this problem as a frontend bug?
vso CLOSED, Lets create separate ticket for that.
|
|
(8) Custom script (remote command in action -> operation) executed on server is shown without host in Reports -> Action log
This causes output like that (command starts with ":" as host is missing):
vjaceslavs RESOLVED in r65091
vso REOPENED for the same reason as (7) command is not executed on host so this message is not appropriate actually, that's why there was blank.
vjaceslavs Command is not executed on host, but it is executed "within host context" and all host related macros will expand to specified host. So we can't say that setting target host for execution on server doesn't change anything.
And this is the reason why we need to show host in the results.
vso Please correct me if I am wrong.
Hostid does not affect anything when custom script is executed on server. Macros are still resolved as if current host was set.
So when 2 or more different targets are set, they are actually duplicates of same command but will have different hostid and get executed more then one time.
Assuming this macro expanding is by design, then I would suggest to make sure that custom script for server is only executed once.
Since there can be only one target for custom script that is executed on server there is no need even to check for duplicates, or to determine host id, we simply must stop on first execute as we know that there can't be more than one, no matter how many are added they are still duplicates of the same command that will be executed on same server.
vjaceslavs Assuming macro expanding is correct there should be no option to specify targets for scripts executed on server as those values will never be used. So problem is present anyway and we should only find out the scope of the problem.
vso CLOSED, Lets create separate ticket for that.
|
|
(9) Error message is never displayed to the user for a custom script operation in actions.
Steps:
In action operation detail set:
Type: Custom script
Commands, path to script that does not exist.
For example:
/home/vso/zabbix_eclipse_dev/ZBXNEXT-1380/share/zabbix/alertscripts/alert_script
Expected:
Message in frontend that is generated by shell:
"/home/vso/zabbix_eclipse_dev/ZBXNEXT-1380/share/zabbix/alertscripts/alert_script: not found"
Actual:
process exited with code: 127
Furthermore this means that buffer is set to NULL so no error messages will be red from custom script.
This means that in simple scenario where script could not be found, there will be no error message indicating exactly that.
vjaceslavs Added script output collection for all script executions so output can be present if error happens. RESOLVED in r65125
vso REOPENED, Double free corruption is one of the issues introduced but only occurs at specific scenarios.
The changes are to complex and buggy, please make a simpler solution so the code is clearer and there are no bugs.
*** Error in `./sbin/zabbix_agentd: listener #1 [processing request]': double free or corruption (top): 0x00000000023122a0 ***
======= Backtrace: =========
/lib/x86_64-linux-gnu/libc.so.6(+0x77725)[0x7fd94fdc0725]
/lib/x86_64-linux-gnu/libc.so.6(+0x7ff4a)[0x7fd94fdc8f4a]
/lib/x86_64-linux-gnu/libc.so.6(cfree+0x4c)[0x7fd94fdccabc]
./sbin/zabbix_agentd: listener #1 [processing request](EXECUTE_STR+0x89)[0x422479]
./sbin/zabbix_agentd: listener #1 [processing request](process+0x31c)[0x4171cc]
./sbin/zabbix_agentd: listener #1 [processing request](listener_thread+0x19d)[0x413aed]
./sbin/zabbix_agentd: listener #1 [processing request](zbx_thread_start+0x45)[0x42b135]
./sbin/zabbix_agentd: listener #1 [processing request](MAIN_ZABBIX_ENTRY+0x223)[0x416373]
./sbin/zabbix_agentd: listener #1 [processing request](daemon_start+0x1c3)[0x42be53]
./sbin/zabbix_agentd: listener #1 [processing request](main+0xb0)[0x40be20]
/lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0xf0)[0x7fd94fd69830]
./sbin/zabbix_agentd: listener #1 [processing request](_start+0x29)[0x40bf29]
======= Memory map: ========
00400000-00456000 r-xp 00000000 08:02 13763939 /home/vso/zabbix_eclipse_dev/ZBXNEXT-1380/sbin/zabbix_agentd
00655000-00656000 r--p 00055000 08:02 13763939 /home/vso/zabbix_eclipse_dev/ZBXNEXT-1380/sbin/zabbix_agentd
00656000-00658000 rw-p 00056000 08:02 13763939 /home/vso/zabbix_eclipse_dev/ZBXNEXT-1380/sbin/zabbix_agentd
00658000-0065d000 rw-p 00000000 00:00 0
022ef000-02330000 rw-p 00000000 00:00 0 [heap]
7fd944000000-7fd944021000 rw-p 00000000 00:00 0
7fd944021000-7fd948000000 ---p 00000000 00:00 0
7fd94b4ed000-7fd94b503000 r-xp 00000000 08:02 4460969 /lib/x86_64-linux-gnu/libgcc_s.so.1
7fd94b503000-7fd94b702000 ---p 00016000 08:02 4460969 /lib/x86_64-linux-gnu/libgcc_s.so.1
7fd94b702000-7fd94b703000 rw-p 00015000 08:02 4460969 /lib/x86_64-linux-gnu/libgcc_s.so.1
7fd94b703000-7fd94b70c000 r-xp 00000000 08:02 4460941 /lib/x86_64-linux-gnu/libcrypt-2.23.so
7fd94b70c000-7fd94b90b000 ---p 00009000 08:02 4460941 /lib/x86_64-linux-gnu/libcrypt-2.23.so
7fd94b90b000-7fd94b90c000 r--p 00008000 08:02 4460941 /lib/x86_64-linux-gnu/libcrypt-2.23.so
7fd94b90c000-7fd94b90d000 rw-p 00009000 08:02 4460941 /lib/x86_64-linux-gnu/libcrypt-2.23.so
7fd94b90d000-7fd94b93b000 rw-p 00000000 00:00 0
7fd94b93b000-7fd94ba0a000 r-xp 00000000 08:02 14165018 /usr/lib/x86_64-linux-gnu/libsqlite3.so.0.8.6
7fd94ba0a000-7fd94bc0a000 ---p 000cf000 08:02 14165018 /usr/lib/x86_64-linux-gnu/libsqlite3.so.0.8.6
7fd94bc0a000-7fd94bc0d000 r--p 000cf000 08:02 14165018 /usr/lib/x86_64-linux-gnu/libsqlite3.so.0.8.6
7fd94bc0d000-7fd94bc0f000 rw-p 000d2000 08:02 14165018 /usr/lib/x86_64-linux-gnu/libsqlite3.so.0.8.6
7fd94bc0f000-7fd94bc10000 rw-p 00000000 00:00 0
7fd94bc10000-7fd94bc56000 r-xp 00000000 08:02 14164557 /usr/lib/x86_64-linux-gnu/libhx509.so.5.0.0
7fd94bc56000-7fd94be56000 ---p 00046000 08:02 14164557 /usr/lib/x86_64-linux-gnu/libhx509.so.5.0.0
7fd94be56000-7fd94be58000 r--p 00046000 08:02 14164557 /usr/lib/x86_64-linux-gnu/libhx509.so.5.0.0
7fd94be58000-7fd94be5a000 rw-p 00048000 08:02 14164557 /usr/lib/x86_64-linux-gnu/libhx509.so.5.0.0
7fd94be5a000-7fd94be5b000 rw-p 00000000 00:00 0
7fd94be5b000-7fd94be69000 r-xp 00000000 08:02 14164539 /usr/lib/x86_64-linux-gnu/libheimbase.so.1.0.0
7fd94be69000-7fd94c068000 ---p 0000e000 08:02 14164539 /usr/lib/x86_64-linux-gnu/libheimbase.so.1.0.0
7fd94c068000-7fd94c069000 r--p 0000d000 08:02 14164539 /usr/lib/x86_64-linux-gnu/libheimbase.so.1.0.0
7fd94c069000-7fd94c06a000 rw-p 0000e000 08:02 14164539 /usr/lib/x86_64-linux-gnu/libheimbase.so.1.0.0
7fd94c06a000-7fd94c091000 r-xp 00000000 08:02 14165183 /usr/lib/x86_64-linux-gnu/libwind.so.0.0.0
7fd94c091000-7fd94c291000 ---p 00027000 08:02 14165183 /usr/lib/x86_64-linux-gnu/libwind.so.0.0.0
7fd94c291000-7fd94c292000 r--p 00027000 08:02 14165183 /usr/lib/x86_64-linux-gnu/libwind.so.0.0.0
7fd94c292000-7fd94c293000 rw-p 00028000 08:02 14165183 /usr/lib/x86_64-linux-gnu/libwind.so.0.0.0
7fd94c293000-7fd94c2a8000 r-xp 00000000 08:02 14164936 /usr/lib/x86_64-linux-gnu/libroken.so.18.1.0
7fd94c2a8000-7fd94c4a7000 ---p 00015000 08:02 14164936 /usr/lib/x86_64-linux-gnu/libroken.so.18.1.0
7fd94c4a7000-7fd94c4a8000 r--p 00014000 08:02 14164936 /usr/lib/x86_64-linux-gnu/libroken.so.18.1.0
7fd94c4a8000-7fd94c4a9000 rw-p 00015000 08:02 14164936 /usr/lib/x86_64-linux-gnu/libroken.so.18.1.0
7fd94c4a9000-7fd94c4d9000 r-xp 00000000 08:02 14164537 /usr/lib/x86_64-linux-gnu/libhcrypto.so.4.1.0
7fd94c4d9000-7fd94c6d9000 ---p 00030000 08:02 14164537 /usr/lib/x86_64-linux-gnu/libhcrypto.so.4.1.0
7fd94c6d9000-7fd94c6da000 r--p 00030000 08:02 14164537 /usr/lib/x86_64-linux-gnu/libhcrypto.so.4.1.0
7fd94c6da000-7fd94c6db000 rw-p 00031000 08:02 14164537 /usr/lib/x86_64-linux-gnu/libhcrypto.so.4.1.0
7fd94c6db000-7fd94c6dc000 rw-p 00000000 00:00 0
7fd94c6dc000-7fd94c77b000 r-xp 00000000 08:02 14163981 /usr/lib/x86_64-linux-gnu/libasn1.so.8.0.0
7fd94c77b000-7fd94c97a000 ---p 0009f000 08:02 14163981 /usr/lib/x86_64-linux-gnu/libasn1.so.8.0.0
7fd94c97a000-7fd94c97b000 r--p 0009e000 08:02 14163981 /usr/lib/x86_64-linux-gnu/libasn1.so.8.0.0
7fd94c97b000-7fd94c97e000 rw-p 0009f000 08:02 14163981 /usr/lib/x86_64-linux-gnu/libasn1.so.8.0.0
7fd94c97e000-7fd94ca02000 r-xp 00000000 08:02 14164633 /usr/lib/x86_64-linux-gnu/libkrb5.so.26.0.0
7fd94ca02000-7fd94cc01000 ---p 00084000 08:02 14164633 /usr/lib/x86_64-linux-gnu/libkrb5.so.26.0.0
7fd94cc01000-7fd94cc04000 r--p 00083000 08:02 14164633 /usr/lib/x86_64-linux-gnu/libkrb5.so.26.0.0
7fd94cc04000-7fd94cc07000 rw-p 00086000 08:02 14164633 /usr/lib/x86_64-linux-gnu/libkrb5.so.26.0.0
7fd94cc07000-7fd94cc08000 rw-p 00000000 00:00 0
7fd94cc08000-7fd94cc10000 r-xp 00000000 08:02 14164541 /usr/lib/x86_64-linux-gnu/libheimntlm.so.0.1.0
7fd94cc10000-7fd94ce0f000 ---p 00008000 08:02 14164541 /usr/lib/x86_64-linux-gnu/libheimntlm.so.0.1.0
7fd94ce0f000-7fd94ce10000 r--p 00007000 08:02 14164541 /usr/lib/x86_64-linux-gnu/libheimntlm.so.0.1.0
7fd94ce10000-7fd94ce11000 rw-p 00008000 08:02 14164541 /usr/lib/x86_64-linux-gnu/libheimntlm.so.0.1.0
7fd94ce11000-7fd94ce14000 r-xp 00000000 08:02 4460992 /lib/x86_64-linux-gnu/libkeyutils.so.1.5
7fd94ce14000-7fd94d013000 ---p 00003000 08:02 4460992 /lib/x86_64-linux-gnu/libkeyutils.so.1.5
7fd94d013000-7fd94d014000 r--p 00002000 08:02 4460992 /lib/x86_64-linux-gnu/libkeyutils.so.1.5
7fd94d014000-7fd94d015000 rw-p 00003000 08:02 4460992 /lib/x86_64-linux-gnu/libkeyutils.so.1.5
7fd94d015000-7fd94d01c000 r-xp 00000000 08:02 14164274 /usr/lib/x86_64-linux-gnu/libffi.so.6.0.4
7fd94d01c000-7fd94d21b000 ---p 00007000 08:02 14164274 /usr/lib/x86_64-linux-gnu/libffi.so.6.0.4
7fd94d21b000-7fd94d21c000 r--p 00006000 08:02 14164274 /usr/lib/x86_64-linux-gnu/libffi.so.6.0.4
7fd94d21c000-7fd94d21d000 rw-p 00007000 08:02 14164274 /usr/lib/x86_64-linux-gnu/libffi.so.6.0.4
7fd94d21d000-7fd94d25a000 r-xp 00000000 08:02 14164462 /usr/lib/x86_64-linux-gnu/libgssapi.so.3.0.0
7fd94d25a000-7fd94d45a000 ---p 0003d000 08:02 14164462 /usr/lib/x86_64-linux-gnu/libgssapi.so.3.0.0
7fd94d45a000-7fd94d45b000 r--p 0003d000 08:02 14164462 /usr/lib/x86_64-linux-gnu/libgssapi.so.3.0.0
7fd94d45b000-7fd94d45d000 rw-p 0003e000 08:02 14164462 /usr/lib/x86_64-linux-gnu/libgssapi.so.3.0.0
7fd94d45d000-7fd94d45e000 rw-p 00000000 00:00 0
7fd94d45e000-7fd94d477000 r-xp 00000000 08:02 14164963 /usr/lib/x86_64-linux-gnu/libsasl2.so.2.0.25
7fd94d477000-7fd94d677000 ---p 00019000 08:02 14164963 /usr/lib/x86_64-linux-gnu/libsasl2.so.2.0.25
7fd94d677000-7fd94d678000 r--p 00019000 08:02 14164963 /usr/lib/x86_64-linux-gnu/libsasl2.so.2.0.25
7fd94d678000-7fd94d679000 rw-p 0001a000 08:02 14164963 /usr/lib/x86_64-linux-gnu/libsasl2.so.2.0.25
7fd94d679000-7fd94d683000 r-xp 00000000 08:02 14164637 /usr/lib/x86_64-linux-gnu/libkrb5support.so.0.1
7fd94d683000-7fd94d882000 ---p 0000a000 08:02 14164637 /usr/lib/x86_64-linux-gnu/libkrb5support.so.0.1
7fd94d882000-7fd94d883000 r--p 00009000 08:02 14164637 /usr/lib/x86_64-linux-gnu/libkrb5support.so.0.1
7fd94d883000-7fd94d884000 rw-p 0000a000 08:02 14164637 /usr/lib/x86_64-linux-gnu/libkrb5support.so.0.1
7fd94d884000-7fd94d887000 r-xp 00000000 08:02 4460940 /lib/x86_64-linux-gnu/libcom_err.so.2.1
7fd94d887000-7fd94da86000 ---p 00003000 08:02 4460940 /lib/x86_64-linux-gnu/libcom_err.so.2.1
7fd94da86000-7fd94da87000 r--p 00002000 08:02 4460940 /lib/x86_64-linux-gnu/libcom_err.so.2.1
7fd94da87000-7fd94da88000 rw-p 00003000 08:02 4460940 /lib/x86_64-linux-gnu/libcom_err.so.2.1
7fd94da88000-7fd94dab4000 r-xp 00000000 08:02 14164629 /usr/lib/x86_64-linux-gnu/libk5crypto.so.3.1
7fd94dab4000-7fd94dcb3000 ---p 0002c000 08:02 14164629 /usr/lib/x86_64-linux-gnu/libk5crypto.so.3.1
7fd94dcb3000-7fd94dcb5000 r--p 0002b000 08:02 14164629 /usr/lib/x86_64-linux-gnu/libk5crypto.so.3.1
7fd94dcb5000-7fd94dcb6000 rw-p 0002d000 08:02 14164629 /usr/lib/x86_64-linux-gnu/libk5crypto.so.3.1
7fd94dcb6000-7fd94dcb7000 rw-p 00000000 00:00 0
7fd94dcb7000-7fd94dd7a000 r-xp 00000000 08:02 14164635 /usr/lib/x86_64-linux-gnu/libkrb5.so.3.3
7fd94dd7a000-7fd94df7a000 ---p 000c3000 08:02 14164635 /usr/lib/x86_64-linux-gnu/libkrb5.so.3.3
7fd94df7a000-7fd94df87000 r--p 000c3000 08:02 14164635 /usr/lib/x86_64-linux-gnu/libkrb5.so.3.3
7fd94df87000-7fd94df89000 rw-p 000d0000 08:02 14164635 /usr/lib/x86_64-linux-gnu/libkrb5.so.3.3
7fd94df89000-7fd94df9a000 r-xp 00000000 08:02 14165035 /usr/lib/x86_64-linux-gnu/libtasn1.so.6.5.1
7fd94df9a000-7fd94e19a000 ---p 00011000 08:02 14165035 /usr/lib/x86_64-linux-gnu/libtasn1.so.6.5.1
7fd94e19a000-7fd94e19b000 r--p 00011000 08:02 14165035 /usr/lib/x86_64-linux-gnu/libtasn1.so.6.5.1
7fd94e19b000-7fd94e19c000 rw-p 00012000 08:02 14165035 /usr/lib/x86_64-linux-gnu/libtasn1.so.6.5.1
7fd94e19c000-7fd94e1f5000 r-xp 00000000 08:02 14155987 /usr/lib/x86_64-linux-gnu/libp11-kit.so.0.1.0
7fd94e1f5000-7fd94e3f4000 ---p 00059000 08:02 14155987 /usr/lib/x86_64-linux-gnu/libp11-kit.so.0.1.0
7fd94e3f4000-7fd94e3fe000 r--p 00058000 08:02 14155987 /usr/lib/x86_64-linux-gnu/libp11-kit.so.0.1.0
7fd94e3fe000-7fd94e400000 rw-p 00062000 08:02 14155987 /usr/lib/x86_64-linux-gnu/libp11-kit.so.0.1.0
7fd94e400000-7fd94e47f000 r-xp 00000000 08:02 14164407 /usr/lib/x86_64-linux-gnu/libgmp.so.10.3.0
7fd94e47f000-7fd94e67e000 ---p 0007f000 08:02 14164407 /usr/lib/x86_64-linux-gnu/libgmp.so.10.3.0
7fd94e67e000-7fd94e67f000 r--p 0007e000 08:02 14164407 /usr/lib/x86_64-linux-gnu/libgmp.so.10.3.0
7fd94e67f000-7fd94e680000 rw-p 0007f000 08:02 14164407 /usr/lib/x86_64-linux-gnu/libgmp.so.10.3.0
7fd94e680000-7fd94e6b2000 r-xp 00000000 08:02 14164543 /usr/lib/x86_64-linux-gnu/libhogweed.so.4.2
7fd94e6b2000-7fd94e8b1000 ---p 00032000 08:02 14164543 /usr/lib/x86_64-linux-gnu/libhogweed.so.4.2
7fd94e8b1000-7fd94e8b2000 r--p 00031000 08:02 14164543 /usr/lib/x86_64-linux-gnu/libhogweed.so.4.2
7fd94e8b2000-7fd94e8b3000 rw-p 00032000 08:02 14164543 /usr/lib/x86_64-linux-gnu/libhogweed.so.4.2
7fd94e8b3000-7fd94e8cb000 r-xp 00000000 08:02 4461077 /lib/x86_64-linux-gnu/libpthread-2.23.so
7fd94e8cb000-7fd94eaca000 ---p 00018000 08:02 4461077 /lib/x86_64-linux-gnu/libpthread-2.23.so
7fd94eaca000-7fd94eacb000 r--p 00017000 08:02 4461077 /lib/x86_64-linux-gnu/libpthread-2.23.so
7fd94eacb000-7fd94eacc000 rw-p 00018000 08:02 4461077 /lib/x86_64-linux-gnu/libpthread-2.23.so
7fd94eacc000-7fd94ead0000 rw-p 00000000 00:00 0
7fd94ead0000-7fd94eae9000 r-xp 00000000 08:02 4461122 /lib/x86_64-linux-gnu/libz.so.1.2.8
7fd94eae9000-7fd94ece8000 ---p 00019000 08:02 4461122 /lib/x86_64-linux-gnu/libz.so.1.2.8
7fd94ece8000-7fd94ece9000 r--p 00018000 08:02 4461122 /lib/x86_64-linux-gnu/libz.so.1.2.8
7fd94ece9000-7fd94ecea000 rw-p 00019000 08:02 4461122 /lib/x86_64-linux-gnu/libz.so.1.2.8
7fd94ecea000-7fd94ed37000 r-xp 00000000 08:02 14164648 /usr/lib/x86_64-linux-gnu/libldap_r-2.4.so.2.10.5
7fd94ed37000-7fd94ef36000 ---p 0004d000 08:02 14164648 /usr/lib/x86_64-linux-gnu/libldap_r-2.4.so.2.10.5
7fd94ef36000-7fd94ef38000 r--p 0004c000 08:02 14164648 /usr/lib/x86_64-linux-gnu/libldap_r-2.4.so.2.10.5
7fd94ef38000-7fd94ef39000 rw-p 0004e000 08:02 14164648 /usr/lib/x86_64-linux-gnu/libldap_r-2.4.so.2.10.5
7fd94ef39000-7fd94ef3b000 rw-p 00000000 00:00 0
7fd94ef3b000-7fd94ef48000 r-xp 00000000 08:02 14164643 /usr/lib/x86_64-linux-gnu/liblber-2.4.so.2.10.5
7fd94ef48000-7fd94f148000 ---p 0000d000 08:02 14164643 /usr/lib/x86_64-linux-gnu/liblber-2.4.so.2.10.5
7fd94f148000-7fd94f149000 r--p 0000d000 08:02 14164643 /usr/lib/x86_64-linux-gnu/liblber-2.4.so.2.10.5
7fd94f149000-7fd94f14a000 rw-p 0000e000 08:02 14164643 /usr/lib/x86_64-linux-gnu/liblber-2.4.so.2.10.5
7fd94f14a000-7fd94f191000 r-xp 00000000 08:02 14164464 /usr/lib/x86_64-linux-gnu/libgssapi_krb5.so.2.2
7fd94f191000-7fd94f390000 ---p 00047000 08:02 14164464 /usr/lib/x86_64-linux-gnu/libgssapi_krb5.so.2.2
7fd94f390000-7fd94f392000 r--p 00046000 08:02 14164464 /usr/lib/x86_64-linux-gnu/libgssapi_krb5.so.2.2
7fd94f392000-7fd94f394000 rw-p 00048000 08:02 14164464 /usr/lib/x86_64-linux-gnu/libgssapi_krb5.so.2.2
7fd94f394000-7fd94f4b7000 r-xp 00000000 08:02 14164425 /usr/lib/x86_64-linux-gnu/libgnutls.so.30.6.2
7fd94f4b7000-7fd94f6b6000 ---p 00123000 08:02 14164425 /usr/lib/x86_64-linux-gnu/libgnutls.so.30.6.2
7fd94f6b6000-7fd94f6c1000 r--p 00122000 08:02 14164425 /usr/lib/x86_64-linux-gnu/libgnutls.so.30.6.2
7fd94f6c1000-7fd94f6c3000 rw-p 0012d000 08:02 14164425 /usr/lib/x86_64-linux-gnu/libgnutls.so.30.6.2
7fd94f6c3000-7fd94f6c4000 rw-p 00000000 00:00 0
7fd94f6c4000-7fd94f6f8000 r-xp 00000000 08:02 14164753 /usr/lib/x86_64-linux-gnu/libnettle.so.6.2
7fd94f6f8000-7fd94f8f7000 ---p 00034000 08:02 14164753 /usr/lib/x86_64-linux-gnu/libnettle.so.6.2
7fd94f8f7000-7fd94f8f9000 r--p 00033000 08:02 14164753 /usr/lib/x86_64-linux-gnu/libnettle.so.6.2
7fd94f8f9000-7fd94f8fa000 rw-p 00035000 08:02 14164753 /usr/lib/x86_64-linux-gnu/libnettle.so.6.2
7fd94f8fa000-7fd94f915000 r-xp 00000000 08:02 14164944 /usr/lib/x86_64-linux-gnu/librtmp.so.1
7fd94f915000-7fd94fb14000 ---p 0001b000 08:02 14164944 /usr/lib/x86_64-linux-gnu/librtmp.so.1
7fd94fb14000-7fd94fb15000 r--p 0001a000 08:02 14164944 /usr/lib/x86_64-linux-gnu/librtmp.so.1
7fd94fb15000-7fd94fb16000 rw-p 0001b000 08:02 14164944 /usr/lib/x86_64-linux-gnu/librtmp.so.1
7fd94fb16000-7fd94fb47000 r-xp 00000000 08:02 14156253 /usr/lib/x86_64-linux-gnu/libidn.so.11.6.15
7fd94fb47000-7fd94fd47000 ---p 00031000 08:02 14156253 /usr/lib/x86_64-linux-gnu/libidn.so.11.6.15
7fd94fd47000-7fd94fd48000 r--p 00031000 08:02 14156253 /usr/lib/x86_64-linux-gnu/libidn.so.11.6.15
7fd94fd48000-7fd94fd49000 rw-p 00032000 08:02 14156253 /usr/lib/x86_64-linux-gnu/libidn.so.11.6.15
7fd94fd49000-7fd94ff09000 r-xp 00000000 08:02 4460931 /lib/x86_64-linux-gnu/libc-2.23.so
7fd94ff09000-7fd950108000 ---p 001c0000 08:02 4460931 /lib/x86_64-linux-gnu/libc-2.23.so
7fd950108000-7fd95010c000 r--p 001bf000 08:02 4460931 /lib/x86_64-linux-gnu/libc-2.23.so
7fd95010c000-7fd95010e000 rw-p 001c3000 08:02 4460931 /lib/x86_64-linux-gnu/libc-2.23.so
7fd95010e000-7fd950112000 rw-p 00000000 00:00 0
7fd950112000-7fd950129000 r-xp 00000000 08:02 4461083 /lib/x86_64-linux-gnu/libresolv-2.23.so
7fd950129000-7fd950329000 ---p 00017000 08:02 4461083 /lib/x86_64-linux-gnu/libresolv-2.23.so
7fd950329000-7fd95032a000 r--p 00017000 08:02 4461083 /lib/x86_64-linux-gnu/libresolv-2.23.so
7fd95032a000-7fd95032b000 rw-p 00018000 08:02 4461083 /lib/x86_64-linux-gnu/libresolv-2.23.so
7fd95032b000-7fd95032d000 rw-p 00000000 00:00 0
7fd95032d000-7fd950330000 r-xp 00000000 08:02 4460955 /lib/x86_64-linux-gnu/libdl-2.23.so
7fd950330000-7fd95052f000 ---p 00003000 08:02 4460955 /lib/x86_64-linux-gnu/libdl-2.23.so
7fd95052f000-7fd950530000 r--p 00002000 08:02 4460955 /lib/x86_64-linux-gnu/libdl-2.23.so
7fd950530000-7fd950531000 rw-p 00003000 08:02 4460955 /lib/x86_64-linux-gnu/libdl-2.23.so
7fd950531000-7fd95059a000 r-xp 00000000 08:02 14166919 /usr/lib/x86_64-linux-gnu/libcurl-gnutls.so.4.4.0
7fd95059a000-7fd95079a000 ---p 00069000 08:02 14166919 /usr/lib/x86_64-linux-gnu/libcurl-gnutls.so.4.4.0
7fd95079a000-7fd95079d000 r--p 00069000 08:02 14166919 /usr/lib/x86_64-linux-gnu/libcurl-gnutls.so.4.4.0
7fd95079d000-7fd95079e000 rw-p 0006c000 08:02 14166919 /usr/lib/x86_64-linux-gnu/libcurl-gnutls.so.4.4.0
7fd95079e000-7fd9507c4000 r-xp 00000000 08:02 4460903 /lib/x86_64-linux-gnu/ld-2.23.so
7fd9508f1000-7fd950992000 rw-s 00000000 00:05 35684393 /SYSV6c020209 (deleted)
7fd950992000-7fd9509a3000 rw-p 00000000 00:00 0
7fd9509c0000-7fd9509c1000 rw-p 00000000 00:00 0
7fd9509c1000-7fd9509c3000 rw-p 00000000 00:00 0
7fd9509c3000-7fd9509c4000 r--p 00025000 08:02 4460903 /lib/x86_64-linux-gnu/ld-2.23.so
7fd9509c4000-7fd9509c5000 rw-p 00026000 08:02 4460903 /lib/x86_64-linux-gnu/ld-2.23.so
7fd9509c5000-7fd9509c6000 rw-p 00000000 00:00 0
7ffdceab2000-7ffdcead3000 rw-p 00000000 00:00 0 [stack]
7ffdceb43000-7ffdceb45000 r--p 00000000 00:00 0 [vvar]
7ffdceb45000-7ffdceb47000 r-xp 00000000 00:00 0 [vdso]
ffffffffff600000-ffffffffff601000 r-xp 00000000 00:00 0 [vsyscall]
23302:20170117:115534.330 One child process died (PID:23305,exitcode/signal:6). Exiting ...
zabbix_agentd [23302]: Error waiting for process with PID 23305: [10] No child processes
23302:20170117:115534.330 Zabbix Agent stopped. Zabbix 3.3.0 (revision {ZABBIX_REVISION}).
vjaceslavs Sorry, I missed the fact that zbx_free alters the value of the pointer. All bugs produced are not intended (even if it seems so), but thanks for advice to make code without bugs. As for simplicity, this solution is the simplest way to get output when errors occur. RESOLVED in r65131.
vso REOPENED, there is memory leak if zbx_execute is called multiple times with same pointer.
==27414== 131,072 bytes in 1 blocks are definitely lost in loss record 25 of 26
==27414== at 0x4C2FD5F: realloc (in /usr/lib/valgrind/vgpreload_memcheck-amd64-linux.so)
==27414== by 0x4317FA: zbx_realloc2 (misc.c:499)
==27414== by 0x4353FF: zbx_strncpy_alloc (str.c:312)
==27414== by 0x43ECA5: zbx_execute (execute.c:426)
==27414== by 0x41E9EE: SYSTEM_SW_PACKAGES (software.c:226)
==27414== by 0x4171CB: process (sysinfo.c:641)
==27414== by 0x413AEC: process_listener (listener.c:58)
==27414== by 0x413AEC: listener_thread (listener.c:145)
==27414== by 0x42B184: zbx_thread_start (threads.c:128)
==27414== by 0x416372: MAIN_ZABBIX_ENTRY (zabbix_agentd.c:966)
==27414== by 0x42BEA2: daemon_start (daemon.c:392)
==27414== by 0x40BE1F: main (zabbix_agentd.c:1203)
vjaceslavs Restored previous logic when free is applied to the param value. RESOLVED in r65132
vso CLOSED with small style fixes in r65133
|
|
(10) IPMI will no be possible to execute on server if not current server target is specified.
Check has been added not to copy communication parameters if executed on server, however if it's not custom script but for example IPMI then server must copy communication parameters.
if (ZBX_SCRIPT_EXECUTE_ON_SERVER != script.execute_on)
{
host.tls_connect = (unsigned char)atoi(row[12]);
#ifdef HAVE_OPENIPMI
host.ipmi_authtype = (signed char)atoi(row[13]);
host.ipmi_privilege = (unsigned char)atoi(row[14]);
strscpy(host.ipmi_username, row[15]);
strscpy(host.ipmi_password, row[16]);
#endif
#if defined(HAVE_POLARSSL) || defined(HAVE_GNUTLS) || defined(HAVE_OPENSSL)
strscpy(host.tls_issuer, row[13 + ZBX_IPMI_FIELDS_NUM]);
strscpy(host.tls_subject, row[14 + ZBX_IPMI_FIELDS_NUM]);
strscpy(host.tls_psk_identity, row[15 + ZBX_IPMI_FIELDS_NUM]);
strscpy(host.tls_psk, row[16 + ZBX_IPMI_FIELDS_NUM]);
#endif
vjaceslavs Check is valid. script.execute_on is initialized only for custom script.
vso CLOSED, yes you are right.
|
|
(11) [F][S] Appropriate message for scripts executed on agent shall be added.
Currently message is "Ran":
Decision shall be made on how it must look.
But I personally would prefer that when executed on agent then it would always say "Executed"
When on server "Succeeded" or "Failed".
For mail and media script "Sent" or "Failed" or "In progress"
vso It was decided:
"Executed" - successfully sent to agent or successfully executed on server.
"Failed" - otherwise.
vjaceslavs Reverted to single status for execution with or without checks
Looks like that for same script executed on agent and server:
RESOLVED in r65163
vso CLOSED, with small style fix in r65165
|
|
(12) zbx_execute release previously allocated resources only in the end, while not using those, this is not very effective.
Before
Logic in zbx_executed was to reallocate previously allocated buffer and then continue with it.
Now:
Allocate new buffer, and free previously allocated buffer in the end.
vjaceslavs RESOLVED in r65175
vso CLOSED
|
|
Successfully tested
|
|
(13) [D] This need to be very well documented, especially in upgrade notes.
vso CLOSED as it was done under (15)
|
|
Available in pre-3.3.0 (trunk) r65179.
|
|
(14) [S] Errors that show up in frontend must start with a capital letter and end with period.
vjaceslavs Should we change all existing errors (introduced in and before this solution) and output of the scripts to this format? If so, there are multiple ways to do it:
1. Wrap output of the script into some message string like Execution failed: "<OUTPUT GOES HERE>".
2. Introduce check and formatting of an output in frontend so all errors are displayed in this format.
3. Fix all existing errors and transform output so first letter is capitalized and output ends with a period.
Should error messages like zbx_waitpid() failed: X be rewritten so they explain what happen in the way users can understand or should they remain the same?
Any combination of those is possible, so decision should be made before implementing.
vso Only static messages that were introduced, everything else is out of scope of this task.
User output must not be altered.
I agree that there are some messages that do not obey this rule, one is example you mentioned, but they all have something in common, they will only occur on some critical error.
vjaceslavs Maybe I am missing difference on how critical some errors are and why unable to create process [X]: X is critical, but Process terminated unexpectedly. is somehow ok, but
RESOLVED in pre-3.3.0 (trunk) r65183
vso CLOSED
|
|
(15) [D] Documented in the following documentation pages for version 3.4:
vso Isn't "Due to command/script exit code check introduction in Zabbix 3.4 commands and scripts can be executed multiple times if exit code is different from 0"
Only for media script ?
Also for media script it would be good to use such example with exit 0 added. And to mention that exit value other than 0 would mean error.
#!/bin/bash
to=$1
subject=$2
body=$3
cat <<EOF | mail -s "$subject" "$to"
$body
EOF
exit 0
VSI Thanks for the comment!
Upgrade notes for 3.4.0 updated, script example on the Custom alertscripts page updated with proposed example and note about error codes.
<dimir> Wouldn't it be better to actually spout error if that occurs:
exit 0
exit $?
vso yes I think you are right about $? It's better working example as exit status of last command will be returned. exit 0 just says that user must explicitly define exit code. Maybe original example should stay as no exit is same as exit $?
vjaceslavs Thats why I was confused about changes made to existing example. If we want to show how to use exit to define exit code then we could make second example like that:
#!/bin/bash
if [ -f "/var/messages/target" ];
then
# some logic goes here
else
exit 1 #exit with an error
fi
VSI Removed "exit 0" entry from the example and mentioned exit code checking basics in the note on Custom alertscripts page. I'd prefer not to complicate description with several examples, especially with additional logic.
Do you think it worth to mention possibility for user to define an exit code within a script regardless of actual execution results? It looks like some very specific case.
vjaceslavs I agree that this is a very specific case. I gave this example just to show the idea on how exit X could be used "in real life", but I think that we can leave things simple and stay with the existing example.
vso REOPENED, There are some info missing in upgrade notes and what's new, all pages from (16) should be mentioned.
This below leaves question, "can be executed" it shall be well define what will happen and how many times and if it's feature or known issue.
alertscripts can be executed multiple times if their exit code is different from 0
It should be confirmed but from code looks like max retries is 3 times, delay in between is 30 seconds, I could not find it documented anywhere.
VSI The main idea is to warn users about the fact of possibility of such behavior after upgrade in case of non-0 exit code. I don't see the point to introduce users in upgrade notes with parameters of a mechanism which cannot be configured. Perhaps it should be documented, but elsewhere. Also these parameters are not actually a result or a dependency of the change introduced in this issue, but their effect is, which is mentioned.
vso Yes I agree, still REOPENED, upgrade notes have been updated under (16) and looks OK.
However what's new does not mention what has been added to (16) in documentation.
For example here https://www.zabbix.com/documentation/3.4/manual/web_interface/frontend_sections/administration/scripts?rev=1478764129&do=diff
New info about user scripts is added but no mention in what's new.
external check items are mentioned in upgrade notes but not it what's new, etc..
I think all new features should be in what's new, please go through (16) pages and double check
Also this below should be deleted from what's new because it's nothing for user to care about.
And for some reason it's in what's new but not in upgrade notes.
system.hw.devices, and system.sw.packages Zabbix agent items
And it's not only place where return code has been added, for example in net.tcp.listen and some other places, probably best to mention that some items that launch commands could also become unsupported. ( if needed, I can provide a list )
VSI What's new updated with:
- generalized term 'scripts' as we don't have anything described as 'user script' in documentation
- mention of external checks
- generalized phrase 'Zabbix agent items executed without “nowait” flag' instead of naming specific items.
vso
Documentation states:
remote commands on Zabbix agent are executed without timeout by the system.run[,nowait] key and are not checked for execution results.
While what's new
Now Zabbix checks exit codes for scripts, including media types, external checks, remote commands, user parameters executed by Zabbix server and for Zabbix agent items executed without “nowait” flag.
VSI What's new fixed and phrased as
"Now Zabbix checks exit codes for user parameters, remote commands and system.run[] items without the “nowait” flag as well as Zabbix server executed scripts (alert, external and global). In case of an error in a script or a command execution process Zabbix provides error description in the Frontend as well as creates corresponding log entries."
vso CLOSED
|
|
(16) [D] No mention about return code in:
- Custom alertscripts
This one looks OK, simply for history and to see whats new.
Should we also mention that it will retry on failure ?
https://www.zabbix.com/documentation/3.4/manual/config/notifications/media/script
VSI Exit code checking and links to whole Command execution page added in:
vso CLOSED
|
|
will this also save the failure output somewhere - server logfile, for example ? if yes, that would solve ZBXNEXT-1284
|
|
Error output of command execution is available in frontend and in server log file (LOG_LEVEL_WARNING).
|
|
vjaceslavs, that sounds really neat, thank you.
|
|
Sub-issues still open: (13)
|
|
Looks like it caused ZBX-12874.
|
Screen Recording 2022-07-04 at 13.14.10.mp4
host-info-1.png
✅ ACC_ Optional interfaces for server-originated checks - v1.0.pdf
Team C
zbx_export_hosts.xml
zbxnext-3006.patch
