[ZBXNEXT-1380] Zabbix server should check the return code of media scripts Created: 2012 Aug 26  Updated: 2024 Apr 10  Resolved: 2017 Feb 17

Status: Closed
Project: ZABBIX FEATURE REQUESTS
Component/s: Agent (G), Server (S)
Affects Version/s: 2.0.2
Fix Version/s: 3.4.0alpha1, 3.4 (plan)

Type: Change Request Priority: Minor
Reporter: rrr Assignee: Unassigned
Resolution: Fixed Votes: 15
Labels: alertscript, externalchecks, media, operations, remotecommands, returncode, scripts, userparameters
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Attachments: PNG File failed-command-and-alert.png     PNG File message_for_executed_on_agent.png     PNG File nohost.png     PNG File reverted.png     PNG File same_command.png    
Issue Links:
Causes
causes ZBX-13607 wrong error message if ping-script ex... Closed
causes ZBX-14328 More information about failed command... Closed
Duplicate
is duplicated by ZBXNEXT-3760 Zabbix Actions - caputre error / exit... Closed
is duplicated by ZBX-6971 Exit code of alert script ignored Closed
is duplicated by ZBX-9333 Check exit code of custom alertscript... Closed
Sub-task
depends on ZBX-12594 Not possible to ignore exit codes of ... Closed
depends on ZBX-12874 Autoregistration script fails to exec... Closed
Team: Team A
Sprint: Sprint 1
Story Points: 1

 Description   

It would be nice if zabbix server check the return code of media scripts to ensure that the recipient received the message.



 Comments   
Comment by Raymond Kuiper [ 2012 Oct 15 ]

+1

Comment by sergio cricca [ 2012 Oct 15 ]

+1

Comment by Raymond Kuiper [ 2012 Oct 15 ]

Please also cast a vote!

Comment by richlv [ 2012 Oct 15 ]

seriously guys, "+1" comments are not helpful. they actually only clutter the issues, result in useless notifications and i'm considering some mass cleanup (read - deletion) of such comments :/

issue voting feature is in the upper right corner. it's also described in bug reporting guidelines at https://zabbix.org/wiki/Docs/bug_reporting_guidelines#Reporting_an_issue

Comment by Marc [ 2013 Sep 10 ]

similar to ZBX-6971

Comment by Timofey [ 2016 Aug 16 ]

Hi guys, i find workaround (bad but works)
Make sleep 3600 on error, so you will see timeouts on dashboard & etc

Comment by Raymond Kuiper [ 2016 Nov 10 ]

How is this still a problem with modern versions of Zabbix?
It can't be that hard to report error on exit code != 0.

Comment by Raymond Kuiper [ 2016 Nov 10 ]

@Timofey, thanks for the 'workaround' btw. Setting 'sleep 120' before every exit at least makes Zabbix see the message as failed.

Comment by Stefan Radman [ 2016 Nov 10 ]

Can someone update the affected versions, update the priority and get this feature request assigned?
With 14 votes and 3.2.1 out it's about time to get this issue out of the way. Thanks

Comment by Raymond Kuiper [ 2016 Nov 10 ]

I would also argue that if a non-zero exit code is received, any stderr script output should be shown as the error message (perhaps truncated) in the frontend.

Comment by richlv [ 2016 Nov 10 ]

stefan.radman, if you have a support agreement with the zabbix company, you might want to raise this topic through your contact person there.
14 votes puts this feature request somewhere at place 120 of the highest voted requests

Comment by Stefan Radman [ 2016 Nov 10 ]

got your point. We have a support agreement and I might raise it there. Thanks

Comment by Oleksii Zagorskyi [ 2016 Nov 10 ]

The timeout for alerter is hardcoded to 40 seconds (as I recall), so sleeping alert scripts on failure could cause alerter processed to be too busy (I saw that a few times on different installations because of simply slow alert script).
So that workaround is risky.

Comment by Stefan Radman [ 2016 Nov 10 ]

I raised an issue through support and it got assigned.

Comment by Oleksii Zagorskyi [ 2016 Nov 13 ]

It would be also nice if SDTERR of alert script would be stored as error message (in alerts), similarly as we log errors for "Email" media type.

I use the this dummy script as alert media (and remmote command, below):

# cat /etc/zabbix/alertscripts/test.pl

#!/usr/bin/perl
#use warnings;
use strict;
print "done\n";
sdfg sf
 # /etc/zabbix/alertscripts/test.pl
done
Can't locate object method "sdfg" via package "sf" (perhaps you forgot to load "sf"?) at /etc/zabbix/alertscripts/test.pl line 6.

# /etc/zabbix/alertscripts/test.pl 1>/dev/null
Can't locate object method "sdfg" via package "sf" (perhaps you forgot to load "sf"?) at /etc/zabbix/alertscripts/test.pl line 6.

# /etc/zabbix/alertscripts/test.pl 2>/dev/null
done
  
 20676:20161113:162315.421 __zbx_zbx_setproctitle() title:'alerter [sending alerts]'
 20676:20161113:162315.421 query [txnlev:0] [select a.alertid,a.mediatypeid,a.sendto,a.subject,a.message,a.status,mt.mediatypeid,mt.type,mt.description,mt.smtp_server,mt.smtp_helo,mt.smtp_email,mt.exec_path,mt.gsm_modem,mt.username,mt.passwd,mt.smtp_port,mt.smtp_security,mt.smtp_verify_peer,mt.smtp_verify_host,mt.smtp_authentication,mt.exec_params,a.retries from alerts a,media_type mt where a.mediatypeid=mt.mediatypeid and a.status=0 and a.alerttype=0 order by a.alertid]
 20676:20161113:162315.422 In execute_action(): alertid [130] mediatype [0]
 20676:20161113:162315.422 In send_email() smtp_server:'mail.zabbix.com' smtp_port:25 smtp_security:0 smtp_authentication:0
 20676:20161113:162320.773 End of send_email():SUCCEED
 20676:20161113:162320.773 End of execute_action():SUCCEED
 20676:20161113:162320.773 alert ID [130] was sent successfully
 20676:20161113:162320.773 query without transaction detected
 20676:20161113:162320.773 query [txnlev:0] [update alerts set status=1,error='' where alertid=130]
 20676:20161113:162320.774 In execute_action(): alertid [131] mediatype [1]
 20676:20161113:162320.774 In zbx_popen() command:'/etc/zabbix/alertscripts/test.pl'
 20676:20161113:162320.775 End of zbx_popen():9
 20838:20161113:162320.776 zbx_popen(): executing script
 20676:20161113:162320.779 In zbx_waitpid()
 20676:20161113:162320.779 zbx_waitpid() exited, status:255
 20676:20161113:162320.779 End of zbx_waitpid():20838
 20676:20161113:162320.779 test.pl output:
Can't locate object method "sdfg" via package "sf" (perhaps you forgot to load "sf"?) at /etc/zabbix/alertscripts/test.pl line 4.
done

 20676:20161113:162320.779 End of execute_action():SUCCEED
 20676:20161113:162320.779 alert ID [131] was sent successfully
 20676:20161113:162320.779 query without transaction detected
 20676:20161113:162320.779 query [txnlev:0] [update alerts set status=1,error='' where alertid=131]
 20676:20161113:162320.780 __zbx_zbx_setproctitle() title:'alerter [sent alerts: 2 success, 0 fail in 5.358552 sec, idle 5 sec]'  

And for consistency .... remote command (performed by escalator, not alerter) should be taken into account too, because they even do not log the STDERR to server log, which is not very nice.
So, to be consistent, wold be good to check exit code for remote command too, print STDERR to the log and store in alerts.

  
 20689:20161113:162306.866 Conditions match our event. Execute operation.
 20689:20161113:162306.866 In execute_commands()
 20689:20161113:162306.866 query [txnlev:1] [select distinct h.hostid,h.host,o.type,o.scriptid,o.execute_on,o.port,o.authtype,o.username,o.password,o.publickey,o.privatekey,o.command,h.tls_connect,h.ipmi_authtype,h.ipmi_privilege,h.ipmi_username,h.ipmi_password from opcommand o,opcommand_grp og,hosts_groups hg,hosts h where o.operationid=og.operationid and og.groupid=hg.groupid and hg.hostid=h.hostid and o.operationid=17 and h.status=0 union select distinct h.hostid,h.host,o.type,o.scriptid,o.execute_on,o.port,o.authtype,o.username,o.password,o.publickey,o.privatekey,o.command,h.tls_connect,h.ipmi_authtype,h.ipmi_privilege,h.ipmi_username,h.ipmi_password from opcommand o,opcommand_hst oh,hosts h where o.operationid=oh.operationid and oh.hostid=h.hostid and o.operationid=17 and h.status=0 union select distinct 0,null,o.type,o.scriptid,o.execute_on,o.port,o.authtype,o.username,o.password,o.publickey,o.privatekey,o.command,1,0,2,null,null from opcommand o,opcommand_hst oh where o.operationid=oh.operationid and o.operationid=17 and oh.hostid is null]
 20689:20161113:162306.868 In substitute_simple_macros() data:'/etc/zabbix/alertscripts/test.pl'
 20689:20161113:162306.868 In zbx_execute_script()
 20689:20161113:162306.868 In zbx_popen() command:'/etc/zabbix/alertscripts/test.pl'
 20689:20161113:162306.870 End of zbx_popen():9
 20827:20161113:162306.870 zbx_popen(): executing script
 20689:20161113:162306.878 In zbx_waitpid()
 20689:20161113:162306.878 zbx_waitpid() exited, status:255
 20689:20161113:162306.878 End of zbx_waitpid():20827
 20689:20161113:162306.878 End of zbx_execute_script():SUCCEED
 20689:20161113:162306.878 In add_command_alert()
 20689:20161113:162306.878 End of add_command_alert()
 20689:20161113:162306.878 In DCget_nextid() table:'alerts' num:1
 20689:20161113:162306.878 End of DCget_nextid() table:'alerts' [128:128]
 20689:20161113:162306.878 query [txnlev:1] [insert into alerts (alertid,actionid,eventid,clock,message,status,error,esc_step,alerttype) values (128,3,1498,1479046986,':/etc/zabbix/alertscripts/test.pl',1,'',1,1);]
 20689:20161113:162306.879 End of execute_commands()
...
 20689:20161113:162306.879 Conditions match our event. Execute operation.
 20689:20161113:162306.879 In execute_commands()
 20689:20161113:162306.879 query [txnlev:1] [select distinct h.hostid,h.host,o.type,o.scriptid,o.execute_on,o.port,o.authtype,o.username,o.password,o.publickey,o.privatekey,o.command,h.tls_connect,h.ipmi_authtype,h.ipmi_privilege,h.ipmi_username,h.ipmi_password from opcommand o,opcommand_grp og,hosts_groups hg,hosts h where o.operationid=og.operationid and og.groupid=hg.groupid and hg.hostid=h.hostid and o.operationid=18 and h.status=0 union select distinct h.hostid,h.host,o.type,o.scriptid,o.execute_on,o.port,o.authtype,o.username,o.password,o.publickey,o.privatekey,o.command,h.tls_connect,h.ipmi_authtype,h.ipmi_privilege,h.ipmi_username,h.ipmi_password from opcommand o,opcommand_hst oh,hosts h where o.operationid=oh.operationid and oh.hostid=h.hostid and o.operationid=18 and h.status=0 union select distinct 0,null,o.type,o.scriptid,o.execute_on,o.port,o.authtype,o.username,o.password,o.publickey,o.privatekey,o.command,1,0,2,null,null from opcommand o,opcommand_hst oh where o.operationid=oh.operationid and o.operationid=18 and oh.hostid is null]
 20689:20161113:162306.881 In substitute_simple_macros() data:'someeeeee'
 20689:20161113:162306.881 In zbx_execute_script()
 20689:20161113:162306.882 In zbx_popen() command:'someeeeee'
 20689:20161113:162306.883 End of zbx_popen():9
 20829:20161113:162306.883 zbx_popen(): executing script
 20689:20161113:162306.885 In zbx_waitpid()
 20689:20161113:162306.885 zbx_waitpid() exited, status:127
 20689:20161113:162306.885 End of zbx_waitpid():20829
 20689:20161113:162306.885 End of zbx_execute_script():SUCCEED
 20689:20161113:162306.885 In add_command_alert()
 20689:20161113:162306.885 End of add_command_alert()
 20689:20161113:162306.885 In DCget_nextid() table:'alerts' num:1
 20689:20161113:162306.885 End of DCget_nextid() table:'alerts' [129:129]
 20689:20161113:162306.885 query [txnlev:1] [insert into alerts (alertid,actionid,eventid,clock,message,status,error,esc_step,alerttype) values (129,3,1498,1479046986,':someeeeee',1,'',1,1);]
 20689:20161113:162306.886 End of execute_commands()  

How does it look like currently in 3.0.5 frontend, falsely saying they were ok:

Comment by richlv [ 2016 Nov 13 ]

remote commands might be much harder - server does not wait for the command to be executed. it just sends the command to the agent and gets back '1' (see system.run[nowait,<command>])

Comment by Oleksii Zagorskyi [ 2016 Nov 14 ]

Rich, for commands executed on agent side - that's correct and you are right. I don't consider such case here.
In previous comment I meant "remote commands" executed on zabbix server, as shown in the debug log above.
Doc https://www.zabbix.com/documentation/3.0/manual/config/notifications/action/operation/remote_command says:

On Zabbix server remote commands are executed with timeout as set in the TrapperTimeout parameter of zabbix_server.conf file

Comment by Alexei Vladishev [ 2016 Dec 05 ]

I hope it will be implemented in Zabbix 3.4, it's in the roadmap.

Comment by Volker Fröhlich [ 2016 Dec 05 ]

Can you please publish this roadmap, maybe here?

http://zabbix.org/mw/index.php?title=Docs/roadmap

Comment by Oleksii Zagorskyi [ 2016 Dec 08 ]

Thinking about, for example, an alert script, used to sent SMSs through a sms-gateway in the Internet, where connections to the gateway randomly could be failed before timeout ...

Just a question - Zabbix uses ALERT_MAX_RETRIES=3 to send alerts.
Does it make any sense to do the same thing for custom alert scripts when it exits with error code (currently it applies only for timeout of script execution) ?

Also, thinking about the 40 seconds hardcoded timeout to be configurable ...

Comment by Stefan Radman [ 2016 Dec 08 ]

No, I don't think it makes sense to retry when the script returns an error code, with one possible exception being exit code 75:

$ grep -A3 TEMPFAIL /usr/include/sysexits.h 
 *	EX_TEMPFAIL -- temporary failure, indicating something that
 *		is not really an error.  In sendmail, this means
 *		that a mailer (e.g.) could not create a connection,
 *		and the request should be reattempted later.
--
#define EX_TEMPFAIL	75	/* temp failure; user is invited to retry */
#define EX_PROTOCOL	76	/* remote error in protocol */
#define EX_NOPERM	77	/* permission denied */
#define EX_CONFIG	78	/* configuration error */

See also: https://tools.ietf.org/html/rfc3463#section-2
Maybe also code 11:

$ grep EAGAIN /usr/include/asm-generic/errno-base.h
#define	EAGAIN		11	/* Try again */

Configurable timeout (and possibly retries) would definitely be an asset.

Comment by Vjaceslavs Bogdanovs [ 2017 Jan 02 ]

Ready for testing in development branch svn://svn.zabbix.com/branches/dev/ZBXNEXT-1380

Comment by Vladislavs Sokurenko [ 2017 Jan 03 ]

(1) [PSA] Memory leak.

output is only freed on SUCCEED, otherwise address is lost and memory is not freed.

	if (SUCCEED == (rc = zbx_execute_script(&host, &script, NULL, &output, error, sizeof(error))))
	{
		zabbix_log(LOG_LEVEL_DEBUG, "%s output:\n%s", script.command, error);
		zbx_free(output);
	}

While it is also allocated on not SUCCEED.

	if (SUCCEED != ret && NULL != result)
		*result = zbx_strdup(*result, "");

vjaceslavs RESOLVED in r64863

vso CLOSED, please review small style fixes in r64868

Comment by Vladislavs Sokurenko [ 2017 Jan 03 ]

(2) [PSA] Unused variable added.
output variable is allocated and freed, however it is not used.

	if (SUCCEED == (rc = zbx_execute_script(&host, &script, NULL, &output, error, sizeof(error))))
	{
		zabbix_log(LOG_LEVEL_DEBUG, "%s output:\n%s", script.command, error);
		zbx_free(output);
	}

vjaceslavs RESOLVED in r64863

vso CLOSED

Comment by Vladislavs Sokurenko [ 2017 Jan 03 ]

(3) Log is not formatted to zabbix style and misleading.

It says printing output while output variable is not printed, instead it prints error variable, and furthermore log is only executed on SUCCEED so there should be no error to print.
It will not be clear that command is printed when reading log, there is just command without description, also there are no apostrophe as in other places so not clear where it ends.
There are new line in the middle of log, there are no such logs in zabbix and this breaks consistency.

		zabbix_log(LOG_LEVEL_DEBUG, "%s output:\n%s", script.command, error);

Example logs:

zabbix_log(LOG_LEVEL_DEBUG, "In %s() host:'%s' hostid:%d", __function_name, host, hostid);
zabbix_log(LOG_LEVEL_DEBUG, "get value from agent result: '%s'", s.buffer);

vjaceslavs There was an error in code (2) - should be "output" (not an "error"). And log "style" was taken from alerter.c (126) to make consistent output so can't agree with "there are no such logs in zabbix". Should I change existing log style in alerter as well?

vso CLOSED, (2) fixes it, it's fine for long data.

Comment by Vladislavs Sokurenko [ 2017 Jan 03 ]

(4) When exec function return error, frontend still display success.

The exec() functions return only if an error has occurred, but we set success on error.
Please see below, the log say "execl() failed but EXIT_SUCCESS is returned."
Same is with setpgid in zbx_popen function.

	zabbix_log(LOG_LEVEL_DEBUG, "%s(): executing script", __function_name);

	execl("/bin/sh", "sh", "-c", command, NULL);

	/* execl() returns only when an error occurs */
	zabbix_log(LOG_LEVEL_WARNING, "execl() failed for [%s]: %s", command, zbx_strerror(errno));
	exit(EXIT_SUCCESS);

vjaceslavs RESOLVED in r64870.

vso CLOSED, now process exited with code: 1 is displayed.

Comment by Vladislavs Sokurenko [ 2017 Jan 03 ]

(5) If child did not terminate normally, still SUCCESS is returned.

The code below means if the child terminated normally and exit status is not success then fail.
else SUCCESS

But what if child did not terminate normally ? It shall also be failure.

else if (WIFEXITED(status) && EXIT_SUCCESS != WEXITSTATUS(status))
{
	...
}
else
	ret = SUCCEED;

vjaceslavs There are multiple options when this kind of "false positive" result can occur:
1. Process was terminated by a signal (no exit code to check).
2. Internal error (GetExitCodeProcess fails, etc.) where is no way to get status code.
So the question is: should we consider absence of exit code as an error code != 0. If we do so, then we should consider if we need to retry execution in this case as it can cause unpredictable results.

I suggest to add extra status code "UNKNOWN" for situations where script execution status is unknown (so we don't have "Done" in frontend).

vjaceslavs Even more, on unix-like systems, command is being executed from shell and even if signal occurs it is handled by shell and exit code is set to non-zero so this is a really specific case when shell crashes.

vjaceslavs It was decided to consider signaled state as a failure without altering the retry logic. RESOLVED in r65091

vso CLOSED

Comment by Vladislavs Sokurenko [ 2017 Jan 04 ]

(6) Script on zabbix_agent gets killed before it finish execution, if it executes longer than "Timeout" which is 3 by default and max 30 seconds.

Previously:
Remote commands on Zabbix agent are executed without timeout by the key

system.run[,nowait]

Meaning that users can have very long running scripts without being afraid that they will get timed out and killed.

Now:
On Zabbix agent remote commands are executed with timeout as set in the Timeout parameter of zabbix_agent.conf file.

Expected:
When user has script that executes longer than Timeout variable from config file, when he upgrades to new version, he must still be able to execute scripts as before, which is without time limit.

Actual:
All users who have script running longer than Timeout variable from config file will get their script killed and see "Timeout while executing a shell script." in the frontend.

This is blocker.

I suggest that such use cases need to be clearly defined in specification and solution discussed and approved.

Why this is happening ?
Output has been added to code below.

			if (SUCCEED == (rc = zbx_execute_script(&host, &script, NULL, &output, error, sizeof(error))))
				zabbix_log(LOG_LEVEL_DEBUG, "%s output:\n%s", script.command, output);

And in function zbx_execute_script_on_agent() it is used to change "nowait" to "wait

item.key = zbx_dsprintf(item.key, "system.run[%s,%s]", param, NULL == result ? "nowait" : "wait");

vjaceslavs There are no other options if we need to get output for log. So the only options are to drop the log to allow users to execute long running scripts or to check result code, but limit execution time.

I am not sure if we can consider that there is no need for exit code check in this case. I agree that behavior changes for multiple cases, but it is not really a blocker as user can create background scripts for cases like that (if it is an intended behavior).

Documentation for scripts does not provide information about this kind of behavior (execute on agent = async without limits, execute on server = limited by a timeout) so as far as I understand, I would say that current solution unifies solutions for server and agent.

Similar situation is with system.run item that will now start to check exit code and can become unsupported because of that.

vso Here is link from documentation Remote commands on Zabbix agent are executed without timeout
All I mean that changes/behavior must be very well documented and decision made. Current consequences are that if user has script that needs to be running infinitely or more than 30 seconds, it suddenly will not be possible, it will get killed at max 30 seconds but by default in 3 seconds.
The reason why I think that current solution for agent is not acceptable is because it will suddenly kill user scripts that previously were not killed and limit functionality of scripts to 30 seconds on agent. (killing scripts in the middle of work could also break something)
Also performance must be evaluated, if agent now waits for commands to execute and server wait for agent response.

I don't see any problems in checking result code on custom alertscript that is executed on server though.

vjaceslavs I added extra description about such cases in specification. We could also implement async remote command execution to bypass this timeout limit but it is not an easy fix so it should be evaluated.

vso Thanks ! Lets wait for approval.

vjaceslavs It was decided to ignore return code of commands executed on agent and to introduce new action status - "RAN" that means "Executed, but not checked". RESOLVED in r65091

vso CLOSED, though exact messages that need to be displayed are to be confirmed, but it will be minor change.

Comment by Vjaceslavs Bogdanovs [ 2017 Jan 13 ]

(7) Custom script (remote command in action -> operation) can be executed multiple times on server if Target list contains multiple records of current host.

Steps to reproduce:
1. Create Action
2. Add operation
3. Set Operation type to "Remote command"
4. Set Type to custom script
5. Set Execute on to "Server"
6. Add host to the Target list multiple times (for example Current host + Host <HOSTNAME>)

Action operation will be executed twice.

vjaceslavs RESOLVED in r65091

vso REOPENED. Fix must be reverted, it's frontend bug and this fix is not the way it can be handled, I suggest to make separate bug report.
Why is it not good solution ?
There can only be one server, that's why there shall be no host checks.
But after new changes there is additional SQL query called:

sql [select distinct h.hostid,h.host,h.tls_connect from functions f,items i,hosts h where f.itemid=i.itemid and i.hostid=h.hostid and h.status=0 and f.triggerid=13558]
  4341:20170116:153222.315 End of get_dynamic_hostid():SUCCEED

As well as bunch of other code for checking host duplicates etc..

All this is done to fill host parameter which in the end is not used by server anyway, and it can't be used because no command is executed on any host, it's executed on server.

case ZBX_SCRIPT_EXECUTE_ON_AGENT:
	ret = zbx_execute_script_on_agent(host, script->command, result,
			error, max_error_len);
	break;
case ZBX_SCRIPT_EXECUTE_ON_SERVER:
	ret = zbx_execute(script->command, result, error, max_error_len,
			CONFIG_TRAPPER_TIMEOUT);
	break;

vjaceslavs Documentation states that "A command on a host is executed only once, even if the host matches more than once (e.g. from several host groups; individually and from a host group)"
And yes, there can be only one server, but nevertheless there can be multiple hosts (host is being set for action operations even for Execute on = Server) and current solution is causing multiple script executions if host is specified multiple times:

How we treat this problem as a frontend bug?

vso CLOSED, Lets create separate ticket for that.

Comment by Vjaceslavs Bogdanovs [ 2017 Jan 13 ]

(8) Custom script (remote command in action -> operation) executed on server is shown without host in Reports -> Action log

This causes output like that (command starts with ":" as host is missing):

vjaceslavs RESOLVED in r65091

vso REOPENED for the same reason as (7) command is not executed on host so this message is not appropriate actually, that's why there was blank.

vjaceslavs Command is not executed on host, but it is executed "within host context" and all host related macros will expand to specified host. So we can't say that setting target host for execution on server doesn't change anything.
And this is the reason why we need to show host in the results.

vso Please correct me if I am wrong.
Hostid does not affect anything when custom script is executed on server. Macros are still resolved as if current host was set.
So when 2 or more different targets are set, they are actually duplicates of same command but will have different hostid and get executed more then one time.
Assuming this macro expanding is by design, then I would suggest to make sure that custom script for server is only executed once.
Since there can be only one target for custom script that is executed on server there is no need even to check for duplicates, or to determine host id, we simply must stop on first execute as we know that there can't be more than one, no matter how many are added they are still duplicates of the same command that will be executed on same server.

vjaceslavs Assuming macro expanding is correct there should be no option to specify targets for scripts executed on server as those values will never be used. So problem is present anyway and we should only find out the scope of the problem.

vso CLOSED, Lets create separate ticket for that.

Comment by Vladislavs Sokurenko [ 2017 Jan 17 ]

(9) Error message is never displayed to the user for a custom script operation in actions.
Steps:
In action operation detail set:
Type: Custom script
Commands, path to script that does not exist.
For example:
/home/vso/zabbix_eclipse_dev/ZBXNEXT-1380/share/zabbix/alertscripts/alert_script

Expected:
Message in frontend that is generated by shell:
"/home/vso/zabbix_eclipse_dev/ZBXNEXT-1380/share/zabbix/alertscripts/alert_script: not found"

Actual:
process exited with code: 127

Furthermore this means that buffer is set to NULL so no error messages will be red from custom script.

This means that in simple scenario where script could not be found, there will be no error message indicating exactly that.

vjaceslavs Added script output collection for all script executions so output can be present if error happens. RESOLVED in r65125

vso REOPENED, Double free corruption is one of the issues introduced but only occurs at specific scenarios.
The changes are to complex and buggy, please make a simpler solution so the code is clearer and there are no bugs.

*** Error in `./sbin/zabbix_agentd: listener #1 [processing request]': double free or corruption (top): 0x00000000023122a0 ***
======= Backtrace: =========
/lib/x86_64-linux-gnu/libc.so.6(+0x77725)[0x7fd94fdc0725]
/lib/x86_64-linux-gnu/libc.so.6(+0x7ff4a)[0x7fd94fdc8f4a]
/lib/x86_64-linux-gnu/libc.so.6(cfree+0x4c)[0x7fd94fdccabc]
./sbin/zabbix_agentd: listener #1 [processing request](EXECUTE_STR+0x89)[0x422479]
./sbin/zabbix_agentd: listener #1 [processing request](process+0x31c)[0x4171cc]
./sbin/zabbix_agentd: listener #1 [processing request](listener_thread+0x19d)[0x413aed]
./sbin/zabbix_agentd: listener #1 [processing request](zbx_thread_start+0x45)[0x42b135]
./sbin/zabbix_agentd: listener #1 [processing request](MAIN_ZABBIX_ENTRY+0x223)[0x416373]
./sbin/zabbix_agentd: listener #1 [processing request](daemon_start+0x1c3)[0x42be53]
./sbin/zabbix_agentd: listener #1 [processing request](main+0xb0)[0x40be20]
/lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0xf0)[0x7fd94fd69830]
./sbin/zabbix_agentd: listener #1 [processing request](_start+0x29)[0x40bf29]
======= Memory map: ========
00400000-00456000 r-xp 00000000 08:02 13763939                           /home/vso/zabbix_eclipse_dev/ZBXNEXT-1380/sbin/zabbix_agentd
00655000-00656000 r--p 00055000 08:02 13763939                           /home/vso/zabbix_eclipse_dev/ZBXNEXT-1380/sbin/zabbix_agentd
00656000-00658000 rw-p 00056000 08:02 13763939                           /home/vso/zabbix_eclipse_dev/ZBXNEXT-1380/sbin/zabbix_agentd
00658000-0065d000 rw-p 00000000 00:00 0 
022ef000-02330000 rw-p 00000000 00:00 0                                  [heap]
7fd944000000-7fd944021000 rw-p 00000000 00:00 0 
7fd944021000-7fd948000000 ---p 00000000 00:00 0 
7fd94b4ed000-7fd94b503000 r-xp 00000000 08:02 4460969                    /lib/x86_64-linux-gnu/libgcc_s.so.1
7fd94b503000-7fd94b702000 ---p 00016000 08:02 4460969                    /lib/x86_64-linux-gnu/libgcc_s.so.1
7fd94b702000-7fd94b703000 rw-p 00015000 08:02 4460969                    /lib/x86_64-linux-gnu/libgcc_s.so.1
7fd94b703000-7fd94b70c000 r-xp 00000000 08:02 4460941                    /lib/x86_64-linux-gnu/libcrypt-2.23.so
7fd94b70c000-7fd94b90b000 ---p 00009000 08:02 4460941                    /lib/x86_64-linux-gnu/libcrypt-2.23.so
7fd94b90b000-7fd94b90c000 r--p 00008000 08:02 4460941                    /lib/x86_64-linux-gnu/libcrypt-2.23.so
7fd94b90c000-7fd94b90d000 rw-p 00009000 08:02 4460941                    /lib/x86_64-linux-gnu/libcrypt-2.23.so
7fd94b90d000-7fd94b93b000 rw-p 00000000 00:00 0 
7fd94b93b000-7fd94ba0a000 r-xp 00000000 08:02 14165018                   /usr/lib/x86_64-linux-gnu/libsqlite3.so.0.8.6
7fd94ba0a000-7fd94bc0a000 ---p 000cf000 08:02 14165018                   /usr/lib/x86_64-linux-gnu/libsqlite3.so.0.8.6
7fd94bc0a000-7fd94bc0d000 r--p 000cf000 08:02 14165018                   /usr/lib/x86_64-linux-gnu/libsqlite3.so.0.8.6
7fd94bc0d000-7fd94bc0f000 rw-p 000d2000 08:02 14165018                   /usr/lib/x86_64-linux-gnu/libsqlite3.so.0.8.6
7fd94bc0f000-7fd94bc10000 rw-p 00000000 00:00 0 
7fd94bc10000-7fd94bc56000 r-xp 00000000 08:02 14164557                   /usr/lib/x86_64-linux-gnu/libhx509.so.5.0.0
7fd94bc56000-7fd94be56000 ---p 00046000 08:02 14164557                   /usr/lib/x86_64-linux-gnu/libhx509.so.5.0.0
7fd94be56000-7fd94be58000 r--p 00046000 08:02 14164557                   /usr/lib/x86_64-linux-gnu/libhx509.so.5.0.0
7fd94be58000-7fd94be5a000 rw-p 00048000 08:02 14164557                   /usr/lib/x86_64-linux-gnu/libhx509.so.5.0.0
7fd94be5a000-7fd94be5b000 rw-p 00000000 00:00 0 
7fd94be5b000-7fd94be69000 r-xp 00000000 08:02 14164539                   /usr/lib/x86_64-linux-gnu/libheimbase.so.1.0.0
7fd94be69000-7fd94c068000 ---p 0000e000 08:02 14164539                   /usr/lib/x86_64-linux-gnu/libheimbase.so.1.0.0
7fd94c068000-7fd94c069000 r--p 0000d000 08:02 14164539                   /usr/lib/x86_64-linux-gnu/libheimbase.so.1.0.0
7fd94c069000-7fd94c06a000 rw-p 0000e000 08:02 14164539                   /usr/lib/x86_64-linux-gnu/libheimbase.so.1.0.0
7fd94c06a000-7fd94c091000 r-xp 00000000 08:02 14165183                   /usr/lib/x86_64-linux-gnu/libwind.so.0.0.0
7fd94c091000-7fd94c291000 ---p 00027000 08:02 14165183                   /usr/lib/x86_64-linux-gnu/libwind.so.0.0.0
7fd94c291000-7fd94c292000 r--p 00027000 08:02 14165183                   /usr/lib/x86_64-linux-gnu/libwind.so.0.0.0
7fd94c292000-7fd94c293000 rw-p 00028000 08:02 14165183                   /usr/lib/x86_64-linux-gnu/libwind.so.0.0.0
7fd94c293000-7fd94c2a8000 r-xp 00000000 08:02 14164936                   /usr/lib/x86_64-linux-gnu/libroken.so.18.1.0
7fd94c2a8000-7fd94c4a7000 ---p 00015000 08:02 14164936                   /usr/lib/x86_64-linux-gnu/libroken.so.18.1.0
7fd94c4a7000-7fd94c4a8000 r--p 00014000 08:02 14164936                   /usr/lib/x86_64-linux-gnu/libroken.so.18.1.0
7fd94c4a8000-7fd94c4a9000 rw-p 00015000 08:02 14164936                   /usr/lib/x86_64-linux-gnu/libroken.so.18.1.0
7fd94c4a9000-7fd94c4d9000 r-xp 00000000 08:02 14164537                   /usr/lib/x86_64-linux-gnu/libhcrypto.so.4.1.0
7fd94c4d9000-7fd94c6d9000 ---p 00030000 08:02 14164537                   /usr/lib/x86_64-linux-gnu/libhcrypto.so.4.1.0
7fd94c6d9000-7fd94c6da000 r--p 00030000 08:02 14164537                   /usr/lib/x86_64-linux-gnu/libhcrypto.so.4.1.0
7fd94c6da000-7fd94c6db000 rw-p 00031000 08:02 14164537                   /usr/lib/x86_64-linux-gnu/libhcrypto.so.4.1.0
7fd94c6db000-7fd94c6dc000 rw-p 00000000 00:00 0 
7fd94c6dc000-7fd94c77b000 r-xp 00000000 08:02 14163981                   /usr/lib/x86_64-linux-gnu/libasn1.so.8.0.0
7fd94c77b000-7fd94c97a000 ---p 0009f000 08:02 14163981                   /usr/lib/x86_64-linux-gnu/libasn1.so.8.0.0
7fd94c97a000-7fd94c97b000 r--p 0009e000 08:02 14163981                   /usr/lib/x86_64-linux-gnu/libasn1.so.8.0.0
7fd94c97b000-7fd94c97e000 rw-p 0009f000 08:02 14163981                   /usr/lib/x86_64-linux-gnu/libasn1.so.8.0.0
7fd94c97e000-7fd94ca02000 r-xp 00000000 08:02 14164633                   /usr/lib/x86_64-linux-gnu/libkrb5.so.26.0.0
7fd94ca02000-7fd94cc01000 ---p 00084000 08:02 14164633                   /usr/lib/x86_64-linux-gnu/libkrb5.so.26.0.0
7fd94cc01000-7fd94cc04000 r--p 00083000 08:02 14164633                   /usr/lib/x86_64-linux-gnu/libkrb5.so.26.0.0
7fd94cc04000-7fd94cc07000 rw-p 00086000 08:02 14164633                   /usr/lib/x86_64-linux-gnu/libkrb5.so.26.0.0
7fd94cc07000-7fd94cc08000 rw-p 00000000 00:00 0 
7fd94cc08000-7fd94cc10000 r-xp 00000000 08:02 14164541                   /usr/lib/x86_64-linux-gnu/libheimntlm.so.0.1.0
7fd94cc10000-7fd94ce0f000 ---p 00008000 08:02 14164541                   /usr/lib/x86_64-linux-gnu/libheimntlm.so.0.1.0
7fd94ce0f000-7fd94ce10000 r--p 00007000 08:02 14164541                   /usr/lib/x86_64-linux-gnu/libheimntlm.so.0.1.0
7fd94ce10000-7fd94ce11000 rw-p 00008000 08:02 14164541                   /usr/lib/x86_64-linux-gnu/libheimntlm.so.0.1.0
7fd94ce11000-7fd94ce14000 r-xp 00000000 08:02 4460992                    /lib/x86_64-linux-gnu/libkeyutils.so.1.5
7fd94ce14000-7fd94d013000 ---p 00003000 08:02 4460992                    /lib/x86_64-linux-gnu/libkeyutils.so.1.5
7fd94d013000-7fd94d014000 r--p 00002000 08:02 4460992                    /lib/x86_64-linux-gnu/libkeyutils.so.1.5
7fd94d014000-7fd94d015000 rw-p 00003000 08:02 4460992                    /lib/x86_64-linux-gnu/libkeyutils.so.1.5
7fd94d015000-7fd94d01c000 r-xp 00000000 08:02 14164274                   /usr/lib/x86_64-linux-gnu/libffi.so.6.0.4
7fd94d01c000-7fd94d21b000 ---p 00007000 08:02 14164274                   /usr/lib/x86_64-linux-gnu/libffi.so.6.0.4
7fd94d21b000-7fd94d21c000 r--p 00006000 08:02 14164274                   /usr/lib/x86_64-linux-gnu/libffi.so.6.0.4
7fd94d21c000-7fd94d21d000 rw-p 00007000 08:02 14164274                   /usr/lib/x86_64-linux-gnu/libffi.so.6.0.4
7fd94d21d000-7fd94d25a000 r-xp 00000000 08:02 14164462                   /usr/lib/x86_64-linux-gnu/libgssapi.so.3.0.0
7fd94d25a000-7fd94d45a000 ---p 0003d000 08:02 14164462                   /usr/lib/x86_64-linux-gnu/libgssapi.so.3.0.0
7fd94d45a000-7fd94d45b000 r--p 0003d000 08:02 14164462                   /usr/lib/x86_64-linux-gnu/libgssapi.so.3.0.0
7fd94d45b000-7fd94d45d000 rw-p 0003e000 08:02 14164462                   /usr/lib/x86_64-linux-gnu/libgssapi.so.3.0.0
7fd94d45d000-7fd94d45e000 rw-p 00000000 00:00 0 
7fd94d45e000-7fd94d477000 r-xp 00000000 08:02 14164963                   /usr/lib/x86_64-linux-gnu/libsasl2.so.2.0.25
7fd94d477000-7fd94d677000 ---p 00019000 08:02 14164963                   /usr/lib/x86_64-linux-gnu/libsasl2.so.2.0.25
7fd94d677000-7fd94d678000 r--p 00019000 08:02 14164963                   /usr/lib/x86_64-linux-gnu/libsasl2.so.2.0.25
7fd94d678000-7fd94d679000 rw-p 0001a000 08:02 14164963                   /usr/lib/x86_64-linux-gnu/libsasl2.so.2.0.25
7fd94d679000-7fd94d683000 r-xp 00000000 08:02 14164637                   /usr/lib/x86_64-linux-gnu/libkrb5support.so.0.1
7fd94d683000-7fd94d882000 ---p 0000a000 08:02 14164637                   /usr/lib/x86_64-linux-gnu/libkrb5support.so.0.1
7fd94d882000-7fd94d883000 r--p 00009000 08:02 14164637                   /usr/lib/x86_64-linux-gnu/libkrb5support.so.0.1
7fd94d883000-7fd94d884000 rw-p 0000a000 08:02 14164637                   /usr/lib/x86_64-linux-gnu/libkrb5support.so.0.1
7fd94d884000-7fd94d887000 r-xp 00000000 08:02 4460940                    /lib/x86_64-linux-gnu/libcom_err.so.2.1
7fd94d887000-7fd94da86000 ---p 00003000 08:02 4460940                    /lib/x86_64-linux-gnu/libcom_err.so.2.1
7fd94da86000-7fd94da87000 r--p 00002000 08:02 4460940                    /lib/x86_64-linux-gnu/libcom_err.so.2.1
7fd94da87000-7fd94da88000 rw-p 00003000 08:02 4460940                    /lib/x86_64-linux-gnu/libcom_err.so.2.1
7fd94da88000-7fd94dab4000 r-xp 00000000 08:02 14164629                   /usr/lib/x86_64-linux-gnu/libk5crypto.so.3.1
7fd94dab4000-7fd94dcb3000 ---p 0002c000 08:02 14164629                   /usr/lib/x86_64-linux-gnu/libk5crypto.so.3.1
7fd94dcb3000-7fd94dcb5000 r--p 0002b000 08:02 14164629                   /usr/lib/x86_64-linux-gnu/libk5crypto.so.3.1
7fd94dcb5000-7fd94dcb6000 rw-p 0002d000 08:02 14164629                   /usr/lib/x86_64-linux-gnu/libk5crypto.so.3.1
7fd94dcb6000-7fd94dcb7000 rw-p 00000000 00:00 0 
7fd94dcb7000-7fd94dd7a000 r-xp 00000000 08:02 14164635                   /usr/lib/x86_64-linux-gnu/libkrb5.so.3.3
7fd94dd7a000-7fd94df7a000 ---p 000c3000 08:02 14164635                   /usr/lib/x86_64-linux-gnu/libkrb5.so.3.3
7fd94df7a000-7fd94df87000 r--p 000c3000 08:02 14164635                   /usr/lib/x86_64-linux-gnu/libkrb5.so.3.3
7fd94df87000-7fd94df89000 rw-p 000d0000 08:02 14164635                   /usr/lib/x86_64-linux-gnu/libkrb5.so.3.3
7fd94df89000-7fd94df9a000 r-xp 00000000 08:02 14165035                   /usr/lib/x86_64-linux-gnu/libtasn1.so.6.5.1
7fd94df9a000-7fd94e19a000 ---p 00011000 08:02 14165035                   /usr/lib/x86_64-linux-gnu/libtasn1.so.6.5.1
7fd94e19a000-7fd94e19b000 r--p 00011000 08:02 14165035                   /usr/lib/x86_64-linux-gnu/libtasn1.so.6.5.1
7fd94e19b000-7fd94e19c000 rw-p 00012000 08:02 14165035                   /usr/lib/x86_64-linux-gnu/libtasn1.so.6.5.1
7fd94e19c000-7fd94e1f5000 r-xp 00000000 08:02 14155987                   /usr/lib/x86_64-linux-gnu/libp11-kit.so.0.1.0
7fd94e1f5000-7fd94e3f4000 ---p 00059000 08:02 14155987                   /usr/lib/x86_64-linux-gnu/libp11-kit.so.0.1.0
7fd94e3f4000-7fd94e3fe000 r--p 00058000 08:02 14155987                   /usr/lib/x86_64-linux-gnu/libp11-kit.so.0.1.0
7fd94e3fe000-7fd94e400000 rw-p 00062000 08:02 14155987                   /usr/lib/x86_64-linux-gnu/libp11-kit.so.0.1.0
7fd94e400000-7fd94e47f000 r-xp 00000000 08:02 14164407                   /usr/lib/x86_64-linux-gnu/libgmp.so.10.3.0
7fd94e47f000-7fd94e67e000 ---p 0007f000 08:02 14164407                   /usr/lib/x86_64-linux-gnu/libgmp.so.10.3.0
7fd94e67e000-7fd94e67f000 r--p 0007e000 08:02 14164407                   /usr/lib/x86_64-linux-gnu/libgmp.so.10.3.0
7fd94e67f000-7fd94e680000 rw-p 0007f000 08:02 14164407                   /usr/lib/x86_64-linux-gnu/libgmp.so.10.3.0
7fd94e680000-7fd94e6b2000 r-xp 00000000 08:02 14164543                   /usr/lib/x86_64-linux-gnu/libhogweed.so.4.2
7fd94e6b2000-7fd94e8b1000 ---p 00032000 08:02 14164543                   /usr/lib/x86_64-linux-gnu/libhogweed.so.4.2
7fd94e8b1000-7fd94e8b2000 r--p 00031000 08:02 14164543                   /usr/lib/x86_64-linux-gnu/libhogweed.so.4.2
7fd94e8b2000-7fd94e8b3000 rw-p 00032000 08:02 14164543                   /usr/lib/x86_64-linux-gnu/libhogweed.so.4.2
7fd94e8b3000-7fd94e8cb000 r-xp 00000000 08:02 4461077                    /lib/x86_64-linux-gnu/libpthread-2.23.so
7fd94e8cb000-7fd94eaca000 ---p 00018000 08:02 4461077                    /lib/x86_64-linux-gnu/libpthread-2.23.so
7fd94eaca000-7fd94eacb000 r--p 00017000 08:02 4461077                    /lib/x86_64-linux-gnu/libpthread-2.23.so
7fd94eacb000-7fd94eacc000 rw-p 00018000 08:02 4461077                    /lib/x86_64-linux-gnu/libpthread-2.23.so
7fd94eacc000-7fd94ead0000 rw-p 00000000 00:00 0 
7fd94ead0000-7fd94eae9000 r-xp 00000000 08:02 4461122                    /lib/x86_64-linux-gnu/libz.so.1.2.8
7fd94eae9000-7fd94ece8000 ---p 00019000 08:02 4461122                    /lib/x86_64-linux-gnu/libz.so.1.2.8
7fd94ece8000-7fd94ece9000 r--p 00018000 08:02 4461122                    /lib/x86_64-linux-gnu/libz.so.1.2.8
7fd94ece9000-7fd94ecea000 rw-p 00019000 08:02 4461122                    /lib/x86_64-linux-gnu/libz.so.1.2.8
7fd94ecea000-7fd94ed37000 r-xp 00000000 08:02 14164648                   /usr/lib/x86_64-linux-gnu/libldap_r-2.4.so.2.10.5
7fd94ed37000-7fd94ef36000 ---p 0004d000 08:02 14164648                   /usr/lib/x86_64-linux-gnu/libldap_r-2.4.so.2.10.5
7fd94ef36000-7fd94ef38000 r--p 0004c000 08:02 14164648                   /usr/lib/x86_64-linux-gnu/libldap_r-2.4.so.2.10.5
7fd94ef38000-7fd94ef39000 rw-p 0004e000 08:02 14164648                   /usr/lib/x86_64-linux-gnu/libldap_r-2.4.so.2.10.5
7fd94ef39000-7fd94ef3b000 rw-p 00000000 00:00 0 
7fd94ef3b000-7fd94ef48000 r-xp 00000000 08:02 14164643                   /usr/lib/x86_64-linux-gnu/liblber-2.4.so.2.10.5
7fd94ef48000-7fd94f148000 ---p 0000d000 08:02 14164643                   /usr/lib/x86_64-linux-gnu/liblber-2.4.so.2.10.5
7fd94f148000-7fd94f149000 r--p 0000d000 08:02 14164643                   /usr/lib/x86_64-linux-gnu/liblber-2.4.so.2.10.5
7fd94f149000-7fd94f14a000 rw-p 0000e000 08:02 14164643                   /usr/lib/x86_64-linux-gnu/liblber-2.4.so.2.10.5
7fd94f14a000-7fd94f191000 r-xp 00000000 08:02 14164464                   /usr/lib/x86_64-linux-gnu/libgssapi_krb5.so.2.2
7fd94f191000-7fd94f390000 ---p 00047000 08:02 14164464                   /usr/lib/x86_64-linux-gnu/libgssapi_krb5.so.2.2
7fd94f390000-7fd94f392000 r--p 00046000 08:02 14164464                   /usr/lib/x86_64-linux-gnu/libgssapi_krb5.so.2.2
7fd94f392000-7fd94f394000 rw-p 00048000 08:02 14164464                   /usr/lib/x86_64-linux-gnu/libgssapi_krb5.so.2.2
7fd94f394000-7fd94f4b7000 r-xp 00000000 08:02 14164425                   /usr/lib/x86_64-linux-gnu/libgnutls.so.30.6.2
7fd94f4b7000-7fd94f6b6000 ---p 00123000 08:02 14164425                   /usr/lib/x86_64-linux-gnu/libgnutls.so.30.6.2
7fd94f6b6000-7fd94f6c1000 r--p 00122000 08:02 14164425                   /usr/lib/x86_64-linux-gnu/libgnutls.so.30.6.2
7fd94f6c1000-7fd94f6c3000 rw-p 0012d000 08:02 14164425                   /usr/lib/x86_64-linux-gnu/libgnutls.so.30.6.2
7fd94f6c3000-7fd94f6c4000 rw-p 00000000 00:00 0 
7fd94f6c4000-7fd94f6f8000 r-xp 00000000 08:02 14164753                   /usr/lib/x86_64-linux-gnu/libnettle.so.6.2
7fd94f6f8000-7fd94f8f7000 ---p 00034000 08:02 14164753                   /usr/lib/x86_64-linux-gnu/libnettle.so.6.2
7fd94f8f7000-7fd94f8f9000 r--p 00033000 08:02 14164753                   /usr/lib/x86_64-linux-gnu/libnettle.so.6.2
7fd94f8f9000-7fd94f8fa000 rw-p 00035000 08:02 14164753                   /usr/lib/x86_64-linux-gnu/libnettle.so.6.2
7fd94f8fa000-7fd94f915000 r-xp 00000000 08:02 14164944                   /usr/lib/x86_64-linux-gnu/librtmp.so.1
7fd94f915000-7fd94fb14000 ---p 0001b000 08:02 14164944                   /usr/lib/x86_64-linux-gnu/librtmp.so.1
7fd94fb14000-7fd94fb15000 r--p 0001a000 08:02 14164944                   /usr/lib/x86_64-linux-gnu/librtmp.so.1
7fd94fb15000-7fd94fb16000 rw-p 0001b000 08:02 14164944                   /usr/lib/x86_64-linux-gnu/librtmp.so.1
7fd94fb16000-7fd94fb47000 r-xp 00000000 08:02 14156253                   /usr/lib/x86_64-linux-gnu/libidn.so.11.6.15
7fd94fb47000-7fd94fd47000 ---p 00031000 08:02 14156253                   /usr/lib/x86_64-linux-gnu/libidn.so.11.6.15
7fd94fd47000-7fd94fd48000 r--p 00031000 08:02 14156253                   /usr/lib/x86_64-linux-gnu/libidn.so.11.6.15
7fd94fd48000-7fd94fd49000 rw-p 00032000 08:02 14156253                   /usr/lib/x86_64-linux-gnu/libidn.so.11.6.15
7fd94fd49000-7fd94ff09000 r-xp 00000000 08:02 4460931                    /lib/x86_64-linux-gnu/libc-2.23.so
7fd94ff09000-7fd950108000 ---p 001c0000 08:02 4460931                    /lib/x86_64-linux-gnu/libc-2.23.so
7fd950108000-7fd95010c000 r--p 001bf000 08:02 4460931                    /lib/x86_64-linux-gnu/libc-2.23.so
7fd95010c000-7fd95010e000 rw-p 001c3000 08:02 4460931                    /lib/x86_64-linux-gnu/libc-2.23.so
7fd95010e000-7fd950112000 rw-p 00000000 00:00 0 
7fd950112000-7fd950129000 r-xp 00000000 08:02 4461083                    /lib/x86_64-linux-gnu/libresolv-2.23.so
7fd950129000-7fd950329000 ---p 00017000 08:02 4461083                    /lib/x86_64-linux-gnu/libresolv-2.23.so
7fd950329000-7fd95032a000 r--p 00017000 08:02 4461083                    /lib/x86_64-linux-gnu/libresolv-2.23.so
7fd95032a000-7fd95032b000 rw-p 00018000 08:02 4461083                    /lib/x86_64-linux-gnu/libresolv-2.23.so
7fd95032b000-7fd95032d000 rw-p 00000000 00:00 0 
7fd95032d000-7fd950330000 r-xp 00000000 08:02 4460955                    /lib/x86_64-linux-gnu/libdl-2.23.so
7fd950330000-7fd95052f000 ---p 00003000 08:02 4460955                    /lib/x86_64-linux-gnu/libdl-2.23.so
7fd95052f000-7fd950530000 r--p 00002000 08:02 4460955                    /lib/x86_64-linux-gnu/libdl-2.23.so
7fd950530000-7fd950531000 rw-p 00003000 08:02 4460955                    /lib/x86_64-linux-gnu/libdl-2.23.so
7fd950531000-7fd95059a000 r-xp 00000000 08:02 14166919                   /usr/lib/x86_64-linux-gnu/libcurl-gnutls.so.4.4.0
7fd95059a000-7fd95079a000 ---p 00069000 08:02 14166919                   /usr/lib/x86_64-linux-gnu/libcurl-gnutls.so.4.4.0
7fd95079a000-7fd95079d000 r--p 00069000 08:02 14166919                   /usr/lib/x86_64-linux-gnu/libcurl-gnutls.so.4.4.0
7fd95079d000-7fd95079e000 rw-p 0006c000 08:02 14166919                   /usr/lib/x86_64-linux-gnu/libcurl-gnutls.so.4.4.0
7fd95079e000-7fd9507c4000 r-xp 00000000 08:02 4460903                    /lib/x86_64-linux-gnu/ld-2.23.so
7fd9508f1000-7fd950992000 rw-s 00000000 00:05 35684393                   /SYSV6c020209 (deleted)
7fd950992000-7fd9509a3000 rw-p 00000000 00:00 0 
7fd9509c0000-7fd9509c1000 rw-p 00000000 00:00 0 
7fd9509c1000-7fd9509c3000 rw-p 00000000 00:00 0 
7fd9509c3000-7fd9509c4000 r--p 00025000 08:02 4460903                    /lib/x86_64-linux-gnu/ld-2.23.so
7fd9509c4000-7fd9509c5000 rw-p 00026000 08:02 4460903                    /lib/x86_64-linux-gnu/ld-2.23.so
7fd9509c5000-7fd9509c6000 rw-p 00000000 00:00 0 
7ffdceab2000-7ffdcead3000 rw-p 00000000 00:00 0                          [stack]
7ffdceb43000-7ffdceb45000 r--p 00000000 00:00 0                          [vvar]
7ffdceb45000-7ffdceb47000 r-xp 00000000 00:00 0                          [vdso]
ffffffffff600000-ffffffffff601000 r-xp 00000000 00:00 0                  [vsyscall]
 23302:20170117:115534.330 One child process died (PID:23305,exitcode/signal:6). Exiting ...
zabbix_agentd [23302]: Error waiting for process with PID 23305: [10] No child processes
 23302:20170117:115534.330 Zabbix Agent stopped. Zabbix 3.3.0 (revision {ZABBIX_REVISION}).

vjaceslavs Sorry, I missed the fact that zbx_free alters the value of the pointer. All bugs produced are not intended (even if it seems so), but thanks for advice to make code without bugs. As for simplicity, this solution is the simplest way to get output when errors occur. RESOLVED in r65131.

vso REOPENED, there is memory leak if zbx_execute is called multiple times with same pointer.

==27414== 131,072 bytes in 1 blocks are definitely lost in loss record 25 of 26
==27414==    at 0x4C2FD5F: realloc (in /usr/lib/valgrind/vgpreload_memcheck-amd64-linux.so)
==27414==    by 0x4317FA: zbx_realloc2 (misc.c:499)
==27414==    by 0x4353FF: zbx_strncpy_alloc (str.c:312)
==27414==    by 0x43ECA5: zbx_execute (execute.c:426)
==27414==    by 0x41E9EE: SYSTEM_SW_PACKAGES (software.c:226)
==27414==    by 0x4171CB: process (sysinfo.c:641)
==27414==    by 0x413AEC: process_listener (listener.c:58)
==27414==    by 0x413AEC: listener_thread (listener.c:145)
==27414==    by 0x42B184: zbx_thread_start (threads.c:128)
==27414==    by 0x416372: MAIN_ZABBIX_ENTRY (zabbix_agentd.c:966)
==27414==    by 0x42BEA2: daemon_start (daemon.c:392)
==27414==    by 0x40BE1F: main (zabbix_agentd.c:1203)

vjaceslavs Restored previous logic when free is applied to the param value. RESOLVED in r65132

vso CLOSED with small style fixes in r65133

Comment by Vladislavs Sokurenko [ 2017 Jan 17 ]

(10) IPMI will no be possible to execute on server if not current server target is specified.
Check has been added not to copy communication parameters if executed on server, however if it's not custom script but for example IPMI then server must copy communication parameters.

if (ZBX_SCRIPT_EXECUTE_ON_SERVER != script.execute_on)
			{
				host.tls_connect = (unsigned char)atoi(row[12]);
#ifdef HAVE_OPENIPMI
				host.ipmi_authtype = (signed char)atoi(row[13]);
				host.ipmi_privilege = (unsigned char)atoi(row[14]);
				strscpy(host.ipmi_username, row[15]);
				strscpy(host.ipmi_password, row[16]);
#endif
#if defined(HAVE_POLARSSL) || defined(HAVE_GNUTLS) || defined(HAVE_OPENSSL)
				strscpy(host.tls_issuer, row[13 + ZBX_IPMI_FIELDS_NUM]);
				strscpy(host.tls_subject, row[14 + ZBX_IPMI_FIELDS_NUM]);
				strscpy(host.tls_psk_identity, row[15 + ZBX_IPMI_FIELDS_NUM]);
				strscpy(host.tls_psk, row[16 + ZBX_IPMI_FIELDS_NUM]);
#endif

vjaceslavs Check is valid. script.execute_on is initialized only for custom script.

vso CLOSED, yes you are right.

Comment by Vladislavs Sokurenko [ 2017 Jan 17 ]

(11) [F][S] Appropriate message for scripts executed on agent shall be added.

Currently message is "Ran":

Decision shall be made on how it must look.
But I personally would prefer that when executed on agent then it would always say "Executed"
When on server "Succeeded" or "Failed".
For mail and media script "Sent" or "Failed" or "In progress"

vso It was decided:
"Executed" - successfully sent to agent or successfully executed on server.
"Failed" - otherwise.

vjaceslavs Reverted to single status for execution with or without checks
Looks like that for same script executed on agent and server:

RESOLVED in r65163

vso CLOSED, with small style fix in r65165

Comment by Vladislavs Sokurenko [ 2017 Jan 19 ]

(12) zbx_execute release previously allocated resources only in the end, while not using those, this is not very effective.

Before
Logic in zbx_executed was to reallocate previously allocated buffer and then continue with it.

Now:
Allocate new buffer, and free previously allocated buffer in the end.

vjaceslavs RESOLVED in r65175

vso CLOSED

Comment by Vladislavs Sokurenko [ 2017 Jan 19 ]

Successfully tested

Comment by Vladislavs Sokurenko [ 2017 Jan 19 ]

(13) [D] This need to be very well documented, especially in upgrade notes.

vso CLOSED as it was done under (15)

Comment by Vjaceslavs Bogdanovs [ 2017 Jan 19 ]

Available in pre-3.3.0 (trunk) r65179.

Comment by Vladislavs Sokurenko [ 2017 Jan 19 ]

(14) [S] Errors that show up in frontend must start with a capital letter and end with period.

vjaceslavs Should we change all existing errors (introduced in and before this solution) and output of the scripts to this format? If so, there are multiple ways to do it:
1. Wrap output of the script into some message string like Execution failed: "<OUTPUT GOES HERE>".
2. Introduce check and formatting of an output in frontend so all errors are displayed in this format.
3. Fix all existing errors and transform output so first letter is capitalized and output ends with a period.

Should error messages like zbx_waitpid() failed: X be rewritten so they explain what happen in the way users can understand or should they remain the same?

Any combination of those is possible, so decision should be made before implementing.

vso Only static messages that were introduced, everything else is out of scope of this task.
User output must not be altered.
I agree that there are some messages that do not obey this rule, one is example you mentioned, but they all have something in common, they will only occur on some critical error.

vjaceslavs Maybe I am missing difference on how critical some errors are and why unable to create process [X]: X is critical, but Process terminated unexpectedly. is somehow ok, but
RESOLVED in pre-3.3.0 (trunk) r65183
vso CLOSED

Comment by Vladimir Silin (Inactive) [ 2017 Jan 24 ]

(15) [D] Documented in the following documentation pages for version 3.4:

vso Isn't "Due to command/script exit code check introduction in Zabbix 3.4 commands and scripts can be executed multiple times if exit code is different from 0"
Only for media script ?

Also for media script it would be good to use such example with exit 0 added. And to mention that exit value other than 0 would mean error.

#!/bin/bash
 
to=$1
subject=$2
body=$3
 
cat <<EOF | mail -s "$subject" "$to"
$body
EOF

exit 0

VSI Thanks for the comment!
Upgrade notes for 3.4.0 updated, script example on the Custom alertscripts page updated with proposed example and note about error codes.

<dimir> Wouldn't it be better to actually spout error if that occurs:

exit 0
exit $?

vso yes I think you are right about $? It's better working example as exit status of last command will be returned. exit 0 just says that user must explicitly define exit code. Maybe original example should stay as no exit is same as exit $?

vjaceslavs Thats why I was confused about changes made to existing example. If we want to show how to use exit to define exit code then we could make second example like that:

#!/bin/bash
if [ -f "/var/messages/target" ];
then
   # some logic goes here
else
   exit 1 #exit with an error
fi

VSI Removed "exit 0" entry from the example and mentioned exit code checking basics in the note on Custom alertscripts page. I'd prefer not to complicate description with several examples, especially with additional logic.
Do you think it worth to mention possibility for user to define an exit code within a script regardless of actual execution results? It looks like some very specific case.

vjaceslavs I agree that this is a very specific case. I gave this example just to show the idea on how exit X could be used "in real life", but I think that we can leave things simple and stay with the existing example.

vso REOPENED, There are some info missing in upgrade notes and what's new, all pages from (16) should be mentioned.

This below leaves question, "can be executed" it shall be well define what will happen and how many times and if it's feature or known issue.

alertscripts can be executed multiple times if their exit code is different from 0

It should be confirmed but from code looks like max retries is 3 times, delay in between is 30 seconds, I could not find it documented anywhere.

VSI The main idea is to warn users about the fact of possibility of such behavior after upgrade in case of non-0 exit code. I don't see the point to introduce users in upgrade notes with parameters of a mechanism which cannot be configured. Perhaps it should be documented, but elsewhere. Also these parameters are not actually a result or a dependency of the change introduced in this issue, but their effect is, which is mentioned.

vso Yes I agree, still REOPENED, upgrade notes have been updated under (16) and looks OK.

However what's new does not mention what has been added to (16) in documentation.

For example here https://www.zabbix.com/documentation/3.4/manual/web_interface/frontend_sections/administration/scripts?rev=1478764129&do=diff
New info about user scripts is added but no mention in what's new.

external check items are mentioned in upgrade notes but not it what's new, etc..

I think all new features should be in what's new, please go through (16) pages and double check

Also this below should be deleted from what's new because it's nothing for user to care about.
And for some reason it's in what's new but not in upgrade notes.

system.hw.devices, and system.sw.packages Zabbix agent items

And it's not only place where return code has been added, for example in net.tcp.listen and some other places, probably best to mention that some items that launch commands could also become unsupported. ( if needed, I can provide a list )

VSI What's new updated with:

  • generalized term 'scripts' as we don't have anything described as 'user script' in documentation
  • mention of external checks
  • generalized phrase 'Zabbix agent items executed without “nowait” flag' instead of naming specific items.

vso
Documentation states:

remote commands on Zabbix agent are executed without timeout by the system.run[,nowait] key and are not checked for execution results.

While what's new

Now Zabbix checks exit codes for scripts, including media types, external checks, remote commands, user parameters executed by Zabbix server and for Zabbix agent items executed without “nowait” flag.

VSI What's new fixed and phrased as
"Now Zabbix checks exit codes for user parameters, remote commands and system.run[] items without the “nowait” flag as well as Zabbix server executed scripts (alert, external and global). In case of an error in a script or a command execution process Zabbix provides error description in the Frontend as well as creates corresponding log entries."

vso CLOSED

Comment by Vladislavs Sokurenko [ 2017 Feb 14 ]

(16) [D] No mention about return code in:

  • Custom alertscripts
    This one looks OK, simply for history and to see whats new.
    Should we also mention that it will retry on failure ?

https://www.zabbix.com/documentation/3.4/manual/config/notifications/media/script

VSI Exit code checking and links to whole Command execution page added in:

vso CLOSED

Comment by richlv [ 2017 Mar 27 ]

will this also save the failure output somewhere - server logfile, for example ? if yes, that would solve ZBXNEXT-1284

Comment by Vjaceslavs Bogdanovs [ 2017 Mar 29 ]

Error output of command execution is available in frontend and in server log file (LOG_LEVEL_WARNING).

Comment by richlv [ 2017 Mar 29 ]

vjaceslavs, that sounds really neat, thank you.

Comment by dimir [ 2017 Mar 30 ]

Sub-issues still open: (13)

Comment by Glebs Ivanovskis (Inactive) [ 2017 Oct 13 ]

Looks like it caused ZBX-12874.

Generated at Tue Apr 23 15:37:14 EEST 2024 using Jira 9.12.4#9120004-sha1:625303b708afdb767e17cb2838290c41888e9ff0.