Zabbix server 7.0.0
Zabbix proxy 7.0.0
Zabbix agent 7.0.0 on Linux

I have to comment that while writing this I tried the remote command numerous times and I haven't got the result shown successfully even once.

But earlier today I saw the hostname command result successfully in the UI at least at two occasions, not sure how those cases were different.

I reproduce this with Zabbix agent2 7.0.0 on windows11.

Also, as document say that active agent would get items from server every 2 mins. I try to adjust Script timeout to 150s, can not help. Then I notice that after few minutes after I click on the agent host and run "Hostname script" (the error message would show and I can close it), the agent2 on windows11 was down.

The finnal log of agent2 is below. The agent2 is stop without error and stop message, which really confused

2024/07/16 21:39:30.192277 [SystemRun] Executing command:'D:\miniconda\python.exe E:\REST-Voyager\stop_try.py'

Agent 7.0.0 connects to proxy/server every 5 seconds as the documentation says: https://www.zabbix.com/documentation/current/en/manual/appendix/config/zabbix_agent2_win#refreshactivechecks

Agent stopping is said to be fixed in 7.0.1: ZBX-24719

sasha maybe this one needs review as well?

Edit: sorry, didn't notice that you already have reviewed this last week against some internal ticket

Hi!

I was able to reproduce this issue by artificially increasing the script execution time by adding the command sleep 3.

When executing scripts, a timeout is used that is specified in the proxy/server configuration file. Try increasing the value of this parameter if it is set too low.

You can also see this timeout in the agent's log file (with DebugLevel=4):

768777:20240814:053325.406 Requested [{"request":"passive checks","data":[{"key":"system.run[hostname]","timeout":4}]}]
768777:20240814:053325.406 Executing command 'hostname'
768777:20240814:053325.407 In zbx_popen() command:'hostname'
768777:20240814:053325.408 End of zbx_popen():6
768891:20240814:053325.409 zbx_popen(): executing script
768777:20240814:053325.414 In zbx_waitpid()
768777:20240814:053325.414 zbx_waitpid() exited, status:0
768777:20240814:053325.414 End of zbx_waitpid():768891
768777:20240814:053325.414 execute_str_local() command:'hostname' len:6 cmd_result:'ubuntu'
768777:20240814:053325.414 Sending back [{"version":"7.0.0","variant":1,"data":[{"value":"ubuntu"}]}] 

In Administration -> General -> Timeouts -> Network timeouts for UI -> Script execution, you can set the timeout for the entire script execution chain:
1. UI -> Server
2. Server -> Proxy
3. Proxy -> Agent
4. Agent -> Proxy
5. Proxy -> Server
6. Server -> UI

In the issue the configured timeouts were not reached:

• Global Script timeout = 10s (script execution on the agent took less than a second)
• Network timeout for Script execution = 60s
• Total time from server sending the script execution request to proxy to proxy sending the response to server was under 6 seconds

What is the value of the Timeout parameter in your proxy configuration file?

The default as supplied by the installer: Timeout=4

That's the connect timeout from proxy process to server trapper, since 7.0, right? Not any kind of item timeout anymore.

I tested this again with 7.0.2 components:

• 14:18:23 script execution clicked on UI
• 14:18:24.9 server responded to active proxy data request with the remote command to execute on agent
• 14:18:25.1 agent requested active checks from proxy, proxy didn't say anything about commands to execute
• 14:18:29.9 proxy sent task data to server: "timeout while retrieving result for remote command"
• 14:18:30.1 agent next time requested active check from proxy, nothing to say about remove commands anymore

I don't see 10s (script timeout) having elapsed yet.

Another example, this time script timeout = 20s:

• 14:50:55 script execution clicked on UI
• 14:50:56.5 server responded to active proxy data request with the remote command to execute on agent
• 14:51:00.0 agent requested active checks from proxy, proxy didn't say anything about commands to execute
• 14:51:04.5 proxy sent task data to server: "timeout while retrieving result for remote command"
• 14:51:05.0 agent next time requested active check from proxy, nothing to say about remove commands anymore

Again, no 20s timeout yet. But, in the previous example (script timeout = 10s) it took 5 seconds for proxy to declare timeout, in this example (script timeout = 20s) it took 8 seconds for proxy to declare timeout.

Note: I also see successful executions, I'm just pointing out that the executions are really unreliable.

Example of a successful command:

• 14:50:07 script execution clicked on UI
• 14:50:08.4 server responded to active proxy data request with the remote command to execute on agent
• 14:50:11.0 agent requested active checks from proxy, proxy gave the command "hostname" to execute:
   

  {"response":"success","commands":[{"command":"hostname","id":4,"wait":1,"timeout":4}]} 

   

• 14:50:11.0 (two milliseconds later) agent sent the result to proxy, proxy said "success"
• 14:50:11.4 proxy sent data to server, nothing about the task result
• 14:50:12.4 proxy sent task data to server (the hostname the agent sent, OK)

(Irrelevant for this particular issue, but from where is that "timeout":4 (the command timeout given by proxy to agent) value coming from? Proxy Timeout=4 configuration directive?)

I am having the same issue, and in some cases even very simple scripts are timing out even when the agent log shows the data being successfully returned.

In zabbix_server.conf, I set Timeout=30 and now my scripts are either successful or it takes approximately 30 seconds before I get an error.

So it seems like some of timeout settings configurable in the GUI are still limited by the settings in the conf file.

I haven't tested with a proxy but it's probably a similar issue.

If you are running scripts on active agents, the RefreshActiveChecks parameter in the agent's configuration file should be set to a value lower than the Timeout in the server or proxy configuration file (depending on where the agent is connected).

The active agent receives the list of scripts to be executed along with the list of active checks.

How often does your agent request the list of active checks?

You can find additional information here: Script timeout

Confirmed: Whatever is configured in zabbix_proxy.conf for Timeout directive, is sent to the active agent in the timeout field with the command to execute.

That conflicts with what is said in the ZCS training materials and in the documentation (https://www.zabbix.com/documentation/7.0/en/manual/appendix/config/zabbix_proxy#timeout). Do you want me to open a new issue on that?

(Actually that timeout field in the response from proxy to agent is not even documented: https://www.zabbix.com/documentation/7.0/en/manual/appendix/items/activepassive#active-checks)

You can find additional information here: Script timeout

Btw the error in the Script timeout documentation ("Timeout while executing a shell script.") is different from the error described in this issue ("timeout while retrieving result for remote command").

Also confirmed: tried tuning every parameter related to timeout. Nothing helped, so my Zabbix is useless now, because it isn't able to  fix remote machines.

Anybody made success config? Or resolved shomehow?

Hey Phil,

Post in the Zabbix forums - https://zabbix.com/forum - (and tag me @MRedbourne). This isn't a place for support questions. But yes, I have this working.

Edit: Maker sure to include your Zabbix versions (agent, server, proxy, etc) along with target OS version.

Funny thing , we did encounter this before - solved by adding one passive item on to the host ( not sure if it has to be working , ours is ). After some time we did encounter it again ( the first time solution was forgotten ) and here we're again - adding passive item to the host solved it again. Try it in your case if using only actove items on the host.

The maximum script execution timeout, depending on the proxy and agent connection type, can be calculated using the following formulas:

gofree's comment about fixing this issue by adding a dummy passive item to the host worked for me.  I would suggest that anybody who has this issue to give this a shot.

I have the same problem,
on the server I have set "Timeout = 30", and on the client "RefreshActiveChecks = 5",
scripts that take up to 24 seconds to execute work correctly, scripts from 25 to 30 seconds randomly (they will work or not),
scripts up to 30 seconds do not work.

The only solution I found to make scripts longer than 30 seconds work is editing the file "src/zabbix_server/server.c"
from

{"Timeout", &zbx_config_timeout, ZBX_CFG_TYPE_INT, ZBX_CONF_PARM_OPT, 1, 30}
,​

to

{"Timeout", &zbx_config_timeout, ZBX_CFG_TYPE_INT, ZBX_CONF_PARM_OPT, 1, 300}
,​

then I compile the program and change "Timeout = 300" on the server and I have no problem running scripts that last about 50 seconds

Does increasing Timeout in Zabbix Server configuration file help with the issue ?

vso The maximum value allowed for Timeout is 30 seconds as per https://www.zabbix.com/documentation/current/en/manual/appendix/config/zabbix_server#timeout so this appears to be the maximum possible time that Zabbix Server will wait to receive the results from a script execution, regardless of other time values that might be in effect.

As the table created by sasha shows, the effective formula is

(server Timeout [30 seconds] - ProxyConfigFrequency - RefreshActiveChecks) = maximum_script_execution_time

As Xin has shown, modifying the source code to allow server Timeout values larger than 30 seconds is necessary to to increase the effective maximum script execution time.

30 seconds should have been enough mrcomps what version of Zabbix agent is used, so it falls due to additional 5 seconds or so ?

Fixed in pull request feature/ZBX-26510-7.0-2

Fixed in:

• pre-7.0.18rc1 2879a2a0faa
• pre-7.2.12rc1 51cf7116568 8a658dd11bd
• pre-7.4.2rc1 cf60260006c
• pre-8.0.0alpha1 (master) 9c00df1e940