OS: Ubuntu 14.04 Trusty
Packages: zabbix-server-mysql,zabbix-frontend-php, zabbix-agent, mysql-server-5.5

Messages in agent log:

28245:20160601:190601.699 TCP expect network error: cannot connect to [[1.1.1.1]:3136]: [4] Interrupted system call
28245:20160601:190601.699 Sending back [0]

this is most likely a support case. note that the reporter also had an open request on serverfault at http://serverfault.com/questions/780012/zabbix-adding-items-make-agents-not-available

@richhlv: Yes, first I tried to find help at zabbix forum and serverfault. I have digged it for some time and now I am sure that this is a bug. I have written explicit steps how to reproduce it.

In what way this is a bug?

By default Timeout's in server/proxy and agent configuration files are equal (3 seconds). For agent this timeout means: "if check takes longer than 3 seconds report not supported item to server". For poller this timeout means: "if there is no response from host within 3 seconds mark this host as unreachable". Since process cannot wait for exactly 3 seconds due to technical limitations, sometimes agent gets bored first, sometimes poller.

Try to reproduce with Timeout=4 in server configuration file.

@glebs.ivanovskis:
> In what way this is a bug?

Server starts to randomly switch trigger "zabbix-agent on <hostname> in not available for 2 minutes". Trigger is not about problematic item itself, but about whole agent connectivity.

> Try to reproduce with Timeout=4 in server configuration file.

Yep, with Timeout=10 in zabbix_server.conf and Timeout=5 in zabbix_agentd.conf problem disappears. But I insist, that this is a real bug:

If:

• timeout on both agent and server are the same (default: timeout = 3)
• there is item net.tcp.port[<IP>,<port>] and trigger using it
• pair [<IP>,<port>] is unavailable by TCP timeout

Then:

"Zabbix-agent on <hostname> is unawailable" ( trigger expression: {agent.ping.nodata(2m)} = 1 ) start spawning on hosts with this item. Not the trigger for specific item, but the trigger for the agent availability. This is bug.

Some time ago we had a problematic behavior with timeouting items. It used to be that a single timeouting item on a host would block other items from processing: that item would timeout, all items would move to unreachable pollers, and unreachable pollers would still try to use that same timeouting item every UnreachableDelay (15) and UnavailableDelay (60) seconds for checking the host.

As far as I remember, that problem was addressed in ZBX-4284. After the fix, Zabbix tries different items for checking the host using unreachable pollers. However, in your case, there are 10 timeouting items on a host, and if it happens that they are all checked in order, then that takes a lot more than 2 minutes, so the "agent.ping" trigger fires. Consequently, there is no bug.

The issue with Timeout parameter being set to 3 by default in all configuration files was addressed in ZBXNEXT-2637: since then server's timeout is 4 seconds by default.

It seems like there is nothing to fix, so closing as "Won't fix".

@asaveljevs: thank you for reply.

It would be a great idea to change default server timeout > client timeout in stable 3.0 branch. It will solve some mystery problems for some zabbix users.

And second thought: it would be great to add optional <timeout> parameter to agent items like net.tcp.port: net.tcp.port[ IP, port, {timeout} ]. And raise warning, if <timeout> is more than timeout on agent. Because users want to have ability to add as much net.tcp.port items, as they need, without getting any implicit problems.

For second thought created separate ticket: https://support.zabbix.com/browse/ZBX-10882