That's interesting. Could it be related to some limits of Linux kernel related to TCP stack? Do you see anything suspicious in syslog or kern.log?

Anton, let me know which error do you see for host error when it's in error state in GUI (temporarily disabling - host unavailable)? Maybe the error is "Invalid port number[]" ?
Are those items inherited from template?
If yes, try for one host Delete and clear template and then link it again.

I had very similar hosts behavior when some part of theirs items did not have defined iterfaceid (they had NULL for iterfaceid) <- because we are using trunk

Here is behavior when one agent item has NULL for iterfaceid:
44813:20120211:133759.283 Zabbix agent item [vfs.fs.size[D:\,pfree]] on host [it5] failed: first network error, wait for 15 seconds
44816:20120211:133814.187 Zabbix agent item [vfs.fs.size[D:\,pfree]] on host [it5] failed: another network error, wait for 15 seconds
44816:20120211:133829.215 Zabbix agent item [vfs.fs.size[D:\,pfree]] on host [it5] failed: another network error, wait for 15 seconds
44816:20120211:133844.235 Zabbix agent item [vfs.fs.size[D:\,pfree]] on host [it5] failed: another network error, wait for 15 seconds
44816:20120211:133859.302 temporarily disabling Zabbix agent checks on host [it5]: host unavailable
44816:20120211:134000.520 enabling Zabbix agent checks on host [it5]: host became available

Try to execute this SQL statement to find zabbix agent items with NULL interfaceid on real hosts:
SELECT DISTINCT h.host,i.itemid,i.name,i.key_,i.interfaceid FROM items i, hosts h WHERE i.type=0 AND i.interfaceid IS NULL AND h.status=0 AND i.hostid=h.hostid;

I found NULL interfaceid on one of the hosts on one of the servers. There were itemes, that were added directly to host, then I copied them to the template, but interfaceid is null.
Even after 'Unlink and clear' and link again - interfaceid is null. And errors continue.
But the second server has no items with null interfaceid.

I use all items inherited from tamplates.
I see no errors in general Linux logs and , as I mentioned before, it's definetly not a network/os/limit issue - I checked these things first.
Sorry, cannot figure out where should I see the state in GUI (Monitoring -> Hosts shows availability icon only)

> Sorry, cannot figure out where should I see the state in GUI (Monitoring -> Hosts shows availability icon only)
Yes, it is. When a host (agent, snmp, etc) in error state then you can move mouse oved RED icon and you will see tool-tip with error description.
Which error do you see?

Note interfaceid=NULL for items in template is ok.

Try to delete this item (copied-recopied ) and create it manually again.

In GUI for one server I see the following 2 errors:
"...error (111): connection refused" (see attachment for full error)
"Invalid port number[]"
Both errors appears on the JMX hosts (we use zapcat agent and port 10052 for monitoring).
We have used zapcat for weeks without any problem, but sometimes we had white spaces in our graphs, I thought it was caused by performance. But it's not.

Trying to reproduce on another server. to see an error.

> We have used zapcat for weeks without any problem, but sometimes we had white spaces in our graphs, I thought it was caused by performance. But it's not.
No, that happened because the hosts (zabbix-agent type I mean) were periodically disabled because even of single problematic item (without interfaceid in the DB)

So you have fix all problems which are generating the error "Invalid port number[]"
Maybe interface is not empty, but the port is empty?

Yes, I can generate the error "Invalid port number []" when "interface.port" is empty (deleted manually in the DB)

Try this:
mysql> SELECT * FROM interface WHERE port="";
and its result should be empty.

I have
SELECT * FROM interface WHERE port="";
Empty set (0.00 sec)

I have no more ideas, sorry.

Wow, it's seems the issue disapeared from one of the servers!
I have unlinked templates with cleanup and linked them back on the problematic hosts.
I have no more NULLs in the DB and no errors in logs
Thanks a lot!

p.s. Problems on the other server can be network issues indeed, there are some hosts monitored over Internet from US to Asia.

Be careful with trunk next time, but thanks for using it in production

Issue closed.