Linux RHEL 6

are all the agents of version 2.0.3 ?

Please start that agent (which has been added with 0.0.0.0) with DebugLevel=4.
Let it run for 5 minutes and attach zabbix_agentd.log here.

Also attach it's config zabbix_agentd.conf

On one screenshot I see TCP port 13684 used for agents, is it so ?

In the attached log we can see:

 15511:20130306:093256.606 refresh_active_checks('10.251.4.233',13685)
 15511:20130306:093256.606 Sending [{
	"request":"active checks",
	"host":"xxx.com",
	"ip":"10.251.5.9",
	"port":13684}]

so zabbix agent itself is ok, thanks.

All Agents 2.0.3, because I couldn't be 100% positive during the initial run, so I just took one host and deleted it and added it back with 2.0.3 agent and 1.9.4 (my previous version). The IP was added correctly both times.

I have attached a log of my 2.0.3 agent starting during normal processing.

I then removed the host again, waited for the host removal to sync with proxy. Then started 2.0.3 agent. That log is attached also. Again it seemed to work fine.

I am using port 13684. Big orgization and the zabbix port was in use.

Here is the config:

Server=10.251.4.233,10.251.4.62
ServerActive=10.251.4.233:13685
LogFile=/tmp/zabbix_agentd.log
PidFile=/tmp/zabbix_agentd.pid
Timeout=30
ListenIP=10.251.5.9
ListenPort=13684
#ServerPort=13685 Deprecated, moved to ServerActive
DebugLevel=4
BufferSend=10
BufferSize=1000

UserParameter=ms.discovery[*],/ms/dist/ipm/PROJ/hpe-zabbix-runtime/$ZBX_ENV/etc/getData.sh -d $1 $2 $3 $4 $5
UserParameter=ms.poll[*],/ms/dist/ipm/PROJ/hpe-zabbix-runtime/$ZBX_ENV/etc/getData.sh -s $1 $2 $3 $4 $5

UserParameter=SetProxy[*],/ms/dist/ipm/PROJ/hpe-zabbix-runtime/prod/etc/setServer.sh $1 $2

Of course it keeps working....Here is the debug level from the proxy of another host. The IP was added correctly at the server level. And the dns populates correctly. I was mistaken by saying this was left out. All of my successful hosts have it populated. I will keep trying to get the proxy snippet of a case in which it fails.

7798:20130306:102655.201 In send_data_to_server() [{
"request":"auto registration",
"host":"vhub.halsey_proxy_1n1a",
"data":[

],
"clock":1362583615}]

Ok. Not a bug. A self inflicted wound. Upon startup I make sure I assign ListenIP so that I don't listen to all available IPs as there is time sensitive traffic on the other IPs. The problem as soon as the box is booted the agent starts and the interface configuration specically to NIC bonding may not be setup and my check doesn't account for this and finds 0.0.0.0 and puts that in the conf file. Which make sense as to why the host I added yesterday failed because it doesn't follow this setup I was looking for.

And due to the fact that by the time I investigate the box was rebooted and setup correctly and the conf file is correct I assumed it was getting dropped. However, the real issue is that I missed that initial registration window.

I do appreciate you time in helping me diagnose. My Apologies.

Here is the bad conf file:

Server=10.251.4.233,10.251.4.62
ServerActive=10.251.4.233:13685
LogFile=/tmp/zabbix_agentd.log
PidFile=/tmp/zabbix_agentd.pid
Timeout=30
ListenIP=0.0.0.0
ListenPort=13684
#ServerPort=13685 Deprecated, moved to ServerActive
DebugLevel=3
BufferSend=10
BufferSize=1000

UserParameter=ms.discovery[*],/ms/dist/ipm/PROJ/hpe-zabbix-runtime/$ZBX_ENV/etc/getData.sh -d $1 $2 $3 $4 $5
UserParameter=ms.poll[*],/ms/dist/ipm/PROJ/hpe-zabbix-runtime/$ZBX_ENV/etc/getData.sh -s $1 $2 $3 $4 $5

UserParameter=SetProxy[*],/ms/dist/ipm/PROJ/hpe-zabbix-runtime/prod/etc/setServer.sh $1 $2

This is the zabbix proxy snippet associated with the bad conf file.

7806:20130306:103723.165 Trapper got [

] len 104
7806:20130306:103723.165 In send_list_of_active_checks_json()
7806:20130306:103723.165 In get_hostid_by_host() host:'ivelocity137.vhub.halsey.ms.com'
7806:20130306:103723.165 query [txnlev:0] [select hostid,status from hosts where host='ivelocity137.vhub.halsey.ms.com' and status in (0,1) and proxy_hostid is null]
7806:20130306:103723.169 query [txnlev:1] [begin;]
7806:20130306:103723.169 query [txnlev:1] [insert into proxy_autoreg_host (clock,host,listen_ip,listen_dns,listen_port) values (1362584243,'ivelocity137.vhub.halsey.ms.com','0.0.0.0',''
,13684)]
7806:20130306:103723.169 query [txnlev:1] [commit;]
7806:20130306:103723.169 End of get_hostid_by_host():FAIL
7806:20130306:103723.169 cannot send list of active checks to [10.251.5.18]: host [ivelocity137.vhub.halsey.ms.com] not found

...

7798:20130306:103723.177 In send_data_to_server() [{
"request":"auto registration",
"host":"vhub.halsey_proxy_1n1a",
"data":[

],
"clock":1362584243}]

Operator Error, Not a bug.