Zabbix agent (v.1.8.3) stops reporting data, I finally managed to capture it in debug mode. It reports data, then infinitely gets stuck in that collector_diskdevice_get() function until agent is manually restarted:

19613:20111007:203502.652 In send_buffer() host:'zabbixserver.someotherdomain.com' port:10051 values:6/100
19613:20111007:203502.819 JSON before sending [{
"request":"agent data",
"data":[

{ "host":"someserver.example.com", "key":"system.cpu.util[,idle,avg1]", "value":"98.461571", "clock":1318019698}

,

{ "host":"someserver.example.com", "key":"system.cpu.util[,iowait,avg1]", "value":"0.012662", "clock":1318019698}

,

{ "host":"someserver.example.com", "key":"system.cpu.util[,nice,avg1]", "value":"0.037986", "clock":1318019698}

,

{ "host":"someserver.example.com", "key":"system.cpu.util[,system,avg1]", "value":"0.700629", "clock":1318019698}

,

{ "host":"someserver.example.com", "key":"system.cpu.util[,user,avg1]", "value":"0.759718", "clock":1318019698}

,

{ "host":"someserver.example.com", "key":"net.if.in[eth0,bytes]", "value":"8215792454", "clock":1318019700}

],
"clock":1318019702}]
19613:20111007:203503.148 JSON back [

{ "response":"success", "info":"Processed 6 Failed 0 Total 6 Seconds spent 0.000104"}

]
19613:20111007:203503.148 In check_response(

{ "response":"success", "info":"Processed 6 Failed 0 Total 6 Seconds spent 0.000104"}

)
19613:20111007:203503.148 Info from server: Processed 6 Failed 0 Total 6 Seconds spent 0.000104
19613:20111007:203503.148 OK
19613:20111007:203503.148 End of send_buffer():SUCCEED
19613:20111007:203503.148 Sleeping for 1 seconds
19613:20111007:203504.148 In send_buffer() host:'zabbixserver.someotherdomain.com' port:10051 values:0/100
19613:20111007:203504.149 End of send_buffer():SUCCEED
19613:20111007:203504.149 Sleeping for 1 seconds
19613:20111007:203505.149 In send_buffer() host:'zabbixserver.someotherdomain.com' port:10051 values:0/100
19613:20111007:203505.149 End of send_buffer():SUCCEED
19613:20111007:203505.149 In process_active_checks('zabbixserver.someotherdomain.com',10051)
19613:20111007:203505.149 For key [net.if.in[eth0,bytes]] received value [8215798070]
19613:20111007:203505.149 In process_value() key:'someserver.example.com:net.if.in[eth0,bytes]' value:'8215798070'
19613:20111007:203505.149 In send_buffer() host:'zabbixserver.someotherdomain.com' port:10051 values:0/100
19613:20111007:203505.149 End of send_buffer():SUCCEED
19613:20111007:203505.149 Buffer: new element 0
19613:20111007:203505.149 In get_min_nextcheck()
19613:20111007:203505.149 Sleeping for 1 seconds
19613:20111007:203506.150 In send_buffer() host:'zabbixserver.someotherdomain.com' port:10051 values:1/100
19613:20111007:203506.150 Will not send now. Now 1318019706 lastsent 1318019702 < 5
19613:20111007:203506.150 End of send_buffer():SUCCEED
19613:20111007:203506.150 Sleeping for 1 seconds
19613:20111007:203507.150 In send_buffer() host:'zabbixserver.someotherdomain.com' port:10051 values:1/100
19613:20111007:203507.316 JSON before sending [{
"request":"agent data",
"data":[

{ "host":"someserver.example.com", "key":"net.if.in[eth0,bytes]", "value":"8215798070", "clock":1318019705}

],
"clock":1318019707}]
19613:20111007:203507.645 JSON back [

{ "response":"success", "info":"Processed 1 Failed 0 Total 1 Seconds spent 0.000029"}

]
19613:20111007:203507.645 In check_response(

{ "response":"success", "info":"Processed 1 Failed 0 Total 1 Seconds spent 0.000029"}

)
19613:20111007:203507.646 Info from server: Processed 1 Failed 0 Total 1 Seconds spent 0.000029
19613:20111007:203507.646 OK
19613:20111007:203507.646 End of send_buffer():SUCCEED
19613:20111007:203507.646 refresh_active_checks('zabbixserver.someotherdomain.com',10051)
19612:20111007:203509.430 In collector_diskdevice_get("")
19612:20111007:203509.430 In collector_diskdevice_get("xvda")
19612:20111007:203509.430 In collector_diskdevice_get("xvdb")
19613:20111007:203510.811 Sending [

{ "request":"active checks", "host":" engine = 'InnoDB';"}

]
19613:20111007:203510.811 Before read
19612:20111007:203609.553 In collector_diskdevice_get("")
19612:20111007:203609.554 In collector_diskdevice_get("xvda")
19612:20111007:203609.554 In collector_diskdevice_get("xvdb")
19612:20111007:203709.674 In collector_diskdevice_get("")
19612:20111007:203709.674 In collector_diskdevice_get("xvda")
19612:20111007:203709.674 In collector_diskdevice_get("xvdb")
19612:20111007:203809.794 In collector_diskdevice_get("")
19612:20111007:203809.794 In collector_diskdevice_get("xvda")
19612:20111007:203809.794 In collector_diskdevice_get("xvdb")
19612:20111007:203909.919 In collector_diskdevice_get("")
19612:20111007:203909.919 In collector_diskdevice_get("xvda")
19612:20111007:203909.919 In collector_diskdevice_get("xvdb")
19612:20111007:204010.051 In collector_diskdevice_get("")
19612:20111007:204010.051 In collector_diskdevice_get("xvda")
19612:20111007:204010.051 In collector_diskdevice_get("xvdb")
19612:20111007:204110.176 In collector_diskdevice_get("")
19612:20111007:204110.176 In collector_diskdevice_get("xvda")
19612:20111007:204110.176 In collector_diskdevice_get("xvdb")
19612:20111007:204210.293 In collector_diskdevice_get("")
19612:20111007:204210.293 In collector_diskdevice_get("xvda")
19612:20111007:204210.294 In collector_diskdevice_get("xvdb")
19612:20111007:204310.418 In collector_diskdevice_get("")
19612:20111007:204310.419 In collector_diskdevice_get("xvda")
19612:20111007:204310.419 In collector_diskdevice_get("xvdb")
19612:20111007:204410.538 In collector_diskdevice_get("")
19612:20111007:204410.539 In collector_diskdevice_get("xvda")
19612:20111007:204410.539 In collector_diskdevice_get("xvdb")
19612:20111007:204510.664 In collector_diskdevice_get("")
19612:20111007:204510.664 In collector_diskdevice_get("xvda")
19612:20111007:204510.664 In collector_diskdevice_get("xvdb")
19612:20111007:204610.786 In collector_diskdevice_get("")
19612:20111007:204610.786 In collector_diskdevice_get("xvda")
19612:20111007:204610.786 In collector_diskdevice_get("xvdb")
19612:20111007:204710.910 In collector_diskdevice_get("")
19612:20111007:204710.910 In collector_diskdevice_get("xvda")
19612:20111007:204710.910 In collector_diskdevice_get("xvdb")