[ZBX-13202] ?Invalid item key format.?message from remote command action Created: 2017 Dec 15 Updated: 2018 Jan 11 Resolved: 2018 Jan 09 |
|
| Status: | Closed |
| Project: | ZABBIX BUGS AND ISSUES |
| Component/s: | Agent (G), Server (S) |
| Affects Version/s: | 3.0.13 |
| Fix Version/s: | None |
| Type: | Problem report | Priority: | Major |
| Reporter: | Hayato Watanabe | Assignee: | Unassigned |
| Resolution: | Duplicate | Votes: | 0 |
| Labels: | actions | ||
| Remaining Estimate: | Not Specified | ||
| Time Spent: | Not Specified | ||
| Original Estimate: | Not Specified | ||
| Attachments: |
RemoteAction.png
debug.patch
|
||||||||||||
| Issue Links: |
|
||||||||||||
| Sprint: | Sprint 23, Sprint 24, Sprint 25 | ||||||||||||
| Description |
|
Hi! I found receive message ?Invalid item key format.? from zabbix-agent. Host: test item: Action: test action Operations: echo 012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123 [Console] test command: [root@localhost tmp]# while sleep 0; do zabbix_sender -z 192.168.10.112 -s "test" -k "test-trapper" -o 1;done
Thank you. |
| Comments |
| Comment by Hayato Watanabe [ 2017 Dec 15 ] |
|
TCP Dump receive log( Host server ) 03:07:22.379545 IP 192.168.10.112.56907 > localhost.localdomain.zabbix-agent: Flags [.], ack 1, win 229, options [nop,nop,TS val 16047752 ecr 9005314], length 0 E..4.D@.@..?.. p.. ..K'B..:...$............ ......i. 03:07:22.379687 IP 192.168.10.112.56907 > localhost.localdomain.zabbix-agent: Flags [.], seq 1:1449, ack 1, win 229, options [nop,nop,TS val 16047752 ecr 9005314], length 1448 E....E@.@..... p.. ..K'B..:...$......$..... ......i.system.run[echo 0123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901 03:07:22.379699 IP localhost.localdomain.zabbix-agent > 192.168.10.112.56907: Flags [.], ack 1449, win 272, options [nop,nop,TS val 9005315 ecr 16047752], length 0 E..4M.@[email protected]... ... p'[email protected]..... ..i..... 03:07:22.379935 IP localhost.localdomain.zabbix-agent > 192.168.10.112.56907: Flags [P.], seq 1:55, ack 1449, win 272, options [nop,nop,TS val 9005315 ecr 16047752], length 54 E..jM.@[email protected]... ... p'B.K..$...@t........... ..i.....ZBXD.).......ZBX_NOTSUPPORTED.Invalid item key format. 03:07:22.380036 IP 192.168.10.112.56907 > localhost.localdomain.zabbix-agent: Flags [P.], seq 1449:1460, ack 1, win 229, options [nop,nop,TS val 16047753 ecr 9005314], length 11 E..?.F@[email protected].. p.. ..K'B..@t..$......%..... ......i.23,nowait] 03:07:22.380080 IP localhost.localdomain.zabbix-agent > 192.168.10.112.56907: Flags [F.], seq 55, ack 1460, win 272, options [nop,nop,TS val 9005315 ecr 16047753], length 0 E..4M.@[email protected]... ... p'[email protected]..... ..i..... 03:07:22.380130 IP localhost.localdomain.zabbix-agent > 192.168.10.112.56907: Flags [R.], seq 56, ack 1460, win 272, options [nop,nop,TS val 9005315 ecr 16047753], length 0 E..4M.@[email protected]... ... p'[email protected]..... ..i..... "seq 1449:1460" packet to late. ex) Custom script : "$ snmptrap .... netSnmpExperimental.1 s {ITEM.VALUE1} ... netSnmpExperimental.1 s {ITEM.VALUE2} ....... (more than 1500 byte)" too failed. |
| Comment by Vladislavs Sokurenko [ 2017 Dec 15 ] |
|
Did this error start with some specific version ? |
| Comment by Hayato Watanabe [ 2017 Dec 15 ] |
|
CentOS 6 -> 2.2.0 |
| Comment by Viktors Tjarve [ 2017 Dec 15 ] |
|
Hi Hayato, |
| Comment by Glebs Ivanovskis (Inactive) [ 2017 Dec 16 ] |
|
It can be seen from TCP dump that agent responds before it gets full command. That's why it complains about malformed key — it's only a part of a key. Why does it respond so early? For backwards compatibility with very old agents passive checks and remote commands use old plain text protocol which does not contain information about message length in the header. Since agent has no idea how long incoming message is it stops reading as soon as read() returns 0 (meaning there is no data in system's TCP layer buffer available to read at the moment). Plain text protocol's reliability hugely depends on network latencies, especially for longer messages. That's why it was surpassed by a better protocol in almost every place of Zabbix except one. |
| Comment by Vladislavs Sokurenko [ 2017 Dec 16 ] |
|
This should be related to MTU, though I am very surprised to hear that it does not contain length in header. While testing on a local machine I believe it will always be bigger than 1500 bytes, that's why it's not reproducible on a development machine. For example mine is 65536: ip ad | grep mtu 1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2: enp0s3: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc fq_codel state UP group default qlen 1000 3: virbr0: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc noqueue state DOWN group default qlen 1000 4: virbr0-nic: <BROADCAST,MULTICAST> mtu 1500 qdisc fq_codel master virbr0 state DOWN group default qlen 1000 it should be easily reproducible when changed to 1500 or even lower. |
| Comment by Vladislavs Sokurenko [ 2017 Dec 18 ] |
|
Dear bsmile as a workaround I suggest you try remote command execution through proxy. |
| Comment by Vladislavs Sokurenko [ 2018 Jan 09 ] |
|
Closing as duplicate of |
