[#ZBX-6230] zabbix server, all trapper messages seem to be truncated around 300 bytes.

Comment by richlv [ 2013 Feb 07 ]

please describe the issue shortly here - it is much more likely developers would take a look at it if they do not have to check lots and lots of forum posts. specifically, include information that shows proper information being sent over the network, but not received by the server

Comment by Stefan Sabolowitsch [ 2013 Feb 07 ]

OK, no problem sorry

zbxlog send this Data to zabbix.

bxlog::Sender::Send zbx_data={"request":"sender data",
"data":[
{
"host":"fwgate-1",
"key":"syslog[]",
"value":"fwgate-1: NetScreen device_id=fwgate-1  [Root]system-error-00601: DNS:QUERY:NULL-QUERY has been detected from 91.XXX.XXX.XXX/5976 t
o 217.XXX.XXX.XXX/53 through policy 114 1 times. (2013-02-06 16:56:19).",
"timestamp":"1360166180",
"source":"local0",
"severity":"14",
"eventid":"131",
}
]
}
Zbxlog::Sender::Send response=OK

and this see zabbix server (debug log):

91227:20130207:131808.083 Trapper got [{"request":"sender data",
"data":[
{
"host":"fwgate-1",
"key":"syslog[]",
"value":"fwgate-1: NetScreen device_id=fwgate-1  [Root]system-critical-00430: Dst IP session limit! From 91.xxx.xxx.xxx:60900 to 217.xxx.xxx
.xxx:53, proto UDP (zone Untrust int  ethernet0/0). Occurred 1 times. (2013-02-07 13:18:05)] len 301
 91226:20130207:131808.083 In process_mass_data()
 91226:20130207:131808.083 End of process_mass_data()
 91227:20130207:131808.084 Trapper got [{"request":"sender data",
"data":[
{
"host":"fwgate-1",
"key":"syslog[]",
"value":"fwgate-1: NetScreen device_id=fwgate-1  [Root]system-critical-00430: Dst IP session limit! From 91.xxx.xxx.xxx:1196 to 217.xxx.xxx.
xxx:53, proto UDP (zone Untrust int  ethernet0/0). Occurred 1 times. (2013-02-07 13:18:07)] len 300

you see, at the end complete missing timestamp, source, severity and event id

On an tcpdump you see zbxlog send the data correct (clear text):

ZBXD{"request":"sender data",
"data":[
{
"host":"fwgate-1",
"key":"syslog[]",
"value":"fwgate-1: NetScreen device_id=fwgate-1  [Root]system-critical-00430: Dst IP session limit! From 149.xxx.xxx.xxx:50955 to 217.xxx.xxx.xxx:53, proto UDP (zone Untrust int  ethernet0/0). Occurred 1 times. (2013-02-07 14:06:42)",
"timestamp":"1360242403",
"source":"local0",
"severity":"15",
"eventid":"130",
}
]
}

The interesting thing is that it with Ubuntu working properly, but with FreeBSD does not.

zabbix_server logfile on Ubuntu, you will here see timestamp, source, severity and eventid

 27781:20130206:134417.026 Trapper got [{"request":"sender data",
"data":[
{
"host":"Zabbix server",
"key":"syslog[]",
"value":"joseph: very long message very long messagevery long message very long message very long message very long message very long message very long message very long message very long message very long message very long message very long message very long message very long message very long message very long message very long message very long message very long message very long message very long message very long message very long message very long message very long message very long message very long message very long message very long message very long message very long messagevery long message very long messagevery long message very long message very long messagevery long message very long messagevery long message very long message very long messagevery long message very long messagevery long message very long message very long messagevery long message very long messagevery long message very long message very long messagevery long message very long messagevery long message very long message very long messagevery long message  very long messagevery long messagevery long message very long messagevery long message very long messagevery long message very long message very long messagevery long message  very long messagevery long message very long message very long message ry long message very long messagevery long message very long message very long message very long message very long message very long message very long message very long message very long message very long mes",
"timestamp":"1360154656",
"source":"user",
"severity":"12",
"eventid":"13",
}
]
}] len 1668

It looks that all trapper messages seem to be truncated around 300 byte on FreeBSD

Comment by richlv [ 2013 Feb 07 ]

if you try to send a longer message using commandline zabbix_sender, is it also truncated ?

Comment by Stefan Sabolowitsch [ 2013 Feb 07 ]

no this should work, see here

93469:20130207:174026.974 Trapper got [{
        "request":"sender data",
        "data":[
                {
                        "host":"fwagte-1",
                        "key":"syslog[]",
                        "value":"local0,11,1360071440,fwgate-1: NetScreen device_id=fwgate-1  [Root]system-information-00536: IKE 153.xxx.
xxx.xxx Phase 2 msg ID fe3c6610: Responded to the peer\\'s first END local0,11,1360071440,fwgate-1: NetScreen device_id=fwgate-1  [Root]sy
stem-information-00536: IKE 153.xxx.xxx.xxx Phase 2 msg ID fe3c6610: Responded to the peer\\'s first"}]}] len 440

Maybe a problem at the end with the Keywords timestamp, source, severity etc ?

Comment by Oleksii Zagorskyi [ 2013 Feb 08 ]

Stefan, there might be a question about correct packet size - a 64-bit number after ZBXD.... and it can be a question to correct calculation at zbxlog size.

Yeah, I know the zbxlog very well (I'm zalex_ua in that forum thread). There might be also question about FreeBSD x32 or x64 ? and is its Perl compiled with x64 support if FreeBSD is x32 ?

Could you check tcpdupm once again and consider precisely the packet size ?
See <DATALEN> here https://www.zabbix.com/documentation/2.0/manual/appendix/items/activepassive

Comment by Stefan Sabolowitsch [ 2013 Feb 08 ]

Hi (Привіт) Oleksiy, thanks for your fast answer.

about your question.

uname -a

FreeBSD zabbix-m01 9.1-RELEASE FreeBSD 9.1-RELEASE #0 r243825: Tue Dec  4 09:23:10 UTC 2012     [email protected]:/usr/obj/usr/src/sys/GENERIC  amd64

perl -v

This is perl 5, version 12, subversion 4 (v5.12.4) built for amd64-freebsd

But this behavior give it also on FreeBSD 32bit (i know, it's not to easy to find out this problem)
packet size / <DATALEN> is vary in length depending on the length of the message.

Comment by Stefan Sabolowitsch [ 2013 Feb 08 ]

JBo (zbxlog developer) wrote a patch to reorders the fields in trapper message so that syslog message is the last one.
This help little, but the value wont wrote in the database.
No values in hystory_log and items (lastvalue) database table

zbxlog logfile:

Zbxlog::Sender::Send item=$VAR1 = [
          'fwgate-1',
          '',
          'syslog[]',
          '134',
          'local0',
          11,
          1360324535,
          'fwgate-1: NetScreen device_id=fwgate-1  [Root]system-information-00536: IKE 153.xxx.xxx.xx Phase 1: Retransmission limit has been
reached. (2013-02-08 12:55:35).'
        ];

Zbxlog::Sender::Send zbx_data={"request":"sender data","data":[{"host":"fwgate-1","key":"syslog[]","timestamp":"1360324535","source":"local0
","severity":"11","eventid":"134","value":"fwgate-1: NetScreen device_id=fwgate-1  [Root]system-information-00536: IKE 153.xxx.xxx.xxx Phase 1
: Retransmission limit has been reached. (2013-02-08 12:55:35).",}]}Zbxlog::Sender::Send response=OK

and here zabbix server logfile:

65021:20130208:125426.256 Trapper got [{"request":"sender data","data":[{"host":"fwgate-1","key":"syslog[]","timestamp":"1360324466","sourc
e":"local0","severity":"11","eventid":"134","value":"fwgate-1: NetScreen device_id=fwgate-1  [Root]system-information-00536: IKE 153.xxx.xxx.
76 Phase 1: Retransmission limit has been reached. (2013-02-08 12:54:26)] len 312

[ZBX-6230] zabbix server, all trapper messages seem to be truncated around 300 bytes. Created: 2013 Feb 07 Updated: 2019 Dec 10
Status:	Open
Project:	ZABBIX BUGS AND ISSUES
Component/s:	Server (S)
Affects Version/s:	2.0.4
Fix Version/s:	None

[ZBX-6230] zabbix server, all trapper messages seem to be truncated around 300 bytes. Created: 2013 Feb 07 Updated: 2019 Dec 10