[ZBX-5383] zbx_tcp_recv_ext causes server crash Created: 2012 Jul 30  Updated: 2020 Jul 16  Resolved: 2012 Aug 30

Status: Closed
Project: ZABBIX BUGS AND ISSUES
Component/s: Agent (G), Proxy (P), Server (S)
Affects Version/s: None
Fix Version/s: 1.8.16rc1, 2.0.3rc1, 2.1.0

Type: Defect (Security) Priority: Critical
Reporter: Uli Stärk Assignee: Unassigned
Resolution: Fixed Votes: 0
Labels: security
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Attachments: Text File zabbix_sender_example.php.txt    
Issue Links:
Duplicate

 Description   

The zabbix sender protocol allows sending huge amouts of data (64 bit integer). If you submit a large amount, the servers stops reading data at 1GB and causes 100% CPU load. This can be used to generate a denial of service and crash the server:
26372:20120730:175909.834 file:str.c,line:204 zbx_realloc: out of memory. Requested 1073741824 bytes.



 Comments   
Comment by Andris Mednis [ 2012 Aug 17 ]

Proposed solution is to limit maximum amount of data accepted by zbx_tcp_recv_ext() when receiving data in the new protocol ("ZBXD\1"):
Zabbix 1.8, 2.0: 128 Mbytes
Zabbix 2.2: 64 Mbytes

Comment by Andris Mednis [ 2012 Aug 22 ]

Fixed in development branch svn://svn.zabbix.com/branches/dev/ZBX-5383

Comment by dimir [ 2012 Aug 28 ]

(1) [S] The error message goes

3630:20120828:173722.531 Message size 18446744073709551615 exceeds the maximum size 13813811479268294656 bytes. Message ignored.

where the limit is not exactly 128 MBytes.

<Andris> RESOLVED in r29957

<dimir> CLOSED

Comment by dimir [ 2012 Aug 30 ]

Successfully tested.

Comment by Andris Mednis [ 2012 Aug 31 ]

Fixed in versions pre-1.8.16 rev. 29972, pre-2.0.3 rev. 30016 and pre-2.1.0 rev.30019.

Comment by richlv [ 2012 Aug 31 ]

documented in :

http://www.zabbix.com/documentation/1.8/manual/about/installation_and_upgrade?&#for_version_1816
http://www.zabbix.com/documentation/1.8/manual/about/what_s_new_1.8.16

http://www.zabbix.com/documentation/2.0/manual/installation/upgrade_notes_203
http://www.zabbix.com/documentation/2.0/manual/introduction/whatsnew203

http://www.zabbix.com/documentation/2.2/manual/installation/upgrade_notes_220
http://www.zabbix.com/documentation/2.2/manual/introduction/whatsnew222#daemon_improvements

Comment by richlv [ 2013 Mar 01 ]

this might have resulted in a regression with 1.4 agents : ZBX-6338

Comment by Oleksii Zagorskyi [ 2013 May 09 ]

From the doc pages: 16MB limit for -> (including older Zabbix protocols)

Rich, according to your doc changes, where did you get the number? From SVN?

rev 30016 SVN log message says:
"After this change: Data larger than 128 MB are ignored. This limit applies only to Zabbix communication protocol. Other protocols (e.g. XML) currently limit data size to 16 MB."

Then maybe this question to Andris?

According to ZBX-4980 I know about some 64KB limit for old XML protocol.
Where I'm wrong ?

Btw, why this is not documented at some dedicated doc page?
For example here https://www.zabbix.com/documentation/2.0/manual/appendix/items/activepassive

Comment by richlv [ 2013 May 09 ]

hmm. if we have extra 64kb limit, the doc changes should be fixed. and yes, we should have this documented in a permanent location, too

Comment by Andris Mednis [ 2013 May 09 ]

The limit on incoming data size from TCP socket is implemented in file src/libs/zbxcomms/comms.c, function zbx_tcp_recv_ext():

int     zbx_tcp_recv_ext(zbx_sock_t *s, char **data, unsigned char flags, int timeout)
{
...
        if (ZBX_TCP_HEADER_LEN == nbytes && 0 == strncmp(s->buf_stat, ZBX_TCP_HEADER, ZBX_TCP_HEADER_LEN))
        {
                left = sizeof(zbx_uint64_t);
                nbytes = ZBX_TCP_READ(s->socket, (void *)&expected_len, left);
                expected_len = zbx_letoh_uint64(expected_len);

                if (ZBX_MAX_RECV_DATA_SIZE < expected_len)    <----- Limit for Zabbix protocol data: Zabbix 1.8, 2.0 -> (128 * ZBX_MEBIBYTE), Zabbix trunk -> (64 * ZBX_MEBIBYTE)
                {
                        zabbix_log(LOG_LEVEL_WARNING, "Message size " ZBX_FS_UI64 " exceeds the maximum size "
                                        ZBX_FS_UI64 " bytes. Message ignored.", expected_len,
                                        (zbx_uint64_t)ZBX_MAX_RECV_DATA_SIZE);
...
                }
...
        }
        else if (ZBX_TCP_ERROR != nbytes)
        {
                read_bytes = nbytes;
                expected_len = 16 * ZBX_MEBIBYTE;              <----- Limit for other protocols: Zabbix 1.8, 2.0, trunk -> 16 MB.
        }
...

In case of XML protocol incoming data are read either until the closing tag </req> is seen, or 16 MB limit is reached, or timeout occurs.

Comment by Oleksii Zagorskyi [ 2013 May 09 ]

Today's IRC:

(14:38:02) Andris: Zalex: Looked into trunk/ code. In Zabbix trapper process, when XML data is read from incoming TCP connection, the function process_trap() imposes 64 kB limit on decoded XML value size due to using fixed size buffer. Probably this limit cam be removed by using a dynamic buffer. The function process_trap() is just intermediary function, perhaps it should not impose such limits. So, receiving from TCP socket imposes 16 MB limit, then 64 KB limit in process_trap(), then some limit on storing into DB.

(14:46:29) Andris: Zalex: in other words - Zabbix will take up to 16 MB of XML Base64-encoded data, but a single decoded value will be no longer longer than 64 kB.

Comment by Oleksii Zagorskyi [ 2013 May 09 ]

Probably at all affected doc pages we could change this:
Previously it was possible to send large amount of data to the Zabbix server
to this:
Previously it was possible to send large amount of data to the Zabbix server in one connection
and then finish this discussion.

Then it will be cleaner that it related to socket receive buffer but not to, for example, a value.

Thank you Andris for explanation !

Details about 64KB limit for single base64 encoded value (outdated XML protocol only) - I'm not sure it wort to be documented, but it still could be added as a small note bottom of the page https://www.zabbix.com/documentation/2.0/manual/appendix/items/activepassive

Comment by Oleksii Zagorskyi [ 2013 May 14 ]

Piece of info about the limits added to:
https://www.zabbix.com/documentation/2.0/manual/appendix/items/activepassive
https://www.zabbix.com/documentation/2.2/manual/appendix/items/activepassive

Probably the discussion CLOSED.

Generated at Fri Mar 29 10:21:14 EET 2024 using Jira 9.12.4#9120004-sha1:625303b708afdb767e17cb2838290c41888e9ff0.