[ZBX-23694] plain text emails timeout (resulting in 3 sent mails) Created: 2023 Nov 10 Updated: 2023 Nov 15 Resolved: 2023 Nov 14 |
|
Status: | Closed |
Project: | ZABBIX BUGS AND ISSUES |
Component/s: | Server (S) |
Affects Version/s: | 6.4.8 |
Fix Version/s: | None |
Type: | Incident report | Priority: | Trivial |
Reporter: | Steven Van Ingelgem | Assignee: | Maksym Buz |
Resolution: | Software failure | Votes: | 0 |
Labels: | None | ||
Remaining Estimate: | Not Specified | ||
Time Spent: | Not Specified | ||
Original Estimate: | Not Specified | ||
Environment: |
$ zabbix_server --version Webinterface: 6.4.8 |
Attachments: |
![]() ![]() ![]() ![]() |
Description |
Steps to reproduce:
Result: The test email goes through just fine, but the interface shows an error. For normal problem triggers, they are sent 3 times. I'm guessing because of a retry-mechanism built into the server. And then they still show as failed. Expected: No errors on sending.
Attached: tcpdump from the server where I did the test via the interface (tcp.stream eq 0): As you can see in packet 18, the "\r\n.\r\n" is sent. But then nothing.
When I run a simple php script mimicking the zabbix flow observed, I get the result in "tcp.stream eq 3": <?php // SMTP server settings $smtpServer = "mail.salvania.be"; $smtpPort = 25; // Sender and recipient details $from = "[email protected]"; $to = "[email protected]"; // Email content $message = "From: \"Zabbix\" <[email protected]>\r\n". "To: <[email protected]>\r\n". "In-Reply-To: <0.e6aaeb1c51ec501da39a0f92f4e6e84e.1.72717f1bd11d285a2415c519590d68d2@zabbix.com>\r\n". "Date: Fri, 10 Nov 2023 09:23:23 +0100\r\n". "Subject: Test subject\r\n". "MIME-Version: 1.0\r\n". "Content-Type: text/plain; charset=\"UTF-8\"\r\n". "Content-Transfer-Encoding: base64\r\n". "\r\n". 'VGhpcyBpcyB0aGUgdGVzdCBtZXNzYWdlIGZyb20gWmFiYml4'; // Function to send command and receive response function sendCommand($smtpConnection, $command, $fetch = true) { foreach( explode("\n", rtrim($command)) as $line ) { $line = rtrim($line); echo "> {$line}\n"; } fputs($smtpConnection, $command); if ($fetch) echo "< " . fgets($smtpConnection); } // Open a TCP connection to the SMTP server $smtpConnection = fsockopen($smtpServer, $smtpPort, $errno, $errstr, 10); // Check if the connection is successful if (!$smtpConnection) { die("Failed to connect to the SMTP server: $errstr ($errno)"); } // Read the welcome message from the SMTP server echo "< " . fgets($smtpConnection); // Send the EHLO command and handle multi-line response sendCommand($smtpConnection, "HELO salvania.be\r\n"); // Send the MAIL FROM command sendCommand($smtpConnection, "MAIL FROM: <$from>\r\n"); // Send the RCPT TO command sendCommand($smtpConnection, "RCPT TO: <$to>\r\n"); // Send the DATA command sendCommand($smtpConnection, "DATA\r\n"); // Send the email headers and body sendCommand($smtpConnection, $message, false); sendCommand($smtpConnection, "\r\n.\r\n"); // Send the QUIT command sendCommand($smtpConnection, "QUIT\r\n"); // Close the connection fclose($smtpConnection); ?>
|
Comments |
Comment by Steven Van Ingelgem [ 2023 Nov 10 ] |
I did notice that there was a ?fix? implemented in https://github.com/zabbix/zabbix/commit/8b69b91f418dd5c5c8c9629c5c08c94e1bde12bd > " So this might be related? |
Comment by Steven Van Ingelgem [ 2023 Nov 10 ] |
Notice 2: in the previous version (6.2) I did not have this issue. I also couldn't find anyone else having similar issues. That's why I did the efforts to see if it is my mail server or not. However, an exact same flow as described in the pcap file attached will show you that both the zabbix and the php stream are the same, but the zabbix one just stops. |
Comment by Andris Zeila [ 2023 Nov 14 ] |
Would it be possible to get debug logs from alerter processes? More specifically I'm interested in In send_email() and End of send_email() messages. Error indicates that there was a timeout while waiting for 250 response. When looking at attached pcap - the 250 response was sent 38.6 seconds after connecting. This is quite close to the hardcoded email timeout in v6.4 zabbix server - 40s. If establishing connection took long time then the timeout could have been triggered. Also there is theoretical possibility of a non alarm signal interrupts (that's why I'm interested in seeing how long the send_email() function took - 40+ seconds would confirm the timeout). On the other hand the php script finished it's work in less than a second. One difference I see is that php script was sending from localhost. Maybe the external connection processing is delayed as spam protection or something, I don't know. |
Comment by Steven Van Ingelgem [ 2023 Nov 14 ] |
Zabbix should be sending to localhost as well. These are the email server settings I'm using: The remarkable thing is that only zabbix is affected. It times out after this 40s... But no other process on my server is. So, somehow the email server doesn't recognize it has received "\r\n.\r\n" via zabbix. I tried to emulate this behaviour in the php script, but I couldn't...
First I thought exactly the same as you: some server process that would have some effect on the throughput. But I couldn't find one. Then I created the PHP script that resembles the C-code I saw, and that went through just fine.
If you can tell me how I can enable the alerter logs for send-email, I'll do that immediately. |
Comment by Andris Zeila [ 2023 Nov 14 ] |
It is sending to localhost, but the connection is not coming from 127.0.0.1. I assume you have SourceIP configured. After receiving data it waits for ~15s, connects to 'itself', sends extended helo, gets response, quits, waits for ~24s and tries sending 250 ok response back to the zabbix server, but connections seems to be already closed. What is is doing - I don't know. The only difference I see is the source address. Based on that my guess was it's some kind of spam protection. To increase log level for alerter processes (the ones sending notifications including emails) try |
Comment by Steven Van Ingelgem [ 2023 Nov 14 ] |
Hi,
I tested it with the SourceIP change (to 127.0.0.1) and the email came through immediately. Afterwards I modified the PHP script to use the ip of the server (seems it was locally redirected to localhost anyhow), and it started to timeout as well. So it's very likely a configuration issue at my side. So, I'm closing this ticket as it doesn't seem to be your problem after all 😏.
Grtz, |