[ZBX-10461] Zabbix (alerter process) crash on CentOS 7 Created: 2016 Feb 28 Updated: 2024 Apr 12 Resolved: 2017 Sep 20 |
|
Status: | Closed |
Project: | ZABBIX BUGS AND ISSUES |
Component/s: | None |
Affects Version/s: | 3.0.0 |
Fix Version/s: | None |
Type: | Incident report | Priority: | Trivial |
Reporter: | wesley wang | Assignee: | Unassigned |
Resolution: | Won't fix | Votes: | 3 |
Labels: | alerter, crash | ||
Remaining Estimate: | Not Specified | ||
Time Spent: | Not Specified | ||
Original Estimate: | Not Specified | ||
Environment: |
OS: CentOS Linux release 7.2.1511 (Core) |
Attachments: |
![]() ![]() ![]() ![]() |
||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
Issue Links: |
|
Description |
Zabbix crash and continue to restart. |
Comments |
Comment by richlv [ 2016 Feb 29 ] |
backtrace for easier searching : 13347:20160228:223018.881 Starting Zabbix Server. Zabbix 3.0.1 (revision 58734). ... 13726:20160228:223021.142 server #266 started [alerter #1] ... 13726:20160228:223021.278 Got signal [signal:11(SIGSEGV),reason:1,refaddr:0x50]. Crashing ... 13726:20160228:223021.278 ====== Fatal information: ====== ... 13726:20160228:223021.279 === Backtrace: === 13726:20160228:223021.279 20: /usr/sbin/zabbix_server: alerter [sending alerts](print_fatal_info+0x287) [0x487d77] 13726:20160228:223021.279 19: /usr/sbin/zabbix_server: alerter [sending alerts]() [0x488076] 13726:20160228:223021.279 18: /lib64/libc.so.6(+0x35670) [0x7eff0035b670] 13726:20160228:223021.279 17: /lib64/libnss3.so(PK11_DigestOp+0x1f) [0x7efefd91161f] 13726:20160228:223021.279 16: /lib64/libcurl.so.4(+0x4274a) [0x7eff00e4974a] 13726:20160228:223021.279 15: /lib64/libcurl.so.4(+0x4e519) [0x7eff00e55519] 13726:20160228:223021.279 14: /lib64/libcurl.so.4(+0x484cc) [0x7eff00e4f4cc] 13726:20160228:223021.280 13: /lib64/libcurl.so.4(+0x4721e) [0x7eff00e4e21e] 13726:20160228:223021.280 12: /lib64/libcurl.so.4(+0x32268) [0x7eff00e39268] 13726:20160228:223021.280 11: /lib64/libcurl.so.4(curl_multi_perform+0xd1) [0x7eff00e3a121] 13726:20160228:223021.280 10: /lib64/libcurl.so.4(curl_easy_perform+0xf3) [0x7eff00e314a3] 13726:20160228:223021.280 9: /usr/sbin/zabbix_server: alerter [sending alerts]() [0x48b197] 13726:20160228:223021.280 8: /usr/sbin/zabbix_server: alerter [sending alerts](send_email+0x1a9) [0x48b3d9] 13726:20160228:223021.280 7: /usr/sbin/zabbix_server: alerter [sending alerts](execute_action+0x33f) [0x41e1bf] 13726:20160228:223021.280 6: /usr/sbin/zabbix_server: alerter [sending alerts](alerter_thread+0x305) [0x41e635] 13726:20160228:223021.280 5: /usr/sbin/zabbix_server: alerter [sending alerts](zbx_thread_start+0x57) [0x488b27] 13726:20160228:223021.280 4: /usr/sbin/zabbix_server: alerter [sending alerts](MAIN_ZABBIX_ENTRY+0x41d) [0x41da8d] 13726:20160228:223021.280 3: /usr/sbin/zabbix_server: alerter [sending alerts](daemon_start+0x1da) [0x4872da] 13726:20160228:223021.280 2: /usr/sbin/zabbix_server: alerter [sending alerts](main+0x39f) [0x41812f] 13726:20160228:223021.280 1: /lib64/libc.so.6(__libc_start_main+0xf5) [0x7eff00347b15] 13726:20160228:223021.280 0: /usr/sbin/zabbix_server: alerter [sending alerts]() [0x4183e5] |
Comment by Aleksandrs Saveljevs [ 2016 Feb 29 ] |
Does it crash reliably? Could you please provide the DebugLevel=4 (or 5) log for the alerter process? What exact SMTP authentication settings do you use? What are the cURL and related crypto library versions? |
Comment by wesley wang [ 2016 Feb 29 ] |
Hi alek, |
Comment by Aleksandrs Saveljevs [ 2016 Mar 01 ] |
Thank you! It can be seen in the log that NTLM authentication is attempted, after which it crashes. What is the version of libnss3.so library? Does it work if cURL is linked with a different crypto library? |
Comment by wesley wang [ 2016 Mar 02 ] |
thanks, alek. BTW, i install all packages with yum, what have to do to make cURL link with other crypto lib. |
Comment by wesley wang [ 2016 Mar 02 ] |
Hi alek, |
Comment by Aleksandrs Saveljevs [ 2016 Mar 02 ] |
Could you please try sending an email using cURL from the command line? It should be something like this. First, prepare an email message: $ cat email.txt Subject: cURL email test This is an email to test cURL. Second, use that file with a command-line similar to the following, but using your Zabbix SMTP parameters: $ curl --mail-from '[email protected]' --mail-rcpt '[email protected]' --user '<username>:<password>' --url smtp://127.0.0.1:25 -T email.txt -v * Hostname was NOT found in DNS cache * Trying 127.0.0.1... % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 * Connected to 127.0.0.1 (127.0.0.1) port 25 (#0) < 220 mail.example.lan ESMTP Postfix (Debian/GNU) > EHLO email.txt < 250-mail.example.lan < 250-PIPELINING < 250-SIZE 10240000 < 250-VRFY < 250-ETRN < 250-STARTTLS < 250-ENHANCEDSTATUSCODES < 250-8BITMIME < 250 DSN > MAIL FROM:<[email protected]> SIZE=57 < 250 2.1.0 Ok > RCPT TO:<[email protected]> < 250 2.1.5 Ok > DATA < 354 End data with <CR><LF>.<CR><LF> } [data not shown] * We are completely uploaded and fine < 250 2.0.0 Ok: queued as 1F0E750E2E3 100 57 0 0 100 57 0 1434 --:--:-- --:--:-- --:--:-- 1461 * Connection #0 to host 127.0.0.1 left intact When Zabbix attempts to send an email, it uses NTLM authentication as seen in the log (because the SMTP server offers NTLM and LOGIN methods): 22217:20160229:212853.560 In send_email() smtp_server:'smtp.example.com' smtp_port:25 smtp_security:0 smtp_authentication:1 22217:20160229:212853.577 * About to connect() to smtp.example.com port 25 (#0) 22217:20160229:212853.591 * Trying 87.65.43.21... 22217:20160229:212853.633 * Connected to smtp.example.com (87.65.43.21) port 25 (#0) 22217:20160229:212853.663 < 220 smtp.example.com Microsoft ESMTP MAIL Service ready at Mon, 29 Feb 2016 21:28:58 +0800 22217:20160229:212853.677 > EHLO example.com 22217:20160229:212853.692 < 250-smtp.example.com Hello [12.34.56.78] 22217:20160229:212853.692 < 250-SIZE 22217:20160229:212853.693 < 250-PIPELINING 22217:20160229:212853.693 < 250-DSN 22217:20160229:212853.693 < 250-ENHANCEDSTATUSCODES 22217:20160229:212853.693 < 250-STARTTLS 22217:20160229:212853.694 < 250-X-ANONYMOUSTLS 22217:20160229:212853.694 < 250-AUTH NTLM LOGIN 22217:20160229:212853.695 < 250-X-EXPS GSSAPI NTLM 22217:20160229:212853.695 < 250-8BITMIME 22217:20160229:212853.695 < 250-BINARYMIME 22217:20160229:212853.696 < 250-CHUNKING 22217:20160229:212853.696 < 250-XEXCH50 22217:20160229:212853.696 < 250 XRDST 22217:20160229:212853.697 > AUTH NTLM TlRMTVNTUAABAAAABoIIAAAAAAAAAAAAAAAAAAAAAAA= 22217:20160229:212853.726 < 334 TlRMTVNTUAACAAAAAgACADgAAAAGgokC8NbEQBlhgZMAAAAAAAAAAKYApgA6AAAABgByFwAAAA9PQQIABABPAEEAAQAOAEkAVABIAFUAQgAwADIABAAgAG8AYQAuAGMAaQB0AGkAYwBiAGEAbgBrAC4AYwBvAG0AAwAwAGkAdABoAHUAYgAwADIALgBvAGEALgBjAGkAdABpAGMAYgBhAG4AawAuAGMAbwBtAAUAIABvAGEALgBjAGkAdABpAGMAYgBhAG4AawAuAGMAbwBtAAcACACcg2kl9XLRAQAAAAA= 22217:20160229:212853.726 Got signal [signal:11(SIGSEGV),reason:1,refaddr:0x50]. Crashing ... You should try to make cURL use NTLM authentication, too, and see whether it crashes or not. This may give us a hint on whether the problem is on Zabbix side. |
Comment by Aleksandrs Saveljevs [ 2016 Mar 02 ] |
Note also that according to the following Zabbix code libcurl either chooses some unspecified login method or PLAIN, depending on libcurl version: if (SMTP_AUTHENTICATION_NORMAL_PASSWORD == smtp_authentication) { #if 0x071e00 >= LIBCURL_VERSION_NUM /* versions 7.20.0 to 7.30.0 do not support specifying login options */ if (CURLE_OK != (err = curl_easy_setopt(easyhandle, CURLOPT_USERNAME, username)) || CURLE_OK != (err = curl_easy_setopt(easyhandle, CURLOPT_PASSWORD, password))) { goto error; } #elif 0x072100 >= LIBCURL_VERSION_NUM /* versions 7.31.0 to 7.33.0 support login options in CURLOPT_USERPWD */ char userpwd[523]; zbx_snprintf(userpwd, sizeof(userpwd), "%s:%s;AUTH=PLAIN", username, password); if (CURLE_OK != (err = curl_easy_setopt(easyhandle, CURLOPT_USERPWD, userpwd))) goto error; #else /* versions 7.34.0 and above support explicit CURLOPT_LOGIN_OPTIONS */ if (CURLE_OK != (err = curl_easy_setopt(easyhandle, CURLOPT_USERNAME, username)) || CURLE_OK != (err = curl_easy_setopt(easyhandle, CURLOPT_PASSWORD, password)) || CURLE_OK != (err = curl_easy_setopt(easyhandle, CURLOPT_LOGIN_OPTIONS, "AUTH=PLAIN"))) { goto error; } #endif } If specifying NTLM or LOGIN method is desired, that would probably be a feature request. What authentication method is expected to be used in your case? |
Comment by Aleksandrs Saveljevs [ 2016 Mar 02 ] |
One way would probably be to compile cURL and Zabbix server from sources. |
Comment by wesley wang [ 2016 Mar 02 ] |
Hi alex, root@zabbix ~]# cat sendmail.sh curl --mail-from '[email protected]' --mail-rcpt '[email protected]' --user 'username:password' --url smtp://smtp.example.com:25 -T email.txt -v OUTPUT: root@zabbix ~]# cat sendmailoutput.txt root@zabbix ~]# sh sendmail.sh % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0* About to connect() to smtp.example.com port 25 (#0) * Trying 1.2.3.4... * Connected to smtp.example.com (1.2.3.4) port 25 (#0) < 220 ithub02.oa.example.com Microsoft ESMTP MAIL Service ready at Wed, 2 Mar 2016 22:09:10 +0800 > EHLO email.txt < 250-ithub02.oa.example.com Hello [1.1.1.1] < 250-SIZE < 250-PIPELINING < 250-DSN < 250-ENHANCEDSTATUSCODES < 250-STARTTLS < 250-X-ANONYMOUSTLS < 250-AUTH NTLM LOGIN < 250-X-EXPS GSSAPI NTLM < 250-8BITMIME < 250-BINARYMIME < 250-CHUNKING < 250-XEXCH50 < 250 XRDST > AUTH NTLM TlRMTVNTUAABAAAABoIIAAAAAAAAAAAAAAAAAAAAAAA= < 334 TlRMTVNTUAACAAAAAgACADgAAAAGgokCeYVftTLVgG0AAAAAAAAAAKYApgA6AAAABgByFwAAAA9PQQIABABPAEEAAQAOAEkAVABIAFUAQgAwADIABAAgAG8AYQAuAGMAaQB0AGkAYwBiAGEAbgBrAC4AYwBvAG0AAwAwAGkAdABoAHUAYgAwADIALgBvAGEALgBjAGkAdABpAGMAYgBhAG4AawAuAGMAbwBtAAUAIABvAGEALgBjAGkAdABpAGMAYgBhAG4AawAuAGMAbwBtAAcACABrgrwXjXTRAQAAAAA= sendmail.sh: line 1: 6293 Segmentation fault curl --mail-from '[email protected]' --mail-rcpt '[email protected]' --user 'username:password' --url smtp://smtp.example.com:25 -T email.txt -v seems got segmentation fault when sending mail with curl command line. |
Comment by Aleksandrs Saveljevs [ 2016 Mar 02 ] |
Great! Seems there is nothing to fix on Zabbix side. You might wish to try the latest cURL and NSS versions, and report a bug there if it still crashes. |
Comment by Aleksandrs Saveljevs [ 2016 Mar 07 ] |
Closing as "Won't fix" as per the comment above. Also, if specifying NTLM or LOGIN authentication method is desired, please create a ZBXNEXT. |
Comment by Sandis Neilands (Inactive) [ 2016 Mar 07 ] |
Let's include this issue along |
Comment by Martins Valkovskis [ 2016 Mar 10 ] |
Well, we cannot include ZBX numbers per se in "Known issues". We need to formulate, in the simplest possible terms, the problem that some users have, the cause and the solution(s) we propose and then we can include the links to the ZBXs. And here I'd appreciate if the developers who did the investigation would formulate the first version of this information. |
Comment by Aleksandrs Saveljevs [ 2016 Mar 14 ] |
Well, this may be a bit non-trivial. The apparent problem is that when cURL 7.29.0 is used with NSS 3.19.1 as the crypto library, then it may crash when sending email alerts using NTLM authentication. The crash could probably be fixed(?) by either upgrading NSS or using cURL with a different crypto backend, but would fixing it make a lot of sense if Zabbix does not currently support neither NTLM, nor LOGIN authentication methods (we specify PLAIN if cURL is sufficiently recent, i.e. at least 7.31.0)? |
Comment by Sandis Neilands (Inactive) [ 2016 Mar 14 ] |
Is there a bug report regarding this somewhere in upstream? |
Comment by Aleksandrs Saveljevs [ 2016 Mar 14 ] |
I have looked for one, but have not found any. |
Comment by Ivan Jose [ 2016 Mar 25 ] |
Just meet the same. cURL - 7.29.0, NSS 3.19.1 as the crypto. Looking for a workaround. |
Comment by evand [ 2016 Apr 20 ] |
CentOS Linux release 7.2.1511 (Core) Got the same situation, and cost lots of time on investigation got frustrated. Once change media types (Email) from “normal password" to "none" then restarted zabbix-server successfully but got lots of " 23344:20160420:165344.674 wrong answer on MAIL FROM "530 5.7.1 Client was not authenticated" errors in zabbix-server.log. |
Comment by Aleksandrs Saveljevs [ 2016 Apr 20 ] |
Has anybody tried using a different crypto library as cURL's backend as a workaround? |
Comment by evand [ 2016 Apr 21 ] |
Ale, Guess I find a way to avoid that crashing...while having completely the same versions of libcurl and nss above. Today I found 3.0.2 released, so based on this , again I chose media types Email uses Security None and Normal password, Windows exchange E-mail account ,the same problem exists(/curl command)(Ok, the problem not mentioned in the release notes). However if set STARTTLS as the security connection then everything comes back normal, server running ok and emails sent out. With a closely attention paid on zabbix-server.log will see those warnings or something like that and I tried a lot of times. If this is the thing, then crash has nothing to do with the versions of libcurl or nss. It's the mail server settings forcing use STARTTLS to send/submit emails. |
Comment by Tairan Wang [ 2017 Jun 16 ] |
Zabbix 3.2 has same issue too. environment: |
Comment by Glebs Ivanovskis (Inactive) [ 2017 Jun 16 ] |
Backtrace is identical. Try suggested workarounds (upgrading libcurl, switching to a different libcurl flavour...). It's not a Zabbix bug. Seems like I've found a bug report upstream (marked as duplicate of a more general issue). Seems it hasn't been fixed on libnss side yet, but maybe newer versions of libcurl work this issue around. |
Comment by Artem Kst [ 2017 Aug 18 ] |
I also have this error. |
Comment by Aviram Alter [ 2017 Sep 03 ] |
So what is the solution here? this happens with the latest versions of libcurl and NSS. |
Comment by Glebs Ivanovskis (Inactive) [ 2017 Sep 03 ] |
Try a different flavour of libcurl (OpenSSL or GnuTLS). |
Comment by dimir [ 2017 Sep 20 ] |
Closing as Won't Fix based on comment above. |
Comment by Vladislavs Sokurenko [ 2018 Nov 29 ] |
Clarification: Where it shows that it crashes even if Zabbix is not involved. So not an issue in Zabbix. |
Comment by Hamid Gholami [ 2018 Dec 15 ] |
Hi dear, vso It is curl version: "curl 7.29.0 (x86_64-redhat-linux-gnu) libcurl/7.29.0 NSS/3.36 zlib/1.2.7 libidn/1.28 libssh2/1.4.3 Also I compile with openssl / Guntls and both of them failed with same error in log.
Are there any solution for upping zabbix server with current database? or should I run new database with new zabbix server? |
Comment by Vladislavs Sokurenko [ 2018 Dec 16 ] |
Hi h.gholami It is possible to move zabbix mysql database between servers if that’s what you need, however this tracker is for bug reports only, for help with upgrade you can contact our support or community, for example forums, irc channel or telegram, thanks ! |
Comment by Martins Valkovskis [ 2019 Jul 16 ] |
Listed in known issues from Zabbix 3.0 onward. |
Comment by Sivanesan Venkatesan [ 2020 Feb 05 ] |
Hi, Please try with the latest CURL and NSS versions will fix the issue.
Thanks and Regards, Sivanesan Venkatesan |