[ZBX-10461] Zabbix (alerter process) crash on CentOS 7 Created: 2016 Feb 28  Updated: 2024 Apr 12  Resolved: 2017 Sep 20

Status: Closed
Project: ZABBIX BUGS AND ISSUES
Component/s: None
Affects Version/s: 3.0.0
Fix Version/s: None

Type: Incident report Priority: Trivial
Reporter: wesley wang Assignee: Unassigned
Resolution: Won't fix Votes: 3
Labels: alerter, crash
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified
Environment:

OS: CentOS Linux release 7.2.1511 (Core)


Attachments: PNG File smtp_settings.png     File zabbix-server.3.2.crashing     Text File zabbix_crash.txt     Text File zabbix_server.log    
Issue Links:
Causes
Duplicate
is duplicated by ZBX-11118 zabbix-server 3.0.4 can't start Closed
is duplicated by ZBX-11599 Zabbix crash on CentOS 7 after update Closed
is duplicated by ZBX-15781 Zabbix problem with SMTP authentication Closed
is duplicated by ZBX-10757 Got signal [signal:11(SIGSEGV),reason... Closed
is duplicated by ZBX-11370 Zabbix Server constantly crash Closed
is duplicated by ZBX-11716 Got signal [signal:11(SIGSEGV),reason... Closed
is duplicated by ZBX-11742 Got signal [signal:11(SIGSEGV),reason... Closed
is duplicated by ZBX-14449 zabbix server constantly crashed Closed
is duplicated by ZBX-15234 One child process died,Zabbix Server ... Closed
is duplicated by ZBX-15292 Zabbix-server not running after restart Closed
is duplicated by ZBX-16367 When enable Exportdir,Zabbixserver crash Closed
is duplicated by ZBX-10898 Zabbix Server 3.0.3 Crashing ... (... Closed
is duplicated by ZBX-12665 Zabbix 3.4.1 crashing Closed
is duplicated by ZBX-13120 Server crash in alerter send_email() ... Closed
is duplicated by ZBX-13700 Zabbix server crashed after trying to... Closed
is duplicated by ZBX-10685 Escalator crash sending mail Zabbix 3... Closed
is duplicated by ZBX-11953 Zabbix server crashed on email send Closed
is duplicated by ZBX-12753 Zabbix server crash after email media... Closed
is duplicated by ZBX-18665 zabbix-server.service failed Closed

 Description   

Zabbix crash and continue to restart.



 Comments   
Comment by richlv [ 2016 Feb 29 ]

backtrace for easier searching :

 13347:20160228:223018.881 Starting Zabbix Server. Zabbix 3.0.1 (revision 58734).
...
 13726:20160228:223021.142 server #266 started [alerter #1]
...
 13726:20160228:223021.278 Got signal [signal:11(SIGSEGV),reason:1,refaddr:0x50]. Crashing ...
 13726:20160228:223021.278 ====== Fatal information: ======
...
 13726:20160228:223021.279 === Backtrace: ===
 13726:20160228:223021.279 20: /usr/sbin/zabbix_server: alerter [sending alerts](print_fatal_info+0x287) [0x487d77]
 13726:20160228:223021.279 19: /usr/sbin/zabbix_server: alerter [sending alerts]() [0x488076]
 13726:20160228:223021.279 18: /lib64/libc.so.6(+0x35670) [0x7eff0035b670]
 13726:20160228:223021.279 17: /lib64/libnss3.so(PK11_DigestOp+0x1f) [0x7efefd91161f]
 13726:20160228:223021.279 16: /lib64/libcurl.so.4(+0x4274a) [0x7eff00e4974a]
 13726:20160228:223021.279 15: /lib64/libcurl.so.4(+0x4e519) [0x7eff00e55519]
 13726:20160228:223021.279 14: /lib64/libcurl.so.4(+0x484cc) [0x7eff00e4f4cc]
 13726:20160228:223021.280 13: /lib64/libcurl.so.4(+0x4721e) [0x7eff00e4e21e]
 13726:20160228:223021.280 12: /lib64/libcurl.so.4(+0x32268) [0x7eff00e39268]
 13726:20160228:223021.280 11: /lib64/libcurl.so.4(curl_multi_perform+0xd1) [0x7eff00e3a121]
 13726:20160228:223021.280 10: /lib64/libcurl.so.4(curl_easy_perform+0xf3) [0x7eff00e314a3]
 13726:20160228:223021.280 9: /usr/sbin/zabbix_server: alerter [sending alerts]() [0x48b197]
 13726:20160228:223021.280 8: /usr/sbin/zabbix_server: alerter [sending alerts](send_email+0x1a9) [0x48b3d9]
 13726:20160228:223021.280 7: /usr/sbin/zabbix_server: alerter [sending alerts](execute_action+0x33f) [0x41e1bf]
 13726:20160228:223021.280 6: /usr/sbin/zabbix_server: alerter [sending alerts](alerter_thread+0x305) [0x41e635]
 13726:20160228:223021.280 5: /usr/sbin/zabbix_server: alerter [sending alerts](zbx_thread_start+0x57) [0x488b27]
 13726:20160228:223021.280 4: /usr/sbin/zabbix_server: alerter [sending alerts](MAIN_ZABBIX_ENTRY+0x41d) [0x41da8d]
 13726:20160228:223021.280 3: /usr/sbin/zabbix_server: alerter [sending alerts](daemon_start+0x1da) [0x4872da]
 13726:20160228:223021.280 2: /usr/sbin/zabbix_server: alerter [sending alerts](main+0x39f) [0x41812f]
 13726:20160228:223021.280 1: /lib64/libc.so.6(__libc_start_main+0xf5) [0x7eff00347b15]
 13726:20160228:223021.280 0: /usr/sbin/zabbix_server: alerter [sending alerts]() [0x4183e5]
Comment by Aleksandrs Saveljevs [ 2016 Feb 29 ]

Does it crash reliably? Could you please provide the DebugLevel=4 (or 5) log for the alerter process? What exact SMTP authentication settings do you use? What are the cURL and related crypto library versions?

Comment by wesley wang [ 2016 Feb 29 ]

Hi alek,
DebugLevel 5 log and smtp settings attached
libcurl version: libcurl-7.29.0-25.el7.centos.x86_64

Comment by Aleksandrs Saveljevs [ 2016 Mar 01 ]

Thank you! It can be seen in the log that NTLM authentication is attempted, after which it crashes.

What is the version of libnss3.so library? Does it work if cURL is linked with a different crypto library?

Comment by wesley wang [ 2016 Mar 02 ]

thanks, alek.
nss version is: nss-3.19.1-19.el7_2.x86_64
actually, i haven't used any NTLM authentication at all. even after i have disabled all media type and remove any media in Users configuration, zabbix server still crash and auto restart repeatedly.

BTW, i install all packages with yum, what have to do to make cURL link with other crypto lib.

Comment by wesley wang [ 2016 Mar 02 ]

Hi alek,
after i change from 'normal password' to 'none' in default email media type setting, zabbix-server start normally.
so why i failed even if media type is disabled. _
thanks very much for your patience and help me out.

Comment by Aleksandrs Saveljevs [ 2016 Mar 02 ]

Could you please try sending an email using cURL from the command line?

It should be something like this. First, prepare an email message:

$ cat email.txt 
Subject: cURL email test

This is an email to test cURL.

Second, use that file with a command-line similar to the following, but using your Zabbix SMTP parameters:

$ curl --mail-from '[email protected]' --mail-rcpt '[email protected]' --user '<username>:<password>' --url smtp://127.0.0.1:25 -T email.txt -v
* Hostname was NOT found in DNS cache
*   Trying 127.0.0.1...
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0
* Connected to 127.0.0.1 (127.0.0.1) port 25 (#0)
< 220 mail.example.lan ESMTP Postfix (Debian/GNU)
> EHLO email.txt
< 250-mail.example.lan
< 250-PIPELINING
< 250-SIZE 10240000
< 250-VRFY
< 250-ETRN
< 250-STARTTLS
< 250-ENHANCEDSTATUSCODES
< 250-8BITMIME
< 250 DSN
> MAIL FROM:<[email protected]> SIZE=57
< 250 2.1.0 Ok
> RCPT TO:<[email protected]>
< 250 2.1.5 Ok
> DATA
< 354 End data with <CR><LF>.<CR><LF>
} [data not shown]
* We are completely uploaded and fine
< 250 2.0.0 Ok: queued as 1F0E750E2E3
100    57    0     0  100    57      0   1434 --:--:-- --:--:-- --:--:--  1461
* Connection #0 to host 127.0.0.1 left intact

When Zabbix attempts to send an email, it uses NTLM authentication as seen in the log (because the SMTP server offers NTLM and LOGIN methods):

 22217:20160229:212853.560 In send_email() smtp_server:'smtp.example.com' smtp_port:25 smtp_security:0 smtp_authentication:1
 22217:20160229:212853.577 * About to connect() to smtp.example.com port 25 (#0)
 22217:20160229:212853.591 *   Trying 87.65.43.21...
 22217:20160229:212853.633 * Connected to smtp.example.com (87.65.43.21) port 25 (#0)
 22217:20160229:212853.663 < 220 smtp.example.com Microsoft ESMTP MAIL Service ready at Mon, 29 Feb 2016 21:28:58 +0800
 22217:20160229:212853.677 > EHLO example.com
 22217:20160229:212853.692 < 250-smtp.example.com Hello [12.34.56.78]
 22217:20160229:212853.692 < 250-SIZE
 22217:20160229:212853.693 < 250-PIPELINING
 22217:20160229:212853.693 < 250-DSN
 22217:20160229:212853.693 < 250-ENHANCEDSTATUSCODES
 22217:20160229:212853.693 < 250-STARTTLS
 22217:20160229:212853.694 < 250-X-ANONYMOUSTLS
 22217:20160229:212853.694 < 250-AUTH NTLM LOGIN
 22217:20160229:212853.695 < 250-X-EXPS GSSAPI NTLM
 22217:20160229:212853.695 < 250-8BITMIME
 22217:20160229:212853.695 < 250-BINARYMIME
 22217:20160229:212853.696 < 250-CHUNKING
 22217:20160229:212853.696 < 250-XEXCH50
 22217:20160229:212853.696 < 250 XRDST
 22217:20160229:212853.697 > AUTH NTLM TlRMTVNTUAABAAAABoIIAAAAAAAAAAAAAAAAAAAAAAA=
 22217:20160229:212853.726 < 334 TlRMTVNTUAACAAAAAgACADgAAAAGgokC8NbEQBlhgZMAAAAAAAAAAKYApgA6AAAABgByFwAAAA9PQQIABABPAEEAAQAOAEkAVABIAFUAQgAwADIABAAgAG8AYQAuAGMAaQB0AGkAYwBiAGEAbgBrAC4AYwBvAG0AAwAwAGkAdABoAHUAYgAwADIALgBvAGEALgBjAGkAdABpAGMAYgBhAG4AawAuAGMAbwBtAAUAIABvAGEALgBjAGkAdABpAGMAYgBhAG4AawAuAGMAbwBtAAcACACcg2kl9XLRAQAAAAA=
 22217:20160229:212853.726 Got signal [signal:11(SIGSEGV),reason:1,refaddr:0x50]. Crashing ...

You should try to make cURL use NTLM authentication, too, and see whether it crashes or not. This may give us a hint on whether the problem is on Zabbix side.

Comment by Aleksandrs Saveljevs [ 2016 Mar 02 ]

Note also that according to the following Zabbix code libcurl either chooses some unspecified login method or PLAIN, depending on libcurl version:

	if (SMTP_AUTHENTICATION_NORMAL_PASSWORD == smtp_authentication)
	{
#if 0x071e00 >= LIBCURL_VERSION_NUM	/* versions 7.20.0 to 7.30.0 do not support specifying login options */
		if (CURLE_OK != (err = curl_easy_setopt(easyhandle, CURLOPT_USERNAME, username)) ||
				CURLE_OK != (err = curl_easy_setopt(easyhandle, CURLOPT_PASSWORD, password)))
		{
			goto error;
		}
#elif 0x072100 >= LIBCURL_VERSION_NUM	/* versions 7.31.0 to 7.33.0 support login options in CURLOPT_USERPWD */
		char	userpwd[523];

		zbx_snprintf(userpwd, sizeof(userpwd), "%s:%s;AUTH=PLAIN", username, password);

		if (CURLE_OK != (err = curl_easy_setopt(easyhandle, CURLOPT_USERPWD, userpwd)))
			goto error;
#else					/* versions 7.34.0 and above support explicit CURLOPT_LOGIN_OPTIONS */
		if (CURLE_OK != (err = curl_easy_setopt(easyhandle, CURLOPT_USERNAME, username)) ||
				CURLE_OK != (err = curl_easy_setopt(easyhandle, CURLOPT_PASSWORD, password)) ||
				CURLE_OK != (err = curl_easy_setopt(easyhandle, CURLOPT_LOGIN_OPTIONS, "AUTH=PLAIN")))
		{
			goto error;
		}
#endif
	}

If specifying NTLM or LOGIN method is desired, that would probably be a feature request. What authentication method is expected to be used in your case?

Comment by Aleksandrs Saveljevs [ 2016 Mar 02 ]

BTW, i install all packages with yum, what have to do to make cURL link with other crypto lib.

One way would probably be to compile cURL and Zabbix server from sources.

Comment by wesley wang [ 2016 Mar 02 ]

Hi alex,
here is my command:

root@zabbix ~]# cat sendmail.sh
curl --mail-from '[email protected]' --mail-rcpt '[email protected]' --user 'username:password' --url smtp://smtp.example.com:25 -T email.txt -v

OUTPUT:

root@zabbix ~]# cat sendmailoutput.txt
root@zabbix ~]# sh sendmail.sh
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0* About to connect() to smtp.example.com port 25 (#0)
*   Trying 1.2.3.4...
* Connected to smtp.example.com (1.2.3.4) port 25 (#0)
< 220 ithub02.oa.example.com Microsoft ESMTP MAIL Service ready at Wed, 2 Mar 2016 22:09:10 +0800
> EHLO email.txt
< 250-ithub02.oa.example.com Hello [1.1.1.1]
< 250-SIZE
< 250-PIPELINING
< 250-DSN
< 250-ENHANCEDSTATUSCODES
< 250-STARTTLS
< 250-X-ANONYMOUSTLS
< 250-AUTH NTLM LOGIN
< 250-X-EXPS GSSAPI NTLM
< 250-8BITMIME
< 250-BINARYMIME
< 250-CHUNKING
< 250-XEXCH50
< 250 XRDST
> AUTH NTLM TlRMTVNTUAABAAAABoIIAAAAAAAAAAAAAAAAAAAAAAA=
< 334 TlRMTVNTUAACAAAAAgACADgAAAAGgokCeYVftTLVgG0AAAAAAAAAAKYApgA6AAAABgByFwAAAA9PQQIABABPAEEAAQAOAEkAVABIAFUAQgAwADIABAAgAG8AYQAuAGMAaQB0AGkAYwBiAGEAbgBrAC4AYwBvAG0AAwAwAGkAdABoAHUAYgAwADIALgBvAGEALgBjAGkAdABpAGMAYgBhAG4AawAuAGMAbwBtAAUAIABvAGEALgBjAGkAdABpAGMAYgBhAG4AawAuAGMAbwBtAAcACABrgrwXjXTRAQAAAAA=
sendmail.sh: line 1:  6293 Segmentation fault      curl --mail-from '[email protected]' --mail-rcpt '[email protected]' --user 'username:password' --url smtp://smtp.example.com:25 -T email.txt -v

seems got segmentation fault when sending mail with curl command line.

Comment by Aleksandrs Saveljevs [ 2016 Mar 02 ]

Great! Seems there is nothing to fix on Zabbix side.

You might wish to try the latest cURL and NSS versions, and report a bug there if it still crashes.

Comment by Aleksandrs Saveljevs [ 2016 Mar 07 ]

Closing as "Won't fix" as per the comment above. Also, if specifying NTLM or LOGIN authentication method is desired, please create a ZBXNEXT.

Comment by Sandis Neilands (Inactive) [ 2016 Mar 07 ]

Let's include this issue along ZBX-10486 to "Known issues" section in documentation.

Comment by Martins Valkovskis [ 2016 Mar 10 ]

Well, we cannot include ZBX numbers per se in "Known issues". We need to formulate, in the simplest possible terms, the problem that some users have, the cause and the solution(s) we propose and then we can include the links to the ZBXs. And here I'd appreciate if the developers who did the investigation would formulate the first version of this information.

Comment by Aleksandrs Saveljevs [ 2016 Mar 14 ]

Well, this may be a bit non-trivial. The apparent problem is that when cURL 7.29.0 is used with NSS 3.19.1 as the crypto library, then it may crash when sending email alerts using NTLM authentication. The crash could probably be fixed(?) by either upgrading NSS or using cURL with a different crypto backend, but would fixing it make a lot of sense if Zabbix does not currently support neither NTLM, nor LOGIN authentication methods (we specify PLAIN if cURL is sufficiently recent, i.e. at least 7.31.0)?

Comment by Sandis Neilands (Inactive) [ 2016 Mar 14 ]

Is there a bug report regarding this somewhere in upstream?

Comment by Aleksandrs Saveljevs [ 2016 Mar 14 ]

I have looked for one, but have not found any.

Comment by Ivan Jose [ 2016 Mar 25 ]

Just meet the same. cURL - 7.29.0, NSS 3.19.1 as the crypto. Looking for a workaround.

Comment by evand [ 2016 Apr 20 ]

CentOS Linux release 7.2.1511 (Core)
zabbix-server-mysql-3.0.1-1.el7.x86_64
libcurl-7.29.0-25.el7.centos.x86_64
nss-3.19.1-19.el7_2.x86_64

Got the same situation, and cost lots of time on investigation got frustrated. Once change media types (Email) from “normal password" to "none" then restarted zabbix-server successfully but got lots of " 23344:20160420:165344.674 wrong answer on MAIL FROM "530 5.7.1 Client was not authenticated" errors in zabbix-server.log.

Comment by Aleksandrs Saveljevs [ 2016 Apr 20 ]

Has anybody tried using a different crypto library as cURL's backend as a workaround?

Comment by evand [ 2016 Apr 21 ]

Ale,

Guess I find a way to avoid that crashing...while having completely the same versions of libcurl and nss above. Today I found 3.0.2 released, so based on this , again I chose media types Email uses Security None and Normal password, Windows exchange E-mail account ,the same problem exists(/curl command)(Ok, the problem not mentioned in the release notes). However if set STARTTLS as the security connection then everything comes back normal, server running ok and emails sent out. With a closely attention paid on zabbix-server.log will see those warnings or something like that and I tried a lot of times.

If this is the thing, then crash has nothing to do with the versions of libcurl or nss. It's the mail server settings forcing use STARTTLS to send/submit emails.

Comment by Tairan Wang [ 2017 Jun 16 ]

Zabbix 3.2 has same issue too.

environment:
CentOS Linux release 7.3.1611 (Core)
Hyper-V 2012 R2

Comment by Glebs Ivanovskis (Inactive) [ 2017 Jun 16 ]

Backtrace is identical. Try suggested workarounds (upgrading libcurl, switching to a different libcurl flavour...).

It's not a Zabbix bug. Seems like I've found a bug report upstream (marked as duplicate of a more general issue). Seems it hasn't been fixed on libnss side yet, but maybe newer versions of libcurl work this issue around.

Comment by Artem Kst [ 2017 Aug 18 ]

I also have this error.
To get the error, I do the following:
set the smtp settings as shown in the screenshot in the header ZBX-10461. Then make the server reboot. After the reboot when triger is true, the server crash.
However, if you set other settings(before rebooting after crash), for example, set the ssl \ tls encryption after this, restart. When trigger is true, I had an error "Timeout was reached: Operation timed out after 40001 milliseconds with 0 out of 0 bytes received", but server is not in crash.
And if now back to return the original settings, without reboot - the EMAIL NOTIFICATION START TO WORK, but before the first server reboot.
I use:
CentOS Linux release 7.3.1611 Minimal
Installation from the package
Zabbix 3.2.7 (revision 70298)
Mariadb

Comment by Aviram Alter [ 2017 Sep 03 ]

So what is the solution here? this happens with the latest versions of libcurl and NSS.

Comment by Glebs Ivanovskis (Inactive) [ 2017 Sep 03 ]

Try a different flavour of libcurl (OpenSSL or GnuTLS).

Comment by dimir [ 2017 Sep 20 ]

Closing as Won't Fix based on comment above.

Comment by Vladislavs Sokurenko [ 2018 Nov 29 ]

Clarification:
Please see comment

Where it shows that it crashes even if Zabbix is not involved. So not an issue in Zabbix.

Comment by Hamid Gholami [ 2018 Dec 15 ]

Hi dear, vso 

It is curl version: "curl 7.29.0 (x86_64-redhat-linux-gnu) libcurl/7.29.0 NSS/3.36 zlib/1.2.7 libidn/1.28 libssh2/1.4.3
Protocols: dict file ftp ftps gopher http https imap imaps ldap ldaps pop3 pop3s rtsp scp sftp smtp smtps telnet tftp
Features: AsynchDNS GSS-Negotiate IDN IPv6 Largefile NTLM NTLM_WB SSL libz unix-sockets"

Also I compile with openssl / Guntls and both of them failed with same error in log.

 

Are there any solution for upping zabbix server with current database? or should I run new database with new zabbix server?

Comment by Vladislavs Sokurenko [ 2018 Dec 16 ]

Hi h.gholami It is possible to move zabbix mysql database between servers if that’s what you need, however this tracker is for bug reports only, for help with upgrade you can contact our support or community, for example forums, irc channel or telegram, thanks !

Comment by Martins Valkovskis [ 2019 Jul 16 ]

Listed in known issues from Zabbix 3.0 onward.

Comment by Sivanesan Venkatesan [ 2020 Feb 05 ]

Hi,

Please try with the latest CURL and NSS versions will fix the issue.

 

Thanks and Regards,

Sivanesan Venkatesan

Generated at Fri Jun 06 20:20:42 EEST 2025 using Jira 9.12.4#9120004-sha1:625303b708afdb767e17cb2838290c41888e9ff0.