Linux fedora-virtualdomain 3.12.9-301.fc20.x86_64 #1 SMP Wed Jan 29 15:56:22 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux

systemd kills the server after the SIGSEGV.

zabbix-server.log doesnt show the problem.

When I make zabbixsvr user logeable then login with it:

bash-4.2$ zabbix_server_mysql
Segmentation fault (core dumped)

bash-4.2$ ps -aux | grep zabbix
root 8845 0.0 0.4 151756 5084 pts/3 S+ 13:35 0:00 vim zabbix_server.conf
root 9153 0.0 0.5 202220 5272 pts/4 S 13:39 0:00 su zabbixsrv
zabbixs+ 9162 0.0 0.1 115344 1948 pts/4 S 13:40 0:00 bash
zabbixs+ 9173 0.0 0.3 247360 3748 ? S 13:40 0:00 zabbix_server_mysql
zabbixs+ 9176 0.0 0.2 247368 2680 ? S 13:40 0:00 zabbix_server_mysql
zabbixs+ 9177 0.0 0.2 247360 2252 ? S 13:40 0:00 zabbix_server_mysql
zabbixs+ 9178 0.0 0.4 351780 4668 ? S 13:40 0:00 zabbix_server_mysql
zabbixs+ 9179 0.0 0.4 351780 4656 ? S 13:40 0:00 zabbix_server_mysql
zabbixs+ 9180 0.0 0.4 351780 4644 ? S 13:40 0:00 zabbix_server_mysql
zabbixs+ 9181 0.0 0.4 351780 4644 ? S 13:40 0:00 zabbix_server_mysql
zabbixs+ 9182 0.0 0.4 351780 4628 ? S 13:40 0:00 zabbix_server_mysql
zabbixs+ 9183 0.0 0.4 351780 4672 ? S 13:40 0:00 zabbix_server_mysql
zabbixs+ 9184 0.0 0.2 247380 2300 ? S 13:40 0:00 zabbix_server_mysql
zabbixs+ 9185 0.0 0.2 247360 2084 ? S 13:40 0:00 zabbix_server_mysql
zabbixs+ 9186 0.0 0.2 247380 2300 ? S 13:40 0:00 zabbix_server_mysql
zabbixs+ 9191 0.0 0.2 247380 2300 ? S 13:40 0:00 zabbix_server_mysql
zabbixs+ 9192 0.0 0.2 247380 2300 ? S 13:40 0:00 zabbix_server_mysql
zabbixs+ 9194 0.0 0.2 247708 2296 ? S 13:40 0:00 zabbix_server_mysql
zabbixs+ 9195 0.0 0.2 247360 2236 ? S 13:40 0:00 zabbix_server_mysql
zabbixs+ 9197 0.0 0.2 247360 2196 ? S 13:40 0:00 zabbix_server_mysql
zabbixs+ 9198 0.0 0.2 247380 2740 ? S 13:40 0:00 zabbix_server_mysql
zabbixs+ 9199 0.0 0.2 247360 2284 ? S 13:40 0:00 zabbix_server_mysql
zabbixs+ 9205 0.0 0.4 351720 4432 ? S 13:40 0:00 zabbix_server_mysql
zabbixs+ 9206 0.0 0.2 247380 2892 ? S 13:40 0:00 zabbix_server_mysql
zabbixs+ 9207 0.0 0.2 247380 2888 ? S 13:40 0:00 zabbix_server_mysql
zabbixs+ 9208 0.0 0.2 247380 2892 ? S 13:40 0:00 zabbix_server_mysql
zabbixs+ 9209 0.0 0.2 247380 2888 ? S 13:40 0:00 zabbix_server_mysql
zabbixs+ 9213 0.0 0.2 247360 2288 ? S 13:40 0:00 zabbix_server_mysql
zabbixs+ 9214 0.0 0.2 247360 2292 ? S 13:40 0:00 zabbix_server_mysql
zabbixs+ 9216 0.0 0.2 247360 2040 ? S 13:40 0:00 zabbix_server_mysql
zabbixs+ 9354 0.0 0.1 123356 1392 pts/4 R+ 13:44 0:00 ps -aux
zabbixs+ 9355 0.0 0.0 112664 952 pts/4 S+ 13:44 0:00 grep zabbix

The servers stays running and 'working'

recompressed logfile with bz2 (and it's smaller as well...)

possibly useful contents from the logfile :

 13223:20140211:155619.238 Starting Zabbix Server. Zabbix 2.0.10 (revision 40809).

 13223:20140211:155619.434 End of DCsync_configuration()
 13223:20140211:155619.435 Got signal [signal:15(SIGTERM),sender_pid:1,sender_uid:0,reason:0]. Exiting ...

 13223:20140211:155621.445 Zabbix Server stopped. Zabbix 2.0.10 (revision 40809).
 13223:20140211:155621.445 Got signal [signal:11(SIGSEGV),reason:1,refaddr:0x18]. Crashing ...
 13223:20140211:155621.446 ====== Fatal information: ======
 13223:20140211:155621.446 Program counter: 0x7fcb34902fe0
 13223:20140211:155621.446 === Registers: ===
 13223:20140211:155621.446 r8      =                0 =                    0 =                    0
 13223:20140211:155621.446 r9      =     7fcb34ed1ea8 =      140510743043752 =      140510743043752
 13223:20140211:155621.446 r10     =     7fffdcaa0350 =      140736895517520 =      140736895517520
 13223:20140211:155621.446 r11     =     7fcb34c18640 =      140510740186688 =      140510740186688
 13223:20140211:155621.446 r12     =     7fcb30fe4bc0 =      140510677060544 =      140510677060544
 13223:20140211:155621.446 r13     =               20 =                   32 =                   32
 13223:20140211:155621.446 r14     =                1 =                    1 =                    1
 13223:20140211:155621.446 r15     =     7fffdcaa05b0 =      140736895518128 =      140736895518128
 13223:20140211:155621.446 rdi     =                8 =                    8 =                    8
 13223:20140211:155621.446 rsi     =                1 =                    1 =                    1
 13223:20140211:155621.446 rbp     =     7fffdcaa07f0 =      140736895518704 =      140736895518704
 13223:20140211:155621.446 rbx     =     7fcb376c39a0 =      140510784928160 =      140510784928160
 13223:20140211:155621.446 rdx     =                1 =                    1 =                    1
 13223:20140211:155621.446 rax     =     7fcb34902fe0 =      140510736953312 =      140510736953312
 13223:20140211:155621.447 rcx     =                0 =                    0 =                    0
 13223:20140211:155621.447 rsp     =     7fffdcaa0588 =      140736895518088 =      140736895518088
 13223:20140211:155621.447 rip     =     7fcb34902fe0 =      140510736953312 =      140510736953312
 13223:20140211:155621.447 efl     =            10202 =                66050 =                66050
 13223:20140211:155621.447 csgsfs  =               33 =                   51 =                   51
 13223:20140211:155621.447 err     =                4 =                    4 =                    4
 13223:20140211:155621.447 trapno  =                e =                   14 =                   14
 13223:20140211:155621.447 oldmask =            14000 =                81920 =                81920
 13223:20140211:155621.447 cr2     =               18 =                   24 =                   24
 13223:20140211:155621.447 === Backtrace: ===
 13223:20140211:155621.449 15: /usr/sbin/zabbix_server_mysql(print_fatal_info+0x287) [0x44fec7]
 13223:20140211:155621.449 14: /usr/sbin/zabbix_server_mysql() [0x44ef26]
 13223:20140211:155621.449 13: /lib64/libc.so.6(+0x35cb0) [0x7fcb34b4bcb0]
 13223:20140211:155621.449 12: /lib64/libpthread.so.0(pthread_mutex_lock+0) [0x7fcb34902fe0]
 13223:20140211:155621.449 11: /lib64/libtspi.so.1(host_table_final+0x14) [0x7fcb30d9abc4]
 13223:20140211:155621.449 10: /lib64/ld-linux-x86-64.so.2(+0xf8ea) [0x7fcb374d68ea]
 13223:20140211:155621.449 9: /lib64/libc.so.6(+0x394c9) [0x7fcb34b4f4c9]
 13223:20140211:155621.449 8: /lib64/libc.so.6(+0x39515) [0x7fcb34b4f515]
 13223:20140211:155621.449 7: /usr/sbin/zabbix_server_mysql(zbx_on_exit+0x15f) [0x41470f]
 13223:20140211:155621.449 6: /lib64/libc.so.6(+0x35cb0) [0x7fcb34b4bcb0]
 13223:20140211:155621.449 5: /lib64/libc.so.6(socket+0x7) [0x7fcb34c0bf67]
 13223:20140211:155621.449 4: /usr/sbin/zabbix_server_mysql(zbx_tcp_listen+0x196) [0x45e0d6]
 13223:20140211:155621.449 3: /usr/sbin/zabbix_server_mysql(MAIN_ZABBIX_ENTRY+0x2b3) [0x4149d3]
 13223:20140211:155621.449 2: /usr/sbin/zabbix_server_mysql(daemon_start+0x23d) [0x44f48d]
 13223:20140211:155621.449 1: /lib64/libc.so.6(__libc_start_main+0xf5) [0x7fcb34b37d65]
 13223:20140211:155621.449 0: /usr/sbin/zabbix_server_mysql() [0x41066d]

I have exactly same issue.

those who see this, how did you install zabbix ?

I installed it with yum.

But after failure, i removed zabbix and compiled it from sources. Results are same..

which package repository did you use ?
when compiling from sources, how do you start the server ?

For RPMs i used Fedora default repos, just typed yum install zabbix zabbix-server etc.

In compiled version, i started it with command: /usr/local/sbin/zabbix_server -c /usr/local/etc/zabbix_server.conf

The way I installed it was very similar to this:
http://tecadmin.net/install-zabbix-network-monitoring-on-centos-rhel-and-fedora/#

but i did not configured any special repositories

it did not worked out of the box, I made the following changes

mkdir /var/run/zabbix-server
mkdir /var/run/zabbix-agent
mkdir /var/log/zabbix

usermod -a -G zabbix zabbix
usermod -a -G zabbixsvr zabbix

chown zabbixsvr:zabbix /var/run/zabbix-server
chown zabbix:zabbix /var/run/zabbix-agent
chown root:zabbix /var/log/zabbix

had troubles with SELinux
tail -f /var/log/audit/audit.log |grep -i avc

setsebool httpd_can_network_connect on
setsebool zabbix_can_network_connect on

and another to the database connection... until audit.log was not showing trouble

attaching my configuration files

About the repositories im using basic fedora 20

[root@fedora-virtualdomain cservin]# yum repolist
Loaded plugins: langpacks, refresh-packagekit
repo id repo name status
fedora/20/x86_64 Fedora 20 - x86_64 38,597
google-chrome google-chrome 3
!updates/20/x86_64 Fedora 20 - x86_64 - Updates 12,620
repolist: 51,220

[root@fedora-virtualdomain cservin]# yum info zabbix-server
Loaded plugins: langpacks, refresh-packagekit
Installed Packages
Name : zabbix-server
Arch : x86_64
Version : 2.0.10
Release : 2.fc20
Size : 14 k
Repo : installed
From repo : updates
Summary : Zabbix server common files
URL : http://www.zabbix.com
License : GPLv2+
Description : Zabbix server common files

btw, if possible, use xz/bzip2/gzip to compress the files (preferably, in that order )

It's all about flavor

I have the same issue after update to Fedora 20.

(gdb) run
Starting program: /usr/sbin/zabbix_server
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/libthread_db.so.1".
[New Thread 0xb6b30b40 (LWP 5643)]
[Thread 0xb6b30b40 (LWP 5643) exited]
Detaching after fork from child process 5644.

Program received signal SIGSEGV, Segmentation fault.
_GI__pthread_mutex_lock (mutex=0x4) at ../nptl/pthread_mutex_lock.c:66
66 unsigned int type = PTHREAD_MUTEX_TYPE_ELISION (mutex);
(gdb) bt
#0 _GI__pthread_mutex_lock (mutex=0x4) at ../nptl/pthread_mutex_lock.c:66
#1 0xb761b594 in pthread_mutex_lock (mutex=0x4) at forward.c:192
#2 0xb6da3084 in host_table_final () from /lib/libtspi.so.1
#3 0xb6d979f5 in my_fini () from /lib/libtspi.so.1
#4 0xb7feda84 in _dl_fini () at dl-fini.c:252
#5 0xb7548011 in _run_exit_handlers (status=status@entry=0, listp=0xb76d03c4 <_exit_funcs>,
run_list_atexit=run_list_atexit@entry=true) at exit.c:82
#6 0xb754806d in __GI_exit (status=0) at exit.c:104
#7 0x0809ec72 in daemon_start ()
#8 0x08054d50 in main ()
(gdb)

Bugreport in Fedora bugtracker: https://bugzilla.redhat.com/show_bug.cgi?id=1064300

https://bugzilla.redhat.com/show_bug.cgi?id=1064300#c18

It seems to be an incompatibility with gnutls 3.1.20. It's not a silent ABI break though, as rebuilding doesn't help.

also see https://bugzilla.redhat.com/show_bug.cgi?id=1063089#c12

Hi, this is affecting me also.

I have tried the packages from EPEL and packages from repo.zabbix.com.

Trousers is up to date (as per one of the bugzilla links).

This host is brand new, fully patched Oracle Linux 7 host, and I am trying to install/run Zabbix 2.2.

Is there any further movement on this bug? Can I provide anything further than what has already been provided?

Thanks

uninstall gnutls or downgrade, its legacy any ways, you will live without it.

There are too many dependencies. Patching trousers seems to be the key.

After updating CentOS 7 to the second release 1503 we have the same issue:

Apr 03 13:47:12 zabbix.tgho.nl systemd[1]: Starting Zabbix Server...
– Subject: Unit zabbix-server.service has begun with start-up
– Defined-By: systemd
– Support: http://lists.freedesktop.org/mailman/listinfo/systemd-devel
–
– Unit zabbix-server.service has begun starting up.
Apr 03 13:47:12 zabbix kernel: zabbix_server[3476]: segfault at 18 ip 00007f7e0a419c00 sp 00007ffd32169c18 error 4 in libpthread-2.17.so[7f7e0a410000+16000]

The same is here: upgraded to Centos 7.1. Fix: downgrade package gnutls from 3.3.8-12.el7 to 3.1.18-10.el7_0.

Hi, same problem, CentOS Linux release 7.1.1503 (Core).

cat /var/log/messages | grep zabbix:
Apr 8 16:08:26 eye kernel: zabbix_server[2188]: segfault at 18 ip 00007ff010083c00 sp 00007fff96446708 error 4 in libpthread-2.17.so[7ff01007a000+16000]
Apr 8 16:08:26 eye kernel: zabbix_server[2189]: segfault at 18 ip 00007ff010083c00 sp 00007fff96446708 error 4 in libpthread-2.17.so[7ff01007a000+16000]
Apr 8 16:08:26 eye systemd: zabbix-server.service: control process exited, code=killed status=11
Apr 8 16:08:26 eye systemd: Unit zabbix-server.service entered failed state.

RHEL 7. Same as everyone else.
segfault at 18 ip 00007ff010083c00 sp 00007fff96446708 error 4 in libpthread-2.17.so[7ff01007a000+16000]

anton.samets's fix worked for me:
yum downgrade gnutls-3.1.18-10.el7_0 gnutls-devel-3.1.18-10.el7_0 gnutls-dane-3.1.18-10.el7_0 gnutls-c++-3.1.18-10.el7_0

As far as I can tell, it seems like there was a silent ABI breakage in gnutls, that is causing zabbix_server to crash through iksemel.

gnutls had 3 breakages between 3.1.18 and 3.3.8: http://upstream.rosalinux.ru/versions/gnutls.html

I would therefore think that a rebuild of iksemel should be enough. Please try these packages together with the latest version of gnutls supplied by EL: http://koji.fedoraproject.org/koji/taskinfo?taskID=9472122

Hello,

I have the very same problem on CentOS Linux release 7.1.1503 (Core)
The behaviour is the same with 2.2 and 2.4 releases:

[root@server]# zabbix_server
Segmentation fault

Also tried to compile zabbix-server from SRPM but it didn't help

I don't think it was caused by gnutls package as this package wasn't updated since I've installed the server.

Same problem here with RHEL 7.1 and Zabbix 2.4...
[root@SomeServer ~]# cat /etc/redhat-release
Red Hat Enterprise Linux Server release 7.1 (Maipo)
[root@SomeServer ~]# /usr/sbin/zabbix_server
Segmentation fault
[root@SomeServer ~]# rpm -qa | grep gnutls
gnutls-3.3.8-12.el7.x86_64

[root@SomeServer ~]# cat /var/log/zabbix/zabbix_server.log
...
2645:20150414:111518.803 using configuration file: /etc/zabbix/zabbix_server.conf
2645:20150414:111518.810 current database version (mandatory/optional): 02040000/02040000
2645:20150414:111518.810 required mandatory version: 02040000
2645:20150414:111518.820 listener failed: zbx_tcp_listen() fatal error: unable to serve on any address [[-]:10051]
2645:20150414:111518.821 Got signal [signal:11(SIGSEGV),reason:1,refaddr:0x18]. Crashing ...
...

According to some people it is related to the upgrade of gnutls from 3.1.8 to 3.3.8-12 ...

The same as others.
The solution is to downgrade gnutls to this one is included in Centos 7 install CD.
I mounted Install CD, add CD repo and:

1. yum downgrade gnutls

gnutls.x86_64 0:3.3.8-12.el7 - was removed
gnutls.x86_64 0:3.1.18-8.el7 - was installed
And all work.

I additionally tested it on a fresh Centos 7 + zabbix 2.4 (worked well) and after "yum update" zabbix_server crushed.

Best
Grzegorz

I investigated the issue quite a bit lately and it is definitely and only connected to the use of iksemel. If you build without-jabber, everything is fine. That also explains why the MySQL proxy has no issues, as it has no business with alerting. What is still a mystery to me, is, why the PostgreSQL server does fine. Also, rebuilding iksemel and even rebuilding Zabbix on top of that shows no effect whatsoever. It must be some really weird conflict with iksemel, gnutls and MySQL.

Confirming here; had tested a zabbix install on RHEL6, and was just updating my setup to 7 (not deployed yet) and found this exact issue. With zabbix-server-2.4.4-1.el7.x86_64, gnutls-3.3.8-12.el7.x86_64 and trousers-0.3.11.2-3.el7.x86_64 (which in the linked ZBX-9449 was mentioned as a possible culprit) the zabbix_server instance segfaults immediately.

Latest insights:

• It's not a packaging artefact. A manual and stripped down build suffers from the same problems.
• A rebuild of gnutls 3.1.18 with everything else left at 7.1 versions works
• A hackish build of gnutls 3.1.28 works as well
• A 3.3.8 rebuild works, if built --without-tpm (trousers)

https://bugzilla.redhat.com/show_bug.cgi?id=1071171

This issue should be gone in any supported version of Fedora. I am not sure whether the corrected EPEL 7 package of trousers is available to users yet, but anyway, there is nothing Zabbix SIA can do any further about this issue. I would thus suggest to close this issue.

as per the previous comment, closing