[ZBX-13441] Zabbix Server stopped Created: 2018 Feb 06  Updated: 2024 Apr 10  Resolved: 2018 Feb 10

Status: Closed
Project: ZABBIX BUGS AND ISSUES
Component/s: Server (S)
Affects Version/s: 3.0.14, 3.4.5
Fix Version/s: 3.0.15rc1, 3.4.7rc2, 4.0.0alpha4, 4.0 (plan)

Type: Problem report Priority: Critical
Reporter: Dmitry Fisher Assignee: Glebs Ivanovskis (Inactive)
Resolution: Fixed Votes: 0
Labels: crash, vmware
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Attachments: PNG File screenshot-1.png     Text File zabbix_server.log     Text File zabbix_server.log     File zabbix_server.log2.old     Text File zabbix_server0802.log     Text File zabbix_server1.log    
Issue Links:
Causes
caused by ZBX-12497 vmware.eventlog only get latest 10 lo... Closed
Duplicate
is duplicated by ZBX-13568 Zabbix Proxy crashed while monitoring... Closed
is duplicated by ZBX-13477 Proxy Crash VMware Collector Closed
Team: Team C
Sprint: Sprint 27
Story Points: 0.5

 Description   

Hi! I have problem with stopping zabbix server every 5-6 hours operation.
There text from server log files:

  1295:20180206:155238.984 Got signal [signal:11(SIGSEGV),reason:1,refaddr:0xffffffb0]. Crashing ...
  1295:20180206:155238.984 ====== Fatal information: ======
  1295:20180206:155238.984 program counter not available for this architecture
  1295:20180206:155238.984 === Registers: ===
  1295:20180206:155238.984 register dump not available for this architecture
  1295:20180206:155238.984 === Backtrace: ===
  1295:20180206:155238.985 3: 0x4a422b <zbx_log_fatal_info+0x9b> at /usr/local/sbin/zabbix_server
  1295:20180206:155238.985 2: 0x4a464b <zbx_set_common_signal_handlers+0x29b> at /usr/local/sbin/zabbix_server
  1295:20180206:155238.985 1: 0x80440ab37 <pthread_sigmask+0x507> at /lib/libthr.so.3
  1295:20180206:155238.985 0: 0x80440a22c <pthread_getspecific+0xe1c> at /lib/libthr.so.3
  1295:20180206:155238.985 === Memory map: ===
  1287:20180206:155238.985 item "500280f6-d8d8-cf3c-e4fa-c37815b0f90b:vmware.vm.net.if.in[{$URL},{HOST.HOST},4000,pps]" became supported
  1295:20180206:155238.985 memory map not available for this platform
  1295:20180206:155238.985 ================================
  1277:20180206:155238.987 One child process died (PID:1295,exitcode/signal:1). Exiting ...
  1319:20180206:155239.003 cannot read preprocessing service request
  1277:20180206:155241.016 syncing history data...
  1277:20180206:155241.024 syncing history data done
  1277:20180206:155241.024 syncing trend data...
  1277:20180206:155241.516 syncing trend data done
  1277:20180206:155241.516 cannot remove shared memory for self-monitoring collector: [22] Invalid argument
  1277:20180206:155241.516 Zabbix Server stopped. Zabbix 3.4.6 (revision 76823).

and

 19769:20180203:232516.217 Got signal [signal:10(SIGBUS),reason:3,refaddr:0x803a1862d]. Crashing ...
 19769:20180203:232516.217 ====== Fatal information: ======
 19769:20180203:232516.217 program counter not available for this architecture
 19769:20180203:232516.217 === Registers: ===
 19769:20180203:232516.217 register dump not available for this architecture
 19769:20180203:232516.217 === Backtrace: ===
 19769:20180203:232516.218 3: 0x4a422b <zbx_log_fatal_info+0x9b> at /usr/local/sbin/zabbix_server
 19769:20180203:232516.218 2: 0x4a464b <zbx_set_common_signal_handlers+0x29b> at /usr/local/sbin/zabbix_server
 19769:20180203:232516.218 1: 0x804409b37 <pthread_sigmask+0x507> at /lib/libthr.so.3
 19769:20180203:232516.218 0: 0x80440922c <pthread_getspecific+0xe1c> at /lib/libthr.so.3
 19769:20180203:232516.218 === Memory map: ===
 19769:20180203:232516.218 memory map not available for this platform
 19769:20180203:232516.218 ================================
 19749:20180203:232516.221 One child process died (PID:19769,exitcode/signal:1). Exiting ...
 19749:20180203:232518.273 syncing history data...
 19749:20180203:232518.276 syncing history data done
 19749:20180203:232518.276 syncing trend data...
 19749:20180203:232518.422 syncing trend data done
 19749:20180203:232518.422 cannot remove shared memory for self-monitoring collector: [22] Invalid argument
 19749:20180203:232518.422 Zabbix Server stopped. Zabbix 3.4.6 (revision 76823).


 Comments   
Comment by Dmitry Fisher [ 2018 Feb 06 ]

PS. i use Freebsd 10.3-RELEASE-p26

Comment by Glebs Ivanovskis (Inactive) [ 2018 Feb 06 ]

Hm, crashes in vmware collector and does so quite frequently. Can you please increase DebugLevel? (Preferably for vmware collectors, if runtime control options are supported on your platform.)

Comment by Dmitry Fisher [ 2018 Feb 06 ]

Which Debug level need using? Now i use 3 level (default)

Comment by Glebs Ivanovskis (Inactive) [ 2018 Feb 07 ]

I think 4 will be enough for a start.

Comment by Dmitry Fisher [ 2018 Feb 07 ]

I am attaching a new log file with Debug 4

Comment by Glebs Ivanovskis (Inactive) [ 2018 Feb 07 ]

Unfortunately, there are no messages from crashing process other than:

 69754:20180207:141701.342 Got signal [signal:10(SIGBUS),reason:3,refaddr:0x803a1862d]. Crashing ...
 69754:20180207:141701.342 ====== Fatal information: ======
 69754:20180207:141701.342 program counter not available for this architecture
 69754:20180207:141701.342 === Registers: ===
 69754:20180207:141701.342 register dump not available for this architecture
 69754:20180207:141701.342 === Backtrace: ===
 69754:20180207:141701.343 3: 0x4a422b <zbx_log_fatal_info+0x9b> at /usr/local/sbin/zabbix_server
 69754:20180207:141701.343 2: 0x4a464b <zbx_set_common_signal_handlers+0x29b> at /usr/local/sbin/zabbix_server
 69754:20180207:141701.343 1: 0x804409b37 <pthread_sigmask+0x507> at /lib/libthr.so.3
 69754:20180207:141701.343 0: 0x80440922c <pthread_getspecific+0xe1c> at /lib/libthr.so.3
 69754:20180207:141701.344 === Memory map: ===
 69754:20180207:141701.344 memory map not available for this platform
 69754:20180207:141701.344 ================================

Looks like log file rotated just before the crash. Do you have older logs?

Comment by Dmitry Fisher [ 2018 Feb 07 ]

Ok i was attached log from 01/02

Comment by Glebs Ivanovskis (Inactive) [ 2018 Feb 07 ]

Oh, just realized that the whole log captures just a couple of seconds

Could you start Zabbix with DebugLevel=3 and then execute the following command:

$ zabbix_server -c /path/to/zabbix_server.conf -R log_level_increase="vmware collector"

This will reduce the amount of information being written into the log file.

Comment by Dmitry Fisher [ 2018 Feb 07 ]

Ok, i was change Debug level on 3.
zabbix_server -c /path/to/zabbix_server.conf -R log_level_increase="vmware collector" ok

zabbix_server -c /path/to/zabbix_server.conf -R log_level_increase="vmware poller"
have error
zabbix_server [72858]: invalid log level control target: unknown process type "vmware poller"

Comment by Glebs Ivanovskis (Inactive) [ 2018 Feb 07 ]

Oops, my bad, there is no such process Just ignore it, vmware collector is enough

Comment by Dmitry Fisher [ 2018 Feb 08 ]

Hi! I am attached new log file, from 08.02.

Comment by Glebs Ivanovskis (Inactive) [ 2018 Feb 08 ]

Thank you! This looks interesting, I think I know the reason. Is your VMwareTimeout set to 30 seconds?

Comment by Dmitry Fisher [ 2018 Feb 08 ]

Yes, 30 seconds.
PS. Version zabbix server 3.4.6) attached screeshot)

Comment by Glebs Ivanovskis (Inactive) [ 2018 Feb 08 ]

Yes, I know that your version is 3.4.6, I've put 3.4.5 in Affected versions for people who may be experiencing same problem.

Do you want a patch?

Index: src/zabbix_server/vmware/vmware.c
===================================================================
--- src/zabbix_server/vmware/vmware.c	(revision 77496)
+++ src/zabbix_server/vmware/vmware.c	(working copy)
@@ -1502,6 +1502,7 @@
 	iter = zbx_malloc(iter, sizeof(zbx_property_collection_iter));
 	iter->property_collector = property_collector;
 	iter->easyhandle = easyhandle;
+	iter->error = NULL;
 	iter->token = NULL;
 
 	if (CURLE_OK == (err = curl_easy_setopt(iter->easyhandle, opt = CURLOPT_POSTFIELDS, property_collection_query)))
Comment by Dmitry Fisher [ 2018 Feb 08 ]

What do you mean? Patch for what? For fix my problem?

Comment by Glebs Ivanovskis (Inactive) [ 2018 Feb 08 ]

Yes.

Comment by Dmitry Fisher [ 2018 Feb 08 ]

Tell me how can i do this?

Comment by Glebs Ivanovskis (Inactive) [ 2018 Feb 08 ]

Have you compiled Zabbix from source? If yes, it should be easy. You need to get sources for 3.4.6, e.g.:

$ svn co svn://svn.zabbix.com/tags/3.4.6

Then you apply the patch (provided that you saved it as patch.diff):

$ cd 3.4.6
$ svn patch ../patch.diff

Then you configure and compile Zabbix.

If you have not compiled Zabbix yourself, then I will need some more time to prepare instructions, because I have never compiled server on FreeBSD too.

Comment by Dmitry Fisher [ 2018 Feb 08 ]

I try to make file with name "patch.diff" and included all your code, but after running, i have some error:

/usr/local/etc/zabbix34 # /usr/local/etc/zabbix34/patch.diff
/usr/local/etc/zabbix34/patch.diff: Index:: not found
/usr/local/etc/zabbix34/patch.diff: ===================================================================: not found
/usr/local/etc/zabbix34/patch.diff: 3: Syntax error: "(" unexpected

And what about your second option with download source and compile it, i dont understanding...
I can download zabbix server tar.gz, but what to do on the next step, dont know..
PS. maybe i can include this code in the some file on running system?

Comment by Glebs Ivanovskis (Inactive) [ 2018 Feb 08 ]

Seems this is the way to do it without SVN:

$ wget https://sourceforge.net/projects/zabbix/files/ZABBIX%20Latest%20Stable/3.4.6/zabbix-3.4.6.tar.gz
$ tar -zxvf zabbix-3.4.6.tar.gz
$ cd zabbix-3.4.6.tar.gz
$ patch -u -p 0 -i ../patch.diff
Comment by Dmitry Fisher [ 2018 Feb 08 ]

I did it!

/home/f1sher/zabbix/zabbix-3.4.6 # patch -u -p 0 -i /usr/local/etc/zabbix34/patch.diff
Hmm...  Looks like a unified diff to me...
The text leading up to this was:
--------------------------
|Index: src/zabbix_server/vmware/vmware.c
|===================================================================
|--- src/zabbix_server/vmware/vmware.c  (revision 77496)
|+++ src/zabbix_server/vmware/vmware.c  (working copy)
--------------------------
Patching file src/zabbix_server/vmware/vmware.c using Plan A...
Hunk #1 succeeded at 1510 (offset 8 lines).
done

What to do then? Use comand in this directory "Make install"? Or all ok and need for wait?

Comment by Glebs Ivanovskis (Inactive) [ 2018 Feb 08 ]

Nice, now you need to compile sources. Follow this guide. Basically:

$ ./configure ...
$ make
$ make install

Then restart Zabbix.

Comment by Dmitry Fisher [ 2018 Feb 09 ]

Option with "configure and make install"didnt solved problem with stopping server.(
Im was attach the last log file.

Comment by Glebs Ivanovskis (Inactive) [ 2018 Feb 09 ]

Please make sure that you are starting the patched binary:

$ diff /path/to/install/dir/zabbix_server /path/to/build/dir/src/zabbix_server/zabbix_server
Comment by Dmitry Fisher [ 2018 Feb 09 ]

Ok! I have this answer:

# diff /usr/local/etc/zabbix34/ /zabbix/zabbix-3.4.6
Only in /zabbix/zabbix-3.4.6: AUTHORS
Only in /zabbix/zabbix-3.4.6: COPYING
Only in /zabbix/zabbix-3.4.6: ChangeLog
Only in /zabbix/zabbix-3.4.6: INSTALL
Only in /zabbix/zabbix-3.4.6: Makefile
Only in /zabbix/zabbix-3.4.6: Makefile.am
Only in /zabbix/zabbix-3.4.6: Makefile.in
Only in /zabbix/zabbix-3.4.6: NEWS
Only in /zabbix/zabbix-3.4.6: README
Only in /zabbix/zabbix-3.4.6: aclocal.m4
Only in /zabbix/zabbix-3.4.6: bin
Only in /zabbix/zabbix-3.4.6: build
Only in /zabbix/zabbix-3.4.6: compile
Only in /zabbix/zabbix-3.4.6: conf
Only in /zabbix/zabbix-3.4.6: config.guess
Only in /zabbix/zabbix-3.4.6: config.log
Only in /zabbix/zabbix-3.4.6: config.status
Only in /zabbix/zabbix-3.4.6: config.sub
Only in /zabbix/zabbix-3.4.6: configure
Only in /zabbix/zabbix-3.4.6: configure.ac
Only in /zabbix/zabbix-3.4.6: database
Only in /zabbix/zabbix-3.4.6: depcomp
Only in /zabbix/zabbix-3.4.6: frontends
Only in /zabbix/zabbix-3.4.6: include
Only in /zabbix/zabbix-3.4.6: install-sh
Only in /zabbix/zabbix-3.4.6: m4
Only in /zabbix/zabbix-3.4.6: man
Only in /zabbix/zabbix-3.4.6: misc
Only in /zabbix/zabbix-3.4.6: missing
Only in /zabbix/zabbix-3.4.6: sass
Only in /zabbix/zabbix-3.4.6: src
Only in /zabbix/zabbix-3.4.6: upgrades
Only in /usr/local/etc/zabbix34/: zabbix
Only in /usr/local/etc/zabbix34/: zabbix_agentd.conf
Only in /usr/local/etc/zabbix34/: zabbix_agentd.conf.d
Only in /usr/local/etc/zabbix34/: zabbix_agentd.conf.sample
Only in /usr/local/etc/zabbix34/: zabbix_server.conf
Only in /usr/local/etc/zabbix34/: zabbix_server.conf.d
Only in /usr/local/etc/zabbix34/: zabbix_server.conf.sample
Only in /usr/local/etc/zabbix34/: zabbix_server_old.conf
Only in /usr/local/etc/zabbix34/: zabbix_server_rmk.conf
Comment by Glebs Ivanovskis (Inactive) [ 2018 Feb 09 ]

I meant comparing binaries:

  • /zabbix/zabbix-3.4.6/src/zabbix_server/zabbix_server - this is the one you compiled with patch (hopefully)
  • the one which is installed and executed when you type zabbix_server, you can find it using 
    $ whereis zabbix_server
Comment by Dmitry Fisher [ 2018 Feb 09 ]

Ok, understand!
I dont have any informathion, when used this comand:

# diff /zabbix/zabbix-3.4.6/src/zabbix_server /usr/local/sbin/zabbix_server
#
Comment by Glebs Ivanovskis (Inactive) [ 2018 Feb 09 ]

So they are identical. Please check

zabbix_server -V
Comment by Dmitry Fisher [ 2018 Feb 09 ]

I was recheck comand and which outputs

 # whereis zabbix_server
zabbix_server: /usr/local/sbin/zabbix_server /usr/local/man/man8/zabbix_server.8.gz
# diff /usr/local/sbin/zabbix_server /zabbix/zabbix-3.4.6/src/zabbix_server
diff: /zabbix/zabbix-3.4.6/src/zabbix_server/zabbix_server: No such file or directory
  1. zabbix_server -V 
    zabbix_server (Zabbix) 3.4.6
Revision 76823 15 January 2018, compilation time: Feb  9 2018 15:25:13

Copyright (C) 2018 Zabbix SIA
License GPLv2+: GNU GPL version 2 or later <http://gnu.org/licenses/gpl.html>.
This is free software: you are free to change and redistribute it according to
the license. There is NO WARRANTY, to the extent permitted by law.
Comment by Glebs Ivanovskis (Inactive) [ 2018 Feb 09 ]

"compilation time: Feb 9 2018 15:25:13"??? It's February 9 15:13 now (in Riga). Are you sure you compiled and installed patched Zabbix server yesterday? Can the log file you attached today be from old binary?

Comment by Dmitry Fisher [ 2018 Feb 09 ]

do not pay attention, it seemed to me that the patch worked incorrectly and I did everything again.
because my server installed on the ESXI and i can using snapshots

Comment by Glebs Ivanovskis (Inactive) [ 2018 Feb 09 ]

Fixed in:

  • pre-3.0.15rc1 r77558;
  • pre-3.4.7rc2 r77559;
  • pre-4.0.0alpha4 (trunk) r77560.
Comment by Dmitry Fisher [ 2018 Feb 10 ]

Thank you for help with patch!
24 hours server is up.
I think problem is resolved on this moment.

Comment by Glebs Ivanovskis (Inactive) [ 2018 Feb 10 ]

No, thank you! For reporting and for your input into investigation.

Generated at Fri Apr 26 08:11:17 EEST 2024 using Jira 9.12.4#9120004-sha1:625303b708afdb767e17cb2838290c41888e9ff0.