[ZBX-15171] Zabbix Server Crash Created: 2018 Nov 15 Updated: 2019 Jan 15 Resolved: 2018 Nov 20 |
|
Status: | Closed |
Project: | ZABBIX BUGS AND ISSUES |
Component/s: | Proxy (P), Server (S) |
Affects Version/s: | 4.0.1, 4.2.0alpha1, 4.2 (plan) |
Fix Version/s: | 4.0.2rc1, 4.2.0alpha1, 4.2 (plan) |
Type: | Problem report | Priority: | Critical |
Reporter: | Leonardo Maza | Assignee: | Alex Kalimulin |
Resolution: | Fixed | Votes: | 0 |
Labels: | crash, discovery, odbc, zabbix_server | ||
Remaining Estimate: | Not Specified | ||
Time Spent: | Not Specified | ||
Original Estimate: | Not Specified | ||
Environment: |
Ubuntu 16.04 and Ubuntu 18.04, MySQL Server |
Attachments: | zabbix_server.log | ||||||||||||||||
Issue Links: |
|
||||||||||||||||
Team: | Team A | ||||||||||||||||
Sprint: | Sprint 46, Nov 2018 | ||||||||||||||||
Story Points: | 0.125 |
Description |
Steps to reproduce:
Result:
More data: Some times them when Zabbix Server crash, It restart automatically, but some times cannot restart. In this cases, when I try stop zabbix server, It can not stop. When I see the proceses running with name zabbix (ps xaf | grep zabbix), Zabbix pinger is running. Then I kill it (kill -i PID) and I can start zabbix normally.
|
Comments |
Comment by Vladislavs Sokurenko [ 2018 Nov 15 ] |
Can you please provide output of: Does increasing CacheSize help with a crash ? |
Comment by Vladislavs Sokurenko [ 2018 Nov 15 ] |
I think that this issue with hanging process is solved in |
Comment by Vladislavs Sokurenko [ 2018 Nov 15 ] |
Backtrace 11992:20181114:153744.306 === Backtrace: === 11992:20181114:153744.307 17: /usr/sbin/zabbix_server: poller #9 [got 4 values in 0.167859 sec, getting values](zbx_backtrace+0x44) [0x5570b7651ce4] 11992:20181114:153744.307 16: /usr/sbin/zabbix_server: poller #9 [got 4 values in 0.167859 sec, getting values](zbx_log_fatal_info+0x153) [0x5570b7651f8f] 11992:20181114:153744.307 15: /usr/sbin/zabbix_server: poller #9 [got 4 values in 0.167859 sec, getting values](+0xff300) [0x5570b7652300] 11992:20181114:153744.307 14: /lib/x86_64-linux-gnu/libpthread.so.0(+0x12890) [0x7f221844e890] 11992:20181114:153744.307 13: /lib/x86_64-linux-gnu/libc.so.6(+0xb1646) [0x7f221504b646] 11992:20181114:153744.307 12: /lib/x86_64-linux-gnu/libc.so.6(__strdup+0xe) [0x7f22150379ae] 11992:20181114:153744.307 11: /usr/sbin/zabbix_server: poller #9 [got 4 values in 0.167859 sec, getting values](zbx_strdup2+0x4f) [0x5570b76680b7] 11992:20181114:153744.307 10: /usr/sbin/zabbix_server: poller #9 [got 4 values in 0.167859 sec, getting values](zbx_odbc_query_result_to_lld_json+0x3e1) [0x5570b75ddb67] 11992:20181114:153744.308 9: /usr/sbin/zabbix_server: poller #9 [got 4 values in 0.167859 sec, getting values](get_value_db+0x28e) [0x5570b75af549] 11992:20181114:153744.308 8: /usr/sbin/zabbix_server: poller #9 [got 4 values in 0.167859 sec, getting values](+0x4bade) [0x5570b759eade] 11992:20181114:153744.308 7: /usr/sbin/zabbix_server: poller #9 [got 4 values in 0.167859 sec, getting values](+0x4e028) [0x5570b75a1028] 11992:20181114:153744.308 6: /usr/sbin/zabbix_server: poller #9 [got 4 values in 0.167859 sec, getting values](poller_thread+0x1a1) [0x5570b75a224f] 11992:20181114:153744.308 5: /usr/sbin/zabbix_server: poller #9 [got 4 values in 0.167859 sec, getting values](zbx_thread_start+0x32) [0x5570b765faf2] 11992:20181114:153744.308 4: /usr/sbin/zabbix_server: poller #9 [got 4 values in 0.167859 sec, getting values](MAIN_ZABBIX_ENTRY+0x9c4) [0x5570b758b486] 11992:20181114:153744.308 3: /usr/sbin/zabbix_server: poller #9 [got 4 values in 0.167859 sec, getting values](daemon_start+0x315) [0x5570b765142d] 11992:20181114:153744.308 2: /usr/sbin/zabbix_server: poller #9 [got 4 values in 0.167859 sec, getting values](main+0x305) [0x5570b758aaac] 11992:20181114:153744.308 1: /lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0xe7) [0x7f2214fbbb97] 11992:20181114:153744.308 0: /usr/sbin/zabbix_server: poller #9 [got 4 values in 0.167859 sec, getting values](_start+0x2a) [0x5570b7589c7a] Possible code fragment that is missing NULL check: while (NULL != (row = zbx_odbc_fetch(query_result))) { zbx_json_addobject(&json, NULL); for (i = 0; i < query_result->col_num; i++) { char *value = NULL; value = zbx_strdup(value, row[i]); zbx_replace_invalid_utf8(value); zbx_json_addstring(&json, macros.values[i], value, ZBX_JSON_TYPE_STRING); zbx_free(value); } zbx_json_close(&json); } Patch: Index: src/zabbix_server/odbc/odbc.c =================================================================== --- src/zabbix_server/odbc/odbc.c (revision 86893) +++ src/zabbix_server/odbc/odbc.c (working copy) @@ -611,6 +611,9 @@ { char *value = NULL; + if (NULL == row[i]) + continue; + value = zbx_strdup(value, row[i]); zbx_replace_invalid_utf8(value); zbx_json_addstring(&json, macros.values[i], value, ZBX_JSON_TYPE_STRING); |
Comment by Leonardo Maza [ 2018 Nov 15 ] |
Hi, thanks for quickly response I increase now the CacheSize from 256M to 512M
This is the ps xaf output The last lines on zabbix_logs when crash ocurre. The output 54136:20181115:043744.698 One child process died (PID:54199,exitcode/signal:1). Exiting ... ^C First I try stop zabbix-server crash, the command never finished and I cancelled it Output ps xaf | grep zabbix. Alway I found icmp pinger running and I only can kill it with kill -9 command
Then I can start zabbix server again
|
Comment by Vladislavs Sokurenko [ 2018 Nov 15 ] |
Yes, this process leftover is fixed under |
Comment by Leonardo Maza [ 2018 Nov 15 ] |
Excellent for ZBX 15027! Increase the CacheSize not hel with crash For now, I patch this issue with the next script, on crontab, until new release, Thanks
- Crontab */2 * * * * /home/seacadm/restart_zabbix_crash.sh > /home/seacadm/logRestartZabbix.log 2>&1
- Script #!/bin/bash MAXAGE=120 #seconds mensaje='Ultima escritura en el archivo de log hace '$fileAge' segundos' test $fileAge -gt $MAXAGE && { |
Comment by Glebs Ivanovskis [ 2018 Nov 15 ] |
Good job, vso! Caused by |
Comment by Vladislavs Sokurenko [ 2018 Nov 15 ] |
Thanks cyclone, updated caused by |
Comment by Glebs Ivanovskis [ 2018 Nov 16 ] |
Please also add Proxy into Component/s. vso done cyclone Thanks! |
Comment by Vladislavs Sokurenko [ 2018 Nov 20 ] |
In 3.0 empty string is returned for NULL values. Cannot create item: item with the same key "trap1[""]" already exists. Cannot create item: item with the same key "trap1[""]" already exists. Cannot create item: item with the same key "trap1[""]" already exists. Cannot create item: item with the same key "trap1[""]" already exists. Cannot create item: item with the same key "trap1[""]" already exists. Cannot create item: item with the same key "trap1[""]" already exists. Cannot create item: item with the same key "trap1[""]" already exists. Cannot create item: item with the same key "trap1[""]" already exists. Cannot create item: item with the same key "trap1[""]" already exists. Cannot create item: item with the same key "trap1[""]" already exists. Cannot create item: item with the same key "trap1[""]" already exists. Cannot create item: item with the same key "trap1[""]" already exists. Cannot create item: item with the same key "trap1[""]" already exists. Cannot create item: item with the same key "trap1[""]" already exists. Cannot create item: item with the same key "trap1[""]" already exists. Cannot create item: item with the same key "trap1[""]" already exists. Cannot create item: item with the same key "trap1[""]" already exists. Cannot create item: item with the same key "trap1[""]" already exists. Cannot create item: item with the same key "trap1[""]" already exists. Cannot create item: item with the same key "trap1[""]" already exists. Cannot create item: item with the same key "trap1[""]" already exists. Cannot create item: item with the same key "trap1[""]" already exists. Cannot create item: item with the same key "trap1[""]" already exists. Cannot create item: item with the same key "trap1[""]" already exists. Cannot create item: item with the same key "trap1[""]" already exists. Cannot create item: item with the same key "trap1[""]" already exists. Cannot create item: item with the same key "trap1[""]" already exists. Cannot create item: item with the same key "trap1[""]" already exists. Cannot create item: item with the same key "trap1[""]" alrea Log:
31836:20181120:143337.720 json '{"data":[{"{#PROXY_HOSTID}":null},{"{#PROXY_HOSTID}":null},{"{#PROXY_HOSTID}":null},{"{#PROXY_HOSTID}":null},{"{#PROXY_HOSTID}":null},{"{#PROXY_HOSTID}":null},{"{#PROXY_HOSTID}":null},{"{#PROXY_HOSTID}":null},{"{#PROXY_HOSTID}":null},{"{#PROXY_HOSTID}":null},{"{#PROXY_HOSTID}":null},{"{#PROXY_HOSTID}":null},{"{#PROXY_HOSTID}":null},{"{#PROXY_HOSTID}":null},{"{#PROXY_HOSTID}":null},{"{#PROXY_HOSTID}":null},{"{#PROXY_HOSTID}":null},{"{#PROXY_HOSTID}":null},{"{#PROXY_HOSTID}":null},{"{#PROXY_HOSTID}":null},{"{#PROXY_HOSTID}":null},{"{#PROXY_HOSTID}":null},{"{#PROXY_HOSTID}":null},{"{#PROXY_HOSTID}":null},{"{#PROXY_HOSTID}":null},{"{#PROXY_HOSTID}":null},{"{#PROXY_HOSTID}":null},{"{#PROXY_HOSTID}":null},{"{#PROXY_HOSTID}":null},{"{#PROXY_HOSTID}":null},{"{#PROXY_HOSTID}":null},{"{#PROXY_HOSTID}":null},{"{#PROXY_HOSTID}":null},{"{#PROXY_HOSTID}":null},{"{#PROXY_HOSTID}":null},{"{#PROXY_HOSTID}":null},{"{#PROXY_HOSTID}":null},{"{#PROXY_HOSTID}":null},{"{#PROXY_HOSTID}":null},{"{#PROXY_HOSTID}":null},{"{#PROXY_HOSTID}":null},{"{#PROXY_HOSTID}":null},{"{#PROXY_HOSTID}":null},{"{#PROXY_HOSTID}":null},{"{#PROXY_HOSTID}":null},{"{#PROXY_HOSTID}":null},{"{#PROXY_HOSTID}":null},{"{#PROXY_HOSTID}":null},{"{#PROXY_HOSTID}":null},{"{#PROXY_HOSTID}":null},{"{#PROXY_HOSTID}":null},{"{#PROXY_HOSTID}":null},{"{#PROXY_HOSTID}":null},{"{#PROXY_HOSTID}":null},{"{#PROXY_HOSTID}":null},{"{#PROXY_HOSTID}":null},{"{#PROXY_HOSTID}":null},{"{#PROXY_HOSTID}":null},{"{#PROXY_HOSTID}":null},{"{#PROXY_HOSTID}":null},{"{#PROXY_HOSTID}":null},{"{#PROXY_HOSTID}":null},{"{#PROXY_HOSTID}":null},{"{#PROXY_HOSTID}":null},{"{#PROXY_HOSTID}":null},{"{#PROXY_HOSTID}":null},{"{#PROXY_HOSTID}":null},{"{#PROXY_HOSTID}":null},{"{#PROXY_HOSTID}":null},{"{#PROXY_HOSTID}":null},{"{#PROXY_HOSTID}":null},{"{#PROXY_HOSTID}":null},{"{#PROXY_HOSTID}":null},{"{#PROXY_HOSTID}":null},{"{#PROXY_HOSTID}":null},{"{#PROXY_HOSTID}":null},{"{#PROXY_HOSTID}":null},{"{#PROXY_HOSTID}":null},{"{#PROXY_HOSTID}":null},{"{#PROXY_HOSTID}":null},{"{#PROXY_HOSTID}":null},{"{#PROXY_HOSTID}":null},{"{#PROXY_HOSTID}":null},{"{#PROXY_HOSTID}":null},{"{#PROXY_HOSTID}":"10267"}]}'
|
Comment by Alex Kalimulin [ 2018 Nov 20 ] |
Fixed in:
|