[ZBX-15573] zabbix server crashes on upgrade from 3.4.15 to 4.0.3 Created: 2019 Feb 01  Updated: 2021 Jan 19  Resolved: 2019 Feb 01

Status: Closed
Project: ZABBIX BUGS AND ISSUES
Component/s: Server (S)
Affects Version/s: None
Fix Version/s: None

Type: Incident report Priority: Major
Reporter: Tomasz Kłoczko Assignee: Unassigned
Resolution: Cannot Reproduce Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified
Environment:

x86_64 Fedora Linux with installed MySQL 5.7.25

Issue Links:
Duplicate
is duplicated by ZBXNEXT-4832 Zabbix Upgrade 3.4 to 4.0 Closed

 Description   

From zabbix-server.log

 

 

31973:20190201:171954.346 Starting Zabbix Server. Zabbix 4.0.3 (revision 87993).
 31973:20190201:171954.347 ****** Enabled features ******
 31973:20190201:171954.347 SNMP monitoring: YES
 31973:20190201:171954.347 IPMI monitoring: YES
 31973:20190201:171954.347 Web monitoring: YES
 31973:20190201:171954.347 VMware monitoring: YES
 31973:20190201:171954.347 SMTP authentication: YES
 31973:20190201:171954.347 Jabber notifications: NO
 31973:20190201:171954.347 Ez Texting notifications: YES
 31973:20190201:171954.347 ODBC: YES
 31973:20190201:171954.347 SSH2 support: NO
 31973:20190201:171954.347 IPv6 support: YES
 31973:20190201:171954.348 TLS support: YES
 31973:20190201:171954.348 ******************************
 31973:20190201:171954.348 using configuration file: /etc/zabbix/zabbix_server.conf
 31973:20190201:171954.489 current database version (mandatory/optional): 03040000/03040007
 31973:20190201:171954.490 required mandatory version: 04000000
 31973:20190201:171954.490 optional patches were found
 31973:20190201:171954.490 starting automatic database upgrade
 31973:20190201:171954.797 completed 0% of database upgrade
 31973:20190201:171954.804 completed 1% of database upgrade
 31973:20190201:171958.684 slow query: 3.879001 sec, "alter table `events` add `name` varchar(2048) default '' not null"
 31973:20190201:171958.982 completed 2% of database upgrade
 31973:20190201:171959.413 completed 3% of database upgrade
 31973:20190201:172000.114 completed 4% of database upgrade
 31973:20190201:172000.143 completed 5% of database upgrade
 31973:20190201:172000.350 completed 6% of database upgrade
 31973:20190201:172000.718 completed 7% of database upgrade
 31973:20190201:172001.092 completed 8% of database upgrade
 31973:20190201:172001.118 completed 9% of database upgrade
 31973:20190201:172001.304 completed 10% of database upgrade
 31973:20190201:172001.312 completed 11% of database upgrade
 31973:20190201:172001.326 completed 12% of database upgrade
 31973:20190201:172001.348 completed 13% of database upgrade
 31973:20190201:172001.358 completed 14% of database upgrade
 31973:20190201:172001.459 completed 15% of database upgrade
 31973:20190201:172001.710 completed 16% of database upgrade
 31973:20190201:172001.750 completed 17% of database upgrade
 31973:20190201:172001.847 completed 18% of database upgrade
 31973:20190201:172001.878 completed 19% of database upgrade
 31973:20190201:172001.892 completed 20% of database upgrade
 31973:20190201:172002.200 completed 21% of database upgrade
 31973:20190201:172002.351 completed 22% of database upgrade
 31973:20190201:172002.607 completed 23% of database upgrade
 31973:20190201:172002.801 completed 24% of database upgrade
 31973:20190201:172002.808 completed 25% of database upgrade
 31973:20190201:172004.198 completed 26% of database upgrade
zabbix_server [3764]: ERROR [file:../../../../src/zabbix_server/taskmanager/taskmanager.c,line:388] Something impossible has just happened.
 3764:20190201:172005.164 === Backtrace: ===
 3764:20190201:172005.164 7: /usr/sbin/zabbix_server: task manager [processing tasks](zbx_backtrace+0x47) [0x5623d6de42c7]
 3764:20190201:172005.164 6: /usr/sbin/zabbix_server: task manager [processing tasks](taskmanager_thread+0x26f) [0x5623d6e3720f]
 3764:20190201:172005.165 5: /usr/sbin/zabbix_server: task manager [processing tasks](zbx_thread_start+0x24) [0x5623d6de43d4]
 3764:20190201:172005.165 4: /usr/sbin/zabbix_server: task manager [processing tasks](MAIN_ZABBIX_ENTRY+0x5bb) [0x5623d6e6a4cb]
 3764:20190201:172005.165 3: /usr/sbin/zabbix_server: task manager [processing tasks](daemon_start+0x1ce) [0x5623d6de94be]
 3764:20190201:172005.165 2: /usr/sbin/zabbix_server: task manager [processing tasks](main+0x347) [0x5623d6d68ab7]
 3764:20190201:172005.165 1: /lib64/libc.so.6(__libc_start_main+0xf3) [0x7f5e58bebee3]
 3764:20190201:172005.165 0: /usr/sbin/zabbix_server: task manager [processing tasks](_start+0x2e) [0x5623d6d68e6e]
 31973:20190201:172007.029 completed 27% of database upgrade
 31973:20190201:172008.439 completed 28% of database upgrade
zabbix_server [3764]: ERROR [file:../../../../src/zabbix_server/taskmanager/taskmanager.c,line:388] Something impossible has just happened.
 3764:20190201:172010.168 === Backtrace: ===
 3764:20190201:172010.168 7: /usr/sbin/zabbix_server: task manager [processing tasks](zbx_backtrace+0x47) [0x5623d6de42c7]
 3764:20190201:172010.168 6: /usr/sbin/zabbix_server: task manager [processing tasks](taskmanager_thread+0x26f) [0x5623d6e3720f]
 3764:20190201:172010.169 5: /usr/sbin/zabbix_server: task manager [processing tasks](zbx_thread_start+0x24) [0x5623d6de43d4]
 3764:20190201:172010.169 4: /usr/sbin/zabbix_server: task manager [processing tasks](MAIN_ZABBIX_ENTRY+0x5bb) [0x5623d6e6a4cb]
 3764:20190201:172010.169 3: /usr/sbin/zabbix_server: task manager [processing tasks](daemon_start+0x1ce) [0x5623d6de94be]
 3764:20190201:172010.169 2: /usr/sbin/zabbix_server: task manager [processing tasks](main+0x347) [0x5623d6d68ab7]
 3764:20190201:172010.169 1: /lib64/libc.so.6(__libc_start_main+0xf3) [0x7f5e58bebee3]
 3764:20190201:172010.169 0: /usr/sbin/zabbix_server: task manager [processing tasks](_start+0x2e) [0x5623d6d68e6e]
 31973:20190201:172011.199 completed 29% of database upgrade
 31973:20190201:172012.684 completed 30% of database upgrade
zabbix_server [3764]: ERROR [file:../../../../src/zabbix_server/taskmanager/taskmanager.c,line:388] Something impossible has just happened.
 3764:20190201:172015.171 === Backtrace: ===
 3764:20190201:172015.172 7: /usr/sbin/zabbix_server: task manager [processing tasks](zbx_backtrace+0x47) [0x5623d6de42c7]
 3764:20190201:172015.172 6: /usr/sbin/zabbix_server: task manager [processing tasks](taskmanager_thread+0x26f) [0x5623d6e3720f]
 3764:20190201:172015.172 5: /usr/sbin/zabbix_server: task manager [processing tasks](zbx_thread_start+0x24) [0x5623d6de43d4]
 3764:20190201:172015.172 4: /usr/sbin/zabbix_server: task manager [processing tasks](MAIN_ZABBIX_ENTRY+0x5bb) [0x5623d6e6a4cb]
 3764:20190201:172015.172 3: /usr/sbin/zabbix_server: task manager [processing tasks](daemon_start+0x1ce) [0x5623d6de94be]
 3764:20190201:172015.172 2: /usr/sbin/zabbix_server: task manager [processing tasks](main+0x347) [0x5623d6d68ab7]
 3764:20190201:172015.172 1: /lib64/libc.so.6(__libc_start_main+0xf3) [0x7f5e58bebee3]
 3764:20190201:172015.172 0: /usr/sbin/zabbix_server: task manager [processing tasks](_start+0x2e) [0x5623d6d68e6e]
 31973:20190201:172015.576 completed 31% of database upgrade
 31973:20190201:172018.447 completed 32% of database upgrade
 31973:20190201:172019.941 completed 33% of database upgrade
zabbix_server [3764]: ERROR [file:../../../../src/zabbix_server/taskmanager/taskmanager.c,line:388] Something impossible has just happened.
 3764:20190201:172020.174 === Backtrace: ===
 3764:20190201:172020.175 7: /usr/sbin/zabbix_server: task manager [processing tasks](zbx_backtrace+0x47) [0x5623d6de42c7]
 3764:20190201:172020.175 6: /usr/sbin/zabbix_server: task manager [processing tasks](taskmanager_thread+0x26f) [0x5623d6e3720f]
 3764:20190201:172020.175 5: /usr/sbin/zabbix_server: task manager [processing tasks](zbx_thread_start+0x24) [0x5623d6de43d4]
 3764:20190201:172020.175 4: /usr/sbin/zabbix_server: task manager [processing tasks](MAIN_ZABBIX_ENTRY+0x5bb) [0x5623d6e6a4cb]
 3764:20190201:172020.175 3: /usr/sbin/zabbix_server: task manager [processing tasks](daemon_start+0x1ce) [0x5623d6de94be]
 3764:20190201:172020.175 2: /usr/sbin/zabbix_server: task manager [processing tasks](main+0x347) [0x5623d6d68ab7]
 3764:20190201:172020.175 1: /lib64/libc.so.6(__libc_start_main+0xf3) [0x7f5e58bebee3]
 3764:20190201:172020.175 0: /usr/sbin/zabbix_server: task manager [processing tasks](_start+0x2e) [0x5623d6d68e6e]
 31973:20190201:172022.800 completed 34% of database upgrade
zabbix_server [3764]: ERROR [file:../../../../src/zabbix_server/taskmanager/taskmanager.c,line:388] Something impossible has just happened.
 3764:20190201:172025.177 === Backtrace: ===
 3764:20190201:172025.178 7: /usr/sbin/zabbix_server: task manager [processing tasks](zbx_backtrace+0x47) [0x5623d6de42c7]
 3764:20190201:172025.178 6: /usr/sbin/zabbix_server: task manager [processing tasks](taskmanager_thread+0x26f) [0x5623d6e3720f]
 3764:20190201:172025.178 5: /usr/sbin/zabbix_server: task manager [processing tasks](zbx_thread_start+0x24) [0x5623d6de43d4]
 3764:20190201:172025.178 4: /usr/sbin/zabbix_server: task manager [processing tasks](MAIN_ZABBIX_ENTRY+0x5bb) [0x5623d6e6a4cb]
 3764:20190201:172025.178 3: /usr/sbin/zabbix_server: task manager [processing tasks](daemon_start+0x1ce) [0x5623d6de94be]
 3764:20190201:172025.178 2: /usr/sbin/zabbix_server: task manager [processing tasks](main+0x347) [0x5623d6d68ab7]
 3764:20190201:172025.178 1: /lib64/libc.so.6(__libc_start_main+0xf3) [0x7f5e58bebee3]
 3764:20190201:172025.178 0: /usr/sbin/zabbix_server: task manager [processing tasks](_start+0x2e) [0x5623d6d68e6e]
 31973:20190201:172025.676 completed 35% of database upgrade
 31973:20190201:172027.044 completed 36% of database upgrade
 31973:20190201:172028.896 completed 37% of database upgrade
 31973:20190201:172028.902 completed 38% of database upgrade
 31973:20190201:172029.157 completed 39% of database upgrade
 31973:20190201:172029.385 completed 40% of database upgrade
 31973:20190201:172029.390 completed 41% of database upgrade
 31973:20190201:172029.401 completed 42% of database upgrade
 31973:20190201:172029.412 completed 43% of database upgrade
 31973:20190201:172029.417 completed 44% of database upgrade
 31973:20190201:172029.429 completed 45% of database upgrade
 31973:20190201:172029.434 completed 46% of database upgrade
 31973:20190201:172029.446 completed 47% of database upgrade
 31973:20190201:172029.481 completed 48% of database upgrade
 31973:20190201:172029.712 completed 49% of database upgrade
 31973:20190201:172029.726 completed 50% of database upgrade
 31973:20190201:172029.741 completed 51% of database upgrade
 31973:20190201:172029.747 completed 52% of database upgrade
 31973:20190201:172029.757 completed 53% of database upgrade
 31973:20190201:172029.770 completed 54% of database upgrade
 31973:20190201:172029.775 completed 55% of database upgrade
 31973:20190201:172029.787 completed 56% of database upgrade
 31973:20190201:172029.793 completed 57% of database upgrade
 31973:20190201:172029.912 completed 58% of database upgrade
 31973:20190201:172029.924 completed 59% of database upgrade
 31973:20190201:172029.930 completed 60% of database upgrade
 31973:20190201:172030.017 completed 61% of database upgrade
 31973:20190201:172030.134 completed 62% of database upgrade
 31973:20190201:172030.140 completed 63% of database upgrade
 31973:20190201:172030.167 completed 64% of database upgrade
zabbix_server [3764]: ERROR [file:../../../../src/zabbix_server/taskmanager/taskmanager.c,line:388] Something impossible has just happened.
 3764:20190201:172030.180 === Backtrace: ===
 3764:20190201:172030.180 7: /usr/sbin/zabbix_server: task manager [processing tasks](zbx_backtrace+0x47) [0x5623d6de42c7]
 3764:20190201:172030.181 6: /usr/sbin/zabbix_server: task manager [processing tasks](taskmanager_thread+0x26f) [0x5623d6e3720f]
 3764:20190201:172030.181 5: /usr/sbin/zabbix_server: task manager [processing tasks](zbx_thread_start+0x24) [0x5623d6de43d4]
 3764:20190201:172030.181 4: /usr/sbin/zabbix_server: task manager [processing tasks](MAIN_ZABBIX_ENTRY+0x5bb) [0x5623d6e6a4cb]
 3764:20190201:172030.181 3: /usr/sbin/zabbix_server: task manager [processing tasks](daemon_start+0x1ce) [0x5623d6de94be]
 3764:20190201:172030.181 2: /usr/sbin/zabbix_server: task manager [processing tasks](main+0x347) [0x5623d6d68ab7]
 3764:20190201:172030.181 1: /lib64/libc.so.6(__libc_start_main+0xf3) [0x7f5e58bebee3]
 3764:20190201:172030.181 0: /usr/sbin/zabbix_server: task manager [processing tasks](_start+0x2e) [0x5623d6d68e6e]
 31973:20190201:172030.201 completed 65% of database upgrade
 

 

 Comments   
Comment by Oleksii Zagorskyi [ 2019 Feb 01 ]

Old and new binaries are running at the same time and logging to the same log?

Comment by Tomasz Kłoczko [ 2019 Feb 01 ]

Yes. that was the case. Looks like I have some issues with my own systemd services descriptions which are causing that stopping zabbux-{server,proxy}{ services not always causes killing all processes.

I had yet another small issue which I think it would be good to handle during upgrade.

I'm trying to keep zabbix server. web frontend and proxy DB users with minimal grated privileges.

It means that usually all those users have not granted privs to create, drop and alter table. During upgrade I'm only temporary granting privs to those operations. Issue is when I'm forgeting to temporary raise privs.

I think that before start upgrade it would be good to add checking does DB user used by proxy/server has table CREAT, DROP, ALTER privs on given DB and do not start upgrade if those privs are missing.

 

 

Comment by Oleksii Zagorskyi [ 2019 Feb 01 ]

Ok, as it's not a but report, I'm closing it.
All ideas and suggestions should be posted in ZBXNEXT project.

Generated at Thu Apr 18 23:16:35 EEST 2024 using Jira 9.12.4#9120004-sha1:625303b708afdb767e17cb2838290c41888e9ff0.