[ZBX-8744] errors after upgrade from 2.2 to 2.4 Created: 2014 Sep 11  Updated: 2017 May 30  Resolved: 2014 Nov 06

Status: Closed
Project: ZABBIX BUGS AND ISSUES
Component/s: Proxy (P), Server (S)
Affects Version/s: 2.4.0
Fix Version/s: 2.4.1rc1, 2.5.0

Type: Incident report Priority: Blocker
Reporter: AndreaConsadori Assignee: Unassigned
Resolution: Fixed Votes: 0
Labels: dbpatches, upgrade
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Issue Links:
Duplicate
is duplicated by ZBX-8745 after upgrade from 2.2 to 2.4 error u... Closed
is duplicated by ZBX-8746 upgrade 2.2 to 2.4 - zabbix server er... Closed

 Description   

If i click under Admin -> Proxy -> for every proxy i reciever
Undefined index: description [ in /usr/share/zabbix/proxies.php:233]



 Comments   
Comment by Alexander Vladishev [ 2014 Sep 13 ]

Whether there were errors while upgrading a database (the first start of the Zabbix server).

Please attach a log file of the Zabbix server.

Comment by AndreaConsadori [ 2014 Sep 15 ]

db upgrade goes to 100% without any error,
the error came after the db upgrade.

22076:20140911:175849.101 Starting Zabbix Server. Zabbix 2.4.0 (revision 48953).
22076:20140911:175849.101 ****** Enabled features ******
22076:20140911:175849.101 SNMP monitoring: YES
22076:20140911:175849.101 IPMI monitoring: YES
22076:20140911:175849.101 WEB monitoring: YES
22076:20140911:175849.101 VMware monitoring: YES
22076:20140911:175849.101 Jabber notifications: YES
22076:20140911:175849.101 Ez Texting notifications: YES
22076:20140911:175849.101 ODBC: YES
22076:20140911:175849.101 SSH2 support: YES
22076:20140911:175849.102 IPv6 support: YES
22076:20140911:175849.102 ******************************
22076:20140911:175849.102 using configuration file: /etc/zabbix/zabbix_server.conf
22076:20140911:175849.107 current database version (mandatory/optional): 02040000/02040000
22076:20140911:175849.108 required mandatory version: 02040000
22076:20140911:175849.108 table 'config' has multiple records
22076:20140911:175849.117 [Z3005] query failed: [1054] Unknown column 'bulk' in 'field list' [select interfaceid,hostid,type,main,useip,ip,dns,port,bulk from interface]
22076:20140911:175849.117 server #0 started [main process]
22078:20140911:175849.118 server #1 started configuration syncer #1
22079:20140911:175849.118 server #2 started db watchdog #1
22080:20140911:175849.118 server #3 started poller #1
22081:20140911:175849.119 server #4 started poller #2
22082:20140911:175849.119 server #5 started poller #3
22083:20140911:175849.119 server #6 started poller #4
22084:20140911:175849.120 server #7 started poller #5
22085:20140911:175849.120 server #8 started poller #6
22086:20140911:175849.121 server #9 started poller #7
22090:20140911:175849.129 server #13 started poller #11
22091:20140911:175849.130 server #14 started poller #12
22095:20140911:175849.130 server #18 started poller #16
22088:20140911:175849.132 server #11 started poller #9
22092:20140911:175849.133 server #15 started poller #13
22093:20140911:175849.133 server #16 started poller #14
22089:20140911:175849.133 server #12 started poller #10
22094:20140911:175849.134 server #17 started poller #15
22100:20140911:175849.141 server #23 started poller #21
22101:20140911:175849.141 server #24 started poller #22
22087:20140911:175849.141 server #10 started poller #8
22096:20140911:175849.141 server #19 started poller #17
22097:20140911:175849.141 server #20 started poller #18
...
22247:20140911:175849.638 server #170 started history syncer #13
22239:20140911:175849.652 server #162 started history syncer #5
22235:20140911:175849.640 server #158 started history syncer #1
22233:20140911:175849.653 [Z3005] query failed: [1054] Unknown column 't.headers' in 'field list' [select h.hostid,h.host,h.name,t.httptestid,t.name,t.variables,t.headers,t.agent,t.authentication,t.http_user,t.http_password,t.http_proxy,t.retries,t.ssl_cert_file,t.ssl_key_file,t.ssl_key_password,t.verify_peer,t.verify_host from httptest t,hosts h where t.hostid=h.hostid and t.nextcheck<=1410451129 and mod(t.httptestid,1)=0 and t.status=0 and h.proxy_hostid is null and h.status=0 and (h.maintenance_status=0 or h.maintenance_type=0)]
22236:20140911:175849.653 server #159 started history syncer #2
22243:20140911:175849.654 server #166 started history syncer #9
22233:20140911:175849.654 [Z3005] query failed: [1054] Unknown column 't.headers' in 'field list' [select h.hostid,h.host,h.name,t.httptestid,t.name,t.variables,t.headers,t.agent,t.authentication,t.http_user,t.http_password,t.http_proxy,t.retries,t.ssl_cert_file,t.ssl_key_file,t.ssl_key_password,t.verify_peer,t.verify_host from httptest t,hosts h where t.hostid=h.hostid and t.nextcheck<=1410451129 and mod(t.httptestid,1)=0 and t.status=0 and h.proxy_hostid is null and h.status=0 and (h.maintenance_status=0 or h.maintenance_type=0)]
22248:20140911:175849.654 server #171 started history syncer #14
....

Comment by richlv [ 2014 Sep 15 ]

do you have server log for the upgrade phase ?
you used distributed monitoring before, right ?

Comment by AndreaConsadori [ 2014 Sep 16 ]

I've no more install log because i notice the error one day after and yes, previously was a node install where i just switch to proxy but master server remain on node 1

Comment by AndreaConsadori [ 2014 Sep 16 ]

Maybe in mysql i've to do a manual cleanup of object with id major of 1002000000? (all my master node obj start with 10010000xxx

Comment by Oleksii Zagorskyi [ 2014 Sep 16 ]

You mentioned ID as 1002000000. It's quite strange.
According to https://www.zabbix.com/documentation/2.2/manual/appendix/faq, let's compare:

  1002000000
NNNSSSDDDDDDDDDDD

I guess you just trimmed unique ID part.
But it looks like you created the object on nodeid=2 for nodeid=1 which means that master node should be nodeid=2 and child is nodeid=1.
Otherwise it could not be possible.
It's a bit unusual.
It should not be any problem but who knows ...

Comment by Oleksii Zagorskyi [ 2014 Sep 16 ]

If you have any db backup before the upgrade to 2.4 - we could play with this, otherwise I don't have any idea how to investigate this case.

Comment by AndreaConsadori [ 2014 Sep 16 ]

where can i send my backup without sharing my host data with others?

Comment by richlv [ 2014 Sep 16 ]

how large is the backup file ?

Comment by AndreaConsadori [ 2014 Sep 16 ]

8mb

Comment by Oleksii Zagorskyi [ 2014 Sep 16 ]

you could send it to support at zabbix dot com

Comment by AndreaConsadori [ 2014 Sep 16 ]

sent with obj ZBX-8744

Comment by Alexander Vladishev [ 2014 Sep 16 ]

The scheme of your database differs from the original. Database must correspond to the original scheme. Except partitioning.

For example:

CREATE TABLE `trigger_discovery` (
  `triggerdiscoveryid` bigint(20) unsigned NOT NULL,
  `triggerid` bigint(20) unsigned NOT NULL,
  `parent_triggerid` bigint(20) unsigned NOT NULL,
  `name` varchar(255) NOT NULL DEFAULT '',
  PRIMARY KEY (`triggerdiscoveryid`),
  UNIQUE KEY `trigger_discovery_1` (`triggerid`,`parent_triggerid`),
  KEY `c_trigger_discovery_2` (`parent_triggerid`),
  KEY `trigger_discovery_2` (`parent_triggerid`)
) ENGINE=InnoDB DEFAULT CHARSET=utf8

Original table definition:

CREATE TABLE `trigger_discovery` (
  `triggerdiscoveryid` bigint(20) unsigned NOT NULL,
  `triggerid` bigint(20) unsigned NOT NULL,
  `parent_triggerid` bigint(20) unsigned NOT NULL,
  `name` varchar(255) NOT NULL DEFAULT '',
  PRIMARY KEY (`triggerdiscoveryid`),
  UNIQUE KEY `trigger_discovery_1` (`triggerid`,`parent_triggerid`),
  KEY `trigger_discovery_2` (`parent_triggerid`),
  CONSTRAINT `c_trigger_discovery_2` FOREIGN KEY (`parent_triggerid`) REFERENCES `triggers` (`triggerid`) ON DELETE CASCADE,
  CONSTRAINT `c_trigger_discovery_1` FOREIGN KEY (`triggerid`) REFERENCES `triggers` (`triggerid`) ON DELETE CASCADE
) ENGINE=InnoDB DEFAULT CHARSET=utf8

We strongly recommends to add FOREIGN KEY constraints to fields which have them in the original db scheme. Otherwise the database will be inconsistent and Zabbix will work incorrectly.

Comment by Alexander Vladishev [ 2014 Sep 16 ]

In any case Zabbix should stop upgrade procedure when error occurred. It will be fixed.

Comment by Alexander Vladishev [ 2014 Sep 16 ]

Fixed in development branch svn://svn.zabbix.com/branches/dev/ZBX-8744

Comment by Daniel W Ottey [ 2014 Sep 16 ]

I am seeing this same error after upgrading zabbix proxy.

17327:20140916:110553.874 [Z3005] query failed: [1054] Unknown column 't.headers' in 'field list' [select h.hostid,h.host,h.name,t.httptestid,t.name,t.variables,t.headers,t.agent,t
.authentication,t.http_user,t.http_password,t.http_proxy,t.retries,t.ssl_cert_file,t.ssl_key_file,t.ssl_key_password,t.verify_peer,t.verify_host from httptest t,hosts h where t.host
id=h.hostid and t.nextcheck<=1410879953 and mod(t.httptestid,1)=0 and t.status=0 and h.proxy_hostid is null and h.status=0 and (h.maintenance_status=0 or h.maintenance_type=0)]

Is there a fix that I can apply?

Comment by tdudas [ 2014 Sep 16 ]

I also have the same error:

[Z3005] query failed: [1054] Unknown column 'bulk' in 'field list' [select interfaceid,hostid,type,main,useip,ip,dns,port,bulk from interface
....
]
[Z3005] query failed: [1054] Unknown column 't.headers' in 'field list'

I also try compile with svn://svn.zabbix.com/branches/dev/ZBX-8744 but still the same error. The upgrade database to 2.4 was failed.

Is there a fix that I can apply?

Comment by richlv [ 2014 Sep 16 ]

8744 branch fix will make it stop upon the first failure. for this to help any, you should restore the db with a backup before the upgrade and attempt the upgrade again. then you will see the original error.

Comment by Andris Zeila [ 2014 Sep 17 ]

Successfully tested

Comment by Alexander Vladishev [ 2014 Sep 17 ]

Fixed in pre-2.4.1 r49087 and pre-2.5.0 (trunk) r49088.

Comment by AndreaConsadori [ 2014 Sep 18 ]

how is possible to fix the scheme of database that differs from the original?

maybe some changes has been made on the fist version when mysql partitioning wasn't usable.

Comment by Oleksii Zagorskyi [ 2014 Sep 18 ]

AndreaConsadori, this issue report is not the place where you need to expect some help on your question.
See http://www.zabbix.org/wiki/Getting_help

In short - you need to fix your db schema yourself and manually, then make sure it corresponds with official db schema and only then do upgrade.

Comment by AndreaConsadori [ 2014 Oct 09 ]

in 2.4.1 i still have the issue
i re upgrade from 2.2.6 to 2.4.1

24437:20141009:103324.970 Starting Zabbix Server. Zabbix 2.4.0 (revision 48953).
24437:20141009:103324.970 ****** Enabled features ******
24437:20141009:103324.970 SNMP monitoring: YES
24437:20141009:103324.970 IPMI monitoring: YES
24437:20141009:103324.970 WEB monitoring: YES
24437:20141009:103324.970 VMware monitoring: YES
24437:20141009:103324.970 Jabber notifications: YES
24437:20141009:103324.970 Ez Texting notifications: YES
24437:20141009:103324.970 ODBC: YES
24437:20141009:103324.971 SSH2 support: YES
24437:20141009:103324.971 IPv6 support: YES
24437:20141009:103324.971 ******************************
24437:20141009:103324.971 using configuration file: /etc/zabbix/zabbix_server.conf
24437:20141009:103325.164 current database version (mandatory/optional): 02020000/02020001
24437:20141009:103325.164 required mandatory version: 02040000
24437:20141009:103325.164 starting automatic database upgrade
24437:20141009:103325.165 completed 0% of database upgrade
24437:20141009:103325.338 completed 1% of database upgrade
24437:20141009:103325.500 completed 2% of database upgrade
24437:20141009:103326.486 completed 3% of database upgrade
24437:20141009:103326.624 completed 4% of database upgrade
24437:20141009:103326.785 completed 5% of database upgrade
24437:20141009:103326.865 completed 6% of database upgrade
24437:20141009:103326.921 completed 7% of database upgrade
24437:20141009:103326.956 [Z3005] query failed: [1452] Cannot add or update a child row: a foreign key constraint fails (`zabbix`.`trigger_discovery`, CONSTRAINT `c_trigger_discovery_1` FOREIGN KEY (`triggerid`) REFERENCES `triggers` (`triggerid`) ON DELETE CASCADE) [insert into trigger_discovery (select triggerid,parent_triggerid from trigger_discovery_tmp)]
24437:20141009:103326.957 completed 8% of database upgrade
24437:20141009:103326.957 database upgrade fully completed
24437:20141009:103327.528 [Z3005] query failed: [1054] Unknown column 'bulk' in 'field list' [select interfaceid,hostid,type,main,useip,ip,dns,port,bulk from interface]
24437:20141009:103327.529 server #0 started [main process]

and start to fill all log of

25388:20141009:103330.352 [Z3005] query failed: [1054] Unknown column 't.headers' in 'field list' [select h.hostid,h.host,h.name,t.httptestid,t.name,t.variables,t.headers,t.agent,t.authentication,t.http_user,t.http_password,t.http_proxy,t.retries,t.ssl_cert_file,t.ssl_key_file,t.ssl_key_password,t.verify_peer,t.verify_host from httptest t,hosts h where t.hostid=h.hostid and t.nextcheck<=1412843610 and mod(t.httptestid,1)=0 and t.status=0 and h.proxy_hostid is null and h.status=0 and (h.maintenance_status=0 or h.maintenance_type=0)]
25388:20141009:103330.468 [Z3005] query failed: [1054] Unknown column 't.headers' in 'field list' [select h.hostid,h.host,h.name,t.httptestid,t.name,t.variables,t.headers,t.agent,t.authentication,t.http_user,t.http_password,t.http_proxy,t.retries,t.ssl_cert_file,t.ssl_key_file,t.ssl_key_password,t.verify_peer,t.verify_host from httptest t,hosts h where t.hostid=h.hostid and t.nextcheck<=1412843610 and mod(t.httptestid,1)=0 and t.status=0 and h.proxy_hostid is null and h.status=0 and (h.maintenance_status=0 or h.maintenance_type=0)]

now polling works but maps are gone, all corrupted

Comment by AndreaConsadori [ 2014 Oct 09 ]

with 2.4.1 issue persist

Comment by Alexander Vladishev [ 2014 Oct 09 ]

In the log I see "Starting Zabbix Server. Zabbix 2.4.0 (revision 48953).". Please check what version you use.

Comment by AndreaConsadori [ 2014 Oct 09 ]

mm... i see

zabbix-release-2.4-1.el6.noarch.rpm

but

zabbix-server-mysql-2.4.0-2.el6.x86_64.rpm

so rpm still uploading....

http://www.zabbix.com/rn2.4.1.php

Comment by Alexander Vladishev [ 2014 Oct 09 ]

Packages 2.4.1 will appear a little bit later. Please wait.

zabbix-release-2.4-1.el6.noarch.rpm <-- it isn't 2.4.1 packet.

<richlv> hmm... somehow we seem to have lost "0" in the version number, which could have helped to avoid this confusion

sasha It is "Zabbix repository configuration" package.

$ rpm -qip zabbix-release-2.4-1.el6.noarch.rpm 
warning: zabbix-release-2.4-1.el6.noarch.rpm: Header V4 DSA/SHA1 Signature, key ID 79ea5ed4: NOKEY
Name        : zabbix-release
Version     : 2.4
Release     : 1.el6
Architecture: noarch
Install Date: (not installed)
Group       : System Environment/Base
Size        : 20118
License     : GPL
Signature   : DSA/SHA1, Thu 11 Sep 2014 11:19:09 AM EEST, Key ID d13d58e479ea5ed4
Source RPM  : zabbix-release-2.4-1.el6.src.rpm
Build Date  : Thu 11 Sep 2014 11:10:26 AM EEST
Build Host  : build.zabbix.com
Relocations : (not relocatable)
Packager    : Zabbix SIA <https://support.zabbix.com>
Vendor      : Zabbix SIA
URL         : http://www.zabbix.com
Summary     : Zabbix repository configuration
Description :
This package contains the Zabbix official repository
GPG key as well as configuration for yum.

<richlv> ah, got it - thanks

Comment by Alexander Vladishev [ 2014 Oct 15 ]

FYI: 2.4.1 packages are published on our repository. Please try again.

Comment by Alexander Vladishev [ 2014 Nov 06 ]

I close the issue. Feel free to reopen.

Comment by Alexander Vladishev [ 2015 Jan 05 ]

This caused a regression: ZBX-9185

Generated at Wed Apr 24 05:31:01 EEST 2024 using Jira 9.12.4#9120004-sha1:625303b708afdb767e17cb2838290c41888e9ff0.