Uploaded image for project: 'ZABBIX BUGS AND ISSUES'
  1. ZABBIX BUGS AND ISSUES
  2. ZBX-21613

Zabbix HA cluster standby node can start database upgrade

XMLWordPrintable

    • Icon: Problem report Problem report
    • Resolution: Fixed
    • Icon: Major Major
    • 6.0.18rc1, 7.0 (plan)
    • 6.2.2
    • Server (S)
    • None
    • Sprint 99 (Apr 2023), Sprint 100 (May 2023)
    • 1

      Steps to reproduce:

      1. Install two Zabbix server HA nodes with Zabbix server 6.0
      2. Upgrade Zabbix to 6.2 on the standby node
      3. Watch log files

      Result:
      Standby node upgrades Zabbix DB schema, because database version check is initialized before HA manager starts:

        1978:20220908:070025.835 Starting Zabbix Server. Zabbix 6.2.2 (revision 35455866073).
        1978:20220908:070025.835 ******************************
        1978:20220908:070025.835 using configuration file: /etc/zabbix/zabbix_server.conf
        1978:20220908:070025.961 current database version (mandatory/optional): 06000000/06000004
        1978:20220908:070025.961 required mandatory version: 06020000
        1978:20220908:070025.961 starting automatic database upgrade
        1978:20220908:070025.978 completed 1% of database upgrade
        1978:20220908:070026.064 completed 2% of database upgrade
        1978:20220908:070026.112 completed 3% of database upgrade
        ........
        1978:20220908:070039.160 completed 97% of database upgrade
        1978:20220908:070039.425 completed 98% of database upgrade
        1978:20220908:070039.443 completed 100% of database upgrade
        1978:20220908:070039.445 database upgrade fully completed
        2058:20220908:070039.486 starting HA manager
        2058:20220908:070039.526 HA manager started in standby mode
        1978:20220908:070039.526 "zbx_node_2" node started in "standby" mode
      

       Active node continues to work with incompatible DB schema and does not even crash:

        7980:20220908:070026.691 [Z3005] query failed: [1412] Table definition has changed, please retry transaction [update triggers set state=1,error='Cannot evaluate function last(/Zabbix server/zabbix.nodes.status[cl7sp6fau0001fpqs4u4nctrw],#2): not enough data.' where triggerid=22963;
      ]
        7963:20220908:070128.948 [Z3005] query failed: [1054] Unknown column 'lastaccess' in 'field list' [select hostid,proxy_hostid,host,ipmi_authtype,ipmi_privilege,ipmi_username,ipmi_password,maintenance_status,maintenance_type,maintenance_from,status,name,lastaccess,tls_connect,tls_accept,tls_issuer,tls_subject,tls_psk_identity,tls_psk,proxy_address,auto_compress,maintenanceid from hosts where status in (0,1,5,6) and flags<>2]
      zabbix_server [7963]: ERROR [file and function: <dbconfig.c,DCsync_configuration>, revision:c7c3044a4a2, line:6687] Something impossible has just happened.
        7963:20220908:070128.948 === Backtrace: ===
        7963:20220908:070128.950 9: /usr/sbin/zabbix_server: configuration syncer [synced configuration in 0.247249 sec, syncing configuration](zbx_backtrace+0x3f) [0x55993573d8c0]
        7963:20220908:070128.950 8: /usr/sbin/zabbix_server: configuration syncer [synced configuration in 0.247249 sec, syncing configuration](DCsync_configuration+0x310f) [0x5599356d3778]
        7963:20220908:070128.950 7: /usr/sbin/zabbix_server: configuration syncer [synced configuration in 0.247249 sec, syncing configuration](dbconfig_thread+0x31e) [0x55993556ab0d]
        7963:20220908:070128.950 6: /usr/sbin/zabbix_server: configuration syncer [synced configuration in 0.247249 sec, syncing configuration](zbx_thread_start+0x37) [0x55993574d9b3]
        7963:20220908:070128.950 5: /usr/sbin/zabbix_server: configuration syncer [synced configuration in 0.247249 sec, syncing configuration](+0x66924) [0x559935558924]
        7963:20220908:070128.950 4: /usr/sbin/zabbix_server: configuration syncer [synced configuration in 0.247249 sec, syncing configuration](MAIN_ZABBIX_ENTRY+0xa8c) [0x559935559d8e]
        7963:20220908:070128.950 3: /usr/sbin/zabbix_server: configuration syncer [synced configuration in 0.247249 sec, syncing configuration](daemon_start+0x384) [0x55993573d4df]
        7963:20220908:070128.950 2: /usr/sbin/zabbix_server: configuration syncer [synced configuration in 0.247249 sec, syncing configuration](main+0x33a) [0x5599355581ce]
        7963:20220908:070128.950 1: /lib64/libc.so.6(__libc_start_main+0xf3) [0x7fb7575a5493]
        7963:20220908:070128.950 0: /usr/sbin/zabbix_server: configuration syncer [synced configuration in 0.247249 sec, syncing configuration](_start+0x2e) [0x559935556f1e]
      
      

      Expected:
      Standby node will not upgrade DB schema, or at least Zabbix server will stop if incompatible DB schema is detected

            vso Vladislavs Sokurenko
            kaspars.mednis Kaspars Mednis
            Team A
            Votes:
            2 Vote for this issue
            Watchers:
            6 Start watching this issue

              Created:
              Updated:
              Resolved: