Uploaded image for project: 'ZABBIX BUGS AND ISSUES'
  1. ZABBIX BUGS AND ISSUES
  2. ZBX-21613

Zabbix HA cluster standby node can start database upgrade

    XMLWordPrintable

Details

    • Problem report
    • Status: Confirmed
    • Major
    • Resolution: Unresolved
    • 6.2.2
    • None
    • Server (S)
    • None

    Description

      Steps to reproduce:

      1. Install two Zabbix server HA nodes with Zabbix server 6.0
      2. Upgrade Zabbix to 6.2 on the standby node
      3. Watch log files

      Result:
      Standby node upgrades Zabbix DB schema, because database version check is initialized before HA manager starts:

        1978:20220908:070025.835 Starting Zabbix Server. Zabbix 6.2.2 (revision 35455866073).
        1978:20220908:070025.835 ******************************
        1978:20220908:070025.835 using configuration file: /etc/zabbix/zabbix_server.conf
        1978:20220908:070025.961 current database version (mandatory/optional): 06000000/06000004
        1978:20220908:070025.961 required mandatory version: 06020000
        1978:20220908:070025.961 starting automatic database upgrade
        1978:20220908:070025.978 completed 1% of database upgrade
        1978:20220908:070026.064 completed 2% of database upgrade
        1978:20220908:070026.112 completed 3% of database upgrade
        ........
        1978:20220908:070039.160 completed 97% of database upgrade
        1978:20220908:070039.425 completed 98% of database upgrade
        1978:20220908:070039.443 completed 100% of database upgrade
        1978:20220908:070039.445 database upgrade fully completed
        2058:20220908:070039.486 starting HA manager
        2058:20220908:070039.526 HA manager started in standby mode
        1978:20220908:070039.526 "zbx_node_2" node started in "standby" mode
      

       Active node continues to work with incompatible DB schema and does not even crash:

        7980:20220908:070026.691 [Z3005] query failed: [1412] Table definition has changed, please retry transaction [update triggers set state=1,error='Cannot evaluate function last(/Zabbix server/zabbix.nodes.status[cl7sp6fau0001fpqs4u4nctrw],#2): not enough data.' where triggerid=22963;
      ]
        7963:20220908:070128.948 [Z3005] query failed: [1054] Unknown column 'lastaccess' in 'field list' [select hostid,proxy_hostid,host,ipmi_authtype,ipmi_privilege,ipmi_username,ipmi_password,maintenance_status,maintenance_type,maintenance_from,status,name,lastaccess,tls_connect,tls_accept,tls_issuer,tls_subject,tls_psk_identity,tls_psk,proxy_address,auto_compress,maintenanceid from hosts where status in (0,1,5,6) and flags<>2]
      zabbix_server [7963]: ERROR [file and function: <dbconfig.c,DCsync_configuration>, revision:c7c3044a4a2, line:6687] Something impossible has just happened.
        7963:20220908:070128.948 === Backtrace: ===
        7963:20220908:070128.950 9: /usr/sbin/zabbix_server: configuration syncer [synced configuration in 0.247249 sec, syncing configuration](zbx_backtrace+0x3f) [0x55993573d8c0]
        7963:20220908:070128.950 8: /usr/sbin/zabbix_server: configuration syncer [synced configuration in 0.247249 sec, syncing configuration](DCsync_configuration+0x310f) [0x5599356d3778]
        7963:20220908:070128.950 7: /usr/sbin/zabbix_server: configuration syncer [synced configuration in 0.247249 sec, syncing configuration](dbconfig_thread+0x31e) [0x55993556ab0d]
        7963:20220908:070128.950 6: /usr/sbin/zabbix_server: configuration syncer [synced configuration in 0.247249 sec, syncing configuration](zbx_thread_start+0x37) [0x55993574d9b3]
        7963:20220908:070128.950 5: /usr/sbin/zabbix_server: configuration syncer [synced configuration in 0.247249 sec, syncing configuration](+0x66924) [0x559935558924]
        7963:20220908:070128.950 4: /usr/sbin/zabbix_server: configuration syncer [synced configuration in 0.247249 sec, syncing configuration](MAIN_ZABBIX_ENTRY+0xa8c) [0x559935559d8e]
        7963:20220908:070128.950 3: /usr/sbin/zabbix_server: configuration syncer [synced configuration in 0.247249 sec, syncing configuration](daemon_start+0x384) [0x55993573d4df]
        7963:20220908:070128.950 2: /usr/sbin/zabbix_server: configuration syncer [synced configuration in 0.247249 sec, syncing configuration](main+0x33a) [0x5599355581ce]
        7963:20220908:070128.950 1: /lib64/libc.so.6(__libc_start_main+0xf3) [0x7fb7575a5493]
        7963:20220908:070128.950 0: /usr/sbin/zabbix_server: configuration syncer [synced configuration in 0.247249 sec, syncing configuration](_start+0x2e) [0x559935556f1e]
      
      

      Expected:
      Standby node will not upgrade DB schema, or at least Zabbix server will stop if incompatible DB schema is detected

      Attachments

        Activity

          People

            zabbix.dev Zabbix Development Team
            kaspars.mednis Kaspars Mednis
            Votes:
            0 Vote for this issue
            Watchers:
            2 Start watching this issue

            Dates

              Created:
              Updated: