Uploaded image for project: 'ZABBIX BUGS AND ISSUES'
  1. ZABBIX BUGS AND ISSUES
  2. ZBX-15635

zabbix_server Segfault when there are NULLs in the host_inventory table

    Details

    • Type: Problem report
    • Status: Closed
    • Priority: Critical
    • Resolution: Won't fix
    • Affects Version/s: 4.0.3, 4.0.4
    • Fix Version/s: None
    • Component/s: Server (S)
    • Labels:
    • Environment:
      Zabbix server 4.0.3 with MariaDB 10.
    • Team:
      Team A
    • Sprint:
      Sprint 49 (Feb 2019)
    • Story Points:
      0

      Description

      Steps to reproduce:

      1. Install Zabbix server 4.0.3 with MariaDB
      2. Update a row in the host_inventory table that it contains some NULLs ex.:**
        select * from host_inventory where type is NULL or type_full is NULL or name is NULL or alias is NULL or os is NULL or os_full is NULL or os_short is NULL or serialno_a is NULL or serialno_b is NULL or tag is NULL or asset_tag is NULL or macaddress_a is NULL or macaddress_b is NULL or hardware is NULL or hardware_full is NULL or software is NULL or software_full is NULL or software_app_a is NULL or software_app_b is NULL or software_app_c is NULL or software_app_d is NULL or software_app_e is NULL or contact is NULL or location is NULL or location_lat is NULL or location_lon is NULL or notes is NULL or chassis is NULL or model is NULL or hw_arch is NULL or vendor is NULL or contract_number is NULL or installer_name is NULL or deployment_status is NULL or url_a is NULL or url_b is NULL or url_c is NULL or host_networks is NULL or host_netmask is NULL or host_router is NULL or oob_ip is NULL or oob_netmask is NULL or oob_router is NULL or date_hw_purchase is NULL or date_hw_install is NULL or date_hw_expiry is NULL or date_hw_decomm is NULL or site_address_a is NULL or site_address_b is NULL or site_address_c is NULL or site_city is NULL or site_state is NULL or site_country is NULL or site_zip is NULL or site_rack is NULL or site_notes is NULL or poc_1_name is NULL or poc_1_email is NULL or poc_1_phone_a is NULL or poc_1_phone_b is NULL or poc_1_cell is NULL or poc_1_screen is NULL or poc_1_notes is NULL or poc_2_name is NULL or poc_2_email is NULL or poc_2_phone_a is NULL or poc_2_phone_b is NULL or poc_2_cell is NULL or poc_2_screen is NULL or poc_2_notes is NULL;+--------+----------------+------+-----------+------+-------+------+---------+----------+------------+------------+------+-----------+--------------+--------------+----------+---------------+----------+---------------+----------------+----------------+----------------+----------------+----------------+---------+----------+--------------+--------------+-------+---------+-------+---------+--------+-----------------+----------------+-------------------+-------+-------+-------+---------------+--------------+-------------+--------+-------------+------------+------------------+-----------------+----------------+----------------+----------------+----------------+----------------+-----------+------------+--------------+----------+-----------+------------+------------+-------------+---------------+---------------+------------+--------------+-------------+------------+-------------+---------------+---------------+------------+--------------+-------------+| hostid | inventory_mode | type | type_full | name | alias | os   | os_full | os_short | serialno_a | serialno_b | tag  | asset_tag | macaddress_a | macaddress_b | hardware | hardware_full | software | software_full | software_app_a | software_app_b | software_app_c | software_app_d | software_app_e | contact | location | location_lat | location_lon | notes | chassis | model | hw_arch | vendor | contract_number | installer_name | deployment_status | url_a | url_b | url_c | host_networks | host_netmask | host_router | oob_ip | oob_netmask | oob_router | date_hw_purchase | date_hw_install | date_hw_expiry | date_hw_decomm | site_address_a | site_address_b | site_address_c | site_city | site_state | site_country | site_zip | site_rack | site_notes | poc_1_name | poc_1_email | poc_1_phone_a | poc_1_phone_b | poc_1_cell | poc_1_screen | poc_1_notes | poc_2_name | poc_2_email | poc_2_phone_a | poc_2_phone_b | poc_2_cell | poc_2_screen | poc_2_notes |+--------+----------------+------+-----------+------+-------+------+---------+----------+------------+------------+------+-----------+--------------+--------------+----------+---------------+----------+---------------+----------------+----------------+----------------+----------------+----------------+---------+----------+--------------+--------------+-------+---------+-------+---------+--------+-----------------+----------------+-------------------+-------+-------+-------+---------------+--------------+-------------+--------+-------------+------------+------------------+-----------------+----------------+----------------+----------------+----------------+----------------+-----------+------------+--------------+----------+-----------+------------+------------+-------------+---------------+---------------+------------+--------------+-------------+------------+-------------+---------------+---------------+------------+--------------+-------------+|  12037 |              1 |      |           | NULL |       | NULL | NULL    |          |            |            |      |           |              |              |          |               |          |               |                |                |                |                | NULL           |         |          |              |              |       |         |       | NULL    |        |                 |                |                   |       |       |       |               |              |             |        |             |            |                  |                 |                |                |                |                |                |           |            |              |          |           |            |            |             |               |               |            |              |             |            |             |               |               |            |              |             |+--------+----------------+------+-----------+------+-------+------+---------+----------+------------+------------+------+-----------+--------------+--------------+----------+---------------+----------+---------------+----------------+----------------+----------------+----------------+----------------+---------+----------+--------------+--------------+-------+---------+-------+---------+--------+-----------------+----------------+-------------------+-------+-------+-------+---------------+--------------+-------------+--------+-------------+------------+------------------+-----------------+----------------+----------------+----------------+----------------+----------------+-----------+------------+--------------+----------+-----------+------------+------------+-------------+---------------+---------------+------------+--------------+-------------+------------+-------------+---------------+---------------+------------+--------------+-------------+1 row in set (0.01 sec)

        {{}}

      3. Wait for zabbix_server to crash

      Result:

      Zabbix_server crashes in DCsync_host_inventory procedure with log:

      27067:20190211:105800.551 Starting Zabbix Server. Zabbix 4.0.3 (revision 87993).
       27067:20190211:105800.551 ****** Enabled features ******
       27067:20190211:105800.551 SNMP monitoring:           YES
       27067:20190211:105800.551 IPMI monitoring:           YES
       27067:20190211:105800.551 Web monitoring:            YES
       27067:20190211:105800.551 VMware monitoring:         YES
       27067:20190211:105800.551 SMTP authentication:       YES
       27067:20190211:105800.551 Jabber notifications:       NO
       27067:20190211:105800.551 Ez Texting notifications:  YES
       27067:20190211:105800.551 ODBC:                      YES
       27067:20190211:105800.551 SSH2 support:              YES
       27067:20190211:105800.551 IPv6 support:               NO
       27067:20190211:105800.551 TLS support:                NO
       27067:20190211:105800.551 ******************************
       27067:20190211:105800.551 using configuration file: /usr/zabbix/etc/zabbix_server.conf
       27067:20190211:105800.551 In zbx_load_modules()
       27067:20190211:105800.551 End of zbx_load_modules():SUCCEED
       27067:20190211:105800.551 In init_database_cache()
       27067:20190211:105800.551 In zbx_mem_create() param:'HistoryCacheSize' size:67108864
       27067:20190211:105800.551 valid user addresses: [0x7f7e1cf2b168, 0x7f7e20f2aff0] total size: 67108488
       27067:20190211:105800.551 End of zbx_mem_create()
       27067:20190211:105800.552 In zbx_mem_create() param:'HistoryIndexCacheSize' size:67108864
       27067:20190211:105800.552 valid user addresses: [0x7f7e18f2b178, 0x7f7e1cf2aff0] total size: 67108472
       27067:20190211:105800.552 End of zbx_mem_create()
       27067:20190211:105800.552 In init_trend_cache()
       27067:20190211:105800.552 In zbx_mem_required_size() size:0 chunks_num:1 descr:'trend cache' param:'TrendCacheSize'
       27067:20190211:105800.552 End of zbx_mem_required_size() size:414
       27067:20190211:105800.552 In zbx_mem_create() param:'TrendCacheSize' size:67108864
       27067:20190211:105800.552 valid user addresses: [0x7f7e14f2b168, 0x7f7e18f2aff0] total size: 67108488
       27067:20190211:105800.552 End of zbx_mem_create()
       27067:20190211:105800.552 End of init_trend_cache()
       27067:20190211:105800.552 End of init_database_cache()
       27067:20190211:105800.552 In init_configuration_cache() size:134217728
       27067:20190211:105800.552 In zbx_mem_create() param:'CacheSize' size:134217728
       27067:20190211:105800.552 valid user addresses: [0x7f7e0cf2b168, 0x7f7e14f2aff0] total size: 134217352
       27067:20190211:105800.552 End of zbx_mem_create()
       27067:20190211:105800.552 End of init_configuration_cache()
       27067:20190211:105800.552 In init_selfmon_collector()
       27067:20190211:105800.553 init_selfmon_collector() size:30960
       27067:20190211:105800.553 End of init_selfmon_collector() collector:0x7f7e2816c000
       27067:20190211:105800.553 In zbx_vmware_init()
       27067:20190211:105800.553 In zbx_mem_required_size() size:0 chunks_num:1 descr:'vmware cache size' param:'VMwareCacheSize'
       27067:20190211:105800.553 End of zbx_mem_required_size() size:421
       27067:20190211:105800.553 In zbx_mem_create() param:'VMwareCacheSize' size:134217307
       27067:20190211:105800.553 valid user addresses: [0x7f7e04f2b170, 0x7f7e0cf2ae50] total size: 134216928
       27067:20190211:105800.553 End of zbx_mem_create()
       27067:20190211:105800.553 End of zbx_vmware_init()
       27067:20190211:105800.553 In zbx_vc_init()
       27067:20190211:105800.553 In zbx_mem_required_size() size:0 chunks_num:1 descr:'value cache size' param:'ValueCacheSize'
       27067:20190211:105800.553 End of zbx_mem_required_size() size:419
       27067:20190211:105800.553 In zbx_mem_create() param:'ValueCacheSize' size:67108864
       27067:20190211:105800.553 valid user addresses: [0x7f7e00f2b168, 0x7f7e04f2aff0] total size: 67108488
       27067:20190211:105800.553 End of zbx_mem_create()
       27067:20190211:105800.553 End of zbx_vc_init()
       27067:20190211:105800.553 In zbx_db_get_database_type()
       27067:20190211:105800.553 In DBconnect() flag:0
       27067:20190211:105800.558 End of DBconnect():0
       27067:20190211:105800.558 query [txnlev:0] [select userid from users limit 1]
       27067:20190211:105800.559 there is at least 1 record in "users" table
       27067:20190211:105800.559 End of zbx_db_get_database_type():ZBX_DB_SERVER
       27067:20190211:105800.559 In DBcheck_version()
       27067:20190211:105800.559 In DBconnect() flag:0
       27067:20190211:105800.565 End of DBconnect():0
       27067:20190211:105800.565 query [txnlev:0] [show tables like 'dbversion']
       27067:20190211:105800.565 query [txnlev:0] [select mandatory,optional from dbversion]
       27067:20190211:105800.566 current database version (mandatory/optional): 04000000/04000003
       27067:20190211:105800.566 required mandatory version: 04000000
       27067:20190211:105800.566 End of DBcheck_version():SUCCEED
       27067:20190211:105800.566 In DBconnect() flag:0
       27067:20190211:105800.567 End of DBconnect():0
       27067:20190211:105800.567 In DCsync_configuration()
       27067:20190211:105800.567 query [txnlev:0] [select refresh_unsupported,discovery_groupid,snmptrap_logging,severity_name_0,severity_name_1,severity_name_2,severity_name_3,severity_name_4,severity_name_5,hk_events_mode,hk_events_trigger,hk_events_internal,hk_events_discovery,hk_events_autoreg,hk_services_mode,hk_services,hk_audit_mode,hk_audit,hk_sessions_mode,hk_sessions,hk_history_mode,hk_history_global,hk_history,hk_trends_mode,hk_trends_global,hk_trends,default_inventory_mode from config order by configid]
       27067:20190211:105800.567 In DCsync_config()
       27067:20190211:105800.567 End of DCsync_config()
       27067:20190211:105800.567 query [txnlev:0] [select hostid,templateid from hosts_templates order by hostid]
       27067:20190211:105800.568 query [txnlev:0] [select globalmacroid,macro,value from globalmacro]
       27067:20190211:105800.568 query [txnlev:0] [select hostmacroid,hostid,macro,value from hostmacro]
       27067:20190211:105800.569 In DCsync_htmpls()
       27067:20190211:105800.570 End of DCsync_htmpls()
       27067:20190211:105800.570 In DCsync_gmacros()
       27067:20190211:105800.570 End of DCsync_gmacros()
       27067:20190211:105800.570 In DCsync_hmacros()
       27067:20190211:105800.570 End of DCsync_hmacros()
       27067:20190211:105800.570 query [txnlev:0] [select hostid,proxy_hostid,host,ipmi_authtype,ipmi_privilege,ipmi_username,ipmi_password,maintenance_status,maintenance_type,maintenance_from,errors_from,available,disable_until,snmp_errors_from,snmp_available,snmp_disable_until,ipmi_errors_from,ipmi_available,ipmi_disable_until,jmx_errors_from,jmx_available,jmx_disable_until,status,name,lastaccess,error,snmp_error,ipmi_error,jmx_error,tls_connect,tls_accept,proxy_address,auto_compress,maintenanceid from hosts where status in (0,1,5,6) and flags<>2]
       27067:20190211:105800.574 query [txnlev:0] [select hostid,inventory_mode,type,type_full,name,alias,os,os_full,os_short,serialno_a,serialno_b,tag,asset_tag,macaddress_a,macaddress_b,hardware,hardware_full,software,software_full,software_app_a,software_app_b,software_app_c,software_app_d,software_app_e,contact,location,location_lat,location_lon,notes,chassis,model,hw_arch,vendor,contract_number,installer_name,deployment_status,url_a,url_b,url_c,host_networks,host_netmask,host_router,oob_ip,oob_netmask,oob_router,date_hw_purchase,date_hw_install,date_hw_expiry,date_hw_decomm,site_address_a,site_address_b,site_address_c,site_city,site_state,site_country,site_zip,site_rack,site_notes,poc_1_name,poc_1_email,poc_1_phone_a,poc_1_phone_b,poc_1_cell,poc_1_screen,poc_1_notes,poc_2_name,poc_2_email,poc_2_phone_a,poc_2_phone_b,poc_2_cell,poc_2_screen,poc_2_notes from host_inventory]
       27067:20190211:105800.579 query [txnlev:0] [select groupid,name from hstgrp]
       27067:20190211:105800.580 query [txnlev:0] [select hg.groupid,hg.hostid from hosts_groups hg,hosts h where hg.hostid=h.hostid and h.status in (0,1) and h.flags<>2 order by hg.groupid]
       27067:20190211:105800.584 query [txnlev:0] [select maintenanceid,maintenance_type,active_since,active_till,tags_evaltype from maintenances]
       27067:20190211:105800.584 query [txnlev:0] [select maintenancetagid,maintenanceid,operator,tag,value from maintenance_tag]
       27067:20190211:105800.584 query [txnlev:0] [select t.timeperiodid,t.timeperiod_type,t.every,t.month,t.dayofweek,t.day,t.start_time,t.period,t.start_date,m.maintenanceid from maintenances_windows m,timeperiods t where t.timeperiodid=m.timeperiodid]
       27067:20190211:105800.585 query [txnlev:0] [select maintenanceid,groupid from maintenances_groups order by maintenanceid]
       27067:20190211:105800.585 query [txnlev:0] [select maintenanceid,hostid from maintenances_hosts order by maintenanceid]
       27067:20190211:105800.585 In DCsync_hosts()
       27067:20190211:105800.587 End of DCsync_hosts()
       27067:20190211:105800.587 In DCsync_host_inventory()
       27067:20190211:105800.591 Got signal [signal:11(SIGSEGV),reason:1,refaddr:(nil)]. Crashing ...
       27067:20190211:105800.591 ====== Fatal information: ======
       27067:20190211:105800.591 Program counter: 0x7f7e24a20f8f
       27067:20190211:105800.591 === Registers: ===
       27067:20190211:105800.591 r8      =                0 =                    0 =                    0
       27067:20190211:105800.591 r9      =          1034aa7 =             16992935 =             16992935
       27067:20190211:105800.591 r10     =                1 =                    1 =                    1
       27067:20190211:105800.591 r11     =                0 =                    0 =                    0
       27067:20190211:105800.591 r12     =     7f7e0cf2c030 =      140179359842352 =      140179359842352
       27067:20190211:105800.591 r13     =     7f7e0d09bf90 =      140179361349520 =      140179361349520
       27067:20190211:105800.591 r14     = fffffffffffffffc = 18446744073709551612 =                   -4
       27067:20190211:105800.591 r15     =               20 =                   32 =                   32
       27067:20190211:105800.591 rdi     =                0 =                    0 =                    0
       27067:20190211:105800.591 rsi     = fffffffffffffffc = 18446744073709551612 =                   -4
       27067:20190211:105800.591 rbp     =                0 =                    0 =                    0
       27067:20190211:105800.591 rbx     =                0 =                    0 =                    0
       27067:20190211:105800.591 rdx     =                0 =                    0 =                    0
       27067:20190211:105800.591 rax     =              fa7 =                 4007 =                 4007
       27067:20190211:105800.591 rcx     =                0 =                    0 =                    0
       27067:20190211:105800.591 rsp     =     7ffc42c23358 =      140721428509528 =      140721428509528
       27067:20190211:105800.591 rip     =     7f7e24a20f8f =      140179757207439 =      140179757207439
       27067:20190211:105800.591 efl     =            10246 =                66118 =                66118
       27067:20190211:105800.591 csgsfs  =               33 =                   51 =                   51
       27067:20190211:105800.591 err     =                4 =                    4 =                    4
       27067:20190211:105800.591 trapno  =                e =                   14 =                   14
       27067:20190211:105800.591 oldmask =                0 =                    0 =                    0
       27067:20190211:105800.591 cr2     =                0 =                    0 =                    0
       27067:20190211:105800.591 === Backtrace: ===
       27067:20190211:105800.592 14: /usr/zabbix/sbin/zabbix_server(zbx_backtrace+0x2b) [0x49dc9b]
       27067:20190211:105800.592 13: /usr/zabbix/sbin/zabbix_server(zbx_log_fatal_info+0x155) [0x49deb5]
       27067:20190211:105800.592 12: /usr/zabbix/sbin/zabbix_server() [0x49e156]
       27067:20190211:105800.592 11: /lib64/libpthread.so.0(+0xf5d0) [0x7f7e278495d0]
       27067:20190211:105800.592 10: /lib64/libc.so.6(+0x145f8f) [0x7f7e24a20f8f]
       27067:20190211:105800.592 9: /usr/zabbix/sbin/zabbix_server(zbx_default_string_hash_func+0x9) [0x49ebe9]
       27067:20190211:105800.592 8: /usr/zabbix/sbin/zabbix_server(zbx_hashset_search+0x1d) [0x4a138d]
       27067:20190211:105800.592 7: /usr/zabbix/sbin/zabbix_server() [0x47c023]
       27067:20190211:105800.592 6: /usr/zabbix/sbin/zabbix_server(DCstrpool_replace+0x17) [0x47f587]
       27067:20190211:105800.592 5: /usr/zabbix/sbin/zabbix_server(DCsync_configuration+0x18f9) [0x482159]
       27067:20190211:105800.592 4: /usr/zabbix/sbin/zabbix_server(MAIN_ZABBIX_ENTRY+0x36e) [0x421d8e]
       27067:20190211:105800.592 3: /usr/zabbix/sbin/zabbix_server(daemon_start+0x1a5) [0x49d7b5]
       27067:20190211:105800.592 2: /usr/zabbix/sbin/zabbix_server(main+0x3ca) [0x42129a]
       27067:20190211:105800.592 1: /lib64/libc.so.6(__libc_start_main+0xf5) [0x7f7e248fd3d5]
       27067:20190211:105800.592 0: /usr/zabbix/sbin/zabbix_server() [0x42159e]
      

      NULLs in host_inventory can appear when API is used. Personally I use ansible with python zabbix-api (0.5.3). Even when I set empty string '' in all columns:

          inventory_mode: automatic
          inventory_zabbix:
              type: ""
              type_full: ""
              name: ""
              alias: ""
              os: ""
              os_full: ""
              os_short: ""
              serialno_a: ""
              serialno_b: ""
              tag: ""
              asset_tag: ""
              macaddress_a: ""
              macaddress_b: ""
              hardware: ""
              hardware_full: ""
              software: ""
              software_full: ""
              software_app_a: ""
              software_app_b: ""
              software_app_c: ""
              software_app_d: ""
              software_app_e: ""
              contact: ""
              location: ""
              location_lat: ""
              location_lon: ""
              notes: ""
              chassis: ""
              model: ""
              hw_arch: ""
              vendor: ""
              contract_number: ""
              installer_name: ""
              deployment_status: ""
              url_a: ""
              url_b: ""
              url_c: ""
              host_networks: ""
              host_netmask: ""
              host_router: ""
              oob_ip: ""
              oob_netmask: ""
              oob_router: ""
              date_hw_purchase: ""
              date_hw_install: ""
              date_hw_expiry: ""
              date_hw_decomm: ""
              site_address_a: ""
              site_address_b: ""
              site_address_c: ""
              site_city: ""
              site_state: ""
              site_country: ""
              site_zip: ""
              site_rack: ""
              site_notes: ""
              poc_1_name: ""
              poc_1_email: ""
              poc_1_phone_a: ""
              poc_1_phone_b: ""
              poc_1_cell: ""
              poc_1_screen: ""
              poc_1_notes: ""
              poc_2_name: ""
              poc_2_email: ""
              poc_2_phone_a: ""
              poc_2_phone_b: ""
              poc_2_cell: ""
              poc_2_screen: ""
              poc_2_notes: ""

      NULLs appear in the columns with automatic inventory update in assigned template.

        Attachments

          Activity

            People

            • Assignee:
              vso Vladislavs Sokurenko
              Reporter:
              bogus Bogusław Sikora
            • Votes:
              0 Vote for this issue
              Watchers:
              5 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: