Uploaded image for project: 'ZABBIX BUGS AND ISSUES'
  1. ZABBIX BUGS AND ISSUES
  2. ZBX-21011

server log is spammed by flapping "error reason ... changed" messages for zabbix.nodes.* items

XMLWordPrintable

    • Sprint 88 (May 2022), Sprint 89 (Jun 2022), Sprint 90 (Jul 2022), Sprint 91 (Aug 2022), Sprint 92 (Sep 2022), Sprint 93 (Oct 2022), Sprint 94 (Nov 2022), Sprint 95 (Dec 2022), Sprint 96 (Jan 2023), Sprint 97 (Feb 2023), Sprint 98 (Mar 2023), Sprint 99 (Apr 2023), Sprint 100 (May 2023), Sprint 101 (Jun 2023), Sprint 102 (Jul 2023), Sprint 103 (Aug 2023), Sprint 104 (Sep 2023), Sprint 105 (Oct 2023), Sprint 106 (Nov 2023), Sprint 107 (Dec 2023), Sprint candidates, S2401
    • 0.125

      Imagine that you have a standalone zabbix server working. Existing "Zabbix server" host will discover 4 items (rule: High availability cluster node discovery), where node name is empty.

      Then you uncomment and set HANodeName= parameter in server conf file, restart.
      After that an existing standalone entry in "ha_node" DB table will be replaced by a new one, generated for the node.
      So now zabbix installation will get 4 additional discovered items, while previously existing ones will be marked for deletion. This is fine.

      But! Those 4 previously existing items will start to spam zabbix_server.log by messages like:

      356996:20220509:134625.878 error reason for "Zabbix server:zabbix.nodes.lastaccess.time[cl1mhfat80001w98k8vvkvxb9]" changed: Preprocessing failed for: [{"id":"cl2yl9hny00011r8kta7i7063","name":"node1","status":3,"lastaccess":1652093184,"address":"l...
      1. Failed: cannot extract value from json by path "$.[?(@.id=="cl1mhfat80001w98k8vvkvxb9")].lastaccess.first()": no data matches the specified path
      1356996:20220509:134625.878 error reason for "Zabbix server:zabbix.nodes.status[cl1mhfat80001w98k8vvkvxb9]" changed: Preprocessing failed for: [{"id":"cl2yl9hny00011r8kta7i7063","name":"node1","status":3,"lastaccess":1652093184,"address":"l...
      1. Failed: cannot extract value from json by path "$.[?(@.id=="cl1mhfat80001w98k8vvkvxb9")].status.first()": no data matches the specified path
      1356996:20220509:134625.878 error reason for "Zabbix server:zabbix.nodes.lastaccess.age[cl1mhfat80001w98k8vvkvxb9]" changed: Preprocessing failed for: [{"id":"cl2yl9hny00011r8kta7i7063","name":"node1","status":3,"lastaccess":1652093184,"address":"l...
      1. Failed: cannot extract value from json by path "$.[?(@.id=="cl1mhfat80001w98k8vvkvxb9")].lastaccess_age.first()": no data matches the specified path
      1356996:20220509:134625.878 error reason for "Zabbix server:zabbix.nodes.address[cl1mhfat80001w98k8vvkvxb9]" changed: Preprocessing failed for: [{"id":"cl2yl9hny00011r8kta7i7063","name":"node1","status":3,"lastaccess":1652093184,"address":"l...
      1. Failed: cannot extract value from json by path "$.[?(@.id=="cl1mhfat80001w98k8vvkvxb9")].address.first()": no data matches the specified path
      

      Reason for that is returned JSON on master internal item is this:

      [{"id":"cl2yl9hny00011r8kta7i7063","name":"node1","status":3,"lastaccess":1652092584,"address":"localhost:10051","db_timestamp":1652092585,"lastaccess_age":1}]
      

      i.e. on beginning, which is included to the not supported error message, it has a dynamic part - clock for "lastaccess". This causes that each time the item is still evaluated, it gets a new not supported error message, which has to be updated in DB.

      So, such messages in server log may be very unclear where they come from and why.

      Suggestion how to fix in template. For JSONPath preprocessing steps (4 prototypes) like:

      $.[?(@.id=="{#NODE.ID}")].address.first()
      

      enable option "Custom on fail" with "Set value to" and value like - "Node is not discoverable anymore".

      This way error message will be fixed and there should be no flapping in the log.

            zabbix.dev Zabbix Development Team
            zalex_ua Oleksii Zagorskyi
            Team INT
            Votes:
            3 Vote for this issue
            Watchers:
            5 Start watching this issue

              Created:
              Updated: