Uploaded image for project: 'ZABBIX BUGS AND ISSUES'
  1. ZABBIX BUGS AND ISSUES
  2. ZBX-23173

memory leak in Agent2 for perf_counter key, when "negative denominator" error happens

    XMLWordPrintable

Details

    • Team B
    • Sprint 103 (Aug 2023), Sprint 104 (Sep 2023), Sprint 105 (Oct 2023)
    • 1

    Description

      There were a few tickets when this error has been discussed:  ZBX-20269, ZBX-20356, ZBX-21661

      Key point of this report - memory leak on zabbix_agent2.exe process.

      Attaching a few graphs, where agent restarts happened.
      Timestamps are taken from agent2 log.

      2023/07/25 07:21:48.214367 Starting Zabbix Agent 2 (6.4.2)
      2023/07/25 12:39:38.655118 Starting Zabbix Agent 2 (6.4.2)
      2023/07/25 13:00:46.133411 Starting Zabbix Agent 2 (6.4.2)
      

      When restarting agent at 13:00:46 it started with DebugLevel=4, log available if needed.

      The "negative denominator" error has been logged only twice, but the memory leak on the graph is constant - increasing linearly.
      As agent2's log does not have PIDs in log headers, it's hard to track log messages in Debug, so here are just those lines:

      2023/07/25 13:13:28.006407 Detected performance counter with negative denominator, retrying in 1 second
      ...
      2023/07/25 13:13:29.006953 Detected performance counter with negative denominator the second time after retry, giving up...
      2023/07/25 13:13:29.009218 [Cpu] cannot obtain CPU#1 utilization counter value: Es wurde ein Leistungsindikator mit einem negativen Nenner gefunden.
      

       

      Attachments

        1. trends.png
          trends.png
          43 kB
        2. reproduce_leak1.diff
          1 kB
        3. leak1_fix.diff
          0.7 kB
        4. history-25-07.png
          history-25-07.png
          44 kB
        5. def.png
          def.png
          34 kB
        6. agent2log.txt
          5 kB

        Issue Links

          Activity

            People

              MVekslers Michael Veksler
              zalex_ua Oleksii Zagorskyi
              Votes:
              1 Vote for this issue
              Watchers:
              11 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: