ExportXMLWordPrintable

    • Type: Problem report
    • Resolution: Unresolved
    • Priority: Trivial
    • None
    • Affects Version/s: 8.0.0
    • None
    • Environment:

      Steps to reproduce:

      1. Run the official web image zabbix/zabbix-web-apache-mysql:ubuntu-8.0.0 (the same happens with zabbix-web-nginx-pgsql:ubuntu-8.0.0) with the default PHP_FPM_PM_* settings.
      2. Open Monitoring -> Hosts -> Graphs for a host with several graphs and leave the page open with auto-refresh. You can also request chart.php / chart2.php in a loop.
      3. Watch the RSS of the php-fpm: pool zabbix workers, or the memory of the web container.

      Result:

      • The RSS of the php-fpm workers keeps growing and is never released.
      • In production, with about 3,500 graph requests per hour per frontend node, the web container grew by about 95 MiB/h: from 150 MiB to 3 GiB in 2.5 days, until RAM and swap ran out.
      • Each worker reached 240+ MB after about 19,000 requests. All workers were still alive since the container start.
      • Almost all of this memory is in [heap] (glibc malloc), not in the Zend memory manager. The php-fpm status page shows "last request memory" of only 4-8 MB.
      • The heap is full of small 32-80 byte blocks with glyph data.

      Root cause:

      • The leak is in FreeType 2.14.2, the Ubuntu 26.04 package libfreetype6 2.14.2+dfsg-1ubuntu0.2. libgd uses it in imagettftext(), which draws all text on graphs.
      • It leaks about 46 bytes for every glyph drawn, whatever the font: NotoSansCJKjp-Regular.otf (the 8.0 graph font) and DejaVuSans.ttf give the same result.
      • Isolated test: PHP CLI inside the official images, 20,000 imagettftext() calls with the 9-character string "12.5 MB/s". The leak per call is the RSS growth from call 5,000 to call 25,000, divided by 20,000.
      Image / library set Leak per call
      ubuntu-7.0.12 (Ubuntu 24.04, PHP 8.3.6, libgd 2.3.3-9ubuntu5, FreeType 2.13.2) ~12 B (no leak)
      ubuntu-8.0.0 (Ubuntu 26.04, PHP 8.5.4, libgd 2.3.3-13ubuntu2, FreeType 2.14.2+dfsg-1ubuntu0.2) 413 B
      alpine-8.0.0 (PHP 8.5.13, bundled gd, FreeType 2.14.3) ~14 B (no leak)
      ubuntu-8.0.0 + libgd 2.3.3-9ubuntu5 from Ubuntu 24.04 413 B (libgd is not the cause)
      ubuntu-8.0.0 + FreeType 2.13.2 (via LD_LIBRARY_PATH) 11 B
      ubuntu-8.0.0 + FreeType 2.14.2+dfsg-1 (Ubuntu release build, without the security patches) 416 B
      ubuntu-8.0.0 + FreeType 2.15.0+dfsg-1 (Debian sid) 11 B

      Why it accumulates:

      • The web images default to PHP_FPM_PM_MAX_REQUESTS=0, so php-fpm workers never restart.
      • The leaked memory therefore stays allocated for the whole life of the container.
      • Ubuntu 26.04 has no newer libfreetype6 yet, and the ubuntu-8.0-latest tag has the same package.

      Workaround (verified):
      PHP_FPM_PM_MAX_REQUESTS=500 for the web container. On a test stand (MySQL 8.4 + web ubuntu-8.0.0, a page that draws 60 graph labels per request, 6 workers):

      Requests php-fpm total RSS, default php-fpm total RSS, MAX_REQUESTS=500
      2,000 148 MiB 148 MiB
      10,000 250 MiB 131 MiB

      Without the setting, memory grows in a straight line (+12.8 MiB per 1,000 requests). With it, memory stays flat. In production, both frontend nodes now run with this setting.

      Expected:

      • The ubuntu web images are rebuilt with a FreeType that does not leak, once Ubuntu ships one.
      • Until then, a non-zero default for PHP_FPM_PM_MAX_REQUESTS in the web images, or an entry in "Known issues -> Graphs". ZBX-25191 got such an entry for a libgd problem.

      Reproduction script (run with php inside the ubuntu-8.0.0 web image):

      <?php
      function rss() { preg_match('/VmRSS:\s+(\d+)/', file_get_contents('/proc/self/status'), $m); return (int)$m[1]; }
      $font = '/usr/share/zabbix/assets/fonts/NotoSansCJKjp-Regular.otf';
      for ($i = 1; $i <= 25000; $i++) {
          $im = imagecreatetruecolor(800, 200);
          imagettftext($im, 7, 0, 10, 20, 0x333333, $font, '12.5 MB/s');
          imagedestroy($im);
          if ($i == 5000) $a = rss();
      }
      printf("leak = %.0f B per call\n", (rss() - $a) * 1024 / 20000);
      

      Swap only the library to compare, for example: LD_LIBRARY_PATH=/path/to/freetype-2.13.2 php leak.php.

      See also ZBX-25191 (an earlier graph problem caused by a system graphics library, documented in Known issues).

            Assignee:
            Zabbix Support Team
            Reporter:
            Vladimir
            Votes:
            0 Vote for this issue
            Watchers:
            1 Start watching this issue

              Created:
              Updated: