Uploaded image for project: 'ZABBIX BUGS AND ISSUES'
  1. ZABBIX BUGS AND ISSUES
  2. ZBX-25543

Agent2 Crash During Log Monitoring Due to Regular Expression Processing (regexp_sub_ex2)

XMLWordPrintable

    • S24-W50/51/52/1, S25-W2/3, S25-W4/5, S25-W6/7
    • 1

      I’m currently investigating this crash issue. I received a report of Zabbix Agent2 crashing with backtrace records, as shown below.

      [2024-11-06 10:01:20.750] 2024/11/05 19:22:51.133195 process_logrt() old file list:
      [2024-11-06 10:01:20.750] 2024/11/05 19:22:51.133200    nr:0 filename:'/var/log/xxx/log_file.log' mtime:1730802120 size:1292897198 processed_size:1478396 seq:1 copy_of:-1 incomplete:0 dev:64768 ino_hi:0 ino_lo:134912620 md5_block_size:512 first_block_md5:9b3958b59a53233463e8099a801f6b1c last_block_offset:1292896686 last_block_md5:f7c07268b383f942a7cf32d89a1551b1
      [2024-11-06 10:01:20.750] 2024/11/05 19:22:51.133206 process_logrt() new file list: (mtime:1730802120 lastlogsize:1478396 start_idx:0)
      [2024-11-06 10:01:20.750] 2024/11/05 19:22:51.133212    nr:0 filename:'/var/log/xxx/log_file.log' mtime:1730802120 size:1292897198 processed_size:1478396 seq:0 copy_of:-1 incomplete:0 dev:64768 ino_hi:0 ino_lo:134912620 md5_block_size:512 first_block_md5:9b3958b59a53233463e8099a801f6b1c last_block_offset:1292896686 last_block_md5:f7c07268b383f942a7cf32d89a1551b1
      [2024-11-06 10:01:20.761] 2024/11/05 19:22:51.133216 In process_log() filename:'/var/log/xxx/log_file.log' lastlogsize:1478396 mtime:1730802120 seek_offset:1478396
      [2024-11-06 10:01:20.761] 2024/11/05 19:22:51.133358 itemid 115537: regexp runtime error: Error occurred while matching regular expression: UTF-8 error: isolated byte with 0x80 bit set
      [2024-11-06 10:01:20.761] 2024/11/05 19:22:51.134464 9: /usr/sbin/zabbix_agent2(zbx_backtrace+0x3c) [0xb22eec]
      [2024-11-06 10:01:20.761] 2024/11/05 19:22:51.134478 8: /usr/sbin/zabbix_agent2(zbx_log_fatal_info+0x475) [0xb23485]
      [2024-11-06 10:01:20.761] 2024/11/05 19:22:51.134482 7: /usr/sbin/zabbix_agent2() [0x9c6ad6]
      [2024-11-06 10:01:20.761] 2024/11/05 19:22:51.134489 6: /lib64/libpthread.so.0(+0x12ce0) [0x7f5150516ce0]
      [2024-11-06 10:01:20.761] 2024/11/05 19:22:51.134495 5: /lib64/libc.so.6(+0xc818e) [0x7f514fa8a18e]
      [2024-11-06 10:01:20.761] 2024/11/05 19:22:51.134498 4: /usr/sbin/zabbix_agent2(regexp_sub_ex2+0x86) [0xb29e56]
      [2024-11-06 10:01:20.761] 2024/11/05 19:22:51.134501 3: /usr/sbin/zabbix_agent2(process_log_check+0x17ad) [0xb3c7bd]
      [2024-11-06 10:01:20.773] 2024/11/05 19:22:51.134504 2: /usr/sbin/zabbix_agent2() [0x9c6bdc]
      [2024-11-06 10:01:20.773] 2024/11/05 19:22:51.134508 1: /usr/sbin/zabbix_agent2(_cgo_6e671d72c461_Cfunc_invoke_process_log_check+0x6c) [0x9c6d1e]
      [2024-11-06 10:01:20.773] 2024/11/05 19:22:51.134511 0: /usr/sbin/zabbix_agent2() [0x48e0e8]
      

      The information below indicates that the issue occurred during log and regular expression matching:

      regexp runtime error: Error occurred while matching regular expression: UTF-8 error: isolated byte with 0x80 bit set
      

      ZBX-25805 has been created for it.

      Zabbix Agent2 version was 6.0.27

      $ zabbix_agent2 -V
      zabbix_agent2 (Zabbix) 6.0.27
      

      I expected ZBX-22871 to fix this issue, but 6.0.27 was expected to be a fixed version. and the crash still occurred even after upgrading from version 6.0.27 to 6.0.35.

      We are investigating further, including the log content and regular expressions.

        1. 0001-.G.-ZBX-25543-added-sanitizing-of-invalid-UTF-8-when.patch
          2 kB
          Dmitrijs Goloscapovs
        2. 0001-.G.-ZBX-25543-test-added-logging.patch
          8 kB
          Dmitrijs Goloscapovs
        3. agent2_patched.log
          64 kB
          Kim Jongkwon

            vso Vladislavs Sokurenko
            JKKim Kim Jongkwon
            Team A
            Votes:
            1 Vote for this issue
            Watchers:
            6 Start watching this issue

              Created:
              Updated:
              Resolved:

                Estimated:
                Original Estimate - Not Specified
                Not Specified
                Remaining:
                Remaining Estimate - Not Specified
                Not Specified
                Logged:
                Time Spent - 11.5h
                11.5h