Uploaded image for project: 'ZABBIX BUGS AND ISSUES'
  1. ZABBIX BUGS AND ISSUES
  2. ZBX-25543

Agent2 Crash During Log Monitoring Due to Regular Expression Processing (regexp_sub_ex2)

XMLWordPrintable

    • S24-W50/51/52
    • 1

      I’m currently investigating this crash issue. I received a report of Zabbix Agent2 crashing with backtrace records, as shown below.

      [2024-11-06 10:01:20.750] 2024/11/05 19:22:51.133195 process_logrt() old file list:
      [2024-11-06 10:01:20.750] 2024/11/05 19:22:51.133200    nr:0 filename:'/var/log/xxx/log_file.log' mtime:1730802120 size:1292897198 processed_size:1478396 seq:1 copy_of:-1 incomplete:0 dev:64768 ino_hi:0 ino_lo:134912620 md5_block_size:512 first_block_md5:9b3958b59a53233463e8099a801f6b1c last_block_offset:1292896686 last_block_md5:f7c07268b383f942a7cf32d89a1551b1
      [2024-11-06 10:01:20.750] 2024/11/05 19:22:51.133206 process_logrt() new file list: (mtime:1730802120 lastlogsize:1478396 start_idx:0)
      [2024-11-06 10:01:20.750] 2024/11/05 19:22:51.133212    nr:0 filename:'/var/log/xxx/log_file.log' mtime:1730802120 size:1292897198 processed_size:1478396 seq:0 copy_of:-1 incomplete:0 dev:64768 ino_hi:0 ino_lo:134912620 md5_block_size:512 first_block_md5:9b3958b59a53233463e8099a801f6b1c last_block_offset:1292896686 last_block_md5:f7c07268b383f942a7cf32d89a1551b1
      [2024-11-06 10:01:20.761] 2024/11/05 19:22:51.133216 In process_log() filename:'/var/log/xxx/log_file.log' lastlogsize:1478396 mtime:1730802120 seek_offset:1478396
      [2024-11-06 10:01:20.761] 2024/11/05 19:22:51.133358 itemid 115537: regexp runtime error: Error occurred while matching regular expression: UTF-8 error: isolated byte with 0x80 bit set
      [2024-11-06 10:01:20.761] 2024/11/05 19:22:51.134464 9: /usr/sbin/zabbix_agent2(zbx_backtrace+0x3c) [0xb22eec]
      [2024-11-06 10:01:20.761] 2024/11/05 19:22:51.134478 8: /usr/sbin/zabbix_agent2(zbx_log_fatal_info+0x475) [0xb23485]
      [2024-11-06 10:01:20.761] 2024/11/05 19:22:51.134482 7: /usr/sbin/zabbix_agent2() [0x9c6ad6]
      [2024-11-06 10:01:20.761] 2024/11/05 19:22:51.134489 6: /lib64/libpthread.so.0(+0x12ce0) [0x7f5150516ce0]
      [2024-11-06 10:01:20.761] 2024/11/05 19:22:51.134495 5: /lib64/libc.so.6(+0xc818e) [0x7f514fa8a18e]
      [2024-11-06 10:01:20.761] 2024/11/05 19:22:51.134498 4: /usr/sbin/zabbix_agent2(regexp_sub_ex2+0x86) [0xb29e56]
      [2024-11-06 10:01:20.761] 2024/11/05 19:22:51.134501 3: /usr/sbin/zabbix_agent2(process_log_check+0x17ad) [0xb3c7bd]
      [2024-11-06 10:01:20.773] 2024/11/05 19:22:51.134504 2: /usr/sbin/zabbix_agent2() [0x9c6bdc]
      [2024-11-06 10:01:20.773] 2024/11/05 19:22:51.134508 1: /usr/sbin/zabbix_agent2(_cgo_6e671d72c461_Cfunc_invoke_process_log_check+0x6c) [0x9c6d1e]
      [2024-11-06 10:01:20.773] 2024/11/05 19:22:51.134511 0: /usr/sbin/zabbix_agent2() [0x48e0e8]
      

      The information below indicates that the issue occurred during log and regular expression matching:

      regexp runtime error: Error occurred while matching regular expression: UTF-8 error: isolated byte with 0x80 bit set
      
      4: /usr/sbin/zabbix_agent2(regexp_sub_ex2+0x86) [0xb29e56]
      3: /usr/sbin/zabbix_agent2(process_log_check+0x17ad) [0xb3c7bd]
      

      Zabbix Agent2 version was 6.0.27

      $ zabbix_agent2 -V
      zabbix_agent2 (Zabbix) 6.0.27
      

      I expected ZBX-22871 to fix this issue, but 6.0.27 was expected to be a fixed version. and the crash still occurred even after upgrading from version 6.0.27 to 6.0.35.

      We are investigating further, including the log content and regular expressions.

            dgoloscapov Dmitrijs Goloscapovs
            JKKim Kim Jongkwon
            Team A
            Votes:
            0 Vote for this issue
            Watchers:
            3 Start watching this issue

              Created:
              Updated: