ZABBIX BUGS AND ISSUES
  1. ZABBIX BUGS AND ISSUES
  2. ZBX-7098

logrt possibly read old log file from very beginning after log rotation

    Details

      Description

      logrt possibly read old log file from very beginning after log rotation if new log and old log has same timestamp

      How to reproduce:

      • Add item: logrt[/tmp/^testlog.*$]
      • Write some text to the log file: echo `date` >> /tmp/testlog
      • Zabbix agent read log file and send contents to Zabbix server
      • Write text and rotate log file: echo `date` >> /tmp/testlog; mv /tmp/testlog /tmp/testlog.1; touch /tmp/testlog /tmp/testlog.1
      • Zabbix agent check log files, but doesn't send last added line
      • Write text to new testlog: echo `date` >> /tmp/testlog
      • Zabbix agent read from very beginning of testlog.1

      The procedure above is simulation of logrotate, but this problem happened with logrotated.

        Activity

        Hide
        Oleksiy Zagorskyi added a comment -

        See also ZBX-7301

        Show
        Oleksiy Zagorskyi added a comment - See also ZBX-7301
        Hide
        Andris Zeila added a comment - - edited

        There are two bugs that might/would happen if the last few log files have the same modification time:

        • the last few lines from the old log file might be ignored
        • the old file might get parsed again, resulting in duplicate item values

        Fixed in development branch svn://svn.zabbix.com/branches/dev/ZBX-7098

        Show
        Andris Zeila added a comment - - edited There are two bugs that might/would happen if the last few log files have the same modification time: the last few lines from the old log file might be ignored the old file might get parsed again, resulting in duplicate item values Fixed in development branch svn://svn.zabbix.com/branches/dev/ZBX-7098
        Hide
        Andris Mednis added a comment -

        (1) The fix has been developed for Zabbix 2.2 but 2.0 also must be fixed.

        Show
        Andris Mednis added a comment - (1) The fix has been developed for Zabbix 2.2 but 2.0 also must be fixed.
        Hide
        Andris Mednis added a comment -

        Before fixing ZBX-7098, it was decided to fix "log[]" and "logrt[]" performance issue ZBX-6731, ZBX-6729 and also ZBX-7740. Currently a fix for ZBX-6731 and ZBX-6729 is developed for Zabbix 2.0 (available in development branch svn://svn.zabbix.com/branches/dev/ZBX-6731) and is being ported to Zabbix 2.2. The performance fix is rather large change in log file processing in Zabbix agent. When completed, the work on ZBX-7098 will continue.

        Show
        Andris Mednis added a comment - Before fixing ZBX-7098 , it was decided to fix "log[]" and "logrt[]" performance issue ZBX-6731 , ZBX-6729 and also ZBX-7740 . Currently a fix for ZBX-6731 and ZBX-6729 is developed for Zabbix 2.0 (available in development branch svn://svn.zabbix.com/branches/dev/ZBX-6731) and is being ported to Zabbix 2.2. The performance fix is rather large change in log file processing in Zabbix agent. When completed, the work on ZBX-7098 will continue.
        Hide
        Andris Mednis added a comment -

        The solution turned out to be more complex, than expected.
        Currently it is being tested.

        Until now we used only file 'mtime' and 'lastlogsize' for following log file changes. We have added using inode numbers and MD5 sums of the first 512-bytes for deciding whether a log[] or logrt[] file checked during the previous check can be counted as the same file and can be checked from 'lastlogsize' position, or it is a different file and should be analyzed from the start.

        One consequence of the coming solution - do not modify 'mtime' of log files with 'touch', do not copy the log file with later restoration of the original name (it will change file's inode number). In both cases the file will be counted as different and will be analyzed from the start, which may result in duplicated alerts.

        On Microsoft Windows, if log files reside on NTFS or ReFS file systems, we have added 64-bit file indexes or 128-bit file IDs to track log file rotation. On file systems where file indexes change (e.g. FAT32) we have developed a fall-back algorithm to take a sensible approach in uncertain conditions.

        Show
        Andris Mednis added a comment - The solution turned out to be more complex, than expected. Currently it is being tested. Until now we used only file 'mtime' and 'lastlogsize' for following log file changes. We have added using inode numbers and MD5 sums of the first 512-bytes for deciding whether a log[] or logrt[] file checked during the previous check can be counted as the same file and can be checked from 'lastlogsize' position, or it is a different file and should be analyzed from the start. One consequence of the coming solution - do not modify 'mtime' of log files with 'touch', do not copy the log file with later restoration of the original name (it will change file's inode number). In both cases the file will be counted as different and will be analyzed from the start, which may result in duplicated alerts. On Microsoft Windows, if log files reside on NTFS or ReFS file systems, we have added 64-bit file indexes or 128-bit file IDs to track log file rotation. On file systems where file indexes change (e.g. FAT32) we have developed a fall-back algorithm to take a sensible approach in uncertain conditions.
        Hide
        Andris Mednis added a comment - - edited

        For Zabbix 2.0 available in development branch svn://svn.zabbix.com/branches/dev/ZBX-7098-20-1

        It is a significant change, need to be thoroughly tested on UNIX, GNU/Linux (with various file systems) and Microsoft Windows (including log files on NTFS, FAT32 (or eXFAT) and ReFS), including complicated log file rotations, truncations, multiple log files with the same modification time, even with identical log files... Try to break it in various ways.

        Andris Zeila successfully tested

        Show
        Andris Mednis added a comment - - edited For Zabbix 2.0 available in development branch svn://svn.zabbix.com/branches/dev/ZBX-7098-20-1 It is a significant change, need to be thoroughly tested on UNIX, GNU/Linux (with various file systems) and Microsoft Windows (including log files on NTFS, FAT32 (or eXFAT) and ReFS), including complicated log file rotations, truncations, multiple log files with the same modification time, even with identical log files... Try to break it in various ways. Andris Zeila successfully tested
        Hide
        Andris Mednis added a comment - - edited

        For Zabbix 2.2 available in development branch svn://svn.zabbix.com/branches/dev/ZBX-7098-22-1

        Andris Zeila successfully tested

        Show
        Andris Mednis added a comment - - edited For Zabbix 2.2 available in development branch svn://svn.zabbix.com/branches/dev/ZBX-7098-22-1 Andris Zeila successfully tested
        Hide
        Andris Mednis added a comment - - edited

        For trunk available in development branch svn://svn.zabbix.com/branches/dev/ZBX-7098-23

        Andris Zeila successfully tested

        Show
        Andris Mednis added a comment - - edited For trunk available in development branch svn://svn.zabbix.com/branches/dev/ZBX-7098-23 Andris Zeila successfully tested
        Hide
        Andris Zeila added a comment -

        Successfully tested!

        Show
        Andris Zeila added a comment - Successfully tested!
        Hide
        Andris Mednis added a comment -

        Fixed in versions pre-2.0.13 r45702, pre-2.2.4 r45706, pre-2.3.0 r45718.

        Show
        Andris Mednis added a comment - Fixed in versions pre-2.0.13 r45702, pre-2.2.4 r45706, pre-2.3.0 r45718.
        Show
        Andris Mednis added a comment - - edited Documented at https://www.zabbix.com/documentation/2.0/manual/config/items/itemtypes/log_items?&#important_notes https://www.zabbix.com/documentation/2.2/manual/config/items/itemtypes/log_items?&#important_notes https://www.zabbix.com/documentation/2.4/manual/config/items/itemtypes/log_items?&#important_notes https://www.zabbix.com/documentation/2.0/manual/introduction/whatsnew2013 https://www.zabbix.com/documentation/2.2/manual/introduction/whatsnew224 Andris Zeila technical review done. Martins Valkovskis Reviewed as well. Some small grammar points improved. CLOSED.
        Hide
        Andris Mednis added a comment -

        Solution for ZBX-7098 caused an issue ZBX-8269.

        Show
        Andris Mednis added a comment - Solution for ZBX-7098 caused an issue ZBX-8269 .

          People

          • Assignee:
            Unassigned
            Reporter:
            Kodai Terashima
          • Votes:
            0 Vote for this issue
            Watchers:
            3 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved: