Improve housekeeper times by batching deletes for items with same retention time

XMLWordPrintable

    • Type: Change Request
    • Resolution: Fixed
    • Priority: Major
    • None
    • Affects Version/s: None
    • Component/s: Server (S)
    • None
    • Environment:
      Any

      Basically, housekeeper takes one delete per item. That's very slow way of doing it, and on large multigigabyte history tables it can be horrific in terms of housekeeping times and loads.

      On most systems, large groups of items have similar retention times for history/trends, i.e. 20000 interface items with retention time of 30 days.

      Housekeeper for surely can take advantage of this. I.e. like that:

      1. Get distinct different retention periods from item list (trivial).
      2. For each retention time, get all item IDs for items with similar retention period from step 1.
      3. Calculate deletion time condition (`clock` < time) which will be common for all items in question using retention period from step 1.
      4. Do DELETE FROM `history_xxx/trends_xxx` WHERE (`clock` < time) AND `itemid` IN (...), where ... is list of item IDs from step 2.

      If number of items in step 4 counts too many, they can be (configurably?) split into lesser amounts. I.e. for 10000 similar items with configured batch of 1000 in one go, it will take only 10 queries to do the actual deletes instead of 10000, giving drastic increase in speed.

            Assignee:
            Unassigned
            Reporter:
            Alexey Asemov
            Votes:
            2 Vote for this issue
            Watchers:
            3 Start watching this issue

              Created:
              Updated:
              Resolved: