Uploaded image for project: 'ZABBIX FEATURE REQUESTS'
  1. ZABBIX FEATURE REQUESTS
  2. ZBXNEXT-8058

History syncer is frequently blocked by waiting on configuration cache to be free

    XMLWordPrintable

Details

    • Team A
    • Sprint 93 (Oct 2022), Sprint 94 (Nov 2022), Sprint 95 (Dec 2022)
    • 4

    Description

      History syncer is frequently blocked by waiting on configuration cache to be free, see stats below, wait is waiting for mutex and busy is how long it hold copying data (customer case).

      For example:
      zbx_dc_get_actions_eval() locked:105940 busy:431.449521 wait:0.655284

      It locked 105940 and was holding for 431 seconds, copying data, there are 40 history syncers.
      For starters History syncer should cache action information locally as it copies it very frequently and there is no reason not to hold it in cache.

      Second improvement:
      DCconfig_get_items_by_itemids_partial() locked:1040567 busy:303.120468 wait:150.530272

      Busy for 303 seconds and waiting for 150 seconds, items can be cached locally instead of copied over and over again, etc..

      Notice total sync 25000 seconds waiting for locks + 2917 holding locks:
      10975:20220919:110619.666 Time spent holding locks:2917.929536 Time spent waiting for locks:25080.168827
      And inserting into database was 6000 seconds:
      zbx_history_add_values() busy:6003.804196

       10975:20220919:110619.666 Profiling information: dbsyncer_thread() busy:42088.771342
      lock_log() locked:403352 busy:3.630379 wait:0.604905
      DCupdate_trends() locked:525279 busy:1.684079 wait:6.668424
      DCmass_update_trends() locked:525279 busy:76.840730 wait:18.116385
      sync_server_history() locked:1345294 busy:247.778051 wait:695.617946
      DCget_nextid() locked:216511 busy:0.065177 wait:0.066877
      DCconfig_get_items_by_itemids_partial() locked:1040567 busy:303.120468 wait:150.530272
      DCconfig_get_hosts_by_itemids() locked:59056 busy:0.166738 wait:1.445685
      DCconfig_get_functions_by_functionids() locked:1112958 busy:153.790326 wait:32.234849
      DCconfig_lock_triggers_by_history_items() locked:546465 busy:82.239748 wait:954.966807
      DCconfig_unlock_triggers() locked:528445 busy:40.460696 wait:25.325980
      DCconfig_get_triggers_by_itemids() locked:525279 busy:172.169006 wait:27.509390
      zbx_dc_get_timer_triggers_by_triggerids() locked:30090 busy:34.771860 wait:28.682465
      zbx_dc_get_timer_triggerids() locked:842988 busy:59.775450 wait:3070.352669
      DCconfig_triggers_apply_changes() locked:104871 busy:0.475808 wait:282.188329
      DCget_user_macro() locked:173 busy:0.000757 wait:0.022280
      DCget_expressions_by_names() locked:20703734 busy:12.918647 wait:72.195879
      DCget_data_expected_from() locked:1786810 busy:0.963294 wait:2.505167
      DCget_hostids_by_functionids() locked:173 busy:0.000335 wait:0.005598
      DCget_internal_action_count() locked:43081 busy:0.007122 wait:1.763838
      zbx_dc_get_actions_eval() locked:105940 busy:431.449521 wait:0.655284
      zbx_dc_correlation_rules_get() locked:92492 busy:0.016456 wait:0.354798
      zbx_dc_get_nested_hostgroupids() locked:624393 busy:0.714630 wait:74.626875
      DCconfig_items_apply_changes() locked:22533 busy:0.100342 wait:54.624219
      DCconfig_update_inventory_values() locked:44112 busy:0.165998 wait:108.681454
      zbx_dc_get_trigger_dependencies() locked:134454 busy:0.740951 wait:2.037974
      zbx_dc_proxy_update_nodata() locked:513939 busy:3.472877 wait:492.421561
      zbx_dc_get_item_tags_by_functionids() locked:490827 busy:0.830222 wait:7.494748
      DCget_proxy_nodata_win() locked:17296173 busy:4.939415 wait:82.897655
      zbx_dc_get_event_maintenances() locked:4221 busy:8.910705 wait:0.002908
      zbx_dc_get_running_maintenanceids() locked:4221 busy:0.023354 wait:0.103875
      vch_item_cache_values_by_time() locked:10946 busy:0.251167 wait:7.224583
      vch_item_cache_values_by_time_and_count() locked:32624 busy:0.168397 wait:21.448718
      zbx_vc_add_values() locked:524858 busy:526.054050 wait:4959.825501
      zbx_vc_get_values() locked:201224415 busy:717.088898 wait:4162.228185
      zbx_vc_flush_stats() locked:528258 busy:27.989302 wait:151.139879
      zbx_history_add_values() busy:6003.804196
      update_selfmon_counter() locked:377133 busy:0.183608 wait:0.416856
      DBupdate_itservices() locked:8472 busy:3.970973 wait:9583.180008
      10975:20220919:110619.666 Time spent holding locks:2917.929536 Time spent waiting for locks:25080.168827

       

      Attachments

        Issue Links

          Activity

            People

              vso Vladislavs Sokurenko
              palivoda Rostislav Palivoda
              Votes:
              0 Vote for this issue
              Watchers:
              9 Start watching this issue

              Dates

                Created:
                Updated: