-
New Feature Request
-
Resolution: Fixed
-
Major
-
None
-
None
-
Team A
-
Sprint 93 (Oct 2022), Sprint 94 (Nov 2022), Sprint 95 (Dec 2022)
-
4
History syncer is frequently blocked by waiting on configuration cache to be free, see stats below, wait is waiting for mutex and busy is how long it hold copying data (customer case).
For example:
zbx_dc_get_actions_eval() locked:105940 busy:431.449521 wait:0.655284
It locked 105940 and was holding for 431 seconds, copying data, there are 40 history syncers.
For starters History syncer should cache action information locally as it copies it very frequently and there is no reason not to hold it in cache.
Second improvement:
DCconfig_get_items_by_itemids_partial() locked:1040567 busy:303.120468 wait:150.530272
Busy for 303 seconds and waiting for 150 seconds, items can be cached locally instead of copied over and over again, etc..
Notice total sync 25000 seconds waiting for locks + 2917 holding locks:
10975:20220919:110619.666 Time spent holding locks:2917.929536 Time spent waiting for locks:25080.168827
And inserting into database was 6000 seconds:
zbx_history_add_values() busy:6003.804196
10975:20220919:110619.666 Profiling information: dbsyncer_thread() busy:42088.771342 lock_log() locked:403352 busy:3.630379 wait:0.604905 DCupdate_trends() locked:525279 busy:1.684079 wait:6.668424 DCmass_update_trends() locked:525279 busy:76.840730 wait:18.116385 sync_server_history() locked:1345294 busy:247.778051 wait:695.617946 DCget_nextid() locked:216511 busy:0.065177 wait:0.066877 DCconfig_get_items_by_itemids_partial() locked:1040567 busy:303.120468 wait:150.530272 DCconfig_get_hosts_by_itemids() locked:59056 busy:0.166738 wait:1.445685 DCconfig_get_functions_by_functionids() locked:1112958 busy:153.790326 wait:32.234849 DCconfig_lock_triggers_by_history_items() locked:546465 busy:82.239748 wait:954.966807 DCconfig_unlock_triggers() locked:528445 busy:40.460696 wait:25.325980 DCconfig_get_triggers_by_itemids() locked:525279 busy:172.169006 wait:27.509390 zbx_dc_get_timer_triggers_by_triggerids() locked:30090 busy:34.771860 wait:28.682465 zbx_dc_get_timer_triggerids() locked:842988 busy:59.775450 wait:3070.352669 DCconfig_triggers_apply_changes() locked:104871 busy:0.475808 wait:282.188329 DCget_user_macro() locked:173 busy:0.000757 wait:0.022280 DCget_expressions_by_names() locked:20703734 busy:12.918647 wait:72.195879 DCget_data_expected_from() locked:1786810 busy:0.963294 wait:2.505167 DCget_hostids_by_functionids() locked:173 busy:0.000335 wait:0.005598 DCget_internal_action_count() locked:43081 busy:0.007122 wait:1.763838 zbx_dc_get_actions_eval() locked:105940 busy:431.449521 wait:0.655284 zbx_dc_correlation_rules_get() locked:92492 busy:0.016456 wait:0.354798 zbx_dc_get_nested_hostgroupids() locked:624393 busy:0.714630 wait:74.626875 DCconfig_items_apply_changes() locked:22533 busy:0.100342 wait:54.624219 DCconfig_update_inventory_values() locked:44112 busy:0.165998 wait:108.681454 zbx_dc_get_trigger_dependencies() locked:134454 busy:0.740951 wait:2.037974 zbx_dc_proxy_update_nodata() locked:513939 busy:3.472877 wait:492.421561 zbx_dc_get_item_tags_by_functionids() locked:490827 busy:0.830222 wait:7.494748 DCget_proxy_nodata_win() locked:17296173 busy:4.939415 wait:82.897655 zbx_dc_get_event_maintenances() locked:4221 busy:8.910705 wait:0.002908 zbx_dc_get_running_maintenanceids() locked:4221 busy:0.023354 wait:0.103875 vch_item_cache_values_by_time() locked:10946 busy:0.251167 wait:7.224583 vch_item_cache_values_by_time_and_count() locked:32624 busy:0.168397 wait:21.448718 zbx_vc_add_values() locked:524858 busy:526.054050 wait:4959.825501 zbx_vc_get_values() locked:201224415 busy:717.088898 wait:4162.228185 zbx_vc_flush_stats() locked:528258 busy:27.989302 wait:151.139879 zbx_history_add_values() busy:6003.804196 update_selfmon_counter() locked:377133 busy:0.183608 wait:0.416856 DBupdate_itservices() locked:8472 busy:3.970973 wait:9583.180008 10975:20220919:110619.666 Time spent holding locks:2917.929536 Time spent waiting for locks:25080.168827
