[#ZBX-19365] Data is not updating to Devices and all option hangs, Zabbix server services required to restart manually every 3 hours.

Dear Team,

Seeking your help with the below issues.

We installed the Production Zabbix system and integrated around "1351" Nodes into them and we have to integrated 800 more devices into the production system. So In total, we will have above 2K integrated nodes.

Please let me know if any more information is required. Thank you.

System Information:

CPU : 8 CPU

RAM: 32 GB

DISK: 3 TB

OS: Red Hat Enterprise Linux release 8.3 (Ootpa)

Zabbix Version: 5.0

Issue: Data is not updating to Devices and all option hangs, Zabbix server services required to restart manually every 3 hours.

Troubleshooting:

As a part of troubleshooting, I have increased the "Start poller" to 375, and the connection to Mysql increase to =1000.
Poller probing in progress but no data updating.
Mysql optimizer_switch='index_condition_pushdown=off'; is switch off

Below cache configuration done in the Zabbix_server.conf

VMwareCacheSize=2048M
HousekeepingFrequency=1
CacheSize=8144M
CacheUpdateFrequency=60
HistoryCacheSize=1024M
HistoryIndexCacheSize=1024M
TrendCacheSize=512M
ValueCacheSize=1024M
Timeout=10

Mysql Configuration :

MariaDB [(none)]> Show variables like 'max_connections';                                                      ----------------------+

Variable_name   Value
----------------------+

max_connections 1000 
----------------------+
1 row in set (0.224 sec)
MariaDB [(none)]> SELECT @@optimizer_switch\G                                                                 *************************** 1. row ***************************
@@optimizer_switch: index_merge=on,index_merge_union=on,index_merge_sort_union=on,index_merge_intersection=on,index_merge_sort_intersection=off,engine_condition_pushdown=off,index_condition_pushdown=on,derived_merge=on,derived_with_keys=on,firstmatch=on,loosescan=on,materialization=on,in_to_exists=on,semijoin=on,partial_match_rowid_merge=on,partial_match_table_scan=on,subquery_cache=on,mrr=off,mrr_cost_based=off,mrr_sort_keys=off,outer_join_with_cache=on,semijoin_with_cache=on,join_cache_incremental=on,join_cache_hashed=on,join_cache_bka=on,optimize_join_buffer_size=off,table_elimination=on,extended_keys=on,exists_to_in=on,orderby_uses_equalities=on,condition_pushdown_for_derived=on,split_materialized=on
1 row in set (0.000 sec)
 
MariaDB [(none)]>
-----------------------------------+

database           size in GB     
-----------------------------------+

information_schema   0.000167846680         mysql                0.000945094042         performance_schema   0.000000000000         zabbix             67.724426269531
-----------------------------------+
4 rows in set (0.050 sec)
MariaDB [zabbix]>

[root@zabbix ~]# iostat 5 10
Linux 4.18.0-240.el8.x86_64 (zabbix.mobily.com.sa)      05/12/2021      x86_64        (8 CPU)
avg-cpu:  %user   %nice %system %iowait  %steal   %idle
           5.56    0.00    3.56   27.36    0.00   63.52
Device             tps    kB_read/s    kB_wrtn/s    kB_read    kB_wrtn
sda             932.92     16913.87     20996.90 3072589362 3814314887
scd0              0.00         0.14         0.00      25612          0
dm-0            935.99     16859.41     20764.38 3062694894 3772075564
dm-1             71.31        53.87       231.37    9785804   42030572
dm-2              0.03         0.38         1.14      68635     206575
 
avg-cpu:  %user   %nice %system %iowait  %steal   %idle
           0.33    0.00    0.07    1.12    0.00   98.47
Device             tps    kB_read/s    kB_wrtn/s    kB_read    kB_wrtn
sda              17.40       124.80        33.40        624        167
scd0              0.00         0.00         0.00          0          0
dm-0              3.60         4.80        33.40         24        167
dm-1             30.00       120.00         0.00        600          0
dm-2              0.00         0.00         0.00          0          0
avg-cpu:  %user   %nice %system %iowait  %steal   %idle
           6.10    0.00    0.68    2.23    0.00   91.00
Device             tps    kB_read/s    kB_wrtn/s    kB_read    kB_wrtn
sda              32.40       428.00        59.40       2140        297
scd0              0.00         0.00         0.00          0          0
dm-0              5.00         6.40        59.40         32        297
dm-1            105.40       421.60         0.00       2108          0
dm-2              0.00         0.00         0.00          0          0
[root@zabbix ~]# vmstat 3 5
procs ----------memory--------- --swap- ----io--- system- -----cpu----
r  b   swpd   free   buff  cache   si   so    bi    bo   in   cs us sy id wa st
0  0 3209508 10206468    148 15904656    7   29  2121  2633   14   25  6  4 64 27  0
0  0 3209508 10206216    148 15904728    0    0     0     0  164  278  0  0 100  0  0
0  0 3209508 10206248    148 15904728    0    0     0    30  175  296  0  0 100  0  0
1  0 3209508 10206020    148 15904492   84    0    84     4  199  310  0  0 99  0  0
2  0 3208484 10203100    148 15904632  400    0   400    49  869  805  4  0 94  1  0
[root@zabbix ~]# free -m
              total        used        free      shared  buff/cache   available
Mem:          31202        5744        9926        1583       15532       23467
Swap:         24575        3132       21443

[ZBX-19365] Data is not updating to Devices and all option hangs, Zabbix server services required to restart manually every 3 hours. Created: 2021 May 12 Updated: 2021 May 13 Resolved: 2021 May 13
Status:	Closed
Project:	ZABBIX BUGS AND ISSUES
Component/s:	Installation (I), Server (S)
Affects Version/s:	5.0.10, 5.0.11
Fix Version/s:	None

[ZBX-19365] Data is not updating to Devices and all option hangs, Zabbix server services required to restart manually every 3 hours. Created: 2021 May 12 Updated: 2021 May 13 Resolved: 2021 May 13