Dear Team, 

Seeking your help with the below issues. 

We installed the Production Zabbix system and integrated around "1351" Nodes into them and we have to integrated 800 more devices into the production system. So In total, we will have above 2K integrated nodes.

Please let me know if any more information is required. Thank you.

System Information:

CPU : 8 CPU

RAM: 32 GB

DISK: 3 TB

OS: Red Hat Enterprise Linux release 8.3 (Ootpa)

Zabbix Version: 5.0

Issue:  Data is not updating to Devices and all option hangs, Zabbix server services required to restart manually every 3 hours.  

Troubleshooting:

• As a part of troubleshooting, I have increased the "Start poller" to 375, and the connection to Mysql increase to =1000.
• Poller probing in progress but no data updating.
• Mysql optimizer_switch='index_condition_pushdown=off'; is switch off

Below cache configuration done in the Zabbix_server.conf

VMwareCacheSize=2048M

HousekeepingFrequency=1

CacheSize=8144M

CacheUpdateFrequency=60

HistoryCacheSize=1024M

HistoryIndexCacheSize=1024M

TrendCacheSize=512M

ValueCacheSize=1024M

Timeout=10

Mysql Configuration :

MariaDB [(none)]> Show variables like 'max_connections';                                                      ----------------------+

----------------------+

----------------------+

1 row in set (0.224 sec)

MariaDB [(none)]>

MariaDB [(none)]> SELECT @@optimizer_switch\G                                                                 *************************** 1. row ***************************

@@optimizer_switch: index_merge=on,index_merge_union=on,index_merge_sort_union=on,index_merge_intersection=on,index_merge_sort_intersection=off,engine_condition_pushdown=off,index_condition_pushdown=on,derived_merge=on,derived_with_keys=on,firstmatch=on,loosescan=on,materialization=on,in_to_exists=on,semijoin=on,partial_match_rowid_merge=on,partial_match_table_scan=on,subquery_cache=on,mrr=off,mrr_cost_based=off,mrr_sort_keys=off,outer_join_with_cache=on,semijoin_with_cache=on,join_cache_incremental=on,join_cache_hashed=on,join_cache_bka=on,optimize_join_buffer_size=off,table_elimination=on,extended_keys=on,exists_to_in=on,orderby_uses_equalities=on,condition_pushdown_for_derived=on,split_materialized=on

1 row in set (0.000 sec)

MariaDB [(none)]>

-----------------------------------+

-----------------------------------+

-----------------------------------+

4 rows in set (0.050 sec)

MariaDB [zabbix]>

[root@zabbix ~]# iostat 5 10

Linux 4.18.0-240.el8.x86_64 (zabbix.mobily.com.sa)      05/12/2021      x86_64        (8 CPU)

avg-cpu:  %user   %nice %system %iowait  %steal   %idle

           5.56    0.00    3.56   27.36    0.00   63.52

Device             tps    kB_read/s    kB_wrtn/s    kB_read    kB_wrtn

sda             932.92     16913.87     20996.90 3072589362 3814314887

scd0              0.00         0.14         0.00      25612          0

dm-0            935.99     16859.41     20764.38 3062694894 3772075564

dm-1             71.31        53.87       231.37    9785804   42030572

dm-2              0.03         0.38         1.14      68635     206575

avg-cpu:  %user   %nice %system %iowait  %steal   %idle

           0.33    0.00    0.07    1.12    0.00   98.47

Device             tps    kB_read/s    kB_wrtn/s    kB_read    kB_wrtn

sda              17.40       124.80        33.40        624        167

scd0              0.00         0.00         0.00          0          0

dm-0              3.60         4.80        33.40         24        167

dm-1             30.00       120.00         0.00        600          0

dm-2              0.00         0.00         0.00          0          0

avg-cpu:  %user   %nice %system %iowait  %steal   %idle

           6.10    0.00    0.68    2.23    0.00   91.00

Device             tps    kB_read/s    kB_wrtn/s    kB_read    kB_wrtn

sda              32.40       428.00        59.40       2140        297

scd0              0.00         0.00         0.00          0          0

dm-0              5.00         6.40        59.40         32        297

dm-1            105.40       421.60         0.00       2108          0

dm-2              0.00         0.00         0.00          0          0

[root@zabbix ~]# vmstat 3 5

procs ----------memory--------- --swap- ----io--- system- -----cpu----

r  b   swpd   free   buff  cache   si   so    bi    bo   in   cs us sy id wa st

0  0 3209508 10206468    148 15904656    7   29  2121  2633   14   25  6  4 64 27  0

0  0 3209508 10206216    148 15904728    0    0     0     0  164  278  0  0 100  0  0

0  0 3209508 10206248    148 15904728    0    0     0    30  175  296  0  0 100  0  0

1  0 3209508 10206020    148 15904492   84    0    84     4  199  310  0  0 99  0  0

2  0 3208484 10203100    148 15904632  400    0   400    49  869  805  4  0 94  1  0

[root@zabbix ~]#

[root@zabbix ~]# free -m

              total        used        free      shared  buff/cache   available

Mem:          31202        5744        9926        1583       15532       23467

Swap:         24575        3132       21443

[root@zabbix ~]#

Attachements:

• Zabbix_server.conf

 ++ 

Snapshot

 ++ 

• Logs /var/log/Zabbix/Zabbix_server.log**

Zabbix Service and Poller information

 Best Regards

Satender Sharma