-
Incident report
-
Resolution: Commercial support required
-
Critical
-
None
-
3.2.10
-
# My configuration is:
- Zabbix Server version: 3.2.10
- Zabbix Server host: Virtual Machine running CentOS 7.3.1611 64 bits, 4 vCPs, 16 GB RAM
- Data Base: mysql Ver 15.1 Distrib 10.1.18-MariaDB, for Linux (x86_64) using readline 5.1. At the moment I´m with just one node.
- Data Base host: Physical Host running CentOS 7.4.1708 (Core), 32 CPUs, 87 GB RAM
- Several proxies version 3.2.10, on VMs CentOS 7.4.
- Partition tables enable on history and trend tables.
# Zabbix Serve Configuration (main parameters changed):
StartTrappers=100
MaxHousekeeperDelete=100000
CacheSize=1536M
CacheUpdateFrequency=300
StartDBSyncers=8 (should I increase it?)
HistoryCacheSize=512M (during the problem, it goes to zero)
TrendCacheSize=256M
ValueCacheSize=1G
Timeout=4
LogSlowQueries=10000
# MariaDB Configuration:
open_files_limit = 16364
max_connections = 500
binlog_format=ROW
default_storage_engine=InnoDB
innodb_autoinc_lock_mode=2
innodb_flush_log_at_trx_commit=0
innodb_additional_mem_pool_size = 16M
innodb_autoextend_increment = 256
innodb_buffer_pool_instances = 32
innodb_buffer_pool_size = 32G
innodb_change_buffer_max_size = 50
innodb_concurrency_tickets = 5000
innodb_file_per_table = 1
innodb_flush_method = O_DIRECT
innodb_log_file_size = 512M
innodb_log_files_in_group = 4
innodb_old_blocks_time = 1000
innodb_open_files = 2048
innodb_stats_on_metadata = OFF
innodb_lock_wait_timeout = 50
innodb_io_capacity = 2000
large-pages
binlog-row-event-max-size = 8192
character_set_server = utf8
collation_server = utf8_bin
expire_logs_days = 1
join_buffer_size = 262144
max_allowed_packet = 32M
max_connect_errors = 10000
max_heap_table_size = 134217728
query_cache_type = 0
query_cache_size = 0
slow-query-log = ON
table_open_cache = 2048
thread_cache_size = 64
tmp_table_size = 134217728
wait_timeout = 86400# My configuration is: - Zabbix Server version: 3.2.10 - Zabbix Server host: Virtual Machine running CentOS 7.3.1611 64 bits, 4 vCPs, 16 GB RAM - Data Base: mysql Ver 15.1 Distrib 10.1.18-MariaDB, for Linux (x86_64) using readline 5.1. At the moment I´m with just one node. - Data Base host: Physical Host running CentOS 7.4.1708 (Core), 32 CPUs, 87 GB RAM - Several proxies version 3.2.10, on VMs CentOS 7.4. - Partition tables enable on history and trend tables. # Zabbix Serve Configuration (main parameters changed): StartTrappers=100 MaxHousekeeperDelete=100000 CacheSize=1536M CacheUpdateFrequency=300 StartDBSyncers=8 (should I increase it?) HistoryCacheSize=512M (during the problem, it goes to zero) TrendCacheSize=256M ValueCacheSize=1G Timeout=4 LogSlowQueries=10000 # MariaDB Configuration: open_files_limit = 16364 max_connections = 500 binlog_format=ROW default_storage_engine=InnoDB innodb_autoinc_lock_mode=2 innodb_flush_log_at_trx_commit=0 innodb_additional_mem_pool_size = 16M innodb_autoextend_increment = 256 innodb_buffer_pool_instances = 32 innodb_buffer_pool_size = 32G innodb_change_buffer_max_size = 50 innodb_concurrency_tickets = 5000 innodb_file_per_table = 1 innodb_flush_method = O_DIRECT innodb_log_file_size = 512M innodb_log_files_in_group = 4 innodb_old_blocks_time = 1000 innodb_open_files = 2048 innodb_stats_on_metadata = OFF innodb_lock_wait_timeout = 50 innodb_io_capacity = 2000 large-pages binlog-row-event-max-size = 8192 character_set_server = utf8 collation_server = utf8_bin expire_logs_days = 1 join_buffer_size = 262144 max_allowed_packet = 32M max_connect_errors = 10000 max_heap_table_size = 134217728 query_cache_type = 0 query_cache_size = 0 slow-query-log = ON table_open_cache = 2048 thread_cache_size = 64 tmp_table_size = 134217728 wait_timeout = 86400
Hi,
I´m having huge performance problem with Zabbix Server since my NVPS has achieved the value of 3.500.
However the problem starts only at midnight, sometimes 21 h, and fineshes 1 or 2 hours later. Sometimes it lasts more too.
During this time, the items aren´t updated and some data is lost. The DBsyncers process stays very slow and
the CPU I/O wait increase very much in the DB Server. The Zabbix Dashboard doen´t work too.
At the DB server we can see a lot of queries like this one:
select clock,ns,value from history_uint where itemid=<id> and clock><end> and clock<=<begin>
During the day, the CPU utilization of the DB server is very low and everything works fine.
Has anyone any idea what is happening at this hour and what can be done do solve this problem?
Do you think that the problem is related with the DB Server or the Zabbix Server?
I think that it is related with the dbsyncer processes, but I´m not sure.
As my environment is very big, the debug level in the Zabbix Server is set to 2 (Error) and I couldn´t see anything.
Nothing on the /var/log/messagestoo, on the DB Server.
Thanks,
Rodney
