-
Type:
Incident report
-
Resolution: Unsupported version
-
Priority:
Critical
-
None
-
Affects Version/s: None
-
Component/s: Server (S)
-
None
-
Environment:# My configuration is:
- Zabbix Server version: 3.2.10
- Zabbix Server host: Virtual Machine running CentOS 7.3.1611 64 bits, 4 vCPs, 16 GB RAM
- Data Base: mysql Ver 15.1 Distrib 10.1.18-MariaDB, for Linux (x86_64) using readline 5.1. At the moment I´m with just one node.
- Data Base host: Physical Host running CentOS 7.4.1708 (Core), 32 CPUs, 87 GB RAM
- Several proxies version 3.2.10, on VMs CentOS 7.4.
- Partition tables enable on history and trend tables.
# Zabbix Serve Configuration (main parameters changed):
StartTrappers=100
MaxHousekeeperDelete=100000
CacheSize=1536M
CacheUpdateFrequency=300
StartDBSyncers=8 (should I increase it?)
HistoryCacheSize=512M (during the problem, it goes to zero)
TrendCacheSize=256M
ValueCacheSize=1G
Timeout=4
LogSlowQueries=10000
# MariaDB Configuration:
open_files_limit = 16364
max_connections = 500
binlog_format=ROW
default_storage_engine=InnoDB
innodb_autoinc_lock_mode=2
innodb_flush_log_at_trx_commit=0
innodb_autoextend_increment = 256
innodb_buffer_pool_instances = 32
innodb_buffer_pool_size = 32G
innodb_concurrency_tickets = 5000
innodb_file_per_table = 1
innodb_flush_method = O_DIRECT
innodb_log_file_size = 512M
innodb_log_files_in_group = 4
innodb_old_blocks_time = 1000
innodb_open_files = 2048
innodb_stats_on_metadata = OFF
innodb_lock_wait_timeout = 50
innodb_io_capacity = 2000
large-pages
binlog-row-event-max-size = 8192
character_set_server = utf8
collation_server = utf8_bin
expire_logs_days = 1
join_buffer_size = 262144
max_allowed_packet = 32M
max_connect_errors = 10000
max_heap_table_size = 134217728
query_cache_type = 0
query_cache_size = 0
slow-query-log = ON
table_open_cache = 2048
thread_cache_size = 64
tmp_table_size = 134217728
wait_timeout = 86400# My configuration is: - Zabbix Server version: 3.2.10 - Zabbix Server host: Virtual Machine running CentOS 7.3.1611 64 bits, 4 vCPs, 16 GB RAM - Data Base: mysql Ver 15.1 Distrib 10.1.18-MariaDB, for Linux (x86_64) using readline 5.1. At the moment I´m with just one node. - Data Base host: Physical Host running CentOS 7.4.1708 (Core), 32 CPUs, 87 GB RAM - Several proxies version 3.2.10, on VMs CentOS 7.4. - Partition tables enable on history and trend tables. # Zabbix Serve Configuration (main parameters changed): StartTrappers=100 MaxHousekeeperDelete=100000 CacheSize=1536M CacheUpdateFrequency=300 StartDBSyncers=8 (should I increase it?) HistoryCacheSize=512M (during the problem, it goes to zero) TrendCacheSize=256M ValueCacheSize=1G Timeout=4 LogSlowQueries=10000 # MariaDB Configuration: open_files_limit = 16364 max_connections = 500 binlog_format=ROW default_storage_engine=InnoDB innodb_autoinc_lock_mode=2 innodb_flush_log_at_trx_commit=0 innodb_autoextend_increment = 256 innodb_buffer_pool_instances = 32 innodb_buffer_pool_size = 32G innodb_concurrency_tickets = 5000 innodb_file_per_table = 1 innodb_flush_method = O_DIRECT innodb_log_file_size = 512M innodb_log_files_in_group = 4 innodb_old_blocks_time = 1000 innodb_open_files = 2048 innodb_stats_on_metadata = OFF innodb_lock_wait_timeout = 50 innodb_io_capacity = 2000 large-pages binlog-row-event-max-size = 8192 character_set_server = utf8 collation_server = utf8_bin expire_logs_days = 1 join_buffer_size = 262144 max_allowed_packet = 32M max_connect_errors = 10000 max_heap_table_size = 134217728 query_cache_type = 0 query_cache_size = 0 slow-query-log = ON table_open_cache = 2048 thread_cache_size = 64 tmp_table_size = 134217728 wait_timeout = 86400
I´ve tried to upgrade the Zabbix Server from 3.2.10 to 3.2.11, using the CentOS 7 repository, and I had a big problem.
The query below does not finish:
select min(clock) from events
where events.source=0 and events.object=0 and not exists
(select null from problem where events.eventid=problem.eventid or events.eventid=problem.r_eventid);
This makes the CPU of the DB server achieve 100% and the Zabbix becomes extremely slow.
I´ve tried even on a small environment, with 130 NVPS, and the problem is the same.
I know that this query is related with the housekeeper process and, as described at the release notes, this process was changed.
My solution was to make a downgrade to the 3.2.10 version, unfortunately.
I also know that the 3.2 releases was deprecated and I want to upgrade to the 3.4 release, but I´m afraid of having the same problem.