[ZBX-16070] Zabbix + Elasticsearch stopped to write trends after upgrade to 4.0.7 Created: 2019 May 02  Updated: 2024 Apr 10  Resolved: 2019 May 19

Status: Closed
Project: ZABBIX BUGS AND ISSUES
Component/s: Documentation (D)
Affects Version/s: 4.0.7
Fix Version/s: 4.4 (plan)

Type: Documentation task Priority: Critical
Reporter: Oleg Morozov Assignee: Martins Valkovskis
Resolution: Fixed Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Attachments: PNG File trend.png    
Issue Links:
Sub-task
depends on ZBX-15160 There is no access to trends if you a... Closed
Team: Team D
Sprint: Sprint 52 (May 2019)
Story Points: 0

 Description   

Steps to reproduce:

  1. Configure to store history in elasticsearch (trends in MySQL)
  2. Upgrade (install) to 4.0.7

Result:

mysql> select * from trends order by clock desc limit 1;

+--------+------------+-----+-----------+-----------+-----------+
| itemid | clock      | num | value_min | value_avg | value_max |
+--------+------------+-----+-----------+-----------+-----------+
|    818 | 1556593200 |  60 | 0.0062    | 0.0062    | 0.0062    |
+--------+------------+-----+-----------+-----------+-----------+

The day I upgraded server ^^^
Current timestamp:

$ date +%s
1556801027

Expected:
Zabbix writes trend values to MySQL.

 

We're using Zabbix + Elasticsearch since last summer everything with trends was fine (except zabbix web ui cannot show trends with ES enabled  See: https://support.zabbix.com/browse/ZBX-15160). We used Grafana to get trend values via Zabbix API. Upgraded server to fix memory leak (https://support.zabbix.com/browse/ZBX-15774) and no new trend values since upgrade. This issue could be related.

 

Server configuration:
LogFile=/var/log/zabbix/zabbix_server.log
LogFileSize=0
PidFile=/var/run/zabbix/zabbix_server.pid
SocketDir=/var/run/zabbix
DBHost=127.0.0.1
DBName=zabbix
DBUser=zabbix
DBPassword=***
DBPort=7001
HistoryStorageURL=http://localhost:9200HistoryStorageDateIndex=1
StartPollers=4
StartTrappers=2
SNMPTrapperFile=/var/log/snmptrap/snmptrap.log
MaxHousekeeperDelete=100000
CacheSize=2G
StartDBSyncers=16
HistoryCacheSize=2G
HistoryIndexCacheSize=256M
TrendCacheSize=128M
ValueCacheSize=6G
Timeout=5
AlertScriptsPath=/usr/lib/zabbix/alertscripts
ExternalScripts=/usr/lib/zabbix/externalscripts
FpingLocation=/usr/bin/fping
Fping6Location=/usr/bin/fping6
LogSlowQueries=5000
ProxyConfigFrequency=60



 Comments   
Comment by Edgar Akhmetshin [ 2019 May 02 ]

Hello Oleg,

Thank you for reporting the issue! Could you increase history syncer log level before trends calculation (every hour) and add debug log to the issue?

zabbix_server -R log_level_increase="history syncer"

Regards,
Edgar

Comment by Edgar Akhmetshin [ 2019 May 02 ]

Hello Oleg,

I'm sorry for the mistake with debug information. Trends are NOT supported/calculated when using Elasticsearch. The thing is, such behaviour is not explicitly described in our official documentation.

Regards,
Edgar

Comment by Oleg Morozov [ 2019 May 02 ]

Edgar, thends worked till 4.0.7 upgrade, we used Zabbix + ES for almost one year.

 

What we should do now if we cannot change history backend in a moment? Rollback to 4.0.4 and restart server every month because of memory leak of stay at 4.0.7 without trends at all?

Comment by Vjaceslavs Bogdanovs [ 2019 May 09 ]

Well, the only option I see is that not all history was stored in Elastic. For example, uints could be stored in SQL, while other types were stored in Elastic. (if we assume that Zabbix was properly configured and restarted after changes to configuration were made).

As for the Elasticsearch implementation - there are no trends. And trends were never calculated for data stored in Elasticsearch.

Comment by Oleg Morozov [ 2019 May 09 ]

Vjaceslavs,

 

As I wrote before, till 4.0.4 trends WERE calulated and stored in MySQL, while history data were stored in Elasticsearch.

We are using Elasticsearch to store all history data sinse last summer and we have trends in MySQL for this period.

For example, the item with 7d history and 6m+ trend data:

Once again: Zabbix, configured to store history data in Elasticsearch, calculated and stored trend data in MySQL. In versions up to 4.0.4.

Comment by Martins Valkovskis [ 2019 May 13 ]

Updated documentation: 4.0, 4.2, 4.4 (the note on trends not being calculated with history in ES)

Comment by Glebs Ivanovskis [ 2019 May 13 ]

Perhaps, trend calculation for Elasticsearch was a "bug" fixed in ZBX-15774?

Morozov, try to increase logging level for housekeeper process.

Generated at Sun Apr 06 18:09:24 EEST 2025 using Jira 9.12.4#9120004-sha1:625303b708afdb767e17cb2838290c41888e9ff0.