-
Problem report
-
Resolution: Fixed
-
Blocker
-
6.0.8
-
None
-
Team A
-
Sprint 93 (Oct 2022), Sprint 94 (Nov 2022), Sprint 95 (Dec 2022), Sprint 96 (Jan 2023)
-
1
An installation (using Oracle DB) has been upgraded from 6.0.1 to 6.0.8 and an error started to be logged after a few minutes:
41571:20220926:200340.883 Zabbix Server stopped. Zabbix 6.0.1 (revision a80cb13868). 44422:20220926:200809.198 Starting Zabbix Server. Zabbix 6.0.8 (revision c7c3044a4a2). 44629:20220926:200927.595 server #27 started [task manager #1] -- 44641:20220926:201107.513 sending configuration data to proxy .... zabbix_server [44629]: ERROR [file and function: <events.c,update_trigger_problem_count>, revision:c7c3044a4a2, line:1562] Something impossible has just happened. 44629:20220926:201150.849 === Backtrace: === 44629:20220926:201150.849 11: /data/zabbix/sbin/zabbix_server: task manager [processing tasks](zbx_backtrace+0x35) [0x596635] 44629:20220926:201150.850 10: /data/zabbix/sbin/zabbix_server: task manager [processing tasks]() [0x51c956] 44629:20220926:201150.850 9: /data/zabbix/sbin/zabbix_server: task manager [processing tasks](zbx_close_problem+0x14a) [0x520bda] 44629:20220926:201150.850 8: /data/zabbix/sbin/zabbix_server: task manager [processing tasks]() [0x4a06da] 44629:20220926:201150.850 7: /data/zabbix/sbin/zabbix_server: task manager [processing tasks](taskmanager_thread+0x195) [0x4a0da5] 44629:20220926:201150.850 6: /data/zabbix/sbin/zabbix_server: task manager [processing tasks](zbx_thread_start+0x3e) [0x5a190e] 44629:20220926:201150.850 5: /data/zabbix/sbin/zabbix_server: task manager [processing tasks]() [0x44acda] 44629:20220926:201150.850 4: /data/zabbix/sbin/zabbix_server: task manager [processing tasks](MAIN_ZABBIX_ENTRY+0x9f6) [0x44bb56] 44629:20220926:201150.850 3: /data/zabbix/sbin/zabbix_server: task manager [processing tasks](daemon_start+0x1b2) [0x5963f2] 44629:20220926:201150.850 2: /data/zabbix/sbin/zabbix_server: task manager [processing tasks](main+0x4dc) [0x449c7c] 44629:20220926:201150.850 1: /lib64/libc.so.6(__libc_start_main+0xf5) [0x7fd258249555] 44629:20220926:201150.850 0: /data/zabbix/sbin/zabbix_server: task manager [processing tasks]() [0x44a06a] zabbix_server [44629]: ERROR [file and function: <events.c,update_trigger_problem_count>, revision:c7c3044a4a2, line:1562] Something impossible has just happened. 44629:20220926:201200.891 === Backtrace: === 44629:20220926:201200.891 11: /data/zabbix/sbin/zabbix_server: task manager [processing tasks](zbx_backtrace+0x35) [0x596635] 44629:20220926:201200.891 10: /data/zabbix/sbin/zabbix_server: task manager [processing tasks]() [0x51c956] 44629:20220926:201200.891 9: /data/zabbix/sbin/zabbix_server: task manager [processing tasks](zbx_close_problem+0x14a) [0x520bda] 44629:20220926:201200.891 8: /data/zabbix/sbin/zabbix_server: task manager [processing tasks]() [0x4a06da] 44629:20220926:201200.891 7: /data/zabbix/sbin/zabbix_server: task manager [processing tasks](taskmanager_thread+0x195) [0x4a0da5] 44629:20220926:201200.891 6: /data/zabbix/sbin/zabbix_server: task manager [processing tasks](zbx_thread_start+0x3e) [0x5a190e] 44629:20220926:201200.891 5: /data/zabbix/sbin/zabbix_server: task manager [processing tasks]() [0x44acda] 44629:20220926:201200.891 4: /data/zabbix/sbin/zabbix_server: task manager [processing tasks](MAIN_ZABBIX_ENTRY+0x9f6) [0x44bb56] 44629:20220926:201200.891 3: /data/zabbix/sbin/zabbix_server: task manager [processing tasks](daemon_start+0x1b2) [0x5963f2] 44629:20220926:201200.891 2: /data/zabbix/sbin/zabbix_server: task manager [processing tasks](main+0x4dc) [0x449c7c] 44629:20220926:201200.891 1: /lib64/libc.so.6(__libc_start_main+0xf5) [0x7fd258249555] 44629:20220926:201200.891 0: /data/zabbix/sbin/zabbix_server: task manager [processing tasks]() [0x44a06a]
The error repeated ~100 times per a day.
Debug level for task manager was increased and here are 2 files where we see when it happens. I've added empty lines just to split blocks/function call, all lines are present and in original order.
Its known that there is some scripts that close certain problems, including the *** problem shown in the logs. They do an event.acknowledge API call.
My findings:
in each file (in each cycle) task manager tries to close 2 or 3 tasks, for the same trigger. 1st (and next one) task/attempt of closing is failed with the error, while last one is without the error.
