|
Steps to reproduce:
- Create lots of hosts
- Run a script that creates many items and triggers on the hosts via LLD. It must be enough LLD to cause there to be an active queue when checking 'zabbix[lld_queue]'
- Run another script at the same time as the LLD script that creates regular items and triggers on the same hosts.
- The system should be very busy.
a "something impossible" error is caused and many locks can be observed in the database.
Here is a snippet of the backtrace from the zabbix_server.log:
zabbix_server [3400]: ERROR [file and function: <db.c,DBget_nextid>, revision:64721203c07, line:752] Something impossible has just happened.
3400:20221102:205022.409 === Backtrace: ===
zabbix_server [3401]: ERROR [file and function: <db.c,DBget_nextid>, revision:64721203c07, line:752] Something impossible has just happened.
3401:20221102:205022.410 === Backtrace: ===
zabbix_server [3410]: ERROR [file and function: <db.c,DBget_nextid>, revision:64721203c07, line:752] Something impossible has just happened.
3410:20221102:205022.410 === Backtrace: ===
zabbix_server [3414]: ERROR [file and function: <db.c,DBget_nextid>, revision:64721203c07, line:752] Something impossible has just happened.
3414:20221102:205022.434 === Backtrace: ===
3412:20221102:205022.686 slow query: 36.484047 sec, "update ids set nextid=nextid+1000 where table_name='functions' and field_name='functionid'"
3404:20221102:205022.730 slow query: 32.742759 sec, "update ids set nextid=nextid+1000 where table_name='functions' and field_name='functionid'"
3399:20221102:205022.781 slow query: 18.273973 sec, "update ids set nextid=nextid+1000 where table_name='functions' and field_name='functionid'"
3416:20221102:205022.822 slow query: 17.632202 sec, "update ids set nextid=nextid+1000 where table_name='functions' and field_name='functionid'"
3400:20221102:205026.383 14: /usr/sbin/zabbix_server: lld worker #2 [processed 2 LLD rules, idle 0.000593 sec during 21.513069 sec](zbx_backtrace+0x3f) [0x56473c73ca19]
3400:20221102:205026.383 13: /usr/sbin/zabbix_server: lld worker #2 [processed 2 LLD rules, idle 0.000593 sec during 21.513069 sec](+0x2ed7d3) [0x56473c7de7d3]
3400:20221102:205026.383 12: /usr/sbin/zabbix_server: lld worker #2 [processed 2 LLD rules, idle 0.000593 sec during 21.513069 sec](DBget_maxid_num+0x129) [0x56473c7de980]
3406:20221102:205026.384 14: /usr/sbin/zabbix_server: lld worker #5 [processed 2 LLD rules, idle 0.000602 sec during 21.238351 sec](zbx_backtrace+0x3f) [0x56473c73ca19]
3410:20221102:205026.384 14: /usr/sbin/zabbix_server: lld worker #7 [processed 2 LLD rules, idle 0.001043 sec during 21.613398 sec](zbx_backtrace+0x3f) [0x56473c73ca19]
3410:20221102:205026.384 13: /usr/sbin/zabbix_server: lld worker #7 [processed 2 LLD rules, idle 0.001043 sec during 21.613398 sec](+0x2ed7d3) [0x56473c7de7d3]
3410:20221102:205026.384 12: /usr/sbin/zabbix_server: lld worker #7 [processed 2 LLD rules, idle 0.001043 sec during 21.613398 sec](DBget_maxid_num+0x129) [0x56473c7de980]
3410:20221102:205026.384 11: /usr/sbin/zabbix_server: lld worker #7 [processed 2 LLD rules, idle 0.001043 sec during 21.613398 sec](+0x194d40) [0x56473c685d40]
3410:20221102:205026.384 10: /usr/sbin/zabbix_server: lld worker #7 [processed 2 LLD rules, idle 0.001043 sec during 21.613398 sec](lld_update_triggers+0x2b6) [0x56473c688cf6]
3410:20221102:205026.384 9: /usr/sbin/zabbix_server: lld worker #7 [processed 2 LLD rules, idle 0.001043 sec during 21.613398 sec](lld_process_discovery_rule+0x51d) [0x56473c6601a4]
3410:20221102:205026.384 8: /usr/sbin/zabbix_server: lld worker #7 [processed 2 LLD rules, idle 0.001043 sec during 21.613398 sec](+0x16be7e) [0x56473c65ce7e]
3410:20221102:205026.384 7: /usr/sbin/zabbix_server: lld worker #7 [processed 2 LLD rules, idle 0.001043 sec during 21.613398 sec](lld_worker_thread+0x30c) [0x56473c65d4f4]
3410:20221102:205026.384 6: /usr/sbin/zabbix_server: lld worker #7 [processed 2 LLD rules, idle 0.001043 sec during 21.613398 sec](zbx_thread_start+0x37) [0x56473c74cb0c]
3410:20221102:205026.384 5: /usr/sbin/zabbix_server: lld worker #7 [processed 2 LLD rules, idle 0.001043 sec during 21.613398 sec](+0x66ee1) [0x56473c557ee1]
3410:20221102:205026.384 4: /usr/sbin/zabbix_server: lld worker #7 [processed 2 LLD rules, idle 0.001043 sec during 21.613398 sec](MAIN_ZABBIX_ENTRY+0x7b8) [0x56473c558afa]
3410:20221102:205026.384 3: /usr/sbin/zabbix_server: lld worker #7 [processed 2 LLD rules, idle 0.001043 sec during 21.613398 sec](daemon_start+0x384) [0x56473c73c638]
3410:20221102:205026.384 2: /usr/sbin/zabbix_server: lld worker #7 [processed 2 LLD rules, idle 0.001043 sec during 21.613398 sec](main+0x33a) [0x56473c55720e]
3410:20221102:205026.384 1: /lib64/libc.so.6(__libc_start_main+0xf3) [0x7fc49bbfccf3]
3410:20221102:205026.384 0: /usr/sbin/zabbix_server: lld worker #7 [processed 2 LLD rules, idle 0.001043 sec during 21.613398 sec](_start+0x2e) [0x56473c555f5e]
3414:20221102:205026.384 14: /usr/sbin/zabbix_server: lld worker #9 [processed 2 LLD rules, idle 0.000401 sec during 21.887992 sec](zbx_backtrace+0x3f) [0x56473c73ca19]
3414:20221102:205026.384 13: /usr/sbin/zabbix_server: lld worker #9 [processed 2 LLD rules, idle 0.000401 sec during 21.887992 sec](+0x2ed7d3) [0x56473c7de7d3]
3414:20221102:205026.384 12: /usr/sbin/zabbix_server: lld worker #9 [processed 2 LLD rules, idle 0.000401 sec during 21.887992 sec](DBget_maxid_num+0x129) [0x56473c7de980]
3414:20221102:205026.384 11: /usr/sbin/zabbix_server: lld worker #9 [processed 2 LLD rules, idle 0.000401 sec during 21.887992 sec](+0x194d40) [0x56473c685d40]
3414:20221102:205026.384 10: /usr/sbin/zabbix_server: lld worker #9 [processed 2 LLD rules, idle 0.000401 sec during 21.887992 sec](lld_update_triggers+0x2b6) [0x56473c688cf6]
3414:20221102:205026.384 9: /usr/sbin/zabbix_server: lld worker #9 [processed 2 LLD rules, idle 0.000401 sec during 21.887992 sec](lld_process_discovery_rule+0x51d) [0x56473c6601a4]
3414:20221102:205026.384 8: /usr/sbin/zabbix_server: lld worker #9 [processed 2 LLD rules, idle 0.000401 sec during 21.887992 sec](+0x16be7e) [0x56473c65ce7e]
3414:20221102:205026.384 7: /usr/sbin/zabbix_server: lld worker #9 [processed 2 LLD rules, idle 0.000401 sec during 21.887992 sec](lld_worker_thread+0x30c) [0x56473c65d4f4]
3414:20221102:205026.384 6: /usr/sbin/zabbix_server: lld worker #9 [processed 2 LLD rules, idle 0.000401 sec during 21.887992 sec](zbx_thread_start+0x37) [0x56473c74cb0c]
3414:20221102:205026.384 5: /usr/sbin/zabbix_server: lld worker #9 [processed 2 LLD rules, idle 0.000401 sec during 21.887992 sec](+0x66ee1) [0x56473c557ee1]
3414:20221102:205026.384 4: /usr/sbin/zabbix_server: lld worker #9 [processed 2 LLD rules, idle 0.000401 sec during 21.887992 sec](MAIN_ZABBIX_ENTRY+0x7b8) [0x56473c558afa]
3414:20221102:205026.384 3: /usr/sbin/zabbix_server: lld worker #9 [processed 2 LLD rules, idle 0.000401 sec during 21.887992 sec](daemon_start+0x384) [0x56473c73c638]
3414:20221102:205026.384 2: /usr/sbin/zabbix_server: lld worker #9 [processed 2 LLD rules, idle 0.000401 sec during 21.887992 sec](main+0x33a) [0x56473c55720e]
3414:20221102:205026.384 1: /lib64/libc.so.6(__libc_start_main+0xf3) [0x7fc49bbfccf3]
3414:20221102:205026.384 0: /usr/sbin/zabbix_server: lld worker #9 [processed 2 LLD rules, idle 0.000401 sec during 21.887992 sec](_start+0x2e) [0x56473c555f5e]
|