Uploaded image for project: 'ZABBIX BUGS AND ISSUES'
  1. ZABBIX BUGS AND ISSUES
  2. ZBX-21854

Zabbix server causes many database locks while under heavy LLD load and trying to create regular items and triggers at the same time

XMLWordPrintable

    • Icon: Problem report Problem report
    • Resolution: Unresolved
    • Icon: Trivial Trivial
    • None
    • 6.0.9
    • None
    • None
    • mariadb 10.5

      Steps to reproduce:

      1. Create lots of hosts
      2. Run a script that creates many items and triggers on the hosts via LLD. It must be enough LLD to cause there to be an active queue when checking 'zabbix[lld_queue]'
      3. Run another script at the same time as the LLD script that creates regular items and triggers on the same hosts.
      4. The system should be very busy.

       

      a "something impossible" error is caused and many locks can be observed in the database.

       

      Here is a snippet of the backtrace from the zabbix_server.log:

       

      zabbix_server [3400]: ERROR [file and function: <db.c,DBget_nextid>, revision:64721203c07, line:752] Something impossible has just happened.
        3400:20221102:205022.409 === Backtrace: ===
      zabbix_server [3401]: ERROR [file and function: <db.c,DBget_nextid>, revision:64721203c07, line:752] Something impossible has just happened.
        3401:20221102:205022.410 === Backtrace: ===
      zabbix_server [3410]: ERROR [file and function: <db.c,DBget_nextid>, revision:64721203c07, line:752] Something impossible has just happened.
        3410:20221102:205022.410 === Backtrace: ===
      zabbix_server [3414]: ERROR [file and function: <db.c,DBget_nextid>, revision:64721203c07, line:752] Something impossible has just happened.
        3414:20221102:205022.434 === Backtrace: ===
        3412:20221102:205022.686 slow query: 36.484047 sec, "update ids set nextid=nextid+1000 where table_name='functions' and field_name='functionid'"
        3404:20221102:205022.730 slow query: 32.742759 sec, "update ids set nextid=nextid+1000 where table_name='functions' and field_name='functionid'"
        3399:20221102:205022.781 slow query: 18.273973 sec, "update ids set nextid=nextid+1000 where table_name='functions' and field_name='functionid'"
        3416:20221102:205022.822 slow query: 17.632202 sec, "update ids set nextid=nextid+1000 where table_name='functions' and field_name='functionid'"
        3400:20221102:205026.383 14: /usr/sbin/zabbix_server: lld worker #2 [processed 2 LLD rules, idle 0.000593 sec during 21.513069 sec](zbx_backtrace+0x3f) [0x56473c73ca19]
        3400:20221102:205026.383 13: /usr/sbin/zabbix_server: lld worker #2 [processed 2 LLD rules, idle 0.000593 sec during 21.513069 sec](+0x2ed7d3) [0x56473c7de7d3]
        3400:20221102:205026.383 12: /usr/sbin/zabbix_server: lld worker #2 [processed 2 LLD rules, idle 0.000593 sec during 21.513069 sec](DBget_maxid_num+0x129) [0x56473c7de980]
        3406:20221102:205026.384 14: /usr/sbin/zabbix_server: lld worker #5 [processed 2 LLD rules, idle 0.000602 sec during 21.238351 sec](zbx_backtrace+0x3f) [0x56473c73ca19]
        3410:20221102:205026.384 14: /usr/sbin/zabbix_server: lld worker #7 [processed 2 LLD rules, idle 0.001043 sec during 21.613398 sec](zbx_backtrace+0x3f) [0x56473c73ca19]
        3410:20221102:205026.384 13: /usr/sbin/zabbix_server: lld worker #7 [processed 2 LLD rules, idle 0.001043 sec during 21.613398 sec](+0x2ed7d3) [0x56473c7de7d3]
        3410:20221102:205026.384 12: /usr/sbin/zabbix_server: lld worker #7 [processed 2 LLD rules, idle 0.001043 sec during 21.613398 sec](DBget_maxid_num+0x129) [0x56473c7de980]
        3410:20221102:205026.384 11: /usr/sbin/zabbix_server: lld worker #7 [processed 2 LLD rules, idle 0.001043 sec during 21.613398 sec](+0x194d40) [0x56473c685d40]
        3410:20221102:205026.384 10: /usr/sbin/zabbix_server: lld worker #7 [processed 2 LLD rules, idle 0.001043 sec during 21.613398 sec](lld_update_triggers+0x2b6) [0x56473c688cf6]
        3410:20221102:205026.384 9: /usr/sbin/zabbix_server: lld worker #7 [processed 2 LLD rules, idle 0.001043 sec during 21.613398 sec](lld_process_discovery_rule+0x51d) [0x56473c6601a4]
        3410:20221102:205026.384 8: /usr/sbin/zabbix_server: lld worker #7 [processed 2 LLD rules, idle 0.001043 sec during 21.613398 sec](+0x16be7e) [0x56473c65ce7e]
        3410:20221102:205026.384 7: /usr/sbin/zabbix_server: lld worker #7 [processed 2 LLD rules, idle 0.001043 sec during 21.613398 sec](lld_worker_thread+0x30c) [0x56473c65d4f4]
        3410:20221102:205026.384 6: /usr/sbin/zabbix_server: lld worker #7 [processed 2 LLD rules, idle 0.001043 sec during 21.613398 sec](zbx_thread_start+0x37) [0x56473c74cb0c]
        3410:20221102:205026.384 5: /usr/sbin/zabbix_server: lld worker #7 [processed 2 LLD rules, idle 0.001043 sec during 21.613398 sec](+0x66ee1) [0x56473c557ee1]
        3410:20221102:205026.384 4: /usr/sbin/zabbix_server: lld worker #7 [processed 2 LLD rules, idle 0.001043 sec during 21.613398 sec](MAIN_ZABBIX_ENTRY+0x7b8) [0x56473c558afa]
        3410:20221102:205026.384 3: /usr/sbin/zabbix_server: lld worker #7 [processed 2 LLD rules, idle 0.001043 sec during 21.613398 sec](daemon_start+0x384) [0x56473c73c638]
        3410:20221102:205026.384 2: /usr/sbin/zabbix_server: lld worker #7 [processed 2 LLD rules, idle 0.001043 sec during 21.613398 sec](main+0x33a) [0x56473c55720e]
        3410:20221102:205026.384 1: /lib64/libc.so.6(__libc_start_main+0xf3) [0x7fc49bbfccf3]
        3410:20221102:205026.384 0: /usr/sbin/zabbix_server: lld worker #7 [processed 2 LLD rules, idle 0.001043 sec during 21.613398 sec](_start+0x2e) [0x56473c555f5e]
        3414:20221102:205026.384 14: /usr/sbin/zabbix_server: lld worker #9 [processed 2 LLD rules, idle 0.000401 sec during 21.887992 sec](zbx_backtrace+0x3f) [0x56473c73ca19]
        3414:20221102:205026.384 13: /usr/sbin/zabbix_server: lld worker #9 [processed 2 LLD rules, idle 0.000401 sec during 21.887992 sec](+0x2ed7d3) [0x56473c7de7d3]
        3414:20221102:205026.384 12: /usr/sbin/zabbix_server: lld worker #9 [processed 2 LLD rules, idle 0.000401 sec during 21.887992 sec](DBget_maxid_num+0x129) [0x56473c7de980]
        3414:20221102:205026.384 11: /usr/sbin/zabbix_server: lld worker #9 [processed 2 LLD rules, idle 0.000401 sec during 21.887992 sec](+0x194d40) [0x56473c685d40]
        3414:20221102:205026.384 10: /usr/sbin/zabbix_server: lld worker #9 [processed 2 LLD rules, idle 0.000401 sec during 21.887992 sec](lld_update_triggers+0x2b6) [0x56473c688cf6]
        3414:20221102:205026.384 9: /usr/sbin/zabbix_server: lld worker #9 [processed 2 LLD rules, idle 0.000401 sec during 21.887992 sec](lld_process_discovery_rule+0x51d) [0x56473c6601a4]
        3414:20221102:205026.384 8: /usr/sbin/zabbix_server: lld worker #9 [processed 2 LLD rules, idle 0.000401 sec during 21.887992 sec](+0x16be7e) [0x56473c65ce7e]
        3414:20221102:205026.384 7: /usr/sbin/zabbix_server: lld worker #9 [processed 2 LLD rules, idle 0.000401 sec during 21.887992 sec](lld_worker_thread+0x30c) [0x56473c65d4f4]
        3414:20221102:205026.384 6: /usr/sbin/zabbix_server: lld worker #9 [processed 2 LLD rules, idle 0.000401 sec during 21.887992 sec](zbx_thread_start+0x37) [0x56473c74cb0c]
        3414:20221102:205026.384 5: /usr/sbin/zabbix_server: lld worker #9 [processed 2 LLD rules, idle 0.000401 sec during 21.887992 sec](+0x66ee1) [0x56473c557ee1]
        3414:20221102:205026.384 4: /usr/sbin/zabbix_server: lld worker #9 [processed 2 LLD rules, idle 0.000401 sec during 21.887992 sec](MAIN_ZABBIX_ENTRY+0x7b8) [0x56473c558afa]
        3414:20221102:205026.384 3: /usr/sbin/zabbix_server: lld worker #9 [processed 2 LLD rules, idle 0.000401 sec during 21.887992 sec](daemon_start+0x384) [0x56473c73c638]
        3414:20221102:205026.384 2: /usr/sbin/zabbix_server: lld worker #9 [processed 2 LLD rules, idle 0.000401 sec during 21.887992 sec](main+0x33a) [0x56473c55720e]
        3414:20221102:205026.384 1: /lib64/libc.so.6(__libc_start_main+0xf3) [0x7fc49bbfccf3]
        3414:20221102:205026.384 0: /usr/sbin/zabbix_server: lld worker #9 [processed 2 LLD rules, idle 0.000401 sec during 21.887992 sec](_start+0x2e) [0x56473c555f5e]

            zabbix.dev Zabbix Development Team
            GRyan337 Ryan Eberly
            Votes:
            0 Vote for this issue
            Watchers:
            3 Start watching this issue

              Created:
              Updated: