Uploaded image for project: 'ZABBIX BUGS AND ISSUES'
  1. ZABBIX BUGS AND ISSUES
  2. ZBX-21713

"Something impossible has just happened" on Oracle DB. events.c,update_trigger_problem_count

    XMLWordPrintable

Details

    • Team A
    • Sprint 93 (Oct 2022), Sprint 94 (Nov 2022), Sprint 95 (Dec 2022)
    • 1

    Description

      An installation (using Oracle DB) has been upgraded from 6.0.1 to 6.0.8 and an error started to be logged after a few minutes:

       41571:20220926:200340.883 Zabbix Server stopped. Zabbix 6.0.1 (revision a80cb13868).
       44422:20220926:200809.198 Starting Zabbix Server. Zabbix 6.0.8 (revision c7c3044a4a2).
       44629:20220926:200927.595 server #27 started [task manager #1]
      --
       44641:20220926:201107.513 sending configuration data to proxy ....
      zabbix_server [44629]: ERROR [file and function: <events.c,update_trigger_problem_count>, revision:c7c3044a4a2, line:1562] Something impossible has just happened.
       44629:20220926:201150.849 === Backtrace: ===
       44629:20220926:201150.849 11: /data/zabbix/sbin/zabbix_server: task manager [processing tasks](zbx_backtrace+0x35) [0x596635]
       44629:20220926:201150.850 10: /data/zabbix/sbin/zabbix_server: task manager [processing tasks]() [0x51c956]
       44629:20220926:201150.850 9: /data/zabbix/sbin/zabbix_server: task manager [processing tasks](zbx_close_problem+0x14a) [0x520bda]
       44629:20220926:201150.850 8: /data/zabbix/sbin/zabbix_server: task manager [processing tasks]() [0x4a06da]
       44629:20220926:201150.850 7: /data/zabbix/sbin/zabbix_server: task manager [processing tasks](taskmanager_thread+0x195) [0x4a0da5]
       44629:20220926:201150.850 6: /data/zabbix/sbin/zabbix_server: task manager [processing tasks](zbx_thread_start+0x3e) [0x5a190e]
       44629:20220926:201150.850 5: /data/zabbix/sbin/zabbix_server: task manager [processing tasks]() [0x44acda]
       44629:20220926:201150.850 4: /data/zabbix/sbin/zabbix_server: task manager [processing tasks](MAIN_ZABBIX_ENTRY+0x9f6) [0x44bb56]
       44629:20220926:201150.850 3: /data/zabbix/sbin/zabbix_server: task manager [processing tasks](daemon_start+0x1b2) [0x5963f2]
       44629:20220926:201150.850 2: /data/zabbix/sbin/zabbix_server: task manager [processing tasks](main+0x4dc) [0x449c7c]
       44629:20220926:201150.850 1: /lib64/libc.so.6(__libc_start_main+0xf5) [0x7fd258249555]
       44629:20220926:201150.850 0: /data/zabbix/sbin/zabbix_server: task manager [processing tasks]() [0x44a06a]
      zabbix_server [44629]: ERROR [file and function: <events.c,update_trigger_problem_count>, revision:c7c3044a4a2, line:1562] Something impossible has just happened.
       44629:20220926:201200.891 === Backtrace: ===
       44629:20220926:201200.891 11: /data/zabbix/sbin/zabbix_server: task manager [processing tasks](zbx_backtrace+0x35) [0x596635]
       44629:20220926:201200.891 10: /data/zabbix/sbin/zabbix_server: task manager [processing tasks]() [0x51c956]
       44629:20220926:201200.891 9: /data/zabbix/sbin/zabbix_server: task manager [processing tasks](zbx_close_problem+0x14a) [0x520bda]
       44629:20220926:201200.891 8: /data/zabbix/sbin/zabbix_server: task manager [processing tasks]() [0x4a06da]
       44629:20220926:201200.891 7: /data/zabbix/sbin/zabbix_server: task manager [processing tasks](taskmanager_thread+0x195) [0x4a0da5]
       44629:20220926:201200.891 6: /data/zabbix/sbin/zabbix_server: task manager [processing tasks](zbx_thread_start+0x3e) [0x5a190e]
       44629:20220926:201200.891 5: /data/zabbix/sbin/zabbix_server: task manager [processing tasks]() [0x44acda]
       44629:20220926:201200.891 4: /data/zabbix/sbin/zabbix_server: task manager [processing tasks](MAIN_ZABBIX_ENTRY+0x9f6) [0x44bb56]
       44629:20220926:201200.891 3: /data/zabbix/sbin/zabbix_server: task manager [processing tasks](daemon_start+0x1b2) [0x5963f2]
       44629:20220926:201200.891 2: /data/zabbix/sbin/zabbix_server: task manager [processing tasks](main+0x4dc) [0x449c7c]
       44629:20220926:201200.891 1: /lib64/libc.so.6(__libc_start_main+0xf5) [0x7fd258249555]
       44629:20220926:201200.891 0: /data/zabbix/sbin/zabbix_server: task manager [processing tasks]() [0x44a06a]
      

      The error repeated ~100 times per a day.

      Debug level for task manager was increased and here are 2 files where we see when it happens. I've added empty lines just to split blocks/function call, all lines are present and in original order.

      Its known that there is some scripts that close certain problems, including the *** problem shown in the logs. They do an event.acknowledge API call.

      My findings:
      in each file (in each cycle) task manager tries to close 2 or 3 tasks, for the same trigger. 1st (and next one) task/attempt of closing is failed with the error, while last one is without the error.

      Attachments

        1. backtrace1-2events
          13 kB
        2. backtrace2-3events
          20 kB
        3. trigger-100190.png
          trigger-100190.png
          54 kB
        4. trigger-25912.png
          trigger-25912.png
          49 kB

        Activity

          People

            vso Vladislavs Sokurenko
            zalex_ua Oleksii Zagorskyi
            Votes:
            0 Vote for this issue
            Watchers:
            6 Start watching this issue

            Dates

              Created:
              Updated: