ZABBIX BUGS AND ISSUES
  1. ZABBIX BUGS AND ISSUES
  2. ZBX-7452

Event synchronization is broken in multinode DM case

    Details

    • Type: Incident report Incident report
    • Status: Closed
    • Priority: Blocker Blocker
    • Resolution: Fixed
    • Affects Version/s: 2.2.0
    • Fix Version/s: 2.2.2rc1, 2.3.0
    • Component/s: Server (S)
    • Environment:
      oracle linux ( RHEL clone ) 6.4, zabbix 2.2 ( rpm from official site ), mysql 5.6, vsphere 5.1

      Description

      Zabbix 2.2, distributed monitoring, nodes. After update from 2.0, event syncronization ( from slave nodes to master node ) is completely broken.
      Other data ( history_log, history_str_sync, history_uint_sync, history_sync ) synchronization works fine.

      As a result, there are obsoleted data on master node. Triggers are actual, mail notification send in time, but monitoring->dashboard show obsoleted information and monitoring->events show obsoleted information. So, multinode configuration become garbage.

      In traffic dump i can see that

      1. zabbix master return constant value for all requests "ZBX_GET_HISTORY_LAST_ID.202.202 events.eventid"
      2. nevertheless, zabbix server response "OK" for every synchronization request "History.202.202.events" with ~ 5k events.

      I rebuild zabbix from source on master node with debug messages ( src/zabbix_server/events.c, zabbix_log(LOG_LEVEL_WARNING,"events num: %i", events_num); ) . After that, i see that events successfully add to an array ( add_event() ), and events_num increase every sync. But in process_events() debug message every time show "events_num: 0" , su events does not push to database

      I create test stand ( 2 node, clear install zabbix 2.2 and mysql 5.6 ). It show the same problem. I.e. it's not an upgrade issue, but zabbix branch 2.2 bug

        Issue Links

          Activity

          Hide
          Gerrit Fluck added a comment -

          Seems to be a related issue: ZBX-7365
          We experience the same problems in 2.2

          Show
          Gerrit Fluck added a comment - Seems to be a related issue: ZBX-7365 We experience the same problems in 2.2
          Hide
          Dilson Tomé added a comment -

          Tested on Zabbix 2.2.1rc1 and same problem.
          Can't sync events.

          NODE 2>
          mysql> select * from events;
          --------------------------------------------------------------------------------+

          eventid source object objectid clock value acknowledged ns

          --------------------------------------------------------------------------------+

          200000000000028 3 0 200200000013575 1386427560 1 0 371474894
          200000000000029 0 0 200200000013575 1386427611 0 0 173221504
          200000000000030 3 0 200200000013575 1386427611 0 0 173221504
          200000000000031 0 0 200200000013579 1386427615 0 0 185215493
          200000000000032 0 0 200200000013580 1386427616 0 0 196416635
          200000000000033 0 0 200200000013581 1386427620 0 0 206510097
          200000000000034 0 0 200200000013582 1386427625 0 0 324965089
          200000000000035 0 0 200200000013584 1386427634 0 0 392126704
          200000000000036 0 0 200200000013588 1386427640 0 0 421204147
          200000000000037 3 0 200200000013586 1386427937 1 0 205579069
          200000000000038 3 0 200200000013574 1386428020 1 0 344295607
          200000000000039 3 0 200200000013576 1386428020 1 0 344295607
          200000000000040 3 0 200200000013577 1386428020 1 0 344295607
          200000000000041 3 0 200200000013578 1386428020 1 0 344295607
          200000000000042 3 0 200200000013579 1386428020 1 0 344295607
          200000000000043 3 0 200200000013580 1386428020 1 0 344295607
          200000000000044 3 0 200200000013581 1386428020 1 0 344295607
          200000000000045 3 0 200200000013582 1386428020 1 0 344295607
          200000000000046 3 0 200200000013583 1386428020 1 0 344295607
          200000000000047 3 0 200200000013584 1386428020 1 0 344295607
          200000000000048 3 0 200200000013585 1386428020 1 0 344295607
          200000000000049 3 0 200200000013587 1386428020 1 0 344295607
          200000000000050 3 0 200200000013588 1386428020 1 0 344295607
          200000000000051 3 0 200200000013589 1386428020 1 0 344295607
          200000000000052 3 0 200200000013590 1386428020 1 0 344295607
          200000000000053 0 0 200200000013575 1386428040 1 0 574789153

          --------------------------------------------------------------------------------+
          26 rows in set (0.00 sec)

          NODE 1
          mysql> select * from events;
          Empty set (0.00 sec)

          Show
          Dilson Tomé added a comment - Tested on Zabbix 2.2.1rc1 and same problem. Can't sync events. NODE 2> mysql> select * from events; ---------------- ------ ------ --------------- ---------- ----- ------------ ----------+ eventid source object objectid clock value acknowledged ns ---------------- ------ ------ --------------- ---------- ----- ------------ ----------+ 200000000000028 3 0 200200000013575 1386427560 1 0 371474894 200000000000029 0 0 200200000013575 1386427611 0 0 173221504 200000000000030 3 0 200200000013575 1386427611 0 0 173221504 200000000000031 0 0 200200000013579 1386427615 0 0 185215493 200000000000032 0 0 200200000013580 1386427616 0 0 196416635 200000000000033 0 0 200200000013581 1386427620 0 0 206510097 200000000000034 0 0 200200000013582 1386427625 0 0 324965089 200000000000035 0 0 200200000013584 1386427634 0 0 392126704 200000000000036 0 0 200200000013588 1386427640 0 0 421204147 200000000000037 3 0 200200000013586 1386427937 1 0 205579069 200000000000038 3 0 200200000013574 1386428020 1 0 344295607 200000000000039 3 0 200200000013576 1386428020 1 0 344295607 200000000000040 3 0 200200000013577 1386428020 1 0 344295607 200000000000041 3 0 200200000013578 1386428020 1 0 344295607 200000000000042 3 0 200200000013579 1386428020 1 0 344295607 200000000000043 3 0 200200000013580 1386428020 1 0 344295607 200000000000044 3 0 200200000013581 1386428020 1 0 344295607 200000000000045 3 0 200200000013582 1386428020 1 0 344295607 200000000000046 3 0 200200000013583 1386428020 1 0 344295607 200000000000047 3 0 200200000013584 1386428020 1 0 344295607 200000000000048 3 0 200200000013585 1386428020 1 0 344295607 200000000000049 3 0 200200000013587 1386428020 1 0 344295607 200000000000050 3 0 200200000013588 1386428020 1 0 344295607 200000000000051 3 0 200200000013589 1386428020 1 0 344295607 200000000000052 3 0 200200000013590 1386428020 1 0 344295607 200000000000053 0 0 200200000013575 1386428040 1 0 574789153 ---------------- ------ ------ --------------- ---------- ----- ------------ ----------+ 26 rows in set (0.00 sec) NODE 1 mysql> select * from events; Empty set (0.00 sec)
          Hide
          Oleg Korchagin added a comment - - edited

          In 2.2.1 bug still actual

          quick & dirty fix:

          see attached file "ZBX-7365_ZBX-7452_quick_and_dirty_fix.patch"

          Show
          Oleg Korchagin added a comment - - edited In 2.2.1 bug still actual quick & dirty fix: see attached file " ZBX-7365 _ ZBX-7452 _quick_and_dirty_fix.patch"
          Hide
          Oleksiy Zagorskyi added a comment -

          Issue CONFIRMED.
          It's a regression caused changes in ZBXNEXT-1575 in rev 34766
          Usage of "process_events()" function has disappeared in that revision.
          https://www.zabbix.org/websvn/wsvn/zabbix.com/trunk/src/zabbix_server/trapper/nodehistory.c?op=diff&rev=34766&peg=34766

          Show
          Oleksiy Zagorskyi added a comment - Issue CONFIRMED. It's a regression caused changes in ZBXNEXT-1575 in rev 34766 Usage of "process_events()" function has disappeared in that revision. https://www.zabbix.org/websvn/wsvn/zabbix.com/trunk/src/zabbix_server/trapper/nodehistory.c?op=diff&rev=34766&peg=34766
          Hide
          Alexander Vladishev added a comment -

          Oleg,

          Thank you for the patch! It will be integrated in version 2.2.2 with little change.

          Show
          Alexander Vladishev added a comment - Oleg, Thank you for the patch! It will be integrated in version 2.2.2 with little change.
          Hide
          Alexander Vladishev added a comment -

          Fixed in the development branch svn://svn.zabbix.com/branches/dev/ZBX-7452

          Show
          Alexander Vladishev added a comment - Fixed in the development branch svn://svn.zabbix.com/branches/dev/ZBX-7452
          Hide
          Andris Zeila added a comment -

          Successfully tested

          Show
          Andris Zeila added a comment - Successfully tested
          Hide
          Alexander Vladishev added a comment -

          Fixed in pre-2.2.2 r41210 and pre-2.3.0 (trunk) r41211.

          Show
          Alexander Vladishev added a comment - Fixed in pre-2.2.2 r41210 and pre-2.3.0 (trunk) r41211.
          Hide
          Andreas Franke added a comment -

          Hello,
          i have updated today my 2 server to 2.2.2rc2 but the problem is still existing. It is possible to clean the master database or the sync status from the events table?

          Show
          Andreas Franke added a comment - Hello, i have updated today my 2 server to 2.2.2rc2 but the problem is still existing. It is possible to clean the master database or the sync status from the events table?
          Hide
          Oleksiy Zagorskyi added a comment -

          Andreas, could you restart master node with DebugLevel=4, let it run for 3 minutes and attach compressed log file here ?

          Show
          Oleksiy Zagorskyi added a comment - Andreas, could you restart master node with DebugLevel=4, let it run for 3 minutes and attach compressed log file here ?
          Hide
          Andreas Franke added a comment -

          Hello Oleksiy i have found the failure. I had some bad entrys in the events table on my child node. There were some entrys which has as objectid some triggers who don't exists.

          Show
          Andreas Franke added a comment - Hello Oleksiy i have found the failure. I had some bad entrys in the events table on my child node. There were some entrys which has as objectid some triggers who don't exists.
          Hide
          Oleksiy Zagorskyi added a comment -

          Andreas, good to know.
          Then there is a question - how they could appear ?
          and it goes to ZBX-3996

          Show
          Oleksiy Zagorskyi added a comment - Andreas, good to know. Then there is a question - how they could appear ? and it goes to ZBX-3996
          Hide
          Andreas Franke added a comment -

          Yes i think this is the same problem.

          Show
          Andreas Franke added a comment - Yes i think this is the same problem.
          Hide
          Giovanni Lovato added a comment -

          I updated to 2.2.2 but I still get

          4046:20140502:094156.452 NODE 3: sending events of node 3 to node 1 datalen 557097
          4046:20140502:094156.467 NOT OK

          Maybe because the DB is unclean from 2.2.1? How can I fix that?

          Show
          Giovanni Lovato added a comment - I updated to 2.2.2 but I still get 4046:20140502:094156.452 NODE 3: sending events of node 3 to node 1 datalen 557097 4046:20140502:094156.467 NOT OK Maybe because the DB is unclean from 2.2.1? How can I fix that?

            People

            • Assignee:
              Unassigned
              Reporter:
              Oleg Korchagin
            • Votes:
              4 Vote for this issue
              Watchers:
              9 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: