-
Incident report
-
Resolution: Won't fix
-
Critical
-
None
-
2.0.0, 2.0.1
-
OS Ubuntu OR CENTOS 6 32/64 (We try with all)
Database MySQL or POSTGRESQL
1 MASTER NODE (MBOX)
4 CHILD NODE (NODE2 to NODE5)
We simulate a lost of zabbix master node and no backup to restore (we will apply zabbix monitoring using nodes in a REALY VERY LARGE enviroment (more than 9.5k hosts) )
In this step we create a new master node and try to sync the child node with this new BOX.
The new MBOX (master node) have a lot of error messages like bellow:
1328:20120704:212708.513 NODE 1: Received auditlog_details from node 5 for node 5 datalen 3754
1328:20120704:212708.514 [Z3005] query failed: [1452] Cannot add or update a child row: a foreign key constraint fails (`zabbix`.`auditlog_details`, CONSTRAINT `c_auditlog_details_1` FOREIGN KEY (`auditid`) REFERENCES `auditlog` (`auditid`) ON DELETE CASCADE) [insert into auditlog_details (auditdetailid,auditid,table_name,field_name,oldvalue,newvalue) values (500500000000001,500500000000003,'hosts','status','1','0'),(500500000000002,500500000000004,'hosts','host','Zabbix server','dfopsrvv0109'),(500500000000003,500500000000004,'hosts','status','1','0'),(500500000000004,500500000000004,'hosts','name','Zabbix server','dfopsrvv0109'),(500500000000005,500500000000042,'items','status','3','1'),(500500000000006,500500000000042,'items','error','Cannot find index [ifAdminStatus] of the OID [IF-MIB::ifInOctets["index","ifAdminStatus","GigabitEthernet1/0/49"]]: NOT FOUND: 1',''),(500500000000007,500500000000043,'items','status','0','1'),(500500000000008,500500000000044,'items','status','1','0'),(500500000000009,500500000000045,'items','status','1','0'),(500500000000010,500500000000046,'items','status','3','0'),(500500000000011,500500000000046,'items','error','Cannot find index [ifAdminStatus] of the OID [IF-MIB::ifInOctets["index","ifAdminStatus","GigabitEthernet1/0/49"]]: NOT FOUND: 1',''),(500500000000012,500500000000047,'items','status','3','1'),(500500000000013,500500000000047,'items','error','Cannot find index [ifAdminStatus] of the OID [IF-MIB::ifAdminStatus["index","ifAdminStatus","GigabitEthernet1/0/49"]]: NOT FOUND',''),(500500000000014,500500000000048,'items','status','1','0'),(500500000000015,500500000000049,'items','status','3','0'),(500500000000016,500500000000049,'items','error','Cannot find index [ifAdminStatus] of the OID [IF-MIB::ifInOctets["index","ifAdminStatus","GigabitEthernet1/0/49"]]: NOT FOUND: 1',''),(500500000000017,500500000000050,'items','status','3','0'),(500500000000018,500500000000050,'items','error','Cannot find index [ifAdminStatus] of the OID [IF-MIB::ifAdminStatus["index","ifAdminStatus","GigabitEthernet1/0/49"]]: NOT FOUND',''),(500500000000019,500500000000059,'items','status','0','1'),(500500000000020,500500000000060,'items','status','0','1'),(500500000000021,500500000000061,'items','status','0','1'),(500500000000022,500500000000062,'items','status','0','1'),(500500000000023,500500000000070,'','error','','Trigger just added. No status update so far.'),(500500000000024,500500000000070,'','templateid','0','500500000013742'),(500500000000025,500500000000070,'','value_flags','0','1'),(500500000000026,500500000000076,'items','status','1','0'),(500500000000027,500500000000077,'items','status','1','0'),(500500000000028,500500000000078,'items','status','1','0'),(500500000000029,500500000000079,'items','status','1','0'),(500500000000030,500500000000080,'items','status','1','0'),(500500000000031,500500000000081,'items','status','1','0'),(500500000000032,500500000000082,'items','status','1','0'),(500500000000033,500500000000083,'items','status','1','0'),(500500000000034,500500000000084,'items','status','1','0'),(500500000000035,500500000000085,'items','status','1','0'),(500500000000036,500500000000087,'items','status','1','0'),(500500000000037,500500000000088,'items','status','0','1');
The NODE5 (and all another nodes) have a lot of error messages:
26838:20120704:213858.629 NODE 5: Sending history_uint_sync of node 5 to node 1 datalen 2399
26838:20120704:213858.676 NODE 5: Sending history_text of node 5 to node 1 datalen 326
26838:20120704:213858.685 NODE 5: Sending auditlog of node 5 to node 1 datalen 22364
26838:20120704:213858.689 NOT OK
26838:20120704:213858.691 NODE 5: Sending auditlog_details of node 5 to node 1 datalen 3754
26838:20120704:213858.693 NOT OK
26838:20120704:213905.699 NODE 5: Sending history_uint_sync of node 5 to node 1 datalen 860
26838:20120704:213905.716 NODE 5: Sending history_text of node 5 to node 1 datalen 258
26838:20120704:213905.725 NODE 5: Sending auditlog of node 5 to node 1 datalen 22364
26838:20120704:213905.739 NOT OK
26838:20120704:213905.741 NODE 5: Sending auditlog_details of node 5 to node 1 datalen 3754
26838:20120704:213905.743 NOT OK
26838:20120704:213915.750 NODE 5: Sending history_uint_sync of node 5 to node 1 datalen 1611
26838:20120704:213915.773 NODE 5: Sending history_text of node 5 to node 1 datalen 1168
26838:20120704:213915.781 NODE 5: Sending auditlog of node 5 to node 1 datalen 22364
26838:20120704:213915.785 NOT OK
26838:20120704:213915.787 NODE 5: Sending auditlog_details of node 5 to node 1 datalen 3754
26838:20120704:213915.789 NOT OK
26838:20120704:213925.796 NODE 5: Sending history_uint_sync of node 5 to node 1 datalen 1621
26838:20120704:213925.819 NODE 5: Sending history_text of node 5 to node 1 datalen 1183
26838:20120704:213925.828 NODE 5: Sending auditlog of node 5 to node 1 datalen 22364
26838:20120704:213925.832 NOT OK
26838:20120704:213925.834 NODE 5: Sending auditlog_details of node 5 to node 1 datalen 3754
26838:20120704:213925.836 NOT OK
26838:20120704:213935.843 NODE 5: Sending history_uint_sync of node 5 to node 1 datalen 1626
26838:20120704:213935.867 NODE 5: Sending history_text of node 5 to node 1 datalen 1108
26838:20120704:213935.876 NODE 5: Sending auditlog of node 5 to node 1 datalen 22364
26838:20120704:213935.880 NOT OK
26838:20120704:213935.881 NODE 5: Sending auditlog_details of node 5 to node 1 datalen 3754
26838:20120704:213935.884 NOT OK
26838:20120704:213945.890 NODE 5: Sending history_uint_sync of node 5 to node 1 datalen 1652
26838:20120704:213945.913 NODE 5: Sending history_text of node 5 to node 1 datalen 218
26838:20120704:213945.921 NODE 5: Sending auditlog of node 5 to node 1 datalen 22364
26838:20120704:213945.925 NOT OK
26838:20120704:213945.927 NODE 5: Sending auditlog_details of node 5 to node 1 datalen 3754
26838:20120704:213945.929 NOT OK