[ZBX-8325] server crashes with Oracle when deleting items that are not discovered anymore Created: 2014 Jun 09  Updated: 2017 May 30  Resolved: 2014 Jun 10

Status: Closed
Project: ZABBIX BUGS AND ISSUES
Component/s: Server (S)
Affects Version/s: 2.2.3
Fix Version/s: 2.2.4rc2, 2.3.2

Type: Incident report Priority: Major
Reporter: Aleksandrs Saveljevs Assignee: Unassigned
Resolution: Fixed Votes: 0
Labels: crash, oracle
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified


 Description   

We have created 3000 items using low-level discovery. After that we changed low-level discovery to yield 750 items only. Server started deleting undiscoverable items and crashed when it was inserting records into "housekeeper" table.



 Comments   
Comment by Aleksandrs Saveljevs [ 2014 Jun 09 ]

Relevant details from the log:

...
18121:20140609:151858.374 In DBdelete_items() values_num:2250
18121:20140609:151858.374 query [txnlev:1] [select distinct itemid from item_discovery where parent_itemid between 24508 and 26757]
18121:20140609:151858.380 In DBdelete_graphs_by_itemids() values_num:2250
18121:20140609:151858.380 query [txnlev:1] [select distinct graphid from graphs_items where itemid between 24508 and 26757]
18121:20140609:151858.385 End of DBdelete_graphs_by_itemids()
18121:20140609:151858.385 In DBdelete_triggers_by_itemids() values_num:2250
18121:20140609:151858.385 query [txnlev:1] [select distinct triggerid from functions where itemid between 24508 and 26757]
18121:20140609:151858.390 End of DBdelete_triggers_by_itemids()
18121:20140609:151858.390 In DBdelete_history_by_itemids() values_num:2250
18121:20140609:151858.390 In DBget_nextid() tablename:'housekeeper'
18121:20140609:151858.390 query [txnlev:1] [select nextid from ids where nodeid=0 and table_name='housekeeper' and field_name='housekeeperid']
18121:20140609:151858.392 query [txnlev:1] [select max(housekeeperid) from housekeeper where housekeeperid between 0 and 9223372036854775807]
18121:20140609:151858.398 query [txnlev:1] [insert into ids (nodeid,table_name,field_name,nextid) values (0,'housekeeper','housekeeperid',0)]
18121:20140609:151858.400 query [txnlev:1] [select nextid from ids where nodeid=0 and table_name='housekeeper' and field_name='housekeeperid']
18121:20140609:151858.401 query [txnlev:1] [update ids set nextid=nextid+15750 where nodeid=0 and table_name='housekeeper' and field_name='housekeeperid']
18121:20140609:151858.403 query [txnlev:1] [select nextid from ids where nodeid=0 and table_name='housekeeper' and field_name='housekeeperid']
18121:20140609:151858.404 End of DBget_nextid():1 table:'housekeeper' recid:'housekeeperid'
18121:20140609:151858.413 query [txnlev:1] [insert into housekeeper (housekeeperid,tablename,field,value) values (:1,:2,:3,:4)]
18121:20140609:151858.413 insert [txnlev:1] [1,'history','itemid',24508]
18121:20140609:151858.477 zbx_db_statement_execute():1
18121:20140609:151858.477 insert [txnlev:1] [2,'history_str','itemid',24508]
...
18121:20140609:151911.625 zbx_db_statement_execute():1
18121:20140609:151911.625 insert [txnlev:1] [13069,'trends_uint','itemid',26374]
18121:20140609:151911.626 zbx_db_statement_execute():1
18121:20140609:151911.626 insert [txnlev:1] [13070,'history','itemid',26375]
18121:20140609:151911.627 zbx_db_statement_execute():1
18121:20140609:151911.627 insert [txnlev:1] [13071,'history_str','itemid',26375]
18121:20140609:151911.628 zbx_db_statement_execute():1
18121:20140609:151911.628 insert [txnlev:1] [13072,'history_uint','itemid',26375]
18121:20140609:151911.629 zbx_db_statement_execute():1
18121:20140609:151911.629 insert [txnlev:1] [13073,'history_log','itemid',26375]
18121:20140609:151911.630 zbx_db_statement_execute():1
18121:20140609:151911.630 insert [txnlev:1] [13074,'history_text','itemid',26375]
18121:20140609:151911.631 zbx_db_statement_execute():1
18121:20140609:151911.631 insert [txnlev:1] [13075,'trends','itemid',26375]
18121:20140609:151911.632 zbx_db_statement_execute():1
18121:20140609:151911.632 insert [txnlev:1] [13076,'trends_uint','itemid',26375]
18121:20140609:151911.632 Got signal [signal:11(SIGSEGV),reason:1,refaddr:0x7fe199464754]. Crashing ...
18121:20140609:151911.632 ====== Fatal information: ======
18121:20140609:151911.632 Program counter: 0x7fe1ae115dde
18121:20140609:151911.632 === Registers: ===
18121:20140609:151911.632 r8      =                6 =                    6 =                    6
18121:20140609:151911.632 r9      =                1 =                    1 =                    1
18121:20140609:151911.632 r10     =                b =                   11 =                   11
18121:20140609:151911.632 r11     =     7fe1aebbcd20 =      140607275912480 =      140607275912480
18121:20140609:151911.632 r12     =          1431cc8 =             21175496 =             21175496
18121:20140609:151911.632 r13     =               16 =                   22 =                   22
18121:20140609:151911.632 r14     =     7fe199464754 =      140606915888980 =      140606915888980
18121:20140609:151911.632 r15     =          142e690 =             21161616 =             21161616
18121:20140609:151911.632 rdi     =          1420958 =             21104984 =             21104984
18121:20140609:151911.632 rsi     =          142e690 =             21161616 =             21161616
18121:20140609:151911.632 rbp     =     7fffc804d9f0 =      140736549149168 =      140736549149168
18121:20140609:151911.632 rbx     =               15 =                   21 =                   21
18121:20140609:151911.632 rdx     =                0 =                    0 =                    0
18121:20140609:151911.632 rax     =          1431cc8 =             21175496 =             21175496
18121:20140609:151911.632 rcx     =               16 =                   22 =                   22
18121:20140609:151911.632 rsp     =     7fffc804d9d0 =      140736549149136 =      140736549149136
18121:20140609:151911.632 rip     =     7fe1ae115dde =      140607264742878 =      140607264742878
18121:20140609:151911.632 efl     =            10202 =                66050 =                66050
18121:20140609:151911.632 csgsfs  =    1000000000033 =      281474976710707 =      281474976710707
18121:20140609:151911.632 err     =                4 =                    4 =                    4
18121:20140609:151911.632 trapno  =                e =                   14 =                   14
18121:20140609:151911.632 oldmask =                0 =                    0 =                    0
18121:20140609:151911.632 cr2     =     7fe199464754 =      140606915888980 =      140606915888980
18121:20140609:151911.632 === Backtrace: ===
18121:20140609:151911.645 27: /home/asaveljevs/projects/zabbix-bin-2.2/sbin/zabbix_server: poller #1 [got 50 values in 0.039941 sec, getting values](print_fatal_info+0x117) [0x482f1a]
18121:20140609:151911.645 26: /home/asaveljevs/projects/zabbix-bin-2.2/sbin/zabbix_server: poller #1 [got 50 values in 0.039941 sec, getting values]() [0x4832bb]
18121:20140609:151911.645 25: /lib/x86_64-linux-gnu/libc.so.6(+0x35420) [0x7fe1a95f1420]
18121:20140609:151911.645 24: /usr/lib/oracle/12.1/client64/lib/libclntsh.so.12.1(ttcv2n+0x26e) [0x7fe1ae115dde]
18121:20140609:151911.645 23: /usr/lib/oracle/12.1/client64/lib/libclntsh.so.12.1(ttcacs+0x39a) [0x7fe1ae12655a]
18121:20140609:151911.645 22: /usr/lib/oracle/12.1/client64/lib/libclntsh.so.12.1(ttcdrv+0x4a7) [0x7fe1ae0f79a7]
18121:20140609:151911.645 21: /usr/lib/oracle/12.1/client64/lib/libclntsh.so.12.1(nioqwa+0x49) [0x7fe1ae0e7a59]
18121:20140609:151911.645 20: /usr/lib/oracle/12.1/client64/lib/libclntsh.so.12.1(upirtrc+0x59f) [0x7fe1ae0d36cf]
18121:20140609:151911.645 19: /usr/lib/oracle/12.1/client64/lib/libclntsh.so.12.1(kpurcsc+0x66) [0x7fe1ae0de1a6]
18121:20140609:151911.645 18: /usr/lib/oracle/12.1/client64/lib/libclntsh.so.12.1(kpuexec+0x2af3) [0x7fe1ae0d6eb3]
18121:20140609:151911.645 17: /usr/lib/oracle/12.1/client64/lib/libclntsh.so.12.1(OCIStmtExecute+0x29) [0x7fe1ae0d4049]
18121:20140609:151911.645 16: /home/asaveljevs/projects/zabbix-bin-2.2/sbin/zabbix_server: poller #1 [got 50 values in 0.039941 sec, getting values]() [0x4e4940]
18121:20140609:151911.645 15: /home/asaveljevs/projects/zabbix-bin-2.2/sbin/zabbix_server: poller #1 [got 50 values in 0.039941 sec, getting values](zbx_db_statement_execute+0x4c) [0x4e4d25]
18121:20140609:151911.645 14: /home/asaveljevs/projects/zabbix-bin-2.2/sbin/zabbix_server: poller #1 [got 50 values in 0.039941 sec, getting values](DBstatement_execute+0x12) [0x4c2147]
18121:20140609:151911.645 13: /home/asaveljevs/projects/zabbix-bin-2.2/sbin/zabbix_server: poller #1 [got 50 values in 0.039941 sec, getting values](zbx_db_insert_execute+0x3e3) [0x4c5e5a]
18121:20140609:151911.645 12: /home/asaveljevs/projects/zabbix-bin-2.2/sbin/zabbix_server: poller #1 [got 50 values in 0.039941 sec, getting values]() [0x4b534d]
18121:20140609:151911.645 11: /home/asaveljevs/projects/zabbix-bin-2.2/sbin/zabbix_server: poller #1 [got 50 values in 0.039941 sec, getting values](DBdelete_items+0x129) [0x4b59d1]
18121:20140609:151911.645 10: /home/asaveljevs/projects/zabbix-bin-2.2/sbin/zabbix_server: poller #1 [got 50 values in 0.039941 sec, getting values]() [0x4d3722]
18121:20140609:151911.645 9: /home/asaveljevs/projects/zabbix-bin-2.2/sbin/zabbix_server: poller #1 [got 50 values in 0.039941 sec, getting values](lld_update_items+0x821) [0x4d40d1]
18121:20140609:151911.645 8: /home/asaveljevs/projects/zabbix-bin-2.2/sbin/zabbix_server: poller #1 [got 50 values in 0.039941 sec, getting values](lld_process_discovery_rule+0x394) [0x4cf194]
18121:20140609:151911.645 7: /home/asaveljevs/projects/zabbix-bin-2.2/sbin/zabbix_server: poller #1 [got 50 values in 0.039941 sec, getting values](dc_add_history+0x94) [0x46568b]
18121:20140609:151911.645 6: /home/asaveljevs/projects/zabbix-bin-2.2/sbin/zabbix_server: poller #1 [got 50 values in 0.039941 sec, getting values]() [0x42a308]
18121:20140609:151911.645 5: /home/asaveljevs/projects/zabbix-bin-2.2/sbin/zabbix_server: poller #1 [got 50 values in 0.039941 sec, getting values](main_poller_loop+0xd8) [0x42aa37]
18121:20140609:151911.645 4: /home/asaveljevs/projects/zabbix-bin-2.2/sbin/zabbix_server: poller #1 [got 50 values in 0.039941 sec, getting values](MAIN_ZABBIX_ENTRY+0x69b) [0x4197dd]
18121:20140609:151911.645 3: /home/asaveljevs/projects/zabbix-bin-2.2/sbin/zabbix_server: poller #1 [got 50 values in 0.039941 sec, getting values](daemon_start+0x296) [0x4824d3]
18121:20140609:151911.645 2: /home/asaveljevs/projects/zabbix-bin-2.2/sbin/zabbix_server: poller #1 [got 50 values in 0.039941 sec, getting values](main+0x1ff) [0x419140]
18121:20140609:151911.645 1: /lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0xf5) [0x7fe1a95ddb45]
18121:20140609:151911.645 0: /home/asaveljevs/projects/zabbix-bin-2.2/sbin/zabbix_server: poller #1 [got 50 values in 0.039941 sec, getting values]() [0x412a49]
Comment by Andris Zeila [ 2014 Jun 10 ]

Fixed in development branch svn://svn.zabbix.com/branches/dev/ZBX-8325

Comment by Andris Zeila [ 2014 Jun 11 ]

Released in:
pre-2.2.4rc2 r46407
pre-2.3.2 r46408

Generated at Fri Apr 26 19:23:28 EEST 2024 using Jira 9.12.4#9120004-sha1:625303b708afdb767e17cb2838290c41888e9ff0.