-
Incident report
-
Resolution: Fixed
-
Critical
-
3.2.3
-
None
-
Linux s3043 3.16.0-4-amd64 #1 SMP Debian 3.16.36-1+deb8u2 (2016-10-19) x86_64 GNU/Linux
zabbix_server (Zabbix) 3.2.3
unixODBC 2.3.1
-
Sprint 3, Sprint 4
-
2
During ODBC Discovery or regular Database item to Informix database makes poller stall.
In 3.0.4 this works without any issues on same unixODBC and driver versions.
Discovery query:
select distinct(p.recv_code) as ID, s.recv_name as NAME, sum(p.paymsum) as SUMM from paym p join paym_subj s on p.recv_code = s.recv_code where reg_date >= current - 1 units day and p.state in (3,5) group by ID,NAME having sum(p.paymsum) > 2000000;
Debug for poller:
7132:20170210:113425.048 __zbx_zbx_setproctitle() title:'poller #6 [got 1 values in 0.000341 sec, getting values]' 7132:20170210:113425.048 In get_values() 7132:20170210:113425.048 In DCconfig_get_poller_items() poller_type:0 7132:20170210:113425.048 End of DCconfig_get_poller_items():1 7132:20170210:113425.048 In substitute_key_macros() data:'db.odbc.discovery[providers,ok_arch2]' 7132:20170210:113425.048 End of substitute_key_macros():SUCCEED data:'db.odbc.discovery[providers,ok_arch2]' 7132:20170210:113425.048 In substitute_simple_macros() data:'select distinct(p.recv_code) as ID, s.recv_name as NAME, sum(p.paymsum) as SUMM from paym p join paym_subj s on p.recv_code = s.recv_code where reg_date >= current - 1 units day and p.state in (3,5) group by ID,NAME having sum(p.paymsum) > 2000000;' 7132:20170210:113425.048 In substitute_simple_macros() data:EMPTY 7132:20170210:113425.048 In substitute_simple_macros() data:EMPTY 7132:20170210:113425.048 In get_value() key:'db.odbc.discovery[providers,ok_arch2]' 7132:20170210:113425.048 In get_value_db() key_orig:'db.odbc.discovery[providers,ok_arch2]' 7132:20170210:113425.048 In db_odbc_discovery() query:'select distinct(p.recv_code) as ID, s.recv_name as NAME, sum(p.paymsum) as SUMM from paym p join paym_subj s on p.recv_code = s.recv_code where reg_date >= current - 1 units day and p.state in (3,5) group by ID,NAME having sum(p.paymsum) > 2000000;' 7132:20170210:113425.048 In odbc_DBconnect() db_dsn:'ok_arch2' user:'' 7132:20170210:113425.135 odbc_DBconnect() connected to Informix(11.70.0000 FC6) using iclis09b.so(4.10.F ) 7132:20170210:113425.135 End of odbc_DBconnect():SUCCEED 7132:20170210:113425.135 In odbc_DBselect() query:'select distinct(p.recv_code) as ID, s.recv_name as NAME, sum(p.paymsum) as SUMM from paym p join paym_subj s on p.recv_code = s.recv_code where reg_date >= current - 1 units day and p.state in (3,5) group by ID,NAME having sum(p.paymsum) > 2000000;' 7132:20170210:113428.494 odbc_DBselect() selected 3 columns 7132:20170210:113428.494 End of odbc_DBselect() 7132:20170210:113428.494 db_odbc_discovery() column[1]:'id' 7132:20170210:113428.494 db_odbc_discovery() column[2]:'name' 7132:20170210:113428.494 db_odbc_discovery() column[3]:'summ' 7132:20170210:113428.494 In odbc_DBfetch()
After odbc_DBfetch() no activity from this poller.
Strace:
1486988910.217319 recvfrom(9, "\0\f", 4096, 0, NULL, NULL) = 2 1486988910.692321 sendto(9, "\0\4\0\0\0\t\0\0\177\377\0\0\0\f", 14, 0, NULL, 0) = 14 1486988910.692581 recvfrom(9, "\0\16\0\0\0\0\0\21\304\fMT54\0\0\0\0\0\0\0\0\0\0\0\0\0\17\0\20\0\0\0\1\0\0\0\0\0\0\0\0\0007\0\0\0\1\0\0\0\3\0\f", 4096, 0, NULL, NULL) = 54 1486988910.698311 brk(0x10b1000) = 0x10b1000 1486988910.713155 mmap(NULL, 790528, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7fcfe8263000 1486988910.732924 mremap(0x7fcfe8263000, 790528, 1576960, MREMAP_MAYMOVE) = 0x7fcfe80e2000 1486988910.770646 mremap(0x7fcfe80e2000, 1576960, 3149824, MREMAP_MAYMOVE) = 0x7fcfe7de1000 1486988910.847196 mremap(0x7fcfe7de1000, 3149824, 6295552, MREMAP_MAYMOVE) = 0x7fcfe77e0000 1486988910.998709 mremap(0x7fcfe77e0000, 6295552, 12587008, MREMAP_MAYMOVE) = 0x7fcfe6bdf000 1486988911.308036 mremap(0x7fcfe6bdf000, 12587008, 25169920, MREMAP_MAYMOVE) = 0x7fcfe53de000 1486988911.915930 mremap(0x7fcfe53de000, 25169920, 50335744, MREMAP_MAYMOVE) = 0x7fcfe23dd000 1486988913.113471 mremap(0x7fcfe23dd000, 50335744, 100667392, MREMAP_MAYMOVE) = 0x7fcfdc3dc000 1486988915.473658 mremap(0x7fcfdc3dc000, 100667392, 201330688, MREMAP_MAYMOVE) = 0x7fcfd03db000 1486988919.934879 mremap(0x7fcfd03db000, 201330688, 402657280, MREMAP_MAYMOVE) = 0x7fcfb83da000 1486988928.679154 mremap(0x7fcfb83da000, 402657280, 805310464, MREMAP_MAYMOVE) = 0x7fcf883d9000 1486988938.769833 --- SIGALRM {si_signo=SIGALRM, si_code=SI_KERNEL} --- 1486988938.769986 rt_sigreturn() = 11 1486988945.954527 mremap(0x7fcf883d9000, 805310464, 1610616832, MREMAP_MAYMOVE) = 0x7fcf283d8000 1486988979.741868 mremap(0x7fcf283d8000, 1610616832, 3221229568, MREMAP_MAYMOVE) = 0x7fce683d7000 1486989046.999187 mremap(0x7fce683d7000, 3221229568, 6442455040, MREMAP_MAYMOVE) = 0x7fcce83d6000 1486989181.357365 mremap(0x7fcce83d6000, 6442455040, 12884905984, MREMAP_MAYMOVE) = 0x7fc9e83d5000
After some time poller crashes.
1486732368.634228 --- SIGTERM {si_signo=SIGTERM, si_code=SI_USER, si_pid=8666, si_uid=941} --- 1486732368.635903 rt_sigprocmask(SIG_BLOCK, [INT USR1 TERM], [TERM], 8) = 0 1486732368.636080 semop(1212418, {{0, -1, SEM_UNDO}}, 1) = 0 1486732368.636275 stat("/var/log/zabbix/zabbix_server.log", {st_mode=S_IFREG|0664, st_size=84325091, ...}) = 0 1486732368.636474 open("/var/log/zabbix/zabbix_server.log", O_RDWR|O_CREAT|O_APPEND, 0666) = 9 1486732368.636595 fstat(9, {st_mode=S_IFREG|0664, st_size=84325091, ...}) = 0 1486732368.636642 mmap(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7f13777f3000 1486732368.636721 write(9, " 8679:20170210:161248.636 Got signal [signal:15(SIGTERM),sender_pid:8666,sender_uid:941,reason:0]. Exiting ...\n", 112) = 112 1486732368.636780 close(9) = 0 1486732368.636824 munmap(0x7f13777f3000, 4096) = 0 1486732368.636869 semop(1212418, {{0, 1, SEM_UNDO}}, 1) = 0 1486732368.636918 rt_sigprocmask(SIG_SETMASK, [TERM], NULL, 8) = 0 1486732368.637430 close(3) = 0 1486732368.716446 exit_group(1) = ? 1486732369.765764 +++ exited with 1 +++
unixODBC 2.3.1 IBM Informix CSDK Version 4.10, IBM Informix-ESQL Version 4.10.FC7DE p.recv_code INTEGER s.recv_name VARCHAR p.paymsum DECIMAL