[ZBX-7729] Crash zabbix server on solaris 10 after start. Created: 2014 Jan 30  Updated: 2017 May 30  Resolved: 2014 May 10

Status: Closed
Project: ZABBIX BUGS AND ISSUES
Component/s: Server (S)
Affects Version/s: 2.2.1
Fix Version/s: 2.2.4, 2.3.0

Type: Incident report Priority: Blocker
Reporter: Sergey Berdnikov Assignee: Unassigned
Resolution: Fixed Votes: 1
Labels: crash, server, solaris
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified
Environment:

uname -a
SunOS test 5.10 Generic_150400-03 sun4v sparc SUNW,SPARC-Enterprise-T5120

gcc -v
Reading specs from /usr/sfw/lib/gcc/sparc-sun-solaris2.10/3.4.3/specs
Configured with: /sfw10/builds/build/sfw10-patch/usr/src/cmd/gcc/gcc-3.4.3/configure --prefix=/usr/sfw --with-as=/usr/ccs/bin/as --without-gnu-as --with-ld=/usr/ccs/bin/ld --without-gnu-ld --enable-languages=c,c++ --enable-shared
Thread model: posix
gcc version 3.4.3 (csl-sol210-3_4-branch+sol_rpath)

Build options:

./configure\
--prefix=/opt/szabbix\
--enable-server\
--with-postgresql=/pub/site/opt/bin/pg_config\
--with-net-snmp=/pub/site/opt/bin/net-snmp-config\
--with-libcurl=/pub/site/opt/bin/curl-config\
--with-libxml2\

Attachments: File zabbix-7729-fix.diff     File zb.output.bz2    
Issue Links:
Duplicate
is duplicated by ZBX-7731 One child process died (PID:36456,exi... Closed

 Description   

Zabbix server crashed after start with error:

2608:20140130:134416.848 Starting Zabbix Server. Zabbix 2.2.1 (revision 40808).
2608:20140130:134416.849 ****** Enabled features ******
2608:20140130:134416.849 SNMP monitoring: YES
2608:20140130:134416.850 IPMI monitoring: NO
2608:20140130:134416.850 WEB monitoring: YES
2608:20140130:134416.850 VMware monitoring: YES
2608:20140130:134416.850 Jabber notifications: NO
2608:20140130:134416.850 Ez Texting notifications: YES
2608:20140130:134416.850 ODBC: NO
2608:20140130:134416.850 SSH2 support: NO
2608:20140130:134416.850 IPv6 support: NO
2608:20140130:134416.850 ******************************
2608:20140130:134416.851 using configuration file: /pub/site/szabbix/etc/zabbix_server.conf
2608:20140130:134417.001 current database version (mandatory/optional): 02020000/02020000
2608:20140130:134417.001 required mandatory version: 02020000
2608:20140130:134417.224 Got signal [signal:10(SIGBUS),reason:1,refaddr:ffffffff7744c33c]. Crashing ...
2608:20140130:134417.224 ====== Fatal information: ======
2608:20140130:134417.224 program counter not available for this architecture
2608:20140130:134417.225 === Registers: ===
2608:20140130:134417.225 register dump not available for this architecture
2608:20140130:134417.225 === Backtrace: ===
2608:20140130:134417.225 backtrace not available for this platform
2608:20140130:134417.225 === Memory map: ===
2608:20140130:134417.225 memory map not available for this platform
2608:20140130:134417.225 ================================

 Comments   
Comment by Sergey Berdnikov [ 2014 Jan 30 ]

objdump -DSswx zabbix_server > file

Comment by Aleksandrs Saveljevs [ 2014 Jan 31 ]

Since program counter is not available for this architecture (i.e., no information is provided about the instruction on which the server crashed), the disassembly is not particularly useful.

Is the crash reliably reproducible? Would it be possible to start the server with DebugLevel=4?

Comment by Oleksii Zagorskyi [ 2014 Jan 31 ]

And as many of crashes fixed already after 2.2.1 - would be good to retry with latest pre-2.2.2rc1
http://www.zabbix.com/developers.php

Comment by Sergey Berdnikov [ 2014 Feb 03 ]

Yer, bug is reliably reproducible. I can't start zabbix server on this host)
Log with DebugLevel=4:

 29272:20140203:092859.006 Starting Zabbix Server. Zabbix 2.2.1 (revision 40808).
 29272:20140203:092859.007 ****** Enabled features ******
 29272:20140203:092859.007 SNMP monitoring:           YES
 29272:20140203:092859.007 IPMI monitoring:            NO
 29272:20140203:092859.007 WEB monitoring:            YES
 29272:20140203:092859.007 VMware monitoring:         YES
 29272:20140203:092859.008 Jabber notifications:       NO
 29272:20140203:092859.008 Ez Texting notifications:  YES
 29272:20140203:092859.008 ODBC:                       NO
 29272:20140203:092859.008 SSH2 support:               NO
 29272:20140203:092859.008 IPv6 support:               NO
 29272:20140203:092859.008 ******************************
 29272:20140203:092859.008 using configuration file: /pub/site/szabbix/etc/zabbix_server.conf
 29272:20140203:092859.008 In load_modules()
 29272:20140203:092859.009 End of load_modules():SUCCEED
 29272:20140203:092859.009 In init_database_cache()
 29272:20140203:092859.009 In zbx_mem_required_size() size:0 chunks_num:4 descr:'history cache' param:'HistoryCacheSize'
 29272:20140203:092859.009 End of zbx_mem_required_size() size:543
 29272:20140203:092859.009 In zbx_mem_create() descr:'history cache' param:'HistoryCacheSize' size:8388575
 29272:20140203:092859.010 zbx_shmget() removing existing shm_id:1778385018
 29272:20140203:092859.022 valid user addresses: [ffffffff79800170, ffffffff79ffffd0] total size: 8388192
 29272:20140203:092859.022 End of zbx_mem_create()
 29272:20140203:092859.022 In zbx_mem_required_size() size:0 chunks_num:1 descr:'history text cache' param:'HistoryTextCacheSize'
 29272:20140203:092859.022 End of zbx_mem_required_size() size:435
 29272:20140203:092859.023 In zbx_mem_create() descr:'history text cache' param:'HistoryTextCacheSize' size:16777216
 29272:20140203:092859.023 zbx_shmget() removing existing shm_id:1778385019
 29272:20140203:092859.035 valid user addresses: [ffffffff78400178, ffffffff793ffff0] total size: 16776824
 29272:20140203:092859.035 End of zbx_mem_create()
 29272:20140203:092859.035 In init_trend_cache()
 29272:20140203:092859.035 In zbx_mem_required_size() size:0 chunks_num:1 descr:'trend cache' param:'TrendCacheSize'
 29272:20140203:092859.035 End of zbx_mem_required_size() size:422
 29272:20140203:092859.035 In zbx_mem_create() descr:'trend cache' param:'TrendCacheSize' size:4194304
 29272:20140203:092859.036 zbx_shmget() removing existing shm_id:1778385020
 29272:20140203:092859.043 valid user addresses: [ffffffff77c00170, ffffffff77fffff0] total size: 4193920
 29272:20140203:092859.043 End of zbx_mem_create()
 29272:20140203:092859.043 End of init_trend_cache()
 29272:20140203:092859.044 End of init_database_cache()
 29272:20140203:092859.044 In init_configuration_cache() size:8388608
 29272:20140203:092859.044 In zbx_mem_create() descr:'configuration cache' param:'CacheSize' size:7130317
 29272:20140203:092859.044 zbx_shmget() removing existing shm_id:1778385021
 29272:20140203:092859.052 valid user addresses: [ffffffff77400170, ffffffff77acccc0] total size: 7129936
 29272:20140203:092859.052 End of zbx_mem_create()
 29272:20140203:092859.052 In zbx_strpool_create()
 29272:20140203:092859.052 In zbx_mem_create() descr:'string pool' param:'CacheSize' size:1258291
 29272:20140203:092859.053 zbx_shmget() removing existing shm_id:1778385022
 29272:20140203:092859.053 valid user addresses: [ffffffff79600168, ffffffff79733328] total size: 1257920
 29272:20140203:092859.053 End of zbx_mem_create()
 29272:20140203:092859.053 End of zbx_strpool_create()
 29272:20140203:092859.054 End of init_configuration_cache()
 29272:20140203:092859.054 In init_selfmon_collector()
 29272:20140203:092859.054 init_selfmon_collector() size:7056
 29272:20140203:092859.054 zbx_shmget() removing existing shm_id:1778385023
 29272:20140203:092859.055 End of init_selfmon_collector() collector:ffffffff7b500000
 29272:20140203:092859.055 In zbx_vc_init()
 29272:20140203:092859.055 In zbx_mem_required_size() size:0 chunks_num:1 descr:'value cache size' param:'ValueCacheSize'
 29272:20140203:092859.055 End of zbx_mem_required_size() size:427
 29272:20140203:092859.055 In zbx_mem_create() descr:'value cache size' param:'ValueCacheSize' size:8388608
 29272:20140203:092859.055 zbx_shmget() removing existing shm_id:1795162112
 29272:20140203:092859.067 valid user addresses: [ffffffff76800170, ffffffff76fffff0] total size: 8388224
 29272:20140203:092859.067 End of zbx_mem_create()
 29272:20140203:092859.067 End of zbx_vc_init()
 29272:20140203:092859.068 In DBcheck_version()
 29272:20140203:092859.068 In DBconnect() flag:0
 29272:20140203:092859.099 query [txnlev:0] [select oid from pg_type where typname='bytea']
 29272:20140203:092859.106 PostgreSQL Server version: 90302
 29272:20140203:092859.106 query [txnlev:0] [set escape_string_warning to off]
 29272:20140203:092859.107 query [txnlev:0] [show standard_conforming_strings]
 29272:20140203:092859.107 query [txnlev:0] [set bytea_output=escape]
 29272:20140203:092859.108 End of DBconnect():0
 29272:20140203:092859.108 query [txnlev:0] [select 1 from information_schema.tables where table_name='dbversion' and table_schema='public']
 29272:20140203:092859.177 query [txnlev:0] [select mandatory,optional from dbversion]
 29272:20140203:092859.180 current database version (mandatory/optional): 02020000/02020000
 29272:20140203:092859.180 required mandatory version: 02020000
 29272:20140203:092859.181 End of DBcheck_version():SUCCEED
 29272:20140203:092859.181 In DBconnect() flag:0
 29272:20140203:092859.238 query [txnlev:0] [select oid from pg_type where typname='bytea']
 29272:20140203:092859.245 PostgreSQL Server version: 90302
 29272:20140203:092859.245 query [txnlev:0] [set escape_string_warning to off]
 29272:20140203:092859.246 query [txnlev:0] [show standard_conforming_strings]
 29272:20140203:092859.247 query [txnlev:0] [set bytea_output=escape]
 29272:20140203:092859.247 End of DBconnect():0
 29272:20140203:092859.247 In DCload_config()
 29272:20140203:092859.247 query [txnlev:0] [select refresh_unsupported,discovery_groupid,snmptrap_logging,severity_name_0,severity_name_1,severity_name_2,severity_name_3,severity_name_4,severity_name_5,hk_events_mode,hk_events_trigger,hk_events_internal,hk_events_discovery,hk_events_autoreg,hk_services_mode,hk_services,hk_audit_mode,hk_audit,hk_sessions_mode,hk_sessions,hk_history_mode,hk_history_global,hk_history,hk_trends_mode,hk_trends_global,hk_trends from config]
 29272:20140203:092859.266 In DCsync_config()
 29272:20140203:092859.266 End of DCsync_config()
 29272:20140203:092859.266 End of DCload_config()
 29272:20140203:092859.266 In DCsync_configuration()
 29272:20140203:092859.267 query [txnlev:0] [select refresh_unsupported,discovery_groupid,snmptrap_logging,severity_name_0,severity_name_1,severity_name_2,severity_name_3,severity_name_4,severity_name_5,hk_events_mode,hk_events_trigger,hk_events_internal,hk_events_discovery,hk_events_autoreg,hk_services_mode,hk_services,hk_audit_mode,hk_audit,hk_sessions_mode,hk_sessions,hk_history_mode,hk_history_global,hk_history,hk_trends_mode,hk_trends_global,hk_trends from config]
 29272:20140203:092859.270 query [txnlev:0] [select i.itemid,i.hostid,h.proxy_hostid,i.type,i.data_type,i.value_type,i.key_,i.snmp_community,i.snmp_oid,i.port,i.snmpv3_securityname,i.snmpv3_securitylevel,i.snmpv3_authpassphrase,i.snmpv3_privpassphrase,i.ipmi_sensor,i.delay,i.delay_flex,i.trapper_hosts,i.logtimefmt,i.params,i.state,i.authtype,i.username,i.password,i.publickey,i.privatekey,i.flags,i.interfaceid,i.snmpv3_authprotocol,i.snmpv3_privprotocol,i.snmpv3_contextname,i.lastlogsize,i.mtime from items i,hosts h where i.hostid=h.hostid and h.status=0 and i.status=0]
 29272:20140203:092859.350 query [txnlev:0] [select distinct t.triggerid,t.description,t.expression,t.error,t.priority,t.type,t.value,t.state,t.lastchange from hosts h,items i,functions f,triggers t where h.hostid=i.hostid and i.itemid=f.itemid and f.triggerid=t.triggerid and h.status=0 and i.status=0 and t.status=0 and t.flags<>2]
 29272:20140203:092859.402 query [txnlev:0] [select d.triggerid_down,d.triggerid_up from trigger_depends d order by d.triggerid_down]
 29272:20140203:092859.407 query [txnlev:0] [select i.itemid,f.functionid,f.function,f.parameter,t.triggerid from hosts h,items i,functions f,triggers t where h.hostid=i.hostid and i.itemid=f.itemid and f.triggerid=t.triggerid and h.status=0 and i.status=0 and t.status=0 and t.flags<>2]
 29272:20140203:092859.434 query [txnlev:0] [select hostid,proxy_hostid,host,ipmi_authtype,ipmi_privilege,ipmi_username,ipmi_password,maintenance_status,maintenance_type,maintenance_from,errors_from,available,disable_until,snmp_errors_from,snmp_available,snmp_disable_until,ipmi_errors_from,ipmi_available,ipmi_disable_until,jmx_errors_from,jmx_available,jmx_disable_until,status,name from hosts where status in (0,5,6) and flags<>2]
 29272:20140203:092859.441 query [txnlev:0] [select hostid,templateid from hosts_templates order by hostid,templateid]
 29272:20140203:092859.445 query [txnlev:0] [select globalmacroid,macro,value from globalmacro]
 29272:20140203:092859.449 query [txnlev:0] [select hostmacroid,hostid,macro,value from hostmacro]
 29272:20140203:092859.452 query [txnlev:0] [select interfaceid,hostid,type,main,useip,ip,dns,port from interface]
 29272:20140203:092859.458 query [txnlev:0] [select r.name,e.expressionid,e.expression,e.expression_type,e.exp_delimiter,e.case_sensitive from regexps r,expressions e where r.regexpid=e.regexpid]
 29272:20140203:092859.467 In DCsync_config()
 29272:20140203:092859.467 End of DCsync_config()
 29272:20140203:092859.467 In DCsync_items()
gs,i.interfaceid,i.snmpv3_authprotocol,i.snmpv3_privprotocol,i.snmpv3_contextname,i.lastlogsize,i.mtime from items i,hosts h where i.hostid=h.hostid and h.status=0 and i.status=0]
 29272:20140203:092859.350 query [txnlev:0] [select distinct t.triggerid,t.description,t.expression,t.error,t.priority,t.type,t.value,t.state,t.lastchange from hosts h,items i,functions f,triggers t
 where h.hostid=i.hostid and i.itemid=f.itemid and f.triggerid=t.triggerid and h.status=0 and i.status=0 and t.status=0 and t.flags<>2]
 29272:20140203:092859.402 query [txnlev:0] [select d.triggerid_down,d.triggerid_up from trigger_depends d order by d.triggerid_down]
 29272:20140203:092859.407 query [txnlev:0] [select i.itemid,f.functionid,f.function,f.parameter,t.triggerid from hosts h,items i,functions f,triggers t where h.hostid=i.hostid and i.itemid=f.itemid
 and f.triggerid=t.triggerid and h.status=0 and i.status=0 and t.status=0 and t.flags<>2]
 29272:20140203:092859.434 query [txnlev:0] [select hostid,proxy_hostid,host,ipmi_authtype,ipmi_privilege,ipmi_username,ipmi_password,maintenance_status,maintenance_type,maintenance_from,errors_from
,available,disable_until,snmp_errors_from,snmp_available,snmp_disable_until,ipmi_errors_from,ipmi_available,ipmi_disable_until,jmx_errors_from,jmx_available,jmx_disable_until,status,name from hosts 
where status in (0,5,6) and flags<>2]
 29272:20140203:092859.441 query [txnlev:0] [select hostid,templateid from hosts_templates order by hostid,templateid]
 29272:20140203:092859.445 query [txnlev:0] [select globalmacroid,macro,value from globalmacro]
 29272:20140203:092859.449 query [txnlev:0] [select hostmacroid,hostid,macro,value from hostmacro]
 29272:20140203:092859.452 query [txnlev:0] [select interfaceid,hostid,type,main,useip,ip,dns,port from interface]
 29272:20140203:092859.458 query [txnlev:0] [select r.name,e.expressionid,e.expression,e.expression_type,e.exp_delimiter,e.case_sensitive from regexps r,expressions e where r.regexpid=e.regexpid]
 29272:20140203:092859.467 In DCsync_config()
 29272:20140203:092859.467 End of DCsync_config()
 29272:20140203:092859.467 In DCsync_items()
 29272:20140203:092859.467 End of DCsync_items()
 29272:20140203:092859.468 In DCsync_triggers()
 29272:20140203:092859.468 End of DCsync_triggers()
 29272:20140203:092859.468 In DCsync_trigdeps()
 29272:20140203:092859.468 Got signal [signal:10(SIGBUS),reason:1,refaddr:ffffffff7744c33c]. Crashing ...
 29272:20140203:092859.468 ====== Fatal information: ======
 29272:20140203:092859.468 program counter not available for this architecture
 29272:20140203:092859.468 === Registers: ===
 29272:20140203:092859.469 register dump not available for this architecture
 29272:20140203:092859.469 === Backtrace: ===
 29272:20140203:092859.469 backtrace not available for this platform
 29272:20140203:092859.469 === Memory map: ===
 29272:20140203:092859.469 memory map not available for this platform
 29272:20140203:092859.469 ================================
Comment by Sergey Berdnikov [ 2014 Feb 03 ]

Version "Pre-2.2.2rc1 (stable) 42098 02 Feb, 2014" does not solve my problem.

  9333:20140203:095150.562 Starting Zabbix Server. Zabbix 2.2.2rc1 (revision 42098).
  9333:20140203:095150.563 ****** Enabled features ******
  9333:20140203:095150.563 SNMP monitoring:           YES
  9333:20140203:095150.563 IPMI monitoring:            NO
  9333:20140203:095150.563 WEB monitoring:            YES
  9333:20140203:095150.563 VMware monitoring:         YES
  9333:20140203:095150.563 Jabber notifications:       NO
  9333:20140203:095150.564 Ez Texting notifications:  YES
  9333:20140203:095150.564 ODBC:                       NO
  9333:20140203:095150.564 SSH2 support:               NO
  9333:20140203:095150.564 IPv6 support:               NO
  9333:20140203:095150.564 ******************************
  9333:20140203:095150.564 using configuration file: /pub/site/szabbix/etc/zabbix_server.conf
  9333:20140203:095150.564 In load_modules()
  9333:20140203:095150.564 End of load_modules():SUCCEED
  9333:20140203:095150.565 In init_database_cache()
  9333:20140203:095150.565 In zbx_mem_required_size() size:0 chunks_num:4 descr:'history cache' param:'HistoryCacheSize'
  9333:20140203:095150.565 End of zbx_mem_required_size() size:543
  9333:20140203:095150.565 In zbx_mem_create() descr:'history cache' param:'HistoryCacheSize' size:8388575
  9333:20140203:095150.566 zbx_shmget() removing existing shm_id:1795162144
  9333:20140203:095150.577 valid user addresses: [ffffffff79800170, ffffffff79ffffd0] total size: 8388192
  9333:20140203:095150.578 End of zbx_mem_create()
  9333:20140203:095150.578 In zbx_mem_required_size() size:0 chunks_num:1 descr:'history text cache' param:'HistoryTextCacheSize'
  9333:20140203:095150.578 End of zbx_mem_required_size() size:435
  9333:20140203:095150.578 In zbx_mem_create() descr:'history text cache' param:'HistoryTextCacheSize' size:16777216
  9333:20140203:095150.578 zbx_shmget() removing existing shm_id:1795162145
  9333:20140203:095150.590 valid user addresses: [ffffffff78400178, ffffffff793ffff0] total size: 16776824
  9333:20140203:095150.590 End of zbx_mem_create()
  9333:20140203:095150.590 In init_trend_cache()
  9333:20140203:095150.591 In zbx_mem_required_size() size:0 chunks_num:1 descr:'trend cache' param:'TrendCacheSize'
  9333:20140203:095150.591 End of zbx_mem_required_size() size:422
  9333:20140203:095150.591 In zbx_mem_create() descr:'trend cache' param:'TrendCacheSize' size:4194304
  9333:20140203:095150.591 zbx_shmget() removing existing shm_id:1795162153
  9333:20140203:095150.598 valid user addresses: [ffffffff77c00170, ffffffff77fffff0] total size: 4193920
  9333:20140203:095150.598 End of zbx_mem_create()
  9333:20140203:095150.599 End of init_trend_cache()
  9333:20140203:095150.599 End of init_database_cache()
  9333:20140203:095150.599 In init_configuration_cache() size:8388608
  9333:20140203:095150.600 In zbx_mem_create() descr:'configuration cache' param:'CacheSize' size:7130317
  9333:20140203:095150.600 zbx_shmget() removing existing shm_id:1795162154
  9333:20140203:095150.607 valid user addresses: [ffffffff77400170, ffffffff77acccc0] total size: 7129936
  9333:20140203:095150.607 End of zbx_mem_create()
  9333:20140203:095150.608 In zbx_strpool_create()
  9333:20140203:095150.608 In zbx_mem_create() descr:'string pool' param:'CacheSize' size:1258291
  9333:20140203:095150.608 zbx_shmget() removing existing shm_id:1795162155
  9333:20140203:095150.609 valid user addresses: [ffffffff79600168, ffffffff79733328] total size: 1257920
  9333:20140203:095150.609 End of zbx_mem_create()
  9333:20140203:095150.609 End of zbx_strpool_create()
  9333:20140203:095150.609 End of init_configuration_cache()
  9333:20140203:095150.609 In init_selfmon_collector()
  9333:20140203:095150.609 init_selfmon_collector() size:7056
  9333:20140203:095150.610 zbx_shmget() removing existing shm_id:1795162156
  9333:20140203:095150.610 End of init_selfmon_collector() collector:ffffffff7b500000
  9333:20140203:095150.610 In zbx_vc_init()
  9333:20140203:095150.610 In zbx_mem_required_size() size:0 chunks_num:1 descr:'value cache size' param:'ValueCacheSize'
  9333:20140203:095150.611 End of zbx_mem_required_size() size:427
  9333:20140203:095150.611 In zbx_mem_create() descr:'value cache size' param:'ValueCacheSize' size:8388608
  9333:20140203:095150.611 zbx_shmget() removing existing shm_id:1795162157
  9333:20140203:095150.622 valid user addresses: [ffffffff76800170, ffffffff76fffff0] total size: 8388224
  9333:20140203:095150.622 End of zbx_mem_create()
  9333:20140203:095150.623 End of zbx_vc_init()
  9333:20140203:095150.623 In DBcheck_version()
  9333:20140203:095150.623 In DBconnect() flag:0
  9333:20140203:095150.705 query [txnlev:0] [select oid from pg_type where typname='bytea']
  9333:20140203:095150.712 PostgreSQL Server version: 90302
  9333:20140203:095150.712 query [txnlev:0] [set escape_string_warning to off]
  9333:20140203:095150.712 query [txnlev:0] [show standard_conforming_strings]
  9333:20140203:095150.713 query [txnlev:0] [set bytea_output=escape]
  9333:20140203:095150.714 End of DBconnect():0
  9333:20140203:095150.714 query [txnlev:0] [select 1 from information_schema.tables where table_name='dbversion' and table_schema='public']
  9333:20140203:095150.769 query [txnlev:0] [select mandatory,optional from dbversion]
  9333:20140203:095150.771 current database version (mandatory/optional): 02020000/02020000
  9333:20140203:095150.772 required mandatory version: 02020000
  9333:20140203:095150.772 End of DBcheck_version():SUCCEED
  9333:20140203:095150.772 In DBconnect() flag:0
  9333:20140203:095150.827 query [txnlev:0] [select oid from pg_type where typname='bytea']
  9333:20140203:095150.834 PostgreSQL Server version: 90302
  9333:20140203:095150.834 query [txnlev:0] [set escape_string_warning to off]
  9333:20140203:095150.835 query [txnlev:0] [show standard_conforming_strings]
  9333:20140203:095150.835 query [txnlev:0] [set bytea_output=escape]
  9333:20140203:095150.836 End of DBconnect():0
  9333:20140203:095150.836 In DCload_config()
  9333:20140203:095150.836 query [txnlev:0] [select refresh_unsupported,discovery_groupid,snmptrap_logging,severity_name_0,severity_name_1,severity_name_2,severity_name_3,severity_name_4,severity_name_5,hk_events_mode,hk_events_trigger,hk_events_internal,hk_events_discovery,hk_events_autoreg,hk_services_mode,hk_services,hk_audit_mode,hk_audit,hk_sessions_mode,hk_sessions,hk_history_mode,hk_history_global,hk_history,hk_trends_mode,hk_trends_global,hk_trends from config]
  9333:20140203:095150.854 In DCsync_config()
  9333:20140203:095150.855 End of DCsync_config()
  9333:20140203:095150.855 End of DCload_config()
  9333:20140203:095150.855 In DCsync_configuration()
  9333:20140203:095150.855 query [txnlev:0] [select refresh_unsupported,discovery_groupid,snmptrap_logging,severity_name_0,severity_name_1,severity_name_2,severity_name_3,severity_name_4,severity_name_5,hk_events_mode,hk_events_trigger,hk_events_internal,hk_events_discovery,hk_events_autoreg,hk_services_mode,hk_services,hk_audit_mode,hk_audit,hk_sessions_mode,hk_sessions,hk_history_mode,hk_history_global,hk_history,hk_trends_mode,hk_trends_global,hk_trends from config]
  9333:20140203:095150.858 query [txnlev:0] [select i.itemid,i.hostid,h.proxy_hostid,i.type,i.data_type,i.value_type,i.key_,i.snmp_community,i.snmp_oid,i.port,i.snmpv3_securityname,i.snmpv3_securitylevel,i.snmpv3_authpassphrase,i.snmpv3_privpassphrase,i.ipmi_sensor,i.delay,i.delay_flex,i.trapper_hosts,i.logtimefmt,i.params,i.state,i.authtype,i.username,i.password,i.publickey,i.privatekey,i.flags,i.interfaceid,i.snmpv3_authprotocol,i.snmpv3_privprotocol,i.snmpv3_contextname,i.lastlogsize,i.mtime from items i,hosts h where i.hostid=h.hostid and h.status=0 and i.status=0]
  9333:20140203:095150.911 query [txnlev:0] [select distinct t.triggerid,t.description,t.expression,t.error,t.priority,t.type,t.value,t.state,t.lastchange from hosts h,items i,functions f,triggers t where h.hostid=i.hostid and i.itemid=f.itemid and f.triggerid=t.triggerid and h.status=0 and i.status=0 and t.status=0 and t.flags<>2]
  9333:20140203:095150.960 query [txnlev:0] [select d.triggerid_down,d.triggerid_up from trigger_depends d order by d.triggerid_down]
  9333:20140203:095150.965 query [txnlev:0] [select i.itemid,f.functionid,f.function,f.parameter,t.triggerid from hosts h,items i,functions f,triggers t where h.hostid=i.hostid and i.itemid=f.itemid and f.triggerid=t.triggerid and h.status=0 and i.status=0 and t.status=0 and t.flags<>2]
  9333:20140203:095150.987 query [txnlev:0] [select hostid,proxy_hostid,host,ipmi_authtype,ipmi_privilege,ipmi_username,ipmi_password,maintenance_status,maintenance_type,maintenance_from,errors_from,available,disable_until,snmp_errors_from,snmp_available,snmp_disable_until,ipmi_errors_from,ipmi_available,ipmi_disable_until,jmx_errors_from,jmx_available,jmx_disable_until,status,name from hosts where status in (0,5,6) and flags<>2]
  9333:20140203:095150.993 query [txnlev:0] [select hostid,templateid from hosts_templates order by hostid,templateid]
  9333:20140203:095150.998 query [txnlev:0] [select globalmacroid,macro,value from globalmacro]
  9333:20140203:095151.001 query [txnlev:0] [select hostmacroid,hostid,macro,value from hostmacro]
  9333:20140203:095151.005 query [txnlev:0] [select interfaceid,hostid,type,main,useip,ip,dns,port from interface]
  9333:20140203:095151.011 query [txnlev:0] [select r.name,e.expressionid,e.expression,e.expression_type,e.exp_delimiter,e.case_sensitive from regexps r,expressions e where r.regexpid=e.regexpid]
  9333:20140203:095151.020 In DCsync_config()
  9333:20140203:095151.021 End of DCsync_config()
  9333:20140203:095151.021 In DCsync_items()
  9333:20140203:095151.021 End of DCsync_items()
  9333:20140203:095151.021 In DCsync_triggers()
  9333:20140203:095151.021 End of DCsync_triggers()
  9333:20140203:095151.021 In DCsync_trigdeps()
  9333:20140203:095151.021 Got signal [signal:10(SIGBUS),reason:1,refaddr:ffffffff7744c33c]. Crashing ...
  9333:20140203:095151.022 ====== Fatal information: ======
  9333:20140203:095151.022 program counter not available for this architecture
  9333:20140203:095151.022 === Registers: ===
  9333:20140203:095151.022 register dump not available for this architecture
  9333:20140203:095151.022 === Backtrace: ===
  9333:20140203:095151.022 backtrace not available for this platform
  9333:20140203:095151.022 === Memory map: ===
  9333:20140203:095151.022 memory map not available for this platform
  9333:20140203:095151.023 ================================
Comment by Sergey Berdnikov [ 2014 Feb 03 ]

Version Zabbix 2.0.10 works fine with no problems.

Comment by Pablo Ruiz [ 2014 May 01 ]

Patch fixing this issue.

Comment by Pablo Ruiz [ 2014 May 01 ]

Hi,

I've just hit this issue while upgrading a zabbix server from 1.8 to 2.2, and after a couple of debugging sessions, I've found the problem to be caused by an unaligned memory access at src/libs/zbxdbcache/dbconfig.c:921:

Program received signal SIGSEGV, Segmentation fault.
0x0000000100089070 in DCsync_items (result=0x1002d83e0) at dbconfig.c:921
921 item->hostid = hostid;

(gdb) list
916 update_index = 1;
917 }
918
919 /* store new information in item structure */
920
921 item->hostid = hostid;
922 item->type = (unsigned char)atoi(row[3]);
923 item->data_type = (unsigned char)atoi(row[4]);
924 DCstrpool_replace(found, &item->key, row[6]);
925 DCstrpool_replace(found, &item->port, row[9]);

At this point 'item' pointer points to an ZBX_DC_ITEM, which was obtained from a hashset hold by global 'config' struct. And after inspecting "include/zbxalgo.h" for hashset's structs declarations, I found ZBX_HASHSET_ENTRY_T typedef/struct, which has it's data member aligned on a 4 bytes boundary, but this (obviouslly) fails on 64bit SPARC, as this platform requires strict aligned memory access.

(gdb) p item
$44 = (ZBX_DC_ITEM *) 0xffffffff5804c3d4

(gdb) p 0xffffffff5804c3d4 % sizeof(uint64_t)
$45 = 4

Si, as a fix I've simple added some padding based on the current pointer length:

ZBX_HASHSET_ENTRY_T
{
  ZBX_HASHSET_ENTRY_T     *next;
  zbx_hash_t              hash;
  /*
   * We need to pad here to avoid unaligned access
   * to data on struct aling archs like SPARC.
   * 2014/05/01 -- pablo.ruiz _at_ gmail.com
   */
  char                    padding[sizeof(void*) - sizeof(zbx_hash_t)];
  char                    data[1];
};

I am attaching a patch fixing this issue.

Regards
Pablo

Comment by Nikolajs Agafonovs (Inactive) [ 2014 May 09 ]

Fix available in svn://svn.zabbix.com/branches/dev/ZBX-7729

Comment by Andris Zeila [ 2014 May 09 ]

I changed procfs.h include to mirror the changes made in ZBX-5243, please review r45227.

Comment by Andris Zeila [ 2014 May 09 ]

Added check to avoid declaring zero length padding array (which is universally supported).
Please review r45266

Moved AC_CHECK_SIZEOF macro after OS check because CGLAGS might be updated during OS check (Solaris) r45273

Comment by Andris Zeila [ 2014 May 09 ]

Successfully tested, please review the changes

nikolajs.agafonovs reviewed. CLOSED

Comment by Alexander Vladishev [ 2014 May 10 ]

The fix should be ported in 2.0 and 2.2.

nikolajs.agafonovs Proposed fix need to be applied to 2.2 and up.
2.0 do not contain this bug.

Fix for 2.2 is available in svn://svn.zabbix.com/branches/dev/ZBX-7729_22

Fixed in pre-2.3.0 (trunk) r45288

wiper tested dev branch for 2.2

nikolajs.agafonovs Fix is available in 2.2 r45521 CLOSED

Generated at Fri Apr 19 23:45:03 EEST 2024 using Jira 9.12.4#9120004-sha1:625303b708afdb767e17cb2838290c41888e9ff0.