Uploaded image for project: 'ZABBIX BUGS AND ISSUES'
  1. ZABBIX BUGS AND ISSUES
  2. ZBX-5502

Zabbix agent fails to start on armv5 host

    XMLWordPrintable

    Details

    • Type: Incident report
    • Status: Closed
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: 2.0.2
    • Fix Version/s: 2.0.3rc1
    • Component/s: Agent (G)
    • Labels:
      None
    • Environment:
      ArchLinux on Guruplug (Marvell arm5tel)

      Description

      zabbix_agentd fails to start with error
      "cannot attach shared memory for disk statistics collector:
      [22] Invalid argument"
      Note this doesn't look like the "cannot allocate shared mem" issue in the docs
      (here we have "cannot attach shared memory", not "allocate")

      Since it is unlikely that you have a similar machine (Raspberry Pi is armv6), I will be happy to follow instructions for debugging.

      Agent works fine with same OS (except kernel rev) on Trimslice with Tegra arm7l architecture wit hFloating Point hardware support.

      working system:-

      [[email protected] ~]# sysctl -a | grep shm
      kernel.shmmax = 33554432
      kernel.shmall = 2097152
      kernel.shmmni = 4096

      failing system:

      [[email protected] ~]$ sysctl -a |grep shm
      kernel.shmmax = 33554432
      kernel.shmall = 2097152
      kernel.shmmni = 4096
      kernel.shm_rmid_forced = 0

      working:
      Linux trimslice 2.6.38-6-ARCH #1 SMP PREEMPT Tue Jan 3 04:23:49 UTC 2012 armv7l

      failing:
      Linux alarm-0 3.1.10-12-ARCH #1 PREEMPT Tue Jul 17 04:51:16 UTC 2012 armv5tel GNU/Linux

      Since it appears to error on disk stats collector, the disk devices do seem to differ:

      [[email protected] ~]# cat /proc/diskstats
      1 0 ram0 0 0 0 0 0 0 0 0 0 0 0
      1 1 ram1 0 0 0 0 0 0 0 0 0 0 0
      <snip>
      1 14 ram14 0 0 0 0 0 0 0 0 0 0 0
      1 15 ram15 0 0 0 0 0 0 0 0 0 0 0
      7 0 loop0 0 0 0 0 0 0 0 0 0 0 0
      7 1 loop1 0 0 0 0 0 0 0 0 0 0 0
      <snip>
      7 6 loop6 0 0 0 0 0 0 0 0 0 0 0
      7 7 loop7 0 0 0 0 0 0 0 0 0 0 0
      179 0 mmcblk0 5623 56 159676 21870 11125 1468 110328 6899170 0 3451170 6921010
      179 1 mmcblk0p1 5548 56 159076 21500 11125 1468 110328 6899170 0 3451120 6920640

      mmcblk0p1 is the partition containing the root fs

      [[email protected] zabbix_agent]# cat /proc/diskstats
      1 0 ram0 0 0 0 0 0 0 0 0 0 0 0
      1 1 ram1 0 0 0 0 0 0 0 0 0 0 0
      <snip>
      1 14 ram14 0 0 0 0 0 0 0 0 0 0 0
      1 15 ram15 0 0 0 0 0 0 0 0 0 0 0
      7 0 loop0 0 0 0 0 0 0 0 0 0 0 0
      7 1 loop1 0 0 0 0 0 0 0 0 0 0 0
      <snip>
      7 6 loop6 0 0 0 0 0 0 0 0 0 0 0
      7 7 loop7 0 0 0 0 0 0 0 0 0 0 0
      31 0 mtdblock0 0 0 0 0 0 0 0 0 0 0 0
      31 1 mtdblock1 0 0 0 0 0 0 0 0 0 0 0
      31 2 mtdblock2 0 0 0 0 0 0 0 0 0 0 0
      8 0 sda 6735 3137 187140 86830 138193 38202 2283488 62720800 0 706720 63017690
      8 1 sda1 6660 3137 186540 86740 138193 38202 2283488 62720800 0 706710 63017600

      sda1 is the partition holding the root fs, not sure what the mtdblock devices are but they're not on the "working" Trimslice.
      Could it be that the error: "cannot attach shared memory for disk statistics collector: [22] Invalid argument" is due to the zabbix agent failing to understand the device 31 mtdblock devices?

      Running zabbix_agentd with p works fine:

      [[email protected] zabbix_agent]# zabbix_agentd -p
      agent.hostname [s|alarm-0]
      agent.ping [u|1]
      agent.version [s|2.0.2]
      system.localtime [u|1345423957]
      system.run [m|ZBX_NOTSUPPORTED]
      web.page.get [t|EOF]
      web.page.perf [d|0.000000]
      web.page.regexp [s|EOF]
      vfs.file.size [u|491]
      vfs.file.time [u|1343061420]
      vfs.file.exists [u|1]
      vfs.file.contents [t|root:x:0:0:root:/root:/bin/bash
      bin:x:1:1:bin:/bin:/bin/false
      daemon:x:2:2:daemon:/sbin:/bin/false
      mail:x:8:12:mail:/var/spool/mail:/bin/false
      ftp:x:14:11:ftp:/srv/ftp:/bin/false
      http:x:33:33:http:/srv/http:/bin/false
      nobody:x:99:99:nobody:/:/bin/false
      dbus:x:81:81:System message bus:/:/bin/false
      ntp:x:87:87:Network Time Protocol:/var/empty:/bin/false
      puppet:x:52:52:Puppet:/var/lib/puppet:/sbin/nologin
      drew:x:1000:1000::/home/drew:/bin/bash
      zabbix:x:1001:1001::/var/lib/zabbix:/bin/bash]
      vfs.file.regexp [s|root:x:0:0:root:/root:/bin/bash]
      vfs.file.regmatch [u|1]
      vfs.file.md5sum [s|52316bf02f91c5d2515d3d5a363cd259]
      vfs.file.cksum [u|3448930709]
      net.dns [u|1]
      net.dns.record [t|zabbix.com SOA ns1.yourhostingaccount.com admin.yourhostingaccount.com 2012081367 10800 3600 604800 3600]
      net.tcp.dns [u|1]
      net.tcp.dns.query [t|zabbix.com SOA ns1.yourhostingaccount.com admin.yourhostingaccount.com 2012081367 10800 3600 604800 3600]
      net.tcp.port [u|0]
      system.hostname [t|alarm-0]
      system.uname [t|Linux alarm-0 3.1.10-12-ARCH #1 PREEMPT Tue Jul 17 04:51:16 UTC 2012 armv5tel GNU/Linux]
      system.users.num [u|1]
      log [m|ZBX_NOTSUPPORTED]
      logrt [m|ZBX_NOTSUPPORTED]
      eventlog [m|ZBX_NOTSUPPORTED]
      kernel.maxfiles [u|51106]
      kernel.maxproc [u|32768]
      vfs.fs.size [u|14013427712]
      vfs.fs.inode [u|928421]
      vfs.fs.discovery [s|{
      "data":[
      {
      "

      {#FSNAME}":"\/",
      "{#FSTYPE}":"rootfs"},
      {
      "{#FSNAME}

      ":"\/",
      "

      {#FSTYPE}":"ext2"},
      {
      "{#FSNAME}":"\/dev",
      "{#FSTYPE}

      ":"devtmpfs"},
      {
      "

      {#FSNAME}":"\/proc",
      "{#FSTYPE}":"proc"},
      {
      "{#FSNAME}

      ":"\/sys",
      "

      {#FSTYPE}":"sysfs"},
      {
      "{#FSNAME}":"\/run",
      "{#FSTYPE}

      ":"tmpfs"},
      {
      "

      {#FSNAME}":"\/dev\/pts",
      "{#FSTYPE}":"devpts"},
      {
      "{#FSNAME}

      ":"\/dev\/shm",
      "

      {#FSTYPE}":"tmpfs"},
      {
      "{#FSNAME}":"\/tmp",
      "{#FSTYPE}

      ":"tmpfs"},
      {
      "

      {#FSNAME}

      ":"\/proc\/sys\/fs\/binfmt_misc",
      "

      {#FSTYPE}

      ":"binfmt_misc"}]}]
      vfs.dev.read [u|6735]
      vfs.dev.write [u|2283880]
      net.tcp.listen [u|0]
      net.udp.listen [u|0]
      net.if.in [u|150284949]
      net.if.out [u|150284949]
      net.if.total [u|300569898]
      net.if.collisions [u|0]
      net.if.discovery [s|{
      "data":[
      {
      "

      {#IFNAME}":"lo"},
      {
      "{#IFNAME}

      ":"eth0"},
      {
      "

      {#IFNAME}

      ":"eth1"}]}]
      vm.memory.size [u|523644928]
      proc.num [u|0]
      proc.mem [u|0]
      system.cpu.switches [u|39578123]
      system.cpu.intr [u|51163897]
      system.cpu.util [m|ZBX_NOTSUPPORTED]
      system.cpu.load [d|0.000000]
      system.cpu.num [u|1]
      system.hw.chassis [m|ZBX_NOTSUPPORTED]
      system.hw.cpu [m|ZBX_NOTSUPPORTED]
      system.hw.devices [m|ZBX_NOTSUPPORTED]
      system.hw.macaddr [s|[eth0] f0:ad:4e:ff:14:61]
      system.sw.arch [s|armv5tel]
      system.sw.os [s|Linux version 3.1.10-12-ARCH ([email protected]) (gcc version 4.7.1 (GCC) ) #1 PREEMPT Tue Jul 17 04:51:16 UTC 2012]
      system.sw.packages [t|[pacman] acl 2.2.51-2, attr 2.4.46-2, autoconf 2.69-1, automake 1.12.2-1, bash 4.2.037-1, binutils 2.22-8, bison 2.6.2-1, bzip2 1.0.6-4, ca-certificates 20120623-1, cloog 0.17.0-2, coreutils 8.17-3, cracklib 2.8.18-2, cronie 1.4.8-3, cryptsetup 1.5.0-1, curl 7.26.0-1, dash 0.5.7-2, db 5.3.21-1, dbus-core 1.6.4-1, device-mapper 2.02.97-1, dhcpcd 5.5.6-1, diffutils 3.2-1.1, dirmngr 1.1.0-4, distcc 3.1-7, dnssec-anchors 20120422-1, e2fsprogs 1.42.4-1, eventlog 0.2.12-3, expat 2.1.0-1, facter 1.6.8-1, fakeroot 1.18.4-1, file 5.11-1.2, filesystem 2012.7-1, findutils 4.4.2-4, flex 2.5.37-1, fuse 2.9.1-1, gawk 4.0.1-1, gcc 4.7.1-5, gcc-libs 4.7.1-5, gdbm 1.10-1, gettext 0.18.1.1-4, git 1.7.11.4-1, glib2 2.32.4-1, glibc 2.16.0-1, gmp 5.0.5-1, gnupg 2.0.19-2, gpgme 1.3.1-4, gpm 1.20.6-10, grep 2.13-2, groff 1.21-2, gzip 1.5-1, heirloom-mailx 12.5-3, hwids 20120730-1, iana-etc 2.30-3, inetutils 1.9.1-2, initscripts 2012.07.5-1, iproute2 3.4.0-2, iputils 20101006-4, isl 0.10-1, jfsutils 1.1.15-3, kbd 1.15.3-3, keyutils 1.5.5-3, kmod 9-2, krb5 1.10.3-1, ldns 1.6.13-1, less 444-3, libarchive 3.0.4-1, libassuan 2.0.3-1, libcap 2.22-3, libedit 20120311_3.0-1, libfetch 2.33-1, libffi 3.0.11-1, libgcrypt 1.5.0-3, libgpg-error 1.10-2, libgssglue 0.3-1, libidn 1.25-1, libksba 1.2.0-1, libldap 2.4.32-1, libltdl 2.4.2-6, libmpc 0.9-2, libnl 3.2.11-1, libpcap 1.3.0-1, libpipeline 1.2.1-1, libsasl 2.1.23-9, libsigsegv 2.10-1, libssh2 1.4.2-1, libsystemd 188-2, libtirpc 0.2.2-2, libtool 2.4.2-6, libusb-compat 0.1.4-2, libusbx 1.0.12-2, libyaml 0.1.4-2, licenses 2.9-1, linux 3.1.10-12, linux-api-headers 3.4.6-1, linux-firmware 20120625-1, linux-headers 3.1.10-12, logrotate 3.8.2-1, lvm2 2.02.97-1, m4 1.4.16-2.1, make 3.82-4, man-db 2.6.2-1, man-pages 3.41-1, mdadm 3.2.5-2, mkinitcpio 0.10-1, mkinitcpio-busybox 1.20.1-1, mpfr 3.1.1-1, nano 2.2.6-2, ncurses 5.9-3, net-tools 1.60.20110819cvs-3, ntfs-3g 2012.1.15-3, openntpd 3.9p1-15, openssh 6.0p1-3, openssl 1.0.1.c-1, package-query 1.0.1-1, pacman 4.0.3-2, pacman-mirrorlist 20120608-1, pam 1.1.5-4, pambase 20120701-1, patch 2.6.1-3, pciutils 3.1.10-1, pcmciautils 017-1, pcre 8.31-1, perl 5.16.1-1, perl-error 0.17018-1, pinentry 0.8.1-4, pkg-config 0.27-1, popt 1.16-5, ppl 1.0-1, ppp 2.4.5-3, procps-ng 3.3.3-3, psmisc 22.19-1, pth 2.0.7-4, puppet 2.7.17-1, readline 6.2.004-1, reiserfsprogs 3.6.21-4, rsync 3.0.9-3, ruby 1.9.3_p194-2, ruby-shadow 1.4.9-4, run-parts 4.3.2-1, screen 4.0.3-13, sed 4.2.1-4, shadow 4.1.5.1-1, strace 4.7-1, sudo 1.8.5.p2-1, sysfsutils 2.1.0-8, syslog-ng 3.3.5-2, systemd-tools 188-2, sysvinit 2.88-6, tar 1.26-2, tcp_wrappers 7.6-15, texinfo 4.13a-7, tzdata 2012e-1, usbutils 006-1, util-linux 2.21.2-5, vi 1:050325-2, vim 7.3.600-1, vim-runtime 7.3.600-1, wget 1.14-1, which 2.20-5, wpa_supplicant 1.0-1, xfsprogs 3.1.8-2, xz 5.0.4-1, yajl 2.0.4-1, yaourt 1.1-1, zabbix-agent 2.0.2-1, zlib 1.2.7-1]
      system.swap.size [u|0]
      system.swap.in [u|0]
      system.swap.out [u|0]
      system.uptime [u|2377393]
      system.boottime [u|1343046566]
      sensor [m|ZBX_NOTSUPPORTED]
      net.tcp.service [u|1]
      net.tcp.service.perf [d|0.044363]

      zabbix_agentd -t <key> for the disk device stats gives mixed results but note the segfaults for the abbreviated Sectors and Operations parameters.

      [[email protected] zabbix_agent]# zabbix_agentd -t vfs.dev.read[,sectors]
      vfs.dev.read[,sectors] [u|373680]
      [[email protected] zabbix_agent]# zabbix_agentd -t vfs.dev.read[,operations]
      vfs.dev.read[,operations] [u|13395]
      [[email protected] zabbix_agent]# zabbix_agentd -t vfs.dev.read[,bytes]
      vfs.dev.read[,bytes] [m|ZBX_NOTSUPPORTED]
      [[email protected] zabbix_agent]# zabbix_agentd -t vfs.dev.read[,sps]
      Segmentation fault
      [[email protected] zabbix_agent]# zabbix_agentd -t vfs.dev.read[,ops]
      Segmentation fault
      [[email protected] zabbix_agent]# zabbix_agentd -t vfs.dev.read[,bps]
      vfs.dev.read[,bps] [m|ZBX_NOTSUPPORTED]

        Attachments

          Activity

            People

            Assignee:
            Unassigned Unassigned
            Reporter:
            drew.gibson Drew Gibson
            Votes:
            0 Vote for this issue
            Watchers:
            0 Start watching this issue

              Dates

              Created:
              Updated:
              Resolved: