Gentoo 34-r6, Oracle 10g2, amd64

troubles only on high load zabbix server:
Количество узлов сети (под наблюдением/без наблюдения/шаблоны) 1117 842 / 25 / 250
Количество элементов данных (активных/деактивированых/не поддерживаются) 51564 49212 / 2114 / 238
Количество триггеров (активированных/деактивированных)[проблема/неизвестно/ок] 14725 14567 / 158 [233 / 1203 / 13130]
Количество пользователей (подключенных в данный момент) 91 10
Требуемое быстродействие сервера, новые значения в секунду 218.7 -

with config:
cat /etc/zabbix/zabbix_server.conf | egrep ^Start
StartPollers=130
StartIPMIPollers=0
StartPollersUnreachable=40
StartTrappers=120
StartPingers=60
StartDiscoverers=6
StartHTTPPollers=4
StartDBSyncers=30

Server have 16GB RAM, RAID 1, and 2 Intel(R) Xeon(R) CPU E5420

We are seeing this issue on some SNMPv2 checks. CentOS 5.5 64-bit, MySQL 5.1 64-bit. Zabbix 1.8.4 final. We have been using Zabbix for a long time and this seems new in 1.8.4.

The proxy is not very heavily loaded but the main server is (we have six proxies total). 3427 total hosts, 150969 total items.

The logs are exactly like Pustovalov posted. Sometimes the value is 0 and the poll interval is totally wrong. Other times it is double the correct value.

Please let me know if you need any other information to help troubleshoot, this is a major issue for us.

Example: Poll interval is 180 seconds (3 minutes)

2011-01-10 14:05:42 1294693542 90382920
2011-01-10 14:02:46 1294693366 0
2011-01-10 13:59:41 1294693181 89447248
2011-01-10 13:53:29 1294692809 90382920
2011-01-10 13:47:29 1294692449 100636448

It seems to be missing every other poll, and the one time it DOES poll after 3 minutes, it just returns 0.

Henry, but as as I see from your latest data your configured Update interval for this item is 6 min, but not 3 min (13:47:29 - 13:53:29).

Jon, is there any chance to get a screenshot of the item configuration screen in order to see its configuration? It looks very much like you are calculating delta for a non-growing counter, so Zabbix just skips negative numbers (deltas).

Item configuration showing 180 sec interval

That last screenshot was mine. I was logged in under a co-worker's account by mistake. As you can see it is configured for a 3 minute poll cycle.

We are seeing another problem with polling on this template, too. Sometimes the values are roughly double what they should be, and sometimes it polls twice within a few seconds. Look at the lines marked with ***** below

2011-01-10 11:05:59 1294682759 68657176 *****
2011-01-10 11:05:54 1294682754 86229008 *****
2011-01-10 10:59:31 1294682371 96623168
2011-01-10 10:53:30 1294682010 96689960
2011-01-10 10:47:29 1294681649 99154472
2011-01-10 10:41:36 1294681296 106047664
2011-01-10 10:38:47 1294681127 231917632 ***** (this shows as a HUGE spike in the graph, to an impossible value for the hardware)
2011-01-10 10:35:29 1294680929 99443488
2011-01-10 10:29:31 1294680571 105943472
2011-01-10 10:23:29 1294680209 90671448
2011-01-10 10:17:30 1294679850 88653544
2011-01-10 10:11:53 1294679513 98244104

last missed interval for SNMP item. I add screenshots in bug for thise.

today i discover what troubles with snmp hosts via zabbix proxy too.
smallest proxy have 45 values per second, if needed we can try enable debug logging for thise proxy. Nedeed?

Fixed in development branch svn://svn.zabbix.com/branches/dev/ZBX-3346

not fixed for me with branch:
17.Янв.2011 21:42:09 1023470720
17.Янв.2011 21:41:09 1141374480
17.Янв.2011 21:40:24 0
17.Янв.2011 21:40:23 966650264
17.Янв.2011 21:39:10 974736736

17.Янв.2011 21:21:09 1044514408
17.Янв.2011 21:20:40 1111817480
17.Янв.2011 21:20:31 1097529192
17.Янв.2011 21:19:27 0
17.Янв.2011 21:19:22 978083784

17.Янв.2011 21:05:10 1195238584
17.Янв.2011 21:04:28 0
17.Янв.2011 21:04:23 1048879800
17.Янв.2011 21:03:15 923030568

Thank you, Alexey.
Please, re-test this issue. Now should work correctly.

any info on what was the cause and when did the problem manifest itself ?

The problem seemed to occur when configuration cache was updating at the same time when an item was already being processed by some poller. An item got inserted into the queue, even though it should not have been.

fixed... now items not checks twice...

The svn://svn.zabbix.com/branches/dev/ZBX-3346 URL stopped working today. Is there another URL I can use to downloaded the fixed code? I'd like to patch our servers since this is causing big problems for us. Unless 1.8.5 is coming out very soon

The very latest 1.8.x code is always available at svn://svn.zabbix.com/branches/1.8. We remove development branches as soon as a change is merged into stable branch and/or trunk.

I see. Thank you!

Fixed in version pre1.8.5, r16995.

I have the latest stable version 1.8.4. Where can I download a "patch" to fixe this problem, as Jon refer url "svn://svn.zabbix.com/branches/dev/ZBX-3346" is not working.

Hi!

This issue has been fixed in the pre-v1.8.5.
Please also find a patch file for v1.8.4 for fixing this issue in the attachment to this issue.

Closing resolved issue.

As Alexei mentioned earlier, development branches are removed as soon as the changes are merged into the main branches. The latest 1.8 code (with this fix included) is always available at svn://svn.zabbix.com/branches/1.8. If you wish to patch 1.8.4, you can do "svn di -c 16995 svn://svn.zabbix.com/branches/1.8" and apply the resulting patch on 1.8.4 distribution. I have not tried though.

Ignore my comment. Igor was faster.

Hi Igor/Aleksandrs

Thanks by your fast reply.

How can I apply the patch ? It is safe to apply the patch can I do a backup.

Many thanks in advance.

Please find useful instructions from Igor for applying the patch:

You should patch your zabbix_server and zabbix_proxy and then re-compile and re-install them.

1. cd zabbix-1.8.4
2. execute command zabbix-1.8.4-delay.patch|patch -p0
3. execute command "make clean"
4. execute command "make install"
5. update your zabbix_server and/or zabbix_proxy binary files on the server with new compiled binary files
6. restart zabbix_server and/or zabbix_proxy

>How can I apply the patch ? Is it safe to apply the patch ? Can I do a backup ?

-->1. Sorry, small mistake in my comment, in the second row the correct command should be: "cat zabbix-1.8.4-delay.patch|patch -p0"
Please note that cat should be in the beginning of this command.

2. Yes, it is absolutely safe to apply this patch.
3. Yes, if you want you can do a backup (backup of the whole zabbix directory). This patch will patch only one file in your zabbix installation: src/libs/zbxdbcache/dbconfig.c

Hi,

This really solve my problem, great support people .

Many thanks especialy to Aleksandrs Saveljevs and Igor Danoshaites.

Regards.

this issue was happened again in 1.8.8 stable
with snmp Speed check of IFHCOUTOCTETS/IFHCINOCTETS oid
interval 10 sec or 15 sec

sample data :
248809160
2011.Oct.11 18:01:28 421387928
2011.Oct.11 18:01:12 218113736
2011.Oct.11 18:00:57 429393864
2011.Oct.11 18:00:42 207361472
2011.Oct.11 18:00:27 421759768
2011.Oct.11 18:00:12 205987552
2011.Oct.11 17:59:57 217666616
2011.Oct.11 17:59:43 386938184
2011.Oct.11 17:59:27 454190920
2011.Oct.11 17:59:13 200073112
2011.Oct.11 17:58:57 419605816

but the real speed is around 200-220Mbps