In src/libs/zbxdbhigh/lld_item.c in lld_item_application_compare_func() we use memcmp() to compare two structures

struct
{
	zbx_uint64_t	x;
	void	*y;
}

This is not good, because on 32-bit platform with 8 byte alignment restrictions for 64-bit integers this structure will contain 4 bytes of padding after pointer. (Raspberry Pi 2 uses ARM Cortex-A7 32-bit CPU. Similar results can be achieved on x86 platform using -m32 -malign-double compiler options.)

Since padding usually contains rubbish, memcmp() is very unlikely to return zero when it compares two structures with equal fields. This results in two identical item-application mappings being treated as different. And Zabbix server tries to insert both of them into the database which causes a failed query.

Issue introduced with ZBXNEXT-1219.

The paspberry pi I'm using is actually not "2", but just "Model B". (issue summary fixed)
This one https://www.raspberrypi.org/products/model-b/
And it's ARMv6 compatible, not ARMv7

Some possibly useful links are:
https://www.raspbian.org/RaspbianFAQ#What_is_Raspbian.3F
https://www.raspbian.org/RaspbianFAQ#What_compilation_options_should_be_set_Raspbian_code.3F

Still a 32-bit processor with 8-byte structure alignment rules. We have a fix in mind, we are just waiting for you to return from vacation to test it.

doh, I'd test it already if it would be posted here

Fix available in development branch svn://svn.zabbix.com/branches/dev/ZBX-10194 revision 57531.

Patch attached as well.

Reviewed the code. zalex_ua, please test on the actual hardware.

Before the patch has been provided, the Raspberry host has been upgraded to Debin 8 Jessie (from 7 Whenzy)

After upgrade:

# cat /proc/cpuinfo
processor       : 0
model name      : ARMv6-compatible processor rev 7 (v6l)
BogoMIPS        : 2.00
Features        : half thumb fastmult vfp edsp java tls
CPU implementer : 0x41
CPU architecture: 7
CPU variant     : 0x0
CPU part        : 0xb76
CPU revision    : 7

Hardware        : BCM2708
Revision        : 000e
Serial          : 00000000dc8125bb

# cat /etc/issue
Raspbian GNU/Linux 8 \n \l

root@kot2:/var/log/zabbix# cat /etc/debian_version
8.0

# uname -a
Linux kot2 4.1.13+ #826 PREEMPT Fri Nov 13 20:13:22 GMT 2015 armv6l GNU/Linux

# mysql -V
mysql  Ver 14.14 Distrib 5.5.46, for debian-linux-gnu (armv7l) using readline 6.3

Because libnetsnmp (as I recall) was updated too, existing zabbix_server binary could not start, complaining on a missing particular version of so file.
So I had to recompile zabbix_server. I used the same zabbix code revision 57353.
I DID NOT APPLY THE PATCH !
With recompiled binary - the issue is not reproducible anymore. I tested new binary on those "production" and "testing" databases.

Before recompilation I copied config.log to another folder, so now I have the file for current binary and for previous one - in case if you will need it.

Let me know if I still need to check the patch on current upgraded OS.

I would like to see both config.log's.

If the issue comes from where I suppose it comes from even a simple server restart could have helped.

Have you tried to reproduce the issue with another freshly created database?

config-prev.log (Raspbian 7 compilation), config.log (Raspbian 8 compilation) are attached.

As for restarts: I may recall that on Raspbian 7 I've restarted zabbix server 10+ times as I think all the time zabbix server logged those SQL errors.

On current Raspbian 8 I've restarted server ~90 times in a loop (after every restart the discovery rules were performed at least twice) and any SQL error was not logged.

Last test includes a case when discovered items deletion (filter was changed temporary), server restart, rediscovering those items - i.e. the same if I'd try freshly created database.

I see that your system upgraded gcc from 4.6 to 4.9 and this could have an effect on Zabbix compilation.

Quoting gcc 4.7 changes:

On ARM, when compiling for ARMv6 (but not ARMv6-M), ARMv7-A, ARMv7-R, or ARMv7-M, the new option -munaligned-access is active by default, which for some sources generates code that accesses memory on unaligned addresses. This requires the kernel of those systems to enable such accesses (controlled by CP15 register c1, refer to ARM documentation). Alternatively, or for compatibility with kernels where unaligned accesses are not supported, all code has to be compiled with -mno-unaligned-access. Upstream Linux kernel releases have automatically and unconditionally supported unaligned accesses as emitted by GCC due to this option being active since version 2.6.28.

I've attached a small program which emulates one particular aspect of LLD item-application mapping processing in Zabbix. Would be very helpful if you could compile and run it on Raspberry Pi. Try to use gcc 4.9 as it is and with option -mno-unaligned-access.

zalex_ua
Results are identical:

# gcc --version
gcc (Raspbian 4.9.2-10) 4.9.2

# ./a.out
size of uint64_t: 8
size of pointer: 4
size of structure: 16
memcmp result: -1

# ./a.out-mno-unaligned-access
size of uint64_t: 8
size of pointer: 4
size of structure: 16
memcmp result: -1

glebs.ivanovskis This means that changes in gcc I quoted above are not related to the issue.

But the issue still exists because the structure is bigger than two its fields and memcmp() result for two structures with identical fields can differ from 0.

Issue of duplicate inserts can be hard to catch "in the wild" because it depends on many factors and one of them is random junk in memory.

Fixed in pre-3.0.0beta1 (trunk) r57735.

Hello

I'm still seeing this on official packages 4.0.6 from Zabbix repo for raspbian...

Similar issue came back again, see ZBX-17073 ...