trunk rev 21658

Rich, we have tried investigating this memory issue and ZBX-4133 by running Zabbix server under Valgrind. However, no problems were found in our environment and, as noted in ZBX-4133, it is pretty difficult to analyze this issue, because heap could become corrupted in any part of Zabbix server code. Please reopen if the problem occurs again.

it just did

and looks like in both cases killed process is timer

Seems to crash right after start. Can you reliably reproduce the problem?

not really, it seems to happen every now and then. i could do some cycle of start/stop and see how often it happens - would that help any ?

maybe some debugging output can be added to the server that i could run with ?

There are two things we can try doing: (a) running Zabbix server under Valgrind and (b) running Zabbix server with additional debugging output.

I propose we start with (b). To that end, could you please try running Zabbix server from svn://svn.zabbix.com/branches/dev/ZBX-4151 ? It adds additional debugging output to memory allocation routines so that we can find out which allocated buffer is most close to the corrupted part of memory. You can probably keep DebugLevel=3, but if you could run it under DebugLevel=4, that would be nice, too.

i created a script to repeatedly start/stop server. after running it, server crashed on the first try...
although this time it was "corrupted double-linked list"

log of that start/crash session at debuglevel4 attached (zabbix_server_dev_branch_corrupted_double-linked_list.log.bz2)

zabbix_server_dev_branch_corrupted_double-linked_list_2.log.bz2 is another crash right after the startup.

additionally, in this case one zabbix_server process did not terminate upon kill -15. stracing it reveals that it got stuck on :

futex(0xb7356380, FUTEX_WAIT_PRIVATE, 2, NULL

Thanks, that was useful! The fix is available in development branch svn://svn.zabbix.com/branches/dev/ZBX-4151 .

The problem was that the buffer for time-based triggers was allocated for 0 triggers, then the configuration cache was synced, and then a non-zero amount of triggers were processed, which resulted in corrupted memory.

Fixed in pre-1.9.7 in r22185.