additional information : this is trunk from april 29th 10:00 (~ revision 19247)

looks like the source of the problem might be a child process that the agent has :

[sh] <defunct>

of course, that one can not be straced

looks like agent is not successfully killing child processes that time out or become dysfunctional

Can not reproduce. Maybe it's already fixed by ZBX-3072 .

just happened again with a debug build from ZBX-3994

this time, pid (30076) had some entries in the logfile :

10496:20110822:194101.619 End of zbx_popen():6 child:30076
30076:20110822:194101.620 zbx_popen(): executing script, parent:10496 [echo "show global status like 'Com_insert';" | HOME=/home/zabbix mysql -N | awk '

']

We have reproduced the problem with the small program in hanging-on-futex.c.

The idea is that some functions are not safe to use in signal handler. In our particular case, the use of function localtime() causes the process to hang on futex() call.

More details are available at http://www.fedoraforum.org/forum/showthread.php?t=187375 . The thread also links to the list of functions that are safe to use in signal handlers at http://pubs.opengroup.org/onlinepubs/000095399/functions/xsh_chap02_04.html .

Note that the problem affects not only agent, but server and proxy, too. The problem can also manifest itself while the process is operating normally, not necessarily during shutdown.

Same example using reentrant version of localtime(), no hanging. See attachment no-hanging-on-futex.c .

One of the biggest problems seems to me localtime() in our logging function ( __zbx_zabbix_log() ). There is lots of logging in our signal handlers. These should be replaced with reentrant functions for sure.

Reproduced again with unknown zabbix_server process:

$ ps u -p 3835
USER PID %CPU %MEM VSZ RSS TTY STAT START TIME COMMAND
vl 3835 0.0 0.0 52960 1048 ? SN Jan25 0:10 sbin/zabbix_server -c /etc/zabbix/zabbix_server.conf

$ grep 3835 /tmp/zabbix_server.log
$

$ strace -p 3835
Process 3835 attached - interrupt to quit
futex(0xb72a6a6c, FUTEX_WAIT_PRIVATE, 2, NULL

$ tail /tmp/zabbix_server.log
3826:20120131:152550.161 In zbx_mem_destroy() descr:'configuration cache'
3826:20120131:152550.161 End of zbx_mem_destroy()
3826:20120131:152550.161 In zbx_strpool_destroy()
3826:20120131:152550.161 In zbx_mem_destroy() descr:'string pool'
3826:20120131:152550.161 End of zbx_mem_destroy()
3826:20120131:152550.161 End of zbx_strpool_destroy()
3826:20120131:152550.161 End of free_configuration_cache()
3826:20120131:152550.162 In free_selfmon_collector() collector:0xb4ba1000
3826:20120131:152550.162 End of free_selfmon_collector()
3826:20120131:152550.162 Zabbix Server stopped. Zabbix 1.9.9 (revision

).

Problem reproduced and investigated.with zabbix_agentd from Zabbix 1.8.11rc1 on Debian GNU/Linux 6.0 (64-bit). More detailed document with proposed solution is in progress.

Signal handlers changed to reduce probability of process hang on exit in development branch svn://svn.zabbix.com/branches/dev/ZBX-3788

How the bug was investigated and proposed solutions - see attached document ZBX-3788_investigation1.odt

Great! Successfully tested!

Please review my changes in r25427.

Fixed in version pre-1.8.11 (revision 25433) and pre-1.9.10 (revision 25434).

i believe correct resolution should be "fixed", not "incomplete"

Andris, great analysis, it was interesting to read.

Thanks for kind words! It was great to learn about "ltrace", which reveals much more than "strace". As of "Fixed" vs."Incomplete" - I believe the change, although significantly reduces probability of process hangup (especially for DebugLevel=4)), does not completely eliminate it. We saw how evil localtime() can be in a signal handler, but even printf() is not acceptable there. Properly designed sIgnal handlers do something simple and lightweight, leaving the real work to others.