-
Type:
Incident report
-
Resolution: Cannot Reproduce
-
Priority:
Critical
-
None
-
Affects Version/s: 2.4.2
-
Component/s: Agent (G)
-
None
-
Environment:RHEL 5.x (various minor releases)
Hello,
On a regular basis, several of our monitored machines experience Zabbix agent crashes with the following message in the agent log:
***
zabbix_agentd [13183]: file:'cpustat.c',line:247 lock failed: [22] Invalid argument
13181:20141204:150824.201 One child process died (PID:13183,exitcode/signal:1). Exiting ...
13181:20141204:150824.201 zbx_on_exit() called
zabbix_agentd [13181]: Error on thread waiting.
13186:20141204:150824.202 Got signal [signal:15(SIGTERM),sender_pid:13181,sender_uid:59898,reason:0]. Exiting ...
13185:20141204:150824.202 Got signal [signal:15(SIGTERM),sender_pid:13181,sender_uid:59898,reason:0]. Exiting ...
13184:20141204:150824.202 Got signal [signal:15(SIGTERM),sender_pid:13181,sender_uid:59898,reason:0]. Exiting ...
13181:20141204:150824.202 Zabbix Agent stopped. Zabbix 2.4.2 (revision 50419).
13181:20141204:150824.202 In unload_modules()
***
The only portion that changes from crash to crash is the PID; everything else is the same (cpustat.c on line 247 with the lock failing due to an invalid argument).
The above log entry was from an instance with the DebugLevel set to 4 (which didn't yield all that much in terms of increased verbosity).
The applicable portions of cpustat.c are:
static void update_cpu_counters(ZBX_SINGLE_CPU_STAT_DATA *cpu, zbx_uint64_t *counter) { int i, index; LOCK_CPUSTATS; if (MAX_COLLECTOR_HISTORY <= (index = cpu->h_first + cpu->h_count)) index -= MAX_COLLECTOR_HISTORY; if (MAX_COLLECTOR_HISTORY > cpu->h_count) cpu->h_count++; else if (MAX_COLLECTOR_HISTORY == ++cpu->h_first) cpu->h_first = 0; if (NULL != counter) { for (i = 0; i < ZBX_CPU_STATE_COUNT; i++) cpu->h_counter[i][index] = counter[i]; cpu->h_status[index] = SYSINFO_RET_OK; } else cpu->h_status[index] = SYSINFO_RET_FAIL; UNLOCK_CPUSTATS; }
Any help would be appreciated.
Thank you!
Cheers,
Nathan Zachary