[ZBX-3974] semaphore lock failed followed by a crash Created: 2011 Jul 21  Updated: 2017 Jul 25  Resolved: 2017 Jul 25

Status: Closed
Project: ZABBIX BUGS AND ISSUES
Component/s: Server (S)
Affects Version/s: None
Fix Version/s: None

Type: Incident report Priority: Critical
Reporter: richlv Assignee: Unassigned
Resolution: Duplicate Votes: 7
Labels: crash, semaphores
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified
Environment:

1.8 branch, rev 20636
opensuse 11.4


Attachments: Text File zabbix_proxy_p.log    
Issue Links:
Duplicate
duplicates ZBXNEXT-3687 Rework semaphore and shared memory ma... Closed
is duplicated by ZBX-11770 Zabbix Server, Proxy, Agent crashs - ... Closed
is duplicated by ZBX-6606 proxy crashing with sqlite Closed
is duplicated by ZBX-11144 agent process crash Closed
is duplicated by ZBX-10657 all zabbix is running but i can't get... Closed

 Description   
  4289:20110719:090517.267 server #25 started [self-monitoring]
...

  4259:20110721:140615.001 History data from active proxy on [192.168.13.2] failed: proxy [Proxy 2] not found
  4264:20110721:140616.016 History data from active proxy on [192.168.13.2] failed: proxy [Proxy 2] not found
  4264:20110721:140617.030 History data from active proxy on [192.168.13.2] failed: proxy [Proxy 2] not found
  4264:20110721:140618.043 History data from active proxy on [192.168.13.2] failed: proxy [Proxy 2] not found
  4264:20110721:140619.045 History data from active proxy on [192.168.13.2] failed: proxy [Proxy 2] not found
zabbix_server [4283]: [file:'selfmon.c',line:353] lock failed: [43] Identifier removed
zabbix_server [4264]: [file:'selfmon.c',line:353] lock failed: [43] Identifier removed
zabbix_server [4282]: [file:'selfmon.c',line:353] lock failed: [43] Identifier removed
zabbix_server [4289]: [file:'selfmon.c',line:353] lock failed: [43] Identifier removed
  4247:20110721:140620.439 One child process died (PID:4289,exitcode/signal:255). Exiting ...
  4247:20110721:140622.442 Syncing history data...
zabbix_server [4247]: [file:'dbcache.c',line:2152] lock failed: [43] Identifier removed


 Comments   
Comment by richlv [ 2011 Jul 21 ]

same just happened in a proxy as well.

 17008:20110721:140928.618 server #12 started [icmp pinger]
...

 17010:20110721:140928.947 server #14 started [discoverer]
 17009:20110721:140929.060 Deleted 23 records from history [0.013868 seconds]
zabbix_proxy_sqlite [17008]: [file:'selfmon.c',line:353] lock failed: [22] Invalid argument
zabbix_proxy_sqlite [17011]: [file:'selfmon.c',line:353] lock failed: [22] Invalid argument
zabbix_proxy_sqlite [17014]: [file:'selfmon.c',line:353] lock failed: [22] Invalid argument
zabbix_proxy_sqlite [17013]: [file:'selfmon.c',line:353] lock failed: [22] Invalid argument
zabbix_proxy_sqlite [17012]: [file:'selfmon.c',line:353] lock failed: [22] Invalid argument
zabbix_proxy_sqlite [16997]: [file:'selfmon.c',line:353] lock failed: [22] Invalid argument
zabbix_proxy_sqlite [16999]: [file:'selfmon.c',line:353] lock failed: [22] Invalid argument
zabbix_proxy_sqlite [17001]: [file:'selfmon.c',line:353] lock failed: [22] Invalid argument
zabbix_proxy_sqlite [17000]: [file:'selfmon.c',line:353] lock failed: [22] Invalid argument
zabbix_proxy_sqlite [16998]: [file:'selfmon.c',line:353] lock failed: [22] Invalid argument
 16996:20110721:144014.150 One child process died (PID:17008,exitcode/signal:255). Exiting ...
 16996:20110721:144016.184 Syncing history data...
zabbix_proxy_sqlite [16996]: [file:'dbcache.c',line:2152] lock failed: [22] Invalid argument
Comment by richlv [ 2011 Jul 22 ]

full proxy log of two crashing sessions (and a bit of the next session startup)

server log has been rotated over already.

Comment by richlv [ 2011 Jul 22 ]

potentially useful information - system was running zabbix server, two active proxies and one passive proxy

Comment by Rudolfs Kreicbergs [ 2011 Jul 29 ]

Changed priority to critical - could not repeat or find the cause. If this is caused by Zabbix then it does not seem like a new problem.

Comment by Rudolfs Kreicbergs [ 2011 Aug 25 ]

Closing as "Cannot Reproduce" - seems that the semaphores are arbitrary removed, no reason was found nor was it reproduced on any other system. Please reopen when additional case to study with LogLevel=4 is acquired.

Comment by richlv [ 2012 May 24 ]

something quite similar happened again. 1.8 branch, rev 27659

 20098:20120524:100317.005 Zabbix agent item [agent.ping] on host [Server 4] failed: first network error, wait for 15 seconds
 20100:20120524:100332.684 resuming Zabbix agent checks on host [Server 4]: connection restored
zabbix_proxy [20094]: [file:'selfmon.c',line:331] lock failed: [22] Invalid argument
 20091:20120524:100343.887 One child process died (PID:20094,exitcode/signal:255). Exiting ...
 20091:20120524:100345.902 syncing history data...
zabbix_proxy [20091]: [file:'dbcache.c',line:2197] lock failed: [22] Invalid argument
Comment by Alexey Pustovalov [ 2012 Sep 08 ]

Semaphore Limit. On some Linux systems, reaches a system limit on the number of semaphores in the Linux configuration. In this case, you will see a message similar to the following:

sem_lock->semop->op_op: Invalid argument
sem_unlock->semctl: Invalid argument

Invalid argument [22] means:

  1. The semaphore identifier is invalid.
  2. The number of semaphores specified is less than 0 or greater than the number in the semaphore set.
  3. The value for cmd is invalid.
  4. The value for cmd is IPC_SET, but the value for sem_perm.uid or sem_perm.gid is invalid.

To increase the number of semaphores, run the following command as owner or root:

echo 256 40000 32 32000 > /proc/sys/kernel/sem

This file contains 4 numbers defining limits for System V IPC semaphores. These fields are, in order:

  • SEMMSL - The maximum semaphores per semaphore set.
  • SEMMNS - A system-wide limit on the number of semaphores in all semaphore sets.
  • SEMOPM - The maximum number of operations that may be specified in a semop(2) call.
  • SEMMNI - A system-wide limit on the maximum number of semaphore identifiers.

This modification takes effect immediately, but is reset at the next reboot. To avoid resetting the limit when rebooting, add the above command to one of your system’s startup scripts by copying the command into a file called mod_sem and then setting up the following links to execute the file each time you restart your system:

cp mod_sem /etc/init.d
ln -s /etc/init.d/mod_sem /etc/rc3.d/S61mod_sem
ln -s /etc/init.d/mod_sem /etc/rc5.d/S61mod_sem

Another method (prefered!):
To use these settings, put the following in sysctl.conf (as root) or use "sysctl -w" to write these settings to sysctl.conf:

 kernel.sem = 250 32000 100 10000

sysctl can be reloaded via the command "sysctl -p" (-p tells it to load from the standard conf file which is /etc/sysctl.conf).

"sysctl -a | grep kernel.sem" can be used to check that the new values have been loaded. This can also be used after a reboot to check that the values have been permanently saved.

Comment by Alexei Vladishev [ 2012 Sep 08 ]

Shall we close it?

<richlv> we don't know for sure what causes this - maybe zabbix passes invalid value, maybe it starts leaking semaphores, or maybe it indeed happens only when system in general is exhausted.

and even in the latter case it is possible we could detect that and as a minimum print a better message, same as when we run out of config cache etc

<richlv> also note that the proxies where crashes happen are not heavily loaded - they monitor some 10 hosts with 3-5 items at most with 3 nvps total...

Comment by richlv [ 2012 Nov 08 ]

happened again. 2.0 branch, rev 31271

zabbix_server [773]: [file:'selfmon.c',line:341] lock failed: [43] Identifier removed
   758:20121108:010738.687 One child process died (PID:773,exitcode/signal:255). Exiting ...
   758:20121108:010740.688 syncing history data...
   758:20121108:010740.688 syncing history data done
   758:20121108:010740.688 syncing trends data...
zabbix_server [758]: [file:'dbcache.c',line:745] lock failed: [43] Identifier removed
Comment by richlv [ 2012 Dec 06 ]

and happened again. 2.0 branch, rev 31853, passive proxy

zabbix_proxy [9162]: [file:'selfmon.c',line:341] lock failed: [22] Invalid argument
  9156:20121206:214137.865 One child process died (PID:9162,exitcode/signal:255). Exiting ...
  9156:20121206:214139.883 syncing history data...
zabbix_proxy [9156]: [file:'dbcache.c',line:2011] lock failed: [22] Invalid argument
Comment by richlv [ 2013 Jan 04 ]

this might also be related to multiple processes attempting to use the same pidfile. just had an agent crash with :

zabbix_agentd [6609]: [file:'cpustat.c',line:155] lock failed: [43] Identifier removed
  6606:20130104:171246.321 One child process died (PID:6609,exitcode/signal:255). Exiting ...

...or maybe by multiple processes attempting to listen on the same port, or some combination of these

Comment by richlv [ 2013 May 16 ]

while seemingly different, maybe ZBX-6606 is related

Comment by richlv [ 2013 Jul 31 ]

happened again. 2.0 branch, r37403

zabbix_server [11650]: [file:'selfmon.c',line:341] lock failed: [43] Identifier removed
 11635:20130731:114207.578 One child process died (PID:11650,exitcode/signal:255). Exiting ...
 11635:20130731:114209.579 syncing history data...
zabbix_server [11635]: [file:'dbcache.c',line:2193] lock failed: [43] Identifier removed
Comment by Alexei Vladishev [ 2014 Mar 05 ]

I had same crash with vanilla 2.2.2 on OpenSuSE. I suspect it is SuSE related issue since we had no reports from users of other distributions.

Comment by richlv [ 2014 Mar 05 ]

these seem to be triggered by running multiple zabbix daemons on the system in cases when these daemons are active and maybe started/stopped more frequently. that is probably less likely to happen during normal system operation.

Comment by Gabriele Armao [ 2014 Mar 18 ]

just got this (or a very similar) issue on one of my customer's server, it's a 2.0.10 Zabbix Server on a Red Hat (not sure which release). Backend database is Oracle.

They were having some issues with Web scenarios (error: error doing curl_easy_perform: Couldn't resolve host name) for a few days and we were trying to debug them before restarting the server, when during the night, the zabbix server crashed with this messages in the server log file:

  4528:20140316:222509.409 web scenario step "WEB2:login_check" error: error doing curl_easy_perform: Couldn't resolve host name
zabbix_server [3279]: cannot open [/var/log/zabbix/zabbix_server.log]: [24] Too many open files

followed, a few seconds later, by:

3114:20140316:222531.947 One child process died (PID:3279,exitcode/signal:255). Exiting ...
  4505:20140316:222602.830 web scenario step "WEB:check_url" error: error doing curl_easy_perform: Couldn't resolve host name
  3114:20140316:222614.540 syncing history data...
  3114:20140316:222615.914 syncing history data done
  3114:20140316:222615.915 syncing trends data...
  4505:20140316:222802.852 web scenario step "WEB:check_url" error: error doing curl_easy_perform: Couldn't resolve host name
  3114:20140316:222853.287 syncing trends data done
  3114:20140316:222853.314 Zabbix Server stopped. Zabbix 2.0.10 (revision 40809).
zabbix_server [4505]: [file:'selfmon.c',line:341] lock failed: [22] Invalid argument

As expected, PID 4505 is an HTTP Poller:

   4505:20140226:101829.725 server #857 started [http poller #3]

and PID 3279 is the DB Watchdog Process:

  3279:20140226:101817.264 server #2 started [db watchdog #1]

So it seems in this case, they run out of system resources (file descriptors, semaphores, ...)

glebs.ivanovskis [24] Too many open files does not line up with the rest of this ticket, so you may be interested in reporting it as a separate issue if it's still happening.

Comment by richlv [ 2014 Mar 18 ]

hmm, "Too many open files" is interesting. it might be related to the way we handle logfiles where each process opens the file individually and if the file is rotated and that process has nothing to write for it for a long time, it still holds open an fd on it. as a result, on a system with lots of processes and files being rotated relatively often we end up with lots of deleted files still being held open and on disk by those processes/fds.

now that is not the case when i observed it as those usually were small instances - but running out of semaphores one way or another could be related.

Comment by richlv [ 2015 Jan 08 ]

had another agent crash like that.

zabbix_agentd [15182]: [file:'cpustat.c',line:247] lock failed: [43] Identifier removed
 15181:20150108:173956.879 One child process died (PID:15182,exitcode/signal:1). Exiting ...
zabbix_agentd [15181]: Error on thread waiting.
Comment by Valery Tishkov [ 2015 Sep 04 ]

reproduced

  5119:20150722:090923.927 Starting Zabbix Agent [_hostname_]. Zabbix 2.2.9 (revision 52686).
  5119:20150722:090923.928 using configuration file: /etc/zabbix/zabbix_agentd.conf
  5125:20150722:090923.932 agent #0 started [collector]
  5126:20150722:090923.933 agent #1 started [listener #1]
  5127:20150722:090923.933 agent #2 started [listener #2]
  5128:20150722:090923.934 agent #3 started [listener #3]
  5129:20150722:090923.935 agent #4 started [listener #4]
  5130:20150722:090923.935 agent #5 started [listener #5]
  5131:20150722:090923.936 agent #6 started [active checks #1]
zabbix_agentd [5125]: [file:'cpustat.c',line:235] lock failed: [43] Identifier removed
zabbix_agentd [5119]: [file:'log.c',line:236] lock failed: [43] Identifier removed
zabbix_agentd [5128]: [file:'diskdevices.c',line:134] lock failed: [43] Identifier removed
zabbix_agentd [5129]: [file:'diskdevices.c',line:134] lock failed: [43] Identifier removed
zabbix_agentd [5127]: [file:'diskdevices.c',line:134] lock failed: [43] Identifier removed
zabbix_agentd [5130]: [file:'diskdevices.c',line:134] lock failed: [43] Identifier removed
zabbix_agentd [5126]: [file:'diskdevices.c',line:134] lock failed: [43] Identifier removed
zabbix_agentd [5131]: [file:'log.c',line:236] lock failed: [43] Identifier removed
$ sysctl -a | grep kernel.sem
kernel.sem = 250        32000   32      128
$ uname -srvmpio
Linux 2.6.32-504.12.2.el6.x86_64 #1 SMP Wed Mar 11 22:03:14 UTC 2015 x86_64 x86_64 x86_64 GNU/Linux
Comment by dimir [ 2015 Nov 20 ]

As to error 43 it seems it means some external entity killed the semaphore during the work flow of Zabbix daemon:

errno.h:
#define EIDRM 43 /* Identifier removed */

Comment by Michael Quigley [ 2016 Feb 02 ]

I've got a Zabbix server running on RedHat 7 in AWS, which has run fine without crashing for around 6 months. I updated to the latest kernel and rebooted, and now the server won't run for more than a handful of minutes without crashing like this:

  2445:20160202:152910.113 server #18 started [http poller #1]
  2451:20160202:152910.115 server #24 started [escalator #1]
  2449:20160202:152910.115 server #22 started [history syncer #3]
  2452:20160202:152910.115 server #25 started [proxy poller #1]
  2450:20160202:152910.116 server #23 started [history syncer #4]
  2445:20160202:153010.901 cannot process step "Login" of web scenario "pilot-stream.xxxxxxx.com" on host "Zabbix Server": response code "503" did not match any of the required status codes "200"
  2433:20160202:153129.235 Zabbix agent item "system.cpu.util[,softirq]" on host "Pilot-Stream-StreamAnalyzer-i-fb53057a" failed: first network error, wait for 15 seconds
  2435:20160202:153144.373 resuming Zabbix agent checks on host "Pilot-Stream-StreamAnalyzer-i-fb53057a": connection restored
zabbix_server [2432]: [file:'dbconfig.c',line:5174] lock failed: [22] Invalid argument
zabbix_server [2427]: [file:'log.c',line:271] lock failed: [22] Invalid argument
zabbix_server [2450]: [file:'selfmon.c',line:391] lock failed: [22] Invalid argument
zabbix_server [2447]: [file:'selfmon.c',line:391] lock failed: [22] Invalid argument
zabbix_server [2448]: [file:'selfmon.c',line:391] lock failed: [22] Invalid argument
zabbix_server [2449]: [file:'selfmon.c',line:391] lock failed: [22] Invalid argument
zabbix_server [2431]: [file:'selfmon.c',line:391] lock failed: [22] Invalid argument
zabbix_server [2433]: [file:'selfmon.c',line:391] lock failed: [22] Invalid argument
zabbix_server [2430]: [file:'selfmon.c',line:391] lock failed: [22] Invalid argument
zabbix_server [2434]: [file:'selfmon.c',line:391] lock failed: [22] Invalid argument
zabbix_server [2453]: [file:'selfmon.c',line:391] lock failed: [22] Invalid argument
zabbix_server [2451]: [file:'selfmon.c',line:391] lock failed: [22] Invalid argument
zabbix_server [2445]: [file:'selfmon.c',line:391] lock failed: [22] Invalid argument
zabbix_server [2435]: [file:'selfmon.c',line:391] lock failed: [22] Invalid argument
zabbix_server [2441]: [file:'selfmon.c',line:391] lock failed: [22] Invalid argument
zabbix_server [2452]: [file:'selfmon.c',line:391] lock failed: [22] Invalid argument
zabbix_server [2446]: [file:'selfmon.c',line:391] lock failed: [22] Invalid argument
zabbix_server [2442]: [file:'selfmon.c',line:391] lock failed: [22] Invalid argument
zabbix_server [2429]: [file:'selfmon.c',line:391] lock failed: [22] Invalid argument
zabbix_server [2428]: [file:'selfmon.c',line:391] lock failed: [22] Invalid argument
zabbix_server [2438]: [file:'selfmon.c',line:391] lock failed: [22] Invalid argument
zabbix_server [2444]: [file:'selfmon.c',line:391] lock failed: [22] Invalid argument
zabbix_server [2436]: [file:'selfmon.c',line:391] lock failed: [22] Invalid argument
zabbix_server [2439]: [file:'selfmon.c',line:391] lock failed: [22] Invalid argument
zabbix_server [2440]: [file:'selfmon.c',line:391] lock failed: [22] Invalid argument
zabbix_server [2437]: [file:'selfmon.c',line:391] lock failed: [22] Invalid argument

I've tried ensuring that the kernel semaphore configuration is adequately configured:

[ec2-user@ip-172-31-3-195 logs]$ sudo sysctl -a | grep kernel.sem
kernel.sem = 512        80000   512     64000

That's made no difference.

Help!

Comment by Michael Quigley [ 2016 Feb 03 ]

This seems pretty repeatable: If I'm ssh-ed into the instance where the zabbix_server is running, tailing the log, the server seems to run. Everytime, as soon as I log out of the instance, zabbix_server crashes like this.

Comment by Michael Quigley [ 2016 Feb 03 ]

Keeping a nohup tail zabbix_server.log > /dev/null 2>&1 & running seems to be keeping my server alive.

Comment by Aleksandrs Saveljevs [ 2016 Feb 03 ]

Michael, could you please specify how exactly you tail the log and how exactly you log out of the instance to make it crash? Does it crash immediately or after some time?

Note that your last command nohup tail zabbix_server.log > /dev/null 2>&1 & does not actually run in the background much, probably due to the missing -f or -F.

Comment by Michael Quigley [ 2016 Feb 03 ]

The good news is my zabbix_server managed to run through the night with the above workaround.Yes, the correct command I executed was nohup tail -f zabbix_server.log > /dev/null 2>&1 &. But I suspect that any command which keeps the ssh session open will keep my zabbix_server alive.

The server was compiled from source and is running on an AWS instance. The server had been running 2.4.5 for 6 months reliably. In an effort to try and resolve this, I upgraded the server to 2.4.7. The server version has no effect on the problem. The zabbix_server is started using systemd with a service definition like this:

[Unit]
Description=Zabbix Server
After=syslog.target network.target

[Service]
Type=forking
ExecStart=/home/ec2-user/local/zabbix/sbin/zabbix_server
ExecReload=/home/ec2-user/local/zabbix/sbin/zabbix_server -R config_cache_reload
RemainAfterExit=yes
PIDFile=/home/ec2-user/local/zabbix/logs/zabbix_server.pid
User=ec2-user

[Install]
WantedBy=multi-user.target

As soon as I kill off my background tail process, and exit my interactive ssh session with the instance, the zabbix_server process crashes.

Comment by Michael Quigley [ 2016 Feb 03 ]

To be even more clear... If I execute the nohup tail -f ... command, I CAN disconnect my ssh session from the instance and the zabbix_server will continue to execute. If I ssh back into the instance again, kill the tail command, and then exit my ssh session, zabbix_server will crash immediately.

glebs.ivanovskis Your issue seems more related to ZBX-11544.

Comment by Jan Klepek [ 2016 Mar 08 ]

Hi,

Why was this removed from 3.0.2rc1? Does it mean that you are not going to fix it in this version? Are you going to fix it anytime soon (like 1-2 months)?

Thank you for information

Comment by Ronny Pettersen [ 2016 Mar 08 ]

I've have 2 test instances of Zabbix running on CentOS 7.2, and I've also noticed that both the Zabbix 2.4.7 and Zabbix 3.0.1 crashes with the same error:

zabbix_server [15728]: [file:'selfmon.c',line:391] lock failed: [22] Invalid argument
zabbix_server [15724]: [file:'log.c',line:271] lock failed: [22] Invalid argument
zabbix_server [15752]: [file:'selfmon.c',line:391] lock failed: [22] Invalid argument
<repeat of above line 5 times>
zabbix_server [15736]: zabbix_server [15738]: [file:'selfmon.c',line:391] lock failed: [22] Invalid argument[file:'selfmon.c',line:391] lock failed: [22] Invalid argument

zabbix_server [15742]: [file:'selfmon.c',line:391] lock failed: [22] Invalid argument
<repeat of above line 9 times>

zabbix_agentd also creashes:

zabbix_agentd [10796]: [file:'cpustat.c',line:252] lock failed: [22] Invalid argument
zabbix_agentd [10795]: [file:'log.c',line:271] lock failed: [22] Invalid argument
zabbix_agentd [10798]: [file:'log.c',line:271] lock failed: [22] Invalid argument

After this, the only processes left running are:
zabbix_agentd: listener #1 [waiting for connection]
zabbix_server: trapper #1 [processed data in 0.000355 sec, waiting for connection]

Since this was test, I've not taken much notice to when it started happening. It could possibly be it started happening after upgrade to CentOS 7.2 on December 17.

In my case, it takes a while before it crashes. Not immediately after disconnect, but I start Zabbix using scripts (although without nohup, using "daemon $exec -c $conf").
It crashes every night at about 02:00 - 02:16 (CET). I'll enable debug and upload that if it would be of any help.

Comment by Ronny Pettersen [ 2016 Mar 14 ]

I've now been able to reproduce this on my system - it may not be the same behavior as for others.
This is triggered by a cronjob that connects to the database every day to update partitioning.
I've done some tests, and found that it is not the actual partitioning that provokes this. This simple shell-script triggered by cron crashes Zabbix:

#!/bin/sh
MYSQL="/usr/bin/mysql"
CONF="--defaults-file=$HOME/.my.zabbix.cnf"
SQL="select now();"
/bin/echo $SQL | $MYSQL $CONF

Cron:

00 15 * * * /opt/zabbix/scripts/test.sh

If I run the script manually, from command line, there are no issues.
I run the same on RHEL 6, and have no issues there. Also no issues with pre CentOS 7.2.

Comment by Ronny Pettersen [ 2016 Mar 14 ]

Also triggerting this run using "at now" gives the same result.

Strace of main Zabbix process (/opt/zabbix/sbin/zabbix_server -c /opt/zabbix/etc/zabbix_server.conf):

Process 13387 attached
wait4(-1, [{WIFEXITED(s) && WEXITSTATUS(s) == 1}], 0, NULL) = 13425
--- SIGCHLD {si_signo=SIGCHLD, si_code=CLD_EXITED, si_pid=13425, si_status=1, si_utime=0, si_stime=0} ---
semop(72220682, {{0, -1, SEM_UNDO}}, 1) = -1 EINVAL (Invalid argument)
write(2, "zabbix_server [13387]: ", 23) = 23
write(2, "[file:'log.c',line:271] lock fai"..., 58) = 58
write(2, "\n", 1)                       = 1
fcntl(3, F_SETLK, {type=F_UNLCK, whence=SEEK_SET, start=0, len=0}) = 0
close(3)                                = 0
munmap(0x7f4897133000, 4096)            = 0
unlink("/opt/zabbix/run/zabbix_server.pid") = 0
exit_group(1)                           = ?
+++ exited with 1 +++

And example of strace of "history syncer #1":

rt_sigprocmask(SIG_BLOCK, [CHLD], [], 8) = 0
rt_sigaction(SIGCHLD, NULL, {SIG_DFL, [CHLD], SA_RESTORER|SA_RESTART, 0x7ffcfc50e670}, 8) = 0
rt_sigprocmask(SIG_SETMASK, [], NULL, 8) = 0
nanosleep({1, 0}, 0x7fff075c3a70)       = 0
rt_sigprocmask(SIG_BLOCK, [CHLD], [], 8) = 0
rt_sigaction(SIGCHLD, NULL, {SIG_DFL, [CHLD], SA_RESTORER|SA_RESTART, 0x7ffcfc50e670}, 8) = 0
rt_sigprocmask(SIG_SETMASK, [], NULL, 8) = 0
nanosleep({1, 0}, 0x7fff075c3a70)       = 0
times({tms_utime=0, tms_stime=0, tms_cutime=0, tms_cstime=0}) = 3128138950
semop(72351754, {{5, -1, SEM_UNDO}}, 1) = -1 EINVAL (Invalid argument)
write(2, "zabbix_server [14415]: ", 23) = 23
write(2, "[file:'selfmon.c',line:391] lock"..., 62) = 62
write(2, "\n", 1)                       = 1
exit_group(1)                           = ?
+++ exited with 1 +++
Comment by Ronny Pettersen [ 2016 Mar 14 ]

I got the same error running any command through cron (even "at now" running date).
Looks like the error I am getting could be a Kernel bug. Similar to this: http://www.postgresql.org/message-id/[email protected]
(also on CentOS 7)
According to this error reported with Postgres and cron, the Semaphore was removed.
Ie:

semop(72351754, {{5, -1, SEM_UNDO}}, 1) = -1 EINVAL (Invalid argument)

I see server have not been restarted after installing latest kernel, so I will attempt that later today.
Current kernel: 3.10.0-229.1.2.el7.x86_64
Latest installed: 3.10.0-327.10.1.el7.x86_64

Comment by Ronny Pettersen [ 2016 Mar 14 ]

Updating to latest kernel (3.10.0-327.10.1.el7.x86_64) + reboot solved it for me
The error reported originally may be something else than what affected me.

Comment by Sheevary [ 2016 Jun 30 ]

I have the same problem.

FreeBSD 10.1 RELEASE p35
MySQL56
Apache24
Zabbix 3.0.2
When server started in semaphores list i see one smaphore
But after a few second zabbix server crash with error in log file:

Zabbix Server stopped. Zabbix 3.0.2 (revision 59540).
zabbix_server [20053]: file:'selfmon.c',line:375 lock failed: [22] Invalid argument

How to fix it?

Upd1
Trying to upgrade 10.1 to 10.3 RELEASE. Maby it helps.

Comment by dimir [ 2016 Jun 30 ]

Had seen the similar case when I was using more than one Zabbix server instances on the box.

Comment by Little Martian [ 2016 Nov 22 ]

I have a fresh install of version 3.2.1 compiled 64bit on Solaris 10. I found the same errors in the log files of both the server and the agent on the same server.
When I start the server, everything is fine, but as soon as I start the agent, the sever log presents the log messages:

zabbix_server [8967]: [file:'selfmon.c',line:380] lock failed: [22] Invalid argument
zabbix_server [8961]: [file:'selfmon.c',line:380] lock failed: [22] Invalid argument
zabbix_server [8965]: [file:'selfmon.c',line:380] lock failed: [22] Invalid argument
zabbix_server [8951]: [file:'selfmon.c',line:380] lock failed: [22] Invalid argument
zabbix_server [8952]: [file:'selfmon.c',line:380] lock failed: [22] Invalid argument

I restart the server and as soon as the process is running, the agent does the same thing:

zabbix_agentd [8997]: [file:'log.c',line:238] lock failed: [22] Invalid argument
zabbix_agentd [8991]: [file:'log.c',line:238] lock failed: [22] Invalid argument
zabbix_agentd [8998]: [file:'log.c',line:238] lock failed: [22] Invalid argument
zabbix_agentd [8995]: [file:'log.c',line:238] lock failed: [22] Invalid argument

I am running Server/Agent v 3.2.1, MySQL Ver 14.14 Distrib 5.7.16, on Solaris 5.10 Generic_150400-15, T4-2, Zabbix and MySQL compiled with gcc 5.2.0 (from OpenCSW).

On the same server there is another zabbix agent running (on different port and user) version 2.x (i do not know exactly) and i just imported two server definitions from an Zabbix 2.2.5. The agents for the imported servers run on two different Solaris 5.10 Generic_150400-15, T4-2 with version: Zabbix Agent (daemon) v2.2.1 (revision 40808) (09 December 2013) Compilation time: Dec 24 2013 20:08:37

For the imported servers I also have enabled SNMP monitoring (that do not work for now, firewall issues)

Comment by Little Martian [ 2016 Nov 22 ]

Sorry, my mistake. The error was from the shutdown script that removed semaphores and shared memory after server stopped. If the server does not stop in 10 sec, one kill -term / sec, then a kill -kill follows and semaphores and shared memory remained orphaned and cleaned with ipcrm.

I removed the semaphores cleanup and the error does not appear in the logs anymore. I have another problem with the zabbix server hanging in check_operation_conditions (the sql finishes instantly in MySQL Workbench). I will address this in another thread.

25260:20161122:145638.165 In check_operation_conditions() operationid:11
25260:20161122:145638.165 query [txnlev:0] [select conditiontype,operator,value from opconditions where operationid=11 order by conditiontype]
Comment by Alexander Khokhlov [ 2016 Dec 09 ]

I have this error after upgrade zabbix-server from version 3.2.1 to 3.2.2 from rpm package on CentOS 7

Linux zabbix 3.10.0-327.36.3.el7.x86_64 #1 SMP Mon Oct 24 16:09:20 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux
[file:'log.c',line:238] lock failed: [22] 

21260:20161209:230000.442 Got signal [signal:11(SIGSEGV),reason:1,refaddr:(nil)]. Crashing ...
 21260:20161209:230000.442 ====== Fatal information: ======
 21260:20161209:230000.442 Program counter: 0x4328ff
 21260:20161209:230000.442 === Registers: ===
 21260:20161209:230000.442 r8      =             ffff =                65535 =                65535
 21260:20161209:230000.442 r9      =                f =                   15 =                   15
 21260:20161209:230000.442 r10     =     7f8110337190 =      140192299315600 =      140192299315600
 21260:20161209:230000.442 r11     =              213 =                  531 =                  531
 21260:20161209:230000.442 r12     =          27ace40 =             41602624 =             41602624
 21260:20161209:230000.443 r13     =          27acc70 =             41602160 =             41602160
 21260:20161209:230000.443 r14     =          27ae4c7 =             41608391 =             41608391
 21260:20161209:230000.443 r15     =           794bb0 =              7949232 =              7949232
 21260:20161209:230000.443 rdi     =           4edb34 =              5167924 =              5167924
 21260:20161209:230000.443 rsi     =                0 =                    0 =                    0
 21260:20161209:230000.443 rdi     =           4edb34 =              5167924 =              5167924
 21260:20161209:230000.443 rsi     =                0 =                    0 =                    0
 21260:20161209:230000.443 rbp     =                0 =                    0 =                    0
 21260:20161209:230000.443 rbx     =     7ffc3ce5ccb0 =      140721330179248 =      140721330179248
 21260:20161209:230000.443 rdx     =                0 =                    0 =                    0
 21260:20161209:230000.443 rax     =                0 =                    0 =                    0
 21260:20161209:230000.443 rcx     =                5 =                    5 =                    5
 21260:20161209:230000.443 rsp     =     7ffc3ce5c870 =      140721330178160 =      140721330178160
 21260:20161209:230000.443 rip     =           4328ff =              4401407 =              4401407
 21260:20161209:230000.443 efl     =            10246 =                66118 =                66118
 21260:20161209:230000.443 csgsfs  =               33 =                   51 =                   51
 21260:20161209:230000.443 err     =                4 =                    4 =                    4
 21260:20161209:230000.443 trapno  =                e =                   14 =                   14
 21260:20161209:230000.443 oldmask =                0 =                    0 =                    0
 21260:20161209:230000.443 cr2     =                0 =                    0 =                    0
 21260:20161209:230000.443 === Backtrace: ===
 21260:20161209:230000.444 12: /usr/sbin/zabbix_server: poller #217 [got 0 values in 0.000000 sec, getting values](print_fatal_info+0xae) [0x48b1ce]
 21260:20161209:230000.444 11: /usr/sbin/zabbix_server: poller #217 [got 0 values in 0.000000 sec, getting values]() [0x48b4a3]
 21260:20161209:230000.444 10: /usr/lib64/libc.so.6(+0x35670) [0x7f8110239670]
 21260:20161209:230000.444 9: /usr/sbin/zabbix_server: poller #217 [got 0 values in 0.000000 sec, getting values](check_vcenter_hv_status+0x5f) [0x4328ff]
 21260:20161209:230000.444 8: /usr/sbin/zabbix_server: poller #217 [got 0 values in 0.000000 sec, getting values](get_value_simple+0x1c9) [0x4301a9]
 21260:20161209:230000.444 7: /usr/sbin/zabbix_server: poller #217 [got 0 values in 0.000000 sec, getting values]() [0x42e3ca]
 21260:20161209:230000.444 6: /usr/sbin/zabbix_server: poller #217 [got 0 values in 0.000000 sec, getting values](poller_thread+0x100) [0x42e500]
 21260:20161209:230000.444 5: /usr/sbin/zabbix_server: poller #217 [got 0 values in 0.000000 sec, getting values](zbx_thread_start+0x45) [0x48bea5]
 21260:20161209:230000.444 4: /usr/sbin/zabbix_server: poller #217 [got 0 values in 0.000000 sec, getting values](MAIN_ZABBIX_ENTRY+0x631) [0x421ae1]
 21260:20161209:230000.444 3: /usr/sbin/zabbix_server: poller #217 [got 0 values in 0.000000 sec, getting values](daemon_start+0x1c0) [0x48aa70]
 21260:20161209:230000.444 2: /usr/sbin/zabbix_server: poller #217 [got 0 values in 0.000000 sec, getting values](main+0x326) [0x419106]
 21260:20161209:230000.444 1: /usr/lib64/libc.so.6(__libc_start_main+0xf5) [0x7f8110225b15]
 21260:20161209:230000.444 0: /usr/sbin/zabbix_server: poller #217 [got 0 values in 0.000000 sec, getting values]() [0x4193ad]
 21260:20161209:230000.444 === Memory map: ===
 21260:20161209:230000.444 00400000-00593000 r-xp 00000000 fd:00 807302354                          /usr/sbin/zabbix_server_mysql
 21260:20161209:230000.444 00792000-00793000 r--p 00192000 fd:00 807302354                          /usr/sbin/zabbix_server_mysql
 21260:20161209:230000.444 00793000-0079a000 rw-p 00193000 fd:00 807302354                          /usr/sbin/zabbix_server_mysql
 21260:20161209:230000.444 0079a000-007a1000 rw-p 00000000 00:00 0
 21260:20161209:230000.444 022a1000-022e5000 rw-p 00000000 00:00 0                                  [heap]
 21260:20161209:230000.444 022e5000-02317000 rw-p 00000000 00:00 0                                  [heap]
 21260:20161209:230000.444 02317000-027de000 rw-p 00000000 00:00 0                                  [heap]
 21260:20161209:230000.444 022e5000-02317000 rw-p 00000000 00:00 0                                  [heap]
 21260:20161209:230000.444 02317000-027de000 rw-p 00000000 00:00 0                                  [heap]
 21260:20161209:230000.444 7f80e3d46000-7f80ea26d000 r--p 00000000 fd:00 537211368                  /usr/lib/locale/locale-archive
 21260:20161209:230000.445 7f80ea26d000-7f80ee26d000 rw-s 00000000 00:04 3702792                    /SYSV760005c7 (deleted)
 21260:20161209:230000.445 7f80ee26d000-7f80eea6d000 rw-s 00000000 00:04 3670023                    /SYSV770005c7 (deleted)
 21260:20161209:230000.445 7f80eea6d000-7f80f10d4000 rw-s 00000000 00:04 3604485                    /SYSV730005c7 (deleted)
 21260:20161209:230000.445 7f80f10d4000-7f80fea6e000 rw-s 00000000 00:04 3571716                    /SYSV670005c7 (deleted)
 21260:20161209:230000.445 7f80fea6e000-7f8102a6e000 rw-s 00000000 00:04 3538947                    /SYSV740005c7 (deleted)
 21260:20161209:230000.445 7f8102a6e000-7f8106a6e000 rw-s 00000000 00:04 3506178                    /SYSV480005c7 (deleted)
 21260:20161209:230000.445 7f8106a6e000-7f810aa6e000 rw-s 00000000 00:04 3473409                    /SYSV680005c7 (deleted)
 21260:20161209:230000.445 7f810aa6e000-7f810aa7a000 r-xp 00000000 fd:00 805308328                  /usr/lib64/libnss_files-2.17.so
 21260:20161209:230000.445 7f810aa7a000-7f810ac79000 ---p 0000c000 fd:00 805308328                  /usr/lib64/libnss_files-2.17.so
 21260:20161209:230000.445 7f810ac79000-7f810ac7a000 r--p 0000b000 fd:00 805308328                  /usr/lib64/libnss_files-2.17.so
 21260:20161209:230000.445 7f810ac7a000-7f810ac7b000 rw-p 0000c000 fd:00 805308328                  /usr/lib64/libnss_files-2.17.so
 21260:20161209:230000.445 7f810ac7b000-7f810ac81000 rw-p 00000000 00:00 0
 21260:20161209:230000.445 7f810ac81000-7f810ace1000 r-xp 00000000 fd:00 805308699                  /usr/lib64/libpcre.so.1.2.0
 21260:20161209:230000.445 7f810ace1000-7f810aee0000 ---p 00060000 fd:00 805308699                  /usr/lib64/libpcre.so.1.2.0
 21260:20161209:230000.445 7f810aee0000-7f810aee1000 r--p 0005f000 fd:00 805308699                  /usr/lib64/libpcre.so.1.2.0
 21260:20161209:230000.445 7f810aee1000-7f810aee2000 rw-p 00060000 fd:00 805308699                  /usr/lib64/libpcre.so.1.2.0
 21260:20161209:230000.445 7f810aee2000-7f810aee4000 r-xp 00000000 fd:00 805307860                  /usr/lib64/libfreebl3.so
 21260:20161209:230000.445 7f810aee4000-7f810b0e3000 ---p 00002000 fd:00 805307860                  /usr/lib64/libfreebl3.so
 21260:20161209:230000.445 7f810b0e3000-7f810b0e4000 r--p 00001000 fd:00 805307860                  /usr/lib64/libfreebl3.so
 21260:20161209:230000.445 7f810b0e4000-7f810b0e5000 rw-p 00002000 fd:00 805307860                  /usr/lib64/libfreebl3.so
 21260:20161209:230000.445 7f810b0e5000-7f810b106000 r-xp 00000000 fd:00 805308741                  /usr/lib64/libselinux.so.1
 21260:20161209:230000.445 7f810b106000-7f810b306000 ---p 00021000 fd:00 805308741                  /usr/lib64/libselinux.so.1
 21260:20161209:230000.445 7f810b306000-7f810b307000 r--p 00021000 fd:00 805308741                  /usr/lib64/libselinux.so.1
 21260:20161209:230000.445 7f810b307000-7f810b308000 rw-p 00022000 fd:00 805308741                  /usr/lib64/libselinux.so.1
 21260:20161209:230000.445 7f810b308000-7f810b30a000 rw-p 00000000 00:00 0
 21260:20161209:230000.445 7f810b30a000-7f810b311000 r-xp 00000000 fd:00 805308911                  /usr/lib64/libffi.so.6.0.1
 21260:20161209:230000.445 7f810b311000-7f810b510000 ---p 00007000 fd:00 805308911                  /usr/lib64/libffi.so.6.0.1
 21260:20161209:230000.445 7f810b510000-7f810b511000 r--p 00006000 fd:00 805308911                  /usr/lib64/libffi.so.6.0.1
 21260:20161209:230000.445 7f810b511000-7f810b512000 rw-p 00007000 fd:00 805308911                  /usr/lib64/libffi.so.6.0.1
 21260:20161209:230000.445 7f810b512000-7f810b519000 r-xp 00000000 fd:00 805308685                  /usr/lib64/librt-2.17.so
 21260:20161209:230000.445 7f810b519000-7f810b718000 ---p 00007000 fd:00 805308685                  /usr/lib64/librt-2.17.so
 21260:20161209:230000.445 7f810b718000-7f810b719000 r--p 00006000 fd:00 805308685                  /usr/lib64/librt-2.17.so
 21260:20161209:230000.445 7f810b719000-7f810b71a000 rw-p 00007000 fd:00 805308685                  /usr/lib64/librt-2.17.so
 21260:20161209:230000.445 7f810b71a000-7f810b722000 r-xp 00000000 fd:00 805308309                  /usr/lib64/libcrypt-2.17.so
 21260:20161209:230000.445 7f810b719000-7f810b71a000 rw-p 00007000 fd:00 805308685                  /usr/lib64/librt-2.17.so
 21260:20161209:230000.445 7f810b71a000-7f810b722000 r-xp 00000000 fd:00 805308309                  /usr/lib64/libcrypt-2.17.so
 21260:20161209:230000.445 7f810b722000-7f810b921000 ---p 00008000 fd:00 805308309                  /usr/lib64/libcrypt-2.17.so
 21260:20161209:230000.446 7f810b921000-7f810b922000 r--p 00007000 fd:00 805308309                  /usr/lib64/libcrypt-2.17.so
 21260:20161209:230000.446 7f810b922000-7f810b923000 rw-p 00008000 fd:00 805308309                  /usr/lib64/libcrypt-2.17.so
 21260:20161209:230000.446 7f810b923000-7f810b951000 rw-p 00000000 00:00 0
 21260:20161209:230000.446 7f810b951000-7f810b954000 r-xp 00000000 fd:00 805309470                  /usr/lib64/libkeyutils.so.1.5
 21260:20161209:230000.446 7f810b954000-7f810bb53000 ---p 00003000 fd:00 805309470                  /usr/lib64/libkeyutils.so.1.5
 21260:20161209:230000.446 7f810bb53000-7f810bb54000 r--p 00002000 fd:00 805309470                  /usr/lib64/libkeyutils.so.1.5
 21260:20161209:230000.446 7f810bb54000-7f810bb55000 rw-p 00003000 fd:00 805309470                  /usr/lib64/libkeyutils.so.1.5
 21260:20161209:230000.446 7f810bb55000-7f810bb62000 r-xp 00000000 fd:00 805407361                  /usr/lib64/libkrb5support.so.0.1
 21260:20161209:230000.446 7f810bb62000-7f810bd62000 ---p 0000d000 fd:00 805407361                  /usr/lib64/libkrb5support.so.0.1
 21260:20161209:230000.446 7f810bd62000-7f810bd63000 r--p 0000d000 fd:00 805407361                  /usr/lib64/libkrb5support.so.0.1
 21260:20161209:230000.446 7f810bd63000-7f810bd64000 rw-p 0000e000 fd:00 805407361                  /usr/lib64/libkrb5support.so.0.1
 21260:20161209:230000.446 7f810bd64000-7f810bdda000 r-xp 00000000 fd:00 805309227                  /usr/lib64/libgmp.so.10.2.0
 21260:20161209:230000.446 7f810bdda000-7f810bfd9000 ---p 00076000 fd:00 805309227                  /usr/lib64/libgmp.so.10.2.0
 21260:20161209:230000.446 7f810bfd9000-7f810bfda000 r--p 00075000 fd:00 805309227                  /usr/lib64/libgmp.so.10.2.0
 21260:20161209:230000.446 7f810bfda000-7f810bfdb000 rw-p 00076000 fd:00 805309227                  /usr/lib64/libgmp.so.10.2.0
 21260:20161209:230000.446 7f810bfdb000-7f810c001000 r-xp 00000000 fd:00 805309523                  /usr/lib64/libhogweed.so.2.5
 21260:20161209:230000.446 7f810c001000-7f810c200000 ---p 00026000 fd:00 805309523                  /usr/lib64/libhogweed.so.2.5
 21260:20161209:230000.446 7f810c200000-7f810c201000 r--p 00025000 fd:00 805309523                  /usr/lib64/libhogweed.so.2.5
 21260:20161209:230000.446 7f810c201000-7f810c202000 rw-p 00026000 fd:00 805309523                  /usr/lib64/libhogweed.so.2.5
 21260:20161209:230000.446 7f810c202000-7f810c231000 r-xp 00000000 fd:00 805309525                  /usr/lib64/libnettle.so.4.7
 21260:20161209:230000.446 7f810c231000-7f810c431000 ---p 0002f000 fd:00 805309525                  /usr/lib64/libnettle.so.4.7
 21260:20161209:230000.446 7f810c431000-7f810c432000 r--p 0002f000 fd:00 805309525                  /usr/lib64/libnettle.so.4.7
 21260:20161209:230000.446 7f810c432000-7f810c433000 rw-p 00030000 fd:00 805309525                  /usr/lib64/libnettle.so.4.7
 21260:20161209:230000.446 7f810c433000-7f810c445000 r-xp 00000000 fd:00 805309504                  /usr/lib64/libtasn1.so.6.2.3
 21260:20161209:230000.446 7f810c445000-7f810c645000 ---p 00012000 fd:00 805309504                  /usr/lib64/libtasn1.so.6.2.3
 21260:20161209:230000.446 7f810c645000-7f810c646000 r--p 00012000 fd:00 805309504                  /usr/lib64/libtasn1.so.6.2.3
 21260:20161209:230000.446 7f810c646000-7f810c647000 rw-p 00013000 fd:00 805309504                  /usr/lib64/libtasn1.so.6.2.3
 21260:20161209:230000.446 7f810c647000-7f810c6b5000 r-xp 00000000 fd:00 805378615                  /usr/lib64/libtspi.so.1.2.0
 21260:20161209:230000.446 7f810c6b5000-7f810c8b4000 ---p 0006e000 fd:00 805378615                  /usr/lib64/libtspi.so.1.2.0
 21260:20161209:230000.446 7f810c8b4000-7f810c8b5000 r--p 0006d000 fd:00 805378615                  /usr/lib64/libtspi.so.1.2.0
 21260:20161209:230000.446 7f810c8b5000-7f810c8b7000 rw-p 0006e000 fd:00 805378615                  /usr/lib64/libtspi.so.1.2.0
 21260:20161209:230000.446 7f810c8b7000-7f810c8b8000 rw-p 00000000 00:00 0
 21260:20161209:230000.446 7f810c8b8000-7f810c8f4000 r-xp 00000000 fd:00 805309056                  /usr/lib64/libp11-kit.so.0.0.0
 21260:20161209:230000.447 7f810c8f4000-7f810caf3000 ---p 0003c000 fd:00 805309056                  /usr/lib64/libp11-kit.so.0.0.0
 21260:20161209:230000.446 7f810c8b8000-7f810c8f4000 r-xp 00000000 fd:00 805309056                  /usr/lib64/libp11-kit.so.0.0.0
 21260:20161209:230000.447 7f810c8f4000-7f810caf3000 ---p 0003c000 fd:00 805309056                  /usr/lib64/libp11-kit.so.0.0.0
 21260:20161209:230000.447 7f810caf3000-7f810cafc000 r--p 0003b000 fd:00 805309056                  /usr/lib64/libp11-kit.so.0.0.0
 21260:20161209:230000.447 7f810cafc000-7f810cafe000 rw-p 00044000 fd:00 805309056                  /usr/lib64/libp11-kit.so.0.0.0
 21260:20161209:230000.447 7f810cafe000-7f810cb13000 r-xp 00000000 fd:00 805306569                  /usr/lib64/libgcc_s-4.8.5-20150702.so.1
 21260:20161209:230000.447 7f810cb13000-7f810cd12000 ---p 00015000 fd:00 805306569                  /usr/lib64/libgcc_s-4.8.5-20150702.so.1
 21260:20161209:230000.447 7f810cd12000-7f810cd13000 r--p 00014000 fd:00 805306569                  /usr/lib64/libgcc_s-4.8.5-20150702.so.1
 21260:20161209:230000.447 7f810cd13000-7f810cd14000 rw-p 00015000 fd:00 805306569                  /usr/lib64/libgcc_s-4.8.5-20150702.so.1
 21260:20161209:230000.447 7f810cd14000-7f810cd46000 r-xp 00000000 fd:00 805309212                  /usr/lib64/libidn.so.11.6.11
 21260:20161209:230000.447 7f810cd46000-7f810cf45000 ---p 00032000 fd:00 805309212                  /usr/lib64/libidn.so.11.6.11
 21260:20161209:230000.447 7f810cf45000-7f810cf46000 r--p 00031000 fd:00 805309212                  /usr/lib64/libidn.so.11.6.11
 21260:20161209:230000.447 7f810cf46000-7f810cf47000 rw-p 00032000 fd:00 805309212                  /usr/lib64/libidn.so.11.6.11
 21260:20161209:230000.447 7f810cf47000-7f810cf81000 r-xp 00000000 fd:00 805311252                  /usr/lib64/libnspr4.so
 21260:20161209:230000.447 7f810cf81000-7f810d180000 ---p 0003a000 fd:00 805311252                  /usr/lib64/libnspr4.so
 21260:20161209:230000.447 7f810d180000-7f810d181000 r--p 00039000 fd:00 805311252                  /usr/lib64/libnspr4.so
 21260:20161209:230000.447 7f810d181000-7f810d183000 rw-p 0003a000 fd:00 805311252                  /usr/lib64/libnspr4.so
 21260:20161209:230000.447 7f810d183000-7f810d185000 rw-p 00000000 00:00 0
 21260:20161209:230000.447 7f810d185000-7f810d189000 r-xp 00000000 fd:00 805311253                  /usr/lib64/libplc4.so
 21260:20161209:230000.447 7f810d189000-7f810d388000 ---p 00004000 fd:00 805311253                  /usr/lib64/libplc4.so
 21260:20161209:230000.447 7f810d388000-7f810d389000 r--p 00003000 fd:00 805311253                  /usr/lib64/libplc4.so
 21260:20161209:230000.447 7f810d389000-7f810d38a000 rw-p 00004000 fd:00 805311253                  /usr/lib64/libplc4.so
 21260:20161209:230000.447 7f810d38a000-7f810d38d000 r-xp 00000000 fd:00 805311254                  /usr/lib64/libplds4.so
 21260:20161209:230000.447 7f810d38d000-7f810d58c000 ---p 00003000 fd:00 805311254                  /usr/lib64/libplds4.so
 21260:20161209:230000.447 7f810d58c000-7f810d58d000 r--p 00002000 fd:00 805311254                  /usr/lib64/libplds4.so
 21260:20161209:230000.447 7f810d58d000-7f810d58e000 rw-p 00003000 fd:00 805311254                  /usr/lib64/libplds4.so
 21260:20161209:230000.447 7f810d58e000-7f810d5b4000 r-xp 00000000 fd:00 805311255                  /usr/lib64/libnssutil3.so
 21260:20161209:230000.447 7f810d5b4000-7f810d7b3000 ---p 00026000 fd:00 805311255                  /usr/lib64/libnssutil3.so
 21260:20161209:230000.447 7f810d7b3000-7f810d7b9000 r--p 00025000 fd:00 805311255                  /usr/lib64/libnssutil3.so
 21260:20161209:230000.447 7f810d7b9000-7f810d7ba000 rw-p 0002b000 fd:00 805311255                  /usr/lib64/libnssutil3.so
 21260:20161209:230000.447 7f810d7ba000-7f810d8d8000 r-xp 00000000 fd:00 805407415                  /usr/lib64/libnss3.so
 21260:20161209:230000.447 7f810d8d8000-7f810dad7000 ---p 0011e000 fd:00 805407415                  /usr/lib64/libnss3.so
 21260:20161209:230000.447 7f810dad7000-7f810dadc000 r--p 0011d000 fd:00 805407415                  /usr/lib64/libnss3.so
 21260:20161209:230000.447 7f810dadc000-7f810dade000 rw-p 00122000 fd:00 805407415                  /usr/lib64/libnss3.so
 21260:20161209:230000.447 7f810dade000-7f810dae0000 rw-p 00000000 00:00 0
 21260:20161209:230000.448 7f810dae0000-7f810db04000 r-xp 00000000 fd:00 805310904                  /usr/lib64/libsmime3.so
 21260:20161209:230000.448 7f810db04000-7f810dd03000 ---p 00024000 fd:00 805310904                  /usr/lib64/libsmime3.so
 21260:20161209:230000.448 7f810dd03000-7f810dd06000 r--p 00023000 fd:00 805310904                  /usr/lib64/libsmime3.so
21260:20161209:230000.448 7f810db04000-7f810dd03000 ---p 00024000 fd:00 805310904                  /usr/lib64/libsmime3.so
 21260:20161209:230000.448 7f810dd03000-7f810dd06000 r--p 00023000 fd:00 805310904                  /usr/lib64/libsmime3.so
 21260:20161209:230000.448 7f810dd06000-7f810dd07000 rw-p 00026000 fd:00 805310904                  /usr/lib64/libsmime3.so
 21260:20161209:230000.448 7f810dd07000-7f810dd45000 r-xp 00000000 fd:00 805310905                  /usr/lib64/libssl3.so
 21260:20161209:230000.448 7f810dd45000-7f810df44000 ---p 0003e000 fd:00 805310905                  /usr/lib64/libssl3.so
 21260:20161209:230000.448 7f810df44000-7f810df48000 r--p 0003d000 fd:00 805310905                  /usr/lib64/libssl3.so
 21260:20161209:230000.448 7f810df48000-7f810df49000 rw-p 00041000 fd:00 805310905                  /usr/lib64/libssl3.so
 21260:20161209:230000.448 7f810df49000-7f810df4a000 rw-p 00000000 00:00 0
 21260:20161209:230000.448 7f810df4a000-7f810df66000 r-xp 00000000 fd:00 805309327                  /usr/lib64/libsasl2.so.3.0.0
 21260:20161209:230000.448 7f810df66000-7f810e165000 ---p 0001c000 fd:00 805309327                  /usr/lib64/libsasl2.so.3.0.0
 21260:20161209:230000.448 7f810e165000-7f810e166000 r--p 0001b000 fd:00 805309327                  /usr/lib64/libsasl2.so.3.0.0
 21260:20161209:230000.448 7f810e166000-7f810e167000 rw-p 0001c000 fd:00 805309327                  /usr/lib64/libsasl2.so.3.0.0
 21260:20161209:230000.448 7f810e167000-7f810e196000 r-xp 00000000 fd:00 805309899                  /usr/lib64/libk5crypto.so.3.1
 21260:20161209:230000.448 7f810e196000-7f810e395000 ---p 0002f000 fd:00 805309899                  /usr/lib64/libk5crypto.so.3.1
 21260:20161209:230000.448 7f810e395000-7f810e397000 r--p 0002e000 fd:00 805309899                  /usr/lib64/libk5crypto.so.3.1
 21260:20161209:230000.448 7f810e397000-7f810e398000 rw-p 00030000 fd:00 805309899                  /usr/lib64/libk5crypto.so.3.1
 21260:20161209:230000.448 7f810e398000-7f810e399000 rw-p 00000000 00:00 0
 21260:20161209:230000.448 7f810e399000-7f810e39c000 r-xp 00000000 fd:00 805308783                  /usr/lib64/libcom_err.so.2.1
 21260:20161209:230000.448 7f810e39c000-7f810e59b000 ---p 00003000 fd:00 805308783                  /usr/lib64/libcom_err.so.2.1
 21260:20161209:230000.448 7f810e59b000-7f810e59c000 r--p 00002000 fd:00 805308783                  /usr/lib64/libcom_err.so.2.1
 21260:20161209:230000.448 7f810e59c000-7f810e59d000 rw-p 00003000 fd:00 805308783                  /usr/lib64/libcom_err.so.2.1
 21260:20161209:230000.448 7f810e59d000-7f810e672000 r-xp 00000000 fd:00 805309909                  /usr/lib64/libkrb5.so.3.3
 21260:20161209:230000.448 7f810e672000-7f810e872000 ---p 000d5000 fd:00 805309909                  /usr/lib64/libkrb5.so.3.3
 21260:20161209:230000.448 7f810e872000-7f810e87f000 r--p 000d5000 fd:00 805309909                  /usr/lib64/libkrb5.so.3.3
 21260:20161209:230000.448 7f810e87f000-7f810e882000 rw-p 000e2000 fd:00 805309909                  /usr/lib64/libkrb5.so.3.3
 21260:20161209:230000.448 7f810e882000-7f810e8cb000 r-xp 00000000 fd:00 805309895                  /usr/lib64/libgssapi_krb5.so.2.2
 21260:20161209:230000.448 7f810e8cb000-7f810eacb000 ---p 00049000 fd:00 805309895                  /usr/lib64/libgssapi_krb5.so.2.2
 21260:20161209:230000.448 7f810eacb000-7f810eacc000 r--p 00049000 fd:00 805309895                  /usr/lib64/libgssapi_krb5.so.2.2
 21260:20161209:230000.448 7f810eacc000-7f810eace000 rw-p 0004a000 fd:00 805309895                  /usr/lib64/libgssapi_krb5.so.2.2
 21260:20161209:230000.448 7f810eace000-7f810ead5000 r-xp 00000000 fd:00 805309509                  /usr/lib64/libgdbm.so.4.0.0
 21260:20161209:230000.448 7f810ead5000-7f810ecd5000 ---p 00007000 fd:00 805309509                  /usr/lib64/libgdbm.so.4.0.0
 21260:20161209:230000.448 7f810ecd5000-7f810ecd6000 r--p 00007000 fd:00 805309509                  /usr/lib64/libgdbm.so.4.0.0
 21260:20161209:230000.448 7f810ecd6000-7f810ecd7000 rw-p 00008000 fd:00 805309509                  /usr/lib64/libgdbm.so.4.0.0
 21260:20161209:230000.448 7f810ecd7000-7f810ecdf000 r-xp 00000000 fd:00 805503563                  /usr/lib64/libOpenIPMIutils.so.0.0.1
 21260:20161209:230000.448 7f810ecdf000-7f810eede000 ---p 00008000 fd:00 805503563                  /usr/lib64/libOpenIPMIutils.so.0.0.1
 21260:20161209:230000.448 7f810eede000-7f810eedf000 r--p 00007000 fd:00 805503563                  /usr/lib64/libOpenIPMIutils.so.0.0.1
 21260:20161209:230000.449 7f810eedf000-7f810eee0000 rw-p 00008000 fd:00 805503563                  /usr/lib64/libOpenIPMIutils.so.0.0.1

...

 21260:20161209:230000.451 7f8112a16000-7f8112a23000 r-xp 00000000 fd:00 805448382                  /usr/lib64/libiksemel.so.3.1.1
 21260:20161209:230000.451 7f8112a23000-7f8112c22000 ---p 0000d000 fd:00 805448382                  /usr/lib64/libiksemel.so.3.1.1
 21260:20161209:230000.451 7f8112c22000-7f8112c23000 r--p 0000c000 fd:00 805448382                  /usr/lib64/libiksemel.so.3.1.1
 21260:20161209:230000.452 7f8112c23000-7f8112c24000 rw-p 0000d000 fd:00 805448382                  /usr/lib64/libiksemel.so.3.1.1
 21260:20161209:230000.452 7f8112c24000-7f8112f43000 r-xp 00000000 fd:00 806489619                  /usr/lib64/libmysqlclient.so.18.0.0
 21260:20161209:230000.452 7f8112f43000-7f8113143000 ---p 0031f000 fd:00 806489619                  /usr/lib64/libmysqlclient.so.18.0.0
 21260:20161209:230000.452 7f8113143000-7f811314b000 r--p 0031f000 fd:00 806489619                  /usr/lib64/libmysqlclient.so.18.0.0
 21260:20161209:230000.452 7f811314b000-7f81131be000 rw-p 00327000 fd:00 806489619                  /usr/lib64/libmysqlclient.so.18.0.0
 21260:20161209:230000.452 7f81131be000-7f81131c8000 rw-p 00000000 00:00 0
 21260:20161209:230000.452 7f81131c8000-7f81131e9000 r-xp 00000000 fd:00 805307863                  /usr/lib64/ld-2.17.so
 21260:20161209:230000.452 7f81133a0000-7f81133a2000 rw-p 00000000 00:00 0
 21260:20161209:230000.452 7f81133a2000-7f81133ca000 rw-s 00000000 00:04 3637254                    /SYSV530005c7 (deleted)
 21260:20161209:230000.452 7f81133ca000-7f81133cb000 rw-p 00000000 00:00 0
 21260:20161209:230000.452 7f81133cb000-7f81133e9000 rw-p 00000000 00:00 0
 21260:20161209:230000.452 7f81133e9000-7f81133ea000 r--p 00021000 fd:00 805307863                  /usr/lib64/ld-2.17.so
 21260:20161209:230000.452 7f81133ea000-7f81133eb000 rw-p 00022000 fd:00 805307863                  /usr/lib64/ld-2.17.so
 21260:20161209:230000.452 7f81133eb000-7f81133ec000 rw-p 00000000 00:00 0
 21260:20161209:230000.452 7ffc3ce31000-7ffc3d0f3000 rw-p 00000000 00:00 0                          [stack]
 21260:20161209:230000.452 7ffc3d14d000-7ffc3d14f000 r-xp 00000000 00:00 0                          [vdso]
 21260:20161209:230000.452 ffffffffff600000-ffffffffff601000 r-xp 00000000 00:00 0                  [vsyscall]
 21260:20161209:230000.452 ================================
 21260:20161209:230000.452 Please consider attaching a disassembly listing to your bug report.
 21260:20161209:230000.452 This listing can be produced with, e.g., objdump -DSswx zabbix_server.
 21260:20161209:230000.452 ================================
 21041:20161209:230000.455 One child process died (PID:21260,exitcode/signal:1). Exiting ...
 21041:20161209:230002.905 Zabbix Server stopped. Zabbix 3.2.2 (revision 64245).
 22283:20161209:230013.237 Starting Zabbix Server. Zabbix 3.2.2 (revision 64245).
Comment by Oleksii Zagorskyi [ 2016 Dec 10 ]

Previous comment looks like duplicate of ZBX-11573

Comment by Glebs Ivanovskis (Inactive) [ 2017 Feb 03 ]

So, anyone who is experiencing this problem, how do you launch Zabbix? I mean from which directory you run a launch command, where is configuration file situated, how many daemons run at the same time, do you launch them with independent configuration files or do you edit one file between launch commands, maybe you generate configuration file on the fly and remove it once daemon has started. More details is better!

Comment by Théo Castelo N. de Araújo [ 2017 Feb 03 ]

We are facing this problem in our environment.
We install zabbix server, agent and proxy, using the sources.
Every default installation, in the / usr / * directory
Here we use Red Hat Enterprise Rhel 7, 3.10.0-327.4.4.el7.x86_64.
Configuration files all with initial values, no changes.
Contents of the startup script in systemctl

[Unit]
Description = Zabbix Server
After = syslog.target network.target mysqld.service

[Service]
Type = oneshot
ExecStart = / usr / local / sbin / zabbix_server -c /usr/local/etc/zabbix_server.conf
ExecReload = / usr / local / sbin / zabbix_server -R config_cache_reload
RemainAfterExit = yes
PIDFile = / tmp / zabbix_server.pid

[Install]
WantedBy = multi-user.target

errors in zabbix_server, in zabbix_proxy and zabbix_agent

  • zabbix_server.log
    zabbix_server [2001]: [file:'selfmon.c',line:380] lock failed: [22] Invalid argument
    zabbix_server [1969]: [file:'log.c',line:238] lock failed: [22] Invalid argument
    
  • zabbix_proxy.log
    zabbix_proxy [1086]: [file:'selfmon.c',line:380] lock failed: [22] Invalid argument
    zabbix_proxy [1089]: zabbix_proxy [1103]: [file:'selfmon.c',line:380] lock failed: [22] Invalid argument[file:'selfmon.c',line:380] lock failed: [22] Invalid argument
    
  • zabbix_agent.log
    zabbix_agentd [930]: [file:'log.c',line:238] lock failed: [22] Invalid argument
    zabbix_agentd [929]: [file:'log.c',line:238] lock failed: [22] Invalid argument
    

glebs.ivanovskis Errm... Running zabbix_server with -R but without -c if you started it with -c is not a good idea. Run time control relies on pid of main process stored in pid file and path to pid file is taken from PidFile parameter in configuration file. Without -c Zabbix will search for default configuration file in default path...

Comment by Théo Castelo N. de Araújo [ 2017 Feb 03 ]

Hummm, thanks for the tip...

but, this conf or parameter can create these errors?

Comment by Glebs Ivanovskis (Inactive) [ 2017 Feb 03 ]

Maybe. I'm trying to figure it out right now. Configuration file path is involved in generation of the key for semaphore array (this is the first column of ipcs -s output). If the path does not exist or we don't have enough permissions to use it, we generate key from '.' as a backup plan. In this case there should be this error in the log:

cannot create IPC key for path '</path/to/config>', try to create for path '.': <error description>

Generating keys from '.' isn't really good because we will end up with key conflict and all Zabbix daemons will compete for the same semaphores.

Comment by Théo Castelo N. de Araújo [ 2017 Feb 03 ]

hummm, but, this parameter, is only in Reload state:

ExecReload = / usr / local / sbin / zabbix_server -R config_cache_reload

Even then, may be generating these errors? Even without reloading services? The Parameter that I should use then, can be:

ExecReload = / usr / local / sbin / zabbix_server -R config_cache_reload -c /usr/local/etc/zabbix_server.conf

Because, the path /usr/local/etc/zabbix_server.conf, is the default path!!! It does not make sense, repeat the path to the configuration file.

Comment by Théo Castelo N. de Araújo [ 2017 Feb 03 ]

Feedback: I removed parameter -c and the command line reload in the systemctl script, so far, no errors appeared.

The script looks like this:

[Unit]
Description = Zabbix Server
After = syslog.target network.target mysqld.service

[Service]
Type = oneshot
ExecStart = / usr / local / sbin / zabbix_server
RemainAfterExit = yes
PIDFile = / tmp / zabbix_server.pid

[Install]
WantedBy = multi-user.target

I can not say that it has completely resolved, but I will monitor if it happens again.

Comment by Glebs Ivanovskis (Inactive) [ 2017 Feb 05 ]

I think it is a bit bad luck on top of misuse of ftok() from our side that causes the crash.

If you compare the first column (semaphore key) of output of this command:

$ ipcs -s | grep zabbix

with the output of

$ printf '%0.2x%0.4x\n' $(($(stat -c '%d' /path/to/config_file) & 0xff)) $(($(stat -c '%i' /path/to/config_file) & 0xffff))

you will see that the only difference is 0x7a in the beginning. This is ASCII code of 'z' and is common for all Zabbix daemons.

If you have multiple Zabbix daemons on one system with configuration files on the same drive and if configuration file inode numbers share 16 lower bits - you are in trouble - your Zabbix daemons will compete for the same semaphore set.

Problem has nothing to do with system-wide limits on semaphore set number and semaphore number in semaphore set mentioned above.

Comment by richlv [ 2017 Feb 06 ]

as far as i recall, most of these were observed during the zabbix training sessions when trainees were starting up 3 zabbix proxies on the same machine - and yes, running them with the wrong -c parameter was very likely to happen at some time.

Comment by Glebs Ivanovskis (Inactive) [ 2017 Feb 06 ]

Wrong config file usually result in a more obvious error. Bind failure due to port conflict, pid file conflict or something like that. Key conflict may happen even if daemons have separate config files. If you are interested, probability of this happening is given by this formula (assuming random inode numbers):

        65536 - 1   65536 - 2         65536 - (N - 1)
P = 1 - --------- * --------- * ... * ---------------
          65536       65536                65536

where N is a number of Zabbix daemons. With N = 2 it is negligible 1/65536 but with N = 10 it comes close to a quite noticeable 0.1%. Plus you need to multiply it by number of students

Comment by Théo Castelo N. de Araújo [ 2017 Feb 06 ]

The problem came back again...

glebs.ivanovskis Can you please check config file inode numbers (for all Zabbix daemons you have on the system) as I described above to prove or disprove my hypothesis?

Comment by richlv [ 2017 Feb 06 ]

ah, thanks a lot for the formula
if that's all the daemons running under the same user (not just daemons of the same type), then it's 5 daemons (3 proxies + agent + server). multiplied by the number of students, by the number of sessions, it indeed does make it quite likely to happen every now and then.

i now recall some users in their prod environments seeing this issue, and they were running 2 or 3 agents on some boxes. i guess restarting agents every now and then makes sure it will happen at some point.

Comment by Théo Castelo N. de Araújo [ 2017 Feb 10 ]

@Glebs Ivanovskis Can you please check config file inode numbers (for all Zabbix daemons you have on the system) as I described above to prove or disprove my hypothesis?

Sorry for the delay, i have been sick last week, this is the result:

[zabbix@srvdfzbxpdf01 ~]$ ipcs -s | grep zabbix
0x7a00e54d 294917     zabbix     600        13        
0x7a00f0a5 327686     zabbix     600        13
[zabbix@srvdfzbxpdf01 ~]$ printf '%0.2x%0.4x\n' $(($(stat -c '%d' /usr/local/etc/zabbix_server.conf) & 0xff)) $(($(stat -c '%i' /usr/local/etc/zabbix_server.conf) & 0xffff))
00f0a5

I think you right

have any way to fix this?

glebs.ivanovskis Please show printf ... for zabbix_agentd.conf and zabbix_proxy.conf. So far we can't decide.

Comment by Théo Castelo N. de Araújo [ 2017 Feb 10 ]

glebs.ivanovskis Thank you, for all your patience and help

here, we have been working in 2 servers

First Server: We have - zabbix_server and zabbix_agent

Second Server, we have only zabbix_proxy

First server:

[root@srvdfzbxpdf02 zabbix]# ipcs -s | grep zabbix
0x7a00e55e 491525     zabbix     600        13        
0x7a00e54d 524294     zabbix     600        13 

zabbix_server.conf

[root@srvdfzbxpdf02 zabbix]# printf '%0.2x%0.4x\n' $(($(stat -c '%d' /usr/local/etc/zabbix_server.conf) & 0xff)) $(($(stat -c '%i' /usr/local/etc/zabbix_server.conf) & 0xffff))
00196a

zabbix_agentd.conf

[root@srvdfzbxpdf02 zabbix]# printf '%0.2x%0.4x\n' $(($(stat -c '%d' /usr/local/etc/zabbix_agentd.conf) & 0xff)) $(($(stat -c '%i' /usr/local/etc/zabbix_agentd.conf) & 0xffff))
00e54d

Second Server:

[root@srvpxzbxpdf04 zabbix]# ipcs -s | grep zabbix
0x7a003415 98304      zabbix     600        13 
[root@srvpxzbxpdf04 zabbix]#  printf '%0.2x%0.4x\n' $(($(stat -c '%d' /usr/local/etc/zabbix_proxy.conf) & 0xff)) $(($(stat -c '%i' /usr/local/etc/zabbix_proxy.conf) & 0xffff))
003415
Comment by Glebs Ivanovskis (Inactive) [ 2017 Feb 10 ]

Looks good, no conflicting inode numbers. Don't worry that printf ... matches ipcs ... output, that's how Zabbix works currently. I guess everything is working fine for you now, isn't it?

Comment by Théo Castelo N. de Araújo [ 2017 Feb 10 ]

No,

in both Servers, in log files, i have this errors:

4039:20170210:101250.615 Got signal [signal:15(SIGTERM),sender_pid:1,sender_uid:0,reason:0]. Exiting ...
  4053:20170210:101250.615 Got signal [signal:15(SIGTERM),sender_pid:1,sender_uid:0,reason:0]. Exiting ...
  4054:20170210:101250.616 Got signal [signal:15(SIGTERM),sender_pid:1,sender_uid:0,reason:0]. Exiting ...
  4055:20170210:101250.617 Got signal [signal:15(SIGTERM),sender_pid:1,sender_uid:0,reason:0]. Exiting ...
  4056:20170210:101250.618 Got signal [signal:15(SIGTERM),sender_pid:1,sender_uid:0,reason:0]. Exiting ...
  4058:20170210:101250.619 Got signal [signal:15(SIGTERM),sender_pid:1,sender_uid:0,reason:0]. Exiting ...
  4057:20170210:101250.619 Got signal [signal:15(SIGTERM),sender_pid:1,sender_uid:0,reason:0]. Exiting ...
  4059:20170210:101250.620 Got signal [signal:15(SIGTERM),sender_pid:1,sender_uid:0,reason:0]. Exiting ...
  4060:20170210:101250.621 Got signal [signal:15(SIGTERM),sender_pid:1,sender_uid:0,reason:0]. Exiting ...
  4062:20170210:101250.621 Got signal [signal:15(SIGTERM),sender_pid:1,sender_uid:0,reason:0]. Exiting ...
  4061:20170210:101250.622 Got signal [signal:15(SIGTERM),sender_pid:1,sender_uid:0,reason:0]. Exiting ...
  4064:20170210:101250.622 Got signal [signal:15(SIGTERM),sender_pid:1,sender_uid:0,reason:0]. Exiting ...
  4063:20170210:101250.623 Got signal [signal:15(SIGTERM),sender_pid:1,sender_uid:0,reason:0]. Exiting ...
  4066:20170210:101250.623 Got signal [signal:15(SIGTERM),sender_pid:1,sender_uid:0,reason:0]. Exiting ...
  4065:20170210:101250.624 Got signal [signal:15(SIGTERM),sender_pid:1,sender_uid:0,reason:0]. Exiting ...
  4068:20170210:101250.624 Got signal [signal:15(SIGTERM),sender_pid:1,sender_uid:0,reason:0]. Exiting ...
  4067:20170210:101250.625 Got signal [signal:15(SIGTERM),sender_pid:1,sender_uid:0,reason:0]. Exiting ...
  4069:20170210:101250.625 Got signal [signal:15(SIGTERM),sender_pid:1,sender_uid:0,reason:0]. Exiting ...
  4070:20170210:101250.626 Got signal [signal:15(SIGTERM),sender_pid:1,sender_uid:0,reason:0]. Exiting ...
  4071:20170210:101250.626 Got signal [signal:15(SIGTERM),sender_pid:1,sender_uid:0,reason:0]. Exiting ...
  4074:20170210:101250.626 Got signal [signal:15(SIGTERM),sender_pid:1,sender_uid:0,reason:0]. Exiting ...
  4072:20170210:101250.627 Got signal [signal:15(SIGTERM),sender_pid:1,sender_uid:0,reason:0]. Exiting ...
  4075:20170210:101250.627 Got signal [signal:15(SIGTERM),sender_pid:1,sender_uid:0,reason:0]. Exiting ...
  4073:20170210:101250.628 Got signal [signal:15(SIGTERM),sender_pid:1,sender_uid:0,reason:0]. Exiting ...
  4076:20170210:101250.628 Got signal [signal:15(SIGTERM),sender_pid:1,sender_uid:0,reason:0]. Exiting ...
  4077:20170210:101250.629 Got signal [signal:15(SIGTERM),sender_pid:1,sender_uid:0,reason:0]. Exiting ...
  4080:20170210:101250.629 Got signal [signal:15(SIGTERM),sender_pid:1,sender_uid:0,reason:0]. Exiting ...
  4078:20170210:101250.630 Got signal [signal:15(SIGTERM),sender_pid:1,sender_uid:0,reason:0]. Exiting ...
  4082:20170210:101250.630 Got signal [signal:15(SIGTERM),sender_pid:1,sender_uid:0,reason:0]. Exiting ...
  4079:20170210:101250.631 Got signal [signal:15(SIGTERM),sender_pid:1,sender_uid:0,reason:0]. Exiting ...
  4083:20170210:101250.631 Got signal [signal:15(SIGTERM),sender_pid:1,sender_uid:0,reason:0]. Exiting ...
  4081:20170210:101250.632 Got signal [signal:15(SIGTERM),sender_pid:1,sender_uid:0,reason:0]. Exiting ...
  4084:20170210:101250.632 Got signal [signal:15(SIGTERM),sender_pid:1,sender_uid:0,reason:0]. Exiting ...
  4039:20170210:101252.623 syncing history data...
  4039:20170210:101252.623 syncing history data done
  4039:20170210:101252.623 syncing trend data...
  4039:20170210:101252.637 syncing trend data done
  4039:20170210:101252.637 Zabbix Server stopped. Zabbix 3.2.3 (revision 64610).
  

glebs.ivanovskis This looks like normal Zabbix termination. No lock failed ... errors?

Théo: Sorry, i Posted wrong log.

Comment by Théo Castelo N. de Araújo [ 2017 Feb 23 ]

I deleted the whole environment.
I installed everything from scratch, following the official tutorial: https://www.zabbix.com/documentation/3.2/manual/installation/install

I have not changed any server configuration, Red Hat Rhel 7, and even then, the error persists.

I do not know what to do anymore.

1303:20170222:173942.979 housekeeper [deleted 267404 hist/trends, 0 items, 0 events, 3 problems, 0 sessions, 0 alarms, 0 audit items in 13.121322 sec, idle for 1 hour(s)]
zabbix_server [1322]: [file:'selfmon.c',line:380] lock failed: [22] Invalid argument
zabbix_server [1213]: [file:'log.c',line:238] lock failed: [22] Invalid argument
zabbix_server [1297]: [file:'log.c',line:238] lock failed: [22] Invalid argument
zabbix_server [1296]: [file:'log.c',line:238] lock failed: [22] Invalid argument
zabbix_server [1299]: [file:'log.c',line:238] lock failed: [22] Invalid argument
zabbix_server [1298]: [file:'log.c',line:238] lock failed: [22] Invalid argument
zabbix_server [1303]: [file:'log.c',line:238] lock failed: [22] Invalid argument
zabbix_server [1300]: [file:'log.c',line:238] lock failed: [22] Invalid argument
zabbix_server [1312]: [file:'log.c',line:238] lock failed: [22] Invalid argument
zabbix_server [1307]: [file:'log.c',line:238] lock failed: [22] Invalid argument
zabbix_server [1316]: [file:'log.c',line:238] lock failed: [22] Invalid argument
zabbix_server [1311]: [file:'log.c',line:238] lock failed: [22] Invalid argument
zabbix_server [1290]: [file:'log.c',line:238] lock failed: [22] Invalid argument
zabbix_server [1315]: [file:'log.c',line:238] lock failed: [22] Invalid argument
zabbix_server [1294]: [file:'log.c',line:238] lock failed: [22] Invalid argument
zabbix_server [1295]: [file:'log.c',line:238] lock failed: [22] Invalid argument
zabbix_server [1323]: [file:'log.c',line:238] lock failed: [22] Invalid argument
zabbix_server [1289]: [file:'log.c',line:238] lock failed: [22] Invalid argument
zabbix_server [1293]: [file:'log.c',line:238] lock failed: [22] Invalid argument
zabbix_server [1291]: [file:'log.c',line:238] lock failed: [22] Invalid argument
zabbix_server [1306]: [file:'log.c',line:238] lock failed: [22] Invalid argument
zabbix_server [1308]: [file:'log.c',line:238] lock failed: [22] Invalid argument
zabbix_server [1302]: [file:'log.c',line:238] lock failed: [22] Invalid argument
zabbix_server [1317]: [file:'log.c',line:238] lock failed: [22] Invalid argument
zabbix_server [1292]: [file:'log.c',line:238] lock failed: [22] Invalid argument
zabbix_server [1288]: [file:'log.c',line:238] lock failed: [22] Invalid argument
zabbix_server [1318]: [file:'log.c',line:238] lock failed: [22] Invalid argument

Could it be some error related to Rhel 7?
Well I remember installing in another environment, Rhel 6 version 3.2.0 of Zabbix and did not give any such error.

Comment by Glebs Ivanovskis (Inactive) [ 2017 Feb 23 ]

Dear theocastelo, have you checked config file inodes after re-installation?

Since you mentioned RHEL 6/7, your issue may be more related to ZBX-11544 (see this comment), all depends on how you start Zabbix and when these errors appear in the log.

Comment by Théo Castelo N. de Araújo [ 2017 Feb 23 ]

glebs.ivanovskis I think this is it.

after changing the REMOVE_IPC parameter, now working fine...

Probably the problem was caused by the application user be the same used for ssh access, when we log off, the error happened.

Thank you so much

Comment by Glebs Ivanovskis (Inactive) [ 2017 Jul 25 ]

Situation must be much better in 3.4 after implementation of ZBXNEXT-3687. I'll dare to close this issue as Duplicate.

Generated at Thu Apr 24 07:12:35 EEST 2025 using Jira 9.12.4#9120004-sha1:625303b708afdb767e17cb2838290c41888e9ff0.