[ZBX-24664] Zabbix Server Terminates upon external item sub process exiting with code 255 Created: 2024 Jun 17 Updated: 2024 Aug 09 Resolved: 2024 Aug 09 |
|
| Status: | Closed |
| Project: | ZABBIX BUGS AND ISSUES |
| Component/s: | Server (S) |
| Affects Version/s: | 7.0.0 |
| Fix Version/s: | None |
| Type: | Incident report | Priority: | Trivial |
| Reporter: | Patrice Gautier | Assignee: | Aigars Kadikis |
| Resolution: | Duplicate | Votes: | 0 |
| Labels: | None | ||
| Remaining Estimate: | Not Specified | ||
| Time Spent: | Not Specified | ||
| Original Estimate: | Not Specified | ||
| Issue Links: |
|
||||||||
| Description |
|
Hi, I have a shell script set up as an external item. It in turn invokes ssh. Since I installed 7.0, the zabbix server and proxies that use that external item exit altogether when that ssh command exits with code 255. This is the log I see in the server: 1:20240616:160731.803 One child process died (PID:18945,exitcode/signal:0). Exiting ... Tracking down pid #18945 it looks like ssh -o PubkeyAcceptedKeyTypes=+ssh-rsa -o HostKeyAlgorithms=+ssh-rsa -o LogLevel=Error -o StrictHostKeyChecking=accept-new -i /var/lib/zabbix/ssh_keys/zb_id_rsa [email protected] This external item has been in place since 5.x, this behavior is new in 7.0 |
| Comments |
| Comment by Aigars Kadikis [ 2024 Jun 18 ] |
|
Please see "dmesg" (or /var/log/messages) and confirm that it's not an OOM (out of memory) situation. dmesg | grep -i oom cat /var/log/messages | grep -i oom cat /var/log/syslog | grep -i oom Please upload all original logs and zabbix_server.conf to https://space.zabbix.com/s/QNXFZSoMBwNprGd I am not sure if an External check (with an SSH command inside) is the root cause. If you see that data from the external script was never delivered to Zabbix, then possibly. |
| Comment by Patrice Gautier [ 2024 Jun 20 ] |
|
Some answers and new details.. • both the server and the one proxy are running in a docker container.. • no OOM errors I can find in either case in the docker logs • I am putting the docker-compose.yml for the server as well as server and proxy logs that exhibit the problem in the area |
| Comment by Aigars Kadikis [ 2024 Jun 25 ] |
|
Thank you for update. From the log, we see that PID 1 receives a signal 255. Since the "zabbix_server" component stays online for a few minutes, try to increase log level for PID:1 podman exec -it zabbixServer zabbix_server -R log_level_increase=1 podman exec -it zabbixServer zabbix_server -R log_level_increase=1 (command needs to be executed 2 times) Upload additional logs. If you say that proxy container exits too with exit code 255, then we should better study proxy logs (less components run in parallel). |
| Comment by Patrice Gautier [ 2024 Jun 26 ] |
|
Ok - I added a new file zp.log that contains logs from the proxy reproducing the problem with the log level upped in the upload area.. |
| Comment by Aigars Kadikis [ 2024 Jul 01 ] |
|
Thank you for providing proxy log. I see the PID 1 goes stopped: 1:20240625:163347.465 In zbx_free_database_cache() 1:20240625:163347.465 In DCsync_all() 1:20240625:163347.465 In sync_history_cache_full() history_num:1 1:20240625:163347.465 syncing history data... .. 1:20240625:163347.468 Zabbix Proxy stopped. Zabbix 7.0.0 (revision 49955f1). 1) Could you increase proxy log to level 5? Run this command 2 times: podman exec -it nameOfConrainer zabbix_proxy -R log_level_increase=1 As a result there must be a line: "log level has been increased to 5". Share the log. 2) Attach characteristics of proxy container: docker inspect nameOfContainer 3) If we think the SSH checks ir problem, could you create another proxy container, but not use SSH agent checks at all. See how stable it stays. We can also create a proxy which use only SSH checks. Set debug level 5 if it's stopping. |
| Comment by Patrice Gautier [ 2024 Jul 03 ] |
|
Hi, I uploaded the logs from the proxy with log level 5 reproducing the problem in the upload area (zb2.log.gz). Here is the docker inspect:
patrice@syn26000:~ % docker inspect zabbixProxy
[
{
"Id": "96b59aa9660d8ec878823ba1a9193ccf93b0bbcab3863253a6425e146db40546",
"Created": "2024-06-25T22:11:19.19311639Z",
"Path": "/usr/bin/docker-entrypoint.sh",
"Args": [
"/usr/sbin/zabbix_proxy",
"--foreground",
"-c",
"/etc/zabbix/zabbix_proxy.conf"
],
"State": {
"Status": "running",
"Running": true,
"Paused": false,
"Restarting": false,
"OOMKilled": false,
"Dead": false,
"Pid": 22452,
"ExitCode": 0,
"Error": "",
"StartedAt": "2024-07-03T19:54:01.328712091Z",
"FinishedAt": "2024-07-03T19:53:58.472011438Z",
"StartedTs": 1720036441,
"FinishedTs": 1720036438
},
"Image": "sha256:e18060e27b7618b9996ee25bbf1c4e0bb1a083fc9511fe9cbade7b00b2a233fd",
"ResolvConfPath": "/var/services/homes/patrice/Developer/utils/sites/resolv26000.conf",
"HostnamePath": "/volume3/@docker/containers/96b59aa9660d8ec878823ba1a9193ccf93b0bbcab3863253a6425e146db40546/hostname",
"HostsPath": "/var/services/homes/patrice/Developer/utils/localHosts",
"LogPath": "/volume3/@docker/containers/96b59aa9660d8ec878823ba1a9193ccf93b0bbcab3863253a6425e146db40546/log.db",
"Name": "/zabbixProxy",
"RestartCount": 11,
"Driver": "btrfs",
"Platform": "linux",
"MountLabel": "",
"ProcessLabel": "",
"AppArmorProfile": "docker-default",
"ExecIDs": null,
"HostConfig": {
"Binds": [
"/var/services/homes/patrice/siteElf/cache/wrapBatch:/var/lib/zabbix/siteElf/cache/wrapBatch:rw",
"/var/services/homes/patrice/Deployment/zabbixProxy/sshKeys:/var/lib/zabbix/ssh_keys:rw",
"/var/services/homes/patrice/Developer/utils:/var/lib/zabbix/Developer/utils:ro",
"/var/services/homes/patrice/Developer/siteElf:/var/lib/zabbix/Developer/siteElf:ro",
"/var/services/homes/patrice/Developer/utils/sites/resolv26000.conf:/etc/resolv.conf:ro",
"/var/services/homes/patrice/Developer/utils/localHosts:/etc/hosts:ro",
"/var/services/homes/patrice/Deployment/logs:/var/lib/zabbix/Deployment/logs:ro",
"/var/services/homes/patrice/Developer/utils/zabbix/zabbixPostStartInContainer.sh:/etc/zabbixPostStartInContainer.sh:ro",
"/var/services/homes/patrice/Developer/utils/zabbix/zabbixPostUpdateInContainer.sh:/etc/zabbixPostUpdateInContainer.sh:ro",
"/var/services/homes/patrice/Deployment/zabbixProxy/dbPassword.txt:/etc/dbPassword.txt:ro",
"/var/services/homes/patrice/Developer/unifiZabbix:/var/lib/zabbix/Developer/unifiZabbix:ro"
],
"ContainerIDFile": "",
"LogConfig": {
"Type": "db",
"Config": {}
},
"NetworkMode": "26000_allaccess_lan",
"PortBindings": {},
"RestartPolicy": {
"Name": "unless-stopped",
"MaximumRetryCount": 0
},
"AutoRemove": false,
"VolumeDriver": "",
"VolumesFrom": null,
"CapAdd": null,
"CapDrop": null,
"CgroupnsMode": "host",
"Dns": [],
"DnsOptions": [],
"DnsSearch": [],
"ExtraHosts": [],
"GroupAdd": null,
"IpcMode": "private",
"Cgroup": "",
"Links": null,
"OomScoreAdj": 0,
"PidMode": "",
"Privileged": false,
"PublishAllPorts": false,
"ReadonlyRootfs": false,
"SecurityOpt": null,
"UTSMode": "",
"UsernsMode": "",
"ShmSize": 67108864,
"Runtime": "runc",
"Env": [
"PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin",
"TERM=xterm",
"ZBX_VERSION=7.0.0",
"ZBX_SOURCES=https://git.zabbix.com/scm/zbx/zabbix.git",
"MIBDIRS=/var/lib/mibs/ietf:/var/lib/mibs/iana:/usr/share/snmp/mibs:/var/lib/zabbix/mibs",
"MIBS=+ALL",
"NMAP_PRIVILEGED=",
"ZBX_ALLOWUNSUPPORTEDDBVERSIONS=1",
"MYSQL_USER=zabbix",
"ZBX_SERVER_HOST=zabbixServerV4.gautiers.name",
"ZBX_HOSTNAME=syn26000mon",
"MYSQL_PASSWORD_FILE=/etc/dbPassword.txt",
"ZBX_VALUECACHESIZE=32M",
"SE_SITE=26000",
"SITE_ELF_DIR=/var/lib/zabbix/bin",
"DB_SERVER_HOST=zabbixProxyDB.gautiers.name",
"ZBX_DEBUGLEVEL=3",
"ZBX_HISTORYCACHESIZE=128M",
"ZBX_STARTDISCOVERERS=3",
"ZBX_TIMEOUT=15",
"ZBX_CACHESIZE=1G",
"ZBX_STARTPOLLERS=10",
"ZBX_STARTPINGERS=4",
"ZBX_HISTORYINDEXCACHESIZE=32M",
"TZ=America/Los_Angeles",
"MYSQL_DATABASE=zabbix_proxy"
],
"ConsoleSize": [
0,
0
],
"Isolation": "",
"CpuShares": 0,
"Memory": 0,
"NanoCpus": 0,
"CgroupParent": "",
"BlkioWeight": 0,
"BlkioWeightDevice": null,
"BlkioDeviceReadBps": null,
"BlkioDeviceWriteBps": null,
"BlkioDeviceReadIOps": null,
"BlkioDeviceWriteIOps": null,
"CpuPeriod": 0,
"CpuQuota": 0,
"CpuRealtimePeriod": 0,
"CpuRealtimeRuntime": 0,
"CpusetCpus": "",
"CpusetMems": "",
"Devices": null,
"DeviceCgroupRules": null,
"DeviceRequests": null,
"KernelMemory": 0,
"KernelMemoryTCP": 0,
"MemoryReservation": 0,
"MemorySwap": 0,
"MemorySwappiness": null,
"OomKillDisable": false,
"PidsLimit": null,
"Ulimits": null,
"CpuCount": 0,
"CpuPercent": 0,
"IOMaximumIOps": 0,
"IOMaximumBandwidth": 0,
"MaskedPaths": [
"/proc/asound",
"/proc/acpi",
"/proc/kcore",
"/proc/keys",
"/proc/latency_stats",
"/proc/timer_list",
"/proc/timer_stats",
"/proc/sched_debug",
"/proc/scsi",
"/sys/firmware"
],
"ReadonlyPaths": [
"/proc/bus",
"/proc/fs",
"/proc/irq",
"/proc/sys",
"/proc/sysrq-trigger"
]
},
"GraphDriver": {
"Data": null,
"Name": "btrfs"
},
"Mounts": [
{
"Type": "bind",
"Source": "/var/services/homes/patrice/Developer/utils/zabbix/zabbixPostStartInContainer.sh",
"Destination": "/etc/zabbixPostStartInContainer.sh",
"Mode": "ro",
"RW": false,
"Propagation": "rprivate"
},
{
"Type": "bind",
"Source": "/var/services/homes/patrice/Developer/utils/zabbix/zabbixPostUpdateInContainer.sh",
"Destination": "/etc/zabbixPostUpdateInContainer.sh",
"Mode": "ro",
"RW": false,
"Propagation": "rprivate"
},
{
"Type": "bind",
"Source": "/var/services/homes/patrice/Deployment/logs",
"Destination": "/var/lib/zabbix/Deployment/logs",
"Mode": "ro",
"RW": false,
"Propagation": "rprivate"
},
{
"Type": "bind",
"Source": "/var/services/homes/patrice/Developer/siteElf",
"Destination": "/var/lib/zabbix/Developer/siteElf",
"Mode": "ro",
"RW": false,
"Propagation": "rprivate"
},
{
"Type": "bind",
"Source": "/var/services/homes/patrice/Developer/unifiZabbix",
"Destination": "/var/lib/zabbix/Developer/unifiZabbix",
"Mode": "ro",
"RW": false,
"Propagation": "rprivate"
},
{
"Type": "bind",
"Source": "/var/services/homes/patrice/Developer/utils",
"Destination": "/var/lib/zabbix/Developer/utils",
"Mode": "ro",
"RW": false,
"Propagation": "rprivate"
},
{
"Type": "bind",
"Source": "/var/services/homes/patrice/Deployment/zabbixProxy/sshKeys",
"Destination": "/var/lib/zabbix/ssh_keys",
"Mode": "rw",
"RW": true,
"Propagation": "rprivate"
},
{
"Type": "bind",
"Source": "/var/services/homes/patrice/Deployment/zabbixProxy/dbPassword.txt",
"Destination": "/etc/dbPassword.txt",
"Mode": "ro",
"RW": false,
"Propagation": "rprivate"
},
{
"Type": "bind",
"Source": "/var/services/homes/patrice/Developer/utils/localHosts",
"Destination": "/etc/hosts",
"Mode": "ro",
"RW": false,
"Propagation": "rprivate"
},
{
"Type": "bind",
"Source": "/var/services/homes/patrice/Developer/utils/sites/resolv26000.conf",
"Destination": "/etc/resolv.conf",
"Mode": "ro",
"RW": false,
"Propagation": "rprivate"
},
{
"Type": "bind",
"Source": "/var/services/homes/patrice/siteElf/cache/wrapBatch",
"Destination": "/var/lib/zabbix/siteElf/cache/wrapBatch",
"Mode": "rw",
"RW": true,
"Propagation": "rprivate"
},
{
"Type": "volume",
"Name": "db0b294d7763db48d038b86f4f20921796d9f1b2bafe1303df853c0a4d1da5cf",
"Source": "/volume3/@docker/volumes/db0b294d7763db48d038b86f4f20921796d9f1b2bafe1303df853c0a4d1da5cf/_data",
"Destination": "/var/lib/zabbix/snmptraps",
"Driver": "local",
"Mode": "",
"RW": true,
"Propagation": ""
}
],
"Config": {
"Hostname": "96b59aa9660d",
"Domainname": "",
"User": "1997",
"AttachStdin": false,
"AttachStdout": true,
"AttachStderr": true,
"ExposedPorts": {
"10051/tcp": {}
},
"Tty": false,
"OpenStdin": false,
"StdinOnce": false,
"Env": [
"PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin",
"TERM=xterm",
"ZBX_VERSION=7.0.0",
"ZBX_SOURCES=https://git.zabbix.com/scm/zbx/zabbix.git",
"MIBDIRS=/var/lib/mibs/ietf:/var/lib/mibs/iana:/usr/share/snmp/mibs:/var/lib/zabbix/mibs",
"MIBS=+ALL",
"NMAP_PRIVILEGED=",
"ZBX_ALLOWUNSUPPORTEDDBVERSIONS=1",
"MYSQL_USER=zabbix",
"ZBX_SERVER_HOST=zabbixServerV4.gautiers.name",
"ZBX_HOSTNAME=syn26000mon",
"MYSQL_PASSWORD_FILE=/etc/dbPassword.txt",
"ZBX_VALUECACHESIZE=32M",
"SE_SITE=26000",
"SITE_ELF_DIR=/var/lib/zabbix/bin",
"DB_SERVER_HOST=zabbixProxyDB.gautiers.name",
"ZBX_DEBUGLEVEL=3",
"ZBX_HISTORYCACHESIZE=128M",
"ZBX_STARTDISCOVERERS=3",
"ZBX_TIMEOUT=15",
"ZBX_CACHESIZE=1G",
"ZBX_STARTPOLLERS=10",
"ZBX_STARTPINGERS=4",
"ZBX_HISTORYINDEXCACHESIZE=32M",
"TZ=America/Los_Angeles",
"MYSQL_DATABASE=zabbix_proxy"
],
"Cmd": [
"/usr/sbin/zabbix_proxy",
"--foreground",
"-c",
"/etc/zabbix/zabbix_proxy.conf"
],
"Image": "zabbix/zabbix-proxy-mysql:ubuntu-7.0.0",
"Volumes": {
"/var/lib/zabbix/snmptraps": {}
},
"WorkingDir": "/var/lib/zabbix",
"Entrypoint": [
"/usr/bin/docker-entrypoint.sh"
],
"OnBuild": null,
"Labels": {
"com.docker.compose.config-hash": "681d4fc41cb896074936881bb3d9ac97f5175eb61efb7b5ee9ae298b72a23bef",
"com.docker.compose.container-number": "1",
"com.docker.compose.depends_on": "",
"com.docker.compose.image": "sha256:e18060e27b7618b9996ee25bbf1c4e0bb1a083fc9511fe9cbade7b00b2a233fd",
"com.docker.compose.oneoff": "False",
"com.docker.compose.project": "zabbixproxy",
"com.docker.compose.project.config_files": "/var/services/homes/patrice/Deployment/zabbixProxy/docker-compose.yml",
"com.docker.compose.project.working_dir": "/var/services/homes/patrice/Deployment/zabbixProxy",
"com.docker.compose.service": "zabbixProxy",
"com.docker.compose.version": "2.9.0",
"org.opencontainers.image.authors": "Alexey Pustovalov <[email protected]>",
"org.opencontainers.image.created": "2024-06-04T08:20:00.720Z",
"org.opencontainers.image.description": "Zabbix proxy with MySQL database support",
"org.opencontainers.image.documentation": "https://www.zabbix.com/documentation/7.0/manual/installation/containers",
"org.opencontainers.image.licenses": "AGPL v3.0",
"org.opencontainers.image.ref.name": "ubuntu",
"org.opencontainers.image.revision": "ab0eb56b4f37916dd38380a4ee99488eeb8d5da4",
"org.opencontainers.image.source": "https://git.zabbix.com/scm/zbx/zabbix.git",
"org.opencontainers.image.title": "Zabbix proxy (MySQL)",
"org.opencontainers.image.url": "https://zabbix.com/",
"org.opencontainers.image.vendor": "Zabbix SIA",
"org.opencontainers.image.version": "7.0.0",
"siteElf.containerReference": "https://hub.docker.com/r/zabbix/zabbix-server",
"siteElf.postRestartCommand": "/etc/zabbixPostStartInContainer.sh",
"siteElf.postRestartUser": "root",
"siteElf.postUpdateCommand": "/etc/zabbixPostUpdateInContainer.sh",
"siteElf.postUpdateUser": "root",
"siteElf.releaseNotes": "https://www.zabbix.com/release_notes",
"siteElf.tagConstraint": ">6.2",
"siteElf.tagPrefix": "ubuntu-"
},
"StopSignal": "SIGTERM",
"DDSM": false
},
"NetworkSettings": {
"Bridge": "",
"SandboxID": "02506073a55a93290dd6861c295eaf63838ac77499b0def3655b3be871e1598e",
"HairpinMode": false,
"LinkLocalIPv6Address": "",
"LinkLocalIPv6PrefixLen": 0,
"Ports": {},
"SandboxKey": "/var/run/docker/netns/02506073a55a",
"SecondaryIPAddresses": null,
"SecondaryIPv6Addresses": null,
"EndpointID": "",
"Gateway": "",
"GlobalIPv6Address": "",
"GlobalIPv6PrefixLen": 0,
"IPAddress": "",
"IPPrefixLen": 0,
"IPv6Gateway": "",
"MacAddress": "",
"Networks": {
"26000_allaccess_lan": {
"IPAMConfig": {
"IPv4Address": "192.168.205.6"
},
"Links": null,
"Aliases": [
"zabbixProxy",
"zabbixProxy",
"96b59aa9660d"
],
"NetworkID": "e45f359de117f45d60ff61064aa7ac6b4a49d1609a73fd13d5eebcc4fe1f6ac6",
"EndpointID": "5bbe9f387f5c5f64fee06c37f8f60e9dad2c0a670f8e998227be69af0bbd11f3",
"Gateway": "192.168.205.1",
"IPAddress": "192.168.205.6",
"IPPrefixLen": 24,
"IPv6Gateway": "2601:647:5800:add0::1",
"GlobalIPv6Address": "2601:647:5800:add0::5",
"GlobalIPv6PrefixLen": 64,
"MacAddress": "02:42:c0:a8:cd:06",
"DriverOpts": null
}
}
}
}
]
Finally if I disabled the ssh items, everything is 100% stable |
| Comment by Aigars Kadikis [ 2024 Jul 04 ] |
|
A similar issue is registered and confirmed under https://support.zabbix.com/browse/ZBX-24779. This means that if we disable all notification capabilities, the crash should not happen. Are there any notifications linked to SSH items that could generate alerts frequently? |
| Comment by Patrice Gautier [ 2024 Jul 23 ] |
|
A couple of new pieces of info: • reading the comments on https://support.zabbix.com/browse/ZBX-24779 I restarted the server and proxy containers with init=true • updated to 7.01 After those 2 changes things have been steady |
| Comment by Vladislavs Sokurenko [ 2024 Aug 07 ] |
|
Thank you for your report, closing as a duplicate of |