Hi! We have the same problems
I'd like to share some details – maybe, it will be useful.

First of all, we have a big enough park of different Linux servers: both a popular Linux brands (like SLES, RHEL, CentOS and Ubuntu) and an appliances (also, both based on some popular distributions like CentOS or SLES and unknown for us sources like VMware Photon or TrendMicro IWSVA). So, possibility to download a single binary (or small limited set of binaries) that able to work on any of these platforms is very valuable for us.

The first time we meet this problem after upgrade of our VMware vCenter's (6.7.0 -> 7.0.1). It is appliance (distributed as VM image), new version of OS was "VMware Photon OS 3.0" (with Linux kernel 4.19.148-5.ph3). The most mysterious fact was: we had 2 vCenter's (test and prod), from the same image; but problem took place only on one of them (of course, prod ).

Now we have a very similar problem on the Ubuntu. This is a test system, so we could experiment with it if necessary.
We had some number of previous versions (from v16.04.6 with kernel 4.4.0 up to v20.04.4 with kernel 5.4.0) – all without any problems, but this one is the first our Ubuntu having version 22.04 (LTS Jammy Jellyfish, kernel 5.15.0-46-generic).

Some additional info:

• We used to use Zabbix Agent version 6.0.4; but we tested the last versions also (6.0.9 and 6.2.3) – the problem is the same.
• Our standard config file for a Zabbix Agent has "Server=" and "ServerActive=" parameters set to FQDN DNS-name of our Zabbix server. This name, of course, presents in our DNS (both in direct and reverse zones) and is resolved successfully.
• When we try to start Zabbix Agent from the root user (like it is in the Description of this ticket):

  /usr/local/sbin/zabbix_agentd_6.0.4 -c /usr/local/etc/zabbix_agentd.conf
  

  then we also have the same message:

  Segmentation fault (core dumped)
  

  The core file itself, however, could not be found anywhere.
  Zabbix Agent does not create or write something into its log file.

• However, if we start Zabbix Agent from the zabbix user (either via sudo command:

  sudo -u zabbix /usr/local/sbin/zabbix_agentd_6.0.4 -c /usr/local/etc/zabbix_agentd.conf
  

  or via systemd unit-file and systemctl), then Zabbix Agent starts successfully, add some records onto its log file and then immediately crashes.
  Part of the log:

      1941470:20221004:100208.338 agent #5 started active checks #1
      1941470:20221004:100208.339 Got signal [signal:11(SIGSEGV),reason:1,refaddr:0x63]. Crashing ...
      1941470:20221004:100208.339 ====== Fatal information: ======
      1941470:20221004:100208.339 Program counter: 0x7fa274831ef4
      1941470:20221004:100208.339 === Registers: ===
      [...]
      1941470:20221004:100208.340 === Backtrace: ===
      1941470:20221004:100208.342 22: [0x42f095]
      1941470:20221004:100208.342 21: [0x42f3ad]
      1941470:20221004:100208.342 20: [0x42fa06]
      1941470:20221004:100208.342 19: [0x496890]
      1941470:20221004:100208.342 18: /lib/x86_64-linux-gnu/libc.so.6(__nss_readline+0x74) [0x7fa274831ef4]
      1941470:20221004:100208.342 17: /lib/x86_64-linux-gnu/libc.so.6(+0x158570) [0x7fa274835570]
      1941470:20221004:100208.342 16: /lib/x86_64-linux-gnu/libc.so.6(_nss_files_gethostbyname3_r+0x85) [0x7fa2748365c5]
      1941470:20221004:100208.342 15: /lib/x86_64-linux-gnu/libc.so.6(_nss_files_gethostbyname2_r+0x15) [0x7fa2748366f5]
      1941470:20221004:100208.342 14: [0x516df1]
      1941470:20221004:100208.342 13: [0x508f36]
  

• According to Zabbix log (and crash dump there), the crash is occurred in the "active checks” process during its first ask for a resolving of Zabbix server DNS-name.
• If we replace Zabbix Server' DNS-name by its IP-address in the "ServerActive=" parameter, then "active checks” process could start successfully, but Zabbix Agent crashes a bit later: in a one of "listener" processes during the first connection from the Zabbix Server (again: during the name resolving, with the same symptoms).
• If we use the IP-address in both "ServerActive=" and "Server=" parameters, then Zabbix Agent starts and works without problems. By the way, we used the same workaround the previous time (with vCenter).
• In both cases Zabbix Agents compiled (with shared libraries) and packed specially for the given platform (Photon or Ubuntu 22) worked without problem. However, it could be perfect to have the universal agent compiled with a static libraries to work successfully also.

Hey there, igorbach, constantin.oshmyan!

Apologies for the delay on this.

The currently published static agent builds were intended for the 2.x and 3.x series of Linux kernels, and provide no guarantees of function on later releases.

We're looking into publishing these builds again, but are currently trying to estimate the userbase of them to understand the trade-off here.

We can build entirely static agents (i.e. including a copy of libc, with the libraries), or an agent only with static libraries (i.e. openssl, pcre, curl and so on) but with a dynamic dependency on libc. The first variant would be limited just like these old static agents are, and may cease functioning with new major kernel releases. The latter would target a libc version of major distros at the time of build and would work as long as the target libc would be compatible, and continue working across different kernel releases. These latter builds would most likely be provided on a per-distro/version basis (i.e. in the case of RHEL 7, 8 and 9 would each have their own agent built).

If I get enough feedback, we'll consider adding such builds and publishing them on a permanent basis.

jlambda, thank you for a good questions

My main interest is to have a minimum number (ideally – only single) of universal agents that are able to work on specific platform (OS/architecture). For example, 1 agent for Linux x86, 1 agent for Linux x86_64 and 1 agent for Linux PowerPC_64 (independently of specific distro). Maybe, one more set for the kernels starting with some version (5.x, for example).

So, if the statically-linked agent depends of distro (has different versions for RHEL, SLES, Ubuntu, etc.), it is useless for us – in this case it will be simpler to use a dynamically-linked agent for the same distro.

an agent only with static libraries (i.e. openssl, pcre, curl and so on) but with a dynamic dependency on libc

In this case (all libraries besides the libc are static), is there a really dependency on the specific distribution?

Hey, constantin.oshmyan!

Regarding

In this case (all libraries besides the libc are static), is there a really dependency on the specific distribution?

I think I misspoke here. The completely static agents would be targeting distros (or at least kernel major versions) to be as compatible as possible. I'd expect maybe 3 unique agents per architecture here (for the 4.x, 5.x and 6.x series of kernels).

The partly static ones would then target incompatible (g)libc versions, which again, would probably be a small number.

Hi again, constantin.oshmyan!

We figured that if we do this, we'll build the partly static agents, with dynamic linking to glibc.

The rationale for this is that glibc has a history of symbol versioning and keeping compatibility with previous versions of glibc this way. This effectively means that we can build a single agent per architecture linking with as old a version of glibc as we can find, as this would be compatible with all following versions.

Relatively recent versions of glibc have introduced a minimum supported kernel version requirement (see below for example), which brings us to the question of what are your oldest and latest running kernels? If they all fall above a certain minimum required version, we could probably build these agents in our existing build environment, making this relatively easy.

glibc info output specifying minimum kernel requirement

% /lib64/libc.so.6
GNU C Library (GNU libc) stable release version 2.37.
Copyright (C) 2023 Free Software Foundation, Inc.
This is free software; see the source for copying conditions.
There is NO warranty; not even for MERCHANTABILITY or FITNESS FOR A
PARTICULAR PURPOSE.
Compiled by GNU CC version 13.0.1 20230401 (Red Hat 13.0.1-0).
libc ABIs: UNIQUE IFUNC ABSOLUTE
Minimum supported kernel: 3.2.0
For bug reporting instructions, please see:
<https://www.gnu.org/software/libc/bugs.html>.

jlambda, great news, thank you!

what are your oldest and latest running kernels?

Actually, the oldest versions of Linux kernel we use are 2.6.32 (2 old appliances based on the CentOS 6.x, should be deprecated and phased out).
There is some number of kernels 3.10.0-1160.x (another appliances, based on CentOS 7.x).
The main park contains more modern kernels (4.x/5.x). The most of systems has kernel 5.4, but there are newer ones (5.15).
The problem mentioned in the descriptions of this issue took place just with these most modern kernels (Ubuntu 22 distro) as well as, by some reason, with 4.19.269-3 (VMware Photon OS v3); all the rest works without any problems with these statically-linked agents built still for the kernel v3.

Hi everyone, including alexei (we talked about our current problems recently on Zabbix Summit 2023)!

What is the current status of this issue and perspectives to have a solution?
I can see that there are some works sometimes; it's scheduled to plan for different sprints; however, I don't see any progress during the entire year. Probably, there are some comments here that are for internal usage only and not accessible in public.

From our side, I can add that we recently updated one of systems (VMware Photon), and the old workaround (using IP-addresses instead of DNS-names in config file) does not help anymore: the statically built agent crashes anyway just after start, so this problem becomes more actual:

  5258:20231006:133807.583 Starting Zabbix Agent [vcenterdev.rietumu.lv]. Zabbix 6.0.4 (revision 3d787ff402e).
  5258:20231006:133807.583 **** Enabled features ****
  5258:20231006:133807.583 IPv6 support:           NO
  5258:20231006:133807.583 TLS support:            NO
  5258:20231006:133807.584 **************************
  5258:20231006:133807.584 using configuration file: /usr/local/etc/zabbix_agentd.conf
  5258:20231006:133807.584 agent #0 started [main process]
  5259:20231006:133807.584 agent #1 started [collector]
  5260:20231006:133807.585 agent #2 started [listener #1]
  5261:20231006:133807.596 agent #3 started [listener #2]
  5262:20231006:133807.596 agent #4 started [listener #3]
  5263:20231006:133807.596 agent #5 started [active checks #1]
  5263:20231006:133807.927 Got signal [signal:11(SIGSEGV),reason:1,refaddr:0xe5]. Crashing ...
  5263:20231006:133807.927 ====== Fatal information: ======
  5263:20231006:133807.927 Program counter: 0x7f13e9f6c2f0
  5263:20231006:133807.928 === Registers: ===
  5263:20231006:133807.928 r8      =          26f2670 =             40838768 =             40838768
  5263:20231006:133807.928 r9      =                0 =                    0 =                    0
  5263:20231006:133807.931 r10     =                0 =                    0 =                    0
  5263:20231006:133807.932 r11     =              246 =                  582 =                  582
  5263:20231006:133807.932 r12     =          26f2a6f =             40839791 =             40839791
  5263:20231006:133807.932 r13     =              400 =                 1024 =                 1024
  5263:20231006:133807.932 r14     =          26f2670 =             40838768 =             40838768
  5263:20231006:133807.932 r15     =     7f13e9d1d010 =      139723503947792 =      139723503947792
  5263:20231006:133807.932 rdi     =          26f2671 =             40838769 =             40838769
  5263:20231006:133807.932 rsi     =     7f13e9d1d1f1 =      139723503948273 =      139723503948273
  5263:20231006:133807.932 rbp     =          26f2670 =             40838768 =             40838768
  5263:20231006:133807.932 rbx     =     7fffdf28c350 =      140736937378640 =      140736937378640
  5263:20231006:133807.932 rdx     =               72 =                  114 =                  114
  5263:20231006:133807.932 rax     =               72 =                  114 =                  114
  5263:20231006:133807.932 rcx     =                0 =                    0 =                    0
  5263:20231006:133807.932 rsp     =     7fffdf28c300 =      140736937378560 =      140736937378560
  5263:20231006:133807.933 rip     =     7f13e9f6c2f0 =      139723506369264 =      139723506369264
  5263:20231006:133807.939 efl     =            10246 =                66118 =                66118
  5263:20231006:133807.939 csgsfs  =   2b000000000033 =    12103423998558259 =    12103423998558259
  5263:20231006:133807.939 err     =                4 =                    4 =                    4
  5263:20231006:133807.939 trapno  =                e =                   14 =                   14
  5263:20231006:133807.941 oldmask =                0 =                    0 =                    0
  5263:20231006:133807.942 cr2     =               e5 =                  229 =                  229
  5263:20231006:133807.942 === Backtrace: ===
  5263:20231006:133807.944 18: [0x42ed45]
  5263:20231006:133807.976 17: [0x42f05d]
  5263:20231006:133807.976 16: [0x42f6b6]
  5263:20231006:133807.976 15: [0x497090]
  5263:20231006:133807.976 14: /lib/libc.so.6(__nss_readline+0x70) [0x7f13e9f6c2f0]
  5263:20231006:133807.976 13: /lib/libnss_files.so.2(+0x635d) [0x7f13ea01635d]
  5263:20231006:133807.976 12: /lib/libnss_files.so.2(_nss_files_getpwnam_r+0x4c) [0x7f13ea01661c]
  5263:20231006:133807.976 11: [0x505584]
  5263:20231006:133807.976 10: [0x5051cd]
  5263:20231006:133807.976 9: [0x419e56]
  5263:20231006:133807.977 8: [0x413b9d]
  5263:20231006:133807.977 7: [0x406b6d]
  5263:20231006:133807.977 6: [0x42d75e]
  5263:20231006:133807.977 5: [0x4040ee]
  5263:20231006:133807.977 4: [0x42e5b2]
  5263:20231006:133807.978 3: [0x40228f]
  5263:20231006:133807.978 2: [0x49d4c4]
  5263:20231006:133807.978 1: [0x49d741]
  5263:20231006:133807.978 0: [0x402616]
  5263:20231006:133807.978 === Memory map: ===
  5263:20231006:133807.979 00400000-005b9000 r-xp 00000000 fe:00 2633554                            /usr/local/sbin/zabbix_agentd
  5263:20231006:133807.979 007b8000-007bc000 rw-p 001b8000 fe:00 2633554                            /usr/local/sbin/zabbix_agentd
  5263:20231006:133807.979 007bc000-007c9000 rw-p 00000000 00:00 0
  5263:20231006:133807.980 026e9000-0270c000 rw-p 00000000 00:00 0                                  [heap]
  5263:20231006:133807.980 7f13e9d1d000-7f13e9e1d000 rw-p 00000000 00:00 0
  5263:20231006:133807.980 7f13e9e1d000-7f13e9e1e000 r--p 00000000 fe:00 2490755                    /usr/lib/ld-2.32.so
  5263:20231006:133807.980 7f13e9e1e000-7f13e9e3e000 r-xp 00001000 fe:00 2490755                    /usr/lib/ld-2.32.so
  5263:20231006:133807.981 7f13e9e3e000-7f13e9e46000 r--p 00021000 fe:00 2490755                    /usr/lib/ld-2.32.so
  5263:20231006:133807.981 7f13e9e46000-7f13e9e47000 ---p 00029000 fe:00 2490755                    /usr/lib/ld-2.32.so
  5263:20231006:133807.981 7f13e9e47000-7f13e9e48000 r--p 00029000 fe:00 2490755                    /usr/lib/ld-2.32.so
  5263:20231006:133807.981 7f13e9e48000-7f13e9e4a000 rw-p 0002a000 fe:00 2490755                    /usr/lib/ld-2.32.so
  5263:20231006:133807.982 7f13e9e4a000-7f13e9e70000 r--p 00000000 fe:00 2490439                    /usr/lib/libc-2.32.so
  5263:20231006:133807.982 7f13e9e70000-7f13e9fba000 r-xp 00026000 fe:00 2490439                    /usr/lib/libc-2.32.so
  5263:20231006:133807.982 7f13e9fba000-7f13ea006000 r--p 00170000 fe:00 2490439                    /usr/lib/libc-2.32.so
  5263:20231006:133807.982 7f13ea006000-7f13ea009000 r--p 001bb000 fe:00 2490439                    /usr/lib/libc-2.32.so
  5263:20231006:133807.982 7f13ea009000-7f13ea00c000 rw-p 001be000 fe:00 2490439                    /usr/lib/libc-2.32.so
  5263:20231006:133807.989 7f13ea00c000-7f13ea010000 rw-p 00000000 00:00 0
  5263:20231006:133807.990 7f13ea010000-7f13ea013000 r--p 00000000 fe:00 2490458                    /usr/lib/libnss_files-2.32.so
  5263:20231006:133807.990 7f13ea013000-7f13ea019000 r-xp 00003000 fe:00 2490458                    /usr/lib/libnss_files-2.32.so
  5263:20231006:133807.990 7f13ea019000-7f13ea01b000 r--p 00009000 fe:00 2490458                    /usr/lib/libnss_files-2.32.so
  5263:20231006:133807.990 7f13ea01b000-7f13ea01c000 r--p 0000a000 fe:00 2490458                    /usr/lib/libnss_files-2.32.so
  5263:20231006:133807.990 7f13ea01c000-7f13ea01d000 rw-p 0000b000 fe:00 2490458                    /usr/lib/libnss_files-2.32.so
  5263:20231006:133807.990 7f13ea01d000-7f13ea023000 rw-p 00000000 00:00 0
  5263:20231006:133807.991 7f13ea028000-7f13ea029000 rw-p 00000000 00:00 0
  5263:20231006:133807.991 7f13ea029000-7f13ea05f000 rw-s 00000000 00:01 13                         /SYSV00000000 (deleted)
  5263:20231006:133807.991 7f13ea05f000-7f13ea060000 rw-p 00000000 00:00 0
  5263:20231006:133807.991 7fffdf26f000-7fffdf290000 rw-p 00000000 00:00 0                          [stack]
  5263:20231006:133807.991 7fffdf3c8000-7fffdf3cc000 r--p 00000000 00:00 0                          [vvar]
  5263:20231006:133807.991 7fffdf3cc000-7fffdf3ce000 r-xp 00000000 00:00 0                          [vdso]
  5263:20231006:133807.991 ffffffffff600000-ffffffffff601000 r-xp 00000000 00:00 0                  [vsyscall]
  5263:20231006:133807.991 ================================
  5263:20231006:133807.991 Please consider attaching a disassembly listing to your bug report.
  5263:20231006:133807.991 This listing can be produced with, e.g., objdump -DSswx zabbix_agentd.
  5263:20231006:133807.991 ================================
  5258:20231006:133807.992 One child process died (PID:5263,exitcode/signal:1). Exiting ...
zabbix_agentd [5258]: Error waiting for process with PID 5263: [10] No child processes
  5258:20231006:133808.003 Zabbix Agent stopped. Zabbix 6.0.4 (revision 3d787ff402e).

We do also need this functionality for either old Linux OS with no repo available, or OS like Vmware vCenter, where we can just drop the binary, like Constantin does.

It has been decided to close this particular ticket due to this not being an actual bug. It is expected that static builds targeting previous Linux kernel mainline versions will be incompatible with future ones.

Internally, we've been discussing on how to address this issue overall, but cannot provide a detailed timeline.

For the time being, we suggest building the agent component yourselves until any further notice.