[ZBX-13055] Use of _res in multithreaded programs is problematical Created: 2017 Nov 16  Updated: 2024 Apr 10  Resolved: 2018 Feb 08

Status: Closed
Project: ZABBIX BUGS AND ISSUES
Component/s: Agent (G)
Affects Version/s: 2.2.21rc1, 3.0.12, 3.2.9, 3.2.10
Fix Version/s: 3.0.15rc1, 3.4.7rc2, 4.0.0alpha4, 4.0 (plan)

Type: Problem report Priority: Major
Reporter: Håvard Eidnes Assignee: Vladislavs Sokurenko
Resolution: Fixed Votes: 0
Labels: patch
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified
Environment:

NetBSD/i386 7.1

Attachments: HTML File diff    
Team: Team A
Team: Team A
Sprint: Sprint 21, Sprint 22, Sprint 23, Sprint 24, Sprint 25, Sprint 27
Story Points: 1

 Description   

Steps to reproduce:

Build zabbix 3.2.x, try to run it.
You end up with this entry in the zabbix_agentd.log file:

_res is not supported for multi-threaded programs.

and zabbix_agentd will abort(), via the "trap" NetBSD has
for this attempted usage in its pthread library.

See attached patch file...
The use of the new resolver interface should probably be
conditional on an autoconf test for whether it is available;
the supplied patch does not contain any machinery for that.

 Comments   
Comment by Andrea Biscuola (Inactive) [ 2017 Nov 17 ]

Indeed, code should be changed for using the proper API. Zabbix itself
is not multi-threaded (multiple processes in the same address space),
but strictly multi-process with shared memory segments for common
data. However, some libraries zabbix depend on could use threads
internally.
Not all the platform advise you in this way, but IMHO, we should fix the
bug starting from your patch and if one day, for some motivations, zabbix
will use pthreads, we will have this already fixed.

Comment by Andrea Biscuola (Inactive) [ 2017 Nov 17 ]

ingus.vilnis

It's not a bug as there is no technical defects in zabbix due to this.
It should be classified as a change request, as NetBSD 7.1 is
suggesting to use a safer API for DNS resolution.
It should also have high priority as NetBSD is a platform that
the zabbix agent support and due to this, the agent is prevented
from executing properly.

Comment by Håvard Eidnes [ 2017 Nov 17 ]

You're correct that zabbix-agentd itself isn't explicitly linked with -lpthread.
In my case, it's libcurl.so which pulls in -lpthread.

Comment by Håvard Eidnes [ 2017 Nov 20 ]

I'm sorry, I have a correction to my patch.
Instead of calling res_nclose(), it should instead
call res_ndestroy(); otherwise I get an fd leak
in the zabbix agent daemon.

Comment by Valdis Kauķis (Inactive) [ 2017 Nov 22 ]

heidnes Can you, please, paste or attach the output of

ldd sbin/zabbix_agentd
ldd /usr/pkg/lib/libcurl.so.4
uname -a

and also your ./configure line, since I can not reproduce the issue:

# ./configure CFLAGS='-Wall' LD_FLAGS=-lpthread --prefix=$PWD  --enable-agent  --with-mysql --with-libpcre=/usr/pkg --with-libcurl
# ldd sbin/zabbix_agentd
sbin/zabbix_agentd:
	-lcurl.4 => /usr/pkg/lib/libcurl.so.4
	-lssl.10 => /usr/lib/libssl.so.10
	-lcrypto.8 => /usr/lib/libcrypto.so.8
	-lcrypt.1 => /usr/lib/libcrypt.so.1
	-lgcc_s.1 => /usr/lib/libgcc_s.so.1
	-lc.12 => /usr/lib/libc.so.12
	-lgssapi.10 => /usr/lib/libgssapi.so.10
	-lkrb5.26 => /usr/lib/libkrb5.so.26
	-lhx509.5 => /usr/lib/libhx509.so.5
	-lasn1.9 => /usr/lib/libasn1.so.9
	-lcom_err.7 => /usr/lib/libcom_err.so.7
	-lroken.19 => /usr/lib/libroken.so.19
	-lutil.7 => /usr/lib/libutil.so.7
	-lwind.0 => /usr/lib/libwind.so.0
	-lheimbase.1 => /usr/lib/libheimbase.so.1
	-lheimntlm.4 => /usr/lib/libheimntlm.so.4
	-lz.1 => /usr/lib/libz.so.1
	-lkvm.6 => /usr/lib/libkvm.so.6
	-lm.0 => /usr/lib/libm.so.0
	-lexecinfo.0 => /usr/lib/libexecinfo.so.0
	-lelf.1 => /usr/lib/libelf.so.1
	-lpcreposix.0 => /usr/pkg/lib/libpcreposix.so.0
	-lpcre.1 => /usr/pkg/lib/libpcre.so.1
# ldd /usr/pkg/lib/libcurl.so.4
/usr/pkg/lib/libcurl.so.4:
	-lssl.10 => /usr/lib/libssl.so.10
	-lcrypto.8 => /usr/lib/libcrypto.so.8
	-lcrypt.1 => /usr/lib/libcrypt.so.1
	-lgcc_s.1 => /usr/lib/libgcc_s.so.1
	-lc.12 => /usr/lib/libc.so.12
	-lgssapi.10 => /usr/lib/libgssapi.so.10
	-lkrb5.26 => /usr/lib/libkrb5.so.26
	-lhx509.5 => /usr/lib/libhx509.so.5
	-lasn1.9 => /usr/lib/libasn1.so.9
	-lcom_err.7 => /usr/lib/libcom_err.so.7
	-lroken.19 => /usr/lib/libroken.so.19
	-lutil.7 => /usr/lib/libutil.so.7
	-lwind.0 => /usr/lib/libwind.so.0
	-lheimbase.1 => /usr/lib/libheimbase.so.1
	-lheimntlm.4 => /usr/lib/libheimntlm.so.4
	-lz.1 => /usr/lib/libz.so.1

As you can see, libpthread does not get linked. Where did you get libcurl? I tried

ftp://ftp.netbsd.org/pub/pkgsrc/packages/NetBSD/x86_64/7.0_current/All/curl-7.54.1.tgz
ftp://ftp.netbsd.org/pub/pkgsrc/packages/NetBSD/x86_64/7.0_2016Q4/All/p5-WWW-Curl-4.17nb4.tgz
ftp://ftp.netbsd.org/pub/pkgsrc/packages/NetBSD/x86_64/8.0_current/All/lua53-curl-0.2nb13.tgz

Also, do I understand correctly, that the issue only affects keys "net.dns", "net.dns.record", "net.tcp.dns" and "net.tcp.dns.query" as nowhere else is the dns_query() function invoked:

# bin/zabbix_get -s localhost -k net.dns.record[,aa.com]
aa.com SOA asia3.akam.net hostmaster.akamai.com 2013041589 9600 3600 604800 900

Comment by Håvard Eidnes [ 2017 Nov 22 ]

Sure, here's the requested output:

$ ldd /usr/pkg/sbin/zabbix_agentd
/usr/pkg/sbin/zabbix_agentd:
        -lssl.10 => /usr/lib/libssl.so.10
        -lcrypto.8 => /usr/lib/libcrypto.so.8
        -lcrypt.1 => /usr/lib/libcrypt.so.1
        -lgcc_s.1 => /usr/lib/libgcc_s.so.1
        -lc.12 => /usr/lib/libc.so.12
        -lldap.4 => /usr/lib/libldap.so.4
        -llber.3 => /usr/lib/liblber.so.3
        -lcurl.4 => /usr/pkg/lib/libcurl.so.4
        -lidn2.0 => /usr/pkg/lib/libidn2.so.0
        -lintl.1 => /usr/lib/libintl.so.1
        -lunistring.2 => /usr/pkg/lib/libunistring.so.2
        -lgssapi.10 => /usr/lib/libgssapi.so.10
        -lkrb5.26 => /usr/lib/libkrb5.so.26
        -lhx509.5 => /usr/lib/libhx509.so.5
        -lasn1.9 => /usr/lib/libasn1.so.9
        -lcom_err.7 => /usr/lib/libcom_err.so.7
        -lroken.19 => /usr/lib/libroken.so.19
        -lutil.7 => /usr/lib/libutil.so.7
        -lwind.0 => /usr/lib/libwind.so.0
        -lheimbase.1 => /usr/lib/libheimbase.so.1
        -lheimntlm.4 => /usr/lib/libheimntlm.so.4
        -lz.1 => /usr/lib/libz.so.1
        -lpthread.1 => /usr/lib/libpthread.so.1
        -lkvm.6 => /usr/lib/libkvm.so.6
        -lm.0 => /usr/lib/libm.so.0
        -lexecinfo.0 => /usr/lib/libexecinfo.so.0
        -lelf.1 => /usr/lib/libelf.so.1
$ 
$ ldd /usr/pkg/lib/libcurl.so
/usr/pkg/lib/libcurl.so:
        -lidn2.0 => /usr/pkg/lib/libidn2.so.0
        -lintl.1 => /usr/lib/libintl.so.1
        -lgcc_s.1 => /usr/lib/libgcc_s.so.1
        -lc.12 => /usr/lib/libc.so.12
        -lunistring.2 => /usr/pkg/lib/libunistring.so.2
        -lssl.10 => /usr/lib/libssl.so.10
        -lcrypto.8 => /usr/lib/libcrypto.so.8
        -lcrypt.1 => /usr/lib/libcrypt.so.1
        -lgssapi.10 => /usr/lib/libgssapi.so.10
        -lkrb5.26 => /usr/lib/libkrb5.so.26
        -lhx509.5 => /usr/lib/libhx509.so.5
        -lasn1.9 => /usr/lib/libasn1.so.9
        -lcom_err.7 => /usr/lib/libcom_err.so.7
        -lroken.19 => /usr/lib/libroken.so.19
        -lutil.7 => /usr/lib/libutil.so.7
        -lwind.0 => /usr/lib/libwind.so.0
        -lheimbase.1 => /usr/lib/libheimbase.so.1
        -lheimntlm.4 => /usr/lib/libheimntlm.so.4
        -lz.1 => /usr/lib/libz.so.1
        -lpthread.1 => /usr/lib/libpthread.so.1
$ 
$ pkg_info | egrep '^curl'
curl-7.56.1         Client that groks URLs
$ 
$ uname -a
NetBSD xxx.xxx.no 7.1_RC1 NetBSD 7.1_RC1 (GENERIC) #0: Wed Feb 15 21:05:14 CET 2017  [email protected]:/usr/obj/sys/arch/amd64/compile/GENERIC amd64
$

I searched the build log for zabbix, and didn't find an explicit linking of -lpthread,
so the dependency pull-in listed above is probably the result of libcurl.so being
pulled in by ldd.

The curl package corresponds to

ftp://ftp.netbsd.org/pub/pkgsrc/packages/NetBSD/amd64/7.1/All/curl-7.56.1.tgz

As for which queries would be affected, I admit that I didn't chase that
dependency; I simply looked for places in the source which explicitly
manipulated the _res "used-to-be-a-global-variable", and converted
to the more modern and thread-safe API.

Comment by Valdis Kauķis (Inactive) [ 2017 Nov 23 ]

Thanks, Håvard for the quick response! I can reproduce the issue now on NetBSD 7.1 (GENERIC.201703111743Z), although with some hacking because my /usr/pkg/lib/libidn2.so: has undefined reference to `strchrnul'.

Comment by Håvard Eidnes [ 2017 Nov 23 ]

I'm a bit curious where your libidn2 package comes from.
The one from ftp.netbsd.org has

$ nm -op libidn2.so | grep strchrnul
libidn2.so:0000000000004fb0 t strchrnul
$

The t indicates it's a text symbol in the library...

Comment by Vladislavs Sokurenko [ 2017 Nov 23 ]

I can also confirm the issue
steps:

./configure --enable-agent --with-libcurl --prefix=$(pwd)

./sbin/zabbix_agentd -t net.dns
_res is not supported for multi-threaded programs.
net.dns                                      [1]   Abort trap (core dumped) ./sbin/zabbix_agentd -t net.dns


pkg_info curl
Information for curl-7.56.1:

Comment:
Client that groks URLs

Requires:
libidn2>=2.0.0

Description:
Curl is a command line tool for transferring files with URL syntax, supporting
FTP, FTPS, HTTP, HTTPS, GOPHER, TELNET, DICT, FILE and LDAP. Curl supports
HTTPS certificates, HTTP POST, HTTP PUT, FTP uploading, HTTP form based upload,
proxies, cookies, user+password authentication (Basic, Digest, NTLM, Negotiate,
kerberos...), file transfer resume, proxy tunneling and a busload of other
useful tricks.

Homepage:
https://curl.haxx.se/

pkg_info
apr-1.5.2           Apache Portable Runtime
apr-util-1.5.4nb7   Apache Portable Runtime utilities
serf-1.3.9          High-performance asynchronous HTTP client library
subversion-base-1.9.5nb1 Version control system, base programs and libraries
pcre-8.40nb1        Perl Compatible Regular Expressions library
mozilla-rootcerts-1.0.20170121nb5 Root CA certificates from the Mozilla Project
libffi-3.2.1nb2     Foreign function interface
python27-2.7.13nb1  Interpreted, interactive, object-oriented programming language
xmlcatmgr-2.2nb1    XML and SGML catalog manager
libxml2-2.9.4nb3    XML parser library from the GNOME project
nghttp2-1.20.0      Implementation of HTTP/2 in C
readline-7.0        GNU library that can recall and edit previous input
apache-2.4.25nb2    Apache HTTP (Web) server, version 2.4
ap24-subversion-1.9.5nb1 WebDAV server (Apache module) for Subversion
perl-5.24.1nb1      Practical Extraction and Report Language
p5-subversion-1.9.5nb1 Perl bindings for Subversion
py27-subversion-1.9.5nb1 Python bindings and tools for Subversion
libyaml-0.1.7       YAML 1.1 parser and emitter written in C
ruby23-base-2.3.3   Ruby 2.3.3 release minimum base package
ruby23-subversion-1.9.5nb1 Ruby bindings for Subversion
subversion-1.9.5    Version control system, meta-package
m4-1.4.17           GNU version of UNIX m4 macro language processor
autoconf-2.69nb7    Generates automatic source code configuration scripts
automake-1.15nb4    GNU Standards-compliant Makefile generator
pkgconf-1.0.1       API-driven pkg-config replacement
pcre++-0.9.5nb2     Wrapper class around the pcre library
vim-share-8.0.0425  Data files for the vim editor (vi clone)
vim-8.0.0425        Vim editor (vi clone) without GUI
libunistring-0.9.7  Unicode string library
libidn2-2.0.4       Convert internationalized domain names to/from ASCII Encoding
curl-7.56.1         Client that groks URLs

cat /proc/version 
NetBSD version 7.1 (netbsd@localhost) (gcc version 4.8.5) NetBSD 7.1 (GENERIC.201703111743Z)
Comment by Valdis Kauķis (Inactive) [ 2017 Nov 28 ]

A short explanation on why libcurl did not link for me:

  uname -v pkg_info|grep idn nm -op /usr/pkg/lib/libidn2.so.0|grep strchrnul
yours NetBSD 7.1_RC1 (GENERIC) libidn2-2.0.4 /usr/pkg/lib/libidn2.so.0:0000000000004fb0 t strchrnul
mine NetBSD 7.1 (GENERIC.201703111743Z) libidn2-2.0.3 /usr/pkg/lib/libidn2.so.0: U strchrnul

More important – it appears that OpenBSD does not support the thread-safe resolver interface, at least not out-of-the-box (v6.2, 2017oct). We will have to support both interfaces with #ifdef-s.

Comment by Valdis Kauķis (Inactive) [ 2017 Nov 28 ]

Fixed in svn://svn.zabbix.com/branches/dev/ZBX-13055

Comment by Sergejs Paskevics [ 2018 Jan 17 ]

Implemented:

  • pre-3.0.15rc1 in r76940;
  • pre-3.4.7rc1 in r76943;
  • pre-4.0.0alpha3 (trunk) in r76944;
Comment by Vladislavs Sokurenko [ 2018 Feb 06 ]

Fixed in:

  • pre-3.0.15rc1 r77407
  • pre-3.4.7rc2 r77410
  • pre-4.0.0alpha4 (trunk) r77411
Generated at Sat Apr 20 05:28:30 EEST 2024 using Jira 9.12.4#9120004-sha1:625303b708afdb767e17cb2838290c41888e9ff0.