FreeBSD 9.2-RELEASE-p10

According to the following lines in the log, the process that crashed was HTTP poller:

 ...
 92537:20140910:082322.699 server #25 started [http poller #1]
 ...
 92537:20140910:082339.791 Got signal [signal:11(SIGSEGV),reason:2,refaddr:0x8020f0120]. Crashing ...
 ...

Would it be possible for you to run Zabbix server with DebugLevel=4, so that we see what exactly HTTP poller was doing?

Also, could you please check whether this issue is the same as ZBX-8005, which is a problem with GnuTLS flavor of libcurl?

Sure can. The issue ZBX-8005 is related to a GnuTLS libcurl and we're not using that:

[root@cosmo ~]# ldd /usr/local/sbin/zabbix_server | grep curl
libcurl.so.7 => /usr/local/lib/libcurl.so.7 (0x801a8d000)

I'll add a log using DebugLevel 4. Strange part is that http things seem to work properly though

Thanks for the quick response! The HTTP poller crashes when accessing an HTTPS website. Which version of libcurl do you have?

No problem. Thanks for the help in trying to find out what's wrong.

[root@cosmo ~]# pkg info | grep curl
curl-7.37.1_2 Non-interactive tool to get files from FTP, GOPHER, HTTP(S) servers

Does it even matter that it actually runs without problems when --with-snmp is changed to --without-snmp? It won't do snmp-things, but that's expected then.

There are two ways we can try moving forward with this issue:

(1) If the crash in HTTP poller depends on SNMP, the SNMP library might be exporting symbols that are used by cURL library or the libraries it depends on. In that way, this issue might be similar to ZBX-8629.

Could you please post the output of "readelf -Ws ..." on the SNMP library and the cURL library? Although that may not give us the full picture, because conflicting symbols might be in different libraries.

(2) Could you please try "curl https://insight2.lico.nl/" from the command line? Is it successful?

Attached output of readelf -Ws for the libraries. A curl for that url runs without any problems and returns the proper HTML

Regarding (2), what if you do "LD_PRELOAD=<snmp-library> curl https://insight2.lico.nl/"?

a LD_PRELOAD=/usr/local/lib/libnetsnmp.so.30 curl https://insight2.lico.nl/ gives me the same (correct) result

As far as I remember, Zabbix printed backtrace on FreeBSD, but you needed to install libexecinfo. Could you please install libexecinfo, reconfigure Zabbix, recompile, make it crash and send the new DebugLevel=4 log?

I've already had libexecinfo installed. The port zabbix22-server depends on it. Reconfigured, compiled and attached log. It doesn't seem to actually use the libexecinfo.

Issue ZBX-2946 was where we dealt with libexecinfo on FreeBSD.

If you are building from sources, could you please try specifying -I and -L directories to configure so that it knows where to look for libexecinfo header and library, like so:

$ ./configure CPPFLAGS='-I/usr/local/include' LDFLAGS='-L/usr/local/lib' --enable-server --with-mysql --with-libcurl --with-net-snmp ...

If you do it like that, does it use libexecinfo?

Added:

CONFIGURE_ARGS+= CPPFLAGS='-I/usr/local/include' LDFLAGS='-L/usr/local/lib'

to Makefile. It does not seem to use the libexecinfo-thingy. Configure output and new server_log attached.

Actually, it seems to use libexecinfo, but the following is the most it can get for some reason:

 65015:20140911:145201.412 === Backtrace: ===
 65015:20140911:145201.412 0: 0x7fffffff9d10

As the last try, could you please add "-g" for debug information to CPPFLAGS?

If that does not work, I shall install FreeBSD 9.2 next week (after the conference) and try to reproduce the issue.

I don't mind trying. New log is attached, but does not seem te produce extra backtrace.

Rene, I have installed FreeBSD 9.2, but its ports collection only has Zabbix 2.0.6 (see http://ftp.freebsd.org/pub/FreeBSD/ports/amd64/packages-9.2-release/net-mgmt/) and cURL 7.31.0 (see http://ftp.freebsd.org/pub/FreeBSD/ports/amd64/packages-9.2-release/www/). Could you please guide me through the necessary steps to obtain an environment as close as possible to yours?

You don't have a net-mgmt/zabbix22-server port? I run portupgrade weekly, so I assume it was updated there.

No, I don't. Here are the ports that I have:

/usr/ports/net-mgmt/zabbix2-agent
/usr/ports/net-mgmt/zabbix2-frontend
/usr/ports/net-mgmt/zabbix2-proxy
/usr/ports/net-mgmt/zabbix2-server

Perhaps, I should specify a different port source?

I didn't specify a different port source. I run 'portsnap fetch && portsnap update' weekly before installing updates'. Maybe that helps

Thank you! Running "portsnap" as you suggested made a difference.

Currently, HTTP pollers on the server seem to work well for your URL on my machine.

Could you please go to /usr/ports/ftp/curl and /usr/ports/net-mgmt/net-snmp, and post screenshots of "make config"?

Screenshots of both. Note that zabbix22_server is the only thing we've installed using ports. Everything else is installed using packages. The reason for installing zabbix22_server using ports is that the pkg version uses MySQL and we use Postgres.

The strange part is that the http poller works correctly when compiled --without-snmp.

So the attached screenshots are not relevant then.

I can make it crash as in ZBX-8770 if I install cURL from ports with GnuTLS, but that is a different case.

Would be possible for you to somehow find out which configuration the installed cURL and Net-SNMP packages were built with?

Would it help if we install both of these using ports? If so, what configuration would it want?

If I have an OpenSSL version of cURL with cookie support from ports, then it works well. If I have a GnuTLS version, it crashes in curl_easy_init(), as mentioned above.

However, what I need is to reproduce your case, so I should create an environment that is as close as possible to yours. Thus, it would probably not help if you would reinstall them from ports, but you can try. I shall now try installing both from packages and retry.

With cURL and Net-SNMP installed from packages, it works well. However, I then have versions of these libraries that are different from yours:

$ curl-config --version
libcurl 7.31.0
$ net-snmp-config --version
5.7.2

What should I do to install version curl-7.37.1_2, as you have done?

What if you run 'pkg install curl'? That gives me:

  Updating FreeBSD repository catalogue...
  FreeBSD repository is up-to-date.
  All repositories are up-to-date.
  Checking integrity... done (0 conflicting)
  The following 1 packages will be affected (of 0 checked):

  Installed packages to be UPGRADED:
    curl: 7.37.1_2 -> 7.37.1_4

  Proceed with this action? [y/N]:

We run 'pkg upgrade' weekly to update them all.

[root@cosmo ~]# cat /etc/pkg/FreeBSD.conf
# $FreeBSD$
#
# To disable this repository, instead of modifying or removing this file,
# create a /usr/local/etc/pkg/repos/FreeBSD.conf file:
#
#   mkdir -p /usr/local/etc/pkg/repos
#   echo "FreeBSD: { enabled: no }" > /usr/local/etc/pkg/repos/FreeBSD.conf
#

FreeBSD: {
  url: "pkg+http://pkg.FreeBSD.org/${ABI}/latest",
  mirror_type: "srv",
  signature_type: "fingerprints",
  fingerprints: "/usr/share/keys/pkg",
  enabled: yes
}

Maybe that helps? It has been a while since we've switched to packages in favor of ports where possible. Sorry for making this something hard to solve. I hope we can create a similar environment for you there.

Thanks to your hint and the help of Google, I have converted the packaging system to pkgng and now I have the exact same version of libcurl and Net-SNMP as you do.

However, my server still does not crash if I access https://insight2.lico.nl/ with Web monitoring. Could you please give the exact ./configure command line that you are using to configure Zabbix server? Alternatively, if "make config" screenshot is more relevant, please post that instead.

Hmm. i see curl isn't even activated. I've added a second screenshot where it shows the current web scenario where the server is compiled --without-snmp. The host where this web action is set isn't monitored by a proxy.

Couldn't it be something else that is failing here?

The fact that cURL is not activated in "make config" is suspicious. If I do the same, then Web monitoring is not performed: server says that it was compiled without cURL support.

Anyway, maybe it does not crash in libcurl after all.

Could you please execute the following queries? These should give information about the configuration of your Web scenario and I shall set it to be exactly the same locally:

\x on
select h.hostid,h.host,h.name,t.httptestid,t.name,t.variables,t.agent,t.authentication,t.http_user,t.http_password,t.http_proxy,t.retries from httptest t,hosts h where t.hostid=h.hostid and t.nextcheck<=1420441712 and mod(t.httptestid,2)=0 and t.status=0 and h.proxy_hostid is null and h.status=0 and (h.maintenance_status=0 or h.maintenance_type=0);
select httpstepid,no,name,url,timeout,posts,required,status_codes,variables from httpstep where httptestid=2 order by no;

Sure can. It's only one step

zabbix=# select h.hostid,h.host,h.name,t.httptestid,t.name,t.variables,t.agent,t.authentication,t.http_user,t.http_password,t.http_proxy,t.retries from httptest t,hosts h where t.hostid=h.hostid and t.nextcheck<=1420441712 and mod(t.httptestid,2)=0 and t.status=0 and h.proxy_hostid is null and h.status=0 and (h.maintenance_status=0 or h.maintenance_type=0);
-[ RECORD 1 ]--+-----------------------------------------------------------------
hostid         | 10087
host           | private.lico.nl
name           | private.lico.nl
httptestid     | 2
name           | insight2.lico.nl
variables      |
agent          | Mozilla/5.0 (X11; Linux i686; rv:8.0) Gecko/20100101 Firefox/8.0
authentication | 0
http_user      |
http_password  |
http_proxy     |
retries        | 1
-[ RECORD 2 ]--+-----------------------------------------------------------------
hostid         | 10154
host           | schoudercom.bluerail.nl
name           | schoudercom.bluerail.nl
httptestid     | 4
name           | Accounttypes
variables      |
agent          | Mozilla/5.0 (compatible; MSIE 10.0; Windows NT 6.1; Trident/6.0)
authentication | 0
http_user      |
http_password  |
http_proxy     |
retries        | 1

zabbix=# select httpstepid,no,name,url,timeout,posts,required,status_codes,variables from httpstep where httptestid=2 order by no;
-[ RECORD 1 ]+--------------------------
httpstepid   | 2
no           | 1
name         | Voorpagina
url          | https://insight2.lico.nl/
timeout      | 30
posts        |
required     | Welkom bij Buzz
status_codes | 200
variables    |

Still works perfectly for me. I have attached a patch, "debug-info.patch", that adds a bit more debugging information to Web monitoring. This way we will find out whether it crashes in libcurl or somewhere else. Would it be possible for you to run server with this patch?

Also, could you please do "ldd /usr/local/lib/libcurl.so" to make sure once again that we are not using a GnuTLS version of libcurl.so?

Sure can. I would have to dig into how one applies a patch to a port. I'm afraid it'll be friday during 'regular' business hours (CEST) before I can do that though. I'll keep you posted.

Not sure whether that is the best way, but I have followed a suggestion on https://www.freebsd.org/doc/en_US.ISO8859-1/books/porters-handbook/slow-patch.html : put the attached patch into files/ directory and name it beginning with "patch-", say, "patch-debug-info.c". Running "make clean" and "make install" yields a server with the patch applied.

And we're back.. attached log with patch applied

According to the log, it indeed crashes in curl_easy_perform().

Could you please post the output of "pkg info curl" and "ldd /usr/local/lib/libcurl.so"?

Attached

in ZBX-8770 upgrade of libcurl helped to avoid some crash.
As I recall in freebsd it's easy to manually edit corresponding file (bump version number from 7.37 to 7.38) and try to reinstall the package.
Maybe would worth to try the same, just in case ?

Upgraded curl to 7.38, 'nothing' changed. Attached new pkg info, ldd and server log files

I seem to have the same environment (the same cURL, at least), but it still does not crash.

Could it be that https://insight2.lico.nl/ yields a different response when accessed from your location than from Zabbix office?

For me, it first gives the following:

$ curl -D - https://insight2.lico.nl/
HTTP/1.1 301 Moved Permanently
Date: Tue, 23 Sep 2014 08:22:12 GMT
Server: Apache
Location: https://buzz.lico.nl/
Vary: Accept-Encoding
Content-Length: 296
Connection: close
Content-Type: text/html; charset=iso-8859-1

<!DOCTYPE HTML PUBLIC "-//IETF//DTD HTML 2.0//EN">
<html><head>
<title>301 Moved Permanently</title>
</head><body>
<h1>Moved Permanently</h1>
<p>The document has moved <a href="https://buzz.lico.nl/">here</a>.</p>
<hr>
<address>Apache Server at insight2.lico.nl Port 443</address>
</body></html>

After that it follows the redirect to https://buzz.lico.nl/ , where there is a proper Web page.

That output is similar to mine. Strange... I don't know what we should do to get your environment similar to mine. Could I get your SSH public key? I can create a user on this system so you can check certain things yourself.

Please send me an email at my first name, dot, my last name, @zabbix.com, and we will arrange something.

Investigation on Rene's server showed that when Zabbix server is compiled with Net-SNMP, the binary is linked to two versions of libcrypto library: libcrypto.so.6 and libcrypto.so.8. When compiled without Net-SNMP, it is linked to only one: libcrypto.so.6. We shall see whether resolving that fixes the crash.

Waiting for https://github.com/freebsd/pkg/issues/936 in attempt to get everything clean

As described above, the problem does not seem to be on Zabbix side. Closing with "Won't fix" for now, please reopen if it turns out otherwise.

I've just created ZBX-8849.
My case I believe is the same as current one.

I'm reopening and closing current case as duplicate.