Oracle Ent Linux, 64 bit, 2.6.18-371.4.1.0.1.el5, Red Hat Enterprise Linux Server release 5.10 (Tikanga)

Is http poller a part that send information to web front end?
Does the last lines suggest that this can be a problem with http poller and libc libraries? Would change anything if I have installed different version of this?
Thanks.

Whatever is different in 2.2.3RC1 version from 2.2.2 concerning crash of http poller - it fixed the problem.
That's a hint probably, now I don't know if I'll be able to test fix for 2.2.2 when released.

Jaroslaw, could you please specify what version/revision you upgraded from, how you did it and what was the server configuration when it was launched? Also, what was the state of the database and what configuration options you passed to both configuration processes of the 2.0 and 2.2.2 builds. Also, could you provide a server log with DebugLevel=4 of a reproduced crash? Thank you.

The version I upgraded from was 2.0 (this is as close as I can say). First upgrade didn't go through to the end as I interrupted this (I've thought nothing is going on, so I stopped zabbix server, upgrade didn't finish, I had to restore database and then started again, this time watching log, it went through to the end successfully).
I will try to run 2.2.2 with debug level 4 and will paste the log.
Config is pretty normal, I haven't compiled server, installed from repositories. I have only compiled version 2.2.3RC1 when discovered I can't run 2.2.2
What do you mean "state of the database"? After I have restored database after first unsuccessful upgrade, database was right after normal server shut down.

Was the 2.0 server compiled by you or was it also from a repository package and how exactly did you 'restore' the database?

I have mentioned this in the last comment - I have installed 2.0 from repositories, it wasn't compiled.
Database was restored from dump made after shutting down 2.0 and before running 2.2.2.

Juris - do you require config file as well?

Attached server log of failing 2.2.2 version, build 42525.
Removed sensitive info.

Jaroslaw, could you please execute these SQL queries in your database and report the results to us:

SELECT httpstepid,no,name,url,timeout,posts,required,status_codes,variables from httpstep;
SELECT httptestid,name,delay,status,authentication,hostid,retries from httptest;

Remove the quotation marks and simply paste it. If it contains confidentail information, mail the results to support[at]zabbix.com

Query result sent to email as requested.

Jaroslaw, I have attached a patchfile patch.p0. Please use that to patch the broken and working versions of the server and repeat the process that leads to the crash with both of them. Also, could you please notify us of the cURL library version(s) that is/are being used.

Thank you.

How do you want me to use this patch? What should I do with this? This is replacing something in source files? If so - I would have to compile 2.2.2 version, which I didn't - it was installed from downloaded binaries, from repository.

Libcurl.so library on my system is libcurl.so.3.0.0, package reported in yum is curl-7.15.5-17.el5_9.i386 and x86_64 (both 32 and 64 bit versions installed).
curl-devel is the same.

Jaroslaw, please execute this command:

strace -o <logfilename> -ffitttxTv -s 1024 </path/to/zabbix_server/binary>

and replace the necessary parts with a valid filename and the path to your server executable respectively. When the strace process finishes, find all the output files generated and archive them and send them to support[at]zabbix.com.

Thank you.

One more question before I run this - should the server be shut down? Like - service zabbix_server stop?
Then run the command above on both binaries separately?

Yes, the server should not be running and for both binaries, the working one and the one that crashes.

How long should this thing run on healthy 2.2.3rc1 version?
For now it produced around 2000 files and still running.
I haven't run it for 2.2.2 yet.

I'm sorry, Jaroslaw. You had to run it for 2.2.2 only, not the healthy binary. I appologize. My mistake. Just gather the 2.2.2 output and send that. Thanks.

Jaroslaw, could you please provide the output of the following commands:

curl -V
yum info curl

I believe I already gave this information but here is again:

[/]# curl -V
curl 7.15.5 (x86_64-redhat-linux-gnu) libcurl/7.15.5 OpenSSL/0.9.8b zlib/1.2.3 libidn/0.6.5
Protocols: tftp ftp telnet dict ldap http file https ftps
Features: GSS-Negotiate IDN IPv6 Largefile NTLM SSL libz

[/]# yum info curl
Loaded plugins: security
Installed Packages
Name : curl
Arch : i386
Version : 7.15.5
Release : 17.el5_9
Size : 464 k
Repo : installed
Summary : A utility for getting files from remote servers (FTP, HTTP, and others).
URL : http://curl.haxx.se/
License : MIT
Description: cURL is a tool for getting files from FTP, HTTP, Gopher, Telnet, and
: Dict servers, using any of the supported protocols. cURL is designed
: to work without user interaction or any kind of interactivity. cURL
: offers many useful capabilities, like proxy support, user
: authentication, FTP upload, HTTP post, and file transfer resume.

Name : curl
Arch : x86_64
Version : 7.15.5
Release : 17.el5_9
Size : 473 k
Repo : installed
Summary : A utility for getting files from remote servers (FTP, HTTP, and others).
URL : http://curl.haxx.se/
License : MIT
Description: cURL is a tool for getting files from FTP, HTTP, Gopher, Telnet, and
: Dict servers, using any of the supported protocols. cURL is designed
: to work without user interaction or any kind of interactivity. cURL
: offers many useful capabilities, like proxy support, user
: authentication, FTP upload, HTTP post, and file transfer resume.

This is likely a related issue to ZBX-7986. Look in the comments section for a possible solution. This is very likely a bug in libcurl.

We were originally incapable of reproducing this issue, but an upgrade of libcurl has halted, or at least mitigated, this bug from occuring. This issue may be reopened if evidence of incorrect code is found in our code, not libcurl, which will then be fixed.

For now, this issue will be closed and marked "Won't fix".