[ZBXNEXT-1816] Use Curl for the agent's web.page items Created: 2013 Jul 09  Updated: 2023 Mar 23  Resolved: 2019 Aug 22

Status: Closed
Project: ZABBIX FEATURE REQUESTS
Component/s: Agent (G), Documentation (D)
Affects Version/s: 2.2.0
Fix Version/s: None

Type: Change Request Priority: Major
Reporter: Volker Fröhlich Assignee: Unassigned
Resolution: Duplicate Votes: 45
Labels: curl, http, httpcheck, webcheck
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Issue Links:
Duplicate
duplicates ZBXNEXT-287 Fix web.page.regexp to be SSL capable... Closed
is duplicated by ZBXNEXT-2520 No way to monitor vhosts using web.pa... Closed
is duplicated by ZBXNEXT-2228 web.page.get[] should support more HT... Closed
is duplicated by ZBXNEXT-2977 allow setting user agent for web.page... Closed
is duplicated by ZBX-9892 web.page.get inserts incorrect data i... Reopened
is duplicated by ZBX-4079 simple check for http dont check vali... Closed
Sub-task
depends on ZBXNEXT-4358 Metrics for easy RestAPI data retriev... Closed
part of ZBXNEXT-3190 Add new REST API simple check with JS... Closed

 Description   

web.page items don't use Curl to do their job. The current implementation in libs/zbxsysinfo/common/http.c does not allow for redirects, SSL negotiation or everything else that is more than trivial. Furthermore, this is not obvious from the documentation, plus the fact that the agent is linked to libcurl.



 Comments   
Comment by Oleksii Zagorskyi [ 2013 Jul 11 ]

If we would use Curl for those agent items ... then logically we would need to provide ability to specify there: http proxy, auth, browser etc things which looks too much.
I'm not sure it would be good.

Also it's another dependency for agent, what would be better to avoid, IMO.
Remember about copying agent binaries (even very outdated) between hosts on platforms like AIX, HP-UX, Solaris.

We once had one problem for FreeBSD because of the package maintainer's mistake - ZBX-2946

But it indeed could be additionally documented, because I recall such a technical detail was a news even for dotneft

Comment by Volker Fröhlich [ 2013 Jul 11 ]

I guess it's arguable whether this item serves a real purpose. You could very easily create a user parameter with curl or wget.

It's no additional dependency though, the agent binary is linked to libcurl already.

I'd be happy if it was just documented.

Comment by richlv [ 2013 Jul 11 ]

i believe agent uses curl for net.tcp.service[https] already, but i can't find that properly documented yet

Comment by Oleksii Zagorskyi [ 2013 Jul 11 ]

hmm, indeed already linked:

# uname -a
FreeBSD mon 8.1-RELEASE FreeBSD 8.1-RELEASE #0: Mon Jul 19 02:55:53 UTC 2010   [email protected]:/usr/obj/usr/src/sys/GENERIC  i386
# ldd /usr/local/sbin/zabbix_agentd | grep curl
	libcurl.so.6 => /usr/local/lib/libcurl.so.6 (0x280b5000)

# uname -a
Linux it0 3.2.0-4-amd64 #1 SMP Debian 3.2.41-2 x86_64 GNU/Linux
# ldd /usr/sbin/zabbix_agentd | grep curl
	libcurl-nss.so.4 => /usr/lib/x86_64-linux-gnu/libcurl-nss.so.4 (0x00007f212aae3000)
Comment by Volker Fröhlich [ 2013 Jul 11 ]

It's also linked to ldap, ber and ssh for the same reasons or probably false reasons.

Comment by Volker Fröhlich [ 2013 Oct 24 ]

I think that you should either purge out those not so important functions that cause the linking or consider utilizing curl.

Comment by dimir [ 2014 Apr 10 ]

As rich stated currently in Zabbix agent there is only one place where we use libcurl (if --with-libcurl was passed to configure script) which is net.tcp.service[https] and we should indeed document that. What if we do the same check for libcurl when processing web.page.get and if we have it, use that otherwise use existing code?

asaveljevs As far as I know, everywhere we now depend on a library, if a library is linked, we work. If a library is not linked, we do not work at all (and make the item "not supported"). With your proposal, this would make "web.page.get[]" the only place where the behavior changes depending on the linked library. That might not be as apparent and clear to the user.

dimir Good point, I agree. Let's keep it simple and clear to the user.

Comment by Aleksandrs Saveljevs [ 2014 Apr 10 ]

Volker, regarding linking:

  • ldap and ber are used for net.tcp.service[ldap];
  • curl is used for net.tcp.service[https], as Rich mentioned;
  • ssh - we do not link it ourselves, it gets linked automatically if curl is used.
Comment by Volker Fröhlich [ 2014 Apr 10 ]

Doesn't linking to ldap and curl sound like a waste in the face of how little code is used?

Comment by dimir [ 2014 Apr 10 ]

We could basically rewrite net.tcp.service to initate SSL connection using OpenSSL and fetch the data when checking "https" service. I have recently written that code, that should be pretty easy. This would aslo allow "https://" in web.page.get . And in that case we would drop libcurl depencency in agent.

Comment by dimir [ 2014 Apr 10 ]

I guess the easier and cleaner solution would be to change web.page.get to use libcurl and as asaveljevs noted, just return NOTSUPPORTED for web.page.get when compiled without curl support.

As for ldap, I don't see any problem. "If ldap service check is needed, recompile with --with-ldap, otherwise get NOTSUPPORTED" approach sounds totally acceptable to me.

Regarding what zalex_ua mentioned about supporting other CURL parameters. Currently web.page.get syntax is:

web.page.get[host,<path>,<port>]

At this point we could leave it as it is and in the future if there will be a need we could add more options, like http proxy etc. This will bring no regression with one exception that agents compiled without curl support will start to fail.

Comment by Aleksandrs Saveljevs [ 2014 Oct 21 ]

One idea to add headers and cURL options would be to extend web.page.get[] syntax with array parameters, like so:

web.page.get[host,<path>,<port>,<headers>,<options>]

This can then be used like so:

web.page.get[127.0.0.1,index.html,80,[Host: www.example.com,User-Agent: Zabbix 2.4.1],[PROXY=127.0.0.2,FOLLOWLOCATION=1]]

Another unrelated idea would be to merge "host", "path" and "port" into "url". If the reason they are currently separated is the fact that we do not use cURL and such separation makes it easy for us to construct the request, we can safely merge them when cURL support is implemented.

Comment by richlv [ 2015 Sep 20 ]

specifying user agent specifically requested in ZBXNEXT-2977

Comment by Volker Fröhlich [ 2015 Sep 28 ]

When you are implementing this, please consider the following:

Resolving names in connection with libcurl follow a different method. This surfaces when you change resolv.conf, which is read on daemon startup. Thus, changes to the file are not reflected in operations – except for web checks. These apparently use libcurl's backend for name resolution.

If your primary nameserver on the Zabbix server becomes unavailable, everything in Zabbix that uses names fails, except for the web checks, if you updated resolv.conf. This behaviour is confusing.

Comment by Marc [ 2015 Oct 30 ]

Could underlying issue of ZBX-9879 be considered here as well (as additional curl option)?

Comment by Viktors Tjarve [ 2015 Nov 11 ]

Specification draft: http://zabbix.org/wiki/Docs/specs/ZBXNEXT-1816

Comment by dimir [ 2016 Jan 20 ]

I definitely vote for idea of asaveljevs to merge 3 first parameters into url. In this case the spec could mention that we need to take care of that in upgrade patch.

Comment by George Machitidze [ 2016 Feb 04 ]

+1 for merging
for compatibility's sake, it's also possible to create alternative to web.page, something like:

web.uri

or

web.urn

URI or URN is for future use - who knows, maybe we want to support not only http and https for access methods, shall we change everything for every new address and access method? RFC 3986 and RFC2141 describes those two.

Comment by MightyDok [ 2016 Feb 29 ]

+1 for merging, need curl support for web page checks

Comment by richlv [ 2016 May 13 ]

note that with chunked transfer encoding, web.page.get actually returns bad data as per ZBX-9892
thus this is not just a feature request to improve functionality, it is also supposed to fix a bug in web.page.get.

Comment by Mickael Martin (Cyres) [ 2017 Aug 04 ]

Nothing since one year (open since 4 years..) ? I need to check a remote https URL from an agent and web.page.get not support https protocol.
The draft (http://zabbix.org/wiki/Docs/specs/ZBXNEXT-1816) is really great !

Comment by dimir [ 2017 Aug 04 ]

We'll try to push this one further, let's see how it goes.

Comment by Glebs Ivanovskis (Inactive) [ 2017 Nov 09 ]

ZBXNEXT-3190 asks for simple check with similar capabilities.

Comment by Mickael Martin (Cyres) [ 2018 Mar 30 ]

Always nothing for #Zbx4...

Comment by dimir [ 2018 Apr 02 ]

Looks like we will have the needed functionality implemented in ZBXNEXT-4359 . When it will be released implementing this task will be really simple. So, hopefully we're getting there!

Comment by richlv [ 2018 Apr 02 ]

that issue is super-vague, though - it doesn't even say whether the new item will work on server/proxy, agent or all of them.

Comment by Mathieu CARBONNEAUX [ 2018 Apr 25 ]

many devices metric now can be pooled with http... now quasi all devices use https...

it's required to have complete http item pool solution.

 

you cant developing the actual solution with adding https support, header support, method, etc...

or using curl and adding argument option...

 

for compatibility you can add new item like :

  web.curl[host,<path>,<port>,<headers>,<options>]

in that way you are explicit that you use curl... and all there functionality...

 

Comment by Oleksii Zagorskyi [ 2019 Aug 22 ]

As I understand, it has been implemented in ZBXNEXT-287, so closing this one as duplicate.

Comment by Robert Szulist [ 2023 Mar 23 ]

How is this implemented in ZBXNEXT-287? It solves the SSL part of this issue.

What about custom headers or the ability to follow redirects?

Comment by Glebs Ivanovskis [ 2023 Mar 23 ]

What about custom headers or the ability to follow redirects?

Checking the latest source code I see that agent sets the CURLOPT_FOLLOWLOCATION option, so redirects are followed.
But there seems no way to send custom headers, indeed. ZBXNEXT-6680 asks for it (currently closed as duplicate of ZBXNEXT-5533).

Comment by Robert Szulist [ 2023 Mar 23 ]

The docs state that `0L` means disabled.

So this code explicitly turns off HTTP redirections.

Comment by Glebs Ivanovskis [ 2023 Mar 23 ]

Oops, my bad. You are right.

Generated at Fri Mar 29 10:32:40 EET 2024 using Jira 9.12.4#9120004-sha1:625303b708afdb767e17cb2838290c41888e9ff0.