Ok, the story started from this:
there in a web application which did not like the fact that by default zabbix sets CURLOPT_ACCEPT_ENCODING option
which resulted this header automatically included to each HTTP request:
and the web application responded with this:
It was not easy to understand where is the issue because "curl" tool in shell does not set the header.
Everyone is using curl for testing, right? So, small question here is - why do we set the header in zabbix? ... well ...
By manual tests using curl in shell I figured out that the app does not like "br" entry in the list.
In the same time the app did not complain on values like "foo" or "bar" or "foo, bar" - which is funny.
I don't know what is libcurl version installed on the system, but on my recent Ubuntu with libcurl 7.81.0 it sets the header with a bit longer list:
Not sure how that is precise, but according to information I have the web app is working on Apache/2.4.6.
To workaround the issue, on the scenario level I've added a custom header "Accept-Encoding" with value "*", which was tested and supposed to work. But .... it did not work in zabbix. The web app complained with the same error.
And reason for that was a missing space character after colon, generated header was:
Luckily it was possible add a trailing space in front of "*" and save that in frontend.
Generated header became this:
and monitoring started to work.
I've tried to reproduce that issue with missing space in custom header on my Apache/2.4.52 (as monitored web app) and I could not - it did not complain.
I've tried to find some standard about HTTP proto, and looks like this is most relevant link:
but it does not mention the space for formatting.
But having this real case and that fact that all other headers are using the space character, I'm sure we also have to add a space in our custom headers.