Debian 12, Zabbix 7.0.3, Mosquitto 2.011, Zabbix Agent 2 7.0.3

Hi,

Since I was not able to reproduce it, could you provide more precise details regarding your config (hosts, items).

Also couple of questions to clarify:

• Do the Zabbix server's logs say anything else that can be helpful to debug the issue?
• Have you tried playing with PluginTimeout as suggested in forum thread?
• When exactly the error occurs? To how many topics are you subscribing? Is that the issue occurring if only more than 2 hosts are involved?
• Not sure if related, but what is the example input sent to the topic? Is the preprocessing step extracting json value correct? Seems like
  $.["InterfaceOrder.Interface.WMS.IndexEnd.PLCSetpoint"] does not follow the syntax from here: https://www.zabbix.com/documentation/current/en/manual/config/items/preprocessing/jsonpath_functionality

• Do the Zabbix server's logs say anything else that can be helpful to debug the issue?

-> the only messages I found was something like this, but I can issue another test if you want

• Have you tried playing with PluginTimeout as suggested in forum thread?

-> Yes, I tried this, but it seems, that it has no effect or the timeout which is max. 30 seconds is not high enough.

• When exactly the error occurs? To how many topics are you subscribing? Is that the issue occurring if only more than 2 hosts are involved?

-> It happens right after the start of the agent.

-> From 1 to some 100

-> it seems, that if you have more than 5 or in lower versions 7 hosts, the number of topics is no longer relevant

• Not sure if related, but what is the example input sent to the topic? Is the preprocessing step extracting json value correct? Seems like
  $.["InterfaceOrder.Interface.WMS.IndexEnd.PLCSetpoint"] does not follow the syntax from here: https://www.zabbix.com/documentation/current/en/manual/config/items/preprocessing/jsonpath_functionality

-> I tried to extract the data like $.["InterfaceOrder.Interface.WMS.IndexEnd.PLCSetpoint"].['value'] but this doesn't work
I get the following error message "cannot extract value from json by path "$.["InterfaceOrder.Interface.WMS.IndexEnd.PLCSetpoint"]['value']": no data matches the specified path". So I decided to add an additional preprocessing step "$.value"

Could you try to capture more detailed logs with setting the DebugLevel to 5. You can try checking that both on agent and server side.

Thanks

Hello,

I found this in the agent2.log:

2024/09/24 07:53:47.528091 [MQTT] cannot subscribe topic 'VSW02/#' to [tcp://10.83.170.106:1883]: timed out while subscribing

and this in server.log

 1033:20240924:075348.312 zbx_substitute_functions_results() expression[0]:'{33577}=1' => 'ERROR(Cannot evaluate function last(/VSW02/InterfaceStatus.Interface.EmergencyStop,#1): item is not supported.)=1'

I attached the files as zip.

Hi,

could you please provide the output of 

nc -zv 10.83.170.106 1883

from the agent where you are getting the error logs.

Hi,

The output of  "nc -zv 10.83.170.106 1883" is

pa@ZabbixTest:/var/log/zabbix$ nc -zv 10.83.170.106 1883
10.83.170.106: inverse host lookup failed: Unknown host
(UNKNOWN) [10.83.170.106] 1883 open

We have tried to reproduce the issue, but with no succes, could you please provide more details on how to reproduce it?

Would it help you if I send you the configuration files from the server, the agent and an export of the hosts of my system?

That might be helpful

I uploaded the files.

Just for information: The issue mostly appears when you restart the agent2 or reboot the system where the agent2 runs on.

What I observed is, if you add one host after another into Zabbix the issue does not appear. But as soon as you do what I mentioned above it will show up.

Hi,

Still unable to reproduce, could you provide the steps to reproduce on clean Zabbix instance with minimum hosts/templates involved (since you mention that the broker can be on the same system, I would like to see the error in this scenario)?

Some additional questions:

• Is your MQTT broker similarly on Debian 12?
• Why are you using mqtt settings such as Plugins.MQTT.Default.Password etc. in zabbix agent config? They are meant to be used in mqtt.conf file but are not necesarry since you are already setting the credentials/topic at the host config site.
  See: https://www.zabbix.com/documentation/6.0/en/manual/appendix/config/zabbix_agent2_plugins/mqtt_plugin

Hi,

I indeed had the issue also on a fresh installed system. Nevertheless I will prepare a separate clean installed test system for this.

I will try to install Zabbix, NodeRed and Mosquitto on the same VM. I'll keep you informed when I'm ready.

• yes the MQTT broker is also installed on Debian 12
• this is, because the Zabbix on the test system was formerly at version 6 (here the configuration wasn't separated) 
  • Question from my side: is this configuration really needed, since you can pass most of these parameters with host macros?

Thanks, I will wait for your follow-up message.

Regarding your question, no it should not be needed so you can remove them form agent config.

Hello, 

here are my test results:

Test1 - New Testsystem (clean install)
-----------------------------
Zabbix Server 7.0.4
Zabbix Frontend 7.0.4
Zabbix Agent2 7.0.4

10 Hosts with 2 items
All on one VM (NodeRed, Mosquitto, Zabbix)

Result=OK
----------------------------

Test2 - Old Testsystem (Updated to 7.0.4)
-----------------------------
Zabbix Server 7.0.4
Zabbix Frontend 7.0.4
Zabbix Agent2 7.0.4

Same hosts as before

Result=NOK
----------------------------

Test3 - New Testsystem
-----------------------------
Zabbix Server 7.0.4
Zabbix Frontend 7.0.4
Zabbix Agent2 7.0.4

10 Hosts with 2 items
Added 7 hosts where the mqtt broker is installed on another VM.

Result=NOK
Remark also hosts where the mqtt broker is installed locally now have errors.

---------

It seems that if the MQTT-Plugin needs to work outside its host (over network) it has problems with the communication.

I changed the Mosquitto config so that there is no difference. But this didn't help.

I followed the above steps but still could not reproduce it. Could you capture the traffic from/to mqtt broker:
tcpdump -npi any -s 0 -w /file/path.pcap host x.x.x.x

You might want to also execute the below to check if packets are dropped, delayed at the point of timeouts:

LC_ALL=C ping -s 1400 -c 100 -i 0.01 <broker_ip>

Sorry for the late answer. I was out of office last week.

I've put the files you requested.

During the recording of the pcap-file I restarted the agent.

Today I tested the 7.0.5 version and I had the same issue.

Zabbix Server 7.0.5
Zabbix Frontend 7.0.5
Zabbix Agent2 7.0.5

Hello,

Do you have any news? May I do something so that you can

better reproduce the problem?

Thanks

Hello,

from the Zabbix-Forum I received that hint.

https://www.zabbix.com/forum/zabbix-help/489802-agent2-mqtt-plugin-issue-time-out-while-subscribing#post493901

https://www.zabbix.com/forum/zabbix-help/456267-mqtt-plugin-in-zabbix-agent-2-subscribe-was-broken-by-timeout/page2

It seems to be a known issue. Please consider to fix it.

Hello,

is there any progress on this one? We started using mqtt.get  This is a perfect way for us. But we have exactly the same porblems (timed out while subscribing/inalid update interval)

Our Setup:

• Zabbix Server 7.06
• Zabbix Agent 2 7.0.6 (Plugin communication protocol version is 6.4.0.
• Zabbix Frontend 7.0.6

Thanks in advance!

Christian

Fixed in:

• 6.0.39rc1 - edf67ccc54e
• 7.0.10rc1 - dbc09657d35
• 7.2.4rc1 - bf9c39dbea9
• 7.4.0alpha1 (master) - cc89dcafbc8