Client operating system: Windows 10 Pro 21H1 Build 19043.1348
Client hardware: Lenovo X1 Carbon and Lenovo ThinkBook 13S G2 ITL

Server: Zabbix Server 5.0.16 on Ubuntu 18.04 LTS

Same problem here with a setup Server-Proxy-Agent

Autoregistration works but then I get the same SSL error.

Hello,

Could you please provide network dump in pcap format for the same time from server and agent?

Do you have any load balancer before Zabbix Server?

Regards,
Edgar

I created and uploaded a network dump in pcap format.

We don't have a load balancer, proxy or anything else in front of the Zabbix server.

Could you please replace Agent (C version) binary and increase logging to debug (5) level, wait for an error and provide it to us?
zabbix_agentd.exe.zip

This binary contains additional debug log and has no signature, so Windows can show errors about publisher of the binary, this is ok. Binary is build on ltsc2022.

Regards,
Edgar

Thank you, Edgar.

I uploaded the log as "zabbix_agentd-2.log".

Thank you, will check.

Let me ask you to gather one more time, added more logging: zabbix_agentd_debug2.zip.

We are very happy if we can help solve the problem.

I uploaded the log as "zabbix_agentd-3.log".

This is probably incorrect:

 11112:20211122:194919.851 Starting Zabbix Agent [TEST]. Zabbix 5.0.17 (revision 0a4ac3d).
...
 11112:20211122:194920.241 OpenSSL library (version OpenSSL 1.1.1l  24 Aug 2021) initialized
...
  9276:20211122:194920.242 agent #2 started [active checks #1]
...
  9276:20211122:194920.245 zbx_tls_init_child() loaded PSK identity "TEST"
...
  9276:20211122:194920.246 zbx_tls_init_child() PSK ciphersuites: TLS_CHACHA20_POLY1305_SHA256 TLS_AES_128_GCM_SHA256 ECDHE-PSK-AES128-CBC-SHA256 ECDHE-PSK-AES128-CBC-SHA PSK-AES128-GCM-SHA256 PSK-AES128-CCM8 PSK-AES128-CCM PSK-AES128-CBC-SHA256 PSK-AES128-CBC-SHA
...
  9276:20211122:194920.247 End of zbx_tls_init_child()
...
  9276:20211122:194920.266 In zbx_tls_connect(): psk_identity:"TEST"
  9276:20211122:194920.267 zbx_psk_client_cb() requested PSK identity "TEST"
  9276:20211122:194920.277 End of zbx_tls_connect():SUCCEED (established TLSv1.3 TLS_CHACHA20_POLY1305_SHA256)
  9276:20211122:194920.277 sending [{"request":"active checks","host":"TEST"}]
  9276:20211122:194920.278 sec1:1637606960.277706
  9276:20211122:194920.278 sec2:1637606960.277706
  9276:20211122:194920.278 timed out1:-1 sec:1637606960.277706
  9276:20211122:194920.279 timed out2:-1 sec:0.000000
  9276:20211122:194920.279 s->timeout:30 now:1637606960.279103 sec:0.000000
  9276:20211122:194920.279 active check configuration update from [monitoring.example.com:10051] started to fail (SSL_write() timed out)
  9276:20211122:194920.280 End of refresh_active_checks():FAIL

It should not fail with timeout issue in <1 ms. And with such error, since network dump looks correct for TLS1.2/TLS1.3, ciphers are identical, start with TLS 1.2 and switch to TLS 1.3 extension:

Confirmed.

Regards,
Edgar

We are very happy if we can help solve the problem.

Also could you please clarify, this is started from some specific version of Agent or Windows, or this is a new installation?

zabbix_agent-5.0.18rc1-windows-amd64-openssl.zip

And unfortunately we need to ask once again for a debug log with new binary.

We cannot say for sure if there is a relationship between an specific agent version or an Windows version.

However, it only affects devices purchased this year (these are all Lenovo notebooks with Windows 10 Pro) . Newly set up virtual machines with Windows 10 Pro are not affected.

I uploaded the log as "zabbix_agentd-4.log".

Thank you, please also try [^zabbix_agent-5.0.18rc1-windows-amd64-openssl-5.zip] if possible, for some reason could not reproduce issue locally and get following log. What language for windows are you using ?

  1584:20211123:162725.873 Starting Zabbix Agent [Windows host]. Zabbix 5.0.18rc1 (revision 61fa18b3ee).
  1584:20211123:162725.874 **** Enabled features ****
  1584:20211123:162725.875 IPv6 support:          YES
  1584:20211123:162725.875 TLS support:           YES
  1584:20211123:162725.876 **************************
  1584:20211123:162725.876 using configuration file: C:\Users\malloc\Documents\zabbix_agentd.conf
  1584:20211123:162725.878 In init_collector_data()
  1584:20211123:162725.879 logical CPU count 4
  1584:20211123:162725.879 End of init_collector_data()
  1584:20211123:162725.880 In init_perf_collector()
  1584:20211123:162725.881 In init_builtin_counter_indexes()
  1584:20211123:162725.881 In get_all_counter_names()
  1584:20211123:162725.963 End of get_all_counter_names()
  1584:20211123:162726.085 End of init_builtin_counter_indexes():SUCCEED
  1584:20211123:162726.086 In set_object_names()
  1584:20211123:162726.194 In get_all_counter_names()
  1584:20211123:162726.198 End of get_all_counter_names()
  1584:20211123:162726.199 In get_all_counter_names()
  1584:20211123:162726.203 End of get_all_counter_names()
  1584:20211123:162726.237 End of set_object_names():SUCCEED
  1584:20211123:162726.238 End of init_perf_collector():SUCCEED
  1584:20211123:162726.240 OpenSSL library (version OpenSSL 1.1.1k  25 Mar 2021) initialized
  1584:20211123:162726.240 agent #0 started [main process]
  5140:20211123:162726.241 agent #1 started [collector]
  5308:20211123:162726.241 agent #2 started [listener #1]
  6208:20211123:162726.242 agent #3 started [listener #2]
  6240:20211123:162726.242 agent #4 started [listener #3]
  3208:20211123:162726.243 agent #5 started [active checks #1]
  5140:20211123:162726.244 In init_cpu_collector()
  5308:20211123:162726.244 In zbx_tls_init_child()
  6208:20211123:162726.245 In zbx_tls_init_child()
  6240:20211123:162726.245 In zbx_tls_init_child()
  3208:20211123:162726.246 In zbx_tls_init_child()
  5140:20211123:162726.247 In get_counter_name() pdhIndex:238
  6240:20211123:162726.247 zbx_tls_init_child() loaded PSK identity "PSK 001"
  5308:20211123:162726.248 zbx_tls_init_child() loaded PSK identity "PSK 001"
  6208:20211123:162726.248 zbx_tls_init_child() loaded PSK identity "PSK 001"
  3208:20211123:162726.249 zbx_tls_init_child() loaded PSK identity "PSK 001"
  5140:20211123:162726.250 End of get_counter_name():SUCCEED
  6240:20211123:162726.250 zbx_tls_init_child() loaded PSK from file "C:\Users\malloc\Documents\zabbix_agentd.psk"
  5308:20211123:162726.251 zbx_tls_init_child() loaded PSK from file "C:\Users\malloc\Documents\zabbix_agentd.psk"
  6208:20211123:162726.251 zbx_tls_init_child() loaded PSK from file "C:\Users\malloc\Documents\zabbix_agentd.psk"
  3208:20211123:162726.252 zbx_tls_init_child() loaded PSK from file "C:\Users\malloc\Documents\zabbix_agentd.psk"
  5140:20211123:162726.252 In get_counter_name() pdhIndex:6
  6240:20211123:162726.253 zbx_tls_init_child() PSK ciphersuites: TLS_CHACHA20_POLY1305_SHA256 TLS_AES_128_GCM_SHA256 ECDHE-PSK-AES128-CBC-SHA256 ECDHE-PSK-AES128-CBC-SHA PSK-AES128-GCM-SHA256 PSK-AES128-CCM8 PSK-AES128-CCM PSK-AES128-CBC-SHA256 PSK-AES128-CBC-SHA
  5308:20211123:162726.254 zbx_tls_init_child() PSK ciphersuites: TLS_CHACHA20_POLY1305_SHA256 TLS_AES_128_GCM_SHA256 ECDHE-PSK-AES128-CBC-SHA256 ECDHE-PSK-AES128-CBC-SHA PSK-AES128-GCM-SHA256 PSK-AES128-CCM8 PSK-AES128-CCM PSK-AES128-CBC-SHA256 PSK-AES128-CBC-SHA
  6208:20211123:162726.254 zbx_tls_init_child() PSK ciphersuites: TLS_CHACHA20_POLY1305_SHA256 TLS_AES_128_GCM_SHA256 ECDHE-PSK-AES128-CBC-SHA256 ECDHE-PSK-AES128-CBC-SHA PSK-AES128-GCM-SHA256 PSK-AES128-CCM8 PSK-AES128-CCM PSK-AES128-CBC-SHA256 PSK-AES128-CBC-SHA
  3208:20211123:162726.255 zbx_tls_init_child() PSK ciphersuites: TLS_CHACHA20_POLY1305_SHA256 TLS_AES_128_GCM_SHA256 ECDHE-PSK-AES128-CBC-SHA256 ECDHE-PSK-AES128-CBC-SHA PSK-AES128-GCM-SHA256 PSK-AES128-CCM8 PSK-AES128-CCM PSK-AES128-CBC-SHA256 PSK-AES128-CBC-SHA
  5140:20211123:162726.255 End of get_counter_name():SUCCEED
  6240:20211123:162726.256 End of zbx_tls_init_child()
  5308:20211123:162726.256 End of zbx_tls_init_child()
  6208:20211123:162726.257 End of zbx_tls_init_child()
  3208:20211123:162726.257 End of zbx_tls_init_child()
  5140:20211123:162726.258 In add_perf_counter() counter:'\Processor(_Total)\% Processor Time' interval:900
  3208:20211123:162726.259 In init_active_metrics()
  5140:20211123:162726.259 add_perf_counter(): PerfCounter '\Processor(_Total)\% Processor Time' successfully added
  3208:20211123:162726.260 buffer: first allocation for 100 elements
  5140:20211123:162726.261 End of add_perf_counter(): SUCCEED
  3208:20211123:162726.261 End of init_active_metrics()
  5140:20211123:162726.262 In add_perf_counter() counter:'\Processor(0)\% Processor Time' interval:900
  5140:20211123:162726.262 add_perf_counter(): PerfCounter '\Processor(0)\% Processor Time' successfully added
  3208:20211123:162726.263 In send_buffer() host:'192.168.0.149' port:10051 entries:0/100
  5140:20211123:162726.263 End of add_perf_counter(): SUCCEED
  3208:20211123:162726.264 End of send_buffer():SUCCEED
  5140:20211123:162726.264 In add_perf_counter() counter:'\Processor(1)\% Processor Time' interval:900
  3208:20211123:162726.265 In refresh_active_checks() host:'192.168.0.149' port:10051
  5140:20211123:162726.266 add_perf_counter(): PerfCounter '\Processor(1)\% Processor Time' successfully added
  3208:20211123:162726.266 In zbx_tls_connect(): psk_identity:"PSK 001"
  5140:20211123:162726.267 End of add_perf_counter(): SUCCEED
  3208:20211123:162726.268 zbx_psk_client_cb() requested PSK identity "PSK 001"
  5140:20211123:162726.268 In add_perf_counter() counter:'\Processor(2)\% Processor Time' interval:900
  5140:20211123:162726.269 add_perf_counter(): PerfCounter '\Processor(2)\% Processor Time' successfully added
  5140:20211123:162726.269 End of add_perf_counter(): SUCCEED
  5140:20211123:162726.270 In add_perf_counter() counter:'\Processor(3)\% Processor Time' interval:900
  5140:20211123:162726.270 add_perf_counter(): PerfCounter '\Processor(3)\% Processor Time' successfully added
  5140:20211123:162726.271 End of add_perf_counter(): SUCCEED
  5140:20211123:162726.272 In get_counter_name() pdhIndex:2
  5140:20211123:162726.272 End of get_counter_name():SUCCEED
  5140:20211123:162726.273 In get_counter_name() pdhIndex:44
  5140:20211123:162726.273 End of get_counter_name():SUCCEED
  5140:20211123:162726.274 In add_perf_counter() counter:'\System\Processor Queue Length' interval:900
  5140:20211123:162726.275 add_perf_counter(): PerfCounter '\System\Processor Queue Length' successfully added
  5140:20211123:162726.275 End of add_perf_counter(): SUCCEED
  5140:20211123:162726.276 End of init_cpu_collector():SUCCEED
  5140:20211123:162726.276 In collect_perfstat()
  5140:20211123:162726.277 End of collect_perfstat()
  3208:20211123:162726.284 End of zbx_tls_connect():SUCCEED (established TLSv1.3 TLS_CHACHA20_POLY1305_SHA256)
  3208:20211123:162726.284 sending [{"request":"active checks","host":"Windows host","port":10052}]
  3208:20211123:162726.285 strlen buffer:2047 buffer2:2047
  3208:20211123:162726.285 sec 1:1637677646.284039
  3208:20211123:162726.286 sec 2:1637677646.284039
  3208:20211123:162726.286 timed out1:-1 sec2:1637677646.284039
  3208:20211123:162726.287 strlen buffer:2047 buffer2:2047
  3208:20211123:162726.288 info ''
  3208:20211123:162726.288 res:75
  3208:20211123:162726.289 timed out2:-1 sec2:1637677646.284039 sec:1637677646.284039

Can you tell what is OpenSSL version on Zabbix server?

It happens on recent Windows business notebooks. Could it be some Windows security software (firewall, antivirus?) trying to protect machine? Or some network device guarding internal network from employee notebooks?

I uploaded the log as "zabbix_agentd-5.log".

Windows language is German.

OpenSSL on Zabbix server:

# openssl version -v
OpenSSL 1.1.1  11 Sep 2018

(Ubuntu 18.04 LTS, openssl 1.1.1-1ubuntu2.1~18.04.13)

The devices do not have an unusual configuration. We mainly installed a few standard apps (Firefox, Chrome, Microsoft Office, TeamViewer, 7-Zip, PDF24, Google Drive).

Only Windows Defender/Windows Firewall and Bitlocker are active. But I also tested it with Windows Defender/Windows Firewall deactivated.

No hardware firewall. Problem also occurs in different networks (our network, customer network, ..).

We are happy to give you access to one of the devices if that helps.

"zabbix_agentd-5.log" is different from all other logs!

In logs zabbix_agentd.log, zabbix_agentd-2.log, zabbix_agentd-3.log, zabbix_agentd-4.log it is sending request what fails (in SSL_write()), e.g.

 15236:20211123:115905.073 sending [{"request":"active checks","host":"TEST"}]
...
 15236:20211123:115905.075 active check configuration update from [monitoring.example.com:10051] started to fail (SSL_write() timed out)

In log zabbix_agentd-5.log sending succeeds but receiving of response fails in SSL_read():

  4120:20211123:181833.009 sending [{"request":"active checks","host":"TEST"}]
...
  4120:20211123:181833.012 before read
  4120:20211123:181833.030 active check configuration update from [monitoring.example.com:10051] started to fail (SSL_read() timed out)

What could cause difference?

How many bytes is PSK key in file C:\Program Files\Zabbix Agent\psk.key ?

• 64 bytes, seems good.

Oh, I didn't notice. The last build from Vladislavs Sokurenko causes reproducible the SSL_read() error. If i go back to the previous build from Edgar Akhmetshin then it causes the SSL_write() error. So something should have changed in the builds!?

I uploaded the psk.key. Since we generate it randomly per client and I change it afterwards, it's okay to publish it.

Sorry for asking again, but please try [^zabbix_agent-5.0.18rc1-windows-amd64-openssl-6.zip] it delays starting of collectors to see if they are the cause of the issue

I uploaded the log as "zabbix_agentd-6.log".

Please try [^zabbix_agent-5.0.18rc1-windows-amd64-openssl-7.zip] also please try setting StartAgents=0 in configuration file.

This build works

We already have "StartAgents = 0" in the configuration file (see uploaded zabbix_agentd.conf).

I uploaded the log as "zabbix_agentd-7.log".

Please also try [^zabbix_agent-5.0.18rc1-windows-amd64-openssl-8.zip] yes, it is possible to fix by putting some values on a stack before calling ssl_write but not yet sure why calling ssl_write results in such behavior, I have fixed debug in new version, with previous it did not show when written to stdout for some reason.

I uploaded the log: zabbix_agentd-8.log

If this is not too much to ask, can you please try different agent versions from https://www.zabbix.com/download_agents and check if it is broken from some version, that way maybe we can check if it was broken in some commit. Currently workaround is found but it is not yet reproducible or clear what could have caused it.
For example:
Check
Zabbix agent v3.0.32

If issue does not exist there then check
Zabbix agent v4.0.35

If issue does not exist there then check
Zabbix agent v5.0.1

If issue does not exist there then check
Zabbix agent v5.0.6

If issue does not exist there then check
Zabbix agent v5.0.9

If issue does not exist there then check
Zabbix agent v5.0.13

If issue does not exist there then check
Zabbix agent v5.0.11

If issue does not exist there then check
Zabbix agent v5.0.12

My team has informed me that we have the same problem with other devices. Not just Lenovo notebooks, but also a Surface Laptop 4 and two Intel NUC systems.

I tested it with Zabbix Agent 3.0.32 and it doesn't work with that either. Not even with the older 3.0.25. Here is the uploaded log with 3.0.32: zabbix_agentd-3.0.32.log

Maybe you can compare with Zabbix Agent 2 what is done differently there? Because with Zabbix Agent 2 it works, as mentioned above.

Thank you, Pah

Does it possible to convert affected Windows system to VMware image (or any other virtualisation software) for example and provide it to us?

https://kb.vmware.com/s/article/1018406

We have tested several Windows releases and versions and currently not able to reproduce locally.

Regards,
Edgar

Yes, that is possible. Can you give me an email address to send the download link to? We don't necessarily want to make them publicly available.

Conversion is in progress. I am curious whether it is reproducible in a VM or whether it is somehow related to the hardware.

How much space is required for the converted image?

I tested the VM image on my PC and unfortunately the problem doesn't appear in the VM. It seems to have something to do with the hardware.

Does affected hardware have something in common, what CPU does it use ?

If it is hardware then maybe concurrency issue due to CPU and not reproducible on slower virtual machine, please try zabbix_agent-5.0.18rc1-windows-amd64-openssl-10.zip added more debug and disabled more code that could execute concurrent threads.

How do you launch zabbix agent ?
Does it behaves the same when launched from command line ?

zabbix_agentd.exe -c zabbix_agentd.conf -f

The devices all have in common that they have an Intel 11th Gen CPU and are connected via WLAN. The WLAN module is also the same (Intel AX201). 25x ThinkPad X1 Carbon Gen 9 with Intel Core i7-1165G7, 3x Lenovo ThinkBook 13s G2 ITL with Intel Core i5-1135G7 and 1x Surface Laptop 4 with Intel Core i5-1135G7. 

Your last build works, but it only sends data once and then terminates. I uploaded the log: zabbix_agentd-10.log

I launch the Zabbix Agent as Windows service. If i start it by command line it behaves the same (tested with 5.0.17 without debugging).

Unfortunately currently issue looks like not reproducible even though we understand that there is something wrong because debug shows it, however issue occurs after invoking ssl_write() in OpenSSL library and not in our code which makes it even harder to debug. We will think about what should be our next steps.

Attached debug version zabbix_agent-5.0-debug.zip running it with drmemory could show something.

"C:\Program Files (x86)\Dr. Memory\bin\drmemory.exe" -light -- zabbix_agentd.exe -c C:\Users\malloc\Documents\zabbix_agentd.conf -f
"C:\Program Files (x86)\Dr. Memory\bin\drmemory.exe" -- zabbix_agentd.exe -c C:\Users\malloc\Documents\zabbix_agentd.conf -f

If I run this:

"C:\Program Files (x86)\Dr. Memory\bin\drmemory.exe" -light -- zabbix_agentd.exe -c "C:\Program Files\Zabbix Agent\zabbix_agentd.conf" -f
"C:\Program Files (x86)\Dr. Memory\bin\drmemory.exe" -- zabbix_agentd.exe -c "C:\Program Files\Zabbix Agent\zabbix_agentd.conf" -f

I get the following two errors:

Translated: The code execution cannot proceed because "libcrypto-1_1-x64.dll" and "libssl-1_1-x64.dll" was not found.

You can take dll from here

Here is the output from Dr. Memory: drmemory.txt

And the corresponding Zabbix log file (contains output from Dr. Memory too): zabbix_agentd-drmemory.log

Does that help?

Is there anything else we can do?

Can you implement a fix or workaround? We would like to be able to use the Zabbix agent on the clients.

Unfortunately drmemory couldn't report correctly because it is missing debug symbols did you launch from same directory as zabbix_agentd.pdb ?
Yes, workaround is simple and possible but ideally we should get to the bottom of this and understand the real cause.

You can try it with the latest OpenSSL 3.0.1 - zabbix-5.0.19.OpenSSL-3.0.1-windows-amd64.tgz.

@Vladislav Sokurenko: Sorry, missed the .pdb file. Here ist the output:
DrMemory-zabbix_agentd.exe.7140.000.zip (with "-light" switch)
DrMemory-zabbix_agentd.exe.10844.000.zip (without "-light" switch)

@Andris Mednis: Unfortunately, the error occurs with it too. Log here: zabbix_agentd-5.0.19.log

Did that help? Is there anything else I can do?
Could you add the workaround until we find the real cause? It would be really really great if we could use the Zabbix Agent again.

Please try zabbix_agent-5.0.21rc1-windows-amd64-openssl.zip it has those functions wrapped.

But that seems to be Zabbix Agent 2 now? Zabbix Agent 2 has always worked on the devices. Does that mean we should switch to the Zabbix Agent 2 or was that a mistake?

That was a mistake, corrected, please try if that helps.

Unfortunately that build doesn't work. See log: zabbix_agentd-5.0.21rc1.log

Yes, same error, zabbix_agent-5.0.21rc1-windows-amd64-openssl-2.zip should do the trick

This build works. Thank you very much.

So can we expect the fix in version 5.0.21? That would be great.

Yes, it should be possible to include it as it removes redundant code that failed, attached patch ZBX-20228-5.0.diff if you wish to build yourself right now.

Fixed in pull request feature/ZBX-20228-5.0-2

Fixed in:

• pre-5.0.21rc1 9096c3a45d1
• pre-5.4.11rc1 a7b1a10b951
• pre-6.0.1rc1 22e66ff4e7a
• pre-6.2.0alpha1 (master) 171d3bf5e53

I just had this crop up on six out of fifty five servers. Windows server 2019 1809 17763.2114. Logs indicate they all got the same problem on the 20th of this month. 

Changing to zabbix_agent-5.0.21rc1-windows-amd64-openssl-2 DID fix the problem. 

I would not say this is of trivial priority, my trigger is set up to alert on 0/1 and does nothing if null/empty. We were unmonitored and had no idea of this failure until something else happened and back tracked on why we weren't getting this alert. I will be checking what I can do about additional triggers to check for this failure going forward.