Hello Javier!

I have tried to reproduce the error, but could not. I installed a fresh centos 5.9 and zabbix_agent from repo. I disabled tcp6 while installing os. And got an answer 0 to the item net.tcp.listen[443].

If there is a service running on port 443, can you disable it and test again?

1. ss -nap | grep :443
2. strace zabbix_agentd -t "net.tcp.listen[443]" &> /tmp/strace-port-443-not-https.txt

Uploading the output:

strace-port-443-not-https.txt

When it is not 443 port open , it works because it return 0, but when I add "Listen 443" to httpd.conf (at ssl.conf) it return 0 too.

I have test net.tcp.listen[5000] with nc -l 5000, and it works, so it is something on our apache installation

We have "Listen 443" directive, but I tested with "Listen 0.0.0.0:443" too

I have investigating even more:

If I test an privileged port (<1024), it fail

If I test a unprivileged port (>= 1024), it works

Not sure why 81 not work with nc, but 80 not work with apache (I cannot stop apache and test nc with 80)

Can the size of /proc/net/tcp be the reason ?

See the attached strace shorted (tail) output, and the file /proc/net/tcp when 443 is listening but net.tcp.listen return 0.

For the last, this server have a tunned (please does not ask me why ! ) sysctl.conf, I'm uploading it too

Thank you very much

Hello Javier.

Just jumping into this topic now. Creating a summary.

Based on the information provided, we do know that Zabbix function net.tcp.listen[] under CentOS 5 uses /proc/net/tcp to get the metric.
This is stated on documentation as well: https://www.zabbix.com/documentation/4.0/manual/config/items/itemtypes/zabbix_agent .

Somehow your strace result shows that zabbix_agentd are trying to access /proc/net/tcp6 . Does the file exist on your system?

Attaching default.sysctl.conf which comes by default in CentOS 5. What is the output of 'netstat -tulpn'?

Regards,

I have been on holidays, tomorrow os a not working day in Spain, I will reply at monday

Thanks!

Hello,

# LANG=C ls /proc/net/tcp6
ls: /proc/net/tcp6: No such file or directory

# netstat -tulpn
Active Internet connections (only servers)
Proto Recv-Q Send-Q Local Address Foreign Address State PID/Program name
tcp 0 0 0.0.0.0:10050 0.0.0.0:* LISTEN 14395/zabbix_agentd
tcp 0 0 0.0.0.0:6051 0.0.0.0:* LISTEN 3215/caagentd
tcp 0 0 0.0.0.0:80 0.0.0.0:* LISTEN 8271/httpd
tcp 0 0 0.0.0.0:22 0.0.0.0:* LISTEN 1817/sshd
tcp 0 0 0.0.0.0:443 0.0.0.0:* LISTEN 8271/httpd
udp 0 0 0.0.0.0:6051 0.0.0.0:* 3215/caagentd
udp 0 0 0.0.0.0:28208 0.0.0.0:* 8271/httpd
udp 0 0 0.0.0.0:41524 0.0.0.0:* 3215/caagentd
udp 0 0 192.168.100.70:123 0.0.0.0:* 2186/ntpd
udp 0 0 192.168.6.160:123 0.0.0.0:* 2186/ntpd
udp 0 0 10.188.7.150:123 0.0.0.0:* 2186/ntpd
udp 0 0 127.0.0.1:123 0.0.0.0:* 2186/ntpd
udp 0 0 0.0.0.0:123 0.0.0.0:* 2186/ntpd

I cloned the vm, and when it boot, zabbix reply as expected

When server boot /proc/net/tcp has 900 bytes.

But when the server have a large uptime:

 # uptime
 11:55:25 up 172 days, 11:18, 4 users, load average: 0.37, 0.56, 0.72
# wc -c /proc/net/tcp 
404550 /proc/net/tcp

Maybe when /proc/net/tcp is larger, zabbix does not recognized well the tcp listen port state ?

Update: After playing with "for x in $(seq 444 3399); do nc -l $x & done" to  force the tcp file size be greater, and restarting httpd, zabbix still is able to reply ok. So simply the size is not the question

Thanks !

I have just installed zabbix 4.0 agent, to test, but:

$ time zabbix_get -s 192.168.6.160 -k 'net.tcp.listen[443]'
zabbix_get [21991]: Check access restrictions in Zabbix agent configuration

At log server:

3460:20181015:134945.089 Message from 192.168.0.195 is missing header. Message ignored.
3460:20181015:134945.089 Process listener error: socket() for [[-]:10050] failed: [97] Address family not supported by protocol

Can I have zabbix agent 4 with zabbix server 3.2  ?

Thank you !

Hello,

Finally I think I fixed this issue at local, at least:

At ./src/libs/zbxsysinfo/linux/net.c:307:

if (++count == 2), when I change to (++count == 1) 

it works fine.

I had made a dirty program attached, which I can compile / execute only for this purpose

Please this .c file is only for testing, nothing serious, obviously

test-tcp-net-listen.c

^{Not sure why if I copy /proc/net/tcp to a file, test-tcp-net-listen.c and change the path of the file read to such file (at 376 line [proc_read_tcp_listen parameter), works without modify the 307 line}

^{Please tell me if you need more info}

^{Thank you very much}

Could you please be so kind and show output of:
cat /proc/net/tcp
cat /proc/net/tcp6

Hello,

Please ,see https://support.zabbix.com/secure/attachment/66321/proc-net-tcp.txt , tpc6 file does not exist as commented at https://support.zabbix.com/browse/ZBX-14899?focusedCommentId=297049&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-297049

Thank you!

Fixed in development branch:

svn://svn.zabbix.com/branches/dev/ZBX-14899

Thank you for your report, issue was reproduced, the problem is that 3, SYN_RECV (A connection request has been received from the network) is present before listening sockets and Zabbix agent stops reading after it encounters non listening sockets.

Thank you for fixing this issue ! And sorry for my wrong patch

Fixed in:

• pre-3.0.24rc1 r86710
• pre-4.0.2rc1 r86711
• pre-4.2.0alpha1 r86712

Is possible to fix it on 3.4 branch ? Or I should upgrade to 4.0.2 when the version arrive to rpm repo ?

Thank you !

Is possible to fix it on 3.4 branch ? Or I should upgrade to 4.0.2 when the version arrive to rpm repo ?

3.4 branch is frozen, please upgrade to 4.0.2.