[#ZBX-21300] net.tcp.socket.count used high cpu

Comment by Edgar Akhmetshin [ 2022 Jul 06 ]

Hello yinyaliang,

Please provide output from this system:

uname -a;
cat /etc/os-release;
zabbix_agent2 -V;
grep model /proc/cpuinfo;
cat /sys/class/dmi/id/sys_vendor;
cat /sys/class/dmi/id/product_name;

And:

ss -ptln

Regards,
Edgar

Comment by yinyaliang [ 2022 Jul 06 ]

uname -a

Linux eeo-wapgcb305 4.14.141-1.el7.x86_64 #1 SMP Tue Sep 3 14:17:45 CST 2019 x86_64 x86_64 x86_64 GNU/Linux

cat /etc/os-release

NAME="CentOS Linux"
VERSION="7 (Core)"
ID="centos"
ID_LIKE="rhel fedora"
VERSION_ID="7"
PRETTY_NAME="CentOS Linux 7 (Core)"
ANSI_COLOR="0;31"
CPE_NAME="cpe:/o:centos:centos:7"
HOME_URL="https://www.centos.org/"
BUG_REPORT_URL="https://bugs.centos.org/"

CENTOS_MANTISBT_PROJECT="CentOS-7"
CENTOS_MANTISBT_PROJECT_VERSION="7"
REDHAT_SUPPORT_PRODUCT="centos"
REDHAT_SUPPORT_PRODUCT_VERSION="7"

zabbix_agent2 -V

zabbix_agent2 (Zabbix) 6.0.0
Revision 5203d2ea7d 14 February 2022, compilation time: 4月 22 2022 11:15:47

grep model /proc/cpuinfo (32t)

model           : 63
model name      : Intel(R) Xeon(R) Gold 6248 CPU @ 2.50GHz
model           : 63
model name      : Intel(R) Xeon(R) Gold 6248 CPU @ 2.50GHz
model           : 63
model name      : Intel(R) Xeon(R) Gold 6248 CPU @ 2.50GHz
model           : 63
model name      : Intel(R) Xeon(R) Gold 6248 CPU @ 2.50GHz
model           : 63
model name      : Intel(R) Xeon(R) Gold 6248 CPU @ 2.50GHz
model           : 63
model name      : Intel(R) Xeon(R) Gold 6248 CPU @ 2.50GHz
model           : 63
model name      : Intel(R) Xeon(R) Gold 6248 CPU @ 2.50GHz
model           : 63
model name      : Intel(R) Xeon(R) Gold 6248 CPU @ 2.50GHz
model           : 63
model name      : Intel(R) Xeon(R) Gold 6248 CPU @ 2.50GHz
model           : 63
model name      : Intel(R) Xeon(R) Gold 6248 CPU @ 2.50GHz
model           : 63
model name      : Intel(R) Xeon(R) Gold 6248 CPU @ 2.50GHz
model           : 63
model name      : Intel(R) Xeon(R) Gold 6248 CPU @ 2.50GHz
model           : 63
model name      : Intel(R) Xeon(R) Gold 6248 CPU @ 2.50GHz
model           : 63
model name      : Intel(R) Xeon(R) Gold 6248 CPU @ 2.50GHz
model           : 63
model name      : Intel(R) Xeon(R) Gold 6248 CPU @ 2.50GHz
model           : 63
model name      : Intel(R) Xeon(R) Gold 6248 CPU @ 2.50GHz
model           : 63
model name      : Intel(R) Xeon(R) Gold 6248 CPU @ 2.50GHz
model           : 63
model name      : Intel(R) Xeon(R) Gold 6248 CPU @ 2.50GHz
model           : 63
model name      : Intel(R) Xeon(R) Gold 6248 CPU @ 2.50GHz
model           : 63
model name      : Intel(R) Xeon(R) Gold 6248 CPU @ 2.50GHz
model           : 63
model name      : Intel(R) Xeon(R) Gold 6248 CPU @ 2.50GHz
model           : 63
model name      : Intel(R) Xeon(R) Gold 6248 CPU @ 2.50GHz
model           : 63
model name      : Intel(R) Xeon(R) Gold 6248 CPU @ 2.50GHz
model           : 63
model name      : Intel(R) Xeon(R) Gold 6248 CPU @ 2.50GHz
model           : 63
model name      : Intel(R) Xeon(R) Gold 6248 CPU @ 2.50GHz
model           : 63
model name      : Intel(R) Xeon(R) Gold 6248 CPU @ 2.50GHz
model           : 63
model name      : Intel(R) Xeon(R) Gold 6248 CPU @ 2.50GHz
model           : 63
model name      : Intel(R) Xeon(R) Gold 6248 CPU @ 2.50GHz
model           : 63
model name      : Intel(R) Xeon(R) Gold 6248 CPU @ 2.50GHz
model           : 63
model name      : Intel(R) Xeon(R) Gold 6248 CPU @ 2.50GHz
model           : 63
model name      : Intel(R) Xeon(R) Gold 6248 CPU @ 2.50GHz
model           : 63
model name      : Intel(R) Xeon(R) Gold 6248 CPU @ 2.50GHz

cat /sys/class/dmi/id/sys_vendor

VMware, Inc.

/sys/class/dmi/id/product_name

VMware Virtual Platform

ss -ptln

State      Recv-Q Send-Q                                                                    Local Address:Port                                                                                   Peer Address:Port              
LISTEN     0      128                                                                                   *:111                                                                                               *:*                   users:(("rpcbind",pid=1020,fd=8))
LISTEN     0      128                                                                                   *:61104                                                                                             *:*                   users:(("sshd",pid=1464,fd=3))
LISTEN     0      128                                                                                   *:47029                                                                                             *:*                   users:(("rpc.statd",pid=4804,fd=9))
LISTEN     0      65535                                                                      172.17.224.5:31350                                                                                             *:*                   users:(("zabbix_agent2",pid=2567,fd=8))
LISTEN     0      64                                                                                    *:27011                                                                                             *:*                  
LISTEN     0      128                                                                                  :::111                                                                                              :::*                   users:(("rpcbind",pid=1020,fd=11))
LISTEN     0      128                                                                                  :::61104                                                                                            :::*                   users:(("sshd",pid=1464,fd=4))
LISTEN     0      128                                                                                  :::20225                                                                                            :::*                   users:(("rpc.statd",pid=4804,fd=11))
LISTEN     0      511                                                                                  :::61185                                                                                            :::*                   users:(("php-fpm",pid=41501,fd=642),("php-fpm",pid=41385,fd=927),("php-fpm",pid=41384,fd=888),("php-fpm",pid=41363,fd=641),("php-fpm",pid=41351,fd=811),("php-fpm",pid=41350,fd=780),("php-fpm",pid=41349,fd=561),("php-fpm",pid=41318,fd=534),("php-fpm",pid=41293,fd=511),("php-fpm",pid=41292,fd=225),("php-fpm",pid=41280,fd=213),("php-fpm",pid=41279,fd=210),("php-fpm",pid=41246,fd=119),("php-fpm",pid=41228,fd=186),("php-fpm",pid=41213,fd=121),("php-fpm",pid=41194,fd=94),("php-fpm",pid=41193,fd=108),("php-fpm",pid=41192,fd=32),("php-fpm",pid=41180,fd=68),("php-fpm",pid=41179,fd=987),("php-fpm",pid=41178,fd=984),("php-fpm",pid=41160,fd=977),("php-fpm",pid=41159,fd=967),("php-fpm",pid=41158,fd=964),("php-fpm",pid=41157,fd=645),("php-fpm",pid=41156,fd=956),("php-fpm",pid=41155,fd=907),("php-fpm",pid=41154,fd=904),("php-fpm",pid=41142,fd=894),("php-fpm",pid=41141,fd=893),("php-fpm",pid=41140,fd=879),("php-fpm",pid=41139,fd=845),("php-fpm",pid=41138,fd=831),("php-fpm",pid=41137,fd=816),("php-fpm",pid=41136,fd=766),("php-fpm",pid=41135,fd=835),("php-fpm",pid=41114,fd=759),("php-fpm",pid=41113,fd=747),("php-fpm",pid=41112,fd=742),("php-fpm",pid=41111,fd=723),("php-fpm",pid=41110,fd=732),("php-fpm",pid=41098,fd=701),("php-fpm",pid=41097,fd=735),("php-fpm",pid=41096,fd=709),("php-fpm",pid=41095,fd=725),("php-fpm",pid=41094,fd=694),("php-fpm",pid=41093,fd=706),("php-fpm",pid=41092,fd=687),("php-fpm",pid=41091,fd=684),("php-fpm",pid=41090,fd=672),("php-fpm",pid=41089,fd=656),("php-fpm",pid=41088,fd=652),("php-fpm",pid=41084,fd=632),("php-fpm",pid=41083,fd=592),("php-fpm",pid=41065,fd=630),("php-fpm",pid=41064,fd=607),("php-fpm",pid=41063,fd=544),("php-fpm",pid=41062,fd=532),("php-fpm",pid=41050,fd=525),("php-fpm",pid=41049,fd=516),("php-fpm",pid=41048,fd=498),("php-fpm",pid=41047,fd=470),("php-fpm",pid=41046,fd=465),("php-fpm",pid=41045,fd=445),("php-fpm",pid=41025,fd=424),("php-fpm",pid=41024,fd=434),("php-fpm",pid=41023,fd=414),("php-fpm",pid=41022,fd=410),("php-fpm",pid=41020,fd=406),("php-fpm",pid=41019,fd=396),("php-fpm",pid=41007,fd=363),("php-fpm",pid=41006,fd=360),("php-fpm",pid=41005,fd=358),("php-fpm",pid=41004,fd=297),("php-fpm",pid=41003,fd=343),("php-fpm",pid=41002,fd=301),("php-fpm",pid=41001,fd=314),("php-fpm",pid=41000,fd=280),("php-fpm",pid=40999,fd=252),("php-fpm",pid=40980,fd=250),("php-fpm",pid=40979,fd=236),("php-fpm",pid=40978,fd=235),("php-fpm",pid=40977,fd=196),("php-fpm",pid=40976,fd=211),("php-fpm",pid=40962,fd=205),("php-fpm",pid=40961,fd=190),("php-fpm",pid=40960,fd=193),("php-fpm",pid=40959,fd=180),("php-fpm",pid=40958,fd=175),("php-fpm",pid=40957,fd=111),("php-fpm",pid=40956,fd=154),("php-fpm",pid=40955,fd=118),("php-fpm",pid=40954,fd=116),("php-fpm",pid=40932,fd=143),("php-fpm",pid=40931,fd=107),("php-fpm",pid=40930,fd=91),("php-fpm",pid=40929,fd=88),("php-fpm",pid=40928,fd=63),("php-fpm",pid=40916,fd=70),("php-fpm",pid=40915,fd=58),("php-fpm",pid=40914,fd=43),("php-fpm",pid=40913,fd=69),("php-fpm",pid=40912,fd=48),("php-fpm",pid=40911,fd=35),("php-fpm",pid=40910,fd=66),("php-fpm",pid=40909,fd=36),("php-fpm",pid=40883,fd=16),("php-fpm",pid=40882,fd=9),("php-fpm",pid=40881,fd=13),("php-fpm",pid=40880,fd=44),("php-fpm",pid=40879,fd=10),("php-fpm",pid=40878,fd=955),("php-fpm",pid=40877,fd=978),("php-fpm",pid=40876,fd=963),("php-fpm",pid=40875,fd=941),("php-fpm",pid=40861,fd=954),("php-fpm",pid=40860,fd=950),("php-fpm",pid=40859,fd=957),("php-fpm",pid=40858,fd=952),("php-fpm",pid=40857,fd=944),("php-fpm",pid=40856,fd=921),("php-fpm",pid=40855,fd=947),("php-fpm",pid=40854,fd=943),("php-fpm",pid=40853,fd=934),("php-fpm",pid=40852,fd=914),("php-fpm",pid=40851,fd=911),("php-fpm",pid=40850,fd=926),("php-fpm",pid=40849,fd=925),("php-fpm",pid=40848,fd=924),("php-fpm",pid=40847,fd=919),("php-fpm",pid=40846,fd=918),("php-fpm",pid=40845,fd=922),("php-fpm",pid=40844,fd=891),("php-fpm",pid=40843,fd=882),("php-fpm",pid=40842,fd=876),("php-fpm",pid=40841,fd=883),("php-fpm",pid=40840,fd=869),("php-fpm",pid=40839,fd=887),("php-fpm",pid=40838,fd=885),("php-fpm",pid=40837,fd=840),("php-fpm",pid=40836,fd=867),("php-fpm",pid=40835,fd=819),("php-fpm",pid=40834,fd=861),("php-fpm",pid=40813,fd=856),("php-fpm",pid=40812,fd=808),("php-fpm",pid=40811,fd=844),("php-fpm",pid=40810,fd=814),("php-fpm",pid=40809,fd=854),("php-fpm",pid=40808,fd=855),("php-fpm",pid=40807,fd=841),("php-fpm",pid=40806,fd=818),("php-fpm",pid=40805,fd=843),("php-fpm",pid=40804,fd=836),("php-fpm",pid=40803,fd=827),("php-fpm",pid=40802,fd=830),("php-fpm",pid=40801,fd=812),("php-fpm",pid=40800,fd=815),("php-fpm",pid=40799,fd=813),("php-fpm",pid=40798,fd=806),("php-fpm",pid=40797,fd=795),("php-fpm",pid=40796,fd=782),("php-fpm",pid=40795,fd=824),("php-fpm",pid=40794,fd=805),("php-fpm",pid=40793,fd=765),("php-fpm",pid=40781,fd=804),("php-fpm",pid=40780,fd=799),("php-fpm",pid=40779,fd=762),("php-fpm",pid=40778,fd=789),("php-fpm",pid=40777,fd=737),("php-fpm",pid=40776,fd=771),("php-fpm",pid=40775,fd=781),("php-fpm",pid=40774,fd=788),("php-fpm",pid=40773,fd=764),("php-fpm",pid=40772,fd=786),("php-fpm",pid=40771,fd=736),("php-fpm",pid=40770,fd=739),("php-fpm",pid=40769,fd=779),("php-fpm",pid=40768,fd=774),("php-fpm",pid=40767,fd=761),("php-fpm",pid=40766,fd=731),("php-fpm",pid=40765,fd=768),("php-fpm",pid=40764,fd=719),("php-fpm",pid=40763,fd=752),("php-fpm",pid=40762,fd=738),("php-fpm",pid=40761,fd=720),("php-fpm",pid=40760,fd=734),("php-fpm",pid=40759,fd=692),("php-fpm",pid=40758,fd=698),("php-fpm",pid=40757,fd=714),("php-fpm",pid=40738,fd=690),("php-fpm",pid=40737,fd=707),("php-fpm",pid=40736,fd=695),("php-fpm",pid=40735,fd=685),("php-fpm",pid=40734,fd=699),("php-fpm",pid=40733,fd=647),("php-fpm",pid=40732,fd=680),("php-fpm",pid=40731,fd=697),("php-fpm",pid=40730,fd=682),("php-fpm",pid=40729,fd=664),("php-fpm",pid=40728,fd=637),("php-fpm",pid=40727,fd=669),("php-fpm",pid=40726,fd=625),("php-fpm",pid=40725,fd=660),("php-fpm",pid=40724,fd=667),("php-fpm",pid=40723,fd=666),("php-fpm",pid=40722,fd=662),("php-fpm",pid=40721,fd=594),("php-fpm",pid=40720,fd=659),("php-fpm",pid=40719,fd=635),("php-fpm",pid=40718,fd=629),("php-fpm",pid=40717,fd=597),("php-fpm",pid=40716,fd=585),("php-fpm",pid=40704,fd=624),("php-fpm",pid=40703,fd=627),("php-fpm",pid=40702,fd=620),("php-fpm",pid=40701,fd=619),("php-fpm",pid=40700,fd=562),("php-fpm",pid=40699,fd=596),("php-fpm",pid=40698,fd=626),("php-fpm",pid=40697,fd=622),("php-fpm",pid=40696,fd=590),("php-fpm",pid=40695,fd=548),("php-fpm",pid=40694,fd=556),("php-fpm",pid=40693,fd=578),("php-fpm",pid=40692,fd=586),("php-fpm",pid=40691,fd=580),("php-fpm",pid=40690,fd=541),("php-fpm",pid=40689,fd=571),("php-fpm",pid=40688,fd=549),("php-fpm",pid=40687,fd=567),("php-fpm",pid=40686,fd=583),("php-fpm",pid=40685,fd=579),("php-fpm",pid=40683,fd=566),("php-fpm",pid=40682,fd=569),("php-fpm",pid=40681,fd=526),("php-fpm",pid=40680,fd=568),("php-fpm",pid=40679,fd=563),("php-fpm",pid=40678,fd=540),("php-fpm",pid=40677,fd=536),("php-fpm",pid=40676,fd=530),("php-fpm",pid=40675,fd=518),("php-fpm",pid=40674,fd=505),("php-fpm",pid=40673,fd=527),("php-fpm",pid=40653,fd=501),("php-fpm",pid=40652,fd=539),("php-fpm",pid=40651,fd=515),("php-fpm",pid=40650,fd=531),("php-fpm",pid=40649,fd=529),("php-fpm",pid=40648,fd=520),("php-fpm",pid=40647,fd=508),("php-fpm",pid=40646,fd=506),("php-fpm",pid=40645,fd=475),("php-fpm",pid=40644,fd=496),("php-fpm",pid=40643,fd=499),("php-fpm",pid=40642,fd=492),("php-fpm",pid=40641,fd=503),("php-fpm",pid=40640,fd=486),("php-fpm",pid=40639,fd=473),("php-fpm",pid=40638,fd=481),("php-fpm",pid=40637,fd=478),("php-fpm",pid=40636,fd=477),("php-fpm",pid=40635,fd=466),("php-fpm",pid=40634,fd=450),("php-fpm",pid=40633,fd=444),("php-fpm",pid=40632,fd=476),("php-fpm",pid=40631,fd=467),("php-fpm",pid=40630,fd=462),("php-fpm",pid=40629,fd=463),("php-fpm",pid=40628,fd=415),("php-fpm",pid=40627,fd=426),("php-fpm",pid=40626,fd=455),("php-fpm",pid=40625,fd=453),("php-fpm",pid=40623,fd=443),("php-fpm",pid=40609,fd=429),("php-fpm",pid=40608,fd=423),("php-fpm",pid=40607,fd=405),("php-fpm",pid=40606,fd=440),("php-fpm",pid=40605,fd=431),("php-fpm",pid=40604,fd=421),("php-fpm",pid=40603,fd=412),("php-fpm",pid=40602,fd=419),("php-fpm",pid=40601,fd=404),("php-fpm",pid=40600,fd=399),("php-fpm",pid=40599,fd=398),("php-fpm",pid=40598,fd=418),("php-fpm",pid=40597,fd=402),("php-fpm",pid=40596,fd=381),("php-fpm",pid=40595,fd=403),("php-fpm",pid=40594,fd=395),("php-fpm",pid=40593,fd=379),("php-fpm",pid=40592,fd=367),("php-fpm",pid=40591,fd=357),("php-fpm",pid=40590,fd=376),("php-fpm",pid=40589,fd=356),("php-fpm",pid=40588,fd=377),("php-fpm",pid=40587,fd=368),("php-fpm",pid=40586,fd=348),("php-fpm",pid=40585,fd=346),("php-fpm",pid=40584,fd=354),("php-fpm",pid=40583,fd=315),("php-fpm",pid=40582,fd=347),("php-fpm",pid=40581,fd=336),("php-fpm",pid=40561,fd=302),("php-fpm",pid=40560,fd=327),("php-fpm",pid=40559,fd=333),("php-fpm",pid=40558,fd=300),("php-fpm",pid=40557,fd=344),("php-fpm",pid=40556,fd=330),("php-fpm",pid=40555,fd=318),("php-fpm",pid=40554,fd=326),("php-fpm",pid=40553,fd=324),("php-fpm",pid=40552,fd=323),("php-fpm",pid=40551,fd=316),("php-fpm",pid=40550,fd=309),("php-fpm",pid=40549,fd=313),("php-fpm",pid=40548,fd=294),("php-fpm",pid=40547,fd=306),("php-fpm",pid=40546,fd=303),("php-fpm",pid=40545,fd=284),("php-fpm",pid=40544,fd=260),("php-fpm",pid=40543,fd=310),("php-fpm",pid=40542,fd=312),("php-fpm",pid=40541,fd=304),("php-fpm",pid=40540,fd=286),("php-fpm",pid=40539,fd=278),("php-fpm",pid=40538,fd=273),("php-fpm",pid=40537,fd=253),("php-fpm",pid=40536,fd=268),("php-fpm",pid=40535,fd=267),("php-fpm",pid=40534,fd=241),("php-fpm",pid=40533,fd=192),("php-fpm",pid=40532,fd=262),("php-fpm",pid=40531,fd=266),("php-fpm",pid=40530,fd=256),("php-fpm",pid=40529,fd=184),("php-fpm",pid=40528,fd=261),("php-fpm",pid=40527,fd=240),("php-fpm",pid=40526,fd=244),("php-fpm",pid=40525,fd=254),("php-fpm",pid=40524,fd=248),("php-fpm",pid=40523,fd=243),("php-fpm",pid=40522,fd=231),("php-fpm",pid=40521,fd=238),("php-fpm",pid=40520,fd=234),("php-fpm",pid=40519,fd=226),("php-fpm",pid=40505,fd=232),("php-fpm",pid=40504,fd=220),("php-fpm",pid=40503,fd=194),("php-fpm",pid=40502,fd=185),("php-fpm",pid=40501,fd=140),("php-fpm",pid=40500,fd=198),("php-fpm",pid=40499,fd=178),("php-fpm",pid=40498,fd=156),("php-fpm",pid=40497,fd=164),("php-fpm",pid=40496,fd=169),("php-fpm",pid=40495,fd=160),("php-fpm",pid=40494,fd=173),("php-fpm",pid=40493,fd=163),("php-fpm",pid=40492,fd=170),("php-fpm",pid=40491,fd=167),("php-fpm",pid=40490,fd=152),("php-fpm",pid=40489,fd=172),("php-fpm",pid=40488,fd=155),("php-fpm",pid=40487,fd=149),("php-fpm",pid=40486,fd=146),("php-fpm",pid=40485,fd=142),("php-fpm",pid=40484,fd=128),("php-fpm",pid=40483,fd=135),("php-fpm",pid=40482,fd=125),("php-fpm",pid=40481,fd=105),("php-fpm",pid=40480,fd=132),("php-fpm",pid=40479,fd=131),("php-fpm",pid=40478,fd=130),("php-fpm",pid=40477,fd=109),("php-fpm",pid=40476,fd=106),("php-fpm",pid=40475,fd=100),("php-fpm",pid=40474,fd=97),("php-fpm",pid=40473,fd=92),("php-fpm",pid=40472,fd=85),("php-fpm",pid=40471,fd=83),("php-fpm",pid=40470,fd=47),("php-fpm",pid=40469,fd=75),("php-fpm",pid=40468,fd=40),("php-fpm",pid=40467,fd=51),("php-fpm",pid=40466,fd=61),("php-fpm",pid=40465,fd=67),("php-fpm",pid=40464,fd=19),("php-fpm",pid=40463,fd=55),("php-fpm",pid=40462,fd=42),("php-fpm",pid=40461,fd=39),("php-fpm",pid=40460,fd=52),("php-fpm",pid=40440,fd=49),("php-fpm",pid=40439,fd=17),("php-fpm",pid=40438,fd=14),("php-fpm",pid=40437,fd=29),("php-fpm",pid=40436,fd=38),("php-fpm",pid=40435,fd=22),("php-fpm",pid=40434,fd=34),("php-fpm",pid=40433,fd=31),("php-fpm",pid=40432,fd=18),("php-fpm",pid=40431,fd=20),("php-fpm",pid=40430,fd=23),("php-fpm",pid=40429,fd=25),("php-fpm",pid=40428,fd=992),("php-fpm",pid=40427,fd=33),("php-fpm",pid=40426,fd=15),("php-fpm",pid=40425,fd=960),("php-fpm",pid=40424,fd=968),("php-fpm",pid=40423,fd=942),("php-fpm",pid=40422,fd=11),("php-fpm",pid=40421,fd=899),("php-fpm",pid=40420,fd=898),("php-fpm",pid=40406,fd=890),("php-fpm",pid=40405,fd=889),("php-fpm",pid=40404,fd=875),("php-fpm",pid=40403,fd=886),("php-fpm",pid=40402,fd=872),("php-fpm",pid=40401,fd=791),("php-fpm",pid=40400,fd=828),("php-fpm",pid=40399,fd=853),("php-fpm",pid=40398,fd=850),("php-fpm",pid=40397,fd=847),("php-fpm",pid=40396,fd=829),("php-fpm",pid=40395,fd=758),("php-fpm",pid=40394,fd=823),("php-fpm",pid=40393,fd=757),("php-fpm",pid=40376,fd=748),("php-fpm",pid=40375,fd=728),("php-fpm",pid=40374,fd=716),("php-fpm",pid=40373,fd=688),("php-fpm",pid=40372,fd=708),("php-fpm",pid=40371,fd=726),("php-fpm",pid=40370,fd=705),("php-fpm",pid=40358,fd=640),("php-fpm",pid=40357,fd=633),("php-fpm",pid=40356,fd=599),("php-fpm",pid=40355,fd=589),("php-fpm",pid=40354,fd=582),("php-fpm",pid=40353,fd=576),("php-fpm",pid=40352,fd=573),("php-fpm",pid=40351,fd=564),("php-fpm",pid=40331,fd=559),("php-fpm",pid=40330,fd=554),("php-fpm",pid=40329,fd=542),("php-fpm",pid=40328,fd=538),("php-fpm",pid=40327,fd=558),("php-fpm",pid=40326,fd=502),("php-fpm",pid=40325,fd=430),("php-fpm",pid=40324,fd=427),("php-fpm",pid=40323,fd=458),("php-fpm",pid=40322,fd=393),("php-fpm",pid=40310,fd=413),("php-fpm",pid=40309,fd=384),("php-fpm",pid=40308,fd=390),("php-fpm",pid=40307,fd=378),("php-fpm",pid=40306,fd=361),("php-fpm",pid=40305,fd=366),("php-fpm",pid=40304,fd=355),("php-fpm",pid=40285,fd=350),("php-fpm",pid=40284,fd=320),("php-fpm",pid=40283,fd=308),("php-fpm",pid=40282,fd=292),("php-fpm",pid=40281,fd=279),("php-fpm",pid=40280,fd=277),("php-fpm",pid=40279,fd=276),("php-fpm",pid=40278,fd=218),("php-fpm",pid=40276,fd=222),("php-fpm",pid=40275,fd=219),("php-fpm",pid=40274,fd=206),("php-fpm",pid=40260,fd=197),("php-fpm",pid=40259,fd=166),("php-fpm",pid=40258,fd=188),("php-fpm",pid=40257,fd=150),("php-fpm",pid=40256,fd=133),("php-fpm",pid=40255,fd=122),("php-fpm",pid=40254,fd=137),("php-fpm",pid=40253,fd=114),("php-fpm",pid=40252,fd=46),("php-fpm",pid=40251,fd=102),("php-fpm",pid=40249,fd=98),("php-fpm",pid=40227,fd=95),("php-fpm",pid=40226,fd=93),("php-fpm",pid=40225,fd=90),("php-fpm",pid=40224,fd=84),("php-fpm",pid=40223,fd=74),("php-fpm",pid=40222,fd=60),("php-fpm",pid=40221,fd=53),("php-fpm",pid=40220,fd=45),("php-fpm",pid=40219,fd=27),("php-fpm",pid=40218,fd=26),("php-fpm",pid=40217,fd=28),("php-fpm",pid=40216,fd=12),("php-fpm",pid=40215,fd=30),("php-fpm",pid=40214,fd=21),("php-fpm",pid=40213,fd=754),("php-fpm",pid=40212,fd=686),("php-fpm",pid=40211,fd=673),("php-fpm",pid=40210,fd=670),("php-fpm",pid=40209,fd=233),("php-fpm",pid=40208,fd=648),("php-fpm",pid=40207,fd=643),("php-fpm",pid=40206,fd=509),("php-fpm",pid=40205,fd=468),("php-fpm",pid=40204,fd=263),("php-fpm",pid=40192,fd=203),("php-fpm",pid=40191,fd=191),("php-fpm",pid=40190,fd=230),("php-fpm",pid=40189,fd=126),("php-fpm",pid=40188,fd=129),("php-fpm",pid=40187,fd=37),("php-fpm",pid=40043,fd=50),("php-fpm",pid=40010,fd=24),("php-fpm",pid=1467,fd=7))
LISTEN     0      64                                                                                   :::6405                                                                                             :::*

Comment by Edgar Akhmetshin [ 2022 Jul 06 ]

Thank you, one more thing, output from numactl utility and lscpu:

# agent should be running
numastat -c zabbix_agent2
lscpu

And:

time ss -a -t state established | wc -l

Comment by yinyaliang [ 2022 Jul 06 ]

numastat -c zabbix_agent2
Per-node process memory usage (in MBs) for PID 791 (zabbix_agent2)
         Node 0 Total
         ------ -----
Huge          0     0
Heap          0     0
Stack         0     0
Private      36    36
-------  ------ -----
Total        36    36

time ss -a -t state established | wc -l
7685
real    0m0.026s
user    0m0.014s
sys     0m0.015s

lscpu
Architecture:          x86_64
CPU op-mode(s):        32-bit, 64-bit
Byte Order:            Little Endian
CPU(s):                32
On-line CPU(s) list:   0-31
Thread(s) per core:    1
Core(s) per socket:    16
Socket(s):             2
NUMA node(s):          1
Vendor ID:             GenuineIntel
CPU family:            6
Model:                 63
Model name:            Intel(R) Xeon(R) Gold 6248 CPU @ 2.50GHz
Stepping:              0
CPU MHz:               2494.140
BogoMIPS:              4988.28
Hypervisor vendor:     VMware
Virtualization type:   full
L1d cache:             32K
L1i cache:             32K
L2 cache:              1024K
L3 cache:              28160K
NUMA node0 CPU(s):     0-31
Flags:                 fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 ss ht syscall nx pdpe1gb rdtscp lm constant_tsc arch_perfmon nopl xtopology tsc_reliable nonstop_tsc cpuid pni pclmulqdq ssse3 fma cx16 pcid sse4_1 sse4_2 x2apic movbe popcnt tsc_deadline_timer aes xsave avx f16c rdrand hypervisor lahf_lm abm cpuid_fault invpcid_single pti ssbd ibrs ibpb stibp fsgsbase tsc_adjust bmi1 avx2 smep bmi2 invpcid xsaveopt arat md_clear flush_l1d arch_capabilities

Comment by yinyaliang [ 2022 Jul 08 ]

Hello edgar.akhmetshin,

Thanks for you replay, do you have any suggestion for me?

Comment by Edgar Akhmetshin [ 2022 Jul 08 ]

Hi,

I've tested with 10000 connections open to simple nginx web server and Zabbix Agent2 executed in different ways, but cannot reproduce the issue.

What kind of checks you have on the Agent2 otherwise this one?

Could you provide perf profile of the Zabbix Agent 2 pid with long running socket checks?

https://www.brendangregg.com/perf.html

Comment by yinyaliang [ 2022 Jul 08 ]

i replace a host for test.

my test command

ab -n 100000000 -c 3000 -r http://test.com/test

my web code:

class TestTCPConnectHandler(BaseActionHandler):
    def __init__(self,*request,**kwargs):
        super(TestTCPConnectHandler,self).__init__(*request, **kwargs)
    async def get(self):
        await gen.sleep(10)
        return

my agent2 code:

the following is the data of my test host:

uname -a;
Linux omt-yaliang2 4.14.141-1.el7.x86_64 #1 SMP Tue Sep 3 14:17:45 CST 2019 x86_64 x86_64 x86_64 GNU/Linux

cat /etc/os-release;
NAME="CentOS Linux"
VERSION="7 (Core)"
ID="centos"
ID_LIKE="rhel fedora"
VERSION_ID="7"
PRETTY_NAME="CentOS Linux 7 (Core)"
ANSI_COLOR="0;31"
CPE_NAME="cpe:/o:centos:centos:7"
HOME_URL="https://www.centos.org/"
BUG_REPORT_URL="https://bugs.centos.org/"
CENTOS_MANTISBT_PROJECT="CentOS-7"
CENTOS_MANTISBT_PROJECT_VERSION="7"
REDHAT_SUPPORT_PRODUCT="centos"
REDHAT_SUPPORT_PRODUCT_VERSION="7"

zabbix_agent2 -V
zabbix_agent2 (Zabbix) 6.0.0
Revision 5203d2ea7d 14 February 2022, compilation time: 7月  8 2022 14:20:19
Copyright (C) 2022 Zabbix SIA
License GPLv2+: GNU GPL version 2 or later <http://gnu.org/licenses/gpl.html>.
This is free software: you are free to change and redistribute it according to
the license. There is NO WARRANTY, to the extent permitted by law.
We use the library Eclipse Paho (eclipse/paho.mqtt.golang), which is
distributed under the terms of the Eclipse Distribution License 1.0 (The 3-Clause BSD License)
available at https://www.eclipse.org/org/documents/edl-v10.php
We use the library go-modbus (goburrow/modbus), which is
distributed under the terms of the 3-Clause BSD License
available at https://github.com/goburrow/modbus/blob/master/LICENSE

grep model /proc/cpuinfo;
model           : 44
model name      : Westmere E56xx/L56xx/X56xx (Nehalem-C)
model           : 44
model name      : Westmere E56xx/L56xx/X56xx (Nehalem-C)

cat /sys/class/dmi/id/sys_vendor;
QEMU

cat /sys/class/dmi/id/product_name;
Standard PC (i440FX + PIIX, 1996)

 ss -ptln
State      Recv-Q Send-Q          Local Address:Port                         Peer Address:Port              
LISTEN     0      128                         *:111                                     *:*                   users:(("rpcbind",pid=559,fd=8))
LISTEN     0      511                         *:80                                      *:*                   users:(("nginx",pid=16251,fd=7),("nginx",pid=16250,fd=7),("nginx",pid=16249,fd=7))
LISTEN     0      128                         *:61104                                   *:*                   users:(("sshd",pid=766,fd=3))
LISTEN     0      65535               10.0.9.77:31350                                   *:*                   users:(("zabbix_agent2",pid=19832,fd=8))
LISTEN     0      80                       [::]:3306                                 [::]:*                   users:(("mysqld",pid=19484,fd=21))
LISTEN     0      128                      [::]:111                                  [::]:*                   users:(("rpcbind",pid=559,fd=11))
LISTEN     0      511                      [::]:80                                   [::]:*                   users:(("nginx",pid=16251,fd=8),("nginx",pid=16250,fd=8),("nginx",pid=16249,fd=8))
LISTEN     0      128                      [::]:61104                                [::]:*                   users:(("sshd",pid=766,fd=4))
LISTEN     0      65535                    [::]:3128                                 [::]:*                   users:(("squid",pid=12878,fd=15))

numastat -c zabbix_agent2
Per-node process memory usage (in MBs)
PID              Node 0 Total
---------------  ------ -----
19832 (zabbix_ag     24    24
19870 (tailf)         2     2
---------------  ------ -----
Total                26    26

lscpu
Architecture:          x86_64
CPU op-mode(s):        32-bit, 64-bit
Byte Order:            Little Endian
CPU(s):                2
On-line CPU(s) list:   0,1
Thread(s) per core:    1
Core(s) per socket:    1
Socket(s):             2
NUMA node(s):          1
Vendor ID:             GenuineIntel
CPU family:            6
Model:                 44
Model name:            Westmere E56xx/L56xx/X56xx (Nehalem-C)
Stepping:              1
CPU MHz:               2526.998
BogoMIPS:              5053.99
Hypervisor vendor:     KVM
Virtualization type:   full
L1d cache:             32K
L1i cache:             32K
L2 cache:              4096K
L3 cache:              16384K
NUMA node0 CPU(s):     0,1
Flags:                 fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 syscall nx lm constant_tsc rep_good nopl xtopology cpuid tsc_known_freq pni pclmulqdq ssse3 cx16 sse4_1 sse4_2 x2apic popcnt aes hypervisor lahf_lm cpuid_fault pti arat

time ss -a -t state established | wc -l
3239
real    0m0.115s
user    0m0.032s
sys     0m0.030s

perf stat -d ./zabbix_agent2 -t net.tcp.socket.count[,,,,established]
net.tcp.socket.count[,,,,established]         [s|3208] Performance counter stats for './zabbix_agent2 -t net.tcp.socket.count[,,,,established]':          1,389.13 msec task-clock                #    0.598 CPUs utilized          
             2,602      context-switches          #    0.002 M/sec                  
                 5      cpu-migrations            #    0.004 K/sec                  
             2,576      page-faults               #    0.002 M/sec                  
   <not supported>      cycles                                                      
   <not supported>      instructions                                                
   <not supported>      branches                                                    
   <not supported>      branch-misses                                               
   <not supported>      L1-dcache-loads                                             
   <not supported>      L1-dcache-load-misses                                       
   <not supported>      LLC-loads                                                   
   <not supported>      LLC-load-misses                                                    2.323596300 seconds time elapsed       1.171183000 seconds user
       0.217171000 seconds sys

perf top -p

Comment by yinyaliang [ 2022 Jul 12 ]

Hello edgar.akhmetshin

it looks like the probelm with iterFdDir function of netstat . will zabbix update this issue?

Comment by Edgar Akhmetshin [ 2022 Jul 14 ]

Hello yinyaliang,

Thank you for the detailed investigation. Reproduced in the suggested way.

Confirming this issue.

Comment by Vladislavs Sokurenko [ 2022 Jul 15 ]

This is because library is calling "extractProcInfo(tabs)" function we however don't need it and could simply delete it, but it is in library.

<palivoda > I think now we dont have any custom build libraries in the project. What about maintaining and updating such libraries later? I think this is not the way we should go. Instead:

a problem should be reported to the library authors;
a patch for the library could be attached to the ticket, so users can make custom build at their own risk.

FYI - sasha, yurii

Comment by yinyaliang [ 2022 Jul 15 ]

may be can use netlink instead of netstat. like linux command ss and netstat. i write a plugins test

package eeoss
import (
    "encoding/json"
    "fmt"
    "syscall"
    "zabbix.com/pkg/plugin"
    "github.com/vishvananda/netlink/nl"
)
// Plugin -
type Plugin struct {
    plugin.Base
}
var impl Plugin
func (p *Plugin) Export(key string, params []string, ctx plugin.ContextProvider) (result interface{}, err error) {
    switch key {
    case "tcp.count":
        return tcpmMain()
    default:
        return nil, plugin.UnsupportedMetricError
    }
}
func init() {
    plugin.RegisterMetrics(&impl, "Eeotcp",
        "tcp.count", "Count tcp state.",
    )
}
func tcp_state_count() (tcpStateCount map[string]int, err error) {
    tcpStateCount = make(map[string]int, 0)
    // Create a new netlink request
    socket_fd := nl.NewNetlinkRequest(nl.SOCK_DIAG_BY_FAMILY, syscall.NLM_F_DUMP)
    {
        // returns a new NetlinkMessage InetDiagReqV2
        // https://github.com/torvalds/linux/blob/v4.0/include/uapi/linux/inet_diag.h#L37
        msg := NewInetDiagReqV2(
            syscall.AF_INET,
            syscall.IPPROTO_TCP,
            AllTCPStates)
        msg.Ext |= (1 << (INET_DIAG_INFO - 1))
        socket_fd.AddData(msg)
    }
    // return [][]byte
    res, err := socket_fd.Execute(syscall.NETLINK_INET_DIAG, 0)
    if err != nil {
        return nil, err
    }
    for _, data := range res {
        state := TcpStateNames[ParseInetDiagMsg(data).State]
        if _, ok := tcpStateCount[state]; ok {
            tcpStateCount[state] += 1
        } else {
            tcpStateCount[state] = 1
        }
    }
    return tcpStateCount, nil
}
func tcpmMain() (interface{}, error) {
    ret, err := tcp_state_count()
    if err != nil {
        fmt.Println(err)
    }
    jsonArray, err := json.Marshal(ret)
    if nil != err {
        return nil, fmt.Errorf("Cannot create JSON array: %s", err)
    }
    return string(jsonArray), nil
}

package eeoss
import (
    "fmt"
    "syscall"
    "unsafe"
)
// https://pkg.go.dev/github.com/elastic/gosigar/sys/linux
const (
    AllTCPStates = ^uint32(0)
)
// https://github.com/torvalds/linux/blob/5924bbecd0267d87c24110cbe2041b5075173a25/include/net/tcp_states.h#L16
const (
    TCP_ESTABLISHED = iota + 1
    TCP_SYN_SENT
    TCP_SYN_RECV
    TCP_FIN_WAIT1
    TCP_FIN_WAIT2
    TCP_TIME_WAIT
    TCP_CLOSE
    TCP_CLOSE_WAIT
    TCP_LAST_ACK
    TCP_LISTEN
    TCP_CLOSING /* Now a valid state */
)
var TcpStateNames = map[uint8]string{
    TCP_ESTABLISHED: "ESTAB",
    TCP_SYN_SENT:    "SYN-SENT",
    TCP_SYN_RECV:    "SYN-RECV",
    TCP_FIN_WAIT1:   "FIN-WAIT-1",
    TCP_FIN_WAIT2:   "FIN-WAIT-2",
    TCP_TIME_WAIT:   "TIME-WAIT",
    TCP_CLOSE:       "UNCONN",
    TCP_CLOSE_WAIT:  "CLOSE-WAIT",
    TCP_LAST_ACK:    "LAST-ACK",
    TCP_LISTEN:      "LISTEN",
    TCP_CLOSING:     "CLOSING",
}
// Extensions that can be used in the InetDiagReqV2 request to ask for
// additional data.
// https://github.com/torvalds/linux/blob/v4.0/include/uapi/linux/inet_diag.h#L103
const (
    INET_DIAG_NONE    = 0
    INET_DIAG_MEMINFO = 1 << iota
    INET_DIAG_INFO
    INET_DIAG_VEGASINFO
    INET_DIAG_CONG
    INET_DIAG_TOS
    INET_DIAG_TCLASS
    INET_DIAG_SKMEMINFO
    INET_DIAG_SHUTDOWN
    INET_DIAG_DCTCPINFO
    INET_DIAG_PROTOCOL /* response attribute only */
    INET_DIAG_SKV6ONLY
    INET_DIAG_LOCALS
    INET_DIAG_PEERS
    INET_DIAG_PAD
    INET_DIAG_MARK
)
// V2 Request
var SizeofInetDiagReqV2 = int(unsafe.Sizeof(InetDiagReqV2{}))
// InetDiagReqV2 (inet_diag_req_v2) is used to request diagnostic data.
// https://github.com/torvalds/linux/blob/v4.0/include/uapi/linux/inet_diag.h#L37
type InetDiagReqV2 struct {
    Family   uint8
    Protocol uint8
    Ext      uint8
    Pad      uint8
    States   uint32
    ID       InetDiagSockID
}
// InetDiagSockID (inet_diag_sockid) contains the socket identity.
// https://github.com/torvalds/linux/blob/v4.0/include/uapi/linux/inet_diag.h#L13
type InetDiagSockID struct {
    SPort  [2]byte    // Source port (big-endian).
    DPort  [2]byte    // Destination port (big-endian).
    Src    [4][4]byte // Source IP
    Dst    [4][4]byte // Destination IP
    If     uint32
    Cookie [2]uint32
}
// InetDiagMsg (inet_diag_msg) is the base info structure. It contains socket
// identity (addrs/ports/cookie) and the information shown by netstat.
// https://github.com/torvalds/linux/blob/v4.0/include/uapi/linux/inet_diag.h#L86
type InetDiagMsg struct {
    Family  uint8 // Address family.
    State   uint8 // TCP State
    Timer   uint8
    Retrans uint8
    ID InetDiagSockID
    Expires uint32
    RQueue  uint32 // Recv-Q
    WQueue  uint32 // Send-Q
    UID     uint32 // UID
    Inode   uint32 // Inode of socket.
}
var FamilyNames = map[uint8]string{
    syscall.AF_INET:  "tcp",
    syscall.AF_INET6: "tcp6",
}
func (req *InetDiagReqV2) Serialize() []byte {
    return (*(*[56]byte)(unsafe.Pointer(req)))[:]
}
func (req *InetDiagReqV2) Len() int {
    return SizeofInetDiagReqV2
}
func NewInetDiagReqV2(family, protocol uint8, states uint32) *InetDiagReqV2 {
    return &InetDiagReqV2{
        Family:   family,
        Protocol: protocol,
        States:   states,
    }
}
func (msg *InetDiagMsg) String() string {
    return fmt.Sprintf("%s", TcpStateNames[msg.State])
}
func ParseInetDiagMsg(data []byte) *InetDiagMsg {
    return (*InetDiagMsg)(unsafe.Pointer(&data[0]))
}

# time ./zabbix_agent2 -t net.tcp.socket.count[,,,,established]
net.tcp.socket.count[,,,,established]         [s|3783]
real    0m2.814s
user    0m2.055s
sys     0m0.413s

# zabbix_agent2 -t tcp.count
tcp.count                                     [s|{"CLOSE-WAIT":530,"ESTAB":3797,"FIN-WAIT-2":527,"LISTEN":4,"SYN-SENT":2,"TIME-WAIT":176}]

this is comparison of cpu usage:

netlink:

netstat:

Comment by Vladislavs Sokurenko [ 2022 Jul 15 ]

Thank you for the screenshot, how many net.tcp.socket.count keys are used ? It could be more performant to implement one key that will return totals for all socket types as JSON.

Comment by yinyaliang [ 2022 Jul 15 ]

I unlink and clear all template on test host. compare netstat and netlink. netlink is faster and used lower cpu. may be you can test it with my code.

Comment by Vladislavs Sokurenko [ 2022 Jul 15 ]

Could you please be so kind and check if ZBX-21300-6.2.diff is any help ?

Comment by yinyaliang [ 2022 Jul 18 ]

I used netlink code on my online server. it worked fine

Comment by Vladislavs Sokurenko [ 2022 Jul 19 ]

Fixed in pull request feature/ZBX-21300-6.0

Comment by Vladislavs Sokurenko [ 2022 Jul 28 ]

Fixed in:

pre-6.0.8rc1 cc5dcc26abf
pre-6.2.2rc1 2221b32ad23
pre-6.4.0alpha1 (master) 4b785508dc6

Comment by Rostislav Palivoda (Inactive) [ 2022 Nov 09 ]

Raised PR to the go-netstat library - https://github.com/cakturk/go-netstat/pull/17

Comment by yinyaliang [ 2022 Dec 08 ]

May I ask why not use netlink? Based on what considerations?

Comment by Vladislavs Sokurenko [ 2022 Dec 08 ]

Sure, netstat works both for Windows and for Linux, isn't netlink just for Linux and have apache2 license ?

[ZBX-21300] net.tcp.socket.count used high cpu Created: 2022 Jul 05 Updated: 2024 Apr 16 Resolved: 2022 Dec 21
Status:	Closed
Project:	ZABBIX BUGS AND ISSUES
Component/s:	Agent (G)
Affects Version/s:	6.0.6, 6.2.0
Fix Version/s:	6.0.8rc1, 6.2.2rc1, 6.4.0alpha1, 6.4 (plan)

[ZBX-21300] net.tcp.socket.count used high cpu Created: 2022 Jul 05 Updated: 2024 Apr 16 Resolved: 2022 Dec 21