[ZBX-19131] Got signal [signal:11(SIGSEGV),reason:1,refaddr:(nil)]. Crashing ... Agent 5.0.7 Created: 2021 Mar 17  Updated: 2024 Apr 10

Status: Confirmed
Project: ZABBIX BUGS AND ISSUES
Component/s: Agent (G)
Affects Version/s: None
Fix Version/s: None

Type: Problem report Priority: Trivial
Reporter: Damian Cancela Assignee: Andrejs Kozlovs
Resolution: Unresolved Votes: 1
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Attachments: File ZBX-19131-1.diff     Text File agent.log     Zip Archive objdump.zip    
Team: Team C

 Description   

Hello,

after upgrading some instances Zabbix Agent into v5.0.7 (C Version) we are getting some Agent crashs in random instances.

Config file not default parameters:

LogFileSize=256

DebugLevel=3

AllowKey=system.run[*]

LogRemoteCommands=1

StartAgents=3

 

Attached log file with the crash

 

Im using a Zabbix Proxy to monitor those instances, same version (5.0.7)

 

Thanks in advance

 Comments   
Comment by Damian Cancela [ 2021 Mar 18 ]

Hi Dev Team, do you know if this is a known issue or at least the root cause? There is some workaround?

 

Thanks in advance!

Comment by Damian Cancela [ 2021 Mar 24 ]

Adding more information from agent at log level 5, you can see the crash almost at the end.

 2564:20210324:054859.503 zbx_setproctitle() title:'collector [idle 1 sec]'

  2574:20210324:054900.401 In send_buffer() host:'zabbix-proxy-saas-a-eu-west-1.as.cert.ascint.sabrecirrus.com' port:10051 entries:0/100

  2574:20210324:054900.401 End of send_buffer():SUCCEED

  2574:20210324:054900.401 zbx_setproctitle() title:'active checks #1 [idle 1 sec]'

  2564:20210324:054900.503 zbx_setproctitle() title:'collector [processing data]'

  2564:20210324:054900.503 In update_cpustats()

  2564:20210324:054900.503 End of update_cpustats()

  2564:20210324:054900.503 zbx_setproctitle() title:'collector [idle 1 sec]'

  2574:20210324:054901.401 In send_buffer() host:'zabbix-proxy-saas-a-eu-west-1.as.cert.ascint.sabrecirrus.com' port:10051 entries:0/100

  2574:20210324:054901.401 End of send_buffer():SUCCEED

  2574:20210324:054901.401 zbx_setproctitle() title:'active checks #1 [getting list of active checks]'

  2574:20210324:054901.401 In refresh_active_checks() host:'zabbix-proxy-saas-a-eu-west-1.as.cert.ascint.sabrecirrus.com' port:10051

  2564:20210324:054901.503 zbx_setproctitle() title:'collector [processing data]'

  2564:20210324:054901.503 In update_cpustats()

  2564:20210324:054901.503 End of update_cpustats()

  2564:20210324:054901.503 zbx_setproctitle() title:'collector [idle 1 sec]'

  2564:20210324:054902.504 zbx_setproctitle() title:'collector [processing data]'

  2564:20210324:054902.504 In update_cpustats()

  2564:20210324:054902.504 End of update_cpustats()

  2564:20210324:054902.504 zbx_setproctitle() title:'collector [idle 1 sec]'

  2566:20210324:054903.008 zbx_setproctitle() title:'listener #2 [processing request]'

  2564:20210324:054903.504 zbx_setproctitle() title:'collector [processing data]'

  2564:20210324:054903.504 In update_cpustats()

  2564:20210324:054903.504 End of update_cpustats()

  2564:20210324:054903.504 zbx_setproctitle() title:'collector [idle 1 sec]'

  2564:20210324:054904.504 zbx_setproctitle() title:'collector [processing data]'

  2564:20210324:054904.504 In update_cpustats()

  2564:20210324:054904.504 End of update_cpustats()

  2564:20210324:054904.504 zbx_setproctitle() title:'collector [idle 1 sec]'

  2564:20210324:054905.504 zbx_setproctitle() title:'collector [processing data]'

  2564:20210324:054905.505 In update_cpustats()

  2564:20210324:054905.505 End of update_cpustats()

  2564:20210324:054905.505 zbx_setproctitle() title:'collector [idle 1 sec]'

  2564:20210324:054906.505 zbx_setproctitle() title:'collector [processing data]'

  2564:20210324:054906.505 In update_cpustats()

  2564:20210324:054906.505 End of update_cpustats()

  2564:20210324:054906.505 zbx_setproctitle() title:'collector [idle 1 sec]'

  2564:20210324:054907.505 zbx_setproctitle() title:'collector [processing data]'

  2564:20210324:054907.505 In update_cpustats()

  2564:20210324:054907.505 End of update_cpustats()

  2564:20210324:054907.505 zbx_setproctitle() title:'collector [idle 1 sec]'

  2564:20210324:054908.505 zbx_setproctitle() title:'collector [processing data]'

  2564:20210324:054908.505 In update_cpustats()

  2564:20210324:054908.505 End of update_cpustats()

  2564:20210324:054908.506 zbx_setproctitle() title:'collector [idle 1 sec]'

  2564:20210324:054909.506 zbx_setproctitle() title:'collector [processing data]'

  2564:20210324:054909.506 In update_cpustats()

  2564:20210324:054909.506 End of update_cpustats()

  2564:20210324:054909.506 zbx_setproctitle() title:'collector [idle 1 sec]'

  2564:20210324:054910.506 zbx_setproctitle() title:'collector [processing data]'

  2564:20210324:054910.506 In update_cpustats()

  2564:20210324:054910.506 End of update_cpustats()

  2564:20210324:054910.506 zbx_setproctitle() title:'collector [idle 1 sec]'

  2564:20210324:054911.506 zbx_setproctitle() title:'collector [processing data]'

  2564:20210324:054911.506 In update_cpustats()

  2564:20210324:054911.506 End of update_cpustats()

  2564:20210324:054911.506 zbx_setproctitle() title:'collector [idle 1 sec]'

  2564:20210324:054912.507 zbx_setproctitle() title:'collector [processing data]'

  2564:20210324:054912.507 In update_cpustats()

  2564:20210324:054912.507 End of update_cpustats()

  2564:20210324:054912.507 zbx_setproctitle() title:'collector [idle 1 sec]'

  2566:20210324:054912.894 Requested [agent.ping]

  2566:20210324:054912.894 Sending back [1]

  2566:20210324:054912.894 zbx_setproctitle() title:'listener #2 [waiting for connection]'

  2571:20210324:054912.912 zbx_setproctitle() title:'listener #7 [processing request]'

  2564:20210324:054913.507 zbx_setproctitle() title:'collector [processing data]'

  2564:20210324:054913.507 In update_cpustats()

  2564:20210324:054913.507 End of update_cpustats()

  2564:20210324:054913.507 zbx_setproctitle() title:'collector [idle 1 sec]'

  2564:20210324:054914.507 zbx_setproctitle() title:'collector [processing data]'

  2564:20210324:054914.507 In update_cpustats()

  2564:20210324:054914.507 End of update_cpustats()

  2564:20210324:054914.507 zbx_setproctitle() title:'collector [idle 1 sec]'

  2564:20210324:054915.507 zbx_setproctitle() title:'collector [processing data]'

  2564:20210324:054915.507 In update_cpustats()

  2564:20210324:054915.508 End of update_cpustats()

  2564:20210324:054915.508 zbx_setproctitle() title:'collector [idle 1 sec]'

  2564:20210324:054916.508 zbx_setproctitle() title:'collector [processing data]'

  2564:20210324:054916.508 In update_cpustats()

  2564:20210324:054916.508 End of update_cpustats()

  2564:20210324:054916.508 zbx_setproctitle() title:'collector [idle 1 sec]'

  2564:20210324:054917.508 zbx_setproctitle() title:'collector [processing data]'

  2564:20210324:054917.508 In update_cpustats()

  2564:20210324:054917.508 End of update_cpustats()

  2564:20210324:054917.508 zbx_setproctitle() title:'collector [idle 1 sec]'

  2574:20210324:054917.615 Got signal [signal:11(SIGSEGV),reason:1,refaddr:(nil)]. Crashing ...

  2574:20210324:054917.615 ====== Fatal information: ======

  2574:20210324:054917.615 Program counter: (nil)

  2574:20210324:054917.615 === Registers: ===

Comment by Damian Cancela [ 2021 Mar 24 ]

Another one here

 

32559:20210323:125603.968 In send_buffer() host:'zabbix-proxy-saas-a-eu-west-1.as.cert.ascint.sabrecirrus.com' port:10051 entries:0/100

32559:20210323:125603.968 End of send_buffer():SUCCEED

32559:20210323:125603.968 zbx_setproctitle() title:'active checks #1 [idle 1 sec]'

32549:20210323:125604.760 zbx_setproctitle() title:'collector [processing data]'

32549:20210323:125604.760 In update_cpustats()

32549:20210323:125604.760 End of update_cpustats()

32549:20210323:125604.760 zbx_setproctitle() title:'collector [idle 1 sec]'

32559:20210323:125604.968 In send_buffer() host:'zabbix-proxy-saas-a-eu-west-1.as.cert.ascint.sabrecirrus.com' port:10051 entries:0/100

32559:20210323:125604.968 End of send_buffer():SUCCEED

32559:20210323:125604.968 zbx_setproctitle() title:'active checks #1 [idle 1 sec]'

32556:20210323:125605.431 Got signal [signal:11(SIGSEGV),reason:1,refaddr:(nil)]. Crashing ...

32556:20210323:125605.431 ====== Fatal information: ======

32556:20210323:125605.431 Program counter: (nil)

32556:20210323:125605.431 === Registers: ===

32556:20210323:125605.431 r8      =                0 =                    0 =                    0

32556:20210323:125605.432 r9      =     7ffed1361be0 =      140732408404960 =      140732408404960

32556:20210323:125605.432 r10     =                8 =                    8 =                    8

32556:20210323:125605.432 r11     =              246 =                  582 =                  582

32556:20210323:125605.432 r12     =     7f919326c908 =      140263215778056 =      140263215778056

32556:20210323:125605.432 r13     =          1a52e30 =             27602480 =             27602480

32556:20210323:125605.432 r14     =                0 =                    0 =                    0

32556:20210323:125605.432 r15     =          1a58c00 =             27626496 =             27626496

32556:20210323:125605.432 rdi     =                3 =                    3 =                    3

32556:20210323:125605.432 rsi     =     7ffed1361ca0 =      140732408405152 =      140732408405152

32556:20210323:125605.432 rbp     =     7f919326c000 =      140263215775744 =      140263215775744

32556:20210323:125605.432 rbx     =          1a51828 =             27596840 =             27596840

32556:20210323:125605.432 rdx     =           800000 =              8388608 =              8388608

32556:20210323:125605.432 rax     =          1a51828 =             27596840 =             27596840

32556:20210323:125605.432 rcx     = fffffffffffff000 = 18446744073709547520 =                -4096

32556:20210323:125605.432 rsp     =     7ffed1361c88 =      140732408405128 =      140732408405128

32556:20210323:125605.432 rip     =                0 =                    0 =                    0

32556:20210323:125605.432 efl     =            10207 =                66055 =                66055

32556:20210323:125605.432 csgsfs  =               33 =                   51 =                   51

32556:20210323:125605.432 err     =               14 =                   20 =                   20

32556:20210323:125605.432 trapno  =                e =                   14 =                   14

32556:20210323:125605.432 oldmask =                0 =                    0 =                    0

32556:20210323:125605.432 cr2     =                0 =                    0 =                    0

32556:20210323:125605.432 === Backtrace: ===

32556:20210323:125605.433 3: [0x427f85]

32556:20210323:125605.433 2: [0x42829d]

32556:20210323:125605.433 1: [0x428896]

32556:20210323:125605.433 0: [0x48d180]

32556:20210323:125605.433 === Memory map: ===

32556:20210323:125605.433 00400000-005aa000 r-xp 00000000 103:03 100671243                         /opt/zabbix/5.0.7/sbin/zabbix_agentd

32556:20210323:125605.433 007a9000-007ad000 rw-p 001a9000 103:03 100671243                         /opt/zabbix/5.0.7/sbin/zabbix_agentd

32556:20210323:125605.433 007ad000-007ba000 rw-p 00000000 00:00 0

32556:20210323:125605.433 01a51000-01a74000 rw-p 00000000 00:00 0                                  [heap]

32556:20210323:125605.433 7f9190f14000-7f9190f15000 rw-p 00000000 00:00 0

32556:20210323:125605.433 7f9190f15000-7f9190f24000 r-xp 00000000 103:03 114371                    /usr/lib64/libbz2.so.1.0.6

32556:20210323:125605.433 7f9190f24000-7f9191123000 ---p 0000f000 103:03 114371                    /usr/lib64/libbz2.so.1.0.6

32556:20210323:125605.433 7f9191123000-7f9191124000 r--p 0000e000 103:03 114371                    /usr/lib64/libbz2.so.1.0.6

32556:20210323:125605.433 7f9191124000-7f9191125000 rw-p 0000f000 103:03 114371                    /usr/lib64/libbz2.so.1.0.6

32556:20210323:125605.433 7f9191125000-7f919114a000 r-xp 00000000 103:03 114254                    /usr/lib64/liblzma.so.5.2.2

32556:20210323:125605.433 7f919114a000-7f9191349000 ---p 00025000 103:03 114254                    /usr/lib64/liblzma.so.5.2.2

32556:20210323:125605.433 7f9191349000-7f919134a000 r--p 00024000 103:03 114254                    /usr/lib64/liblzma.so.5.2.2

32556:20210323:125605.433 7f919134a000-7f919134b000 rw-p 00025000 103:03 114254                    /usr/lib64/liblzma.so.5.2.2

32556:20210323:125605.433 7f919134b000-7f9191360000 r-xp 00000000 103:03 114246                    /usr/lib64/libz.so.1.2.7

32556:20210323:125605.433 7f9191360000-7f919155f000 ---p 00015000 103:03 114246                    /usr/lib64/libz.so.1.2.7

32556:20210323:125605.433 7f919155f000-7f9191560000 r--p 00014000 103:03 114246                    /usr/lib64/libz.so.1.2.7

32556:20210323:125605.433 7f9191560000-7f9191561000 rw-p 00015000 103:03 114246                    /usr/lib64/libz.so.1.2.7

32556:20210323:125605.433 7f9191561000-7f9191578000 r-xp 00000000 103:03 114221                    /usr/lib64/libelf-0.176.so

32556:20210323:125605.433 7f9191578000-7f9191777000 ---p 00017000 103:03 114221                    /usr/lib64/libelf-0.176.so

32556:20210323:125605.433 7f9191777000-7f9191778000 r--p 00016000 103:03 114221                    /usr/lib64/libelf-0.176.so

32556:20210323:125605.433 7f9191778000-7f9191779000 rw-p 00017000 103:03 114221                    /usr/lib64/libelf-0.176.so

32556:20210323:125605.433 7f9191779000-7f919177d000 r-xp 00000000 103:03 114470                    /usr/lib64/libattr.so.1.1.0

32556:20210323:125605.433 7f919177d000-7f919197c000 ---p 00004000 103:03 114470                    /usr/lib64/libattr.so.1.1.0

32556:20210323:125605.433 7f919197c000-7f919197d000 r--p 00003000 103:03 114470                    /usr/lib64/libattr.so.1.1.0

32556:20210323:125605.433 7f919197d000-7f919197e000 rw-p 00004000 103:03 114470                    /usr/lib64/libattr.so.1.1.0

32556:20210323:125605.434 7f919197e000-7f9191995000 r-xp 00000000 103:03 108518                    /usr/lib64/libpthread-2.17.so

32556:20210323:125605.434 7f9191995000-7f9191b94000 ---p 00017000 103:03 108518                    /usr/lib64/libpthread-2.17.so

32556:20210323:125605.434 7f9191b94000-7f9191b95000 r--p 00016000 103:03 108518                    /usr/lib64/libpthread-2.17.so

32556:20210323:125605.434 7f9191b95000-7f9191b96000 rw-p 00017000 103:03 108518                    /usr/lib64/libpthread-2.17.so

32556:20210323:125605.434 7f9191b96000-7f9191b9a000 rw-p 00000000 00:00 0

32556:20210323:125605.434 7f9191b9a000-7f9191baf000 r-xp 00000000 103:03 115                       /usr/lib64/libgcc_s-4.8.5-20150702.so.1

32556:20210323:125605.434 7f9191baf000-7f9191dae000 ---p 00015000 103:03 115                       /usr/lib64/libgcc_s-4.8.5-20150702.so.1

32556:20210323:125605.434 7f9191dae000-7f9191daf000 r--p 00014000 103:03 115                       /usr/lib64/libgcc_s-4.8.5-20150702.so.1

32556:20210323:125605.434 7f9191daf000-7f9191db0000 rw-p 00015000 103:03 115                       /usr/lib64/libgcc_s-4.8.5-20150702.so.1

32556:20210323:125605.434 7f9191db0000-7f9191db2000 r-xp 00000000 103:03 28970                     /usr/lib64/libdl-2.17.so

32556:20210323:125605.434 7f9191db2000-7f9191fb2000 ---p 00002000 103:03 28970                     /usr/lib64/libdl-2.17.so

32556:20210323:125605.434 7f9191fb2000-7f9191fb3000 r--p 00002000 103:03 28970                     /usr/lib64/libdl-2.17.so

32556:20210323:125605.434 7f9191fb3000-7f9191fb4000 rw-p 00003000 103:03 28970                     /usr/lib64/libdl-2.17.so

32556:20210323:125605.434 7f9191fb4000-7f9192002000 r-xp 00000000 103:03 504043                    /usr/lib64/libdw-0.176.so

32556:20210323:125605.434 7f9192002000-7f9192202000 ---p 0004e000 103:03 504043                    /usr/lib64/libdw-0.176.so

32556:20210323:125605.434 7f9192202000-7f9192204000 r--p 0004e000 103:03 504043                    /usr/lib64/libdw-0.176.so

32556:20210323:125605.434 7f9192204000-7f9192205000 rw-p 00050000 103:03 504043                    /usr/lib64/libdw-0.176.so

32556:20210323:125605.434 7f9192205000-7f919220c000 r-xp 00000000 103:03 108524                    /usr/lib64/librt-2.17.so

32556:20210323:125605.434 7f919220c000-7f919240b000 ---p 00007000 103:03 108524                    /usr/lib64/librt-2.17.so

32556:20210323:125605.434 7f919240b000-7f919240c000 r--p 00006000 103:03 108524                    /usr/lib64/librt-2.17.so

32556:20210323:125605.434 7f919240c000-7f919240d000 rw-p 00007000 103:03 108524                    /usr/lib64/librt-2.17.so

32556:20210323:125605.434 7f919240d000-7f919250e000 r-xp 00000000 103:03 28972                     /usr/lib64/libm-2.17.so

32556:20210323:125605.434 7f919250e000-7f919270d000 ---p 00101000 103:03 28972                     /usr/lib64/libm-2.17.so

32556:20210323:125605.434 7f919270d000-7f919270e000 r--p 00100000 103:03 28972                     /usr/lib64/libm-2.17.so

32556:20210323:125605.434 7f919270e000-7f919270f000 rw-p 00101000 103:03 28972                     /usr/lib64/libm-2.17.so

32556:20210323:125605.434 7f919270f000-7f9192713000 r-xp 00000000 103:03 108546                    /usr/lib64/libcap.so.2.22

32556:20210323:125605.434 7f9192713000-7f9192912000 ---p 00004000 103:03 108546                    /usr/lib64/libcap.so.2.22

32556:20210323:125605.434 7f9192912000-7f9192913000 r--p 00003000 103:03 108546                    /usr/lib64/libcap.so.2.22

32556:20210323:125605.434 7f9192913000-7f9192914000 rw-p 00004000 103:03 108546                    /usr/lib64/libcap.so.2.22

32556:20210323:125605.434 7f9192914000-7f9192926000 r-xp 00000000 103:03 503999                    /usr/lib64/libnss_myhostname.so.2

32556:20210323:125605.434 7f9192926000-7f9192b25000 ---p 00012000 103:03 503999                    /usr/lib64/libnss_myhostname.so.2

32556:20210323:125605.434 7f9192b25000-7f9192b28000 r--p 00011000 103:03 503999                    /usr/lib64/libnss_myhostname.so.2

32556:20210323:125605.434 7f9192b28000-7f9192b29000 rw-p 00014000 103:03 503999                    /usr/lib64/libnss_myhostname.so.2

32556:20210323:125605.434 7f9192b29000-7f9192b3f000 r-xp 00000000 103:03 108520                    /usr/lib64/libresolv-2.17.so

32556:20210323:125605.434 7f9192b3f000-7f9192d3f000 ---p 00016000 103:03 108520                    /usr/lib64/libresolv-2.17.so

32556:20210323:125605.434 7f9192d3f000-7f9192d40000 r--p 00016000 103:03 108520                    /usr/lib64/libresolv-2.17.so

32556:20210323:125605.434 7f9192d40000-7f9192d41000 rw-p 00017000 103:03 108520                    /usr/lib64/libresolv-2.17.so

32556:20210323:125605.434 7f9192d41000-7f9192d43000 rw-p 00000000 00:00 0

32556:20210323:125605.434 7f9192d43000-7f9192d49000 r-xp 00000000 103:03 28983                     /usr/lib64/libnss_dns-2.17.so

32556:20210323:125605.434 7f9192d49000-7f9192f48000 ---p 00006000 103:03 28983                     /usr/lib64/libnss_dns-2.17.so

32556:20210323:125605.434 7f9192f48000-7f9192f49000 r--p 00005000 103:03 28983                     /usr/lib64/libnss_dns-2.17.so

32556:20210323:125605.434 7f9192f49000-7f9192f4a000 rw-p 00006000 103:03 28983                     /usr/lib64/libnss_dns-2.17.so

32556:20210323:125605.434 7f9192f4a000-7f919304a000 rw-p 00000000 00:00 0

32556:20210323:125605.434 7f919304a000-7f919306c000 r-xp 00000000 103:03 28957                     /usr/lib64/ld-2.17.so

32556:20210323:125605.434 7f919306c000-7f919326b000 ---p 00022000 103:03 28957                     /usr/lib64/ld-2.17.so

32556:20210323:125605.434 7f919326b000-7f919326c000 r--p 00021000 103:03 28957                     /usr/lib64/ld-2.17.so

32556:20210323:125605.434 7f919326c000-7f919326d000 rw-p 00022000 103:03 28957                     /usr/lib64/ld-2.17.so

32556:20210323:125605.434 7f919326d000-7f919326e000 rw-p 00000000 00:00 0

32556:20210323:125605.434 7f919326e000-7f9193431000 r-xp 00000000 103:03 28964                     /usr/lib64/libc-2.17.so

32556:20210323:125605.434 7f9193431000-7f9193631000 ---p 001c3000 103:03 28964                     /usr/lib64/libc-2.17.so

32556:20210323:125605.434 7f9193631000-7f9193635000 r--p 001c3000 103:03 28964                     /usr/lib64/libc-2.17.so

32556:20210323:125605.434 7f9193635000-7f9193637000 rw-p 001c7000 103:03 28964                     /usr/lib64/libc-2.17.so

32556:20210323:125605.434 7f9193637000-7f919363c000 rw-p 00000000 00:00 0

32556:20210323:125605.434 7f919363c000-7f9193648000 r-xp 00000000 103:03 28986                     /usr/lib64/libnss_files-2.17.so

32556:20210323:125605.434 7f9193648000-7f9193847000 ---p 0000c000 103:03 28986                     /usr/lib64/libnss_files-2.17.so

32556:20210323:125605.434 7f9193847000-7f9193848000 r--p 0000b000 103:03 28986                     /usr/lib64/libnss_files-2.17.so

32556:20210323:125605.435 7f9193848000-7f9193849000 rw-p 0000c000 103:03 28986                     /usr/lib64/libnss_files-2.17.so

32556:20210323:125605.435 7f9193849000-7f919384f000 rw-p 00000000 00:00 0

32556:20210323:125605.435 7f919384f000-7f919385f000 r--p 00000000 103:03 2794942                   /etc/ld.so.cache

32556:20210323:125605.435 7f919385f000-7f91938b9000 rw-s 00000000 00:04 1489698868                 /SYSV00000000 (deleted)

32556:20210323:125605.435 7f91938b9000-7f91938ba000 rw-p 00000000 00:00 0

32556:20210323:125605.435 7ffed1344000-7ffed1365000 rw-p 00000000 00:00 0                          [stack]

32556:20210323:125605.435 7ffed1376000-7ffed1378000 r-xp 00000000 00:00 0                          [vdso]

32556:20210323:125605.435 ffffffffff600000-ffffffffff601000 r-xp 00000000 00:00 0                  [vsyscall]

32556:20210323:125605.435 ================================

32556:20210323:125605.435 Please consider attaching a disassembly listing to your bug report.

32556:20210323:125605.435 This listing can be produced with, e.g., objdump -DSswx zabbix_agentd.

32556:20210323:125605.435 ================================

32546:20210323:125605.435 One child process died (PID:32556,exitcode/signal:1). Exiting ...

32546:20210323:125605.435 zbx_on_exit() called

zabbix_agentd [32546]: Error waiting for process with PID 32556: [10] No child processes

32546:20210323:125605.435 In zbx_dshm_destroy() shmid:-1

32546:20210323:125605.435 End of zbx_dshm_destroy():SUCCEED

32546:20210323:125605.436 In zbx_unload_modules()

32546:20210323:125605.436 End of zbx_unload_modules()

32546:20210323:125605.436 Zabbix Agent stopped. Zabbix 5.0.7 (revision 1b3db7f980).

  8161:20210323:130002.100 Starting Zabbix Agent [ip-10-181-1-63.eu-west-1.compute.internal]. Zabbix 5.0.7 (revision 1b3db7f980).

  8161:20210323:130002.101 **** Enabled features ****

  8161:20210323:130002.101 IPv6 support:           NO

  8161:20210323:130002.101 TLS support:            NO

  8161:20210323:130002.101 **************************

  8161:20210323:130002.101 using configuration file: /opt/zabbix/stable/etc/zabbix_agentd.conf

  8161:20210323:130002.101 In zbx_load_modules()

  8161:20210323:130002.101 End of zbx_load_modules():SUCCEED

  8161:20210323:130002.101 In init_collector_data()

  8161:20210323:130002.101 In zbx_dshm_create() size:0

  8161:20210323:130002.101 End of zbx_dshm_create():SUCCEED shmid:-1

  8161:20210323:130002.101 End of init_collector_data()

  8161:20210323:130002.101 agent #0 started [main process]

  8166:20210323:130002.106 agent #3 started listener #2

  8166:20210323:130002.106 zbx_setproctitle() title:'listener #2 [waiting for connection]'

Comment by Andrejs Kozlovs [ 2021 Mar 25 ]

Hello, damiancancela Is it possible to get some additional info:

  1. objdump -Swx zabbix_agentd
  2. OS version
  3. Details of scenario - what is the difference between instances where crash happens and where not
  4. Is it installed from sources, package, container, etc?
Comment by Andrejs Kozlovs [ 2021 Mar 29 ]

Hello, damiancancela can you reproduce issue with more new version of Zabbix (5.0.9)?

Comment by Damian Cancela [ 2021 Mar 29 ]
  1. objdump -Swx zabbix_agentd (i'll check)
  2. OS version (check at the end)
  3. Details of scenario - what is the difference between instances where crash happens and where not ( I don't know, nothing strange, i tried to get that info from logs with debug level 5, but as you can see is not showing anything out of normal)
  4. Is it installed from sources, package, container, etc? (sources)

 

OS version

-sh-4.2$ uname -a
3.10.0-1127.el7.x86_64 #1 SMP Tue Mar 31 23:36:51 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux
-sh-4.2$ cat /etc/*-release
CentOS Linux release 7.8.2003 (Core)
NAME="CentOS Linux"
VERSION="7 (Core)"
ID="centos"
ID_LIKE="rhel fedora"
VERSION_ID="7"
PRETTY_NAME="CentOS Linux 7 (Core)"
ANSI_COLOR="0;31"
CPE_NAME="cpe:/o:centos:centos:7"
HOME_URL="https://www.centos.org/"
BUG_REPORT_URL="https://bugs.centos.org/"

CENTOS_MANTISBT_PROJECT="CentOS-7"
CENTOS_MANTISBT_PROJECT_VERSION="7"
REDHAT_SUPPORT_PRODUCT="centos"
REDHAT_SUPPORT_PRODUCT_VERSION="7"

CentOS Linux release 7.8.2003 (Core)
CentOS Linux release 7.8.2003 (Core)

 

Hope this helps

 

 

 

Comment by Damian Cancela [ 2021 Mar 29 ]

Regarding question about 5.0.9... current version of server and proxy is 5.0.7.

So silly question, i should upgrade first proxy and server into 5.0.9 right?

 

thanks

Comment by Damian Cancela [ 2021 Mar 29 ]

FYI,

zabbix proxy and zabbix were installed with RPM's

But the zabbix agent has been created from sources.

Comment by Damian Cancela [ 2021 Mar 29 ]

Attached objdump export

 

objdump.zip

Comment by Vladislavs Sokurenko [ 2021 Mar 30 ]

Unfortunately we could not reproduce the issue, could you please apply the patch ZBX-19131-1.diff and provide log ? Also information about how agent was configured prior to build is very important and if it was built/used on same system ?

Comment by Damian Cancela [ 2021 Mar 30 ]

How can i apply that patch?

 

One clarification, sorry we didn't use sources to create the agent, was downloaded from binaries site: 

https://www.zabbix.com/download_agents?version=5.0+LTS&release=5.0.7&os=Linux&os_version=3.0&hardware=amd64&encryption=No+encryption&packaging=Archive

 

Thanks

Comment by Vladislavs Sokurenko [ 2021 Mar 30 ]

Please try downloading from here:
https://repo.zabbix.com/zabbix/5.0/rhel/7/x86_64/

Comment by Damian Cancela [ 2021 Mar 30 ]

im sorry but for us use rpm's for clients is not an option, should not be the same with binaries/package?

 

Thanks

Comment by Vladislavs Sokurenko [ 2021 Mar 30 ]

Could try building from sources in that case or trying to extract binary from package and see if it works for test purposes, meanwhile we can double check the binary that you downloaded, thanks you for information.

Comment by Damian Cancela [ 2021 Mar 31 ]

We've downloaded binaries one more time, same result

Comment by Andrejs Kozlovs [ 2021 Mar 31 ]

You got this with 5.0.9 or still 5.0.7? Did you have a chance to build Agent from sources for your system?

Comment by Damian Cancela [ 2021 Mar 31 ]

No, i didn't use the sources to build the agent. I just use again the binaries/package.

Im still in 5.0.7, i should upgrade?

 

Because there is a security bug in 5.0.9

Zabbix CSRF security tests identified CVE-2021-27927 potential vulnerability in Zabbix versions 4.0.28rc1 - 4.0.29, 5.0.8rc1-5.0.9, 5.2.4rc1-5.2.5, 5.4.0alpha1-5.4.0beta1. There is a threat of configuration data corruption or Zabbix user access override in the affected versions.

An attacker doesn't have to know Zabbix user login credentials but has to know the correct Zabbix URL and contact information of an existing user with sufficient privileges. Therefore the possibility of the attack from the public domain is limited.

Comment by Andrejs Kozlovs [ 2021 Mar 31 ]

Please try to build agent 5.0.7 from sources for test to resolve doubts that binaries does not match your system. Please build it exactly on your system.
Also is it possible to get list of your checks (agent items) on crashed system?

Comment by Vladislavs Sokurenko [ 2021 Apr 13 ]

Reproduced similar issue on Fedora 33 with zabbix_agent-5.2.6-linux-3.0-amd64-static
Steps:

  1. Download latest binary agent and add "ServerActive=localhost:10051" to config file
NAME=Fedora
VERSION="33 (Workstation Edition)"
ID=fedora
VERSION_ID=33
VERSION_CODENAME=""
PLATFORM_ID="platform:f33"
PRETTY_NAME="Fedora 33 (Workstation Edition)"
ANSI_COLOR="0;38;2;60;110;180"
LOGO=fedora-logo-icon
CPE_NAME="cpe:/o:fedoraproject:fedora:33"
HOME_URL="https://fedoraproject.org/"
DOCUMENTATION_URL="https://docs.fedoraproject.org/en-US/fedora/f33/system-administrators-guide/"
SUPPORT_URL="https://fedoraproject.org/wiki/Communicating_and_getting_help"
BUG_REPORT_URL="https://bugzilla.redhat.com/"
REDHAT_BUGZILLA_PRODUCT="Fedora"
REDHAT_BUGZILLA_PRODUCT_VERSION=33
REDHAT_SUPPORT_PRODUCT="Fedora"
REDHAT_SUPPORT_PRODUCT_VERSION=33
PRIVACY_POLICY_URL="https://fedoraproject.org/wiki/Legal:PrivacyPolicy"
VARIANT="Workstation Edition"
VARIANT_ID=workstation

ldd --version
ldd (GNU libc) 2.32
Copyright (C) 2020 Free Software Foundation, Inc.
This is free software; see the source for copying conditions.  There is NO
warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.
Written by Roland McGrath and Ulrich Drepper.

Observe

Starting Zabbix Agent [Zabbix server]. Zabbix 5.2.6 (revision 798506596c).
Press Ctrl+C to exit.

 69584:20210413:112617.248 Starting Zabbix Agent [Zabbix server]. Zabbix 5.2.6 (revision 798506596c).
 69584:20210413:112617.248 **** Enabled features ****
 69584:20210413:112617.248 IPv6 support:           NO
 69584:20210413:112617.248 TLS support:            NO
 69584:20210413:112617.248 **************************
 69584:20210413:112617.248 using configuration file: /etc/zabbix/zabbix_agentd.conf
 69584:20210413:112617.248 In zbx_load_modules()
 69584:20210413:112617.248 End of zbx_load_modules():SUCCEED
 69584:20210413:112617.248 In init_collector_data()
 69584:20210413:112617.248 In zbx_dshm_create() size:0
 69584:20210413:112617.248 End of zbx_dshm_create():SUCCEED shmid:-1
 69584:20210413:112617.248 End of init_collector_data()
 69584:20210413:112617.248 agent #0 started [main process]
 69585:20210413:112617.248 agent #1 started [collector]
 69586:20210413:112617.248 agent #2 started [listener #1]
 69585:20210413:112617.248 In init_cpu_collector()
 69587:20210413:112617.248 agent #3 started [listener #2]
 69586:20210413:112617.248 zbx_setproctitle() title:'listener #1 [waiting for connection]'
 69588:20210413:112617.249 agent #4 started [listener #3]
 69589:20210413:112617.249 agent #5 started [active checks #1]
 69585:20210413:112617.249 End of init_cpu_collector():SUCCEED
 69587:20210413:112617.249 zbx_setproctitle() title:'listener #2 [waiting for connection]'
 69588:20210413:112617.249 zbx_setproctitle() title:'listener #3 [waiting for connection]'
 69589:20210413:112617.249 In init_active_metrics()
 69585:20210413:112617.249 zbx_setproctitle() title:'collector [processing data]'
 69589:20210413:112617.249 buffer: first allocation for 100 elements
 69585:20210413:112617.249 In update_cpustats()
 69589:20210413:112617.249 End of init_active_metrics()
 69589:20210413:112617.249 In send_buffer() host:'localhost' port:10051 entries:0/100
 69589:20210413:112617.249 End of send_buffer():SUCCEED
 69589:20210413:112617.249 zbx_setproctitle() title:'active checks #1 [getting list of active checks]'
 69589:20210413:112617.249 In refresh_active_checks() host:'localhost' port:10051
 69585:20210413:112617.250 End of update_cpustats()
 69585:20210413:112617.250 zbx_setproctitle() title:'collector [idle 1 sec]'
 69589:20210413:112617.250 Got signal [signal:11(SIGSEGV),reason:1,refaddr:0x63]. Crashing ...
 69589:20210413:112617.250 ====== Fatal information: ======
 69589:20210413:112617.250 Program counter: 0x7f7ba73dacb4
 69589:20210413:112617.250 === Registers: ===
 69589:20210413:112617.250 r8      =     7fffa4715990 =      140735952279952 =      140735952279952
 69589:20210413:112617.250 r9      =                0 =                    0 =                    0
 69589:20210413:112617.250 r10     =               70 =                  112 =                  112
 69589:20210413:112617.250 r11     =              246 =                  582 =                  582
 69589:20210413:112617.250 r12     =     7fffa4715d6f =      140735952280943 =      140735952280943
 69589:20210413:112617.250 r13     =              3e0 =                  992 =                  992
 69589:20210413:112617.250 r14     =     7fffa4715990 =      140735952279952 =      140735952279952
 69589:20210413:112617.250 r15     =     7f7ba71862a0 =      140169061098144 =      140169061098144
 69589:20210413:112617.250 rdi     =     7fffa4715991 =      140735952279953 =      140735952279953
 69589:20210413:112617.250 rsi     =     7f7ba7186481 =      140169061098625 =      140169061098625
 69589:20210413:112617.250 rbp     =     7fffa4715990 =      140735952279952 =      140735952279952
 69589:20210413:112617.250 rbx     =     7fffa4715430 =      140735952278576 =      140735952278576
 69589:20210413:112617.250 rdx     =               31 =                   49 =                   49
 69589:20210413:112617.250 rax     =               31 =                   49 =                   49
 69589:20210413:112617.250 rcx     =                0 =                    0 =                    0
 69589:20210413:112617.250 rsp     =     7fffa47153b0 =      140735952278448 =      140735952278448
 69589:20210413:112617.250 rip     =     7f7ba73dacb4 =      140169063541940 =      140169063541940
 69589:20210413:112617.250 efl     =            10246 =                66118 =                66118
 69589:20210413:112617.250 csgsfs  =   2b000000000033 =    12103423998558259 =    12103423998558259
 69589:20210413:112617.250 err     =                4 =                    4 =                    4
 69589:20210413:112617.250 trapno  =                e =                   14 =                   14
 69589:20210413:112617.250 oldmask =                0 =                    0 =                    0
 69589:20210413:112617.250 cr2     =               63 =                   99 =                   99
 69589:20210413:112617.250 === Backtrace: ===
 69589:20210413:112617.252 20: [0x4283e5]
 69589:20210413:112617.252 19: [0x4286fd]
 69589:20210413:112617.252 18: [0x428d66]
 69589:20210413:112617.252 17: [0x48dc60]
 69589:20210413:112617.252 16: /lib64/libc.so.6(__nss_readline+0x74) [0x7f7ba73dacb4]
 69589:20210413:112617.252 15: /lib64/libnss_files.so.2(+0x44fd) [0x7f7ba74834fd]
 69589:20210413:112617.252 14: /lib64/libnss_files.so.2(_nss_files_gethostbyname3_r+0x87) [0x7f7ba7484537]
 69589:20210413:112617.252 13: /lib64/libnss_files.so.2(_nss_files_gethostbyname2_r+0x15) [0x7f7ba7484665]
 69589:20210413:112617.252 12: [0x50a871]
 69589:20210413:112617.252 11: [0x4fc9d6]
 69589:20210413:112617.252 10: [0x4fd2eb]
 69589:20210413:112617.252 9: [0x429491]
 69589:20210413:112617.252 8: [0x4052f0]
 69589:20210413:112617.252 7: [0x40636e]
 69589:20210413:112617.252 6: [0x426d8e]
 69589:20210413:112617.252 5: [0x4040aa]
 69589:20210413:112617.252 4: [0x427c9c]
 69589:20210413:112617.252 3: [0x402235]
 69589:20210413:112617.252 2: [0x494094]
 69589:20210413:112617.252 1: [0x494311]
 69589:20210413:112617.252 0: [0x4025b6]
 69589:20210413:112617.252 === Memory map: ===
 69589:20210413:112617.252 00400000-005aa000 r-xp 00000000 fd:00 2370658                            /home/vso/Downloads/zabbix_agent-5.2.6-linux-3.0-amd64-static/sbin/zabbix_agentd
 69589:20210413:112617.252 007aa000-007ae000 rw-p 001aa000 fd:00 2370658                            /home/vso/Downloads/zabbix_agent-5.2.6-linux-3.0-amd64-static/sbin/zabbix_agentd
 69589:20210413:112617.252 007ae000-007bb000 rw-p 00000000 00:00 0 
 69589:20210413:112617.252 01879000-0189c000 rw-p 00000000 00:00 0                                  [heap]
 69589:20210413:112617.252 7f7ba7186000-7f7ba7286000 rw-p 00000000 00:00 0 
 69589:20210413:112617.252 7f7ba7286000-7f7ba7287000 r--p 00000000 fd:00 14552364                   /usr/lib64/ld-2.32.so
 69589:20210413:112617.252 7f7ba7287000-7f7ba72a8000 r-xp 00001000 fd:00 14552364                   /usr/lib64/ld-2.32.so
 69589:20210413:112617.252 7f7ba72a8000-7f7ba72b1000 r--p 00022000 fd:00 14552364                   /usr/lib64/ld-2.32.so
 69589:20210413:112617.252 7f7ba72b1000-7f7ba72b2000 r--p 0002a000 fd:00 14552364                   /usr/lib64/ld-2.32.so
 69589:20210413:112617.252 7f7ba72b2000-7f7ba72b4000 rw-p 0002b000 fd:00 14552364                   /usr/lib64/ld-2.32.so
 69589:20210413:112617.252 7f7ba72b4000-7f7ba72da000 r--p 00000000 fd:00 14550029                   /usr/lib64/libc-2.32.so
 69589:20210413:112617.252 7f7ba72da000-7f7ba7429000 r-xp 00026000 fd:00 14550029                   /usr/lib64/libc-2.32.so
 69589:20210413:112617.252 7f7ba7429000-7f7ba7474000 r--p 00175000 fd:00 14550029                   /usr/lib64/libc-2.32.so
 69589:20210413:112617.252 7f7ba7474000-7f7ba7475000 ---p 001c0000 fd:00 14550029                   /usr/lib64/libc-2.32.so
 69589:20210413:112617.252 7f7ba7475000-7f7ba7478000 r--p 001c0000 fd:00 14550029                   /usr/lib64/libc-2.32.so
 69589:20210413:112617.252 7f7ba7478000-7f7ba747b000 rw-p 001c3000 fd:00 14550029                   /usr/lib64/libc-2.32.so
 69589:20210413:112617.253 7f7ba747b000-7f7ba747f000 rw-p 00000000 00:00 0 
 69589:20210413:112617.253 7f7ba747f000-7f7ba7482000 r--p 00000000 fd:00 14552583                   /usr/lib64/libnss_files-2.32.so
 69589:20210413:112617.253 7f7ba7482000-7f7ba7489000 r-xp 00003000 fd:00 14552583                   /usr/lib64/libnss_files-2.32.so
 69589:20210413:112617.253 7f7ba7489000-7f7ba748b000 r--p 0000a000 fd:00 14552583                   /usr/lib64/libnss_files-2.32.so
 69589:20210413:112617.253 7f7ba748b000-7f7ba748c000 r--p 0000b000 fd:00 14552583                   /usr/lib64/libnss_files-2.32.so
 69589:20210413:112617.253 7f7ba748c000-7f7ba7493000 rw-p 00000000 00:00 0 
 69589:20210413:112617.253 7f7ba74b2000-7f7ba74b3000 rw-p 00000000 00:00 0 
 69589:20210413:112617.253 7f7ba74b3000-7f7ba759b000 rw-s 00000000 00:01 131104                     /SYSV00000000 (deleted)
 69589:20210413:112617.253 7f7ba759b000-7f7ba759d000 rw-p 00000000 00:00 0 
 69589:20210413:112617.253 7fffa46fb000-7fffa471d000 rw-p 00000000 00:00 0                          [stack]
 69589:20210413:112617.253 7fffa4760000-7fffa4764000 r--p 00000000 00:00 0                          [vvar]
 69589:20210413:112617.253 7fffa4764000-7fffa4766000 r-xp 00000000 00:00 0                          [vdso]
 69589:20210413:112617.253 ffffffffff600000-ffffffffff601000 r-xp 00000000 00:00 0                  [vsyscall]
 69589:20210413:112617.253 ================================
 69589:20210413:112617.253 Please consider attaching a disassembly listing to your bug report.
 69589:20210413:112617.253 This listing can be produced with, e.g., objdump -DSswx zabbix_agentd.
 69589:20210413:112617.253 ================================
 69584:20210413:112617.253 One child process died (PID:69589,exitcode/signal:1). Exiting ...
 69584:20210413:112617.253 zbx_on_exit() called
zabbix_agentd [69584]: Error waiting for process with PID 69589: [10] No child processes
 69584:20210413:112617.253 In zbx_dshm_destroy() shmid:-1
 69584:20210413:112617.253 End of zbx_dshm_destroy():SUCCEED
 69584:20210413:112617.253 In zbx_unload_modules()
 69584:20210413:112617.253 End of zbx_unload_modules()
 69584:20210413:112617.253 Zabbix Agent stopped. Zabbix 5.2.6 (revision 798506596c).

Note that after compiling from sources the issue is gone.

Same problem with Zabbix agent v5.0.10 and Zabbix agent v5.2.0 amd64

file ./zabbix_agentd 
./zabbix_agentd: ELF 64-bit LSB executable, x86-64, version 1 (GNU/Linux), statically linked, for GNU/Linux 2.6.32, BuildID[sha1]=be1e6187e8ba8f56a790e5fcdbd2e82d9c5a2c30, stripped

ldd ./zabbix_agentd 
	not a dynamic executable
Comment by Damian Cancela [ 2021 Jun 30 ]

I've never update on this, FYI @ak i've tested building the agent from one the affected servers, seems that helped because since then the issue is not showing in any of the affected servers.

Thanks

Generated at Sun May 31 19:47:29 EEST 2026 using Jira 10.3.18#10030018-sha1:5642e4ad348b6c2a83ebdba689d04763a2393cab.