Please look into /etc/default/zabbix-agent for these lines:

# User to run zabbix agent as
ZABBIX_AGENT_USER=zabbix

This is the evidence that the init script can't work properly.

The -c option is passed to start-stop-daemon in the init script.

# start-stop-daemon --help
...
    -c|--chuid <name|uid[:group|gid]> change to this user/group before starting process
...

I've checked. It works fine with default configuration.

root@ubuntu1404:~# service zabbix-agent status
 * zabbix_agentd is running
root@ubuntu1404:~# ps aux | grep zabbix
zabbix    1831  0.0  0.0  92612  2528 ?        S    10:54   0:00 /usr/sbin/zabbix_agentd
zabbix    1832  0.0  0.0  92612  2776 ?        S    10:54   0:00 /usr/sbin/zabbix_agentd: collector [idle 1 sec]
zabbix    1834  0.0  0.1  92612  4184 ?        S    10:54   0:00 /usr/sbin/zabbix_agentd: listener #1 [waiting for connection]
zabbix    1835  0.0  0.1  92612  4184 ?        S    10:54   0:00 /usr/sbin/zabbix_agentd: listener #2 [waiting for connection]
zabbix    1836  0.0  0.1  92612  4184 ?        S    10:54   0:00 /usr/sbin/zabbix_agentd: listener #3 [waiting for connection]
zabbix    1837  0.0  0.1  92612  4944 ?        S    10:54   0:00 /usr/sbin/zabbix_agentd: active checks #1 [idle 1 sec]
root      1850  0.0  0.0  11764  2128 pts/0    S+   10:55   0:00 grep --color=auto zabbix
root@ubuntu1404:~# service zabbix-agent stop
 * Stopping Zabbix agent zabbix_agentd                                                                                                                                                                                                                         [ OK ] 
root@ubuntu1404:~# service zabbix-agent status
 * zabbix_agentd is not running
root@ubuntu1404:~# ps aux | grep zabbix
root      1875  0.0  0.0  11760  2196 pts/0    S+   10:55   0:00 grep --color=auto zabbix
root@ubuntu1404:~# service zabbix-agent start
 * Starting Zabbix agent zabbix_agentd                                                                                                                                                                                                                         [ OK ] 
root@ubuntu1404:~# service zabbix-agent status
 * zabbix_agentd is running
root@ubuntu1404:~# ps aux | grep zabbix
zabbix    1890  0.0  0.0  92612  2500 ?        S    10:55   0:00 /usr/sbin/zabbix_agentd
zabbix    1892  0.0  0.0  92612  2712 ?        S    10:55   0:00 /usr/sbin/zabbix_agentd: collector [idle 1 sec]
zabbix    1893  0.0  0.1  92612  4320 ?        S    10:55   0:00 /usr/sbin/zabbix_agentd: listener #1 [waiting for connection]
zabbix    1894  0.0  0.1  92612  4320 ?        S    10:55   0:00 /usr/sbin/zabbix_agentd: listener #2 [waiting for connection]
zabbix    1895  0.0  0.1  92612  4320 ?        S    10:55   0:00 /usr/sbin/zabbix_agentd: listener #3 [waiting for connection]
zabbix    1896  0.0  0.1  92612  4924 ?        S    10:55   0:00 /usr/sbin/zabbix_agentd: active checks #1 [idle 1 sec]
root@ubuntu1404:~# tail -f /var/log/zabbix/zabbix_agentd.log 
  1890:20200213:105549.502 TLS support:           YES
  1890:20200213:105549.502 **************************
  1890:20200213:105549.502 using configuration file: /etc/zabbix/zabbix_agentd.conf
  1890:20200213:105549.503 agent #0 started [main process]
  1892:20200213:105549.504 agent #1 started [collector]
  1893:20200213:105549.505 agent #2 started [listener #1]
  1894:20200213:105549.506 agent #3 started [listener #2]
  1895:20200213:105549.506 agent #4 started [listener #3]
  1896:20200213:105549.509 agent #5 started [active checks #1]
  1896:20200213:105549.516 active check configuration update from [127.0.0.1:10051] started to fail (cannot connect to [[127.0.0.1]:10051]: [111] Connection refused)

The problem must be with something else.

What alterations did you make in /etc/zabbix/zabbix_agentd.conf?
Did you alter the User setting?

We use this specific configuration (User is not altered):

PidFile=/var/run/zabbix/zabbix_agentd.pid
LogFile=/var/log/zabbix-agent/zabbix_agentd.log
LogFileSize=0
Server=xxx.xxx.xxx.xxx, 2001:xxxx:xxxx::xxxx
ListenPort=10050
Timeout=10
Include=/etc/zabbix/zabbix_agentd.conf.d/

And the files included by Include contain only UserParameter lines (which vary between different servers).

The config listed above works well with 4.0.16 and older versions.

Resolved in 4.0.18, closing the issue.

Today i upgraded to 4.0.18 yet this issue is not resolved. Please reopen the ticket. 

OS : Ubuntu 14.04.5 LTS

I ran init script in debug mode. Here is the output:

root@capl-attrib01:~# sh -x /etc/init.d/zabbix-agent start
+ set -e
+ NAME=zabbix_agentd
+ DAEMON=/usr/sbin/zabbix_agentd
+ DESC=Zabbix agent
+ test -x /usr/sbin/zabbix_agentd
+ DIR=/var/run/zabbix
+ PID=/var/run/zabbix/zabbix_agentd.pid
+ RETRY=15
+ test ! -d /var/run/zabbix
+ export PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/sbin:/sbin
+ . /lib/lsb/init-functions
+ run-parts --lsbsysinit --list /lib/lsb/init-functions.d
+ [ -r /lib/lsb/init-functions.d/01-upstart-lsb ]
+ . /lib/lsb/init-functions.d/01-upstart-lsb
+ unset UPSTART_SESSION
+ _RC_SCRIPT=/etc/init.d/zabbix-agent
+ [ -r /etc/init//etc/init.d/zabbix-agent.conf ]
+ _UPSTART_JOB=zabbix-agent
+ [ -r /etc/init/zabbix-agent.conf ]
+ which initctl
+ initctl version
+ grep -q upstart
+ initctl status zabbix-agent
+ exec start zabbix-agent
zabbix-agent stop/pre-start, process 6264
root@capl-attrib01:~# ps -ef | grep 6264
root 6270 5679 0 03:26 pts/1 00:00:00 grep 6264
root@capl-attrib01:~# 

root@capl-attrib01:~# sh -x /etc/init.d/zabbix-agent status
+ set -e
+ NAME=zabbix_agentd
+ DAEMON=/usr/sbin/zabbix_agentd
+ DESC=Zabbix agent
+ test -x /usr/sbin/zabbix_agentd
+ DIR=/var/run/zabbix
+ PID=/var/run/zabbix/zabbix_agentd.pid
+ RETRY=15
+ test ! -d /var/run/zabbix
+ export PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/sbin:/sbin
+ . /lib/lsb/init-functions
+ run-parts --lsbsysinit --list /lib/lsb/init-functions.d
+ [ -r /lib/lsb/init-functions.d/01-upstart-lsb ]
+ . /lib/lsb/init-functions.d/01-upstart-lsb
+ unset UPSTART_SESSION
+ _RC_SCRIPT=/etc/init.d/zabbix-agent
+ [ -r /etc/init//etc/init.d/zabbix-agent.conf ]
+ _UPSTART_JOB=zabbix-agent
+ [ -r /etc/init/zabbix-agent.conf ]
+ which initctl
+ initctl version
+ grep -q upstart
+ initctl status zabbix-agent
+ status zabbix-agent
+ output=zabbix-agent stop/waiting
+ echo zabbix-agent stop/waiting
zabbix-agent stop/waiting
+ exit 3

looks like it doesn't run start-stop-daemon at all. can someone please help me understand & resolve the issue.

when i manually run start-stop-daemon, It starts. 

root@capl-attrib01:~# start-stop-daemon -c zabbix --oknodo --start --pidfile /var/run/zabbix/zabbix_agentd.pid --exec /usr/sbin/zabbix_agentd >/dev/null 2>&1
root@capl-attrib01:~# ps -ef | grep zabbix
zabbix 6317 1 0 03:32 ? 00:00:00 /usr/sbin/zabbix_agentd
zabbix 6318 6317 0 03:32 ? 00:00:00 /usr/sbin/zabbix_agentd: collector [idle 1 sec]
zabbix 6319 6317 0 03:32 ? 00:00:00 /usr/sbin/zabbix_agentd: listener #1 [waiting for connection]
zabbix 6320 6317 0 03:32 ? 00:00:00 /usr/sbin/zabbix_agentd: listener #2 [waiting for connection]
zabbix 6321 6317 0 03:32 ? 00:00:00 /usr/sbin/zabbix_agentd: listener #3 [waiting for connection]
zabbix 6322 6317 0 03:32 ? 00:00:00 /usr/sbin/zabbix_agentd: active checks #1 [idle 1 sec]
root 6324 5679 0 03:32 pts/1 00:00:00 grep zabbix

You should have the rights to re-open the ticket?

I don't have rights to reopen this ticket.

I'll try to get it some attention.

Re-opened as requested by Deshmukh.

root@capl-attrib01:~# bash -x /usr/sbin/service zabbix-agent status
++ basename /usr/sbin/service
+ VERSION='service ver. 0.91-ubuntu1'
++ basename /usr/sbin/service
+ USAGE='Usage: service < option > | --status-all | [ service_name [ command | --full-restart ] ]'
+ SERVICE=
+ ACTION=
+ SERVICEDIR=/etc/init.d
+ OPTIONS=
+ '[' 2 -eq 0 ']'
+ cd /
+ '[' 2 -gt 0 ']'
+ case "${1}" in
+ '[' -z '' -a 2 -eq 1 -a zabbix-agent = --status-all ']'
+ '[' 2 -eq 2 -a status = --full-restart ']'
+ '[' -z '' ']'
+ SERVICE=zabbix-agent
+ shift
+ '[' 1 -gt 0 ']'
+ case "${1}" in
+ '[' -z zabbix-agent -a 1 -eq 1 -a status = --status-all ']'
+ '[' 1 -eq 2 -a '' = --full-restart ']'
+ '[' -z zabbix-agent ']'
+ '[' -z '' ']'
+ ACTION=status
+ shift
+ '[' 0 -gt 0 ']'
+ '[' -r /etc/init/zabbix-agent.conf ']'
+ which initctl
+ initctl version
+ grep -q upstart
+ case "${ACTION}" in
+ exec status zabbix-agent
zabbix-agent stop/waiting

I moved /etc/init/zabbix-agent.conf to /tmp/ and init script works, so this config file maybe source of the issue.

 Deshmukh, can you please share the output of

dpkg-query -L zabbix-agent

We only provide old style init file in /etc/init.d/zabbix_agent on trusty.

Therefore I'm confused by the /etc/init/zabbix-agent.conf upstart config file on your system.

root@capl-attrib01:~# zabbix_agentd -V
zabbix_agentd (daemon) (Zabbix) 4.0.18
Revision 688cd21456 21 February 2020, compilation time: Feb 21 2020 15:26:38

Copyright (C) 2020 Zabbix SIA
License GPLv2+: GNU GPL version 2 or later <http://gnu.org/licenses/gpl.html>.
This is free software: you are free to change and redistribute it according to
the license. There is NO WARRANTY, to the extent permitted by law.

This product includes software developed by the OpenSSL Project
for use in the OpenSSL Toolkit (http://www.openssl.org/).

Compiled with OpenSSL 1.0.1f 6 Jan 2014
Running with OpenSSL 1.0.1f 6 Jan 2014

root@capl-attrib01:~# dpkg-query -L zabbix-agent
/.
/usr
/usr/bin
/usr/sbin
/usr/sbin/zabbix_agentd
/usr/share
/usr/share/doc
/usr/share/doc/zabbix-agent
/usr/share/doc/zabbix-agent/userparameter_mysql.conf
/usr/share/doc/zabbix-agent/copyright
/usr/share/doc/zabbix-agent/changelog.Debian.gz
/usr/share/man
/usr/share/man/man8
/usr/share/man/man8/zabbix_agentd.8.gz
/usr/lib
/usr/lib/tmpfiles.d
/usr/lib/tmpfiles.d/zabbix-agent.conf
/etc
/etc/zabbix
/etc/zabbix/zabbix_agentd.d
/etc/zabbix/zabbix_agentd.conf
/etc/logrotate.d
/etc/logrotate.d/zabbix-agent
/etc/default
/etc/default/zabbix-agent
/etc/init.d
/etc/init.d/zabbix-agent
/var
/var/log
/var/log/zabbix
/lib
/lib/systemd
/lib/systemd/system
/lib/systemd/system/zabbix-agent.service
/etc/zabbix/zabbix_agentd.d/userparameter_mysql.conf

So i was on Zabbix agent 2.2.23 and then i performed upgrade using these steps. 

wget https://repo.zabbix.com/zabbix/4.0/ubuntu/pool/main/z/zabbix-release/zabbix-release_4.0-3+trusty_all.deb
apt-get remove -y zabbix-release
apt-get update
dpkg -i zabbix-release_4.0-3+trusty_all.deb
apt-get update
apt-get install --only-upgrade zabbix-agent

Output of following command doesn't has /etc/init/zabbix-agent.conf

dpkg-query -L zabbix-agent 

Was this a residual file of Zabbix agent 2.2 ?

Was this a residual file of Zabbix agent 2.2 ?

Yes. This seems to be part of zabbix-agent provided by debian. Our official zabbix-4.0 one overrode it.

Indeed. This was a residual file of debian maintained zabbix agent 2.2.x version. I verified the same by downloading a package from ubuntu website.

ubuntu@accessdev1:~$ dpkg -c zabbix-agent_2.2.2+dfsg-1ubuntu1_amd64.deb | grep zabbix-agent.conf
rw-rr- root/root 845 2014-02-16 17:08 ./etc/init/zabbix-agent.conf

why didn't 4.0 remove that file if it doesn't need it?

is that the way --only-upgrade normally behaves ?

I've checked.

It seems that even simply running apt remove zabbix-agent leaves residual configuration files, regardless of whether they were modified by the user or not. That's why /etc/init/zabbix-agent.conf was left there.

Please check if apt-get purge zabbix-agent helps.

I suggest you to make necessary changes in agent upgrade documentation to reflect these findings.

Please check if apt-get purge zabbix-agent helps.

Yes, of course it does help. But beware that this will also purge your /etc/zabbix/zabbix.conf file, including any changes that you may have made.

I suggest you to make necessary changes in agent upgrade documentation to reflect these findings

We definitely will.

Documentation updated:

• Installation from packages in 3.0, 4.0, 5.0, 5.2
• Upgrade from packages in 4.0, 5.0, 5.2