This is due to different behavior of sudo on debian vs ubuntu and has nothing to do with user creation or packages.

When using sudo on debian the HOME variable is set to the one of the target user, but on ubuntu the caller's home variable is used.
The latter behavior can be ovverriden with -H flag.

# sudo -Hu zabbix env | grep HOME
HOME=/var/lib/zabbix/

See here: https://unix.stackexchange.com/questions/91384/how-is-sudo-set-to-not-change-home-in-ubuntu-and-how-to-disable-this-behavior

vzhuravlev: I think that sudo was given as an example, but there is no sudo in use, when mysql client or other shell command is run under user zabbix, right? So the problem should be something else, why HOME is not set to /var/lib/zabbix when command is executed under under zabbix user.

Is it possible to make sure that zabbix agent can locate and use the home directory to read the configuration file?

For Mysql/PostgreSQL templates login/password are stored in ~/.my.cnf file and zabbix-agent is not reading it.

#For OS Linux: You need create .my.cnf in zabbix-agent home directory (/var/lib/zabbix by default)

Eg.

This user parameter runs fine on Centos and Debian as it is checking for ~/.my.cnf UserParameter=mysql.get_status_variables[*], mysql -h"$1" -P"$2" -sNX -e "show global status" but it is not checked on Ubuntu,

yurii do you have ideas on how to make agent behave consistent on different Linux distributive?  

I thought we agreed to use specific file with MySQL credentials for agent. Something like /etc/zabbix/userparameters-mysql.conf . Did something change and we decided to stick with default file in user home directory called .my.cnf ? This is not good as this file may be used by other programs. What if a user wants to use different account for Zabbix agent?

Depending on any environment variables is always a security risk for Zabbix agent. If we go for a separate file we do not depend on HOME variable at all and this issue could be closed.

vzhuravlev has pointed out that if we use custom path, it breaks compatibility with other OS (eg. Windows) as we need to provide full path to config.

Additionally while it was a trivial fix for MySQL it is problematic to use it for PostgreSQL. 
Migration to go Agent plugins should resolve the issue though.

<dimir> I don't agree:

$ PGPASSFILE=/etc/zabbix/userparameters-pgsql.conf strace psql 2>&1 | grep userparameter
stat("/etc/zabbix/userparameters-pgsql.conf", 0x7ffc8d954190) = -1 ENOENT (No such file or directory)

$ strace mysql --defaults-extra-file=/etc/zabbix/userparameter-mysql.conf 2>&1 | grep userparameter
stat("/etc/zabbix/userparameter-mysql.conf", 0x7ffce9646610) = -1 ENOENT (No such file or directory)

Absolutely the same thing with Windows, see here.

Now all these auto-connects from Zabbix agent is mess. So I propose stop it and standardize paths to configuration files:

Document it and starting from 4.4 there will be no more questions on how to set up auto-connect to the database on Zabbix agent host.

How was this issue resolved?

The problem is with all agents for ubuntu from zabbix repo. They return the value
(3.0.28-1+bionic from repo.zabbix.com)
zabbix_get -s 192.168.7.126 -k system.run["echo ~"]
~
instead expected
(3.0.12+dfsg-1 from lv.archive.ubuntu.com)
zabbix_get -s 192.168.7.126 -k system.run["echo ~"]
/var/lib/zabbix

Tested on Ubuntu 18.04.
The difference is the presence of keys
User=zabbix
Group=zabbix
in the file zabbix-agent.service.

After installing 4.2.7-1+bionic from repo.zabbix.com, adding
User=zabbix
Group=zabbix
in the file zabbix-agent.service and
systemctl daemon-reload && systemctl restart zabbix-agent
I have the result
zabbix_get -s 192.168.7.126 -k system.run["echo ~"]
/var/lib/zabbix

UPD:
unfortunately Dabian has the same problem
root@debian10:~# zabbix_get -s 127.0.0.1 -k system.run["echo ~"]
~
root@debian10:~# zabbix_get -s 127.0.0.1 -k system.run["env"]
JOURNAL_STREAM=9:52421
CONFFILE=/etc/zabbix/zabbix_agentd.conf
PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
INVOCATION_ID=6d954d170a6d4cc29d484a0c1df7d5dc
LANG=en_US.UTF-8
PWD=/

after adding User=zabbix in /usr/lib/systemd/system/zabbix-agent.service
root@debian10:~# zabbix_get -s 127.0.0.1 -k system.run["echo ~"]
/var/lib/zabbix/

root@debian10:~# zabbix_get -s 127.0.0.1 -k system.run["env"]
USER=zabbix
HOME=/var/lib/zabbix/
LOGNAME=zabbix
JOURNAL_STREAM=9:52120
CONFFILE=/etc/zabbix/zabbix_agentd.conf
PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
INVOCATION_ID=b3db11181edb4a91b5e43c76c989d928
LANG=en_US.UTF-8
PWD=/

at the same time on CentOS without any manipulation
[mchudinov@c7amd64-dock ~]$ zabbix_get -s 127.0.0.1 -k system.run["echo ~"]
/var/lib/zabbix

[mchudinov@c7amd64-dock ~]$ zabbix_get -s 127.0.0.1 -k system.run["env"]
PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin
CONFFILE=/etc/zabbix/zabbix_agentd.conf
PWD=/
LANG=en_US.UTF-8
SHLVL=1
_=/usr/bin/env

Why don't we define and use specific file for the database credentials and not depend on the HOME variable? If users would like to use it from home directory of zabbix they can just create a symlink.

Our packages already contain config files in a specific static locations, in absolute paths, no home directory is needed:

# dpkg -L zabbix-agent | grep /etc/zabbix/
/etc/zabbix/zabbix_agentd.conf
/etc/zabbix/zabbix_agentd.d
/etc/zabbix/zabbix_agentd.d/userparameter_mysql.conf

Let's not depend on the home directory at all, and add two more files

/etc/zabbix/userparameter.my.cnf
/etc/zabbix/userparameter.pgpass

to the package, with contents:

# ls -l /etc/zabbix/userparameter.pgpass
-r-------- 1 zabbix zabbix 41 Oct 7 13:37 /etc/zabbix/userparameter.pgpass

# cat /etc/zabbix/userparameter.pgpass
hostname:port:database:username:password

~# ls -l /etc/zabbix/userparameter.my.cnf
-r-------- 1 zabbix zabbix 47 Oct 7 13:41 /etc/zabbix/userparameter.my.cnf

# cat /etc/zabbix/userparameter.my.cnf
[client]
user=mysqluser
password=mysqlpassword

and after installation user will just edit those files.

Keep in mind that it would be best (security wise) if Zabbix agent process would start with no environment variables at all.

kodai suggests that it may be that HOME is not set on Debian/Ubuntu because we use

Type=forking

Listing available Type values:

simple  - A long-running process that does not background its self and stays attached to the shell.
forking - A typical daemon that forks itself detaching it from the process that ran it, effectively backgrounding itself.
oneshot - A short-lived process that is expected to exit.
dbus    - Like simple, but notification of processes startup finishing is sent over dbus.
notify  - Like simple, but notification of processes startup finishing is sent over inotify.
idle    - Like simple, but the binary is started after the job has been dispatched.

Continue to quoting kodai:

Type=simple is default mode of systemd, and processes need to run on foreground mode in container environment.

Please check, if it solves the strange HOME env in debian/ubuntu.

In general, use Type=simple is good for systemd to control processes.

And the change must be in major version because service file need to have User and Group parameters when using Type=simple

Perhaps switching to simple would solve that issue?

I've tried it. Seems to make no difference.

Setting User=zabbix in the service file fixes the issue regardless of Type being Simple or Forking.

Ah, right, because for Type=simple to work we have to set the User= anyway.

HOME is not setting because we use forking into the agent code. Not only on Debian/Ubuntu. Probably on all OS.

Changes are in https://git.zabbix.com/projects/ZBX/repos/zabbix/pull-requests/652/diff
Please review.

<dimir> No go from my side. Zabbix agent itself does not need this environment variables, so no sense in setting those. If anything around agent needs those, that thing should set it around for itself.

Keep in mind that it would be best (security wise) if Zabbix agent process would start with no environment variables at all.

Makes total sense! Here are a couple of quotes from Debian Wiki on Environment Variables and RedHat 7 System Administrator's Guide, respectively:

Using text console

Text console logins end up with a login shell. Variables are acquired in stages, by multiple processes in sequence. Each process adds some more variables.

1. At the end of boot the mother of all processes init is started. init's environment, including PATH, is defined in its source code and cannot be changed at run time.
2. init runs the start-up scripts from /lib/systemd/system (under systemd) or /etc/rc?.d based on the run level set in /etc/inittab (under sysvinit). Each service or script defines its own required environment variables.
3. At the end of booting, init runs a getty process on one or more virtual consoles. getty waits for the user to type their name, after which it sets the TERM variable and executes login to prompt for a password.
4. login checks /etc/passwd and /etc/shadow for the user's account details. If the password is acceptable, login sets HOME, SHELL, PATH, LOGNAME and MAIL based on the contents of /etc/passwd, checks PAM, and finally runs the user's login shell.
   1. PAM may instruct login to read the variables in /etc/environment also. This is the default.
5. The login shell starts and reads its shell-specific configuration files.
   1. bash first reads /etc/profile to get values that are defined for all users. After reading that file, it looks for _{/.bash_profile}, /.bash_login, and ~/.profile, in that order, and reads and executes commands from the first of these files that exists and is readable. See DotFiles for full details.
   2. (please fill in other shells as well)

Now the environment variables are ready to be used by the applications you start from the terminal.

• System services do not inherit any context (such as the HOME and PATH environment variables) from the invoking user and their session. Each service runs in a clean execution context.

Please note, that the proposed solution only affects the situation when the User setting in zabbix_agentd.conf file is used to change the user.
This is the case only when the AllowRoot is 0 and the agent is started as root.
In that case we do not inherit the environment from the calling user, but instead are getting values from /etc/passwd for the user that we switch to.
Currently, HOME and USER values  are inherited from root and are "/root" and "root" respectively. I think that this is wrong and inconsistent. Unless this behavior was implemented deliberately with some specific intent in mind.

An alternative solution is to set User and Group in the systemd service file for the user, as was proposed before. This is known to fix the issue with HOME variable as well as completely bypasses the custom logic of changing users implemented by zabbix. This is the case with the new agent, since it lacks the aforementioned logic for switching users to begin with.

This has to be decided upon.

Thanks, cyclone. The case with Debian is clear. There's systemd and it should be used. So no need to change agent code.

The issue is with *NIX systems that don't have or have poor init mechanism. It is still bad to run user as root, so what could we do?

1. deprecate old agent (since new one is coming) and leave the code as is
2. remove the user switching mechanism from old agent and instruct users to use su, sudo or such
3. instruct users to start agent as HOME=/var/lib/zabbix /path/to/agentd (will that work?)
4. set HOME environment variable only if not set, when switching a user

To me, #3 is the best option.

5. Pass --login-path option to MySQL client explicitly in UserParameters or ask users to set MYSQL_TEST_LOGIN_FILE environment variable?

I think templates are not well thought through if they require changes in Zabbix agent code.

I see the same problem in CentOS 7 with zabbix-agent 4.2.8.
I agree with @Glebs Ivanovskis . Zabbix agent should not depend on HOME environment variable, hence UserParameters neither.

Also, on CentOS/RedHat SELinux by default prohibits Zabbix Agent from reading in /var/lib en would require an extra rule:
extra allow zabbix_agent_t zabbix_var_lib_t:file read;

The version of MariaDB I currently have (5.5.60)  however does not seem to support the --login-path option. but with --defaults-extra-file I can point mysql to the correct .cnf file containing username and password which then can be placed in /etc/zabbix, circumventing SELinux problems on reading that file.