[ZBX-15565] Zabbix 4.0.3 ssh-agent not working Created: 2019 Jan 31  Updated: 2019 May 29  Resolved: 2019 May 29

Status: Closed
Project: ZABBIX BUGS AND ISSUES
Component/s: Server (S)
Affects Version/s: 4.0.3
Fix Version/s: None

Type: Problem report Priority: Trivial
Reporter: Benny Boy Assignee: Arturs Lontons
Resolution: Won't fix Votes: 0
Labels: agentless, server, ssh
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified
Environment:

Production


Attachments: PNG File create_the_item.png     PNG File edit_the_item.png    

 Description   

I follow the documentation except for the location of the home of the zabbix user.

https://www.zabbix.com/documentation/4.0/manual/config/items/itemtypes/ssh_checks

I choose Key file authentication so I specified the `SSHKeyLocation=/var/lib/zabbix/.ssh` and restart `systemctl restart zabbix-server` and `systemctl restart zabbix-agent`

I generate ssh key without passphrase. I use those commande to generate both key I try to use.

First key I try `ssh-keygen -o -a 100 -t ed25519`

Second key I try `ssh-keygen -o -a 100 -t rsa -b 4096`

I generate those keys as zabbix user in the zabbix server.

On the destination server I copy both key in authorized_key and confirm it's working using those command.

`sudo -u zabbix ssh -i ~/.ssh/id_ed25519 zabbix_ssh_agent@hostname`

`sudo -u zabbix ssh -i ~/.ssh/id_rsa zabbix_ssh_agent@hostname`

Both key are working, on the destination side I create the user zabbix_ssh_agent with both public key. I choose to test both key because I started with ed25519 and I want to make sure this is not the cause of the problem.

 I create the item in zabbix frontend with those parameters.

 

Name: SSH syntetic check
Type: SSH agent
Key: ssh.run[testing123]
Authentification method: Public key
User name: zabbix_ssh_agent
Public key file: id_rsa.pub
Private key file: id_rsa
Execute script: ls /tmp
Type of information: Text
Update Interval: 1m
History storage period: 90d

 

When I go back to the same screen to edit or validate the information I just provided to Zabbix, I have several other fields and a duplicate field. I see 2 User Name field.

I attache two screenshots.

The item start to collect information and I got that error in the frontend web page.

`Public key authentication failed: Callback returned error`

I try couple way to specify the location of the key file. When I try to specify the relative path I got that error message `Cannot access public key file /var/lib/zabbix/.ssh//var/lib/zabbix/.ssh/id_rsa.pub`

The relavite path : 

```

Public key file: /var/lib/zabbix/.ssh/id_rsa.pub

Private key file: /var/lib/zabbix/.ssh/id_rsa

```

I enable debug log and collect that.

```

26326:20190131:150855.564 End of substitute_key_macros():SUCCEED data:'ssh.run[testing123]'
26326:20190131:150855.564 In substitute_simple_macros() data:'id_rsa.pub'
26326:20190131:150855.564 In substitute_simple_macros() data:'id_rsa'
26326:20190131:150855.564 In substitute_simple_macros() data:'ls /tmp'
26326:20190131:150855.564 In substitute_simple_macros() data:'SSH_DESTINATION_USER'
26326:20190131:150855.564 In substitute_simple_macros() data:EMPTY
26326:20190131:150855.564 In get_value() key:'ssh.run[testing123]'
26840:20190131:150855.564 End of preprocessor_enqueue()
26840:20190131:150855.564 In preprocessor_assign_tasks()
26840:20190131:150855.564 In preprocessor_get_queued_item()
26840:20190131:150855.564 End of preprocessor_get_queued_item()
26326:20190131:150855.564 In ssh_run()

26326:20190131:150855.684 ssh_run() supported authentication methods:'publickey'
26326:20190131:150855.685 End of ssh_run():
26326:20190131:150855.685 Item [HOSTNAME_DESTINATION:ssh.run[testing123]] error: Public key authentication failed: Callback returned error

20776:20190130:201919.772 item "HOSTNAME_DESTINATION:ssh.run[testing123]" became not supported: Public key authentication failed: Username/PublicKey combination invalid

26247:20190131:150853.823 ssh:[username:'SSH_DESTINATION_USER' password:'' authtype:1 params:'ls /tmp']

26247:20190131:150853.823 ssh:[publickey:'id_rsa.pub' privatekey:'id_rsa']

```

Thank you!!



 Comments   
Comment by Benny Boy [ 2019 Jan 31 ]

I move all the file to `/home/zabbix/` and modify `SSHKeyLocation=/home/zabbix/.ssh` restart server and agent and I have the same problem.

Comment by Alexey Pustovalov [ 2019 Jan 31 ]

is it possible to enable debug mode 5?
zabbix_server -R log_level_increase till 5th debug level.

Comment by Benny Boy [ 2019 Feb 01 ]

The log I already post are in the debug log level. I redo it to make sure but I have +- the same information.

I also see the in the zabbix_server.log :  SSH2 support:              YES.

I use Ubuntu 16.04 with package version of Zabbix.

```

30470:20190201:164230.551 In substitute_key_macros() data:'ssh.run[testing123]'
30470:20190201:164230.551 End of substitute_key_macros():SUCCEED data:'ssh.run[testing123]'
30470:20190201:164230.551 In substitute_simple_macros() data:'id_rsa.pub'
30470:20190201:164230.551 In substitute_simple_macros() data:'id_rsa'
30470:20190201:164230.551 In substitute_simple_macros() data:'ls /tmp'
30470:20190201:164230.551 In substitute_simple_macros() data:'zabbix_ssh_agent'
30470:20190201:164230.551 In substitute_simple_macros() data:EMPTY
30470:20190201:164230.551 In get_value() key:'ssh.run[testing123]'
30470:20190201:164230.551 In ssh_run()
30505:20190201:164230.551 In zbx_preprocess_item_value()
30505:20190201:164230.552 End of zbx_preprocess_item_value()
30389:20190201:164230.552 End substitute_simple_macros() data:'{75877}=1 or {75878}<>0'
30505:20190201:164230.552 In zbx_ipc_socket_write()
30389:20190201:164230.552 In substitute_simple_macros() data:'{101666}=1'
30389:20190201:164230.552 End substitute_simple_macros() data:'{101666}=1'
30699:20190201:164230.552 End of DCconfig_get_poller_nextcheck():1549039351
30699:20190201:164230.552 End of get_values():0
30815:20190201:164230.552 __zbx_zbx_setproctitle() title:'unreachable poller #132 [got 0 values in 0.000731 sec, getting values]'
30815:20190201:164230.552 In get_values()
30815:20190201:164230.552 In DCconfig_get_poller_items() poller_type:1
30470:20190201:164230.691 ssh_run() supported authentication methods:'publickey'
30470:20190201:164230.692 End of ssh_run():NOTSUPPORTED
30470:20190201:164230.692 Item [DESTINATION_HOSTNAME:ssh.run[testing123]] error: Public key authentication failed: Callback returned error

```

Comment by Benny Boy [ 2019 Feb 05 ]

If you need other info, I'm ready to provide

Thx!

Comment by Frank [ 2019 Feb 18 ]

metabsd I was having a similar issue on our Zabbix-Proxy (worked fine on Zabbix-Server).
Turns out, when we added a newline after the "----END RSA PRIVATE KEY----" it solved our problem. Perhaps this will also resolve your issue.
I have reported this in ZBX-15673 

Comment by Arturs Lontons [ 2019 Feb 19 ]

Hi,

Thank you for your comment, Frank.
Benny Boy, please do some further troubleshooting by executing the steps that Frank provided, just so we know if the cause of the issue is the same or if these are two separate issues.

Thanks.

Comment by Glebs Ivanovskis [ 2019 Feb 19 ]

ZBX-9817 may be related.

Comment by Benny Boy [ 2019 Feb 19 ]

After spending time trying to find a solution with the SSH agent. I decided to try it with some Python programming with the paramiko library. I had several problems with my key type ed25519. I want to specify that my key is ed25519. So the padding for this type of key when it is generated on Ubuntu is rather like that.
----- BEGIN OPENSSH PRIVATE KEY -----
----- END OPENSSH PRIVATE KEY -----

During my journey with Paramiko I discovered that if I use my key without passphrase I have errors but when I specify a passphrase it works well.

I explain all this because now I use a passphrase with my key. So I specify the passphrase in zabbix as well as the location of the private key and the public key. By cons I did not remove the padding. ----- END OPENSSH PRIVATE KEY -----

I had the following error: 

Public key authentication failed: Callback returned error

I want to specify that now everything works with my piece of code in python.

I also test without the padding ------ END OPENSSH PRIVATE KEY ------ but I also had the error

Public key authentication failed: Callback returned error
Comment by Edgars Melveris [ 2019 May 29 ]

Closing this, as I believe there is no bug in Zabbix side.
I'd like to point you at this part of the documentation:

Key types other than “rsa” may be supported by the ssh-keygen tool and SSH servers but they may not be supported by libssh2, used by Zabbix.

Feel free to reopen, if you still think this is a bug in Zabbix

Generated at Fri Apr 19 19:23:53 EEST 2024 using Jira 9.12.4#9120004-sha1:625303b708afdb767e17cb2838290c41888e9ff0.