[ZBX-4857] pollers hang when processing SSH checks if "public key" field is empty Created: 2012 Apr 10  Updated: 2017 May 30  Resolved: 2012 May 02

Status: Closed
Project: ZABBIX BUGS AND ISSUES
Component/s: Frontend (F), Server (S)
Affects Version/s: None
Fix Version/s: 1.8.13rc1, 2.0.0rc4

Type: Incident report Priority: Blocker
Reporter: Oleksii Zagorskyi Assignee: Unassigned
Resolution: Fixed Votes: 0
Labels: ssh
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Attachments: File ltrace_12507_empty_pub_key_ssh2-141ssl.out     File ltrace_12741_empty_priv_key_ssh2-141ssl.out    

 Description   

First of all - I'm almost sure it's a new libssh2's bug (which will be reported to libssh2 devs).

But it observed because of not very good behavior of the zabbix frontend and zabbix server.

Zabbix frontend allows to create an "SSH agent" item with empty values of: User name, Public key file, Private key file.
But these values cannot be empty anyhow. Frontend should disallow to save such items!
*NOTE*: "Key passphrase" should be allowed to be empty!
Btw, SSH command operation in actions do not allow to save operation if one of three mentioned fields is empty, and that's great !

That was frontend related part.

Next is server's part.
To perform SSH checks, a path to a folder, where public and private keys are storing, must be defined in server configuration parameter "SSHKeyLocation"

I know that to connect to a remote server by ssh client (not the libssh2 library) I need just a private key.
So in the frontend I'm trying to leave the field "Public key file" empty.

(Note: public key actually is stored in the private one and can be extracted,
Even in the libssh2 a function "_libssh2_pub_priv_keyfile" -> "Computing public key from private key file" if it is not passed to a "libssh2_userauth_publickey_fromfile_ex" function.

And when server starting process this ssh item - a poller hangs ! and consumes 100% of cpu/core.
A strace command shows constant tries to read a directory with crazy speed:

  1. strace -p 18694
    read(10, 0x7fdc565fa000, 4096) = -1 EISDIR (Is a directory)
    read(10, 0x7fdc565fa000, 4096) = -1 EISDIR (Is a directory)
    read(10, 0x7fdc565fa000, 4096) = -1 EISDIR (Is a directory)
    read(10, 0x7fdc565fa000, 4096) = -1 EISDIR (Is a directory)
    ...

I'm attaching two outputs of "ltrace -tt -n2 -s 80 -T -S -p <PID>"

ltrace_12507_empty_pub_key_ssh2-141ssl.out - with the empty *public* key in zabbix configuration
where we can see that problem occurs in the libssh2 namely.
When the key file field is empty - zabbix_server pass only path to folder (/home/zabbix/.ssh/) to the libssh2's "libssh2_userauth_publickey_fromfile_ex" function and it hangs as I explained above !
Moreover, I have reproduced this bug without zabbix_server (libssh2 has many code examples which can be used).

Actually it's bad zabbix_server behavior as well, but if frontend will be fixed as described above - it will be suitable solution, IMO

ltrace_12741_empty_priv_key_ssh2-141ssl.out - with the empty *private* key in zabbix configuration
we can see than libssh2's function "file_read_privatekey" doesn't affect by this problem when the private key file it's only a path to a folder.

I used latest release 1.4.1 of libssh2 (from April 4, 2012) compiled from sources.

I suppose the problem in a libssh2's function "file_read_publickey"
lines:
/* Read Public Key */
fd = fopen(pubkeyfile, "r");
if (!fd)

{ return _libssh2_error(session, LIBSSH2_ERROR_FILE, "Unable to open public key file"); }

while (!feof(fd) && (c = fgetc(fd)) != '\r' && c != '\n')
pubkey_len++;

That's all for now, later I'll add some comments bellow.



 Comments   
Comment by Oleksii Zagorskyi [ 2012 Apr 11 ]

Posted to libssh2 developers here:
http://www.libssh2.org/mail/libssh2-devel-archive-2012-04/0021.shtml

<zalex> that bug has been fixed here:
http://git.libssh2.org/?p=libssh2.git;a=commitdiff;h=11f9dce3d708a9c0ad43702ea3eb6d93a3bbe527

I suppose libssh2 version 1.4.2 will be without this bug.
CLOSED

Comment by dimir [ 2012 Apr 18 ]

Confirmed, libssh hangs if pubkey is a directory. Reproducible with libssh 1.3.0 too.

Comment by dimir [ 2012 Apr 18 ]

Fix for 1.8 is available in development branch: svn://svn.zabbix.com/branches/dev/ZBX-4857 .

Until bug in libssh2 is fixed we do a regular file check before doing a libssh2 call.

Symbolic links as key files are supported.

Added more error information when libssh2 call fails.

Comment by dimir [ 2012 Apr 18 ]

There will be a separate branch for 2.0 .

Comment by dimir [ 2012 Apr 26 ]

Fix "Fix Version/s"

Comment by dimir [ 2012 Apr 26 ]

(1) For 2.0 we decided to fix "ssh check" item form to not allow empty key files and username in case of public key authentication method.

<zalex> I do not see it implemented here Will it be performed in another development?

<dimir> We will fix it in a separate branch.

<zalex> Thanks, but would be great to know that separate branch/issue

<sasha> GUI will be fixed in a new development branch svn://svn.zabbix.com/branches/dev/ZBX-4857

Comment by Alexander Vladishev [ 2012 Apr 27 ]

Tested! Please review my changes in r27165.

Comment by dimir [ 2012 Apr 28 ]

Fixed in pre-1.8.13 r27192, pre-2.0.0 r27194.

Comment by Alexander Vladishev [ 2012 May 02 ]

Validation of username, private and public key has been added in the development branch svn://svn.zabbix.com/branches/dev/ZBX-4857 r27211.

<zalex> successfully TESTED.

Comment by Eduards Samersovs (Inactive) [ 2012 May 07 ]

Frontend tested.

Comment by dimir [ 2012 May 07 ]

Fixed in pre-2.0.0 r27307.

Comment by Sandis Neilands (Inactive) [ 2015 Jul 14 ]

The correction for this issue has introduced a minor memory leak. See ZBX-9703.

Generated at Fri Apr 19 09:16:51 EEST 2024 using Jira 9.12.4#9120004-sha1:625303b708afdb767e17cb2838290c41888e9ff0.