This issue is an attempt to report an observed behavior of libssh2 under Debian.

The libssh2 can use two different crypto libraries to perform encryption/decryption: OpenSSL or Libgcrypt

Configure options for libssh2:

  --with-libgcrypt        Use Libgcrypt for crypto
  --with-openssl          Use OpenSSL for crypto

By default OpenSSL is udsed if --with-libgcrypt is not specified.

Debian's package is using Libgcrypt:
http://packages.debian.org/squeeze/libssh2-1

Why it happened?
Here: http://bugs.debian.org/cgi-bin/bugreport.cgi?bug=409362
And as result Debian's package is using Libgcrypt

In its package in a file "rules" a line exists:
CONFIGURE_EXTRA_FLAGS = --with-libgcrypt

The package's Changelog excerpt:

libssh2 (0.17-1) unstable; urgency=low
...
  * Build using libgcrypt, not OpenSSL (Closes: #409362).
 ...
 -- Mikhail Gusarov <[email protected]>  Wed, 04 Jul 2007 15:21:46  0700

The library libssh2 for several other checked distros (centos, gentoo) and freebsd is using default OpenSSL

In the same time openssh-client is using only OpenSSL:
http://packages.debian.org/squeeze/openssh-client
So, the problen is not visible when try to use an private key with passphrase from console (by openssh-client).

Quoting Simon:
"The Libgcrypt backend in libssh2 contains a hand written
slimmed down ASN.1 parser to read out the RSA key, but it does not
support any of the PKCS* encrypted forms of RSA keys. The OpenSSL
backend in libssh2 uses OpenSSL to read the keys, so it supports
whatever private key formats that OpenSSL supports."
and
"Are you using libgcrypt or OpenSSL as the backend? The libgcrypt
backend can only read unencrypted private keys."

Sources:
http://www.mail-archive.com/[email protected]/msg02226.html
http://www.mail-archive.com/[email protected]/msg02057.html

From a #libssh2 on a Freenode I learned that Simon is the author of the libgcrypt backend for libssh

How to distinguish easily:
Libgcrypt variant:

# ldd /usr/lib/libssh2.so
	linux-vdso.so.1 =>  (0x00007fff67dc3000)
	libgcrypt.so.11 => /usr/lib/libgcrypt.so.11 (0x00007f2e54679000)
	libz.so.1 => /usr/lib/libz.so.1 (0x00007f2e54462000)
	libc.so.6 => /lib/libc.so.6 (0x00007f2e540ff000)
	libgpg-error.so.0 => /usr/lib/libgpg-error.so.0 (0x00007f2e53efc000)
	/lib64/ld-linux-x86-64.so.2 (0x00007f2e54b37000)

OpenSSL variant:

# ldd /usr/lib/libssh2.so
	linux-vdso.so.1 =>  (0x00007fff49bff000)
	libssl.so.0.9.8 => /usr/lib/libssl.so.0.9.8 (0x00007f42d9ff3000)
	libcrypto.so.0.9.8 => /usr/lib/libcrypto.so.0.9.8 (0x00007f42d9c52000)
	libz.so.1 => /usr/lib/libz.so.1 (0x00007f42d9a3a000)
	libc.so.6 => /lib/libc.so.6 (0x00007f42d96d8000)
	libdl.so.2 => /lib/libdl.so.2 (0x00007f42d94d4000)
	/lib64/ld-linux-x86-64.so.2 (0x00007f42da48d000)

So, for now we have to add a note to documentation. Here: http://www.zabbix.com/documentation/2.0/manual/config/items/itemtypes/ssh_checks