OpenSSH at CERN FAQ


Table of Contents
1. Frequently Asked Questions

1. Frequently Asked Questions

Here is a list of problems seen after the introduction of OpenSSH at CERN. Before contacting ssh.support@cern.ch, we ask you to check here - perhaps your problem has already been solved before!


1.1. General troubleshooting hints

  • In order for us to help you, the problem has to be repeatable.

  • When reporting problems, always send the output of

    $ ssh -v -l <user> <destination>
  • If you have a root account on the destination host, please run

    # /usr/sue/etc/sshd -d -p 222

    or (on Linux)

    # /usr/sbin/sshd -d -p 222

    as root there and connect using

    $ ssh -v -p 222 <user> <destination>

    (the sshd server will exit after each connection, and you may run into trouble with a local firewall that prevents you from connecting from a different machine. Same-machine connections to "localhost" should work)

  • If you do not have root access on the server, you can generate your own "server" key pair and run on an unprivileged port:

    $ ssh-keygen -P "" -f /tmp/sshtest
    $ pagsh -c "/usr/sue/etc/sshd -d -p 2222 -h /tmp/sshtest"

    or (on Linux)

    $ pagsh -c "/usr/sbin/sshd -d -p 2222 -h /tmp/sshtest"

    Then connect using

    $ ssh -v -p 2222 <user> <destination>

1.2. Recommended versions of ssh at CERN

Operating systemAFS @sys idVersionRemarks
CERN Linux SLC3, SLC4{i386,ia64,x85_64}_linux2{4,6}OpenSSH_4.3p2deployed as RPMs from the installation server and on CD. See installation instructions
MS NICE 2000, XP?PuTTY(client only, see the installation instructions)


1.3. Operating systems no longer supported by CERN-IT

Operating systemAFS @sys idRemarks
SUN Solaris 9sun4x_59Previously deployed through SUE "ssh" and "sshd" features, see installation instructions.
SUN Solaris 8sun4x_58
SUN Solaris 7sun4x_57
SUN Solaris 2.6sun4x_56
Digital UNIX V4.0Falpha_dux40 
CERN RedHat Linux 7.3.4i386_redhat73, i386_linux24
CERN RedHat Linux 7.2.1i386_redhat72 >
CERN RedHat Linux 6.1.1i386_redhat61
HP-UX 10.20hp_ux102 
AIXrs_aix42Binaries are available through a collaboration with CASPUR


1.4. SSH-2 protocol, or why it is not the default at CERN

Background: the original version of ssh as written by Tatu Ylonen was found to have some protocol design flaws, for which workarounds had to be developed. To overcome these shortcomings, the protocol between client and server has been redesigned. This is new protocol then was called SSH-2, and the previous one has been dubbed SSH-1. Please see this FAQ from the O'Reilly "snail" book on the topic for further discussion.

SSH.COM[1] has pushed for protocol SSH-1 to be declared obsolete. Other bodies like the US DOE advisory group CIAC have followed this trend in their recent advisories. However, please note that the current SSH-1 protocol shortcomings are all handled, and that both SSH-1 and SSH-2 protocols have had security-relevant implementation errors.

At CERN, the OpenSSH implementation of both SSH-1 and SSH-2 is used. This implementation has dealt very quickly in the past with security holes appearing in their code, and the recommended versions of ssh at CERN should not be vulnerable to known exploits.

Unfortunately for CERN, some of the ssh protocol extensions CERN users depend on heavily (namely AFS and Kerberos-4 support) are not available in SSH-2. A future move toward Kerberos-5 could help in this respect since an implementation of GSSAPI over SSH-2 is available. For the time being, SSH-1 is still the recommended protocol at CERN.


1.5. log in using RSA keys

Do you really want to do this? Using RSA for login means you will not get an AFS token, so you cannot access most of your home directory on the public servers. There is no way to "translate" between RSA key and AFS tokens.

If you want to give it a try, check the following common errors:

  • the UNIX permissions must be correct: 0600 for ~/.ssh/authorized_keys, 0755 for ~/.ssh (and AFS read access for everybody!), home directory not writable by anybody but you.

    Warning

    Please make sure that your private key is somewhere safe (e.g. in ~/private, with a symlink to ~.ssh), and encrypted using a good pass phrase.

  • in ~/.ssh/authorized_keys, there has to be one key per line (no linebreaks allowed)

The debugging tips at the beginning of this chapter (running the server in debug mode) should point out the reason for failure pretty quickly.


1.6. New warning messages

OpenSSH stores both the host name and the IP number together with the host key. This leads to some new messages:

    Warning: Permanently added 'lxplus001,137.138.161.126' (RSA) to the list of known hosts.
    Warning: Permanently added the RSA host key for IP address '137.138.161.126' to the list of known hosts.

If these annoy you, use "CheckHostIP no" in your $HOME/.ssh/config file. However, please be aware that you are turning off an intentional security feature of ssh.

Some warning that may appear while connecting to the PLUS servers under their common DNS name (e.g. RSPLUS, HPPLUS) is due to the fact that for load-balancing purposes, these servers' DNS entry is constantly changing. This is detected and reported by ssh (as it should be).

    @@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
    @       WARNING: POSSIBLE DNS SPOOFING DETECTED!          @
    @@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
    The RSA host key for rsplus has changed,
  5 and the key for the according IP address 137.138.246.82
    is unknown. This could either mean that
    DNS SPOOFING is happening or the IP address for the host
    and its host key have changed at the same time
    @@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
 10 @    WARNING: REMOTE HOST IDENTIFICATION HAS CHANGED!     @
    @@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
    IT IS POSSIBLE THAT SOMEONE IS DOING SOMETHING NASTY!
    Someone could be eavesdropping on you right now (man-in-the-middle attack)!
    It is also possible that the RSA host key has just been changed.
 15 Please contact your system administrator.
    Add correct host key in /afs/cern.ch/user/i/iven/.ssh/known_hosts to get rid of this message.
    Offending key in /afs/cern.ch/user/i/iven/.ssh/known_hosts:246
    Password authentication is disabled to avoid Trojan horses.
    Agent forwarding is disabled to avoid Trojan horses.

To avoid these, use qualified hostnames like rsplus01, hpplus01 etc.. (LXPLUS and SUNDEV are not prone to this problem, since a common host key is used on all the servers in the cluster)

An alternative is to (manually) insert into $HOME/.ssh/known_hosts the PLUS name after each qualified machine name that belongs to this PLUS service:

    rsplus01,rsplus 1024 37 15457042575...
    rsplus02,rsplus 1024 37 10734479336...

To remove the above error message, simply edit the file ~/.ssh/known_hosts (or ~/.ssh/known_hosts2 for the SSH-2 protocol) and remove the line (which should start with the hostname and/or IP address). Be careful not to break the long lines, it has to have one line per host/key. Next time you connect, ssh should ask you whether you actually want to connect, etc..


1.7. Statistics options for scp

OpenSSH scp does not support a few of the command line options from ssh-1.2.26. Besides, the statistics output is different. The environment variables controlling statistics output (SSH_SCP_STATS, SSH_NO_SCP_STATS, SSH_ALL_SCP_STATS, SSH_NO_ALL_SCP_STATS) are not supported, either. The changed options are

ssh-1.2.26 optionmeaningOpenSSH option
-aTurn on statistics display for each file (on by default)(on by default)
-ATurn off statistics display for each file. This appears to be a no-op for ssh-1.2.26(n.a., use -q to turn off all statistics)
-LUse non privileged port-o UsePriviledgedPort=no (works as well on ssh-1.2.26)
-QTurn on statistics display(on by default)

Sample statistics output from OpenSSH scp (no explicit options)

    junk                 100% |*****************************| 22867       00:00
    zeroes               100% |*****************************|   512 KB    00:00

and output from ssh-1.2.26 scp:

    junk                      |         22 KB |  22.3 kB/s | ETA: 00:00:00 | 100%
    zeroes                    |        512 KB | 512.0 kB/s | ETA: 00:00:00 | 100%

If you actually parse this output in scripts, you would have to change them.


1.8. Errors on exit regarding X11 applications

Since the ssh client does forwarding for the X11 traffic from the remote host, it won't exit until the last X11 application has been closed. It appears that this mechanism sometimes fails, and the ssh program will report errors like below even if all remote X11 applications are done:

    (logout)
    Waiting for forwarded connections to terminate...
    The following connections are open:
    X11 connection from xxxxx.cern.ch port 2352

The session will appear to hang. It can be closed by typing "~." (without the quotes), and this should return you to your previous shell. You could use "~&" as well to leave the current connection as a background process.

If you are sure that there are no X11 windows or icons from the remote server around, and if you can reproduce the problem, please contact ssh.support@cern.ch.

A current suspicion is that the regular network scanning mechanism plays a role in this: by opening a connection to the remote X11 port, but failing to connect through the forwarded channel, this could mess up the internal bookkeeping done by ssh. To be confirmed.


1.9. Other FAQs

  • FAQ from R.Silverman's book on ssh (O'Reilly "Snail Book")

  • FAQ from the OpenSSH project

Notes

[1]

(they may have a vested interest, since they sell ssh client and servers. The original ssh was freely distributable, subsequent releases (including the ones that understood SSH-2) were more and more restricted.)