Desktop via XSession

Instead of choosing your favourite dektop environment when logging in via a display manager, use XSession to gain more control over the login process. This will invoke ~/.xsession (permissions: 0755):

#!/bin/bash

# resources
xrdb -merge ~/.Xresources

# key bindings
xmodmap ~/.Xmodmap
xbindkeys

# ssh agent
export SSH_ASKPASS="$(which x11-ssh-askpass)"
eval $(ssh-agent -s)
ssh-add </dev/null

# desktop
exec startxfce4

Also note the last line which does not simply invoke startxfce4 but substitutes it for the current process (bash).

See also: XFree Resources, XFree Key Bindings, SSH Agent and ssh-askpass