There are 2 main configuration files: /etc/ssh/sshd_config file for server-side configurations, and ~/.ssh/config/ for client-side configurations. Worth noting that SSH key(s) are in the ~/.ssh/ folder.

Client Side

This is my standard configuration. With this configuration, I can ssh into my server by issuing command ssh server1. This configuration features TCPKeepAlive, which basically keeps the connection alive despite client being idle.

Host server1
    Port 22
    User user1
    ServerAliveInterval 30
    ServerAliveCountMax 120
    TCPKeepAlive yes

Server Side

First, it is a great idea to use nonstandard port (not 22) for ssh, because port 22 is always being the subject of penetration. Second, do not allow ssh root access, because that's very dangerous. Third, do not allow password access, since we will only allow access via asymmetric public-private key pair.

This config is package-generated (Ubuntu Xenial 16.04 LTS, Debian 8 Jessie), with fewer inline comments. Pay attention to Port, PasswordAuthentication and AllowUser directives!

# use non-standard port
Port *****

# using SSH version 2 standards
Protocol 2

# HostKeys
HostKey /etc/ssh/ssh_host_rsa_key
HostKey /etc/ssh/ssh_host_dsa_key
HostKey /etc/ssh/ssh_host_ecdsa_key
HostKey /etc/ssh/ssh_host_ed25519_key

# lifetime and size of ephemeral version 1 server key
KeyRegenerationInterval 3600
ServerKeyBits 1024

# authentication and allowed user(s)
LoginGraceTime 120
PermitRootLogin no
StrictModes yes
AllowUsers user1
RSAAuthentication yes
PubkeyAuthentication yes

# do not enable password login, change to yes if needed
PasswordAuthentication no

# privilege Separation is turned on for security
UsePrivilegeSeparation yes

# logging
SyslogFacility AUTH
LogLevel INFO

# do not allow rhosts
IgnoreRhosts yes
RhostsRSAAuthentication no
HostbasedAuthentication no

# to enable empty passwords, change to yes (NOT RECOMMENDED)
PermitEmptyPasswords no

# Change to yes to enable challenge-response passwords (beware issues with some PAM modules and threads)
ChallengeResponseAuthentication no

# GSSAPI options
GSSAPIAuthentication no
GSSAPICleanupCredentials yes

# X11 GUI forwarding
X11Forwarding no
X11DisplayOffset 10
PrintMotd no
PrintLastLog yes
TCPKeepAlive yes

# allow client to pass locale environment variables
AcceptEnv LANG LC_*

# allows SFTP to access this instance
Subsystem sftp /usr/lib/openssh/sftp-server

# to enable PAM authentication
UsePAM yes

# added by DigitalOcean build process
ClientAliveInterval 120
ClientAliveCountMax 2

Usually I will create a backup copy of the default, package-generated sshd_config, then create a new sshd_config with above directives.

cd /etc/ssh
sudo mv sshd_config sshd_config.default
sudo vim sshd_config