IPsec is a protocol suite on the same level as ipv4 and ipv6 that provides confidentiality and authentication of each IP packet. It that was originally designed for ipv6 and then back-ported to legacy IP (IPv4). Since the Internet Protocol is a network-layer protocol, IPsec is used mostly for Virtual Private Networks (vpn).
IPsec has many similarities to ssl: It protect data only in transit, and the secure channel is established using symmetric cryptographic algorithms. The key exchange is guarded by public-key-authentication although the key exchange is strictly speaking not part of the IPsec specification. The key exchange is specified on its own and is known as Internet Key Exchange (IKE). Like SSL, IKE uses x509 certificates to establish a trusted key exchange. Unlike to SSL, keys for IPsec can also be distributed manually although that doesn't scale well and is error prone, and therefore this is rarely an option.
A striking difference between IPsec and SSL/TLS is that IPsec is dealt with by the kernel, so IPsec is transparent to applications - at least that's the idea. SSL on the other hand is at the responsibility of the application. Another difference is that an IPsec security association (the IPsec term for the secure channel) is a simplex channel (i.e. only one-way) - for full duplex secure communication with IPsec two security associations must be established.
IPsec can be used in several modes. Originally there were just tunnel mode, where entire IP packets were encapsulated in containing IP packets, and transport mode, which operated on the original IP packets. In addition, there are two options to protect data:
- Authentication Header (AH) is used for integrity protection
- Encapsulating Security Payload (ESP) is used for confidentiality protection and optionally for payload integrity protection.
A typical setup for VPNs is IPsec in tunnel mode with ESP with authentication.
The design of IPsec is said to be overly complex (see Ferguson and Schneier). As a matter of fact, IPsec was felt to be so troublesome and hard to manage that alternatives like openvpn were developed, which use SSL/TLS as base technology.
