IPsec: How it works and why we need it

Computerworld - The IP Security (IPsec) architecture comprises a suite of protocols [7, 9, 10] developed to ensure the integrity, confidentiality and authentication of data communications over an IP network [1, 3]. While the flexibility of the IPsec standards has drawn the interest of the commercial sector, this same flexibility has resulted in the identification of several problems with the protocols because of their complexity [8]. As with other security systems, poor maintenance can easily lead to a critical system failure [5].
IPsec may be used in three different security domains: virtual private networks, application-level security and routing security. At this time, IPsec is predominately used in VPNs. When used in application-level security or routing security, IPsec is not a complete solution and must be coupled with other security measures to be effective, hindering its deployment in these domains [4].

IPsec operation
IPsec has two modes of operation, transport mode and tunnel mode. When operating in transport mode, the source and destination hosts must directly perform all cryptographic operations. Encrypted data is sent through a single tunnel that is created with L2TP (Layer 2 Tunneling Protocol). Data (ciphertext) is created by the source host and retrieved by the destination host. This mode of operation establishes end-to-end security.
When operating in tunnel mode, special gateways perform cryptographic processing in addition to the source and destination hosts. Here, many tunnels are created in series between gateways, establishing gateway-to-gateway security [6]. When using either of these modes, it's important to provide all gateways with the ability to verify that a packet is real and to authenticate the packet at both ends. Any invalid packets must be dropped [2].
Two types of data packet encodings (DPE) are required in IPsec. These are the authentication header (AH) and the encapsulating security payload (ESP) DPEs. These encodings offer network-level security for the data [4]. The AH provides authenticity and integrity of the packet. The authentication is made available through keyed hash functions, also known as MACs (message authentication codes). This header also prohibits illegal modification and has the option of providing antireplay security. The AH can establish security between multiple hosts, multiple gateways, or multiple hosts and gateways, all implementing AH [7]. The ESP header provides encryption, data encapsulation and data confidentiality. Data confidentiality is made available through symmetric key encryption [3].
During its journey through the various tunnels and gateways, additional headers are added to the packet. On each pass through a gateway, a datagram is wrapped in a new header. Included in this header is the