IP Security For Broadcasters: Part 6 - NAT And VPN

NAT will operate without IPsec and vice versa, but making them work together is a fundamental challenge that needs detailed configuration and understanding.


All 12 articles in this series are available in our free 82 page eBook ‘IP Security For Broadcasters’ – download it HERE.

Articles in this series:


IPsec (as discussed in Part 3) provides a very high level of security based on the exchange of encryption keys and the validation of the IP header information. NAT, on the other hand, provides some extra security as it hides the IP addresses of the program-sensitive equipment from hackers.

For applications requiring access to public networks, such as the internet, NAT must be used, but it does rely on changing the source IP address and the source UDP/TCP port numbers. This conflicts with how IPsec and VPNs operate as they actively validate the IP and port addresses, the very fields that are changed during NAT, causing a conflict between the two protocols.

Solving Address Validation

There are several solutions to providing VPN over NAT including secure sockets, a method of relaying traffic between networks. This relies on the implementation of additional proxy servers and NAT Hole Punching, but it is limited to protocols such as UDP, TCP and ICMP.

There are many more NAT-Traversal solutions and each has their own benefits. The IPsec solution maintains the method of encapsulating security payload, but it does require compliance with multiple protocols that must be enabled to move through the firewalls and comply with network translators. The protocols that address the compatibility issues between IPsec and NAT are highlighted in RFC 3715 (IPsec Network Address Translation Compatibility Requirements), and the solutions are addressed in:

  • RFC3947 – Negotiation of NAT-Traversal in IKE (Internet Key Exchange)
  • RFC3948 – UDP Encapsulation of IPsec ESP (Encapsulating Security Payload) Packets
  • RFC5996 – IKE (Internet Key Exchange) Protocol

Although there is no dedicated RFC (Request-for-Comments) document for IPsec NAT-T, RFC3947 describes the process of detection and key negotiation.

The essence of achieving VPN over NAT-T consists of two parts. During phase 1, the sending router checks whether the receiving router supports NAT-T and whether there are any routers along the path which also support NAT-T. Then phase 2 initiates the UDP encapsulation of the IPsec packets so that the receiving NAT-T router can validate and use them.

Establishing IKE Connection

Phase 1 is started by the initiator issuing a specific vendor ID payload to the receiver, to advertise support for various features within RFC3947. These messages consist of defined strings that are “hashed” to provide 16-byte representations of the specific vendor ID string. Upon receipt, the receiver will send their own vendor ID payload commands back to the initiator to verify compliance. The two end routers, also known as hosts or IKE peers, have now established an intention to exchange data between each other.

The next process is for the initiator to determine if any NAT devices exist between the two IKE peer protocols. Each IKE peer may or may not exist in the NAT router so we cannot assume it has any knowledge of a NAT. To verify this, the initiator sends a NAT-D message consisting of its original IP address and UDP port number (from the UDP wrapper). These are then hashed, added to the payload, and sent to the receiver IKE peer.

Upon receipt of the NAT-D message, the IKE peer un-hashes the source IP address and UDP port number and compares them to the IP header. If they are the same then no NAT took place (otherwise, they would be different). And importantly, the receiver now has two source IP address and UDP port combinations: one for the original device prior to Network Address Translation, and the one after NAT.

Keeping Address Records

Having the original IP address is important as the receiver can use this to validate the authentication header and encrypted segment of the VPN packet. This is needed as the authentication and encryption calculations would otherwise be based on the device’s original source IP address and port number. The NAT-T process has no knowledge of the IPsec (VPN) process as it will probably have taken place upstream on an entirely different device.

Furthermore, having the source IP and UDP port addresses after the NAT allows the sender to populate the destination IP address and UDP port number when sending data back to the initiator. When the NAT router associated with the IKE peer’s initiator receives the packet, it will have a copy of them in its lookup table and change the destination IP address and UDP port numbers back to those of the original device so that the device can perform and validate its authentication checks and decryption.

Figure 1 – In this diagram the original packet is carrying TCP data. It is encrypted using the IPsec (VPN) keys with the ESP data added to it. During NAT-T the original source IP address and TCP port number are sent to the receiver IKE peer using NAT-D, allowing it to replace the IP header for correct decryption and authentication. The UDP wrapper allows the NAT to change the UDP port number to facilitate NAT translation.

Figure 1 – In this diagram the original packet is carrying TCP data. It is encrypted using the IPsec (VPN) keys with the ESP data added to it. During NAT-T the original source IP address and TCP port number are sent to the receiver IKE peer using NAT-D, allowing it to replace the IP header for correct decryption and authentication. The UDP wrapper allows the NAT to change the UDP port number to facilitate NAT translation.

Once the NAT-D negotiation has been completed and the respective IKE peers can exchange IPsec (VPN) packets, then phase 2 takes over, that is, the UDP ESP encapsulation. Assuming at least one NAT process exists within the route, the outer IP address and UDP port header are likely to be changed. However, each IKE peer now has the key information regarding the original source IP address and UDP/TCP port number so it can decrypt and authenticate the critical part of the message and maintain high levels of security through a NAT translation.

Maintaining Connections

A method of keep-alive is used to allow each IKE peer to maintain the correct original source IP address and UDP/TCP port number. This consists of periodically sending NAT-D messages between the two IKE peers so that they know the critical IP address and port number are still the same. If they do change, then the connection must be terminated, and a new session established, starting with phase 1.

Although NAT-T provides a solution to the interaction of IPsec (VPN) and NAT translation, it does so at the cost of latency and care must be taken when using it, especially if IP addresses and port numbers change. For normal transactional internet traffic this isn’t usually an issue. For latency-sensitive video and audio streaming applications, on the other hand, it can be a challenge.

Part of a series supported by

You might also like...

Designing IP Broadcast Systems - The Book

Designing IP Broadcast Systems is another massive body of research driven work - with over 27,000 words in 18 articles, in a free 84 page eBook. It provides extensive insight into the technology and engineering methodology required to create practical IP based broadcast…

IP Security For Broadcasters: Part 1 - Psychology Of Security

As engineers and technologists, it’s easy to become bogged down in the technical solutions that maintain high levels of computer security, but the first port of call in designing any secure system should be to consider the user and t…

Demands On Production With HDR & WCG

The adoption of HDR requires adjustments in workflow that place different requirements on both people and technology, especially when multiple formats are required simultaneously.

If It Ain’t Broke Still Fix It: Part 2 - Security

The old broadcasting adage: ‘if it ain’t broke don’t fix it’ is no longer relevant and potentially highly dangerous, especially when we consider the security implications of not updating software and operating systems.

Standards: Part 21 - The MPEG, AES & Other Containers

Here we discuss how raw essence data needs to be serialized so it can be stored in media container files. We also describe the various media container file formats and their evolution.