In a recent post here on the INE blog, we received some follow-up questions similar to the following:
"Why do IPSec peers end up using tunnel mode, even though we had explicitly configured transport mode in the IPSec transform-set?"
It is an excellent question, and here is the answer. In a site to site IPSec tunnel the "mode transport" setting is only used when the traffic to be protected (traffic matching the Crypto ACLs) has the same IP addresses as the IPSec peers, and excludes all other IP addresses. When Crypto ACLs include IP addresses beyond of the 2 peer endpoints the "mode transport" setting is ignored, and tunnel mode is negotiated (due to IP addresses, other than the 2 peers, being part of the crypto ACL). There is also an option for the key word "require" after "mode transport" which will prevent the peers from negotiating tunnel mode, and if the IP addresses in the Crypto ACLs are outside of the peers's own IP addresses, IKE phase 2 will not successfully complete.
One notable exception to this, is GET VPN, where the KS policy of tunnel mode or transport mode will be used by the group members (whichever mode the KS has configured), regardless of the IP addresses used in the KS ACL for policy.
Below is a site to site example. Let's use the following topology, with R1 and R3 being peers, and a Crypto ACL that says to encrypt all ICMP traffic, regardless of the IP addresses. This Crypto ACL will cause our peers to ignore the mode transport option, and negotiate tunnel mode.
Below are the full configs, some debug output, and show commands to demonstrate that even with transport mode explicitly configured in the transform sets, if the crypto ACLs don't exclusively include the endpoints of the VPN tunnel, the two peers go ahead and negotiate tunnel mode instead of transport mode. Note the Crypto ACL includes all ICMP from any source to any destination.
First, here is R1: