Troubleshooting GRE and GRE over IPSec on Palo Alto Networks Firewalls

Generic Routing Encapsulation (GRE) is a tunneling protocol that can encapsulate a wide variety of network layer protocols inside virtual point-to-point links. When combined with IPSec, GRE tunnels can provide secure, flexible connections between networks. However, configuring and troubleshooting these tunnels on Palo Alto Networks firewalls can sometimes be challenging. This guide provides a comprehensive approach to troubleshooting common GRE and GRE over IPSec issues.

Detailed Explanation: GRE and GRE over IPSec

Understanding GRE (Generic Routing Encapsulation)

GRE is a tunneling protocol developed by Cisco that allows the encapsulation of a wide variety of network layer protocols inside IP tunnels. Think of it as putting a data packet (the "passenger protocol") inside another IP packet (the "delivery protocol" or "carrier protocol") to traverse an IP network. This is particularly useful for sending non-IP protocols over an IP network or for connecting discontinuous subnets. GRE itself is stateless, meaning the tunnel endpoints don't maintain information about the state or availability of the remote endpoint, unless keepalives are used.

Key Characteristics of GRE:

• Multiprotocol Support: GRE can encapsulate various Layer 3 protocols, including IP (IPv4, IPv6), IPX, and AppleTalk, making it highly versatile.
• Simplicity: GRE is relatively simple to configure compared to other tunneling protocols.
• No Inherent Security: GRE does not provide encryption or authentication for the encapsulated traffic. Data sent over a plain GRE tunnel is not secure.
• IP Protocol 47: GRE is identified by IP protocol number 47 in the outer IP header.
• Supports Multicast and Broadcast Traffic: A significant advantage of GRE is its ability to transport multicast (e.g., for routing protocols like OSPF, EIGRP) and broadcast traffic, which IPsec alone typically does not support.

GRE Packet Structure

When GRE encapsulates a packet, it adds its own header and a new IP header (delivery header). A typical GRE packet encapsulation looks like this:

• Original Packet: [Original IP Header | Original Payload (e.g., TCP/UDP + Data)]
• GRE Encapsulated Packet: [New (Outer) IP Header (Protocol 47) | GRE Header | Original IP Header | Original Payload]

The GRE header itself is typically 4 bytes but can be larger if optional fields like a key or sequence number are used (defined in RFC 2890). The minimal GRE header includes flags and the protocol type of the encapsulated passenger protocol.

GRE Packet Encapsulation

How GRE Works:

1. A packet (passenger protocol) destined for a remote network arrives at the ingress router (tunnel source).
2. The router determines that the packet needs to be sent through a GRE tunnel.
3. The original packet is encapsulated with a GRE header. The GRE header includes the protocol type of the original packet.
4. A new IP header (delivery header) is prepended. The source IP of this new header is the local tunnel endpoint, and the destination IP is the remote tunnel endpoint's public IP address. The protocol field in this new IP header is set to 47.
5. This newly formed GRE packet is then routed over the transport network (e.g., the internet) towards the remote tunnel endpoint.
6. Intermediate routers in the transport network only examine the outer IP header for routing.
7. When the GRE packet arrives at the egress router (tunnel destination), the router decapsulates it by removing the outer IP header and the GRE header.
8. The original inner packet is then routed to its final destination in the remote network.

Common Use Cases for GRE:

• Connecting non-contiguous networks over an IP backbone.
• Transporting non-IP protocols over an IP network.
• Enabling routing protocols (which often use multicast) to run between sites over an IP network.
• Creating VPNs (though security must be added separately, e.g., with IPSec).
• Forwarding IPv6 traffic over IPv4 networks (and vice-versa).

Understanding IPSec (Internet Protocol Security)

IPSec is a suite of protocols used to secure Internet Protocol (IP) communications by authenticating and encrypting each IP packet of a communication session. It provides confidentiality, data integrity, and origin authentication.

Key IPSec Services:

• Confidentiality (Encryption): Prevents eavesdropping by encrypting the data. Commonly uses ESP (Encapsulating Security Payload).
• Data Integrity: Ensures that data has not been tampered with in transit, often using hash algorithms within ESP or AH (Authentication Header).
• Origin Authentication: Verifies the identity of the sender, provided by AH and ESP.
• Anti-Replay Protection: Prevents attackers from capturing and retransmitting packets.

IPSec Modes of Operation:

• Tunnel Mode: The entire original IP packet (header and payload) is encrypted and/or authenticated, and then encapsulated with a new IP header. This mode is typically used for site-to-site VPNs where gateways are connecting networks. The new IP header has the source and destination of the IPSec tunnel endpoints.
• Transport Mode: Only the IP payload (and ESP/AH headers) is encrypted and/or authenticated. The original IP header is retained and used for routing. This mode is typically used for end-to-end communication between two hosts, or when another tunneling protocol (like GRE) is already providing the outer IP header.

GRE over IPSec

Since GRE does not provide encryption, it is often combined with IPSec to create secure tunnels. This is known as GRE over IPSec. In this scenario, GRE first encapsulates the original data packet, and then this entire GRE packet is encapsulated and protected by IPSec.

Why Combine GRE and IPSec?

• Secure Multicast/Broadcast Traffic: IPSec, by itself, typically does not support multicast or broadcast traffic, which is often required by dynamic routing protocols (OSPF, EIGRP, RIP) or certain applications. GRE can encapsulate this multicast/broadcast traffic into unicast GRE packets, which can then be encrypted by IPSec.
• Support for Non-IP Protocols Securely: GRE can carry non-IP protocols, and IPSec can then secure the GRE tunnel carrying these protocols.
• Simplified VPN Configuration for Complex Routing: Using GRE allows for simpler routing over the VPN. Dynamic routing protocols can run over the GRE tunnel, and the underlying IPSec tunnel simply provides a secure point-to-point link for the GRE traffic.
• Flexibility: GRE provides protocol flexibility, and IPSec adds the necessary security layer.

How GRE over IPSec Works:

1. The original packet (e.g., an OSPF multicast packet or a data packet from a host) arrives at the firewall/router.
2. GRE encapsulates the original packet: [GRE Header | Original IP Header | Original Payload].
3. This GRE packet is then treated as the payload for IPSec.
4. IPSec (typically in Tunnel Mode for site-to-site) encrypts and/or authenticates the GRE packet and adds its own headers (e.g., ESP header) and a new outer IP header: [New (Outer) IP Header | IPSec (ESP) Header | GRE Header | Original IP Header | Original Payload | ESP Trailer | ESP Auth].
5. This final IPSec packet is sent over the public network (e.g., internet) to the peer IPSec gateway.
6. The peer IPSec gateway receives the packet, decrypts it (removes IPSec headers), revealing the GRE packet.
7. The GRE packet is then decapsulated (GRE header removed), revealing the original packet.
8. The original packet is then routed to its final destination.

Essentially, GRE creates a virtual tunnel, and IPSec creates a secure tunnel to protect the GRE traffic.

GRE over IPSec Packet Structure (using IPSec Tunnel Mode with ESP)

GRE over IPSec (Tunnel Mode with ESP) Packet Encapsulation. The GRE packet becomes the payload for IPSec.

When configuring GRE over IPSec on Palo Alto Networks firewalls, you would typically:

1. Configure the IPSec tunnel (Phase 1 and Phase 2 SAs) between the public IP addresses of the firewalls.
2. Configure a GRE tunnel object, specifying local and peer *private or loopback* IP addresses that will be the endpoints of the GRE tunnel itself. These private IPs are often routed *through* the IPSec tunnel.
3. Create tunnel interfaces for both IPSec (if using route-based IPSec, which is common) and GRE.
4. Ensure routing directs the GRE tunnel's source/destination IP traffic over the IPSec tunnel.
5. Ensure routing directs the actual data traffic into the GRE tunnel interface.
6. Configure security policies to allow:
   • IPSec traffic (UDP 500, UDP 4500, ESP) between the public IPs.
   • GRE traffic (protocol 47) between the private/loopback IPs used as GRE endpoints *through* the IPSec tunnel. On Palo Alto, if IPSec is route-based, this might mean allowing traffic to/from the zone containing the IPSec tunnel interface, with the GRE endpoint IPs as source/destination.
   • The actual data traffic through the GRE tunnel interface (between the appropriate zones).

Benefits of GRE over IPSec:

• Secure transmission of multicast and broadcast traffic.
• Secure transmission of non-IP protocols.
• Scalable VPNs with dynamic routing protocols.
• Enhanced data security with layered encryption.
• Maintains data integrity across insecure networks.

Key Configuration Steps for GRE on Palo Alto Networks Firewalls:

• Tunnel Interface: Create a tunnel interface (e.g., tunnel.1 ), assign it an IP address (this will be an endpoint of the GRE tunnel's private network). This interface must be assigned to a virtual router and a security zone.
• GRE Tunnel Object: Define the GRE tunnel object ( Network > GRE Tunnels ), specifying the local physical interface or its IP that will source the GRE packets, and the peer's public IP address as the destination for the outer GRE packet. Link it to the created tunnel interface.
• Routing:
  • Configure static routes or a dynamic routing protocol (like OSPF or BGP) to direct traffic destined for remote private networks *through* the GRE tunnel interface.
  • Ensure the route to the GRE peer's public IP address (the Peer Address in the GRE tunnel object) does *not* go through the GRE tunnel itself to avoid recursive routing.
• Security Policies:
  • Allow traffic intended to pass *through* the GRE tunnel (e.g., from internal zones to the zone of the tunnel interface, and vice-versa, permitting the actual application traffic).
  • A crucial step often missed: If the GRE tunnel terminates on the firewall and the physical ingress interface (receiving the GRE packet from the internet) is in a different zone than the GRE tunnel interface, you must create a security policy to allow the incoming GRE protocol (IP protocol 47). This policy would typically be from the zone of the physical interface (e.g., untrust) to the zone of the tunnel interface (e.g., gre-zone), with the source IP being the public IP of the remote GRE peer, and the destination IP being the public IP of the local firewall.
• NAT (Network Address Translation):
  • If the firewall is behind another NAT device, ensure appropriate static NAT rules (port forwarding for IP Protocol 47) are in place for the firewall's external IP.
  • When sending traffic through GRE to a service like Zscaler or Netskope, you might need to create NAT policies to *exclude* the source IP from being NAT'd if the service needs to see the original source IP.

Common GRE Troubleshooting Steps

1. Verify Tunnel Interface Status

The first step is to check the status of the GRE tunnel interface.

• CLI Command: show interface tunnel. (e.g., show interface tunnel.1 ).
• Expected Output: Look for "tunnel interface state: Up".
• This command also shows statistics for keepalives (if enabled) and sent/received packets/bytes.

2. Check GRE Packet Counters

Verify if GRE packets are being encapsulated and decapsulated successfully.

• CLI Command: show counter global filter value all | match flow_gre .
• Expected Output: The counters flow_gre_tunnel_encap_success and flow_gre_tunnel_decap_success should be increasing when traffic is flowing.
• Look for any error counters like flow_gre_tunnel_decap_err or flow_gre_tunnel_encap_err .

3. Examine System Logs

System logs can provide information about tunnel status changes (up/down events).

• CLI Command: show log system subtype equal gre or show log system | match .
• Look for messages indicating the tunnel is up and not flapping.

4. Verify Routing

Incorrect routing is a common cause of GRE tunnel failures.

• Recursive Routing: This is a frequent issue where the route to the GRE peer's public IP address resolves through the GRE tunnel interface itself. This creates a loop, causing the tunnel to go down or flap.
  • CLI Command to Check: test routing fib-lookup virtual-router ip .
  • If the output shows the next hop is the GRE tunnel interface, it confirms recursive routing.
  • Resolution: Ensure the route to the GRE peer's public IP resolves through a physical interface or another (non-GRE) tunnel, not the GRE tunnel itself. This might involve adding a more specific static route for the peer's public IP.
• Route to Remote Networks: Ensure there's a route (static or dynamic) pointing to the remote internal networks via the GRE tunnel interface.
• CLI Command to Check Routing Table: show routing route .

5. Check Security Policies

• Allowing GRE Protocol (IP Protocol 47): Ensure a security policy allows IP protocol 47 (GRE) from the peer's public IP to the local firewall's public IP if the GRE tunnel terminates on the firewall. The destination zone in this policy should be the zone assigned to the tunnel interface.
• Allowing Traffic *Through* the Tunnel: Verify security policies permit the actual data traffic (e.g., TCP, UDP, ICMP) between the source and destination zones that traverse the GRE tunnel.
• Check traffic logs ( Monitor > Traffic ) for any denied GRE packets or denied application traffic that should be going over the tunnel.

6. GRE Keepalives

GRE keepalives can help detect when the other end of the tunnel is unreachable. However, they can sometimes cause issues. Palo Alto Networks firewalls support GRE keepalives. The keepalive mechanism involves sending GRE packets (often with protocol type 0 in the GRE header) and expecting responses.

• Keepalives are configured under the GRE tunnel object ( Network > GRE Tunnels ).
• Ensure keepalive settings (interval, retries) are compatible if the peer device also uses them.
• In some cases, disabling GRE keepalives has resolved tunnel flapping issues, even when keepalive packets appeared to be exchanged correctly. This might be a last resort if other troubleshooting fails.
• GRE itself does not have a built-in mechanism to bring the tunnel interface down if the peer is unreachable without keepalives; the tunnel interface state often reflects the state of the underlying physical interface and route availability. Keepalives add this liveness detection.

7. MTU and Fragmentation Issues

GRE adds overhead (typically 24 bytes for the outer IP header + GRE header, though the basic GRE header is 4 bytes minimum). This can lead to packets exceeding the MTU of an intermediate link, causing fragmentation or drops.

• CLI Command to Check for Fragmentation: show counter global | match frag or look for flow_fwd_mtu_exceeded .
• Palo Alto Networks firewalls can automatically adjust TCP MSS for traffic going through the tunnel (configurable on the tunnel interface).
• You may need to manually set the MTU on the tunnel interface to a lower value (e.g., 1400, 1476) and/or configure TCP MSS clamping on the firewall or the peer device.
  • PAN-OS path for MTU on tunnel interface: Network > Interfaces > Tunnel > (select interface) > Advanced > Other info > MTU . The default is often 1500.

8. NAT Traversal and Intermediate Devices

• If the Palo Alto Networks firewall or the GRE peer is behind a NAT device, ensure that IP protocol 47 is permitted and correctly NATted (port forwarding for protocol 47 to the firewall's GRE endpoint IP).
• Some NAT devices might have issues with ESP (if using GRE over IPSec) because it's a Layer 3 protocol without port numbers, potentially leading to silent drops. While this is more specific to IPSec, ensure no intermediate device is unintentionally blocking GRE.
• If you are using a service that requires the original source IP (like Zscaler or Netskope), ensure NAT policies are configured to *exclude* GRE-bound traffic from source NAT.

9. Ping and Traffic Tests

• Ping Across the Tunnel: Use the ping source host command from the firewall's CLI to test basic connectivity over the tunnel interface.
• Test traffic from a host in the local network to a host in the remote network.

10. Packet Captures

Packet captures are invaluable for in-depth analysis.

• Capture packets on the physical interface that carries the GRE traffic.
• Capture packets on the tunnel interface itself.
• Filters can be applied based on IP addresses, protocol (47 for GRE), etc.
• Use tcpdump or the GUI packet capture feature ( Monitor > Packet Capture ).
• CLI for basic tcpdump-like captures (filtered):

  Setting up a filter:

  > debug dataplane packet-diag set filter match source destination protocol 47

  Enabling the filter:

  > debug dataplane packet-diag set filter on

  Viewing captured packets (dp-log):

  > less dp-log pan_packet_diag.log

  Or view live captures (can be resource intensive):

  > debug dataplane packet-diag set capture stage FWD layer7 no file off terminal yes filter-id
> debug dataplane packet-diag set capture on

  (Remember to turn off debugging after use: debug dataplane packet-diag set capture off , debug dataplane packet-diag set filter off )

Troubleshooting GRE over IPSec

When GRE is encapsulated within an IPSec tunnel, you must troubleshoot both layers. An issue in the IPSec tunnel will prevent the GRE tunnel from functioning.

1. Verify IPSec Tunnel Status

Ensure the IPSec tunnel (Phase 1 and Phase 2) is up and stable.

• GUI: Network > IPSec Tunnels . Check the status indicators.
• CLI Commands:
  • show vpn ike-sa gateway (for Phase 1)
  • show vpn ipsec-sa tunnel (for Phase 2)
  • A more general command to see tunnel status is also test vpn ike-sa gateway and test vpn ipsec-sa tunnel .
• Common IPSec issues include mismatched pre-shared keys, crypto profiles, proxy IDs (if policy-based), or peer IP addresses.

2. Check IPSec Packet Encapsulation/Decapsulation

• CLI Command: show vpn flow tunnel-id
• Look for encrypt and decrypt packet counts increasing.
• Check traffic logs for ESP (protocol 50) or UDP 4500 (NAT-T) traffic being allowed by security policies if the IPSec tunnel terminates on the firewall.

3. IPSec Tunnel Configuration for GRE

When configuring the IPSec tunnel on the Palo Alto Networks firewall for GRE over IPSec, the traffic selectors (Proxy IDs for policy-based VPN or routing for route-based VPN) must be configured to carry the GRE traffic.

• Route-Based IPSec (Recommended): This is generally preferred.
  • The IPSec tunnel interface is created and assigned to a zone and virtual router.
  • A static route (or dynamic routing) is configured to route traffic destined for the remote GRE tunnel endpoint's *private/internal* IP address via the IPSec tunnel interface.
  • The "interesting traffic" for the IPSec tunnel effectively becomes the GRE packets exchanged between the GRE tunnel endpoints.
• Policy-Based IPSec: If used, the crypto ACL (proxy ID) must explicitly permit IP protocol 47 between the public IP addresses of the GRE tunnel source and destination (which are also the IPSec tunnel endpoints in this specific common setup).
  • Local Proxy ID: Local public IP, Remote public IP, Protocol 47
  • Remote Proxy ID: Remote public IP, Local public IP, Protocol 47
  However, it's more common and flexible to use route-based IPSec and route the GRE packets (which will have their own private source/destination IPs for the GRE tunnel itself, and public IPs for the outer GRE delivery header) through the IPSec tunnel. The IPSec proxy IDs in a route-based setup would then cover the private IPs of the GRE tunnel interface endpoints if the IPSec tunnel is specifically for these.
• On some platforms, there might be a specific option to "enable GRE encapsulation" within the IPSec tunnel configuration, but on Palo Alto Networks, proper routing and security policies for GRE traffic over the IPSec tunnel interface are key. (Note: Juniper SRX has specific considerations where GRE and IPSec endpoints might need to differ unless using loopbacks. Palo Alto's approach is generally more straightforward with routing.)

4. Routing for GRE over IPSec

• The route for the remote GRE tunnel endpoint (the *private IP* assigned to the remote end of the GRE tunnel interface) must point towards the local GRE tunnel interface.
• The route for the GRE peer's *public IP address* (the actual destination of the outer IPSec packets) must point out a physical interface towards the internet, not the GRE or IPSec tunnel itself.
• Traffic destined for the remote private network (behind the peer's GRE endpoint) should be routed into the local GRE tunnel interface.
• The IPSec tunnel interface itself (if route-based) needs routes pointing to the networks that will be secured by IPSec. In the case of GRE over IPSec, this typically means routing the GRE packets (identified by the GRE tunnel endpoints' IPs) through the IPSec tunnel interface.
• Be extremely cautious of recursive routing.

5. MTU Considerations for GRE over IPSec

IPSec adds its own overhead in addition to the GRE overhead. This further reduces the available MTU for the payload.

• The typical IPSec overhead (ESP) can be around 52-76 bytes (variable based on encryption/authentication algorithms and mode).
• Combined with GRE (e.g., 24 bytes), the total overhead can be significant (e.g., 76+ bytes).
• You will likely need to set a lower MTU on the GRE tunnel interface (e.g., 1350-1400, sometimes as low as 1280) and configure TCP MSS clamping accordingly.

6. NAT Traversal (NAT-T) for IPSec

• If either IPSec peer is behind a NAT device, NAT-T (which encapsulates ESP in UDP port 4500) should be enabled on the IKE Gateway configuration.
• Ensure that UDP 500 (IKE) and UDP 4500 (NAT-T) are allowed by any intermediate firewalls/ACLs and by the security policies on the Palo Alto Networks firewall itself, destined for the firewall.

Other Important Information and CLI Commands

General Troubleshooting Commands:

• show system info : General system status.
• show session info : Overview of current session count, packet rate, etc.
• show session id : Details for a specific session.
• show counter global filter severity drop : Shows packet drop counters. Use delta yes to see changes since the last command.
• show running resource-monitor : Check CPU and buffer utilization. High dataplane CPU can impact tunnel performance.
• less mp-log (e.g., less mp-log ikemgr.log , less mp-log pan_system_event.log ): View management plane logs.
• less dp-log (e.g., less dp-log devsrv.log ): View data plane logs.

Recursive Routing Fixes:

• Some PAN-OS versions have specific fixes for GRE recursive routing, especially in HA scenarios or during commits. Ensure you are on a recommended release if encountering persistent issues. Recursive routing for the GRE peer's *public IP* is a common misconfiguration.

Performance:

• GRE traffic processing might have different performance characteristics than pure IPSec on Palo Alto Networks firewalls. IPSec encryption/decryption is often hardware-offloaded. While GRE encapsulation/decapsulation is generally efficient, the overall throughput can be affected by various factors including CPU load if specific features interact with GRE traffic in software.
• If experiencing low throughput, check for high CPU, packet buffer issues, or fragmentation.

Specific Scenarios:

• Connecting to Cloud Services (Zscaler, Netskope): These often involve GRE tunnels, sometimes over IPSec. Pay close attention to NAT exclusion rules and ensure the service provider's public IP ranges are correctly routed and permitted. These services typically validate the source IP of the GRE tunnel.
• Dynamic Routing over GRE/IPSec: Ensure routing protocol adjacencies form correctly over the tunnel. Mismatched BGP ASNs or incorrect BGP peering IPs can cause issues. Asymmetric routing might also be a factor to consider with some cloud providers.

Considerations:

• Order of Operations: When troubleshooting GRE over IPSec, it's generally advisable to ensure the IPSec tunnel is stable first, then verify GRE.
• Intermediate Firewalls/Routers: Any device in the path between the GRE/IPSec peers must permit IP protocol 47 (GRE) and ESP (IP protocol 50) or UDP 4500/500 (for IPSec).
• Minimum PAN-OS Version: GRE tunnel functionality was introduced in PAN-OS 9.0.

By systematically checking these areas, you can effectively diagnose and resolve most GRE and GRE over IPSec tunnel issues on Palo Alto Networks firewalls.

Interactive Quiz: GRE and GRE over IPSec