Custom Service Routes in Palo Alto Networks Firewalls: Per-VSYS vs. Destination-Based

1. Understanding Firewall-Sourced Traffic in PAN-OS

Before diving into specific service route configurations, it's important to understand that "service routes" pertain to traffic generated by the Palo Alto Networks firewall itself . This is distinct from transit traffic (traffic passing through the firewall from one network segment to another). Services that generate such traffic include:

• DNS (Domain Name System): For resolving hostnames used in FQDN address objects, logs, update servers, etc.
• NTP (Network Time Protocol): For time synchronization, essential for accurate logging and certificate validation.
• Syslog: For sending logs to external syslog servers.
• SNMP (Simple Network Management Protocol): For sending traps or responding to queries from NMS.
• Panorama: For communication between the firewall and a Panorama management server.
• Palo Alto Networks Updates: For downloading software updates, content updates (App-ID, Threat-ID, Antivirus, WildFire), etc.
• WildFire: For submitting samples and retrieving verdicts from the WildFire cloud.
• User-ID Agent: When the firewall acts as a User-ID agent or connects to one.
• Cortex Data Lake (CDL): For sending logs to the CDL.
• Email Alerts: For sending email notifications.
• GlobalProtect Services: Certain communications related to GlobalProtect portal/gateway functionality.
• External Dynamic Lists (EDL): For fetching updates to EDLs.

By default, the firewall often uses its management interface and its associated routing table for these services. However, this default behavior may not be suitable for all network designs or security requirements, necessitating custom service routes.

2. Per-Virtual System (Per-VSYS) Service Routes

Palo Alto Networks firewalls support multiple Virtual Systems (VSYS), allowing a single physical firewall to be partitioned into multiple logical firewalls. Each VSYS can have its own set of administrators, security policies, interfaces, and routing tables (Virtual Routers). Per-VSYS service routes enable administrators to define distinct source interfaces and IP addresses for predefined services on a per-VSYS basis.

2.1. Key Characteristics and Purpose

• Scope: Specific to an individual Virtual System. If no per-VSYS route is configured for a service within a VSYS, it typically inherits the global service route settings or uses its default routing behavior.
• Granularity: Allows different VSYS instances on the same physical firewall to use different source interfaces (and thus, potentially different network paths or source IP addresses) for the same service (e.g., VSYS1 uses DNS server A via interface X, while VSYS2 uses DNS server B via interface Y).
• Applicable Services: Applies to a predefined list of services configurable within PAN-OS (e.g., DNS, NTP, Syslog, Panorama, WildFire, Palo Alto Networks Updates, etc.).
• Routing Dependency: The firewall determines the actual egress interface for a per-VSYS service route based on the routing table of the Virtual Router associated with the VSYS and the chosen source interface. The specified source interface in the service route configuration must be part of a Virtual Router accessible by that VSYS.

2.2. Common Use Cases

• Managed Service Providers (MSPs): MSPs hosting multiple tenants (each as a VSYS) on a single firewall can ensure each tenant's firewall-generated traffic (e.g., syslog to their specific SIEM, DNS queries to their specific resolvers) is sourced correctly and uses their designated network paths.
• Large Enterprises: Organizations with distinct departments or security zones managed as separate VSYS can isolate their operational traffic. For example, a PCI-compliant VSYS might need to send logs to a specific, secured syslog server using a dedicated interface, different from other VSYS.
• Network Segmentation: Ensuring firewall-sourced traffic from a particular VSYS uses an interface within that VSYS's security domain, rather than a global management interface.

2.3. Configuration in PAN-OS

Per-VSYS service routes are configured in the PAN-OS WebGUI:

1. Navigate to Device > Setup > Services .
2. In the Service Route Configuration section, click Customize .
3. A "Service Route Configuration" window will appear. From the Virtual System dropdown menu at the top, select the specific VSYS for which you want to customize service routes (e.g., `vsys1`, `vsys2`). To configure global settings that VSYS can inherit, leave it as "Global".
4. Once a VSYS is selected, you can then customize the settings for each service listed (e.g., DNS, NTP, Palo Alto Networks Update Server). For each service, you can typically specify:
   • Source Interface: The layer 3 interface (e.g., `ethernet1/1`, `tunnel.5`, `ae1.100`) the firewall should use to source traffic for this service from this VSYS.
   • Source Address: The specific IP address on the selected source interface to use. Usually, you can select "Use Interface IP Address" or specify a different IP if the interface has multiple IPs (e.g., for specific services using secondary IPs).
5. Click OK to save the VSYS-specific service route settings.
6. Click OK again on the Setup > Services page.
7. Commit the changes to the firewall.

CLI Configuration Example (Conceptual):

To set a per-VSYS DNS service route for `vsys1` to use `ethernet1/5` with its primary IP:

set deviceconfig system service route vsys vsys1 service dns source-interface ethernet1/5 source-address interface-ip
 

To view configured per-VSYS service routes:

show deviceconfig system service route vsys vsys1 service dns
show deviceconfig system service route vsys all
 

2.4. Impact on Routing and Potential for Asymmetry

When a per-VSYS service route is configured, the firewall will attempt to send traffic for that service from the specified source interface and IP. The actual path taken depends on the routing table of the Virtual Router to which that source interface belongs. It's crucial that this Virtual Router has a route to the destination service.

A common pitfall is asymmetric routing. If the per-VSYS service route forces traffic out via one path, but return traffic from the service comes back via a different path (e.g., to a different interface on the firewall or via a different VSYS's routing domain), it can lead to session failures or unexpected behavior. Careful planning of both service routes and the underlying network routing is essential.

3. Destination-Based Service Routes

Destination-Based Service Routes provide a more direct way to control routing for firewall-sourced traffic by specifying the egress interface and next-hop based on the destination IP address of the service. These routes act as a form of Policy-Based Forwarding (PBF) specifically for traffic originating from the firewall itself.

3.1. Key Characteristics and Purpose

• Scope: Global. Destination-based service routes apply to all VSYS on the firewall. They cannot be configured on a per-VSYS basis and cannot be overridden at the VSYS level.
• Granularity: Allows precise routing decisions for firewall-sourced traffic to specific IP addresses or subnets.
• Overrides FIB: These routes take precedence over entries in the firewall's Forwarding Information Base (FIB) derived from the virtual router's routing table for matching firewall-generated traffic.
• Applicable Services: Can be used for any service based on its destination IP address, whether it's a predefined service or a custom internal service not listed in the standard service route options.

3.2. Common Use Cases

• Custom or Non-Standard Services: Forcing traffic to an internal update server, a specific log collector, or any other service not explicitly listed under the predefined per-VSYS service route options.
• Specific Path Selection: When the firewall has multiple uplinks (e.g., multiple internet connections, dedicated management network) and certain firewall-sourced traffic (e.g., to a specific Panorama IP, a particular NTP server) must use a specific path, regardless of the default route.
• VPN Tunnel Routing: Ensuring firewall-sourced management traffic (e.g., syslog, SNMP traps) to a central NMS/SIEM is routed over a specific VPN tunnel.
• Overriding Default Management Interface Behavior: If the default management interface route is not desired for certain destinations.

3.3. Configuration in PAN-OS

Destination-Based Service Routes are configured in the PAN-OS WebGUI:

1. Navigate to Device > Setup > Services .
2. In the Service Route Configuration section, click Customize .
3. A "Service Route Configuration" window will appear. Select the Destination tab.
4. Click Add to create a new destination-based service route.
5. Configure the following parameters:
   • Name: A descriptive name for the route.
   • Destination: The destination IP address or network (e.g., `192.168.100.50/32` or `10.20.0.0/16`).
   • Interface: The egress interface the firewall should use (e.g., `ethernet1/2`, `tunnel.10`).
   • Source Address: The source IP address the firewall should use. This must be an IP address configured on the selected egress interface.
   • Next Hop Type: Typically "IP Address".
   • Next Hop Value: The IP address of the next-hop gateway.
   • Metric: Optional, default is 10. Lower metrics are preferred if multiple destination-based routes match.
6. Click OK to save the route.
7. Click OK again on the Setup > Services page.
8. Commit the changes to the firewall.

CLI Configuration Example:

To route firewall-sourced traffic destined for `172.16.50.100` via `ethernet1/4` using source IP `10.0.4.5` and next-hop `10.0.4.1`:

set deviceconfig system service route destination 172.16.50.100/32 interface ethernet1/4 nexthop 10.0.4.1 source-address 10.0.4.5
 

To view configured destination-based service routes:

show deviceconfig system service route destination
 

4. Service Route Precedence and Decision Logic

Understanding the order in which PAN-OS evaluates and applies service routes is critical for predicting behavior and troubleshooting issues. The general precedence is as follows for traffic originating from the firewall:

1. Destination-Based Service Route: If the destination IP of the firewall-sourced traffic matches a configured destination-based service route, this route is used. It has the highest precedence and applies globally.
2. Per-VSYS Customized Service Route: If no destination-based route matches, and the specific service (e.g., DNS, NTP) has a custom configuration for the originating VSYS (or for "Global" if the VSYS inherits), that VSYS-specific (or global customized) source interface/IP is used. The firewall then uses the routing table (FIB) of the Virtual Router associated with that source interface to forward the packet.
3. Global Default Service Settings (Source Interface): If neither of the above applies, PAN-OS checks the global settings for that specific service (e.g., the primary DNS server IP entered under `Device > Setup > Services`). These global settings may also allow specifying a source interface. If a source interface is specified here, the firewall uses that interface and the routing table of the Virtual Router that interface belongs to.
4. Default System Routing (e.g., Management Interface): If no specific service route (destination-based, per-VSYS custom, or global custom with source interface) dictates the source, the firewall typically falls back to using its management interface and its associated routing table (often the default virtual router, `vr_default` or `management`) for management plane-originated traffic. For services initiated from a dataplane interface (like some User-ID agent communications from a dataplane IP), it would use that dataplane interface's Virtual Router.

Diagram illustrating the decision logic for service route selection in Palo Alto Networks firewalls. Destination-based routes are checked first, followed by per-VSYS (or global customized) service settings, and finally default system routing behavior.

5. Comparison Table: Per-VSYS vs. Destination-Based Service Routes

6. Practical Scenarios and Examples

Scenario 1: Per-VSYS DNS for Multi-Tenancy

An MSP uses a Palo Alto Networks firewall with `vsys1` for Tenant A and `vsys2` for Tenant B.

• Tenant A (vsys1) must use their internal DNS server `10.100.1.53` via their dedicated interface `ethernet1/1` (IP `10.100.1.1`).
• Tenant B (vsys2) must use public DNS `8.8.8.8` via a shared internet interface `ethernet1/5` (IP `203.0.113.10`).

Configuration for Tenant A (vsys1):

1. Navigate to Device > Setup > Services > Customize.
2. Select `vsys1` from the Virtual System dropdown.
3. For the DNS service row, set Source Interface to `ethernet1/1` and Source Address to `10.100.1.1` (or "Use Interface IP Address").
4. Ensure `vsys1` has DNS server `10.100.1.53` configured (under `Device > Setup > Services` while `vsys1` is the selected context, or via Panorama template for `vsys1`).

A similar configuration would be done for `vsys2` pointing to `ethernet1/5` and `203.0.113.10` for its DNS service, with `8.8.8.8` as its configured DNS server.

Per-VSYS service route configuration ensuring Tenant A and Tenant B use their designated DNS servers and source interfaces for firewall-originated DNS queries.

Scenario 2: Destination-Based Route for Syslog over VPN

A firewall needs to send all its syslog messages to a central SIEM server at `192.168.200.75`. This SIEM is reachable via an IPsec tunnel interface `tunnel.12`, and the next-hop router on the other side of the tunnel is `10.254.0.1`. The firewall's source IP for this traffic should be `10.254.0.2` (the IP of `tunnel.12`).

Configuration:

1. Navigate to Device > Setup > Services > Customize.
2. Select the Destination tab.
3. Click Add.
   • Name: `Syslog_to_SIEM_via_VPN`
   • Destination: `192.168.200.75/32`
   • Interface: `tunnel.12`
   • Source Address: `10.254.0.2`
   • Next Hop Type: IP Address
   • Next Hop Value: `10.254.0.1`

This ensures any syslog traffic generated by the firewall (regardless of VSYS, as it's global) to `192.168.200.75` will use `tunnel.12`.

Destination-based service route forcing firewall-generated syslog traffic to a specific SIEM server over a designated VPN tunnel interface.

7. Verification and Troubleshooting

Properly verifying and troubleshooting service routes involves checking configurations, observing traffic, and understanding logs.

7.1. CLI Verification Commands

• View Destination-Based Service Routes:

  show deviceconfig system service route destination

• View Per-VSYS Service Route for a specific service and VSYS:

  show deviceconfig system service route vsys <vsys_name> service <service_name>

  Example: `show deviceconfig system service route vsys vsys1 service dns`
• View all Per-VSYS Service Routes:

  show deviceconfig system service route vsys all

• Check active sessions using service routes:

  show session all filter-service-route yes

  This command is invaluable for seeing if traffic is actually using a configured service route.
• Test DNS resolution (uses service routes if applicable):

  test dns-proxy dns-server <dns_server_ip> host <hostname_to_resolve> vsys <vsys_id>

  Replace ` ` with `vsys1`, `vsys2`, etc.
• Test NTP synchronization:

  test ntp-service host <ntp_server_ip_or_hostname>

  If you have per-VSYS NTP, the sourcing will depend on the VSYS context from which you run the command (typically implies global or the context of your management session). For VSYS specific test, generate actual NTP traffic from the VSYS by having it configured and observe sessions.
• General routing lookup (doesn't directly show service route override but helps understand underlying FIB):

  test routing fib-lookup virtual-router <vr_name> ip <destination_ip>

7.2. Troubleshooting Steps

1. Verify Configuration: Double-check the service route settings in the GUI (`Device > Setup > Services > Customize`) and via CLI. Ensure correct interfaces, IPs, and next-hops.
2. Check Precedence: If behavior is unexpected, review the service route precedence. A destination-based route might be overriding a per-VSYS setting, or vice-versa depending on what is configured.
3. Interface and VR State: Ensure the chosen source interface is up, has the correct IP, and is part of the intended Virtual Router. Verify the VR has routes to the destination (especially for per-VSYS routes).
4. Security Policies: Firewall-sourced traffic is still subject to security policies. Ensure policies allow the traffic from the zone of the source interface to the zone of the destination. Check traffic logs for denies.
   Gotcha! (Palo Alto Networks): A common oversight is forgetting that firewall-generated traffic, even with service routes, must be permitted by security policies. If the source interface is in 'trust' and the destination is in 'untrust', a policy allowing 'trust' to 'untrust' for that specific application/service is needed.
5. Packet Captures: Use packet captures on the expected source interface and/or management interface to see if traffic is leaving with the correct source IP and going to the correct destination.

   # Example: Capture on ethernet1/1 for NTP traffic
   debug dataplane packet-diag set capture stage firewall file pcap_ntp.pcap interface ethernet1/1
   debug dataplane packet-diag set capture filter match destination-port 123 protocol 17
   debug dataplane packet-diag set capture on
   # ... let traffic flow ...
   debug dataplane packet-diag set capture off
   # Then view pcap_ntp.pcap
      

6. System Logs: Check system logs for any errors related to the specific service (e.g., DNS resolution failures, NTP sync issues, Panorama connection problems).
7. Isolate the Issue: Temporarily remove or disable the custom service route. Does the service work with default behavior? This helps pinpoint if the service route itself is the problem.

A systematic approach to troubleshooting service route issues on Palo Alto Networks firewalls.

8. Best Practices for Service Route Configuration

• Use Per-VSYS for Logical Separation: When different virtual systems genuinely require distinct source addressing or paths for standard firewall services, per-VSYS routes offer clean separation.
• Employ Destination-Based Routes Selectively: Use destination-based routes for specific overrides, custom services, or when a global routing decision for firewall traffic is needed. Avoid overusing them, as they can make routing harder to follow if not well-documented.
• Document Extensively: Clearly document all custom service routes, especially destination-based ones, including the reason for their implementation. This is vital for future troubleshooting and network changes.
• Ensure Underlying Routing is Sound: Service routes are an overlay. The Virtual Routers must have correct and stable routing for the paths service routes intend to use.
• Always Consider Security Policies: Remember that firewall-sourced traffic needs to be permitted by security policies. Design policies accordingly.
• Regular Audits: Periodically review service route configurations to ensure they are still necessary and correctly configured, especially after network topology changes or VSYS modifications.
• Leverage Panorama: For managing multiple firewalls, use Panorama templates and template stacks to consistently deploy and manage service route configurations. This is especially helpful for global settings or standardized per-VSYS configurations across many devices.
• Test Changes: After implementing or modifying service routes, thoroughly test the affected services to confirm they are working as expected.