Palo Alto Networks URL Filtering Solution

Advanced URL Filtering (preceded by URL Filtering) is a subscription service that protects your network and its users against malicious and evasive web based threats—both known and unknown . The subscription provides the same functionality as URL Filtering—granular URL filtering control, visibility into user web activity, safe search enforcement, and credential phishing prevention—with the addition of full web content inspection using an inline machine learning based web security engine . The inline web security engine enables real time analysis and categorization of URLs that are not present in PAN DB , Palo Alto Networks cloud based URL database. Then, the engine determines the action the firewall takes.

Advanced URL Filtering protects against malicious URLs that are updated or introduced before PAN DB has analyzed and added them to the database. With Advanced URL Filtering enabled, URL requests are:

• Analyzed in realtime using the cloud based Advanced URL Filtering detection modules. This is in addition to URLs being compared to entries in PAN DB. The ML powered web protection engine detects and blocks the malicious websites that PAN DB cannot.
• Inspected for phishing and malicious JavaScript using local inline categorization , a firewall based analysis solution, which can block unknown malicious web pages in real time.

Advanced URL Filtering licenses are supported on next generation firewalls running PAN OS 9.1 and later . You can manage URL filtering features on the PAN OS and Panorama web interface, Prisma Access, and Cloud NGFW platforms. However, some URL filtering features are not available on each platform.

If network security requirements in your enterprise prohibit the firewalls from directly accessing the Internet, Palo Alto Networks provides an offline URL filtering solution with the PAN DB private cloud . You can deploy a PAN DB private cloud on one or more M 600 appliances that function as PAN DB servers within your network; however, the private cloud does not support any of the cloud based URL analysis features provided by the Advanced URL Filtering solution.

Legacy URL Filtering Subscription

URL Filtering enforces policy rules for websites stored in your local cache or PAN DB. When a user requests a website, the firewall checks the local cache for its URL category. If the website isn't in the cache, the firewall queries PAN DB to decide which action to apply. As a result, attackers are better able to launch precision attack campaigns using URLs that aren't present in the cloud based database .

Legacy subscription holders can continue using their URL filtering deployment until the end of the license term.

Comparison: Advanced URL Filtering vs. Legacy URL Filtering

URL Filtering Support

Advanced URL Filtering features are available on various Palo Alto Networks platforms. Feature support depends on the platform and type of URL filtering license. Features that are only available with an Advanced URL Filtering license are indicated by an Advanced URL Filtering label.

The following table shows the compatibility of Advanced URL Filtering features with each Palo Alto Networks platform that supports URL Filtering.

How Advanced URL Filtering Works

Advanced URL Filtering classifies websites based on site content, features, and safety . A URL can have up to four URL categories that indicate the likelihood that the site will expose you to threats. As PAN DB categorizes sites, firewalls with Advanced URL Filtering enabled can leverage that knowledge to enforce your organization’s security policies. In addition to the protection offered by PAN DB, Advanced URL Filtering provides realtime analysis using machine learning (ML) to defend against new and unknown threats. This provides protection against malicious URLs that are updated or introduced before URL filtering databases have an opportunity to analyze and add the content.

Websites that indicate the presence of certain advanced threats are additionally processed though a cloud based inline deep learning system , using detectors and analyzers that complement the ML models. Deep learning detectors can process larger data sets and identify complex malicious patterns. When Advanced URL Filtering receives HTTP response data from the firewall upon receipt of a suspicious web request, the data is further analyzed through the deep learning detectors and provides inline protection against evasive zero day web attacks . This includes cloaked websites, multi step attacks, CAPTCHA challenges, and previously unseen one time use URLs. Because evasive malicious websites are in a constant state of flux, the detectors and analyzers are updated and deployed automatically as Palo Alto Networks threat researchers improve the detection logic, all without requiring administrator updates.

Diagram illustrating how Advanced URL Filtering uses both the PAN DB cloud database and real time analysis mechanisms like inline ML and deep learning to categorize and protect against web threats.

When a user requests a web page, the firewall queries user added exceptions and PAN DB for the site’s risk category. PAN DB uses URL information from Unit 42, WildFire, passive DNS, Palo Alto Networks telemetry data, data from the Cyber Threat Alliance , and applies various analyzers to determine the category. If the URL displays risky or malicious characteristics, the web payload data is also submitted to Advanced URL Filtering in the cloud for real time analysis and generates additional analysis data. The resulting risk category is then retrieved by the firewall and is used to enforce web access rules.

This diagram illustrates the typical URL lookup process on a Palo Alto Networks firewall with URL Filtering. The firewall checks local cache, custom lists (Custom URL Categories, EDLs), and finally queries the cloud-based PAN DB database. Advanced URL Filtering adds real time analysis steps within or alongside the cloud query for unknown threats.

Comparison: PAN DB vs. Advanced URL Filtering

The firewall caches site categorization information for new entries to enable fast retrieval for subsequent requests, while it removes URLs that users have not accessed recently so that it accurately reflects the traffic in your network. Checks built into PAN DB cloud queries ensure that the firewall receives the latest URL categorization information. If you do not have Internet connectivity or an active URL filtering license, no queries are made to PAN DB.

Screenshot illustrating the order of precedence for URL category lookup: Custom URL Categories > External Dynamic Lists > Predefined URL Categories.

The firewall determines a website’s URL category by comparing it to entries in 1) custom URL categories , 2) external dynamic lists (EDLs) , and 3) predefined URL categories , in order of precedence .

Firewalls configured to analyze URLs in realtime using machine learning on the dataplane provides an additional layer of security against phishing websites and JavaScript exploits. To keep up with the latest changes, local inline categorization ML models are added or updated via content releases .

When the firewall checks PAN DB for a URL, it also looks for critical updates, such as URLs that previously qualified as benign but are now malicious.

If you believe PAN DB has incorrectly categorized a site, you can submit a change request in your browser through Test A Site or directly from the firewall logs.

Did you know?

Technically, the firewall caches URLs on both the management plane and the dataplane:

• PAN OS 9.0+ do not download PAN DB seed databases. The cache is populated as URL queries are made.
• The management plane holds more URLs and communicates directly with PAN DB.
• The dataplane holds fewer URLs and receives information from the management plane. The lookup order is: URL category exception lists (custom URL categories, EDLs) > dataplane cache > management plane cache > PAN DB.

URL Filtering Profiles

URL Filtering profiles define how the firewall handles traffic to specific URL categories . A URL Filtering profile is a collection of URL filtering controls that you apply to individual Security policy rules that allow access to the internet. You can configure site access for URL categories, allow or disallow user credential submissions, enable safe search enforcement, and various other settings. To enforce the actions defined in a URL Filtering profile, apply the profile to Security policy rules .

The firewall comes with a default profile that blocks threat-prone categories. You can use the default profile, clone it, or add a new one. By default, site access is set to allow when creating a basic profile, meaning traffic is allowed but not logged unless the action is changed to 'alert' .

Create a best practice URL Filtering profile to ensure protection against URLs observed hosting malware or exploitive content.

URL Filtering Profile Actions

In a URL Filtering profile, you can define Site Access for URL categories, allow or disallow User Credential Submissions , and enable safe search enforcement .

For categories that you alert on, instead of block, you can strictly control how users interact with site content. For example, give users access to resources but take precautions:

• Follow Anti Spyware, Vulnerability Protection, and File Blocking best practices (e.g., block dangerous file types, obfuscated JavaScript).
• Target decryption based on URL category (e.g., decrypt high risk/medium risk sites).
• Display a response page to warn users when visiting high risk/medium risk sites.
• Prevent credential phishing by blocking corporate credential submissions to high risk/medium risk sites.

Mermaid Sequence Diagram: URL Filtering Profile Application

Predefined URL Categories

PAN DB's predefined categories classify websites based on content and purpose (e.g., Adult, Social Networking, Business and Economy). Some are security focused, indicating potential risk or malicious intent.

Security Focused URL Categories

PAN DB automatically assigns risk categories (high, medium, low) to URLs that are not or are no longer classified as explicitly malicious. High and medium risk sites are associated with potential threats. Malicious URL categories (e.g., malware, phishing, command and control) are blocked by default (except Private IP Addresses).

Blocking malicious categories is strongly recommended. For high/medium risk categories, you might choose to alert, use continue pages, decrypt, inspect, and limit interactions (e.g., block credential submissions, risky file downloads).

Note the default actions for malicious categories: Command and Control, Grayware, Malware, Phishing, Ransomware, Scanning Activity are blocked by default . Private IP Addresses are allowed by default .

URL Filtering Profiles

URL Filtering profiles define how the firewall handles traffic to specific URL categories . A URL Filtering profile is a collection of URL filtering controls that you apply to individual Security policy rules that allow access to the internet. You can configure site access for URL categories, allow or disallow user credential submissions, enable safe search enforcement, and various other settings. To enforce the actions defined in a URL Filtering profile, apply the profile to Security policy rules .

The firewall comes with a default profile that blocks threat-prone categories. You can use the default profile, clone it, or add a new one. By default, site access is set to allow when creating a basic profile, meaning traffic is allowed but not logged unless the action is changed to 'alert' .

Create a best practice URL Filtering profile to ensure protection against URLs observed hosting malware or exploitive content.

URL Filtering Profile Actions

In a URL Filtering profile, you can define Site Access for URL categories, allow or disallow User Credential Submissions , and enable safe search enforcement .

For categories that you alert on, instead of block, you can strictly control how users interact with site content. For example, give users access to resources but take precautions:

• Follow Anti Spyware, Vulnerability Protection, and File Blocking best practices (e.g., block dangerous file types, obfuscated JavaScript).
• Target decryption based on URL category (e.g., decrypt high risk/medium risk sites).
• Display a response page to warn users when visiting high risk/medium risk sites.
• Prevent credential phishing by blocking corporate credential submissions to high risk/medium risk sites.

URL Categories

Palo Alto Networks categorizes websites based on their content, features, and safety into URL categories. PAN DB assigns up to four categories, including risk categories (high, medium, low) .

URL categories enable category based filtering and granular policy control. You can use them in URL Filtering profiles (attached to Security policy rules) or as match criteria in Security policy rules (e.g., for decryption policy).

Check categories via Test A Site. Submit change requests for incorrect categorization (cannot request changes for risk categories, insufficient content, or newly registered domains).

Custom URL Categories

Create custom URL categories to define exceptions to category based enforcement or define a new category from multiple existing ones.

• URL List: Specify a list of URLs to enforce independently or group as a custom category ( block/allow lists ). Use URL list guidelines (wildcards * and ^, token separators, case insensitive).
• Category Match: Define a new category for sites matching *all* selected PAN DB categories.

PAN DB evaluates URLs against custom URL categories before external dynamic lists and predefined categories . The strictest action among matching policies applies.

This diagram illustrates the order of precedence for URL category evaluation on a Palo Alto Networks firewall. Custom URL Categories are checked first, followed by External Dynamic Lists, then Predefined Categories , and finally realtime Detection mechanisms (part of ATP/Advanced URL Filtering) if no prior match is found or for specific checks like malicious content analysis. The first match or highest risk verdict determines the category used for policy enforcement.

Predefined URL Categories

PAN DB's predefined categories classify websites based on content and purpose (e.g., Adult, Social Networking, Business and Economy). Some are security focused, indicating potential risk or malicious intent.

Security Focused URL Categories

PAN DB automatically assigns risk categories (high, medium, low) to URLs that are not or are no longer classified as explicitly malicious. High and medium risk sites are associated with potential threats. Malicious URL categories (e.g., malware, phishing, command and control) are blocked by default (except Private IP Addresses).

Blocking malicious categories is strongly recommended. For high/medium risk categories, you might choose to alert, use continue pages, decrypt, inspect, and limit interactions (e.g., block credential submissions, risky file downloads).

Note the default actions for malicious categories: Command and Control, Grayware, Malware, Phishing, Ransomware, Scanning Activity are blocked by default . Private IP Addresses are allowed by default .

Configure URL Filtering

To configure URL Filtering, you need an active URL Filtering license . Configuration involves creating URL Filtering profiles and applying them to Security policy rules . Best practices are recommended for an effective deployment.

Activate Advanced URL Filtering License

The Advanced URL Filtering license includes access to PAN DB. If the license expires, all URL filtering functions cease. Obtain and install the license via the Customer Support Portal. Download and install the latest Applications and Threats content release (8390 6607+ for realtime detection category). Schedule dynamic updates .

PAN OS & Panorama Configuration Steps Overview:

1. Obtain and install Advanced URL Filtering license (Device > Licenses). Verify activation date.

   Screenshot showing the Device > Licenses page in PAN OS, listing the Advanced URL Filtering license status.

2. Download and install the latest PAN OS content release (Device > Dynamic Updates). Follow best practices for content updates.
3. Schedule the firewall to download dynamic updates for Applications and Threats (Device > Dynamic Updates > Schedule).
4. (Next Steps) Configure URL filtering profiles and apply to Security policy rules. Test configuration.

Get Started with URL Filtering

The first step is understanding user web activity patterns. Review predefined categories, use Test A Site, and create a passive profile that alerts on most categories to gain visibility . Block known bad categories (malware, C2, phishing) immediately.

Consider pairing URL filtering with User ID to control access based on users/groups and prevent credential theft. Leverage URL filtering use cases beyond simple block/allow. Follow best practices.

PAN OS & Panorama Configuration Steps Overview:

1. Use Test A Site to check URL categorization.
2. Create a passive URL Filtering profile that alerts on all categories except known bad ones (Objects > Security Profiles > URL Filtering). Clone default, rename. Set action for most categories to 'alert'.

   Screenshot showing how to set the action for multiple URL categories simultaneously in a PAN OS URL Filtering profile categories tab.

3. Block access to known dangerous URL categories (malware, C2, phishing, etc.) in the same profile.
4. Click OK.
5. Apply the URL Filtering profile to Security policy rules that allow traffic from clients in the trust zone to the Internet (Policies > Security > [Rule] > Actions tab). Make sure the Source Zone is set to a protected internal network .
6. Commit the configuration.
7. View URL filtering logs ( Monitor > Logs > URL Filtering ) to see web activity.
8. (Next Steps) Use risk categories, pair with User ID, explore other use cases, follow best practices.

Configure Inline Categorization

Inline categorization enables realtime analysis using ML/DL for malicious phishing variants and JavaScript exploits ( local inline categorization ) and advanced zero day web attacks ( cloud inline categorization ). It requires an Advanced URL Filtering license .

Local inline categorization (Inline ML before PAN OS 10.2) is not supported on VM 50/VM50L . Cloud inline categorization uses cloud based ML detectors.

PAN OS & Panorama Configuration Steps Overview (10.2 & Later):

1. Log in to the PAN OS web interface.
2. Verify active Advanced URL Filtering license (Device > Licenses). (Image 003 applies here).
3. Update or create a new URL Filtering profile (Objects > Security Profiles > URL Filtering).

   Screenshot showing the Inline Categorization options within a URL Filtering profile in PAN OS (10.2+).

   • Select an existing or add a new profile.
   • Go to the ' Inline Categorization ' tab and enable Enable cloud inline categorization and/or Enable local inline categorization .
   • Click OK and Commit changes.
4. (Optional) Add URL exceptions to your URL Filtering profile (Inline Categorization tab > Add). Exceptions apply to both cloud and local inline categorization.

   Screenshot showing where to add external dynamic list exceptions for Inline ML (Local Inline Categorization) in older PAN OS UI.

5. Commit changes.
6. (Required if firewall is deployed with an explicit proxy server) Configure the proxy server for inline cloud services (Device > Setup > Services > Proxy Server).

   Screenshot showing where to configure Proxy Server settings for Inline Cloud Services under Device > Setup > Services in PAN OS.

7. (Optional) Set the Cloud Content FQDN for inline categorization (Device > Setup > content id > Content Cloud Setting).
8. (Optional) Verify connectivity to inline categorization cloud services using CLI:

   show mlav cloud status

   
   Displays the status of the connection to the ML cloud server (ml.service.paloaltonetworks.com).

   show ctd agent status security client

   
   Displays the status of the connection to the cloud delivered threat prevention service (hawkeye.services edge.paloaltonetworks.com). Verify necessary FQDNs/ports are not blocked.
9. Install an updated firewall device certificate for cloud authentication.
10. Test your URL filtering deployment using specific test URLs. Monitor URL Filtering logs filtered by Inline ML Verdict (PAN OS 10.0/10.1) or Inline Categorization Verdict (PAN OS 10.2+).

    Screenshot showing URL Filtering logs filtered by Inline ML Verdict, displaying malicious-javascript and phishing detections (older UI).

    

    Screenshot showing URL Filtering logs filtered by Inline Categorization Verdict, displaying Local and Cloud detections (newer UI).

URL Category Exceptions

You can exclude specific websites from URL category enforcement using custom URL categories or external dynamic lists (EDLs) , ensuring these sites are blocked or allowed regardless of their predefined category.

• Add IP addresses or URLs to a custom URL category of URL List type. Define site access in a URL Filtering profile or use the category as match criteria in a Security policy rule (place exception rule above other rules).
• Add URLs to an external dynamic list (EDL) of URL List type. Use the EDL in a URL Filtering profile or as match criteria in a Security policy rule. Benefit: update list without commit.

URL exceptions are evaluated before EDLs and predefined categories . Custom URL categories of URL List type function as block/allow lists. Custom URL categories of Category Match type define a new category for sites matching *all* specified categories.

Guidelines for URL Category Exceptions

Guidelines for populating custom URL categories or external dynamic lists of URLs:

• The firewall automatically appends a trailing slash (/) to domain entries not ending in '/' or '*' (PAN OS 10.2+ default). This limits matches to the exact domain and subdirectories (e.g., example.com becomes example.com/).
• In wildcard entries, trailing slash limits matches to patterns ending before the slash (e.g., *.example.com/ matches subdomains but not example.com itself).
• Manually adding trailing slashes is recommended for clarity.
• List entries are case insensitive . Omit 'http' and 'https'. Max length 255 characters.
• Use exact match (example.com) or wildcards (*, ^) for pattern matching. Wildcards must be the only character in a token. Caret (^) matches exactly one token, asterisk (*) matches one or more.
• Do not create entries with consecutive asterisks (**) or > 9 consecutive carets (^^^^^^^^^) in wildcard entries as it affects performance.
• Palo Alto Networks does not support regular expressions .

Create a Custom URL Category

Create a custom URL category (Objects > Custom Objects > URL Category).

• Set Type to URL List or Category Match .
• Add Items (URLs or existing categories).
• Save.
• Define Site Access and User Credential Submissions settings for the custom category in a URL Filtering profile.
• Apply the URL Filtering profile to a Security policy rule.

This diagram illustrates the creation and use of Custom URL Categories. You define the type (URL List or Category Match) and add the corresponding items. The resulting Custom URL Category object is then used within a URL Filtering profile to define specific actions or directly as match criteria in a Security policy rule, allowing tailored policy enforcement.

Configure URL Filtering

After planning your deployment and understanding user activity, configure URL Filtering profiles.

PAN OS & Panorama Configuration Steps Overview:

1. Create a URL Filtering profile (Objects > Security Profiles > URL Filtering > Add).
2. Define site access for each URL category (Categories tab > Set Access). Use allow, alert, block, continue, override, none.

   Screenshot showing how to set the action for multiple URL categories simultaneously in a PAN OS URL Filtering profile categories tab.

3. Detect corporate credential submissions (User Credential Detection tab). Choose method: Use IP User Mapping , Use Domain Credential Filter , or Use Group Mapping . Configure log severity. Define actions (alert, allow, block, continue) for credential submissions per category.
4. Detect phishing and malicious JavaScript in realtime using local inline categorization (Inline ML tab PAN OS 10.1; Inline Categorization tab PAN OS 10.2+).
5. Define URL category exceptions (URL Category Exceptions tab - PAN OS 10.1; Inline ML/Categorization tab - PAN OS 10.2+). Add URLs/patterns to allow or block regardless of category.
6. Enable Safe Search Enforcement (URL Filtering Settings tab).
7. Log container page only (URL Filtering Settings tab).
8. Enable HTTP Header Logging (URL Filtering Settings tab). Select fields to log.
9. Save the URL Filtering profile.
10. Apply the URL Filtering profile to Security policy rules that allow web access (Policies > Security > [Rule] > Actions tab).
11. Commit the configuration.
12. Test your URL filtering configuration.
13. (Best Practice) Enable Hold client request for category lookup (Device > Setup > content id). Default is Allow.

    Critical Point (PCNSA/PCNSE): Enabling 'Hold client request for category lookup' (Device > Setup > content id) is a security best practice to prevent users from accessing potentially malicious sites before the firewall can determine the category and apply policy. It introduces latency during the first access to an uncached URL.

14. Set the Category lookup timeout (sec) (Device > Setup > content id > gear icon).

Monitoring URL Filtering

Monitoring web activity is essential for protecting your organization and ensuring the efficacy of your URL filtering policy. Palo Alto Networks platforms generate detailed logs, dashboards, and reports.

Tools for monitoring: ACC , URL filtering logs , reports , Test A Site. Features for log control: HTTP header logging (more detail), log container page only (less volume).

PAN OS & Panorama Monitoring Overview:

• ACC: View common categories, threats, and blocked activity (ACC > Network Activity, Threat Activity, Blocked Activity widgets).

  Screenshot of the ACC showing Network Activity widgets, which can be sorted by URL categories.

• URL Filtering Logs: View detailed logs for categories set to alert, block, continue, or override ( Monitor > Logs > URL Filtering ). Log columns show Action, Category, User, URL , etc. Credential Detected column shows submissions. Inline ML/Categorization verdicts are shown.

  Screenshot showing URL Filtering logs in PAN OS, highlighting the 'Credential Detected' column.

  

  Screenshot of the detailed view of a URL Filtering log entry, showing details related to a credential submission event.

Important Point (PCNSA/PCNSE): URL Filtering logs (Monitor > Logs > URL Filtering) are the primary source for monitoring web activity enforced by the URL Filtering profile. They include Action, Category, URL, User, and Credential Detection status.

• Predefined URL Filtering Reports: Generate reports on categories, users, websites, blocked categories ( Monitor > Reports > URL Filtering Reports ).
• Custom Reports: Create custom reports based on URL Filtering logs ( Monitor > Manage Custom Reports ).

  Screenshot showing the Query Builder interface for creating a custom report based on URL logs.

  

  Screenshot showing an example custom report result based on URL logs.

• User Activity Report: View browsing patterns for users/groups ( Monitor > PDF Reports > User Activity Report ). Requires User ID enabled.

  Screenshot showing the configuration interface for the User Activity Report in PAN OS.

  

  Screenshot showing the table of contents for a generated User Activity Report PDF.

  

  Screenshot showing the 'Traffic Summary by URL Category' section within a User Activity Report.

Advanced Threat Prevention Report Headings

The Advanced Threat Prevention Report provides detailed analysis and detection information, as well as information about the transaction, session, and other related processes in JSON format.