Regular expressions (also known as REGEX or REGEXP) help you find URLs or text that match a particular pattern. REGEX is supported in Site Explorer, Site Audit and certain endpoints in our API.

We use RE2.

You can test if your Regex expression works using this website: https://regex101.com/. Note that you should select “Golang” from the left menu.

Reports that allow you to filter for specific URLs, domains or text fields support REGEX filters. You can choose to filter for "Match regex", or "Doesn't match regex":

Here is an example of where to find the option to filter by REGEX on the Referring Domains report:

You can also check More filters for fields that support REGEX:

You can search for specific URLs or text in a page's HTML if you select the "Use regular expression" option:

Inside the Crawl settings of any project, you can define a specific set of URLs that you wish to include or exclude from any crawl. These fields support REGEX:

Any invalid REGEX expressions will be ignored. It would be helpful to first check that your expressions work correctly using a tool like https://regex101.com/

Keep in mind that these rules also apply to the seeds. So whenever you set a new pattern, make sure that our crawler has something to begin the crawl with.

If you use both fields and some URL matches both “Include” and “Exclude” patterns, Site Audit will exclude that URL from the crawl.

Page explorer and Link explorer support Advanced filters, and many have the option to select "Matching RegExp" when setting a filter:

Certain API endpoints for Site Explorer and Keywords Explorer tools allow the where parameter, which allows you to use REGEX patterns.

See the full documentation of how they can be applied in our API documentation.