All Collections
Site Audit
FAQs and Troubleshooting
How do I finish crawling my website faster in Site Audit?
How do I finish crawling my website faster in Site Audit?

How to speed up your crawl speed and optimise your crawl if it's taking too long to complete

Rebekah avatar
Written by Rebekah
Updated over a week ago

1. Increasing crawl speed

Verifying ownership is a required step to increase speed but in itself, does not increase speed. Instead, this enables advanced settings, allowing you to crawl as fast as you like.  (in other words, this only applies if you are the owner.) 

  1. From the Projects settings click on "Ownership verification".

You can use the following verification methods:

  • Google Search Console: Connect your Google Account and allow access to your Search Console data. If your website is verified in your Google Search Console, it will be automatically verified in Ahrefs.

  • HTML file upload: Download the verification file, then upload it to your domain's root folder. You do not need to tamper or rename with the file in any way.

  • TXT record: Add the text record to your domain's DNS settings.

  • HTML tag: Add the meta tag to your website's homepage code. 

Note: For sites on Wordpress, please see specific instructions here to verify using HTML tag.

  2. Proceed to "Crawl settings" under Site Audit to tweak crawl speed settings for your verified domain. The more requests we send, the faster the crawl speed.

For example: you can decrease the delay between requests to 0.

2. Optimising/decreasing crawled pages

If your site has 500k URLs, crawling all of these pages will take a long time even if you verify ownership to increase crawl speed.

In many cases, another option is to optimise (decrease) the number of pages that are crawled so that you receive important data from your crawl faster.

For example, if you run a huge e-commerce store with thousands of product pages that are created and deleted by the day, you can simply use an option like "Remove URL parameters" to avoid crawling these pages and focus on the data that matters.

Did this answer your question?