1. Increasing crawl speed

Verifying ownership is a required step to increase speed but in itself, does not increase speed. Instead, this enables advanced settings, allowing you to crawl as fast as you like.  (in other words, this only applies if you are the owner.) 

  1. From Site Audit, simply start a new project and click on "Ownership verification". Enter your domain URL to begin the process!
  • HTML file upload: Download the verification file, then upload it to your domain's root folder. You do not need to tamper or rename with the file in any way.
  • TXT record: Add the text record to your domain's DNS settings.
  • HTML tag: Add the meta tag to your website's homepage code. 

Note: For sites on Wordpress, please see specific instructions here to verify using HTML tag.

  2. Proceed to "Crawl settings" to tweak crawl speed settings for your verified domain. The more requests we send, the faster the crawl speed.

For example: you can decrease the delay between requests to 0.

2. Optimising/decreasing crawled pages

If your site has 500k URLs, crawling all of these pages will take a long time even if you verify ownership to increase crawl speed.

In many cases, another option is to optimise (decrease) the number of pages that are crawled so that you receive important data from your crawl faster.

Please see this link to learn how to exclude pages from your crawl.

For example, if you run a huge e-commerce store with thousands of product pages that are created and deleted by the day, you can simply use an option like "Remove URL parameters" to avoid crawling these pages and focus on the data that matters.

Related

Did this answer your question?