Due to immense amounts of data that Ahrefs bot has to crawl/re-crawl constantly, its algorithm includes certain priority rules. In simplest terms, our crawler indexes some pages faster and more often than the others. 

Above all, the order of priority depends on the Domain Rating (DR) metric of a website and the number of pages it has. You can read more about Ahrefs DR here: https://ahrefs.com/blog/seo-metrics/.

For instance, if a website with a low DR has a lot of pages, we won’t be able to index all of those pages. Moreover, the re-crawling frequency will be comparatively lower.

On the other hand, if a website has a high DR score and just a few pages, our bot will be re-crawling all of them as often as possible. 

The other important part is the number of new backlinks. Once Ahrefs bot discovers each new backlink pointing to a given page, it will be raising its crawling priority. Correspondingly, if it doesn’t see any new backlinks coming to a certain page for a long time, it won’t re-crawl it often.

Sure, our algorithm is way more complicated than that and includes tons of other rules that determine the priority level for each particular page. 

Did this answer your question?