At Ahrefs, we crawl pages according to a specific algorithm that ensures an efficient use of our resources without compromising the quality of data. 

With such an algorithm, it is not possible to manually submit any links/URLs or XML sitemap or ping our service for an on-demand crawl. 

Crawl priority for sites

In general, the more “popular” a site is (ie. a higher Domain Rating (DR), more quality backlinks), the more likely a domain will be crawled. Domains that are less “popular” (ie. lower DR, less quality backlinks) will be partially crawled. 

Speed of crawl / update for specific pages

The speed of crawling and updating specific pages depends on their URL Rating (UR). Pages with higher UR have a higher chance of getting indexed and updated more often, as compared to pages with a low rating.

Did this answer your question?