Ahrefs runs our own web crawler, or bot known as AhrefsBot. Our crawler visits millions of websites daily to retrieve information and store them in our records. In other words, this is how Ahrefs gets all its data of the whole wide web.
Factors that affect if/when pages get crawled
Crawl priority: in general, the more “popular” a site is (ie. a higher Domain rating (
Speed of crawling/updating: this depends on a page's URL Rating (UR). Pages with a high UR are more likely to be indexed and are updated more often than pages with a low UR.
When our bots have yet to crawl the target site, you'll see the following message when checking the Page inspect report: No HTML code snapshot found for this page.
Since the target page has not been crawled yet, on-page data such as the Title, Page content, HTML source code etc are not available in our database. Similarly, any outgoing links from this page would also not be captured.
If the domain is new, please give our crawler some time to pick up the new website (1-2 weeks is generally a good bet.). Same goes for a new page that has just been created.
Note: Backlinks reports will be available if we crawled other sites that have backlinks to the target site.
It's also possible that the domain might be blocking us from crawling it. To find out if the website's settings is blocking our crawler, enter the URL into our crawler's page: https://ahrefs.com/robot
To fix that, please read this article: https://help.ahrefs.com/site-explorer/crawling/how-do-i-enable-ahrefs-bot-to-crawl-my-website-and-index-its-pages