Skip to main content
All CollectionsSite ExplorerCrawling
How do I see all the pages that have been crawled by Ahrefs?
How do I see all the pages that have been crawled by Ahrefs?

Find out where to find the list of pages that have been added to Ahrefs' index.

Anna avatar
Written by Anna
Updated over 9 months ago

Due to immense amounts of data that Ahrefs bot has to crawl/re-crawl constantly, its algorithm includes certain priority rules. 

Because of that, there is a specific list of pages that we have in our index.

If you open any site in Site Explorer, you can see Crawled pages count when the "Crawled pages" metric is selected for the Performance chart.

This is the exact number of pages currently in our index for the mode you chose (domain with subdomains, domains without subdomains, exact URL or prefix).

Now if you want to see what those pages are, you can go to Best by links report in Site Explorer. You can sort it based on best internal or external links:

It is expected that the total pages in the Best-by-links report does not match the Crawled pages number on the Overview. This is because it also includes links to pages that we have found in the target website, but not yet indexed.

Did this answer your question?