💡 A full guide on how Ahrefs crawls and indexes the web can be found on our "How to use Ahrefs" guide in our academy.
In simplest terms, a web crawler (or spider) is a computer program that browses the web on a step by step basis.
It starts with indexing an initial list of URLs (so-called “seeds”).
When it finds all the hyperlinks that the seeds contain, it adds them to the list of URLs which it will index next. Once added, the crawler visits those new pages and repeats the whole process all over again.
Our AhrefsBot crawls the web strictly following the rules from robots.txt. More details here: https://ahrefs.com/robot