Skip to main content
All CollectionsSite ExplorerCrawling
How do I control your Bot's crawling behaviour?
How do I control your Bot's crawling behaviour?

Find out how you can control AhrefsBot's behaviour on your website.

Helen avatar
Written by Helen
Updated over a week ago

Our Web Crawler, named AhrefsBot crawls your website making notes of outbound links and adding them to our database. It will periodically re-crawl your website to check the current status of previously found links. 

The crawler does not generate URLs, it only follows links found on the Internet and it is always using GET (which according to HTTP specification should not trigger any destructive actions on server side).

AhrefsBot strictly respects robots.txt, both disallow and allow rules.

To change the frequency of AhrefsBot visiting your site, you can specify the minimum acceptable delay between two consecutive requests from our bot in your robots.txt file:

User-agent: AhrefsBot
Crawl-Delay: [value]

Where Crawl-Delay value is time in seconds.

If for some reason you want to prevent AhrefsBot from visiting your site, put the two following lines into the robots.txt file on your server:

User-agent: AhrefsBot
Disallow: /

Please note that AhrefsBot may need some time to pick the changes in your robots.txt file. This will be made prior to each next scheduled crawl.

Did this answer your question?