Our Web Crawler, named AhrefsBot crawls your website making notes of outbound links and adding them to our database. It will periodically re-crawl your website to check the current status of previously found links. 

The crawler does not generate URLs, it only follows links found on the Internet and it is always using GET (which according to HTTP specification should not trigger any destructive actions on server side).

AhrefsBot strictly respects robots.txt, both disallow and allow rules.

To change the frequency of AhrefsBot visiting your site, you can specify the minimum acceptable delay between two consecutive requests from our bot in your robots.txt file:

User-agent: AhrefsBot
​Crawl-Delay: [value]

Where Crawl-Delay value is time in seconds.

If for some reason you want to prevent AhrefsBot from visiting your site, put the two following lines into the robots.txt file on your server:

User-agent: AhrefsBot
​Disallow: /

Please note that AhrefsBot may need some time to pick the changes in your robots.txt file. This will be made prior to each next scheduled crawl.