Sometimes, our bot: AhrefsBot might be blocked by the Cloudflare Firewall preventing it from crawling the website.
If you want to lift this restriction, follow the steps below.
1. Log in to Cloudflare and select the domain.
2. Go to Security > Security rules.
3. Click Create rule, then choose Custom rule.
4. Give the rule a name, for example “Ahrefs IPs.”
5. Under “When incoming requests match,” set Field to “IP Source Address,” Operator to “is in,”.
5.1 Copy the list of ranges from https://help.ahrefs.com/en/articles/78658-what-is-the-list-of-your-ip-ranges "Our crawlers use the following ranges:" section
5.2 Return to Cloudflare page and press Edit expression on the right side of Expression preview
5.3 Paste the list of ranges between curled brackets {}
5.4 Copy the list of individual IP addresses from https://help.ahrefs.com/en/articles/78658-what-is-the-list-of-your-ip-ranges "And the following individual IP addresses outside these ranges:"
5.5 Go back to Cloudflare page, put cursor after 202.8.40.0/22 and before }) , press Space bar
5.6 Paste the list of individual addresses from clipboard
5.7 Press Use expression builder to see the list of networks and IP addresses in a nicer view.
6. Under “Then take action,” set the action to “Skip.”
7. Optionally keep “Log matching requests” enabled.
8. Under “WAF components to skip,” enable: All remaining custom rules, All rate limiting rules, All managed rules, and All Super Bot Fight Mode rules.
9. Set placement order as needed and deploy the rule.
The above tips should work for most websites with a generic setup. If it does not work for you, please contact Cloudflare Support.