If you see your Site Audit's crawl in Data Explorer returning a 400 HTTP status code (Bad Request error) for your main page and a resource page having Incapsula in the URL, this is due to the Incapsula plugin on your website. 

Our crawler user agent is blocked by the Incapsula firewall, preventing us from accessing the content of your pages. 

Find out how you can set security exceptions with your provider in order for us to crawl your website:

Whitelist Ahrefs IPs

In your Incapsula control panel, navigate to:

Settings > Security > Whitelist Specific Sources

Please add in the following Ahrefs IP ranges:
54.36.148.0/24
54.36.149.0/24
54.36.150.0/24

A full up-to-date list of our IPs can be found here: https://help.ahrefs.com/about-ahrefs/what-is-the-list-of-your-current-ip-ranges

Scroll to the top of the page and click 'Save'

Whitelist our user-agents

Please ensure that you have also whitelisted our user agents ‘AhrefsBot' and ‘AhrefsSiteAudit’. If you need more information on our AhrefsBot, do refer to this page here: https://ahrefs.com/robot

In your Incapsula control panel, navigate to:

Settings > Security > Bot Access Control 

Click on 'Add exception' and the Whitelist Rules for Bot Access window will pop up.

Under the dropdown menu, select 'User agent' and key in ‘AhrefsBot' and ‘AhrefsSiteAudit’ then click 'Confirm'. 

The crawling error usually comes from user’s configuration and if you still face any problem with crawling your website after whitelisting the above, please check in with your provider.

Related

Did this answer your question?