All Collections
Site Audit
FAQs and Troubleshooting
How do I whitelist Ahrefs user agents for Incapsula?
How do I whitelist Ahrefs user agents for Incapsula?

Troubleshooting issues with Incapsula plugin

Helen avatar
Written by Helen
Updated over a week ago

If you see your Site Audit's crawl in Data Explorer returning a 400 HTTP status code (Bad Request error) for your main page and a resource page having Incapsula in the URL, this is due to the Incapsula plugin on your website. 

Our crawler user agent is blocked by the Incapsula firewall, preventing us from accessing the content of your pages. 

Find out how you can set security exceptions with your provider in order for us to crawl your website:

Whitelist Ahrefs IPs

In your Incapsula control panel, navigate to:

Settings > Security > Whitelist Specific Sources

Scroll to the top of the page and click 'Save'

Whitelist our user-agents

Please ensure that you have also whitelisted our user agents ‘AhrefsBot' and ‘AhrefsSiteAudit’. If you need more information on our AhrefsBot, do refer to this page here: https://ahrefs.com/robot

In your Incapsula control panel, navigate to:

Settings > Security > Bot Access Control 

Click on 'Add exception' and the Whitelist Rules for Bot Access window will pop up.

Under the dropdown menu, select 'User agent' and key in ‘AhrefsBot' and ‘AhrefsSiteAudit’ then click 'Confirm'. 

The crawling error usually comes from user’s configuration and if you still face any problem with crawling your website after whitelisting the above, please check in with your provider.

Did this answer your question?