What is Bot Analytics?
Bot Analytics helps you monitor which bots visit your website, how often they visit, and which pages they crawl. It tracks bots across 12 categories including AI assistants, search engines, SEO tools, and more.
Bot Analytics is free during beta.
Why monitor bot traffic?
A sizebable share of the traffic hitting your website comes from bots, with Cloudflare estimating that 53% of the crawler traffic is wasted effort.
Monitoring your website's bot traffic can help you:
Set appropriate rate-limiting rules to manage bad bots and keep your website stable
Know which AI crawlers (like ChatGPT and Perplexity) visit your site to collect content before citing it in AI-generated answers
Viewing bot traffic data in the reports
Bot Analytics has four reports, each giving a different view of your bot traffic.
Overview: a top-level summary of all bot activity across your site
Bots: traffic broken down by individual bot (e.g., Googlebot, GPTBot, ClaudeBot)
Categories: traffic grouped by bot type, such as Search Engine, AI Crawler, or SEO Tool
Crawled pages: specific pages on your site bots have visited
Each report includes a line chart you can configure to suit your analysis.
Toggle between Absolute (raw request counts) and Relative (percentage share) views. The latter option makes it easier to directly compare pages or bots against each other.
Set the time granularity to hourly, daily, weekly, or monthly, and use custom date picker to compare different time periods.
Filtering in reports
You can filter data for bots names, categories, page URLs, or metrics like clicks, etc. Multiple filters can be stacked together to curate the report you need.
There is also an “AI bots” button to instantly filter everything to AI bot traffic only. This handy feature lets you quickly see how the AI bots crawl your site, and which pages they visit the most.
Exporting the bot traffic data
Export your bot traffic data out in a .csv or as a Google sheet.
How do I inspect a page’s content?
In any report that lists pages, hover over the magnifying glass icon next to a page URL to open Page Inspect and view that page’s content. You can also see how the page has changed over time, which is useful for checking what bots actually see when they crawl it.
Page inspect data comes from AhrefsBot.
How do I set up Bot Analytics?
Bot Analytics requires a Cloudflare integration to collect bot traffic data. It works independently from Web Analytics and collects data server-side, so you do not need a JavaScript tracking script on your site.
First, open your project in Ahrefs, and go to Project Settings → Bot Analytics.
Next, choose your integration method: Cloudflare Logpush or Cloudflare Worker
Method 1: Cloudflare Logpush
Cloudflare Logpush sends your site’s HTTP request logs to Ahrefs automatically. It’s the more efficient option for high-traffic sites. This method needs a Cloudflare Enterprise plan.
Method 2: Cloudflare Worker
A Cloudflare Worker runs a small script on each request and sends the data to Ahrefs. This method works with any Cloudflare plan, including the free tier.
Configure your custom token (Cloudflare Worker)
To use the Cloudflare Worker method, you need to create a custom API token in Cloudflare with the correct permissions.
When creating the token, make sure to fill in your Zone (domain) under Zone Resources. The token cannot be saved without this step.
Once your token is set up correctly, your worker configuration should look like this:
What are the bot categories?
Bot Analytics groups all detected bots into the following 12 categories.
Category | Description |
Monitoring | Bots that check whether your website is online and performing correctly (e.g., uptime monitors) |
AI Assistant | Bots from AI platforms like ChatGPT and Perplexity that crawl content to generate answers |
AI Search | Crawlers powering AI-driven search engines |
AI Crawler | Bots that collect web content to train AI models or build AI datasets |
Search Engine | Crawlers from Google, Bing, and other search engines that index your pages |
SEO Tool | Bots from SEO platforms like Ahrefs that analyze your site |
Social Media | Bots from platforms like Facebook and Twitter/X that fetch link previews |
Service Agent | Bots that perform automated tasks on behalf of a service or platform |
Security | Bots that scan for vulnerabilities or malware |
Feed Fetcher | Bots that fetch RSS or Atom feeds to syndicate your content |
Advertising | Bots from ad networks that verify placements and detect click fraud |
Other | Any bot that doesn’t fit into the above categories |