What is Bot Analytics?
Bot Analytics helps you monitor which bots visit your website, how often they visit, and which pages they crawl. It tracks bots across 12 categories including AI assistants, search engines, SEO tools, and more. Bot Analytics is free during beta.
For more information about how it works and how it can be set up, check out its technical documentation here.
Why monitor bot traffic?
A sizeable share of traffic hitting your website comes from bots, with Cloudflre estimating that 53% of the crawler traffic is wasted effort.
Monitoring your website's bot traffic can help you:
Set appropriate rate-limiting rules to manage bad bots and keep your website stable
Know which AI crawlers (like ChatGPT and Perplexity) visit your site to collect content before citing it in AI-generated answers
Viewing bot traffic data in the reports
Bot Analytics has six reports, each giving a different view of your bot traffic.
Overview: a top-level summary of all bot activity across your site
Bots: traffic broken down by individual bot (e.g., Googlebot, GPTBot, ClaudeBot)
Categories: traffic grouped by bot type, such as Search Engine, AI Crawler, or SEO Tool
Pages: View and sort pages by the most number of visits, status code, or bots that is made to any page.
Status Codes: View pages and visits by the status codes bots experience when visiting any page on your website
Site Structure: View bot traffic data for pages on your site, organized in it's site structure hierarchy
Each report includes a line chart you can configure to suit your analysis.
Toggle between Absolute (raw request counts) and Relative (percentage share) views. The latter option makes it easier to directly compare pages or bots against each other.
Set the time granularity to hourly, daily, weekly, or monthly, and use custom date picker to compare different time periods.
Filtering in reports
You can filter data for bots names, categories, page URLs, or metrics like clicks, etc. Multiple filters can be stacked together to curate the report you need.
There is also an “AI bots” button to instantly filter everything to AI bot traffic only. This handy feature lets you quickly see how the AI bots crawl your site, and which pages they visit the most.
Exporting the bot traffic data
Export your bot traffic data out in a .csv or as a Google sheet.
Inspecting a page's content
In any report that lists pages, hover over the magnifying glass icon next to a page URL to open Page Inspect and view that page’s content. You can also see how the page has changed over time, which is useful for checking what bots actually see when they crawl it.
Page inspect data comes from AhrefsBot.
How do I set up Bot Analytics?
Bot Analytics supports many ways of monitoring non-human traffic logs, check out the CDN services we support below:
Cloudflare
Method 1: Cloudflare Logpush
Cloudflare Logpush sends your site’s HTTP request logs to Ahrefs automatically. It’s the more efficient option for high-traffic sites. This method needs a Cloudflare Enterprise plan.
Method 2: Cloudflare Worker
A Cloudflare Worker runs a small script on each request and sends the data to Ahrefs. This method works with any Cloudflare plan, including the free tier.
Vercel
Connect your Vercel team to your Ahrefs workspace, then enable Bot Analytics for your project. Ahrefs matches the project by domain and creates a Log Drain automatically.
→ Setup guide
→ Ahrefs on Vercel Marketplace
Amazon CloudFront
Forward standard access logs to Ahrefs through an Amazon Data Firehose stream.
Fastly
Send request logs to Ahrefs through an HTTPS logging endpoint using Fastly’s real-time log streaming.
How Bot Analytics verifies real and fake bots
Sometimes, bots that visit your website may declare a different name / User Agent than they actually are (e.g. bots pretending to be Googlebot but aren't actually from Google). Bot analytics verifies bots, and flag the ones that fail.
Each bot is verified using two independent sources:
Known bot IP ranges, fetched from public endpoints (Google, Bing, Anthropic and others). We check if the visitor IP belongs to the bot it claims to be.
Cloudflare verified bots, a fallback for any bot we don't know the IP ranges for.
Bot analytics flags a visitor as spoofed when it claims to be a bot from our known list of verified bots, but fails both checks. Spoofed bots keep their claimed name with a “(spoofed)” suffix, for example “Googlebot (spoofed)“, and move into a new Spoofed Bot category.
You can filter by that category to exclude spoofed traffic or to see only spoofed bots.
Note: Bot analytics can only verify if a bot is fake or not when we have at least one of the two supported sources of verification (Known bot IP ranges and Cloudflare verified bots). For any bot that we can’t tell real from fake, it will keep its claimed name in Bot analytics and will not get marked as spoofed.
Verified bot coverage will grow as we add more IP ranges and verification methods.
What are the bot categories?
Bot Analytics groups all detected bots into the following 12 categories.
Category | Description |
Monitoring | Bots that check whether your website is online and performing correctly (e.g., uptime monitors) |
AI Assistant | Bots from AI platforms like ChatGPT and Perplexity that crawl content to generate answers |
AI Search | Crawlers powering AI-driven search engines |
AI Crawler | Bots that collect web content to train AI models or build AI datasets |
Search Engine | Crawlers from Google, Bing, and other search engines that index your pages |
SEO Tool | Bots from SEO platforms like Ahrefs that analyze your site |
Social Media | Bots from platforms like Facebook and Twitter/X that fetch link previews |
Service Agent | Bots that perform automated tasks on behalf of a service or platform |
Security | Bots that scan for vulnerabilities or malware |
Feed Fetcher | Bots that fetch RSS or Atom feeds to syndicate your content |
Advertising | Bots from ad networks that verify placements and detect click fraud |
Other | Any bot that doesn’t fit into the above categories |