Know Exactly Which AI Bots Visit Your Site
A lightweight script tag identifies GPTBot, ClaudeBot, PerplexityBot, and 15+ AI crawlers in real time. See event logs, traffic trends, and page-level breakdowns.
Accurate Bot Identification
Rutt identifies 15+ distinct AI crawlers by analyzing user-agent strings against our continuously updated database. We distinguish between training crawlers (GPTBot, ClaudeBot), search indexing bots (OAI-SearchBot, PerplexityBot), and user-triggered fetches (ChatGPT-User, Claude-Web) so you know exactly why each bot is visiting.
Real-Time Event Stream
Every AI crawler visit appears in your dashboard within seconds, showing the page URL, crawler identity, company (OpenAI, Anthropic, Google, etc.), visit type, and timestamp. Filter events by date range, crawler, or page to drill into specific activity patterns.
Daily and Weekly Traffic Trends
Visualize AI crawler activity over time with interactive charts. Spot trends like increasing GPTBot activity before a new model release, identify seasonal crawling patterns, or detect anomalies like sudden spikes from a new crawler you have not seen before.
Page-Level Analytics
See exactly which pages on your site attract the most AI crawler attention. Discover whether bots focus on your blog content, product pages, documentation, or pricing. Use this data to prioritize content updates on the pages that AI models actually consume.
Most Crawled Pages
Crawler Type Breakdown
Not all AI bot visits are the same. Training crawlers collect data for model updates. Search indexing bots build knowledge bases for AI-powered search. User-triggered fetches happen when someone asks ChatGPT to visit a URL in conversation. Our breakdown shows you the mix of each type so you can make targeted access decisions.
One-Line Installation
Add a single script tag to your site and tracking begins immediately. The snippet weighs under 1KB compressed, loads asynchronously so it never blocks page rendering, and uses the Beacon API to transmit data without affecting your site's performance metrics or Core Web Vitals scores.
<script
src="https://cdn.aimonitor.dev/v1/tracker.js"
data-site-id="YOUR_SITE_ID"
async
></script>Paste this snippet before the closing </head> tag. Under 1KB, async loading, zero render blocking.
AI Crawlers Are Visiting Your Site Right Now
Every day, AI companies send automated crawlers to your website. Without monitoring, you are making critical decisions about your content and infrastructure without knowing who is consuming what.
Your content is being used for AI training without your knowledge
GPTBot and ClaudeBot crawl websites to collect data for model training and fine-tuning. These crawlers can visit hundreds of your pages in a single day, extracting text that becomes part of the training data for the next version of ChatGPT or Claude. Without visibility into this activity, you cannot make informed decisions about what content to make available or restrict.
AI traffic can consume significant server resources
Some websites see 20% to 40% of their total HTTP requests coming from AI bots alone. These crawlers do not render JavaScript or load images, so they do not show up in traditional analytics, but they still consume bandwidth and server resources. High-frequency crawling can even degrade performance for real visitors during peak traffic.
You cannot control what you cannot see
Decisions about robots.txt rules, rate limiting, and content access policies require data. Should you block GPTBot from your pricing page? Should you rate limit ClaudeBot to reduce server load? Should you allow PerplexityBot full access because it drives referral traffic? Without crawler analytics, these decisions are just guesses.
Our JavaScript snippet (under 1KB compressed) loads asynchronously after your page renders. On each page view, it analyzes the visitor's user-agent string against our database of known AI crawlers, search bots, and common bot patterns. It then sends a lightweight beacon to our API with the page URL, visitor classification, and timestamp. The entire process happens in milliseconds with zero impact on page load or rendering.