AI Crawlability Checker
Check if ChatGPT, Claude, Perplexity, Google Gemini, and other AI search engines can crawl and index your website. Get your AI crawlability score and discover exactly what's blocking AI bots from discovering your content.
Enter Your Website URL
We'll check robots.txt, llms.txt, meta tags, and HTTP headers for all major AI crawlers.
Now Find Out Who's Actually Visiting From AI Search with website analytics.
Getting crawled is step one — step two is knowing which AI platforms send you real traffic. UseClick tracks visits from ChatGPT, Perplexity, and every other source with website analytics and short link tracking. No cookies, GDPR compliant, free to start.
What Is AI Crawlability?
AI crawlability is the ability of AI search engines and large language models to discover, read, and index your website's content for use in AI-generated answers, recommendations, and search results.
AI-Powered Search Is Growing
ChatGPT, Perplexity, Claude, and Google Gemini are becoming primary ways people find information. If your site isn't crawlable by these AI engines, you're invisible to millions of searches.
robots.txt Controls Access
Your robots.txt file is the main gatekeeper for AI crawlers. A single Disallow: / rule for GPTBot or ClaudeBot blocks that AI engine from ever indexing your content.
llms.txt Is the New Standard
Just like robots.txt revolutionized traditional SEO, llms.txt is becoming the standard for AI-first content discovery. Early adopters gain a significant advantage in AI search visibility.
AI Crawlers We Check
Our tool checks your site against all major AI search engine crawlers to give you a complete picture of your AI search visibility.
ChatGPT / OpenAI
GPTBotPowers ChatGPT's web search and knowledge base. Largest AI search user base.
Claude / Anthropic
ClaudeBot + anthropic-aiPowers Claude AI by Anthropic. Checks both bot identifiers used by Anthropic.
Perplexity AI
PerplexityBotPowers Perplexity.ai — one of the fastest-growing AI search engines.
Google Gemini
Google-ExtendedPowers Google's AI features including Gemini and Search Generative Experience.
Meta AI
meta-externalagentPowers Meta AI across Facebook, Instagram, and WhatsApp.
Common Crawl
CCBotOpen dataset used to train most major LLMs. Being indexed here helps many AI models.
Amazon Alexa
AmazonbotPowers Amazon Alexa and Amazon's AI-powered search features.
Cohere AI
cohere-aiEnterprise AI platform used by many businesses for internal AI search and RAG systems.
How to Optimize Your Site for AI Crawlers
Follow these steps to maximize your AI search visibility and ensure all major AI engines can discover and index your content.
Audit Your robots.txt
Open your robots.txt at yourdomain.com/robots.txt. Look for Disallow: / rules for specific AI bot names. Remove any that are blocking bots you want to allow.
# Allow all AI crawlers User-agent: GPTBot Allow: / User-agent: ClaudeBot Allow: / User-agent: PerplexityBot Allow: / User-agent: Google-Extended Allow: /
Create an llms.txt File
Place an llms.txt file at your domain root. This emerging standard helps LLMs understand your site's purpose, key pages, and how to use your content.
# My Company — Short description ## About We provide [what you do]. ## Key Pages - [Homepage](https://example.com/) - [About](https://example.com/about) - [Blog](https://example.com/blog) ## Optional - [Sitemap](https://example.com/sitemap.xml)
Check Meta Robots Tags
Ensure your pages don't have noai or noindex in their meta robots tags unless intentional.
<!-- Good: allows AI indexing --> <meta name="robots" content="index, follow"> <!-- Bad: blocks AI crawlers --> <meta name="robots" content="noai, noimageai"> <!-- Bad: blocks all crawling --> <meta name="robots" content="noindex, nofollow">
Add a sitemap.xml
A sitemap helps AI crawlers discover all pages on your site. Generate one with your CMS or a tool, then reference it in robots.txt.
# In robots.txt, add: Sitemap: https://yourdomain.com/sitemap.xml # Most CMSes auto-generate sitemaps: # WordPress: /sitemap.xml or /sitemap_index.xml # Shopify: /sitemap.xml # Next.js: use next-sitemap package
Know Which AI Platforms Drive Traffic to Your Site
UseClick's real-time link analytics shows you exactly which AI search engines — ChatGPT, Perplexity, Claude — are sending clicks your way. Privacy-first, GDPR compliant, no cookies needed.
Frequently Asked Questions
Everything you need to know about AI crawlability and optimizing your site for AI search engines.
Frequently Asked Questions
AI crawlability refers to how easily AI search engines and large language models (LLMs) like ChatGPT, Claude, and Perplexity can discover, read, and index your website's content. As AI-powered search grows, being crawlable by AI bots is becoming as important as traditional SEO. If AI crawlers can't access your site, your content won't appear in AI-generated answers or recommendations.
GPTBot is OpenAI's web crawler used to collect training data for ChatGPT and other OpenAI models. When GPTBot can index your site, your content may be included in ChatGPT responses and recommendations. You can allow or block GPTBot via your robots.txt file. Blocking GPTBot means your site won't appear in ChatGPT's knowledge base.
ClaudeBot and anthropic-ai are Anthropic's web crawlers used to collect data for Claude AI. Both identifiers should be treated the same way in robots.txt. If you want Claude to be able to reference your content, ensure neither is blocked. Our tool checks both bot names to give you an accurate result.
llms.txt is an emerging standard (similar to robots.txt) designed specifically for AI crawlers. It's a plain text file placed at /llms.txt on your domain that helps LLMs understand your site's content structure, important pages, and how you'd like your content used in AI-generated responses. Having an llms.txt signals that you are AI-search-friendly and can improve visibility in AI tools.
robots.txt is the primary way to control which bots can access your site. You can use User-agent directives with specific bot names (e.g., GPTBot, ClaudeBot, PerplexityBot) to allow or block each AI crawler individually. A Disallow: / rule for a specific bot completely blocks it. Without any restrictions, all bots are allowed by default.
It depends on your goals. Allowing AI crawlers means your content may be referenced in AI-generated answers, potentially driving traffic and building authority. Blocking them gives you more control over how your content is used but reduces your AI search visibility. For most businesses, allowing reputable AI crawlers (ChatGPT, Claude, Perplexity, Gemini) is beneficial for visibility.
The AI Crawlability Score (0-100) measures how well-optimized your site is for AI search engines. It factors in: HTTPS security (10pts), robots.txt presence (10pts), sitemap.xml (10pts), llms.txt file (20pts), and accessibility to each AI crawler (50pts). A score of 75+ is excellent, 50-74 is good, and below 50 needs improvement.
Google-Extended is the user-agent for Google's AI products including Gemini, Bard, and Google's Search Generative Experience (SGE). Blocking Google-Extended prevents your content from being used in Google's AI-powered search features. Since Google AI search is increasingly prominent, keeping Google-Extended allowed can significantly impact your AI search visibility.
More Free SEO & Marketing Tools
Ready to track smarter?
UseClick.io makes link management effortless. Create branded short links that are clean, memorable, and built to strengthen your brand identity.