// Access · Agent Analytics & Crawler Access
You can't be the answer if the engines can't read you.
Sixteen AI crawlers feed the answer engines — training bots, search indexers, and the live user-agents that fetch your page while a person waits for their answer. Synthic checks your robots.txt against all of them, then reads your server logs to show which ones actually came.
16
AI crawlers in the registry
3
kinds: live · search · training
live
robots.txt fetched at check time
your logs
parsed, never sampled
01 — Access
Blocked by accident is the most common failure.
A blanket Disallow written for one bot quietly blocks them all. Synthic fetches your live robots.txt and evaluates each crawler's access with the exact rule that matched, plus llms.txt and sitemap presence — the three things an engine needs to read you properly.
- Per-crawler verdict with the matched rule
- Vendor and purpose for every agent
- llms.txt and sitemap.xml presence checks
Live user agents
A person is pulling you into an answer right now.
- ChatGPT-UserOpenAI
- Claude-UserAnthropic
- Perplexity-UserPerplexity
Search index
Feeds the retrieval layer engines cite from.
- OAI-SearchBotOpenAI
- Claude-SearchBotAnthropic
- PerplexityBotPerplexity
- GooglebotGoogle
- BingbotMicrosoft
Training crawlers
Feeds the next model version.
- GPTBotOpenAI
- ClaudeBotAnthropic
- Google-ExtendedGoogle
- AmazonbotAmazon
- Applebot-ExtendedApple
- Meta-ExternalAgentMeta
- BytespiderByteDance
- CCBotCommon Crawl
02 — Analytics
Who reads you, and who is answering with you.
Paste or upload your access logs. Synthic finds every AI agent hit and segments them: live user-agents mean a real person just pulled your page into an answer; search-index bots feed retrieval; training crawlers feed the next model. The paths they hit tell you which pages are doing the work.
- Hits by bot, by kind, by day
- Top paths, and the paths live agents fetched
- Parsed from the lines you provide — nothing estimated
Access
- AI crawlers can read the site16
- llms.txt present11
- sitemap.xml present7
Structured data
- JSON-LD structured data18
- FAQ schema for answer retrieval9
Meta
- Descriptive <title>7
- Meta description6
- Canonical + Open Graph4
Content
- Clear heading hierarchy9
- Answer-first / Q&A structure9
- Substantive content6
// Where it lives
The dashboard modules that deliver it.
Every capability on this page is a working surface in the product today, not a roadmap item.
app/site-audit
AEO Site Audit
A technical audit for the answer era, scored into one AEO grade.
app/crawlers
Crawler Access
Your robots.txt against the 16 crawlers that feed AI engines.
app/crawler-logs
Agent Analytics
Which AI agents hit your server, from your own access logs.
// Questions
Straight answers.
Which log format is supported?
Combined Log Format — what Nginx and Apache write by default. Paste a window of lines or upload the file.
Should I block training crawlers?
It depends on your business. Synthic shows you the trade: training crawlers feed future models, search and live agents feed today's answers. Blocking the second group makes you invisible now.
The answer is being written right now.
Run a live audit and see exactly where you stand — before your competitor does.