Access · Agent Analytics & Crawler Access

You can't be the answer if the engines can't read you.

Sixteen AI crawlers feed the answer engines. Synthic checks your robots.txt against all of them, then reads your server logs to show which ones actually came.

Start free audit All capabilities

Crawler access · robots.txt16 agents trackedSample

Agents tracked

Allowed

Blocked

answering without you

GPTBot

OAI-SearchBot

ChatGPT-User

ClaudeBot

Claude-SearchBot

Claude-User

PerplexityBot

Perplexity-User

Google-Extended

Googlebot

Bingbot

Amazonbot

Applebot-Extended

Meta-ExternalAgent

Bytespider

CCBot

robots.txt · the two lines that matter

User-agent: CCBot
Disallow: /          ← blocks a live answer engine

User-agent: GPTBot
Allow: /

FIG.00 — agents · red = blocked · illustrativeACCESS

AI crawlers in the registry

kinds: live · search · training

live

robots.txt fetched at check time

your logs

parsed, never sampled

Anatomy

How it's built, part by part.

01Access

Blocked by accident is the most common failure.

A blanket Disallow written for one bot quietly blocks them all. Synthic evaluates your live robots.txt per crawler, with the exact rule that matched, plus llms.txt and sitemap presence.

Live user agents

A person is pulling you into an answer right now.

ChatGPT-UserOpenAI
Claude-UserAnthropic
Perplexity-UserPerplexity

Search index

Feeds the retrieval layer engines cite from.

OAI-SearchBotOpenAI
Claude-SearchBotAnthropic
PerplexityBotPerplexity
GooglebotGoogle
BingbotMicrosoft

Training crawlers

Feeds the next model version.

GPTBotOpenAI
ClaudeBotAnthropic
Google-ExtendedGoogle
AmazonbotAmazon
Applebot-ExtendedApple
Meta-ExternalAgentMeta
BytespiderByteDance
CCBotCommon Crawl

FIG.01 — THE REGISTRY · LIB/CRAWLERS.TS16 AGENTS

02Analytics

Who reads you, and who is answering with you.

Paste your access logs. Every AI agent hit, segmented: live user-agents mean a person just pulled your page into an answer; search bots feed retrieval; training crawlers feed the next model.

11 checks · 90 weight points → score out of 100lib/aeo.ts

Access · 30

AI crawlers can read the site
llms.txt present
sitemap.xml present

Structured · 24

JSON-LD structured data
FAQ schema for answer retrieval

Meta · 15

Descriptive <title>
Meta description
Canonical + Open Graph

Content · 21

Clear heading hierarchy
Answer-first / Q&A structure
Substantive content

FIG.02 — THE ACCESS CHECKS THE SITE AUDIT SCORES

What you get

Three things, done properly.

Crawler access checks

Your live robots.txt against sixteen AI agents, with the exact rule that matched each verdict.

Agent analytics from your logs

Every AI agent hit in your access logs, segmented into live, search and training traffic.

llms.txt and sitemap presence

The two files that tell an engine what you are and where to look — checked, and generated if missing.

Specification

Inputs, outputs, cadence, evidence.

Agent Analytics & Crawler Access · specv1

Registry: 16 AI crawlers in three kinds — live user agents, search indexers, training crawlers — with vendor and purpose
Access check: Live fetch of robots.txt; per-agent verdict with the exact matching rule; llms.txt and sitemap.xml presence
Log input: Combined Log Format lines (Nginx and Apache defaults), pasted or uploaded
Log outputs: Hits by bot, by kind and by day; top paths; paths fetched by live agents
Downstream: Blocked-crawler findings feed the Action Center and the AEO site audit
Never shown: Estimated crawl traffic — only lines you provided are counted

Where it lives · sidebar

Next: Optimization Studio

The answer is being written right now.

Run a live audit and see exactly where you stand — before your competitor does.

Start free audit See pricing

Free first audit
No credit card
Live engine data
Your data stays yours
Cancel anytime