BlogTechnical
Technical

AI crawlability audit: what Writesonic found when they tested 62 webpage elements across 6 AI crawlers — and what it means for your site

JSON-LD scored zero out of six for readability across all major AI crawlers tested. The title tag was the only metadata element that 5 of 6 crawlers could read. A Writesonic study of 62 webpage elements reveals which technical signals AI systems actually ingest — and what a proper AI crawlability audit should check.

JSON-LD, the format Google recommends for structured data, scored zero out of six for readability across the six major AI crawlers that Writesonic tested. If your AI crawlability strategy is built on schema markup, this study changes the calculation.

0 / 6
AI crawlers that could read JSON-LD structured data
Writesonic, 62 webpage elements tested across 6 AI crawlers, March 2026.

What the study tested

In March 2026, Writesonic published a systematic test of 62 webpage elements across six major AI crawlers: GPTBot (OpenAI), ClaudeBot (Anthropic), PerplexityBot, Gemini crawler (Google), Meta-ExternalAgent, and Applebot. For each element, they scored whether the crawler could reliably extract the information. The score was the number of crawlers out of six that read the element accurately.

The results are significantly different from what most AI optimisation advice assumes.

What AI crawlers cannot read

AI crawlers cannot read JSON-LD, Open Graph tags, meta descriptions, or Twitter card tags — all four scored zero out of six in Writesonic's test.

JavaScript-rendered content fared only slightly better, scoring one out of six. These are among the most commonly recommended technical SEO elements, and none of them appear to function as inputs AI crawlers actually use.

Pages where the body content is populated by React, Vue, or similar frameworks — and is empty in the page's HTML source — were largely unreadable across every crawler tested. This has significant implications for single-page applications and any site relying on client-side rendering for its primary content.

Element typeAI crawler readability (out of 6)Notes
Title tag5 / 6Only metadata element with strong cross-crawler readability
Visible body text6 / 6Primary input for all crawlers; must be in HTML source
H1 / H2 / H3 headings6 / 6Semantic heading structure strongly readable
Alt text (images)4 / 6Varies by crawler; GPTBot and Perplexitybot strongest
JSON-LD structured data0 / 6Not read by any tested crawler
Meta description0 / 6Not extracted by AI crawlers in tests
Open Graph tags0 / 6Not extracted by AI crawlers in tests
JS-rendered content1 / 6Most AI crawlers do not execute JavaScript

Source: Writesonic, 62 webpage elements, 6 AI crawlers, March 2026 (abridged)

What AI crawlers do read

Four page elements reach AI crawlers cleanly. Body text and heading structure each score six out of six, the title tag scores five out of six, and image alt text scores four out of six.

Visible body text is the primary content channel for AI retrieval — every crawler tested could read the text a human reader sees on the page.

Semantic heading structure was just as strong: H1, H2, and H3 tags were reliably parsed across all crawlers. This confirms what content structure research suggests — clear heading hierarchy is a machine-readability signal for AI systems, not just a user experience one.

The title tag was the only metadata element with meaningful cross-crawler readability, with Meta-ExternalAgent (Meta's crawler) the sole exception. For every other major crawler, the title tag is the one metadata element worth keeping accurate and descriptive.

Image alt text trailed at four out of six, with GPTBot and PerplexityBot showing the strongest readability. For image-heavy content, alt text remains a worthwhile signal — just not a universal one.

What a proper AI crawlability audit should check

A traditional technical SEO audit focuses on crawl errors, redirect chains, canonical tags, and structured data validation. An AI crawlability audit has a different checklist.

1. Robots.txt permissions for AI crawlers

The first check is whether your robots.txt is blocking AI crawlers. The major crawlers and their user-agent strings are listed below. Any disallow rule that matches these agents prevents those AI systems from indexing your content for potential citations.

robots.txt: allow all AI crawlers
User-agent: GPTBot
Allow: /

User-agent: ChatGPT-User
Allow: /

User-agent: ClaudeBot
Allow: /

User-agent: anthropic-ai
Allow: /

User-agent: PerplexityBot
Allow: /

User-agent: Google-Extended
Allow: /

User-agent: Meta-ExternalAgent
Allow: /

2. Server-side vs client-side rendering

If your primary content is rendered by JavaScript, check whether AI crawlers can access it. The simplest test is to view the page source (Ctrl+U in Chrome) and search for your key body text. If the text is not present in the raw HTML, most AI crawlers are not reading it.

Next.js, Nuxt, and similar frameworks with server-side rendering (SSR) or static generation (SSG) produce HTML that AI crawlers can read. Client-side-only rendering (CSR) does not.

3. Heading hierarchy and content structure

Heading hierarchy and content structure directly affect AI citation rates: an AirOps analysis of 815,484 AI-cited pages found that pages with 7 to 20 subheadings were cited more often than pages with fewer or more.

Over-structuring and under-structuring both reduce citation probability, so the target band matters more than simply adding headings. Each page should have exactly one H1 describing the primary topic, H2s for major subtopics, and H3s for subdivisions within those sections. Heading text should be descriptive and include relevant topic terms — AI crawlers use headings to understand content structure the same way human readers do.

4. Crawl rate and server stability

AI crawlers typically use the crawl-delay directive in robots.txt and respect Retry-After headers. If your server responds slowly or intermittently, crawlers may skip pages or fail to complete a crawl. Check your server logs for GPTBot, ClaudeBot, and PerplexityBot access patterns to confirm they are completing successful crawls.

5. Content accessibility: no login walls, no paywalls

AI crawlers do not authenticate. Any content behind a login, paywall, or cookie consent gate that prevents page load is invisible to them. If you want AI systems to cite specific content, that content must be accessible without authentication.

What schema markup still does

Schema markup still matters for Google AI Overviews, even though JSON-LD failed to register with any of the six AI crawlers tested for direct readability.

Google AI Overviews builds on Google Search infrastructure that does read structured data, so Article and Person schema remain relevant there. For ChatGPT, Anthropic, and Perplexity, the evidence suggests schema is not a direct input to their retrieval systems at all.

The more productive reframe: visible content signals are the universal AI crawlability layer, while schema markup is a Google-specific amplifier. Both have value; they just operate in different channels.

Visible content is the universal AI crawlability layer

AI crawlability is fundamentally different from traditional search crawlability. The technical elements that matter most are the ones visible to human readers: body text, heading structure, title tags, and clean HTML.

The metadata layer that traditional SEO builds on — JSON-LD, Open Graph, meta descriptions — does not appear to reach most AI crawlers. Start any audit with what crawlers can actually read, not what traditional SEO tooling tells you to add. The two checklists overlap significantly in outcomes: both reward structured, accessible content, but the checklist itself is different.

Frequently asked questions

What is an AI crawlability audit?

An AI crawlability audit reviews your website to determine how effectively AI crawlers (including GPTBot (OpenAI), ClaudeBot (Anthropic), PerplexityBot, and Googlebot-Image) can access, parse, and extract content from your pages. Unlike a traditional SEO technical audit, an AI crawlability audit focuses on the elements that AI retrieval systems actually read: visible page content, title tags, semantic HTML structure, and robots.txt allowances. Metadata that traditional crawlers rely on (JSON-LD, Open Graph, meta descriptions) scores near-zero on AI crawler readability tests.

Can AI crawlers read JSON-LD schema markup?

JSON-LD scored zero out of six for readability in a Writesonic study that tested 62 webpage elements across 6 major AI crawlers in March 2026: none of the tested crawlers reliably extracted information from JSON-LD structured data. This is a significant finding because JSON-LD is the recommended implementation format for schema markup and is widely included on pages as an AEO signal. The evidence suggests it does not directly influence AI crawler behaviour.

What technical elements do AI crawlers actually read?

The technical elements AI crawlers actually read come down to one gate: server-rendered HTML. A JavaScript-heavy page that renders content client-side stays invisible to most AI crawlers no matter how well its metadata is optimised, while a server-rendered page with clean HTML gets read by every crawler tested even with zero metadata work. Confirm this with view-source, not DevTools — DevTools shows the post-render DOM, which is exactly what a client-side-rendering crawler will never see. Anchor text and paragraph formatting are worth polishing only after rendering is confirmed clean.

Which AI crawlers should I allow in robots.txt?

The major AI crawlers and their robots.txt user-agent strings are: GPTBot (OpenAI), ClaudeBot (Anthropic), PerplexityBot (Perplexity), ChatGPT-User (OpenAI browsing), anthropic-ai (Anthropic), and Google-Extended (Google AI training). Blocking these agents prevents the respective AI systems from indexing your content for potential citations. If you want your content considered for AI search citations, these agents should be allowed.

Does page speed affect AI crawlability?

Page speed affects AI crawlability, but indirectly. AI crawlers typically crawl at lower rates than Googlebot, and slow or unstable server response times can cause a crawler to skip a page entirely. The bigger risk is JavaScript-heavy pages that require rendering before content is accessible: most AI crawlers do not execute JavaScript, so dynamically loaded content stays invisible to them regardless of how fast the server responds.

BE

BetterAISearch Editorial Team

BetterAISearch

The BetterAISearch team synthesises peer-reviewed studies, platform documentation, and independent research into actionable, scored tactics.

Get the next finding first.

New research moves scores. We send what changed and what it means, before it becomes LinkedIn noise.