llms.txt and Schema Markup in 2026: What Actually Makes Your Site AI-Readable

Key terms used in this guide

• llms.txt — a proposed plain-text/Markdown file at the root of a domain that summarises a site's content for large language models.
• AI-readable — content an AI crawler can fetch, parse, attribute and quote without executing JavaScript or guessing at meaning.
• Schema markup — structured data (usually JSON-LD) that states machine-readable facts about a page: who published it, what it is about, what entities it references.
• AEO (Answer Engine Optimization) — optimising so an engine can extract a direct answer from your page.
• GEO (Generative Engine Optimization) — optimising so a generative model selects and cites your page when it synthesises an answer.
• Entity clarity — how unambiguously a machine can resolve "who or what this is" to a single real-world thing.

What is llms.txt?

llms.txt is a proposed standard file placed at the root of a website — for example yourdomain.com/llms.txt — containing a Markdown-formatted summary of the site and links to its most important pages. The idea, proposed in 2024, was to give language models a clean, curated map of a site instead of forcing them to crawl noisy HTML.

It is a genuinely reasonable idea. It is also, as of now, a standard nobody enforces.

Does llms.txt actually work? Here is what has been said on the record

This is the part most agency blog posts skip, because "you must add llms.txt immediately" is easier to sell than the truth.

Google's John Mueller addressed llms.txt directly and compared it to the keywords meta tag — a self-declared claim about what a site contains, which search engines abandoned decades ago precisely because it could not be trusted. His reasoning, paraphrased:

• No major AI service has stated it uses llms.txt.
• Server logs show major AI crawlers do not request the file.
• A crawler that has already downloaded your actual pages gains nothing from a summary it would have to verify anyway.
• The format invites cloaking — showing models one thing and users another — which is exactly why self-declared metadata files die.

Google's own documentation on AI features and your website is unusually blunt on the wider point: "You don't need to create new machine readable files, AI text files, or markup to appear in these features." Google also states there is "no special schema.org structured data that you need to add" for AI Overviews or AI Mode.

So should you delete your llms.txt? No — and this is where the honest answer gets more interesting than the marketing answer.

The one use case where llms.txt is genuinely useful

If you publish technical documentation, an API reference, or a knowledge base, an llms.txt (or the fuller llms-full.txt) is useful because humans paste it into a chat window. A developer who wants an AI assistant to help them integrate your product can hand the model one clean file instead of forty rendered pages. Several major developer-tools companies publish one for exactly this reason.

That is a developer-experience win. It is not a ranking or citation mechanism. Judge it on the right axis and it looks fine; sell it as an AI visibility lever and you are selling a keywords meta tag in 2026.

What actually makes your site AI-readable: the seven layers

Every one of these is verifiable, and every one of them is something an AI crawler or model demonstrably relies on. This is the substance of AI Visibility & Optimization work, in priority order.

Layer 1: Your content must exist in the HTML source

Most AI crawlers are far less capable than Googlebot at rendering JavaScript. If your product descriptions, pricing, or key claims are injected client-side by React, Vue, or a page-builder widget that loads after paint, a large share of AI crawlers will fetch a near-empty shell.

Test it in ten seconds: open your page, view-source:, and use Ctrl+F to search for a sentence you can see on the rendered page. If the source does not contain it, neither does the crawler's copy. Fixing this is a build-level concern — server-side rendering, static generation, or pre-rendering — which is why AI readiness is often a web and app development problem before it is a content problem.

Layer 2: AI crawlers must be allowed in robots.txt

An enormous number of sites are invisible in AI search for a boring reason: a developer, plugin, or CDN blocked the crawlers. Worse, the distinctions matter — blocking a training bot is a defensible business choice, but blocking a search bot removes you from answers entirely.

User agent	Operator	What it does	Block it?
OAI-SearchBot	OpenAI	Surfaces sites in ChatGPT search answers	Never
ChatGPT-User	OpenAI	Fetches a page when a user asks ChatGPT to look at it	Never
GPTBot	OpenAI	Collects data for model training	Your call
Claude-SearchBot	Anthropic	Indexes content for Claude's search results	Never
Claude-User	Anthropic	Retrieves pages in response to a user's request	Never
ClaudeBot	Anthropic	Collects training data	Your call
PerplexityBot	Perplexity	Indexes for Perplexity answers and citations	Never
Google-Extended	Google	Controls Gemini training/grounding use — not Search indexing	Your call
Applebot-Extended	Apple	Controls Apple Intelligence training use	Your call
CCBot	Common Crawl	Open dataset used by many model builders	Your call

The one-line rule: allow every search and user-triggered agent, and make a deliberate business decision about the training agents. Blocking a training bot costs you nothing today; blocking a search bot deletes you from the answer.

Note the asymmetry in how the providers describe it. Anthropic states that blocking Claude-SearchBot "may reduce your site's visibility and accuracy in user search results," and OpenAI states plainly that sites opted out of OAI-SearchBot "will not be shown in ChatGPT search answers." That is not a ranking penalty. That is deletion.

Layer 3: Answer-first structure

Models extract. They do not appreciate a slow build-up. The pattern that gets quoted is:

1. A heading phrased as the question a real person types.
2. A 40–60 word direct answer immediately beneath it, in a single self-contained paragraph.
3. Supporting detail, caveats and evidence after that.

The test for a well-formed answer paragraph: if you cut it out and paste it somewhere with no surrounding context, does it still make sense and still name the subject? If it starts with "This means that…" or "As we saw above…", it cannot be quoted.

Layer 4: Entity clarity and Organization schema

Before a model can recommend you, it has to be confident you are a single, real, identifiable organisation — not three inconsistent name variants across the web. Entity clarity is unglamorous consistency work:

• One exact legal/brand name, used identically everywhere. Not "Kesar Technolab", "Kesar TechnoLab" and "Kesar Technolab Pvt Ltd" in rotation.
• Organization schema on the homepage with name, url, logo, description, address, telephone — and critically sameAs pointing to every owned profile (LinkedIn, Crunchbase, GitHub, X, Google Business Profile).
• An About page that states, in plain sentences, what the company is, where it operates, when it was founded, and who runs it. Models read this to resolve ambiguity.
• Named authors with real credentials on every article, marked up with author as a Person.

The sameAs array is the single highest-leverage line of schema most sites are missing. It is how you tell a machine "the LinkedIn company page, the Crunchbase profile and this website are the same entity" — which is how corroboration gets attached to you rather than to a competitor with a similar name.

Layer 5: The schema types that are actually worth the effort

Google says no special schema is required for AI features, and that is true. It is also not the whole story: schema is how you make your facts unambiguous and machine-checkable, which reduces the chance a model paraphrases you wrongly or attributes your claim to someone else. Deploy it in this order.

Schema type	Put it on	Why it pays	Priority
Organization + sameAs	Homepage, sitewide	Establishes you as one resolvable entity	Do first
Article / BlogPosting	Every post	Attaches author, publisher and dates to your claims	Do first
BreadcrumbList	Every page	Communicates site hierarchy and topical context	Do first
Service / Product + Offer	Service and product pages	Makes what you sell, where, and at what price machine-legible	High
FAQPage	Pages with real Q&A	Pre-packages extractable question/answer pairs	High
Person (author)	Author bios	Supplies the expertise signal models weigh heavily	High
HowTo	Genuine step-by-step guides	Turns procedures into ordered, quotable steps	Medium
LocalBusiness	Contact page, if you serve locally	Ties you to a place for "near me" style prompts	Medium
Dataset	Original research you publish	Original data is the most-cited content type there is	Medium

One hard rule: schema must describe content a human can see on the page. Marking up FAQs that are not on the page, or reviews you invented, is a spam signal in traditional search engine optimization and a trust problem in AI search. Google is explicit that structured data must match visible content.

Layer 6: Dates, freshness and version signals

Models weight recency heavily for anything time-sensitive, and they read dates from your markup, not your vibes. Publish datePublished and, when you genuinely revise a page, dateModified — and show the date in the visible copy too. Also state the year inside the text of time-bound claims ("as of August 2026, OpenAI operates four documented crawlers"), because that sentence survives being quoted out of context.

The lazy version of this — bumping dateModified nightly with no content change — is detectable and pointless.

Layer 7: Off-site corroboration (the layer that actually decides it)

Here is the uncomfortable part. Layers 1–6 are necessary and entirely within your control, and they are still the smaller half of the problem.

Analysis of AI citation behaviour through 2026 has repeatedly found that AI answers draw heavily on sources outside the classic top ten organic results, and that domain-reference breadth correlates with being cited. SE Ranking's November 2025 analysis found sites referenced across more than 32,000 domains were roughly 3.5x more likely to be cited by ChatGPT. Being cited is a function of how many independent places on the internet say the same thing about you.

Which means the highest-return AI visibility work is often not on your website at all:

• Getting listed and reviewed on the comparison sites, directories and marketplaces in your category — these are disproportionately cited because they aggregate.
• Being an answer on Reddit, Quora, and industry forums where real practitioners discuss your category.
• Publishing original data or benchmarks other people quote — the single most durable citation asset.
• Consistent, verifiable presence on the profiles you listed in sameAs.

Any agency that pitches AI visibility as purely on-site technical work is selling you the easy half. We cover the strategic side of this in our complete AEO and GEO guide for startups, and the mechanics of how agents choose sources in agentic AI search in 2026.

The 20-minute AI-readability self-test

Run this on your own site before you talk to any agency. Each check is pass/fail and takes about two minutes.

1. Source test. view-source: your top service page. Search for a sentence you can see rendered. Present in source? Pass.
2. robots.txt test. Load yourdomain.com/robots.txt. Any Disallow: / for OAI-SearchBot, Claude-SearchBot, Claude-User, ChatGPT-User or PerplexityBot? That is a fail, and it is your first fix.
3. Schema test. Run your homepage and one blog post through Google's Rich Results Test and Schema.org's validator. Do you have Organization with a populated sameAs? Most sites do not.
4. Name test. Search your brand name in Google. Do the top results agree on one spelling, one description, one location?
5. Extraction test. Take your best page's opening paragraph out of context. Does it name the subject and answer something on its own?
6. Author test. Does any article on your site name a real human with credentials? "Admin" and "Team" supply zero expertise signal.
7. Prompt test. Ask ChatGPT, Claude, Perplexity and Gemini the five questions a buyer would actually type — "best [your category] agency in [your city]", "[competitor] alternatives", "how much does [your service] cost". Record whether you appear, who does, and what is cited. That transcript is your real baseline.

Check 7 is the one people skip and it is the only one that measures the outcome rather than a proxy. Do it monthly, in a logged-out or temporary session so your own history does not bias the answer.

Five mistakes we see on nearly every audit

1. Blocking search bots while trying to block training bots. The user agents look similar; the consequences are not. This single misconfiguration is the most common cause of total AI invisibility we find.
2. Treating llms.txt as the deliverable. An hour of work sold as a strategy.
3. Schema that contradicts the page. FAQ markup for questions no visitor can see; aggregate ratings with no reviews behind them.
4. Burying the answer. 400 words of narrative before the page says what it is about. Human readers bounce; models cannot extract.
5. Doing all of it on-site and none of it off-site. Perfect markup, zero third-party mentions, no citations. Necessary conditions are not sufficient conditions.

What to do this week

1. Audit robots.txt for the six search and user agents in the table above. Unblock anything blocked. Cost: 30 minutes. Highest return of anything on this page.
2. Add Organization schema with a complete sameAs array to your homepage.
3. Add Article schema with a named Person author to every blog post.
4. Rewrite the opening 60 words of your five most commercially important pages into self-contained direct answers.
5. Run the seven-question prompt test and save the transcript as your baseline.

Frequently asked questions

Should I create an llms.txt file in 2026?

Only if you publish documentation that people will paste into an AI assistant. It takes about an hour and does no harm. It will not improve your chances of being cited in AI search, because no major AI provider has confirmed reading it and Google has publicly compared it to the deprecated keywords meta tag.

Does schema markup help you rank in AI search?

Google states there is no special schema required to appear in AI Overviews or AI Mode. But schema still matters for AI visibility in a different way: it makes your facts — who published this, when, about what entity — unambiguous and machine-verifiable, which reduces misattribution and misquoting. Treat schema as accuracy insurance rather than a ranking lever.

Which AI crawlers should I allow in robots.txt?

Always allow the search and user-triggered agents: OAI-SearchBot, ChatGPT-User, Claude-SearchBot, Claude-User and PerplexityBot. Blocking these removes you from AI answers entirely. Training-focused agents such as GPTBot, ClaudeBot, Google-Extended, Applebot-Extended and CCBot are a separate business decision with no effect on whether you appear in today's answers.

How do I check whether AI crawlers are actually visiting my site?

Look in your server access logs or CDN analytics and filter by user agent string. Cloudflare, Fastly and most hosting panels expose this. If you see zero requests from OAI-SearchBot or PerplexityBot over 30 days, either you are blocking them or nothing has pointed them at you yet.

Is AEO different from schema markup?

Yes. Schema markup is one technical input. AEO is the broader practice of structuring content so an engine can extract a direct answer — which is mostly about heading phrasing, paragraph structure and factual specificity. You can do excellent AEO with no schema at all, and you can have flawless schema on content no engine would ever quote.

How long before technical AI-readability fixes show up in AI answers?

Crawler-access fixes can register within days to a few weeks, because they remove a hard block rather than compete for position. Content restructuring and entity work typically take one to three months to influence citation behaviour, and off-site corroboration is a six-month-plus programme. Anyone promising AI citations in two weeks is guessing.

Where to read next

• What is AI visibility? The complete AEO and GEO guide for startups in 2026 — start here if the concepts are new.
• Agentic AI search in 2026 — how AI agents choose which sources to cite.
• AEO agency vs SEO agency — which kind of help you actually need, and when neither is worth paying for.
• AI visibility for e-commerce — the product-feed and PDP version of everything on this page.
• AI visibility agency pricing in 2026 — what this work costs across India, the US and the EU.

Sources referenced: Google Search Central documentation on AI features and your website; John Mueller's public remarks on llms.txt as reported by Search Engine Journal (April 2025); OpenAI crawler documentation; Anthropic crawler documentation (February 2026); SE Ranking AI citation analysis (November 2025). Last reviewed August 2026.