---
title: "llms.txt vs robots.txt vs MCP: what each one actually does"
description: "Understand the different roles of robots.txt, llms.txt, MCP and WebMCP, and choose the right mechanism for crawling, documentation or actions."
canonical: "https://getdedica.com/blog/llms-txt-vs-robots-txt-vs-mcp"
last-updated: "2026-10-30"
---

# llms.txt vs robots.txt vs MCP: what each one actually does

When a new file or protocol gets attention, site owners often ask whether they should add it to every project. That is understandable, but `robots.txt`, `llms.txt`, MCP and WebMCP do different jobs. None is a universal “make my site work with AI” switch.

## robots.txt expresses crawl preferences

The file at `/robots.txt` tells compliant crawlers which paths a site owner asks them to request or avoid. It is a public convention, not a password or firewall. A crawler can ignore it, and the file does not let a blocked request through a CDN challenge. It also does not tell an agent what a pricing page means.

Use it when you want to express crawl preferences for particular bots or paths. Check the crawler's current documentation before writing rules. Some organizations operate separate crawlers for search, training and user-directed fetching, and the rules for one do not automatically control the others. A request that is blocked at the network layer needs a firewall review as well.

## llms.txt is a proposed guide for language-model readers

The `llms.txt` proposal suggests a Markdown file that points language models toward useful site information. Some documentation projects publish one to help a reader find their reference material. It does not block or authorize crawlers, and publishing it does not guarantee a model will read or rely on it.

Google Search Central says Google Search, including its generative AI features, does not use `llms.txt` as a special signal. Google says the file is optional for other services or systems that may use it. So add one when it makes your documentation easier for a known consumer to navigate, not because you expect it to improve Google rankings. The [official Search guidance](https://developers.google.com/search/docs/fundamentals/ai-optimization-guide) is explicit on this point.

## MCP exposes tools to an AI application

MCP is a protocol for an AI host to discover and call tools made available by a server. For example, an authenticated customer might connect a support tool that can find their open cases. An MCP server is an integration surface, not a crawl directive and not a replacement for public pages that explain your product.

Choose MCP when a real customer task benefits from a structured operation and you are prepared to maintain authentication, permissions, schemas, rate limits and error handling. A tool that can read a customer's account needs server-side authorization on every call. A tool that changes a record should make that side effect clear and preserve the product's confirmation model. See the [MCP tools specification](https://github.com/modelcontextprotocol/modelcontextprotocol/blob/main/docs/specification/2026-07-28/server/tools.mdx).

## WebMCP exposes tools from a page

WebMCP is an early browser API for registering structured actions from a web page. Chrome documents an imperative JavaScript API and a declarative form API. It can help an agent interact with a form or page task after it has opened the site in a supported browser. It does not create a server-side integration for a remote assistant that never visits the page.

Chrome currently describes WebMCP as an origin-trial feature and says clients must visit the site directly to discover the tools. Treat it as a progressive enhancement and check the current [Chrome documentation](https://developer.chrome.com/docs/ai/webmcp) for browser support and limitations.

## Choose by the problem

| If you need to… | Start with… | Remember… |
|---|---|---|
| Tell compliant crawlers which paths they may request | `robots.txt` | It does not enforce access or grant permission through a firewall. |
| Help a known reader find long-form documentation | `llms.txt` or a well-organized docs index | It is optional, and does not promise Google visibility. |
| Let an AI application call an authenticated product capability | MCP | Build and secure it like a production API. |
| Help a supported browser agent use a page or form | WebMCP | It is experimental and tied to a browser page context. |

Most businesses should first make key public pages reachable, accurate, linked and understandable. Add a protocol when it solves a specific task for a specific client. This keeps the implementation tied to a user need instead of chasing a checklist item.

Dedica's [free website readiness scan](/) focuses on public site access, content and usability signals. It does not require `llms.txt`, create an MCP server or certify WebMCP. Use the scan to find site issues, then test whichever integration matches the task you want to support.

For the firewall side of crawler access, read [How to let AI agents through Cloudflare and other bot protection](/blog/let-ai-agents-through-bot-protection).

## Sources

- [Optimizing your website for generative AI features on Google Search](https://developers.google.com/search/docs/fundamentals/ai-optimization-guide), Google Search Central.
- [WebMCP](https://developer.chrome.com/docs/ai/webmcp), Chrome for Developers.
- [MCP server tools specification](https://github.com/modelcontextprotocol/modelcontextprotocol/blob/main/docs/specification/2026-07-28/server/tools.mdx), Model Context Protocol.
- [Overview of OpenAI Crawlers](https://developers.openai.com/api/docs/bots), OpenAI Developers.