robots.txt, llms.txt and agents.json: different files, different purposes

Files placed at the root of a website can look deceptively similar. A robots.txt file, an llms.txt document and a JSON manifest may all describe something about the site, but they do not have the same status or purpose. Treating them as interchangeable can lead to both poor search decisions and unsafe assumptions about access control.

The useful question is not which file makes a website “AI-ready”. It is which audience or integration needs information, what recognised mechanism that audience uses and how the website enforces any real permissions.

robots.txt: instructions for cooperative crawlers

robots.txt is used to communicate crawling preferences to crawlers that support the Robots Exclusion Protocol. Rules can apply to particular user agents and paths. Search providers may operate different crawlers for different purposes, so a decision about one bot should not automatically be applied to every service from that provider.

For example, OpenAI documents separate user agents for search, training-related crawling and user-initiated page retrieval. Their purposes and treatment of site preferences need to be checked in the provider's current documentation before changing a policy.

A disallow rule is not authentication. It does not stop a visitor or an uncooperative client from requesting a public URL. It is also not a reliable substitute for controlling whether a URL appears in search: a crawler may need access to the page to see a noindex instruction.

Private documents, customer records and administrative endpoints require actual access controls. Listing a sensitive path in robots.txt does not make it private.

llms.txt: an optional proposal for contextual information

The llms.txt project proposes a way to make useful information available to language-model tools. It can provide a concise introduction and links to relevant resources, but it is not a universal requirement adopted by every search engine or AI service.

Google's guidance for AI features explicitly says that no additional machine-readable AI file is required for inclusion. Creating llms.txt therefore should not be sold as a condition for appearing in AI Overviews, AI Mode or a model's answer.

If a particular integration consumes such a document, make it useful for that integration: accurate, concise and maintained alongside the public site. Do not assume that adding the file forces external systems to read it, refresh their knowledge or cite the business.

agents.json: a custom manifest, unless a specific integration defines otherwise

In the approach described here, agents.json is a devBoys-specific convention for describing selected public capabilities or resources. It is not presented as a universal web protocol or an official requirement of Google or OpenAI.

A custom manifest can be useful when both sides agree on its structure. For example, an internal assistant might use it to discover a documented endpoint or the public scope of an integration. The format, versioning and consumer must be defined explicitly.

A declaration in JSON cannot grant or enforce permission by itself. An endpoint that changes an order, accesses customer data or starts a paid operation still needs authentication, authorisation, validation and appropriate approval. The server must enforce those rules even if a client ignores the manifest completely.

Compare the roles before adding more files

These mechanisms can coexist, but they solve different problems. A public website should remain understandable through its normal pages, navigation and accurate metadata even when no optional AI document is present.

Generate integration documents from a controlled source

A Laravel application can generate a text or JSON response from approved public data. This can reduce drift between the website and an integration document, provided the same publication rules apply to both.

Select fields deliberately. A database model may contain internal notes, unpublished prices or administrative values that must never be serialised simply because they are available. Use an explicit output structure, the correct content type and tests for both included and excluded information.

Caching should reflect how often the underlying public information changes. Consider how updates invalidate the cached document and how a consumer can recognise a version change. Dynamic generation is useful, but it does not make external providers fetch the result in real time.

Separate discovery from actions

An assistant finding a product description is a different situation from an assistant placing an order. Read-only public resources can often be exposed with modest risk. Actions involving money, personal data or business state require a much more carefully defined boundary.

For an actionable integration, document the input schema, required permissions, possible side effects and error responses. Decide which operations require user confirmation. Restrict credentials to the intended service and record enough information to investigate failures without leaking secrets into logs.

MCP is one established protocol for connecting AI applications to tools and resources. Whether it is the right choice depends on the intended client and workflow; a normal authenticated API may be sufficient.

A practical review for website owners

• Identify the actual crawlers and integration clients you want to support.
• Keep ordinary search requirements, indexability and useful content in good order.
• Label optional or custom formats honestly.
• Generate public documents only from approved information.
• Enforce permissions at the application boundary.
• Test updates, failures and sensitive-data exclusion.

The objective is a clear, maintainable interface, not a collection of fashionable filenames. A custom MCP integration can help when there is a defined business task and a consumer that will use it. Optional discovery files should support that task without being mistaken for security or a search guarantee.