llms.txt and a custom agents.json: useful context, not search requirements

Why examine these files at all?

AI tools can consume website information through several routes, including search, direct retrieval and an integration configured by the user. A concise description of a site or its supported capabilities may help a tool designed to use it. That does not make a particular filename a universal protocol.

The llms.txt project is a proposal for presenting information to language-model tools. In this article, agents.json means a devBoys-specific machine-readable description of a site's services or capabilities. It is not a universal Google or OpenAI standard, and publishing either file is not a condition for appearing in generated answers.

What llms.txt is intended to provide

A text resource can offer a brief introduction and links to useful documentation or service pages. Its practical value depends on whether the intended consumer retrieves and uses it. Keep its information consistent with the public website rather than treating it as a separate place for claims you would not show customers.

The llms.txt proposal describes the format and its evolution. Check the current proposal when implementing a file; do not assume every tool follows the same version. A concise resource may reduce irrelevant material for a configured workflow, but it cannot force an external assistant to read it or prevent hallucinations.

How it differs from robots.txt

Robots.txt expresses crawl preferences to cooperating crawlers. It is not a binary index/noindex switch and is not an access-control mechanism. A URL can require separate indexing instructions, and confidential content needs actual authentication and authorization.

An llms.txt resource supplies information or navigation for consumers that support it. It does not override crawl rules, license terms or server permissions. Keeping the two purposes distinct avoids the dangerous assumption that a helpful text file can also protect private data.

What a custom agents.json can describe

A site-specific JSON document might identify a business, link to public information or describe a supported integration endpoint. Its fields need an explicit schema agreed with the consuming application. Without a consumer that understands it, the file is simply another public document.

Do not place credentials, private records or privileged instructions in a public manifest. Describing an API operation does not authorize a caller to perform it. The endpoint must enforce its own permissions and validation.

MCP, OpenAPI descriptions and a custom agents.json serve different roles. An integration may use one or more of them, but renaming a JSON file does not create compatibility with an AI product.

What an agent actually does with the information

A configured tool may fetch the resource, follow selected links and use the resulting content as context. That is a possible integration path, not a documented universal sequence followed by Perplexity, ChatGPT or every crawler.

Test the specific client you intend to support. Record which endpoint it requested, which content it received and whether its answer or action was correct. Server logs alone do not reveal the model's interpretation or prove that it cited your website.

An implementation process

• Identify the consumer and the business task before choosing the format.
• Select public, accurate information and links that already work.
• Generate valid text or JSON and return an appropriate content type.
• Test the resource and referenced URLs in the actual consumer.
• Apply authentication to any private integration separately.
• Include updates and broken-link checks in the normal content workflow.

A root-level URL may be appropriate for the proposed format, while a custom integration should use the documented discovery arrangement its client expects. Do not assume that placing files in the root causes automatic registration with external AI services.

Generating resources from a Laravel application

An application can generate a resource from the same approved data used by its public pages. That reduces the risk of conflicting service descriptions or prices. Version the custom schema where needed, validate the output and include cache invalidation when source content changes.

Separate public summaries from private business tools. A public route should never serialize an entire internal model merely because some of its fields are useful. Test the output for unintended attributes and stale language variants.

Security, intellectual property and private information

A manifest can document intended use, but it does not enforce licensing or stop a hostile scraper. Use server-side access controls for non-public material. Review what an AI client sends to its model provider and which data-handling arrangements apply.

If a tool can edit content or perform a transaction, define narrowly scoped permissions and require appropriate review before consequential changes. Input from a fetched page must not become an unrestricted instruction to your application.

A practical review checklist

• Is the purpose and supported consumer documented?
• Do the files contain only intended public information?
• Do their facts and links match the website?
• Is a custom schema clearly labeled as site-specific?
• Are private endpoints protected independently?
• Can you test real use without assuming search-provider adoption?

What this means for visibility

Google explicitly says its AI search features do not require new AI text files or special markup. Treat public search improvements and a bespoke agent integration as separate proposals with separate evidence. A file may be useful for your own workflow without being a ranking factor.