=== MSG LLMs TXT Manager ===
Contributors: cropllm
Donate link: https://myseogrowth.com/donate
Tags: llms-txt, markdown, ai, agents, crawlers
Requires at least: 6.0
Tested up to: 7.1
Requires PHP: 7.4
Stable tag: 2.0.6
License: GPLv2 or later
License URI: https://www.gnu.org/licenses/gpl-2.0.html

Serve your content as clean Markdown to AI agents, publish llms.txt and llms-full.txt, and see which agents actually fetch what.

== Description ==

Most llms.txt plugins publish one file and stop. Measured server logs show that file is fetched mainly by SEO bots rather than by AI assistants, while the technique that does deliver content to agents is ordinary HTTP content negotiation. This plugin does both, and then shows you which one is working on your site.

**Content negotiation.** When a client requests a page with `Accept: text/markdown`, it gets a clean Markdown version instead of HTML. Coding agents use this. Browsers, search crawlers and everything else are unaffected and still receive HTML.

**Markdown mirrors.** Any page is available as Markdown by appending `.md` to its URL. Mirrors are served with a noindex header so they never compete with your pages in search.

**llms.txt.** A spec-compliant index: one H1, a blockquote summary, H2 sections of links with descriptions, and an `Optional` section. Built from your published content, or written by hand.

**llms-full.txt.** Every listed page as Markdown in a single file, so an agent can read the whole site in one request. Size-capped so it stays inside a model's context window.

**Agent activity log.** These clients do not run JavaScript, so they never appear in your analytics. The plugin records which agents request which paths and in which format, including requests for pages that do not exist, which tells you the titles agents expect you to have.

**Crawler rules.** Optional Allow and Disallow rules for AI user agents, written into robots.txt where crawlers actually read them.

Everything is virtual. No files are written to disk.

= What this will and will not do =

It will make your content cheap and accurate for an agent to read, and it will tell you who is reading it.

It will not lift your Google rankings. Google has stated that llms.txt has no effect on Search or AI Overviews, and no major AI provider has committed to consuming the file. Treat this as infrastructure for agents, not as an SEO tactic. If your logs show your main AI reader is fetching HTML, the highest-value work is still keeping your rendered HTML clean.

= For developers =

Filters: `msgltm_index_document`, `msgltm_full_document`, `msgltm_post_document`, `msgltm_posts_for_type`, `msgltm_is_excluded`.

== Installation ==

1. Upload and activate the plugin.
2. Go to **Settings → LLMs TXT Manager**.
3. On the Content tab, choose which post types to include and check the preview.
4. On the Settings tab, write a summary, then run the diagnostics to confirm every endpoint responds correctly.

== Frequently Asked Questions ==

= What is llms.txt? =

A Markdown file at the root of a site giving AI models a curated map of the content. It was proposed by Jeremy Howard of Answer.AI in September 2024. It is a community convention, not a standard from any recognised body.

= Do AI crawlers actually read it? =

Rarely. Log studies consistently show the file is fetched mostly by SEO tools and generic bots rather than by AI assistants. Markdown delivered through content negotiation sees far more real agent use. Both are enabled here, and the Agent activity tab will tell you which is true for your site rather than leaving you to guess.

= Will the .md mirrors cause duplicate content problems? =

No. Every mirror is sent with an `X-Robots-Tag: noindex, nofollow` header, so search engines will not index them.

= Why does the negotiation check say "blocked by" my cache? =

Page caches such as Breeze, WP Super Cache and Varnish store the HTML of a URL and serve it to every later request for that URL before WordPress loads. They key on the URL alone, so an agent asking for Markdown gets the stored HTML and the plugin never sees the request. This is a property of how those caches work, not a bug. Agents still get Markdown through the `.md` mirror and through the `rel="alternate"` link on every page, both of which work behind any cache. Negotiation works fully on hosts whose cache honours `Vary: Accept` or where pages are not cached.

= Does content negotiation break my caching? =

It should not, and the plugin works hard to make sure. Negotiated responses carry `Vary: Accept`, `Cache-Control: no-store`, the `DONOTCACHEPAGE` constant that WordPress cache plugins honour, and the LiteSpeed and CDN equivalents. Even so, some caches ignore all of that. The diagnostics test for exactly this: they request the page as a browser, then as an agent, then as a browser again. If the second browser request comes back as Markdown, negotiation is switched off automatically and the cache is purged. Run the diagnostics after enabling negotiation and after changing cache plugins.

= Is llms.txt the same as robots.txt? =

No. robots.txt controls crawler access. llms.txt recommends content and cannot block anything. Allow and Disallow lines inside an llms.txt file are ignored by every crawler, which is why this plugin writes those rules to robots.txt instead.

= Will this slow my site down? =

Generated files are cached and rebuilt when you publish. Markdown conversion happens once per post and is cached until the post changes. Nothing calls out to the network.

== Screenshots ==

1. Settings tab: surfaces, index file and crawler rules.
2. Content tab: post type selection and a live preview.
3. Agent activity: which agents read your site, and in which format.

== Changelog ==

= 2.0.6 =
* Fixed: a fatal error on /llms.txt when WordPress core sitemaps are enabled (sites without an SEO plugin). The sitemap URL was read from the wrong object.
* Fixed: root-relative links in converted Markdown dropped a non-standard port from the site URL.
* Verified end to end inside WordPress 7.1: every endpoint, header, negotiation path, admin screen, form handler, nonce check, the meta box, activation, upgrade migration, and uninstall.

= 2.0.5 =
* Changed: diagnostics now recognise when a page cache answered a request before WordPress ran. Content negotiation blocked in this way is reported as a limitation of the cache, with an explanation, rather than as a failure.
* Changed: the content negotiation setting explains which caching setups it can and cannot work behind.

= 2.0.4 =
* Changed: responses are now sent with `Cache-Control: public, max-age=0, must-revalidate` instead of a five-minute lifetime. Browsers and CDNs revalidate on every request and receive a 304 when nothing has changed, so edits appear immediately.

= 2.0.3 =
* Fixed: caches that capture output through PHP's output buffer (Breeze, WP Super Cache, WP Rocket, W3 Total Cache, Cache Enabler and others) could store every plugin response and re-serve it labelled text/html, regardless of headers or DONOTCACHEPAGE. The plugin now closes all output buffers before writing any response, so such caches capture nothing. Verified against Breeze's own cache handler.
* Fixed: purging Breeze now uses its full purge, which also clears Cloudways Varnish.
* Added: diagnostics name the active page cache and give cache-specific exclusion advice when needed.

= 2.0.2 =
* Fixed: page caches that ignore the Vary header could store a negotiated Markdown response under the HTML URL and serve it to browsers. Negotiated responses are now marked uncacheable at every layer (DONOTCACHEPAGE, LiteSpeed API, Cache-Control no-store, CDN and surrogate headers).
* Fixed: the "browsers get HTML" diagnostic trusted the Content-Type header, which a cache can fake. It now inspects the body, runs before and after the Markdown request, and if it finds Markdown being served to browsers it switches negotiation off and purges the cache.
* Added: a purge button that clears the plugin's caches and asks WP Super Cache, W3 Total Cache, WP Rocket, LiteSpeed, WP Fastest Cache, SiteGround Optimizer, Cache Enabler, Breeze, Hummingbird, WP Engine or Kinsta to purge.
* Changed: upgrading from 2.0.0 or 2.0.1 switches content negotiation off and purges caches, with a notice asking you to run the diagnostics before re-enabling it.

= 2.0.1 =
* Fixed: Manual mode was saved and shown in the admin but never served. /llms.txt always returned the generated index. Manual content is now served correctly.
* Changed: the endpoint cache header dropped from one hour to five minutes, so edits appear quickly. ETags still keep repeat requests cheap.
* Added: the Settings tab states which document is currently being served, and the Content tab warns when its preview is not what visitors receive.

= 2.0.0 =
* Added: content negotiation. A request with `Accept: text/markdown` receives Markdown, with a `Vary: Accept` header for caches.
* Added: Markdown mirrors at any URL plus `.md`, served noindex.
* Added: `llms-full.txt`, a single size-capped file containing every listed page as Markdown.
* Added: an HTML to Markdown converter handling headings, lists, nested lists, tables, code blocks, blockquotes, links and images.
* Added: agent activity logging, showing per-agent HTML and Markdown counts and requests for pages that do not exist.
* Added: per-post controls to pin, exclude, or override the description.
* Added: descriptions are taken from Yoast, Rank Math, SEOPress or AIOSEO when present, and pages those plugins mark noindex are excluded.
* Added: diagnostics that request every endpoint and report what came back.
* Added: `rel="alternate"` Markdown link in the page head, ETag and conditional request support, and developer filters.
* Security: text and titles are escaped so page content can never become live markup or forge a link in a generated file.

= 1.6.0 =
* Fixed: the served file was HTML-escaped, which corrupted Markdown characters such as `>` and `&`.
* Changed: the index now follows the llms.txt specification instead of robots.txt-style directives.
* Changed: per-bot rules write to robots.txt, where crawlers read them.
* Changed: the generator reads local posts instead of making an HTTP request per URL.
* Fixed: works with plain permalinks and on subdirectory installs.

= 1.5.5 =
* Readme alignment and metadata updates.

== Upgrade Notice ==

= 2.0.6 =
Fixes a fatal error on sites using WordPress core sitemaps. Recommended for everyone.
