=== FoundByAI — SEO for AI Bots ===
Contributors: thenooch
Donate link: https://venmo.com/u/Daniel-Neumann-7
Tags: ai, seo, llms, chatgpt, markdown
Requires at least: 5.6
Tested up to: 7.0
Requires PHP: 7.4
Stable tag: 1.1.10
License: GPL-2.0-or-later
License URI: https://www.gnu.org/licenses/gpl-2.0.html

Audit every page for AI visibility, then serve clean Markdown + llms.txt so ChatGPT, Claude & Perplexity can find and cite your content.

== Description ==

AI tools like ChatGPT, Claude, and Perplexity now send real traffic to the sites they cite — but most WordPress pages are invisible to them. FoundByAI shows you exactly what's holding each page back, then fixes the plumbing for you: it audits every page for AI readiness, serves clean Markdown to AI crawlers, and builds your llms.txt automatically.

Built specifically for AI visibility — not bolted onto a legacy SEO plugin. It tracks 20 AI bots, runs a per-page AI audit with concrete fixes, and works alongside Yoast, Rank Math, and AIOSEO (reading their scores, never fighting them).

= Who is this for? =

* ✅ **Bloggers & publishers** who want to get quoted when people ask an AI a question
* ✅ **Store owners** (WooCommerce) who want products discoverable by AI shopping assistants
* ✅ **Marketers & agencies** managing AI visibility across client sites

No code required, and no need to replace the SEO plugin you already use.

= What FoundByAI does for you =

**Content Serving**
* HTTP Content Negotiation: Automatically serve Markdown when crawlers request `Accept: text/markdown`
* .md URL Endpoints: Visit `/post-slug.md` to download any post as clean Markdown
* Custom Post Type Support: Serve Markdown for Posts, Pages, or custom post types
* WooCommerce Integration: Auto-detects WooCommerce and displays dedicated settings for product discovery
* YAML Frontmatter: Each Markdown file includes metadata in YAML format

**AI Discovery (llms.txt)**
* Dual Endpoints: Generate llms.txt at both `/.well-known/llms.txt` and `/llms.txt`
* Auto-Detection: Auto-discover and prioritize by SEO plugin scores
* 3-Tier Prioritization: SEO score > post date > manual pinning rules
* Smart Filtering: Auto-exclude <200 word posts, password-protected, private, noindex content
* 1-Hour Cache: Intelligent caching with smart invalidation
* Dynamic Pin Rules: Boost posts by tag, category, or manual selection

**Dynamic Bot Management**
* 20 Curated Bots: GPTBot, Claude-Web, PerplexityBot, Google-Extended, Cohere, Meta AI, ByteDance, and more
* Enable/Disable Controls: Manage which bots to track and serve
* Extensibility Filter: Register custom bots with `sfab_register_bots`

**SEO Plugin Integration**
* Auto-Detect: Yoast SEO, Rank Math, All In One SEO
* Score Reading: Use SEO scores for llms.txt prioritization
* Noindex Respect: Respect SEO plugin's noindex settings
* Manual Override: Force specific plugin for edge cases

**Advanced Controls**
* Per-Post Discovery: Include/exclude individual posts from llms.txt
* Word Count Threshold: Filter posts by minimum word count
* ACF Field Support (Optional): Include custom fields in Markdown frontmatter
* Bot Tracking: Monitor which AI crawlers visit your content
* Logs Dashboard: View and analyze bot visit history

**Admin UI**
* 5-Tab Settings: Detection, Content, Discovery, Bots, Advanced
* llms.txt Preview: See generated discovery file in admin
* Pin Rules Manager: Add/remove tag, category, and post-specific boosts
* Responsive Design: Works on desktop and mobile

= How It Works =

1. **AI crawlers discover** your Markdown endpoints via `rel="alternate"` links
2. **They request** your content with `Accept: text/markdown` or visit `.md` URLs
3. **Plugin serves** clean Markdown + YAML frontmatter
4. **Discovery file** (llms.txt) lists your best posts in priority order
5. **Analytics** track which AI crawlers visit and how they engage
6. **SEO stays intact** - no interference with your SEO plugin

= Strategic Positioning =

This plugin is a **specialist tool**, not a competitor to SEO plugins:
* ✅ Owns: Markdown serving, AI crawler discovery, llms.txt generation, AI bot analytics
* ✅ Complements: Yoast, Rank Math, All In One SEO (reads their scores, respects their rules)
* ✅ Smart robots.txt: Auto-injects llms.txt rules when safe, shows manual instructions otherwise
* ✅ Stays out of: XML sitemaps, core SEO meta tags

= Why optimize for AI at all? =

AI language models need clean, structured content. Instead of parsing messy HTML filled with navigation, ads, and scripts, they prefer semantic Markdown. By offering this format, you:

* Improve AI understanding of your content
* Increase likelihood of being cited or referenced by AI tools
* Attract GPTBot, ClaudeBot, Perplexity, and other AI crawlers
* Maintain full control over content distribution
* Boost visibility to AI applications

= Compatibility =

✅ Yoast SEO, Rank Math, All In One SEO (full integration)
✅ WooCommerce (dedicated product discovery settings)
✅ Advanced Custom Fields (optional ACF field support)
✅ All custom post types
✅ WordPress 5.0+ and PHP 7.4+
✅ No external dependencies, single-file plugin

= Test It =

After activation, test your setup:

1. Visit SEO for AI Bots > Settings
2. View any tab to verify configuration
3. Visit your site's `/llms.txt` endpoint
4. Visit any post with `.md` extension: `yoursite.com/your-post.md`
5. View bot visits in SEO for AI Bots > Bot Visits Log

== Installation ==

1. Download the plugin
2. Go to Plugins > Add New > Upload Plugin
3. Select `seo-for-ai-bots.zip` and Install
4. Activate Plugin
5. Go to SEO for AI Bots > Settings to configure

= Manual Installation =

1. Upload `seo-for-ai-bots` folder to `/wp-content/plugins/`
2. Activate through the Plugins menu
3. Configure in SEO for AI Bots

== Frequently Asked Questions ==

= What is llms.txt? =

llms.txt is a simple, standardized index file (like robots.txt, but for AI) that tells large language models such as ChatGPT, Claude, and Perplexity which pages on your site matter most and how to read them. FoundByAI generates and maintains yours automatically, in priority order.

= Which AI bots does FoundByAI detect? =

FoundByAI tracks 20 bots out of the box — including GPTBot and ChatGPT-User (OpenAI), ClaudeBot and Claude-Web (Anthropic), PerplexityBot, Google-Extended, Applebot, Meta AI, ByteDance, Cohere, xAI Grok, and more. You can enable or disable any of them, and register your own with a simple filter.

= How do I enable llms.txt? =

llms.txt is enabled by default after activation. Visit SEO for AI Bots > Discovery to:
* Preview your llms.txt file
* Add pinning rules (boost posts by tag, category, or ID)
* View posts that will be included

= Can I control which posts appear in llms.txt? =

Yes, three ways:
1. **Global threshold**: Settings > Content sets minimum word count (default 200)
2. **Per-post checkbox**: Edit any post to include/exclude from llms.txt
3. **Pinning rules**: Settings > Discovery to boost specific posts or categories

= What's the difference between .md URLs and llms.txt? =

* **/.md URLs** (e.g., `/my-post.md`): Individual post downloads in Markdown format
* **/llms.txt**: Directory file listing your best posts for AI crawler discovery

= How does prioritization work? =

3-tier system:
1. **SEO Score** (0-1000 points): From your SEO plugin (Yoast, Rank Math, etc.)
2. **Post Date** (0-100 points): Newer posts get higher score, decays over 1 year
3. **Manual Pins** (highest): Boost specific posts or categories

= Will it slow down my site? =

No. Processing only happens for:
* Markdown requests (visitors requesting .md files or Markdown content type)
* llms.txt generation (cached for 1 hour)
* Regular page loads are unaffected

= Do I need an SEO plugin? =

No. The plugin works standalone. With an SEO plugin, it reads scores for smarter prioritization. Without one, it uses post dates and manual pinning rules.

= What if my SEO plugin isn't detected? =

Go to Settings > Detection and manually select your plugin from the dropdown. This is rarely needed.

= Can I add custom AI bots? =

Yes, use the filter: `sfab_register_bots`

Example:
```php
add_filter( 'sfab_register_bots', function( $bots ) {
    $bots[] = array(
        'pattern' => 'MyCustomBot',
        'name'    => 'My Custom AI Bot',
        'enabled' => true,
        'custom'  => true,
    );
    return $bots;
} );
```

= Does it work with ACF (Advanced Custom Fields)? =

Yes! If ACF is installed, go to Settings > Content to select which ACF fields to include in Markdown frontmatter.

= How do I test Markdown output? =

Method 1: Visit any post with `.md`:
```
yoursite.com/my-post.md
```

Method 2: Use curl:
```
curl -H "Accept: text/markdown" https://yoursite.com/my-post/
```

Method 3: Check llms.txt at:
```
yoursite.com/llms.txt
```

= Can I customize llms.txt format? =

Use the filter: `sfab_llms_posts` to modify the posts list before generation.

= How many logs does it keep? =

Currently keeps last 500 bot visits with rolling expiration. Logs older than 14 days are auto-removed.

= Is there multisite support? =

Not in v2.0. Each site needs the plugin configured separately.

= Does this plugin collect any data? =

**Anonymous Telemetry (Opt-in, Disabled by Default)**

The plugin includes optional anonymous telemetry. When you enable it in Settings > Privacy:

* **Collected:** Anonymized site hash (not your URL), plugin version, active SEO plugin name, active FAQ plugin names, and aggregated bot visit counts (7-day totals)
* **NOT collected:** Your site URL, page content, personal data, or user information
* **Frequency:** Sent weekly to our analytics endpoint (Google Apps Script)
* **Purpose:** Helps us understand which plugins to prioritize for compatibility
* **Control:** Enable/disable anytime in Settings > Privacy

**Bot Visit Logging (Local Only)**

The Bot Activity feature stores visit data locally in your WordPress database:

* **Stored:** Timestamp, page visited, bot name, user agent, IP address, request type
* **Purpose:** Analytics to show which AI bots visit your content
* **Retention:** Last 500 visits, auto-deleted after 14 days
* **Location:** Your database only - never sent externally
* **GDPR Note:** Bot IPs are typically datacenter IPs, not personal. If you prefer not to store IPs, you can disable bot tracking in Settings.

= Does the plugin connect to external services? =

The core plugin works entirely offline. Optional features that connect externally — see the "External Services" section below for full details on each service, what data is sent, and links to their terms and privacy policies.

== Screenshots ==

1. See exactly how AI reads each page — with the specific fixes to improve it (AI Readiness audit).
2. Your weekly AI visibility snapshot at a glance.
3. Preview the llms.txt AI crawlers will read — before it goes live.
4. Know which AI crawlers actually visit your content, with 90-day history.
5. Catch AI-readiness issues before you hit publish, right in the editor.
6. Set it up once — five simple tabs, no code.

== External Services ==

This plugin optionally connects to the following external services. All connections are initiated by user action or explicitly opt-in — the core plugin functions entirely offline.

= 1. SEO for AI Bots Proxy (seoforaibots.com) =

* **What it does:** Powers WAF/firewall bot access checks and AI search visibility checks (Pro), and receives anonymized usage telemetry if you opt in.
* **Data sent:** Site URL (for visibility checks), anonymized site hash (not your URL), plugin version. Telemetry is never sent unless you explicitly enable it.
* **When:** Visibility and firewall checks only when you click the relevant "Check" button. Telemetry weekly only if you opt in under Settings > Privacy (off by default). The plugin does not fetch configuration or contact this server on its own.
* **Terms of Service:** https://seoforaibots.com/terms-of-service/
* **Privacy Policy:** https://seoforaibots.com/privacy-policy/

= 2. Google Search Console API (googleapis.com) — Pro, opt-in =

* **What it does:** Retrieves your site's search performance data to power the AI Coverage Gap dashboard (shows pages with Google traffic but no AI bot visits).
* **Data sent:** OAuth access token for authentication; your site's GSC property URL to filter results.
* **When:** Only when you connect GSC via the API Keys tab (one-click OAuth flow) and visit the AI Coverage Gap dashboard. No data is sent until you initiate the connection.
* **Terms of Service:** https://policies.google.com/terms
* **Privacy Policy:** https://policies.google.com/privacy

= 3. Wikipedia API (en.wikipedia.org) — on-demand =

* **What it does:** Checks whether your brand has a Wikipedia article, shown as an advisory in the AI Readiness dashboard.
* **Data sent:** Your brand name (from plugin settings).
* **When:** Only when you view the AI Readiness dashboard with a brand name configured. Results are cached for 24 hours.
* **Terms of Use:** https://foundation.wikimedia.org/wiki/Policy:Terms_of_Use
* **Privacy Policy:** https://foundation.wikimedia.org/wiki/Policy:Privacy_policy

== Upgrade Notice ==

= 1.1.10 =
Privacy & security update: the plugin no longer makes any automatic calls to our server (configuration now ships with the plugin), and JSON-LD handling is hardened. Recommended for all users.

= 1.1.9 =
Big free upgrade: content refactoring suggestions, per-post AI context & JSON-LD, content handler controls, WooCommerce/custom post type support, all llms.txt power controls, llms-full.txt, and extended bot history are now free for everyone. Also fixes a crash on the AI Readiness page.

= 1.1.8 =
New: AI Coverage Gap dashboard (Pro) — connects Google Search Console to show which pages get Google traffic but zero AI bot visits. New API Keys tab in Configuration for managing GSC connection and AI Simulator keys.

= 1.1.7 =
Security and compatibility fixes: eliminates duplicate OG/schema tags when Rank Math, Yoast, or AIOSEO is active; adds nonce verification to admin filter pages and AJAX handlers; hardens JSON-LD output and SQL queries per WordPress.org guidelines.

= 1.1.6 =
Hotfix: fatal error on Pages/Posts list when Markdown column is enabled. Update immediately.

= 1.1.5 =
Fix: AI Checklist now correctly detects Rank Math meta descriptions. Meta box scores refresh after saving.

= 1.1.4 =
Hotfix: resolves a fatal error in wp-admin affecting Pro users. Update immediately.

= 1.1.0 =
Public launch — Free + Pro now available. UI improvements and Freemius product rename fix.

= 1.0.2 =
Freemius integration improvements.

= 1.0.1 =
Code quality improvements for WordPress.org submission.

= 1.0.0 =
Initial public release on WordPress.org.

== Changelog ==

= 1.1.10 - 2026-06-22 =
* Privacy: Removed all automatic background calls to our server. The plugin no longer fetches remote configuration or bot definitions on a schedule — bot definitions ship with the plugin and update with each release. Nothing is sent off-site unless you explicitly opt in to telemetry.
* Security: Custom JSON-LD entered per post is now decoded and re-encoded (sanitized) before it is stored, rather than saving the raw input.
* Security: Speakable schema output no longer uses JSON flags that could weaken escaping inside the script tag.
* Housekeeping: Removed the legacy daily config-refresh cron from existing installs.

= 1.1.9 - 2026-06-14 =
* Free: Content refactoring suggestions are now available to everyone — actionable structural tips to make each page more AI-friendly
* Free: Per-post AI context and custom JSON-LD schema fields
* Free: Per-post content handler controls (summarize / extract / remove how AI bots see specific content)
* Free: WooCommerce and custom post type support, plus ACF field inclusion
* Free: All llms.txt power controls — custom intro text, cache duration, pinning/priority rules, taxonomy filtering, per-type post limits, and override mode
* Free: llms-full.txt endpoint (full-content export per the llmstxt.org spec)
* Free: Extended 90-day bot activity history, Insights tab, and bot management tab
* Free: Wikipedia brand-presence advisory and the full AI Readiness dashboard
* Fix: AI Readiness page could fail to load all pages due to an internal error — now lists every page correctly
* Note: Pro now focuses on the features that rely on external AI/data services — AI Search Simulator, Competitor Intelligence, AI Coverage Gap (Google Search Console), and the firewall bot-access check

= 1.1.8 - 2026-05-05 =
* New: AI Coverage Gap dashboard (Pro) — connect Google Search Console to see pages with high Google impressions but zero AI bot visits, with HIGH/MED/LOW gap scoring
* New: API Keys tab in Configuration — centralizes AI Simulator API keys and GSC OAuth connection in one place
* New: Google Search Console OAuth integration via proxy relay — client secret stays server-side, one-click connect flow

= 1.1.7 - 2026-05-05 =
* Fix: Rank Math, Yoast, and AIOSEO compatibility — no longer outputs duplicate OG tags or Article JSON-LD when a major SEO plugin is active
* Fix: llms-full.txt rules removed from robots.txt copy-paste block and auto-inject for free users (that URL returns a Pro-only error page)
* Fix: Plugin no longer intercepts /llms.txt when Rank Math, Yoast, or AIOSEO is active — SEO plugin owns that URL
* Fix: /.well-known/llms.txt now proxies the active SEO plugin's /llms.txt content instead of serving a duplicate
* Fix: Plugin activation no longer hard-flushes .htaccess rewrite rules, preventing conflicts with other plugins' rewrite rules
* New: AI discovery meta tags (llms-txt, ai-sitemap) now output on every page, not just singular posts
* New: robots.txt health checks panel in AI Readiness dashboard — 5 checks including well-known/llms.txt accessibility, AI crawler rules, and Sitemap redirect detection
* Fix: Inline JSON-LD validator in meta box now uses wp_add_inline_script() instead of a raw script tag
* Fix: AI Readiness filter parameters (search, status, type, sort) now require a valid nonce — prevents unauthorized filtering
* Fix: Competitor Intelligence page GET parameters now require a valid nonce
* Fix: Referral tracker AJAX handler now verifies a nonce before processing POST data
* Fix: JSON-LD schema output uses wp_json_encode() without flags that could bypass WordPress escaping rules
* Fix: Database visit log query now passes LIMIT and OFFSET through wpdb::prepare() placeholders

= 1.1.6 - 2026-03-19 =
* Fix: Fatal error on Pages/Posts list — sfab_render_markdown_column used bare constant instead of string

= 1.1.5 - 2026-03-10 =
* Fix: AI Checklist meta description check now correctly detects Rank Math descriptions (including auto-generated from excerpt)
* Fix: Bottom panel meta box now shows a "Refresh" notice after saving in Gutenberg so scores stay current
* Fix: SEO Score check now shows your actual score (e.g. "14/100 via Rank Math") alongside the fix recommendation

= 1.1.4 - 2026-03-09 =
* Fix: Fatal error in wp-admin — analytics-pages.php now loads before ai-readiness.php so ai_markdown_server_get_analytics_cache() is always defined

= 1.1.3 - 2026-03-05 =
* Fix: Fatal error on frontend pages — moved remote config functions to core-functions.php so they're always available, not just on admin pages
* Added error notification system: plugin emails the site admin on critical errors with rate-limiting (max once per hour per error type); falls back gracefully instead of crashing

= 1.1.2 - 2026-03-04 =
* Internal build fix: resolved premium package deployment issue

= 1.1.1 - 2026-03-04 =
* Improved AI Readiness page: rewrote help accordion sections in plain English with friendlier tone
* Redesigned help cards to match site visual style (warm palette, serif headers, no colored borders)
* Added ? trigger buttons on table headers to expand relevant help sections
* Improved Sitemap tab: added help note explaining how to exclude pages from AI Markdown Sitemap; added smart link to active SEO plugin's sitemap settings
* Updated per-post checkbox label to "Include in AI discovery (llms.txt & ai-sitemap)"
* Fixed h3 font size across admin pages (1.4rem)

= 1.1.0 - 2026-03-02 =
* Improved empty state UI: replaced emoji with custom robot logo on Bot Activity pages
* Added dismissable helpful notice on AI Readiness page explaining that low-scoring pages may be intentional (login, password reset, dashboard pages) with suggestion to use noindex

= 1.0.3 - 2026-02-05 =
* Added Rule Analysis tools for Custom Handlers (Pro): Find Conflicts, Find Overlaps, Consolidate Rules
* Improved rule management with automatic detection of duplicate and conflicting patterns

= 1.0.2 - 2026-02-04 =
* Improved Freemius uninstall integration for better user feedback tracking

= 1.0.1 - 2026-02-04 =
* Improved internationalization with proper translator comments
* Code quality improvements for WordPress.org compliance
* Updated to WordPress 6.9 compatibility

= 1.0.0 - 2026-02-04 =
Initial public release.

**Content Serving**
* Serve clean Markdown to AI crawlers via HTTP Content Negotiation
* .md URL endpoints for any post (e.g., /your-post.md)
* YAML frontmatter with metadata
* Custom post type support
* WooCommerce product integration

**AI Discovery (llms.txt)**
* Dual endpoints at /.well-known/llms.txt and /llms.txt
* llms-full.txt comprehensive variant
* 3-tier prioritization (SEO score > date > manual pins)
* Smart filtering (word count, noindex, private content)
* Pin rules for tags, categories, and specific posts

**Bot Management**
* 18+ curated AI bots (GPTBot, ClaudeBot, Perplexity, etc.)
* Enable/disable individual bots
* Bot visit analytics dashboard
* Unknown bot tracking
* Bot blocking via .htaccess

**SEO Plugin Integration**
* Auto-detect Yoast SEO, Rank Math, All In One SEO
* Read SEO scores for prioritization
* Respect noindex settings

**Content Analysis**
* AI Readiness scoring
* Image alt text checker
* Content structure analysis
* Citation potential calculator
* Competitor llms.txt analysis

**Universal Content Handler**
* Extract content from 20+ plugins (WPForms, Gravity Forms, Elementor, etc.)
* Custom extraction rules with CSS/XPath selectors
* FAQ pattern detection

**Additional Features**
* Onboarding wizard with site scan
* AI Search Simulator to check visibility
* AI referral tracking
* Content freshness alerts
* Per-post AI context customization
* Extensibility filters for developers
* Optional anonymous telemetry (disabled by default)

== Support ==

For issues, feature requests, or bug reports, please use the WordPress.org support forum or visit:
https://seoforaibots.com

== Translations ==

This plugin is ready for translation. Contribute at:
https://translate.wordpress.org/projects/wp-plugins/seo-for-ai-bots/

== Additional Resources ==

* [Dries Buytaert's "The Third Audience"](https://dri.es/the-third-audience) - The concept behind this plugin
* [llms.txt Standard](https://llmstxt.com/) - AI-Friendly Discovery Format
* [Content Negotiation (MDN)](https://developer.mozilla.org/en-US/docs/Web/HTTP/Content_negotiation)
* [GPTBot Documentation](https://platform.openai.com/docs/gptbot)
* [Markdown Guide](https://www.markdownguide.org/)
* [Yoast SEO](https://yoast.com/wordpress/plugins/seo/)
* [Rank Math](https://rankmath.com/)
* [All In One SEO](https://aioseo.com/)
