=== SEOmarketeer Bot Flagger ===
Contributors: seomarketeer
Tags: bots, ai crawlers, security, traffic, mcp
Requires at least: 6.0
Tested up to: 7.1
Requires PHP: 7.4
Stable tag: 1.0.0
License: GPLv2 or later

Logs every request and tells you what it really is: a person, a search engine, an AI crawler, your own cache, or an attack.

== Description ==

Most analytics trust what a visitor says it is. Attack tools know that, so they call themselves Chrome, or Googlebot, or an AI crawler, and walk straight past. SM Bot Flagger checks what they actually do: a request for your cloud credentials is an attack, whatever name it gives.

= What it does =
Each front-end request is written to its own table with:

* date and time, in site time and UTC
* IP address, stored whole or masked
* user agent
* referrer and referring domain
* requested path and query string
* HTTP status code
* whether the request started a new visit, which makes it a landing page
* the category and client name the rules assigned
* how long WordPress took to build the page

Nothing is written to postmeta, and there is one INSERT per request.

= Categories =

Traffic is sorted into six buckets:

* Regular traffic — looks like a person in a browser
* Search bots — Googlebot, Bingbot, DuckDuckBot and friends
* AI bots — GPTBot, ClaudeBot, PerplexityBot, Bytespider, CCBot and around thirty others
* Malicious traffic — sqlmap, Nikto, WPScan, plus around 75 path signatures:
  config and env files, backup leftovers, SSH and private keys, AWS, Azure,
  Google Cloud, Oracle, OVH, Linode and Hetzner credentials, Docker and
  Kubernetes, Terraform state, Vite and Spring Actuator file disclosure, path
  traversal and SQL injection, and credential files for AI coding agents
* Other bots — SEO crawlers, uptime monitors, link previews, HTTP libraries
* Caching — your own cache plugins warming pages: WP Rocket preload and bot,
  WP Fastest Cache, LiteSpeed crawler, W3 Total Cache, WP Super Cache,
  FlyingPress, NitroPack, Cache Enabler, Swift Performance, Cloudflare Always
  Online
* Unclassified — empty or unrecognised user agents

Cache warming is the site requesting its own pages, so it is not a visit and it
is not really a bot either. Keeping it separate stops preload runs from inflating
the traffic numbers. Filter the log to Caching to check a preload actually ran.

Settings has a "Do not log" row where you can tick categories to discard. Those
requests are still classified, then dropped before anything is written, so they
cost one pass over the rules and no database queries at all. This is worth doing
for Caching on a busy site, where a preload run can easily be the largest source
of requests. Nothing is stored, so skipped traffic cannot be recovered later and
will not show up in the log, the charts or the API.

A rule matches on one of three things: the user agent, the request path, or the
IP address. User agent and path patterns match anywhere in the value, with *
as a wildcard and an optional regular expression. Address patterns understand an
exact address, a prefix such as 203.0.113, a wildcard such as 203.0.113.*, and
CIDR ranges such as 203.0.113.0/24 or 2001:db8::/32. Prefixes only match on a
boundary, so 192.168.1 will not match 192.168.10.4.

This is how you handle a crawler that lies about its user agent, or a scraper
running from one hosting range: match the address instead. It is also how you
mark an internal monitoring path or a staging endpoint as something other than
real traffic.

Rules live under SM Bot Flagger → Classification. Each rule matches a substring
or a regular expression against the user agent, the path or the IP. They are
evaluated top to bottom and the first match wins, so put narrow rules above broad
ones. You can edit, reorder, disable, delete and restore them.

= Privacy =

SM Bot Flagger processes personal data in the sense of the GDPR, because an IP
address is personal data. The defaults are chosen to keep as little as
possible:

* IP addresses are masked before storage, to /24 for IPv4 and /48 for IPv6.
  Full addresses are stored only if you switch that on.
* Visitors are counted through a hash of the masked address, the user agent
  and a secret that changes every day, so nobody can be followed from one day
  to the next.
* Values that look like secrets in URLs, such as password reset keys and
  tokens, are removed before a row is written.
* Administrators are not logged by default.
* Entries are deleted after 90 days by default.
* Nothing leaves your server. There is no tracking script, no external
  service, no call home. The plugin author never sees your data.

The plugin adds suggested text to your site's privacy policy guide under
Settings > Privacy > Policy Guide. Adapt it to your configuration. If you
store full addresses or keep entries longer, say so there, and check your
legal basis: for most sites that is a legitimate interest in security, which
should be recorded in your processing register.

The optional MCP endpoint lets a tool you connect read the log. Whoever holds
a token can read the same data you can, so treat tokens as passwords and
revoke the ones you no longer use.

= The MCP endpoint =

Enable it under Settings, then create a token under MCP API.

Endpoint: `https://your-site.tld/wp-json/sm-bot-flagger/v1/mcp`
Transport: streamable HTTP, JSON-RPC 2.0 over POST
Auth: `Authorization: Bearer <token>`, or `X-MCP-Token: <token>` if a proxy
strips the Authorization header

Tools:

* `get_traffic_summary` — totals and the category split for a period
* `get_timeseries` — hourly or daily counts per category
* `get_top` — ranks pages, landing pages, referrers, referring domains, user
  agents, clients, categories, status codes, IPs, rules or hour of day
* `search_hits` — individual requests, with filters
* `classify_user_agent` — test a user agent against the current rules
* `list_classification_rules` — the rules in evaluation order
* `upsert_classification_rule` — add or change a rule, write token only
* `delete_classification_rule` — remove a rule, write token only

Read tokens can only read. Give a token write access only if you want the client
to change how traffic is categorised.

If Authorization never arrives, add this to .htaccess on Apache:

`SetEnvIf Authorization "(.*)" HTTP_AUTHORIZATION=$1`

= Hooks =

Extension points for add-ons:

`sm_bot_flagger_categories` (array) — add your own traffic categories. They appear in the
rules editor, the charts, the filters and the MCP schemas automatically.
`sm_bot_flagger_classification` (array) — override the category, label or rule for a
request, after the rules have run.
`sm_bot_flagger_mcp_tools` (array) — register MCP tools. Include a `callback` key and the
free plugin will dispatch to it.
`sm_bot_flagger_stats_where` (array) — append conditions to every stats query.
`sm_bot_flagger_admin_menu` (action) — add a submenu page under SM Bot Flagger.
`sm_bot_flagger_dashboard_cards` / `sm_bot_flagger_dashboard_panels` (actions) — add to the dashboard.
`sm_bot_flagger_log_columns` (array) and `sm_bot_flagger_log_column` (action) — add columns to the log.
`sm_bot_flagger_settings_fields` (action) and `sm_bot_flagger_sanitize_settings` (array) — add settings.

`SM_Bot_Flagger_Logger::current_classification()` classifies the current request once and
caches it, so an add-on can act on a crawler early rather than at shutdown.

`sm_bot_flagger_should_log` (bool) — return false to skip logging a request.
`sm_bot_flagger_hit_data` (array) — change the row before it is written, or return an empty
array to drop it. Useful for adding a country lookup.
`sm_bot_flagger_mcp_tools` (array) — add your own MCP tools. Give each one a handler named
`tool_<name>` on a class of your choosing and hook `tools/call` accordingly.

= Notes on accuracy =

* Logging happens on `shutdown`, so requests served entirely from a page cache
  or a CDN never reach PHP and are not counted. On a cached site, treat these
  numbers as a lower bound.
* Bot detection relies on self-reported user agents. Anything determined to look
  like a browser will be counted as regular traffic.
* Only pick a proxy header for the IP if a proxy actually sets it. A client can
  send X-Forwarded-For itself, which would let anyone forge their address.

= Security notes =

* Query string values for parameters like key, token, code, password and nonce
  are replaced with [redacted] before a row is written, so password reset links
  and OAuth callbacks do not end up sitting in the log table.
* CSV exports prefix any cell starting with =, +, - or @ with an apostrophe, so
  a crafted user agent cannot run as a formula when the file is opened.
* The MCP endpoint blocks an address for 15 minutes after 20 failed token
  attempts.
* Tokens are not accepted as a ?token= query parameter by default, because
  query strings leak into access logs and referrers. Enable it with the
  `sm_bot_flagger_mcp_allow_query_token` filter if a client cannot send headers.
* A regular expression that does not compile is switched off rather than being
  treated as literal text, so a typo cannot silently widen what a rule matches.
* Rule patterns are matched against at most the first 1000 characters of a
  header, which bounds the cost of a badly written expression.

A write-scoped MCP token can change classification rules, and those rules run on
every front-end request. Treat a write token as trusted, and hand out read
tokens for anything that only needs to look at the numbers.

= Behaviour rules =

Pattern rules judge a request on what it looks like. Behaviour rules judge a
client on what it has been doing. They run at log time, because they need the
response status and a short memory, and they need a stored IP address, so they
do nothing when IP storage is off. Confirmed search and AI crawlers are never
demoted by them.

Random paths. A 404 on a single path segment of 4 to 12 alphanumeric characters
with no vowels and mixed case, such as /6bXX29bt, is treated as scanning. Real
slugs contain vowels; generated tokens do not. The 404 is required, so a short
consonant heavy page of your own is never caught.

Repeated 404s. Flagged on both a count and a share: by default 8 misses within
10 minutes, and at least half of that client's requests in the window missing.
The share is what makes this safe. A busy legitimate client can collect a
handful of 404s among hundreds of good requests, and counting alone would flag
it. Scanners miss on almost everything they ask for.

Login attempts. A cumulative count of requests to wp-login.php and xmlrpc.php,
10 per hour by default. Deliberately not a rate: credential stuffing is usually
slow and spread over many addresses, so a per-second limit never sees it.

Request rate, off by default. On real traffic the fastest clients are usually
legitimate, while scanners and credential stuffing run slowly enough to pass any
threshold worth setting. Counting 404s separates the two far better than
counting requests. Measure your own traffic before switching this on.

Once a client trips a rule it stays flagged for an hour by default, and what it
already did in that window is recategorised too.

= Path precedence =

A user agent is a claim the client makes about itself. Anything can send
"OAI-SearchBot" or "Googlebot", and plenty of attack tooling does, because it
gets past naive filtering.

Settings has "Trust the path", on by default, which tests path and address rules
before user agent rules. A request for /root/.aws/credentials is then recorded
as an attack whatever it claims to be, and your genuine crawler rules stay
correct for genuine crawlers.

Turn it off if you would rather rules were evaluated strictly in the order shown
on the Classification screen.

= Filtering =

Dashboard tiles toggle categories, and several can be combined. Rows in the top
lists are links: click a page, landing page, status code, client or referring
domain to open the traffic log filtered to it.

The log itself filters on category, status family (2xx to 5xx), an exact status
code, a page path, landing pages only, and a free text search across path, user
agent, referrer, IP and client. Combining a path with "landing pages only" gives
you the entry traffic for one page. Filters survive paging and are applied to
CSV exports.

= Correcting a classification =

Open the Traffic log and click the client name on any row. You get the user
agent as an editable pattern, a name, and a category. Shorten the pattern to the
distinctive part before saving, for example `MyScraper/2.` rather than the whole
string, so the rule catches other versions too.

Saving puts the rule first in the evaluation order, so it beats the shipped
defaults. Tick the box to also rewrite requests that are already in the log,
which is matched on the user agent containing your pattern.

The chart above the table follows the filters, and can be grouped by category,
by client or by the raw user agent. The eight busiest values get their own band
and the rest are folded into "Everything else". "Show full user agent" prints
the whole string under each client name.

== Installation ==

1. Install through Plugins > Add New, or upload the zip.
2. Activate. Logging starts straight away with privacy-friendly defaults.
3. Open SM Bot Flagger in the admin menu. The dashboard fills as traffic arrives.
4. Optional: under Settings, turn on the MCP endpoint and create a token to
   connect Claude or another MCP client.

== Frequently Asked Questions ==

= Will it slow my site down? =

Logging happens at the very end of each request, as one database write.
Behaviour rules only keep counters for clients that have already requested a
missing page.

= Does it work with page caching? =

Pages served straight from a cache never reach WordPress, so they are not
logged. On a heavily cached site, read the numbers as a floor rather than a
total. Your cache plugin's own warming requests are recognised and kept out
of your visitor numbers.

= Is it GDPR compliant? =

It is built to keep as little as possible: masked addresses, a visitor hash
that changes daily, secrets removed from URLs, administrators excluded, and
90-day retention. See the Privacy section. Compliance also depends on how you
run your site: adapt the suggested privacy policy text, and record your legal
basis.

= Does any data leave my server? =

No. There is no tracking script, no external service and no call home. The
plugin author never sees your traffic.

= What is the MCP endpoint? =

A small server inside WordPress that lets an AI assistant or any MCP client
read the traffic log and ask questions in plain language. It is off by
default, needs a token you create, and read-only tokens cannot change
anything.

= Can I correct a classification? =

Yes. Click a client in the traffic log, choose the category, and optionally
match on the path or address instead of the user agent. Past requests are
corrected too.

= Is there a paid version? =

Yes. The Pro add-on adds verified crawlers, so a fake Googlebot is caught by
DNS rather than by name; blocking by category, behaviour or address, with a
monitor mode; and priority support. See https://seomarketeer.eu.

= Where do I get help? =

In this plugin's support forum on WordPress.org, or through
https://seomarketeer.eu.

== Screenshots ==

1. The dashboard: who is visiting, over time and by category.
2. The traffic log with the chart grouped by user agent.
3. Correcting a classification from a log row.
4. The classification rules.
5. The MCP endpoint with a token and client configuration.
6. Settings, including the behaviour rules and privacy options.

== Changelog ==

= 1.0.0 =
* First public release.
