Configuration Guide
Complete reference for configuringBadBehaviour via config/bb_config.php (and related files). Covers every option, the strictness levels, bot-registry customization, logging, and operational recipes.1. Quick start
The minimum viable configuration:
<?php
// config/bb_config.php
return [
'preset' => 'minimal',
'strictness' => 'monitor-only', // start safe, ramp up later
'logging' => true,
];
That's it. Drop the file in
config/, restart your app, and you'll start logging to the bad_behaviour table. From there, expand to:<?php
return [
// Required-ish
'preset' => 'minimal',
'strictness' => 'monitor-only',
'logging' => true,
// Behind Cloudflare (set true + add CF ranges)
'reverse_proxy' => [
'enabled' => true,
'header' => 'CF-Connecting-IP',
'addresses' => [
'173.245.48.0/20',
'103.21.244.0/22',
// ... (all CF ranges)
],
],
// Custom category overrides
'bot_categories' => [
'blocked' => ['residential_proxy'],
'challenge' => ['ai_crawler'],
'log_only' => ['security_scanner'],
'allowed' => ['feed_reader'],
],
// See also: custom_rules for per-bot or per-IP rules
'custom_rules' => [
['type' => 'ip', 'value' => '203.0.113.0/24', 'action' => 'block'],
],
];
Verify it's loaded:
$ php bin/diagnose.php
If
config_loaded: true and safe_mode: false, you're set.2. How configuration works
2.1 Where the file lives
Adapters look for
config/bb_config.php in this order:1. CONFIG_DIR/bb_config.php (if CONFIG_DIR constant defined — WackoWiki) 2. config/bb_config.php (CWD-relative — CLI tools) 3. <package_root>/config/bb_config.php (most installations)
The file is a PHP script that must
return an array. It's not a JSON/YAML/INI file — it's PHP, which lets you do conditional logic, env var lookups, and computed defaults.2.2 The merge order
When you provide a config, three layers merge in this exact order:
defaults ← Configuration::get_defaults()
↓
strictness layer ← Configuration::strictness_overrides($strictness)
↓
user config ← YOUR bb_config.php
↓
final config ← Configuration::from_array() result
User config always wins. This is the cardinal rule — you can override anything by setting it in your
bb_config.php, including features that strictness-level overrides would normally enable or disable.Example:
'normal' strictness enables DNS verification. But you can disable it explicitly:return [
'strictness' => 'normal',
'dns_verification' => ['enabled' => false], // overrides 'normal' default
];
2.3 Safe mode
If
config/bb_config.php is missing or broken, the library falls back to safe mode:- All active defenses are OFF
- DNS/network-touching features are OFF
- Logging stays ON (you can still observe traffic)
- A warning is logged once per process
Safe mode is never a hard error — the library always boots. This ensures a misconfigured BadBehaviour can't take your site down.
You can check safe-mode status at runtime:
if ($bb->is_in_safe_mode()) {
// Config file missing or invalid — operator needs to fix it
}
2.4 The Configuration object
Configuration is a readonly class. Once built via Configuration::from_array(), properties are immutable. If you need to change config, build a new Configuration.$config = Configuration::from_array([
'preset' => 'minimal',
'strictness' => 'monitor-only',
'logging' => true,
], $adapter);
// Read properties:
echo $config->strictness; // 'monitor-only'
echo $config->log_table; // 'bad_behaviour' (injected by adapter)
echo $config->dns_verification_enabled; // false (monitor-only override)
// Round-trip back to array:
$array = $config->to_array();
// Inject into BadBehaviour:
$bb = new BadBehaviour($config);
2.5 Validation & clamping
All numeric inputs are clamped to safe ranges. Invalid values fall back to defaults rather than throwing:
| Input | Clamp behavior |
|---|---|
dns_verification_timeout_ms: 50 |
min 50, max 2000 |
dns_verification_positive_ttl: 100 |
min 3600 |
rate_limits.global.requests: -5 |
max(1, $value) |
strictness: 'invalid-level' |
falls back to 'normal' |
preset: 'nonexistent' |
falls back to 'full' (with error log) |
This means you can be sloppy with values and the library will still boot.
3. Strictness levels
Three levels control how aggressively BadBehaviour defends your site:
return [
'strictness' => 'monitor-only', // or 'normal' or 'strict'
];
| Strictness | What it does | What it doesn't do |
|---|---|---|
monitor-only |
Log everything, block only obvious attacks (empty UA, raw XSS). Zero risk of breaking real users. | No DNS verification, no behavioral, no rate limits, no fingerprinting, no DNSBL. |
normal (default) |
DNS verification ON (catches bot spoofers), rate limiting ON (catches scrapers), unverified bots logged not blocked. Experimental detectors stay OFF. | No fingerprinting, no behavioral, no client hints. |
strict |
Everything ON. Forward DNS confirmation (catches PTR spoofing). Unverified AI blocked. Tighter rate limits. All experimental detectors enabled. | — |
Per-feature behavior per strictness
| Feature | monitor-only | normal | strict |
|---|---|---|---|
dns_verification_enabled |
OFF | ON | ON |
dns_verification_require_forward_confirm |
OFF | OFF | ON |
dns_verification_positive_ttl |
7d | 7d | 30d |
dynamic_ip_ranges_enabled |
OFF | ON | ON |
rate_limit_enabled |
OFF | ON (1000/hr, 60/min) | ON (500/hr, 30/min) |
enable_fingerprinting |
OFF | OFF | ON |
enable_behavioral_analysis |
OFF | OFF | ON |
enable_client_hints_validation |
OFF | OFF | ON |
enable_agentic_detection |
OFF | OFF | ON |
enable_head_request_detection |
OFF | OFF | ON |
enable_asset_scraping_detection |
OFF | OFF | ON |
block_unverified_ai |
OFF | OFF | ON |
strict_search_engines |
OFF | OFF | ON |
dnsbl_enabled |
OFF | OFF | ON |
When to use which
-
monitor-only: First 1–2 weeks of deployment. Or any time blocking real users is worse than letting bots through (e.g., public-facing site, e-commerce checkout). -
normal: Production baseline. Catches the worst offenders without breaking search engine indexing or mobile app traffic. -
strict: When actively under attack — spam flood, credential stuffing, content scraping. Enable briefly, evaluate FP rate, disable if FP rate is too high.
User values override strictness
You can enable a feature even in monitor-only:
return [
'strictness' => 'monitor-only',
'rate_limit_enabled' => true, // ← enable rate limits but keep monitor-only demotion
'enable_behavioral_analysis' => true, // ← see behavioral patterns in logs without enforcing
];
Or disable a feature even in strict:
return [
'strictness' => 'strict',
'block_unverified_ai' => false, // ← AI crawlers logged not blocked even at strict
];
4. Bot categories (Option A)
Pin entire bot categories to a specific action regardless of their default behavior:
return [
'bot_categories' => [
'blocked' => [], // hard-block by category
'challenge' => [], // force CAPTCHA by category
'log_only' => [], // log but never block
'allowed' => [], // allow verified-by-default categories
],
];
Priority order (most severe wins)
If you put
social_crawler in both blocked and allowed, blocked wins:blocked[] > challenge[] > log_only[] > allowed[]
If you don't list a category anywhere, the default category-specific logic runs (see [Bot Registry wiki page] for per-category defaults).
Available category values
The full list of values you can put in any of the four sub-keys:
| Value | What it is | Default behavior |
|---|---|---|
search_engine |
Google, Bing, Yandex, Baidu, etc. | verified → allow; unverified → block |
ai_crawler |
GPTBot, ClaudeBot, Gemini, etc. | depends on ai_crawlers.* config |
social_crawler |
Facebook, Twitter, Discord, etc. | verified → allow; unverified → log_only |
seo_crawler |
Ahrefs, Semrush, MJ12bot, etc. | verified → default_action; unverified → block |
archive_crawler |
Internet Archive, Common Crawl, BnF, etc. | allow verified |
feed_reader |
Feedly, Inoreader, Apple News, etc. | allow verified |
shopping_crawler |
Google Shopping, Bing Shopping, etc. | allow verified |
monitoring |
UptimeRobot, Pingdom, StatusCake, etc. | allow verified |
cloud_infrastructure |
Cloudflare, AWS ELB, GCP LB, Azure, Fastly | HARD allow — cannot be overridden |
security_scanner |
Qualys, Shodan, Censys, Detectify, Rapid7 | log_only |
residential_proxy |
Bright Data, etc. | block |
malicious |
User-defined bad bots (no built-in entries) | block (if listed in blocked[]) |
The CLOUD_INFRASTRUCTURE safety override
cloud_infrastructure always returns ALLOW — even if you put it in blocked[]. Blocking these takes your origin offline because CDN/LB health probes will be denied and your origin will be marked unhealthy.The check runs before user overrides, so this is structurally impossible to bypass by configuration.
Examples
Block residential proxies (commercial scraping networks):
'bot_categories' => [
'blocked' => ['residential_proxy'],
],
Challenge all social media scrapers (stop Facebook/Twitter link previews):
'bot_categories' => [
'challenge' => ['social_crawler'],
],
Log security scanners without blocking:
'bot_categories' => [
'log_only' => ['security_scanner'],
],
Force RSS feed readers to be allowed:
'bot_categories' => [
'allowed' => ['feed_reader', 'archive_crawler'],
],
Multiple categories, mixed actions:
'bot_categories' => [
'blocked' => ['residential_proxy'],
'challenge' => ['ai_crawler'], // challenge ALL AI crawlers
'log_only' => ['security_scanner'],
'allowed' => ['feed_reader'],
],
Round-trip guarantee
All four sub-keys are read in
from_array() and written in to_array(). If you write:'bot_categories' => ['challenge' => ['ai_crawler']]
You'll see it preserved in
$config->to_array()['bot_categories']. No silent drops.5. Bot registry
Configure which bots are recognized and how they're matched. Lives in
config/bb_registry.php (separate from bb_config.php).5.1 Presets
return [
'preset' => 'minimal', // default
];
| Preset | Bots loaded | Use case |
|---|---|---|
full |
100 (everything shipped) | Most sites; max coverage |
minimal |
30 (high-traffic bots only) | Default. Fastest matching. |
verified-only |
70 (only bots with DNS verification or IP ranges) | Tighter, may miss regional bots |
no-ai |
85 (everything except AI crawlers) | Sites blocking AI training scrapers |
no-seo |
88 (everything except SEO crawlers) | Sites blocking SEO crawlers |
eu-only |
25 (European + GDPR-relevant bots) | EU-hosted sites |
human-only |
0 (empty registry) | Use with additions to ship only your custom bots |
custom |
varies (only bots[]) |
Define the complete registry yourself |
5.2 Filters
return [
'preset' => 'minimal',
// Drop whole categories
'exclude_categories' => ['seo_crawler', 'shopping_crawler'],
// Re-add categories (overrides exclude). CRITICAL for cloud_infrastructure.
'include_categories' => ['cloud_infrastructure'],
// Drop specific bots by ID
'exclude_bots' => ['petal'],
// Add custom bots on top
'additions' => [
'my_internal_bot' => [
'name' => 'My Internal Crawler',
'user_agent_patterns' => ['MyBot', 'MyBot/1.0'],
'category' => 'monitoring',
'ip_ranges' => ['10.0.0.0/8'],
'default_action' => 'allow',
],
],
];
5.3 Cloud infrastructure safety net
If you filter aggressively (
human-only, no-ai excludes monitoring, etc.), your CDN's health probes may be blocked — origin gets marked unhealthy — site goes offline.Always force-include
cloud_infrastructure:return [
'preset' => 'human-only',
'include_categories' => ['cloud_infrastructure'], // ← safety net
'additions' => [
'your_bot' => [...],
],
];
The shipped
config/bb_registry.php has this safety net built in.5.4 Custom bot schema
For
additions and custom preset:'additions' => [
'my_bot' => [
// Required
'name' => 'Human-readable bot name',
'user_agent_patterns' => ['MyBot', 'MyBot/1.0'], // ≥1 pattern, ≥3 chars each
'category' => 'search_engine', // see category list above
// Optional
'host_patterns' => ['bot.example.com'],
'ip_ranges' => ['192.0.2.0/24', '2001:db8::/32'],
'verify_dns' => true,
'dns_suffixes' => ['bot.example.com'],
'robots_txt_token' => 'MyBot',
'default_action' => 'allow', // allow|challenge|block|log_only
'description' => 'What this bot does',
],
],
Invalid entries are logged via
error_log() and skipped — the rest of your registry still loads.6. Logging & enforcement tracking
6.1 Basic logging
'logging' => true, // default
Writes blocked/challenged requests to the
bad_behaviour table. Set to false to disable logging entirely (you'll have no forensics — only do this if you have another log source).6.2 Verbose mode
'verbose' => true,
When
true, allowed requests are also logged. Without this, only blocked/challenged/monitored requests appear in the log. Set verbose=true for the first 1–2 weeks to see your full traffic pattern.Storage impact: 50–500 MB/day for a small site. Plan accordingly.
6.3 The enforcement_action column
Every log row carries one of three values:
| Value | Meaning | HTTP outcome |
|---|---|---|
enforced |
Detection ran and the response was actually changed (403 served). | Request blocked |
monitored |
A block/challenge was detected but suppressed (monitor-only mode). | Request served normally |
allowed |
No block was detected; request was allowed to proceed. | Request served normally (only logged when verbose=true) |
6.4 The original_code column
For
monitored rows: the blocked.X code that WOULD have applied. For enforced rows: NULL.Example row interpretation:
status_code enforcement_action original_code meaning monitored.bot monitored blocked.bot Would have blocked, didn't monitored.malicious_ua monitored blocked.malicious_ua Would have blocked, didn't blocked.malicious_ua enforced NULL Empty UA, actually 403'd blocked.attack_pattern enforced NULL Raw XSS, actually 403'd allowed allowed NULL Normal request (verbose=true only)
6.5 Log schema indexes
The
bad_behaviour table includes indexes optimized for monitoring dashboards:KEY idx_ip (ip)
KEY idx_status (status_code)
KEY idx_date (date)
KEY idx_bot (bot_category, bot_verified)
KEY idx_enforcement (enforcement_action, date) -- monitor-only queries
6.6 Verification queries
After running for a week in monitor-only mode:
-- 1. Is monitor-only actually working?
SELECT enforcement_action, COUNT(*) AS n
FROM bad_behaviour
WHERE date >= CURDATE()
GROUP BY enforcement_action;
-- Expected (monitor-only):
-- enforced ~0-20 (only empty UA + raw XSS)
-- monitored ~50-5000 (bots, attacks, rate limits)
-- allowed 0 (set verbose=true if you want these)
-- 2. What would have been blocked if I'd been at 'normal' strictness?
SELECT status_code, COUNT(*) AS n
FROM bad_behaviour
WHERE enforcement_action = 'monitored'
AND date >= CURDATE()
GROUP BY status_code
ORDER BY n DESC;
-- 3. Top offenders
SELECT ip, COUNT(*) AS hits, MIN(date) AS first_seen
FROM bad_behaviour
WHERE enforcement_action = 'monitored'
AND date >= NOW() - INTERVAL 7 DAY
GROUP BY ip
ORDER BY hits DESC
LIMIT 20;
-- 4. Bot category breakdown
SELECT
JSON_UNQUOTE(JSON_EXTRACT(metadata, '$.bot_category')) AS bot_category,
COUNT(*) AS n
FROM bad_behaviour
WHERE metadata LIKE '%bot_category%'
AND date >= CURDATE()
GROUP BY bot_category
ORDER BY n DESC;
7. Rate limiting
Detect and block volumetric abuse:
'rate_limits' => [
'enabled' => true,
// All requests per IP per hour
'global' => ['requests' => 1000, 'window' => 3600],
// Burst protection per IP per minute
'per_minute' => ['requests' => 60, 'window' => 60],
// Form submissions per IP per hour
'post' => ['requests' => 30, 'window' => 3600],
// Login attempts per IP per 15 minutes (credential stuffing)
'login' => ['requests' => 10, 'window' => 900],
],
What each bucket catches
| Bucket | Triggers on | Catches | |||
|---|---|---|---|---|---|
global |
Every request | Volumetric scraping | |||
per_minute |
Every request | Burst attacks, download accelerators | |||
post |
POST/PUT/PATCH | Form spam, comment spam | |||
login |
POST to URL matching ##(login | signin | auth | password)## | Credential stuffing |
Tuning for shared NAT IPs
Shared IPs (corporate networks, mobile carriers, VPN exit nodes) legitimately aggregate many users. Default thresholds assume one-user-per-IP. If you serve corporate users:
'rate_limits' => [
'global' => ['requests' => 5000, 'window' => 3600], // 5x headroom
'per_minute' => ['requests' => 300, 'window' => 60], // 5x headroom
],
Check your logs first:
-- What are realistic request rates per legitimate IP?
SELECT
ip,
COUNT(*) AS requests_per_hour,
COUNT(DISTINCT DATE_FORMAT(date, '%Y-%m-%d %H:00:00')) AS hours_active,
COUNT(*) / NULLIF(COUNT(DISTINCT DATE_FORMAT(date, '%Y-%m-%d %H:00:00')), 0) AS avg_per_hour
FROM bad_behaviour
WHERE enforcement_action = 'allowed'
AND date >= NOW() - INTERVAL 7 DAY
GROUP BY ip
HAVING requests_per_hour > 100
ORDER BY requests_per_hour DESC
LIMIT 50;
Set your limits to 5x the 95th percentile of legitimate traffic.
Custom bucket matching
The
login bucket matches URLs containing login, signin, auth, or password (case-insensitive). To add more patterns, use custom_rules:'custom_rules' => [
[
'type' => 'header',
'header' => 'X-Login-Path',
'value' => 'yes',
'action' => 'challenge',
],
],
Then in your app, set
X-Login-Path: yes on your login route.8. DNS verification
Synchronous reverse-DNS + optional forward-confirm verification for bots claiming to be Google/Bing/etc:
'dns_verification' => [
'enabled' => true,
'timeout_ms' => 300,
'require_forward_confirm' => false,
'positive_ttl' => 604800, // 7 days
'negative_ttl' => 3600, // 1 hour
],
How it works
1. Bot UA claims "Googlebot" 2. Library checks IP against static Google ranges 3. If no static match: reverse-DNS lookup (PTR record) 4. Result: "crawl-1-2-3.googlebot.com" 5. Check suffix matches "googlebot.com" 6. If yes → ALLOW (cached for positive_ttl seconds) 7. If no → CHALLENGE or BLOCK depending on config
Cost: 40–300ms on FIRST request per bot IP. Subsequent requests hit the cache. Most production traffic is cached after the first 50 bot requests.
When to enable require_forward_confirm
Forward confirmation also verifies the host's A/AAAA records contain the original IP. This catches PTR spoofing (attacker sets their own PTR to
crawl-1–2-3.googlebot.com).Catches: PTR spoofing, hijacked hostnames.
FP risk: HIGH on IPv6. Many IPv6 setups have inconsistent forward/reverse DNS, and enabling forward confirm will block legitimate bots. Default OFF.
Only enable when you observe actual PTR spoofing abuse:
'dns_verification' => [
'require_forward_confirm' => true, // turn on if you're seeing PTR spoofers
],
TTL tuning
| TTL | Behavior |
|---|---|
positive_ttl: 604800 (7d, default) |
Verified IPs cached 7 days. Standard. |
positive_ttl: 2592000 (30d, strict) |
Cache longer; less re-checking but stale on IP rotation. |
negative_ttl: 3600 (1h, default) |
Failed lookups re-checked every hour. |
negative_ttl: 86400 (1d, strict) |
Less re-checking, faster table growth. |
If you observe transient DNS failures (your authoritative DNS hiccups), lower
negative_ttl so retries happen sooner.Performance impact
DNS lookups happen synchronously in the request path. For a busy site:
- 95% of bot traffic is cached after the first 50 requests
- Cache is adapter-backed (file cache, Redis, Memcached, MediaWiki WAN cache)
- Negative TTL of 1h means transient DNS failures self-heal within an hour
For very high traffic (>1000 RPS), consider:
- Increasing
dns_verification_timeout_msif your DNS is slow (default 300ms is generous) - Monitoring
log_request_failureevents in your adapter logger - Falling back to
'strictness' => 'monitor-only'during DNS outages
9. AI crawler control
Manage GPTBot, ClaudeBot, and other AI training scrapers:
'ai_crawlers' => [
'allowed' => ['GPTBot', 'ClaudeBot', 'Google-Extended'],
'block_unverified' => false,
'strict' => false,
],
The allowed list
These bots bypass DNS verification — they're allowed regardless of whether their DNS resolves correctly:
'allowed' => [
'GPTBot', // OpenAI
'ChatGPT-User', // OpenAI on-demand
'OAI-SearchBot', // OpenAI search
'ClaudeBot', // Anthropic
'Claude-Web', // Anthropic search
'Google-Extended', // Google AI training
'PerplexityBot', // Perplexity
'GrokBot', // xAI
'MistralBot', // Mistral
'YouBot', // You.com
'Meta-ExternalAgent', // Meta AI
],
The token is the bot's
robots_txt_token (the name used in robots.txt). Check the shipped registry for the exact token for each bot. block_unverified vs strict
These are independent settings:
| Setting | Behavior |
|---|---|
block_unverified: false (default) |
Unverified AI bots are CHALLENGED (CAPTCHA served). Pass = allow. |
block_unverified: true |
Unverified AI bots are BLOCKED outright. No CAPTCHA option. |
strict: false (default) |
AI bots not in allowed list are CHALLENGED. |
strict: true |
AI bots not in allowed list are BLOCKED. |
Recommended for publishers:
block_unverified: true + allowed: [] (or only the AI services you have a partnership with).Recommended for general sites: defaults (false/false) — challenge is more humane than block, and bot operators can fix their DNS to gain access.
robots.txt integration
Add AI bots to your
robots.txt to declare your policy:User-agent: GPTBot Disallow: / User-agent: ClaudeBot Disallow: / User-agent: Google-Extended Disallow: /
BadBehaviour doesn't parse your robots.txt (that's the bot's job to honor), but the bot categories in your config should mirror your robots.txt policy.
10. DNSBL & http:BL
Network-dependent IP reputation lookups.
DNSBL (Spamhaus, Spamcop, etc.)
'dnsbl' => [
'enabled' => true,
'lists' => ['zen.spamhaus.org', 'bl.spamcop.net'],
],
Cost: One DNS query per IPv4 request. IPv6 is skipped (DNSBLs are IPv4-only).
FP risk: Moderate. Spamhaus PBL flags residential IPs (correct for mail, wrong for web). Listed IPs are often compromised devices, not the actual attackers.
Use when: Receiving spam that gets through the rest of the pipeline. Disable if you observe false positives on residential IPs.
http:BL (Project Honeypot)
'httpbl' => [
'key' => 'your-access-key', // from https://www.projecthoneypot.org/
'threat' => 25, // minimum threat score (0-255)
'maxage' => 30, // max days since last activity
],
Requires a free Project Honeypot API key.
FP risk: Low. Well-curated, low false-positive rate.
Use when: Specifically seeing comment spam, form spam, or harvester traffic.
When to enable either
| Symptom | Enable | Why |
|---|---|---|
| Form/comment spam getting through | http:BL | Honeypot catches spammers specifically |
| Generic volumetric abuse | DNSBL | Catches compromised hosts |
| Mostly browser traffic, few bots | neither | Detection overhead > value |
| Public-facing site, low FP tolerance | neither | FP risk on residential IPs |
11. Fingerprinting (JA3 / HTTP/2)
Block known-bad TLS and HTTP/2 fingerprints:
'enable_fingerprinting' => true,
'fingerprints' => [
'bad_ja3' => ['e7d705a3286e19ea42f587b344ee6865'],
'bad_h2' => ['a1b2c3d4e5f6...'],
'bot_header_orders' => ['...'],
'expected_ja3' => ['chrome-ja3-here'],
],
What each field catches
| Field | Catches |
|---|---|
bad_ja3[] |
Specific JA3 hashes known to belong to scraper frameworks |
bad_h2[] |
Specific HTTP/2 SETTINGS hashes known to be abusive |
bot_header_orders[] |
Specific header ordering patterns (e.g., python-requests) |
expected_ja3[] |
(Future) expected JA3s per browser version for consistency checks |
Where fingerprints come from
BadBehaviour doesn't compute JA3 itself — your reverse proxy must forward it via header:
| Source | Header |
|---|---|
| Cloudflare Enterprise | CF-Ray-Ja3, X-Client-Ja3 |
| HAProxy | SSL-Client-Ja3 |
| nginx + ja3-nginx-module | X-Client-Ja3 |
| Generic | X-Ja3-Fingerprint |
If your proxy doesn't forward JA3, the field is NULL and detection is skipped. BadBehaviour never inspects TLS itself (would require a TLS-terminating proxy anyway).
When to use
Only enable if:
- You have a reverse proxy that forwards JA3
- You've identified specific abusive JA3 hashes via traffic analysis
- You can verify the FPs are acceptable
Default is OFF — the FP risk of blacklisting a JA3 hash is “you've locked out an entire TLS library version for one misclassification.”
12. GeoIP blocking
Block traffic by country or ASN:
'geoip' => [
'enabled' => true,
'database_path' => '/usr/share/GeoIP/GeoLite2-Country.mmdb',
'blocked_countries' => ['CN', 'RU'],
'blocked_asns' => ['AS15169'], // Google
],
Setup
- Download MaxMind GeoLite2-Country.mmdb (free, requires registration): https://www.maxmind.com/en/geolite2/signup
- Place it at a path readable by your PHP process
- Set
geoip.enabledtotrueand the database path
Limitations
- GeoIP databases are 60MB — adds startup memory cost
- VPNs/proxies make the IP→country mapping meaningless (a request from a Russian VPN exit appears to come from Finland)
- ASN blocking is blunt — blocking AS15169 blocks ALL Google traffic including Search, Drive, etc.
When to use
Rarely. GeoIP blocking is a blunt instrument with high FP risk. Use only for:
- Sanctioned countries with regulatory requirements
- Specific high-abuse ASNs after you've confirmed they're not mixed-use
Better alternative: Use
custom_rules with country-based allow (allowlist your serving countries) rather than block.13. Challenge / CAPTCHA
Force suspicious requests through a CAPTCHA before allowing:
'challenge' => [
'enabled' => true,
'provider' => 'turnstile', // builtin | recaptcha | hcaptcha | turnstile
'site_key' => 'your-site-key',
'secret_key' => 'your-secret-key',
'recaptcha_min_score' => 0.5, // reCAPTCHA v3 only
],
Providers
| Provider | Cost | User experience | When to use |
|---|---|---|---|
builtin |
Free | 3–10s JS proof-of-work (timeproof) | Sites without CAPTCHA accounts; low-traffic challenges |
turnstile (Cloudflare) |
Free | Usually invisible | Sites already on Cloudflare |
hcaptcha |
Free tier + paid | Visible puzzle | Sites wanting ethical CAPTCHA |
recaptcha |
Free tier + paid | Often invisible (v3) | Sites with existing reCAPTCHA setup |
Built-in challenge (no external service)
The builtin challenge is a JS-based timeproof: serves an HTML page with JavaScript that delays submission by 3–10 seconds. Bots without JavaScript (curl, scrapers) fail immediately. Browser-based bots (Puppeteer, Playwright) are slower but still pass.
FP risk: Very low. Only affects the JS-disabled or JS-slow.
When to enable
Challenge adds latency for users (3–10s on first request). Enable selectively:
// Per-rule, not library-wide:
'custom_rules' => [
[
'type' => 'country',
'value' => 'XX', // specific high-risk country
'action' => 'challenge',
],
[
'type' => 'ua_contains',
'value' => 'GPTBot', // specific bot
'action' => 'challenge',
],
],
14. Custom rules
Per-bot, per-IP, per-UA rules. Runs first in the detection pipeline.
Schema
'custom_rules' => [
[
'id' => 'my_rule_1', // optional, for logging
'type' => 'ip', // see types below
'value' => '203.0.113.0/24', // see values below
'action' => 'block', // allow | block | challenge | log
],
[
'id' => 'my_rule_2',
'type' => 'ua_regex',
'value' => '/MyInternalScraper\/\d+\./i',
'action' => 'log',
],
],
Rule types
type |
value format |
Matches |
|---|---|---|
ip |
CIDR string or array of CIDRs | package->ip |
ua_regex |
PCRE regex (with delimiters) | package->user_agent |
ua_contains |
Substring | package->user_agent (case-insensitive) |
asn |
ASN string (e.g., 'AS15169') |
package->asn |
country |
ISO 3166–1 alpha-2 (e.g., 'RU') |
package->country |
header |
Array 'header' => 'X-Foo', 'value' => 'bar' |
request headers |
Actions
action |
Result |
|---|---|
allow |
Request allowed (overrides other detectors) |
block |
BLOCKED_CUSTOM_RULE, served as 403 |
challenge |
CHALLENGE_REQUIRED, served as CAPTCHA |
log |
Logged but not enforced (request continues) |
Examples
Allowlist your office IP (overrides all detectors):
['type' => 'ip', 'value' => '198.51.100.0/24', 'action' => 'allow'],
Block known scraper:
[
'type' => 'ua_regex',
'value' => '/DataScraper\/[0-9]/i',
'action' => 'block',
],
Challenge specific country:
['type' => 'country', 'value' => 'XX', 'action' => 'challenge'],
Block specific header signature:
[
'type' => 'header',
'header' => 'User-Agent',
'value' => 'HeadlessChrome',
'action' => 'challenge',
],
Multiple IPs to block:
[
'type' => 'ip',
'value' => ['203.0.113.0/24', '198.51.100.0/24'],
'action' => 'block',
],
Rule processing order
Rules are evaluated in declaration order. First match wins. To override a built-in detector, put the rule early in the array.
15. Reverse proxy
If you're behind Cloudflare, AWS, GCP, or any other CDN/LB:
'reverse_proxy' => [
'enabled' => true,
'header' => 'CF-Connecting-IP', // or 'X-Forwarded-For' for AWS ALB
'addresses' => [
// Cloudflare IPv4 ranges (keep updated):
'173.245.48.0/20',
'103.21.244.0/22',
'103.22.200.0/22',
'103.31.4.0/22',
// ... full list at https://www.cloudflare.com/ips/
],
],
Why this matters
Without it, every request appears to come from your CDN's edge IP. You'll:
- Rate-limit yourself — one user hitting 60/min triggers the
per_minutebucket, blocking all subsequent users - Log wrong IPs — your analytics show Cloudflare's IP, not the real client
- Ban yourself — blocklisting your CDN IP bans every user behind it
Configuration by provider
| Provider | Header | Setup |
|---|---|---|
| Cloudflare | CF-Connecting-IP |
Add all CF ranges to addresses[] |
| AWS ALB | X-Forwarded-For |
Add VPC CIDR (e.g., 10.0.0.0/8) |
| GCP Load Balancer | X-Forwarded-For |
Add GCP ranges |
| nginx (forward proxy) | X-Forwarded-For |
Add 127.0.0.1 (only nginx is trusted) |
| Fastly | Fastly-Client-IP |
Add Fastly ranges |
Security: only trust your proxies
The
addresses[] list is the trusted proxy list. Only IPs in this list can set the forwarded header. This prevents IP spoofing via header injection:// BAD: empty addresses = trust everyone = anyone can spoof their IP
'reverse_proxy' => ['enabled' => true, 'header' => 'X-Forwarded-For', 'addresses' => []],
// GOOD: trust only your CDN
'reverse_proxy' => ['enabled' => true, 'header' => 'CF-Connecting-IP', 'addresses' => $cf_ranges],
If you forget to list your proxy's IPs, BadBehaviour falls back to trusting only private IPs (RFC 1918 ranges), which is safer than trusting everything but means your public proxy IPs won't be trusted.
16. Performance tuning
16.1 Skip static resources
The single biggest performance win — skip detection for static assets:
'performance' => [
'skip_extensions' => [
'css', 'js', 'png', 'jpg', 'jpeg', 'gif', 'ico', 'svg',
'woff', 'woff2', 'ttf', 'eot', 'webp', 'avif', 'map', 'txt',
],
'skip_paths' => [
'/static/', '/assets/', '/media/', '/images/', '/css/',
'/js/', '/fonts/', '/dist/', '/build/', '/vendor/', '/node_modules/',
],
],
By default, 95% of web traffic (CSS, JS, images, fonts) is skipped before detection runs. Without this, every static asset triggers a full detection pipeline including DNS lookups.
16.2 Cache backend
BadBehaviour uses an injectable cache (
CacheInterface). Default implementations:| Adapter | Default cache |
|---|---|
GenericAdapter |
In-memory (per-process, not shared) |
MediaWikiAdapter |
MediaWiki WAN cache |
WackoWikiAdapter |
File cache (in CACHE_DIR) |
For multi-server deployments, inject a shared cache (Redis, Memcached):
$redis_cache = new RedisCache('redis://10.0.0.1:6379');
$config = new Configuration(
// ... other settings
cache: $redis_cache,
);
$bb = new BadBehaviour($config);
The cache holds:
- DNS verification results (positive + negative)
- Rate limit counters
- JA3/HTTP2 verification cache
- Bot detection result cache
Without shared cache, each server builds its own DNS verification cache — first request to each bot IP hits the full 300ms lookup on every server.
16.3 Detection caching
BotDetector caches detection results for 5 minutes per IP+UA combo (LRU, 5000 entries). Bots hammering you with the same UA get a fast-path response after the first detection.17. Diagnostics & monitoring
17.1 The diagnose CLI
$ php bin/diagnose.php
Outputs:
- Smoke tests — verifies
bot_categoriesround-trip +determine_actionpriority order + CLOUD_INFRASTRUCTURE safety override - Library diagnostics — current effective state
Sample output:
{
"safe_mode": false,
"monitor_only_effective": true,
"monitor_only": true,
"strictness": "monitor-only",
"preset": "minimal",
"config_loaded": true,
"logging_enabled": true,
"detectors_active": {
"blacklist": true,
"bot": true,
"dns_verify": false,
"dyn_ranges": false,
"rate_limit": false,
"dnsbl": false,
"behavioral": false,
"fingerprint": false,
"client_hints": false,
"agentic": false,
"head": false,
"asset": false
},
"hint": "BadBehaviour is in monitor-only mode by configuration. ..."
}
Exit codes: 0 on success, 1 on smoke test failure.
17.2 Runtime diagnostics
$bb = new BadBehaviour($config);
$state = $bb->diagnostics();
// Use in health check endpoints:
return new JsonResponse($state);
17.3 Verifying monitor-only is working
After running in monitor-only for a week:
-- 1. Did anything actually get a 403? (Should be tiny — empty UA + raw XSS only)
SELECT enforcement_action, COUNT(*)
FROM bad_behaviour
WHERE date >= CURDATE()
GROUP BY enforcement_action;
-- 2. What would have been blocked at 'normal' strictness?
SELECT status_code, COUNT(*)
FROM bad_behaviour
WHERE enforcement_action = 'monitored'
AND date >= CURDATE()
GROUP BY status_code
ORDER BY COUNT(*) DESC;
-- 3. False positive candidates (legitimate-looking bots in monitored)
SELECT ip, user_agent, status_code, date
FROM bad_behaviour
WHERE enforcement_action = 'monitored'
AND user_agent LIKE '%Googlebot%' -- known search engines
LIMIT 100;
-- 4. Bot category breakdown
SELECT
JSON_UNQUOTE(JSON_EXTRACT(metadata, '$.bot_category')) AS category,
COUNT(*) AS n
FROM bad_behaviour
WHERE date >= CURDATE()
AND JSON_EXTRACT(metadata, '$.bot_category') IS NOT NULL
GROUP BY category
ORDER BY n DESC;
If you're seeing legitimate search engines (Googlebot, Bingbot) in
monitored, your bot_categories.allowed list probably needs to include search_engine.18. Common recipes
Recipe 1: Pure observation (safest starting point)
<?php
return [
'preset' => 'minimal',
'strictness' => 'monitor-only',
'logging' => true,
'verbose' => true, // log allowed too
];
Run for 1–2 weeks. Look at the logs. Move to Recipe 2 when you understand your traffic.
Recipe 2: News / publisher site (block AI training scrapers)
<?php
return [
'preset' => 'minimal',
'strictness' => 'normal',
'logging' => true,
// Aggressive AI crawler blocking
'ai_crawlers' => [
'allowed' => [], // allow NO AI crawlers
'block_unverified' => true, // any unverified AI = block
'strict' => true, // any AI not on allowlist = block
],
// robots.txt policy mirrors
'bot_categories' => [
'blocked' => ['ai_crawler', 'residential_proxy'],
'allowed' => ['search_engine'], // explicitly allow Google/Bing/etc
],
// Be loud about scrapers
'rate_limits' => [
'enabled' => true,
'global' => ['requests' => 500, 'window' => 3600],
'per_minute' => ['requests' => 30, 'window' => 60],
],
];
Recipe 3: E-commerce (maximize search visibility)
<?php
return [
'preset' => 'minimal',
'strictness' => 'normal',
'logging' => true,
'bot_categories' => [
'blocked' => ['residential_proxy'],
'challenge' => ['ai_crawler'],
'allowed' => [
'search_engine', // Google, Bing — critical for SEO
'shopping_crawler', // Google Shopping, Bing Shopping — critical for product visibility
'cloud_infrastructure', // safety: CDN probes
],
],
// Strict rate limits on checkout
'rate_limits' => [
'enabled' => true,
'global' => ['requests' => 2000, 'window' => 3600],
'post' => ['requests' => 100, 'window' => 3600],
'login' => ['requests' => 20, 'window' => 900],
],
];
Recipe 4: Behind Cloudflare
<?php
return [
'preset' => 'minimal',
'strictness' => 'normal',
'logging' => true,
// Critical: trust Cloudflare's IP forwarding
'reverse_proxy' => [
'enabled' => true,
'header' => 'CF-Connecting-IP',
'addresses' => [
// Current full list at https://www.cloudflare.com/ips/
'173.245.48.0/20',
'103.21.244.0/22',
'103.22.200.0/22',
'103.31.4.0/22',
'141.101.64.0/18',
'108.162.192.0/18',
'190.93.240.0/20',
'188.114.96.0/20',
'197.234.240.0/22',
'198.41.128.0/17',
'162.158.0.0/15',
'104.16.0.0/13',
'104.24.0.0/14',
'172.64.0.0/13',
'131.0.72.0/22',
],
],
// Use CF's JA3 forwarding
'enable_fingerprinting' => true,
'fingerprints' => [
'bad_ja3' => [], // populate after traffic analysis
],
];
Recipe 5: Migration from Bad Behaviour 2.x
<?php
return [
'preset' => 'minimal',
'strictness' => 'normal', // start safe
'logging' => true,
'verbose' => true, // log everything for comparison
// Carry over BB 2.x settings
'reverse_proxy' => [
'enabled' => true,
'header' => 'X-Forwarded-For',
'addresses' => ['10.0.0.0/8', '172.16.0.0/12', '192.168.0.0/16'],
],
'offsite_forms' => false,
'httpbl' => [
'key' => 'your-key-here',
'threat' => 25,
'maxage' => 30,
],
];
Run side-by-side with BB 2.x for a week. Compare logs. Adjust before cutting over.
19. Troubleshooting
“Safe mode won't exit”
Symptom:
diagnostics()['safe_mode'] === true even though bb_config.php exists.Cause: Adapter path resolution. The adapter looks at
DIR . '/../../../config/bb_config.php' (or similar relative paths) which can resolve wrong depending on repo layout.Fix: Either:
- Move
bb_config.phpto where the adapter looks - Define
CONFIG_DIRconstant pointing to your config directory (WackoWiki-style) - Place
bb_config.phprelative to CWD if running CLI tools
Verify the path:
$ php -r 'echo realpath("config/bb_config.php") ?: "NOT FOUND";'
“My config keys are being ignored”
Symptom: You set
'bot_categories' => ['challenge' => ['social_crawler']] but it's not taking effect.Diagnostic:
$ php bin/diagnose.php
If
test_bot_categories_round_trip fails, the keys aren't being read. Check:- File syntax (run
php -l config/bb_config.php) - Trailing comma (PHP 8+ allows but some configs don't)
-
return [...]at the end (not just[...])
If
test_bot_categories_round_trip passes but runtime doesn't honor the config, check your strictness — monitor-only demotes overrides to logged-only.“Logs not appearing”
Symptom: Detection runs but no rows in
bad_behaviour table.Diagnostic:
// 1. Is logging enabled?
echo $config->logging; // should be true
// 2. Is the table writable?
// Check DB permissions for the bb user
// 3. Is install_once() running?
// Look for 'install_once' errors in your error log
Most common cause:
define('BB2_NO_CREATE', true) is set (skips auto table creation). Run bin/install-bb.php manually.“False positives on search engines”
Symptom: Googlebot/Bingbot/etc. getting blocked.
Diagnostic:
SELECT ip, user_agent, status_code, date
FROM bad_behaviour
WHERE enforcement_action = 'enforced'
AND (user_agent LIKE '%Googlebot%' OR user_agent LIKE '%Bingbot%')
ORDER BY date DESC
LIMIT 50;
Fixes:
- Add to
allowed[]:
'bot_categories' => ['allowed' => ['search_engine']],
- If DNS verification is failing (PTR record doesn't resolve to
*.googlebot.com):
- Check your reverse DNS is actually working
- For
strictstrictness: disablerequire_forward_confirm - Increase
dns_verification_positive_ttlso verified IPs don't re-check
- If the bot's IP isn't in static ranges (e.g., Google using a new range):
- Run
bin/update-ip-ranges.phpto refresh dynamic IP ranges - The cache may be stale
“False positives on residential IPs”
Symptom: Legitimate users behind shared NAT (corporate, mobile, VPN exit) hitting rate limits.
Fix: Increase rate limits based on your 95th percentile:
-- What's realistic for legitimate users?
SELECT
ip,
COUNT(*) AS reqs,
COUNT(*) / COUNT(DISTINCT DATE_FORMAT(date, '%Y-%m-%d %H:00:00')) AS per_hour
FROM bad_behaviour
WHERE enforcement_action = 'allowed'
AND date >= NOW() - INTERVAL 7 DAY
GROUP BY ip
HAVING reqs > 100
ORDER BY per_hour DESC
LIMIT 50;
Set
rate_limits.global.requests to 5x your 95th percentile per-hour.“Cloudflare CDN probes are getting blocked”
Symptom: Origin server is marked unhealthy by Cloudflare; site goes down.
Cause:
cloud_infrastructure category is somehow not being matched.Diagnostic:
SELECT ip, user_agent, status_code
FROM bad_behaviour
WHERE user_agent LIKE 'Cloudflare%' OR user_agent LIKE 'ELB-HealthChecker%'
ORDER BY date DESC
LIMIT 20;
Fix:
- Verify cloud_infrastructure is included in your registry:
return [
'preset' => 'minimal',
'include_categories' => ['cloud_infrastructure'], // safety net
];
- If using
preset='custom', manually add the cloud_infrastructure bots — they're not auto-included in custom mode. - Check
dynamic_ip_ranges— if disabled, the static CIDR list is your only defense. Runbin/update-ip-ranges.phpto get fresh ranges.
“Bot detection seems slow”
Symptom: First request from a bot IP adds 200–500ms latency.
Cause: DNS verification synchronous lookup.
Fixes:
- Check cache hit rate — first request is slow, subsequent should be fast
- For high-traffic sites: use a shared cache (Redis/Memcached) so caches are warm across servers
- During DNS outages: temporarily set
'strictness' => 'monitor-only' - Reduce
dns_verification_timeout_ms— but not below 100ms (some legit DNS takes longer)
“How do I disable X without losing Y?”
The strictness system is a shortcut. For granular control, override individual features:
return [
'strictness' => 'normal',
'dnsbl_enabled' => false, // disable just DNSBL
'enable_behavioral_analysis' => true, // enable just behavioral
'rate_limit_enabled' => false, // disable just rate limits
];
strictness sets the baseline; your keys override individual features.See also
-
bin/diagnose.php— runtime diagnostics + smoke tests -
STRICTNESS.md— detailed strictness-level semantics - [Bot Registry wiki page] — full bot registry reference
- [Understanding Error Codes] —
ResultCodeenum reference - [Writing a Custom Adapter] — integrating with your application