Bot Analytics

Logs & Bot Analytics

Data Collection

Bot visits are logged to the custom table {prefix}cybermaps_logs:

Column Type Description
id bigint(20) Auto-increment PK
time datetime GMT timestamp
bot varchar(100) Identified bot name
category varchar(50) Bot category
url varchar(255) Requested URL
ip_address varchar(45) Visitor IP (IPv6-safe)
user_agent text Raw User-Agent string

Logging fires at template_redirect priority 99, after all discovery handlers have had a chance to serve content. Non-bot traffic (unidentified User-Agents) is rejected early to avoid filling the table.

Dashboard

KPI Cards:

  • Total bot hits (last 30 days)
  • Top 5 endpoints by request count
  • Active bot signatures (unique bots in last 30 days)
  • Category distribution (AI Training vs. Search Engine vs. Social vs. Other)

Velocity Chart: 30-day HTML bar chart generated by ChartGenerator. Each bar represents one day. Hover for exact count.

Activity Stream: Last 50 entries with date-range presets:

  • Last hour
  • Today
  • Last 7 days
  • Last 30 days

Click any bot name to filter the stream to only that bot.

Admin Bar Indicator: Shows today’s bot hit count. Green (active), yellow (no hits in last 5 minutes). Cached 5 minutes.

Cache Invalidation

A dirty counter mechanism prevents stale dashboard data:

  1. log_ai_ingestion() increments cybermaps_logs_dirty on each insert
  2. Dashboard render methods check the dirty counter before serving cached transients
  3. If the counter exceeds the threshold (10), the dashboard cache is rebuilt

This is more efficient than deleting transients on every log insert, one counter update vs. multiple transient deletions.

Data Retention

  • Configurable retention in days (default: 30)
  • Daily cron job (cybermaps_cleanup_logs_event) purges entries older than the retention window
  • Cron is scheduled on activation, cleared on deactivation

Export

CSV export with:

  • Date-range filter (start date, end date)
  • LIMIT 10000 safety cap
  • Proper CSV headers and escaping

Table Maintenance

OPTIMIZE TABLE button available in the admin UI for defragmentation after many deletes.