Clockwork Control v1.10.0: Email Authentication Posture via DoH, Real-User Core Web Vitals (CrUX), and AiRemedy Diagnostic Accuracy Scorecards

Every agency that manages WordPress websites has experienced this dreaded client phone call:

“Our customer inquiry emails are going straight to spam, our transactional invoices are bouncing, and Google says our domain is unauthenticated. What happened?”

You investigate only to discover that the client’s internal IT department or a third-party marketing agency added another marketing tool to the domain’s DNS, pushing the SPF record past the strict RFC 7208 10-DNS-lookup limit. Or an expired DMARC policy record was accidentally deleted during a domain registrar transfer, causing Gmail and Yahoo Mail to reject every outbound email under their strict bulk-sender enforcement rules.

Because agencies typically don’t manage client DNS day-to-day, email authentication posture breaks silently. You rarely discover the issue until deliverability collapses and the client is in a panic.

With Clockwork Control v1.10.0, we’re giving agency owners proactive, automated visibility over their fleet’s entire email deliverability posture.

Alongside the brand-new EmailAuth module powered by DNS-over-HTTPS, Clockwork Control v1.10.0 delivers Real-User Core Web Vitals (Google Chrome UX Report / CrUX) field data, AiRemedy Diagnostic Accuracy Reporting & 60-Minute Resolution Outcomes, Automated Plugin Update Failure Streaks, Settings Card Reordering, a fleet-wide Visual Hierarchy & Icon Standardization, and Client Uptime Alert Subscribers via SMS and transactional email.

Here is the complete breakdown of what Clockwork Control v1.10.0 brings to your agency.

1. Introducing EmailAuth: Fleet-Wide SPF, DMARC & DKIM Deliverability Defense

The new EmailAuth module (modules/EmailAuth, Composer package clockwork/email-auth) brings commercial-grade email authentication monitoring directly into your self-hosted control panel, categorized under the unified Security hub (/security/email-auth or /email-auth).

Monitored across root apex domains (deduplicated across sites so subdomains share their primary apex check), EmailAuth performs deep inspection into every layer of email posture:

A. Strict RFC 7208 SPF Validation & Mechanism Expansion

• Single Record Enforcement: Verifies that exactly one v=spf1 TXT record exists. Multiple SPF records cause instant SPF permerror and inbox rejection.
• Recursive 10-DNS-Lookup Counter: Recursively expands include:, redirect=, a, mx, and exists mechanisms. Flags advisory warnings when a domain reaches 8–10 lookups, and triggers a hard failure the moment it exceeds 10.
• Circular Include Loop Detection: Gracefully handles circular include references without infinite recursion.
• Void Lookup Counter: Flags records containing non-resolving domains (NXDOMAIN or empty responses); warns if void lookups exceed 2.
• Qualifier Grading: Recommends -all (hard fail) or ~all (soft fail), warns on ?all (neutral), and fails on +all or missing terminal qualifiers.
• Deprecated Mechanisms: Flags deprecated ptr lookups.

B. RFC 7489 DMARC Policy & Reporting Audits

• Inspects _dmarc.{apex} records to verify valid RFC 7489 compliance.
• Policy Enforcement: Passes on p=reject or p=quarantine; displays an advisory warning on p=none (monitoring-only mode).
• Reporting Destinations: Verifies that valid rua= aggregate mailto report destinations are configured.
• Rollout Percentages: Evaluates pct= enforcement percentage and sp= subdomain policies.

C. Multi-Selector DKIM Probing

• Because DNS does not allow listing DKIM keys without knowing selector names, EmailAuth probes 12 standard industry selectors (google, selector1, selector2, k1, s1, s2, mx, smtp, pm, mandrill, default, dkim).
• Allows operators to configure custom DKIM selectors per domain directly from the web drawer.
• Probing misses produce advisory unknown/warn indicators—never false negatives.

D. Parked & Non-Sending Domain Intelligence

• Detects absent or null MX records (0 .).
• Automatically shifts recommendations to v=spf1 -all and p=reject, confirming that parked client domains cannot be spoofed by spammers.

2. Why DNS-over-HTTPS (DoH) Matters for Reliable Automation

Traditional PHP dns_get_record() is notorious for causing headaches in automated fleet monitoring:

1. Host Resolver Caching: Local resolver caches (systemd-resolved, dnsmasq, or local ISP resolvers) can cache old TXT records for hours or days, masking recent DNS updates.
2. Ambiguous Failures: PHP’s native DNS functions conflate NXDOMAIN (domain does not exist), empty TXT responses (domain exists but has no TXT record), and network timeouts into false or an empty array. This routinely triggers false-positive “DMARC record missing” alerts whenever a local resolver stumbles.
3. Deterministic Testing: Native DNS functions cannot be reliably faked in automated test suites.

To solve this, EmailAuth implements the DnsTxtResolver contract with DNS-over-HTTPS (DohDnsTxtResolver). By default, it queries Cloudflare’s privacy-focused DoH endpoint (https://cloudflare-dns.com/dns-query) with an automatic failover to Google Public DNS (https://dns.google/resolve).

A transient lookup timeout is classified as unknown, never a false fail. For offline or air-gapped agency deployments, a fallback NativeDnsTxtResolver is available.

3. Fleet Hub, Interactive Slide-Over Drawer & State-Transition Alerts

EmailAuth is deeply woven into Clockwork Control’s operational workflow:

• Unified Security Strip: Available at /security/email-auth alongside Security Scans, WP Admins, and fail2ban IP Bans. View apex domains, total linked WordPress sites, SPF lookup counts, DMARC policies, DKIM status, and overall health grades.
• Global Fleet Scan (“Scan all domains”): Trigger an immediate background verification scan across all monitored apex domains (POST /email-auth/scan-all). Soft-deleted domains are automatically skipped.
• Slide-Over Detail Drawer:
  • One-click copyable raw DNS records (v=spf1..., v=DMARC1...).
  • Plain-English findings categorized by severity (fail, warn, info).
  • Suggested fix records ready to send to client IT teams.
  • Custom selector manager.
  • One-click Ignore Domain toggle to mute alerts for client-managed domains.
  • Instant Re-scan Domain button (POST /email-auth/scan) for real-time verification after DNS edits.
  • Remove from Tracking: Safely soft-deletes the domain (DELETE /email-auth/domains/{domain}), excluding it from weekly scans.
• State-Transition Chat Alerts: Fires email_auth_degraded notifications to Slack and Mattermost ops channels exclusively when a domain’s status degrades (e.g. from pass/warn to fail), eliminating repetitive alert noise.
• Weekly Automated Cadence: Scheduled weekly via clockwork:check-email-auth on Mondays at 04:00 UTC.

4. Real-User Core Web Vitals: Google Chrome UX Report (CrUX) Field Data

While simulated Lighthouse and GTmetrix scans provide valuable lab diagnostics from fixed test servers, they do not reflect how real visitors experience your clients’ websites on real devices and real cellular connections.

Clockwork Control v1.10.0 introduces direct integration with Google’s Chrome UX Report API (ChromeUxReportClient), pulling authentic 28-day rolling real-user field metrics:

• Stored Metrics (site_field_metrics):
  • LCP (Largest Contentful Paint) at the 75th percentile.
  • INP (Interaction to Next Paint) at p75—Google’s official responsiveness metric.
  • CLS (Cumulative Layout Shift) at p75 (stored as cls_p75_x1000 to prevent floating-point drift).
  • FCP (First Contentful Paint) and TTFB (Time to First Byte) at p75.
  • Overall Core Web Vitals Assessment (cwv_pass): Evaluates whether LCP ($\le 2500\text{ms}$), INP ($\le 200\text{ms}$), and CLS ($\le 0.1$) all achieve Google’s “Good” threshold.
  • Histogram Distributions (good_pct): The exact percentage of real user visits falling into the “Good” category.
• Graceful no_data State: Small client sites often lack sufficient Chrome traffic for Google to report data. The CrUX API returns HTTP 404 in these instances. Clockwork Control treats this as a first-class no_data status: it displays an informative notice in the UI without triggering error logs, alerts, or circuit breakers.
• Where It Appears:
  • Performance Tab (/sites/{id}?tab=performance): Dedicated “Real users, last 28 days” card featuring official Google threshold badges.
  • Site Overview: CWV pass/fail status badge.
  • Monthly Client Reports: ClientReportCompiler::compilePerformance() automatically populates real-user Core Web Vitals and TTFB in client PDF reports.
• Weekly Collection: Runs Sundays at 05:30 UTC via clockwork:collect-field-metrics.

5. AiRemedy: Diagnostic Accuracy Reports, Operator Verdicts & 60-Minute Resolution Outcomes

Following the release of AiRemedy (Self-Healing Servers) in v1.9.0, agency operators requested concrete data to evaluate diagnostic accuracy before promoting servers from Shadow Mode to Interactive Copilot or Autonomous Self-Healing.

Clockwork Control v1.10.0 delivers complete reliability observability:

A. Accuracy Report (/ai-remedy/accuracy)

• Operator Verdict Distributions: Authenticated operators can review incidents on the drawer or /ai-remedy/runs/{id} and submit verdicts (Correct, Partial, Wrong, or Unsure) with optional feedback notes, recorded immutably in action_logs.
• Cost Efficiency Metrics: Displays total token spend alongside cost per run and cost per correct diagnosis.
• Rollout Readiness Scorecard: Plain-English evaluation (e.g., “45 verdicts recorded, 91% correct, 0 wrong Tier 1 proposals in the last 30 days”) to give operators objective data before enabling autonomous command execution.

B. Automated 60-Minute Resolution Outcomes (OutcomeClassifier)

AiRemedy doesn’t rely solely on human verdicts. A scheduled background classifier (clockwork:ai-remedy-evaluate-outcomes, running every 15 minutes) automatically inspects the incident 60 minutes after diagnosis:

• Server Spikes are evaluated using server_metrics.
• Site Downtime is evaluated using site_uptime_events.
• Human Actions are cross-referenced from action_logs.
• Classifies incidents into self_resolved, human_resolved, persisted, escalated, or unknown.
• “Would Have Acted, But Self-Resolved” Metric: Tracks incidents where AiRemedy proposed Tier 1 fixes, but the spike resolved itself without human or AI action. This prevents over-intervention and helps fine-tune CPU spike thresholds.

C. Copilot Review Queue & Expiration

• Needs Review Filter: Filter the incident log by pending Copilot runs awaiting operator sign-off.
• Main Nav Badge: Real-time counter badge on the main navigation AiRemedy link showing unreviewed runs.
• 24-Hour Expiration: Hourly command clockwork:ai-remedy-expire-unreviewed auto-expires unreviewed Copilot incidents after 24 hours to prevent stale commands from lingering.

D. Sudo Password Age Tracking

• Stored server SSH/sudo passwords now track servers.ssh_password_updated_at.
• Displays password age and a prominent 90-day rotation warning on /servers/{id}/credentials.

E. Service Rate Limits & .env Persistence

• Registered ai-remedy in ServiceRateLimitRegistry with OpenRouter vendor limits (200 requests/minute, 45s timeout, concurrency 2, 100ms pacing delay, 2 retry attempts).
• OPENROUTER_API_KEY is persisted directly to .env via EnvCredentialManager, avoiding database storage.

6. Automated Plugin Update Failure Streaks & Exception Synchronization

When an automated nightly plugin update fails repeatedly (e.g., due to an expired commercial license key, a fatal PHP syntax error, or an incompatible dependency), continuing to retry the update every night generates endless error logs and risks breaking client sites.

Clockwork Control v1.10.0 introduces automated failure streak management:

• Tracks consecutive nightly update failures per site and plugin slug (plugin_update_failure_streaks).
• Automatically inserts failing plugins into plugin_update_ignores with a machine-generated audit note once the streak reaches threshold (default: 5 consecutive failures).
• Pushes synchronized exception notices to Clockwork Companion and Renegade via clockwork:push-update-exceptions, rendering update coverage notices and paused update indicators directly in WordPress wp-admin.
• Broadcasts state transition alerts via the plugin_update_auto_ignored chat event.
• Provides a one-click “Resume Management” button in Control to unignore the plugin, reset the streak counter, and clear wp-admin notices.

7. Settings Card Reordering & Fleet-Wide UX Cleanup

Clockwork Control v1.10.0 includes significant quality-of-life and visual hierarchy refinements:

• Settings & Operations Card Reordering:
  • Drag-and-drop handles and accessible up/down shift controls on all 5 Settings & Operations category cards (Agency Branding, Fleet Policies, Integrations & Alerts, Operations & Tools, System & Workspace).
  • Order persistence via localStorage (cw_settings_cards_order) with zero-FOUC initial script render and a one-click “Reset default order” action.
• Standardized Neutral Icons:
  • Standardized card and row icons across site detail tabs, fleet sites list, and infrastructure dashboard to neutral ink-muted.
  • Reserved green, yellow, orange, and red strictly for semantic health status indicators, drastically reducing visual fatigue.
• Servers Dashboard Telemetry:
  • Replaced duplicate refresh forms in Metric Card 4 with genuine OS Maintenance telemetry ($patchCounts['patches'] and $patchCounts['reboots']), complete with direct deep links to fleet updates.
• Compact Maintenance History & Updates:
  • Consolidated stacked Month picker and Filters cards into a single compact toolbar with inline submit and instant filter clearing.
  • Standardized unstyled text links with btn-pill-nav button pills across updates and server updates.
• Simplified Sites Directory:
  • Relocated host and server information to a clean secondary line beneath the domain, eliminating the cluttered badge wall.
• Security Scans Disclosure:
  • Replaced the 4-line explanatory paragraph with an expandable Schedule & details disclosure, pulling metric rollup cards directly below the tab strip.
• Feedback Backlog Optimization:
  • Consolidated duplicate CTAs into page header actions, removed the 160px banner, and suppressed the floating pin overlay on /feedback.

8. Client Uptime Alert Subscribers (SMS & Transactional Email)

In addition to internal engineering paging, care-plan sites can now have dedicated Alert Subscribers attached (site_notification_subscribers pivot):

• Client-Friendly Tone: Clients receive clean, reassuring notifications with zero operational jargon or server internals:
  • SMS (via Twilio): “{Name}, the website {domain} is currently unreachable. Our team has been notified and is investigating.”
  • Email (via Laravel Mail): Transactional HTML and plain-text emails (SiteDownClientMail and SiteUpClientMail) with clean status badges and incident duration.
• Flexible Management:
  • Manage all contacts and site assignments centrally at /settings/notifications under Client Alert Contacts.
  • Or configure alert subscribers directly from any site’s Settings tab (/sites/{site}?tab=settings → Card 6 Alert Subscribers).
• Strict Isolation: Client alert contacts never participate in internal on-call rotations or off-windows. They receive notifications strictly for the specific sites they are assigned to, and only when those sites are on an active care plan.

9. What’s in the v1.10.0 Release

10. Upgrading to Clockwork Control v1.10.0

Clockwork Control v1.10.0 is available immediately on GitHub for all self-hosted agency installations.

To upgrade your local or LAN installation:

# Navigate to your Clockwork Control directory
cd clockwork-control

# Fetch latest releases and checkout v1.10.0
git fetch --tags
git checkout v1.10.0

# Install PHP and frontend dependencies
composer install --no-dev --optimize-autoloader
npm install

# Run database migrations
php artisan migrate --force

# Rebuild frontend assets
npm run build

# Clear and rebuild caches
php artisan optimize:clear
php artisan optimize

# Verify that the test suite passes
php artisan test

If you manage client domains with custom DKIM selectors, visit Security → Email Auth (/security/email-auth), click on any apex domain, and configure custom selectors from the slide-over drawer. To collect initial real-user Core Web Vitals, ensure CLOCKWORK_CRUX_API_KEY (or CLOCKWORK_PSI_API_KEY) is present in your .env and run php artisan clockwork:collect-field-metrics.

Sovereign Control for WordPress Agencies

Clockwork Control is 100% free and open-source software under the MIT License. We believe agencies should own their operational tools, hold their own cryptographic keys, and manage their clients’ hosting without recurring per-site SaaS penalties.

Have questions or want to contribute a new hosting or cloud provider module? Check out our Documentation Portal, review our Contributing Guide, or join the conversation on GitHub.