Clockwork Control v1.11.0: Email Auth Client Notice Composer, Full Inline Report Emails, Chronic Server Saturation Detection, and AiRemedy Hardening

We are excited to announce the release of Clockwork Control v1.11.0!

This release bridges client communication and fleet intelligence: agency operators can now compose white-labeled email authentication advisory notices directly to client contacts from the Email Auth dashboard, deliver comprehensive monthly client reports rendered completely inline in HTML email, detect chronic server hardware saturation before clients experience slowdowns, and rely on battle-tested AiRemedy autonomous healing resilience across ephemeral process spikes.

Here is a full tour of what is included in Clockwork Control v1.11.0.

1. Email Auth: Interactive Client Notice Composer (/email-auth/domains/{domain}/notify)

Following our v1.10.0 release of fleet-wide email posture monitoring across SPF, DMARC, and DKIM, agency operators loved the high-level visibility but needed an effortless bridge to turn technical findings into actionable client conversations. When a client’s DNS records are missing a DMARC policy or hitting the RFC 7208 10-lookup limit, copying error codes into an external email client creates unnecessary friction.

Clockwork Control v1.11.0 introduces the Client Notice Composer:

• Drawer-Integrated Action: Open any domain’s inspection drawer on /email-auth and click “Compose email” (or navigate directly to /email-auth/domains/{domain}/notify).
• Auto-Discovered Recipients: Pre-fills recipient email addresses by resolving client site ownership against the apex domain via RootDomainResolver. You can add, edit, or remove contacts freely (comma-separated, up to 20 recipients).
• Plain-English Finding Explanations: Powered by Modules\EmailAuth\Support\FindingExplainer, technical DNS errors are automatically translated into clear, non-technical explanations that clients can understand and appreciate.
• Selective Finding Checkboxes: Findings classified as fail or warn are pre-ticked by default; informational items (info) are offered unticked for optional inclusion, while passing checks (pass) are omitted.
• Custom Operator Note & Direct Mailto Reply: Includes an editable operator note (default: “We’ve noticed some issues with your domain’s email settings (DNS)… Would you like us to fix this for you?”) and routes client replies directly to your agency support address via mailto:—eliminating broken links to private, self-hosted dashboards.
• Side-by-Side Live Email Preview: A real-time preview pane renders the exact white-labeled email layout right beside your composition form as you toggle findings and adjust your notes.
• Agency White-Labeling: Formatted using your agency’s Client Reports branding theme (custom primary header background color, accent styling, and embedded brand logo via inline CID).
• Operator BCC & Audit Log: Optional “Send me a copy” BCC checkbox, with every transmission recorded in the action log table (email_auth_client_notice_sent) including recipient addresses, subject line, finding codes, and mailer provider.

2. Client Reports: Full Inline HTML Maintenance Emails

In agency client management, convenience and polish are paramount. Clients rarely want to click through external portals or log into private dashboards just to see that their WordPress site was updated and backed up.

Clockwork Control v1.11.0 elevates scheduled monthly client reports to full inline HTML emails:

• Complete Inline Layout: Scheduled maintenance reports now render the complete executive report directly within the client’s email client: branded header with embedded agency logo CID (BrandLogo), KPI summary tiles, plugin update lists with clean titles, uptime statistics, security scans, Core Web Vitals performance (CrUX LCP, FID/INP, CLS), traffic analytics, contact form test results, and timestamped backup logs.
• Zero Broken Links: Because Clockwork Control is a self-hosted platform running on private agency infrastructure, external links back to localhost or internal URLs have been replaced with direct support mailto: call-to-actions.
• Plain-Text Alternative: Automatically generates a clean, readable plain-text counterpart for clients using terminal or text-based email readers.
• Schedule Snapping & Drift Prevention: Resolved a recurring cron calculation issue where monthly reports drifted forward by one day on each cycle. Scheduled runs now reliably snap to midnight (00:00) on the 1st of each month so long-running queue batches never push the next run past the daily 06:00 UTC execution window.

3. AiRemedy: Chronic Saturation Tracking & Autonomous Healing Resilience

Clockwork Control’s autonomous self-healing server engine, AiRemedy, receives major enhancements in v1.11.0 focused on long-term infrastructure health and edge-case execution resilience.

Chronic Server Saturation Detection (recent_spikes_48h)

When a server spikes in CPU or memory utilization, terminating rogue worker processes can restore immediate responsiveness. However, if a server repeatedly spikes multiple times a week, the root problem isn’t a runaway process—it is insufficient compute capacity or unbuffered web traffic.

• 48-Hour Spike Counter: Tracks trailing incident frequency across monitored servers over the past 48 hours.
• Advisory Diagnosis Alert: If a server experiences $\ge 5$ spikes within 48 hours, AiRemedy automatically flags the incident with a Chronic Saturation Advisory. The diagnosis advises operators that repeated process termination is only treating symptoms, recommending server resizing, PHP worker pool tuning, or edge CDN caching (such as Cloudflare or Fastly) instead.

Autonomous Healing Hardening

• Graceful Ephemeral PID Handling: When CPU spikes are caused by brief bursts (e.g., intensive image processing or quick search queries), the process PID may terminate naturally milliseconds before SSH commands execute. Commands like kill, pkill, renice, or ionice returning exit code 1 (No such process) are now treated as non-fatal notices rather than failed runs; subsequent remediation steps (such as cache flushes or service reloads) proceed smoothly.
• PHP-FPM Service Unit Normalization: Language models occasionally suggest restarting per-site service units (e.g., systemctl reload php8.3-fpm-<site>.service). AiRemedy now normalizes these into the canonical system-wide service unit (php8.3-fpm), avoiding exit code 5 service unit errors.
• Diagnostic Command Filtering: WatchServerSpikes now filters out legitimate diagnostic probe commands (ps aux, top, grep) from being misidentified as rogue processes.
• Debounce & Routine Task Exclusion: Extended CPU spike persistence threshold from 20s to 60s, and excluded routine scheduled maintenance tasks (wp-cron, wp backup, mysqldump, duplicity, composer) from triggering unnecessary remediation runs.
• Outcome Scoring Accuracy: Corrected scoring evaluation so incidents where autonomous actions successfully restored system health are accurately recorded as ai_resolved.

4. Navigation Architecture Polish (v1.10.2 & v1.10.3 Refinements)

Over the past few maintenance releases, we also streamlined navigation across both layout styles:

• Command Center Rail — Collapsible Groups:
  • The left-hand operations rail now organizes secondary navigation into collapsible groups: Operations, Configuration, and Modules (Workspaces remains open by default).
  • Groups automatically open when navigating to a contained page and persist their open/closed state in localStorage (cw_rail_group_*).
  • When the rail is collapsed to icons, all group items remain directly clickable.
  • The Operations group heading displays the live AiRemedy review badge count even when collapsed.
  • The new Modules group lists every enabled module’s page (Code Snippets, Clients, Client Reports, Mattermost/SMS notifications, Email Auth, Feedback, Bill.com sync) directly on desktop.
• Modern Studio Top Bar — Operations Dropdown:
  • Groups Capacity, Fleet Updates, Maintenance, and AiRemedy into an Operations ▾ dropdown menu.
  • Teleported to <body> to prevent horizontal scroll clipping on smaller desktop displays, complete with keyboard Escape and outside-click dismissal.

5. Metrics Post-Reboot Resilience & Dependency Updates

• DigitalOcean CPU Metrics Post-Reboot Resilience: Enhanced DigitalOceanMetricsParser::percentCpuUsed() to detect Linux counter resets (/proc/stat jiffies) when droplets reboot mid-polling window. Utilization is now calculated across synchronized post-reboot data points instead of discarding the entire window and reporting an “Unknown” status for 15 minutes.
• Security & Dependency Maintenance: Resolved npm security advisories in shell-quote and source-map-js, and bumped vite to 8.3.2 and @biomejs/biome to 2.5.15.

6. Upgrading to Clockwork Control v1.11.0

Clockwork Control v1.11.0 is available immediately on GitHub. Existing self-hosted installations can update via standard Git pull:

# Navigate to your Clockwork Control root directory
cd /path/to/clockwork-control

# Fetch tags and check out v1.11.0
git fetch --tags
git checkout v1.11.0

# Update backend dependencies & run database migrations
composer install --no-dev --optimize-autoloader
php artisan migrate --force

# Clear and rebuild application caches
php artisan optimize:clear
php artisan optimize

# Restart background queue workers and scheduled workers
sudo supervisorctl restart all

Take a look at the updated Documentation or browse the Visual Tour to see all the latest features in Clockwork Control!