Server updates + reboots

The server detail page → Updates tab handles apt-get upgrades and reboots one server at a time. For multi-server work, use the fleet view at /operations/server-updates — that page exposes bulk selection (“Select all with packages”, “Security only”) plus a single button that queues every selected server at once. Both surfaces share the same backend pipeline: clockwork:process-server-updates drains the queue once per minute, one server per tick.

Both go over SSH (we don’t use the DO API for power actions). Both queue rather than run synchronously, so a click doesn’t block the request for ten minutes.

Updates tab

/servers/{id}/updates shows:

• Pending packages & security count — ground-truth package list pulled via SSH (apt-get -s dist-upgrade).
• Ground-Truth vs. Hosting Mirror: When a local SSH snapshot exists and confirms 0 pending packages, the tab and header status pill report Up to date, preventing false-positive “Patches available” flags even if a third-party panel (like SpinupWP’s API) is lagging behind.
• Reboot required flag — driven by /var/run/reboot-required on the server.
• Update queue state — idle, queued, running, completed, failed.
• Optional deferred reboot time — schedule a maintenance window for a post-upgrade restart.
• Last update log — captured apt-get stdout/stderr output from the last run.

Queue an update

Click Run updates now. The form POSTs to /servers/{id}/update/queue, sets update_status='queued', and returns immediately. The actual upgrade runs asynchronously via clockwork:process-server-updates (executing every minute, one server per tick).

The runner connects via SSH and executes:

apt-get update
DEBIAN_FRONTEND=noninteractive apt-get -y dist-upgrade
apt-get -y autoremove

Auto-Reboot Behavior After Updates

• No reboot needed: If the installed packages do not touch the kernel or core C libraries, the server does not restart.
• Kernel / Libc update installed (/var/run/reboot-required exists):
  • If a time was entered in “Optional deferred reboot” (e.g. 03:00): The reboot is scheduled for that server-local hour (shutdown -r 03:00).
  • If left blank: The server automatically reboots in +1 minute (shutdown -r +1), allowing SSH to exit cleanly while ensuring the new kernel is applied immediately.

After the upgrade, ServerUpdater also validates Nginx’s service state (systemctl is-active nginx) and self-heals by restarting Nginx if a major version bump stalled it.

Cancel a queued update with the Cancel button before it picks up. Once running, it must complete.

Issues page → “Patches available”

/issues lists every server with upgrade_required=true under Patches available. That flag means non-security apt packages are pending; it is cleared only when a poll (clockwork:poll-system-updates, nightly, or “Re-poll fleet now”) finds nothing left to upgrade. Rebooting does not install anything, so a reboot never removes a row from this card — the Reboot now link is shown only on rows whose reboot_required flag is also set, next to a “reboot pending” marker.

Each row has an Install updates button, and the card’s toolbar has Install updates on all N plus an optional Reboot at (server-local HH:MM; blank = reboot immediately after the upgrade if it asks for one). Both POST to the same queueBulk endpoint the fleet page uses, with Accept: application/json, and flip the affected rows to a queued spinner in place; the banner above the card reports how many were queued, what was skipped (ignored / already in flight / no SSH credentials) and a rough ETA. Rows already queued/running show that state instead of a button, and rows with no SSH credentials show no SSH.

Timing: the processor drains one server per minute and an apt run typically takes 1–4 minutes, so queueing 16 servers means the last one finishes roughly 20–60 minutes later. Track progress on the fleet page.

Fleet page (/operations/server-updates)

A single dashboard for the whole fleet:

• 5 rollup tiles — Servers (with polled/never/failed breakdown), Packages pending, Security, Reboot pending, In flight.
• Per-server table — server name, state badge (up to date / patches / security / queued / running / failed / ssh failed), package + security counts, reboot flag, last polled, an inline Update & reboot button (for instant single-box cycles on servers with pending packages or required reboots; omitted when the server is up to date), and a Manage link to the per-server Updates tab.
• Sticky bulk-action toolbar — pinned to the viewport bottom so the controls are visible the moment the page loads, not buried under 50 rows of table:
  • Select all with packages — checks every row with total_updates > 0.
  • Security only — checks rows with security_updates > 0.
  • Clear — unchecks everything.
  • Reboot at time field — optional. Blank = reboot immediately upon completion (with a 1-minute grace for SSH to exit cleanly).
  • Install updates & reboot immediately (N selected) — prompts with "are you sure you want to run updates and reboot on all selected servers?" before queueing. Disabled until at least one box is checked. Rows for up-to-date servers are disabled from selection since no updates are pending.

Re-poll fleet now

The button next to the rollup tiles kicks off clockwork:poll-system-updates --all through BackgroundArtisan (the shared Cache::add + PhpExecutableFinder + nohup helper every other fleet-wide “run now” button uses). A poll-in-progress banner appears at the top showing “polled X of Y” and the page auto-refreshes every 10s until the marker clears. The poll is idempotent across clicks — a cache lock (operations.system_updates.poll_in_progress_since) means a double-click only starts one background process; the second click sees “still running” and bails.

The same poll runs automatically every day at 04:15 UTC, but the on-demand path is what you reach for after a manual upgrade so the dashboard reflects reality immediately.

What gets skipped at queue time

The controller (OperationsUpdatesController::queueBulk) is defensive — even if a stale form re-submits a row that shouldn’t move, the row stays put. Skip reasons surface in the response flash message:

• ignored — server.is_ignored=true.
• already running/queued — row is already in flight.
• missing SSH credentials — no jail provisioned and no stored password.

Auto-refresh of snapshot after a successful upgrade

ProcessServerUpdates inline-re-polls each server right after apt-get upgrade succeeds — same probe clockwork:poll-system-updates uses, scoped to one row. Without this, a successful upgrade left the snapshot showing the pre-upgrade pending counts until the next scheduled poll at 04:15 UTC, making the dashboard look like nothing had changed for up to 24 hours. Failed upgrades skip the re-poll (the failure log on last_update_log is enough forensics; re-polling could mask the prior state).

Stale-running reaper

clockwork:reap-stale-server-updates runs hourly and flips any row stuck in update_status='running' for more than 2 hours back to failed so it can be re-queued. Real apt-upgrades top out at ~30 minutes on the heaviest server; 2h is comfortably past the worst-case real path. Each reap writes an action_logs entry (server_update_reaped) so when you see a row in failed state after the fact, the trail’s there.

The 2h threshold is the STALE_THRESHOLD_MINUTES constant on ReapStaleServerUpdates — bump it if you ever genuinely have an upgrade that legitimately takes longer.

Reboot

Click Reboot server:

1. The button POSTs to /servers/{id}/reboot, sets scheduled_reboot_at = now() + 1 minute.
2. Backend SSH-issues sudo shutdown -r +1.
3. The server schedules the reboot in /run/systemd/shutdown/scheduled and the SSH command returns cleanly with STATUS: reboot-scheduled.
4. After ~2 minutes, the post-reboot probe runs every poll cycle until SSH responds — at which point we clear reboot_required and scheduled_reboot_at.

Why +1, not now: shutdown -r now would kill the SSH session mid-command, leaving us guessing whether the schedule was even accepted. The 1-minute delay lets the SSH command return cleanly so we get confirmation.

Default behavior when no reboot_at is specified on a bulk upgrade: if /var/run/reboot-required exists after the upgrade, ServerUpdater defaults to shutdown -r +1 (same 1-minute grace as the manual reboot button). Previously a blank reboot_at silently skipped the reboot step and the kernel update sat un-applied until somebody noticed and clicked the manual button — defeating the point of bulk-patching for security. Set reboot_at=HH:MM explicitly if you want to schedule for a maintenance window instead.

Why not the DO API: DO’s reboot endpoint is a hard hypervisor reboot — can corrupt InnoDB. We always use OS-level shutdown so filesystems flush cleanly.

Cancel a scheduled reboot with the Cancel reboot button (runs sudo shutdown -c over SSH).

Sync from fresh probe

clockwork:poll-system-updates re-SSHs to each server and syncs upgrade_required + reboot_required from the live probe so the dashboard row reflects the latest state even when no manual update has been queued. This is how the Issues page “Patches Available” and “Reboot Required” counts stay current between manual operator actions.

It’s scheduled twice, at different scopes (Scheduled jobs has the exact times): a daily run and a weekly --all sweep covering every monitored server unconditionally. The daily run’s default (non---all) query targets servers SpinupWP’s mirror has already flagged upgrade_required=true, plus every server with no spinupwp_id at all (i.e. not SpinupWP-managed) — because upgrade_required is only ever set by the SpinupWP import, a GridPane/Hetzner/custom-VPS server would otherwise never trip that flag and would sit unprobed until the next weekly --all sweep, up to a week away. Including every non-SpinupWP server in the daily query closes that gap: those boxes now get probed daily like everything else, rather than only weekly.

Probe state

/servers/{id}/reboot/probe re-runs the SSH probe on demand. Useful when you suspect the server is back up but the post-reboot scan hasn’t fired yet.

Tags, and the staging exclusion

TagsController manages the fleet’s server tags (Settings → Tags, /settings/tags) — free-form labels like Shared, Dedicated, or Staging, each with a name, hex color, optional description, and sort order. TagsController::syncServer() (PATCH /servers/{server}/tags, called from the server detail page) attaches/detaches tags on one server via $server->tags()->sync(...).

One tag matters here: a server tagged staging is excluded from the probing/alerting side of this pipeline, but not from actually applying system updates. Server::scopeMonitored() (is_ignored=false AND NOT tagged staging) still gates clockwork:poll-system-updates and every other probe/scan/alert loop — a staging server’s apt-update snapshot isn’t refreshed on the daily/weekly schedule, and it won’t fire alerts. But queuing and draining an update is a separate, narrower scope: Server::scopeEligibleForSystemUpdates() (is_ignored=false only) is what ServerUpdateController::queue() checks and what clockwork:process-server-updates queries through, so a staging server can be queued from /servers/{id} and will be drained by the scheduler like any other non-ignored server. A successful run also re-probes that one server inline (bypassing the daily/weekly monitored() gate), so its package-count card reflects the upgrade immediately rather than waiting for the next scheduled poll.

This split is deliberate: staging boxes are noisy and not client-facing, so paging on their patch state wastes attention — but operators still need to be able to patch the box itself on demand. A staging server’s package-count card only refreshes right after you run an update (the inline re-probe above) or on a schedule if you remove the staging tag (putting it back in the full monitored() set) — there’s no separate manual “poll now” for the apt snapshot; Recheck state on this page only re-probes reboot-required/uptime, not package counts.

Symptoms of a stuck queue

The most common cause: the scheduler isn’t running. clockwork:process-server-updates runs every minute via the schedule — without it, queued updates sit forever. See Runbooks → Scheduler stuck.

What this isn’t

• Not WordPress core / plugin updates. That’s WordPress plugin inventory.
• Not provisioning. Servers are provisioned by SpinupWP. We don’t create or destroy them.
• Not a security update tracker. clockwork:composer-audit (weekly Mon 06:00 UTC) audits this app’s Composer dependencies. The clockwork:security-check command at 06:30 runs system-side checks. Per-server CVE tracking is on the backlog.

Gotchas

• shutdown on Ubuntu 22.04 is a symlink to systemctl (/usr/sbin/shutdown -> /bin/systemctl). shutdown -r +1 is systemctl reboot with a delay. shutdown -c clears it. Both write /run/systemd/shutdown/scheduled.
• apt-get output can be huge. We cap captured logs at 1 MB; over that, the tail is preserved.
• Concurrent SSH sessions to one server can step on each other. The every minute, one server per tick cadence is deliberate — don’t increase it unless you’ve thought about lock contention.
• Don’t run clockwork:process-server-updates --limit=N (N > 1) by hand while the scheduler is also draining the queue. Both processes race for queued rows; the atomic-claim filter prevents double-running the same server, but you can end up with N concurrent SSH sessions across the fleet which spikes load on the Clockwork Control box and on the Apt mirrors. The single-server-per-minute scheduler cadence is the right pace 99% of the time. If you need a one-shot bulk drain, kill schedule:work first, run --limit=N, then restart the scheduler.
• Form re-submit landmine (fixed). A POST/302 response from the bulk-queue endpoint used to let the browser remember the URL as a POST target — a Cmd+R on the resulting page could then re-submit the form and quietly re-queue the servers that had just completed. We now return a 303 See Other to the index URL, so reloading the landing page is unambiguously a fresh GET. If you ever see the dashboard counts stop dropping while the queue keeps draining, check the URL bar — that’s the symptom.