Standby

Changelog

1. v0.9.5439/11/2026

   agentd 0.124.0 — Vector host label is agentd's name, not the kernel's (alpi shipped as host=":"); Windows Vector set to AUTO_START with a reconcile heal for existing hosts; OpenRC vector.status arm.

2. v0.9.5429/9/2026

   FIX FOR TWO BUGS IN v0.9.541's COLLECTOR, one of which is publicly visible. Deploy this promptly.
   
   The collector itself works — real scans upload again and fleet-summary moved from 07/08/26 to live. But its first run drained a month of accumulated done/*.rc markers, which I had not designed for:
   
   1. A missing results artefact was uploaded as a FAILED SCAN. A cron scan has no pre-created row, so each one INSERTED — ~6,600 junk rows on one host in a single pass. An unreadable artefact is the ABSENCE of a result, not a compliance failure.
   
   2. Markers whose results file had <score>0</score> and no rule-results uploaded as ok at 0.00%. On the live fleet that moved the average 72.0% → 63.0% and Failing 2 → 4 — and /api/oscap/fleet-summary feeds the PUBLIC TRUST PAGE. That published a worse compliance posture than we have, from scans that measured nothing.
   
   Fixes: the agent discards artefact-less debris locally and tells the server nothing; zero-rule evaluations are demoted with a reason and no score; the upload route enforces the same rule server-side, because the server owns what becomes public and the fleet is routinely three agent versions wide. A genuine 0% (with rule-results) still survives — the guard tests the evidence, not the number.
   
   Migration 20270324100000 clears what already landed: deletes the artefact-less rows, DEMOTES the zero-rule ones rather than deleting, since an evaluation did run and the fix for a compliance surface is never to make an inconvenient row vanish. Both statements are narrow by construction and not reversible; down says so rather than pretending.
   
   Verified live on noc before this change: collector working, PAM sweep auto-checked 18 hosts with the buttons gone, 75-min reaper held. Clean serialized build; both Linux targets selftest on alpine and ubuntu:24.04. 1191 agentd tests, 44 oncall tests, all agentd and migration gates green.
   
   The Windows .exe still ships unsigned.

3. v0.9.5419/9/2026

   agentd 0.122.0 — the OpenSCAP result collector. This is the fix for the month-long compliance-evidence gap; v0.9.540 only made the ack honest about it.
   
   TWO bugs had to both be present for the silence, and both were:
   
   1. No collector. oscap.scan is detached and its outcome lands on disk; backup.run and av.clamav.scan have collectors in agent.rs for exactly this reason (the AV one's comment says it was "missing for the same reason" as the backup one). oscap is the third and never got one. The Python agent uploaded its own results; agentd had no reference to /api/agents/:id/oscap/upload at all.
   
   2. The oscap wrapper created its data dirs root:root while the agent runs unprivileged — the same bug 0.43.0 fixed for the backup and av wrappers in the same file. It stayed invisible because there was no collector to be denied.
   
   Ships: collect_results/discard_result + agent.rs ship_oscap_results; discard only after the server acks; BOUNDED retry (this body is megabytes where its siblings are bytes) that sets a scan aside rather than deleting it; agent-owned wrapper dirs; /var/lib/standby-agentd/oscap in AGENT_STATE_DIRS; and `statedirs` added to the post-install loop so the ownership repair lands on THIS upgrade rather than the next hourly one.
   
   score_pct is the number oscap computed, never recomputed from counts — a weighted benchmark would give a different number and disagree with the HTML report from the same scan.
   
   A new class gate globs the handlers directory (a list would have been written when there were two) and immediately found a fourth detached handler, winupdate, which now DECLARES its exemption rather than having it by omission. Mutation-testing the gate found a hole in it: a dead ship_* helper satisfied a plain contains() — the exact shape of the bug — so it now strips comments and checks the enclosing fn is invoked.
   
   Build: clean serialized build after discarding a raced one whose bytes could not be attributed to a source tree. Both Linux targets selftest on alpine AND ubuntu:24.04. 1187 tests pass. All agentd release gates green.
   
   STILL UNVERIFIED: no scan has completed end to end on a real host. Deploy, update one agent, run one scan.
   
   The Windows .exe ships unsigned, as before.

4. v0.9.5409/9/2026

   Audit of the OpenSCAP subsystem. Contains the "Scored / No score" tile split and the agentd collector finding.
   
   ROOT CAUSE OF THE MONTH-LONG EVIDENCE GAP, and it is not the reaper: agentd has no OpenSCAP result collector. oscap.scan starts a detached 10-20 min evaluation and acks "poll oscap.scan_status for the result" — a capability that does not exist. handle_scan_status is registered nowhere and carries #[allow(dead_code)], so the warning that would have flagged it was switched off; its module header says it "is the collector the heartbeat WILL use". The Python agent uploads results via POST /api/agents/:id/oscap/upload; agentd never calls that endpoint. When the fleet moved to agentd in early August the collection path was left behind. Verified live: hourly dispatches, every ack started:true, last result of any kind 07/08/26.
   
   THE COLLECTOR IS NOT FIXED HERE. This release makes the ack honest (result_collected:false, no false instruction to poll) and adds a gate that fails when an ack promises an unregistered capability.
   
   Also: the tile split — a host whose last scan FAILED is not a host that was never scanned. Live on noc, "Never scanned 12" sat beside a table printing "failed 32m ago" on the same host.
   
   Live-verified after the box auto-updated to v0.9.539: reaper fix holds (scan ran 462s, ended by operator cancel, not the sweeper — every prior scan died at 304-358s); tiles read Scored 16 / No score 12 / Failing 2 / Average 72.0%; the three manual-repair buttons are gone.

5. v0.9.5399/9/2026

   Removes three OpenSCAP page buttons that were each a manual workaround for something that should heal itself. Supersedes v0.9.538 — deploy this one.
   
   "Repair agent permissions" (button + route) deleted. It dispatched agent.refresh_install with args:{}, and both of that handler's flags default to false, so a control labelled "Re-apply agent sudoers + systemd unit" re-applied neither. The job is already done hourly: update --apply runs reconcile_host_state() before its version check, precisely so a converged host still gets its unit, grant and wrappers re-applied.
   
   "Check PAM profiles" / "Repair PAM profiles" replaced by sweepPamCollisions in the scheduler. NOT a blind fleet-wide apply — the agent's own handler argues against that in writing, which is why apply defaults to false there. The read-only check runs on a schedule and apply goes only to a host whose own report came back needs_apply. `ambiguous` is never auto-applied (no packaged profile to fall back to; apply cannot fix it) and is logged for a human. Bounded at 8 dispatches/tick and 3/host/day. A gate asserts exactly one apply site, inside that branch.
   
   The PAM verdict now arrives on the coverage rows as a per-host state, so it is visible without anyone clicking; the old pill only existed while an operator watched a command they had just fired. Healthy renders nothing.
   
   Kept: "Rollback last fleet-wide" — a deliberate destructive undo of a remediation an operator chose, not a workaround for our bug.
   
   Also carried: v0.9.538's orphan-reaper fix (OpenSCAP had produced zero successful scans since 07/08/26) and v0.9.537's agent staging-directory fix.

6. v0.9.5389/9/2026

   Two fixes.
   
   1. OpenSCAP produced ZERO successful scans between 07/08/26 and 09/09/26 — the orphan reaper killed every one. `ORPHAN_MAX_MIN = 5` was calibrated on the Python agent ("real CIS L1 scans complete in <2 min"); the eval now runs deliberately starved under systemd-run (CPUWeight/IOWeight=10, Nice=19) and takes 46-1210s, and agentd detaches it while continuing to heartbeat — so the sweeper's "agent is alive, therefore not scanning" test became true of every healthy in-flight scan. 21 of 23 failed rows carry its message, each dying at 304-358s. Threshold is now DERIVED from the agent's own SB_TIMEOUT=3600 (75 min) and pinned there by a gate that reads the agent source.
   
   2. The same page's stat tiles counted never-scanned hosts as 0%: Number(null) is 0, not NaN, so an isFinite guard passed them. "Never scanned 0" displayed beside a table printing `never`; the average was real scores diluted with a zero apiece.
   
   Also carried from v0.9.537: the agent's staging directory was created only by `update --apply`, which is a timer up to 45 min out — so a freshly enrolled host failed every staging-backed capability (vector.install, backup.install, containment, hostops, coraza, cred, ha, ipcontrol) with EACCES until its first self-update. The installer now creates it.

7. v0.9.5369/7/2026

   Vector 0.58.0: config gate pinned to the 0.58.0 debian + alpine images (7 variants green, both canaries still rejected). No config adaptation needed — none of the 0.58.0 breaking changes touch the rendered pipeline. 0.58.0 deb/rpm/msi/musl packages registered on noc (19 active rows, sha256 cross-checked against upstream SHA256SUMS). Fleet rollout is a vector.install dispatch from Hosts.

8. v0.9.5359/6/2026

   Fixes the retention-drift check, which could never have worked: it read the logs table's TTL from system.tables using the read-only metrics reader, whose grants are correctly limited to standby.logs and standby.metrics_rollup. Every query was refused, the verdict was permanently "unknown", and a claimed 90-day retention sitting over a 365-day table TTL could not be reported. system.* reads now use the admin credentials that already perform the ALTER TABLE MODIFY TTL, while standby.logs continues to use the reader. Missing admin credentials are reported distinctly from a refused query, since the two need different fixes.

9. v0.9.5349/6/2026

   Worker liveness. Standby could not answer "did my watchdog run?" — the only uptime available was the API process's, while in production the schedulers run in a separate unit. Since a check that exists and says nothing looks identical from outside to a check that never ran, every alert in the product rested on an unverifiable assumption. Workers now record a heartbeat on every tick including failing ones, staleness is judged against each worker's own declared cadence (with a floor, and never-ran treated as the worst case rather than skipped), and the staleness check itself runs in the API process because a check among the workers cannot report the workers being dead. Adds GET /api/admin/workers/health, split-brain detection for both units running schedulers, and makes the log-store checks report when they cannot measure instead of returning silently.

10. v0.9.5339/6/2026

    Fixes a design flaw in v0.9.531's log-store capacity check, found by auditing it live. The check computed days-until-full and then gated the whole alert behind the occupancy threshold, so the countdown could only appear once the disk was already past 80% — at which point the rate adds nothing. It now fires on the rate alone within a 30-day horizon, takes the worse of occupancy and runway for severity, and says explicitly when it was raised on the fill rate rather than the percentage. Also corrects figures in the source comments: df -h prints GiB, so the reference volume is 160.9 GB rather than the "150G" displayed, and the disk is at 77.4% rather than the 82.8% a bad division produced.

11. v0.9.5329/6/2026

    Log retention: the setting now reaches the table. The ALTER TABLE ... MODIFY TTL fired only when the retention value CHANGED, so a value written once at install never applied — on the reference install the UI stated 90 days while ClickHouse enforced 365, indefinitely. Saving the retention field now re-applies it even when unchanged, and a new check reports drift between the claimed and enforced windows every telemetry tick, distinguishing "keeps longer than claimed" (a storage-limitation problem) from "keeps less than claimed" (silent data loss). The detector deliberately issues no ALTER of its own: applying a shorter window schedules deletion of everything past it, so that stays an explicit action in Settings.

12. v0.9.5319/6/2026

    The log store now reports its disk by trajectory rather than occupancy. On the reference install 117 GB of 150 GB with 22 hosts reads as "room for five more hosts" — but the table held 97 days of data under a 365-day TTL, so it had evicted nothing and was ~23 days from full with no new hosts at all. The new check leads with days-until-full, only projects while the table is still filling its retention window (a countdown at steady state would be meaningless), parses both TTL spellings ClickHouse and our own installer emit, and says when its size figure comes from the whole mount rather than ClickHouse itself. Also reported and NOT fixed: the log-collection retention setting only reaches ClickHouse when it changes, so a displayed 90-day retention can sit over a 365-day table TTL indefinitely.

13. v0.9.5309/6/2026

    Standby now monitors its own log store. A ClickHouse table that fails to load leaves the server up and answering, so systemd reads "active" and a metrics-based blackout detector sees nothing — on the reference install that combination hid an unqueryable logs table for 18 hours. The new probe queries the TABLE rather than the server (a bare SELECT 1 answers happily throughout), treats "up and refusing" as its own state, never clears on an unconfigured install, and carries ClickHouse's own exception text into the alert. The Logs tab also now shows the upstream HTTP status and full error body, which the API had always returned and the page discarded.

14. v0.9.5299/6/2026

    Privileged-session evidence. The brokered web terminal has recorded an Ed25519-verified root PTY per session since August, and nothing claimed it: no compliance control read a transcript, and the recordings had no retention class at all — every keystroke and every byte of output of a root shell held in Postgres indefinitely (SOC 2 C1.1/C1.2 disposal, GDPR Art. 5(1)(e), ISO A.8.10). Transcripts now age out at 365 days, operator-tunable, while the session record itself is never pruned because that is the audit trail. Adds a session review workflow (self-review refused), makes the session justification mandatory, and scores all of it through privilegedSessionAudited() against SOC 2 CC6.1, ISO A.8.18 and NIS2 11.4 — added alongside the existing questionnaires rather than replacing them.

15. v0.9.5289/6/2026

    Compliance evidence: two automated checks were grading whether a feature was switched on rather than whether it worked. monitoringConfigured returned "met" for any install with at least one enabled alert rule — satisfying 23 controls across SOC2, ISO 27001, ISO 20000, DORA, PCI DSS, NIS2 and CRA, including SOC 2 A1.1 (processing capacity) — on evidence of the form {"alert_rules": 4}. It now grades coverage: which hosts are matched by a rule, and whether a filesystem-space rule exists at all. incidentResponseConfigured returned "met" for "a chain exists and someone has a phone", and now reads the live pager-channel-broken signal, because a chain that cannot deliver is not incident response. WARNING: scores will FALL where coverage was previously assumed — that is intended, and the route back to "met" is remediation rather than a better proxy.

16. v0.9.5279/6/2026

    Fixes a permanent-mute bug affecting every alert source. ingestExternalAlert's re-fire branch bumped a dedupe counter and returned without ever checking whether the incident that alert belongs to still existed, and resolveIncident does not resolve the alert — so an incident closed under a live alert meant the condition kept firing every tick with nothing on the board, forever. On the reference install this had muted two CRITICAL "pager channel broken" alerts since 19 August. WARNING: on first deploy this will open roughly 11 incidents at once (3 critical) for conditions that are real and mostly weeks old.

17. v0.9.5269/6/2026

    The backup sink can now raise its own alert when it fills. Previously the Backups page showed 141/148 GB (96%), a collapsing data-per-day curve and 6 failed runs, while capacity-status returned ok:true (which meant "the number is readable", not "the number is fine"), the only filesystem alert rule watched inodes (at 0.15%), and the reachability card claimed all hosts "can write to their sink" on the strength of an auth + path probe. The only thing that paged was backup-stale on the innocent hosts underneath. New verdict: WARN 80 / HIGH 90 / CRITICAL 95, measured against used+available rather than used/total (a filesystem reserves ~5% for root, so the writer sees a higher figure than used/total suggests), with an explicit unknown state that never clears a firing alert.

18. v0.9.5259/6/2026

    Two fixes. (8) The incident command timeline shipped ~1 MB per incident — a 200-char output preview on every row plus five more fields no component renders; trimmed to the nine fields the page actually reads, with a guard that names each field's reader. (9) ip_control rule #765 was never retried on ten hosts for 26 days: the hard re-verify paced off updated_at, which the reconcile loop re-stamped every 2 minutes, so it could never elapse. It now paces off last_dispatched_at, and the reconcile no longer makes the no-op write. The hard-fail alert also claimed the addresses were unenforced while fail2ban was holding them — it now checks sibling layers and downgrades to warning when one is enforcing. Includes v0.9.524's browser-tab hang fix.

19. v0.9.5249/6/2026

    Fixes a hard browser-tab hang on 30 of 134 live incidents. An incident title carrying raw firewall-engine stderr (a literal newline) was assigned to document.title, which reads back with whitespace collapsed — so the Shell's "keep our title" MutationObserver re-assigned a value that could never compare equal to itself, in an unbounded microtask loop. Cold page load only; the same incident reached by clicking a link was fine. Fixed in the shell (fixed-point normalisation + a non-self-triggering observer), at both alert/incident title write boundaries, at the ip_control producer, and with a migration that collapses the rows already written.

20. v0.9.5239/5/2026

    Contract §20.3.1 merged (v9): kstack v1.61.0 reads Standby's hosts and CVE findings with its pairing token. The CVE read gate now lets a service-account token holding host.view GET /api/cve-findings and /api/cve/summary only; a signed-in person still needs super_admin or security_admin; OpenAPI corrected to say which is which. No migration, no UI change.