Standby

Choose the networks people may use Standby from, and a status page installer that no longer leaves the page offline when it is run again.

Added

• Settings → Console access limits where anyone can sign in to Standby and use it from. Add your office, your VPN, or each engineer's address, with a label and an optional expiry date. A person must pass this list and their own IP allow-list, if they have one. The card shows the address Standby sees you on, including your IPv6 range, and lists refused attempts, which are also audited. Host agents, the status page, alert and webhook ingest, install commands and service-account tokens are not affected.
• Standby will not save a list that locks you out — and there is a way back in if one ever does. If an address changes or an entry expires, run sudo standby-access --allow <address> or sudo standby-access --off on the Standby server.

Fixed

• Re-running the status page install command no longer leaves nginx stopped and the status page offline. The installer now starts nginx when it is not running, enables it at boot, and stops with a clear error if it still cannot run. The Standby server's installer also enables nginx at boot after moving it to nginx.org.

Upgrading

If a status page re-run ended with "nginx.service is not active", run sudo systemctl enable --now nginx on that server, then mint a new install command in Status admin and run it there again to finish the certificate and connection steps.

Why

A console that holds your on-call, compliance and fleet controls should only be reachable from places you trust. And a status page must come back up when its installer is re-run, not go dark.

Host agent 0.140.0 is included. Update all agents now works on Alpine (OpenRC) hosts.

Fixed

• Update all agents did nothing on Alpine (OpenRC) hosts. The agent tried to start a systemd service that does not exist there, and the result said the host had a sudo permission problem. An Alpine host now runs its own daily update script on demand, and the result names that script.
• The result says when an Alpine host's daily update cannot run. If crond is missing, stopped or not set to start at boot, the update result says so and gives the command that fixes it.
• The Alpine update script keeps a log. What each update did is written to /var/log/standby-agentd-update.log on the host, instead of being discarded.

Upgrading

• After updating, open Fleet → Update all agents. Alpine hosts take 0.140.0 on their next daily update run; from then on, Update on an Alpine host starts the update straight away.

Why

On Alpine hosts, Update and Update all agents did nothing, and the result blamed a sudo permission problem. Those hosts only updated once a day, and only when their scheduler was running.

Security release. Update promptly.

Status page app 0.3.54 is included. Install tokens now open only the install they were made for, the status page gets its own credential toward Standby, and updates no longer roll back over an unused status page setup.

Fixed

• Updating to 0.16.17 rolled back on servers with an unused status page setup left by an earlier install. The update meant to remove that setup could miss it, and its post-update health check then waited for a status page that had never run, and rolled a good update back. The leftover is now removed reliably, and the health check counts the status page only if it was running before the update.
• Incidents the status page detected itself never reached on-call. It sent them to a Standby address that nothing had configured. It now uses the Standby address and token every installed status page already has.
• A new status page server came up on older PostgreSQL, Redis and nginx than the Standby server. Its installer now uses the same sources as a Standby server: PostgreSQL 18, and current Redis and nginx from their own signed repositories. It falls back to the system's packages, with a warning, if one cannot be verified.
• Host pages showed an IPv6 address instead of the host's own IPv4. The host page, the host peek, the external-scan page, the infrastructure register PDF and the AI assistant now show the host's own address: its public IPv4 when it has one. The host page still shows the connection address when it differs ("seen from"). The fleet known_hosts file now lists both addresses.
• Fleet naming could publish an IPv6 address as an A record, and suggest an invalid SPF ip4: entry for mail hosts. It now uses the host's own address: an A record for IPv4, AAAA for IPv6, and ip4: or ip6: to match.
• DNS nodes on a server with IPv6 never had their IPv4 allowed to transfer zones. Their IPv6 address was stored as their IPv4 one. Both addresses are now recorded correctly, and the update corrects existing nodes.
• Re-running the status install command kept the old credentials. After reconnecting, every call between Standby and the status page failed. A re-run now brings Standby's address, credentials and release key up to date and keeps anything you added.
• A status-page update that needed a new or newer dependency could leave the page down. It now installs dependencies whenever a release pins different versions, into a cache it can write.
• Status admin offered to forget a working status page when it could not reach it, for any reason. It now says why (credential refused, certificate, timeout, nothing answering) and asks before forgetting.
• Every status page read through Standby wrote an audit entry. Including background refreshes every 10–20 seconds on the status board and every host page. Only changes are audited now.

Security

• Any install token could open any one-server installer. The status page, telemetry, backup and edge-firewall install commands accepted every enrolment token Standby had issued, including the agent install token that appears in every fleet install command. The status install script also carries the status page's credential. Each install command's token now opens only the install it was made for, for that one server; fetching the status install script uses it up.
• The status page's phone-home could set the credential Standby stores. It now sends only the page's address. Standby connects only after that address answers an authenticated check over HTTPS with the credential Standby already holds. Private addresses are refused unless ALLOW_PRIVATE_STATUS_URL=1.
• One status credential worked in both directions. The status page now reaches Standby with its own token, separate from the credential Standby uses to manage it. Once a status page has used its own token, Standby stops accepting the old shared one for alert silences, incidents and release downloads. Status settings → Rotate credentials replaces both at any time.
• A status-page maintenance could silence every alert for as long as the request said. Standby now reads the maintenance back from the status page, uses its own scheduled times, never silences for more than 24 hours, and notifies admins whenever alerts are silenced this way.

Removed

• "Connect automatically" from the licence is gone from Status settings. It could not work. A status page connects itself when you install it with the command from Status admin.

Upgrading

• If 0.16.17 rolled back on your server, install 0.16.18 from Settings → Updates.
• Update the status page (Status settings → Update status app), then Rotate credentials on the same card. Until you do, the status page keeps working on the old shared credential and the card shows a warning.
• If you run DNS nodes with kstack, push them once after updating (Integrations → kstack → Secondaries → kstack), so kstack allows their IPv4 addresses to transfer zones.
• To bring an existing status page server up to date, re-run its install command from Status admin on that server. nginx moves over automatically. To move Redis and PostgreSQL as well, add REDIS_SOURCE=redis.io PG_UPGRADE=1 after sudo. Your PostgreSQL data is copied to the new version, and the old cluster is kept, stopped, until you remove it with pg_dropcluster <old version> main.

Why

An agent install token could fetch the status page install script, which carries the status page's credential, and that credential could silence every alert. This release closes that path and fixes the update that rolled back on 0.16.17.

Moving hosts or telemetry to a new server now reconnects by itself. Host agent 0.139.0 is included.

Added

• Telemetry → Test connection. It checks that the telemetry server accepts Standby's credentials for metrics and logs, and says how to fix it if not.

Fixed

• Moving a host to a different Standby server meant deleting its agent folder by hand. Running the agent install command from the new server now re-enrols the host by itself. The old identity is kept beside the new one, and nothing has to be deleted.
• After moving telemetry, hosts stopped shipping and Install Vector (fleet) seemed to do nothing. Re-running the telemetry installer now always hands its credentials back to Standby. When the telemetry server or its credentials change, every host's Vector is re-configured automatically. Install Vector (fleet) shows each host's outcome.
• An unused status page setup left on the Standby server is now removed by the update instead of warned about. A status page that runs or holds data is never touched.

Upgrading

After updating, open Fleet → Update all agents. If hosts show "agent alive but not shipping metrics", open Telemetry → Test connection. If it reports the token was rejected, re-run the telemetry install command from that page; hosts then catch up on their own.

Every release is now proven by a fresh Debian 13 install and an update through the console button.

Changed

• Every release is checked by a fresh Debian 13 install followed by an update through the console's button, before it can be published.

Fixed

• A fresh install failed on a minimal Debian 13 server without rsync.
• The update log wrongly said the maintenance page was missing on Debian and Ubuntu installs.

Status admin now recognises a status page that no longer exists and lets you install a new one.

Fixed

• Status admin kept a status page that no longer exists, and hid the install command because of it. It now checks whether a status page answers at the saved address. If none does, it says so and offers Forget this address, after which the install command for a server of its own is back.

The updater now updates itself first, so fixes to the update process always arrive through the update button.

Changed

• Every update runs the updater from the release being installed. After downloading and verifying a release, the updater installs that release's own updater and lets it carry out the update. A server never again needs a command run by hand to get past an updater bug.

Upgrading

Servers on 0.16.13 or earlier need this release installed once with the two-line install command or sudo standby-update. From then on the console's update button is always enough.

Updates no longer stall on the status database, and the hosts list shows each server's own address.

Fixed

• Updates were refused with "pre-update database snapshot failed" on servers carrying a status database. The database check failed on a database with no migrations yet, and a status database owned by its own role could not be read. The updater now backs such a database up as the database superuser, and the check works on an empty one.
• The hosts list showed IPv6 addresses, and 127.0.0.1 for the Standby server. It showed where each agent's connection came from. It now shows the host's own address from its network interfaces, public IPv4 first.

Upgrading

On a server where updates were refused, run the two-line install command once. The in-app button works again after that.

The public status page now always runs on a server of its own.

Added

• A fresh install has a status release ready to install from Status admin.

Changed

• The public status page always runs on a server of its own. New installs no longer create a status database, service, nginx site or certificate on the Standby server, and no longer ask for a status domain. Install it from Status admin on a fresh Debian 13 or Ubuntu 26.04 server; the command refuses to run on the Standby server itself.

Fixed

• The console connects to the status server's public https address, never a local one.

Upgrading

An existing install whose status page runs on the Standby server keeps it running. Each update reminds you to move it to its own server (see the Operations guide, "Moving a co-located status page").

The telemetry server now installs on Debian 13.

Fixed

• The telemetry server installer stopped at ClickHouse on Debian 13 ("the repository is not signed"). It now installs ClickHouse's published key in the format Debian 13 expects, and a re-run repairs a half-installed server.
• On Debian, the telemetry installer took nginx from the wrong distribution's repository.
• The telemetry install command no longer pipes into bash.
• The status-page connection said "Connected" even when no status page answered.

Upgrading

On a telemetry server where the install stopped, mint a new command on the Telemetry page and run it again. It repairs what the earlier attempt left behind.

Updates work again on servers that do not run the status page.

Fixed

• Every update was refused on a server that does not run the status page, with a false "DATABASE RESTORE FAILED". The updater tried to back up the status page's unused database, could not, and refused to continue. It then reported a failed restore of a database nothing had touched; your data was never changed. The updater now backs up the status database only when a status page runs on the server, and never restores a database it did not back up.

Upgrading

On an affected server, run the two-line install command once. The in-app button and standby-update work again after that.

The in-app Install update button now works on Debian 13 and Ubuntu 26.04 servers.

Fixed

• Installing an update from the console failed with "Access denied" on Debian 13 and Ubuntu 26.04. The button relies on polkit, which their minimal images do not include. The installer and every update now install it.

Upgrading

On an affected server, update once from the command line (sudo standby-update, or the install command). After that the button works.

The public status page can now run on a server of its own.

Added

• The status page can run on its own server. Run the command from Status admin on a fresh Debian 13 or Ubuntu 26.04 server. It installs what the server needs (Node 26, PostgreSQL, Redis, nginx, certbot), then installs the status page, connects it to Standby and gets its certificate. It still installs on the Standby server itself as before.

Fixed

• A status page's certificate stayed self-signed; it is now issued by Let's Encrypt.
• The status-page install command no longer pipes into bash.

Installs without an uploaded logo now show the Standby mark.

Fixed

• An install with no logo uploaded showed a plain coloured square in the sidebar and on the sign-in page. It now shows the Standby mark. Upload your own under Settings → Branding to replace it.

Finishing disaster-recovery setup now opens the console straight away.

Fixed

• After finishing disaster-recovery setup, every menu click went back to the setup page until the browser was reloaded. The console now refreshes its setup status when setup finishes, and checks with the server before sending anyone back to setup.

The certificate repair now works on sites where an earlier certificate request failed.

Fixed

• The certificate repair skipped sites where an earlier, failed request had left a broken record. The repair now checks the certificate actually served. A temporary or nearly expired one is replaced with a Let's Encrypt certificate, and the update shows each site's certificate status, including Let's Encrypt's reason if it refuses.

Upgrading

Run the two-line install command again, or sudo standby-update. The update prints each site's certificate status.

Updates now repair a site that was left on a temporary certificate.

Fixed

• Installs made before 0.16.3 stayed on a temporary self-signed certificate. Every update now checks each site Standby serves. A site still on the temporary certificate gets a Let's Encrypt certificate. If the domain's DNS does not point at the server yet, the update carries on and the next one tries again.

Upgrading

Run the two-line install command again, or sudo standby-update. The certificate is repaired as part of the update.

New installs get a real Let's Encrypt certificate.

Fixed

• A new install kept a temporary self-signed certificate, so browsers showed the site as not secure. The temporary certificate sat where Let's Encrypt keeps its own, and the certificate request was refused. It now lives elsewhere, and Let's Encrypt replaces it as intended.

First installs on a new server now run to the end.

Fixed

• A first install stopped silently right after "license valid — proceeding". On a new server there is no PostgreSQL yet, and a check for an existing database ended the installer instead of moving on. The installer now continues to the domain, email and password questions and provisions the server.

Upgrading

If a first install stopped after "license valid — proceeding", run the same two-line install command again.

Fixes first installs on a new server made with the two-line install command.

Fixed

• A first install on a new server stopped part-way with no message. The installer mistook the licence file it had just saved for an earlier install and stopped. A new server is now provisioned in full.
• A first install now records its update server. Licence answers are trusted and later updates know where to check.

Upgrading

If a first install of 0.16.0 stopped right after "in-place update", run the same two-line install command again.

Security release. Update promptly.

GA hardening release. It tightens sign-in, makes licensing tamper-proof, verifies every release and agent update end to end, puts disaster recovery on by default, and closes several gaps where paging could reach nobody. Host agent 0.138.0, status page 0.3.53 and probe 0.3.8 are included.

Added

• Disaster recovery is on by default. New installs walk the first administrator through an off-box backup, a dead-man URL and a recovery passphrase. standby-restore --from-restic latest rebuilds a server straight from the off-box copy. The audit log is archived to verifiable files after 400 days instead of growing for ever.
• Operator guides ship with every release and can be read in the app under Documentation. The guides cover administration, operations, recovery, high availability, the API and white-labelling.
• Support contacts in branding, a redacted support bundle command and a security.txt fallback.
• A DoD SPRS score for NIST SP 800-171 and CMMC Level 2.

Changed

• Licensing can no longer be bypassed. A licence key that has never been verified, or whose answer cannot be trusted, allows the free 5 devices for new enrolment. Hosts already enrolled, and paging, are never affected.
• New installs take nginx from nginx.org and Redis from the Redis repository, alongside Node 26 and PostgreSQL 18.

Fixed

• An escalation step whose channels could not reach a person paged nobody. It now falls back to email. If email cannot reach them either, an alarm goes straight to on-call and admins.
• Turning on two-factor authentication failed on some installs.
• Rotation handoffs happen at the rotation's own hour and time zone, including midnight and across daylight-saving changes.
• In-app updates show a maintenance page, pause background work while migrating and check that the new release starts before switching.

Security

• Email sign-in links are removed. Sign in with a password and two-factor authentication, or with single sign-on. An old link now points to the sign-in page.
• Single sign-on no longer links an administrator account just because the email matches. Link it from Profile → Single sign-on, or a super admin can allow a one-time link. The allowed email domains setting is now enforced for SSO, SCIM, invitations and email changes.
• New installs put fleet scripts and the web terminal on an allow-list, with two-person approval for scripts. Upgraded installs keep their settings and are shown the recommended ones. Root on any host can switch remote execution off locally.
• External scans need proof that you own the target. Proof is a DNS TXT record or a super-admin Attest ownership. A host name reported by an agent is no longer enough.
• Releases and agent updates are verified end to end. The installer checks the release signature before unpacking anything. Agent updates must carry a vendor signature, and an agent that has not reported back within 10 minutes rolls itself back.
• The API listens on the local interface only; nginx is the single public entry.

Upgrading

• After updating, open Fleet → Update all agents. This agent version is the first that checks the signature on the next update.
• Sign in as a super admin and finish disaster recovery: set a recovery passphrase, download the recovery bundle and store it off the server. A banner stays until you do. Also set an off-box backup repository and a dead-man URL.
• If you use push notifications, set your own ntfy server under Settings → Telephony. Each person then sets their topic on Profile → Push to my phone.
• Check that PUBLIC_BASE_URL is set in oncall.env. Installer and enrolment links are built only from it.

ClamAV signatures now stay up to date, and software installed on a host shows up without restarting its agent. Includes host agent 0.137.0.

Added

• A warning when a host's ClamAV signatures are more than 3 days old. It pages nobody, names the updater to start, and clears by itself once signatures are current again.

Fixed

• Installing ClamAV or updating its signatures no longer stops the agent or leaves signatures stale. The signature download ran inside the agent's own memory limit, ran out of memory and took the agent down with it, after the signature updater had been switched off and before it was switched back on. The download now runs on its own with enough memory, and the updater is always switched back on afterwards. Needs agent 0.137.0.
• Software installed on a host shows up without restarting its agent. After installing ClamAV, restic, OpenSCAP, fail2ban or the WAF, the agent kept reporting the host as it was before, so scanning or backups stayed unavailable. It now checks again straight after any install, and every 10 minutes for anything installed by hand.
• The ClamAV install confirmation describes what actually happens. It described an apt-only install that started the clamd daemon. It now says the scanner and signature updater are installed with the host's own package manager, without clamd.

Upgrading

After updating, open Fleet → Update all agents. If a host's ClamAV signatures were stale before this release, the new warning names it; running Update signatures on its Antivirus tab, once the host runs agent 0.137.0, brings its updater back.

Why

Installing ClamAV or updating its signatures could stop the host agent and leave the signature updater switched off, so signatures went stale for days with nothing said about it.

Installing Standby now takes two lines: your licence, then the installer. The installer asks for the domain, email and admin password itself, and the licence no longer travels in the URL.

Changed

• Installing Standby takes two lines: your licence, then the installer. Run these on the new server:

  export STANDBY_LICENSE='zlic_...'
  sudo -E bash -c "$(curl -fsSL -H "Authorization: Bearer $STANDBY_LICENSE" https://<your-update-edge>/install/standby)"
  

  The installer asks for the domain, the email and the admin password on the terminal. Run it exactly as shown; piping the download into bash takes the keyboard away from the installer, so it cannot ask its questions. Unattended installs still set DOMAIN, EMAIL and AUTO=1 (see the install guide).

Security

• The licence key no longer travels in the install URL. It is sent as a request header, so it stays out of web-server logs. The older command with the key in the URL still works.

Why

The install command used to carry four settings and put the licence in the URL, where it could end up in web-server logs and shell history.

Antivirus now runs on every host that has it, and installs on any Linux distribution. Every antivirus scan and backup had been skipping the last item in its list; that is fixed. A network card that stops transmitting now raises an alert. Includes host agent 0.136.0.

Added

• ClamAV installs on Alpine, Rocky, AlmaLinux, RHEL, openSUSE and Arch. It was Debian and Ubuntu only. RHEL-family hosts install it from EPEL.
• An alert when ClamAV has not completed a scan for 8 days. A warning that pages nobody. It is not raised for a host that has been offline for more than a day.
• Alerts for a network card that has stopped transmitting. Some Intel onboard cards (the e1000e driver, common on dedicated servers) can stop sending while the server keeps running. Four new log rules watch every host for the kernel's warnings; a hang pages, an unexpected adapter reset warns. They need Log rules switched on and telemetry configured, and can only fire while the card still sends something.
• Runbook: Network card transmit hang. How to confirm the fault, the immediate fix, making it survive a reboot, what to do when the host is unreachable, and when to ask your provider for a different card. The new alerts link to it.

Changed

• Installing ClamAV no longer starts the clamd daemon. Scans never used it, and it holds about 1 GB of memory. Hosts that already run it keep it.

Fixed

• Antivirus scans no longer skip the last folder in their list. A scan of a single folder, such as the weekly full-system scan of /, scanned nothing and reported that none of the requested paths existed. Needs agent 0.136.0.
• Backups no longer leave out the last folder, exclusion or tag in their lists. The same fault affected backup policies. After the agent update, every item in a list is used. If a policy's last folder matters to you, check that the next run includes it.
• Hosts with ClamAV are now actually scanned. Only hosts with a policy assigned by hand were ever scanned. Hosts with ClamAV and no policy now get the weekly full-system scan; a host you unassign stays unassigned.
• Full-system scans finish within their time limit. Scans that start at / now always skip container storage and the agent's own data.
• kstack node alerts are linked to their host. An incident kstack raised about one of its nodes, such as a node that stopped reporting, now links to that host in the fleet.

Upgrading

After updating, open Fleet → Update all agents. The scan and backup fixes take effect only once a host runs agent 0.136.0. Every host with ClamAV and no scan policy is given ClamAV — full system (weekly) (Sundays 01:00 UTC). To keep a host off a schedule, unassign its policy on the host's Antivirus tab; it then stays unassigned.

Why

Most hosts with ClamAV had never been scanned, because nothing gave them a schedule. Scans of a single folder (including the weekly full-system scan) scanned nothing, and backups were silently leaving out the last folder in their list.

SSDs near the end of their rated life now warn, a server model chosen by hand that contradicts the hardware is flagged, and Locate is no longer offered on drives that have no light. Host agent 0.135.0 is included.

Fixed

• An SSD near the end of its rated life warns. The host page's disk list warned at 80 % wear, but the drive report only warned at 90 %, and on a server that reports drive bays the drive report decides — so a drive at 86 % or 89 % raised nothing. Drives now warn at 80 % wear, or when less than 20 % of their spare capacity is left. These warnings page nobody.
• A server model chosen by hand that contradicts the hardware is flagged. If you pick a rack model for a server that reports itself as a desktop or tower, the front panel says that the drawn bays may not exist, and how to go back to Automatic.
• Locate is no longer offered on drives that have no locate light. On an internal drive (in a desktop or tower, an M.2 inside the case, or a VM's virtual disk) it failed with "ledctl is not installed (package: ledmon)", which suggested that installing ledmon would help. It would not. The drive panel now says there is no light to blink.

Upgrading

After updating, open Fleet → Update all agents.