We paused new features to give WEOUR OS a production safety net

This is not a feature release. It records why WEOUR OS paused new features to add a layer of production safety net: backup, scheduling, release, rollback.

Scope of this round

Production infrastructure (record of this round of work): backup · scheduling · release · rollback

Backup cadence

Every 6 hours (launchd timer)

Backup success contract

Four gates: stream / decrypt / structure / durable

Backup health tiers

HEALTHY ≤ 7 hours · DEGRADED ≤ 12 hours · STALE > 12 hours (FAILED means a single attempt failed, not a tier)

Scheduler stagger budget

Default 30 seconds (floor 0 / ceiling 60), decoupled from the tick interval

Scheduler tick P50 (4 stores)

119.5 s → 31.9 s

Scheduler tick P95 (4 stores)

369.9 s → 55.3 s

Scheduler duty cycle

83.7% → 17.1%

Scheduler acceptance sample

11 ticks (4 stores)

First canonical deployment Trial

Failed on 2026-09-23, rollback executed by the pipeline

Production validation for this round

release 20260923-rpg13-7f685ca (CANONICAL, validation / smoke PASS)

Release governance test cases

123 → 157

Retention state at this round's observation point

release 19 · deletion candidates 1 · unknown provenance 6 · protected 5 (retention dry-run)

Scheduler correction commit

Anxin360/weour-os@e92b059

readiness / iowait correction commit

Anxin360/weour-os@d4cbe01

Foundation V1.1 trial branch tip

Anxin360/weour-os@b8cc817

Governance V1.3 commit

Anxin360/weour-os@7f685ca

Why we paused new features first

WEOUR OS is already running real data in production: real stores, a real scheduling loop, real release records. The more features there are, the higher the cost of a mistake; but before this, several things had never really been closed out:

• Immutable releases only accumulate, and the disk does not clean itself;
• Backups existed, but "did this backup actually succeed" was never automatically checked, and there was no health signal that could be consumed automatically;
• The scheduler looked busy, but most of that busyness was not business work.

None of these are feature problems; they are problems with how the system runs. Fixing them first makes every feature added afterwards cheaper.

What changed

1. Release retention: immutable releases need governed cleanup

Releases are immutable whole builds: switching means pointing a symlink at the new release directory, and rollback means pointing it back at the old one. This design makes rollback cheap, at the cost of a disk that keeps growing — at audit time, release directories accounted for about 74% of the host's used disk (44.12 GB / 59.44 GB used), while the same disk also had to leave headroom for builds.

So cleanup was written as a classification priority, not "delete the oldest":

CURRENT > IMMEDIATE_ROLLBACK > DIAGNOSTIC > FAILED_TRIAL > VALIDATED_ANCHOR
      > MIGRATION_BOUNDARY > MANUAL_PROTECT > UNKNOWN_PROVENANCE > ROUTINE

Only ROUTINE enters routine quota cleanup; no other category is ever swept away in passing. One rule matters especially: the release of a failed Trial is not downgraded to ROUTINE just because its commit later landed on main. Classification rests only on lifecycle evidence, not on directory names, timestamps, or chat logs.

2. Backup: from "there is a backup" to "a backup that can be checked"

The shape of backup was not rebuilt from scratch: the backup host pulls, production only exposes a restricted forced-command stream, and the decryption private key never enters the production host. The changes happened on the pull side:

• Cadence and RPO: every 6 hours (launchd timer), with a target RPO of 6 hours. The cadence is explainable and does not depend on a drifting interval timer.
• Four gates: a run counts as successful only after all four gates — stream, decrypt, structure, durable — pass. A pg_dump exit code of 0 was never success by itself.
• Failure is failure: if any gate fails, the run is FAILED, last-success is not updated, and cleanup does not happen.
• Health tiers: HEALTHY (≤ 7 hours) / DEGRADED (≤ 12 hours) / STALE (> 12 hours) are derived from hours since last success and can be consumed directly by monitoring as an exit code; FAILED describes a single attempt, not a tier.
• Failure-injection tests: covering remote failure, undersized artifact, decryption failure, invalid structure, overlapping runs, stale lock, retention policy, and corrupted health state.

The last step was wiring the health signal back into production: after the four gates pass, the backup host pushes the snapshot to production through a restricted identity that can write only one fixed file — it cannot switch releases, cannot restart services, and cannot read any key.

3. Scheduler: separating the semantics of tick and stagger

Diagnosis first produced an uncomfortable conclusion: over the run window with 4 stores, 96.8% of tick wall-clock time was spent in deterministic stagger sleeps — 84 seconds of real work out of 2610 seconds. Counted per action, the sleeps alone came to 4150 seconds/hour, close to 115% of wall clock.

The root cause was mixed semantics: the stagger window was bound to the tick interval (180 seconds), and the fixed sleeps of multiple actions on the same serial runner were accumulated serially.

The correction separated the two: the tick interval is used only for reporting cadence and no longer participates in stagger computation; stagger has its own budget, default 30 seconds, selectable 0 / 30 / 60, with invalid values falling back to the default. Measured results after deployment:

• tick P50 119.5 s → 31.9 s, P95 369.9 s → 55.3 s;
• duty cycle 83.7% → 17.1%;
• planned sleep matches actual sleep, and the remaining time is explainable API and DB work;
• timer, cadence, store switches, and concurrency budget were not changed.

4. One canonical release pipeline

Release is no longer a string of manual commands but an ordered, reproducible pipeline:

provenance → preflight → release lock → scheduler quiet window → source
  → build backend → monitored build → immutable release + manifest
  → candidate validation → rollback readiness → atomic switch → restart
  → readiness → functional smoke → state write → history → retention dry-run

Several parts are deliberate: builds happen only inside the scheduler's quiet window; the switch is atomic; the rollback target is verified once before and once after the switch; and the exit code of systemctl restart is explicitly recorded as "unit started", not "application ready".

5. The first Trial failed, and was preserved in full

The first canonical deployment Trial ran through its own success path, then failed at the smoke stage, and the pipeline executed its own canonical rollback.

The cause was not a problem with the application: systemctl restart returning only means the unit started, not that the port is bound. The smoke at the time was an immediate HTTP check with no readiness wait, so it observed connection refused and gateway errors.

The handling is the point: nobody bypassed the pipeline to fix this step by hand. The pipeline stopped according to its own failure contract and executed the rollback, that release was preserved for diagnosis, and the cleanup stage performed not a single deletion. Production kept serving normally on the pre-switch release.

The correction released afterwards turned readiness into bounded polling and made build monitoring actually measure iowait. On the re-run, the earlier failure mode was correctly absorbed by the new logic as transient unreadiness rather than a deployment failure.

6. Governance: leaving evidence for both "who may release" and "what was released"

The rules added this round are for our future selves:

• canonical deploy only: day-to-day releases go only through the pipeline; exceptions must be explicitly marked.
• deploymentMethod: every new lifecycle record carries a method enum, history is appended rather than rewritten, and records that cannot be determined are marked UNKNOWN_LEGACY, never guessed as CANONICAL.
• Failed Trial classification: FAILED_TRIAL is determined by lifecycle evidence and does not consume routine cleanup quota.
• Diagnostic evidence first: before deletion, a KB-level diagnostic record containing no keys must be written back and verified; if it cannot be written, deletion is refused.

At this round's observation point (production release 20260923-rpg13-7f685ca), the retention policy was presented as a dry-run: the record showed 19 releases, of which 1 deletion candidate, 6 unknown provenance, and 5 protected. Cleanup happens only when the evidence is complete (the target is confirmed unreferenced, and the diagnostic record has been written back and verified).

How we verified it

At every step, verification was anchored as far as possible in reviewable evidence rather than "it looks fine":

• Backup was verified by failure-injection tests and real four-gate runs, and the health signal is consumed by monitoring;
• The scheduler correction was first simulated in an isolated environment with measured offsets, then compared before and after deployment using 11 real ticks;
• The release pipeline was verified by letting it run one real release end to end: readiness passed, functional smoke passed on six routes, state was written by the pipeline, and only then was the retention dry-run performed;
• Governance rules are covered by test cases, from "a failed Trial is not downgraded" to "the releaser rejects anomalous payloads".

What it does not mean

It is not a feature release, and it does not mean the infrastructure is finished.

In this round's record, the limitations of backup are written down explicitly: there is only one decryption key and it sits on the same machine as the ciphertext; there is no second off-site copy; there is no PITR, so the RPO is the 6-hour cadence; media backups have no retention policy; and roles and global objects are not in the dump. These are known trade-offs, not overlooked details.

The scheduler correction only addresses where time is spent; it is not a concurrency or performance optimization.

Governance does not cover all history either: at this round's observation point there were still old releases whose provenance could not be determined, and they were conservatively retained.

This note belongs to the round of work from 2026-09-22 to 2026-09-23. The production states in it are facts at this round's observation point, not the state at any later moment; subsequent production changes are outside the scope of this note.

Finally, it is not a promise about future versions, timelines, or capabilities. What is recorded here is an infrastructure upgrade that already happened, and the evidence supporting it.

Where the evidence is

Every factual statement in this note points to an object that can be looked up: the correspondence between commits, deployment release identifiers, and test case counts is registered in the facts table above and in this note's evidence entries. The body of this note only explains meaning and does not repeat the identifiers.