Shuttle Failure Modes
| Failure |
MVP Behavior |
| Shuttle pod crashes |
Kubernetes restarts it. Cache rebuilds. ~10s downtime. Customer workloads unaffected. |
| Starbase unreachable |
Snapshots queue in the bounded replay buffer (default 60 batches) and replay on reconnect — idempotent by snapshot_id (FR-081). Capacity retries 3x then drop (stateless). Desired State Runnable retries on next tick. Workloads stay running. |
| K8s API unreachable |
Informer's built-in resync handles reconnection. Brief gap in snapshots. |
Desired-state sequence regression |
Shuttle refuses the payload (stale-read / split-brain guard, §25.1), keeps the last-applied state, and increments starform_desired_state_regressions_total. A fleet-wide regression = a control-plane Postgres restore, not a stale read: on recovery Starbase re-derives each cluster's counter from the reported sequence_observed (§25.4) and resumes above it, with an operator sequence_reseed for clusters it cannot reach (§39.3 #34). |
| Tombstone flood (delete breaker) |
Deletions beyond max(10, 20% of managed services) in one tick are skipped wholesale; creates/updates still apply; starform_gc_breaker_tripped pages the SRE; operator override via the payload's force_gc flag (§20.3, §25.1). |
Empty services array while the cache holds managed resources |
Safe by construction (no tombstones ⇒ nothing deletes); Shuttle applies nothing and reports the anomaly via §25.4. |
| Status POST fails |
Retry 3× then drop; status is derived from the Informer cache, so the next 10s tick regenerates it — no loss (§25.4). |
| Pod stuck in terminating |
Runnable logs warning, does not force delete in MVP. |
Cross-references
Retry/loss semantics for snapshots → §19.4 · the
level-driven loop that recovers from drift → §19.3 · the
silent-staleness alert that catches a frozen Informer →
§26.1 · resilience of running workloads during a control-plane
outage → §32.