Skip to content

Not Yet Designed

The following require detailed design before or during implementation. Priority tiers reflect MVP blocking, launch blocking, and post-launch scope. (Source: §39.) When a design touches one of these, surface it.

§39.1 MVP-Blocking — Must Design Before Building Starbase

  1. Database schema — DESIGNED 2026-07-11 → Starbase › Database Schema (services are project-scoped definitions + service_environments instantiation; builds/deployments split with rollback-as-insert; leased job queue; billing raw snapshots + totals; clusters + minted tokens; soft deletes for the tombstone protocol). Remaining downstream: the desired-state serialization code (implementation-time per §32) and #67's session tables. Original scope: Postgres tables for users, workspaces, projects, environments, services, deployments, builds, clusters, billing, regions, var_groups, var_group_entries, service_var_groups, github_installations, webhook_deliveries (for dedup), buckets (object storage primitive — now post-MVP, §39.3 #16). RBAC tables (users, workspaces, workspace_members, projects, project_members, environments) defined in §15.7. Identity model (v1.9): there is no customers table — the entity hierarchy is workspace → project → environment → service. The canonical tenant key carried through desired state, snapshots, metrics, and logs is project_id + environment + service_id, with workspace_id as the billing-boundary label; the stale customer_id field was removed throughout. Desired state serialization code (internal/service/desiredstate.go) is downstream of this work. Environments are customer-named, validated as an RFC 1123 label (≤30 chars, §20.2); the environments table carries an is_ephemeral flag so preview/ephemeral environments are identified structurally, not by name-matching — this is what the per-route preview-metric retention treatment keys off (FR-067; previews are metriced like any env, short-retention post-MVP). Placement (2026-07-04): cluster_id lives on the service-per-environment record (and on database records), not on projects — projects carry only a default_region (FR-077). Regions (2026-07-11): the regions table's columns (id, cloud, status, transport, endpoints, vpc_cidr) are defined at Region Onboarding; cluster registration + token minting ride the same operator admin surface.
  2. Starbase API endpoints — DESIGNED 2026-07-11 → Starbase › Dashboard API (flat-by-UUID URLs; per-resource endpoint catalog with §15 permission mapping incl. the Developer* protected-env notation; deploy-and-rollback as one endpoint; 202+status async pattern; tombstone deletes; FR-065-filtered observability reads; cursor pagination; one error envelope). Remaining downstream: the OpenAPI spec (implementation-time) and the deferred surfaces noted inline (custom domains #27, buckets #16, API tokens #20, invoices #14)
  3. Customer onboarding flow — DESIGNED 2026-07-13 → Overview › First Deploy (signup → live-URL journey as a composition of designed pieces; default target production). Small section, to be refined. Original scope: Signup → create workspace → create project → connect Git → first deploy (end-to-end happy path)
  4. Deployments & rollbacks — DESIGNED 2026-07-11 → §16.12 rollout semantics (maxSurge 1 / maxUnavailable 0 zero-downtime; deploy-during-deploy supersedes; rollback = deploy an old build, pins nothing; rows kept forever + registry GC keeps deployed + last 10 images). Remaining: Stardeck's rollback button UX
  5. Health checks — DESIGNED 2026-07-11 → §20.3 mapping (single path + timeout input → startup/readiness/liveness; web default = TCP-connect readiness; workers none; FR-020 reworded). Traffic-shift = readiness-gated rolling update; nothing further at the Gateway
  6. Service types — DESIGNED 2026-07-13 — explicit type (web/worker) carried in the §25.1 payload; web renders a ClusterIP Service (+ HTTPRoute when public), worker renders neither and gets no probes (§20.2); a UI type change is a delete-and-rebuild of the type-specific resources with service_id preserved (Services › Networking). Cron re-tiered post-MVP on 2026-07-02 → §39.3 #48.
  7. Auto-generated hostnames — DESIGNED 2026-07-13 → Networking › Hostname generation (single-label, always-suffixed <name>[-<env>]-<rand4>.starform.app — Render/Railway/Vercel model: unique by construction, no squatting, no "name taken"; hostname UNIQUE backstop; editable via one DNSProvider update; single-label forced by Cloudflare Universal SSL). (Per-hostname routing was already canonical at DNS & Domains.)
  8. Horizontal autoscaling (HPA) — DESIGNED 2026-07-13 (UX = enable + min + max + Target CPU %, memory % advanced; bounds 1 ≤ min ≤ max ≤ plan-quota-cap + 30 ≤ target ≤ 90; explicit scaleDown.stabilizationWindowSeconds: 300 — §20.3, Stardeck). Per-service scaling rules, min/max replica limits, scale-up/scale-down cooldown. MVP scope: CPU/memory only, driven by the Kubernetes Metrics API (metrics-server — self-installed on Talos at bootstrap, §26.3). Scaling on RPS/latency is deferred — custom-metric HPA would need an adapter exposing the regional ClickHouse telemetry (the L7 request rows) through the custom-metrics API, which has no off-the-shelf path today (v2 Read & query). Note that FR-050's customer-facing CPU/memory display comes from the cAdvisor → OTel Collector → ClickHouse path and is separate from HPA's metrics-server source — do not conflate the two. Separate from cluster autoscaling (already designed) and scale-to-zero (post-MVP). The §25.1 autoscaling payload block and the HPA replicas-ownership rule (§20.3) are specified; scaling-rule UX, cooldowns, and bounds validation are now designed (marker above).
  9. Internal / private networking — DESIGNED 2026-07-13 → Services › Networking — public flag (web only: false = internal, no HTTPRoute/DNS); a stable internal address http://<service-name>-<environment>:<port> shown to the customer (short form, never the .svc.cluster.local FQDN); Connections auto-inject a linked service's URL per environment (service_connections, same path as DB-credential injection); cross-service ${{…}} reference variables deferred. Cross-env isolation stays §20.4.
  10. Live log streaming UX — WebSocket or SSE streaming from ClickHouse to dashboard, CLI logs -f support, filtering by level/service/time, search
  11. Build overrides — DESIGNED 2026-07-11 → §16.5 build overrides (build_config: root_directory, dockerfile_path, build/start commands, and the pre-deploy command — a rollout-gating one-shot Job, FR-086, §20.3)
  12. Customer-facing metrics dashboard — Default dashboards per service (request rate, error rate, latency, CPU, memory), ability to build custom dashboards
  13. Stripe integration detail — DESIGNED 2026-08-01 → Billing › Stripe lifecycle (one Stripe Customer per workspace; monthly cron → auto_advance invoice, US$0.50 floor rolls forward; Smart Retries dunning → suspended via desired state, auto-cleared on payment; next-cycle plan changes, no proration machinery; manual refunds; stripe_events dedup; invoices endpoint). The billing math landed with it — Billing › Calculation, FR-087. Original scope: Invoice lifecycle, payment failure handling, dunning flow, refunds, proration edge cases
  14. Build pipeline error handling — DESIGNED 2026-07-11 → §16.12 rollout & error semantics (45-min build timeout; deterministic failures never auto-retry, infra failures ride the §14 job lease; no hard image-size limit, warn >5 GiB; Spaces log-upload failure non-fatal)
  15. Per-environment branch configuration — (resolved in schema 2026-07-11: branch + auto_deploy columns on the service-environment record) Explicit mapping from Git branch to environment (e.g., main → production, develop → staging); default auto-deploy on push, opt-out per service-environment. Remaining: the dashboard UX for editing the mapping.
  16. Encryption-at-rest catalog — Cross-cutting spec covering all encrypted fields in Postgres: Var Group entry values (§38.2), DB credentials, bucket credentials, GitHub installation tokens, and cloud-provider credentials/kubeconfigs Starform holds for its own clusters and future multi-cloud regions (§12). All use AES-256-GCM with a single key from ENCRYPTION_KEY env var; key rotation is a post-MVP batch job. Customer-cluster etcd (where Shuttle renders Var Group values into K8s Secrets) is a distinct at-rest layer: encrypted via Talos LUKS2 disk encryption (STATE+EPHEMERAL, AES-XTS-256) enabled in the bootstrap machine-config, not the app-level GCM key (added 2026-07-06).
  17. DB sell-pricing in the Financial Model (added 2026-07-03) — Customer prices for the DB tier ladder (Managed Databases) are finalized in the Financial Model before launch; the PRD records DO list costs only as vendor constraints (Depot/Tigris precedent). Includes validating the Mininova defaults (250m/512Mi, 1 GiB PVC) against real workloads.
  18. DB-pod telemetry attribution (added 2026-07-03; mechanism reworded to v2 2026-07-11) — Mininova DB pods carry the label set minus service-id plus starform.io/database-id (§24.1). Design the logs/metrics attribution for the project · environment · database identity: the k8sattributes extraction of database_id, store columns/queries for rows without service_id, and the v2 guide's sync — done together so the observability guide is updated once, not drifted.
  19. Dashboard auth & sessions — DESIGNED 2026-07-11 → Starbase › Auth & Sessions (SSO flow with verified-email auto-linking; skinny 15-min JWT + rotating server-side refresh tokens with family-revocation on reuse; httpOnly .starform.io cookies; sessions + sso_identities DDL). Extras (password reset, MFA) stay deferred (§39.3 #25) (re-tiered from #25, 2026-07-05)

§39.2 Pre-Launch-Blocking — Must Exist Before Public Launch

Pre-launch-blocking

These gate public launch (SOC 2 / enterprise expectations), even though they're not needed to build the MVP.

  1. Audit logs — Who did what and when, across all customer-facing operations. Required for SOC 2, expected by enterprise
  2. API tokens / service accounts — Scoped tokens for CI/CD integration (deploy from GitHub Actions without user credentials), per-machine tokens
  3. Status page / SLA / incident response — Public status page (status.starform.io), incident response runbooks, SLA commitments for uptime
  4. GDPR / account deletion / data retention — Right-to-be-forgotten workflow, account closure data cleanup, data export before deletion. Includes confirming Grafana Cloud platform-telemetry data residency — either an EU-residency stack or enforced exclusion of customer identifiers from the platform series (§35.5). Customer-facing residency posture: Region Onboarding · what lives where.
  5. Rate limiting — Per-customer API quotas (target 100 req/min/user), deploy rate limits, build queue limits, protection from runaway customer impact. Redis-backed token bucket implementation.
  6. Admission policy for label enforcement — Kyverno or custom admission webhook rejecting pods missing required starform.io/* labels. Prevents silent NetworkPolicy bypass (§20.4 failure mode).
  7. Abuse controls (added 2026-07-05) — the public untrusted tier's real defense against resource abuse (crypto-mining): a valid-payment gate, per-pod CPU/memory limits, and mining/abuse detection (sustained-CPU + known-pool egress signatures) with fast automated suspension. Distinct from sandboxing — gVisor does not address abuse (FR-084).

  8. Customer-cluster etcd DR (added 2026-07-05) — scheduled in-cluster etcd snapshots → DO Spaces (platform object storage — not Tigris; corrected 2026-07-09) + a rehearsed talosctl bootstrap --recover-from restore. Softened by desired state (a lost cluster rebuilds from Starbase), but still required before real customers (Kubernetes Runtime).

  9. Talos machine logs → platform ClickHouse (added 2026-07-05) — ship node system logs (kubelet, machined; Talos has no journald) via machine.logging.destinations (TCP json_lines, extraTags: cluster_id) → the regional OTel Collector gateway (tcplog receiver) → a platform_logs ClickHouse table with TTL. System-log retention for SOC 2; never Grafana Cloud — log bodies carry customer identifiers that can't be label-dropped (§35.5; spec 2026-07-05-substrate-observability-design.md)

  10. Telemetry-droplet health (added 2026-07-05) — make FR-069's telemetry-droplet watching real: a static Alloy scrape job over the VPC to ClickHouse's native /metrics endpoint and the OTel gateway's own telemetry, or one Alloy unit on the droplet (embedded node-exporter); paging keep-list → Grafana Cloud (§35.5)
  11. Manual fleet-upgrade runbook (added 2026-07-05; carved from #36) — written and rehearsed-once procedures for every layer Starform now owns: Talos OS (the gVisor extension rides it), Kubernetes, Cilium, Envoy Gateway (re-verify the §20.2 access-log CEL path before any bump), Shuttle version rollout, telemetry-droplet patching (ClickHouse + the OTel gateway; pin the Collector, stage upgrades), DOKS maintenance windows. Rehearsed on a throwaway cluster before launch — the first CVE with customers aboard is not the time to improvise
  12. K8s Secret encryption in etcd (secretbox) (added 2026-07-06) — enable Talos cluster.secretboxEncryptionSecret so the kube-apiserver encrypts Secret objects within customer-cluster etcd — defense-in-depth beyond the MVP LUKS2 disk encryption (§39.1 #18); a live etcd read/dump then returns ciphertext. Key rotation rides §39.3 #41. SOC 2 hardening
  13. Public Suffix List entry for starform.app (added 2026-07-11) — submit the private-section entry (mandatory-template PR + a permanently-maintained _psl TXT record + rationale) so each customer subdomain becomes its own registrable domain — tenant-to-tenant cookie isolation (DNS & Domains). Free but volunteer-reviewed (weeks–months) plus browser-propagation lag after merge, and effectively one-way — submit during private beta, months before public launch. Until propagation, tenant cookie isolation on starform.app does not exist (acceptable only for a small closed beta)

§39.3 Post-MVP / Seed-Stage

Post-MVP / seed-stage

Deferred beyond MVP. Preview environments (#28) are Decision pending — ship at launch (Railway parity) or defer to seed.

  1. Volumes / persistent storage — (re-tiered post-MVP 2026-07-05; MVP state = managed DBs) DO Block Storage CSI configuration for customer volumes (the CSI driver is already installed for Mininova PVCs), attach/resize/snapshot workflows, survives pod restart/node failure
  2. Managed object storage primitive (Tigris) — (re-tiered post-MVP 2026-07-09 — was MVP-blocking; parity note: Railway and Render ship no S3 primitive either) S3-compatible buckets provisioned per environment via the Tigris Partner Integration API. One Tigris Organization per Starform workspace (isolated tenants). Credential injection into services via auto-generated system Secret (endpoint, access key ID, secret access key). Public vs. private visibility flag. Credential display in dashboard after provisioning. Zero egress fees pass through to customer. The StorageProvider port (§10) stays defined-but-dormant. Action moves with it: negotiate partner pricing and SLA with Tigris sales (help@tigrisdata.com) before this ships. Tigris is customer-facing only — platform object storage is DO Spaces (etcd snapshots #59, build-log archive §16.6, throwaway-DB dumps #51)
  3. Auth extras — (narrowed 2026-07-05; the core dashboard session/SSO design is MVP-blocking → §39.1 #67) password reset (N/A while login is SSO-only), MFA, additional SSO providers
  4. Cluster bootstrap workflow — (Talos, reworded 2026-07-05; MVP = manual/one-off talosctl per Kubernetes Runtime — this item is the automated, repeatable workflow) Step-by-step: droplets + cloud-init Talos machine config (gVisor extension) → etcd bootstrap → Cilium → DO CCM/CSI → metrics-server + kube-state-metrics → Envoy Gateway → OTel Collector + Grafana Alloy → Shuttle → active. Plus the out-of-band per-region provisioning (Terraform/cloud-init) of the ClickHouse telemetry droplet (with its OTel gateway) and the ops/jump droplet (later Shipyard) (§4.1, §26.3). The region-level wrapper (VPC → telemetry → registry → cluster → registration → go-live) is Region Onboarding
  5. Custom domains — (decided 2026-07-05: Cloudflare for SaaS, no cert-manager — spec 2026-07-05-review-batch-design.md) custom-hostname lifecycle + DNS validation via the Cloudflare API, HTTPRoute lifecycle; Cloudflare issues and renews the edge certs (100 hostnames free, then $0.10/mo each) and customer domains stay behind the WAF
  6. Preview environments — Ephemeral environments per PR, full replica of production with Git-branch-based lifecycle, flagged is_ephemeral on the environments table (§39.1 #1). Per-route Envoy L7 metric handling resolved 2026-07-06: previews are metriced like any env (competitor parity), not suppressed (the parse-only path can't drop ephemeral routes without forbidden name-matching, FR-067); short-retention-for-ephemeral is a post-MVP refinement (FR-051). Strategic decision remaining: ship at launch (Railway parity) or defer to seed stage
  7. Scale-to-zero — Free tier only, HPAScaleToZero (K8s 1.36), custom activator for request buffering
  8. Auth primitive — JWKS endpoint hosting, SecurityPolicy lifecycle, claim-to-header mapping UI
  9. Email primitive — AWS SES integration, per-customer domain management, bounce handling
  10. Queue primitive — NATS as customer-facing managed service
  11. CLI tool — starform deploy, starform logs, starform status, device code auth flow
  12. Platform-side disaster recovery — Control plane Postgres backup/restore, registry compromise response, runbooks. Includes the desired-state sequence restore path (§25.1): the monotonic counter lives in the control-plane Postgres, so a naive restore re-issues lower sequences and freezes every Shuttle. On recovery Starbase re-derives each cluster's counter from the fleet's reported sequence_observed (§25.4) and resumes above it, plus an operator-authenticated per-cluster sequence_reseed break-glass for clusters that are also lost
  13. Multi-region failover — Region drain workflow, customer migration between regions, SLA commitments
  14. Automated platform upgrades — (narrowed 2026-07-05) fleet-wide rollout automation via Shipyard, canary/staged strategies; the manual procedures are pre-launch (§39.2 #66)
  15. Capacity planning & quota enforcement — Cluster-level quota (1000 pods soft limit), auto-provision next cluster at threshold, signup throttling
  16. External secret store integration — AWS Secrets Manager, HashiCorp Vault as customer-facing secret-store backends (lets a customer source Var Group values from their own vault). Post-MVP. (Not BYOC, which is out of scope, §37.)
  17. Per-environment value overrides in Var Groups — currently requires separate groups per environment
  18. Placement-migration workflow — (reworded 2026-07-04) Moving a service/environment between clusters or regions (placement is per service-per-environment, FR-077) requires coordinated source drain + target provision plus an image copy to the target region's registry and the service's DNS-record move to the target region's LB (DNS & Domains); env-granular moves (staging first, then prod) are the intended path; not in MVP scope
  19. Encryption key rotation — Batch job to re-encrypt all encrypted fields with new key without downtime
  20. Self-hosted platform monitoring — migrate platform self-monitoring off Grafana Cloud (MVP-only, §35.5) to a self-hosted stack once the SRE team can operate it; path: shrink the Grafana Cloud keep-list toward the per-cluster liveness series, stand up self-hosted evaluation/paging against the #65 store, then cancel. Re-evaluate cost vs. the external-watcher benefit
  21. Mass Relay — egress-only telemetry query channel — (reshaped 2026-07-09; was the optional cross-region read-API/cache — Mass Relay is that read-API, connection-inverted) a per-region agent (repo starform/massrelay, §1) on the telemetry droplet that dials out to Starbase and holds a persistent multiplexed gRPC stream: Starbase sends already-FR-065-filtered queries down, Mass Relay executes them against localhost ClickHouse as starbase_read and streams rows back; the FR-051 retention upserts ride the same channel. It extends the outbound-pull inversion (Shuttle / Shipyard, FR-085) to the telemetry plane wherever no private path exists. Transport is a per-region choice behind the QueryTransport seam (§13), settled 2026-07-10: DO regions keep the direct read over VPC peering permanently (intra-region $0, cross-region $0.01/GiB on the private backbone, zero moving parts — in a healthy peered region the agent adds ~nothing, and a rare peering outage degrading dashboard reads is within the §4.1 accepted risk class); Mass Relay is the transport for regions on clouds without DO peering — its build trigger is the first non-DO region; DO-only expansion never needs it — and the emergency transport for a peered region during a prolonged peering outage (a droplet process + a token, deployable in minutes; flip the region's seam). Where it is primary it runs two replicas / two streams. The #46 non-overlap constraint stays for peered (DO) regional VPCs. Trust domain: a read-only ClickHouse credential + a Starbase token — deliberately not Shipyard (talosconfig) and not Shuttle (K8s creds). Design when built: stream protocol (multiplexing, backpressure, reconnect), the Starbase-side terminator. Design record: specs/2026-07-09-egress-only-telemetry-and-scope-cuts-design.md
  22. Multi-region telemetry-gateway placement — (reworded to v2 2026-07-11 — was the Vector aggregator) at MVP (single region) the regional OTel Collector gateway runs on the telemetry droplet next to ClickHouse; for added regions, decide whether each region's gateway stays on the droplet or moves to a small regional platform cluster
  23. Multi-region control plane — the control plane is central by decision (Railway's model; one Starbase + one Postgres) and a single-region SPOF at MVP (§4.1). Running workloads survive an outage via Shuttle's level-driven loop (§32), but new deploys + dashboards stop. Distributing it (Fly's model) needs a distributed/replicated DB — out of near-term scope. Design control-plane HA before relying on multi-region
  24. IPAM / subnet allocation registry — (resolved & absorbed 2026-07-11 → Region Onboarding · stage 0) no registry system exists or is needed: the regional /16s (mutually non-overlapping — they all peer the same control-plane VPC) are picked as one Terraform-vars line per region, reviewed in the region's infra PR. Pod/service CIDRs remain cluster-local overlays (§4.4)
  25. Account-quota runbook — (reframed 2026-07-03) DO soft-quotas (Droplets 25, Managed DB clusters 10, K8s clusters gated by Droplets) are administrative numbers lifted through the DO reseller partnership (agreed 2026-07-03), not architectural limits. Keep the operational runbook: track per-region Droplet/DB-cluster/K8s counts and request raises ahead of growth through the partnership channel
  26. Cron services — K8s CronJob rendering in Shuttle, schedule UX, batch RBAC addition to §22, service-type=cron handling (§24.2). Post-MVP per the 2026-07-02 decision; carved out of §39.1 #6. The entity model keeps cron as a service type throughout
  27. DB promote-to-dedicated — the Mininova→Micronova boundary migration (dump/restore job, surfaced as a tier upgrade — the natural upgrade funnel). Blocked at MVP: crossing the boundary means "create new instance, move your data" (Managed Databases)
  28. Managed-DB HA standby toggle — DO supports +1/+2 standby nodes per cluster; MVP is single-node at every managed tier. Design the toggle (pricing via Financial Model, failover expectations, desired-state field)
  29. Best-effort throwaway-DB backups — nightly pg_dump of Mininova Postgres instances to DO Spaces (platform object storage — not Tigris; corrected 2026-07-09); converts "throwaway" into "best-effort" without making promises. Cheap goodwill; explicitly not a durability SLA
  30. Shared multiplexed managed DB pools — rejected for MVP (out-of-band SQL provisioning side-channel, password-only isolation, per-region fixed floor, pool-wide blast radius; see the 2026-07-03 spec). Revisit only if Mininova data-loss churn materializes as a retention problem
  31. Long-polling on the desired-state GET — ?wait=25s on §25.1 (server answers early on change) for near-instant propagation while staying outbound-only; interim step before the gRPC upgrade (§31 #10). MVP ships the conditional GET only
  32. Cross-region private networking overlay — a Railway-class private network spanning regions. Explicitly not offered at MVP (FR-078, Render's model: public hostnames across regions); evaluate only if customer demand materializes post-expansion — it is a service-mesh-scale investment
  33. Per-region registry quota — DO multi-registry allows 10 registries per team (Professional); raise via the reseller partnership before region count approaches it; add to the #47 runbook
  34. Shipyard — machine-lifecycle agent (added 2026-07-05) — design/build the per-region agent (hardened in-VPC droplet, outbound-pull from Starbase, holds regional talosconfig) that runs recurring control-plane day-2 ops (OS/K8s upgrade, CA rotation, etcd-snapshot scheduling, bootstrap, DR restore) for every cluster in its region. MVP = manual talosctl (Shipyard)
  35. gVisor performance / per-tier sandbox (added 2026-07-05) — measure gVisor I/O overhead on customer workloads; decide whether any tier opts out of the sandbox
  36. Regional platform metrics store (added 2026-07-05) — a second, platform-scoped metrics store on the telemetry droplet for deep substrate history (full etcd/apiserver/Cilium/node series; $0 intra-VPC ingest); build when an incident proves the need — Alloy already supports the split (chained prometheus.relabel → second remote_write). With v2 this would add the stack's only PromQL store (a small VictoriaMetrics) — weigh ClickHouse tables instead when triggered. Companion to #42 (§35.5)
  37. Reconcile the docs to Observability v2 — RECONCILED 2026-07-11 — Observability v2 (single-store ClickHouse) is now the design of record everywhere: topology SVGs (home + system-topology), §4.1/§4.2, §0.2, §6, the observability FRs (FR-050/063/064/066/067/069/071/080; FR-049/051 were already v2), §20.2 (the envoy_cluster_name parse retired for the access-log CEL read), §24/§24B consumers, §26, Security & Isolation, Stardeck, glossary, canonical-sources (§35.1–35.4 marked superseded with v2 homes), and CLAUDE.md §2/§8. The v1 chapters remain archived under Observability as superseded. Design record: specs/2026-07-07-single-store-clickhouse-telemetry-design.md
  38. Internal SRE observability tool (added 2026-07-07) — separable from the customer pipeline: decide whether Starform's own SRE observability runs a self-hosted ClickStack/SigNoz (natural now that ClickHouse is the telemetry store) vs staying on Grafana Cloud (§35.5, MVP). Companion to #42
  39. ClickHouse billing audit trail + delivery-gap reconciliation (added 2026-07-09; re-tiered out of MVP — §36 keeps raw snapshot rows in Postgres at MVP: snapshot_id PK, bounded ~60–90 d retention, dispute record) — when snapshot volume outgrows the Postgres window, the immutable audit copy is written in-region by Shuttle via the regional OTel gateway (per-cluster token; the gateway stays the sole ClickHouse writer; its on-disk queue rides out ClickHouse outages) — not by the central API. Independent write paths upgrade the nightly PG↔CH reconciliation to detect delivery gaps (a batch that never reached Starbase), not just dual-write bugs — "in ClickHouse, not in Postgres" is a delivery alarm, not noise. Design record: specs/2026-07-09-egress-only-telemetry-and-scope-cuts-design.md
  40. Non-DO region readiness spikes (added 2026-07-11) — verify with the first non-DO region, not before: the Talos image-import pipeline on the target cloud (Image Factory nocloud image → custom-image import; metadata/user-data path unverified), that cloud's CCM + CSI on self-managed Talos, Cloudflare per-zone DNS record limits vs projected public-service count, and AZ handling on AZ-having clouds (node/etcd spread across zones, per-AZ subnets, LB cross-AZ targets — DO has no AZ concept) (Region Onboarding · non-DO deltas)
  41. Region resilience tiers (seeded 2026-07-11) — a DO region is a single failure domain (one datacenter, no AZs) — accepted at MVP; verified posture matches the own-infra camp (Fly: host-pinned Machines, "run two + multi-region"; Railway Metal), not Heroku's 3-AZ Private Spaces. Revisit on Enterprise demand: metro region pairs (NYC1+NYC3 as two ordinary peered regions — Region Onboarding handles it unchanged), cross-region managed-DB read replicas (DO supports for PG) as the data-DR offering, and AZ-spread clusters on AZ-having clouds (#73). Never imply AZ redundancy in the SLA (§39.2 #21)
  42. One-click templates (added 2026-07-13) — deploy a curated app (Ghost/n8n/Strapi) in one click — Railway's biggest acquisition loop; no coverage before now. The schema substrate landed early (services.source_type / image_ref, 2026-07-13): a source_type: image service skips the build entirely (its image_ref → desired-state image, Depot untouched). Remaining design: the template manifest (services + DBs + var groups + pre-deploy), the repo-copy path for git templates (the build pipeline assumes a user-owned repo — builds.commit_sha is NOT NULL), auto-generated secret entries (Var Groups are static today), and the catalog UI. Ship-at-launch vs seed is the same decision shape as previews (#28).