Document History¶
The living decision log (the successor to §41). Read it to understand why a decision was made before changing it. Versions track the legacy single-file PRD through v1.13, after which the docs were migrated to this MkDocs site. (Source: §41.)
Versioning after the migration
The single-file PRD's Version: header ended at v1.13. The documentation is now this docs/
site; new changes are recorded here (and in git history), not in a single file header. The legacy
PRD is preserved at _legacy/master_prd.md.
| Version | Date | Changes |
|---|---|---|
| 1.0 | April 2026 | Initial architecture finalized. Shuttle PRD complete. Starbase architecture defined. Gateway decision (Envoy Gateway) locked. Naming system established. |
| 1.1 | April 2026 | Added §15 (RBAC & Permissions Model) with workspace roles (Owner/Admin/Billing/Member), project roles (Admin/Developer/Viewer), environment protection flag, and database schema. Renumbered subsequent sections. Frontend named Stardeck (Ship's command deck). |
| 1.2 | April 2026 | Rewrote §16 (Starforge): MVP adapter switched from Google Cloud Build + Nixpacks to Depot SaaS with Dockerfile frontend. Added decision rationale, per-customer project isolation model, log streaming to ClickHouse, known limitations, and migration triggers for future self-hosted BuildKit. Updated Tech Stack Summary, External Services, and ports detail table to reflect Depot as MVP build backend. |
| 1.3 | April 2026 | Moved Railpack from post-MVP to primary build frontend at MVP, with Dockerfile as explicit fallback. Rationale: requiring a Dockerfile at launch was a visible UX regression vs Railway, Render, Heroku, Vercel. Railpack (MIT, Railway-maintained, BuildKit LLB-native) integrates as a Go library in Starbase Worker and submits LLB directly to Depot's BuildKitService. Adds ~2–3 weeks to MVP scope; removes the largest UX gap vs competitors. Updated §16.5, 16.8 limitations table, Tech Stack Summary, External Services row, adapter layout, and ports detail table. |
| 1.4 | April 2026 | Corrected Depot pricing throughout §16. Previous versions stated Depot Docker build rate as $0.004/min — actual public rate is $0.04/min (10× higher). Build-minute sell price set to $0.05/min (~20% margin over cost at list rates). Added explicit note that Business-plan negotiation is required before launch to improve margins. Added Depot cache storage ($0.20/GB/mo) as tracked infrastructure overhead. Revised migration triggers with concrete Depot-spend thresholds. Financial Model v2.1 regenerated with matching corrections. |
| 1.5 | April 2026 | Added §38 (Var Groups): Render-style environment groups as the primitive for env vars and secrets. Separate K8s Secret per Var Group with attach_order precedence, env-scoped via annotations, conflicts allowed with UI warning. Added paragraph to §32 pinning desired state computation code location (internal/service/desiredstate.go) and clarifying contract-vs-implementation split. Rewrote §39 (Not Yet Designed) into three priority tiers with ~35 items. Auth system and cluster bootstrap moved to post-MVP. Added MVP-blocking items: deployments & rollbacks, health checks, service types, volumes, wildcard domains, HPA, internal networking, live log streaming, build overrides, customer metrics dashboard. Added pre-launch-blocking items: audit logs, API tokens, status page/SLA, GDPR, rate limiting. Renumbered old §38 to 39 and old §39 to 40. |
| 1.6 | April 2026 | Namespace model change. Switched from namespace-per-customer to namespace-per-project with environments as first-class labels (§20). Rationale: project-level cluster affinity (all environments of a project schedule to the same cluster), operational manageability at scale, and alignment with scheduling-unit = isolation-unit principle. NetworkPolicies now use label selectors to enforce cross-environment isolation within a shared namespace. Labels expanded (§24): added starform.io/environment, starform.io/workspace-id, starform.io/service-name, starform.io/service-type, starform.io/observability-enabled, plus the standard K8s recommended label set (app.kubernetes.io/*). New §24B (Annotations): comprehensive annotation catalog organized by resource type — provenance, rollout triggers (with Var Group checksum spec), DO CCM, Prometheus scrape, cert-manager, HTTPRoute, namespace-level. §38.7 updated with concrete checksum computation. §16 additions: §16.11 (Git integration — webhook signature verification, deduplication via webhook_deliveries table, 5-second push debouncing, GitHub App lifecycle events, stale installation detection, rate limiting), §16.12 (build lifecycle UX — state machine with cancellation, 4-step progress visibility, post-deploy credential display). §39 additions: managed object storage primitive (MVP), per-environment branch config, encryption-at-rest catalog, admission policy for label enforcement, project-to-cluster migration workflow, encryption key rotation. New §40 (Functional Requirements & Success Criteria): 62 FRs and 15 SCs in normative FR-###/SC-### format for test mapping and due diligence. Document History renumbered to 41. |
| 1.7 | April 2026 | Label reclassification (§24). Split identity labels into two subsections: 24.1 Load-Bearing (8 labels required for correctness — system breaks if missing) and 24.2 Operational (3 labels including tier, honestly described as denormalized convenience for kubectl/dashboard/analytics — not billing-critical). Added explicit billing note: labels are not used for billing math; Starbase computes per-pod cost from PodSpec resource requests. The tier label was previously framed as billing-critical — that framing was wrong and has been corrected. Namespace Labels renumbered from 24.3 to 24.4. Object storage provider change. Switched from DO Spaces to Tigris via Partner Integration API. Rationale: Tigris has a purpose-built partner API for platforms (one API call creates an isolated tenant organization per customer), zero egress fees (strengthens Starform's egress-differentiation story), S3-compatible, globally distributed by default (no CDN integration needed). Updated §4.3 External Services, §6 Tech Stack Summary, §9 Package Layout (new adapter/tigris/ folder, removed storage.go from DO adapter), §10 Ports Detail (StorageProvider description updated), §11 Adapter Wiring example, §37 BYOC table (with note that Tigris is cloud-agnostic and can remain in BYOC deployments), and §39.1 item 16 (MVP object storage primitive now references Tigris with explicit action item to negotiate partner pricing and SLA before launch). No pricing numbers added to PRD pending confirmed partner terms. |
| 1.9 | June 2026 | Monitoring architecture finalized end-to-end. HTTPRoute naming pinned (§20.2): route name = <project_uuid><service_uuid>-<environment> with hyphen-stripped 32-char UUIDs and a positional parse, because Envoy's per-route metrics expose customer identity only via the cluster-name string and environments share a namespace. This supersedes the earlier "recover project_id via a kube-state-metrics namespace-label join" approach — folding all three IDs into the route name removes the cross-metric join and the kube-state-metrics dependency for Envoy attribution. Metrics pipeline rewritten (§35.2): three sources documented — Envoy Gateway (latency, RPS, throughput, error rate), kubelet/cAdvisor (CPU, memory, network), and kube-state-metrics — closing the CPU/memory collection gap. Three attribution mechanisms (cAdvisor joined on (namespace, pod) to kube_pod_labels for resource metrics — making kube-state-metrics load-bearing; envoy_cluster_name parsing for Envoy L7 metrics; direct pod-label promotion only for optional app /metrics), with the two customer-facing paths emitting identical label keys. Mandatory Envoy cardinality discipline added (stat-inclusion matcher, custom histogram buckets, no per-route metrics for preview envs). New §35.4 (Telemetry Transport): VPC peering ($0 egress), vmagent remote_write → vmauth, Vector agent → regional aggregator → ClickHouse, per-cluster bearer auth — the layer prior versions understated under "autonomous, zero shipping code." New §35.5 (Platform Self-Monitoring): self-hosted vmalert + Alertmanager against a separate internal VictoriaMetrics watching Starbase, Postgres, ClickHouse, VM, Envoy, LBs, and Shuttle; one vmagent routing to two stores by per-URL relabeling. Tenant-key reconciliation: removed the stale customer_id field throughout (§19.3, 25.1, 25.2, 26.2, 35.1, 35.2); canonical identity is project_id + environment + service_id, with workspace_id as the billing-boundary label — §24's catalog was already correct; the prose lagged it. Bootstrap (§26.3) + Cluster topology (§4.2): added metrics-server and kube-state-metrics; metrics-server is not bundled on DOKS and must be installed (HPA/kubectl top depend on it). HPA (§39.1 #9): MVP scales on CPU/memory via metrics-server only; RPS/latency scaling deferred (needs prometheus-adapter); clarified FR-050's display path is cAdvisor→VM, distinct from HPA's source. Retention (FR-051): split — logs via per-tier ClickHouse partition TTL; metrics global 90d on single-node VM at MVP, per-tier deferred. Distributed tracing confirmed out of MVP scope. Structure & readability: added Part 0 (Orientation) — system primer, end-to-end deploy walkthrough, a canonical-source map (one home per concept), and an MVP-scope-at-a-glance table. Softened the status line (was "Architecture Finalized" despite ~40 open items in §39). Synced the §2 diagram to the six-component customer cluster. BYOC reframed as out of scope (not offered) across §12, §17, §37, and §39 — content retained as a portability proof, not deleted; clarified the distinction from Starform-operated multi-cloud regions, which remain on the geographic-expansion roadmap. No content removed in this version. Consistency sweep: propagated the monitoring decisions into the summary tables and requirements — §4.1 (internal VictoriaMetrics, vmalert, Alertmanager, Vector aggregator added; customer-vs-internal store distinguished), the §2 infra diagram (platform-monitoring + aggregator row), §6 Tech Stack (metric sources, metrics-server, platform alerting, telemetry transport rows), and §40 (new FR-063–FR-069 for L7 metrics, tenant key, query isolation, private transport, Envoy cardinality, metrics-server, and platform self-monitoring; new SC-016–SC-018 for tenant isolation, alert latency, and metric freshness). Existing FR/SC numbering preserved (additions only). Identity propagation: carried the tenant-tuple decision through the desired-state contract (§32 — new "Identity in the payload" note; environments added to the computation SELECT), the §9 store layout (added environment.go, var_group.go, bucket.go), and the §39.1 #1 schema item (added the environments table and an explicit identity-model note: no customers table; canonical key project_id + environment + service_id; workspace_id as billing label). |
| 1.10 | June 2026 | Monitoring topology revised for MVP; observability split into a self-contained SRE guide. Platform self-monitoring → Grafana Cloud (§35.5, FR-069): retired the self-hosted internal VictoriaMetrics + vmalert + Alertmanager for Grafana Cloud (hosted metrics + Grafana Alerting + OnCall) — an external watcher survives infra failure; MVP-only, revisit post-MVP (§39.3 #42). Net: per region VictoriaMetrics drops from two instances (customer + internal) to one (customer); ClickHouse unchanged. GDPR guardrail: customer identifiers excluded from platform series, or an EU-residency stack (§39.2 #22). Central control-plane region (§4.1): Starbase + Stardeck + Postgres run in one region at MVP; telemetry stores are per region. The only cross-region traffic is the desired-state pull and Starbase's dashboard reads, routed per project's region via a region→store-endpoint lookup (no fan-out, §35.4); single-region control plane is an accepted SPOF — workloads survive via Shuttle's level-driven loop (§39.3 #45). Telemetry stores on VMs (§4.1, §26.3): ClickHouse and VictoriaMetrics move from in-cluster to dedicated per-region DO VM droplets (separate droplet per store), provisioned out-of-band via Terraform/cloud-init — consistent with DO Managed Postgres/Valkey already off-cluster; backups via volume snapshots. Log pipeline → hybrid Fluent Bit → Vector (§35.1, §35.4): customer-cluster agent switched from Vector to Fluent Bit (~64Mi/node, lighter on tenant nodes, less kube-apiserver load at high namespace counts); the regional Vector aggregator keeps the native batching clickhouse sink (avoids "too many parts"), receiving via its fluent source. Arbitrary environment names (§20.2, §24.1, FR-070): no fixed dev/staging/prod enum — names are customer-chosen, validated as an RFC 1123 label (≤30 chars) because the name is load-bearing in the HTTPRoute parse and K8s labels. Preview/ephemeral environments are identified by a structural is_ephemeral flag, not name-matching — what the per-route preview-metric suppression keys off (FR-067, §39.1 #1). New open items (§39.3): self-hosted-monitoring migration (#42), cross-region query-proxy/cache (#43), multi-region aggregator placement (#44), multi-region control plane (#45). Companion doc: the Observability Architecture guide (architecture/observability.html) is rewritten as a self-contained SRE implementation guide — the SRE does not receive this PRD, so the guide inlines the observability-relevant contracts (route parse §20.2, label catalog §24, tenant key §24.1, query filter FR-065) with provenance tags ("owned by PRD §X; PRD wins on conflict") and a sync checklist, plus worked config examples per chapter. A scoped exception to the "companion docs never duplicate" rule (CLAUDE.md §6). Consistency sweep across §2 (topology diagram), §4 (control-plane region + regional telemetry tier + Fluent Bit customer cluster), §6 (tech stack), §40 (FR-067/069 amended, FR-070 added; SC-017 → Grafana). |
| 1.11 | June 2026 | Cross-region read path, VPC/IP topology, and versionless rename. Control plane stays central (§4.1, §39.3 #45): one Starbase + one Postgres (Railway's model) — distributing it (Fly's model) needs a distributed DB / Corrosion-class build; out of near-term scope, SPOF accepted for MVP. Cross-region telemetry reads (§35.4, FR-071): the central control plane reaches each region's VPC-private stores over private cross-region VPC peering, through the existing authenticated front-doors — vmauth (VictoriaMetrics) + a read-only ClickHouse user — injecting the FR-065 filter itself. No new component; nothing internet-facing. The regional read-API/cache (§39.3 #43) is demoted to an optional post-MVP latency optimization. VPC & IP topology (§4.4): clusters in a region share one /16 VPC (tenant isolation is namespace/NetworkPolicy, not VPC) → peerings track regions (~12 ≪ 50/account), not clusters. VPC-native DOKS (Cilium 1.31+) because Starform resells DO Managed DBs — customer pods reach their DB directly (DB sees the pod IP; isolated via Trusted Sources scoped to the pod subnet). IP plan: per-region node /16 + per-cluster pod /18 + service /22 from a systematic 10/8 scheme, non-overlapping, in an IPAM registry (§39.3 #46) — not resizable after creation. DO viability stress-tested: every limit examined (peering 50, VPC /16, NIC bandwidth 2–25 Gbps, cluster/DB counts) is either huge headroom or a support-raised soft-quota (#47) — DO is a sound launch-and-scale foundation, with ports/adapters portability as the hedge. Rename: the master PRD is now versionless (master_prd.md); the header is the source of truth. Header → 1.11. Added FR-071; §39.3 #43 reframed, #45 updated, #46/#47 added. |
| 1.12 | June 2026 | Platform self-monitoring → dedicated Grafana Alloy agent. §35.5, §4.2, §6, §26.3, FR-069: the platform plane no longer rides a vmagent remote_write split. A dedicated Grafana Alloy agent (Grafana's k8s-monitoring Helm chart) runs in each customer cluster, scrapes Starform-component series (Shuttle, Envoy gateway-self, node-exporter, KSM platform objects), and remote_writes them to Grafana Cloud — an independent path so a vmagent fault can't blind the SRE to platform health, and the standard Grafana Cloud Kubernetes onboarding. vmagent simplifies to a single destination (the regional customer VictoriaMetrics); the GDPR labeldrop of customer identifiers moves from vmagent's relabel rules to Alloy's. Net: the customer cluster goes six → seven components — the accepted trade-off (raised and committed) for an independent external watcher. Verified: Grafana Agent reached EOL Nov 2025; Alloy is its supported OpenTelemetry-collector successor. Companion doc (architecture/observability.html): rebuilt Diagrams 0/1/2/3 + bootstrap — added the Alloy box, split the read front-door into its two real parts (vmauth proxy deployment + a native read-only ClickHouse user; chproxy ruled out as unnecessary at MVP), and gave the metrics vs logs pipelines distinct arrow colours (blue = metrics, amber = logs). Header → 1.12. |
| 1.13 | June 2026 | Fixed an Envoy stat-inclusion contradiction (§35.2). The cardinality matcher is a strict allowlist, so "keep only the three upstream_* customer families" silently dropped Envoy's gateway-self metrics (server.*, control_plane.*, listener_manager.*, cluster_manager.*) — the very metrics §35.2/§35.5 say must reach platform self-monitoring. §35.2 step 1 now allowlists both the per-route customer families and the bounded gateway-self families (per-instance, so no per-route cardinality cost; vmagent keeps the customer set → VictoriaMetrics, the Alloy agent keeps the gateway-self set → Grafana Cloud). Mirrored in architecture/observability.html ch.2 (EnvoyProxy example + step text + gotcha). Header → 1.13. |
| — | June 2026 | Migrated to MkDocs. The single-file PRD + the hand-built observability HTML were converted into this component-first MkDocs Material site under docs/ (the new source of truth); originals deprecated under _legacy/. Diagrams converted to theme-aware Mermaid; light + dark themes from the Starform frontend tokens; the §-cross-reference web preserved via stable #section-N-M anchors + the Canonical Sources index. |
| — | July 2026 | Closed the reconcile loop; made deletion safe; settled HPA/cron tiering. New §25.4 Shuttle→Starbase status report (fourth Status Runnable: 10s tick, POST-on-change + 5-min resend) carrying per-service rollout state, replica counts, failure reasons, observed image digest, tombstone acks, orphan reports, and breaker state — Starbase now derives deploying → deployed \| failed from it (FR-072, SC-019); §16.12 cancel semantics corrected to a desired-state revert. Deletion is tombstone-only (§20.3/§25.1): absence never deletes; orphans are reported, not deleted; a delete breaker (max(10, 20%), force_gc override) plus a monotonic sequence guard on §25.1 bound control-plane-bug blast radius (FR-073); §27 rewritten accordingly. §25.1 also gains deployment_id and an autoscaling block. HPA confirmed in MVP end-to-end (HPA resource in §20.2, autoscaling RBAC in §22, payload block, HPA-owns-replicas rule in §20.3); cron re-tiered post-MVP (§0.4, §39.1 #6 → §39.3 #48). Design spec: specs/2026-07-02-status-contract-and-tombstone-gc-design.md. |
| — | July 2026 | Placement model replaced; correctness batch applied; per-region registries. Placement moves from project-pinning to per service-per-environment with a per-project default region (FR-077 supersedes FR-008) — Railway/Render granularity. No private networking across regions (FR-078; regions never peered to each other; cross-region calls via public hostnames; databases region-local, UI warns). Namespace renamed to proj-<project_uuid> (32-hex, identical on every cluster; readability via the project-slug label; collision fallback and rename-drift eliminated) — SRE guide synced. Cross-env networking now opt-in (allow_cross_env_networking, FR-079). §20.4 egress corrected (was blocking DNS and the tenant's own DBs): allow-all-external except 10.0.0.0/8…, DNS, same-env, per-env DB /32s. cluster_id on every metric series; joins key (cluster_id, namespace, pod) (FR-080). Snapshot replay buffer (FR-081) replaces the false "max 60s loss" claim; nightly PG↔CH reconciliation. ClickHouse partitioning fixed (daily, not per-project — parts explosion). Image pulls specified (registry↔DOKS integration + SA imagePullSecrets reference). Conditional GET on §25.1 (If-None-Match/304); long-poll seeded (#55). Registries one per region; builds push to the placement region (§16.2). §39.3 #40 reworded, #55–57 added; per-cluster quotas (§20.6); CLAUDE.md §7/§8 updated. Design spec: specs/2026-07-04-placement-model-and-correctness-batch-design.md. |
| — | July 2026 | Managed-database tiers & topology decided. New canonical page (Starbase › Managed Databases, §-less): the customer-chosen DB tier alone decides the engine — plans never gate (Railway-style). Ladder (DO-verified): Mininova DB/Valkey = containerized throwaway in-cluster pod ("not for production"; no backup/PITR/upgrade promises; forced for is_ephemeral envs; PG 250m/512Mi + 1 GiB PVC, Valkey emptyDir) · Micronova→Ultranova = dedicated DO Managed single-node (1/2/4/8/16 GiB; DO list $15.15–$244.35 recorded as vendor constraints; sell prices in the Financial Model). Dedicated isolation = Trusted Sources (cluster-granular) + per-env egress allowlist (FR-075). Billing split: Mininova via existing pod snapshots; dedicated via the provisioning ledger (FR-076); FR-074 added. §25.1 gains a databases array, §25.4 a databases status array; §20.2/§20.3/§22/§24.1 extended (PVC verbs, database-id label). Multiplexed shared pools rejected (side-channel provisioning, password-only isolation; §39.3 #52 revisit-on-evidence). SC-004 withdrawn; §39.3 #47 reframed around the DO reseller partnership; §39.3 #49–52 + §39.1 #53–54 added. Design spec: specs/2026-07-03-managed-db-topology-design.md. |
| — | July 2026 | Substrate pivot: DOKS platform + Talos/gVisor customer clusters; Shipyard machine plane; SNAT/IP simplification. Public launch runs untrusted code, and DOKS can't expose a custom RuntimeClass — so Starbase + Stardeck stay on DOKS while customer workload clusters move to self-managed Talos Linux + gVisor on DO droplets (managed DBs stay DO). New page Overview › Kubernetes Runtime. Workload/kernel isolation added as the 5th security pillar: gVisor RuntimeClass on web/worker pods (FR-082) + a hardened PodSpec on every customer pod (FR-083); abuse controls pre-launch (FR-084). New per-region machine-lifecycle agent Shipyard — outbound-pull, holds regional talosconfig, runs day-2 control-plane ops; Starbase never dials nodes (FR-085); "Starbase decides; Shuttle applies workloads; Shipyard applies machines"; MVP = manual talosctl, Omni fallback. §4.4 reconciled: Cilium overlay → pods SNAT to node IP → Trusted Sources = cluster/node tag → per-tenant DB isolation is Cilium egress; pod/service CIDRs become cluster-local overlays → /16 holds only real infra → IPAM #46 shrinks. Bootstrap (both renderings) → Talos-first; ClusterProvider/worker-binary → Talos; §37 portability strengthened. SC-020; §39.2 #58–59, §39.3 #60–62; CLAUDE.md §1/§2/§7/§8/§131/§137/§140. Design spec: specs/2026-07-05-compute-substrate-and-isolation-design.md. |
| — | July 2026 | Substrate observability decided — extend the §35.5 plane; zero new components at MVP. The Talos pivot made Starform own the K8s control plane and node OS on customer clusters; §35 never absorbed it (nothing watched etcd/apiserver/Cilium; node-exporter was referenced but deployed nowhere; FR-069's droplet + DOKS coverage was aspirational). Decision: the existing per-cluster Grafana Alloy (v1.12) additionally scrapes the substrate (etcd, kube-apiserver, scheduler/controller-manager, Cilium, node-exporter host metrics, etcd-backup age via KSM kube_job_*) and ships only a paging keep-list (~200–500 series/cluster — free-tier scale at $6.50/1k) plus a per-cluster liveness series alerted on absence; the same Alloy is documented onto the DOKS platform cluster. node-exporter recognized as a real component (bundled by the k8s-monitoring chart; the chart's KSM disabled) — §4.2 row added. Customer pipeline untouched (vmagent/Fluent Bit; the customer VictoriaMetrics stays customer-only). Pre-launch: Talos machine logs → Vector → platform ClickHouse (§39.2 #63), telemetry-droplet health (§39.2 #64); seeded: regional platform metrics store (§39.3 #65, companion to #42). Rejected: unfiltered→Grafana Cloud (~40–70k series ≈ $260–450/mo/cluster), all-self-hosted-now (builds paging pre-customer; loses the external watcher for region-down), platform store at MVP (no proven need). FR-069 widened; §35.5, §4.2, §26.3 (both renderings) updated. Design spec: specs/2026-07-05-substrate-observability-design.md. |
| — | July 2026 | Omni fallback removed. There is no buy alternative for machine lifecycle — Shipyard is the sole plan. §39.3 #61 deleted; the Shipyard "Build vs. buy" section and the Kubernetes Runtime license note scrubbed. (The compute-substrate design spec retains Omni as an option considered — historical record only.) |
| — | July 2026 | Review batch: custom-domain certs, volumes tier, TLS posture, upgrade runbook, consistency nits. Custom domains → Cloudflare for SaaS (§5, §39.3 #27): CF validates/issues/renews custom-hostname edge certs (100 free, then $0.10/mo — verified), domains stay behind the WAF; cert-manager removed from the docs entirely ("no cert-manager" is permanent — §24B.8 rows replaced with custom-hostname-id, consumer contract row dropped). Competitor check: Sevalla + Render ride Cloudflare; Railway/Fly run their own ACME with public rate-limit pain. Volumes re-tiered post-MVP (§39.1 #7 → §39.3 #7; MVP state = managed DBs + Tigris buckets; CSI stays for Mininova PVCs). FR-061 reworded: TLS 1.2 minimum + 1.3 enabled at the public edge (Cloudflare zone setting, recorded in §5); internal transport under FR-066/FR-071. Manual fleet-upgrade runbook → pre-launch (§39.2 #66, carved from #36; Talos OS/gVisor, K8s, Cilium, Envoy GW incl. §20.2 parse re-verify, Shuttle rollout, droplet patching, DOKS windows; #36 narrows to Shipyard automation). Auth: #25 split — dashboard sessions/JWT/SSO-callbacks re-tiered MVP-blocking (§39.1 #67), #25 keeps password-reset/MFA extras; "out of scope at MVP" wording fixed to post-MVP (§39.3 #30). Labels: app.starform.io/* prefixes fixed in the Var-Group Secret example (starform.io/managed-by, var-group-id); glossary "8 labels" → 9 with membership matching §24.1 (database-id in, tier = operational); CLAUDE.md §7 aligned. Design spec: specs/2026-07-05-review-batch-design.md. |
| — | July 2026 | Architecture review — Batch 1 (mechanical consistency fixes). Applied the no-decision findings from the 8-dimension review (specs/2026-07-06-architecture-review.md): FR/SC ranges bumped to FR-085/SC-020 (reference index, canonical-sources, CLAUDE §4); Shipyard added as the 5th binary (06-binaries-and-repos) and to the §4.1 regional-droplet inventory; the §16.12 build-state diagram cancel edges corrected to pending|building|deploying → cancelled (was missing building/deploying, had a spurious built→cancelled) per FR-023; the duplicate #section-15-9 anchor + forked RBAC content removed from Stardeck (now links the canonical §15.9); is_ephemeral added to the §15.7 environments schema + §15.2 diagram; SSO wording fixed to GitHub+Google at MVP (04-integrations, 08-tech-stack) per FR-005; RPS upstream_rq_total added to the Envoy allowlist; network added to FR-050; retention reworded to row-level TTL (FR-051, §35.5 isolation table); tier qualifiers on FR-059 (pre-launch), FR-069 (store-scrape pre-launch), open-work #26 (MVP=manual) and glossary cron (post-MVP); Diagram 2c label box aligned to the 9-label set; managed-DB egress "fully effective once…" → "as of the 2026-07-04 correction"; Foundations→Starbase link labels (5 files); #60 link label Shipyard; bootstrap Helm order (metrics-server/KSM before Envoy); runtime day-2 op table reconciled with the Shipyard 6-op set + worker-vs-control-plane upgrade split; canonical-sources §5 label → "Traffic Flow & TLS"; CLAUDE §2 node-exporter + §8 substrate-observability note. Batch 2 (design decisions — the blocker + ~8 items) deferred to individual review. mkdocs build --strict clean. |
| — | July 2026 | Architecture review — Batch 2 · B1 (blocker): desired-state sequence survives a control-plane restore. The monotonic per-cluster sequence lives in the control-plane Postgres; a naive DR restore would re-issue lower sequences and freeze every Shuttle fleet-wide (running workloads survive, all deploys stop, no escape — force_gc bypasses only the delete breaker). Fix: on control-plane recovery Starbase re-derives each cluster's counter from the max sequence_observed that cluster already reports (§25.4) and resumes above it — automatic, per-cluster, desired-state-pure (the fleet holds the high-water mark, reusing an existing status field); plus an operator-authenticated per-cluster sequence_reseed break-glass for total loss (cluster + Postgres both gone). §25.1 sequence contract + §25.4 sequence_observed bullet + §27 failure-mode row + §39.3 #34 DR item updated. Spec: specs/2026-07-06-architecture-review.md (B1). |
| — | July 2026 | Architecture review — Batch 2 · M-SEC1: customer-cluster secrets encrypted at rest. Shuttle renders Var Group values into K8s Secrets that live as base64 in each customer cluster's etcd — plaintext at rest on self-managed Talos running untrusted co-tenant code, contradicting FR-060's "all sensitive data at rest." Decision (verified Talos machine-config capabilities): MVP = Talos LUKS2 disk encryption on STATE+EPHEMERAL (EPHEMERAL holds etcd data; AES-XTS-256, key sealed to the node) in the bootstrap machine-config — closes stolen-disk/DO-volume-snapshot/offline-etcd; pre-launch (§39.2 #68) = secretbox EncryptionConfiguration (encrypts Secret objects within etcd) as defense-in-depth. Honest scope: neither stops a live gVisor-escape reading a mounted secret — that's the sandbox's job. FR-060 reworded (two at-rest layers), §39.1 #18 catalog extended, §39.2 #68 added, the runtime Talos-stack + security-pillar updated, the misleading encrypted-at-rest annotation reworded. Spec: specs/2026-07-06-architecture-review.md (M-SEC1). |
| — | July 2026 | Egress-only telemetry direction; build-log rework; MVP scope cuts (2026-07-09). A non-DO-region pressure test found VPC peering to be the only control-plane→region inbound dependency, carrying four flows — each now resolved. Build logs leave ClickHouse entirely (§16.6 rewritten): the Worker relays Depot's live SolveStatus stream to Stardeck and archives <build_id>.log.zst per build to DO Spaces; verified 2026-07-09 that Depot's public API (ProjectService / BuildService.createBuild / BuildKitService) has no completed-build log retrieval — a log-export API joins the §16.7 Depot negotiation checklist; FR-049 carve-out; §16.2 diagram, §14, §0.2, Stardeck, v1 §35.1 + v2 store/index/read synced. Billing audit trail re-tiered post-MVP (§36): MVP = raw snapshot rows in Postgres (snapshot_id PK — idempotent dedup + dispute record, bounded ~60–90 d retention); one write path, reconciliation deferred; post-MVP (new §39.3 #71) the ClickHouse audit copy is written in-region by Shuttle via the OTel gateway, upgrading the nightly PG↔CH check to delivery-gap detection. Tigris re-tiered post-MVP and scoped customer-facing only (§39.1 #16 → §39.3 #16; parity: Railway/Render ship no S3 primitive): platform object storage = DO Spaces, never Tigris — etcd DR (#59), throwaway-DB dumps (#51), runtime page corrected ×2. Mass Relay christened (§1 naming + §3 binaries): per-region egress-only telemetry query channel superseding §39.3 #43 — outbound persistent gRPC stream from the telemetry droplet; Starbase sends FR-065-filtered queries down, Mass Relay executes against localhost ClickHouse (starbase_read) and streams rows back; retention upserts ride the same channel in its regions. Transport settled as a per-region choice behind the QueryTransport seam (2026-07-10): DO regions keep the direct read over VPC peering permanently (intra-region $0, private, zero moving parts — in a healthy peered region the agent adds ~nothing, and a rare peering outage degrading dashboard reads sits within the §4.1 accepted risk class); Mass Relay is the transport for clouds without DO peering (built with the first non-DO region; two replicas where primary) and the emergency transport for a peered region during a prolonged peering outage (deploy + flip the seam); #46 unchanged. §13 also gains the previously undocumented telemetry-query-broker responsibility; FR-071 + §35.4 + §4.4 carry forward notes. Rejected and recorded: direct customer ClickHouse access (row-policy tenant boundary, shared-store DoS, schema-as-public-API, RBAC/plan/audit bypass — the future version is a brokered query endpoint). CLAUDE.md §2/§8 updated. Design spec: specs/2026-07-09-egress-only-telemetry-and-scope-cuts-design.md. |
| — | July 2026 | Region onboarding + DNS & domains designed (2026-07-11). Two new canonical pages. Starbase › Region Onboarding: the region model is factories-are-code, regions-are-data (Crossplane Provider/ProviderConfig precedent; §11 house pattern) — regions columns defined (id, cloud, status provisioning→active→draining, transport, vpc_cidr, endpoints; creds in the §39.1 #18 catalog), registered via a PR-reviewed manifest → operator-authenticated admin endpoint with pre-write validation + connectivity probes (KubeFed-join precedent); stages 0–6 (allocate → peering wherever the cloud supports it → v2 telemetry droplet → registry → §26.3 cluster bootstrap → registration → go-live canary through the normal customer pipeline); token minting made concrete (cluster_id + Shuttle bearer + FR-066 ingest token minted at cluster registration, delivered as bootstrap Secrets); §39.3 #46 resolved & absorbed (one tfvars line per region; no IPAM system); §13 gains the admin-surface bullet; §12 gains the code/data note. Networking › DNS & Domains: two-domain split (starform.io product / starform.app customer-only) with the PSL as the tenant-isolation linchpin (new pre-launch §39.2 #72, lead-time warning — submit during private beta); hostname→region routing = per-hostname proxied records to the placement region's LB via DNSProvider (Sevalla/Render camp — verified; Fly/Railway own-anycast camp rejected per #56), wildcard becomes a 404 catch-all, record lifecycle tied to deploy/migrate/delete (#40 gains the record move; #8 narrowed to hostname-generation rules). LB contract fix: §25.1 cluster_config.lb_size_units → adapter-rendered opaque gateway_service_annotations (Shuttle applies without interpreting; §34.2/§32/§7 synced) — the last DO-ism out of the cloud-neutral contract. New §39.3 #73 (non-DO readiness spikes: Talos image import, CCM/CSI on self-managed, CF record limits). §0.4 domains row updated; canonical-sources + nav + CLAUDE.md §8 updated. Design spec: specs/2026-07-11-region-onboarding-dns-design.md. |
| — | July 2026 | Track-A closers: FR-075 cloud-neutral · jump droplet · residency posture · region resilience (2026-07-11). FR-075 reworded goal-first (was written in DO product terms — unsatisfiable elsewhere): the DatabaseProvider restricts DB-side access to the hosting cluster's nodes by the cloud's native mechanism (DO: tag-scoped Trusted Sources; others: security groups / IP allowlists); the pre-SNAT Cilium per-environment egress allowlist stays the per-tenant boundary (§10 port row + managed-databases synced). Jump droplet decided: the Talos API stays VPC-private, never public — each region gets a hardened SSH jump droplet at onboarding (the only holder of the regional talosconfig; per-region blast radius); it is proto-Shipyard — the agent later installs onto the same box (runtime machine-lifecycle, onboarding stages 2/4, §4.1 row renamed from "Shipyard droplet", Shipyard page synced). Residency posture recorded (onboarding page, #22 crossref): runtime + data residency per region (workloads, DBs, telemetry, images in-region); metadata, secrets, and billing central; source + build artifacts transit US/EU via Depot — full-stack residency is Enterprise territory, not offered; §16.9 #3 is the path if ever needed (a numbered seed item was deliberately not added — §16.9/§16.10 already carry it). DO has no AZs (verified): a DO region = one datacenter = a single failure domain — honest note added to runtime Control-plane HA; AZ handling folded into #73 (spread on AZ-having clouds); new §39.3 #74 region resilience tiers (metro pairs NYC1+NYC3 as two ordinary peered regions · cross-region managed-DB read replicas · never imply AZ redundancy in the SLA; competitive posture verified: Fly host-pinned/no-AZ, Heroku 3-AZ Private Spaces). Track A (region-onboarding gap list) closed. |
| — | July 2026 | Database schema designed — §39.1 #1 (2026-07-11). New canonical page Starbase › Database Schema with full DDL. The one fork settled: services are project-scoped definitions instantiated per environment — a service_environments junction row (FR-077's "service-per-environment record", literally) carries placement cluster_id, branch + auto_deploy (#17 resolved), current deployment, tier/replicas/autoscaling, hostname, suspended; a service exists in an env iff a row exists (Railway-parity per-env freedom; UNIQUE(project_id, name) on definitions). Delivery: builds (the §16.12 state machine + Depot handle + Spaces log key) vs deployments (a build released to one service-env, §25.4-driven, first_ready_at/FR-072) — rollback = a new deployment row pointing at an old build. Ops: the §14 job queue gains leases (locked_until, heartbeat-extended — a dead Worker's job re-runs), attempts/max_attempts (poison-job stop), dedupe_key (§16.11 debounce), and resumable build jobs (re-poll by depot_build_id) — closes the Worker-dies-mid-build gap. Billing (per 2026-07-09): billing_snapshots raw (PK (snapshot_id, snapshot_at), daily partitions, ~60–90 d), usage_totals (invoice-cron source), usage_events; workspaces gains stripe_customer_id (§15.7 extended). Infra: clusters (holds desired_state_sequence, lb_ip) + cluster_tokens (hashed, minted per Region Onboarding). Soft deletes on placed entities are the §25.1 tombstone protocol (deleted_at + deletion_confirmed_at). Out by decision: buckets (#16), sessions (#67), Mass Relay/audit tables (#43/#71); desired state stays computed, never stored (§32). §39.1 #1 marked designed; #17 resolved-in-schema. Design spec: specs/2026-07-11-database-schema-design.md. |
| — | July 2026 | Auth & sessions designed — §39.1 #67 (2026-07-11). New canonical page Starbase › Auth & Sessions: SSO login flow (GitHub + Google via the §11 SSOProvider) with verified-email auto-linking (one person, one account, any provider; sso_identities keyed on provider+provider_user_id); skinny 15-min JWT (sub/sid only — roles resolved per request by §15.6, so changes are immediate) + server-side rotating refresh tokens (hashed sessions rows, 30 d sliding / 90 d cap, reuse of a rotated-out token revokes the session family); transport = Secure; HttpOnly; SameSite=Lax cookies on .starform.io — never JS-readable, and unreachable from customer code on starform.app (the two-domain split's second payoff); CSRF = SameSite + required custom header. sessions + sso_identities DDL on the page (RBAC-page precedent); #67 marked designed; extras (password reset, MFA) stay §39.3 #25. Design spec: specs/2026-07-11-auth-sessions-design.md. |
| — | July 2026 | Dashboard REST API contract designed — §39.1 #2 (2026-07-11). New canonical page Starbase › Dashboard API: conventions (flat-by-UUID URLs, lists nesting one level; cookie auth + CSRF header, §39.2 #20 tokens will reuse the endpoints; one error envelope with stable code; cursor pagination) · a per-resource endpoint catalog (workspaces/members with §15.3 anti-escalation, projects, environments, service definitions vs service-environment instantiation — the Model-A split made API-visible, builds, deployments, databases incl. tier-boundary 409, Var Groups with §38.3 warning responses, FR-065-filtered logs/metrics reads, usage, regions) with §15 permission mapping incl. the Developer* protected-env notation · key payloads (deploy/rollback — one endpoint, an old build_id is the rollback; the service-environment PUT; logs SSE frames; metrics series; var-group entry warnings) · cross-cutting semantics (202+status async, tombstone deletes per FR-073, middleware-only enforcement). #2 marked designed — all §39.1 "design before building Starbase" gates now closed. Design spec: specs/2026-07-11-dashboard-api-design.md. |
| — | July 2026 | Starbase tab restructured — dev-order nav, feature-vertical pages, history sweep (2026-07-11). Content moved, nothing deleted. Nav regrouped as a build sequence: Orientation (index, §9, §13, §14) → Foundation-build-in-order (1 Database Schema · 2 Auth & Sessions · 3 RBAC · 4 Dashboard API) → Wiring (§10–§12, §32) → Features (Starforge, §38, Managed DBs, §36) → Operate & Reference (Region Onboarding, §37). Feature-vertical rule (extends the RBAC/§38.2 precedent): each feature page owns its DDL and its API endpoints — moved with their anchors intact: billing tables + usage endpoints → §36 (#db-billing/#api-billing); jobs + lease semantics → §14 (#db-jobs); builds/deployments DDL + delivery endpoints + deploy payload → §16.12 (#db-delivery/#api-delivery); github_installations/webhook_deliveries → §16.11 (#db-git); clusters/cluster_tokens → Region Onboarding (#db-infra); DB endpoints → Managed Databases (#api-databases); Var Group endpoints + entries payload → §38 (#api-var-groups). The two central pages became spine + map: Database Schema = ER diagram + core-graph DDL (services/service_environments/databases) + a where-every-table-lives index; Dashboard API = conventions + core-resource catalog + a feature-endpoint index. History sweep: dated change-narration ("decided/reworded/re-tiered/settled YYYY-MM-DD — was…") removed from all Starbase pages — current truth only; the was/why remains here and in specs/ (external-fact verification dates kept as freshness metadata); each page keeps one Design record: pointer. Paragraph pass: walls split into bullets (managed-databases intro, onboarding stages 2/4). Starbase scope only; other tabs sweep later if wanted. |
| — | July 2026 | Remaining tabs restructured — Overview · Shuttle · Networking · Runtime · Shipyard · Stardeck (2026-07-11). The Starbase treatment (row above) applied to the rest of the site; excluded by design: Observability (self-contained SRE guide; the v1→v2 flip is open-work #69), Reference (keeps dated change-notes), _legacy/. Nav: Shuttle's flat 16-entry list regrouped as a build sequence — Orientation (§19, Security & Isolation) → Build-in-order (1 · §21 Repo Structure · 2 · §29 Scaffolding · 3 · §23 Configuration · 4 · §25 API Contract · 5 · §20 Per-Project Resources · 6 · §22 Cluster RBAC) → Conventions (§24, §24B) → Operate & Reference (§26, §27, §28, §30, §31); Networking reordered to the request path (Traffic & TLS → DNS & Domains → Gateway → Load Balancer → VPC & IP); Overview already read in order — unchanged; no URLs changed. Feature-vertical / dedup: Security & Isolation's verbatim copy of the §20.4 env-isolation NetworkPolicy YAML (drifted from the canonical) replaced with a bulleted posture summary + pointer — §20.4 stays canonical and complete; no other hoards found (§25/§24/§24B/§22/§23 already own their contracts). History sweep: dated change-narration removed across the six tabs — §20.1 namespace-rename note, §20.6 quota date, §25.1 lb_size_units replacement note, "Field semantics (July 2026)", NetworkPolicy-title "corrected" suffixes, the §0.4 Tigris re-tier date, the §4.4 "settled" date, the DNS/Runtime/Shipyard intro dates, the talosctl/etcd-backup dates, and the undated §19.3 customer_id + §24.4 plan-tier "was removed" notes — current truth only; the was/why stays in this table and in specs/; the §24.1 "no customer_id (v1.9)" canonical guard kept deliberately (CLAUDE.md §7 mirror). Design record: pointers added (§20 → 2026-07-04-placement…, §25 → 2026-07-02-status-contract…, Security & Isolation → 2026-07-05-compute-substrate…) and normalized (Runtime, DNS & Domains, Shipyard). Stale DO-ism fixed: two "LB size unit" examples (§19.3, §30) → gateway Service annotations, matching §25.1. Paragraph pass: walls split into bullets — §4.1 intro + telemetry-tier notes, §19.3 Snapshot/Status Runnables, §20.3 deletion-safety + replicas-ownership, §20.4 posture, §25.4 Starbase handling, the security-page etcd pillar, two §4.4 bricks, the DNS proxied-record paragraph (the §2 topology-SVG caption deliberately left as a caption). Stardeck: already clean, untouched. Content moved/deduped, nothing lost. |
| — | July 2026 | Architecture review — Batch 2 · M-DES3: multi-cloud ClusterProvider aligned to Talos-on-plain-VMs. The ClusterProvider adapter mapped AWS→EKS / GCP→GKE (byoc-portability, 03-ports, 02-package-layout) — managed control planes that can't expose the gVisor RuntimeClass FR-082 mandates, contradicting byoc's own "identical Talos+gVisor on any vendor with plain VMs" (the pivot updated the DO column but left AWS/GCP stale). Fixed: the future multi-cloud cells now read Talos on EC2 / Talos on GCE (plain VMs, self-managed Talos — never managed EKS/GKE). Illustrative only; BYOC stays out-of-scope. Spec: specs/2026-07-06-architecture-review.md (M-DES3). |
| — | July 2026 | Architecture review — Batch 2 · M-DES5: preview-env metrics provided, not suppressed. FR-067 claimed a vmagent "drop-list skips is_ephemeral routes," but the parse-only Envoy path can't know a route is ephemeral without the name-matching FR-067 forbids — the mechanism was un-implementable. Competitor check (Railway, Render both expose preview observability per-service; Sevalla via Cloudflare analytics) + the L7-attribution review (Envoy-stats identity is inherently in the cluster name — the route-name parse is an Envoy constraint, not a fixable-cheaply hack; escaping it means access-log-derived or Cloudflare-edge metrics, deferred) → decision: previews are metriced like any environment (logs + CPU/mem/network + per-route L7), not suppressed; churn-series cardinality is bounded by metric retention, short-retention-for-ephemeral a post-MVP refinement (FR-051). FR-067 reworded, §35.2 drop-list claim removed, SRE-guide preview stub → Decided, §39.3 #28 metric-handling resolved (launch-vs-seed still open). L7 gateway/attribution redesign (Envoy access logs vs Cloudflare analytics) noted as a future option; Envoy Gateway retained. Spec: specs/2026-07-06-architecture-review.md (M-DES5). |
| — | July 2026 | Architecture review — Batch 2 · M-DES2: failed-payment suspension stops autoscaled services. Suspension set replicas: 0, but the HPA owns replicas under autoscaling (Shuttle excludes the field from its diff) — so a non-paying customer's autoscaled service kept running. Fix: a dedicated suspended boolean on the §25.1 service payload; when true Shuttle scales to 0 and removes any HPA regardless of the autoscaling config (stored config untouched; clearing the flag restores the HPA). billing.md failed-payment row + §25.1 field + §20.3 HPA-ownership rule updated. Spec: specs/2026-07-06-architecture-review.md (M-DES2). |
| — | July 2026 | Architecture review — Batch 2 · M-CON3: workspace_id telemetry claim corrected to server-side resolution. FR-064/§26.2 claimed workspace_id is "carried" on every metric/log line, but the KSM allowlist and ClickHouse log schema carry only the tenant key (project_id+environment+service_id) — a workspace-boundary query was unsatisfiable as documented. Decision (no pipeline change; avoids adding a redundant label to every series): workspace-boundary scoping is resolved server-side via Starbase's project→workspace map (expand workspace_id → its project_ids, filter on those). workspace_id remains a resource label (§24.1) + on billing snapshots (§25.2), but is not a metric/log series label. FR-064, §26.2, and the two SRE-guide "rides along" restatements reworded. Spec: specs/2026-07-06-architecture-review.md (M-CON3). |
| — | July 2026 | Architecture review — Batch 2 · M-CON1: MVP is single-region, multi-region built-but-dormant (Batch 2 complete). Three statements said "no cross-region read at MVP" while FR-071 (untagged = MVP MUST) + the query-broker fan-out + §4.1 implied active multi-region reads. Decision: the whole platform launches in one region (control plane + clusters + telemetry co-located); the multi-region read path — query-broker fan-out, per-region peering, FR-071, cluster_id joins — is built but dormant until a 2nd region is added (post-MVP geographic expansion). Matches the single-region control-plane SPOF and how Railway/Render launched. FR-071 tagged dormant-at-MVP (was reading as an active MUST); §4.1 clarified to single-region-whole-platform; the "no cross-region read at MVP" statements are now correct. FR-077/078/080 (placement, region-local DBs, cluster_id) keep their multi-region readiness. This closes the 2026-07-06 architecture review — Batch 1 (mechanical) + Batch 2 (1 blocker + 8 design decisions) all applied. Spec: specs/2026-07-06-architecture-review.md. |
| — | July 2026 | Observability v2 — single-store ClickHouse (forward design; added additively). Reconsidered the two-store telemetry design before any code exists: one ClickHouse store replaces VictoriaMetrics + vmagent + vmauth + Vector + Fluent Bit. L7 metrics (latency/RPS/error/throughput) derive from Envoy access logs → clean tenant columns, so the fragile envoy_cluster_name route-name parse and its EG-version coupling disappear; CPU/mem/net scrape into ClickHouse rows. One OpenTelemetry Collector (ClickStack/SigNoz blueprint) is the only agent. Per-tier retention via ClickHouse TTL for logs and metrics — closes the FR-051 gap VM couldn't. Raw-first; rollup MVs deferred (ClickStack guidance). Borrows ClickStack's collector + schema (Apache-2.0); builds only the tenant-filtered query broker (FR-065) + Stardeck display (not HyperDX — no per-tenant isolation). Documented additively: new docs/observability/v2/ (architecture · ingest · schema+retention · metrics-from-logs · read+isolation · bootstrap) with SRE build steps + titled configs + Mermaid; the two-store guide kept as v1 with a superseded banner. Verified sources: OTel clickhouseexporter (logs/traces beta, metrics alpha), EG OTel access-log sink, ClickStack schema/TTL, SigNoz. Cross-cutting pages (topology SVGs, FRs, vendor map, CLAUDE.md) still read v1 — reconciling them to v2 is a tracked follow-up. Design record: specs/2026-07-07-single-store-clickhouse-telemetry-design.md. |
| — | July 2026 | Observability: Platform Self-Monitoring extracted to a shared section + v1↔v2 consistency pass. §35.5 (Grafana Alloy → Grafana Cloud) is independent of the customer-telemetry store, so it was promoted in the nav out of the v1 subgroup to a shared entry (file path + #section-35-5 anchor unchanged → zero link breakage), neutralized of its v1-store references (vmagent/VictoriaMetrics/Vector), and reframed as shared; the stale "(ch.5)" refs + v1's missing inbound link fixed. A thorough v1↔v2 consistency check (2 agents, 15 findings) then fixed: the retention tier driver (was never populated + mislabeled — it's the namespace starform.io/plan-tier, not the pod's Nova tier; added the k8sattributes extraction, corrected the gotcha, added the DDL backstop branch); the dropped FR-066 per-cluster ingest token (restored on the agent→gateway hop); http_events→otel_http; the OTel gateway placement (on the telemetry droplet, matching the SVG); "no route-name parse" qualified + the route_name fallback column wired; throughput → bytes_sent + bytes_received (v1 parity); "tenant tuple"→"tenant key" (CLAUDE §7); an FR-067 preview-env coverage row; Alloy + KSM clarified in the v2 bootstrap; a superseded banner on every v1 chapter; and a spec typo (7/20/30/90→7/30/90). Cross-cutting FR/SVG/vendor-map reconciliation stays open-work #69. Design record: specs/2026-07-07-single-store-clickhouse-telemetry-design.md. |
| — | July 2026 | Two-VPC separation made explicit (home topology SVG + §4.1 text). The master topology SVG (home + §2) grouped the system by plane and never showed that the control plane and the regional stuff live in separate VPCs — and the M-CON1 §4.1 edit had implied they share a VPC at MVP ("co-located", Starbase "reads … directly", "per-region peering built but dormant"). Corrected to the authoritative networking model (§4.4): the control-plane VPC peers each regional VPC, so the VPCs are always separate, and single-region MVP is simply one intra-region peering — the control plane reads the regional stores over it. Only the cross-region fan-out + additional peerings are dormant until a 2nd region (M-CON1's single-region / multi-region-dormant decision otherwise stands). Redrew the master SVG (both synced copies) with a Control-plane VPC box, a private VPC peering connector, and one enclosing Regional VPC /16 box holding the cluster + telemetry-droplet sub-boxes; fixed the §4.1, FR-071, and §35.4 prose to match. The master SVG still shows the v1 telemetry names — the v1→v2 swap stays open-work #69. |
| — | July 2026 | Observability v2 restructured to a linear pipeline + two verified correctness fixes. The v2 docs were fragmented (separate Metrics/Logs pages, a Collect page that told you to "go to the Metrics page") and the Observability tab landed on the old v1 guide. Rebuilt v2 as a straight line — Architecture · Collect · Store · Read & query · Bootstrap (6 pages → 4; metrics + logs + the read-filter merged into one "Read & query", since one ClickHouse store makes them just different tables). The landing (observability/index.md) is now a lean v2-first pointer; the v1 read-first overview (component map, VPC Diagram 1, lifecycle, scope) moved to observability/00-v1-overview.md. Cut the meta-bloat notes ("Self-contained — build from v2 alone"). Two fixes folded in, both verified against vendor docs: (1) L7 attribution — the %METADATA(…:starform.io/*)% label pull was invalid (Envoy Gateway propagates name/namespace/kind/annotations, not arbitrary labels); v2 now reads the HTTPRoute name/namespace via the access-log CEL operator and splits the tenant columns once in the Store DDL — the same §20.2 naming, from a stable Gateway-API field, not v1's per-scrape envoy_cluster_name regex. (2) Retention — dropped the starform.io/plan-tier namespace label and the tier column; retention now derives from the customer's plan at the store via a Starbase-maintained project_id → retention_days ClickHouse dictionary + a retention_days MATERIALIZED dictGet(…) column + TTL (dictGet is forbidden in TTL but allowed in a materialized column). Swept the tier overload: §24.4 label removed with a note, FR-051 reconciled to the store-derived model, spec corrected. Canonical §20.2/§35.2 + CLAUDE.md still describe v1's parse — open-work #69. Design record: specs/2026-07-07-single-store-clickhouse-telemetry-design.md. |
| — | July 2026 | Observability v2: runnable-config pass + edge-case review. User-prompted double-check of every v2 config against vendor docs (after the Envoy-labels miss) found the configs directionally right but not literally runnable; all corrected. Store: the ClickHouse exporter INSERTs a fixed schema — tables rewritten to the stock columns + our MATERIALIZED derived columns pulled from the LogAttributes/ResourceAttributes Maps (SigNoz/ClickStack pattern), with store-side hyphen-stripping so both tables share 32-hex IDs; write side defined (otel_ingest for the gateway, starbase_write for retention upserts; build logs enter via the gateway; billing audit stays §36's — Postgres remains the invoicing source, ClickHouse is the audit copy). Collect: filelog container parser + pod_association (file logs have no connection IP); cAdvisor scrape authed + pinned to its own node; two gateway doors fix a double-write bug (one shared receiver fed both tables); bearertokenauth (token file) realizes FR-066; file_storage disk queue rides out store outages; unrouted rows dropped. Rollups documented in full (1-minute AggregatingMergeTree + MV from raw attributes, concrete add-trigger, backfill note); quantile() correctly described as approximate. Edge cases captured as gotchas: self-tailing exclusion, fast-dying-pod unattributed rows (+ watchdog), websocket latency skew (response_code != 101), per-tenant log-spam monitoring, TTL-is-lazy, plan-changes-affect-new-rows-only, maintenance-blackout expectations, plan-window UI note, bloom-index lever. Alpha/beta strategy: test-and-run with a fallback ladder (pin → SigNoz exporter fork → swap writer only). Mermaid diagrams enlarged to content width (CSS). Verify-at-build flags inline: EG JSON→attributes-vs-body, CEL path on pinned EG, alpha metrics exporter. Spec updated (specs/2026-07-07-single-store-clickhouse-telemetry-design.md, "Correctness pass"). |
| — | July 2026 | Deploy vertical designed — §39.1 #4/#5/#12/#15 (2026-07-11). Pre-deploy command in MVP (FR-086): build_config.pre_deploy_command → §25.1 pre_deploy → Shuttle runs a one-shot Job (new image, the env's Var Groups, gVisor + hardened spec, backoffLimit 0, 600 s deadline) that gates the Deployment update; failure ⇒ §25.4 rollout: failed reason PreDeployFailed, previous version keeps serving; superseding deployments cancel a running Job; Job pods carry standard labels so their logs attribute for free — chosen over the startup-path workaround (N replicas racing the same migration). Health checks (FR-020 reworded): Railway-style single input — path + timeout (default 300 s) — rendered by Shuttle as all three probes (startup ceil(timeout/10) @10 s · readiness @10 s ×3 · liveness @15 s ×3); web default when unset = TCP-connect readiness (never ready-at-process-start); workers = none. Rollout semantics (#4): maxSurge 1 / maxUnavailable 0 everywhere; deploy-during-deploy supersedes (in-flight → superseded); rollback pins nothing; registry GC keeps currently-deployed + last 10 images per service (Worker cleanup). Build error handling (#15): 45-min build timeout; deterministic failures never auto-retry, infra failures ride the §14 job lease; no hard image-size limit (warn >5 GiB); Spaces log-upload failure non-fatal. §20.2 Job row + §20.3 apply-order step 8 + probe/rollout subsections; §16.12 gains the semantics section + a 5th progress stage (Pre-deploy); §16.5 build-overrides table (root_directory/dockerfile_path/commands); §0.4 Config row; FR-086 added (range mentions bumped). Design spec: specs/2026-07-11-deploy-vertical-design.md. |
| — | July 2026 | Docs reconciled to Observability v2 — §39.3 #69 closed (2026-07-11). The cross-cutting pages now state the single-store ClickHouse + OTel Collector design everywhere; the v1 chapters stay archived as superseded. Topology: both master SVGs (home + §2) — vmagent/Fluent Bit → one OTel Collector box, VictoriaMetrics/Vector → OTel gateway + ClickHouse (logs + metrics · plan-based TTL), one blue telemetry flow (amber legend retired), the Shipyard box relabeled to the jump droplet; captions/legends rewritten. §4.1/§4.2: telemetry tier = one droplet (ClickHouse + gateway rows); cluster table = one OTel Collector row; KSM re-scoped to the platform plane; the vmalert note dropped; §0.2 step 7 and the §6 vendor rows flipped. §20.2: the v1 envoy_cluster_name per-scrape parse retired — identity is read from Envoy route metadata via the access-log CEL operator and split once in the store DDL; the EG-bump rule is now "confirm the CEL path"; the "why fold project_id" rationale restated in v2 terms (networking/index note synced). FRs: FR-050/063/064/066/067/069/071/080 reworded to v2 with was-notes (FR-067 loses the series-cardinality mandate — L7 metrics are rows, not series; FR-080 becomes pod-origin cluster_id columns, no joins; FR-066 = per-cluster token at the OTel gateway; FR-071's front-door = the read-only ClickHouse user alone); FR-049/051 were already v2. Shuttle: §26.2/§26.3 rewritten (agent/gateway table, v2 flows, bootstrap step 6, single-droplet provisioning note); §17 goal, §30 criterion 3, §24 intro, §24B.5 (reserved for a future app-metrics scrape) + the consumer table, the §21 servicemonitor comment. Security & Isolation: ingress-auth pillar = token validated at the gateway's two doors; single read front-door (vmauth gone); query-isolation pillar cites v2. Elsewhere: Stardeck reads ClickHouse via the broker; the BYOC portability row; glossary vendor map + 7-metrics pointer; §32's attribution cite; canonical-sources — v2 marked current design, §35.1–§35.4 rows marked (v1, superseded) with current v2 homes; §35.5's three v1 hedges dropped; open-work #9/#26/#44/#54/#63/#64/#65/#66 reworded to v2 and #69 marked RECONCILED; CLAUDE.md §2/§8 + the verify-list flipped. Design record: specs/2026-07-07-single-store-clickhouse-telemetry-design.md. |
| — | July 2026 | Starbase feature-vertical completed; nav numbering dropped (2026-07-12). The hybrid left by the restructure (a central schema page still holding services/service_environments/databases DDL while every other table lived with its feature) is dissolved — one rule, no exceptions: a page owns a concern; its tables and endpoints live on it; the central pages own nothing. New page Starbase › Services & Environments (#services-model) — the Model-A service model's home: services + service_environments DDL (#db-services) and the service/service-environment endpoints + instantiation PUT payload (#api-services), moved from the schema and Dashboard-API pages. databases DDL → Managed Databases (#db-databases, beside its endpoints). The central page retitled Data Model (same URL/anchors): ER diagram + the where-every-table-lives index + cross-cutting notes, zero DDL; Dashboard API keeps conventions + tenancy CRUD + observability reads + the endpoint index. Nav: the new page slots into Foundation; the "N ·" numbering prefixes dropped from the Starbase Foundation and Shuttle Build groups (the group labels already say "in order"). Inbound #db-core links retargeted; canonical-sources rows updated. Content moved, nothing deleted. |
| — | July 2026 | Feature-semantics batch — §39.1 #3/#6/#8/#9/#10 designed + templates seeded (2026-07-13). Closed the remaining MVP feature-semantics items, mostly composition over the existing contract. #6 Service types: explicit type (web/worker) added to the §25.1 payload (the services.type column + service-type label already existed — the gap was propagation); web renders a ClusterIP Service (+ HTTPRoute when public), worker neither/no probes; a UI type change is a delete-and-rebuild with service_id preserved. #10 Private networking + connections: new service_environments.public (web only: false = internal, no HTTPRoute/DNS); a stable internal_host = the ClusterIP Service name <service-name>-<environment>, shown to the customer as a clean http://host:port (never the .svc.cluster.local FQDN — verified Render exposes a bare <name>-<hash>:port, Sevalla the full FQDN; we chose the short form); Connections (service_connections table) auto-inject a linked service's URL per environment via the env map — the same path as DB-credential injection — sidestepping the per-env Var Group gap (#39); cross-service ${{…}} refs stay deferred. #8 Hostnames: single-label, always-suffixed <name>[-<env>]-<rand4>.starform.app (Render/Railway/Vercel — unique by construction, no squatting/"name taken"; single-label forced by Cloudflare Universal SSL). #9 HPA UX: enable + min + max + Target CPU % (memory % advanced); bounds 1 ≤ min ≤ max ≤ plan-quota-cap, 30 ≤ target ≤ 90; explicit scaleDown.stabilizationWindowSeconds: 300; CPU-based (the frontend demo's min/max-only + "based on traffic" caption corrected). #3 First Deploy: new small Overview page (signup → live URL, default target production; to be refined). Templates: services.source_type/image_ref schema landed early (image-source skips the build) — full flow is new §39.3 #75. Image scanning / security_hold (seen in the frontend demo): explicitly out of scope. Two Starforge doc bugs fixed (§16.2 "logs → ClickHouse" → DO Spaces; §16.12 "four" → five stages). Pending-review draft specs/2026-07-11-remaining-items-proposals-PENDING-REVIEW.md sections applied. |
| — | August 2026 | Billing calculation + Stripe lifecycle — the §36-calc gap closed, §39.1 #14 designed (2026-08-01). The billing pipeline was designed but the math never was; both now canonical on Billing. FR-087 (calculation): only Running snapshots bill; one snapshot = one pod-minute; a missed snapshot bills nothing — the under-billing bias is deliberate (outages can only under-charge). Rate is calendar-month normalized — size_monthly ÷ (days_in_month × 1440) — so a 24/7 pod bills exactly list price every month (replaces the draft's fixed-43,200 "≈/mo" rate). Priced at Starbase ingest: cents resolved in the same transaction as the idempotent raw insert (PK conflict → no increment, FR-081-safe); usage_totals.quantity → amount_cents (fractional NUMERIC, rounded half-up once at invoice) — settling the missing-tier-dimension inconsistency (one quantity per category couldn't price mixed instance sizes); Shuttle payloads carry no pricing; raw rows stay money-free (the recompute/dispute path). Dedicated-DB cents materialize from the FR-076 ledger into usage_totals at invoice time — the invoice reads one table. Egress deferred at MVP (absorbed as COGS; category reserved; measurement lands with the §39.3 #71 audit trail — rejected billing-reads-ClickHouse now: billing stays Postgres-only, the access-log buffer drops first under pressure, Hobby rows expire at 7-day TTL; COGS check → Financial Model). Storage: inside tier prices. Invoice: max(0, round(Σ) − credit); totals under Stripe's US$0.50 minimum charge (verified 2026-08-01) roll forward. #14 Stripe lifecycle: one Stripe Customer per workspace; card required at first paid action (#58 gate); monthly cron → auto_advance invoice; webhook set (invoice.paid / payment_failed / payment_method.attached / subscription.*) signature-verified + deduped via a new stripe_events table; dunning = Smart Retries (verified: default 8 attempts / 2 weeks) → final failure sets suspended through desired state (halts autoscaled services, §36.3), payment success auto-clears; plan changes next-cycle, refunds manual; GET /workspaces/{id}/invoices. §13 Stripe-webhook bullet + §14 cron bullet reconciled (the stale "handles proration" claim removed); CLAUDE.md FR range → 087. Design record: specs/2026-07-11-remaining-items-proposals-PENDING-REVIEW.md (§36-calc + #14 sections, amended per review). |
This is a living document. Pages will be expanded as components are designed and built.