v1 · Overview — the whole system in plain words¶
v1 — superseded by v2 (ClickHouse only)
The two-store design (VictoriaMetrics for metrics + ClickHouse for logs), kept as the reference and
decision trail. This page is the plain-words overview; the chapters that follow build each piece. The
load-bearing contracts it depends on (the HTTPRoute route-name format, the starform.io/* labels, the
tenant key, the server-side query filter) are restated inline per chapter and collected in the
Reference. Owned by §20.2 / §24 / §35 — those pages win on conflict.
The components — what each piece is¶
Everything obeys one rule — Starbase decides, Shuttle applies; every change flows through desired state. The pieces, grouped by where they run:
The component map. Brand-blue = Starform-built · gray = third-party · darker = a data store · dashed = external SaaS. The telemetry path threads bottom-up — cluster collectors (Fluent Bit, vmagent) → regional stores → Stardeck.
The mental model¶
The mental model
Monitoring is two separate pipes — one for metrics, one for logs — that never touch until they meet on the dashboard. Alongside them runs a third plane (Grafana Cloud, external) that watches Starform's own machinery for the SRE team.
Nobody "queries a new pod." Metrics are pulled on a timer, logs are tailed off the machine, and traces (later) are pushed by the app. Three signals, three habits.
Diagram 1 — VPC topology. The regional VPC (one per region) holds the customer cluster and its
telemetry VM stores, so ingest is intra-VPC at $0. The central control plane (its own VPC) peers
in to read each region's stores — only through the authed front-door, in two parts: vmauth (a
proxy deployment) for VictoriaMetrics, and a read-only ClickHouse user for logs. The stores stay
private; Starbase injects the tenant filter. One VPC + one peering per region (~12 ≪ 50). A dedicated
Grafana Alloy agent ships platform series to Grafana Cloud (external). Arrow colour:
blue = metrics pipe · amber = logs
pipe.
The lifecycle in one breath¶
A pod is born → on the next timer tick its meters get read and its log file gets tailed (no one contacts
the pod) → each signal is tagged with project · environment · service → shipped over the private
network → stored (metrics in VictoriaMetrics, logs in ClickHouse) → the dashboard shows it through a
filter the user can't remove. A pod dies → readings stop, series age out by retention, the log file
disappears. Nothing to clean up.
| Step | What happens | |
|---|---|---|
| 01 | Born | Shuttle creates the pod, its labels, and its HTTPRoute. |
| 02 | Read & tailed | Next tick: vmagent scrapes kubelet + Envoy; Fluent Bit picks up the new log file. |
| 03 | Tagged | Identity attached — by join, by parse, or by label. |
| 04 | Shipped | Over the regional VPC (intra-VPC, $0) with a per-cluster token. |
| 05 | Stored | Metrics → VictoriaMetrics. Logs → ClickHouse. |
| 06 | Served | API forces a tenant filter; Stardeck renders it. |
What's in, and what's a stub¶
| Area | Status | Notes |
|---|---|---|
| Customer metrics — latency, RPS, throughput, error rate, CPU, memory, network | In MVP | Seven signals from two sources (Diagram 3). |
| Customer logs — runtime + build, live tail | In MVP | Fluent Bit → Vector aggregator → ClickHouse. |
| Platform self-monitoring — alerts on Starform's own components | In MVP | Grafana Cloud (external), fed by a dedicated Grafana Alloy agent. |
| Distributed tracing | Stub | Push model, needs app instrumentation. Future plane — sketched, not built. |
| Per-tier metric retention | Deferred | MVP = global 90d. Logs already tier cleanly. |
Next: Chapter 1 · Architecture & components — namespaces, the HTTPRoute name, and the label set, with the three identity conventions everything below depends on.