Skip to content

v2 · Bootstrap — bring-up order

In plain words

Stand the pieces up bottom-up. The store goes first — nothing can write telemetry until it exists. Then the retention dictionary and the tables (the collectors are told not to create tables, and a table's retention lookup needs the dictionary present first). Then the gateway that writes the store, then the agents that feed the gateway, and last you point Envoy at an agent. Finally, send one request and check it lands.

The v2 telemetry tier is one stateful store per region — ClickHouse. No VictoriaMetrics droplet, no vmagent, no Vector.

flowchart LR
  classDef built fill:#3434DC22,stroke:#3434DC,color:#5B5EE8;
  classDef third fill:transparent,stroke:#808080,color:#808080;
  classDef store fill:#3434DC0f,stroke:#3434DC,color:#5B5EE8,stroke-dasharray:6 4;

  S1["1 · ClickHouse droplet<br/>Terraform, out-of-band"]:::store
  S2["2 · dict + schema + users"]:::store
  S3["3 · OTel gateway (region)"]:::third
  S4["4 · OTel agent (per cluster)"]:::third
  S5["5 · Envoy access-log sink"]:::third
  S6["6 · verify end-to-end"]:::built

  S1 --> S2 --> S3 --> S4 --> S5 --> S6
Diagram — v2 bring-up. Store → schema → gateway collector → node agents → Envoy access-log sink → verify. Dashed = ClickHouse (the one store).

SRE steps

  1. ClickHouse droplet — first, because everything downstream writes to it. Provisioned per region out-of-band (Terraform + cloud-init), once, independent of the cluster lifecycle; backups are DO volume snapshots. (Same pattern as v1's stores — one droplet now, not two.)
  2. Retention dictionary → tables → users, in that order — each table's retention_days column calls dictGet and errors on attach if the dictionary is missing. Create proj_retention_src + the proj_retention dictionary, apply otel_http.sql + otel_logs.sql, then create the users (otel_ingest, starbase_read, starbase_write) and wire the Starbase→proj_retention_src upsert (Store). (The resource-metric ALTER waits for step 3 — the gateway's first start creates those tables.)
  3. Regional OTel gateway — before the agents, so they have somewhere to forward. Provision the per-cluster token file (/etc/otel/cluster-tokens) and the disk-queue dir (/var/lib/otelcol/queue), deploy otel-gateway.yaml on the telemetry droplet, confirm it connects to ClickHouse — its first start creates the otel_metrics_* tables; then run metrics-retention.sql (Store · step 3).
  4. Per-cluster OTel agent — the DaemonSet that feeds the gateway. Deploy otel-agent.yaml; it tails app logs, scrapes cAdvisor, and listens on :4317 for access logs.
  5. Envoy Gateway access-log sink — last, once the agent is listening: apply the EnvoyProxy telemetry config so requests start flowing to it.
  6. Verify — drive one request, then: SELECT count() FROM otel.otel_http WHERE Timestamp > now() - 60 returns > 0, and the latency query returns a row.
  7. Watchdogs — add three pipeline alerts to the platform plane (§35.5): unattributed-row rate (SELECT count() FROM otel.otel_logs WHERE project_id = '' AND Timestamp > now() - 300 — fast-dying pods that missed their identity stamp), gateway queue depth (the collector's own otelcol_exporter_queue_size metric, scraped by Alloy), and ClickHouse disk % (page before full).

Install-order note

Within a customer cluster, the OTel Collector agent replaces v1's Fluent Bit and vmagent in the §26.3 bring-up sequence. Everything else is unchanged — Cilium, DO CCM/CSI, metrics-server, kube-state-metrics, Envoy Gateway, Shuttle, and Grafana Alloy (the platform plane, §35.5, retained as-is).

Note: kube-state-metrics leaves the customer-metric path — v2 attributes via the OTel k8sattributes processor, not v1's cAdvisor↔KSM join — so KSM stays only for the platform plane (Alloy's kube_job_*, §35.5).

Gotchas

  • Dictionary before tables, tables before collectors. A table's dictGet retention column errors on attach if proj_retention is missing, and the collectors run create_schema: false so an absent table drops inserts. Order: dictionary → tables → collectors.
  • The droplet is ClickHouse-only — don't carry the v1 VictoriaMetrics droplet into a v2 region; it has no consumer.
  • metrics-server stays — it feeds Kubernetes HPA (CPU/memory autoscaling) and is unrelated to the telemetry store; do not remove it with vmagent.
  • Maintenance = a short blackout, by design. A ClickHouse restart or upgrade pauses ingest (the gateway's disk queue holds it) and blanks dashboards for the window; customer workloads are untouched — Shuttle's loop doesn't depend on telemetry. Schedule it; don't fear it.