Skip to content

Shuttle Observability

§26.1 Shuttle Self-Observability

  • /healthz — Liveness probe. 200 if Manager is running.
  • /readyz — Readiness probe. 200 when Informer cache is synced and Starbase contacted at least once.
  • /metrics — Prometheus endpoint. Standard controller-runtime metrics plus custom: starform_snapshots_sent_total, starform_snapshots_failed_total, starform_capacity_reports_sent_total, starform_reconcile_errors_total, starform_starbase_request_duration_seconds, starform_informer_last_event_timestamp_seconds{resource}, starform_informer_watch_errors_total{resource}, starform_status_reports_sent_total, starform_status_reports_failed_total, starform_gc_breaker_tripped (gauge), starform_orphaned_resources (gauge), starform_desired_state_regressions_total, starform_snapshot_buffer_depth (gauge — alerts before the FR-081 replay buffer overflows).

Why starform_informer_last_event_timestamp_seconds matters: the Snapshot Runnable derives billing from the Informer cache. If a watch stream silently dies and fails to reconnect cleanly, the cache freezes and Shuttle bills off a stale view of reality without any reconcile error. This metric is a per-resource-type gauge of the unix timestamp of the last received watch event; alerts fire on time() - starform_informer_last_event_timestamp_seconds{resource="pods"} > 300s. Pairs with starform_informer_watch_errors_total to distinguish silent staleness from flapping reconnects.

starform_gc_breaker_tripped and starform_desired_state_regressions_total are the two loud-failure signals of the deletion-safety design (§20.3, §27) — both page via the platform plane (§35.5).

§26.2 Customer Workload Observability

Shuttle does NOT ship customer logs or metrics. One autonomous agent handles this:

Tool Type Role
OTel Collector (agent) DaemonSet Tails app-pod stdout files (filelog), receives Envoy Gateway access logs (EG's OTel sink), scrapes kubelet/cAdvisor (CPU, memory, network); stamps tenant identity (k8sattributes pod labels · route metadata) and ships to the regional gateway (v2 Collect)
OTel Collector (gateway) Process on the regional telemetry droplet Validates the per-cluster ingest tokens (FR-066), batches, and is the sole ClickHouse writer

Log flow: Customer Pod stdout → node filesystem → OTel agent (filelog + k8sattributes) → regional OTel gateway → ClickHouse otel_logs (tenant-first sort: project_id, environment, service_id)

Metric flow: Envoy access logs (latency, RPS, error rate, throughput — one row per request → otel_http) + cAdvisor meters (CPU, memory, network → otel_metrics_*) → OTel agent → regional OTel gateway → ClickHouse. Identity is columns on every row; workspace-boundary queries resolve server-side via the project→workspace map — workspace_id is not carried on rows (FR-064). See v2 Read & query for the 7 metrics as SQL.

Shuttle's only role: apply the standard label set (§24) to every customer resource; the OTel Collector uses these labels automatically.

§26.3 Who Installs the Telemetry Agents

Customer clusters are self-managed Talos — the whole stack below is self-installed at bootstrap (nothing is bundled; MVP = manual talosctl, post-MVP = Shipyard, Kubernetes Runtime). The provisioning sequence:

  1. Create the Talos cluster: DO API provisions droplets with the Talos machine config (gVisor extension) as cloud-init user-data; run the one-time etcd bootstrap. Enable the region's registry↔cluster integration — the registry-<name> pull secret is maintained in every namespace and Shuttle's rendered ServiceAccounts reference it in imagePullSecrets (§20.2), since custom SAs don't inherit the default SA's pull secrets
  2. Install Cilium (CNI, overlay + kube-proxy replacement; SYS_MODULE dropped, KubePrism :7445)
  3. Install DO CCM (DO_CLUSTER_VPC_ID) and DO CSI (do-block-storage)
  4. Install metrics-server and kube-state-metrics via Helm
  5. Install Envoy Gateway via Helm
  6. Install the OTel Collector and Grafana Alloy via Helm (OTel Collector: DaemonSet agent — filelog tail + Envoy access-log receiver + node-pinned cAdvisor scrape, k8sattributes identity, two authed doors to the regional gateway, v2 Collect; Grafana Alloy → Grafana Cloud with the platform + substrate paging keep-list — the chart bundles node-exporter, its own KSM disabled, §35.5)
  7. Install Shuttle (deploy/ manifests) and register the cluster as active

Shuttle contains no Helm logic, no collector config, no Alloy config.

The regional telemetry store is not part of cluster bootstrap. The per-region ClickHouse VM droplet (with its OTel Collector gateway) is provisioned out-of-band (Terraform + cloud-init/systemd; bring-up order: retention dictionary → tables → users → gateway, v2 Bootstrap), once per region, independent of the K8s cluster lifecycle — a cluster can be rebuilt without touching telemetry data. Backups are DO volume snapshots (SOC2, §39.2).

Cross-references

Collection, store & queries → v2 Collect / v2 Read & query · platform self-monitoring via Grafana Alloy → §35.5 · the labels the OTel Collector consumes → §24.1 · billing derived from snapshots → Billing · the Snapshot Runnable behind the billing feed → §19.3.