Platform self-monitoring (the SRE plane)¶

Shared section — referenced by both v1 and v2

Platform self-monitoring is independent of the customer-telemetry store (identical for v1 and v2) — it watches Starform's own components, not customer data. It restates the GDPR labeldrop rule and the Shuttle-informer staleness signal inline so it stands alone. Owned by PRD §35.5 / §26.1 — the PRD wins on conflict.

In plain words

A dedicated Grafana Alloy agent in every cluster — each customer cluster and the DOKS platform cluster — scrapes the health of Starform's own pieces and, on self-managed Talos, the Kubernetes substrate itself (etcd, kube-apiserver, Cilium, node disks), and ships a small paging keep-list plus a per-cluster liveness series to Grafana Cloud (external SaaS). There, Grafana Alerting checks rules and OnCall pages the SRE — and a cluster gone silent is itself the page. It runs as its own agent, separate from the customer pipeline, for two reasons: an external watcher survives the infra dying (which a plane co-located in your own cluster cannot), and an independent pipeline means a customer-pipeline problem can't blind you to the platform's own health. MVP-only; revisit post-MVP (decision record: specs/2026-07-05-substrate-observability-design.md).

How to build it

Two pieces: an agent that ships platform health out, and the rules that act on it.

1 · Deploy Grafana Alloy per cluster (Grafana's k8s-monitoring Helm chart — it bundles the node-exporter DaemonSet; disable its kube-state-metrics, we run our own). It scrapes Starform-component series — Shuttle /metrics, Envoy gateway-self, node-exporter, KSM platform objects — and the substrate we own on Talos: etcd, kube-apiserver, kube-scheduler, kube-controller-manager, cilium-agent/operator, plus etcd-backup job age via KSM kube_job_*. Only the paging keep-list (~200–500 series/cluster) is remote_writen to Grafana Cloud — deep substrate history is deliberately not stored at MVP (§39.3 #65); the customer pipeline separately ships customer telemetry to the customer ClickHouse store. GDPR: drop customer identifiers before they leave the cluster (the labeldrop below), or use an EU-residency stack.

What ships, per job — the keep-list is exactly the rule inputs below plus up; nothing else leaves the cluster:

Job	Series kept	Pages when
`etcd` (static targets — see config)	`up` · `etcd_server_has_leader` · `etcd_server_proposals_failed_total` · `etcd_disk_wal_fsync_duration_seconds_bucket` · `etcd_mvcc_db_total_size_in_bytes`	no leader · proposals failing · slow fsync · DB near quota
`apiserver`	`up` · `apiserver_request_total` (5xx only, non-5xx dropped) · `apiserver_client_certificate_expiration_seconds_bucket`/`_count`	down · 5xx rate · client cert expiring
`kube-scheduler` / `kube-controller-manager`	`up`	down
`cilium` (agent `:9962` + operator `:9963`)	`up` · `cilium_unreachable_nodes`	CNI down · nodes unreachable
`node-exporter`	`node_filesystem_{avail_bytes,size_bytes,files,files_free}` (real mounts only) · `node_memory_MemAvailable_bytes` · `node_memory_MemTotal_bytes`	disk full · inodes low · memory pressure
`kube-state-metrics`	`kube_job_status_completion_time` (etcd-backup jobs)	backup stale
`shuttle`	`starform_` (incl. informer staleness) · `up` — `up{job="shuttle"}` doubles as the per-cluster liveness series*	informer stale · cluster silent
`envoy-self`	`envoy_{server,control_plane,listener_manager,cluster_manager}_*` (bounded, per-instance — §35.2)	gateway unhealthy

Grafana Alloy · config.alloy (River)

// Alloy (River): keep only platform jobs, strip tenant labels (GDPR), remote_write to Grafana Cloud
discovery.kubernetes "pods" { role = "pod" }

prometheus.scrape "platform" {
  targets    = discovery.kubernetes.pods.targets
  forward_to = [prometheus.relabel.platform.receiver]
}

// Talos runs etcd as a HOST service, not a pod — pod discovery can't see it. Static targets on
// the 3 control-plane IPs; prerequisite: machine config exposes etcd's metrics listener (:2381).
prometheus.scrape "etcd" {
  job_name   = "etcd"
  targets    = [{ "__address__" = "10.x.x.1:2381" }, { "__address__" = "10.x.x.2:2381" }, { "__address__" = "10.x.x.3:2381" }]
  forward_to = [prometheus.relabel.platform.receiver]
}

prometheus.relabel "platform" {
  forward_to = [prometheus.remote_write.grafana.receiver]
  rule { source_labels = ["job"], regex = "shuttle|node-exporter|kube-state-metrics|envoy-self|etcd|apiserver|kube-scheduler|kube-controller-manager|cilium", action = "keep" }
  // THE keep-list — exactly the rule inputs + up; this line is the Grafana Cloud cost model
  rule {
    source_labels = ["__name__"]
    regex = "up|etcd_server_has_leader|etcd_server_proposals_failed_total|etcd_disk_wal_fsync_duration_seconds_bucket|etcd_mvcc_db_total_size_in_bytes|apiserver_request_total|apiserver_client_certificate_expiration_seconds_(bucket|count)|cilium_unreachable_nodes|node_filesystem_(avail_bytes|size_bytes|files|files_free)|node_memory_(MemAvailable|MemTotal)_bytes|kube_job_status_completion_time|starform_.*|envoy_(server|control_plane|listener_manager|cluster_manager)_.*"
    action = "keep"
  }
  rule { source_labels = ["__name__","code"], separator = ";", regex = "apiserver_request_total;[234]..", action = "drop" }   // keep 5xx only — the full counter is the cardinality trap
  rule { source_labels = ["fstype"], regex = "tmpfs|overlay|squashfs", action = "drop" }   // real mounts only
  rule { regex = "project_id|service_id|workspace_id", action = "labeldrop" }   // never ship customer IDs
}

prometheus.remote_write "grafana" {
  endpoint {
    url = "https://prometheus-prod-NN.grafana.net/api/prom/push"
    basic_auth {
      username = "<grafana-cloud-instance-id>"
      password = "<grafana-cloud-token>"   // inject from a Secret, not inline
    }
  }
}

2 · Load the platform alert rules into Grafana Cloud (mimirtool) or author them in Grafana Alerting; Grafana OnCall pages the SRE. Watch Starbase API/Worker, control-plane Postgres, the telemetry droplet — ClickHouse + the OTel gateway; droplet scrape lands pre-launch, §39.2 #64 — Envoy health, DO Load Balancers, Shuttle's /metrics — and the substrate: etcd (leader, fsync p99, DB size), apiserver (up, 5xx rate, cert expiry), cilium-agent, node disk/inodes/memory, etcd-backup age, and per-cluster liveness alerted on absence:

Grafana alert rules · mimirtool

groups:
  - name: starform-platform
    rules:
      - alert: ShuttleInformerStale       # billing meters off a frozen view
        expr: time() - starform_informer_last_event_timestamp_seconds{resource="pods"} > 300
        for: 2m
        labels:      { severity: critical }
        annotations: { summary: "Shuttle informer stale on {{ $labels.cluster_id }}" }
      - alert: StarbaseAPIDown
        expr: up{job="starbase-api"} == 0
        for: 1m
        labels:      { severity: critical }
      - alert: ClusterSilent                # liveness by absence — a dark cluster pages by itself
        expr: count by (cluster_id) (up{job="shuttle"} offset 10m) unless count by (cluster_id) (up{job="shuttle"})
        labels:      { severity: critical }
  - name: starform-substrate              # the Kubernetes we own on Talos (thresholds: round numbers, tune in service)
    rules:
      - alert: EtcdNoLeader
        expr: etcd_server_has_leader == 0
        for: 1m
        labels:      { severity: critical }
      - alert: EtcdProposalsFailing        # leader churn or quorum loss
        expr: rate(etcd_server_proposals_failed_total[15m]) > 0
        for: 15m
        labels:      { severity: warning }
      - alert: EtcdFsyncSlow               # disk too slow for etcd; precursor to leader loss
        expr: histogram_quantile(0.99, sum by (le, cluster_id) (rate(etcd_disk_wal_fsync_duration_seconds_bucket[5m]))) > 0.1
        for: 10m
        labels:      { severity: warning }
      - alert: EtcdDBSizeHigh              # 80% of the 2 GiB default backend quota
        expr: etcd_mvcc_db_total_size_in_bytes > 1.6e9
        for: 30m
        labels:      { severity: warning }
      - alert: EtcdBackupStale             # the §39.2 #59 DR path is only real if this never fires
        expr: time() - max by (cluster_id) (kube_job_status_completion_time{job_name=~"etcd-backup.*"}) > 93600
        labels:      { severity: critical }
      - alert: ApiserverDown
        expr: sum by (cluster_id) (up{job="apiserver"}) == 0
        for: 1m
        labels:      { severity: critical }
      - alert: Apiserver5xxHigh            # absolute rate — the non-5xx counter is dropped by the keep-list
        expr: sum by (cluster_id) (rate(apiserver_request_total{code=~"5.."}[5m])) > 1
        for: 5m
        labels:      { severity: warning }
      - alert: ApiserverClientCertExpirySoon   # < 7 days (kube-prometheus pattern)
        expr: apiserver_client_certificate_expiration_seconds_count > 0 and histogram_quantile(0.01, sum by (le, cluster_id) (rate(apiserver_client_certificate_expiration_seconds_bucket[5m]))) < 604800
        labels:      { severity: warning }
      - alert: SchedulerDown
        expr: up{job="kube-scheduler"} == 0
        for: 5m
        labels:      { severity: warning }
      - alert: ControllerManagerDown
        expr: up{job="kube-controller-manager"} == 0
        for: 5m
        labels:      { severity: warning }
      - alert: CiliumAgentDown             # CNI down = cluster down
        expr: up{job="cilium"} == 0
        for: 5m
        labels:      { severity: critical }
      - alert: CiliumUnreachableNodes
        expr: cilium_unreachable_nodes > 0
        for: 10m
        labels:      { severity: warning }
      - alert: NodeDiskFull                # a full control-plane disk kills etcd
        expr: node_filesystem_avail_bytes / node_filesystem_size_bytes < 0.10
        for: 15m
        labels:      { severity: critical }
      - alert: NodeInodesLow
        expr: node_filesystem_files_free / node_filesystem_files < 0.05
        for: 15m
        labels:      { severity: warning }
      - alert: NodeMemoryPressure
        expr: node_memory_MemAvailable_bytes / node_memory_MemTotal_bytes < 0.10
        for: 10m
        labels:      { severity: warning }

Gotchas & what lives elsewhere

Nothing ships automatically. Grafana Cloud stores and evaluates only what the keep rules forward. The k8s-monitoring chart's defaults do not scrape Talos control-plane components — the etcd / apiserver / scheduler / controller-manager jobs are explicit scrape configs, and etcd needs its metrics listener exposed in the Talos machine config first (bootstrap prerequisite, §39.3 #26; build-time verifies in the design spec §5).
Watch the watcher's blind spot: alert on starform_informer_last_event_timestamp_seconds — if Shuttle's informer silently freezes, billing meters off a stale view with no reconcile error.
Grafana Cloud is MVP-only — the post-MVP option is self-hosting platform monitoring once the SRE team can operate it (§39.3 #42: shrink the keep-list toward the liveness series, then cancel). Either way, customer telemetry never goes to Grafana Cloud — only Starform-component series.
Talos machine logs are not this plane: node system logs (kubelet, machined) ship via Talos machine.logging.destinations → the regional OTel gateway → a platform ClickHouse table — never Grafana Cloud, since log bodies carry customer identifiers that can't be label-dropped (pre-launch, §39.2 #63).
The keep-list is the cost model: Grafana Cloud bills per active series; an unfiltered substrate scrape is ~40–70k series/cluster. Don't widen the keep rules casually — deep history belongs in the seeded regional store (§39.3 #65).

PRD reference & inlined contracts

Owned by §35.5 (platform self-monitoring), §26.1 (Shuttle self-obs); FR-069, SC-017. The GDPR labeldrop rule and the informer-staleness signal are restated above so this shared section stands alone — if they ever diverge, the PRD wins. Decision record: specs/2026-07-05-substrate-observability-design.md. Canonical map: Canonical Sources.