Skip to content

v2 · Read & query

In plain words

Getting data back out has one rule: a customer only ever sees their own. So the dashboard never talks to ClickHouse directly — it calls Starbase, and Starbase runs the query with the customer's identity taken from their logged-in session, forcing a WHERE project_id = … the request can't change or remove.

Once that filter is in place, the "7 metrics" and "logs" are just SQL over the tables: the four L7 metrics are math on the request rows (otel_http), CPU/memory/network come from the scraped rows (otel_metrics_*), and logs are a tail of otel_logs. There is no metrics database and no pre-computing — ClickHouse does the arithmetic at query time.

The isolation rule — read this first

How every query is made safe

1 · The read-only account. Starbase connects as a user that can only SELECT, only in the otel database. Even a bug in query-building can't write, drop, or reach another database. v2 has one front-door — this user — where v1 also needed a vmauth proxy for VictoriaMetrics.

Read-only telemetry user · clickhouse-read-user.sql — run once in ClickHouse on the telemetry droplet
CREATE USER starbase_read IDENTIFIED BY '<from-secret>';
GRANT SELECT ON otel.* TO starbase_read;   -- SELECT only, and only the otel database

2 · Inject the filter server-side. The tenant identity comes from the authenticated session and is passed as bound parameters — values handed to ClickHouse separately from the SQL text. That is the isolation boundary (FR-065): the client can't inject into it, and no request field sets the scope. Never build the filter by pasting strings into SQL.

Starbase query broker · Go — the tenant filter is injected from the session, not the request
// identity comes from the session — NOT from the HTTP request body
p := session.Tenant()  // { ProjectID, Environment, ServiceID }

rows, err := ch.Query(ctx, `
    SELECT toStartOfMinute(Timestamp) AS t, quantile(0.99)(duration_ms) AS p99
    FROM otel.otel_http
    WHERE project_id = {pid:String} AND environment = {env:String}
      AND Timestamp >= now() - INTERVAL 1 HOUR
    GROUP BY t ORDER BY t`,
    clickhouse.Named("pid", p.ProjectID),   // bound params — the client cannot widen the scope
    clickhouse.Named("env", p.Environment),
)

3 · Cross-region reads. With more than one region, Starbase reads the region that holds the environment's data over the private VPC peering, through the same read-only user — never a public endpoint. At single-region MVP this is a local read; the cross-region path is built but dormant until region two (FR-071, transport §35.4). The broker is written against a QueryTransport seam (§13) — the per-region transport switch: DO regions keep this direct read over the peering permanently; regions on clouds without DO peering read over Mass Relay — an agent on the telemetry droplet holding an outbound stream to Starbase, executing these same filtered queries locally (§39.3 #43). The store exposes no public endpoint on either path.

The 7 metrics — SQL over the rows

The seven numbers come from two tables. Four are computed from the request log (otel_http, one row per request); three are the machine's own meters (otel_metrics_*, scraped a few times a minute). Columns like duration_ms, response_code and project_id are computed at insert from the ingest attributes (Store) — the queries just use them. Every query below carries the injected tenant filter from step 2.

Metric From Computed as
Latency p50/p95/p99 otel_http quantile(0.5/0.95/0.99)(duration_ms)
Requests/sec otel_http count() ÷ window seconds
Error rate otel_http countIf(response_code >= 500) / count()
Throughput otel_http sum(bytes_sent + bytes_received) (egress + ingress)
CPU otel_metrics_* container_cpu_* rate
Memory otel_metrics_* container_memory_*
Network otel_metrics_* container_network_* rate

The queries

quantile gives a percentile (p99 = "99% of requests were faster than this"); count() over a minute is the request rate; error rate is the share of rows that ended >= 500; throughput sums the bytes.

Latency percentiles · Starbase query broker → ClickHouse (per-minute series)
SELECT toStartOfMinute(Timestamp) AS t,
       quantile(0.50)(duration_ms) AS p50,
       quantile(0.95)(duration_ms) AS p95,
       quantile(0.99)(duration_ms) AS p99
FROM otel.otel_http
WHERE project_id = {project_id:String}      -- injected by Starbase, not the client
  AND environment = {environment:String}
  AND service_id  = {service_id:String}
  AND Timestamp >= now() - INTERVAL 1 HOUR
GROUP BY t ORDER BY t;
RPS · error rate · throughput · Starbase query broker → ClickHouse (per-minute series)
SELECT toStartOfMinute(Timestamp) AS t,
       count() / 60.0                          AS rps,
       countIf(response_code >= 500) / count() AS error_rate,
       sum(bytes_sent + bytes_received)        AS throughput_bytes   -- egress + ingress
FROM otel.otel_http
WHERE project_id = {project_id:String} AND environment = {environment:String}
  AND Timestamp >= now() - INTERVAL 1 HOUR
GROUP BY t ORDER BY t;

Because identity is columns, a per-path breakdown is free — add path to the SELECT and GROUP BY. That is the drill-down v1 couldn't do without blowing up metric cardinality. CPU / memory / network read the same way from otel_metrics_*, filtered on the same tenant columns.

Logs — tail the rows

Live tail + search

Logs are the simplest read: newest rows for a service. The dashboard live-tail polls this on a short interval with a moving timestamp cursor (FR-049). otel_logs holds runtime logs; build logs live outside ClickHouse — live relay + DO Spaces archive (§16.6, 2026-07-09).

Live tail · Starbase query broker → ClickHouse (newest lines for a service)
SELECT Timestamp, SeverityText, Body
FROM otel.otel_logs
WHERE project_id = {project_id:String}      -- injected by Starbase, not the client
  AND environment = {environment:String}
  AND service_id  = {service_id:String}
  AND Timestamp >= now() - INTERVAL 5 MINUTE
ORDER BY Timestamp DESC
LIMIT 500;

Filter by level with AND SeverityText = {level:String}; full-text search with AND Body ILIKE {q:String}.

Gotchas

  • Never interpolate the identity into SQL text. Bound parameters are the isolation boundary; a CI test should confirm a crafted project_id in a request body cannot change a query's scope.
  • One front-door. v2 has only the read-only ClickHouse user — no vmauth proxy to run or secure.
  • Percentiles are approximate, on purpose. quantile() samples (error typically under 1% — invisible on a dashboard; quantileExact exists if a case ever demands it) and scans the window. When a busy tenant's L7 queries slow, add the per-minute rollup — the exact SQL is written down in Store · Rollups — not before.
  • Exclude upgraded connections from latency. Websocket/SSE rows are logged at connection close with hours-long durations that wreck p99 — add AND response_code != 101 to the latency query.
  • A Hobby user picking a 30-day window sees 7 days of data — Stardeck should say "your plan retains 7 days", not render what looks like an outage.
  • Log search stays a scan (within the tenant's slice). Fine at MVP; when search slows, add a tokenbf_v1 bloom-filter index on Body — one ALTER, the standard ClickHouse lever.
  • cAdvisor network under gVisor is unverified — customer pods run in the gVisor netstack, so validate container_network_* against a known load before trusting the network panel (tracked with gVisor overhead, §39.3 #62).
  • No PromQL. A future RPS/latency autoscaler (§39.1 #9, deferred) reads these tables through a ClickHouse adapter; CPU/memory autoscaling via metrics-server is unaffected.

Reference

Tables read here: Store & retention · who fills them: Collect. Server-side filter FR-065; private read path FR-071; live streaming FR-049. Tenant key + 7 metrics owned by §24.1 / CLAUDE §7.