v2 · Collect — the one agent

In plain words

v2 has a single agent — the OpenTelemetry Collector — where v1 ran three (vmagent, Fluent Bit, Vector). It does three jobs at once: tails app-log files, receives Envoy's request logs, and scrapes the machine's CPU/memory/network meters. It tags every row with which customer it belongs to and ships it to ClickHouse.

It runs in two tiers. On every node a light agent collects and stamps identity. In the region a gateway batches everything and is the only thing that writes ClickHouse — many tiny inserts choke ClickHouse ("too many parts"), so the gateway holds them and writes in bulk (the job Vector held in v1). App logs and request logs enter the gateway through separate doors, so each lands in its own table.

flowchart LR
  classDef built fill:#3434DC22,stroke:#3434DC,color:#5B5EE8;
  classDef third fill:transparent,stroke:#808080,color:#808080;
  classDef store stroke-dasharray:4 3,stroke:#808080,color:#808080;

  L["app logs"]:::third
  A["Envoy access logs"]:::third
  M["cAdvisor meters"]:::third
  AG["OTel agent<br/>per node · stamps identity"]:::built
  GW["OTel gateway<br/>on the telemetry droplet · batches"]:::built
  CH[("ClickHouse")]:::store

  L --> AG
  A --> AG
  M --> AG
  AG -->|per-cluster token · two doors| GW --> CH

Diagram — Collect. One agent, three inputs, identity stamped on the way in; the regional gateway batches and is the sole ClickHouse writer. App logs and access logs use separate doors (ports) so each lands in its own table.

Three signals, one agent

Signal	Source	How it's received	Gateway door	Lands in
App logs	pod stdout files	filelog (tail)	:4317	otel_logs
L7 request logs	Envoy Gateway	otlp (EG's OTel access-log sink)	:4319	otel_http
Resource meters	cAdvisor / kubelet	prometheus (scrape)	:4317	otel_metrics_* alpha exporter

How to build it

Deploy two things — the per-node agent and the regional gateway — then wire identity and auth.

1 · The node agent — three receivers. filelog tails the container log files — the container operator parses the runtime format, stitches split lines back together, and extracts k8s.namespace.name / k8s.pod.name from the file path (that's how a file gets matched to its pod). The exclude list keeps the agent from tailing platform/system pods — otherwise a ClickHouse outage makes the agent ship its own error logs in a feedback loop. otlp receives Envoy's request logs; prometheus scrapes cAdvisor — over HTTPS with the ServiceAccount token, pinned to its own node (without the keep relabel, every agent scrapes every node → N× duplicate rows):

OTel Collector node agent · receivers · otel-agent.yaml — runs as a DaemonSet on every customer node (NODE_NAME injected via fieldRef spec.nodeName)

receivers:
  filelog:                            # tail app-pod stdout → otel_logs
    include: [/var/log/pods/*/*/*.log]
    exclude:                          # never tail platform/system pods (self-logging feedback loop)
      - /var/log/pods/kube-system_*/*/*.log
      - /var/log/pods/observability_*/*/*.log      # this collector itself
      - /var/log/pods/starform-system_*/*/*.log    # Shuttle · Envoy Gateway · Alloy
    include_file_path: true
    operators:
      - type: container               # parse runtime format + stamp k8s.pod.name / k8s.namespace.name from the path
  otlp:                               # receive Envoy access logs → otel_http
    protocols: { grpc: { endpoint: 0.0.0.0:4317 } }
  prometheus:                         # scrape cAdvisor CPU/mem/net → otel_metrics_*
    config:
      scrape_configs:
        - job_name: cadvisor
          scheme: https
          authorization: { credentials_file: /var/run/secrets/kubernetes.io/serviceaccount/token }
          tls_config: { insecure_skip_verify: true }   # or pin the kubelet CA
          kubernetes_sd_configs: [{ role: node }]
          relabel_configs:
            - source_labels: [__meta_kubernetes_node_name]   # scrape ONLY this node —
              regex: ${env:NODE_NAME}                        # else every agent scrapes every node
              action: keep
          metrics_path: /metrics/cadvisor

2 · Stamp identity, then forward — three pipelines, two doors. For pod-origin data (app logs, meters), k8sattributes looks the pod up by the name/namespace the container parser extracted (file logs have no network connection to match on) and copies its starform.io/* labels onto every row. Access logs skip k8sattributes — their identity rides the route (step 3) — and a small filter drops rows for requests that matched no route (health probes, 404s, IP scans). Each pipeline exports to its own gateway door, carrying the per-cluster token (FR-066):

OTel Collector node agent · identity + pipelines · otel-agent.yaml — same DaemonSet, on every customer node

processors:
  k8sattributes:
    pod_association:                  # match file logs to their pod by parsed name/namespace (no connection IP exists)
      - sources:
          - { from: resource_attribute, name: k8s.pod.name }
          - { from: resource_attribute, name: k8s.namespace.name }
    extract:
      labels:                         # pod starform.io/* labels → tenant identity
        - { tag_name: project_id,  key: starform.io/project-id }
        - { tag_name: service_id,  key: starform.io/service-id }
        - { tag_name: environment, key: starform.io/environment }
        - { tag_name: cluster_id,  key: starform.io/cluster-id }
  filter/unrouted:                    # drop access-log rows with no route (health probes, 404s, scans)
    logs:
      log_record:
        - 'attributes["route_name"] == ""'
  batch: {}
exporters:
  otlp/app:                           # app logs + meters → gateway :4317, over the VPC
    endpoint: otel-gateway.region.internal:4317
    headers: { authorization: "Bearer ${env:CLUSTER_BEARER}" }   # per-cluster ingest token (FR-066)
    tls: { insecure: false }
  otlp/access:                        # Envoy request logs → gateway :4319 (their own door → their own table)
    endpoint: otel-gateway.region.internal:4319
    headers: { authorization: "Bearer ${env:CLUSTER_BEARER}" }
    tls: { insecure: false }
service:
  pipelines:
    logs/app:    { receivers: [filelog],    processors: [k8sattributes, batch],   exporters: [otlp/app] }
    logs/access: { receivers: [otlp],       processors: [filter/unrouted, batch], exporters: [otlp/access] }   # no k8sattributes — identity rides the route
    metrics:     { receivers: [prometheus], processors: [k8sattributes, batch],   exporters: [otlp/app] }

3 · Identity on request logs — from the route, not a label. The access log is written by Envoy, not a pod, so it can't be label-stamped. Envoy Gateway exposes the HTTPRoute's name and namespace as route metadata (it does not propagate arbitrary labels), and our naming already encodes identity there: name = <project32><service32>-<env> (§20.2), namespace = proj-<project_uuid>. Read them with the CEL operator; the Store DDL splits the name into the tenant columns once, at insert:

Envoy Gateway access logs → OTel · EnvoyProxy CRD spec.telemetry.accessLog — kubectl apply, one per gateway

spec:
  telemetry:
    accessLog:
      settings:
        - format:
            type: JSON                 # structured fields, nothing to re-parse
            json:
              duration:       "%DURATION%"          # → latency
              response_code:  "%RESPONSE_CODE%"      # → error rate
              response_flags: "%RESPONSE_FLAGS%"     # → debugging (UH, LR, …)
              bytes_sent:     "%BYTES_SENT%"         # → throughput (egress)
              bytes_received: "%BYTES_RECEIVED%"     # → throughput (ingress)
              method: "%REQ(:METHOD)%"
              path:   "%REQ(X-ENVOY-ORIGINAL-PATH?:PATH)%"
              # tenant identity from the HTTPRoute's route metadata (name/namespace), NOT labels:
              route_name: "%CEL(xds.route_metadata.filter_metadata['envoy-gateway']['resources'][0]['name'])%"
              namespace:  "%CEL(xds.route_metadata.filter_metadata['envoy-gateway']['resources'][0]['namespace'])%"
          sinks:
            - type: OpenTelemetry     # → the node agent's otlp receiver, on :4317
              openTelemetry: { host: otel-agent.observability.svc.cluster.local, port: 4317 }

Verify once at build (5 minutes): EG's docs don't pin (a) whether the JSON fields land in the OTLP attributes or the body, or (b) the exact CEL path on your pinned EG version. Send one request and look at the row: fields in LogAttributes → done; in Body → flip the Store expressions to JSONExtract(Body, …) (Store gotcha). Envoy's own logger maps a JSON struct to attributes, so attributes-first is the likely answer.

4 · The regional gateway — the sole writer. Two receivers (one per door) that validate the per-cluster tokens via the bearertokenauth extension; an on-disk queue (file_storage) so a ClickHouse outage or gateway restart doesn't drop data; batching (the "too many parts" defence); and one ClickHouse exporter per table. create_schema: true on the app exporter creates only the missing otel_metrics_* tables — our pre-created tables win (IF NOT EXISTS):

OTel Collector regional gateway · otel-gateway.yaml — one per region, a process on the telemetry droplet next to ClickHouse

extensions:
  bearertokenauth:                    # validates per-cluster ingest tokens (FR-066)
    scheme: Bearer
    filename: /etc/otel/cluster-tokens   # one token per line — one per cluster
  file_storage:                       # on-disk queue: survives ClickHouse outages + gateway restarts
    directory: /var/lib/otelcol/queue
receivers:
  otlp/app:                           # app logs + resource meters
    protocols: { grpc: { endpoint: 0.0.0.0:4317, auth: { authenticator: bearertokenauth } } }
  otlp/access:                        # Envoy request logs
    protocols: { grpc: { endpoint: 0.0.0.0:4319, auth: { authenticator: bearertokenauth } } }
processors:
  batch: { timeout: 5s, send_batch_size: 10000 }   # bulk inserts → avoids too-many-parts
exporters:
  clickhouse:                         # app logs → otel_logs · meters → otel_metrics_*
    endpoint: tcp://clickhouse.region.internal:9000
    database: otel
    username: otel_ingest
    password: ${env:CH_INGEST_PASSWORD}
    async_insert: true
    create_schema: true               # creates ONLY the missing otel_metrics_* tables (IF NOT EXISTS)
    logs_table_name: otel_logs
    sending_queue: { storage: file_storage }
  clickhouse/http:                    # request logs → otel_http
    endpoint: tcp://clickhouse.region.internal:9000
    database: otel
    username: otel_ingest
    password: ${env:CH_INGEST_PASSWORD}
    async_insert: true
    create_schema: false              # otel_http is ours — declared on the Store page
    logs_table_name: otel_http
    sending_queue: { storage: file_storage }
service:
  extensions: [bearertokenauth, file_storage]
  pipelines:
    logs/app:    { receivers: [otlp/app],    processors: [batch], exporters: [clickhouse] }
    logs/access: { receivers: [otlp/access], processors: [batch], exporters: [clickhouse/http] }
    metrics:     { receivers: [otlp/app],    processors: [batch], exporters: [clickhouse] }

5 · Verify. After one request and one log line, rows land in the right tables — run against the telemetry droplet's ClickHouse: SELECT count() FROM otel.otel_http WHERE Timestamp > now() - 60 returns > 0, and the same on otel.otel_logs — and not vice versa (the doors are split).

Gotchas & what to verify

• Attribution is the route name, read cleanly — not the v1 parse. No per-scrape envoy_cluster_name regex; the HTTPRoute name is a stable Gateway-API field, split once in the Store DDL. Confirm the CEL path on your pinned EG version (step 3's 5-minute test).
• ID forms match at the store. Route names are hyphen-stripped 32-hex; pod labels carry hyphenated UUIDs — the Store expressions strip hyphens so otel_http and otel_logs share the same project_id form.
• Alpha/beta — run, watch, fall back. L7 metrics ride the logs path (beta, run in production by ClickStack/SigNoz); only the cAdvisor meters ride the alpha metrics path. The ladder if it misbehaves: pin the last-known-good Collector → switch to SigNoz's production-hardened exporter fork (Apache-2.0) → worst case swap only the writer (Vector's ClickHouse sink as the gateway; the agents don't change).
• Outage behavior, in order. ClickHouse down → the gateway queues on disk (file_storage) → agents back off; app logs survive naturally (the files sit on the node until rotation), but Envoy's access-log buffer is the smallest — L7 rows drop first in a long outage. Size /var/lib/otelcol/queue for the outage window you want to ride out.
• A pod that dies fast can miss its stamp. If k8sattributes hasn't seen the pod yet, rows arrive without project_id — invisible to tenant queries, expired by the 90-day default. Watch the unattributed-row rate (Bootstrap · watchdogs).
• No per-tenant log limit at MVP. One tenant logging at full speed fills the shared disk — watch per-project bytes/day (one SQL on otel_logs) and alert; a hard quota is post-MVP.
• Long-lived connections (websockets/SSE) log at close — an hour-long connection shows up as an hour of "latency." Exclude/flag upgrades in the latency panel (Read & query).

Reference

Where it lands: Store & retention · read it back: Read & query. HTTPRoute name contract §20.2; label set §24.1; ingest auth FR-066. Design record: specs/2026-07-07-single-store-clickhouse-telemetry-design.md.