Self-Hosted Observability Architecture

Why put a collector in the middle?

Why not send straight from the application to Loki/Tempo/Mimir? There are several reasons:

1. Buffering and retry: if the backend is temporarily unavailable, the collector holds on to the data instead of losing it. The application doesn’t have to worry about this.
2. Enrichment: the collector can add metadata the application doesn’t have (node name, Kubernetes namespace, environment, cluster).
3. Filtering and sampling: you can decide which traces to keep, drop noisy logs, and aggregate metrics before sending them to storage. All in one central place.
4. Format conversion: the collector receives OTLP from the application and converts it to each backend’s native format (Loki, Prometheus remote-write, Tempo).
5. Single point of configuration: switching backends becomes a change to a single file.

Pipelines

Alloy is configured as pipelines. Each pipeline defines a processing chain. A simplified example for traces:

otelcol.receiver.otlp "default" {
  grpc { endpoint = "0.0.0.0:4317" }
  http { endpoint = "0.0.0.0:4318" }
  output {
    traces = [otelcol.processor.k8sattributes.default.input]
  }
}

otelcol.processor.k8sattributes "default" {
  output {
    traces = [otelcol.processor.batch.default.input]
  }
}

otelcol.processor.batch "default" {
  output {
    traces = [otelcol.exporter.otlp.tempo.input]
  }
}

otelcol.exporter.otlp "tempo" {
  client {
    endpoint = "tempo-distributor:4317"
    tls { insecure = true }
  }
}

Telemetry flows receiver → processors → exporter. Each component is an independent plugin, and you can build arbitrarily complex pipelines.

Span metrics

An important trick is the span metrics connector. It automatically turns traces into RED metrics (Rate, Errors, Duration). Every time an application sends a trace, the collector computes latency and an error count and exports them as metrics. This is what feeds the latency alerts later on.

otelcol.connector.spanmetrics "default" {
  histogram {
    explicit {
      buckets = ["50ms", "100ms", "250ms", "500ms", "1s", "2s", "5s", "10s"]
    }
  }
  dimension { name = "http.method" }
  dimension { name = "http.status_code" }
}

Without it, you’d have to manually instrument every endpoint to emit latency metrics. With the connector, you get them for free from the traces.

MinIO: self-hosted S3

Before talking about the backends, it’s worth pointing out where the data actually lives. Loki, Tempo, Mimir and Pyroscope were designed to use S3-compatible object storage for long-term storage. In cloud environments, the natural choice would be Amazon S3, GCS or Azure Blob. But since the whole stack is self-hosted, we run MinIO inside the Kubernetes cluster itself.

MinIO is an object storage server that is 100% compatible with the S3 API. To Loki, Tempo and Mimir, it’s indistinguishable from AWS S3: you just point the endpoint at MinIO’s internal service (minio.minio.svc.cluster.local:9000) and provide the access key/secret key.

storage_config:
  aws:
    s3: http://access-key:secret-key@minio.minio.svc:9000/loki-chunks
    s3forcepathstyle: true

The bucket setup is the usual one: each backend has its own bucket, created before the first install:

loki-chunks       # log chunks
loki-ruler        # Loki rules
mimir-tsdb        # metric blocks
mimir-ruler       # Mimir rules
tempo-traces      # traces
pyroscope         # profiles

Having MinIO in the same cluster brings two direct benefits: very low latency (traffic never leaves the cluster) and predictable cost (you only pay for the disk you provision, with no charges per GB transferred or per PUT/GET request like on AWS S3).

MinIO’s data is persisted on Kubernetes volumes (in our case, a StorageClass with Ceph RBD via CSI). That means even if the MinIO pod restarts, the data stays on disk. For high availability you can run MinIO in distributed mode (with erasure coding across multiple nodes), but for a small to medium cluster a single instance is enough.

From here on, whenever I say “S3”, read it as MinIO running inside the cluster. The interface is the same.

Loki for logs

Loki is “Prometheus for logs”. Instead of indexing the full content of each log line (like Elasticsearch does), it only indexes labels (service, namespace, severity). The log content is stored in compressed chunks in S3.

This brings two benefits:

• Much lower storage cost (compressed chunks in S3 vs an inverted index on disk).
• Controlled cardinality, because labels are few and bounded. You can’t use a request_id or trace_id as a label, or the index blows up.

For fields that change on every request (like trace_id), Loki has the concept of structured metadata, which is stored inside the chunk and searchable, but not indexed. You get the best of both: you can filter by trace_id without killing cardinality.

Loki’s golden rule: labels for low-cardinality things (up to a few hundred unique values), structured metadata for everything else.

Tempo for traces

Tempo is a trace backend that stores everything in S3. The clever part is that it has no index over span content. Instead, Tempo relies on the trace_id as the primary key and uses columnar formats (Parquet) for complex queries (TraceQL).

For queries that need to be fast, you can configure dedicated columns with span attributes that show up often in filters (in our case, business IDs). This turns the attribute into a native Parquet column instead of leaving it inside a generic JSON field.

Tempo also has what they call the metrics-generator, which plays a role similar to the collector’s spanmetrics: turning spans into metrics at ingestion time.

Mimir for metrics

Mimir is a distributed Prometheus with storage in S3. The API is 100% compatible with Prometheus, so any tool that speaks PromQL works without modification.

The main difference is scale. Traditional Prometheus runs as a single instance and keeps data on local disk. Mimir scales horizontally (you add replicas as you grow) and keeps months or years of retention in S3, instead of a few days.

We use the remote-write protocol to send metrics from Alloy to Mimir:

prometheus.remote_write "mimir" {
  endpoint {
    url = "http://mimir-gateway/api/v1/push"
  }
}

To understand which metric types exist (Counter, Gauge, Histogram, Summary) and how to pick the right one when instrumenting, I wrote a companion post: Prometheus metrics on /metrics and when to use each one.

Correlation between datasources

The coolest part of Grafana is automatic correlation across the 3 pillars. With the datasources configured correctly, you can:

• Logs → Traces: click a log and open the matching trace, using the trace_id in the structured fields.
• Traces → Logs: click a span and see the logs it produced, filtered by service.name and the span’s time window.
• Metrics → Traces: click a latency spike on a dashboard and open the traces from that period (Exemplars).

This is configured in Loki’s derivedFields and Tempo’s tracesToLogsV2:

- name: Loki
  jsonData:
    derivedFields:
      - name: trace_id
        matcherType: regex
        matcherRegex: '"trace_id":"([a-f0-9]+)"'
        url: '$${__value.raw}'
        datasourceUid: tempo
- name: Tempo
  jsonData:
    tracesToLogsV2:
      datasourceUid: loki
      tags:
        - key: service.name
          value: service_name

This correlation is what makes debugging in production so much faster. You go from a metric to the trace, and from the trace to the logs, in seconds.

Defining rules

An alert rule is a query that runs at defined intervals. If the result crosses the threshold for more than X minutes, the alert fires.

Example alert for p95 latency:

groups:
  - name: latency
    folder: Latency
    interval: 1m
    rules:
      - title: API checkout p95 > 5s
        condition: C
        data:
          - refId: A
            datasourceUid: prometheus
            model:
              expr: |
                histogram_quantile(0.95,
                  sum by (le) (
                    rate(traces_spanmetrics_duration_milliseconds_bucket{
                      service_name="api",
                      span_name="POST /checkout"
                    }[5m])
                  )
                )
          - refId: B
            datasourceUid: __expr__
            model:
              type: reduce
              expression: A
              reducer: last
          - refId: C
            datasourceUid: __expr__
            model:
              type: threshold
              expression: B
              conditions:
                - evaluator:
                    type: gt
                    params: [5000]
        for: 5m
        labels:
          severity: critical
        annotations:
          summary: 'Checkout p95 latency above 5s'

Notice that the alert uses traces_spanmetrics_duration_milliseconds_bucket, which is exactly the metric the collector generates from traces. Without instrumenting anything in the application, we can alert on latency per endpoint.

Contact points

Contact points define where the notification goes. We set up multiple channels for different severities:

contactPoints:
  - name: discord-alerts
    receivers:
      - type: discord
        settings:
          url: ${DISCORD_WEBHOOK_URL}
  - name: whatsapp-alerts
    receivers:
      - type: webhook
        settings:
          url: ${EVOLUTION_API_URL}
          httpMethod: POST
  - name: email-alerts
    receivers:
      - type: email
        settings:
          addresses: oncall@empresa.com

WhatsApp in particular is interesting because Grafana has no native integration, but you can use a webhook that calls an API like Evolution API (also self-hosted), which sends messages to a WhatsApp group.

Notification policies: routing by severity

This is the most important architectural part. We don’t want every alert going to every channel. A low-priority queue alert doesn’t need to wake anyone up at 3 AM.

The solution is to build a routing hierarchy based on labels:

policies:
  - receiver: discord-alerts
    group_by: [alertname, namespace]
    routes:
      - matchers: [notify=email-only]
        receiver: email-alerts
        repeat_interval: 168h
      - matchers: [severity=~"critical|emergency"]
        continue: true
        receiver: whatsapp-alerts
        mute_time_intervals: [nighttime]
      - matchers: [severity=~"critical|emergency"]
        continue: true
        receiver: email-alerts
      - matchers: [severity=~"critical|emergency"]
        receiver: discord-alerts

The logic is clearer visually:

flowchart TB
  Alerta["Alert fired"] --> Match{severity?}
  Match -->|warning| Discord["Discord"]
  Match -->|critical / emergency| FanOut[/"fan-out (continue: true)"/]
  Match -->|notify=email-only| EmailOnly["Email<br/>repeat_interval: 168h"]
  FanOut --> Discord
  FanOut --> Email["Email"]
  FanOut --> WppCheck{time of day?}
  WppCheck -->|01h-11h UTC| Mute["muted<br/>(mute time)"]
  WppCheck -->|outside the window| Whatsapp["WhatsApp"]

The logic is:

• By default, every alert lands in Discord (low noise, the team sees it when they’re online).
• If the severity is critical or emergency, it fires on 3 channels at once (Discord, Email, WhatsApp).
• WhatsApp has a mute time overnight for alerts that can wait, but Discord and Email keep firing.
• Alerts tagged with notify=email-only (e.g. infra warnings) only go to email, with a long repeat_interval.

continue: true is important: without it, the alert stops at the first match. With it, routing keeps evaluating the other policies.

Severity matters

The severity vocabulary we use is:

• warning: something needs attention, but it isn’t breaking production. Goes to Discord only.
• critical: something is broken or about to break. Notifies every channel.
• emergency: production is down. Notifies every channel, with no mute time.

The difference between critical and emergency comes down to whether the overnight mute time is respected. A database saturating at 3 AM can be critical (it waits until morning), but a site that’s down is emergency (it wakes up whoever needs to be woken up).

YAML anchors to avoid duplication

Since every alert rule has a repetitive structure (data, reduce expression, threshold), we use YAML anchors to reuse blocks:

x-anchors:
  - &latency-base
    condition: C
    for: 5m
    isPaused: false
  - &reduce-data
    refId: B
    datasourceUid: __expr__
    model:
      type: reduce
      expression: A
      reducer: last

groups:
  - name: latency
    rules:
      - <<: *latency-base
        title: API checkout p95 > 5s
        data:
          - refId: A
            ... # query
          - <<: *reduce-data
          - refId: C
            ... # threshold

This cuts down the amount of YAML drastically and avoids inconsistencies between rules.