Derived metrics

Precompute metrics from telemetry you already send. Two worker jobs write derived series into metric_points, so they query through the ordinary metrics source, chart with the same aggregations as ingested metrics, and roll up into the 5m and 1h tables automatically. Because they are stored, a derived signal outlives the raw spans it came from once those age out.

Both jobs run on the worker role and append exactly once per bucket under an advisory lock, so a multi-worker deployment never double-counts.

Span-to-metrics RED series

The span-to-metrics connector turns spans into rate, errors, and duration series, grouped by service, span name, and span kind, in one-minute buckets. It is off by default; enable it in [span_metrics]:

[span_metrics]
enabled        = true
retention_days = 90

It writes three metrics:

Metric	Meaning
traces.span.calls	Number of spans in the bucket.
traces.span.errors	Number of spans with an error status.
traces.span.duration	Sum of span durations in nanoseconds.

All three are delta sums, so they re-aggregate exactly across coarser windows. Query the request rate, error ratio, and average latency by summing them:

metrics | where name == "traces.span.calls" | stats sum(value) as calls by service, bin(1m)

metrics
  | where name == "traces.span.duration"
  | stats sum(value) as ns by service, bin(5m)

Average latency is sum(traces.span.duration) / sum(traces.span.calls). Recording duration as a sum keeps that average exact under rollup. Percentiles are not derived here; query the raw spans for those while they are still retained.

Cardinality follows your spans: one series per service, span name, and kind. Leave the connector off on projects with high span-name cardinality, or scope its cost by tuning span sampling at the collector.

Recording rules

A recording rule materializes an OQL query into a named metric on an interval, so an expensive aggregation is computed once and charts cheaply. It is the user-defined version of the span-to-metrics and service-map rollups.

Each rule defines:

Field	Purpose
Name	Display name for the rule.
Metric	The output metric name written to metric_points.
Query	An OQL query that produces exactly one aliased aggregation, optionally grouped.
Step	The bucket width and evaluation interval, such as 1m or 1h.
Retention	How many days the derived series are kept.

The query must end in one stats stage with a single aliased aggregation and no bin of its own; the step supplies the bin. Grouping dimensions become the output metric’s attributes, and a service dimension maps to the service column:

spans | where is_error | stats count() as errors by service

A rule with that query, the metric name app.errors, and a step of 1m writes one app.errors point per service per minute. Query it like any metric:

metrics | where name == "app.errors" | stats sum(value) as errors by service, bin(5m)

Derived points are gauges marked with the rule that produced them, so a rule’s output never collides with an ingested metric of the same name, and rolls up by sum, average, min, and max rather than rate. Rules apply going forward from when they are created; they do not backfill history.

Recording rules are managed through the API under /api/v1/workspaces/{ws}/recording-rules and are gated by the SLO permissions, since both define worker-evaluated derived metrics over OQL.

Downsampling

Every metric, ingested or derived, is downsampled into 5-minute and 1-hour rollup tables as it is written. The query planner reads the rollup tables automatically when a query bins at 5 minutes or coarser, so long windows stay fast without any configuration. Each signal keeps its own retention, so raw points can expire while the rollups remain.