Experiments

An experiment pairs a flag’s percentage rollout with goal metrics and measures how each variation moves them. It reuses telemetry you already send: the flag records a per-unit exposure when it serves a tracked rollout bucket, and the worker joins those exposures to metric outcomes over ClickHouse on each tick. The flag still serves variations; the experiment only reads what happened.

You configure the split on the flag (the Rollout tab, or PUT .../flags/{key}) and create the experiment over it. Mark rollout buckets tracked so their evaluations emit exposures; an untracked bucket is served but never analyzed.

Define an experiment

GET  /api/v1/workspaces/{ws}/projects/{p}/experiments
POST /api/v1/workspaces/{ws}/projects/{p}/experiments
GET  /api/v1/workspaces/{ws}/projects/{p}/experiments/{id}
PUT  /api/v1/workspaces/{ws}/projects/{p}/experiments/{id}

Field	Meaning
flag_key, environment	The flag whose rollout assigns variations, and the environment it runs in. Both are fixed once set.
baseline_variation	The control arm. Lift and significance are measured against it.
unit_attribute	The attribute carrying the unit id on outcome telemetry, joined to the exposure’s unit. Defaults to key.
stats_method	bayes leads with probability to beat control; frequentist leads with a p-value. Both are always computed.
cuped	Enable CUPED variance reduction (below). Off by default; set it when designing the experiment.
exposure_sample	Optional deterministic sample fraction. Omit, set 0, or set 1 to analyze every exposed unit.
analysis_window	How far back results look (for example 30d).
warehouse_dsn_env	Optional. Read assignments and outcomes from a BYO warehouse instead of ClickHouse (see Warehouse-native mode).
metrics	One or more goals (below). Exactly one must be primary.

A goal metric is defined over existing spans, logs, or metrics:

Field	Meaning
kind	binary (a conversion rate) or continuous (a per-unit value).
source	spans, logs, or metrics.
selector	An OQL predicate selecting the outcome events, validated up front.
unit_attribute	The attribute on the outcome carrying the unit id.
value_field, per_unit_agg	For a continuous goal, the numeric attribute and how it aggregates per unit (sum, avg, min, max).
direction	Whether increase or decrease is the improvement.

A draft experiment is fully editable. Start it to freeze its structure and begin analysis:

POST /api/v1/workspaces/{ws}/projects/{p}/experiments/{id}/start
POST /api/v1/workspaces/{ws}/projects/{p}/experiments/{id}/stop
POST /api/v1/workspaces/{ws}/projects/{p}/experiments/{id}/archive

Starting snapshots the flag’s current tracked arms and weights, so editing the rollout later does not distort the sample-ratio check.

Stopping may include a decision record:

POST /api/v1/workspaces/{ws}/projects/{p}/experiments/{id}/stop
{"winner_variation": 1, "decision_note": "Ship treatment after seven days"}

Both fields are optional; winner_variation must be one of the tracked variation indices.

After a stopped experiment records a winner, apply it to the flag environment:

POST /api/v1/workspaces/{ws}/projects/{p}/experiments/{id}/apply-winner
{"create_guarded_rollout": true}

Applying the winner enables the flag in the experiment’s environment and pins the fallthrough to the winning variation, replacing the experiment’s percentage rollout. The request body is optional. When create_guarded_rollout is true, Oriel also creates a disarmed guarded release with the experiment’s control arm as the baseline and a default 1h analysis window, so an operator can attach guardrails before arming it.

When exposure_sample is greater than 0 and below 1, analysis deterministically hashes each exposed unit by flag and unit id and keeps only that fraction before first-touch assignment. Repeated worker runs use the same sampled units, and the sample-ratio-mismatch check runs on the sampled exposure counts.

Sample-size calculator

Use the stateless calculator to estimate the per-arm sample size needed before launching an experiment:

GET /api/v1/workspaces/{ws}/projects/{p}/experiments/calculator

Query parameters:

Parameter	Meaning
metric_kind	binary for conversion rates, or continuous for per-unit means.
baseline	Baseline conversion rate for binary metrics, or baseline mean for continuous metrics.
stddev	Standard deviation for continuous metrics. Ignored for binary metrics.
lift	Relative effect to detect.
alpha	False-positive rate. Defaults to 0.05.
beta	False-negative rate. Defaults to 0.20.

The response returns sample_size_per_arm and the realized power after rounding the sample size up.

Results

GET /api/v1/workspaces/{ws}/projects/{p}/experiments/{id}/results
GET /api/v1/workspaces/{ws}/projects/{p}/experiments/{id}/results/timeseries?metric={id}

Per variation the results carry exposed units, the conversion rate or mean, relative lift against control with a 95% confidence interval, and the headline significance for the chosen method. The worker also runs a chi-square sample-ratio-mismatch test across the arms: if observed exposure counts diverge from the configured split, the results page warns that assignment looks broken and a winner should not be trusted yet.

The Bayesian view pairs the probability to beat control with an expected loss: the posterior-weighted amount you would give up by shipping that variation if it turns out to be worse than control, relative to control. A clear winner has an expected loss near zero; a toss-up keeps real downside on the table. Read it as the risk of calling the experiment now.

CUPED variance reduction

CUPED (Controlled-experiment Using Pre-Existing Data) tightens the intervals without changing assignment. For each unit it measures the same goal metric over the window immediately before the experiment started and uses that pre-period value as a covariate, subtracting the part of each unit’s outcome the covariate already predicts. Units that were always going to convert (or always going to be slow) stop adding noise, so the same data reaches significance sooner.

It is a read-side choice: enabling cuped re-analyzes the existing exposures, it never re-buckets anyone, and a covariate that turns out to be uncorrelated leaves the estimate unchanged rather than hurting it. When a metric’s variance drops, its results panel shows how much (for example CUPED · −24% variance). CUPED compares every metric, binary or continuous, on the adjusted values, so a CUPED conversion rate reads as a continuous mean.

Warehouse-native mode

By default the worker reads exposures and outcomes from Oriel’s ClickHouse. A warehouse-native experiment instead reads its assignments and outcome facts from a warehouse you own, then runs the same statistics and writes the same results. The flag still defines the variations, weights, and baseline; the warehouse holds which unit got which variation index and what each unit then did.

Set warehouse_dsn_env on the experiment to the name of an environment variable holding the warehouse DSN. The connection speaks the Postgres wire protocol over the pgx driver, so Redshift, Postgres, CockroachDB, and Materialize work directly; front Snowflake or BigQuery with a Postgres-wire proxy. The DSN lives in the environment, never in the control database, matching how Oriel handles its other DSNs. The variable name and every table and column below are validated, and the time window and experiment key bind as query parameters, so a mapping cannot inject SQL.

warehouse_assignment maps the table recording assignments:

Field	Meaning
table	The assignment table. A unit’s first row in the window fixes its arm.
unit_column	The unit id.
variation_column	The flag’s 0-based variation index.
timestamp_column	Bounds assignments to the analysis window.
experiment_column	Optional. When set, rows are filtered to the experiment’s key, so one table can serve many experiments.

Each goal metric carries a warehouse mapping instead of a telemetry source and selector:

Field	Meaning
table	The facts table whose rows are this goal’s outcomes.
unit_column	Joins a fact to the assignment’s unit.
value_column	For a continuous goal, the numeric column aggregated per unit with per_unit_agg. A binary goal counts any matching row as a conversion and ignores it.
timestamp_column	Bounds facts to the analysis window.

Everything downstream is unchanged: the worker analyzes on the same tick, writes the same snapshots, and the results page renders lift, intervals, significance, and the sample-ratio-mismatch warning the same way. CUPED is not yet available in warehouse mode and is rejected when set on a warehoused experiment.

Mutually-exclusive layers and holdouts

A layer keeps overlapping experiments from confounding each other: every experiment in a layer draws from one shared hash of the unit, so a unit is allocated to at most one experiment per layer. The rest of the layer is a holdout, a reserved control that no experiment in the layer touches.

Set a layer when you create an experiment:

Field	Meaning
layer_key	The layer name. Experiments sharing it in the same project are mutually exclusive. Empty opts out and the experiment splits the flag’s full traffic.
layer_allocation	The share this experiment claims, in parts-per-100000. The UI collects it as a percentage.
layer_holdout	The reserved control, in parts-per-100000. The first experiment in a layer fixes it; later ones inherit it.

Each experiment packs into the next free slot of the layer, so adding one never moves an existing one. An allocation that no longer fits beside the others and the holdout is rejected with ORL-8005. Starting a layered experiment writes the slot onto the flag’s fallthrough rollout, so SDK evaluations gate on it immediately; stopping, archiving, or deleting the experiment clears it again. A layered experiment needs a fallthrough percentage rollout on its flag.

Layers only change bucketing for the flags that join one. Flags that predate layers, and any experiment with no layer_key, bucket exactly as before, so turning on layers never re-buckets a live flag.