Experiments

An experiment shows two versions of one quiz to different shoppers at the same time and lets one of them decide what the store keeps. One version is the one that is live (the control), the other is a version you built in the editor; a shopper always sees the same version, and every metric is reported for both with the difference between them and a range around that difference. A metric reads as collecting (too early to say), leaning (a difference is showing but could still turn) or clear (settled). A test can end at the first clear result, on a date, when you end it, or by asking our AI to decide: the AI option costs credits, charged each time it runs, and you can cap how many times it may run (1 to 20).

Reads need experiments:read; creating, changing, starting, republishing and ending need experiments:write (see Authentication). Reads cost 1 unit, writes 2 and the results and attention routes count as analytics (see Rate limits). On a plan without experiments, create, change, start and republish answer 403 experiments_disabled; reading and ending still answer.

The lifecycle

`status`	What it means	What you can do
`draft`	Set up, not started. No shopper has seen it.	Change every setting, then start it
`running`	Shoppers are being split between the two versions.	Change the settings, move the split, end it
`waiting`	Still splitting, and it needs you: the test cannot decide on its own until you act.	Read why on the attention route, then change the settings or end it
`ended`	No longer splitting. The settings are frozen and the results stay readable.	Read it

A quiz has at most one test that is not ended. Starting a test never changes what is published: the live version stays live, and a winner is published only when the test ends with one.

The endpoints

Task	Call	Scope
The store’s tests	`GET /v1/experiments`	`experiments:read`
What needs you	`GET /v1/experiments/attention`	`experiments:read`
One quiz’s tests	`GET /v1/quizzes/{quiz_id}/experiments`	`experiments:read`
Read one test	`GET /v1/experiments/{experiment_id}`	`experiments:read`
Read one version’s side	`GET /v1/experiments/{experiment_id}/arms/{arm_id}`	`experiments:read`
Read the results	`GET /v1/experiments/{experiment_id}/results`	`experiments:read`
Create a draft	`POST /v1/experiments`	`experiments:write`
Change the settings or the split	`PATCH /v1/experiments/{experiment_id}`	`experiments:write`
Swap in a new version	`POST /v1/experiments/{experiment_id}/arms/{arm_id}/republish`	`experiments:write`
Start splitting shoppers	`POST /v1/experiments/{experiment_id}/start`	`experiments:write`
End it, with or without a winner	`POST /v1/experiments/{experiment_id}/end`	`experiments:write`

An unknown test, another store’s test and another store’s quiz all answer 404 not_found; every error is a Problem body.

Find tests

Both lists answer newest first and page with limit (1 to 500, 200 by default) and offset.

curl "https://api.octaneai.com/v1/experiments?limit=50" \
  -H "Authorization: Bearer $OCTANE_API_KEY"

{
  "data": [
    {
      "id": "expt_7c9e6679742540de944be07fc1f90ae7",
      "app_id": "quiz_7c9e6679742540de944be07fc1f90ae7",
      "name": "Skip the email page",
      "status": "draft",
      "decision_metric": "completion_rate",
      "decision_option": "when_clear",
      "timezone": "UTC",
      "fallback": "notify",
      "ai_fallback": "when_clear",
      "revenue_tiers": ["added_from_quiz", "shown_bought_later", "bought_not_shown"],
      "protect_metrics": [],
      "weekly_digest": true,
      "long_run_reminders": true,
      "period": 0,
      "period_reason": "started",
      "revision": 0,
      "created_at": "2026-09-16T12:00:00Z",
      "updated_at": "2026-09-16T12:00:00Z",
      "arms": [
        {
          "id": "arm_00000000000000000000000000000001",
          "position": 0,
          "label": "Live version",
          "weight": 1,
          "version_id": "ver_00000000000000000000000000000001",
          "version_number": 1,
          "is_control": true
        },
        {
          "id": "arm_00000000000000000000000000000002",
          "position": 1,
          "label": "Variant B",
          "weight": 1,
          "version_id": "ver_00000000000000000000000000000002",
          "version_number": 2,
          "is_control": false
        }
      ]
    }
  ]
}

GET /v1/quizzes/{quiz_id}/experiments answers the same shape for one quiz. A field that is null is left off the wire, so read every optional member with a default.

What the test says about itself:

name is your label for the test, and app_id is the quiz_id it belongs to.
arms are the two sides, in order: the control is the version that is live, the other is the variant you are testing. Each carries the ver_ id and the version number it renders, the arm_ id the rest of the API names it by, and its weight.
weight is a whole number of at least 1; the two weights are shares of the traffic, so 1 and 1 send half the shoppers to each version and 9 and 1 send a tenth to the variant.
decision_metric is the metric that decides the test, and decision_option is how it ends: when_clear (publish the winner at the first clear result), on_date (decide on decision_date, read at local midnight in timezone), ai (our AI decides) or manual (it runs until you end it).
fallback is what happens on the date when the result is not clear: keep (keep running), revert (put the live version back) or notify (tell you and wait).
ai_max_runs caps how many times the AI may run, and ai_fallback says what the test does after that: when_clear decides at the first clear result without the AI, notify tells you and waits.

revenue_tiers are the kinds of revenue the money metrics count:

`revenue_tiers`	Revenue from
`added_from_quiz`	Products added to the cart from the quiz; always counted
`shown_bought_later`	Products the quiz showed and the shopper bought later
`bought_not_shown`	Products a quiz taker bought without the quiz showing them

protect_metrics are metrics that must not get worse: the test will not decide on its own while one of them is harmed.
weekly_digest and long_run_reminders control the emails the store gets about the test.
period, period_started_at and period_reason say which counting period the results are read from.

Once the test is over, ended_at, ended_by, outcome and winner_arm_id say how it finished, and winner_arm_id names the arm whose version was published. outcome is also set while a test is waiting, to say why:

`outcome`	Status	What happened
`winner_manual`	`ended`	You published a winner
`winner_auto`	`ended`	The test published the winner at a clear result
`winner_ai`	`ended`	Our AI published a winner
`ai_no_change`	`ended`	Our AI ended the test and kept the live version
`reverted`	`ended`	The decision date passed without a clear result and the live version was put back
`plan_downgraded`	`ended`	The store’s plan no longer includes tests
`ended_manual`	`ended`	You ended a started test without a winner
`cancelled`	`ended`	You ended a draft
`ai_recommended`	`waiting`	Our AI recommends a winner and waits for you
`ai_keep_running`	`waiting`	Our AI decided to keep the test running
`ai_blocked_credits`	`waiting`	Our AI could not run for lack of credits
`srm_flagged`	`waiting`	The split looks unhealthy; automatic decisions are paused
`no_winner_notify`	`waiting`	The decision date passed without a clear result and the test waits for you

ai_verdict is what our AI decided last, and ai_rationale why:

ai_verdict What the AI decided
publish Publish a version
keep_running Keep the test running
no_change End with no change; the live version stays

`ai_verdict`	What the AI decided
`publish`	Publish a version
`keep_running`	Keep the test running
`no_change`	End with no change; the live version stays

The response also carries a few fields the dashboard uses to draw the test’s own page.

What needs you

The attention route answers one row per test that is waiting on a decision, across the store. Nothing to do means an empty list.

curl https://api.octaneai.com/v1/experiments/attention \
  -H "Authorization: Bearer $OCTANE_API_KEY"

{ "items": [] }

Each row carries experiment_id, app_id, name, since and a message to show as it is, plus a kind:

`kind`	What it means
`split_unhealthy`	The split between the versions looks wrong and the results cannot be trusted
`ai_recommended`	Our AI has a recommendation waiting for you
`waiting`	Automatic decisions are paused; you decide
`clear`	The decision metric is clear and you may want to end the test
`long_run`	The test has been running a long time; `week` says how many weeks

Create a draft

A draft needs the quiz_id of the quiz it tests, a name and the variant: the ver_ id of the version to test against the live one. Everything else has a default, so the shortest create is three fields; the control is whatever version is live when the test starts.

curl -X POST https://api.octaneai.com/v1/experiments \
  -H "Authorization: Bearer $OCTANE_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "quiz_id": "quiz_7c9e6679742540de944be07fc1f90ae7",
    "name": "Skip the email page",
    "variant": { "version_id": "ver_00000000000000000000000000000002" }
  }'

The answer is 201 with the draft, in the shape the list shows. By default it decides on completion rate at the first clear result, counts all three kinds of revenue, and splits the traffic evenly. To set the rest at once, send control_weight, decision_metric ({"kind": ..., "target": ...}), decision_option, decision_date, timezone, fallback, ai_trigger, ai_max_runs, ai_fallback, revenue_tiers, protect_metrics, weekly_digest, long_run_reminders or merchant_note (what the AI should know about this test) in the same body.

Every metric has a kind, the name decision_metric and protect_metrics take and the results’ metrics[].kind and protection[].kind report:

`kind`	What it measures	Counted over (`population`)	Can be protected
`completion_rate`	Completion rate	`starts`	Yes
`email_opt_in`	Email opt-in rate	`starts`	Yes
`phone_opt_in`	Phone opt-in rate	`starts`	Yes
`order_conversion`	Visits with an order	`visits`	Yes
`rpv`	Revenue per visit	`visits`	Yes
`aov`	Average order value	`orders`	No
`reached_page`	Reached a page; `target` is `{"mode": "specific" or "after", "page_key"}`	`visits`	No
`product_page_clicked`	Went to a product page	`visits`	No
`custom_url_clicked`	Clicked a custom link	`visits`	No
`button_clicked`	Clicked a button; `target` is `{"page_key", "component_key"}`	`visits`	No
`chose_answer`	Chose an answer; `target` is `{"page_key", "component_key", "option_id"}`	`visits`	No

Only reached_page, button_clicked and chose_answer take a target, and that target has to exist in both versions; every other kind takes none.

Start it

Starting is deliberate: the body must say so.

curl -X POST https://api.octaneai.com/v1/experiments/expt_7c9e6679742540de944be07fc1f90ae7/start \
  -H "Authorization: Bearer $OCTANE_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{"confirm": true}'

The answer is the test with status: "running". From that moment shoppers are split between the two versions; the live version stays live for everyone the test sends to it.

Change the settings or the split

PATCH takes any of the create fields except the variant, and carries the traffic split as control_weight and variant_weight. Either weight may be sent alone; the other arm keeps the weight it has.

curl -X PATCH https://api.octaneai.com/v1/experiments/expt_7c9e6679742540de944be07fc1f90ae7 \
  -H "Authorization: Bearer $OCTANE_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{"control_weight": 1, "variant_weight": 1}'

On a draft the weights simply change. On a test that has started, a new split opens a new counting period, and only the live version’s share may go down, so that no shopper who has seen the variant is moved back; to show everyone the live version again, end the test. Sending "ai_max_runs": null lifts an existing cap.

Swap in a new version

Republishing points one side of the test at a different version, and says what to do with what has been measured so far: keep_results: true for a fix that cannot change what shoppers do (a typo, a colour), false to start a new counting period.

curl -X POST https://api.octaneai.com/v1/experiments/expt_7c9e6679742540de944be07fc1f90ae7/arms/arm_00000000000000000000000000000002/republish \
  -H "Authorization: Bearer $OCTANE_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{"version_id": "ver_00000000000000000000000000000002", "keep_results": false}'

Shoppers stay in the version they are in either way. The answer is the test after the swap.

End it

Ending stops the split. Send winner_arm_id to publish that arm’s version as you end, or leave it out to end with the live version untouched.

curl -X POST https://api.octaneai.com/v1/experiments/expt_7c9e6679742540de944be07fc1f90ae7/end \
  -H "Authorization: Bearer $OCTANE_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{"confirm": true, "winner_arm_id": "arm_00000000000000000000000000000002"}'

Ending a test that has already ended changes nothing and answers the same test. Publishing a winner this way fires quiz.published like any other publish, and ending fires experiment.ended.

Read one test

The detail route answers the test with events, the ordered record of what happened to it: when it started, what was changed, what our AI said, how it ended. Each event has at, kind, the arm_id it concerns (or null) and data:

`kind`	What happened
`created`	The test was created
`started`	The test started
`period_opened`	A new counting period opened
`weights_changed`	The traffic split changed
`control_published`	The live version changed
`arm_republished`	A side was pointed at another version
`decision_metric_changed`	The decision metric changed
`decision_option_changed`	How the test decides changed
`protect_changed`	The protected metrics changed
`revenue_tiers_changed`	Which revenue counts changed
`settings_changed`	Other settings changed
`evaluation_delayed`	The decision date moved
`decision_traced`	A decision was recorded
`ai_decided`	Our AI decided
`srm_flagged`	The split looks unhealthy; automatic decisions paused
`clear`	The decision metric became clear
`nudged`	A reminder was sent
`digest_sent`	The weekly update was sent
`warning_sent`	The three-day reminder was sent
`result_sent`	The result was sent
`ended`	The test ended
`target_missing`	The decision metric’s target is missing on one version

curl https://api.octaneai.com/v1/experiments/expt_7c9e6679742540de944be07fc1f90ae7 \
  -H "Authorization: Bearer $OCTANE_API_KEY"

{
  "id": "expt_7c9e6679742540de944be07fc1f90ae7",
  "app_id": "quiz_7c9e6679742540de944be07fc1f90ae7",
  "name": "Skip the email page",
  "status": "draft",
  "decision_metric": "completion_rate",
  "decision_option": "when_clear",
  "timezone": "UTC",
  "fallback": "notify",
  "ai_fallback": "when_clear",
  "revenue_tiers": ["added_from_quiz", "shown_bought_later", "bought_not_shown"],
  "protect_metrics": [],
  "weekly_digest": true,
  "long_run_reminders": true,
  "period": 0,
  "period_reason": "started",
  "revision": 0,
  "created_at": "2026-09-16T12:00:00Z",
  "updated_at": "2026-09-16T12:00:00Z",
  "arms": [
    {
      "id": "arm_00000000000000000000000000000001",
      "position": 0,
      "label": "Live version",
      "weight": 1,
      "version_id": "ver_00000000000000000000000000000001",
      "version_number": 1,
      "is_control": true
    },
    {
      "id": "arm_00000000000000000000000000000002",
      "position": 1,
      "label": "Variant B",
      "weight": 1,
      "version_id": "ver_00000000000000000000000000000002",
      "version_number": 2,
      "is_control": false
    }
  ],
  "events": []
}

One arm reads on its own too, when all you need is which version a side is showing:

curl https://api.octaneai.com/v1/experiments/expt_7c9e6679742540de944be07fc1f90ae7/arms/arm_00000000000000000000000000000001 \
  -H "Authorization: Bearer $OCTANE_API_KEY"

{
  "id": "arm_00000000000000000000000000000001",
  "position": 0,
  "label": "Live version",
  "weight": 1,
  "version_id": "ver_00000000000000000000000000000001",
  "version_number": 1,
  "is_control": true,
  "experiment_id": "expt_7c9e6679742540de944be07fc1f90ae7"
}

Read the results

The results route answers everything the test’s page in the dashboard shows: the headline, the state of every metric on both versions, the counts per version, what still stands between the test and a decision, and the evidence behind a decision that has been made. It is a live read, not a cached one. The answer below is cut to the fields the bullets explain; the reference lists every one, among them the counting window (window_start, window_end, admission_cutoff), the share of visits without cookies (cookieless_share) and the frozen evidence behind a decision (decision_trace, ai_decision_trace).

curl https://api.octaneai.com/v1/experiments/expt_7c9e6679742540de944be07fc1f90ae7/results \
  -H "Authorization: Bearer $OCTANE_API_KEY"

{
  "experiment": { "id": "expt_7c9e6679742540de944be07fc1f90ae7", "...": "..." },
  "as_of": "2026-09-16T12:00:00Z",
  "frozen": false,
  "attribution_window_days": 7,
  "attribution_line": "Revenue keeps arriving for up to 7 days after a visit.",
  "arms": [],
  "header": {
    "testing": "Live version (v1) vs Variant B (v2)",
    "current_read": "Not enough data yet.",
    "next": "Start this test to split shoppers."
  },
  "changes": [],
  "progress": {
    "day": 0,
    "observed": { "control": 0, "variant": 0 },
    "tier": "min_days",
    "line": ""
  },
  "gates": [],
  "metrics": [],
  "protection": [],
  "split": { "units_per_arm": [0, 0], "expected": [0.5, 0.5], "log_odds": 0.0, "latched": false, "surfaced": false },
  "periods": [],
  "events": [],
  "changed_since": false,
  "explainer_keys": []
}

header is the test in words, ready to show: testing names the two versions, current_read says where it stands, next says what happens next, and difference reads the gap on the decision metric.
arms carries the counts per version: views, starts, completions, completion_rate, email_opt_ins, phone_opt_ins, orders, visits_with_order, revenue with its revenue_by_bucket, counted_revenue (the part of it in the revenue kinds the test counts, what the money metrics read) and assignment_units (the distinct shoppers assigned to the version).
metrics is one readout per metric, on both versions, with the difference, the range around it (lo and hi), its state (collecting, leaning or clear) and its role (decider or exploratory). population is what the metric is counted over and unit how its difference reads:

population unit
starts percentage points
visits percentage points; currency per visit for rpv
orders currency per order
protection is one row per protected metric, labelled not checked yet, no harm or worse.
gates are the five things a test clears before it can decide on its own, each with met and a sentence to show: the decision metric is measured on both versions, the 7-day minimum, enough data, a healthy split, and no protected metric harmed.

`population`	`unit`
`starts`	`percentage points`
`visits`	`percentage points`; `currency per visit` for `rpv`
`orders`	`currency per order`

progress says how far along the test is, changes lists the pages that differ between the two versions, and events is the same timeline the detail route answers. progress.tier names the stage:

`tier`	What it means
`min_days`	Inside the 7-day minimum of the current period
`estimating`	Not enough traffic yet to say how long the test needs
`on_track`	Enough traffic is on its way within six weeks at this pace
`slow`	At this pace the test needs longer than six weeks
`reached`	Enough shoppers have been through and the decision metric is clear
`reached_unclear`	Enough shoppers have been through but the difference is too small to be sure

attribution_window_days and attribution_line say how long orders keep arriving for a visit: the newest visits are undercounted on both versions until their orders land, which is why the money metrics settle later than completion and opt-in.
frozen is true once the test has ended, and changed_since is true when the numbers have moved a lot since the decision was recorded. The decision itself never changes.

Counting periods

A test counts visits from its current counting period only. The first period opens when the test starts, and a new one opens whenever something changes what the numbers mean; from then on the results count only the visits of the new period.

period is the number of the current period: 0 on a draft, 1 once the test starts, and one higher each time a new period opens.
period_started_at is when the current period began; a draft has none.
period_reason says why it began:

`period_reason`	What opened the period
`started`	The test started
`split_changed`	The split between the versions changed
`control_published`	A new live version was published, and the live side of the test follows it
`variant_republished`	The variant was republished with `keep_results: false`
`metric_changed`	The decision metric or its target changed on a started test
`revenue_tiers_changed`	The kinds of revenue the test counts changed on a started test

A split change on a running test opens a new period so that shoppers assigned under different odds are never mixed: visits counted under two different splits can point one way each and the other way added together. A new period never moves a shopper; everyone keeps the version they were shown, and only the counting starts again.

The 7-day minimum before a test may decide counts from the start of the current period, so a new period starts that week again; a decision date that falls inside it waits until the week is over. Earlier periods stay in the results as periods, each with its period number, its reason (one of the period_reason values above), when it started and ended, and a label to show.

Errors you will meet

Every one is a Problem body.

Status	`title`	When
`403`	`experiments_disabled`	The store’s plan does not include experiments. Creating, changing, starting and republishing are refused; reading and ending still answer
`403`	`insufficient_scope`	The key lacks `experiments:read` or `experiments:write`
`404`	`not_found`	No such quiz, test or arm for this store
`409`	`conflict`	The test is not in a state that takes this call: a test is already open on the quiz, only a draft can be started, or the test has ended and its settings are frozen
`422`	`invalid_experiment`	The settings do not hold together

The detail of an invalid_experiment is written to be shown to the person who sent it. The ones you will meet most:

A decision date less than 7 days out: a test runs at least 7 days, whatever the numbers look like earlier.
ai_max_runs outside 1 to 20.
Raising the live version’s share on a test that has started: while a test runs the live version’s share can only go down; end the test to show everyone the live version.
A variant that is the live version, revenue_tiers without added_from_quiz, a metric that is both the decision metric and a protected one, or a decision metric whose target is missing from one of the versions.

Creating a test on a quiz that has never been published answers 409 conflict: a test needs a live version to run the variant against.

What else carries the test

The experiment.started and experiment.ended webhooks fire when a test starts and when it ends, with the arms and the winner.
Every session webhook carries experiment_id and arm_id, so you can tell which version a shopper saw.
An assistant connected through the MCP server can read and run tests when you grant it the Experiments permission.