Test result to Slack the moment it ends

Goal: the moment a test ends, one Slack message says which version won, how the test ended, what the decision metric moved by and how long the test ran, so the team hears the result without opening the dashboard.

Uses: the experiment.ended webhook to know the test ended and how, GET /v1/experiments/{experiment_id}/results from Experiments for the decision metric and the shoppers counted, and GET /v1/quizzes/{quiz_id} for the quiz’s name. Scopes: webhooks:manage to create the endpoint, experiments:read for the results, quizzes:read for the quiz. The Slack incoming webhook URL goes in SLACK_WEBHOOK_URL.

The flow

Subscribe an endpoint to experiment.ended once.
When a test ends, the receiver verifies the signature, drops a webhook-id it has already handled, and reads the test’s results and the quiz’s name.
It picks the sentence for the event’s outcome, fills in the numbers and posts it to Slack:

Skip the email page on Find your routine: Variant B won on Email opt-in rate, +23% (range +8% to +41%), 2,140 shoppers over 16 days. Published automatically.

curl -X POST https://api.octaneai.com/v1/webhooks \
  -H "Authorization: Bearer $OCTANE_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{"url": "https://hooks.example.com/octane", "topics": ["experiment.ended"], "description": "Test results to Slack"}'

Keep the whole secret from the answer in OCTANEAI_WEBHOOK_SECRET. The event the endpoint will receive (the fields this recipe reads; the full object is on the Webhooks page):

{
  "id": "evt_6ba7b8109dad11d180b400c04fd430c8",
  "type": "experiment.ended",
  "created": "2026-10-02T12:00:00Z",
  "api_version": "v1",
  "data": {
    "object": {
      "experiment_id": "expt_7c9e6679742540de944be07fc1f90ae7",
      "quiz_id": "quiz_7c9e6679742540de944be07fc1f90ae7",
      "name": "Skip the email page",
      "status": "ended",
      "arms": [
        { "arm_id": "arm_00000000000000000000000000000001", "label": "Live version", "is_control": true, "version_id": "ver_15ce0be06b904d0180b1155da02a6c6f", "version_number": 2 },
        { "arm_id": "arm_00000000000000000000000000000002", "label": "Variant B", "is_control": false, "version_id": "ver_9f1c1a0d5f2b4e2b8a3c6d7e8f901234", "version_number": 3 }
      ],
      "winner": { "arm_id": "arm_00000000000000000000000000000002", "label": "Variant B", "is_control": false, "version_id": "ver_9f1c1a0d5f2b4e2b8a3c6d7e8f901234", "version_number": 3 },
      "started_at": "2026-09-16T12:00:00Z",
      "ended_at": "2026-10-02T12:00:00Z",
      "outcome": "winner_auto"
    }
  }
}

2. The receiver

The receiver checks the signature with the verify() from Verify a webhook signature, repeated so the block runs on its own; that page explains each line. It keeps the webhook-id of every delivery it has posted, so a retry or a redelivery of the same event does not post twice (Flask and httpx, pip install flask httpx; Express, npm install express):

Python
TypeScript

import base64
import datetime as dt
import hashlib
import hmac
import os
import time
import httpx
from flask import Flask, request

app = Flask(__name__)
SECRET = os.environ["OCTANEAI_WEBHOOK_SECRET"]  # whsec_...
HEADERS = {"Authorization": f"Bearer {os.environ['OCTANE_API_KEY']}"}
API = "https://api.octaneai.com/v1"
SLACK = os.environ["SLACK_WEBHOOK_URL"]
SEEN = set()  # in production: the table the signature recipe's accept() writes

METRICS = {
    "completion_rate": "Completion rate",
    "email_opt_in": "Email opt-in rate",
    "phone_opt_in": "Phone opt-in rate",
    "order_conversion": "Order conversion",
    "rpv": "Revenue per visit",
    "aov": "Average order value",
    "reached_page": "Reached a page",
    "product_page_clicked": "Went to a product page",
    "custom_url_clicked": "Clicked a custom link",
    "button_clicked": "Clicked a button",
    "chose_answer": "Chose an answer",
}

SENTENCES = {
    "winner_manual": "{test} on {quiz}: {winner} won on {metric}, {read}, {shoppers} shoppers over {days}. You published it.",
    "winner_auto": "{test} on {quiz}: {winner} won on {metric}, {read}, {shoppers} shoppers over {days}. Published automatically.",
    "winner_ai": "{test} on {quiz}: {winner} won on {metric}, {read}, {shoppers} shoppers over {days}. Our AI published it.",
    "ai_no_change": "{test} on {quiz}: our AI reviewed it and kept the live version. {metric}: {read}, {shoppers} shoppers over {days}.",
    "reverted": "{test} on {quiz}: no clear result, so the live version was put back. {metric}: {read}, {shoppers} shoppers over {days}.",
    "plan_downgraded": "{test} on {quiz}: ended because the store's plan no longer includes tests; the live version stays. {metric}: {read}, {shoppers} shoppers over {days}.",
    "ended_manual": "{test} on {quiz}: ended with no change, the live version stays. {metric}: {read}, {shoppers} shoppers over {days}.",
}

def verify(secret: str, headers, body: bytes) -> bool:
    try:
        key = base64.b64decode(secret[len("whsec_"):], validate=True)
        msg_id, ts, sigs = headers["webhook-id"], headers["webhook-timestamp"], headers["webhook-signature"]
        if abs(time.time() - int(ts)) > 300:
            return False
        expected = hmac.new(key, f"{msg_id}.{ts}.".encode() + body, hashlib.sha256).digest()
        for sig in sigs.split(" "):
            version, _, value = sig.partition(",")
            if version == "v1" and hmac.compare_digest(base64.b64decode(value, validate=True), expected):
                return True
    except (KeyError, ValueError):
        pass
    return False

def get(path: str) -> dict:
    r = httpx.get(f"{API}{path}", headers=HEADERS, timeout=30)
    r.raise_for_status()
    return r.json()

def when(stamp: str) -> dt.datetime:
    return dt.datetime.fromisoformat(stamp.replace("Z", "+00:00"))

def message(ended: dict) -> str:
    results = get(f"/experiments/{ended['experiment_id']}/results")
    quiz = get(f"/quizzes/{ended['quiz_id']}")
    decider = next((m for m in results["metrics"] if m["role"] == "decider"), None)
    if decider is None or decider["lift"] is None:
        read = "too early to measure"
    else:
        read = f"{decider['lift']:+.0%} (range {decider['lo']:+.0%} to {decider['hi']:+.0%})"
    days = max(1, round((when(ended["ended_at"]) - when(ended["started_at"])).total_seconds() / 86400))
    return SENTENCES[ended["outcome"]].format(
        test=ended["name"],
        quiz=quiz["name"],
        winner=ended["winner"]["label"] if ended["winner"] else None,
        metric=METRICS[results["experiment"]["decision_metric"]],
        read=read,
        shoppers=f"{results['split']['n']:,}",
        days="1 day" if days == 1 else f"{days} days",
    )

@app.post("/octane")
def octane():
    if not verify(SECRET, request.headers, request.get_data()):
        return "bad signature", 401
    delivery_id, event = request.headers["webhook-id"], request.get_json()
    if delivery_id in SEEN or event["type"] != "experiment.ended" or event["data"]["object"]["outcome"] not in SENTENCES:
        return "", 200
    text = message(event["data"]["object"])  # in production: enqueue, then answer; see the signature recipe
    print(text)
    httpx.post(SLACK, json={"text": text}, timeout=30).raise_for_status()
    SEEN.add(delivery_id)
    return "", 200

if __name__ == "__main__":
    app.run(port=8080)

import express from "express";
import { createHmac, timingSafeEqual } from "node:crypto";

const SECRET = process.env.OCTANEAI_WEBHOOK_SECRET!; // whsec_...
const HEADERS = { Authorization: `Bearer ${process.env.OCTANE_API_KEY}` };
const API = "https://api.octaneai.com/v1";
const SLACK = process.env.SLACK_WEBHOOK_URL!;
const SEEN = new Set<string>(); // in production: the table the signature recipe's accept() writes

type Ended = {
  experiment_id: string;
  quiz_id: string;
  name: string;
  winner: { label: string } | null;
  started_at: string;
  ended_at: string;
  outcome: string;
};
type Parts = { test: string; quiz: string; winner: string | undefined; metric: string; read: string; shoppers: string; days: string };

const METRICS: Record<string, string> = {
  completion_rate: "Completion rate",
  email_opt_in: "Email opt-in rate",
  phone_opt_in: "Phone opt-in rate",
  order_conversion: "Order conversion",
  rpv: "Revenue per visit",
  aov: "Average order value",
  reached_page: "Reached a page",
  product_page_clicked: "Went to a product page",
  custom_url_clicked: "Clicked a custom link",
  button_clicked: "Clicked a button",
  chose_answer: "Chose an answer",
};

const SENTENCES: Record<string, (p: Parts) => string> = {
  winner_manual: (p) => `${p.test} on ${p.quiz}: ${p.winner} won on ${p.metric}, ${p.read}, ${p.shoppers} shoppers over ${p.days}. You published it.`,
  winner_auto: (p) => `${p.test} on ${p.quiz}: ${p.winner} won on ${p.metric}, ${p.read}, ${p.shoppers} shoppers over ${p.days}. Published automatically.`,
  winner_ai: (p) => `${p.test} on ${p.quiz}: ${p.winner} won on ${p.metric}, ${p.read}, ${p.shoppers} shoppers over ${p.days}. Our AI published it.`,
  ai_no_change: (p) => `${p.test} on ${p.quiz}: our AI reviewed it and kept the live version. ${p.metric}: ${p.read}, ${p.shoppers} shoppers over ${p.days}.`,
  reverted: (p) => `${p.test} on ${p.quiz}: no clear result, so the live version was put back. ${p.metric}: ${p.read}, ${p.shoppers} shoppers over ${p.days}.`,
  plan_downgraded: (p) => `${p.test} on ${p.quiz}: ended because the store's plan no longer includes tests; the live version stays. ${p.metric}: ${p.read}, ${p.shoppers} shoppers over ${p.days}.`,
  ended_manual: (p) => `${p.test} on ${p.quiz}: ended with no change, the live version stays. ${p.metric}: ${p.read}, ${p.shoppers} shoppers over ${p.days}.`,
};

const pct = new Intl.NumberFormat("en-US", { style: "percent", signDisplay: "exceptZero", maximumFractionDigits: 0 });

function verify(secret: string, headers: Record<string, string | undefined>, body: Buffer): boolean {
  const id = headers["webhook-id"], ts = headers["webhook-timestamp"], sigs = headers["webhook-signature"];
  if (!id || !ts || !sigs || !/^\d+$/.test(ts)) return false;
  if (Math.abs(Date.now() / 1000 - Number(ts)) > 300) return false;
  const key = Buffer.from(secret.slice("whsec_".length), "base64");
  const expected = createHmac("sha256", key).update(`${id}.${ts}.`).update(body).digest();
  for (const sig of sigs.split(" ")) {
    const [version, value] = sig.split(",");
    if (version !== "v1" || !value) continue;
    const given = Buffer.from(value, "base64");
    if (given.length === expected.length && timingSafeEqual(given, expected)) return true;
  }
  return false;
}

async function get(path: string) {
  const res = await fetch(`${API}${path}`, { headers: HEADERS });
  if (!res.ok) throw new Error(`${res.status} ${await res.text()}`);
  return res.json();
}

async function message(ended: Ended): Promise<string> {
  const results = await get(`/experiments/${ended.experiment_id}/results`);
  const quiz = await get(`/quizzes/${ended.quiz_id}`);
  const decider = results.metrics.find((m: { role: string }) => m.role === "decider");
  const read = !decider || decider.lift === null
    ? "too early to measure"
    : `${pct.format(decider.lift)} (range ${pct.format(decider.lo)} to ${pct.format(decider.hi)})`;
  const days = Math.max(1, Math.round((Date.parse(ended.ended_at) - Date.parse(ended.started_at)) / 86400000));
  return SENTENCES[ended.outcome]({
    test: ended.name,
    quiz: quiz.name,
    winner: ended.winner?.label,
    metric: METRICS[results.experiment.decision_metric],
    read,
    shoppers: results.split.n.toLocaleString("en-US"),
    days: days === 1 ? "1 day" : `${days} days`,
  });
}

const app = express();
app.post("/octane", express.raw({ type: "application/json" }), async (req, res) => {
  const headers = { "webhook-id": req.get("webhook-id"), "webhook-timestamp": req.get("webhook-timestamp"), "webhook-signature": req.get("webhook-signature") };
  if (!verify(SECRET, headers, req.body)) return res.status(401).send("bad signature");
  const deliveryId = headers["webhook-id"]!;
  const event = JSON.parse(req.body.toString("utf8"));
  if (SEEN.has(deliveryId) || event.type !== "experiment.ended" || !(event.data.object.outcome in SENTENCES)) return res.sendStatus(200);
  const text = await message(event.data.object); // in production: enqueue, then answer
  console.log(text);
  const posted = await fetch(SLACK, { method: "POST", headers: { "Content-Type": "application/json" }, body: JSON.stringify({ text }) });
  if (!posted.ok) return res.sendStatus(503);
  SEEN.add(deliveryId);
  res.sendStatus(200);
});
app.listen(8080);

The message

The event’s outcome picks one sentence; nothing else in the message is written on the fly.

`outcome`	How the test ended	Sentence
`winner_manual`	You picked the winner and published it	{test} on {quiz}: {winner} won on {metric}, {lift} (range {lo} to {hi}), {shoppers} shoppers over {days}. You published it.
`winner_auto`	The test published the winner itself	{test} on {quiz}: {winner} won on {metric}, {lift} (range {lo} to {hi}), {shoppers} shoppers over {days}. Published automatically.
`winner_ai`	Our AI decided and published the winner	{test} on {quiz}: {winner} won on {metric}, {lift} (range {lo} to {hi}), {shoppers} shoppers over {days}. Our AI published it.
`ai_no_change`	Our AI reviewed the test and kept the live version	{test} on {quiz}: our AI reviewed it and kept the live version. {metric}: {lift} (range {lo} to {hi}), {shoppers} shoppers over {days}.
`reverted`	No clear result on the decision date, so the live version was put back	{test} on {quiz}: no clear result, so the live version was put back. {metric}: {lift} (range {lo} to {hi}), {shoppers} shoppers over {days}.
`plan_downgraded`	The store’s plan no longer includes tests, so the test ended	{test} on {quiz}: ended because the store’s plan no longer includes tests; the live version stays. {metric}: {lift} (range {lo} to {hi}), {shoppers} shoppers over {days}.
`ended_manual`	You ended it without a winner	{test} on {quiz}: ended with no change, the live version stays. {metric}: {lift} (range {lo} to {hi}), {shoppers} shoppers over {days}.

Where each part comes from:

{test}, {winner} and {days}: the event’s name, the label of its winner (null when the test ended without one), and the time from started_at to ended_at, rounded to whole days and never less than one.
{quiz}: name from GET /v1/quizzes/{quiz_id}; the event carries only the quiz_id.
{metric}: the results’ experiment.decision_metric, one of the keys the Experiments page lists, turned into a readable name.
{lift}, {lo} and {hi}: the entry of the results’ metrics whose role is decider. lift is the variant’s difference from the live version as a share of the live version’s figure (0.23 is +23%), and lo and hi are the range around it on the same scale. They are null when there is too little data or the live version’s figure is too close to zero to size a change; the message then says too early to measure.
{shoppers}: the results’ split.n, the shoppers counted on both versions.

The lift is always the variant against the live version, so when the live version wins, the figure is the variant’s shortfall: Live version won on Completion rate, -9% (range -15% to -3%).

Things to know

Deliveries are at least once. SEEN keeps the webhook-id of each event already posted and is recorded only after Slack accepted the message, so a failed post answers an error and Octane AI retries it. It lives in memory and is lost on a restart; in production, record the id with the job as Verify a webhook signature does.
Answer within 10 seconds. The receiver makes two reads and one Slack post inside the request, so a slow Slack or API makes the attempt fail and be retried; in production enqueue the event, answer 200 and post from the job.
Other events are acknowledged and ignored. The first delivery to a new endpoint is a ping; cancelled is a draft ended before it started, with no results to read; and an outcome the table above does not name gets no sentence. All three answer 200 and post nothing, so a value added later never fails a delivery.
A winner publishes a version. A quiz.published for the winning version arrives beside experiment.ended; this endpoint subscribes only to experiment.ended, so it posts once.
Cost. The results route is an analytics call (3 units) and the quiz read a plain read (1 unit), so each ended test costs 4 units (see Rate limits).

Test result to Slack the moment it ends

The flow

1. Subscribe

2. The receiver

The message

Things to know