Skip to content
Octane AIOctane AI

Alert when a page loses shoppers after a publish

Goal: when an edit to a question makes shoppers leave, know which page it was within a day, without watching the funnel by hand.

Uses: the quiz.published webhook to know the moment a version went live, and GET /v1/quizzes/{quiz_id}/analytics/funnel from Analytics for each page’s dropoff_share_of_starts before and after. Scopes: webhooks:manage to create the endpoint, analytics:read for the funnel.

The flow

  1. Subscribe an endpoint to quiz.published once.
  2. When a publish arrives, the receiver reads the funnel for the seven closed days before published_at and stores it as the baseline for that quiz: starts and, per page, dropoff_share_of_starts. Those days will not change any more, so the snapshot can be taken at once.
  3. A daily job reads the funnel from the publish day to today for every quiz published in the last seven days and compares each page with its baseline: a page whose share of starts lost rose by five points or more, with enough starts to mean it, is the alert.

1. Subscribe

Terminal window
curl -X POST https://api.octaneai.com/v1/webhooks \
  -H "Authorization: Bearer $OCTANE_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{"url": "https://hooks.example.com/octane", "topics": ["quiz.published"], "description": "Funnel watch"}'

The event the endpoint will receive:

{
  "id": "evt_6ba7b8109dad11d180b400c04fd430c8",
  "type": "quiz.published",
  "created": "2026-09-03T09:00:00Z",
  "api_version": "v1",
  "data": {
    "object": {
      "quiz_id": "quiz_7c9e6679742540de944be07fc1f90ae7",
      "quiz_name": "Find your routine",
      "version_id": "ver_15ce0be06b904d0180b1155da02a6c6f",
      "version_number": 2,
      "published_at": "2026-09-03T09:00:00Z"
    }
  }
}

2. Snapshot the funnel when a publish arrives

The receiver verifies the signature as Verify a webhook signature does, then reads the funnel for the seven days before the publish and writes baseline-<quiz_id>.json (Flask and httpx, pip install flask httpx; Express; Go’s net/http):

import base64
import datetime as dt
import hashlib
import hmac
import json
import os
import time
from zoneinfo import ZoneInfo
import httpx
from flask import Flask, request


app = Flask(__name__)
SECRET = os.environ["OCTANEAI_WEBHOOK_SECRET"]  # whsec_...
HEADERS = {"Authorization": f"Bearer {os.environ['OCTANE_API_KEY']}"}
API = "https://api.octaneai.com/v1"
TZ = "America/New_York"  # the timezone the daily job uses too


def verify(secret: str, headers, body: bytes) -> bool:
    try:
        key = base64.b64decode(secret[len("whsec_"):], validate=True)
        msg_id, ts, sigs = headers["webhook-id"], headers["webhook-timestamp"], headers["webhook-signature"]
        if abs(time.time() - int(ts)) > 300:
            return False
        expected = hmac.new(key, f"{msg_id}.{ts}.".encode() + body, hashlib.sha256).digest()
        for sig in sigs.split(" "):
            version, _, value = sig.partition(",")
            if version == "v1" and hmac.compare_digest(base64.b64decode(value, validate=True), expected):
                return True
    except (KeyError, ValueError):
        pass
    return False


def snapshot(published: dict) -> None:
    day = dt.datetime.fromisoformat(published["published_at"].replace("Z", "+00:00")).astimezone(ZoneInfo(TZ)).date()
    r = httpx.get(f"{API}/quizzes/{published['quiz_id']}/analytics/funnel", headers=HEADERS, timeout=30,
                  params={"from": day - dt.timedelta(days=7), "to": day - dt.timedelta(days=1), "tz": TZ})
    r.raise_for_status()
    funnel = r.json()
    baseline = {
        "quiz_id": published["quiz_id"], "quiz_name": published["quiz_name"], "version_number": published["version_number"],
        "published_day": day.isoformat(), "starts": funnel["starts"],
        "pages": {p["page_key"]: {"title": p["title"], "dropoff_share": p["dropoff_share_of_starts"]} for p in funnel["pages"] if not p["removed"]},
    }
    with open(f"baseline-{published['quiz_id']}.json", "w") as f:
        json.dump(baseline, f)
    print("baseline", published["quiz_id"], "version", published["version_number"], funnel["starts"], "starts")  # baseline quiz_... version 2 1840 starts


@app.post("/octane")
def octane():
    if not verify(SECRET, request.headers, request.get_data()):
        return "bad signature", 401
    event = request.get_json()
    if event["type"] == "quiz.published":
        snapshot(event["data"]["object"])  # in production: enqueue, then answer; see the signature recipe
    return "", 200


if __name__ == "__main__":
    app.run(port=8080)

3. Compare, once a day

For every baseline younger than seven days, read the funnel from the publish day to today and compare page by page:

import datetime as dt
import glob
import json
import os
from zoneinfo import ZoneInfo
import httpx


HEADERS = {"Authorization": f"Bearer {os.environ['OCTANE_API_KEY']}"}
API = "https://api.octaneai.com/v1"
SLACK = os.environ["SLACK_WEBHOOK_URL"]
TZ = "America/New_York"
MIN_STARTS, MIN_POINTS = 100, 5.0  # enough starts to mean it; a rise in the share of starts lost, in points


today = dt.datetime.now(ZoneInfo(TZ)).date()
alerts = []
for path in glob.glob("baseline-*.json"):
    with open(path) as f:
        before = json.load(f)
    published = dt.date.fromisoformat(before["published_day"])
    if (today - published).days > 7:
        continue
    r = httpx.get(f"{API}/quizzes/{before['quiz_id']}/analytics/funnel", headers=HEADERS, timeout=30,
                  params={"from": published, "to": today, "tz": TZ})
    r.raise_for_status()
    after = r.json()
    print(before["quiz_name"], "version", before["version_number"], "starts before", before["starts"], "after", after["starts"])
    if after["starts"] < MIN_STARTS:
        continue
    for page in after["pages"]:
        if page["removed"] or page["dropoff_share_of_starts"] is None:
            continue
        was = before["pages"].get(page["page_key"])
        if was is None:
            print(f"  new page {page['page_key']} ({page['title']}): {page['dropoff_share_of_starts']}% of starts leave here")
            continue
        rise = page["dropoff_share_of_starts"] - (was["dropoff_share"] or 0)
        print(f"  {page['page_key']} {was['dropoff_share']}% -> {page['dropoff_share_of_starts']}%")
        if rise >= MIN_POINTS:
            alerts.append(f"{before['quiz_name']} v{before['version_number']}: page '{page['title']}' ({page['page_key']}) now loses "
                          f"{page['dropoff_share_of_starts']:.1f}% of starts, was {was['dropoff_share']:.1f}% the week before the publish.")
if alerts:
    httpx.post(SLACK, json={"text": "\n".join(alerts)}, timeout=30).raise_for_status()
print(len(alerts), "alerts")

The alert reads: Find your routine v2: page 'Your skin' (skin) now loses 19.4% of starts, was 11.1% the week before the publish.

Errors to handle

StatusWhyWhat to do
401 from your receiverThe signature did not verify: a wrong secret, a replay older than five minutes, or a body you modified before hashing.Compare the raw bytes; rotate the secret from the dashboard if it leaked.
403 insufficient_scopeThe key lacks webhooks:manage (subscribe) or analytics:read (funnel).Mint a key with both.
404 not_foundThe quiz was deleted between the publish and the comparison.Drop its baseline.
422 validation_errorto before from (a publish dated in the future by a wrong tz), or country on funnel, which takes no country filter.Use one tz in both jobs.
429 rate_limitedMany quizzes published the same week.Sleep for Retry-After and continue; every read is idempotent.

Things to know

  • What dropoff_share_of_starts is. dropoffs counts the visits that ended on the page: the shopper closed the quiz, went idle or started again. The share divides it by the quiz’s starts in the range so pages and ranges of different sizes compare; a shopper can leave the first page without answering, which counts as a drop-off but not as a start, so a busy first page can read above 100. It is the number to compare; dropoffs alone grows with traffic.
  • Two ranges, one timezone. The baseline is the seven closed days before the publish day; the comparison runs from the publish day to today (today is read live). Both jobs send the same tz, or the publish day lands in different days on each side. The publish day itself mixes both versions; a job that runs from the day after is stricter.
  • A second publish resets the baseline. Each quiz.published overwrites baseline-<quiz_id>.json, so the comparison always reads against the version before the latest one. The quiz overview’s markers[] lists every publish with its day if you want the history.
  • Pages that changed. A page the new version no longer has answers removed: true and is skipped; a page the old version did not have has no baseline and is reported as new rather than compared. A renamed page keeps its page_key, so it is compared normally.
  • A new endpoint is pending until its ping lands. Events are queued only to active endpoints: the endpoint answers the verification ping with a 2xx first (GET /v1/webhooks/{endpoint_id} shows status), so start the receiver before you subscribe, or a publish made in between is not delivered. Creating an endpoint with a URL you already registered answers 200 with the existing endpoint instead of a second one.
  • Deliveries are at least once. A redelivery of the same publish rewrites the same baseline with the same numbers, so the receiver needs no dedup; keep the webhook-id check from the signature recipe if the handler ever does more.
  • Cost. A funnel read is 3 units; the snapshot and the daily comparison are one read per published quiz.