A/B test which discount converts best

Why this matters

The classic coupon mistake: marketing increases the discount from 10% to 20%, conversion rate goes up 15%, everyone celebrates. Six months later finance points out revenue-per-visitor dropped 4% because the bigger discount cannibalized margin faster than it grew the funnel.

Optimizing for conversion rate alone is a trap. The real metric is revenue per visitor (or per session, or per distinct_id) — that’s the number that pays for ads, payroll, and stock. A 20% coupon that converts 15% better but earns 4% less per visitor is worse, not better.

Every traditional A/B tool tracks conversion. Few make it dead-easy to track conversion and downstream revenue in the same experiment. Agentry does both in one prompt: the agent creates the flag, captures the cohort assignment, and prepares protected metric views from live events.

What you get

• A feature flag with control/variant rollout that’s stable per distinct_id
• Two protected metric views: conversion-rate-by-variant and revenue-per-visitor-by-variant
• A clear winner after ~7 days at typical traffic levels (or a clear “ran longer, no signal”)

Walk through it

Endpoint: POST /v1/projects/:project_id/feature-flags
Exact shape: /v1/openapi.json?path=%2Fv1%2Fprojects%2F%7Bproject_id%7D%2Ffeature-flags&method=post
Concept fields:
  key: "coupon_discount_pct",
  name: "Coupon discount A/B test",
  variants: [
    { key: "control", rollout_percentage: 50, payload: { discount: 10 } },
    { key: "variant", rollout_percentage: 50, payload: { discount: 20 } }
  ]

{
  "flag_id": "flag_cpn_1",
  "key": "coupon_discount_pct",
  "variants": [
    { "key": "control", "rollout_percentage": 50 },
    { "key": "variant", "rollout_percentage": 50 }
  ]
}

# Coupon A/B comparison is a custom analysis in this playbook.
# Keep the result in a protected server-side view using analytics/query.
# Only publish a browser-safe public-query URL later if the user asks and a real
# query blueprint backs this comparison.

const res = await fetch(`https://api.agentry.sh/v1/analytics/`, { /* see Setting it up */ });
// in checkout page:
const variant = await evaluateFlag("coupon_discount_pct", distinct_id);
const discountPct = variant.payload?.discount ?? 10;

Run for 7 days at your current traffic (~3k orders/day) — that gives ~10k orders per arm, enough for a ~3% MDE on both conversion and RPV. I’ll check back next Tuesday.

The output

Coupon A/B test (7-day check-in)

Variant      Visitors    Orders     Conv %     Revenue/Visitor
─────────────────────────────────────────────────────────────────
control       21,432      1,287     6.0%       $4.21
variant       21,189      1,498     7.1%       $4.08

Conversion lift: +18% (variant) ★
Revenue/Visitor: -3.1% (variant) ★

Verdict: VARIANT LOSES.
  The bigger discount converts more shoppers but earns 3% less per
  visitor — the extra orders don't pay for the extra discount.
  At your annual volume that's ~$340k of margin lost if you ship it.

Next experiment to consider:
  - 15% (split-the-difference)
  - 20% but only for new customers
  - 10% + free shipping (vs 20% raw)

Setting it up

1. Apply the coupon based on the flag. The flag tells the front-end which discount to render. Read at the moment the cart total is computed:

// Server-side eval (recommended — keeps the api_key off the client).
async function evaluateFlag(key: string, distinct_id: string) {
  const res = await fetch(
    `https://api.agentry.sh/v1/projects/${PROJECT_ID}/feature-flags/evaluate`,
    {
      method: "POST",
      headers: {
        "Authorization": `Bearer ${process.env.AGENTRY_PRIVATE_API_KEY}`,  // agentry_sk_... (NOT AGENTRY_PUBLIC_API_KEY)
        "Content-Type": "application/json",
        "User-Agent": "myshop/1.0",  // REQUIRED — Cloudflare 403s default UAs
      },
      body: JSON.stringify({ key, distinct_id }),
    },
  );
  return res.json();  // { value: "control" | "variant" } — deterministic per user
}

2. Include total (post-discount revenue) on order_completed. Otherwise revenue-per-visitor can’t compute:

await fetch(`https://api.agentry.sh/v1/analytics/`, {
  method: "POST",
  headers: {
    "Authorization": `Bearer ${process.env.AGENTRY_PUBLIC_API_KEY}`,
    "Content-Type": "application/json",
    "User-Agent": "myshop/1.0",
  },
  body: JSON.stringify({
    event: "order_completed",
    distinct_id: user.email,
    properties: {
      total: order.total_after_discount,
      discount_applied: order.discount_pct,
      coupon_variant: variant.variant,  // for cross-check
    },
  }),
});

3. Stable assignment. The flag hashes distinct_id to a variant — same visitor always sees the same coupon. Don’t pass a random seed; pass the stable per-shopper id (email or cookie visitor id).

Variations

• “Same test but only for first-time visitors — repeat customers always get 10%.”
• “Three-arm: 0% (baseline), 10% (control), 20% (variant). Is the coupon even helping vs no coupon?”
• “Stop the experiment now and ship the winner — also clean up the flag so I don’t have a dead toggle.”
• “Re-run, but segment results by traffic source — does paid traffic behave differently than organic?”