Run a pricing A/B test with zero infrastructure

Why this matters

A/B testing pricing is the highest-leverage experiment most SaaS companies run. A 15% lift in checkout conversion is worth more than 6 months of “growth hacks.”

The reason teams don’t do it: infrastructure tax. Setting up Statsig / LaunchDarkly / Optimizely is a 1-week procurement + integration cycle. Doing it in-house means flag-eval code + experiment-config table + conversion-tracking pipeline + stats calculation. Founders pick “ship the new price and pray” because the alternative takes a week.

Agentry can coordinate the experiment from an agent-first API, but the core proof is still analytics: define the conversion events, verify the flag/exposure events and subscription outcome exist, then compare the variants with HogQL. Feature flags, cohorts, and public dashboards are advanced follow-up surfaces, not the default first install path.

What you get

• A feature flag your app checks to decide which price to show
• A variant comparison grounded in exposure and subscription events
• Optional cohorts or browser-safe output only when you explicitly ask for those advanced surfaces
• A “stop the test early if X” rule

Walk through it

Endpoint: POST /v1/projects/:project_id/feature-flags
Exact shape: /v1/openapi.json?path=%2Fv1%2Fprojects%2F%7Bproject_id%7D%2Ffeature-flags&method=post
Concept fields:
  key: "pricing_v2_test",
  name: "Pricing V2 — $49/$149 vs $29/$99",
  filters: {
    multivariate: {
      variants: [
        { key: "control", rollout_percentage: 50 },
        { key: "variant", rollout_percentage: 50 }
      ]
    }
  }

{ "flag_id": "ff_8c12", "key": "pricing_v2_test", "active": true }

Endpoint: GET /v1/projects/:project_id/verify-report
Endpoint: GET /v1/projects/:project_id/event-property-keys?events=$feature_flag_called,subscription_started
Proof needed before comparison:
  $feature_flag_called has feature_flag + value
  subscription_started has plan + price_variant or enough user/account identity to join back to exposure

{ "answer_ready": true, "properties": ["feature_flag", "value", "plan"] }

# Pricing experiment comparison is a custom analysis in this playbook.
# Keep the result in a protected server-side view using analytics/query, or add
# a real query blueprint for this comparison before publishing a public-query URL.

{
  "status": "internal_result_ready",
  "publication": "requires a real query blueprint before public-query publishing"
}

The output

A protected/internal results view until this custom analysis has a real query blueprint:

📊 Pricing V2 Test — Day 7 of 14

                     Control ($29/$99)    Variant ($49/$149)
Exposures            2,140                2,108
Subscriptions         97                   84
Conversion           4.53%                3.98%
Lift                 baseline             -12.1%
95% CI               [3.69%, 5.49%]       [3.20%, 4.91%]
p-value              —                    0.346 (not significant)

VERDICT (day 7): Variant is trending worse but not significantly.
Continue test to day 14 unless lift becomes definitive (>30%).

Effective ARPU:
  Control:  $29 × 4.53%  = $1.31 expected revenue per visitor
  Variant:  $49 × 3.98%  = $1.95 expected revenue per visitor (+48%)

Even with lower conversion, variant has higher revenue per visitor.
This is the metric that matters.

Setting it up

Your app needs to check the flag and behave accordingly. Agentry has no SDK — the flag-eval call is a raw POST to /v1/projects/<project_id>/feature-flags/evaluate. Note this endpoint takes an API key (agentry_sk_...), not AGENTRY_PUBLIC_API_KEY — flag config is owner-side, so it lives behind the same auth as cases, deploys, and other owner APIs.

// Server-side flag eval (recommended — fast, no race with page render,
// and the API key never reaches the browser)
const projectId = process.env.AGENTRY_PUBLIC_API_KEY!.slice("agentry_pk_".length).split(".")[0];

async function evaluateFlag(key: string, distinctId: string) {
  const res = await fetch(
    `https://api.agentry.sh/v1/projects/${projectId}/feature-flags/evaluate`,
    {
      method: "POST",
      headers: {
        "Authorization": `Bearer ${process.env.AGENTRY_PRIVATE_API_KEY}`,  // agentry_sk_... (NOT AGENTRY_PUBLIC_API_KEY)
        "Content-Type": "application/json",
        "User-Agent": "myapp/1.0",  // REQUIRED — Cloudflare 403s default UAs
      },
      body: JSON.stringify({ key, distinct_id: distinctId }),
    },
  );
  const { value } = await res.json() as { value: string };
  return value;  // "control" | "variant" — deterministic per user
}

export async function loader({ request }) {
  const userId = await getUserIdFromRequest(request);
  const variant = await evaluateFlag("pricing_v2_test", userId);
  return { variant };
}

// In your component
function PricingPage({ variant }) {
  const prices = variant === "variant"
    ? { pro: 49, scale: 149 }
    : { pro: 29, scale: 99 };
  return <PricingCards prices={prices} />;
}

// Capture the conversion event when user upgrades — analytics ingest uses
// AGENTRY_PUBLIC_API_KEY (different token from the flag eval above)
async function handleUpgrade(plan: string, userEmail: string, variant: string) {
  await fetch(`https://api.agentry.sh/v1/analytics/`, {
    method: "POST",
    headers: {
      "Authorization": `Bearer ${process.env.AGENTRY_PUBLIC_API_KEY}`,  // agentry_pk_… (NOT the api key)
      "Content-Type": "application/json",
      "User-Agent": "myapp/1.0",
    },
    body: JSON.stringify({
      event: "subscription_started",
      distinct_id: userEmail,
      properties: {
        plan,
        amount_cents: prices[plan] * 100,
        pricing_variant: variant,  // optional but useful for ad-hoc queries later
      },
    }),
  });
}

If you’d rather evaluate client-side, proxy the flag call through your own backend — the agentry_sk_ key must never reach the browser.

Variations

• “3-way test: $29, $39, $49. Adjust the variant comparison accordingly.”
• “Only show variant B to users who came from organic search, but first verify the source property exists.”
• “Add a guardrail: if variant conversion drops below 50% of control, kill the variant automatically.”
• “Run this experiment but on enterprise tier specifically — different signal than self-serve.”
• “After the test ends, draft a writeup post for the company blog summarizing what we learned.”