Goals and A/B experiments

A goal says what counts as a successful call, for example "the caller booked a slot". An experiment splits calls between variants of your agent, for example a formal and a casual greeting, so you can see which variant reaches your goals more often. You declare both in your agent's code, next to the agent itself, and they go live with your next deployment.

Declare goals and experiments

Add goals and experiments to defineAgent in src/agent/agent.tsx:

import { defineAgent } from "curb";

export const agent = defineAgent({
  app: BookingAgent,
  goals: {
    booked: { description: "The caller booked a webinar slot", tool: "book_slot" },
    felt_easy: {
      description: "Booking felt easy",
      judge: "The caller says booking with the assistant was easy or pleasant.",
    },
  },
  experiments: {
    greeting: {
      description: "Formal vs casual opening",
      variants: { formal: 1, casual: 1 },
      fromClient: "greeting",
    },
  },
});

A goal is one of two kinds:

• Tool goal (tool): reached when the agent ran that tool during the call and the tool succeeded. Name one of your agent's tools. npx curb check warns when no tool of your agent has that name.
• Judge goal (judge): a sentence describing what should be true of the conversation, 1 to 1000 characters. Judge goals are not judged yet: every call shows them as not judged for now, so declare them today and their results follow once judging is available.

An experiment lists 2 to 6 variants, each with a weight from 1 to 100. The weights are relative: { formal: 1, casual: 1 } gives each variant half the calls, and { formal: 1, casual: 3 } gives casual three calls in four. Each call gets one variant of every experiment when it starts and keeps it until it ends. The same call always gets the same variant, so you can trace any call back to the variant it had.

The limits:

• Goal, experiment and variant names start with a lowercase letter and have at most 40 lowercase letters, digits and underscores, such as booked or casual_v2.
• At most 20 goals and 5 experiments per agent.
• A description is optional and holds at most 200 characters. It is for the people reading the results; the agent never sees it.

npx curb check reports goals and experiments outside these limits and names every problem, and a deployment with them fails to build, so a mistake never reaches a live call.

Use the variant in your agent

useVariant returns the variant the current call got, so the agent can behave differently per variant:

import { Instruction, useVariant } from "curb";

function Greeting() {
  return useVariant("greeting") === "casual" ? (
    <Instruction>Greet the caller casually and use their first name when you know it.</Instruction>
  ) : (
    <Instruction>Greet the caller formally.</Instruction>
  );
}

Render it inside your <Agent>, like any other instruction. The variant is set before the agent's first words, so the greeting can depend on it. useVariant returns undefined for an experiment your agent does not declare.

Choose a variant from the page

fromClient names a key of the context your web page passes to the call. When the call's context has that key and its value is one of the experiment's variants, the call gets that variant instead of a random one. Any other value is ignored and the call gets a variant by weight as usual.

With fromClient: "greeting", a widget tag with data-context-params="greeting" lets a link choose the variant: https://example.com/call?greeting=casual always starts a call with the casual greeting. Use this to send one audience to one variant, or to try a variant yourself on the live agent. How a page passes context is in Tell the agent where a call comes from.

Try each variant locally

Every conversation in npx curb dev gets its own variants, chosen the same way as on a live call, so a new conversation can get another variant. To try one variant, add the experiment's fromClient key to the preview's address:

http://127.0.0.1:4000/?client.greeting=casual

In a test scenario, experiments fixes the variant for that conversation. An experiment the scenario leaves out gets a fixed variant chosen from the scenario's file and name, so the same scenario runs with the same variants every time:

import { scenario } from "curb/testing";

export const casualGreeting = scenario({
  experiments: { greeting: "casual" },
  turns: ["Hallo, ich möchte einen Termin buchen."],
  expect: ({ called }) => [called("book_slot")],
});

Keep one scenario per variant for every conversation that has to work in each of them, and run them with npx curb test.

See the results

The dashboard shows goals and experiments; you change them only in code.

• Each call on your agent's Calls page shows the variant it got in every experiment and, shortly after the call ends, the result of every goal. The calls table has a Goals column and an A/B test filter that lists only the calls of one variant.
• The agent's Goals page shows each goal of the live version with how it is checked and how often calls reached it over the last 30 days, the same rates for every version, and for each experiment the share of calls, the number of calls and the goal rates of every variant.

Each rate comes with a 95% range. While the ranges of two variants or versions overlap, the page says there are too few calls to tell them apart rather than naming a winner. Calls whose goals are not judged or not evaluable are left out of the rates and counted beside them.

A goal's result is one of:

Changing a goal or an experiment takes effect with the next deployment. Calls keep the results and variants they had, so the calls of an earlier deployment still show what they were measured against.

Next steps