Answer from your backend
Look up what the caller asks for in your own systems and have the agent tell them the answer.
On this page
When a turn needs your systems, you get session.delegation.created. It carries an id and no
text, so keep the transcript yourself. Do the work in a task of its own so audio keeps flowing,
then send the result under that id. The examples on this page replace functions in talk.py
from the quickstart.
Send the result
Find the order in what the caller said and send the facts back as commentary. The agent says them in its own words. Callers say numbers as words, so this lookup reads both.
import re
ORDERS = {"4711": "Order 4711 shipped yesterday and arrives tomorrow."}
DIGITS = {"zero": "0", "one": "1", "two": "2", "three": "3", "four": "4",
"five": "5", "six": "6", "seven": "7", "eight": "8", "nine": "9"}
def look_up_order(caller_said: str) -> str:
words = re.findall(r"[a-z0-9]+", caller_said.lower())
number = "".join(DIGITS.get(word, word) for word in words if word in DIGITS or word.isdigit())
return ORDERS.get(number, "There is no order with that number. Ask the caller to repeat it.")Send facts, not finished sentences. From "Order 4711 arrives tomorrow", the agent makes a reply that fits the conversation.
Choose how the agent uses what you send
| Send | The agent |
|---|---|
connection.session.commentary.append | Tells the caller. Send results here. |
connection.session.thinking.append | Knows it and does not say it. Send progress here. |
connection.session.instructions.append | Changes how it behaves from now on. |
Each takes content of at most 500 tokens and a delegation_id, the id of the delegation it
belongs to. These are OpenAI's append events.
Use the caller's whole turn
session.input_transcript.delta streams words while the caller speaks. For a lookup, use the
final text of the turn instead. Add "kugel": {} to the session and keep the last
kugel.input_transcript.committed, which arrives before the delegation:
turn = ""
async for event in connection:
if event.type == "kugel.input_transcript.committed":
turn = event.text
elif event.type == "session.delegation.created":
tasks.add(asyncio.create_task(answer(connection, event.delegation.id, turn)))Report progress on slow work
The agent tells the caller it is checking, then waits for your result. While a slow lookup runs, send progress as thinking so the agent knows what is happening without reading it out:
async def answer(connection, delegation_id: str, caller_said: str) -> None:
await connection.session.thinking.append(
delegation_id=delegation_id,
content="Checking the order now. Nothing is confirmed yet.",
)
await asyncio.sleep(5) # your slow lookup
result = await asyncio.to_thread(look_up_order, caller_said)
await connection.session.commentary.append(delegation_id=delegation_id, content=result)Ask the caller a question
To ask something and carry on with the answer, send a commentary with expects_reply. The agent
says it word for word, and the caller's next finished turn opens a new delegation. The SDK's
append has no argument for it, so send the event yourself.
This answer confirms before it cancels an order:
confirming = False
async def answer(connection, delegation_id: str, caller_said: str) -> None:
global confirming
if not confirming:
confirming = True
await connection.send(
{
"type": "session.commentary.append",
"delegation_id": delegation_id,
"content": "Shall I cancel order 4711?",
"expects_reply": True,
}
)
return
confirming = False
said_yes = "yes" in re.findall(r"[a-z]+", caller_said.lower())[-6:]
await connection.session.commentary.append(
delegation_id=delegation_id,
content="Order 4711 is cancelled." if said_yes else "Order 4711 stays as it is.",
)Set the greeting and what the agent knows
The agent speaks first. Tell it what to open with, and what you already know about the caller,
in instructions:
session = {
"model": "fluid-1",
"instructions": (
"You answer the phone for a parcel service. Keep replies short. "
"Open every call with exactly: Good morning, parcel service, this is Mia. "
"The caller has one open order, number 4711."
),
"delegation": {"type": "client"},
}Say something at any time
Send a commentary with no delegation, and the agent says it word for word at the next opportunity:
await connection.session.commentary.append(
delegation_id=None,
content="A quick note: our depot closes at six today.",
)If the caller finishes a turn before the agent has said it, the line is dropped, because that turn may have changed the subject.
Change how the agent behaves
Add to the agent's instructions during the call, for example when your system learns something that changes the rules:
await connection.session.instructions.append(
delegation_id=None,
content="The caller's identity is confirmed. You can now give delivery details.",
)Choose what the agent delegates
By default the agent delegates the work your instructions list for a back office or tools, or,
when they list none, anything it needs your backend for. To choose exactly, describe your tools
in session.kugel.tools. With [] the agent never delegates. Your code still runs the work.
session = {
"model": "fluid-1",
"instructions": "You answer the phone for a parcel service. Keep replies short.",
"delegation": {"type": "client"},
"kugel": {
"tools": [
{
"name": "find_order",
"description": "Find an order and where it is by its number.",
"parameters": {"type": "object", "properties": {"number": {"type": "string"}}},
}
]
},
}The agent decides when a turn needs delegating, so test your calls with the instructions and tools you plan to use.