Vapi, Retell, LiveKit and Braintrust

#Vapi

Set the assistant's Server URL to the agent's Vapi webhook, and in Vapi create a Bearer Token credential (header Authorization, Bearer prefix on) whose token is a workspace API key, and select it as the server's credential. Duet Machines ingests only end-of-call-report messages. Every other message type Vapi documents that expects no answer is answered (202, {"ignored": "<type>"}) and counted on the Connect panel, so Vapi logs no error; limit the assistant's server messages to the report to skip that traffic. Messages that expect an answer from your server (assistant-request, tool-calls, transfer, handoff, knowledge-base, voice, endpointing and campaign requests) are refused (400): point them at your own server. An undocumented type or a malformed report is refused (422, naming the fields and reasons, never their values), as is any message sent without the key (401).

https://platform.duetmachines.com/api/integrations/vapi/<agent uid>

{
  "server": {
    "url": "https://platform.duetmachines.com/api/integrations/vapi/<agent uid>",
    "credentialId": "<a Bearer Token credential whose token is your DUET_API_KEY>"
  },
  "serverMessages": ["end-of-call-report"]
}

Turns

artifact.messages: user is a customer turn timed at its endTime, bot (or assistant) an agent turn timed at its time, and a tool_calls call answered by a later tool_call_result a tool turn (status executed, the result as sent, its duration from the call's time to the result's). A call the report holds no result for is not a turn; a result that answers no call, or a message role not listed here, refuses the report.

Config

The assistant's system messages as the instructions, its tools, and artifact.variableValues for the instructions' {{placeholders}}. A tool counts with the function definition it holds (function, code and handoff tools). Of Vapi's built-in tools, endCall becomes an endCall function with no parameters, and transferCall a transferCall function whose required destination is one of its destinations (number, SIP URI or assistant name), each described by its description. A transferCall without destinations and other built-ins (dtmf, sms, ...) have no published definition, so an assistant using one records no config rather than a guessed definition. Without a config the conversation is judged against the agent's live version, and the Connect panel says why the latest report had none.

Vapi API key

When a report has no assistant (a saved assistant's call.assistantId only) or uses saved tools (toolIds), save the workspace's Vapi private API key on the agent's Traffic (owners and admins; stored encrypted, never shown again, audited). Duet Machines then reads GET /assistant/{id} and GET /tool/{id} from Vapi's API, reusing each answer for 5 minutes per assistant version (reports without assistantVersion are read every time). An assistant or tool edited after the call started, or an assistant whose latest version is not the call's, is not what the call ran, so no config is recorded; a Vapi error is shown, never filled in.

Other fields

call.id is the conversation's external_id, startedAt its start (required), assistantVersion its release; endedReason, call.type, call.assistantId and artifact.recordingUrl become vapi.* metadata. The recording is never downloaded.

#Retell

Set the agent's Retell webhook URL as the Retell agent's webhook_url with webhook_events call_ended and call_analyzed, and save the Retell API key that has the webhook badge on the agent's Traffic (owners and admins; it is stored encrypted and never shown again). Retell signs each webhook with that key: X-Retell-Signature is v=<ms>,d=<hex HMAC-SHA256 of the raw body followed by the ms>; Duet Machines refuses a missing or wrong signature, or one older than 5 minutes (401), and a webhook for an agent with no key (404). Retell's other documented events (call_started, which is in its default set, transcript_updated, transfer and chat events) are answered (202) and counted on the Connect panel, since Retell retries any other answer 3 times; an undocumented event or a malformed call is refused (422).

https://platform.duetmachines.com/api/integrations/retell/<agent uid>

Turns

transcript_with_tool_calls: user and agent utterances, timed from start_timestamp by the customer's last word's end and the agent's first word's start (no time without words); a tool_call_invocation answered by a later tool_call_result is a tool turn, failed when successful is false and executed otherwise (Retell leaves it out when it was not told). Node transitions are skipped; DTMF, SMS, injected context and a transfer target's speech refuse the call.

Config

Retell's webhook carries no prompt or tools, so its conversations are judged against the agent's live version in Duet Machines.

Other fields

call_id is the external_id, start_timestamp the start (required), agent_version the release; call.metadata is kept as metadata (non-string values as JSON), with retell.agent_id, retell.call_type, retell.disconnection_reason and retell.recording_url. The recording is never downloaded.

#LiveKit Agents

With LiveKit Agents 1.7 or later, record each AgentSession with the Python SDK from its events: conversation_item_added adds the user's and the assistant's committed messages, function_tools_executed each call with its output (failed when is_error) and the time between them, and close ends the conversation. Pass the config to dm.conversation as in the SDKs to have it judged against what the agent ran with.

# pip install duetmachines  (LiveKit Agents 1.7 or later)
import json
import os

from duetmachines import DuetMachines
from livekit.agents import AgentSession, CloseEvent, ConversationItemAddedEvent, FunctionToolsExecutedEvent
from livekit.agents.llm import ChatMessage

dm = DuetMachines(os.environ["DUET_API_KEY"], agent="<agent uid>")


def record(session: AgentSession, conversation_id: str) -> None:
    """Send the session's turns and tool calls as one conversation; call it before session.start()."""
    conversation = dm.conversation(conversation_id, version=os.environ.get("GIT_SHA"))

    @session.on("conversation_item_added")
    def item_added(event: ConversationItemAddedEvent) -> None:
        item = event.item
        if not isinstance(item, ChatMessage) or not item.text_content:
            return
        if item.role == "user":
            conversation.customer(item.text_content)
        elif item.role == "assistant":
            conversation.agent(item.text_content)

    @session.on("function_tools_executed")
    def tools_executed(event: FunctionToolsExecutedEvent) -> None:
        for call, output in event.zipped():
            status = "failed" if output.is_error else "executed"
            # From the model's call to the tool's output, as LiveKit stamped them.
            seconds = round(output.created_at - call.created_at, 3)
            conversation.tool(call.name, json.loads(call.arguments), output.output, status, duration_seconds=seconds)

    @session.on("close")
    def closed(event: CloseEvent) -> None:
        conversation.end()


# In your entrypoint: record(session, ctx.room.name), then await session.start(...)

#Braintrust

If the agent is already traced in Braintrust, Duet Machines imports its conversations from the project with no code change: save a Braintrust API key (owners and admins; stored encrypted, never shown again, audited; US or EU data plane) on the agent's Traffic, pick the project, and Duet Machines reads it every 60 seconds through Braintrust's documented API (GET /v1/project, SQL over project_logs with POST /btql). It only reads: nothing is written to Braintrust. Each read starts after the last change it stored, so a trace that changes is read again and its conversation updated, and a trace read twice stores nothing twice. One agent per project in a workspace; removing the key stops every import, and imported conversations stay.

Conversation

One Braintrust trace (its root_span_id is the external_id), as Braintrust recommends for multi-turn agents. If you log one trace per turn (Braintrust's "separate trace per turn"), set Thread metadata key when you link the project: the root span's metadata key that holds the thread id (thread_id, or a dotted path like metadata.thread_id for LangChain's handler). Every trace whose root holds the same id is then one conversation, its external_id the thread id and its metadata braintrust.thread_key the key; a root without a string id there is refused, never guessed. The key cannot change once conversations were imported under another one (409 thread_key_differs_from_imported).

Conversation span

Without one, every model-call span may hold the conversation and the latest-ending one is read. A classifier or summarizer on the same client logs the same span name, so set Conversation span name (span_attributes.name, e.g. a traced function named reply around the agent's calls, or span_info={"name": ...} on the call) and only spans with that name, or inside one, are read. When the model calls of a conversation hold different histories (a side call), the trace is refused asking for the name instead of picking one.

Formats

The span's logged input, output and metadata, as Braintrust's own integrations write them: OpenAI Chat Completions (messages and choices); OpenAI Responses (input items with function_call and function_call_output, the response's output items, instructions as the prompt; a call continuing an OpenAI-stored conversation with previous_response_id or conversation is refused, its earlier turns are not in the span); Anthropic Messages (tool_use and tool_result blocks, is_error a failed call, the system prompt); Vercel AI SDK 5 and later model calls (doGenerate and doStream spans with wrapAISDK: the prompt, tool-call and tool-result parts, error-text/error-json a failed call, execution-denied a blocked one); LangChain and LangGraph chat model runs from the Python callback handler (human, ai, tool messages; that handler logs a run's reply without its tool calls, so a conversation whose last call asked for tools is read once a later call holds them) and from the JS handler (LangChain's serialized messages, read as for LangSmith runs, tools from invocation_params); and Duet Machines' endpoint contract. Reasoning is never a turn. Anything else a known format holds (an image, a file, a tool the provider runs itself) refuses the trace naming it.

Turns

The span's input is the history and its output the last step: user messages are customer turns, assistant text agent turns, a tool call answered by a later result a tool turn, a turn_failure a failed agent turn. A call with no answer is not a turn. Spans Braintrust's own scoring and Topics automations write are skipped. Turns carry no times: none of these formats records them.

End

The conversation ends when the trace's root span closes (metrics.end), and is judged on the watch's next pass. A root span still open leaves it open; it ends on the 30-minute idle rule if it never closes. A thread's traces each close with their turn, so a thread ends on the idle rule.

Config

The prompt and tool schemas the call was logged with: system and developer messages, Responses instructions, Anthropic's system, the AI SDK's system message; tools from the request (metadata.tools, or the AI SDK call's tools). Braintrust's provider wrappers log every request parameter and set metadata.provider, so such a span without tools offered none. Reproductions run this snapshot: your real prompt and declared tools. Without a prompt, or a span logged by hand without tools, there is no snapshot and the conversation is judged against the agent's live version.

Usage

Each model call's prompt_tokens and completion_tokens at its metadata.model, on the turn it wrote: an llm span's own, or the model-call spans beneath a step's span (a call inside another counts once). When any call's tokens or model are missing, or calls cannot be matched to messages, no turn carries usage and the cost reads unknown, never a partial sum.

Scores and tags

The root span's scores become metadata braintrust.score.<name> and its tags braintrust.tags, beside its own metadata (non-string values as JSON) and braintrust.project_id: filter, export and compare on them. The judge never reads them; a Braintrust score is context, never proof. Comments are not imported: Braintrust does not document their shape.

Refused traces

A trace with no span in a readable format, something a format holds that Duet Machines cannot represent, model calls holding different conversations, or a tool message answering no call is not stored; the agent's Traffic lists the latest with the reason, and the count.

Limits

Braintrust's Starter and Pro plans allow about 20 SQL queries a minute: a sync reads up to 150 changed traces in a few queries, and a rate-limited sync stops and goes on from there the next time. The first sync reads everything the plan still retains (14 days on Starter).