# Analysis Plans Source: https://docs.transluce.org/analysis/analysis-plans Specify and verify what your coding agent analyzes. With Analysis Plans, you can define, execute, and verify analysis over collections of agent traces. An Analysis Plan contains executable steps and optional markdown notes: * A [**DQL step**](/analysis/dql-steps) displays and executes a structured query. These steps can help filter, group, or aggregate over metadata, transcripts, or prior reading results. DQL steps are fast and deterministic. * A [**Reading step**](/analysis/reading-steps) uses a language model to evaluate the results of a DQL query, which may return transcripts, metadata, prior Reading results, or text. * **Markdown notes** are queued in script order like other steps but are not executed. They are meant for plan context (for example, the behavior a rubric measures), usually one note at the top of the plan. See [Best Practices](/analysis/best-practices) for tips on generating and revising Analysis Plans. For a deeper look at how Analysis Plans work on a real investigation, see our walkthrough on [identifying suspicious behaviors in SWE-rebench](https://transluce.org/docent/blog/analysis-plans). An Analysis Plan in the Docent UI: a sequence of DQL and Reading steps with their inputs, prompts, and outputs. ## Common use-cases A reader evaluates each transcript independently for a behavior. A separate Reading step clusters the results. The per-transcript step applies a rubric to each transcript one at a time. For example: "Does the agent attempt to access files that don't exist? If so, describe what it tried to access and why." The reduce step takes those per-transcript results and groups them: "Cluster these file-access failures by root cause." ```text wrap theme={null} /docent What are the main reasons why fails on ? Search each failed run for the primary failure mode. Then cluster common failure modes across all runs. - Collection ID: ``` After clustering your transcripts, create reading steps that cluster within categories to increase specificity. ```text wrap theme={null} /docent What are the main reasons why fails on ? Identify runs where failed. Summarize the primary failure modes and explain why you think they were decisive. Cluster common failure modes across all runs. For each of the top three failure modes, re-cluster the transcripts around more specific failures. The goal is to identify failures that are prevalent (common in the data) and specific (a developer can identify a concrete fix). - Collection ID: ``` Compare two models on the same tasks. A DQL step selects runs where one model regresses relative to the other, then a reading step identifies the main differences between a successful and a failed run on the same task. ```text wrap theme={null} /docent What are the main reasons why underperforms ? Identify tasks where regresses on average. On those tasks, compare a failed run against the successful runs. Summarize the main failure modes and analyze whether avoiding those failures was material to the result of the successful runs. - Collection ID: ``` We used this workflow to investigate why GPT-5.1 Codex underperformed GPT-5 Codex on Terminal-Bench. See [the writeup](https://transluce.org/docent/blog/terminal-bench) for the full report. ## Analysis Plans are programs Under the hood, an Analysis Plan is a Python script built with the Docent SDK. Your coding agent writes these scripts for you, but they're ordinary code: you can read them, edit them, and re-run them. Analysis Plans is a lazily evaluated computation framework. Each call to `client.query()` or `client.read()` registers a step and immediately returns a lightweight handle. Handles feed into later steps, forming a dependency graph. When you ask for results, or when the script exits, the graph is submitted as an Analysis Plan. Docent executes the steps in dependency order, and the results flow back into your script as plain Python objects. Here's a complete search-and-cluster pipeline: ```python theme={null} from docent import Docent client = Docent() collection_id = "" # Step $1: a DQL step selecting inputs (no LLM involved) sampled = client.query( collection_id, "SELECT transcripts.id AS transcript FROM transcripts ORDER BY transcripts.id LIMIT 100", name="Sample 100 transcripts", ) # Step $2: a Reading step that runs once per row of $1 summarize = client.read( prompt_template=[ sampled.transcript.as_type("transcript"), "Write a 1-2 sentence summary of any mistakes the agent made.", ], model="openai/gpt-5.4-mini", name="Summarize mistakes per transcript", ) # Step $3: a DQL step that gathers all of $2's outputs into one row. # The f-string interpolates the Reading handle as its alias ('$2'); # the server substitutes the real reading ID at execution time. summaries = client.query( collection_id, f""" SELECT array_agg(rr.id ORDER BY rr.id) AS summaries FROM reading_results rr JOIN reading_result_links rrl ON rrl.result_id = rr.id WHERE rrl.reading_id = '{summarize}' """, name="Collect all summaries", ) # Step $4: a Reading step that sees every summary at once clusters = client.read( prompt_template=[ "Cluster these mistake summaries into 5-10 categories: ", summaries.summaries.as_type("reading_result", is_list=True), ], model="openai/gpt-5.5", name="Cluster mistake summaries", ) # Nothing has run yet. Accessing .results forces evaluation of $4 and # everything upstream, blocks until complete, and returns the output. print(clusters.results[0].output) ``` ### How DQL rows feed into readings Every Reading step takes its inputs from a DQL query, which returns a table. Each row of that table becomes one LLM call. The columns of the table fill in the prompt. Accessing an attribute on a query handle, like `sampled.transcript` in step `$2`, gives a reference to that column. Where the reference appears in the prompt template, each row's value is substituted. The `.as_type(...)` annotation controls how the value is rendered: `as_type("text")` embeds the value literally, while types like `"transcript"` or `"agent_run"` treat the value as an ID and render the full object for the judge. This means the DQL query controls both what each judge call sees and how many calls there are. Step `$1` returns 100 rows, so step `$2` makes 100 LLM calls, one per transcript. Step `$3` uses `array_agg` to collapse all of step `$2`'s outputs into a single row with a list-valued column, so step `$4` makes one LLM call that sees every summary at once. Going from "one call per item" to "one call over all items" is just a change to the query. ### Working with the graph A few properties fall out of this design: * **Plans are built with ordinary Python.** You can create steps in loops, build prompts with string formatting, and wrap common patterns in functions. * **Dependencies are tracked for you.** Referencing a query's column in a prompt ties the reading to that query. Interpolating a `Reading` handle into a DQL string ties the query to that reading. Docent infers the execution order from these references, so you never schedule anything yourself. * **Results are available whenever you want them.** Accessing `reading.results` mid-script runs that step and everything upstream of it. The outputs come back as plain dicts, so you can use ordinary Python to shape later steps. For example, you can take the cluster names proposed by one reading and use them as the `enum` values in the next reading's output schema. * **Re-running is cheap.** Steps are content-addressed. When you re-run a script, any step whose inputs and configuration are unchanged reuses its cached results. The standard way to iterate is to append steps to the script and re-run the whole thing; only the new steps execute. When the plan is submitted, its Reading steps appear in the UI. By default, they follow your account's auto-approval preference under **Settings → Preferences**. Call `client.flush(auto_approve=True)` to auto-approve a specific submission or `client.flush(auto_approve=False)` to require manual approval; an explicit value overrides your account preference. ## Creating and executing Analysis Plans Analysis Plans display in the UI after your coding agent writes and executes a script calling Docent's analysis tools. The UI view is read-only: to make revisions, instruct your coding agent to change the plan. Reading steps may require your approval before running, depending on your account preference or an explicit choice made by your coding agent. You can change the default under **Settings → Preferences**. When approval is required, approve individual Reading steps by clicking the Approve button in the top right corner of the step, or approve all pending steps by clicking the Approve All button in the top right of the page. Steps waiting on your approval display in purple on the minimap. The approval view for an Analysis Plan in the Docent UI, where you review each step before it runs. Steps will display a Results table after they have run. # Best Practices Source: https://docs.transluce.org/analysis/best-practices Tips for prompting your agent * **Ask questions whose answers are evident from your transcripts.** Language models are more liable to hallucinate if you ask questions that are unanswerable without access to data that's not in your collection or details of your infra. * **Ask for data instead of telling the coding agent to validate a hypothesis.** For example, if you ask an LLM to compare two traces, it is likely to generate a difference even if that difference is not valid. One way to guard against that is to avoid prompts like "does this model spend less time exploring files before implementing?" and instead have the LLM evaluate time spent exploring on different transcripts, and then look at the resulting chart yourself to make a determination. * Asking the language model to compare two models is helpful for **proposing hypotheses**, not validating them. * **Modify your analysis to manage long context.** Try the following when working with long transcripts or many transcripts. * **Random sampling:** Randomly select a few transcripts if you are working with many of them. * **Recursive summarization:** Instruct the coding agent to summarize the content of each transcript. Then, cluster the resulting summaries instead of clustering the transcripts themselves. If your data contains a sensible pivot key, you can instruct the agent to group by that key (e.g., group by task and summarize). # Structured Queries (DQL) Source: https://docs.transluce.org/analysis/dql A read-only SQL subset for structured, quantitative questions about your data You usually don't write DQL by hand anymore. The [Docent plugin](/installation) produces [DQL steps](/analysis/dql-steps) inside an [Analysis Plan](/analysis/analysis-plans); this page is the language reference if you want to inspect or edit what your coding agent generated. Docent Query Language (DQL) is a read-only SQL subset for ad-hoc queries over a Docent collection. Queries can only run over a single collection by design (if you need multi-collection support, please reach out to us!) Your coding agent (using the Docent plugin) is the easiest way to get DQL written for your collection. To write DQL by hand, see the [DQL schema reference](/sdk/dql/schema) for the column schema of each table and the [execute\_dql reference](/sdk/dql/execute) for Python SDK methods, operators, and common patterns. For filtering agent runs (`select_agent_run_ids`), see [Query Agent Runs](/sdk/agent-runs/query). ## When to use DQL DQL is great for structured queries like getting the average reward by model, or identifying tasks where one model regressed compared to another. Ask the agent: ```text wrap theme={null} /docent What's the average reward by model across this collection? ``` ```text wrap theme={null} /docent Which tasks had one model scoring lower than another when averaging across rollouts? ``` ## A few illustrative queries ### A simple `SELECT` ```sql theme={null} SELECT id, name, metadata_json->'model'->>'name' AS model_name, created_at FROM agent_runs WHERE metadata_json->>'status' = 'completed' ORDER BY created_at DESC LIMIT 10; ``` ### Aggregating with CTEs Per-environment success rates, normalized via a CTE. ```sql theme={null} WITH normalized_runs AS ( SELECT metadata_json->>'environment' AS environment, metadata_json->>'status' AS status FROM agent_runs WHERE metadata_json ? 'environment' ) SELECT environment, COUNT() AS total_runs, SUM(CASE WHEN status = 'completed' THEN 1 ELSE 0 END) AS completed_runs, CAST(SUM(CASE WHEN status = 'completed' THEN 1 ELSE 0 END) AS DOUBLE PRECISION) / NULLIF(COUNT(), 0) AS completion_rate FROM normalized_runs GROUP BY environment ORDER BY total_runs DESC; ``` ### Joining runs with judge results Pulls the most recent rubric result per run, then joins to surface the model responsible for each score. ```sql theme={null} WITH latest_scores AS ( SELECT agent_run_id, MAX(rubric_version) AS rubric_version FROM judge_results WHERE rubric_id = 'helpful_response_v1' GROUP BY agent_run_id ) SELECT ar.id, ar.metadata_json->'model'->>'name' AS model_name, jr.output->>'score' AS score, jr.result_metadata->>'label' AS label FROM latest_scores ls JOIN judge_results jr ON jr.agent_run_id = ls.agent_run_id AND jr.rubric_version = ls.rubric_version AND jr.rubric_id = 'helpful_response_v1' JOIN agent_runs ar ON ar.id = jr.agent_run_id WHERE ar.metadata_json->>'environment' = 'prod' ORDER BY CAST(jr.output->>'score' AS DOUBLE PRECISION) DESC LIMIT 15; ``` ## What DQL does and doesn't do * **Read-only.** Only `SELECT`-style queries are permitted. * **Single statement.** Batches and multiple statements are rejected. * **Explicit projection.** Wildcard `*` is disallowed; list columns explicitly so downstream tooling stays predictable. * **Single-collection scope.** A query runs against one collection at a time. * **10,000 row cap.** Every query is capped by the server. Use `LIMIT`/`OFFSET` for pagination, or export offline for larger sets. * **JSON performance.** Metadata is stored as JSON; heavy traversal across large collections is slower than filtering on indexed scalar columns. * **Type awareness.** JSON paths expose a generic `json` type. Cast explicitly (e.g., `CAST(metadata_json->>'duration_ms' AS BIGINT)`) when precision matters. ## Writing DQL by hand See the [full schema reference](/sdk/dql/schema) for the column schema of each table. # DQL Steps Source: https://docs.transluce.org/analysis/dql-steps Structured queries generated by your coding agent. A DQL step is a structured query (read-only SQL) over your collection's data so you can quickly filter, group, and aggregate. You can run DQL over metadata, transcripts, and the output of prior [Reading steps](/analysis/reading-steps). Your coding agent generates DQL steps as part of [Analysis Plans](/analysis/analysis-plans). Example use cases for DQL steps include: ```text wrap theme={null} /docent What's the average reward by model across this collection? ``` ```text wrap theme={null} /docent Get me the tasks where model A regresses compared to model B when averaging score across rollouts. ``` DQL steps are read-only in the UI. To revise them, prompt your coding agent. We recommend using a coding agent to generate DQL instead of writing it by hand. For the full DQL syntax (operators, JSON access, CTEs, joins), see the [DQL language reference](/analysis/dql). ## Reading and revising DQL steps In an analysis plan, DQL steps run by default. You can see the results as a table in the UI, and the DQL query that generated them above it. The results table does not sort or filter in place. To order rows, ask your coding agent to put the ordering in the query (e.g., "sort by suspiciousness" adds an `ORDER BY` clause). DQL steps have deeply linked IDs. When sorting or filtering, ask your coding agent to include an ID column (e.g., `agent_runs.id`) in the `SELECT`. Any ID column in the results table renders as a clickable link that opens the linked transcript inline, so you can read the run that produced each row without leaving the plan. ### Example queries **Average reward by model** ```sql theme={null} SELECT metadata_json->'model'->>'name' AS model_name, AVG((metadata_json->>'reward')::float) AS avg_reward FROM agent_runs GROUP BY metadata_json->'model'->>'name' ``` **Success rate by environment** ```sql theme={null} SELECT metadata_json->>'environment' AS environment, COUNT(*) AS total_runs, SUM(CASE WHEN metadata_json->>'status' = 'completed' THEN 1 ELSE 0 END) AS completed, CAST(SUM(CASE WHEN metadata_json->>'status' = 'completed' THEN 1 ELSE 0 END) AS DOUBLE PRECISION) / NULLIF(COUNT(*), 0) AS completion_rate FROM agent_runs WHERE metadata_json ? 'environment' GROUP BY metadata_json->>'environment' ORDER BY total_runs DESC ``` # Exporting Data Source: https://docs.transluce.org/analysis/exporting Export transcripts and agent run metadata for local analysis A common task is downloading transcripts alongside their agent run metadata for local analysis. This page shows how to export using DQL. ## Exporting with DQL ### Export all transcripts and metadata ```python theme={null} from docent import Docent import json client = Docent() collection_id = "your-collection-id" # Download all transcripts joined with agent run metadata result = client.execute_dql( collection_id, """SELECT t.id AS transcript_id, t.name AS transcript_name, t.messages, t.metadata_json AS transcript_metadata, ar.id AS agent_run_id, ar.name AS agent_run_name, ar.metadata_json AS agent_run_metadata FROM transcripts t JOIN agent_runs ar ON ar.id = t.agent_run_id""" ) rows = client.dql_result_to_dicts(result) # Parse the messages JSON from each transcript for row in rows: row["messages"] = json.loads(row["messages"]) if isinstance(row["messages"], str) else row["messages"] print(f"Downloaded {len(rows)} transcripts") print(f"First transcript has {len(rows[0]['messages'])} messages") ``` ### Paginating large collections DQL caps results at 10,000 rows. If your collection has more transcripts, use `LIMIT` and `OFFSET` to paginate: ```python theme={null} page_size = 1000 offset = 0 all_rows = [] while True: result = client.execute_dql( collection_id, f"""SELECT t.id AS transcript_id, t.name AS transcript_name, t.messages, ar.id AS agent_run_id, ar.name AS agent_run_name, ar.metadata_json AS agent_run_metadata FROM transcripts t JOIN agent_runs ar ON ar.id = t.agent_run_id ORDER BY t.id LIMIT {page_size} OFFSET {offset}""" ) rows = client.dql_result_to_dicts(result) if not rows: break all_rows.extend(rows) offset += page_size print(f"Downloaded {len(all_rows)} total transcripts") ``` ### Filtering by metadata You can narrow the export to specific runs using metadata filters: ```python theme={null} result = client.execute_dql( collection_id, """SELECT t.id AS transcript_id, t.messages, ar.name AS agent_run_name, ar.metadata_json->>'model' AS model FROM transcripts t JOIN agent_runs ar ON ar.id = t.agent_run_id WHERE ar.metadata_json->>'environment' = 'prod'""" ) ``` ## See also * [Structured queries (DQL)](/analysis/dql) — when to reach for DQL and how your coding agent uses it * [DQL schema reference](/sdk/dql/schema) — full column schema * [execute\_dql reference](/sdk/dql/execute) — operators, syntax, and SDK mechanics # Labeling Agent Runs Source: https://docs.transluce.org/analysis/labeling Annotate agent runs with structured data Labels let you annotate agent runs with structured data. Use labels to measure judge performance or keep track of interesting agent runs. ## Creating a Label Set Label sets are collections of labels with the same [schema](https://json-schema.org/). You will need to create a label set in order to upload labels to Docent. ```python theme={null} import os from docent import Docent client = Docent( api_key=os.getenv("DOCENT_API_KEY"), ) # Define your label schema using JSON Schema label_schema = { "type": "object", "properties": { "label": { "enum": [ "match", "no match" ], "type": "string" }, "explanation": { "type": "string", # Custom field for citations in the UI "citations": true } } } # Create the label set label_set_id = client.create_label_set( collection_id="your-collection-id", name="Auditor Labels", label_schema=label_schema, description="Labels from human auditors." ) print(f"Created label set: {label_set_id}") ``` ## Adding Labels to Agent Runs Once you've created a label set, you can upload labels into Docent. ```python theme={null} from docent.data_models.judge import Label # Create a label for a specific agent run label = Label( label_set_id=label_set_id, agent_run_id="your-agent-run-id", label_value={ "label": "match", "explanation": "The agent..." } ) client.add_label(collection_id="your-collection-id", label=label) ``` For bulk uploads: ```python theme={null} labels = [ Label( label_set_id=label_set_id, agent_run_id="run-1", label_value={"label": "match", "explanation": "..."} ), Label( label_set_id=label_set_id, agent_run_id="run-2", label_value={"label": "no match", "explanation": "..."} ), ] client.add_labels(collection_id="your-collection-id", labels=labels) ``` # Analysis Quickstart Source: https://docs.transluce.org/analysis/quickstart Run your first analysis in Docent If you have not installed Docent yet, start with [Installation](/installation) and run `uvx docent@latest setup` to configure the SDK, coding-agent plugin, and API key. Paste one of these prompts into your coding agent to analyze our sample Terminal-Bench collection. ```text wrap theme={null} /docent What are the main reasons why GPT-5.1 Codex fails? Identify runs where GPT-5.1 failed. Summarize the primary failure modes in those runs and explain why you think they were decisive. Cluster common failure modes or failing strategies across all runs. Continue to cluster within clusters until you reach failures that are prevalent (i.e. common in the data) and specific (i.e. it is evident to a developer what a concrete fix would look like). - Collection ID: 479b7093-5a33-47f1-8d7b-fc9f6f16bb75 ``` ```text wrap theme={null} /docent What are the main reasons why GPT-5.1 Codex underperforms GPT-5 Codex? Identify tasks where GPT-5.1 regresses on average. On those tasks, compare a failed GPT-5.1 run against the successful GPT-5 runs. Summarize the main failure modes and analyze whether avoiding those failures was material to the result of the successful runs. - Collection ID: 479b7093-5a33-47f1-8d7b-fc9f6f16bb75 ``` ```text wrap theme={null} /docent Give me an overview of this collection. - Collection ID: 479b7093-5a33-47f1-8d7b-fc9f6f16bb75 ``` To analyze your own data instead, swap in the collection ID from the top left corner of your collection, next to the collection name. Turn on your coding agent's auto-approval mode so it can generate and run scripts without stopping at each permission prompt — see [Claude Code Auto mode](https://code.claude.com/docs/en/permission-modes#eliminate-prompts-with-auto-mode) or [Codex Auto-review](https://developers.openai.com/codex/concepts/sandboxing/auto-review). Analysis plan approval is separate, and is controlled in Docent under [**Settings → Preferences**](https://docent.transluce.org/settings/preferences). ## Next steps Learn how Analysis Plans are structured, reviewed, and revised. Ready to analyze your own agent runs? Follow the ingestion guide to load your logs into Docent. # Reading Steps Source: https://docs.transluce.org/analysis/reading-steps Use language models to evaluate, classify, and synthesize agent-run data inside an Analysis Plan. A Reading step uses a language model to evaluate the results of a DQL query, which may return transcripts, metadata, prior Reading results, or text. Reading steps are most helpful for operations like classifying transcripts, extracting structured information, or synthesizing results across runs. Your coding agent generates Reading steps inside [Analysis Plans](/analysis/analysis-plans). Their approval behavior follows your account preference unless you tell your coding agent to auto-approve or require approval for a specific plan. ## Inputs Every Reading step is paired with a [DQL query](/analysis/dql-steps) that selects its inputs. The DQL query determines which runs (or which prior results) the Reading step sees and how they're grouped, giving you precise control over what it evaluates. Each parameter in a Reading step's prompt template has a type that determines what data it carries: | Type | Description | | ------------------ | ------------------------------------------------------------- | | `transcript` | A full transcript and transcript-level metadata | | `transcript_slice` | A contiguous portion of a transcript, e.g., the last N turns. | | `agent_run` | All transcripts in an AgentRun and run-level metadata | | `reading_result` | The output of a prior Reading step. | | `text` | A string derived from a metadata field or DQL expression. | Any parameter can be a list selected with `ARRAY_AGG`. This is useful when the number of desired inputs varies row to row. ## Context selection Beyond choosing *which* runs a Reading step sees, you can control *how much* of each run is rendered into the judge's context. Two mechanisms are available: transcript slices select a window of messages, and context configs select which metadata fields appear alongside them. By default, the judge sees the full message content of every input — for an `agent_run` parameter, that's all transcripts plus the names of transcripts and transcript groups. Metadata at every level (agent run, transcript group, transcript, message) is hidden unless a context config includes it. ### Transcript slices A `transcript_slice` parameter renders a contiguous range of messages instead of the whole transcript. Slices are produced in the DQL step with the `transcript_slice(transcript_id, start_idx, end_idx)` function: ```sql theme={null} -- The last 5 messages of each transcript SELECT transcript_slice(transcripts.id, -5, -1) AS ending FROM transcripts ``` Slice behavior: * `start_idx` and `end_idx` are 0-based message indices, inclusive on both ends. They may be equal to render a single message. * Negative indices count from the end using Python's index direction (`-1` is the last message, `-2` is the second-to-last), but the range is still inclusive on both ends: `(-5, -1)` includes the last five messages. * Out-of-range indices don't error; the slice just renders fewer messages. * Rendered messages keep their original indices, so the judge's citations still point to absolute positions in the full transcript. Because the bounds are ordinary DQL expressions, they can be computed per row — for example, a window around an error location stored in metadata: ```sql theme={null} SELECT transcript_slice( t.id, GREATEST(0, CAST(t.metadata_json->>'first_error_idx' AS INTEGER) - 3), CAST(t.metadata_json->>'first_error_idx' AS INTEGER) + 3 ) AS error_window FROM transcripts t WHERE t.metadata_json ? 'first_error_idx' ``` ### Context configs Each `agent_run`, `transcript`, or `transcript_slice` parameter can carry a context config that controls which metadata fields — and, for agent runs, which transcripts — are rendered for the judge. Context configs don't change which rows the DQL step selects; they only change how each selected item is formatted. The available filters depend on the parameter type: | Filter | `agent_run` | `transcript` / `transcript_slice` | Default | | --------------------------- | ----------- | --------------------------------- | -------- | | `agent_run_metadata` | ✓ | | Excluded | | `transcript_group_names` | ✓ | | Included | | `transcript_group_metadata` | ✓ | | Excluded | | `transcript_names` | ✓ | | Included | | `transcript_metadata` | ✓ | ✓ | Excluded | | `message_metadata` | ✓ | ✓ | Excluded | Each filter is a pair of glob pattern lists, `include` and `exclude`, matched against dot-separated paths in the metadata (e.g., `task.difficulty`, `usage.prompt_tokens`). Including a parent path includes its whole subtree; more specific patterns win, and on a tie, exclude wins. The name filters (`transcript_names`, `transcript_group_names`) match object names rather than metadata paths, which lets you render only specific transcripts from a multi-transcript run. In an SDK script, context configs are passed to `client.read()` as a dict keyed by parameter name: ```python theme={null} from docent.data_models.context_config import AgentRunContextConfig from docent.data_models.metadata_util import GlobFilter runs = client.query( collection_id, "SELECT agent_runs.id AS run FROM agent_runs LIMIT 50", name="Sample runs", ) reading = client.read( prompt_template=[ "Evaluate this run, using the included metadata when relevant: ", runs.run.as_type("agent_run"), ], context_configs={ "run": AgentRunContextConfig( # Show task config and score, hide all other run metadata agent_run_metadata=GlobFilter(include=("task.*", "score")), # Render only the main and solver transcripts transcript_names=GlobFilter(include=("main", "solver-*")), ), }, model="openai/gpt-5.4-mini", name="Evaluate runs with task metadata", ) ``` Context configs are part of a Reading step's identity: changing one produces a different content hash, so cached results from the old configuration won't be reused. ## Output schema The output schema is a [JSON Schema](https://json-schema.org/) object that constrains the judge's response. Standard types (`string`, `number`, `boolean`) and `enum` values are supported. Set `"citations": true` on a string field to have the judge ground its output in specific passages from the input. The coding agent proposes a schema based on your question; you can ask it to add, remove, or rename fields. ```json theme={null} { "type": "object", "properties": { "summary": { "type": "string", "citations": true } }, "required": ["summary"] } ``` ```json theme={null} { "type": "object", "properties": { "match": { "type": "string", "enum": ["match", "no match"] }, "explanation": { "type": "string", "citations": true } }, "required": ["match", "explanation"] } ``` ## Reading steps in the plan UI When your coding agent generates an Analysis Plan, each Reading step appears in the UI with four sections: 1. **Summary.** A one-line description of what the step does, with the step's alias (e.g., `$2`) and the step it reads from (e.g., "Data from `$1`"). Other steps reference this step's output by alias. 2. **Prompt template.** The full prompt the judge receives for each input row. Parameters appear as labeled pills. When the step is run, the parameters are filled from the input data. 3. **Output schema.** The JSON Schema that constrains the judge's response. In order to view the output schema, click on the Output Schema pill in the UI. 4. **Results.** A table of the judge's output, displaying one result row per input. You can toggle between **Compact** and **Detailed** view to see more of the output inline. Clicking on a result will open it in the sidebar, and you can use the up and down arrow keys to review different result rows. A Reading step in the plan UI: summary header with step ID and data source, prompt template, output schema, and a results table. # Agent Run Source: https://docs.transluce.org/concepts/agent-run # Agent Run An `AgentRun` represents a complete agent run. It contains a collection of [Transcript](/concepts/transcript) objects, as well as metadata (scores, experiment info, etc.). * In single-agent (most common) settings, each `AgentRun` contains a single `Transcript`. * In multi-agent settings, an `AgentRun` may contain multiple `Transcript` objects. For example, in a two-agent debate setting, you'll have one `Transcript` per agent in the same `AgentRun`. * Docent's LLM search features operate over complete `AgentRun` objects. Runs are passed to LLMs in their rendered text form (for example, via `AgentRunView.to_text()`). ### Usage `AgentRun` objects require a list of [Transcript](/concepts/transcript) objects, as well as a metadata dictionary whose keys are strings. The metadata should be JSON-serializable. Passing `dict[str, Transcript]` is still accepted for backwards compatibility, but is deprecated. ```python theme={null} from docent.data_models import AgentRun, Transcript from docent.data_models.chat import UserMessage, AssistantMessage transcripts = [ Transcript( messages=[ UserMessage(content="Hello, what's 1 + 1?"), AssistantMessage(content="2"), ] ) ] agent_run = AgentRun( transcripts=transcripts, metadata={ "scores": {"correct": True, "reward": 1.0}, } ) ``` ### Rendering To see how your `AgentRun` is being rendered to an LLM, render a view and call `to_text()`: ```python theme={null} from docent.data_models.agent_run import AgentRunView print(AgentRunView.from_agent_run(agent_run).to_text()) ``` ### **AgentRun** Bases: `BaseModel` Represents a complete run of an agent with transcripts and metadata. An AgentRun encapsulates the execution of an agent, storing all communication transcripts and associated metadata. It must contain at least one transcript. **Attributes:** | Name | Type | Description | | | ------------------- | ----------------------- | ------------------------------------------------------------------------------------- | ----------------------------------------------- | | `id` | `str` | Unique identifier, auto-generated; cannot be set by callers. | | | `name` | \`str | None\` | Optional human-readable name for the agent run. | | `description` | \`str | None\` | Optional description of the agent run. | | `transcripts` | `list[Transcript]` | List of Transcript objects. | | | `transcript_groups` | `list[TranscriptGroup]` | List of TranscriptGroup objects. | | | `metadata` | `dict[str, Any]` | Additional structured metadata about the agent run as a JSON-serializable dictionary. | | ```python theme={null} class AgentRun(BaseModel): """Represents a complete run of an agent with transcripts and metadata. An AgentRun encapsulates the execution of an agent, storing all communication transcripts and associated metadata. It must contain at least one transcript. Attributes: id: Unique identifier, auto-generated; cannot be set by callers. name: Optional human-readable name for the agent run. description: Optional description of the agent run. transcripts: List of Transcript objects. transcript_groups: List of TranscriptGroup objects. metadata: Additional structured metadata about the agent run as a JSON-serializable dictionary. """ id: str = Field(default_factory=lambda: str(uuid4()), frozen=True) name: str | None = None description: str | None = None transcripts: list[Transcript] transcript_groups: list[TranscriptGroup] = Field(default_factory=list) metadata: dict[str, Any] = Field(default_factory=dict) def __setattr__(self, name: str, value: Any) -> None: if name == "id": raise ValueError( "Cannot set `id` on AgentRun. Docent assigns IDs automatically; " "the assigned value is already available as `agent_run.id`." ) super().__setattr__(name, value) ############## # Validators # ############## @field_validator("transcripts", mode="before") @classmethod def _validate_transcripts_type(cls, v: Any) -> Any: if isinstance(v, dict): logger.warning( "dict[str, Transcript] for transcripts is deprecated. Use list[Transcript] instead." ) v = cast(dict[str, Transcript], v) return [Transcript.model_validate(t) for t in v.values()] return v @field_validator("transcript_groups", mode="before") @classmethod def _validate_transcript_groups_type(cls, v: Any) -> Any: if isinstance(v, dict): logger.warning( "dict[str, TranscriptGroup] for transcript_groups is deprecated. Use list[TranscriptGroup] instead." ) v = cast(dict[str, TranscriptGroup], v) return [TranscriptGroup.model_validate(tg) for tg in v.values()] return v @model_validator(mode="after") def _validate_transcripts_not_empty(self): """Validates that the agent run contains at least one transcript. Raises: ValueError: If the transcripts list is empty. Returns: AgentRun: The validated AgentRun instance. """ if len(self.transcripts) == 0: raise ValueError("AgentRun must have at least one transcript") return self @property def transcript_dict(self) -> dict[str, Transcript]: """Returns a dictionary mapping transcript IDs to Transcript objects.""" return {t.id: t for t in self.transcripts} @property def transcript_group_dict(self) -> dict[str, TranscriptGroup]: """Returns a dictionary mapping transcript group IDs to TranscriptGroup objects.""" return {tg.id: tg for tg in self.transcript_groups} def to_text( self, children_text: str, agent_run_alias: int | str = 0, indent: int = 0, render_metadata: bool = True, agent_run_metadata_comments: list[Comment] | None = None, ) -> str: if not isinstance(agent_run_alias, str): agent_run_alias = f"R{agent_run_alias}" if render_metadata: metadata_text = dump_metadata(self.metadata) if metadata_text is not None: if indent > 0: metadata_text = textwrap.indent(metadata_text, " " * indent) metadata_alias = f"{agent_run_alias}M" children_text += f"\n<|agent run metadata {metadata_alias}|>\n{metadata_text}\n" # Add agent run metadata comments right underneath the metadata block if agent_run_metadata_comments: metadata_comments_text = render_metadata_comments(agent_run_metadata_comments) if metadata_comments_text: if indent > 0: metadata_comments_text = textwrap.indent( metadata_comments_text, " " * indent ) children_text += f"\n<|agent run metadata comments|>\n{metadata_comments_text}\n" if indent > 0: children_text = textwrap.indent(children_text, " " * indent) return ( f"<|agent run {agent_run_alias}|>\n{children_text}\n\n" ) ``` #### **transcript\_dict** `property` ```python theme={null} transcript_dict: dict[str, Transcript] ``` Returns a dictionary mapping transcript IDs to Transcript objects. #### **transcript\_group\_dict** `property` ```python theme={null} transcript_group_dict: dict[str, TranscriptGroup] ``` Returns a dictionary mapping transcript group IDs to TranscriptGroup objects. ### **AgentRunTree** Bases: `BaseModel` ```python theme={null} class AgentRunTree(BaseModel): nodes: dict[str, AgentRunTreeNode] transcript_id_to_idx: dict[str, int] parent_map: dict[str, str] # child_id -> parent_id @property def nodes_pruned(self): return self._prune_transcriptless_nodes(self.nodes) @classmethod def from_agent_run(cls, agent_run: AgentRun) -> AgentRunTree: t_dict = agent_run.transcript_dict tg_dict = agent_run.transcript_group_dict # Init tree and add the root AgentRun node nodes: dict[str, AgentRunTreeNode] = { GLOBAL_ROOT_ID: AgentRunTreeNode( id=GLOBAL_ROOT_ID, node_type=NodeType.AGENT_RUN, children_ids=[], ) } parent_map: dict[str, str] = {} # Add all transcript groups to the tree for tg_id, tg in tg_dict.items(): # Add this tg if tg_id not in nodes: nodes[tg_id] = AgentRunTreeNode( id=tg_id, node_type=NodeType.TRANSCRIPT_GROUP, children_ids=[], ) # Add parent and mark the relationship # If the stated ID is None, then it's the global root par_id = tg.parent_transcript_group_id or GLOBAL_ROOT_ID if par_id not in nodes: nodes[par_id] = AgentRunTreeNode( id=par_id, node_type=( NodeType.AGENT_RUN if par_id == GLOBAL_ROOT_ID else NodeType.TRANSCRIPT_GROUP ), children_ids=[], ) nodes[par_id].children_ids.append(tg_id) parent_map[tg_id] = par_id # Now add all the transcripts for t_id, t in t_dict.items(): # Add this transcript nodes[t_id] = AgentRunTreeNode( id=t_id, node_type=NodeType.TRANSCRIPT, children_ids=[], ) # Mark parent relationship par_id = t.transcript_group_id or GLOBAL_ROOT_ID # This should never happen, but check anyways for safety; fallback to global root if par_id not in nodes: logger.error( f"Parent {par_id} not found for transcript {t_id}. Assigning to global root as a fallback" ) par_id = GLOBAL_ROOT_ID nodes[par_id].children_ids.append(t_id) parent_map[t_id] = par_id # Go through each node and sort its children by created_at timestamp def _cmp(obj_id: str) -> datetime: obj_type = nodes[obj_id].node_type if obj_type == NodeType.TRANSCRIPT_GROUP: # This should never happen, but check anyways for safety if obj_id not in tg_dict: logger.error(f"Transcript group {obj_id} not found") return datetime.max return tg_dict[obj_id].created_at or datetime.max elif obj_type == NodeType.TRANSCRIPT: # This should never happen, but check anyways for safety if obj_id not in t_dict: logger.error(f"Transcript {obj_id} not found") return datetime.max return t_dict[obj_id].created_at or datetime.max else: raise ValueError(f"Unknown node type: {obj_type}") for node in nodes.values(): node.children_ids = sorted(node.children_ids, key=_cmp) # Combined DFS: mark has_transcript_in_subtree and assign transcript indices t_id_to_idx: dict[str, int] = {} def _dfs(u_id: str, next_idx: int) -> tuple[bool, int]: """Mark has_transcript_in_subtree and assign indices in a single traversal. Returns (contains_transcript, next_idx_after). """ node = nodes.get(u_id) if node is None: return False, next_idx if node.node_type == NodeType.TRANSCRIPT: # Leaf node: assign index immediately (pre-order) t_id_to_idx[u_id] = next_idx node.has_transcript_in_subtree = True return True, next_idx + 1 # Non-transcript node: recurse into children contains_transcript = False for child_id in node.children_ids: child_contains, next_idx = _dfs(child_id, next_idx) contains_transcript = contains_transcript or child_contains node.has_transcript_in_subtree = contains_transcript return contains_transcript, next_idx _dfs(GLOBAL_ROOT_ID, 0) return cls(nodes=nodes, transcript_id_to_idx=t_id_to_idx, parent_map=parent_map) def _prune_transcriptless_nodes(self, nodes: dict[str, AgentRunTreeNode]): """Return a view of the canonical tree that only includes transcript-bearing branches.""" return { node_id: node for node_id, node in nodes.items() if node.has_transcript_in_subtree or node_id == GLOBAL_ROOT_ID } ``` ### **SelectionSpec** Bases: `BaseModel` ```python theme={null} class SelectionSpec(BaseModel): nodes: dict[str, SelectionSpecNode] @classmethod def from_agent_run_tree(cls, agent_run_tree: AgentRunTree) -> SelectionSpec: return cls( nodes={ node_id: SelectionSpecNode(node_id=node_id) for node_id in agent_run_tree.nodes.keys() } ) def is_default(self) -> bool: """Return True if all nodes have default settings (show everything).""" return all( node.render_children_default is True and len(node.render_children_overrides) == 0 and node.render_self_metadata is True for node in self.nodes.values() ) ``` #### **is\_default** ```python theme={null} is_default() -> bool ``` Return True if all nodes have default settings (show everything). ```python theme={null} def is_default(self) -> bool: """Return True if all nodes have default settings (show everything).""" return all( node.render_children_default is True and len(node.render_children_overrides) == 0 and node.render_self_metadata is True for node in self.nodes.values() ) ``` ### **AgentRunView** ```python theme={null} class AgentRunView: def __init__( self, agent_run: AgentRun, selection_spec: SelectionSpec | None = None, comments: list[Comment] | None = None, ): self.agent_run = agent_run self._cached_tree: AgentRunTree | None = None if selection_spec is None: self.selection_spec = SelectionSpec.from_agent_run_tree(self.tree) else: self.selection_spec = selection_spec self.comments = comments # We also need to build an index of which comments belong to each location # There are 4 types of comments: AR metadata, transcript metadata, message metadata, and message content metadata # TODO(mengk): there's quite a bit of data duplication here # agent_run_id -> ... self._agent_run_metadata_comment_index: dict[str, list[Comment]] = {} # transcript_id -> ... self._transcript_metadata_comment_index: dict[str, list[Comment]] = {} # (transcript_id, block_idx) -> ... self._block_metadata_comment_index: dict[tuple[str, int], list[Comment]] = {} # (transcript_id, block_idx) -> ... self._block_content_comment_index: dict[tuple[str, int], list[Comment]] = {} for comment in self.comments or []: for citation in comment.citations: citation_item = citation.target.item if isinstance(citation_item, AgentRunMetadataItem): self._agent_run_metadata_comment_index.setdefault( citation_item.agent_run_id, [] ).append(comment) elif isinstance(citation_item, TranscriptMetadataItem): self._transcript_metadata_comment_index.setdefault( citation_item.transcript_id, [] ).append(comment) elif isinstance(citation_item, TranscriptBlockMetadataItem): self._block_metadata_comment_index.setdefault( (citation_item.transcript_id, citation_item.block_idx), [] ).append(comment) else: # Must be TranscriptBlockContentItem self._block_content_comment_index.setdefault( (citation_item.transcript_id, citation_item.block_idx), [] ).append(comment) @property def tree(self) -> AgentRunTree: if self._cached_tree is None: self._cached_tree = AgentRunTree.from_agent_run(self.agent_run) return self._cached_tree @classmethod def from_agent_run( cls, agent_run: AgentRun, comments: list[Comment] | None = None ) -> AgentRunView: return cls(agent_run=agent_run, comments=comments) def to_dict(self) -> dict[str, Any]: """Serialize the view for storage. Omits selection_spec if it's default.""" return { "agent_run_id": self.agent_run.id, "selection_spec": ( None if self.selection_spec.is_default() else self.selection_spec.model_dump(mode="json") ), } @classmethod def from_dict(cls, data: dict[str, Any], agent_run: AgentRun) -> AgentRunView: """Reconstruct a view from serialized data and an AgentRun.""" spec_data = data.get("selection_spec") if spec_data is None: return cls(agent_run=agent_run) else: selection_spec = SelectionSpec.model_validate(spec_data) return cls(agent_run=agent_run, selection_spec=selection_spec) ####################### # Core text rendering # ####################### def to_text( self, agent_run_alias: int | str = 0, t_idx_map: dict[str, int] | None = None, indent: int = 0, full_tree: bool = False, ): ar_tree_nodes = self.tree.nodes if full_tree else self.tree.nodes_pruned if t_idx_map is None: t_idx_map = self.tree.transcript_id_to_idx t_dict = self.agent_run.transcript_dict tg_dict = self.agent_run.transcript_group_dict # Traverse the tree and render the string def _recurse(u_id: str) -> str: if (u := ar_tree_nodes.get(u_id)) is None: return "" children_texts: list[str] = [] for v_id in u.children_ids: # Check if this child should be rendered if not self.should_render_child(u_id, v_id): continue # Get the node object if (v := ar_tree_nodes.get(v_id)) is None: continue # Casework on the node type if v.node_type == NodeType.TRANSCRIPT_GROUP: children_texts.append(_recurse(v_id)) elif v.node_type == NodeType.TRANSCRIPT: # Gather comments for this transcript transcript_metadata_comments = self._transcript_metadata_comment_index.get(v_id) block_metadata_comments = { block_idx: comments for ( t_id, block_idx, ), comments in self._block_metadata_comment_index.items() if t_id == v_id } or None block_content_comments = { block_idx: comments for (t_id, block_idx), comments in self._block_content_comment_index.items() if t_id == v_id } or None cur_text = t_dict[v_id].to_text( transcript_alias=t_idx_map[v_id], indent=indent, render_metadata=self.should_render_metadata(v_id), transcript_metadata_comments=transcript_metadata_comments, block_metadata_comments=block_metadata_comments, block_content_comments=block_content_comments, ) children_texts.append(cur_text) else: raise ValueError(f"Unknown node type: {v.node_type}") children_text = "\n".join(children_texts) # No wrapper for global root if u_id == GLOBAL_ROOT_ID: # Get agent run metadata comments agent_run_metadata_comments = self._agent_run_metadata_comment_index.get( self.agent_run.id ) return self.agent_run.to_text( children_text, agent_run_alias=agent_run_alias, indent=indent, render_metadata=self.should_render_metadata(GLOBAL_ROOT_ID), agent_run_metadata_comments=agent_run_metadata_comments, ) # Delegate rendering to TranscriptGroup else: tg = tg_dict[u_id] return tg.to_text( children_text=children_text, indent=indent, render_metadata=self.should_render_metadata(u_id), ) return _recurse(GLOBAL_ROOT_ID) ################# # Query methods # ################# def should_render_child(self, parent_id: str, child_id: str) -> bool: """Determine if a child should be rendered based on parent's render settings.""" # Default to rendering if no spec if (parent_spec := self.selection_spec.nodes.get(parent_id)) is None: return True # Default include: render all except those in overrides if parent_spec.render_children_default: return child_id not in parent_spec.render_children_overrides # Default exclude: render only those in overrides else: return child_id in parent_spec.render_children_overrides def should_render_metadata(self, node_id: str) -> bool: """Determine if a node's metadata should be rendered.""" # Default to rendering if no spec if (node_spec := self.selection_spec.nodes.get(node_id)) is None: return True return node_spec.render_self_metadata ######################################### # Show/hide parts of the canonical tree # ######################################### def set_metadata_selection(self, node_id: str, selected: bool) -> None: """Set whether a node's metadata is rendered. When enabling (True), this also ensures the path from the root to this node is visible by adjusting parent render settings. Args: node_id: The ID of the node to modify. selected: Whether the node's metadata should be rendered. """ if (spec := self.selection_spec.nodes.get(node_id)) is not None: spec.render_self_metadata = selected if selected: self._ensure_path_to_root_selected(node_id) def set_node_selection(self, node_id: str, selected: bool) -> None: """Set whether a node and its descendants are rendered. This recursively sets children selection state for all descendants. When enabling (True), this also ensures the path from the root to this node is visible by adjusting parent render settings. When disabling (False), this ensures the parent excludes this node. Notably, this does _not_ affect the metadata rendering state of each node. Args: node_id: The ID of the node to modify. selected: Whether the node and its descendants should be rendered. """ self._set_children_selected_recursive(node_id, selected=selected) if selected: self._ensure_path_to_root_selected(node_id) else: self._ensure_node_excluded_from_parent(node_id) def _set_children_selected_recursive(self, node_id: str, selected: bool) -> None: """Recursively set children selection state for a node and all its descendants.""" if (node := self.tree.nodes.get(node_id)) is None: return if (spec := self.selection_spec.nodes.get(node_id)) is None: return spec.render_children_default = selected spec.render_children_overrides.clear() for child_id in node.children_ids: self._set_children_selected_recursive(child_id, selected=selected) def _get_parent_id(self, node_id: str) -> str | None: """Get the parent ID for a node, or None if it's the root or not found.""" # The root node has no parent if node_id == GLOBAL_ROOT_ID: return None return self.tree.parent_map.get(node_id) def _set_parent_renders_child(self, parent_id: str, child_id: str, renders: bool) -> None: """Update parent's overrides so that it renders (or doesn't render) the child.""" if (parent_spec := self.selection_spec.nodes.get(parent_id)) is None: return if renders == parent_spec.render_children_default: parent_spec.render_children_overrides.discard(child_id) else: parent_spec.render_children_overrides.add(child_id) def _ensure_path_to_root_selected(self, node_id: str) -> None: """Traverse from node_id up to root, ensuring each parent renders its child.""" u_id = node_id while (parent_id := self._get_parent_id(u_id)) is not None: self._set_parent_renders_child(parent_id, u_id, renders=True) u_id = parent_id def _ensure_node_excluded_from_parent(self, node_id: str) -> None: """Ensure the parent does not render this node.""" if (parent_id := self._get_parent_id(node_id)) is not None: self._set_parent_renders_child(parent_id, node_id, renders=False) ``` #### **to\_dict** ```python theme={null} to_dict() -> dict[str, Any] ``` Serialize the view for storage. Omits selection\_spec if it's default. ```python theme={null} def to_dict(self) -> dict[str, Any]: """Serialize the view for storage. Omits selection_spec if it's default.""" return { "agent_run_id": self.agent_run.id, "selection_spec": ( None if self.selection_spec.is_default() else self.selection_spec.model_dump(mode="json") ), } ``` #### **from\_dict** `classmethod` ```python theme={null} from_dict(data: dict[str, Any], agent_run: AgentRun) -> AgentRunView ``` Reconstruct a view from serialized data and an AgentRun. ```python theme={null} @classmethod def from_dict(cls, data: dict[str, Any], agent_run: AgentRun) -> AgentRunView: """Reconstruct a view from serialized data and an AgentRun.""" spec_data = data.get("selection_spec") if spec_data is None: return cls(agent_run=agent_run) else: selection_spec = SelectionSpec.model_validate(spec_data) return cls(agent_run=agent_run, selection_spec=selection_spec) ``` #### **should\_render\_child** ```python theme={null} should_render_child(parent_id: str, child_id: str) -> bool ``` Determine if a child should be rendered based on parent's render settings. ```python theme={null} def should_render_child(self, parent_id: str, child_id: str) -> bool: """Determine if a child should be rendered based on parent's render settings.""" # Default to rendering if no spec if (parent_spec := self.selection_spec.nodes.get(parent_id)) is None: return True # Default include: render all except those in overrides if parent_spec.render_children_default: return child_id not in parent_spec.render_children_overrides # Default exclude: render only those in overrides else: return child_id in parent_spec.render_children_overrides ``` #### **should\_render\_metadata** ```python theme={null} should_render_metadata(node_id: str) -> bool ``` Determine if a node's metadata should be rendered. ```python theme={null} def should_render_metadata(self, node_id: str) -> bool: """Determine if a node's metadata should be rendered.""" # Default to rendering if no spec if (node_spec := self.selection_spec.nodes.get(node_id)) is None: return True return node_spec.render_self_metadata ``` #### **set\_metadata\_selection** ```python theme={null} set_metadata_selection(node_id: str, selected: bool) -> None ``` Set whether a node's metadata is rendered. When enabling (True), this also ensures the path from the root to this node is visible by adjusting parent render settings. **Parameters:** | Name | Type | Description | Default | | ---------- | ------ | ----------------------------------------------- | ---------- | | `node_id` | `str` | The ID of the node to modify. | *required* | | `selected` | `bool` | Whether the node's metadata should be rendered. | *required* | ```python theme={null} def set_metadata_selection(self, node_id: str, selected: bool) -> None: """Set whether a node's metadata is rendered. When enabling (True), this also ensures the path from the root to this node is visible by adjusting parent render settings. Args: node_id: The ID of the node to modify. selected: Whether the node's metadata should be rendered. """ if (spec := self.selection_spec.nodes.get(node_id)) is not None: spec.render_self_metadata = selected if selected: self._ensure_path_to_root_selected(node_id) ``` #### **set\_node\_selection** ```python theme={null} set_node_selection(node_id: str, selected: bool) -> None ``` Set whether a node and its descendants are rendered. This recursively sets children selection state for all descendants. When enabling (True), this also ensures the path from the root to this node is visible by adjusting parent render settings. When disabling (False), this ensures the parent excludes this node. Notably, this does *not* affect the metadata rendering state of each node. **Parameters:** | Name | Type | Description | Default | | ---------- | ------ | -------------------------------------------------------- | ---------- | | `node_id` | `str` | The ID of the node to modify. | *required* | | `selected` | `bool` | Whether the node and its descendants should be rendered. | *required* | ```python theme={null} def set_node_selection(self, node_id: str, selected: bool) -> None: """Set whether a node and its descendants are rendered. This recursively sets children selection state for all descendants. When enabling (True), this also ensures the path from the root to this node is visible by adjusting parent render settings. When disabling (False), this ensures the parent excludes this node. Notably, this does _not_ affect the metadata rendering state of each node. Args: node_id: The ID of the node to modify. selected: Whether the node and its descendants should be rendered. """ self._set_children_selected_recursive(node_id, selected=selected) if selected: self._ensure_path_to_root_selected(node_id) else: self._ensure_node_excluded_from_parent(node_id) ``` # Chat Messages Source: https://docs.transluce.org/concepts/chat-messages # Chat messages We support 4 types of message objects: * `SystemMessage`: Instructions and context for the conversation * `UserMessage`: Messages from end users to the assistant * `AssistantMessage`: Responses from the AI assistant, potentially including tool calls * `ToolMessage`: Results from tools invoked during the conversation Each message has a `content` field, which can either be a `str` or a list of `Content` objects with `text` and/or `reasoning`. We don't support audio/image/video content yet. Each message also has an optional `metadata` field that can store additional structured information about the message as a dictionary. ### Usage The easiest way to convert a `dict` into a `ChatMessage` is to use `parse_chat_message`: ```python theme={null} from docent.data_models.chat import parse_chat_message message_data = [ { "role": "user", "content": "What is the capital of France?", }, { "role": "assistant", "content": "Paris", }, ] messages = [parse_chat_message(msg) for msg in message_data] ``` The function will automatically raise validation errors if the input message does not conform to the schema. You may also want to create messages manually: ```python theme={null} from docent.data_models.chat import ( SystemMessage, UserMessage, AssistantMessage, ToolMessage, ContentText, ContentReasoning, ToolCall, ToolCallContent, ) messages = [ SystemMessage(content="You are a helpful assistant."), UserMessage(content=[ContentText(text="Help me with this problem.")]), AssistantMessage(content="I'll help you solve that.", tool_calls=[ToolCall(id="call_123", function="calculator", arguments={"operation": "add", "a": 5, "b": 3}, view=ToolCallContent(format="markdown", content="Calculating: 5 + 3"))]), ToolMessage(content="8", tool_call_id="call_123", function="calculator"), # Example with metadata SystemMessage(content="Processing user request", metadata={"source": "api", "priority": "high", "timestamp": "2024-01-01T12:00:00Z"}), ] ``` ### Note on tool calls There are two parts to a tool call: * On the `AssistantMessage` itself, the `tool_calls` field contains a list of `ToolCall` objects. These represent calls to various tools the agent has access to. * The `ToolMessage` object is the output of the tool, e.g. a list of files after calling `ls`. Tool Call Agent Run Example For an example of parsing tool calls, check out the [τ-Bench tab in the SDK ingestion guide](/ingestion/sdk#τ-bench). ### **ChatMessage** `module-attribute` ```python theme={null} ChatMessage = Annotated[SystemMessage | UserMessage | AssistantMessage | ToolMessage, Discriminator('role')] ``` Type alias for any chat message type, discriminated by the role field. This is the base message union used in Transcript and AgentRun contexts. For chat sessions, use DocentChatMessage instead. ### **DocentChatMessage** `module-attribute` ```python theme={null} DocentChatMessage = Annotated[SystemMessage | UserMessage | DocentAssistantMessage | ToolMessage, Discriminator('role')] ``` Type alias for chat session messages with chat-specific assistant metadata. ### **BaseChatMessage** Bases: `BaseModel` Base class for all chat message types. **Attributes:** | Name | Type | Description | | | ---------- | ------------------------------------------------ | --------------------------------------------------------------- | ------------------------------------------------------------------- | | `id` | \`str | None\` | Optional unique identifier for the message. | | `content` | \`str | list\[Content]\` | The message content, either as a string or list of Content objects. | | `role` | `Literal['system', 'user', 'assistant', 'tool']` | The role of the message sender (system, user, assistant, tool). | | | `metadata` | \`dict\[str, Any] | None\` | Additional structured metadata about the message. | ```python theme={null} class BaseChatMessage(BaseModel): """Base class for all chat message types. Attributes: id: Optional unique identifier for the message. content: The message content, either as a string or list of Content objects. role: The role of the message sender (system, user, assistant, tool). metadata: Additional structured metadata about the message. """ id: str | None = None content: str | list[Content] role: Literal["system", "user", "assistant", "tool"] metadata: dict[str, Any] | None = None @property def text(self) -> str: """Get the text content of the message. Returns: str: The text content of the message. If content is a list, concatenates all text content elements with newlines. """ if isinstance(self.content, str): return self.content else: all_text = [content.text for content in self.content if content.type == "text"] return "\n".join(all_text) ``` #### **text** `property` ```python theme={null} text: str ``` Get the text content of the message. **Returns:** | Name | Type | Description | | ----- | ----- | ------------------------------------------------------------------------------------------------------------ | | `str` | `str` | The text content of the message. If content is a list, concatenates all text content elements with newlines. | ### **SystemMessage** Bases: `BaseChatMessage` System message in a chat conversation. **Attributes:** | Name | Type | Description | | ------ | ------------------- | ----------------------- | | `role` | `Literal['system']` | Always set to "system". | ```python theme={null} class SystemMessage(BaseChatMessage): """System message in a chat conversation. Attributes: role: Always set to "system". """ role: Literal["system"] = "system" # type: ignore ``` #### **text** `property` ```python theme={null} text: str ``` Get the text content of the message. **Returns:** | Name | Type | Description | | ----- | ----- | ------------------------------------------------------------------------------------------------------------ | | `str` | `str` | The text content of the message. If content is a list, concatenates all text content elements with newlines. | ### **UserMessage** Bases: `BaseChatMessage` User message in a chat conversation. **Attributes:** | Name | Type | Description | | | -------------- | ----------------- | --------------------- | ------------------------------------------------------------- | | `role` | `Literal['user']` | Always set to "user". | | | `tool_call_id` | \`list\[str] | None\` | Optional list of tool call IDs this message is responding to. | ```python theme={null} class UserMessage(BaseChatMessage): """User message in a chat conversation. Attributes: role: Always set to "user". tool_call_id: Optional list of tool call IDs this message is responding to. """ role: Literal["user"] = "user" # type: ignore tool_call_id: list[str] | None = None ``` #### **text** `property` ```python theme={null} text: str ``` Get the text content of the message. **Returns:** | Name | Type | Description | | ----- | ----- | ------------------------------------------------------------------------------------------------------------ | | `str` | `str` | The text content of the message. If content is a list, concatenates all text content elements with newlines. | ### **AssistantMessage** Bases: `BaseChatMessage` Assistant message in a chat conversation. **Attributes:** | Name | Type | Description | | | ------------ | ---------------------- | -------------------------- | -------------------------------------------------------------- | | `role` | `Literal['assistant']` | Always set to "assistant". | | | `model` | \`str | None\` | Optional identifier for the model that generated this message. | | `tool_calls` | \`list\[ToolCall] | None\` | Optional list of tool calls made by the assistant. | ```python theme={null} class AssistantMessage(BaseChatMessage): """Assistant message in a chat conversation. Attributes: role: Always set to "assistant". model: Optional identifier for the model that generated this message. tool_calls: Optional list of tool calls made by the assistant. """ role: Literal["assistant"] = "assistant" # type: ignore model: str | None = None tool_calls: list[ToolCall] | None = None ``` #### **text** `property` ```python theme={null} text: str ``` Get the text content of the message. **Returns:** | Name | Type | Description | | ----- | ----- | ------------------------------------------------------------------------------------------------------------ | | `str` | `str` | The text content of the message. If content is a list, concatenates all text content elements with newlines. | ### **DocentAssistantMessage** Bases: `AssistantMessage` Assistant message in a chat session with additional chat-specific metadata. This extends AssistantMessage with fields that are only relevant in Docent chat contexts **Attributes:** | Name | Type | Description | | | -------------------- | ----------------------- | ----------- | ------------------------------------------------------------- | | `citations` | \`list\[InlineCitation] | None\` | Optional list of citations referenced in the message content. | | `suggested_messages` | \`list\[str] | None\` | Optional list of suggested followup messages. | ```python theme={null} class DocentAssistantMessage(AssistantMessage): """Assistant message in a chat session with additional chat-specific metadata. This extends AssistantMessage with fields that are only relevant in Docent chat contexts Attributes: citations: Optional list of citations referenced in the message content. suggested_messages: Optional list of suggested followup messages. """ citations: list[InlineCitation] | None = None suggested_messages: list[str] | None = None ``` #### **text** `property` ```python theme={null} text: str ``` Get the text content of the message. **Returns:** | Name | Type | Description | | ----- | ----- | ------------------------------------------------------------------------------------------------------------ | | `str` | `str` | The text content of the message. If content is a list, concatenates all text content elements with newlines. | ### **ToolMessage** Bases: `BaseChatMessage` Tool message in a chat conversation. **Attributes:** | Name | Type | Description | | | -------------- | ----------------- | --------------------- | ----------------------------------------------------------- | | `role` | `Literal['tool']` | Always set to "tool". | | | `tool_call_id` | \`str | None\` | Optional ID of the tool call this message is responding to. | | `function` | \`str | None\` | Optional name of the function that was called. | | `error` | \`dict\[str, Any] | None\` | Optional error information if the tool call failed. | ```python theme={null} class ToolMessage(BaseChatMessage): """Tool message in a chat conversation. Attributes: role: Always set to "tool". tool_call_id: Optional ID of the tool call this message is responding to. function: Optional name of the function that was called. error: Optional error information if the tool call failed. """ role: Literal["tool"] = "tool" # type: ignore tool_call_id: str | None = None function: str | None = None error: dict[str, Any] | None = None ``` #### **text** `property` ```python theme={null} text: str ``` Get the text content of the message. **Returns:** | Name | Type | Description | | ----- | ----- | ------------------------------------------------------------------------------------------------------------ | | `str` | `str` | The text content of the message. If content is a list, concatenates all text content elements with newlines. | ### **parse\_chat\_message** ```python theme={null} parse_chat_message(message_data: dict[str, Any] | ChatMessage) -> ChatMessage ``` Parse a message dictionary or object into the appropriate ChatMessage subclass. This parses base messages without chat-specific fields. For chat sessions, use parse\_docent\_chat\_message instead. **Parameters:** | Name | Type | Description | Default | | | -------------- | ----------------- | ------------- | --------------------------------------------------------------- | ---------- | | `message_data` | \`dict\[str, Any] | ChatMessage\` | A dictionary or ChatMessage object representing a chat message. | *required* | **Returns:** | Name | Type | Description | | ------------- | ------------- | -------------------------------------------------------- | | `ChatMessage` | `ChatMessage` | An instance of a ChatMessage subclass based on the role. | **Raises:** | Type | Description | | ------------ | ------------------------------- | | `ValueError` | If the message role is unknown. | ```python theme={null} def parse_chat_message(message_data: dict[str, Any] | ChatMessage) -> ChatMessage: """Parse a message dictionary or object into the appropriate ChatMessage subclass. This parses base messages without chat-specific fields. For chat sessions, use parse_docent_chat_message instead. Args: message_data: A dictionary or ChatMessage object representing a chat message. Returns: ChatMessage: An instance of a ChatMessage subclass based on the role. Raises: ValueError: If the message role is unknown. """ if isinstance(message_data, (SystemMessage, UserMessage, AssistantMessage, ToolMessage)): return message_data role = message_data.get("role") if role == "system": return SystemMessage.model_validate(message_data) elif role == "user": return UserMessage.model_validate(message_data) elif role == "assistant": return AssistantMessage.model_validate(message_data) elif role == "tool": return ToolMessage.model_validate(message_data) else: raise ValueError(f"Unknown message role: {role}") ``` ### **parse\_docent\_chat\_message** ```python theme={null} parse_docent_chat_message(message_data: dict[str, Any] | DocentChatMessage) -> DocentChatMessage ``` Parse a message dictionary or object into the appropriate DocentChatMessage subclass. This handles chat session messages which may include DocentAssistantMessage with citations and suggested\_messages fields. **Parameters:** | Name | Type | Description | Default | | | -------------- | ----------------- | ------------------- | ----------------------------------------------------------------------------- | ---------- | | `message_data` | \`dict\[str, Any] | DocentChatMessage\` | A dictionary or DocentChatMessage object representing a chat session message. | *required* | **Returns:** | Name | Type | Description | | ------------------- | ------------------- | -------------------------------------------------------------- | | `DocentChatMessage` | `DocentChatMessage` | An instance of a DocentChatMessage subclass based on the role. | **Raises:** | Type | Description | | ------------ | ------------------------------- | | `ValueError` | If the message role is unknown. | ```python theme={null} def parse_docent_chat_message( message_data: dict[str, Any] | DocentChatMessage, ) -> DocentChatMessage: """Parse a message dictionary or object into the appropriate DocentChatMessage subclass. This handles chat session messages which may include DocentAssistantMessage with citations and suggested_messages fields. Args: message_data: A dictionary or DocentChatMessage object representing a chat session message. Returns: DocentChatMessage: An instance of a DocentChatMessage subclass based on the role. Raises: ValueError: If the message role is unknown. """ if isinstance( message_data, (SystemMessage, UserMessage, DocentAssistantMessage, AssistantMessage, ToolMessage), ): return message_data role = message_data.get("role") if role == "system": return SystemMessage.model_validate(message_data) elif role == "user": return UserMessage.model_validate(message_data) elif role == "assistant": return DocentAssistantMessage.model_validate(message_data) elif role == "tool": return ToolMessage.model_validate(message_data) else: raise ValueError(f"Unknown message role: {role}") ``` ### **Content** `module-attribute` ```python theme={null} Content = Annotated[ContentText | ContentReasoning, Discriminator('type')] ``` Discriminated union of possible content types using the 'type' field. Can be either ContentText or ContentReasoning. ### **BaseContent** Bases: `BaseModel` Base class for all content types in chat messages. Provides the foundation for different content types with a discriminator field. **Attributes:** | Name | Type | Description | | ------ | --------------------------------------------------------- | --------------------------------------------------------------------------- | | `type` | `Literal['text', 'reasoning', 'image', 'audio', 'video']` | The content type identifier, used for discriminating between content types. | ```python theme={null} class BaseContent(BaseModel): """Base class for all content types in chat messages. Provides the foundation for different content types with a discriminator field. Attributes: type: The content type identifier, used for discriminating between content types. """ type: Literal["text", "reasoning", "image", "audio", "video"] ``` ### **ContentText** Bases: `BaseContent` Text content for chat messages. Represents plain text content in a chat message. **Attributes:** | Name | Type | Description | | | --------- | ----------------- | ---------------------------------------------- | ------------------------------------------------------ | | `type` | `Literal['text']` | Fixed as "text" to identify this content type. | | | `text` | `str` | The actual text content. | | | `refusal` | \`bool | None\` | Optional flag indicating if this is a refusal message. | ```python theme={null} class ContentText(BaseContent): """Text content for chat messages. Represents plain text content in a chat message. Attributes: type: Fixed as "text" to identify this content type. text: The actual text content. refusal: Optional flag indicating if this is a refusal message. """ type: Literal["text"] = "text" # type: ignore text: str refusal: bool | None = None ``` ### **ContentReasoning** Bases: `BaseContent` Reasoning content for chat messages. Represents reasoning or thought process content in a chat message. **Attributes:** | Name | Type | Description | | | ----------- | ---------------------- | --------------------------------------------------- | ------------------------------------------------- | | `type` | `Literal['reasoning']` | Fixed as "reasoning" to identify this content type. | | | `reasoning` | `str` | The actual reasoning text. | | | `signature` | \`str | None\` | Optional signature associated with the reasoning. | | `redacted` | `bool` | Flag indicating if the reasoning has been redacted. | | ```python theme={null} class ContentReasoning(BaseContent): """Reasoning content for chat messages. Represents reasoning or thought process content in a chat message. Attributes: type: Fixed as "reasoning" to identify this content type. reasoning: The actual reasoning text. signature: Optional signature associated with the reasoning. redacted: Flag indicating if the reasoning has been redacted. """ type: Literal["reasoning"] = "reasoning" # type: ignore reasoning: str signature: str | None = None redacted: bool = False ``` ### **ToolCall** `dataclass` Tool call information. **Attributes:** | Name | Type | Description | | | ------------- | ---------------------- | -------------------------------- | -------------------------------------------------- | | `id` | `str` | Unique identifier for tool call. | | | `type` | \`Literal\['function'] | None\` | Type of tool call. Can only be "function" or None. | | `function` | `str` | Function called. | | | `arguments` | `dict[str, Any]` | Arguments to function. | | | `parse_error` | \`str | None\` | Error which occurred parsing tool call. | | `view` | \`ToolCallContent | None\` | Custom view of tool call input. | ```python theme={null} @dataclass class ToolCall: """Tool call information. Attributes: id: Unique identifier for tool call. type: Type of tool call. Can only be "function" or None. function: Function called. arguments: Arguments to function. parse_error: Error which occurred parsing tool call. view: Custom view of tool call input. """ id: str function: str arguments: dict[str, Any] type: Literal["function"] | None = None parse_error: str | None = None view: ToolCallContent | None = None ``` ### **ToolCallContent** Bases: `BaseModel` Content to include in tool call view. **Attributes:** | Name | Type | Description | | | --------- | ----------------------------- | -------------------------- | -------------------------------------------------- | | `title` | \`str | None\` | Optional (plain text) title for tool call content. | | `format` | `Literal['text', 'markdown']` | Format (text or markdown). | | | `content` | `str` | Text or markdown content. | | ```python theme={null} class ToolCallContent(BaseModel): """Content to include in tool call view. Attributes: title: Optional (plain text) title for tool call content. format: Format (text or markdown). content: Text or markdown content. """ title: str | None = None format: Literal["text", "markdown"] content: str ``` ### **ToolParam** Bases: `BaseModel` A parameter for a tool function. **Parameters:** | Name | Type | Description | Default | | -------------- | ---- | ----------------------------------------------------------------- | ---------- | | `name` | | The name of the parameter. | *required* | | `description` | | A description of what the parameter does. | *required* | | `input_schema` | | JSON Schema describing the parameter's type and validation rules. | *required* | ```python theme={null} class ToolParam(BaseModel): """A parameter for a tool function. Args: name: The name of the parameter. description: A description of what the parameter does. input_schema: JSON Schema describing the parameter's type and validation rules. """ name: str description: str input_schema: dict[str, Any] ``` ### **ToolParams** Bases: `BaseModel` Description of tool parameters object in JSON Schema format. **Parameters:** | Name | Type | Description | Default | | ---------------------- | ---- | ------------------------------------------------------------------------------- | ---------- | | `type` | | The type of the parameters object, always 'object'. | *required* | | `properties` | | Dictionary mapping parameter names to their ToolParam definitions. | *required* | | `required` | | List of required parameter names. | *required* | | `additionalProperties` | | Whether additional properties are allowed beyond those specified. Always False. | *required* | ```python theme={null} class ToolParams(BaseModel): """Description of tool parameters object in JSON Schema format. Args: type: The type of the parameters object, always 'object'. properties: Dictionary mapping parameter names to their ToolParam definitions. required: List of required parameter names. additionalProperties: Whether additional properties are allowed beyond those specified. Always False. """ type: Literal["object"] = "object" properties: dict[str, ToolParam] = Field(default_factory=dict) required: list[str] = Field(default_factory=list) additionalProperties: bool = False ``` ### **ToolInfo** Bases: `BaseModel` Specification of a tool (JSON Schema compatible). If you are implementing a ModelAPI, most LLM libraries can be passed this object (dumped to a dict) directly as a function specification. For example, in the OpenAI provider: ```python theme={null} ChatCompletionToolParam( type="function", function=tool.model_dump(exclude_none=True), ) ``` In some cases the field names don't match up exactly. In that case call `model_dump()` on the `parameters` field. For example, in the Anthropic provider: ```python theme={null} ToolParam( name=tool.name, description=tool.description, input_schema=tool.parameters.model_dump(exclude_none=True), ) ``` **Attributes:** | Name | Type | Description | | ------------- | ------------ | -------------------------------------- | | `name` | `str` | Name of tool. | | `description` | `str` | Short description of tool. | | `parameters` | `ToolParams` | JSON Schema of tool parameters object. | ````python theme={null} class ToolInfo(BaseModel): """Specification of a tool (JSON Schema compatible). If you are implementing a ModelAPI, most LLM libraries can be passed this object (dumped to a dict) directly as a function specification. For example, in the OpenAI provider: ```python ChatCompletionToolParam( type="function", function=tool.model_dump(exclude_none=True), ) ``` In some cases the field names don't match up exactly. In that case call `model_dump()` on the `parameters` field. For example, in the Anthropic provider: ```python ToolParam( name=tool.name, description=tool.description, input_schema=tool.parameters.model_dump(exclude_none=True), ) ``` Attributes: name: Name of tool. description: Short description of tool. parameters: JSON Schema of tool parameters object. """ name: str description: str parameters: ToolParams = Field(default_factory=ToolParams) ```` # Collection Source: https://docs.transluce.org/concepts/collection A Collection is a workspace for one experiment. It holds the AgentRuns from that experiment along with the rubrics, labels, and metadata you build up about them. Queries, search, and your coding agent all operate inside one Collection at a time. ## What a Collection contains A Collection is the home for everything tied to one experiment: the runs themselves, the analysis you generate about them, and the access list for the whole set. * **AgentRuns.** The runs themselves, one per execution you want to analyze. * **Rubrics and their results.** Judges you have defined and the scores they produced. * **Labels.** Human annotations attached to AgentRuns. * **Collection metadata.** Fields that describe the dataset as a whole, like eval config, environment, or dataset name. * **Shared access.** The list of users who can view or edit this Collection. ## Analysis tools operate within a single collection All analysis tools currently operate within a collection. These tools include: * DQL queries * Rubric runs * Search and clustering * Your coding agent's context * Filter state in the web UI Your account and the data model sit above the workspace level: * Your account and API keys * Sharing permissions (one user can belong to many Collections) * The shapes of AgentRuns, Transcripts, and ChatMessages To compare data across Collections, export it and join it yourself. Put things you want to compare in the same Collection from the start. For example, if you want to compare the performance of different models on one benchmark, you should include runs from many models in the same collection. ### Collection metadata vs. AgentRun metadata Collection metadata describes the dataset as a whole. AgentRun metadata describes each run. * **Put it on the Collection** when every run shares the value, like dataset name, eval version, environment, or the date the eval ran. * **Put it on the AgentRun** when you would ever filter or group by it, like `model`, `checkpoint`, `task_id`, or scores. * **When unsure, put it on the AgentRun.** DQL can query AgentRun metadata. Collection metadata carries context, not slicing. See [Metadata](/concepts/metadata) for the full pattern and the tracing equivalents. ## Create, update, and share Collections You create, update, and share a Collection through the SDK or the web UI. Deletion happens only in the web UI. Create a Collection in one SDK call and keep the returned `collection_id` for everything downstream: ```python theme={null} from docent import Docent client = Docent() collection_id = client.create_collection( name="Terminal-Bench: GPT-5 vs GPT-5.1", description="December 2025 head-to-head run", metadata={"eval": "terminal-bench", "date": "2025-12-10"}, ) ``` The SDK covers the common operations, with one exception: * **List, update, or remove runs:** [Manage collections](/sdk/collections/manage). * **Read or merge Collection metadata:** [Collection metadata](/sdk/collections/metadata). * **Share a Collection:** done in the web UI. * **Delete a Collection:** web UI only. The SDK removes AgentRuns from a Collection but does not delete the Collection itself. ## Next steps * [AgentRun](/concepts/agent-run): the unit that lives inside a Collection. * [Metadata](/concepts/metadata): how to shape the fields you will query against. * [Ingestion Quickstart](/ingestion/quickstart): pick a path to load your first Collection. # LLM Output Source: https://docs.transluce.org/concepts/llm-output # LLM output This module defines data models that are standardized across different LLM providers. ### **FinishReasonType** `module-attribute` ```python theme={null} FinishReasonType = Literal['error', 'stop', 'length', 'tool_calls', 'content_filter', 'function_call', 'streaming', 'refusal'] ``` Possible reasons for an LLM completion to finish. ### **LLMCompletion** Bases: `BaseModel` A single completion from an LLM. **Attributes:** | Name | Type | Description | | | ------------------ | -------------------------- | ----------- | ----------------------------------------------- | | `text` | \`str | None\` | The generated text content. | | `tool_calls` | \`list\[ToolCall] | None\` | List of tool calls made during the completion. | | `finish_reason` | \`FinishReasonType | None\` | Reason why the completion finished. | | `top_logprobs` | \`list\[list\[TopLogprob]] | None\` | Probability distribution for top token choices. | | `reasoning_tokens` | \`str | None\` | Extended thinking tokens for reasoning models. | ```python theme={null} class LLMCompletion(BaseModel): """A single completion from an LLM. Attributes: text: The generated text content. tool_calls: List of tool calls made during the completion. finish_reason: Reason why the completion finished. top_logprobs: Probability distribution for top token choices. reasoning_tokens: Extended thinking tokens (for reasoning models). """ text: str | None = None tool_calls: list[ToolCall] | None = None finish_reason: FinishReasonType | None = None top_logprobs: list[list[TopLogprob]] | None = None reasoning_tokens: str | None = None @property def no_text(self) -> bool: """Check if the completion has no text. Returns: bool: True if text is None or empty, False otherwise. """ return self.text is None or len(self.text) == 0 ``` #### **no\_text** `property` ```python theme={null} no_text: bool ``` Check if the completion has no text. **Returns:** | Name | Type | Description | | ------ | ------ | ----------------------------------------------- | | `bool` | `bool` | True if text is None or empty, False otherwise. | ### **LLMOutput** `dataclass` Container for LLM output, potentially with multiple completions. Aggregates completions from an LLM along with metadata and error information. **Attributes:** | Name | Type | Description | | ------------- | --------------------- | -------------------------------------------------- | | `model` | `str` | The name/identifier of the model used. | | `completions` | `list[LLMCompletion]` | List of individual completions. | | `errors` | `list[LLMException]` | List of error types encountered during generation. | ```python theme={null} @dataclass class LLMOutput: """Container for LLM output, potentially with multiple completions. Aggregates completions from an LLM along with metadata and error information. Attributes: model: The name/identifier of the model used. completions: List of individual completions. errors: List of error types encountered during generation. """ model: str completions: list[LLMCompletion] errors: list[LLMException] = field(default_factory=list) usage: UsageMetrics = field(default_factory=UsageMetrics) duration: float | None = None @property def non_empty(self) -> bool: """Check if there are any completions. Returns: bool: True if there's at least one completion, False otherwise. """ return len(self.completions) > 0 @property def first(self) -> LLMCompletion | None: """Get the first completion if available. Returns: LLMCompletion | None: The first completion or None if no completions exist. """ return self.completions[0] if self.non_empty else None @property def first_text(self) -> str | None: """Get the text of the first completion if available. Returns: str | None: The text of the first completion or None if no completion exists. """ return self.first.text if self.first else None @property def did_error(self) -> bool: """Check if any errors occurred during generation. Returns: bool: True if there were errors, False otherwise. """ return bool(self.errors) def to_dict(self) -> dict[str, Any]: return { "model": self.model, "completions": [comp.model_dump() for comp in self.completions], "errors": [e.error_type_id for e in self.errors], "usage": self.usage.to_dict(), "duration": self.duration, } @classmethod def from_dict(cls, data: dict[str, Any]) -> "LLMOutput": error_type_map = {e.error_type_id: e for e in LLM_ERROR_TYPES} errors = data.get("errors", []) error_types_to_not_log: list[str] = [ CompletionTooLongException.error_type_id, ContextWindowException.error_type_id, ] errors_to_log = [e for e in errors if e not in error_types_to_not_log] if errors_to_log: logger.error(f"Loading LLM output with errors: {errors}") errors = [error_type_map.get(e, LLMException)() for e in errors] completions = data.get("completions", []) completions = [LLMCompletion.model_validate(comp) for comp in completions] usage: dict[TokenType, int] = {} if data_usage := data.get("usage"): usage = cast(dict[TokenType, int], data_usage) return cls( model=data["model"], completions=completions, errors=errors, usage=UsageMetrics(**usage), duration=data.get("duration"), ) ``` #### **non\_empty** `property` ```python theme={null} non_empty: bool ``` Check if there are any completions. **Returns:** | Name | Type | Description | | ------ | ------ | --------------------------------------------------------- | | `bool` | `bool` | True if there's at least one completion, False otherwise. | #### **first** `property` ```python theme={null} first: LLMCompletion | None ``` Get the first completion if available. **Returns:** | Type | Description | | | | --------------- | ----------- | ------------- | ----------------------------------------------------------- | | \`LLMCompletion | None\` | LLMCompletion | None: The first completion or None if no completions exist. | #### **first\_text** `property` ```python theme={null} first_text: str | None ``` Get the text of the first completion if available. **Returns:** | Type | Description | | | | ----- | ----------- | --- | ----------------------------------------------------------------------- | | \`str | None\` | str | None: The text of the first completion or None if no completion exists. | #### **did\_error** `property` ```python theme={null} did_error: bool ``` Check if any errors occurred during generation. **Returns:** | Name | Type | Description | | ------ | ------ | ------------------------------------------- | | `bool` | `bool` | True if there were errors, False otherwise. | ### **ToolCallPartial** `dataclass` Partial representation of a tool call before full processing. Used as an intermediate format before finalizing into a complete ToolCall. **Parameters:** | Name | Type | Description | Default | | | --------------- | --------------------- | --------------------------------------------- | ---------------------------------------------- | ---------- | | `id` | \`str | None\` | The identifier for the tool call. | *required* | | `function` | \`str | None\` | The name of the function to call. | *required* | | `arguments_raw` | \`str | None\` | Raw JSON string of arguments for the function. | *required* | | `type` | `Literal['function']` | The type of the tool call, always "function". | *required* | | ```python theme={null} @dataclass class ToolCallPartial: """Partial representation of a tool call before full processing. Used as an intermediate format before finalizing into a complete ToolCall. Args: id: The identifier for the tool call. function: The name of the function to call. arguments_raw: Raw JSON string of arguments for the function. type: The type of the tool call, always "function". """ id: str | None function: str | None arguments_raw: str | None type: Literal["function"] ``` ### **LLMCompletionPartial** Bases: `LLMCompletion` Partial representation of an LLM completion before finalization. Extends LLMCompletion but with tool\_calls being a list of ToolCallPartial. This is used during the processing stage before tool calls are fully parsed. **Attributes:** | Name | Type | Description | | | | ------------ | ----------------------- | ----------- | ------ | ------------------------------------------ | | `tool_calls` | \`list\[ToolCallPartial | None] | None\` | List of partial tool call representations. | ```python theme={null} class LLMCompletionPartial(LLMCompletion): """Partial representation of an LLM completion before finalization. Extends LLMCompletion but with tool_calls being a list of ToolCallPartial. This is used during the processing stage before tool calls are fully parsed. Attributes: tool_calls: List of partial tool call representations. """ tool_calls: list[ToolCallPartial | None] | None = None # type: ignore ``` #### **no\_text** `property` ```python theme={null} no_text: bool ``` Check if the completion has no text. **Returns:** | Name | Type | Description | | ------ | ------ | ----------------------------------------------- | | `bool` | `bool` | True if text is None or empty, False otherwise. | ### **LLMOutputPartial** `dataclass` Bases: `LLMOutput` Partial representation of LLM output before finalization. Extends LLMOutput but with completions being a list of LLMCompletionPartial. Used as an intermediate format during processing. **Attributes:** | Name | Type | Description | | ------------- | ---------------------------- | ---------------------------- | | `completions` | `list[LLMCompletionPartial]` | List of partial completions. | ```python theme={null} class LLMOutputPartial(LLMOutput): """Partial representation of LLM output before finalization. Extends LLMOutput but with completions being a list of LLMCompletionPartial. Used as an intermediate format during processing. Attributes: completions: List of partial completions. """ completions: list[LLMCompletionPartial] # type: ignore ``` #### **non\_empty** `property` ```python theme={null} non_empty: bool ``` Check if there are any completions. **Returns:** | Name | Type | Description | | ------ | ------ | --------------------------------------------------------- | | `bool` | `bool` | True if there's at least one completion, False otherwise. | #### **first** `property` ```python theme={null} first: LLMCompletion | None ``` Get the first completion if available. **Returns:** | Type | Description | | | | --------------- | ----------- | ------------- | ----------------------------------------------------------- | | \`LLMCompletion | None\` | LLMCompletion | None: The first completion or None if no completions exist. | #### **first\_text** `property` ```python theme={null} first_text: str | None ``` Get the text of the first completion if available. **Returns:** | Type | Description | | | | ----- | ----------- | --- | ----------------------------------------------------------------------- | | \`str | None\` | str | None: The text of the first completion or None if no completion exists. | #### **did\_error** `property` ```python theme={null} did_error: bool ``` Check if any errors occurred during generation. **Returns:** | Name | Type | Description | | ------ | ------ | ------------------------------------------- | | `bool` | `bool` | True if there were errors, False otherwise. | ### **AsyncLLMOutputStreamingCallback** Bases: `Protocol` Protocol for asynchronous streaming callbacks with batch index. Defines the expected signature for callbacks that handle streaming output with a batch index. **Parameters:** | Name | Type | Description | Default | | ------------- | ---- | ------------------------------------- | ---------- | | `batch_index` | | The index of the current batch. | *required* | | `llm_output` | | The LLM output for the current batch. | *required* | ```python theme={null} class AsyncLLMOutputStreamingCallback(Protocol): """Protocol for asynchronous streaming callbacks with batch index. Defines the expected signature for callbacks that handle streaming output with a batch index. Args: batch_index: The index of the current batch. llm_output: The LLM output for the current batch. """ async def __call__( self, batch_index: int, llm_output: LLMOutput, ) -> None: ... ``` ### **AsyncSingleLLMOutputStreamingCallback** Bases: `Protocol` Protocol for asynchronous streaming callbacks without batch indexing. Defines the expected signature for callbacks that handle streaming output without batch indexing. **Parameters:** | Name | Type | Description | Default | | ------------ | ---- | -------------------------- | ---------- | | `llm_output` | | The LLM output to process. | *required* | ```python theme={null} class AsyncSingleLLMOutputStreamingCallback(Protocol): """Protocol for asynchronous streaming callbacks without batch indexing. Defines the expected signature for callbacks that handle streaming output without batch indexing. Args: llm_output: The LLM output to process. """ async def __call__( self, llm_output: LLMOutput, ) -> None: ... ``` ### **AsyncEmbeddingStreamingCallback** Bases: `Protocol` Protocol for sending progress updates for embedding generation. ```python theme={null} class AsyncEmbeddingStreamingCallback(Protocol): """Protocol for sending progress updates for embedding generation.""" async def __call__(self, progress: int) -> None: ... ``` ### **finalize\_llm\_output\_partial** ```python theme={null} finalize_llm_output_partial(partial: LLMOutputPartial) -> LLMOutput ``` Convert a partial LLM output into a finalized LLM output. Processes tool calls by parsing their arguments from raw JSON strings, handles errors in JSON parsing, and provides warnings for truncated completions. **Parameters:** | Name | Type | Description | Default | | --------- | ------------------ | ----------------------------------- | ---------- | | `partial` | `LLMOutputPartial` | The partial LLM output to finalize. | *required* | **Returns:** | Name | Type | Description | | ----------- | ----------- | --------------------------------------------------- | | `LLMOutput` | `LLMOutput` | The finalized LLM output with processed tool calls. | **Raises:** | Type | Description | | ---------------------------- | ------------------------------------------------------------------------- | | `CompletionTooLongException` | If the completion was truncated due to length and resulted in empty text. | | `ValueError` | If tool call ID or function is missing in the partial data. | ```python theme={null} def finalize_llm_output_partial(partial: LLMOutputPartial) -> LLMOutput: """Convert a partial LLM output into a finalized LLM output. Processes tool calls by parsing their arguments from raw JSON strings, handles errors in JSON parsing, and provides warnings for truncated completions. Args: partial: The partial LLM output to finalize. Returns: LLMOutput: The finalized LLM output with processed tool calls. Raises: CompletionTooLongException: If the completion was truncated due to length and resulted in empty text. ValueError: If tool call ID or function is missing in the partial data. """ def _parse_tool_call(tc_partial: ToolCallPartial): if tc_partial.id is None: raise ValueError("Tool call ID not found in partial; check for parsing errors") if tc_partial.function is None: raise ValueError("Tool call function not found in partial; check for parsing errors") arguments: dict[str, Any] = {} # Attempt to load arguments into JSON try: arguments = json.loads(tc_partial.arguments_raw or "{}") parse_error = None # If the tool call arguments are not valid JSON, return an empty dict with the error except Exception as e: arguments = {"__parse_error_raw_args": tc_partial.arguments_raw} parse_error = f"Couldn't parse tool call arguments as JSON: {e}. Original input: {tc_partial.arguments_raw}" return ToolCall( id=tc_partial.id, function=tc_partial.function, arguments=arguments, parse_error=parse_error, type=tc_partial.type, ) output = LLMOutput( model=partial.model, completions=[ LLMCompletion( text=c.text, tool_calls=[_parse_tool_call(tc) for tc in (c.tool_calls or []) if tc is not None], finish_reason=c.finish_reason, reasoning_tokens=c.reasoning_tokens, ) for c in partial.completions ], usage=partial.usage, ) # If the completion is empty and was truncated (likely due to too much reasoning), raise an exception if output.first and output.first.finish_reason == "length" and output.first.no_text: raise CompletionTooLongException( "Completion empty due to truncation. Consider increasing max_new_tokens." ) for c in output.completions: if c.finish_reason == "length": logger.warning( "Completion truncated due to length; consider increasing max_new_tokens." ) return output ``` # Metadata Source: https://docs.transluce.org/concepts/metadata # Metadata Docent supports metadata at multiple levels: * Collection metadata attached to the collection itself * Agent run metadata attached to an [AgentRun](/concepts/agent-run) * Transcript group metadata attached to a `TranscriptGroup` * Transcript metadata attached to a [Transcript](/concepts/transcript) Any metadata should be JSON serializable. When metadata is rendered/stored, Docent converts it to JSON-compatible values using Pydantic's serializer (which supports common Python collections and nested Pydantic models). ## Choosing a metadata level * Use collection metadata for information shared by the entire collection, such as dataset provenance, eval configuration, environment, or model family. * Use agent run metadata for values that vary run to run, especially scores or other fields you want to analyze across a collection. * Use transcript group or transcript metadata for finer-grained context within a single run. ## Collection metadata Collection metadata lives on the collection rather than on individual runs. It is a good fit for collection-wide configuration and provenance. ### From the Python SDK ```python theme={null} from docent import Docent client = Docent() collection_id = "..." client.update_collection_metadata( collection_id, { "dataset": "helpdesk_jan_2026", "config": { "model": "gpt-5", "prompt_version": "v3", }, }, ) metadata = client.get_collection_metadata(collection_id) metadata_after_delete, not_found = client.delete_collection_metadata_keys( collection_id, ["config.prompt_version"], ) ``` Updates are deep-merged into the existing collection metadata, so patching `config.model` does not remove unrelated keys under `config`. Deletions support dot paths for nested keys. ### From tracing ```python theme={null} from docent.trace import collection_metadata, initialize_tracing initialize_tracing("customer-support-evals") collection_metadata( { "dataset": "helpdesk_jan_2026", "environment": "staging", "config": { "model": "gpt-5", "prompt_version": "v3", }, } ) ``` You can call `collection_metadata()` any time after `initialize_tracing()`. Unlike `agent_run_metadata()`, it does not require an active agent run or transcript context. ## Agent run, transcript group, and transcript metadata We recommend including information about metrics / scores in metadata, as well as other information about the agent or task setup. Scoring fields are useful for tracking metrics, like task completion or reward, but they are a convention rather than a required schema. Neither `AgentRun` nor `Transcript` enforces required metadata keys. Here's an example of what a typical agent run metadata dict might look like: ```python theme={null} metadata = { # Optional conventional fields "scores": {"reward_1": 0.1, "reward_2": 0.5, "reward_3": 0.8}, # Custom fields "episode": 42, "policy_version": "v1.2.3", "training_step": 12500, } ``` If you're using Inspect, `docent.loaders.load_inspect` also contains a `load_inspect_log` function which reads the standard scoring and metadata information from Inspect logs and copies them into Docent metadata. # Data Models Overview Source: https://docs.transluce.org/concepts/overview This guide explains how Docent formats and organizes your transcripts. Use it to decide how to ingest your data for the analysis you want to do. ## Collection A Collection is a Docent workspace for one experiment. It holds the set of agent runs you want to analyze together. The typical unit is a single benchmark or eval, with metadata (model, checkpoint, scaffold) attached to each run so you can slice and compare across runs within it. Filters, DQL, and Analysis Plans all operate within a single Collection. The Collection is the frame that all analysis sits inside. See the [Collection reference](/concepts/collection) or the [SDK collections API](/sdk/collections/manage) for details. ## Agent Run An AgentRun is one execution of an agent against one task. Think of it as the row in your dataset. Filters, DQL queries, and clustering return sets of AgentRuns. A rubric grades one AgentRun at a time. Search returns AgentRuns that match your prompt. An AgentRun bundles one or more Transcripts. Metadata can attach to the AgentRun as a whole, or to individual Transcripts within it. See the [Agent Run reference](/concepts/agent-run) for details. ## Transcript A Transcript is the sequence of ChatMessages from one agent's point of view. Single-agent runs have one Transcript per AgentRun. See the [Transcript reference](/concepts/transcript). *Multi-agent setup? Transcripts can be organized into optional [TranscriptGroups](/concepts/transcript#transcriptgroup) within a single AgentRun. Most single-agent evals ignore them.* ## Chat Message A ChatMessage is one turn in a conversation: `SystemMessage`, `UserMessage`, `AssistantMessage` (optionally with tool calls), or `ToolMessage`. The schema is OpenAI-compatible. If your messages are already in that format, `parse_chat_message` converts them directly. See the [Chat Messages reference](/concepts/chat-messages). ## Metadata Metadata is a JSON dict you attach to Collections, AgentRuns, TranscriptGroups, and Transcripts. Richer metadata makes everything else in Docent more useful: DQL queries join on it, rubric prompts reference it, and dashboard filters slice by it. Common fields on an AgentRun: * `scores` (e.g. `{"reward": 0.7, "passed": true}`). Scoring info goes under `metadata["scores"]` by convention. * `model`, `checkpoint`, `agent_scaffold` for cross-run comparisons. * `task_id`, `difficulty`, `category` for slicing. * `cost`, `latency_ms`, `token_count` for quantitative rollups. "Average reward per model" is a one-line DQL query when `model` and `reward` are in metadata. It's impossible when they're not. See the [Metadata reference](/concepts/metadata). ## Next steps * [Installation](/installation): set up Docent. * [Analysis Quickstart](/analysis/quickstart): run your first analysis. * [Ingestion Quickstart](/ingestion/quickstart): pick a path to load your data. * [Analysis Plans](/analysis/analysis-plans): pick a mode to analyze it. # Transcript Source: https://docs.transluce.org/concepts/transcript # Transcript A `Transcript` object represents a sequence of chat messages (user, assistant, system, tool) from the perspective of *a single* agent. See [here for more details on the chat message schemas](/concepts/chat-messages). ### **TranscriptGroup** Bases: `BaseModel` Represents a group of transcripts that are logically related. A transcript group can contain multiple transcripts and can have a hierarchical structure with parent groups. This is useful for organizing transcripts into logical units like experiments, tasks, or sessions. **Attributes:** | Name | Type | Description | | | ---------------------------- | ---------------- | ------------------------------------------------------------ | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `id` | `str` | Unique identifier, auto-generated; cannot be set by callers. | | | `name` | \`str | None\` | Optional human-readable name for the transcript group. | | `description` | \`str | None\` | Optional description of the transcript group. | | `agent_run_id` | `str` | ID of the agent run this transcript group belongs to. | | | `parent_transcript_group_id` | \`str | None\` | Optional ID of the parent transcript group. | | `created_at` | \`datetime | None\` | Optional creation timestamp. Both naive and timezone-aware `datetime` values are accepted; tz-aware values are converted to UTC on ingest and stored as naive UTC. Leave as `None` to let the server assign one. | | `metadata` | `dict[str, Any]` | Additional structured metadata about the transcript group. | | ```python theme={null} class TranscriptGroup(BaseModel): """Represents a group of transcripts that are logically related. A transcript group can contain multiple transcripts and can have a hierarchical structure with parent groups. This is useful for organizing transcripts into logical units like experiments, tasks, or sessions. Attributes: id: Unique identifier, auto-generated; cannot be set by callers. name: Optional human-readable name for the transcript group. description: Optional description of the transcript group. agent_run_id: ID of the agent run this transcript group belongs to. parent_transcript_group_id: Optional ID of the parent transcript group. metadata: Additional structured metadata about the transcript group. """ id: str = Field(default_factory=lambda: str(uuid4()), frozen=True) name: str | None = None description: str | None = None agent_run_id: str parent_transcript_group_id: str | None = None created_at: datetime | None = None metadata: dict[str, Any] = Field(default_factory=dict) def __setattr__(self, name: str, value: Any) -> None: if name == "id": raise ValueError( "Cannot set `id` on TranscriptGroup. Docent assigns IDs automatically; " "the assigned value is already available as `group.id`." ) super().__setattr__(name, value) def to_text(self, children_text: str, indent: int = 0, render_metadata: bool = True) -> str: """Render this transcript group with its children and metadata. Metadata appears below the rendered children content. Args: children_text: Pre-rendered text of this group's children (groups/transcripts). indent: Number of spaces to indent the rendered output. render_metadata: Whether to include metadata in the output. Returns: str: XML-like wrapped text including the group's metadata. """ # Prepare YAML metadata if render_metadata: metadata_text = dump_metadata(self.metadata) if metadata_text is not None: if indent > 0: metadata_text = textwrap.indent(metadata_text, " " * indent) inner = f"{children_text}\n<|{self.name} metadata|>\n{metadata_text}\n" else: inner = children_text else: inner = children_text # Compose final text: content first, then metadata, all inside the group wrapper if indent > 0: inner = textwrap.indent(inner, " " * indent) return f"<|{self.name}|>\n{inner}\n" ``` #### **to\_text** ```python theme={null} to_text(children_text: str, indent: int = 0, render_metadata: bool = True) -> str ``` Render this transcript group with its children and metadata. Metadata appears below the rendered children content. **Parameters:** | Name | Type | Description | Default | | ----------------- | ------ | ---------------------------------------------------------------- | ---------- | | `children_text` | `str` | Pre-rendered text of this group's children (groups/transcripts). | *required* | | `indent` | `int` | Number of spaces to indent the rendered output. | `0` | | `render_metadata` | `bool` | Whether to include metadata in the output. | `True` | **Returns:** | Name | Type | Description | | ----- | ----- | ----------------------------------------------------- | | `str` | `str` | XML-like wrapped text including the group's metadata. | ```python theme={null} def to_text(self, children_text: str, indent: int = 0, render_metadata: bool = True) -> str: """Render this transcript group with its children and metadata. Metadata appears below the rendered children content. Args: children_text: Pre-rendered text of this group's children (groups/transcripts). indent: Number of spaces to indent the rendered output. render_metadata: Whether to include metadata in the output. Returns: str: XML-like wrapped text including the group's metadata. """ # Prepare YAML metadata if render_metadata: metadata_text = dump_metadata(self.metadata) if metadata_text is not None: if indent > 0: metadata_text = textwrap.indent(metadata_text, " " * indent) inner = f"{children_text}\n<|{self.name} metadata|>\n{metadata_text}\n" else: inner = children_text else: inner = children_text # Compose final text: content first, then metadata, all inside the group wrapper if indent > 0: inner = textwrap.indent(inner, " " * indent) return f"<|{self.name}|>\n{inner}\n" ``` ### **Transcript** Bases: `BaseModel` Represents a transcript of messages in a conversation with an AI agent. A transcript contains a sequence of messages exchanged between different roles (system, user, assistant, tool) and provides methods to organize these messages into logical units of action. **Attributes:** | Name | Type | Description | | | --------------------- | ------------------- | ------------------------------------------------------------ | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `id` | `str` | Unique identifier, auto-generated; cannot be set by callers. | | | `name` | \`str | None\` | Optional human-readable name for the transcript. | | `description` | \`str | None\` | Optional description of the transcript. | | `transcript_group_id` | \`str | None\` | Optional ID of the transcript group this transcript belongs to. | | `created_at` | \`datetime | None\` | Optional creation timestamp. Both naive and timezone-aware `datetime` values are accepted; tz-aware values are converted to UTC on ingest and stored as naive UTC. Leave as `None` to let the server assign one. | | `messages` | `list[ChatMessage]` | List of chat messages in the transcript. | | | `metadata` | `dict[str, Any]` | Additional structured metadata about the transcript. | | ```python theme={null} class Transcript(BaseModel): """Represents a transcript of messages in a conversation with an AI agent. A transcript contains a sequence of messages exchanged between different roles (system, user, assistant, tool) and provides methods to organize these messages into logical units of action. Attributes: id: Unique identifier, auto-generated; cannot be set by callers. name: Optional human-readable name for the transcript. description: Optional description of the transcript. transcript_group_id: Optional ID of the transcript group this transcript belongs to. messages: List of chat messages in the transcript. metadata: Additional structured metadata about the transcript. """ id: str = Field(default_factory=lambda: str(uuid4()), frozen=True) name: str | None = None description: str | None = None transcript_group_id: str | None = None created_at: datetime | None = None messages: list[ChatMessage] metadata: dict[str, Any] = Field(default_factory=dict) def __setattr__(self, name: str, value: Any) -> None: if name == "id": raise ValueError( "Cannot set `id` on Transcript. Docent assigns IDs automatically; " "the assigned value is already available as `transcript.id`." ) super().__setattr__(name, value) def _enumerate_messages(self) -> Iterable[tuple[int, ChatMessage]]: """Yield (index, message) tuples for rendering. Override in subclasses to customize index assignment. """ return enumerate(self.messages) def to_text( self, transcript_alias: int | str = 0, indent: int = 0, render_metadata: bool = True, transcript_metadata_comments: list[Comment] | None = None, block_metadata_comments: dict[int, list[Comment]] | None = None, block_content_comments: dict[int, list[Comment]] | None = None, ) -> str: """Render this transcript as formatted text with optional comments. Args: transcript_alias: Identifier for the transcript (e.g., 0 becomes "T0"). indent: Number of spaces to indent nested content. render_metadata: Whether to include transcript metadata in the output. transcript_metadata_comments: Comments on this transcript's metadata. Rendered after the transcript metadata block. block_metadata_comments: Mapping from block index to comments on that block's metadata. Keyed by block index because comments need to be rendered inline with each block at the correct position. block_content_comments: Mapping from block index to comments on that block's content. Keyed by block index because comments need to be rendered inline with each block, and may include text range selections that highlight specific portions of the block content. Returns: Formatted text representation of the transcript. """ if isinstance(transcript_alias, int): transcript_alias = f"T{transcript_alias}" # Format individual message blocks blocks: list[str] = [] for msg_idx, message in self._enumerate_messages(): block_label = f"{transcript_alias}B{msg_idx}" # Get block-level comments for this message index msg_metadata_comments = ( block_metadata_comments.get(msg_idx) if block_metadata_comments else None ) msg_content_comments = ( block_content_comments.get(msg_idx) if block_content_comments else None ) block_text = format_chat_message( message, block_label, block_metadata_comments=msg_metadata_comments, block_content_comments=msg_content_comments, indent=indent, ) blocks.append(block_text) blocks_str = "\n".join(blocks) if indent > 0: blocks_str = textwrap.indent(blocks_str, " " * indent) content_str = f"<|{transcript_alias} blocks|>\n{blocks_str}\n" # Gather metadata and add to content if render_metadata: metadata_text = dump_metadata(self.metadata) if metadata_text is not None: if indent > 0: metadata_text = textwrap.indent(metadata_text, " " * indent) metadata_label = f"{transcript_alias}M" content_str += f"\n<|transcript metadata {metadata_label}|>\n{metadata_text}\n" # Add transcript metadata comments after the metadata if transcript_metadata_comments: metadata_comments_text = render_metadata_comments(transcript_metadata_comments) if metadata_comments_text: if indent > 0: metadata_comments_text = textwrap.indent( metadata_comments_text, " " * indent ) content_str += f"\n<|transcript metadata comments|>\n{metadata_comments_text}\n" # Format content and return if indent > 0: content_str = textwrap.indent(content_str, " " * indent) return f"<|transcript {transcript_alias}|>\n{content_str}\n\n" ``` #### **to\_text** ```python theme={null} to_text(transcript_alias: int | str = 0, indent: int = 0, render_metadata: bool = True, transcript_metadata_comments: list[Comment] | None = None, block_metadata_comments: dict[int, list[Comment]] | None = None, block_content_comments: dict[int, list[Comment]] | None = None) -> str ``` Render this transcript as formatted text with optional comments. **Parameters:** | Name | Type | Description | Default | | | ------------------------------ | ---------------------------- | ----------------------------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ------ | | `transcript_alias` | \`int | str\` | Identifier for the transcript (e.g., 0 becomes "T0"). | `0` | | `indent` | `int` | Number of spaces to indent nested content. | `0` | | | `render_metadata` | `bool` | Whether to include transcript metadata in the output. | `True` | | | `transcript_metadata_comments` | \`list\[Comment] | None\` | Comments on this transcript's metadata. Rendered after the transcript metadata block. | `None` | | `block_metadata_comments` | \`dict\[int, list\[Comment]] | None\` | Mapping from block index to comments on that block's metadata. Keyed by block index because comments need to be rendered inline with each block at the correct position. | `None` | | `block_content_comments` | \`dict\[int, list\[Comment]] | None\` | Mapping from block index to comments on that block's content. Keyed by block index because comments need to be rendered inline with each block, and may include text range selections that highlight specific portions of the block content. | `None` | **Returns:** | Type | Description | | ----- | ------------------------------------------------ | | `str` | Formatted text representation of the transcript. | ```python theme={null} def to_text( self, transcript_alias: int | str = 0, indent: int = 0, render_metadata: bool = True, transcript_metadata_comments: list[Comment] | None = None, block_metadata_comments: dict[int, list[Comment]] | None = None, block_content_comments: dict[int, list[Comment]] | None = None, ) -> str: """Render this transcript as formatted text with optional comments. Args: transcript_alias: Identifier for the transcript (e.g., 0 becomes "T0"). indent: Number of spaces to indent nested content. render_metadata: Whether to include transcript metadata in the output. transcript_metadata_comments: Comments on this transcript's metadata. Rendered after the transcript metadata block. block_metadata_comments: Mapping from block index to comments on that block's metadata. Keyed by block index because comments need to be rendered inline with each block at the correct position. block_content_comments: Mapping from block index to comments on that block's content. Keyed by block index because comments need to be rendered inline with each block, and may include text range selections that highlight specific portions of the block content. Returns: Formatted text representation of the transcript. """ if isinstance(transcript_alias, int): transcript_alias = f"T{transcript_alias}" # Format individual message blocks blocks: list[str] = [] for msg_idx, message in self._enumerate_messages(): block_label = f"{transcript_alias}B{msg_idx}" # Get block-level comments for this message index msg_metadata_comments = ( block_metadata_comments.get(msg_idx) if block_metadata_comments else None ) msg_content_comments = ( block_content_comments.get(msg_idx) if block_content_comments else None ) block_text = format_chat_message( message, block_label, block_metadata_comments=msg_metadata_comments, block_content_comments=msg_content_comments, indent=indent, ) blocks.append(block_text) blocks_str = "\n".join(blocks) if indent > 0: blocks_str = textwrap.indent(blocks_str, " " * indent) content_str = f"<|{transcript_alias} blocks|>\n{blocks_str}\n" # Gather metadata and add to content if render_metadata: metadata_text = dump_metadata(self.metadata) if metadata_text is not None: if indent > 0: metadata_text = textwrap.indent(metadata_text, " " * indent) metadata_label = f"{transcript_alias}M" content_str += f"\n<|transcript metadata {metadata_label}|>\n{metadata_text}\n" # Add transcript metadata comments after the metadata if transcript_metadata_comments: metadata_comments_text = render_metadata_comments(transcript_metadata_comments) if metadata_comments_text: if indent > 0: metadata_comments_text = textwrap.indent( metadata_comments_text, " " * indent ) content_str += f"\n<|transcript metadata comments|>\n{metadata_comments_text}\n" # Format content and return if indent > 0: content_str = textwrap.indent(content_str, " " * indent) return f"<|transcript {transcript_alias}|>\n{content_str}\n\n" ``` ### **render\_metadata\_comments** ```python theme={null} render_metadata_comments(comments: list[Comment]) -> str ``` Render metadata comments (agent run, transcript, or block metadata). For metadata comments, we render the key on which the comment was written and the user's content. TODO(mengk): known limitation: does not highlight text\_range selections, if available. I'm not sure if it's supported in the UI, but just pointing this out for the backend. **Parameters:** | Name | Type | Description | Default | | ---------- | --------------- | ------------------------------------------- | ---------- | | `comments` | `list[Comment]` | List of Comment objects targeting metadata. | *required* | **Returns:** | Type | Description | | ----- | ----------------------------------- | | `str` | Formatted string with all comments. | ```python theme={null} def render_metadata_comments(comments: list[Comment]) -> str: """Render metadata comments (agent run, transcript, or block metadata). For metadata comments, we render the key on which the comment was written and the user's content. TODO(mengk): known limitation: does not highlight text_range selections, if available. I'm not sure if it's supported in the UI, but just pointing this out for the backend. Args: comments: List of Comment objects targeting metadata. Returns: Formatted string with all comments. """ if not comments: return "" lines: list[str] = [] for comment in comments: # Iterate through citations to find the right target metadata_key = "unknown" for citation in comment.citations: item = citation.target.item if isinstance(item, TranscriptMetadataItem): metadata_key = item.metadata_key break elif isinstance(item, AgentRunMetadataItem): metadata_key = item.metadata_key break lines.append(f'{comment.content}') return "\n".join(lines) ``` ### **render\_block\_content\_comments** ```python theme={null} render_block_content_comments(comments: list[Comment], content: str, comment_index_offset: int = 0) -> tuple[str, str] ``` Render block content comments with text range highlighting. For block content comments with text\_range, we surround the range with tags and render the comment content below with a reference to the selection. **Parameters:** | Name | Type | Description | Default | | ---------------------- | --------------- | -------------------------------------------------------- | ---------- | | `comments` | `list[Comment]` | List of Comment objects targeting block content. | *required* | | `content` | `str` | The block content text to annotate. | *required* | | `comment_index_offset` | `int` | Starting index for comment numbering (local to message). | `0` | **Returns:** | Type | Description | | ----- | -------------------------------------------------------------------------- | | `str` | Tuple of (annotated\_content, comments\_text) where annotated\_content has | | `str` | selection tags inserted and comments\_text contains the rendered comments. | ```python theme={null} def render_block_content_comments( comments: list[Comment], content: str, comment_index_offset: int = 0, ) -> tuple[str, str]: """Render block content comments with text range highlighting. For block content comments with text_range, we surround the range with tags and render the comment content below with a reference to the selection. Args: comments: List of Comment objects targeting block content. content: The block content text to annotate. comment_index_offset: Starting index for comment numbering (local to message). Returns: Tuple of (annotated_content, comments_text) where annotated_content has selection tags inserted and comments_text contains the rendered comments. """ if not comments: return content, "" # Build a list of (position, tag_text) for all tag insertions. # By treating start and end tags as independent insertions sorted by position # descending, we correctly handle overlapping/nested ranges. Each insertion # only affects positions after it, so processing from the end backward # preserves all indices. insertions: list[tuple[int, str]] = [] comments_with_text_range: set[str] = set() for i, comment in enumerate(comments): # Iterate through citations to find the right target text_range: CitationTargetTextRange | None = None for citation in comment.citations: item = citation.target.item if isinstance(item, TranscriptBlockContentItem): text_range = citation.target.text_range break # If the text range exists, add the start and end tags if ( text_range and text_range.target_start_idx is not None and text_range.target_end_idx is not None ): start_idx = text_range.target_start_idx end_idx = text_range.target_end_idx if 0 <= start_idx < len(content) and start_idx < end_idx <= len(content): # End tag goes at end_idx, start tag goes at start_idx comment_idx = comment_index_offset + i insertions.append((end_idx, f"")) insertions.append((start_idx, f"")) # Keep track of comments with text ranges comments_with_text_range.add(comment.id) # Sort by position descending. For ties (e.g., end of one range = start of another), # end tags (closing) should come before start tags (opening) at the same position # to produce valid nesting, but since our tags don't need to be valid XML, order # at ties doesn't matter for correctness. insertions.sort(key=lambda x: x[0], reverse=True) # Apply insertions from the end backward to preserve indices annotated_content = content for pos, tag in insertions: annotated_content = annotated_content[:pos] + tag + annotated_content[pos:] # Build comment text comment_lines: list[str] = [] for i, comment in enumerate(comments): if comment.id in comments_with_text_range: comment_idx = comment_index_offset + i comment_lines.append( f'{comment.content}' ) else: comment_lines.append(f"{comment.content}") return annotated_content, "\n".join(comment_lines) ``` # Agentic Ingestion Source: https://docs.transluce.org/ingestion/agentic How the /docent plugin generates ingestion scripts This page explains how the `/docent` plugin handles ingestion under the hood. To ingest your data, point your coding agent at a directory containing your trajectories. For best results, sort your trajectories by format before invoking it. ```text wrap theme={null} /docent Ingest the trajectories at ``` ## How the Docent plugin handles ingestion Using the `/docent` plugin, your coding agent uploads your agent logs into Docent by writing a Python script that converts them into `AgentRun` format. It investigates your file structure, examines your trajectory format, and maps each field in your schema to a Docent object. It produces: 1. `ingestion-plan.md`: A mapping of your trajectory fields to Docent's data model, including any fields that will be intentionally omitted. Whether your coding agent pauses for you to review this plan follows **Settings → Preferences**. You can override that preference by asking the agent to auto-approve or require review for a specific ingestion. 2. `ingest.py`: A Python script that reads your logs and uploads them via the SDK. You can modify and rerun this as needed. Your coding agent still asks clarifying questions if your data format is ambiguous or it needs more context about how you want the data structured. Auto-approval only controls whether it pauses after producing a complete ingestion plan. If your coding agent can't infer your format or you need fine-grained control, write the ingest script directly. See [SDK ingestion](/ingestion/sdk). ## Best practices * **Ingest one trajectory format at a time.** If you have multiple formats, sort your trajectories by the scaffold that generated them (e.g., `openhands/`, `mini-swe-agent/`, `custom/`) before invoking `/docent`. * **Include metadata relevant to your analysis.** Fields like reward, model name, and task ID enable downstream filtering, DQL queries, and rubric evaluation. * **Verify your uploaded data in the web interface.** After upload, check that transcripts display correctly and metadata appears where you expect. It's normal to iterate a few times on different organization structures. ## What's next * **Ready to analyze?** See [Analysis Plans](/analysis/analysis-plans). * **Want to tweak the ingestion script yourself?** See [SDK ingestion](/ingestion/sdk) for the underlying API. # Harbor Source: https://docs.transluce.org/ingestion/integrations/harbor Ingest Harbor/ATIF trajectories into Docent Harbor ([web](https://www.harborframework.com/), [GitHub](https://github.com/harbor-framework/harbor)) is a framework for evaluating and optimizing AI agents. ATIF ([docs](https://www.harborframework.com/docs/agents/trajectory-format)) is a trajectory format developed by Harbor. We offer helpers that import Harbor outputs and ATIF trajectories into Docent. ## Use this when Use Harbor ingestion when your source data is either: * a Harbor trial directory containing `agent/`, `verifier/`, `config.json`, and `result.json` (see [output structure](https://www.harborframework.com/docs/run-jobs/results-and-artifacts#output-structure)) * a raw ATIF trajectory JSON payload that you want to convert directly ## Main helpers * `convert_atif_to_agent_run(atif)` converts one parsed ATIF payload into one Docent `AgentRun`. * `convert_harbor_trial_to_agent_run(trial_dir, path_root=None)` converts one Harbor trial directory into one `AgentRun`. * `convert_harbor_directory_to_agent_runs(root)` recursively discovers Harbor trials and converts each one. All three helpers are available from `docent.sdk.integrations`. ```python theme={null} from docent.sdk.integrations import ( convert_atif_to_agent_run, convert_harbor_directory_to_agent_runs, convert_harbor_trial_to_agent_run, ) ``` ## Example To convert every Harbor trial under a root directory: ```python theme={null} from docent.sdk.integrations import convert_harbor_directory_to_agent_runs agent_runs = convert_harbor_directory_to_agent_runs("/path/to/harbor/root") ``` To convert a raw ATIF payload already loaded in memory: ```python theme={null} import json from docent.sdk.integrations import convert_atif_to_agent_run with open("trajectory.json", "r", encoding="utf-8") as infile: atif = json.load(infile) agent_run = convert_atif_to_agent_run(atif) ``` After conversion, upload normally: ```python theme={null} from docent import Docent from docent.sdk.integrations import convert_harbor_directory_to_agent_runs client = Docent() agent_runs = convert_harbor_directory_to_agent_runs("/path/to/harbor/root") client.add_agent_runs(collection_id, agent_runs) ``` ## More on the conversion process Each Harbor trial or ATIF payload becomes one Docent `AgentRun`. At a high level, the converter: * turns the ATIF trajectory into a single Docent transcript * preserves ATIF metadata under `agent_run.metadata["atif"]` * preserves Harbor trial metadata under `agent_run.metadata["harbor"]` * stores the raw `config.json` and `result.json` payloads in Harbor metadata when converting a Harbor trial For Harbor trial directories specifically, Docent expects exactly one ATIF JSON file directly under `agent/`. The converter is strict. If the source data uses unsupported ATIF features such as image content, continued trajectories, subagent references, malformed step sequences, or invalid Harbor trial structure, it raises `ConversionError` instead of trying to approximate the data. # Inspect Source: https://docs.transluce.org/ingestion/integrations/inspect Ingest Inspect logs into Docent Inspect ([web](https://inspect.aisi.org.uk/), [GitHub](https://github.com/UKGovernmentBEIS/inspect_ai)) is an open-source framework for large language model evaluations. We offer helpers that import Inspect `.eval` logs into Docent. ## Use this when Use Inspect ingestion when your source data is already in Inspect `.eval` format and you want either: * a pure conversion step that gives you `AgentRun` objects back * a recursive conversion-and-upload workflow for a directory of `.eval` files ## Main helpers * `convert_inspect_eval_file_to_agent_runs(file_path)` converts one Inspect `.eval` archive into a list of `AgentRun` objects. * `convert_inspect_directory_to_agent_runs(root)` recursively finds `.eval` files and returns all converted runs. * `ingest_inspect_directory(collection_id, fpath, *, upload_agent_run_batch, batch_size=100)` batches conversion and upload if you want a lower-level ingestion loop. * `Docent.recursively_ingest_inspect_logs(collection_id, fpath)` is the highest-level client wrapper for recursive ingestion. ```python theme={null} from docent.sdk.integrations import ( convert_inspect_directory_to_agent_runs, convert_inspect_eval_file_to_agent_runs, ) ``` ## Example To convert a single `.eval` file: ```python theme={null} from docent.sdk.integrations import convert_inspect_eval_file_to_agent_runs agent_runs = convert_inspect_eval_file_to_agent_runs("evals/my-run.eval") ``` To convert every `.eval` file under a directory: ```python theme={null} from docent.sdk.integrations import convert_inspect_directory_to_agent_runs agent_runs = convert_inspect_directory_to_agent_runs("/path/to/inspect/logs") ``` To recursively convert and upload in one step: ```python theme={null} from docent import Docent client = Docent() client.recursively_ingest_inspect_logs(collection_id, "/path/to/inspect/logs") ``` After conversion, upload normally: ```python theme={null} from docent import Docent from docent.sdk.integrations import convert_inspect_directory_to_agent_runs client = Docent() agent_runs = convert_inspect_directory_to_agent_runs("/path/to/inspect/logs") client.add_agent_runs(collection_id, agent_runs) ``` ## More on the conversion process Each Inspect sample becomes one Docent `AgentRun`. ### How Inspect data is mapped The converter: * reads archive header metadata such as task and model * converts each sample's messages with `parse_chat_message(...)` * normalizes sample scores into `agent_run.metadata["scores"]` * preserves raw score payloads in `agent_run.metadata["scoring_metadata"]` * includes sample-level fields such as `sample_id`, `epoch`, and `target` in run metadata * merges sample metadata on top of header metadata when both are present Because Inspect messages already fit Docent's chat schema closely, this conversion is usually low-friction. ### Recursive ingestion helper If you want to control uploads yourself, use `ingest_inspect_directory(...)`. It: * recursively finds `.eval` files * converts them lazily * uploads them in batches through your callback That lower-level helper is what `Docent.recursively_ingest_inspect_logs(...)` uses internally. # NeMo Gym Source: https://docs.transluce.org/ingestion/integrations/nemogym Ingest NeMo Gym rollouts into Docent NeMo Gym ([docs](https://docs.nvidia.com/nemo/gym/latest/index.html), [GitHub](https://github.com/NVIDIA-NeMo/Gym)) is a library for building reinforcement learning environments for large language models. We offer helpers that import NeMo Gym rollout exports into Docent. ## Use this when Use NeMo Gym ingestion when your source data is either: * one parsed rollout dictionary already loaded in Python * a JSONL export where each line is one NeMo Gym rollout object ## Main helpers * `convert_nemogym_rollout_to_agent_run(rollout)` converts one parsed rollout object into one `AgentRun`. * `convert_nemogym_jsonl_file_to_agent_runs(file_path)` reads a JSONL file and converts each line into one `AgentRun`. ```python theme={null} from docent.sdk.integrations import ( convert_nemogym_jsonl_file_to_agent_runs, convert_nemogym_rollout_to_agent_run, ) ``` ## Example To convert a JSONL export: ```python theme={null} from docent.sdk.integrations import convert_nemogym_jsonl_file_to_agent_runs agent_runs = convert_nemogym_jsonl_file_to_agent_runs("rollouts.jsonl") ``` To convert one rollout dictionary already loaded in memory: ```python theme={null} from docent.sdk.integrations import convert_nemogym_rollout_to_agent_run agent_run = convert_nemogym_rollout_to_agent_run(rollout) ``` After conversion, upload normally: ```python theme={null} from docent import Docent from docent.sdk.integrations import convert_nemogym_jsonl_file_to_agent_runs client = Docent() agent_runs = convert_nemogym_jsonl_file_to_agent_runs("rollouts.jsonl") client.add_agent_runs(collection_id, agent_runs) ``` ## More on the conversion process Each NeMo Gym rollout becomes one Docent `AgentRun`. At a high level, the converter: * turns the rollout input and output into a single Docent transcript * maps `developer` messages to Docent `system` messages * converts NeMo Gym function calls into Docent tool calls and tool messages * stores the rollout reward in `agent_run.metadata["scores"]["reward"]` * preserves extra request and response data under `agent_run.metadata["source"]` The converter is strict. It expects the rollout to include `responses_create_params.input`, `response.output`, `agent_ref.name`, `_ng_task_index`, and `_ng_rollout_index`. `responses_create_params.input` can be either: * a string, which Docent converts into a single user message * an array of structured input items If the rollout uses unsupported message shapes, unsupported content-part types, invalid tool-call wiring, or missing required fields, the converter raises `ConversionError` instead of trying to approximate the data. # Ingestion Quickstart Source: https://docs.transluce.org/ingestion/quickstart Upload your data to Docent This section will help you upload your data to Docent. We recommend using the Docent plugin with a coding agent to ingest logs after you run an evaluation. You can also [trace your agents](/ingestion/tracing) to capture data as they run. ## Instructions 1. **[Install the Docent plugin](/installation).** The `/docent` command below comes from the plugin. 2. **Read the [Data Models Overview](/concepts/overview)** to understand Docent's core abstractions (collections, agent runs, transcripts, metadata) so you can better instruct your coding agent on how you'd like your data organized. 3. **Point your coding agent at a directory containing your trajectories.** For best results, sort your trajectories by format before invoking the agent, and ingest one format at a time. ```text wrap theme={null} /docent Ingest the trajectories at ``` ## Notes * Find [best practices](/ingestion/agentic#best-practices) for ingesting your data in our agentic ingestion guide. * Using Inspect, Harbor, or NeMo-Gym? The [integration guides](/ingestion/integrations/inspect) handle those formats directly. * You can read more about the [Python SDK](/ingestion/sdk) that our plugin writes under the hood if you want to write a script by hand, understand what your agent produced, or correct an existing ingestion script. ## Next steps Walk through a sample analysis on Terminal-Bench data and learn the core workflows. # Ingest via SDK Source: https://docs.transluce.org/ingestion/sdk Ingest agent runs into Docent using the Python SDK ## Before you start We generally recommend using the [`/docent` plugin](/installation) to [ingest your traces](/ingestion/quickstart). The plugin writes the SDK script for you from your existing logs. Use this page if you want to debug what `/docent` produced, have unusual data formats your coding agent can't infer, or need fine-grained control. If you already have an Inspect `.eval` file, the fastest path is [drag-and-drop upload](/ingestion/integrations/inspect). Otherwise, follow the steps below. ## Setup Install the SDK: ```bash theme={null} uv add docent ``` Go to the [API keys page](https://docent.transluce.org/settings/api-keys), create a key, and instantiate a client object with that key: ```python theme={null} import os from docent import Docent client = Docent( api_key=os.getenv("DOCENT_API_KEY"), # is default and can be omitted # Uncomment and adjust these if you're self-hosting # server_url="http://localhost:8889", # web_url="http://localhost:3001", ) ``` ## Create a collection ```python theme={null} collection_id = client.create_collection( name="sample collection", description="example that comes with the Docent repo", ) ``` ## Convert your data There are three end-to-end examples below; pick whichever matches your data. If your messages are already in OpenAI chat format (`{"role": ..., "content": ..., "tool_calls": ...}`), use `parse_chat_message` to convert each one into a `ChatMessage`. All three examples below use this helper. Say we have three simple agent runs. ```python theme={null} transcript_1 = [ { "role": "user", "content": "What's the weather like in New York today?" }, { "role": "assistant", "content": "The weather in New York today is mostly sunny with a high of 75°F (24°C)." } ] metadata_1 = {"model": "gpt-3.5-turbo", "agent_scaffold": "foo", "hallucinated": True} transcript_2 = [ { "role": "user", "content": "What's the weather like in San Francisco today?" }, { "role": "assistant", "content": "The weather in San Francisco today is mostly cloudy with a high of 65°F (18°C)." } ] metadata_2 = {"model": "gpt-3.5-turbo", "agent_scaffold": "foo", "hallucinated": True} transcript_3 = [ { "role": "user", "content": "What's the weather like in Paris today?" }, { "role": "assistant", "content": "I'm sorry, I don't know because I don't have access to weather tools." } ] metadata_3 = {"model": "gpt-3.5-turbo", "agent_scaffold": "bar", "hallucinated": False} transcripts = [transcript_1, transcript_2, transcript_3] metadata = [metadata_1, metadata_2, metadata_3] ``` We need to convert each input into an [AgentRun](/concepts/agent-run) object, which holds Transcript objects where each message needs to be a [ChatMessage](/concepts/chat-messages). We could construct the messages manually, but it's easier to use the `parse_chat_message` function, since the raw dicts already conform to the expected schema. ```python theme={null} from docent.data_models.chat import parse_chat_message from docent.data_models import Transcript parsed_transcripts = [ Transcript(messages=[parse_chat_message(msg) for msg in transcript]) for transcript in transcripts ] ``` Now we can create the [AgentRun](/concepts/agent-run) objects. ```python theme={null} from docent.data_models import AgentRun agent_runs = [ AgentRun( transcripts=[t], metadata={ "model": m["model"], "agent_scaffold": m["agent_scaffold"], "scores": {"hallucinated": m["hallucinated"]}, } ) for t, m in zip(parsed_transcripts, metadata) ] ``` For a more complex case that involves tool calls, Docent ships with a sample τ-bench log file, generated by running Sonnet 3.5 (new) on *one* task from the τ-bench-airline dataset. To inspect the log, we can load it as a dictionary. ```python theme={null} from docent.samples import get_tau_bench_airline_fpath import json with open(get_tau_bench_airline_fpath(), "r") as f: tb_log = json.load(f) print(tb_log) ``` Next, we write a function that parses the dict into an [AgentRun](/concepts/agent-run) object, complete with metadata. Most of the effort is in converting the raw tool calls into the expected format. ```python theme={null} from docent.data_models import AgentRun, Transcript from docent.data_models.chat import ChatMessage, ToolCall, parse_chat_message def load_tau_bench_log(data: dict[str, Any]) -> AgentRun: traj, info, reward, task_id = data["traj"], data["info"], data["reward"], data["task_id"] messages: list[ChatMessage] = [] for msg in traj: # Extract raw message data role = msg.get("role") content = msg.get("content", "") raw_tool_calls = msg.get("tool_calls") tool_call_id = msg.get("tool_call_id") # Create a message data dictionary message_data = { "role": role, "content": content, } # For tool messages, include the tool name if role == "tool": message_data["name"] = msg.get("name") message_data["tool_call_id"] = tool_call_id # For assistant messages, include tool calls if present if role == "assistant" and raw_tool_calls: # Convert tool calls to the expected format parsed_tool_calls: list[ToolCall] = [] for tc in raw_tool_calls: tool_call = ToolCall( id=tc.get("id"), function=tc.get("function", {}).get("name"), arguments=tc.get("function", {}).get("arguments", {}), type="function", parse_error=None, ) parsed_tool_calls.append(tool_call) message_data["tool_calls"] = parsed_tool_calls # Parse the message into the appropriate type chat_message = parse_chat_message(message_data) messages.append(chat_message) # Extract metadata from the sample task_id = info["task"]["user_id"] scores = {"reward": round(reward, 3)} # Build metadata metadata = { "benchmark_id": task_id, "task_id": task_id, "model": "sonnet-35-new", "scores": scores, "additional_metadata": info, "scoring_metadata": info["reward_info"], } # Create the transcript and wrap in AgentRun transcript = Transcript( messages=messages, metadata=metadata, ) agent_run = AgentRun( transcripts=[transcript], metadata=metadata, ) return agent_run ``` Let's just load the single run in, and print its string representation. ```python theme={null} agent_runs = [load_tau_bench_log(tb_log)] print(agent_runs[0].text) ``` You can upload Inspect files directly into Docent! After making a collection on the website, just click "Add Data" and then "Upload Inspect Log". Alternatively, you can also add Inspect logs via the SDK; keep reading for an example of how to do this. Our [ChatMessage](/concepts/chat-messages) schema is compatible with Inspect AI's format (as of `inspect-ai==0.3.93`), which means you can directly use the `parse_chat_message` function to parse Inspect messages. Docent ships with a sample Inspect log file, generated by running GPT-4o on a subset of the Intercode CTF benchmark. First install [Inspect](https://inspect.aisi.org.uk/): ```bash uv theme={null} uv add inspect-ai ``` ```bash pip theme={null} pip install inspect-ai ``` Inspect provides a library function to read the log; we can convert it to a dictionary for easier viewing. ```python theme={null} from docent.samples import get_inspect_fpath from inspect_ai.log import read_eval_log from pydantic_core import to_jsonable_python ctf_log = read_eval_log(get_inspect_fpath()) ctf_log_dict = to_jsonable_python(ctf_log) ``` Now we can write a function that takes the Inspect log and converts it into an [AgentRun](/concepts/agent-run) object. ```python theme={null} from inspect_ai.log import EvalLog from docent.data_models import AgentRun, Transcript from docent.data_models.chat import parse_chat_message def load_inspect_log(log: EvalLog) -> list[AgentRun]: if log.samples is None: return [] agent_runs: list[AgentRun] = [] for s in log.samples: # Extract sample_id from the sample ID sample_id = s.id epoch_id = s.epoch # Gather scores scores: dict[str, int | float | bool] = {} # Evaluate correctness (for this CTF benchmark) if s.scores and "includes" in s.scores: scores["correct"] = s.scores["includes"].value == "C" # Set metadata metadata = { "task_id": log.eval.task, "sample_id": str(sample_id), "epoch_id": epoch_id, "model": log.eval.model, "scores": scores, "additional_metadata": s.metadata, "scoring_metadata": s.scores, } # Create transcript agent_runs.append( AgentRun( transcripts=[ Transcript( messages=[parse_chat_message(m.model_dump()) for m in s.messages] ) ], metadata=metadata, ) ) return agent_runs ``` Let's check on our loaded run: ```python theme={null} agent_runs = load_inspect_log(ctf_log) print(agent_runs[0].text) ``` ## Upload the runs ```python theme={null} client.add_agent_runs(collection_id, agent_runs) ``` If you navigate to the frontend URL printed by `client.create_collection(...)`, you should see the run available for viewing. Docent assigns the `id` field on `AgentRun`, `Transcript`, and `TranscriptGroup` automatically. You cannot set these IDs yourself — reassigning `id` after construction raises a `ValueError`, and the upload path rejects payloads whose IDs were set by the caller (for example, runs round-tripped through `client.get_agent_run(...)` or loaded from a JSON dump). To wire references between objects in the same upload, construct the parent first and read its assigned `id`: ```python theme={null} group = TranscriptGroup(agent_run_id=run.id) transcript = Transcript(transcript_group_id=group.id, messages=[...]) ``` To re-upload runs that already have IDs, regenerate them first with the clone helper: ```python theme={null} from docent import clone_agent_runs_with_random_ids agent_runs = clone_agent_runs_with_random_ids(agent_runs) client.add_agent_runs(collection_id, agent_runs) ``` A single-run variant, `clone_agent_run_with_random_ids`, is also exported from `docent`. ## Tips and tricks ### Including sufficient context Docent can only catch issues that are evident from the context it has about your evaluation. For example: * If you're looking to catch issues with solution labels, you should provide the exact label in the metadata, not just the agent's score. * For software engineering tasks, if you want to know *why* agents failed, you should include information about what tests were run and their traceback/execution logs. # Tracing Source: https://docs.transluce.org/ingestion/tracing Automatically capture LLM interactions with Docent's tracing system Docent provides a comprehensive tracing system that automatically captures LLM interactions, organizes them, and enables detailed analysis of your AI applications. ## Overview The Docent tracing system allows you to: * Automatically instrument LLM SDK calls, including OpenAI and Anthropic. If you use OpenRouter through the OpenAI SDK, tracing should also work without extra Docent setup. * Organize code into logical agent runs with metadata and scores * Track chat conversations and tool calls * Analyze performance and quality metrics * Resume agent runs across different parts of your codebase ## Getting Started ### 1. Installation Docent tracing is included with the main Docent SDK package: ```bash theme={null} pip install docent ``` ### 2. API Key Setup You'll need a Docent API key to send traces to the Docent backend. You can get one by: 1. Signing up or logging in at [Docent](https://docent.transluce.org) 2. On your dashboard, click on your account icon in the top right 3. Select "API Keys" 4. Generate an API key Set your API key as an environment variable: ```bash theme={null} export DOCENT_API_KEY="your-api-key-here" ``` Or pass it directly to the initialization function. ### 3. Initialize Tracing The primary entry point for setting up Docent tracing is `initialize_tracing()`: ```python theme={null} from docent.trace import initialize_tracing # Basic initialization initialize_tracing("my-application") # With custom configuration initialize_tracing( collection_name="my-application", endpoint="https://docent.transluce.org/rest/telemetry", # Optional, uses default if not provided api_key="your-api-key", # Optional, uses env var if not provided ) # Add new agent runs to an existing collection initialize_tracing( collection_id="c49ef42c-7493-4af3-84d5-ac2b67556005", # Your collection's ID from the dashboard ) ``` **Parameters:** * `collection_name`: Name for your application/collection * `endpoint`: Optional OTLP endpoint URL (defaults to Docent's hosted service) * `api_key`: Optional API key (uses `DOCENT_API_KEY` environment variable if not provided) * `enable_console_export`: Whether to also export traces to console for debugging (default: False) ## Four Levels of Organization Docent organizes your traces into four hierarchical levels: ### 1. Collection A **collection** is the top-level organization unit. It represents a set of agent runs that you want to analyze together. ### 2. Agent Run An **agent run** typically represents a single execution of your entire system. It could include: * Multiple LLM calls * Tool calls and responses * Associated metadata and scores * One or more chat sessions (transcripts) ### 3. Transcript Group A **transcript group** is a logical grouping of related transcripts. Transcript groups are entirely optional. It allows you to organize transcripts that are conceptually related, such as: * Different phases of a multi-step process * Related experiments or iterations * Multiple conversations with the same user ### 4. Transcript A **transcript** is essentially a chat session - a sequence of messages with an LLM. Transcripts are automatically created by detecting consistent chat messages from within LLM calls that are tagged to the same agent run (or Transcript Group if you use them). ## Creating Agent Runs ### Using the Decorator The simplest way to create an agent run is using the `@agent_run` decorator: ```python theme={null} from docent.trace import agent_run @agent_run def analyze_document(document_text: str): # This entire function will be wrapped in an agent run response = client.chat.completions.create( model="gpt-5", messages=[{"role": "user", "content": f"Analyze this document: {document_text}"}] ) return response.choices[0].message.content ``` ### Using Context Managers For more control, use the context manager approach: ```python theme={null} from docent.trace import agent_run_context def process_user_query(query: str): with agent_run_context(): # Your agent code here response = client.chat.completions.create( model="gpt-5", messages=[{"role": "user", "content": query}] ) return response.choices[0].message.content ``` ### Async Support Both decorators and context managers work with async code: ```python theme={null} from docent.trace import agent_run_context async def async_agent_function(): async with agent_run_context(): # Async agent code here response = await client.chat.completions.create( model="gpt-5", messages=[{"role": "user", "content": "Hello"}] ) return response.choices[0].message.content ``` ## Attaching Scores to Agent Runs You can attach scores to agent runs to track performance metrics and quality indicators. It will automatically be associated with the agent run that is currently in context. ```python theme={null} from docent.trace import agent_run_score def evaluate_response(response: str, expected: str): # Calculate some score accuracy = calculate_accuracy(response, expected) # Attach the score to the current agent run agent_run_score("accuracy", accuracy) # You can attach multiple scores agent_run_score("response_length", len(response)) agent_run_score("processing_time", 1.23) agent_run_score("user_satisfaction", 4.5) return accuracy @agent_run def run_system(): # ... evaluate_response(response, expected) ``` ## Attaching Metadata to Agent Runs You can attach metadata to agent runs to provide context and enable filtering: ```python theme={null} from docent.trace import agent_run_context, agent_run_metadata # Using context manager with metadata with agent_run_context( metadata={ "user_id": "user_123", "session_id": "session_456", "temperature": 0.7 } ): # Your agent code here pass # Using decorator with metadata @agent_run(metadata={"task_type": "document_analysis", "priority": "high"}) def analyze_document_with_metadata(document_text: str): # Function code here pass # Adding metadata during execution def process_with_dynamic_metadata(): with agent_run_context() as (agent_run_id, transcript_id): # Do some work result = process_data() # Add metadata based on results agent_run_metadata({ "processing": { "input_size": len(input_data), "output_size": len(result), "success": True } }) return result ``` ## Attaching Metadata to Collections Use collection metadata for information shared by the whole collection, such as dataset version, environment, or common experiment configuration. ```python theme={null} from docent.trace import collection_metadata, initialize_tracing initialize_tracing("my-application") collection_metadata( { "environment": "prod", "dataset": "customer_support_v2", "config": { "model": "gpt-5", "prompt_version": "v3", }, } ) ``` Unlike `agent_run_metadata()`, `collection_metadata()` does not require an active agent run. It updates metadata on the collection itself and is applied immediately. For SDK-based read, update, and delete workflows, see [Metadata](/concepts/metadata). ## Working with Transcript Groups Transcript groups allow you to organize related transcripts into logical hierarchies. This is useful for organizing conversations that span multiple interactions or for grouping related experiments. ### Creating Transcript Groups #### Using the Decorator ```python theme={null} from docent.trace import transcript_group @transcript_group(name="ask_all_agents", description="send the query to all agents") def ask_all_agents(user_id: str): # This function will be wrapped in a transcript group context # All transcripts created within this function will be grouped together pass ``` #### Using Context Managers ```python theme={null} from docent.trace import transcript_group_context def process_user_session(user_id: str): with transcript_group_context( name=f"user_session_{user_id}", description="Complete user interaction session" ) as transcript_group_id: # All transcripts created within this context will be grouped # You can access the transcript_group_id if needed pass ``` ### Hierarchical Transcript Groups You can create nested transcript groups to represent hierarchical relationships: ```python theme={null} from docent.trace import transcript_group_context def run_experiment_batch(): with transcript_group_context(name="experiment_batch"): for experiment_id in range(3): with transcript_group_context(name=f"experiment_{experiment_id}"): run_single_experiment(experiment_id) ``` ## Automatic Transcript Creation Docent automatically creates transcripts by detecting consistent chat completions. When you make LLM calls within an agent run, they're automatically grouped into logical conversation threads. ## Advanced Agent Run Usage ### Resuming Agent Runs You can resume agent runs across different parts of your codebase by passing the `agent_run_id`. This is useful for connecting related work that happens in different modules or at different times. #### Resuming Using Context Managers With context managers, you can explicitly pass and resume agent runs: ```python theme={null} from docent import agent_run_context def run_agent(state): with agent_run_context() as (agent_run_id, _): # Agent logic here response = client.chat.completions.create( model="gpt-5", messages=[{"role": "user", "content": user_input}] ) # save the agent_run_id state.metadata["agent_run_id"] = agent_run_id return response.choices[0].message.content def evaluate_agent(state): # Resume the same agent run by passing the agent_run_id with agent_run_context(agent_run_id=state.metadata.agent_run_id): # This continues the same agent run accuracy = calculate_accuracy(response, expected_answer) agent_run_score("evaluation_accuracy", accuracy) # Add evaluation metadata agent_run_metadata({ "evaluation": { "method": "human_review", "reviewer": "expert_1", "timestamp": "2024-01-15T10:30:00Z" } }) return accuracy ``` #### Resuming Using Decorators With decorators, you can access the agent run ID from the function's attributes: ```python theme={null} from docent import agent_run @agent_run def run_agent(user_input: str): # Agent logic here response = client.chat.completions.create( model="gpt-5", messages=[{"role": "user", "content": user_input}] ) # The agent run ID is available as an attribute agent_run_id = run_agent.docent.agent_run_id # Pass to evaluation function evaluate_agent_response(agent_run_id, response.choices[0].message.content) return response.choices[0].message.content ``` # Installation Source: https://docs.transluce.org/installation Install the Docent SDK and coding agent plugin Before starting, [create a Docent account](https://docent.transluce.org/signup), install [`uv`](https://docs.astral.sh/uv/getting-started/installation/), and install either [Claude Code](https://claude.com/claude-code) or [Codex](https://help.openai.com/en/articles/11096431). Open your project directory. Replace `your-project` with its path. ```shell theme={null} cd your-project ``` Run setup. ```shell theme={null} uvx docent@latest setup ``` This installs or updates the SDK and agent plugin, then helps you save and validate a Docent API key. If this is not already a Python project, setup explains what it will create and asks before running `uv init`. When setup finishes, restart Claude Code or start a new Codex session. Go to your project directory, replacing `your-project` with its path: ```shell theme={null} cd your-project ``` Then initialize Python project metadata when needed and add Docent: ```shell theme={null} [ -f pyproject.toml ] || uv init --bare uv add docent ``` If Docent is already declared in the project, update it within the project's existing version constraint: ```shell theme={null} uv sync --upgrade-package docent ``` ```shell theme={null} claude plugin marketplace add TransluceAI/claude-code-plugins claude plugin install docent@transluce-plugins ``` To update an existing installation: ```shell theme={null} claude plugin marketplace update transluce-plugins claude plugin update docent@transluce-plugins ``` Restart Claude Code after installation or update. ```shell theme={null} codex plugin marketplace add TransluceAI/codex-plugins codex plugin add docent@transluce-plugins ``` To update an existing installation, refresh the marketplace. Codex refreshes installed plugins from the new snapshot automatically: ```shell theme={null} codex plugin marketplace upgrade transluce-plugins ``` Start a new Codex session afterward. Download the skill files directly and pass the file contents directly in your prompt context: * [Claude Code Docent SKILL.md](https://github.com/TransluceAI/claude-code-plugins/blob/main/plugins/docent/skills/docent/SKILL.md) * [Codex Docent SKILL.md](https://github.com/TransluceAI/codex-plugins/blob/main/plugins/docent/skills/docent/SKILL.md) ```shell theme={null} mkdir -p ~/.docent cat <<'EOF' > ~/.docent/docent.env DOCENT_API_KEY= DOCENT_DOMAIN=docent.transluce.org EOF chmod 600 ~/.docent/docent.env ``` Project-level `docent.env` files remain available when a repository needs to override the global configuration. After setup, [run your first analysis](/analysis/quickstart). # Welcome to Docent Source: https://docs.transluce.org/introduction Understand and improve your agents Docent is a behavior analysis platform for agents. After you run an evaluation, Docent analyzes your traces and explains what failure modes or environment issues are driving your team's evaluation results. Teams use Docent to: * **Iterate on scaffolds.** Docent returns actionable insights to inform prompt tuning, tool instructions, or orchestration logic. * **Post-train models.** Compare behavior across checkpoints or training steps to identify what's driving shifts in eval results. * **Build better benchmarks.** Catch reward hacking, evaluation awareness, broken environments, and ambiguous task specifications. Read about how Docent helped [align Claude 4](https://www-cdn.anthropic.com/4263b940cabb546aa0e3283f35b686f4f3b2ff47.pdf) and [debug a regression between two Codex checkpoints on Terminal-Bench](https://transluce.org/docent/blog/terminal-bench). ## Get started Set up the SDK, coding-agent plugin, and API key. Join our Slack community to ask questions and chat with our team. # Rubrics and Judges Source: https://docs.transluce.org/legacy/rubrics Define evaluation criteria and run LLM-based judges on agent runs We no longer recommend authoring behavior rubrics by hand. The [Docent plugin](/installation) generates [Reading steps](/analysis/reading-steps) inside an [Analysis Plan](/analysis/analysis-plans) for you. This page is kept for users with existing rubrics. Docent helps you create and optimize LLM-based judges that evaluate agent runs against your criteria. A **rubric** is a configuration object that defines how a judge works, and a **judge** is a callable that evaluates an [AgentRun](/concepts/agent-run) using that rubric configuration. You can create and manage judges directly in the Docent web UI. This page focuses on the underlying data model and how to use the SDK to create and run judges programmatically. ## The Data Model A rubric defines the complete configuration for a judge. Here's an example: ```python theme={null} from docent.judges.types import Rubric, OutputParsingMode, PromptTemplateMessage from docent._llm_util.providers.preference_types import ModelOption rubric = Rubric( prompt_templates=[ PromptTemplateMessage( role="user", content=""" Evaluate this agent run against the rubric. Rubric: {rubric} Agent run: {agent_run} Output your evaluation as JSON in ... tags. Schema: {output_schema} """, ), ], rubric_text=""" Evaluate whether the agent successfully completed the user's request. Decision procedure: 1. Identify what the user asked for 2. Check if the agent's final response addresses the request 3. Verify the response is accurate and complete """, output_schema={ "type": "object", "properties": { "label": {"type": "string", "enum": ["pass", "fail"]}, "explanation": {"type": "string", "citations": True}, }, "required": ["label", "explanation"], "additionalProperties": False, }, judge_model=ModelOption( provider="openai", model_name="gpt-4o", ), output_parsing_mode=OutputParsingMode.XML_KEY, response_xml_key="response", ) ``` Let's break down each of the key configuration options shown above. ### prompt\_templates A list of messages that form the judge's prompt. Templates must collectively include all three variables: `{agent_run}`, `{rubric}`, and `{output_schema}`. Variables can be distributed across multiple messages in the list. ```python theme={null} from docent.judges.types import PromptTemplateMessage rubric = Rubric( rubric_text="...", output_schema={...}, prompt_templates=[ PromptTemplateMessage( role="user", content=""" Evaluate this agent run against the rubric. Rubric: {rubric} Agent run: {agent_run} Output your evaluation as JSON in ... tags. Schema: {output_schema} """, ), ], ) ``` **Template Variables** Prompt templates must collectively include all three of these variables, which are automatically substituted: | Variable | Description | | ------------------------------ | ----------------------------------------------------------------------------------------------------- | | `{agent_run}` | The rendered transcript of the agent run being evaluated | | `{rubric}` | The `rubric_text` field content | | `{output_schema}` | JSON-formatted output schema | | `{output_format_instructions}` | Optional. Format-specific instructions (JSON or YAML) selected by the rubric's `output_format` field. | Validation will fail if any required variable is missing or if templates contain other undefined variables. `{output_format_instructions}` is optional — include it only if you want the prompt to surface format-specific guidance to the judge. ### rubric\_text The core evaluation criteria that the judge follows. This text is substituted into the `{rubric}` template variable in the prompt. Write clear decision procedures that the judge can follow step-by-step. Be specific about what constitutes success or failure. ### output\_schema A JSON schema defining the structure of the judge's output. ```python theme={null} output_schema={ "type": "object", "properties": { "label": {"type": "string", "enum": ["pass", "fail"]}, "score": {"type": "integer", "minimum": 1, "maximum": 5}, "explanation": {"type": "string", "citations": True}, }, "required": ["label", "score", "explanation"], "additionalProperties": False, } ``` **Metaschema Rules** We will post a link to the full JSON metadata schema soon. In the meantime, judge output schemas must generally follow these rules: * Root must be `type: "object"` with `properties` * `additionalProperties` is optional, but if present must be `false` * Supported types: `string`, `integer`, `number`, `boolean`, `array`, `object` (recursive objects are allowed) * Arrays require an `items` schema; objects require `properties` * Special: `"citations": true` on string fields enables transcript references * `anyOf`, `oneOf`, `allOf` are not supported **Labels share this meta-schema.** Label sets, which define structured annotations for agent runs, use the same meta-schema validation. Any schema you define for a label set must follow these same rules. **Example with Citations** ```python theme={null} output_schema={ "type": "object", "properties": { "issues": { "type": "array", "items": { "type": "object", "properties": { "description": {"type": "string", "citations": True}, "severity": {"type": "string", "enum": ["low", "medium", "high"]}, }, "required": ["description", "severity"], "additionalProperties": False, }, }, "overall_score": {"type": "integer", "minimum": 1, "maximum": 10}, }, "required": ["issues", "overall_score"], "additionalProperties": False, } ``` When `"citations": true` is set on a string field, the judge must include citations to specific parts of the transcript in its response. The Docent web UI automatically parses and links these citations. SDK users must manually convert results using `JudgeResultWithCitations.from_judge_result()` to resolve citation references. ### judge\_model Specifies which LLM to use for evaluation. Uses `ModelOption` with provider, model name, and optional reasoning effort. ```python theme={null} from docent._llm_util.providers.preference_types import ModelOption rubric = Rubric( rubric_text="...", output_schema={...}, judge_model=ModelOption( provider="openai", model_name="gpt-5", reasoning_effort="high", ), ) ``` **Supported Providers** | Provider String | Description | | --------------- | ----------- | | `openai` | OpenAI | | `anthropic` | Anthropic | | `google` | Google | | `openrouter` | OpenRouter | **Model Names** Any model from the supported providers can be used as a judge. Use the exact model string that the provider uses: * OpenAI: `gpt-4o`, `gpt-4o-mini`, `o1`, `o3-mini`, etc. * Anthropic: `claude-sonnet-4-5`, `claude-sonnet-4-20250514`, etc. * Google: `gemini-2.0-flash`, `gemini-1.5-pro`, etc. * OpenRouter: Uses a different format with the provider prefix, e.g., `anthropic/claude-3-opus`, `openai/gpt-4o` **Reasoning Effort** Some reasoning models support a `reasoning_effort` parameter that controls how much computation the model uses. Typical values are `minimal`, `low`, `medium`, and `high`. Not all models support this parameter—it is primarily available for OpenAI's reasoning models (o1, o3-mini, etc.). ### output\_parsing\_mode Defines how the LLM output is parsed: * `XML_KEY` (default): Extract JSON from within XML tags (e.g., `...`). When using this mode, at least one prompt template must contain the XML tag `<{response_xml_key}>` (e.g., `` by default). * `CONSTRAINED_DECODING`: Parse entire output as JSON (uses structured output). Supported by OpenAI, OpenRouter, and Anthropic. Not yet implemented for Google (will raise `NotImplementedError`). ```python theme={null} from docent.judges.types import OutputParsingMode rubric = Rubric( rubric_text="...", output_schema={...}, output_parsing_mode=OutputParsingMode.CONSTRAINED_DECODING, ) ``` ### response\_xml\_key When using `XML_KEY` parsing mode, specifies the tag name to extract the response from. Defaults to `"response"`. Note: At least one prompt template must contain the corresponding XML tag (e.g., `...` if using `response_xml_key="answer"`). ```python theme={null} rubric = Rubric( rubric_text="...", output_schema={...}, response_xml_key="answer", # Extract from ... ) ``` ### output\_format Selects the serialization format the judge is instructed to emit and that the SDK parses. Supported values are `"yaml"` (default for new rubrics) and `"json"`. ```python theme={null} rubric = Rubric( rubric_text="...", output_schema={...}, output_format="yaml", ) ``` The `{output_format_instructions}` template variable, when present in a prompt template, is substituted with format-specific guidance derived from this field (for example, instructions on escaping for JSON or `yaml.safe_load`-compatible output for YAML). Output is still validated against `output_schema` regardless of format. Rubrics created before the `output_format` field existed continue to behave as if `output_format="json"` was set, preserving backward compatibility. ## SDK Methods ### create\_rubric() Upload a rubric to a collection. Returns the rubric ID. ```python theme={null} rubric_id = client.create_rubric(collection_id, rubric) ``` ### start\_rubric\_eval\_job() Start a rubric evaluation job for agent runs in a collection. ```python theme={null} job_id = client.start_rubric_eval_job( collection_id, rubric_id, max_agent_runs=100, n_rollouts_per_input=1, ) ``` Use the method below to track the job progress and retrieve the results. ### get\_rubric\_run\_state() Retrieve the current rubric evaluation results and job progress. This method does not start evaluation; use `start_rubric_eval_job()` first. ```python theme={null} import time job_id = client.start_rubric_eval_job(collection_id, rubric_id) while True: run_state = client.get_rubric_run_state(collection_id, rubric_id) if run_state["job_id"] is None: break time.sleep(2) results = run_state["results"] print(f"Retrieved {len(results)} evaluated agent runs") ``` The response includes the current grouped judge results in `results`, plus progress metadata such as `job_id`, `job_status`, `total_results_needed`, and `current_results_count` while a job is still running. ### get\_rubric() Retrieve a rubric configuration object by ID. Optionally specify a version. ```python theme={null} rubric = client.get_rubric(collection_id, rubric_id) rubric = client.get_rubric(collection_id, rubric_id, version=2) ``` ### get\_judge() Get a callable `BaseJudge` instance for running evaluations. Optionally specify a version. ```python theme={null} judge = client.get_judge(collection_id, rubric_id) # Run the judge on an agent run (async) result = await judge(agent_run) ``` ### list\_rubrics() List all rubrics in a collection. ```python theme={null} rubrics = client.list_rubrics(collection_id) ``` ## Running the Judge The `build_judge` function creates an async callable that wraps LLM providers. It takes a `Rubric` configuration and an LLM service, and returns a judge you can call directly on any `AgentRun`. ```python theme={null} import asyncio from docent._llm_util.llm_svc import BaseLLMService from docent.judges.impl import build_judge from docent.judges.types import Rubric, ResultType from docent.data_models.agent_run import AgentRun from docent.data_models.transcript import Transcript from docent.data_models.chat.message import UserMessage, AssistantMessage # Define the rubric rubric = Rubric( rubric_text=""" Evaluate whether the agent provided a helpful and accurate response. Decision procedure: 1. Check if the agent understood the user's question 2. Verify the response directly addresses the question 3. Assess accuracy of any factual claims """, output_schema={ "type": "object", "properties": { "label": {"type": "string", "enum": ["helpful", "not helpful"]}, "explanation": {"type": "string", "citations": True}, }, "required": ["label", "explanation"], "additionalProperties": False, }, ) # Create the LLM service (reads API keys from environment variables) llm_svc = BaseLLMService() # Build the judge judge = build_judge(rubric, llm_svc) # Create an agent run to evaluate agent_run = AgentRun( transcripts=[ Transcript( messages=[ UserMessage(content="What is the capital of France?"), AssistantMessage(content="The capital of France is Paris."), ] ) ] ) # Run the judge (async) async def evaluate(): result = await judge(agent_run) if result.result_type == ResultType.DIRECT_RESULT: print(f"Label: {result.output['label']}") print(f"Explanation: {result.output['explanation']}") else: print(f"Judge failed: {result.result_metadata}") asyncio.run(evaluate()) ``` The `BaseLLMService` reads API keys from environment variables depending on the model provider: * OpenAI: `OPENAI_API_KEY` * Anthropic: `ANTHROPIC_API_KEY` * Google: `GOOGLE_API_KEY` * OpenRouter: `OPENROUTER_API_KEY` ### JudgeResult The judge returns a `JudgeResult` object with these fields: | Field | Type | Description | | ----------------- | ------------------------ | ----------------------------------------------------------- | | `id` | `str` | Unique identifier for this result | | `agent_run_id` | `str` | ID of the evaluated agent run | | `rubric_id` | `str` | ID of the rubric used | | `rubric_version` | `int` | Version of the rubric | | `output` | `dict[str, Any]` | Parsed output matching your `output_schema` | | `result_metadata` | `dict[str, Any] \| None` | Additional metadata (contains errors on failure), or `None` | | `result_type` | `ResultType` | `DIRECT_RESULT` on success, `FAILURE` on error | # Rubric Refinement Source: https://docs.transluce.org/legacy/rubrics/refinement Quantifying behaviors that require judgment to detect We no longer recommend rubric refinement as a primary workflow. The [Docent plugin](/installation) generates [Reading steps](/analysis/reading-steps) inside an [Analysis Plan](/analysis/analysis-plans), which you review and edit directly. This page is kept for users with existing rubrics. ## Overview The rubric refinement agent helps you turn a high-level behavioral description into a precise rubric that an LLM judge can apply consistently at scale. This helps overcome several challenges when specifying a rubric: * Behavioral concepts that are easy to recognize ("cheating," "sycophancy”) can be hard to describe precisely * LLM judges interpret rubrics literally and inconsistently, often latching onto phrasings in ways you didn't intend * Edge cases may be genuinely ambiguous, even from a human's perspective * Understanding the target behavior in detail can be difficult before you evaluate specific examples The refinement agent addresses this through collaborative specification writing: you describe what you want to measure, and it iterates with you until the rubric captures your actual intent. **Accessing the refinement agent:** When you search using "Direct Search" or "Guided Search," the refinement agent appears automatically. ## When to use refinement **You want to measure the prevalence of a fuzzy behavior.** You want to measure something like "cheating" or "rambling," but the concept is ambiguous. The refinement agent proposes an initial rubric, asks clarifying questions, and surfaces potential ambiguities for you to review. **You want to debug an existing rubric.** Your rubric produces wrong results on specific examples. You can recognize the issue, but how to revise the rubric is not obvious. The refinement agent takes your labeled disagreements as feedback and proposes rewrites that address those failure modes. **You have existing labels.** You have already annotated transcripts with your own taxonomy. The refinement agent extracts patterns from your labels, handles inconsistencies, and proposes rubrics that capture your current intent. **You want to explore.** You suspect something is wrong with your agent but lack a specific hypothesis. The refinement agent can quickly generate an exploratory rubric informed by a sample of your transcripts, helping surface behaviors worth investigating. ## Accessing refinement Open your collection and click the **Rubrics** icon in the left sidebar. In the search bar, write a natural-language description of your target behavior — for example, *"Cases where the agent repeatedly calls a missing utility."* You don't need to be precise; the refinement agent will help you sharpen it. Click **Guided Search** to start the refinement agent. It will sample transcripts from your collection and propose an initial rubric. Starting a guided search: navigate to Rubrics, describe the behavior, and click Guided Search Once the agent starts, you'll see two panels: * **Left panel: the generated rubric.** This is a fully operationalized rubric with a decision procedure the LLM judge will follow. You can review it, and the agent will update it as the conversation progresses. * **Right panel: the refinement chat.** The agent explains how it interpreted your description, surfaces edge cases, and asks clarifying questions. Reply in natural language to steer the rubric toward your intent. The refinement view: a generated rubric on the left and a guided chat on the right Keep chatting until the rubric captures your target behavior or the refinement agent indicates it is confident in the specification. When you're satisfied, click **Run rubric...** to apply it across your collection. To quantify how many runs display your target behavior, add a filter for rubric matches. Click the Filter button. Select `rubric.[your_rubric_id].label` as the field to filter over, and select `match` as the target value. If you created a rubric with a different output schema than the default `match` or `no match`, you may need to filter over a different field. Buttons for filtering by results flagged as a match You can see the number of matches next to the filter button. Displayed number of matches # Search and Clustering Source: https://docs.transluce.org/legacy/search-and-clustering Search for specific behaviors in agent runs and cluster results We no longer recommend search and clustering as a primary workflow. Use the [Docent plugin](/installation) — your coding agent generates an [Analysis Plan](/analysis/analysis-plans) with reduce-style [Reading steps](/analysis/reading-steps) for clustering and synthesis. This page is kept for users with existing saved searches. Docent allows you to search for specific behaviors in agent runs. We refer to search queries as *rubrics*, which specify what kinds of results you are looking for. Your rubrics can be arbitrarily complex, since we use frontier-level language models to evaluate them. ## Walkthrough Let's check for issues with the agent scaffolding that might have caused spurious failures. First, we filter to runs where the agent failed, then search for `potential issues with the environment the agent is operating in`: We can then cluster the results and see what the most common issues are: ## Sharing results You can open access permissions to share these results with anyone: You can also link to specific parts of the agent run: ## Tips for using search * If you don't precisely know what you're looking for, start with a general rubric (e.g., "cases of cheating" or "types of environment issues"). Then, based on initial results, refine your rubric. * If you *do* know what you're looking for, feel free to provide lots of detail in your rubric; that's why the text box is so large. * Use appropriate metadata filters to narrow the scope of your search. ## Customizing the judge output schema Judges produce data in a JSON format. Each judge has an associated schema that is used to prompt the language model and validate its output. The default schema is: ```json theme={null} { "type": "object", "properties": { "explanation": { "type": "string", "citations": true }, "label": { "type": "string", "enum": ["match", "no match"] }, } } ``` You can edit this schema in the rubric editor. For example, if you want the judge to classify transcripts into one of three categories, you can put those categories in `properties.label.enum`. See the [JSON Schema documentation](https://json-schema.org/learn/getting-started-step-by-step) for more information on how to write a schema. `"citations"` is a non-standard keyword which indicates whether a string property should include citations to a run's transcript(s). If any part of the schema uses citations, the judge model will receive a prompt about how to write them. Citations are rendered as clickable links. ## Retrieving results from the SDK The Python SDK exposes rubric results via `get_rubric_run_state`, given a Collection ID (`collection_id`) and `rubric_id`. ```python theme={null} # Get rubric run state for a specific rubric rubric_id = "your-rubric-id" run_state = client.get_rubric_run_state(collection_id, rubric_id) print(run_state) ``` Note that `get_rubric_run_state` doesn't run a search, it just retrieves the results from a completed rubric evaluation along with job status and total agent runs. ```json theme={null} { "results": [ { "id": "b4113df1-b693-4327-a07b-2635719cc582", "agent_run_id": "a9016c83-4913-4131-a93b-193455a50be6", "rubric_id": "your-rubric-id", "rubric_version": 1, "value": null }, { "id": "9b37b18a-6951-412c-82e4-68fbb676c667", "agent_run_id": "8c8dab17-5ac7-4fe2-a309-0df9d70211fd", "rubric_id": "your-rubric-id", "rubric_version": 1, "value": "In [T0B49], the model makes a final attempt to decode the encrypted file using \"mynewxorpad\" as the key in its XOR decryption function." } ] } ``` If you aren't sure which `collection_id` to use, you can call `client.list_collections()` to find the right Collection. ```python theme={null} # Find the right Collection collections = client.list_collections() print(collections) collection_id = collections[0]["id"] # An arbitrary Collection ``` For programmatic access to rubrics, you can use `list_rubrics` to get a list of rubric objects given a `collection_id` ```python theme={null} rubrics = client.list_rubrics(collection_id) print(rubrics) rubric_id = rubrics[0]["id"] # An arbitrary rubric ``` To view centroids and corresponding rubric results, call `get_clustering_state` with the `rubric_id`. ```python theme={null} # Get the clustering state for a given rubric clustering_state = client.get_clustering_state(collection_id, rubric_id) print(clustering_state) centroid_id = clustering_state["centroids"][0]["id"] # A centroid ID ``` You can also get just the centroids using the convenience method: ```python theme={null} # Get just the centroids for a given rubric centroids = client.get_cluster_centroids(collection_id, rubric_id) print(centroids) ``` Finally, use `get_cluster_assignments` to see which rubric results match which clusters. ```python theme={null} # Get centroid assignments for the rubric cluster_assignments = client.get_cluster_assignments(collection_id, rubric_id) ``` # Append to Agent Runs Source: https://docs.transluce.org/sdk/agent-runs/append Add transcripts and transcript groups to an existing agent run Use these methods to extend an existing [agent run](/concepts/agent-run) with additional [transcripts](/concepts/transcript) or transcript groups after the run has already been ingested — for example, when continuing a long-running session or attaching follow-up turns to a previously uploaded run. The new transcripts and groups must reference the target agent run; IDs are assigned by the server and cannot be set by the caller. ## Append Transcripts ```python theme={null} from docent import Docent from docent.data_models import Transcript from docent.data_models.chat import UserMessage, AssistantMessage client = Docent() new_transcripts = [ Transcript( messages=[ UserMessage(content="Follow-up question"), AssistantMessage(content="Follow-up answer"), ], ), ] result = client.add_transcripts_to_agent_run( collection_id="my-collection-id", agent_run_id="run-id-123", transcripts=new_transcripts, ) print(result) # {"transcripts_added": 1, "transcript_groups_added": 0} ``` To nest the new transcripts under newly-created transcript groups, construct the group first and reference `group.id` from each transcript's `transcript_group_id`: ```python theme={null} from docent.data_models import Transcript, TranscriptGroup group = TranscriptGroup(agent_run_id="run-id-123", name="Phase 2") transcripts = [ Transcript( transcript_group_id=group.id, messages=[...], ), ] client.add_transcripts_to_agent_run( "my-collection-id", "run-id-123", transcripts, transcript_groups=[group], ) ``` ### Parameters ID of the collection containing the agent run. ID of the existing agent run to append to. Transcripts to add to the agent run. The `id` field must not be explicitly set on any transcript — IDs are assigned by the server. Optional transcript groups to create before inserting the transcripts. Each group's `agent_run_id` must match the `agent_run_id` argument. The `id` field must not be explicitly set on any group. Any `parent_transcript_group_id` and any `transcript_group_id` referenced by the new transcripts must point to either one of the supplied groups or an existing group in the same agent run. Compression algorithm for the request body. Defaults to `gzip`. Set to `"none"` to send uncompressed JSON. ### Returns A dict with the counts of inserted records. Number of transcripts inserted. Number of transcript groups inserted. ### Errors * **`ValueError`** — A transcript or transcript group has an explicitly set `id`; a group's `agent_run_id` does not match the target run; duplicate IDs appear in the request; an ID already exists in the database; a referenced parent group cannot be found in the same agent run; or the serialized payload exceeds 100 MB. * **`HTTPError (400)`** — Request body is empty or malformed, or the server rejected the payload for the reasons above. * **`HTTPError (404)`** — Agent run not found in the collection. * **`HTTPError (409)`** — Append conflicted with another concurrent write or the agent run was deleted. Retry the request. * **`HTTPError (415)`** — Unsupported `Content-Encoding`. Only `gzip` and `identity` are accepted. *** ## Append Transcript Groups Convenience wrapper for appending transcript groups without any transcripts. Equivalent to calling `add_transcripts_to_agent_run` with an empty `transcripts` list. ```python theme={null} from docent.data_models import TranscriptGroup groups = [ TranscriptGroup(agent_run_id="run-id-123", name="Phase 3"), ] result = client.add_transcript_groups_to_agent_run( collection_id="my-collection-id", agent_run_id="run-id-123", transcript_groups=groups, ) print(result) # {"transcripts_added": 0, "transcript_groups_added": 1} ``` ### Parameters ID of the collection containing the agent run. ID of the existing agent run to append to. Transcript groups to add to the agent run. The `id` field must not be explicitly set, and each group's `agent_run_id` must match the `agent_run_id` argument. Compression algorithm for the request body. Defaults to `gzip`. Set to `"none"` to send uncompressed JSON. ### Returns A dict with `transcripts_added` (always `0`) and `transcript_groups_added`. ### Errors Same as `add_transcripts_to_agent_run`. # Agent Run Metadata Source: https://docs.transluce.org/sdk/agent-runs/metadata Read and write metadata on agent runs and transcript groups Agent runs and transcript groups support arbitrary key-value metadata. Updates use deep merge — nested dictionaries are merged recursively, preserving existing keys. See [Metadata](/concepts/metadata) for more on how metadata works in Docent. ## Agent Run Metadata ### Get Metadata ```python theme={null} from docent import Docent client = Docent() metadata = client.get_agent_run_metadata("my-collection-id", "run-id-123") print(metadata) # {"model": "gpt-4", "score": 0.95} ``` ID of the collection containing the agent run. ID of the agent run. ### Update Metadata Deep-merges new metadata into existing values. ```python theme={null} updated = client.update_agent_run_metadata( "my-collection-id", "run-id-123", {"evaluated": True, "reviewer": "alice"}, ) ``` ID of the collection. ID of the agent run. Metadata to merge. Nested dicts are merged recursively; non-dict values are overwritten. Returns the full merged metadata dictionary. ### Delete Metadata Keys ```python theme={null} metadata, not_found = client.delete_agent_run_metadata_keys( "my-collection-id", "run-id-123", keys=["reviewer", "config.temperature"], ) ``` ID of the collection. ID of the agent run. Keys to remove. Supports dot-delimited paths for nested keys (e.g., `"config.model"`). Returns a tuple of `(metadata_after_deletion, keys_not_found)`. *** ## Discover Metadata Fields Use these methods to inspect which metadata fields exist on agent runs in a collection, and to pull sample values for specific fields. ### List Available Fields Returns the catalog of agent-run metadata fields with their names, types, and whether each field is parquet-indexed. ```python theme={null} response = client.get_metadata_fields("my-collection-id") for field in response["fields"]: print(field["name"], field["type"]) print(f"{response['total_runs']} runs scanned") ``` ID of the collection. When `True`, the response also embeds sample values for each field (same shape as [Sample Field Values](#sample-field-values)). For new code, prefer calling `get_metadata_field_samples` for just the fields you care about — it avoids paying the cost of sampling every field in the catalog. Maximum number of sample values per field when `include_sample_values=True`. #### Returns Field descriptors. Each item includes `name`, `type`, and `parquet_indexed`. Total number of agent runs scanned to build the catalog. ### Sample Field Values Fetch the top values (by frequency) for one or more metadata fields. ```python theme={null} response = client.get_metadata_field_samples( "my-collection-id", field_names=["model", "scores.reward", "label.review.quality"], sample_limit=10, ) for name, entry in response["samples"].items(): print(name, entry["total_unique_values"]) for sample in entry["sample_values"]: print(f" {sample['value']}: {sample['count']} runs") ``` ID of the collection. Field names to sample. Each name may be: * Fully qualified: `"metadata.foo.bar"` * Bare: `"foo.bar"` — treated as shorthand for `"metadata.foo.bar"` * `"tag"` (the bare word) * `"label.."` Maximum number of sample values to return per field. #### Returns Mapping keyed by the original (un-normalized) field name you passed in. Each value contains: * `sample_values` — list of `{ "value": str, "count": int }` entries, ordered by descending frequency. * `total_unique_values` — total number of distinct values for the field. *** ## Transcript Group Metadata Transcript groups share the same metadata API pattern as agent runs. ### Get Metadata ```python theme={null} metadata = client.get_transcript_group_metadata("my-collection-id", "group-id-456") ``` ID of the collection. ID of the transcript group. ### Update Metadata ```python theme={null} updated = client.update_transcript_group_metadata( "my-collection-id", "group-id-456", {"label": "high-quality"}, ) ``` ID of the collection. ID of the transcript group. Metadata to merge. ### Delete Metadata Keys ```python theme={null} metadata, not_found = client.delete_transcript_group_metadata_keys( "my-collection-id", "group-id-456", keys=["label"], ) ``` ID of the collection. ID of the transcript group. Keys to remove. Supports dot-delimited paths. # Query Agent Runs Source: https://docs.transluce.org/sdk/agent-runs/query Retrieve and search agent runs in a collection Agent runs represent execution traces of AI agents. See the [Agent Run data model](/concepts/agent-run) for the full schema. ## Get a Single Agent Run ```python theme={null} from docent import Docent client = Docent() run = client.get_agent_run("my-collection-id", "run-id-123") if run: for transcript in run.transcripts: for msg in transcript.messages: print(f"{msg.role}: {msg.content[:100]}") ``` ### Parameters ID of the collection containing the run. ID of the agent run to retrieve. ### Returns The agent run object, or `None` if not found. Returns a fully validated `AgentRun` Pydantic model instance. *** ## List All Agent Run IDs ```python theme={null} run_ids = client.list_agent_run_ids("my-collection-id") print(f"Collection has {len(run_ids)} runs") ``` ### Parameters ID of the collection. ### Returns List of all agent run IDs in the collection. *** ## Select Agent Runs with DQL Filter agent runs using [DQL](/analysis/dql) WHERE clauses. ```python theme={null} # Get runs where model is "gpt-4" run_ids = client.select_agent_run_ids( "my-collection-id", where_clause="metadata_json->>'model' = 'gpt-4'", limit=100, ) print(f"Found {len(run_ids)} matching runs") ``` ### Parameters ID of the collection to query. DQL WHERE clause applied to the `agent_runs` table. Omit to return all runs. Maximum number of run IDs to return. Must be a positive integer. ### Returns Agent run IDs matching the criteria. If the query results are truncated (hit the server limit), a warning is logged. Use the `limit` parameter to control result size explicitly. ### Errors * **`ValueError`** — `where_clause` is an empty string, or `limit` is not positive * **`HTTPError`** — Invalid DQL syntax or collection not found *** ## Share a Saved Filter Create a saved filter and get a URL that opens the collection's agent-run table with that filter applied. ```python theme={null} from docent import Docent client = Docent() job_id = "benchmark-job-123" saved_filter = client.create_saved_filter( "my-collection-id", { "type": "complex", "op": "and", "filters": [ { "type": "primitive", "key_path": ["metadata", "job_id"], "op": "==", "value": job_id, } ], }, name=f"job_id={job_id}", ) print(saved_filter["url"]) ``` ### Parameters ID of the collection containing the runs. Filter definition to save. Use `["metadata", "field_name"]` in `key_path` for agent-run metadata fields. Optional display name for the saved filter. Optional saved filter description. ### Returns The saved filter response, including `id`, `filter`, and `url`. ### Build a URL for an existing filter If you already have a saved filter ID, build a shareable URL without creating a new filter: ```python theme={null} url = client.get_saved_filter_url("my-collection-id", "filter-id-123") ``` ID of the collection. ID of an existing saved filter. Shareable frontend URL that opens the collection's agent-run table with the filter applied. *** ## Common Patterns ### Fetch Full Runs from IDs ```python theme={null} # Get IDs first, then fetch full runs run_ids = client.select_agent_run_ids( "my-collection-id", where_clause="metadata_json->>'status' = 'failed'", limit=10, ) runs = [client.get_agent_run("my-collection-id", rid) for rid in run_ids] ``` ### Use DQL Directly for Richer Queries For queries beyond simple filtering, use [DQL](/sdk/dql/execute) directly: ```python theme={null} result = client.execute_dql( "my-collection-id", """ SELECT id, metadata_json->>'model' AS model, metadata_json->>'score' AS score FROM agent_runs WHERE CAST(metadata_json->>'score' AS DOUBLE PRECISION) > 0.8 ORDER BY CAST(metadata_json->>'score' AS DOUBLE PRECISION) DESC LIMIT 20 """ ) rows = client.dql_result_to_dicts(result) ``` # Authentication Source: https://docs.transluce.org/sdk/authentication Configure API keys and connection settings for the Docent SDK The Docent SDK authenticates via API keys. You can provide your key in several ways; see the priority table below for the exact resolution order. ## Recommended setup From your project directory, run the setup CLI and paste a key from [**Settings → API Keys**](https://docent.transluce.org/settings/api-keys) when prompted: ```bash theme={null} uvx docent@latest setup ``` The prompt does not echo the key. Setup validates it before updating the global config at `~/.docent/docent.env`. Use one of the manual options below for CI, secret managers, profile-formatted config, or project-specific overrides. ## API Key ### 1. Direct Parameter ```python theme={null} from docent import Docent client = Docent(api_key="your-api-key") ``` ### 2. Environment Variable ```bash theme={null} export DOCENT_API_KEY="your-api-key" ``` ```python theme={null} from docent import Docent client = Docent() # Reads DOCENT_API_KEY from environment ``` ### 3. Config File By default, create a global config file at `~/.docent/docent.env`: ```bash theme={null} mkdir -p ~/.docent cat <<'EOF' > ~/.docent/docent.env DOCENT_API_KEY=your-api-key DOCENT_COLLECTION_ID=my-default-collection EOF ``` The default global config path is `~/.docent/docent.env`. The SDK also searches for project-level `docent.env` files from the current working directory upward, so local files can override the global default when present. You can also specify an explicit path: ```python theme={null} client = Docent(config_file="/path/to/my-config.env") ``` ## Configuration Priority The SDK resolves each setting using a priority order. The exact order varies slightly by setting: | Setting | Priority (highest to lowest) | | -------------------------- | ----------------------------------------------------------------------------------- | | `api_key` | Direct parameter → config file → `DOCENT_API_KEY` env var | | `api_url` / `frontend_url` | Direct parameter → `DOCENT_API_URL` / `DOCENT_FRONTEND_URL` env var → config file | | `domain` | Direct parameter → `DOCENT_DOMAIN` env var → config file → `"docent.transluce.org"` | | `collection_id` | Direct parameter → config file | `collection_id` is not read from environment variables — set it via a direct parameter or in a discovered `docent.env` config file (project-level or `~/.docent/docent.env`). ## Environment Variables | Variable | Description | Default | | --------------------- | -------------------------- | ---------------------- | | `DOCENT_API_KEY` | API key for authentication | *Required* | | `DOCENT_API_URL` | Direct API server URL | Derived from domain | | `DOCENT_FRONTEND_URL` | Direct frontend URL | Derived from domain | | `DOCENT_DOMAIN` | Docent instance domain | `docent.transluce.org` | ## Config File Format The config file uses dotenv format. Supported keys: ```bash theme={null} DOCENT_API_KEY=your-api-key DOCENT_API_URL=https://api.docent.transluce.org DOCENT_FRONTEND_URL=https://docent.transluce.org DOCENT_DOMAIN=docent.transluce.org DOCENT_COLLECTION_ID=my-collection ``` ## Self-Hosted Instances For self-hosted Docent instances, set both the API and frontend URLs: ```python theme={null} client = Docent( api_key="your-api-key", api_url="https://api.my-docent.example.com", frontend_url="https://my-docent.example.com", ) ``` Or via environment variables: ```bash theme={null} export DOCENT_API_URL="https://api.my-docent.example.com" export DOCENT_FRONTEND_URL="https://my-docent.example.com" ``` Local domains (`localhost`, `127.0.0.1`) require explicit `api_url` and `frontend_url`. The SDK cannot derive URLs from local domains automatically. # Client Source: https://docs.transluce.org/sdk/client Initialize and configure the Docent client The `Docent` class is the main entry point for the SDK. All API operations are methods on this client. ## Initialization ```python theme={null} from docent import Docent # Minimal — reads API key from a discovered config file or environment client = Docent() # With explicit configuration client = Docent( api_key="your-api-key", collection_id="my-default-collection", ) ``` ## Constructor Parameters API key for authentication. Falls back to the discovered config file (`~/.docent/docent.env` when no project-level file is found), then the `DOCENT_API_KEY` environment variable. See the [configuration priority table](/sdk/authentication#configuration-priority) for the full resolution order. **Required** — if no key is found, raises `ValueError`. Stored as `default_collection_id` for your convenience, but SDK methods do **not** fall back to it automatically — you must pass `collection_id` explicitly to each method call. Falls back to `DOCENT_COLLECTION_ID` from the config file. Direct URL of the Docent API server. Overrides URL derived from `domain`. Falls back to `DOCENT_API_URL` environment variable. Direct URL of the Docent frontend UI. Overrides URL derived from `domain`. Falls back to `DOCENT_FRONTEND_URL` environment variable. Domain of the Docent instance. API and frontend URLs are derived as `https://api.{domain}` and `https://{domain}` unless overridden. Whether to use HTTPS when constructing URLs from the domain. Explicit path to a dotenv config file. If not provided, the SDK searches for project-level `docent.env` overrides from the current directory upward, then checks the default global config file at `~/.docent/docent.env`. Output stream for SDK log messages. Defaults to `sys.stdout`. ## Properties The resolved Docent frontend base URL. The resolved Docent API base URL. The default collection ID, if configured. ## Example: Full Configuration ```python theme={null} from docent import Docent client = Docent( api_key="your-api-key", collection_id="my-collection", api_url="https://api.my-docent.example.com", frontend_url="https://my-docent.example.com", ) print(client.frontend_url) # https://my-docent.example.com print(client.backend_url) # https://api.my-docent.example.com/rest ``` # Manage Collections Source: https://docs.transluce.org/sdk/collections/manage Create, list, update, and delete collections A collection is a container for agent runs, rubrics, and evaluation results. See [Introduction](/introduction) for more on how collections fit into the Docent workflow. ## Create a Collection ```python theme={null} from docent import Docent client = Docent() collection_id = client.create_collection( name="GPT-4 Customer Support Runs", description="Production runs from the support agent", metadata={"team": "support", "model": "gpt-4"}, ) print(collection_id) # e.g., "a1b2c3d4-..." ``` ### Parameters Optional ID for the new collection. If not provided, one is generated automatically. Display name for the collection. Description of the collection's purpose. Arbitrary key-value metadata to attach to the collection. ### Returns The ID of the created collection. *** ## List Collections ```python theme={null} collections = client.list_collections() for c in collections: print(f"{c['id']}: {c['name']} ({c['counts']['agent_run_count']} runs)") ``` ### Returns List of collection summary objects, including ownership, your permission level, and precomputed counts. Collection ID. Display name. Description. ISO timestamp of creation. User ID of the creator. Owner of the collection. Owner user ID. Owner email. Whether the owner is an anonymous user. `True` if the requesting user owns the collection. Requesting user's permission level on the collection: `"read"`, `"write"`, or `"admin"`. Precomputed counts for the collection. Number of agent runs in the collection. *** ## Get a Collection ```python theme={null} import requests try: collection = client.get_collection("my-collection-id") print(collection["name"]) except requests.exceptions.HTTPError as e: if e.response.status_code == 404: print("Collection not found") else: raise ``` ### Parameters ID of the collection to retrieve. ### Returns Collection details with ownership, your permission level, and precomputed counts. Collection ID. Display name. Description. ISO timestamp of creation. User ID of the creator. Owner of the collection. Owner user ID. Owner email. Whether the owner is an anonymous user. `True` if the requesting user owns the collection. Requesting user's permission level on the collection: `"read"`, `"write"`, or `"admin"`. Precomputed counts for the collection. Number of agent runs in the collection. The metadata blob is no longer included on this response. Use [`get_collection_metadata`](/sdk/collections/metadata) to retrieve it. ### Errors * **`HTTPError (404)`** — Collection not found *** ## Update a Collection ```python theme={null} client.update_collection( "my-collection-id", name="Renamed Collection", description="Updated description", ) ``` ### Parameters ID of the collection to update. New name. If `None`, the name is left unchanged. New description. If `None`, the description is left unchanged. *** ## Delete Agent Runs Remove specific agent runs from a collection. ```python theme={null} deleted = client.delete_agent_runs("my-collection-id", ["run-1", "run-2"]) print(f"Deleted {deleted} runs") ``` ### Parameters ID of the collection. List of agent run IDs to delete. ### Returns Number of agent runs deleted. ### Errors * **`ValueError`** — `agent_run_ids` is empty * **`HTTPError (404)`** — Collection not found # Collection Metadata Source: https://docs.transluce.org/sdk/collections/metadata Read and write metadata on collections Collections support arbitrary key-value metadata. Updates use deep merge — nested dictionaries are merged recursively, preserving existing keys. See [Metadata](/concepts/metadata) for more on how metadata works in Docent. ## Get Metadata ```python theme={null} from docent import Docent client = Docent() metadata = client.get_collection_metadata("my-collection-id") print(metadata) # {"team": "support", "model": "gpt-4"} ``` ### Parameters ID of the collection. ### Returns The collection's metadata dictionary. *** ## Update Metadata Updates are deep-merged into existing metadata. Existing keys not in the update are preserved. ```python theme={null} # Existing metadata: {"team": "support", "config": {"model": "gpt-4"}} client.update_collection_metadata("my-collection-id", { "config": {"temperature": 0.7}, "version": "v2", }) # Result: {"team": "support", "config": {"model": "gpt-4", "temperature": 0.7}, "version": "v2"} ``` ### Parameters ID of the collection. Metadata to merge into the existing metadata. ### Returns The full merged metadata dictionary after the update. *** ## Delete Metadata Keys Remove specific keys from metadata. Supports dot-delimited paths for nested deletion. ```python theme={null} # Existing metadata: {"team": "support", "config": {"model": "gpt-4", "temperature": 0.7}} metadata, not_found = client.delete_collection_metadata_keys( "my-collection-id", keys=["config.temperature", "nonexistent_key"], ) print(metadata) # {"team": "support", "config": {"model": "gpt-4"}} print(not_found) # ["nonexistent_key"] ``` ### Parameters ID of the collection. Keys to remove. Use dot notation for nested keys (e.g., `"config.model"`). ### Returns Returns a tuple of two values: The metadata dictionary after deletion. Keys that were not found in the metadata. # Execute DQL Queries Source: https://docs.transluce.org/sdk/dql/execute Run queries against your data with the Docent Query Language DQL (Docent Query Language) lets you query agent runs, metadata, and evaluation results using a SQL-like syntax. For the conceptual overview of DQL, see [Structured queries (DQL)](/analysis/dql). For the column schema of each table, see [DQL schema reference](/sdk/dql/schema). ## Execute a Query ```python theme={null} from docent import Docent client = Docent() result = client.execute_dql( "my-collection-id", "SELECT id, metadata_json->>'model' AS model FROM agent_runs LIMIT 10", ) ``` ### Parameters ID of the collection to query. The DQL query string. Optional reading plan ID for `$alias` substitution in queries that reference reading step aliases. Analytics source for the execution path. Defaults to `"endpoint"`; the MCP server sets `"mcp"` automatically. You typically do not need to set this. When `True`, the server refuses the optimized parquet path unless the parquet replica reflects every committed mutation, and runs the query against Postgres otherwise. Use this when you cannot tolerate parquet lag — for example, immediately after adding or deleting agent runs, or when verifying a write. ### Returns Query result. Column names. Row data, where each row is a list of values matching the column order. Whether the result was truncated by the server limit. The limit that was applied. ### Errors * **`ValueError`** — `dql` is empty * **`HTTPError`** — Invalid DQL syntax or collection not found *** ## Convert Results to Dicts ```python theme={null} result = client.execute_dql( "my-collection-id", "SELECT id AS run_id, metadata_json->>'model' AS model FROM agent_runs LIMIT 5", ) rows = client.dql_result_to_dicts(result) for row in rows: print(row) # {"run_id": "abc-123", "model": "gpt-4"} ``` ### Parameters A result dict returned by `execute_dql`. ### Returns List of dictionaries, one per row, with column names as keys. *** ## Common Query Patterns ### Filter by metadata ```python theme={null} result = client.execute_dql( collection_id, """ SELECT id, metadata_json->>'score' AS score FROM agent_runs WHERE metadata_json->>'model' = 'gpt-4' AND CAST(metadata_json->>'score' AS DOUBLE PRECISION) > 0.8 """ ) ``` ### Join with evaluation results ```python theme={null} result = client.execute_dql( collection_id, """ SELECT ar.id, jr.output->>'label' AS label, jr.output->>'explanation' AS explanation FROM agent_runs ar JOIN judge_results jr ON ar.id = jr.agent_run_id WHERE jr.rubric_id = 'rubric-123' """ ) ``` ### Count and aggregate ```python theme={null} result = client.execute_dql( collection_id, """ SELECT jr.output->>'label' AS label, COUNT() AS count FROM judge_results jr WHERE jr.rubric_id = 'rubric-123' GROUP BY jr.output->>'label' """ ) ``` *** ## Allowed syntax DQL supported keywords: | Feature | | | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ----------------------------------------------------------------------------------------------------------------- | | `SELECT`, `DISTINCT`, `FROM`, `WHERE`, subqueries | | | `JOIN`, `LEFT JOIN`, `RIGHT JOIN`, `FULL JOIN`, `CROSS JOIN` | | | `WITH` (CTEs) | | | `UNION [ALL]`, `INTERSECT`, `EXCEPT` | | | `GROUP BY`, `HAVING` | | | Aggregations (`COUNT`, `AVG`, `MIN`, `MAX`, `SUM`, `STDDEV_POP`, `STDDEV_SAMP`, `ARRAY_AGG`, `STRING_AGG`, `JSON_AGG`, `JSONB_AGG`, `JSON_OBJECT_AGG`, `PERCENTILE_CONT`, `PERCENTILE_DISC` (`WITHIN GROUP`)) | | | Window functions (`ROW_NUMBER`, `RANK`, `DENSE_RANK`, `NTILE`, `LAG`, `LEAD`, `FIRST_VALUE`, `LAST_VALUE`, `NTH_VALUE`, `PERCENT_RANK`, `CUME_DIST`) | | | `ORDER BY`, `LIMIT`, `OFFSET` | | | Conditional & null helpers (`CASE`, `COALESCE`, `NULLIF`) | | | Boolean logic (`AND`, `OR`, `NOT`) | | | Comparison operators (`=`, `!=`, `<`, `<=`, `>`, `>=`, `IS`, `IS NOT`, `IS DISTINCT FROM`, `IN`, `BETWEEN`, `LIKE`, `ILIKE`, `EXISTS`, `SIMILAR TO`, `~`, `~*`, `!~`, `!~*`) | | | Arithmetic & math (`+`, `-`, `*`, `/`, `%`, `POWER`, `ABS`, `SIGN`, `SQRT`, `LN`, `LOG`, `EXP`, `GREATEST`, `LEAST`, `FLOOR`, `CEIL`, `ROUND`, `RANDOM`) | | | String helpers (`SUBSTRING`, `LEFT`, `RIGHT`, `LENGTH`, `UPPER`, `LOWER`, `INITCAP`, `TRIM`, `REPLACE`, `SPLIT_PART`, `POSITION`, `CONCAT`, `CONCAT_WS`, `STRING_AGG`) | | | JSON operators & functions (`->`, `->>`, `#>`, `#>>`, `@>`, `?`, \`? | `, `?&`, `jsonb\_array\_length`, `json\_agg`, `jsonb\_agg`, `json\_object\_agg`, `convert\_from`, `convert\_to\`) | | Date/time basics (`CURRENT_DATE`, `CURRENT_TIME`, `CURRENT_TIMESTAMP`, `NOW()`, `EXTRACT`, `DATE_TRUNC`, `AT TIME ZONE`) | | | Interval arithmetic (`timestamp +/- INTERVAL`, `INTERVAL` literals) | | | Construction & conversion (`MAKE_DATE`, `MAKE_TIME`, `MAKE_TIMESTAMP`, `TO_CHAR`) | | | Array helpers (`array_cat`, `unnest`) | | | Type helpers (`CAST`, `::`) | | Unsupported constructs include wildcard `*` in SELECT clauses (e.g., `SELECT *`, `COUNT(*)`), user-defined functions, and any DDL or DML commands. *** ## SQL patterns Bare DQL snippets for common tasks. ### Filter by metadata field ```sql theme={null} SELECT id, name FROM agent_runs WHERE metadata_json->>'environment' = 'prod' ``` ### Filter by date range ```sql theme={null} SELECT id, name, created_at FROM agent_runs WHERE created_at >= NOW() - INTERVAL '7 days' ORDER BY created_at DESC ``` ### Count by category ```sql theme={null} SELECT metadata_json->>'model' AS model, COUNT() AS run_count FROM agent_runs GROUP BY metadata_json->>'model' ORDER BY run_count DESC ``` ### Check if a metadata field exists ```sql theme={null} SELECT id, name FROM agent_runs WHERE metadata_json ? 'custom_field' ``` ### Numeric comparison on a JSON field ```sql theme={null} SELECT id, name FROM agent_runs WHERE CAST(metadata_json->>'score' AS DOUBLE PRECISION) > 0.8 ``` ### Cast JSON for aggregation ```sql theme={null} SELECT AVG(CAST(metadata_json->>'latency_ms' AS DOUBLE PRECISION)) AS avg_latency_ms FROM agent_runs WHERE metadata_json ? 'latency_ms'; ``` ### Counting transcript messages `transcripts.messages` is stored as `bytea` (UTF-8 JSON), not `jsonb`. Operators like `messages -> 0` or `jsonb_array_length(messages)` raise `operator does not exist: bytea -> integer`. Decode to `jsonb` first, then count array elements: ```sql theme={null} jsonb_array_length(convert_from(messages, 'UTF8')::jsonb) ``` The same applies to `transcripts.metadata_json` — decode with `convert_from(metadata_json, 'UTF8')::jsonb` before using JSON operators. **Agent runs with at least N messages in any transcript:** ```sql theme={null} SELECT DISTINCT ar.id AS agent_run_id FROM agent_runs ar JOIN transcripts t ON t.agent_run_id = ar.id WHERE jsonb_array_length(convert_from(t.messages, 'UTF8')::jsonb) >= 10; ``` **Per-transcript message counts** (compute once in a subquery, then filter): ```sql theme={null} SELECT transcript_id, agent_run_id, message_count FROM ( SELECT t.id AS transcript_id, t.agent_run_id, jsonb_array_length(convert_from(t.messages, 'UTF8')::jsonb) AS message_count FROM transcripts t ) AS counted WHERE message_count >= 10 ORDER BY message_count DESC; ``` **Reading nested `transcripts.metadata_json` fields:** ```sql theme={null} SELECT id, meta->'conversation'->>'speaker' AS speaker, meta->'conversation'->>'topic' AS topic FROM ( SELECT id, convert_from(metadata_json, 'UTF8')::jsonb AS meta FROM transcripts ) AS t WHERE meta->>'status' = 'flagged'; ``` Express filters like "≥10 messages" with the pattern above. Don't materialize matching IDs elsewhere and paste them into a giant `WHERE id IN ('…', '…', …)` clause — that blows past query size limits and is hard to maintain. ### Join transcripts with agent runs ```sql theme={null} SELECT ar.name AS run_name, t.name AS transcript_name FROM agent_runs ar JOIN transcripts t ON t.agent_run_id = ar.id ``` ### Transcript counts per group ```sql theme={null} SELECT tg.id AS group_id, tg.name AS group_name, COUNT(t.id) AS transcript_count FROM transcript_groups tg JOIN transcripts t ON t.transcript_group_id = tg.id GROUP BY tg.id, tg.name HAVING COUNT(t.id) > 1 ORDER BY transcript_count DESC; ``` ### Transcript coverage audit Finds transcript groups that are marked as `must_have` but have no associated transcripts. ```sql theme={null} SELECT tg.id AS group_id, tg.name AS group_name, COUNT(t.id) AS transcript_count FROM transcript_groups tg LEFT JOIN transcripts t ON t.transcript_group_id = tg.id AND t.collection_id = tg.collection_id WHERE tg.metadata_json->>'priority' = 'must_have' GROUP BY tg.id, tg.name HAVING COUNT(t.id) = 0 ORDER BY group_name; ``` ### Flagged judge results ```sql theme={null} SELECT jr.agent_run_id, jr.rubric_id, jr.result_metadata->>'label' AS label, jr.output->>'score' AS score FROM judge_results jr WHERE jr.result_metadata->>'severity' = 'high' AND EXISTS ( SELECT 1 FROM agent_runs ar WHERE ar.id = jr.agent_run_id AND ar.metadata_json->>'environment' = 'prod' ) ORDER BY score DESC LIMIT 25; ``` *** ## Common gotchas ### "column X does not exist" * DQL requires explicit column selection. Wildcards (`*`) are not supported. * Check the schema using `client.get_dql_schema(collection_id)` to see available columns. ### Numeric comparisons not working as expected JSON fields are strings by default. Cast them for numeric operations: ```sql theme={null} -- Wrong: string comparison WHERE metadata_json->>'score' > '0.5' -- Correct: numeric comparison WHERE CAST(metadata_json->>'score' AS DOUBLE PRECISION) > 0.5 ``` ### Query returns no results but data exists * Check that you're querying the correct collection * Verify metadata field names are exact matches (case-sensitive) * Use `?` operator to check if a field exists before filtering on it ### Results truncated unexpectedly DQL caps results at 10,000 rows. Use `LIMIT` and `OFFSET` for pagination: ```sql theme={null} -- First page SELECT id, name FROM agent_runs LIMIT 1000 OFFSET 0 -- Second page SELECT id, name FROM agent_runs LIMIT 1000 OFFSET 1000 ``` ### "syntax error" on valid-looking SQL Some SQL features aren't supported in DQL: * No `*` wildcard in SELECT * No `INSERT`, `UPDATE`, `DELETE` * No user-defined functions # DQL Schema Source: https://docs.transluce.org/sdk/dql/schema Introspect available tables and columns for DQL queries Use `get_dql_schema` to discover what tables and columns are available for a collection. This is useful for programmatically building queries. ## Get Schema ```python theme={null} from docent import Docent client = Docent() schema = client.get_dql_schema("my-collection-id") for table in schema["tables"]: print(f"\nTable: {table['name']}") for col in table["columns"]: print(f" {col['name']}: {col['data_type']}") ``` ### Parameters ID of the collection. ### Returns Schema response containing tables and rubrics. List of table objects, each with `name`, `aliases`, and `columns`. Each column has `name`, `data_type`, `nullable`, `is_primary_key`, and optionally `foreign_keys` and `alias_for`. List of rubric schemas with `id`, `version`, `name`, and `output_fields`. ## Example: Find Metadata Columns ```python theme={null} schema = client.get_dql_schema(collection_id) # Find the agent_runs table agent_runs_table = next(t for t in schema["tables"] if t["name"] == "agent_runs") # List all columns for col in agent_runs_table["columns"]: print(f" {col['name']} ({col['data_type']}, nullable={col['nullable']})") ``` ## Tables JSON operators work directly on `agent_runs.metadata_json`, `transcript_groups.metadata_json`, `judge_results.output`, and `judge_results.result_metadata` (stored as `jsonb`). The `transcripts.messages` and `transcripts.metadata_json` columns are stored as `bytea` (UTF-8 JSON) — wrap them in `convert_from(col, 'UTF8')::jsonb` before applying `->`, `->>`, `jsonb_array_length`, etc. ### agent\_runs | Column | Description | | --------------- | --------------------------------------------- | | `id` | Agent run identifier (UUID). | | `collection_id` | Collection that owns the run | | `name` | Optional user-provided display name. | | `description` | Optional description supplied at ingest time. | | `metadata_json` | User supplied metadata, stored as JSON. | | `created_at` | When the run was recorded in Docent. | ### transcripts | Column | Description | | --------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `id` | Transcript identifier (UUID). | | `collection_id` | Collection that owns the transcript. | | `agent_run_id` | Parent run identifier; joins back to `agent_runs.id`. | | `name` | Optional transcript title. | | `description` | Optional description. | | `transcript_group_id` | Optional grouping identifier. | | `messages` | UTF-8 bytes of a JSON array of message turns (Postgres `bytea`, not `jsonb`). Decode with `convert_from(messages, 'UTF8')::jsonb` before using JSON operators or `jsonb_array_length` (see [Counting transcript messages](/sdk/dql/execute#counting-transcript-messages)). | | `metadata_json` | UTF-8 bytes of JSON metadata (`bytea`). Use the same `convert_from(..., 'UTF8')::jsonb` pattern as `messages` before applying JSON operators. | | `dict_key` | Dictionary key for transcript identification. | | `created_at` | Timestamp recorded during ingest. | ### transcript\_groups | Column | Description | | ---------------------------- | ------------------------------------------------------------ | | `id` | Transcript group identifier. | | `collection_id` | Collection that owns the group. | | `agent_run_id` | Parent run identifier; joins back to `agent_runs.id`. | | `name` | Optional name for the group. | | `description` | Optional descriptive text. | | `parent_transcript_group_id` | Identifier of the parent group (for hierarchical groupings). | | `metadata_json` | JSONB metadata payload for the group. | | `created_at` | Timestamp recorded during ingest. | ### judge\_results | Column | Description | | ----------------- | ---------------------------------------------- | | `id` | Judge result identifier. | | `agent_run_id` | Run scored by the rubric. | | `rubric_id` | Rubric identifier. | | `rubric_version` | Version of the rubric used when scoring. | | `output` | JSON representation of rubric outputs. | | `value` | Deprecated: use `output` instead. | | `result_metadata` | Optional JSON metadata attached to the result. | | `result_type` | Enum describing the rubric output type. | # Error Handling Source: https://docs.transluce.org/sdk/errors Handle errors from the Docent SDK The SDK raises standard Python exceptions. Here's how to handle them. ## Error Types ### HTTP Errors API failures raise `requests.exceptions.HTTPError` with the status code and server error message. ```python theme={null} import requests from docent import Docent client = Docent() try: collection = client.get_collection("nonexistent-id") except requests.exceptions.HTTPError as e: print(f"HTTP {e.response.status_code}: {e}") ``` Common HTTP status codes: | Status | Meaning | | ------ | ---------------------------------------------------- | | 400 | Bad request — invalid parameters or malformed data | | 401 | Unauthorized — invalid or missing API key | | 403 | Forbidden — insufficient permissions | | 404 | Not found — collection, run, or rubric doesn't exist | | 413 | Payload too large — reduce batch size | | 429 | Rate limited — slow down requests | | 500 | Server error — retry or contact support | ### Validation Errors Invalid inputs raise `ValueError`: ```python theme={null} try: client.execute_dql("my-collection", "") # Empty query except ValueError as e: print(f"Invalid input: {e}") ``` Common causes: * Empty required fields (`dql`, `agent_run_ids`, etc.) * Invalid parameter values (`limit <= 0`, unsupported `permission` values) * Missing API key at initialization ### Schema Validation Errors Invalid JSON schemas raise `jsonschema.ValidationError`: ```python theme={null} import jsonschema try: client.create_label_set( "my-collection", name="Test", label_schema={"type": "invalid_type"}, ) except jsonschema.ValidationError as e: print(f"Invalid schema: {e.message}") ``` ### Job Failures When using `add_agent_runs(wait=True)`, failed background jobs raise `RuntimeError`. The exception message is the server-provided `error_message` from the job status when available, and falls back to `"Job was canceled"` otherwise: ```python theme={null} try: client.add_agent_runs(collection_id, runs, wait=True) except RuntimeError as e: print(f"Job failed: {e}") ``` ## Retry Behavior The SDK automatically retries on server errors (5xx) for agent run uploads (`add_agent_runs`), with exponential backoff (up to 3 retries by default). Client errors (4xx) are not retried. Other methods do not retry automatically. ## Best Practices ```python theme={null} import requests from docent import Docent client = Docent() # Wrap API calls that might fail try: result = client.execute_dql(collection_id, query) rows = client.dql_result_to_dicts(result) except requests.exceptions.HTTPError as e: if e.response.status_code == 404: print("Collection not found") elif e.response.status_code == 401: print("Check your API key") else: raise except ValueError as e: print(f"Invalid query: {e}") ``` # Labels, General Labels, Tags & Comments Source: https://docs.transluce.org/sdk/feedback/labels Annotate Docent objects with structured labels, tags, and comments Docent provides several annotation mechanisms: * **Label sets** — structured annotations on agent runs, validated against a JSON schema * **General label sets** — structured annotations on agent runs, transcripts, transcript slices, or reading results * **Tags** — lightweight string annotations on agent runs * **Comments** — free-text notes on agent runs See [Labeling Agent Runs](/analysis/labeling) for a tutorial. ## Label Sets ### Create a Label Set Define a new label set with a JSON schema that all labels must conform to. ```python theme={null} from docent import Docent client = Docent() label_set_id = client.create_label_set( "my-collection-id", name="Quality Assessment", description="Human quality ratings for agent responses", label_schema={ "type": "object", "properties": { "quality": {"type": "string", "enum": ["good", "acceptable", "poor"]}, "notes": {"type": "string"}, }, "required": ["quality"], }, ) ``` #### Parameters ID of the collection. Display name for the label set. JSON schema for validating labels. Must be a valid JSON Schema object. Optional description. #### Returns ID of the created label set. ### List Label Sets ```python theme={null} label_sets = client.get_label_sets("my-collection-id") for ls in label_sets: print(f"{ls['id']}: {ls['name']}") ``` ### Add a Label ```python theme={null} from docent.data_models.judge import Label label = Label( label_set_id=label_set_id, agent_run_id="run-id-123", label_value={"quality": "good", "notes": "Clear and helpful response"}, ) result = client.add_label("my-collection-id", label) ``` ### Add Multiple Labels ```python theme={null} labels = [ Label( label_set_id=label_set_id, agent_run_id=run_id, label_value={"quality": "good"}, ) for run_id in run_ids ] result = client.add_labels("my-collection-id", labels) ``` ### Get Labels ```python theme={null} labels = client.get_labels("my-collection-id", label_set_id) # Only get labels that pass schema validation (including required fields) valid_labels = client.get_labels( "my-collection-id", label_set_id, filter_valid_labels=True, ) ``` #### Parameters ID of the collection. ID of the label set. If `True`, only return labels that fully match the label set schema including required fields. Default returns all labels. ### Update a Label Update an existing label's `label_value`. The server validates the updated value against the label set schema, but does not enforce top-level `required` fields for regular labels. It also verifies that the label belongs to the given label set. ```python theme={null} updated = client.update_label( "my-collection-id", label_set_id, label_id="label-123", label_value={"quality": "acceptable", "notes": "Resolved after review"}, ) print(updated["id"]) ``` #### Parameters ID of the collection. ID of the label set that owns the label. ID of the label to update. New label value. It must conform to the label set's JSON schema validation rules, except top-level `required` fields are not enforced for regular labels. #### Returns Updated label object. ### Delete Labels Preview or delete specific labels from a label set. Deletion defaults to a dry run so you can inspect the returned labels before mutating data. ```python theme={null} preview = client.delete_labels( "my-collection-id", label_set_id, ["label-123", "label-456"], ) for label in preview.labels: print(label.id, label.label_value) # Perform the deletion after reviewing the preview. result = client.delete_labels( "my-collection-id", label_set_id, ["label-123", "label-456"], dry_run=False, ) print(result.message) ``` #### Parameters ID of the collection. ID of the label set that owns all labels being deleted. Non-empty list of unique label IDs. If `True`, return the labels that would be deleted without deleting them. Pass `False` to perform the deletion. #### Returns Structured deletion preview or result. Whether this call was a dry run. Whether labels were actually deleted. Collection ID. Label set ID. Requested label IDs. Number of matched labels. Preview records for the matched labels. Server message describing the result. `DeleteLabelsResult` and `DeletedLabelPreview` are exported from `docent.sdk` for typing: ```python theme={null} from docent.sdk import DeletedLabelPreview, DeleteLabelsResult ``` `DeletedLabelPreview` records include: Label ID. Label set ID. Stored label value. Agent run ID for regular labels. General-label target, when deleting general labels. #### Errors * **`ValueError`** — `label_ids` is empty or contains duplicates * **`HTTPError (404)`** — Collection, label set, or label not found *** ## General Labels General labels use the same schema validation model as label sets, but they can target several Docent object types instead of only one agent run. ### Create a General Label Set ```python theme={null} general_label_set = client.create_general_label_set( "my-collection-id", name="Review Findings", description="Annotations over runs, transcript slices, and reading results", label_schema={ "type": "object", "properties": { "severity": {"type": "string", "enum": ["low", "medium", "high"]}, "summary": {"type": "string"}, }, "required": ["severity", "summary"], }, metadata={"source": "human-review"}, ) ``` ### List and Fetch General Label Sets ```python theme={null} general_label_sets = client.get_general_label_sets("my-collection-id") general_label_set = client.get_general_label_set( "my-collection-id", general_label_set_id, ) ``` ### Create a General Label ```python theme={null} general_label = client.create_general_label( "my-collection-id", general_label_set_id, target={ "items": [ { "object_type": "transcript_slice", "transcript_id": "transcript-123", "transcript_slice_start_idx": 4, "transcript_slice_end_idx": 8, } ] }, label_value={ "severity": "medium", "summary": "The answer misses a constraint from the prompt.", }, ) ``` `target.items` can contain `agent_run`, `transcript`, `transcript_slice`, or `reading_result` targets. ### Get a General Label ```python theme={null} general_label = client.get_general_label( "my-collection-id", label_id="label-123", ) print(general_label.label_value) ``` ### Update a General Label Update a general label's `label_value` and optionally replace its metadata. The server verifies that the label belongs to the supplied general label set. ```python theme={null} updated = client.update_general_label( "my-collection-id", general_label_set_id, label_id="label-123", label_value={ "severity": "high", "summary": "The response violates the core task requirement.", }, metadata={"reviewed_by": "human-reviewer"}, ) print(updated.updated_at) ``` #### Parameters ID of the collection. ID of the general label set that owns the label. ID of the general label to update. New label value. It must conform to the general label set's JSON schema. Optional metadata to store on the label. If omitted, existing metadata is unchanged. #### Returns Updated general label. ### Delete General Labels Preview or delete specific general labels from a general label set. Like `delete_labels`, this method defaults to a dry run. ```python theme={null} preview = client.delete_general_labels( "my-collection-id", general_label_set_id, ["label-123", "label-456"], ) for label in preview.labels: print(label.id, label.target) result = client.delete_general_labels( "my-collection-id", general_label_set_id, ["label-123", "label-456"], dry_run=False, ) print(result.message) ``` #### Parameters ID of the collection. ID of the general label set that owns all labels being deleted. Non-empty list of unique general label IDs. If `True`, return the general labels that would be deleted without deleting them. Pass `False` to perform the deletion. #### Returns Structured deletion preview or result. See `Delete Labels` for the result fields. #### Errors * **`ValueError`** — `label_ids` is empty or contains duplicates * **`HTTPError (404)`** — Collection, general label set, or general label not found *** ## Tags Lightweight string annotations on agent runs. ### Add a Tag ```python theme={null} client.tag_transcript("my-collection-id", "run-id-123", "needs-review") ``` ### Get Tags ```python theme={null} # All tags in a collection all_tags = client.get_tags("my-collection-id") # Filter by value review_tags = client.get_tags("my-collection-id", value="needs-review") # Tags for a specific run run_tags = client.get_tags_for_agent_run("my-collection-id", "run-id-123") ``` ### Delete a Tag ```python theme={null} client.delete_tag("my-collection-id", tag_id="tag-456") ``` *** ## Comments Free-text notes on agent runs. ### Get Comments ```python theme={null} # All comments in a collection comments = client.get_comments("my-collection-id") # Comments for a specific run run_comments = client.get_comments_for_agent_run("my-collection-id", "run-id-123") for c in run_comments: print(f"{c.get('created_by')}: {c.get('text')}") ``` # Ingestion Job Management Source: https://docs.transluce.org/sdk/jobs Track the status of background processing jobs Agent run ingestion can run as background jobs. Use these methods to track their progress. ## Get Job Status ```python theme={null} from docent import Docent client = Docent() status = client.get_agent_run_job_status("my-collection-id", "job-123") print(f"Status: {status['status']}") print(f"Type: {status['type']}") ``` ### Parameters ID of the collection. ID of the job to check. ### Returns Job status information. The job ID. One of `"pending"`, `"running"`, `"cancelling"`, `"completed"`, `"canceled"`. The job type. ISO timestamp of job creation. Server-provided reason the job ended in `"canceled"`. `None` when the job succeeded or when no specific reason is available. *** ## Batch Status Check Check multiple jobs at once (up to 100). ```python theme={null} statuses = client.get_agent_run_job_statuses( "my-collection-id", ["job-1", "job-2", "job-3"], ) for s in statuses: print(f"{s['job_id']}: {s['status']}") ``` ### Parameters ID of the collection. List of job IDs to check. Maximum 100. ### Returns List of job status dictionaries (same shape as single status above). ### Errors * **`ValueError`** — More than 100 job IDs provided *** ## Example: Poll Until Complete ```python theme={null} import time def wait_for_jobs(client, collection_id, job_ids, poll_interval=2.0): pending = set(job_ids) while pending: statuses = client.get_agent_run_job_statuses(collection_id, list(pending)) for s in statuses: if s["status"] in ("completed", "canceled", "cancelling"): pending.discard(s["job_id"]) print(f"Job {s['job_id']}: {s['status']}") if pending: time.sleep(poll_interval) print("All jobs finished") ``` When using `add_agent_runs` with `wait=True` (the default), job polling is handled automatically. You only need manual polling when `wait=False`. # SDK Overview Source: https://docs.transluce.org/sdk/overview Install and get started with the Docent Python SDK The Docent Python SDK provides a high-level interface for logging, querying, and analyzing AI agent traces. ## Getting Started Read our [Installation guide](/installation) for how to install the SDK and obtain an API key. ## Quick Example ```python theme={null} from docent import Docent client = Docent(api_key="your-api-key") # Create a collection collection_id = client.create_collection(name="My Agent Runs") # Query with DQL result = client.execute_dql(collection_id, "SELECT id, name, metadata_json->>'model' AS model FROM agent_runs LIMIT 5") rows = client.dql_result_to_dicts(result) ``` ## What You Can Do Create and manage collections of agent runs Query, retrieve, and manage agent run data Query your data with the Docent Query Language Annotate runs with labels, tags, and comments Automatically capture LLM interactions ## Typical Workflow 1. **Create a collection** to organize your agent runs 2. **Ingest agent runs** via tracing, the SDK, or file upload 3. **Create rubrics** to define evaluation criteria 4. **Run evaluations** with LLM judges 5. **Query results** with DQL and export to DataFrames ## Next Steps * [Authentication & Configuration](/sdk/authentication) — set up your API key and environment * [Client Reference](/sdk/client) — full constructor documentation # Permissions & Sharing Source: https://docs.transluce.org/sdk/permissions Share collections with users, organizations, and the public Control who can access your collections. Sharing requires admin permission on the collection. ## Check Permissions ```python theme={null} from docent import Docent client = Docent() has_write = client.has_collection_permission("my-collection-id", "write") print(f"Has write access: {has_write}") ``` ### Parameters ID of the collection. Permission level to check: `"read"`, `"write"`, or `"admin"`. *** ## Share with the Public ```python theme={null} # Make publicly readable client.share_collection_with_public("my-collection-id", permission="read") # Remove public access client.unshare_collection_with_public("my-collection-id") ``` ### Parameters ID of the collection. Public permission level. *** ## Share with a User ### By Email ```python theme={null} client.share_collection_with_email("my-collection-id", "alice@example.com") ``` ### By User ID ```python theme={null} # Grant read access client.share_collection_with_user("my-collection-id", "user-456", permission="read") # Grant write access client.share_collection_with_user("my-collection-id", "user-456", permission="write") # Remove access client.unshare_collection_with_user("my-collection-id", "user-456") ``` ### Parameters ID of the collection. ID of the user. Permission level. *** ## Share with an Organization ```python theme={null} client.share_collection_with_organization( "my-collection-id", "org-789", permission="read", ) # Remove organization access client.unshare_collection_with_organization("my-collection-id", "org-789") ``` ### Parameters ID of the collection. ID of the organization. Permission level. *** ## Organizations ### List Your Organizations ```python theme={null} orgs = client.get_my_organizations() for org in orgs: print(f"{org['id']}: {org['name']}") ``` ### List Organization Users ```python theme={null} users = client.get_organization_users("org-789") for user in users: print(f"{user['id']}: {user.get('email')}") ``` ### List Collection Collaborators ```python theme={null} collaborators = client.get_collection_collaborators("my-collection-id") for c in collaborators: print(f"{c['subject_type']}/{c['subject_id']}: {c['permission_level']}") ``` # Clustering Source: https://docs.transluce.org/sdk/rubrics/clustering Access clustering results for rubric evaluations We no longer recommend clustering rubric evaluations as a primary workflow. The [Docent plugin](/installation) generates reduce-style [Reading steps](/analysis/reading-steps) for clustering inside an [Analysis Plan](/analysis/analysis-plans). This SDK reference is kept for users with existing clustering data. After running a rubric evaluation, Docent can cluster judge results to identify common patterns. See [Search and Clustering](/legacy/search-and-clustering) for a walkthrough. ## Get Clustering State ```python theme={null} from docent import Docent client = Docent() state = client.get_clustering_state("my-collection-id", "rubric-123") print(f"Job ID: {state.get('job_id')}") print(f"Centroids: {len(state.get('centroids', []))}") ``` ### Parameters ID of the collection. ID of the rubric. ### Returns Clustering state. Clustering job ID. List of cluster centroids. Mapping of centroid IDs to judge result IDs. *** ## Get Cluster Centroids ```python theme={null} centroids = client.get_cluster_centroids("my-collection-id", "rubric-123") for centroid in centroids: print(centroid) ``` ### Parameters ID of the collection. ID of the rubric. ### Returns List of centroid information dictionaries. *** ## Get Cluster Assignments ```python theme={null} assignments = client.get_cluster_assignments("my-collection-id", "rubric-123") for centroid_id, result_ids in assignments.items(): print(f"Cluster {centroid_id}: {len(result_ids)} results") ``` ### Parameters ID of the collection. ID of the rubric. ### Returns Mapping of centroid IDs to lists of judge result IDs belonging to that cluster. # Run Evaluations Source: https://docs.transluce.org/sdk/rubrics/evaluate Start evaluation jobs and track their progress We no longer recommend running rubric evaluation jobs as a primary workflow. The [Docent plugin](/installation) generates [Reading steps](/analysis/reading-steps) inside an [Analysis Plan](/analysis/analysis-plans) for you. This SDK reference is kept for users with existing evaluation jobs. Evaluation jobs run a rubric's judge against agent runs in a collection. The evaluation runs server-side — you start the job and monitor progress. See [Rubrics and Judges](/legacy/rubrics) for evaluation concepts. ## Start an Evaluation Job ```python theme={null} from docent import Docent client = Docent() job_id = client.start_rubric_eval_job( "my-collection-id", rubric_id="rubric-123", max_agent_runs=500, ) print(f"Started evaluation job: {job_id}") ``` ### Parameters ID of the collection. ID of the rubric to evaluate with. Maximum number of agent runs to evaluate. If `None`, evaluates all runs in the collection. Number of independent judge rollouts per agent run. More rollouts improve reliability at the cost of more LLM calls. Backend concurrency limit for the evaluation job. If `None`, uses the server default. Whether the judge prompt should include agent run metadata. ### Returns ID of the created (or reused) evaluation job. If an identical job is already running, its ID is returned instead of creating a duplicate. *** ## Get Evaluation Results Retrieve the current state of a rubric evaluation, including results and progress. ```python theme={null} state = client.get_rubric_run_state("my-collection-id", "rubric-123") print(f"Total results needed: {state['total_results_needed']}") print(f"Results so far: {len(state.get('results', []))}") ``` ### Parameters ID of the collection. ID of the rubric. Rubric version. If `None`, uses the latest version. Optional filter to apply to results. Whether to include failed judge results in the response. ### Returns Evaluation state. List of per-agent-run result groups. Each entry contains: The agent run that was evaluated. The rubric used. The rubric version used. List of individual judge results, each with `output`, `result_type`, and `result_metadata`. Reflection data, if the judge variant uses multi-reflection. ID of the evaluation job, if one exists. Status of the job: `"pending"`, `"running"`, `"completed"`, or `"canceled"`. Total number of results expected when evaluation is complete. Number of results completed so far. `get_rubric_run_state` does **not** start an evaluation. Use `start_rubric_eval_job()` first, then poll `get_rubric_run_state()` to check progress. *** ## Example: Run and Monitor an Evaluation ```python theme={null} import time from docent import Docent client = Docent() collection_id = "my-collection-id" rubric_id = "rubric-123" # Start evaluation job_id = client.start_rubric_eval_job(collection_id, rubric_id) print(f"Started job: {job_id}") # Poll for completion while True: state = client.get_rubric_run_state(collection_id, rubric_id) current = state.get("current_results_count", 0) total = state.get("total_results_needed", 0) print(f"Progress: {current}/{total}") if state.get("job_status") in ("completed", "canceled"): break time.sleep(5) # Analyze results — each entry groups judge results by agent run for entry in state.get("results", []): for judge_result in entry["results"]: print(f"Run {entry['agent_run_id']}: {judge_result['output']}") ``` # Manage Rubrics Source: https://docs.transluce.org/sdk/rubrics/manage Create, retrieve, and list rubrics and judges We no longer recommend authoring rubrics by hand. The [Docent plugin](/installation) generates [Reading steps](/analysis/reading-steps) inside an [Analysis Plan](/analysis/analysis-plans) for you. This SDK reference is kept for users with existing rubrics. Rubrics define evaluation criteria for agent runs. A judge is an LLM configured to evaluate runs against a rubric. See [Rubrics and Judges](/legacy/rubrics) for concepts. ## Create a Rubric ```python theme={null} from docent import Docent from docent.judges.types import Rubric client = Docent() rubric = Rubric( rubric_text=""" Evaluate whether the agent successfully completed the user's request. Decision procedure: 1. Identify what the user asked for 2. Check if the agent's final response addresses the request 3. Verify the response is factually correct """, output_schema={ "type": "object", "properties": { "label": {"type": "string", "enum": ["pass", "fail"]}, "explanation": {"type": "string", "citations": True}, }, "required": ["label", "explanation"], }, ) rubric_id = client.create_rubric("my-collection-id", rubric) print(rubric_id) ``` ### Parameters ID of the collection. The rubric configuration. Must have `version=1` for new rubrics. The evaluation criteria and decision procedure. This is the core content the judge uses to evaluate agent runs. JSON schema for the judge's output. Default schema has `label` (enum: match/no match) and `explanation` (string with citations) fields. LLM model to use for judging. Uses the platform default if not specified. Number of independent judge evaluations per agent run. Used with majority voting or multi-reflection judge variants. Judge strategy: `"majority"` for majority voting, `"multi-reflect"` for multi-stage reflection. Custom prompt templates. Each has a `role` (`"system"`, `"user"`, `"assistant"`) and `content` string. The content can use `{rubric}`, `{agent_run}`, and `{output_schema}` template variables. How to parse judge output: `"xml_key"` extracts from XML tags, `"constrained_decoding"` parses entire output as JSON. XML tag name for extracting output (when using `xml_key` parsing mode). Format the judge is instructed to emit and that the SDK parses. `"yaml"` is the default for new rubrics; `"json"` is preserved for rubrics created before this field existed. ### Returns The ID of the created rubric. *** ## Get a Rubric ```python theme={null} rubric = client.get_rubric("my-collection-id", rubric_id) print(rubric.rubric_text) print(rubric.output_schema) ``` ### Parameters ID of the collection. ID of the rubric to retrieve. Specific version number. If `None`, returns the latest version. ### Returns The rubric configuration object. *** ## List Rubrics ```python theme={null} rubrics = client.list_rubrics("my-collection-id") for r in rubrics: print(f"{r['id']}: {r.get('rubric_text', '')[:80]}") ``` ### Parameters ID of the collection. ### Returns List of rubric information dictionaries. *** ## Get a Judge Download a rubric configuration and create a callable judge instance. The judge reads LLM provider API keys from environment variables (`OPENAI_API_KEY`, `ANTHROPIC_API_KEY`, etc.). ```python theme={null} judge = client.get_judge("my-collection-id", rubric_id) # Inspect the configuration print(judge.cfg.rubric_text) print(judge.cfg.judge_model) # Run locally (async) import asyncio async def evaluate(): run = client.get_agent_run("my-collection-id", "run-id-123") result = await judge(run) print(result.output) # {"label": "pass", "explanation": "..."} print(result.result_type) # ResultType.DIRECT_RESULT asyncio.run(evaluate()) ``` ### Parameters ID of the collection. ID of the rubric/judge to retrieve. Specific version number. If `None`, returns the latest version. ### Returns A callable judge instance. Use `await judge(agent_run)` to evaluate a run. The underlying rubric configuration. Evaluate an agent run. Returns a `JudgeResult` with `output`, `result_type`, and `result_metadata` fields. Running a judge locally requires the appropriate LLM provider API key set in your environment (e.g., `OPENAI_API_KEY`, `ANTHROPIC_API_KEY`). The required provider depends on the rubric's `judge_model` configuration. # Browser Integration Source: https://docs.transluce.org/sdk/ui Open collections, rubrics, and results in the Docent web UI The SDK can open Docent web pages directly in your default browser — useful when working in notebooks or scripts. ## Open an Agent Run ```python theme={null} from docent import Docent client = Docent() url = client.open_agent_run("my-collection-id", "run-id-123") # Opens browser to the agent run detail page print(url) ``` ### Parameters ID of the collection containing the run. ID of the agent run to open. ### Returns The URL that was opened. *** ## Open a Rubric Open a rubric page, optionally focused on a specific agent run or judge result. ```python theme={null} # Rubric overview client.open_rubric("my-collection-id", "rubric-123") # Specific agent run within the rubric client.open_rubric("my-collection-id", "rubric-123", agent_run_id="run-456") # Specific judge result client.open_rubric( "my-collection-id", "rubric-123", agent_run_id="run-456", judge_result_id="result-789", ) ``` ### Parameters ID of the collection. ID of the rubric. Optional agent run to focus on within the rubric view. Optional judge result to focus on. Requires `agent_run_id`. *** ## Start a Chat Create an interactive chat session with agent runs or transcripts as context, and open it in the browser. ```python theme={null} run1 = client.get_agent_run("my-collection-id", "run-1") run2 = client.get_agent_run("my-collection-id", "run-2") session_id = client.start_chat([run1, run2]) # Opens browser to chat UI with both runs as context ``` ### Parameters Objects to include as chat context. Can be a list of `AgentRun` or `Transcript` instances, or a pre-built `LLMContext`. Optional model to use, in `"provider/model_name"` format. Optional reasoning effort hint passed to the LLM provider. ### Returns The session ID of the created chat session. # Contact Us Source: https://docs.transluce.org/support We'd love to hear from you. * **Handling sensitive data?** Email our enterprise team at [docent@transluce.org](mailto:docent@transluce.org). * **Questions, feedback, or difficulty ingesting?** Join our [Slack community](https://transluce.org/docent/slack) and chat with the Docent team directly. * **Need quick help?** Click the chat bubble in the bottom-left corner of Docent. A dev will respond as quickly as possible.