Run Evaluations

We no longer recommend running rubric evaluation jobs as a primary workflow. The Docent plugin generates Reading steps inside an Analysis Plan for you. This SDK reference is kept for users with existing evaluation jobs.

Evaluation jobs run a rubric’s judge against agent runs in a collection. The evaluation runs server-side — you start the job and monitor progress. See Rubrics and Judges for evaluation concepts.

Start an Evaluation Job

from docent import Docent

client = Docent()

job_id = client.start_rubric_eval_job(
    "my-collection-id",
    rubric_id="rubric-123",
    max_agent_runs=500,
)
print(f"Started evaluation job: {job_id}")

Parameters

str

required

ID of the collection.

str

required

ID of the rubric to evaluate with.

int | None

Maximum number of agent runs to evaluate. If None, evaluates all runs in the collection.

int

default:"1"

Number of independent judge rollouts per agent run. More rollouts improve reliability at the cost of more LLM calls.

int | None

Backend concurrency limit for the evaluation job. If None, uses the server default.

bool

default:"True"

Whether the judge prompt should include agent run metadata.

Returns

str

ID of the created (or reused) evaluation job. If an identical job is already running, its ID is returned instead of creating a duplicate.

Get Evaluation Results

Retrieve the current state of a rubric evaluation, including results and progress.

state = client.get_rubric_run_state("my-collection-id", "rubric-123")
print(f"Total results needed: {state['total_results_needed']}")
print(f"Results so far: {len(state.get('results', []))}")

Parameters

str

required

ID of the collection.

str

required

ID of the rubric.

int | None

Rubric version. If None, uses the latest version.

dict | None

Optional filter to apply to results.

bool

default:"False"

Whether to include failed judge results in the response.

Returns

dict

Evaluation state.

Show Fields

list[dict]

List of per-agent-run result groups. Each entry contains:

Show AgentRunJudgeResults fields

str

The agent run that was evaluated.

str

The rubric used.

int

The rubric version used.

list[dict]

List of individual judge results, each with output, result_type, and result_metadata.

dict | None

Reflection data, if the judge variant uses multi-reflection.

str | None

ID of the evaluation job, if one exists.

str | None

Status of the job: "pending", "running", "completed", or "canceled".

int | None

Total number of results expected when evaluation is complete.

int | None

Number of results completed so far.

get_rubric_run_state does not start an evaluation. Use start_rubric_eval_job() first, then poll get_rubric_run_state() to check progress.

Example: Run and Monitor an Evaluation

import time
from docent import Docent

client = Docent()
collection_id = "my-collection-id"
rubric_id = "rubric-123"

# Start evaluation
job_id = client.start_rubric_eval_job(collection_id, rubric_id)
print(f"Started job: {job_id}")

# Poll for completion
while True:
    state = client.get_rubric_run_state(collection_id, rubric_id)
    current = state.get("current_results_count", 0)
    total = state.get("total_results_needed", 0)
    print(f"Progress: {current}/{total}")

    if state.get("job_status") in ("completed", "canceled"):
        break
    time.sleep(5)

# Analyze results — each entry groups judge results by agent run
for entry in state.get("results", []):
    for judge_result in entry["results"]:
        print(f"Run {entry['agent_run_id']}: {judge_result['output']}")

Getting Started

Collections

Agent Runs

DQL

Feedback & Labels

More

Legacy

Run Evaluations

Start an Evaluation Job

Parameters

Returns

Get Evaluation Results

Parameters

Returns

Example: Run and Monitor an Evaluation

​Start an Evaluation Job

​Parameters

​Returns

​Get Evaluation Results

​Parameters

​Returns

​Example: Run and Monitor an Evaluation

Start an Evaluation Job

Parameters

Returns

Get Evaluation Results

Parameters

Returns

Example: Run and Monitor an Evaluation