Docent provides three annotation mechanisms for agent runs:
- Label sets — structured annotations validated against a JSON schema
- Tags — lightweight string annotations
- Comments — free-text notes
See Labeling Agent Runs for a tutorial.
Label Sets
Create a Label Set
Define a new label set with a JSON schema that all labels must conform to.
from docent import Docent
client = Docent()
label_set_id = client.create_label_set(
"my-collection-id",
name="Quality Assessment",
description="Human quality ratings for agent responses",
label_schema={
"type": "object",
"properties": {
"quality": {"type": "string", "enum": ["good", "acceptable", "poor"]},
"notes": {"type": "string"},
},
"required": ["quality"],
},
)
Parameters
Display name for the label set.
JSON schema for validating labels. Must be a valid JSON Schema object.
Returns
ID of the created label set.
List Label Sets
label_sets = client.get_label_sets("my-collection-id")
for ls in label_sets:
print(f"{ls['id']}: {ls['name']}")
Add a Label
from docent.data_models.judge import Label
label = Label(
label_set_id=label_set_id,
agent_run_id="run-id-123",
label_value={"quality": "good", "notes": "Clear and helpful response"},
)
result = client.add_label("my-collection-id", label)
Add Multiple Labels
labels = [
Label(
label_set_id=label_set_id,
agent_run_id=run_id,
label_value={"quality": "good"},
)
for run_id in run_ids
]
result = client.add_labels("my-collection-id", labels)
Get Labels
labels = client.get_labels("my-collection-id", label_set_id)
# Only get labels that pass schema validation (including required fields)
valid_labels = client.get_labels(
"my-collection-id",
label_set_id,
filter_valid_labels=True,
)
Parameters
If True, only return labels that fully match the label set schema including
required fields. Default returns all labels.
Lightweight string annotations on agent runs.
Add a Tag
client.tag_transcript("my-collection-id", "run-id-123", "needs-review")
# All tags in a collection
all_tags = client.get_tags("my-collection-id")
# Filter by value
review_tags = client.get_tags("my-collection-id", value="needs-review")
# Tags for a specific run
run_tags = client.get_tags_for_agent_run("my-collection-id", "run-id-123")
Delete a Tag
client.delete_tag("my-collection-id", tag_id="tag-456")
Free-text notes on agent runs.
# All comments in a collection
comments = client.get_comments("my-collection-id")
# Comments for a specific run
run_comments = client.get_comments_for_agent_run("my-collection-id", "run-id-123")
for c in run_comments:
print(f"{c.get('created_by')}: {c.get('text')}")
