Docent provides several annotation mechanisms:
Label sets — structured annotations on agent runs, validated against a JSON schemaGeneral label sets — structured annotations on agent runs, transcripts, transcript slices, or reading resultsTags — lightweight string annotations on agent runsComments — free-text notes on agent runs
See Labeling Agent Runs for a tutorial.
Label Sets Create a Label Set Define a new label set with a JSON schema that all labels must conform to.
from docent import Docent client = Docent() label_set_id = client.create_label_set( "my-collection-id" , name = "Quality Assessment" , description = "Human quality ratings for agent responses" , label_schema = { "type" : "object" , "properties" : { "quality" : { "type" : "string" , "enum" : [ "good" , "acceptable" , "poor" ]}, "notes" : { "type" : "string" }, }, "required" : [ "quality" ], }, )
Parameters Display name for the label set.
JSON schema for validating labels. Must be a valid JSON Schema object.
Returns ID of the created label set.
List Label Sets label_sets = client.get_label_sets( "my-collection-id" ) for ls in label_sets: print ( f " { ls[ 'id' ] } : { ls[ 'name' ] } " )
Add a Label from docent.data_models.judge import Label label = Label( label_set_id = label_set_id, agent_run_id = "run-id-123" , label_value = { "quality" : "good" , "notes" : "Clear and helpful response" }, ) result = client.add_label( "my-collection-id" , label)
Add Multiple Labels labels = [ Label( label_set_id = label_set_id, agent_run_id = run_id, label_value = { "quality" : "good" }, ) for run_id in run_ids ] result = client.add_labels( "my-collection-id" , labels)
Get Labels labels = client.get_labels( "my-collection-id" , label_set_id) # Only get labels that pass schema validation (including required fields) valid_labels = client.get_labels( "my-collection-id" , label_set_id, filter_valid_labels = True , )
Parameters If True, only return labels that fully match the label set schema including
required fields. Default returns all labels.
Update a Label Update an existing label’s label_value. The server validates the updated value against
the label set schema, but does not enforce top-level required fields for regular labels.
It also verifies that the label belongs to the given label set.
updated = client.update_label( "my-collection-id" , label_set_id, label_id = "label-123" , label_value = { "quality" : "acceptable" , "notes" : "Resolved after review" }, ) print (updated[ "id" ])
Parameters ID of the label set that owns the label.
ID of the label to update.
New label value. It must conform to the label set’s JSON schema validation rules,
except top-level required fields are not enforced for regular labels.
Returns Delete Labels Preview or delete specific labels from a label set. Deletion defaults to a dry run so
you can inspect the returned labels before mutating data.
preview = client.delete_labels( "my-collection-id" , label_set_id, [ "label-123" , "label-456" ], ) for label in preview.labels: print (label.id, label.label_value) # Perform the deletion after reviewing the preview. result = client.delete_labels( "my-collection-id" , label_set_id, [ "label-123" , "label-456" ], dry_run = False , ) print (result.message)
Parameters ID of the label set that owns all labels being deleted.
Non-empty list of unique label IDs.
If True, return the labels that would be deleted without deleting them.
Pass False to perform the deletion.
Returns Structured deletion preview or result. Show DeleteLabelsResult fields
Whether this call was a dry run.
Whether labels were actually deleted.
Number of matched labels.
labels
list[DeletedLabelPreview]
Preview records for the matched labels.
Server message describing the result.
DeleteLabelsResult and DeletedLabelPreview are exported from docent.sdk for typing:from docent.sdk import DeletedLabelPreview, DeleteLabelsResult
DeletedLabelPreview records include:Agent run ID for regular labels.
General-label target, when deleting general labels.
Errors
ValueErrorlabel_ids is empty or contains duplicatesHTTPError (404)
General Labels General labels use the same schema validation model as label sets, but they can
target several Docent object types instead of only one agent run.
Create a General Label Set general_label_set = client.create_general_label_set( "my-collection-id" , name = "Review Findings" , description = "Annotations over runs, transcript slices, and reading results" , label_schema = { "type" : "object" , "properties" : { "severity" : { "type" : "string" , "enum" : [ "low" , "medium" , "high" ]}, "summary" : { "type" : "string" }, }, "required" : [ "severity" , "summary" ], }, metadata = { "source" : "human-review" }, )
List and Fetch General Label Sets general_label_sets = client.get_general_label_sets( "my-collection-id" ) general_label_set = client.get_general_label_set( "my-collection-id" , general_label_set_id, )
Create a General Label general_label = client.create_general_label( "my-collection-id" , general_label_set_id, target = { "items" : [ { "object_type" : "transcript_slice" , "transcript_id" : "transcript-123" , "transcript_slice_start_idx" : 4 , "transcript_slice_end_idx" : 8 , } ] }, label_value = { "severity" : "medium" , "summary" : "The answer misses a constraint from the prompt." , }, )
target.items can contain agent_run, transcript, transcript_slice, or
reading_result targets.Get a General Label general_label = client.get_general_label( "my-collection-id" , label_id = "label-123" , ) print (general_label.label_value)
Update a General Label Update a general label’s label_value and optionally replace its metadata. The
server verifies that the label belongs to the supplied general label set.
updated = client.update_general_label( "my-collection-id" , general_label_set_id, label_id = "label-123" , label_value = { "severity" : "high" , "summary" : "The response violates the core task requirement." , }, metadata = { "reviewed_by" : "human-reviewer" }, ) print (updated.updated_at)
Parameters ID of the general label set that owns the label.
ID of the general label to update.
New label value. It must conform to the general label set’s JSON schema.
Optional metadata to store on the label. If omitted, existing metadata is unchanged.
Returns Delete General Labels Preview or delete specific general labels from a general label set. Like
delete_labels, this method defaults to a dry run.
preview = client.delete_general_labels( "my-collection-id" , general_label_set_id, [ "label-123" , "label-456" ], ) for label in preview.labels: print (label.id, label.target) result = client.delete_general_labels( "my-collection-id" , general_label_set_id, [ "label-123" , "label-456" ], dry_run = False , ) print (result.message)
Parameters ID of the general label set that owns all labels being deleted.
Non-empty list of unique general label IDs.
If True, return the general labels that would be deleted without deleting
them. Pass False to perform the deletion.
Returns Structured deletion preview or result. See Delete Labels for the result fields.
Errors
ValueErrorlabel_ids is empty or contains duplicatesHTTPError (404)
Lightweight string annotations on agent runs.
Add a Tag client.tag_transcript( "my-collection-id" , "run-id-123" , "needs-review" )
# All tags in a collection all_tags = client.get_tags( "my-collection-id" ) # Filter by value review_tags = client.get_tags( "my-collection-id" , value = "needs-review" ) # Tags for a specific run run_tags = client.get_tags_for_agent_run( "my-collection-id" , "run-id-123" )
Delete a Tag client.delete_tag( "my-collection-id" , tag_id = "tag-456" )
Free-text notes on agent runs.
# All comments in a collection comments = client.get_comments( "my-collection-id" ) # Comments for a specific run run_comments = client.get_comments_for_agent_run( "my-collection-id" , "run-id-123" ) for c in run_comments: print ( f " { c.get( 'created_by' ) } : { c.get( 'text' ) } " )
