# Create an evaluator version

`POST /evaluators/{name}/versions`

Creates a new evaluator version with an auto-incremented version identifier.

## Parameters

### Path parameters

| Name | Type | Required | Description |
|------|------|----------|-------------|
| `name` | `string` | Yes | The name of the resource |

### Other parameters

| Name | Type | Required | Description |
|------|------|----------|-------------|
| `undefined` | `string` | No | — |

## Request Body

Content-Type: `application/json` *(required)*

Evaluator Definition

| Property | Type | Required | Description |
|----------|------|----------|-------------|
| `display_name` | `string` | No | Display Name for evaluator. It helps to find the evaluator easily in AI Foundry. It does not need to be unique. |
| `metadata` | `object` | No | Metadata about the evaluator |
| `evaluator_type` | `enum` | Yes | The type of the evaluator _Enum:_ `builtin`, `custom` |
| `categories` | `enum[]` | Yes | The categories of the evaluator _Enum:_ `quality`, `safety`, `agents` |
| `supported_evaluation_levels` | `enum[]` | No | Evaluation levels this evaluator supports (e.g., `turn`, `conversation`). When omitted on create, the service defaults to `["turn"]`. On update, omitting this field leaves it unchanged; an empty list is rejected. Custom code-based evaluators support only `turn`; custom prompt-based evaluators support exactly one level (`turn` or `conversation`). _Default:_ `["turn"]` _Enum:_ `turn`, `conversation` |
| `definition` | `CodeBasedEvaluatorDefinition \| PromptBasedEvaluatorDefinition \| RubricBasedEvaluatorDefinition \| EndpointBasedEvaluatorDefinition` | Yes | Base evaluator configuration with discriminator |
| `description` | `string` | No | The asset description text. |
| `tags` | `object` | No | Tag dictionary. Tags can be added, removed, and updated. |

### `definition` — `CodeBasedEvaluatorDefinition | PromptBasedEvaluatorDefinition | RubricBasedEvaluatorDefinition | EndpointBasedEvaluatorDefinition`

Base evaluator configuration with discriminator

**One of the following:**

#### `code` — `code (CodeBasedEvaluatorDefinition)`

Code-based evaluator definition using python code

| Property | Type | Required | Description |
|----------|------|----------|-------------|
| `type` | `enum` | Yes | _Constant:_ `code` |
| `init_parameters` | `object` | No | The JSON schema (Draft 2020-12) for the evaluator's input parameters. This includes parameters like type, properties, required. |
| `data_schema` | `object` | No | The JSON schema (Draft 2020-12) for the evaluator's input data. This includes parameters like type, properties, required. |
| `metrics` | `object` | No | List of output metrics produced by this evaluator |
| `code_text` | `string` | No | Inline code text for the evaluator |
| `entry_point` | `string` | No | The entry point Python file name for the uploaded evaluator code (e.g. 'answer_length_evaluator.py') |
| `image_tag` | `string` | No | The container image tag to use for evaluator code execution |
| `blob_uri` | `string` | No | The blob URI for the evaluator storage _Format:_ `uri` |

#### `prompt` — `prompt (PromptBasedEvaluatorDefinition)`

Prompt-based evaluator

| Property | Type | Required | Description |
|----------|------|----------|-------------|
| `type` | `enum` | Yes | _Constant:_ `prompt` |
| `init_parameters` | `object` | No | The JSON schema (Draft 2020-12) for the evaluator's input parameters. This includes parameters like type, properties, required. |
| `data_schema` | `object` | No | The JSON schema (Draft 2020-12) for the evaluator's input data. This includes parameters like type, properties, required. |
| `metrics` | `object` | No | List of output metrics produced by this evaluator |
| `prompt_text` | `string` | Yes | The prompt text used for evaluation |

#### `rubric` — `rubric (RubricBasedEvaluatorDefinition)`

Rubric-based evaluator definition — stores dimensions produced by the generate API. Used for both quality and safety evaluators.

| Property | Type | Required | Description |
|----------|------|----------|-------------|
| `type` | `enum` | Yes | _Constant:_ `rubric` |
| `init_parameters` | `object` | No | The JSON schema (Draft 2020-12) for the evaluator's input parameters. This includes parameters like type, properties, required. |
| `data_schema` | `object` | No | The JSON schema (Draft 2020-12) for the evaluator's input data. This includes parameters like type, properties, required. |
| `metrics` | `object` | No | List of output metrics produced by this evaluator |
| `dimensions` | `Dimension[]` | Yes | The set of dimensions — the scoring blueprint used by the LLM judge. Quality evaluators include a non-editable residual dimension with id 'general_quality' (always_applicable: true); safety evaluators include 'general_policy_compliance'. Both use the same Dimension structure. |
| `pass_threshold` | `number` | No | Pass/fail threshold for the aggregate rubric score, on the same normalized 0.0-1.0 scale as the emitted `score`. When the runtime weighted average meets or exceeds this value, the result is `pass`. Defaults to 0.5 (equivalent to a raw 1-5 weighted average of 3.0). The 'any dimension scored 1 → fail' rule still applies regardless of this threshold. _Default:_ `0.5` _Format:_ `float` |

##### `dimensions` — `Dimension[]`

The set of dimensions — the scoring blueprint used by the LLM judge. Quality evaluators include a non-editable residual dimension with id 'general_quality' (always_applicable: true); safety evaluators include 'general_policy_compliance'. Both use the same Dimension structure.

**Array of** `Dimension`**:**

A single dimension — one independent, measurable quality dimension within a rubric evaluator's scoring blueprint.

| Property | Type | Required | Description |
|----------|------|----------|-------------|
| `id` | `string` | Yes | Stable identifier for this dimension (snake_case, e.g., `correct_resolution`). Required. Provided by the user when manually creating a rubric evaluator or during human-in-the-loop review of a generated set; the generation pipeline produces an initial value the user can edit. Editable when saving new versions. |
| `description` | `string` | Yes | What this dimension measures (e.g., 'Correctly identifies the user's reservation intent and pursues the appropriate workflow'). |
| `weight` | `integer` | Yes | Relative weight of this dimension (1-10). The generation pipeline assigns exactly one dimension weight 8-10; all others use 1-6. User edits are not constrained by this heuristic. _Format:_ `int32` |
| `always_applicable` | `boolean` | No | When true, the LLM judge always scores this dimension regardless of relevance (skips applicability assessment). The service-generated general quality/policy dimension has this set to true and is non-editable. Users may set this on their own custom dimensions. The service defaults to `false` if a value is not specified by the caller. _Default:_ `false` |

#### `endpoint` — `endpoint (EndpointBasedEvaluatorDefinition)`

Endpoint-based evaluator definition. The customer owns and hosts an HTTP endpoint that implements the evaluation contract. The evaluator references a Project Connection by name; the connection stores the endpoint URL and credentials (API Key or Entra ID). At execution time, the service resolves the connection to obtain the endpoint URL and authentication details, then calls the endpoint for each evaluation row.

| Property | Type | Required | Description |
|----------|------|----------|-------------|
| `type` | `enum` | Yes | _Constant:_ `endpoint` |
| `init_parameters` | `object` | No | The JSON schema (Draft 2020-12) for the evaluator's input parameters. This includes parameters like type, properties, required. |
| `data_schema` | `object` | No | The JSON schema (Draft 2020-12) for the evaluator's input data. This includes parameters like type, properties, required. |
| `metrics` | `object` | No | List of output metrics produced by this evaluator |
| `connection_name` | `string` | Yes | Name of the Project Connection that stores the endpoint URL and credentials. The connection must exist on the project and have a non-empty target URL. Supported auth types: ApiKey (sends `api-key` header) and AAD/Entra ID (acquires a bearer token via the project's Managed Identity). |

## Responses

### 201 — The request has succeeded and a new resource has been created as a result.

Content-Type: `application/json`

Evaluator Definition

| Property | Type | Required | Description |
|----------|------|----------|-------------|
| `display_name` | `string` | No | Display Name for evaluator. It helps to find the evaluator easily in AI Foundry. It does not need to be unique. |
| `metadata` | `object` | No | Metadata about the evaluator |
| `evaluator_type` | `enum` | Yes | The type of the evaluator _Enum:_ `builtin`, `custom` |
| `categories` | `enum[]` | Yes | The categories of the evaluator _Enum:_ `quality`, `safety`, `agents` |
| `supported_evaluation_levels` | `enum[]` | No | Evaluation levels this evaluator supports (e.g., `turn`, `conversation`). When omitted on create, the service defaults to `["turn"]`. On update, omitting this field leaves it unchanged; an empty list is rejected. Custom code-based evaluators support only `turn`; custom prompt-based evaluators support exactly one level (`turn` or `conversation`). _Default:_ `["turn"]` _Enum:_ `turn`, `conversation` |
| `definition` | `CodeBasedEvaluatorDefinition \| PromptBasedEvaluatorDefinition \| RubricBasedEvaluatorDefinition \| EndpointBasedEvaluatorDefinition` | Yes | Base evaluator configuration with discriminator |
| `generation_artifacts` | `EvaluatorGenerationArtifacts` | No | Service-managed provenance artifacts produced by an evaluator generation job. Present only on EvaluatorVersion resources created via the generation pipeline. The combined-JSONL Foundry Dataset is read-only and resolves to a versioned dataset in a service-reserved namespace. |
| `generation_job_id` | `string` | No | Read-only provenance link back to the EvaluatorGenerationJob that produced this version. Present only on evaluator versions created via the generation pipeline; absent for manually-created versions and unaffected by subsequent `PATCH` calls. |
| `warnings` | `enum[]` | No | Categories of warnings surfaced on this generated evaluator version. Present only on versions created via an EvaluatorGenerationJob when the paired job produced non-empty warnings. Absent (treat as no warnings) when the version is not from generation, when the paired job was clean, or when a subsequent `PATCH` to `definition` cleared the paired job's advisories. Follow `generation_job_id` to fetch the detailed warning payloads. _Constant:_ `input_quality` |
| `created_by` | `string` | Yes | Creator of the evaluator |
| `created_at` | `string` | Yes | Creation date/time of the evaluator |
| `modified_at` | `string` | Yes | Last modified date/time of the evaluator |
| `id` | `string` | No | Asset ID, a unique identifier for the asset |
| `name` | `string` | Yes | The name of the resource |
| `version` | `string` | Yes | The version of the resource |

#### `definition` — `CodeBasedEvaluatorDefinition | PromptBasedEvaluatorDefinition | RubricBasedEvaluatorDefinition | EndpointBasedEvaluatorDefinition`

Base evaluator configuration with discriminator

**One of the following:**

##### `code` — `code (CodeBasedEvaluatorDefinition)`

Code-based evaluator definition using python code

| Property | Type | Required | Description |
|----------|------|----------|-------------|
| `type` | `enum` | Yes | _Constant:_ `code` |
| `init_parameters` | `object` | No | The JSON schema (Draft 2020-12) for the evaluator's input parameters. This includes parameters like type, properties, required. |
| `data_schema` | `object` | No | The JSON schema (Draft 2020-12) for the evaluator's input data. This includes parameters like type, properties, required. |
| `metrics` | `object` | No | List of output metrics produced by this evaluator |
| `code_text` | `string` | No | Inline code text for the evaluator |
| `entry_point` | `string` | No | The entry point Python file name for the uploaded evaluator code (e.g. 'answer_length_evaluator.py') |
| `image_tag` | `string` | No | The container image tag to use for evaluator code execution |
| `blob_uri` | `string` | No | The blob URI for the evaluator storage _Format:_ `uri` |

##### `prompt` — `prompt (PromptBasedEvaluatorDefinition)`

Prompt-based evaluator

| Property | Type | Required | Description |
|----------|------|----------|-------------|
| `type` | `enum` | Yes | _Constant:_ `prompt` |
| `init_parameters` | `object` | No | The JSON schema (Draft 2020-12) for the evaluator's input parameters. This includes parameters like type, properties, required. |
| `data_schema` | `object` | No | The JSON schema (Draft 2020-12) for the evaluator's input data. This includes parameters like type, properties, required. |
| `metrics` | `object` | No | List of output metrics produced by this evaluator |
| `prompt_text` | `string` | Yes | The prompt text used for evaluation |

##### `rubric` — `rubric (RubricBasedEvaluatorDefinition)`

Rubric-based evaluator definition — stores dimensions produced by the generate API. Used for both quality and safety evaluators.

| Property | Type | Required | Description |
|----------|------|----------|-------------|
| `type` | `enum` | Yes | _Constant:_ `rubric` |
| `init_parameters` | `object` | No | The JSON schema (Draft 2020-12) for the evaluator's input parameters. This includes parameters like type, properties, required. |
| `data_schema` | `object` | No | The JSON schema (Draft 2020-12) for the evaluator's input data. This includes parameters like type, properties, required. |
| `metrics` | `object` | No | List of output metrics produced by this evaluator |
| `dimensions` | `Dimension[]` | Yes | The set of dimensions — the scoring blueprint used by the LLM judge. Quality evaluators include a non-editable residual dimension with id 'general_quality' (always_applicable: true); safety evaluators include 'general_policy_compliance'. Both use the same Dimension structure. |
| `pass_threshold` | `number` | No | Pass/fail threshold for the aggregate rubric score, on the same normalized 0.0-1.0 scale as the emitted `score`. When the runtime weighted average meets or exceeds this value, the result is `pass`. Defaults to 0.5 (equivalent to a raw 1-5 weighted average of 3.0). The 'any dimension scored 1 → fail' rule still applies regardless of this threshold. _Default:_ `0.5` _Format:_ `float` |

###### `dimensions` — `Dimension[]`

The set of dimensions — the scoring blueprint used by the LLM judge. Quality evaluators include a non-editable residual dimension with id 'general_quality' (always_applicable: true); safety evaluators include 'general_policy_compliance'. Both use the same Dimension structure.

**Array of** `Dimension`**:**

A single dimension — one independent, measurable quality dimension within a rubric evaluator's scoring blueprint.

| Property | Type | Required | Description |
|----------|------|----------|-------------|
| `id` | `string` | Yes | Stable identifier for this dimension (snake_case, e.g., `correct_resolution`). Required. Provided by the user when manually creating a rubric evaluator or during human-in-the-loop review of a generated set; the generation pipeline produces an initial value the user can edit. Editable when saving new versions. |
| `description` | `string` | Yes | What this dimension measures (e.g., 'Correctly identifies the user's reservation intent and pursues the appropriate workflow'). |
| `weight` | `integer` | Yes | Relative weight of this dimension (1-10). The generation pipeline assigns exactly one dimension weight 8-10; all others use 1-6. User edits are not constrained by this heuristic. _Format:_ `int32` |
| `always_applicable` | `boolean` | No | When true, the LLM judge always scores this dimension regardless of relevance (skips applicability assessment). The service-generated general quality/policy dimension has this set to true and is non-editable. Users may set this on their own custom dimensions. The service defaults to `false` if a value is not specified by the caller. _Default:_ `false` |

##### `endpoint` — `endpoint (EndpointBasedEvaluatorDefinition)`

Endpoint-based evaluator definition. The customer owns and hosts an HTTP endpoint that implements the evaluation contract. The evaluator references a Project Connection by name; the connection stores the endpoint URL and credentials (API Key or Entra ID). At execution time, the service resolves the connection to obtain the endpoint URL and authentication details, then calls the endpoint for each evaluation row.

| Property | Type | Required | Description |
|----------|------|----------|-------------|
| `type` | `enum` | Yes | _Constant:_ `endpoint` |
| `init_parameters` | `object` | No | The JSON schema (Draft 2020-12) for the evaluator's input parameters. This includes parameters like type, properties, required. |
| `data_schema` | `object` | No | The JSON schema (Draft 2020-12) for the evaluator's input data. This includes parameters like type, properties, required. |
| `metrics` | `object` | No | List of output metrics produced by this evaluator |
| `connection_name` | `string` | Yes | Name of the Project Connection that stores the endpoint URL and credentials. The connection must exist on the project and have a non-empty target URL. Supported auth types: ApiKey (sends `api-key` header) and AAD/Entra ID (acquires a bearer token via the project's Managed Identity). |

#### `generation_artifacts` — `EvaluatorGenerationArtifacts`

Service-managed provenance artifacts produced by an evaluator generation job. Present only on EvaluatorVersion resources created via the generation pipeline. The combined-JSONL Foundry Dataset is read-only and resolves to a versioned dataset in a service-reserved namespace.

| Property | Type | Required | Description |
|----------|------|----------|-------------|
| `dataset` | `DatasetReference` | Yes | Reference to a versioned Foundry Dataset. |
| `kinds` | `string[]` | Yes | The kinds of rows present in `dataset`. Always contains `"spec"` (the generated evaluation specification, a Markdown document describing what the evaluator measures). May additionally contain `"tools"` (when the generation pipeline produced or inferred OpenAI tool schemas) and/or `"context"` (when supplementary materials such as file uploads or trace samples were used during generation). |

##### `dataset` — `DatasetReference`

Reference to a versioned Foundry Dataset.

| Property | Type | Required | Description |
|----------|------|----------|-------------|
| `name` | `string` | Yes | Dataset name. |
| `version` | `string` | Yes | Dataset version. |

### 4XX — Client error

Content-Type: `application/json`

A response containing error details.

| Property | Type | Required | Description |
|----------|------|----------|-------------|
| `error` | `Azure.Core.Foundations.Error` | Yes | The error object. |

#### `error` — `Azure.Core.Foundations.Error`

The error object.

| Property | Type | Required | Description |
|----------|------|----------|-------------|
| `code` | `string` | Yes | One of a server-defined set of error codes. |
| `message` | `string` | Yes | A human-readable representation of the error. |
| `target` | `string` | No | The target of the error. |
| `details` | `Azure.Core.Foundations.Error[]` | No | An array of details about specific errors that led to this reported error. |
| `innererror` | `Azure.Core.Foundations.InnerError` | No | An object containing more specific information about the error. As per Azure REST API guidelines - https://aka.ms/AzureRestApiGuidelines#handling-errors. |

##### `innererror` — `Azure.Core.Foundations.InnerError`

An object containing more specific information about the error. As per Azure REST API guidelines - https://aka.ms/AzureRestApiGuidelines#handling-errors.

| Property | Type | Required | Description |
|----------|------|----------|-------------|
| `code` | `string` | No | One of a server-defined set of error codes. |
| `innererror` | `Azure.Core.Foundations.InnerError` | No | (circular reference) |

### 5XX — Server error

Content-Type: `application/json`

A response containing error details.

| Property | Type | Required | Description |
|----------|------|----------|-------------|
| `error` | `Azure.Core.Foundations.Error` | Yes | The error object. |

#### `error` — `Azure.Core.Foundations.Error`

The error object.

| Property | Type | Required | Description |
|----------|------|----------|-------------|
| `code` | `string` | Yes | One of a server-defined set of error codes. |
| `message` | `string` | Yes | A human-readable representation of the error. |
| `target` | `string` | No | The target of the error. |
| `details` | `Azure.Core.Foundations.Error[]` | No | An array of details about specific errors that led to this reported error. |
| `innererror` | `Azure.Core.Foundations.InnerError` | No | An object containing more specific information about the error. As per Azure REST API guidelines - https://aka.ms/AzureRestApiGuidelines#handling-errors. |

##### `innererror` — `Azure.Core.Foundations.InnerError`

An object containing more specific information about the error. As per Azure REST API guidelines - https://aka.ms/AzureRestApiGuidelines#handling-errors.

| Property | Type | Required | Description |
|----------|------|----------|-------------|
| `code` | `string` | No | One of a server-defined set of error codes. |
| `innererror` | `Azure.Core.Foundations.InnerError` | No | (circular reference) |