# Run a grader `POST /openai/v1/fine_tuning/alpha/graders/run` Runs the specified grader and returns its evaluation results. ## Request Body Content-Type: `application/json` *(required)* | Property | Type | Required | Description | |----------|------|----------|-------------| | `grader` | `OpenAI.GraderStringCheck \| OpenAI.GraderTextSimilarity \| OpenAI.GraderPython \| OpenAI.GraderScoreModel \| OpenAI.GraderMulti` | Yes | The grader used for the fine-tuning job. | | `item` | `OpenAI.RunGraderRequestItem` | No | The dataset item provided to the grader. This will be used to populate the `item` namespace. See [the guide](/docs/guides/graders) for more details. | | `model_sample` | `string` | Yes | The model sample to be evaluated. This value will be used to populate the `sample` namespace. See [the guide](/docs/guides/graders) for more details. The `output_json` variable will be populated if the model sample is a valid JSON string. | ### `grader` — `OpenAI.GraderStringCheck | OpenAI.GraderTextSimilarity | OpenAI.GraderPython | OpenAI.GraderScoreModel | OpenAI.GraderMulti` The grader used for the fine-tuning job. **One of the following:** #### `OpenAI.GraderStringCheck` — `OpenAI.GraderStringCheck` A StringCheckGrader object that performs a string comparison between input and reference using a specified operation. | Property | Type | Required | Description | |----------|------|----------|-------------| | `type` | `enum` | Yes | The object type, which is always `string_check`. _Constant:_ `string_check` | | `name` | `string` | Yes | The name of the grader. | | `input` | `string` | Yes | The input text. This may include template strings. | | `reference` | `string` | Yes | The reference text. This may include template strings. | | `operation` | `enum` | Yes | The string check operation to perform. One of `eq`, `ne`, `like`, or `ilike`. _Enum:_ `eq`, `ne`, `like`, `ilike` | #### `OpenAI.GraderTextSimilarity` — `OpenAI.GraderTextSimilarity` A TextSimilarityGrader object which grades text based on similarity metrics. | Property | Type | Required | Description | |----------|------|----------|-------------| | `type` | `enum` | Yes | The type of grader. _Default:_ `"text_similarity"` _Constant:_ `text_similarity` | | `name` | `string` | Yes | The name of the grader. | | `input` | `string` | Yes | The text being graded. | | `reference` | `string` | Yes | The text being graded against. | | `evaluation_metric` | `enum` | Yes | The evaluation metric to use. One of `cosine`, `fuzzy_match`, `bleu`, `gleu`, `meteor`, `rouge_1`, `rouge_2`, `rouge_3`, `rouge_4`, `rouge_5`, or `rouge_l`. _Enum:_ `cosine`, `fuzzy_match`, `bleu`, `gleu`, `meteor`, `rouge_1`, `rouge_2`, `rouge_3`, `rouge_4`, `rouge_5`, `rouge_l` | #### `OpenAI.GraderPython` — `OpenAI.GraderPython` A PythonGrader object that runs a python script on the input. | Property | Type | Required | Description | |----------|------|----------|-------------| | `type` | `enum` | Yes | The object type, which is always `python`. _Constant:_ `python` | | `name` | `string` | Yes | The name of the grader. | | `source` | `string` | Yes | The source code of the python script. | | `image_tag` | `string` | No | The image tag to use for the python script. | #### `OpenAI.GraderScoreModel` — `OpenAI.GraderScoreModel` A ScoreModelGrader object that uses a model to assign a score to the input. | Property | Type | Required | Description | |----------|------|----------|-------------| | `type` | `enum` | Yes | The object type, which is always `score_model`. _Constant:_ `score_model` | | `name` | `string` | Yes | The name of the grader. | | `model` | `string` | Yes | The model to use for the evaluation. | | `sampling_params` | `OpenAI.EvalGraderScoreModelSamplingParams` | No | The sampling parameters for the model. | | `input` | `OpenAI.EvalItem[]` | Yes | The input messages evaluated by the grader. Supports text, output text, input image, and input audio content blocks, and may include template strings. | | `range` | `OpenAI.numeric[]` | No | The range of the score. Defaults to `[0, 1]`. | ##### `sampling_params` — `OpenAI.EvalGraderScoreModelSamplingParams` The sampling parameters for the model. | Property | Type | Required | Description | |----------|------|----------|-------------| | `seed` | `OpenAI.integer \| null` | No | | | `top_p` | `OpenAI.numeric \| null` | No | _Default:_ `1` | | `temperature` | `OpenAI.numeric \| null` | No | | | `max_completions_tokens` | `OpenAI.integer \| null` | No | | | `reasoning_effort` | `enum` | No | Constrains effort on reasoning for [reasoning models](https://platform.openai.com/docs/guides/reasoning). Currently supported values are `none`, `minimal`, `low`, `medium`, `high`, and `xhigh`. Reducing reasoning effort can result in faster responses and fewer tokens used on reasoning in a response. - `gpt-5.1` defaults to `none`, which does not perform reasoning. The supported reasoning values for `gpt-5.1` are `none`, `low`, `medium`, and `high`. Tool calls are supported for all reasoning values in gpt-5.1. - All models before `gpt-5.1` default to `medium` reasoning effort, and do not support `none`. - The `gpt-5-pro` model defaults to (and only supports) `high` reasoning effort. - `xhigh` is supported for all models after `gpt-5.1-codex-max`. _Enum:_ `none`, `minimal`, `low`, `medium`, `high`, `xhigh` | ##### `input` — `OpenAI.EvalItem[]` The input messages evaluated by the grader. Supports text, output text, input image, and input audio content blocks, and may include template strings. **Array of** `OpenAI.EvalItem`**:** A message input to the model with a role indicating instruction following hierarchy. Instructions given with the `developer` or `system` role take precedence over instructions given with the `user` role. Messages with the `assistant` role are presumed to have been generated by the model in previous interactions. | Property | Type | Required | Description | |----------|------|----------|-------------| | `role` | `enum` | Yes | The role of the message input. One of `user`, `assistant`, `system`, or `developer`. _Enum:_ `user`, `assistant`, `system`, `developer` | | `content` | `OpenAI.EvalItemContentText \| OpenAI.EvalItemContentItemObject \| (OpenAI.EvalItemContentText \| OpenAI.EvalItemContentItemObject)[]` | Yes | Inputs to the model - can contain template strings. Supports text, output text, input images, and input audio, either as a single item or an array of items. | | `type` | `enum` | No | The type of the message input. Always `message`. _Constant:_ `message` | ###### `content` — `OpenAI.EvalItemContentText | OpenAI.EvalItemContentItemObject | (OpenAI.EvalItemContentText | OpenAI.EvalItemContentItemObject)[]` Inputs to the model - can contain template strings. Supports text, output text, input images, and input audio, either as a single item or an array of items. **One of the following:** - **`OpenAI.EvalItemContentItem`** — `OpenAI.EvalItemContentText | OpenAI.EvalItemContentItemObject` A single content item: input text, output text, input image, or input audio. **One of the following:** - **`OpenAI.EvalItemContentText`** — `OpenAI.EvalItemContentText` A text input to the model. | Type | Description | |------|-------------| | `OpenAI.EvalItemContentText` | A text input to the model. | - **`OpenAI.EvalItemContentItemObject`** — `OpenAI.EvalItemContentItemObject` A single content item: input text, output text, input image, or input audio. | Property | Type | Required | Description | |----------|------|----------|-------------| | `type` | `enum` | Yes | _Enum:_ `input_text`, `output_text`, `input_image`, `input_audio` | - **`OpenAI.EvalItemContentArray`** — `OpenAI.EvalItemContentArray ((OpenAI.EvalItemContentText | OpenAI.EvalItemContentItemObject)[])` A list of inputs, each of which may be either an input text, output text, input image, or input audio object. **One of the following:** - **`OpenAI.EvalItemContentText`** — `OpenAI.EvalItemContentText` A text input to the model. | Type | Description | |------|-------------| | `OpenAI.EvalItemContentText` | A text input to the model. | - **`OpenAI.EvalItemContentItemObject`** — `OpenAI.EvalItemContentItemObject` A single content item: input text, output text, input image, or input audio. | Property | Type | Required | Description | |----------|------|----------|-------------| | `type` | `enum` | Yes | _Enum:_ `input_text`, `output_text`, `input_image`, `input_audio` | #### `OpenAI.GraderMulti` — `OpenAI.GraderMulti` A MultiGrader object combines the output of multiple graders to produce a single score. | Property | Type | Required | Description | |----------|------|----------|-------------| | `type` | `enum` | Yes | The object type, which is always `multi`. _Default:_ `"multi"` _Constant:_ `multi` | | `name` | `string` | Yes | The name of the grader. | | `graders` | `OpenAI.GraderStringCheck \| OpenAI.GraderTextSimilarity \| OpenAI.GraderPython \| OpenAI.GraderScoreModel \| OpenAI.GraderLabelModel` | Yes | | | `calculate_output` | `string` | Yes | A formula to calculate the output based on grader results. | ##### `graders` — `OpenAI.GraderStringCheck | OpenAI.GraderTextSimilarity | OpenAI.GraderPython | OpenAI.GraderScoreModel | OpenAI.GraderLabelModel` **One of the following:** ###### `OpenAI.GraderStringCheck` — `OpenAI.GraderStringCheck` A StringCheckGrader object that performs a string comparison between input and reference using a specified operation. | Property | Type | Required | Description | |----------|------|----------|-------------| | `type` | `enum` | Yes | The object type, which is always `string_check`. _Constant:_ `string_check` | | `name` | `string` | Yes | The name of the grader. | | `input` | `string` | Yes | The input text. This may include template strings. | | `reference` | `string` | Yes | The reference text. This may include template strings. | | `operation` | `enum` | Yes | The string check operation to perform. One of `eq`, `ne`, `like`, or `ilike`. _Enum:_ `eq`, `ne`, `like`, `ilike` | ###### `OpenAI.GraderTextSimilarity` — `OpenAI.GraderTextSimilarity` A TextSimilarityGrader object which grades text based on similarity metrics. | Property | Type | Required | Description | |----------|------|----------|-------------| | `type` | `enum` | Yes | The type of grader. _Default:_ `"text_similarity"` _Constant:_ `text_similarity` | | `name` | `string` | Yes | The name of the grader. | | `input` | `string` | Yes | The text being graded. | | `reference` | `string` | Yes | The text being graded against. | | `evaluation_metric` | `enum` | Yes | The evaluation metric to use. One of `cosine`, `fuzzy_match`, `bleu`, `gleu`, `meteor`, `rouge_1`, `rouge_2`, `rouge_3`, `rouge_4`, `rouge_5`, or `rouge_l`. _Enum:_ `cosine`, `fuzzy_match`, `bleu`, `gleu`, `meteor`, `rouge_1`, `rouge_2`, `rouge_3`, `rouge_4`, `rouge_5`, `rouge_l` | ###### `OpenAI.GraderPython` — `OpenAI.GraderPython` A PythonGrader object that runs a python script on the input. | Property | Type | Required | Description | |----------|------|----------|-------------| | `type` | `enum` | Yes | The object type, which is always `python`. _Constant:_ `python` | | `name` | `string` | Yes | The name of the grader. | | `source` | `string` | Yes | The source code of the python script. | | `image_tag` | `string` | No | The image tag to use for the python script. | ###### `OpenAI.GraderScoreModel` — `OpenAI.GraderScoreModel` A ScoreModelGrader object that uses a model to assign a score to the input. | Property | Type | Required | Description | |----------|------|----------|-------------| | `type` | `enum` | Yes | The object type, which is always `score_model`. _Constant:_ `score_model` | | `name` | `string` | Yes | The name of the grader. | | `model` | `string` | Yes | The model to use for the evaluation. | | `sampling_params` | `OpenAI.EvalGraderScoreModelSamplingParams` | No | The sampling parameters for the model. | | `input` | `OpenAI.EvalItem[]` | Yes | The input messages evaluated by the grader. Supports text, output text, input image, and input audio content blocks, and may include template strings. | | `range` | `OpenAI.numeric[]` | No | The range of the score. Defaults to `[0, 1]`. | - **`sampling_params`** — `OpenAI.EvalGraderScoreModelSamplingParams` The sampling parameters for the model. | Property | Type | Required | Description | |----------|------|----------|-------------| | `seed` | `OpenAI.integer \| null` | No | | | `top_p` | `OpenAI.numeric \| null` | No | _Default:_ `1` | | `temperature` | `OpenAI.numeric \| null` | No | | | `max_completions_tokens` | `OpenAI.integer \| null` | No | | | `reasoning_effort` | `enum` | No | Constrains effort on reasoning for [reasoning models](https://platform.openai.com/docs/guides/reasoning). Currently supported values are `none`, `minimal`, `low`, `medium`, `high`, and `xhigh`. Reducing reasoning effort can result in faster responses and fewer tokens used on reasoning in a response. - `gpt-5.1` defaults to `none`, which does not perform reasoning. The supported reasoning values for `gpt-5.1` are `none`, `low`, `medium`, and `high`. Tool calls are supported for all reasoning values in gpt-5.1. - All models before `gpt-5.1` default to `medium` reasoning effort, and do not support `none`. - The `gpt-5-pro` model defaults to (and only supports) `high` reasoning effort. - `xhigh` is supported for all models after `gpt-5.1-codex-max`. _Enum:_ `none`, `minimal`, `low`, `medium`, `high`, `xhigh` | - **`input`** — `OpenAI.EvalItem[]` The input messages evaluated by the grader. Supports text, output text, input image, and input audio content blocks, and may include template strings. **Array of** `OpenAI.EvalItem`**:** A message input to the model with a role indicating instruction following hierarchy. Instructions given with the `developer` or `system` role take precedence over instructions given with the `user` role. Messages with the `assistant` role are presumed to have been generated by the model in previous interactions. | Property | Type | Required | Description | |----------|------|----------|-------------| | `role` | `enum` | Yes | The role of the message input. One of `user`, `assistant`, `system`, or `developer`. _Enum:_ `user`, `assistant`, `system`, `developer` | | `content` | `OpenAI.EvalItemContentText \| OpenAI.EvalItemContentItemObject \| (OpenAI.EvalItemContentText \| OpenAI.EvalItemContentItemObject)[]` | Yes | Inputs to the model - can contain template strings. Supports text, output text, input images, and input audio, either as a single item or an array of items. | | `type` | `enum` | No | The type of the message input. Always `message`. _Constant:_ `message` | - **`content`** — `OpenAI.EvalItemContentText | OpenAI.EvalItemContentItemObject | (OpenAI.EvalItemContentText | OpenAI.EvalItemContentItemObject)[]` Inputs to the model - can contain template strings. Supports text, output text, input images, and input audio, either as a single item or an array of items. **One of the following:** - **`OpenAI.EvalItemContentItem`** — `OpenAI.EvalItemContentText | OpenAI.EvalItemContentItemObject` A single content item: input text, output text, input image, or input audio. **One of the following:** - **`OpenAI.EvalItemContentText`** — `OpenAI.EvalItemContentText` A text input to the model. | Type | Description | |------|-------------| | `OpenAI.EvalItemContentText` | A text input to the model. | - **`OpenAI.EvalItemContentItemObject`** — `OpenAI.EvalItemContentItemObject` A single content item: input text, output text, input image, or input audio. | Property | Type | Required | Description | |----------|------|----------|-------------| | `type` | `enum` | Yes | _Enum:_ `input_text`, `output_text`, `input_image`, `input_audio` | - **`OpenAI.EvalItemContentArray`** — `OpenAI.EvalItemContentArray ((OpenAI.EvalItemContentText | OpenAI.EvalItemContentItemObject)[])` A list of inputs, each of which may be either an input text, output text, input image, or input audio object. **One of the following:** - **`OpenAI.EvalItemContentText`** — `OpenAI.EvalItemContentText` A text input to the model. | Type | Description | |------|-------------| | `OpenAI.EvalItemContentText` | A text input to the model. | - **`OpenAI.EvalItemContentItemObject`** — `OpenAI.EvalItemContentItemObject` A single content item: input text, output text, input image, or input audio. | Property | Type | Required | Description | |----------|------|----------|-------------| | `type` | `enum` | Yes | _Enum:_ `input_text`, `output_text`, `input_image`, `input_audio` | ###### `OpenAI.GraderLabelModel` — `OpenAI.GraderLabelModel` A LabelModelGrader object which uses a model to assign labels to each item in the evaluation. | Property | Type | Required | Description | |----------|------|----------|-------------| | `type` | `enum` | Yes | The object type, which is always `label_model`. _Constant:_ `label_model` | | `name` | `string` | Yes | The name of the grader. | | `model` | `string` | Yes | The model to use for the evaluation. Must support structured outputs. | | `input` | `OpenAI.EvalItem[]` | Yes | | | `labels` | `string[]` | Yes | The labels to assign to each item in the evaluation. | | `passing_labels` | `string[]` | Yes | The labels that indicate a passing result. Must be a subset of labels. | - **`input`** — `OpenAI.EvalItem[]` **Array of** `OpenAI.EvalItem`**:** A message input to the model with a role indicating instruction following hierarchy. Instructions given with the `developer` or `system` role take precedence over instructions given with the `user` role. Messages with the `assistant` role are presumed to have been generated by the model in previous interactions. | Property | Type | Required | Description | |----------|------|----------|-------------| | `role` | `enum` | Yes | The role of the message input. One of `user`, `assistant`, `system`, or `developer`. _Enum:_ `user`, `assistant`, `system`, `developer` | | `content` | `OpenAI.EvalItemContentText \| OpenAI.EvalItemContentItemObject \| (OpenAI.EvalItemContentText \| OpenAI.EvalItemContentItemObject)[]` | Yes | Inputs to the model - can contain template strings. Supports text, output text, input images, and input audio, either as a single item or an array of items. | | `type` | `enum` | No | The type of the message input. Always `message`. _Constant:_ `message` | - **`content`** — `OpenAI.EvalItemContentText | OpenAI.EvalItemContentItemObject | (OpenAI.EvalItemContentText | OpenAI.EvalItemContentItemObject)[]` Inputs to the model - can contain template strings. Supports text, output text, input images, and input audio, either as a single item or an array of items. **One of the following:** - **`OpenAI.EvalItemContentItem`** — `OpenAI.EvalItemContentText | OpenAI.EvalItemContentItemObject` A single content item: input text, output text, input image, or input audio. **One of the following:** - **`OpenAI.EvalItemContentText`** — `OpenAI.EvalItemContentText` A text input to the model. | Type | Description | |------|-------------| | `OpenAI.EvalItemContentText` | A text input to the model. | - **`OpenAI.EvalItemContentItemObject`** — `OpenAI.EvalItemContentItemObject` A single content item: input text, output text, input image, or input audio. | Property | Type | Required | Description | |----------|------|----------|-------------| | `type` | `enum` | Yes | _Enum:_ `input_text`, `output_text`, `input_image`, `input_audio` | - **`OpenAI.EvalItemContentArray`** — `OpenAI.EvalItemContentArray ((OpenAI.EvalItemContentText | OpenAI.EvalItemContentItemObject)[])` A list of inputs, each of which may be either an input text, output text, input image, or input audio object. **One of the following:** - **`OpenAI.EvalItemContentText`** — `OpenAI.EvalItemContentText` A text input to the model. | Type | Description | |------|-------------| | `OpenAI.EvalItemContentText` | A text input to the model. | - **`OpenAI.EvalItemContentItemObject`** — `OpenAI.EvalItemContentItemObject` A single content item: input text, output text, input image, or input audio. | Property | Type | Required | Description | |----------|------|----------|-------------| | `type` | `enum` | Yes | _Enum:_ `input_text`, `output_text`, `input_image`, `input_audio` | ## Responses ### 200 — The request has succeeded. Content-Type: `application/json` | Property | Type | Required | Description | |----------|------|----------|-------------| | `reward` | `OpenAI.numeric` | Yes | _Format:_ `double` | | `metadata` | `OpenAI.RunGraderResponseMetadata` | Yes | | | `sub_rewards` | `object` | Yes | | | `model_grader_token_usage_per_model` | `object` | Yes | | #### `metadata` — `OpenAI.RunGraderResponseMetadata` | Property | Type | Required | Description | |----------|------|----------|-------------| | `name` | `string` | Yes | | | `type` | `string` | Yes | | | `errors` | `OpenAI.RunGraderResponseMetadataErrors` | Yes | | | `execution_time` | `OpenAI.numeric` | Yes | _Format:_ `double` | | `scores` | `object` | Yes | | | `token_usage` | `OpenAI.integer \| null` | Yes | | | `sampled_model_name` | `string \| null` | Yes | | ##### `errors` — `OpenAI.RunGraderResponseMetadataErrors` | Property | Type | Required | Description | |----------|------|----------|-------------| | `formula_parse_error` | `boolean` | Yes | | | `sample_parse_error` | `boolean` | Yes | | | `truncated_observation_error` | `boolean` | Yes | | | `unresponsive_reward_error` | `boolean` | Yes | | | `invalid_variable_error` | `boolean` | Yes | | | `other_error` | `boolean` | Yes | | | `python_grader_server_error` | `boolean` | Yes | | | `python_grader_server_error_type` | `string \| null` | Yes | | | `python_grader_runtime_error` | `boolean` | Yes | | | `python_grader_runtime_error_details` | `string \| null` | Yes | | | `model_grader_server_error` | `boolean` | Yes | | | `model_grader_refusal_error` | `boolean` | Yes | | | `model_grader_parse_error` | `boolean` | Yes | | | `model_grader_server_error_details` | `string \| null` | Yes | | ### 4XX — Client error Content-Type: `application/json` Error response for API failures. | Property | Type | Required | Description | |----------|------|----------|-------------| | `error` | `OpenAI.Error` | Yes | | #### `error` — `OpenAI.Error` | Property | Type | Required | Description | |----------|------|----------|-------------| | `code` | `string \| null` | Yes | | | `message` | `string` | Yes | | | `param` | `string \| null` | No | | | `type` | `string` | No | | | `details` | `OpenAI.Error[]` | No | | | `additionalInfo` | `object` | No | | | `debugInfo` | `object` | No | | ### 5XX — Server error Content-Type: `application/json` Error response for API failures. | Property | Type | Required | Description | |----------|------|----------|-------------| | `error` | `OpenAI.Error` | Yes | | #### `error` — `OpenAI.Error` | Property | Type | Required | Description | |----------|------|----------|-------------| | `code` | `string \| null` | Yes | | | `message` | `string` | Yes | | | `param` | `string \| null` | No | | | `type` | `string` | No | | | `details` | `OpenAI.Error[]` | No | | | `additionalInfo` | `object` | No | | | `debugInfo` | `object` | No | | ## Code Examples ### Score text alignment #### cURL ```bash curl -X POST https://api.openai.com/v1/fine_tuning/alpha/graders/run \ -H "Content-Type: application/json" \ -H "Authorization: Bearer $OPENAI_API_KEY" \ -d '{ "grader": { "type": "score_model", "name": "Example score model grader", "input": [ { "role": "user", "content": [ { "type": "input_text", "text": "Score how close the reference answer is to the model answer on a 0-1 scale. Return only the score. Reference answer: {{item.reference_answer}} Model answer: {{sample.output_text}}" } ] } ], "model": "gpt-5-mini", "sampling_params": { "temperature": 1, "top_p": 1, "seed": 42 } }, "item": { "reference_answer": "fuzzy wuzzy was a bear" }, "model_sample": "fuzzy wuzzy was a bear" }' ``` #### Python ```python import os from openai import OpenAI client = OpenAI( api_key=os.environ.get("OPENAI_API_KEY"), # This is the default and can be omitted ) response = client.fine_tuning.alpha.graders.run( grader={ "input": "input", "name": "name", "operation": "eq", "reference": "reference", "type": "string_check", }, model_sample="model_sample", ) print(response.metadata) ``` #### JavaScript ```javascript import OpenAI from "openai"; const openai = new OpenAI(); const result = await openai.fineTuning.alpha.graders.run({ grader: { type: "score_model", name: "Example score model grader", input: [ { role: "user", content: [ { type: "input_text", text: "Score how close the reference answer is to the model answer on a 0-1 scale. Return only the score. Reference answer: {{item.reference_answer}} Model answer: {{sample.output_text}}", }, ], }, ], model: "gpt-5-mini", sampling_params: { temperature: 1, top_p: 1, seed: 42 }, }, item: { reference_answer: "fuzzy wuzzy was a bear" }, model_sample: "fuzzy wuzzy was a bear", }); console.log(result); ``` #### Node.js ```javascript import OpenAI from 'openai'; const client = new OpenAI({ apiKey: process.env['OPENAI_API_KEY'], // This is the default and can be omitted }); const response = await client.fineTuning.alpha.graders.run({ grader: { input: 'input', name: 'name', operation: 'eq', reference: 'reference', type: 'string_check', }, model_sample: 'model_sample', }); console.log(response.metadata); ``` #### Go ```go package main import ( "context" "fmt" "github.com/openai/openai-go" "github.com/openai/openai-go/option" ) func main() { client := openai.NewClient( option.WithAPIKey("My API Key"), ) response, err := client.FineTuning.Alpha.Graders.Run(context.TODO(), openai.FineTuningAlphaGraderRunParams{ Grader: openai.FineTuningAlphaGraderRunParamsGraderUnion{ OfStringCheck: &openai.StringCheckGraderParam{ Input: "input", Name: "name", Operation: openai.StringCheckGraderOperationEq, Reference: "reference", }, }, ModelSample: "model_sample", }) if err != nil { panic(err.Error()) } fmt.Printf("%+v ", response.Metadata) } ``` #### Java ```java package com.openai.example; import com.openai.client.OpenAIClient; import com.openai.client.okhttp.OpenAIOkHttpClient; import com.openai.models.finetuning.alpha.graders.GraderRunParams; import com.openai.models.finetuning.alpha.graders.GraderRunResponse; import com.openai.models.graders.gradermodels.StringCheckGrader; public final class Main { private Main() {} public static void main(String[] args) { OpenAIClient client = OpenAIOkHttpClient.fromEnv(); GraderRunParams params = GraderRunParams.builder() .grader(StringCheckGrader.builder() .input("input") .name("name") .operation(StringCheckGrader.Operation.EQ) .reference("reference") .build()) .modelSample("model_sample") .build(); GraderRunResponse response = client.fineTuning().alpha().graders().run(params); } } ``` #### Ruby ```ruby require "openai" openai = OpenAI::Client.new(api_key: "My API Key") response = openai.fine_tuning.alpha.graders.run( grader: {input: "input", name: "name", operation: :eq, reference: "reference", type: :string_check}, model_sample: "model_sample" ) puts(response) ``` #### Response ```json { "reward": 1.0, "metadata": { "name": "Example score model grader", "type": "score_model", "errors": { "formula_parse_error": false, "sample_parse_error": false, "truncated_observation_error": false, "unresponsive_reward_error": false, "invalid_variable_error": false, "other_error": false, "python_grader_server_error": false, "python_grader_server_error_type": null, "python_grader_runtime_error": false, "python_grader_runtime_error_details": null, "model_grader_server_error": false, "model_grader_refusal_error": false, "model_grader_parse_error": false, "model_grader_server_error_details": null }, "execution_time": 4.365238428115845, "scores": {}, "token_usage": { "prompt_tokens": 190, "total_tokens": 324, "completion_tokens": 134, "cached_tokens": 0 }, "sampled_model_name": "gpt-4o-2024-08-06" }, "sub_rewards": {}, "model_grader_token_usage_per_model": { "gpt-4o-2024-08-06": { "prompt_tokens": 190, "total_tokens": 324, "completion_tokens": 134, "cached_tokens": 0 } } } ``` ### Score an image caption #### cURL ```bash curl -X POST https://api.openai.com/v1/fine_tuning/alpha/graders/run \ -H "Content-Type: application/json" \ -H "Authorization: Bearer $OPENAI_API_KEY" \ -d '{ "grader": { "type": "score_model", "name": "Image caption grader", "input": [ { "role": "user", "content": [ { "type": "input_text", "text": "Score how well the provided caption matches the image on a 0-1 scale. Only return the score. Caption: {{sample.output_text}}" }, { "type": "input_image", "image_url": "https://example.com/dog-catching-ball.png", "file_id": null, "detail": "high" } ] } ], "model": "gpt-5-mini", "sampling_params": { "temperature": 0.2 } }, "item": { "expected_caption": "A golden retriever jumps to catch a tennis ball" }, "model_sample": "A dog leaps to grab a tennis ball mid-air" }' ``` #### Node.js ```javascript import OpenAI from 'openai'; const client = new OpenAI({ apiKey: process.env['OPENAI_API_KEY'], // This is the default and can be omitted }); const response = await client.fineTuning.alpha.graders.run({ grader: { input: 'input', name: 'name', operation: 'eq', reference: 'reference', type: 'string_check', }, model_sample: 'model_sample', }); console.log(response.metadata); ``` #### Python ```python import os from openai import OpenAI client = OpenAI( api_key=os.environ.get("OPENAI_API_KEY"), # This is the default and can be omitted ) response = client.fine_tuning.alpha.graders.run( grader={ "input": "input", "name": "name", "operation": "eq", "reference": "reference", "type": "string_check", }, model_sample="model_sample", ) print(response.metadata) ``` #### Go ```go package main import ( "context" "fmt" "github.com/openai/openai-go" "github.com/openai/openai-go/option" ) func main() { client := openai.NewClient( option.WithAPIKey("My API Key"), ) response, err := client.FineTuning.Alpha.Graders.Run(context.TODO(), openai.FineTuningAlphaGraderRunParams{ Grader: openai.FineTuningAlphaGraderRunParamsGraderUnion{ OfStringCheck: &openai.StringCheckGraderParam{ Input: "input", Name: "name", Operation: openai.StringCheckGraderOperationEq, Reference: "reference", }, }, ModelSample: "model_sample", }) if err != nil { panic(err.Error()) } fmt.Printf("%+v ", response.Metadata) } ``` #### Java ```java package com.openai.example; import com.openai.client.OpenAIClient; import com.openai.client.okhttp.OpenAIOkHttpClient; import com.openai.models.finetuning.alpha.graders.GraderRunParams; import com.openai.models.finetuning.alpha.graders.GraderRunResponse; import com.openai.models.graders.gradermodels.StringCheckGrader; public final class Main { private Main() {} public static void main(String[] args) { OpenAIClient client = OpenAIOkHttpClient.fromEnv(); GraderRunParams params = GraderRunParams.builder() .grader(StringCheckGrader.builder() .input("input") .name("name") .operation(StringCheckGrader.Operation.EQ) .reference("reference") .build()) .modelSample("model_sample") .build(); GraderRunResponse response = client.fineTuning().alpha().graders().run(params); } } ``` #### Ruby ```ruby require "openai" openai = OpenAI::Client.new(api_key: "My API Key") response = openai.fine_tuning.alpha.graders.run( grader: {input: "input", name: "name", operation: :eq, reference: "reference", type: :string_check}, model_sample: "model_sample" ) puts(response) ``` ### Score an audio response #### cURL ```bash curl -X POST https://api.openai.com/v1/fine_tuning/alpha/graders/run \ -H "Content-Type: application/json" \ -H "Authorization: Bearer $OPENAI_API_KEY" \ -d '{ "grader": { "type": "score_model", "name": "Audio clarity grader", "input": [ { "role": "user", "content": [ { "type": "input_text", "text": "Listen to the clip and return a confidence score from 0 to 1 that the speaker said: {{item.target_phrase}}" }, { "type": "input_audio", "input_audio": { "data": "{{item.audio_clip_b64}}", "format": "mp3" } } ] } ], "model": "gpt-audio", "sampling_params": { "temperature": 0.2, "top_p": 1, "seed": 123 } }, "item": { "target_phrase": "Please deliver the package on Tuesday", "audio_clip_b64": "" }, "model_sample": "Please deliver the package on Tuesday" }' ``` #### Node.js ```javascript import OpenAI from 'openai'; const client = new OpenAI({ apiKey: process.env['OPENAI_API_KEY'], // This is the default and can be omitted }); const response = await client.fineTuning.alpha.graders.run({ grader: { input: 'input', name: 'name', operation: 'eq', reference: 'reference', type: 'string_check', }, model_sample: 'model_sample', }); console.log(response.metadata); ``` #### Python ```python import os from openai import OpenAI client = OpenAI( api_key=os.environ.get("OPENAI_API_KEY"), # This is the default and can be omitted ) response = client.fine_tuning.alpha.graders.run( grader={ "input": "input", "name": "name", "operation": "eq", "reference": "reference", "type": "string_check", }, model_sample="model_sample", ) print(response.metadata) ``` #### Go ```go package main import ( "context" "fmt" "github.com/openai/openai-go" "github.com/openai/openai-go/option" ) func main() { client := openai.NewClient( option.WithAPIKey("My API Key"), ) response, err := client.FineTuning.Alpha.Graders.Run(context.TODO(), openai.FineTuningAlphaGraderRunParams{ Grader: openai.FineTuningAlphaGraderRunParamsGraderUnion{ OfStringCheck: &openai.StringCheckGraderParam{ Input: "input", Name: "name", Operation: openai.StringCheckGraderOperationEq, Reference: "reference", }, }, ModelSample: "model_sample", }) if err != nil { panic(err.Error()) } fmt.Printf("%+v ", response.Metadata) } ``` #### Java ```java package com.openai.example; import com.openai.client.OpenAIClient; import com.openai.client.okhttp.OpenAIOkHttpClient; import com.openai.models.finetuning.alpha.graders.GraderRunParams; import com.openai.models.finetuning.alpha.graders.GraderRunResponse; import com.openai.models.graders.gradermodels.StringCheckGrader; public final class Main { private Main() {} public static void main(String[] args) { OpenAIClient client = OpenAIOkHttpClient.fromEnv(); GraderRunParams params = GraderRunParams.builder() .grader(StringCheckGrader.builder() .input("input") .name("name") .operation(StringCheckGrader.Operation.EQ) .reference("reference") .build()) .modelSample("model_sample") .build(); GraderRunResponse response = client.fineTuning().alpha().graders().run(params); } } ``` #### Ruby ```ruby require "openai" openai = OpenAI::Client.new(api_key: "My API Key") response = openai.fine_tuning.alpha.graders.run( grader: {input: "input", name: "name", operation: :eq, reference: "reference", type: :string_check}, model_sample: "model_sample" ) puts(response) ```