Skip to main content
Evaluation

Model evaluation overview

Evaluate model performance across multiple dimensions on QwenCloud, with LLM-based scoring and manual annotation.

Model evaluation quantifies model output quality across multiple dimensions. You can create evaluation dimensions and tasks on the Model Evaluation console page.
Model evaluation is available only in Singapore and China North 2 (Beijing). It is not available in China (Hong Kong), Germany (Frankfurt), Japan (Tokyo), or US (Virginia), and its menu is hidden in the console in those regions. To use it, switch to Singapore or China North 2 (Beijing) with the region selector in the upper-right corner of the console. See Feature availability by region.
Evaluation jobs and their results belong to the region where you run them and cannot be used in another region.

Core concepts

Model evaluation consists of two sub-modules:
  • Evaluation dimensions — Define the scoring criteria. Supports LLM-based scoring (numeric / classification) and manual annotation.
  • Evaluation tasks — Select dimensions + dataset + model to evaluate, then view results.

Evaluation workflow

1

Define evaluation dimensions

Create one or more evaluation dimensions, choosing the dimension type, judge model, and scoring template. See Evaluation dimensions.
2

Prepare evaluation data

Create and publish an evaluation dataset in Datasets.
3

Create an evaluation task

Select the data source, evaluation model, and evaluation dimensions, then submit the task. See Evaluation tasks.
4

View results

Check scores, token usage, and per-sample scoring details in the task list.

Billing

Model evaluation costs consist of two parts:
Cost typeBilled?Description
Evaluated model inferenceYesWhen using an "Evaluation dataset" as the data source, the system calls the evaluated model to run inference, billed at standard token rates. No inference cost when using "Inference results".
Judge model scoringFree (limited time)The judge model scoring process for LLM-based dimensions (numeric and classification) is currently free for a limited time.

Next steps