Create evaluation tasks on QwenCloud to quantify model output quality using custom dimensions and datasets.
Evaluation tasks apply evaluation dimensions to specific data and models, producing quantified results. You can create and manage tasks on the Evaluation Tasks console page.
The evaluation task list displays the following information:
Click Details in the actions column to open the results page, which has two tabs:
Create an evaluation task
1
Open the create page
On the evaluation tasks list page, click Create. The system auto-generates a task name (format:
eval_YYYYMMDD_HHMMSS), which you can modify.2
Select the data source
Choose a data source type from the dropdown. The two sources differ in upload entry point and subsequent configuration:
| Data source | Upload entry | Description | Data format (JSONL) | Use cases |
|---|---|---|---|---|
| Evaluation dataset | Create and publish on the Datasets page | The system runs inference with the evaluation model, then scores with evaluation dimensions | Prompt (question) + Completion (reference answer) | First-time evaluation, comparing multiple models |
| Inference results | Upload on the Evaluation page | Scores directly with evaluation dimensions, skipping inference | Prompt + Output (model output), Completion (reference answer) optional | Already have inference results, reducing inference cost |
3
Select data and configure
The configuration options shown vary by data source:
- Evaluation dataset: Choose a published evaluation dataset from the dropdown, then select the evaluation model to run inference on the dataset. Click Manage datasets to create or publish a dataset.
- Inference results: Upload or select existing inference results. No evaluation model is needed (the results already contain model output) — proceed directly to selecting evaluation dimensions.
Recommended data volume: 50-100 samples for initial validation, 200-500 for reliable statistical results, and 500+ for comprehensive evaluation covering edge cases.
4
Select evaluation dimensions
Choose one or more evaluation dimensions. Click Manage dimensions to go to the dimensions management page.
5
Review billing details
Expand Billing details to view estimated costs. The scoring process is free; inference is billed at standard token rates.
6
Submit the task
Click Create to submit the evaluation task.
Manage evaluation tasks
The evaluation task list displays the following information:
| Column | Description |
|---|---|
| Task name | Name of the evaluation task |
| Score | Scores per evaluation dimension (numeric for numeric type, pass rate for classification) |
| Status | Task execution status |
| Evaluation model | The model used for inference |
| Dataset | The associated evaluation dataset |
| Dimensions | Evaluation dimensions used |
| Tokens | Token consumption during inference |
| Actions | View details, delete, etc. |
View evaluation results
Click Details in the actions column to open the results page, which has two tabs:
- Metrics — Aggregated performance per evaluation dimension. Numeric dimensions show average score, score distribution, and pass rate; classification dimensions show label proportions and overall pass rate. Useful for quickly comparing scores across dimensions.
- Data details — Per-sample scoring breakdown, including the input prompt, model output (uploaded
outputfor inference results, online inference result for evaluation datasets), reference answer, and each dimension's score along with the judge model's reasoning. Filter by dimension or by pass/fail to locate samples where the model underperformed.
Next steps
- Evaluation dimensions -- Create custom scoring criteria.
- Create a dataset -- Prepare an evaluation dataset.