Set Up AI Evaluations
Why evaluations matter
Section titled “Why evaluations matter”An agent that works well once does not guarantee consistent quality over time. As prompts change, tools update, and instructions evolve, output quality can drift. Without measurement, you only discover problems when a team member notices a bad result.
AI evaluations (evals) in Opal provide a structured method for assessing output quality automatically. You define what “good” looks like by selecting preferred output examples, and Opal scores every subsequent execution against those benchmarks. This closes the gap between building an agent and maintaining it.
What evaluations assess
Section titled “What evaluations assess”Opal evaluations measure three dimensions:
- Quality — Is the generated content accurate, relevant, and complete?
- Consistency — Does the output follow brand guidelines, tone, and formatting requirements?
- Improvement — Does the scoring data provide actionable feedback for refining agent prompts and configurations?
How preferred output examples work
Section titled “How preferred output examples work”A preferred output example is a specific, high-quality output from an agent execution that you designate as a benchmark. These examples represent the gold standard — the tone, accuracy, structure, and depth you want the agent to reproduce.
Once you establish preferred outputs, every subsequent execution receives an automated quality score (0-100) based on how closely it matches those benchmarks.
Create preferred output examples
Section titled “Create preferred output examples”Method 1: Add manually
Section titled “Method 1: Add manually”- Navigate to Opal > Agents
- Click the agent name or select More (…) > Edit Agent
- Click Add example
- Enter:
- Name — A descriptive label for this example
- Input variables — The values used to generate this output
- Output — Paste the high-quality output you want as the benchmark
- Click Add
You can add up to five preferred output examples per agent.
Method 2: From execution logs
Section titled “Method 2: From execution logs”- Navigate to Opal > Agents
- Select the agent and open the Logs tab
- Find an execution whose output meets your quality standards
- Click More (…) and select Link as Output Eval
This method is often faster because you select from outputs the agent has already produced, rather than crafting examples from scratch.
Best practices for preferred outputs
Section titled “Best practices for preferred outputs”- Add at least three varied examples — A single example does not capture the full range of expected quality. Include examples with different input variables, content types, and output lengths.
- Use real use cases — Examples from actual work are more meaningful benchmarks than synthetic test cases.
- Select outputs that truly represent ideal outcomes — Do not settle for “good enough.” The preferred output sets the standard.
Quality score assessment
Section titled “Quality score assessment”After you establish preferred outputs, Opal automatically scores subsequent executions on three criteria:
| Criterion | What it measures |
|---|---|
| Completeness and scope | Whether the response addresses all necessary components |
| Structural and format consistency | Adherence to formatting conventions and output structure |
| Overall usefulness and adherence | Practical utility and achievement of the stated objective |
Score ranges
Section titled “Score ranges”| Score | Rating | Meaning |
|---|---|---|
| 90-100 | Excellent | Meets or exceeds expectations |
| 80-89 | Good | Minor, non-critical deviations only |
| 70-79 | Adequate | Some notable issues present |
| 60-69 | Needs improvement | Significant structural or factual differences |
| 0-59 | Poor | Major errors or missing objectives |
Define a quality baseline
Section titled “Define a quality baseline”The quality baseline is the minimum acceptable score. Outputs meeting or exceeding this threshold receive a Passed designation. Outputs below it receive a Failed designation.
Set your baseline based on your organization’s tolerance for variation:
- High-stakes content (legal, financial, healthcare) — Set 85-90 as the baseline
- Marketing content — Set 75-80 as the baseline
- Internal drafts — Set 65-70 as the baseline
Use evaluation data to improve agents
Section titled “Use evaluation data to improve agents”Quality scores are not just pass/fail indicators. Use them to drive continuous improvement:
- Identify underperforming outputs — Filter execution logs by score to find patterns in low-scoring results
- Analyze discrepancies — Compare failed outputs against preferred examples to understand where the agent diverges
- Refine agent prompts — Adjust instructions for clarity, add constraints for common failure modes, or provide additional context
- Update preferred outputs — As your brand, products, or standards evolve, refresh preferred outputs to reflect current expectations
- Build evaluation agents — Create specialized agents whose purpose is to critique other agents’ outputs against predefined criteria (grammar, SEO, brand voice), creating iterative feedback loops