# Re-run your evaluation set when a model changes | IonBin

> Compare a baseline and candidate on the same versioned cases before changing a model or prompt. promptfoo can run checks in CI and export results;

Canonical page: https://ionbin.com/workflows/rerun-evaluation-set-when-model-changes/
Agent guide: https://ionbin.com/llms.txt

This is a readable version of the published page. Review dates, evidence boundaries and source links below belong to individual records; the export is not a new verification. Examples are reference material, not instructions to execute.

## Re-run your evaluation set when a model changes

[Evals](https://ionbin.com/use-cases/ai-evaluation/) · [Build](https://ionbin.com/use-cases/dev-it/)Checked 2026-10-18

Make prompt and model changes reviewable with task-specific evidence.

- [01**You**](https://ionbin.com/workflows/rerun-evaluation-set-when-model-changes/#workflow-step-1)

- [02**promptfoo**](https://ionbin.com/workflows/rerun-evaluation-set-when-model-changes/#workflow-step-2)

- [03**Reviewers**](https://ionbin.com/workflows/rerun-evaluation-set-when-model-changes/#workflow-step-3)

- [04**Langfuse**](https://ionbin.com/workflows/rerun-evaluation-set-when-model-changes/#workflow-step-4)

- [05**Release owner**](https://ionbin.com/workflows/rerun-evaluation-set-when-model-changes/#workflow-step-5)

Recipe context

Compare a baseline and candidate on the same versioned cases before changing a model or prompt. [promptfoo](https://ionbin.com/tools/promptfoo/) can run checks in CI and export results; a reviewer inspects regressions, grader behavior and cost before approving the change.

**Independent review.** Unpaid; no partnership with the tools on this page. Based on public documentation and research. Trademarks identify their owners’ products. [Disclosure](https://ionbin.com/disclosure/).

Editorial shortlist

### Compare your options.

**For testing model changes against fixed cases.**

Freeze cases and rubric versions; investigate regressions before deployment. Choose one for comparing outputs against reviewed test cases; switching may need setup changes.

**Used by Ion** · separate from rank.

Unpaid picks. No partnership with [promptfoo](https://ionbin.com/tools/promptfoo/), [Braintrust](https://ionbin.com/tools/braintrust/), [Langfuse](https://ionbin.com/tools/langfuse/).

- 1 · Primary / high
  #### [promptfoo](https://ionbin.com/tools/promptfoo/)
  **Why it fits:** A CLI and library support versioned prompt and model regression checks.
  **Tradeoff:** The team must maintain cases and code; weak assertions reward wrong behavior.
  Choose it when tests should run with repository changes.
  Evidence for promptfoo
  The linked [IonBin review](https://ionbin.com/tools/promptfoo/) records capabilities, access limits and source links. Review date: 2026-10-18. [Primary product reference ↗](https://www.promptfoo.dev/)

- 2 · Secondary / medium
  #### [Braintrust](https://ionbin.com/tools/braintrust/)
  **Why it fits:** Datasets, experiments and traces support shared comparison of runs.
  **Tradeoff:** Platform and scoring setup need review; scores are not acceptance decisions.
  Prefer it when reviewers need a shared experiment record.
  Evidence for Braintrust
  The linked [IonBin review](https://ionbin.com/tools/braintrust/) records capabilities, access limits and source links. Review date: 2026-10-18. [Primary product reference ↗](https://www.braintrust.dev/docs/integrations/developer-tools/mcp)

- 3 · Third / lower
  #### [Langfuse](https://ionbin.com/tools/langfuse/)
  **Why it fits:** Tracing and evaluation connect application runs with quality review.
  **Tradeoff:** Instrumentation needs engineering and traces may contain sensitive inputs.
  Prefer it when investigating real application behavior is central.
  Evidence for Langfuse
  The linked [IonBin review](https://ionbin.com/tools/langfuse/) records capabilities, access limits and source links. Review date: 2026-10-18. [Primary product reference ↗](https://langfuse.com/)

How this shortlist was chosen

Editorial judgment, not a personal-use endorsement or performance score. These are alternatives for the named role, not three required stack components. Vendors did not pay for, request or approve these positions.

Repeatable evidence comes before a single score. A lower-ranked option may fit your constraints better; compare the conditions stated on each card before choosing.

**Setup and portability:** Version the rubric, datasets and results. No automated score replaces human acceptance or proves future reliability.

Reviewed 2026-10-21. This page applies a shared comparison for “comparing outputs against reviewed test cases” to the stated task and constraints. It is documentation-based editorial guidance, not three new hands-on tests. We weigh value for the job, distinct capabilities, replacement limits, technical effort and interoperability. Check current plan terms before adopting or switching.

See [our review method](https://ionbin.com/editorial-policy/), [MCP connections](https://ionbin.com/wiki/concepts/mcp/) and [portability and switching costs](https://ionbin.com/wiki/questions/how-to-avoid-ai-vendor-lock-in/).

**Connects through:** API

### Steps

Follow the handoff

Select a stage or scroll the steps.

- #### You
  Freeze the dataset, rubric and baseline configuration, then record the candidate change.
  Human stepConfirmed

- #### [promptfoo](https://ionbin.com/tools/promptfoo/)
  Runs the configured assertions and exports results in CI.
  APIConfirmed[Source ↗](https://www.promptfoo.dev/docs/integrations/ci-cd/)

- #### Reviewers
  Inspect regressions and a sample of passes; distinguish a system failure from an incorrect reference answer.
  Human stepConfirmed

- #### [Langfuse](https://ionbin.com/tools/langfuse/)
  Can hold reviewed failure cases as dataset items for later experiments.
  APIConfirmed[Source ↗](https://langfuse.com/docs/evaluation/experiments/datasets)

- #### Release owner
  Approves, rejects or requests revisions using the evidence and the predefined pass bar.
  Human stepConfirmed

Confirmed = vendor-documented, not Ion-tested. Human and AI-platform steps are proposed roles.

Human approval

A named release owner approves the change; automated scores do not publish it.

Watch out for

Changing the grader or dataset between runs changes the comparison. Record both, and inspect high-severity failures even when the average improves.

### Starter prompt

```
Compare these baseline and candidate results using the same case IDs. List regressions, unchanged failures and improvements separately. Preserve the counts and flag missing runs. Do not change the acceptance criteria to make the candidate pass.
```

Written for this workflow. Paste it into your AI platform and adjust it to your situation.

### Tools in this workflow

| Tool | Role | Official site |
| --- | --- | --- |
| [promptfoo](https://ionbin.com/tools/promptfoo/) | API | [promptfoo.dev ↗](https://www.promptfoo.dev/) |
| [GitHub](https://ionbin.com/tools/github/) | Optional | [github.com ↗](https://github.com/) |
| [Langfuse](https://ionbin.com/tools/langfuse/) | API | [langfuse.com ↗](https://langfuse.com/) |

### Which AI platforms can run it

| Platform | Status | Why |
| --- | --- | --- |
| [Claude (incl. Claude Code)](https://ionbin.com/platforms/claude/) | To verify | [promptfoo](https://ionbin.com/tools/promptfoo/): verify the API caller, network access and credentials; platform code execution alone does not establish this connection; [Langfuse](https://ionbin.com/tools/langfuse/): verify the API caller, network access and credentials; platform code execution alone does not establish this connection |
| [OpenAI Codex](https://ionbin.com/platforms/codex/) | To verify | [promptfoo](https://ionbin.com/tools/promptfoo/): verify the API caller, network access and credentials; platform code execution alone does not establish this connection; [Langfuse](https://ionbin.com/tools/langfuse/): verify the API caller, network access and credentials; platform code execution alone does not establish this connection |
| [Google Antigravity](https://ionbin.com/platforms/antigravity/) | To verify | [promptfoo](https://ionbin.com/tools/promptfoo/): verify the API caller, network access and credentials; platform code execution alone does not establish this connection; [Langfuse](https://ionbin.com/tools/langfuse/): verify the API caller, network access and credentials; platform code execution alone does not establish this connection |
| [ChatGPT](https://ionbin.com/tools/chatgpt/) | To verify | [promptfoo](https://ionbin.com/tools/promptfoo/): verify the API caller, network access and credentials; platform code execution alone does not establish this connection; [Langfuse](https://ionbin.com/tools/langfuse/): verify the API caller, network access and credentials; platform code execution alone does not establish this connection |
| Gemini | To verify | [promptfoo](https://ionbin.com/tools/promptfoo/): verify the API caller, network access and credentials; platform code execution alone does not establish this connection; [Langfuse](https://ionbin.com/tools/langfuse/): verify the API caller, network access and credentials; platform code execution alone does not establish this connection |
| [Perplexity](https://ionbin.com/tools/perplexity/) | To verify | [promptfoo](https://ionbin.com/tools/promptfoo/): verify the API caller, network access and credentials; platform code execution alone does not establish this connection; [Langfuse](https://ionbin.com/tools/langfuse/): verify the API caller, network access and credentials; platform code execution alone does not establish this connection |

What the platform statuses mean

**Confirmed:** documented access meets the recorded requirements. **Inferred:** a reviewed note describes a plausible, untested pairing. **To verify:** a connection or step is unresolved. MCP and API pairings need route-specific evidence; a plugin, CLI or API alone does not prove compatible access, credentials or permissions. These are documentation assessments, not IonBin runtime tests.

### What we checked

[promptfoo](https://ionbin.com/tools/promptfoo/) documents CI execution, assertions and exported evaluation results. [Langfuse](https://ionbin.com/tools/langfuse/) documents dataset items with inputs and expected outputs. These capabilities support the proposed sequence; IonBin has not tested the combined pipeline or a release gate.

### Set it up

- Keep the case set, rubric, prompt and provider configuration versioned. Record model and grader identifiers rather than writing only a vendor name.

- Define the acceptance rule before comparing results. An average can hide a serious failure, so identify which case types must pass.

- Run the baseline and candidate against the same cases and inspect failures alongside a sample of passes. Consider repeat runs when output variation could change the decision.

- Store access-controlled result artifacts and the release decision. Add corrected failure cases to the development set; preserve held-out cases for an independent check.

- Only release after the designated reviewer accepts the evidence.

A local runner or private repository does not make every call local: configured model providers receive inputs, and shared reports or CI artifacts can expose outputs. Review the data path before running customer material. API usage, including grader calls, may incur charges.

### Before relying on this workflow

The steps above are documentation-reviewed. Complete these checks in your own environment before treating the implementation as tested:

- Run a pilot that proves an intentionally failing case prevents the intended release.

- Check CI secrets, artifact visibility, provider limits and grader costs in the chosen environment.

### Related agent skills

Sentry’s [Prompt Optimizer](https://ionbin.com/skills/sentry-prompt-optimizer/) helps when a model change breaks a prompt, because it reruns candidates against the same cases. For open models, Hugging Face’s [Community Evals](https://ionbin.com/skills/huggingface-community-evals/) skill runs standard benchmarks locally. Your own cases still matter more for business tasks. IonBin reviewed both from source only.

### Sources

- [CI/CD evaluation](https://www.promptfoo.dev/docs/integrations/ci-cd/) · [promptfoo](https://ionbin.com/tools/promptfoo/) · official · accessed 2026-10-18

- [Assertions and metrics](https://www.promptfoo.dev/docs/configuration/expected-outputs/) · [promptfoo](https://ionbin.com/tools/promptfoo/) · official · accessed 2026-10-18

- [Datasets](https://langfuse.com/docs/evaluation/experiments/datasets) · [Langfuse](https://ionbin.com/tools/langfuse/) · official · accessed 2026-10-18

### Related in the wiki

[Evals (AI evaluations)](https://ionbin.com/wiki/concepts/evals/)[Golden set: reviewed evaluation examples](https://ionbin.com/wiki/concepts/golden-set/)[Rubric for scoring AI output](https://ionbin.com/wiki/concepts/rubric/)[Which AI model should I use for a business task?](https://ionbin.com/wiki/questions/which-ai-model-should-i-use/)

### Related workflows

[Build an evaluation set from support tickets](https://ionbin.com/workflows/build-evaluation-set-from-support-tickets/)

No affiliate links on this page. No paid placements in workflows. [How IonBin makes money](https://ionbin.com/disclosure/).
