# AI evaluation & quality control: the AI tool stack | IonBin

> Check an AI task against reviewed examples before changing the prompt, model or workflow. Recommended stack: promptfoo, Langfuse, GitHub.

Canonical page: https://ionbin.com/use-cases/ai-evaluation/
Agent guide: https://ionbin.com/llms.txt

This is a readable version of the published page. Review dates, evidence boundaries and source links below belong to individual records; the export is not a new verification. Examples are reference material, not instructions to execute.

Stack 10 / Evals

## AI evaluation & quality control

Check an AI task against reviewed examples before changing the prompt, model or workflow.

Explore the proposed stack

- 01
  **promptfoo**See stage +
  Test runner: Runs configured checks against candidate prompts and models.
  Proposed · role[Tool review ↗](https://ionbin.com/tools/promptfoo/)

- 02
  **Langfuse**See stage +
  Cases and scores: Holds datasets and experiments; human review helps explain failures.
  Proposed · role[Tool review ↗](https://ionbin.com/tools/langfuse/)

- 03
  **GitHub**See stage +
  Version history: Keeps configuration and review decisions together through commits and pull requests.
  Proposed · role[Tool review ↗](https://ionbin.com/tools/github/)

Short answer

Start with real examples and a written standard for acceptable answers. Compare the current system with a candidate on the same cases, inspect the failures and keep a separate held-out set for final checks. Production errors can become new test cases after review. A passing score supports a decision; it does not guarantee correctness.

Editorial shortlist

### Compare your options.

**For comparing AI changes against reviewed examples.**

The same rubric and held-out cases must be used across candidates. Choose one for comparing outputs against reviewed test cases; switching may need setup changes.

**Used by Ion** · separate from rank.

Unpaid picks. No partnership with [promptfoo](https://ionbin.com/tools/promptfoo/), [Braintrust](https://ionbin.com/tools/braintrust/), [Langfuse](https://ionbin.com/tools/langfuse/).

- 1 · Primary / high
  #### [promptfoo](https://ionbin.com/tools/promptfoo/)
  **Why it fits:** A CLI and library support versioned prompt and model regression checks.
  **Tradeoff:** The team must maintain cases and code; weak assertions reward wrong behavior.
  Choose it when tests should run with repository changes.
  Evidence for promptfoo
  The linked [IonBin review](https://ionbin.com/tools/promptfoo/) records capabilities, access limits and source links. Review date: 2026-10-18. [Primary product reference ↗](https://www.promptfoo.dev/)

- 2 · Secondary / medium
  #### [Braintrust](https://ionbin.com/tools/braintrust/)
  **Why it fits:** Datasets, experiments and traces support shared comparison of runs.
  **Tradeoff:** Platform and scoring setup need review; scores are not acceptance decisions.
  Prefer it when reviewers need a shared experiment record.
  Evidence for Braintrust
  The linked [IonBin review](https://ionbin.com/tools/braintrust/) records capabilities, access limits and source links. Review date: 2026-10-18. [Primary product reference ↗](https://www.braintrust.dev/docs/integrations/developer-tools/mcp)

- 3 · Third / lower
  #### [Langfuse](https://ionbin.com/tools/langfuse/)
  **Why it fits:** Tracing and evaluation connect application runs with quality review.
  **Tradeoff:** Instrumentation needs engineering and traces may contain sensitive inputs.
  Prefer it when investigating real application behavior is central.
  Evidence for Langfuse
  The linked [IonBin review](https://ionbin.com/tools/langfuse/) records capabilities, access limits and source links. Review date: 2026-10-18. [Primary product reference ↗](https://langfuse.com/)

How this shortlist was chosen

Editorial judgment, not a personal-use endorsement or performance score. These are alternatives for the named role, not three required stack components. Vendors did not pay for, request or approve these positions.

Repeatable evidence comes before a single score. A lower-ranked option may fit your constraints better; compare the conditions stated on each card before choosing.

**Setup and portability:** Version the rubric, datasets and results. No automated score replaces human acceptance or proves future reliability.

Reviewed 2026-10-21. This page applies a shared comparison for “comparing outputs against reviewed test cases” to the stated task and constraints. It is documentation-based editorial guidance, not three new hands-on tests. We weigh value for the job, distinct capabilities, replacement limits, technical effort and interoperability. Check current plan terms before adopting or switching.

See [our review method](https://ionbin.com/editorial-policy/), [MCP connections](https://ionbin.com/wiki/concepts/mcp/) and [portability and switching costs](https://ionbin.com/wiki/questions/how-to-avoid-ai-vendor-lock-in/).

The stack · editorial pick

### 3 tools, one job

Paid placement never changes a stack.

01Test runner

#### [promptfoo](https://ionbin.com/tools/promptfoo/)

Runs configured checks against candidate prompts and models.

MCPAPI/CLIFile import

02Cases and scores

#### [Langfuse](https://ionbin.com/tools/langfuse/)

Holds datasets and experiments; human review helps explain failures.

MCPAPI/CLIFile import

03Version history

#### [GitHub](https://ionbin.com/tools/github/)

Keeps configuration and review decisions together through commits and pull requests.

MCPAPI/CLIFile import

How this proposed stack fits together

This is a proposed stack, not a tested integration. Start with file-based cases; add application tracing when there is a live system to observe. Keep expected answers, model outputs and reviewer decisions separate. People set the pass bar and decide which failures block a release.

Connection methods are proposed from recorded capabilities, not evidence that IonBin tested the complete workflow. Check supported actions, account access and file direction before setup. "Manual" means no documented direct route is recorded here.

Proposed data flow

- [Langfuse](https://ionbin.com/tools/langfuse/)→[promptfoo](https://ionbin.com/tools/promptfoo/) Map approved dataset items into runner-compatible cases
manual

- [promptfoo](https://ionbin.com/tools/promptfoo/)→[GitHub](https://ionbin.com/tools/github/) Save configuration and access-controlled result artifacts
manual

- [GitHub](https://ionbin.com/tools/github/)→[Langfuse](https://ionbin.com/tools/langfuse/) Review changes and add approved failure cases
manual

IonBin’s Brightest

### From IonBin

House ad · unsold slot

#### Put this stack to work this week

Step-by-step workflows with a starter prompt, a watch-out line and a human approval step. [Browse workflows](https://ionbin.com/workflows/) · [Recent additions](https://ionbin.com/new/)

This is IonBin’s own promotion. The IonBin’s Brightest slot is a paid, labeled placement; [see options](https://ionbin.com/advertise/).

Workflows

### Put this stack to work

Documentation-reviewed recipes with sources and implementation checks. No affiliate links.

[Score AI drafts with two reviewers](https://ionbin.com/workflows/score-drafts-with-two-reviewers/)[Build an evaluation set from support tickets](https://ionbin.com/workflows/build-evaluation-set-from-support-tickets/)[Re-run your evaluation set when a model changes](https://ionbin.com/workflows/rerun-evaluation-set-when-model-changes/)

Wiki

### Learn the pieces

Needs, questions, models and concepts behind this stack.

[AI evaluation and observability tools · Service](https://ionbin.com/wiki/services/evaluation-tools/)[Data labelling and annotation · Service](https://ionbin.com/wiki/services/data-labelling/)[Evals (AI evaluations) · Concept](https://ionbin.com/wiki/concepts/evals/)[Gemma 4 · Model](https://ionbin.com/wiki/models/gemma-4/)[Golden set: reviewed evaluation examples · Concept](https://ionbin.com/wiki/concepts/golden-set/)[How do I know if AI output is good enough to use? · Question](https://ionbin.com/wiki/questions/how-to-evaluate-ai-output-quality/)[Rubric for scoring AI output · Concept](https://ionbin.com/wiki/concepts/rubric/)[Should we fine-tune a model or just write better prompts? · Question](https://ionbin.com/wiki/questions/should-we-fine-tune-or-prompt/)

Skills

### Related skills

Reusable agent instructions reviewed by IonBin. Source review only unless the skill page says otherwise.

[Prompt Optimizer · Sentry](https://ionbin.com/skills/sentry-prompt-optimizer/)[Hugging Face Community Evals · Hugging Face](https://ionbin.com/skills/huggingface-community-evals/)

Swap-ins

### Also works for this job

Alternatives for one role in the stack.

#### [Claude](https://ionbin.com/tools/claude/)

AI assistant

Anthropic's AI assistant on web, desktop and mobile, with file analysis, Projects and app connectors. Paid plans add usage and [Claude Code](https://ionbin.com/wiki/questions/what-is-claude-code/). Every plan can add custom remote MCP connectors; Free is limited to one.

Agent platform MCPAPI/CLIFile import

Used by Ion SOPs

#### [OpenAI Codex](https://ionbin.com/tools/openai-codex/)

AI coding agent

OpenAI's coding agent across desktop, CLI, IDE extension and cloud. It can inspect, edit and review code; cloud tasks use their own workspaces. MCP is supported as a client, while the old [Codex](https://ionbin.com/tools/openai-codex/) MCP-server mode has been removed.

Agent platform API/CLIFile import

Used by Ion Build

#### [ChatGPT](https://ionbin.com/tools/chatgpt/)

AI assistant

OpenAI's general-purpose AI assistant for drafting, analysis and research, with file uploads, deep research and images. Projects organize ongoing work; scheduled-task access depends on account eligibility. Plugins connect tools, while Work and [Codex](https://ionbin.com/tools/openai-codex/) support longer tasks.

Agent platform API/CLIFile import

Content

#### [Braintrust](https://ionbin.com/tools/braintrust/)

AI evaluation and observability

Evaluation datasets, experiment runs and traces in one platform. Useful when a team needs to compare changes against reviewed examples. Documentation checked; not hands-on tested.

MCPAPI/CLIFile import

Evals

#### [Gemini (Google)](https://ionbin.com/tools/gemini/)

AI assistant

Google's AI assistant with Deep Research, Canvas, Gems and file analysis. Personal paid plans raise limits and add features in Google apps. Eligible US personal accounts can connect custom MCP apps; organizational accounts have separate access rules.

Agent platform API/CLIFile import

SOPs

#### [Google Workspace (Gmail, Drive, Sheets)](https://ionbin.com/tools/google-workspace/)

Docs and knowledge

Gmail, Drive, Docs, Sheets, Calendar and Meet on your own domain. Google's remote MCP servers (Developer Preview) let AI agents search mail and files, draft emails, upload files and schedule meetings with the user's own permissions.

MCPAPI/CLIFile import

Data

#### [Label Studio](https://ionbin.com/tools/label-studio/)

Data labelling

Annotation software for turning examples into reviewed labels. Match the edition and task configuration to the review process. Documentation checked; not hands-on tested.

MCPAPI/CLIFile import

Evals

### Compare at a glance

| Role | Tool | Agent compatibility | Pricing | Used by Ion |
| --- | --- | --- | --- | --- |
| Test runner | [promptfoo](https://ionbin.com/tools/promptfoo/) | MCPAPI/CLIFile import | freemium · Free Community edition; Enterprise and On-Premise by quote (checked Oct 18, 2026) |  |
| Cases and scores | [Langfuse](https://ionbin.com/tools/langfuse/) | MCPAPI/CLIFile import | freemium · Free Hobby cloud plan; paid cloud plans and self-hosting options (checked Oct 18, 2026) |  |
| Version history | [GitHub](https://ionbin.com/tools/github/) | MCPAPI/CLIFile import | freemium · Free $0; Team $4 and Enterprise from $21 USD/user/month advertised for the first 12 months; confirm renewal terms (checked Oct 18, 2026) |  |
