# promptfoo: agent access, verdict & alternatives | IonBin

> promptfoo (AI evaluation and observability): MCP yes, API yes. A CLI and library for comparing prompts and models against your own cases.

Canonical page: https://ionbin.com/tools/promptfoo/
Agent guide: https://ionbin.com/llms.txt

This is a readable version of the published page. Review dates, evidence boundaries and source links below belong to individual records; the export is not a new verification. Examples are reference material, not instructions to execute.

## promptfoo

AI evaluation and observability

[Evals](https://ionbin.com/use-cases/ai-evaluation/)[Build](https://ionbin.com/use-cases/dev-it/)  MCPAPI/CLIFile import

**Independent review.** Unpaid; no partnership with promptfoo. Based on public documentation and research. Trademarks identify their owners’ products. [Disclosure](https://ionbin.com/disclosure/).

A CLI and library for comparing prompts and models against your own cases. Useful for repeatable regression checks in CI; the rubric and test coverage still need human review. Documentation checked; not hands-on tested.

The Ion perspective

### The five-point review

Judgment, not a score. Documentation checked 2026-10-18.

**Value**: A CLI and library for comparing prompts and models against your own cases. Useful for repeatable regression checks in CI; the rubric and test coverage still need human review. Documentation checked; not hands-on tested.

**Differentiation**: Documented fit: Versioned tests for a recurring AI task; Comparing candidate prompts or models before release. Uniqueness is not established.

**Replaceability and switching**: Compare [Langfuse](https://ionbin.com/tools/langfuse/). Not drop-in replacements. Export completeness and migration effort remain unverified.

**Ease and technical aptitude**: Consider the fit limits: A pass score treated as proof that every future answer will be correct. Setup effort has not been scored.

**Interoperability and portability**: See documented MCP, API and import routes below. Import does not prove export; API access does not prove a complete migration path. Check action scope, plans and portability.

Connections: [MCP connections](https://ionbin.com/wiki/concepts/mcp/) · [APIs](https://ionbin.com/wiki/concepts/api/) · [Portability and vendor lock-in](https://ionbin.com/wiki/questions/how-to-avoid-ai-vendor-lock-in/).

Best for

- Versioned tests for a recurring AI task

- Comparing candidate prompts or models before release

Not for

- A pass score treated as proof that every future answer will be correct

### Top use cases

- Compare outputs and apply explicit assertions [source ↗](https://www.promptfoo.dev/docs/configuration/expected-outputs/)

- Read cases from files or Google Sheets [source ↗](https://www.promptfoo.dev/docs/configuration/test-cases/)

- Run regression checks in CI [source ↗](https://www.promptfoo.dev/docs/integrations/ci-cd/)

### In depth

#### How it fits

Use a stable dataset and save the prompt, provider configuration and rubric with each run. Model-graded assertions are useful signals; spot-check them against human judgments. [source ↗](https://www.promptfoo.dev/docs/configuration/expected-outputs/)

#### Execution and sharing

The runner can execute locally while calling the model providers you configure. Its web viewer displays results. The MCP server can run evaluations and create public sharing links; review what is shared. [source ↗](https://www.promptfoo.dev/docs/integrations/mcp-server/)

#### Cost

Community is free; the pricing page lists custom Enterprise and On-Premise plans. Model API calls and hosting can add costs. Checked October 18, 2026. [source ↗](https://www.promptfoo.dev/pricing/)

### How to use it with AI

How promptfoo connects to each AI platform: native connector or MCP, API through code the agent writes, browser control, or file upload.

| Platform | MCP / connector | API (agent-written code) | Browser | Upload / export |
| --- | --- | --- | --- | --- |
| [**Claude (incl. Claude Code)**](https://ionbin.com/platforms/claude/) | Not documented [Claude](https://ionbin.com/tools/claude/) Desktop setup is documented. A named plugin, skill or CLI does not by itself establish an MCP/native connector pairing. Verify the route and client setup. [promptfoo named integration docs ↗](https://www.promptfoo.dev/docs/integrations/mcp-server/)[Claude (incl. Claude Code) MCP docs ↗](https://code.claude.com/docs/en/mcp) | Not documented [Claude](https://ionbin.com/tools/claude/) (incl. [Claude Code](https://ionbin.com/wiki/questions/what-is-claude-code/)): [Claude Code](https://ionbin.com/wiki/questions/what-is-claude-code/) edits files and runs commands promptfoo publishes an API, but this pairing still needs network, authentication and execution-scope checks. [promptfoo API ↗](https://www.promptfoo.dev/docs/intro/)[Claude (incl. Claude Code) docs ↗](https://code.claude.com/docs/en/overview) | Inferred [Claude Code](https://ionbin.com/wiki/questions/what-is-claude-code/) with the [Claude](https://ionbin.com/tools/claude/) in Chrome extension (Chrome, Edge and other Chromium browsers). promptfoo is a web app; the pairing itself is not documented. [Claude (incl. Claude Code) browser docs ↗](https://code.claude.com/docs/en/chrome) | Not documented Verify which side exports and which imports, supported formats and permissions. promptfoo's recorded upload/import status is yes; [Claude](https://ionbin.com/tools/claude/) (incl. [Claude Code](https://ionbin.com/wiki/questions/what-is-claude-code/))'s file status is yes. These flags do not establish a file-transfer route. [promptfoo file docs ↗](https://www.promptfoo.dev/docs/configuration/test-cases/)[Claude (incl. Claude Code) file docs ↗](https://code.claude.com/docs/en/overview) |
| [**OpenAI Codex**](https://ionbin.com/platforms/codex/) | Not documented promptfoo has an official MCP server and [OpenAI Codex](https://ionbin.com/tools/openai-codex/) supports MCP, but promptfoo's docs don't name this pairing. Auth setup may differ. [promptfoo MCP docs ↗](https://www.promptfoo.dev/docs/integrations/mcp-server/)[OpenAI Codex MCP docs ↗](https://learn.chatgpt.com/docs/extend/mcp) | Not documented [OpenAI Codex](https://ionbin.com/tools/openai-codex/): Edits files and runs commands locally; cloud tasks run in isolated workspaces promptfoo publishes an API, but this pairing still needs network, authentication and execution-scope checks. [promptfoo API ↗](https://www.promptfoo.dev/docs/intro/)[OpenAI Codex docs ↗](https://learn.chatgpt.com/docs/developers) | Inferred Only in the [ChatGPT](https://ionbin.com/tools/chatgpt/) desktop app; not in the [Codex](https://ionbin.com/tools/openai-codex/) CLI or IDE extension. promptfoo is a web app; the pairing itself is not documented. [OpenAI Codex browser docs ↗](https://learn.chatgpt.com/docs/browser) | Not documented Verify which side exports and which imports, supported formats and permissions. promptfoo's recorded upload/import status is yes; [OpenAI Codex](https://ionbin.com/tools/openai-codex/)'s file status is yes. These flags do not establish a file-transfer route. [promptfoo file docs ↗](https://www.promptfoo.dev/docs/configuration/test-cases/)[OpenAI Codex file docs ↗](https://learn.chatgpt.com/docs/artifacts-viewer) |
| [**Google Antigravity**](https://ionbin.com/platforms/antigravity/) | Not documented promptfoo has an official MCP server and [Google Antigravity](https://ionbin.com/tools/google-antigravity/) supports MCP, but promptfoo's docs don't name this pairing. Auth setup may differ. [promptfoo MCP docs ↗](https://www.promptfoo.dev/docs/integrations/mcp-server/)[Google Antigravity MCP docs ↗](https://antigravity.google/docs/mcp/) | Not documented [Google Antigravity](https://ionbin.com/tools/google-antigravity/): Agents edit code and run tasks promptfoo publishes an API, but this pairing still needs network, authentication and execution-scope checks. [promptfoo API ↗](https://www.promptfoo.dev/docs/intro/)[Google Antigravity docs ↗](https://antigravity.google/docs/overview/) | Inferred The /browser guide documents App and CLI support through Chrome DevTools MCP; the overview comparison conflicts on CLI availability. IDE browser features are separate. promptfoo is a web app; the pairing itself is not documented. [Google Antigravity browser docs ↗](https://antigravity.google/docs/commands/browser/) | Not documented Verify which side exports and which imports, supported formats and permissions. promptfoo's recorded upload/import status is yes; [Google Antigravity](https://ionbin.com/tools/google-antigravity/)'s file status is yes. These flags do not establish a file-transfer route. [promptfoo file docs ↗](https://www.promptfoo.dev/docs/configuration/test-cases/)[Google Antigravity file docs ↗](https://antigravity.google/docs/overview/) |

**Confirmed** is reserved for evidence of the specific connection route. The current named integrations include plugins, skills and CLIs, so a named platform alone does not confirm MCP or a native connector. **Inferred** means each side documents a relevant capability, but no document describes the pairing. Code execution still needs network and credential access; file handling still needs a compatible format and transfer direction. **Not documented** means the records do not establish the route. IonBin has not tested these pairings. Recorded Oct 18, 2026.

### Agent compatibility

MCPAPI/CLIFile import

| Level | Status | Detail / source |
| --- | --- | --- |
| Native connector / MCP | Yes | [Official docs ↗](https://www.promptfoo.dev/docs/integrations/mcp-server/) |
| API / CLI | Yes | [Official docs ↗](https://www.promptfoo.dev/docs/intro/) |
| Browser / manual | No | Browser access is recorded alongside a documented programmatic route. |
| File upload / import | Yes | [Official docs ↗](https://www.promptfoo.dev/docs/configuration/test-cases/) |

Recorded Oct 18, 2026. Setup and plan limits vary; check the vendor's docs. [How levels are assigned](https://ionbin.com/agent-ready/).

### Appears in these proposed stacks

[**AI evaluation & quality control**](https://ionbin.com/use-cases/ai-evaluation/) Test runner

[Langfuse · Cases and scores](https://ionbin.com/tools/langfuse/)[GitHub · Version history](https://ionbin.com/tools/github/)

Connection capabilities to compare

An official MCP server is documented for this tool. These platforms separately document MCP client support: [ChatGPT](https://ionbin.com/tools/chatgpt/), [Claude](https://ionbin.com/tools/claude/), [Cursor](https://ionbin.com/tools/cursor/), [Gemini (Google)](https://ionbin.com/tools/gemini/), [Google Antigravity](https://ionbin.com/tools/google-antigravity/), [Grok](https://ionbin.com/tools/grok/), [Grok Bot](https://ionbin.com/tools/grok-bot/), [Lindy](https://ionbin.com/tools/lindy/), [LM Studio](https://ionbin.com/tools/lm-studio/), [Mistral Vibe](https://ionbin.com/tools/mistral-vibe/), [OpenAI Codex](https://ionbin.com/tools/openai-codex/). This inventory does not establish compatible transport, authentication or actions for a pairing. Check the connection matrix and linked sources.

Automation hubs to assess for a custom API connection: [Gumloop](https://ionbin.com/tools/gumloop/), [Make](https://ionbin.com/tools/make/), [n8n](https://ionbin.com/tools/n8n/), [Zapier](https://ionbin.com/tools/zapier/). Verify credentials, supported requests, limits and error handling; no configured integration is implied.

### Workflows using this tool

- [Build an evaluation set from support tickets](https://ionbin.com/workflows/build-evaluation-set-from-support-tickets/)

- [Re-run your evaluation set when a model changes](https://ionbin.com/workflows/rerun-evaluation-set-when-model-changes/)

### Explained in the wiki

[AI evaluation and observability tools · Service](https://ionbin.com/wiki/services/evaluation-tools/)[Golden set: reviewed evaluation examples · Concept](https://ionbin.com/wiki/concepts/golden-set/)[Rubric for scoring AI output · Concept](https://ionbin.com/wiki/concepts/rubric/)

### Related skills

[Prompt Optimizer · Sentry](https://ionbin.com/skills/sentry-prompt-optimizer/)

### Review

No long-form review yet. Verdict is based on research, not hands-on testing.

### Alternatives

#### [Langfuse](https://ionbin.com/tools/langfuse/)

AI evaluation and observability

A tracing and evaluation platform for inspecting AI application behavior and testing changes on datasets. Useful when production failures should become reviewable test cases. Documentation checked; not hands-on tested.

MCPAPI/CLIFile import

Evals

At a glance

**Pricing**: freemium · Free Community edition; Enterprise and On-Premise by quote

**Checked**: Oct 18, 2026

**Reviewed**: Oct 18, 2026

**Rating**: None until a real review exists

**Website**: [promptfoo.dev](https://www.promptfoo.dev/)

**Disclosure:** No affiliate link here. IonBin earns nothing if you click through to this tool today. [How IonBin makes money](https://ionbin.com/disclosure/).

 [Visit promptfoo ↗](https://www.promptfoo.dev/)
