Independent review. Unpaid; no partnership with promptfoo. Based on public documentation and research. Trademarks identify their owners’ products. Disclosure.
A CLI and library for comparing prompts and models against your own cases. Useful for repeatable regression checks in CI; the rubric and test coverage still need human review. Documentation checked; not hands-on tested.
The Ion perspective
The five-point review
Judgment, not a score. Documentation checked 2026-10-18.
Value
A CLI and library for comparing prompts and models against your own cases. Useful for repeatable regression checks in CI; the rubric and test coverage still need human review. Documentation checked; not hands-on tested.
Differentiation
Documented fit: Versioned tests for a recurring AI task; Comparing candidate prompts or models before release. Uniqueness is not established.
Replaceability and switching
Compare Langfuse. Not drop-in replacements. Export completeness and migration effort remain unverified.
Ease and technical aptitude
Consider the fit limits: A pass score treated as proof that every future answer will be correct. Setup effort has not been scored.
Interoperability and portability
See documented MCP, API and import routes below. Import does not prove export; API access does not prove a complete migration path. Check action scope, plans and portability.
Use a stable dataset and save the prompt, provider configuration and rubric with each run. Model-graded assertions are useful signals; spot-check them against human judgments. source ↗
Execution and sharing
The runner can execute locally while calling the model providers you configure. Its web viewer displays results. The MCP server can run evaluations and create public sharing links; review what is shared. source ↗
Cost
Community is free; the pricing page lists custom Enterprise and On-Premise plans. Model API calls and hosting can add costs. Checked October 18, 2026. source ↗
How to use it with AI
How promptfoo connects to each AI platform: native connector or MCP, API through code the agent writes, browser control, or file upload.
Not documentedOpenAI Codex: Edits files and runs commands locally; cloud tasks run in isolated workspaces promptfoo publishes an API, but this pairing still needs network, authentication and execution-scope checks.promptfoo API ↗OpenAI Codex docs ↗
InferredOnly in the ChatGPT desktop app; not in the Codex CLI or IDE extension. promptfoo is a web app; the pairing itself is not documented.OpenAI Codex browser docs ↗
Not documentedVerify which side exports and which imports, supported formats and permissions. promptfoo's recorded upload/import status is yes; OpenAI Codex's file status is yes. These flags do not establish a file-transfer route.promptfoo file docs ↗OpenAI Codex file docs ↗
InferredThe /browser guide documents App and CLI support through Chrome DevTools MCP; the overview comparison conflicts on CLI availability. IDE browser features are separate. promptfoo is a web app; the pairing itself is not documented.Google Antigravity browser docs ↗
Not documentedVerify which side exports and which imports, supported formats and permissions. promptfoo's recorded upload/import status is yes; Google Antigravity's file status is yes. These flags do not establish a file-transfer route.promptfoo file docs ↗Google Antigravity file docs ↗
Confirmed is reserved for evidence of the specific connection route. The current named integrations include plugins, skills and CLIs, so a named platform alone does not confirm MCP or a native connector. Inferred means each side documents a relevant capability, but no document describes the pairing. Code execution still needs network and credential access; file handling still needs a compatible format and transfer direction. Not documented means the records do not establish the route. IonBin has not tested these pairings. Recorded Oct 18, 2026.
Automation hubs to assess for a custom API connection: Gumloop, Make, n8n, Zapier. Verify credentials, supported requests, limits and error handling; no configured integration is implied.
A tracing and evaluation platform for inspecting AI application behavior and testing changes on datasets. Useful when production failures should become reviewable test cases. Documentation checked; not hands-on tested.