# How IonBin evaluates agent skills

> The Skills.md review rubric: source provenance, setup requirements, task baselines, permissions, security testing, licensing, evidence and version

Canonical page: https://ionbin.com/skills/methodology/
Agent guide: https://ionbin.com/llms.txt

This is a readable version of the published page. Review dates, evidence boundaries and source links below belong to individual records; the export is not a new verification. Examples are reference material, not instructions to execute.

Our evaluation standard

## Useful is a claim.
Show the evidence.

A skill can change an agent’s instructions, tools and access. A useful review explains the task it serves, what it requires, what we observed and what remains unknown.

Current collection

**20 source-only assessments · 7 runtime probes · 16 decision scenarios.** Each record states its actual coverage. A helper probe is not a complete task evaluation; neither establishes comprehensive security. The rubric below defines the evidence required for broader conclusions.

Keep the labels honest

### Evidence and experience are different.

Source inspection, helper execution and task evaluation support different conclusions. We record coverage and our relationship to a skill separately.

20 records in this collection

#### Source reviewed

Named sources were inspected. No runtime results are established by this label.

7 records in this collection

#### Runtime probed

A helper or bounded runtime behavior was probed. This does not establish model task effectiveness or comprehensive security.

16 records in this collection

#### Scenario checked

A text-only decision scenario was checked with and without skill instructions. This is not an executed workflow or proof of effectiveness.

A result label does not mean every rubric item passed. Reports distinguish completed, partial and blocked work, give actual case counts and failures, and preserve the limitations. Compare case definitions before comparing counts.

**Explored**: We investigated the skill and recorded the scope. This label does not imply real-world use or replication.

**Used**: A documented task used the skill. The public provenance summary explains what was used and, where publishable, links evidence. Installation or availability alone does not count.

**Replicated**: We recreated or adapted behavior and record exactly what was replicated, the original attribution, license, changes and validation. This does not imply identical behavior or an independently reproduced evaluation.

Reproducing an evaluation is separate from replicating a skill. It requires another run or reviewer, the same revision and fixtures, and recorded differences. Private work remains private; public records expose only the permitted summary and public artifact.

Six questions. A traceable answer.

### The comprehensive evaluation rubric.

This is the target standard for a full evaluation. An assessment may cover only part of it; unchecked work stays explicitly unverified.

Identity & provenance

#### What are we actually evaluating?

Record the author, repository, exact skill path, immutable revision and retrieval date. Trace bundled scripts, linked instructions and copied or adapted material. Separate upstream facts from our interpretation.

**Evidence to retain**Source manifest, revision and attribution record.

Setup & compatibility

#### What must exist before it works?

Record agent host and version, model, operating system, runtime, packages, browser, connectors and account requirements. Identify paid plans, usage costs and missing setup steps. Check the license of the skill and bundled assets separately.

**Evidence to retain**Environment manifest, dependency list and setup log.

Task success

#### Does it improve the task?

Define a representative task, fixed inputs and an acceptance rubric before running it. Compare the same task with and without the skill under the same conditions. Record repetitions, success and failure counts, human corrections, time and cost. Inspect the output itself.

**Evidence to retain**Task fixtures, baseline, scored outputs and failure examples.

Permissions & data flows

#### What can it touch, change or send?

Trace local reads and writes, shell execution, network destinations, external services and credential handling. Identify destructive actions, publishing and other side effects. Observe requested versus actual access, approval points, sandbox limits and any sensitive information in logs or outputs.

**Evidence to retain**Permission and data-flow map, observed calls and network log.

Adversarial behavior & recovery

#### What happens when something goes wrong?

Use controlled test fixtures for untrusted content, instruction injection, secret exposure, path escapes, unexpected downloads and oversized inputs. Test missing dependencies, revoked access, interrupted runs and rollback. Distinguish a suspected concern from a reproduced finding.

**Evidence to retain**Test cases, reproducible findings, impact and recovery notes.

Usability & maintenance

#### Can someone else repeat the result?

Have another reviewer follow the setup and acceptance steps. Record ambiguous instructions, unnecessary access, platform differences and maintenance signals. Tie the conclusion to the tested revision and environment, then define what change triggers a new review.

**Evidence to retain**Reproduction notes, limitations, reviewer and recheck triggers.

Specific concerns, bounded conclusions

### A risk note is not a safety certificate.

#### How concerns are labeled

**Review before use** calls out an access requirement, assumption or practical concern that needs checking in the intended environment.

**Higher attention** flags a potentially consequential exposure or side effect that deserves closer inspection before use. It is an editorial priority, not a vulnerability severity score.

Source-derived concerns appear separately from evaluation findings. An **observation** records behavior; **needs attention** identifies a finding to examine. A label alone does not claim a reproduced exploit.

#### What a tested finding needs

A finding needs the affected revision, preconditions, reproduction steps, observed result, likely impact and evidence. Separate actual behavior from assumptions and note mitigations and retest results.

All executions need an appropriately isolated environment and permitted test data. The skill, its dependencies and its agent host each affect the risk. A quiet test run is not proof of safety.

**No universal “safe to install” badge.**

Permissions, credentials, data sensitivity and the task change the consequences. We show the evidence and conditions so readers can make that decision in their own context.

Versioned, not timeless

### The review ends at its evidence boundary.

#### Record the version

Each assessment states its source date and revision when captured. An unpinned source is labeled unpinned. A runtime review also needs dependency versions, host, model, operating system and configuration; an upstream commit alone is insufficient.

#### Recheck when the behavior changes

New scripts, network destinations, dependency changes, permission expansion and major instruction rewrites trigger review. Older findings stay attached to their reviewed revision; they are not silently carried forward to a new version.

The current collection is a dated snapshot. No continuous monitoring or automatic re-review is claimed. Unresolved requirements, licensing and compatibility stay visible as limitations.

Discovery without borrowed credibility

### Why a skill appears here.

We use upstream collections and public discovery signals to identify candidates. A record can be useful to investigate without being a recommendation.

#### Popularity needs a denominator

Downloads, installs, stars and ranking positions mean different things. When a number is included, the record names the metric, source, scope and observation date. Repository-level attention does not establish the popularity of every skill in that repository.

A single snapshot cannot prove a trend. “Trending” needs a defined time window and a change over that period.

#### Link to the actual upstream work

Each record links the repository, exact skill source and license notes. We describe the useful requirements and concerns in our own words. Inclusion does not imply endorsement by the author or transfer of rights.

Discovery services can contribute leads. Our assessment must still identify what IonBin actually examined.

Read with the evidence in view.

### Explore the assessments.

Compare setup, access, recorded results and open questions before choosing what to evaluate yourself.

[Browse Skills.md](https://ionbin.com/skills/)
