FreeBench

What FreeBench measures

What FreeBench measures: exact free model-and-harness setups on real coding workloads, how qualification works, what unmeasured means, and the limits of a result. No ranking is published.

FreeBench v2 agent benchmark rankings are pending execution qualification, calibration and publication review. No approved public model cohort exists.

FreeBench measures how a specific free or low-cost model, reached through a specific coding harness, performs on a defined engineering workload, and what its practical access limits are. A result describes that complete setup. It is not a verdict about the model in isolation.

Publication status

FreeBench does not publish a ranking. Source-backed offer facts, setup guides and catalog records are available now. Measured results appear only after the exact setup passes qualification, a frozen campaign is calibrated, and a scoped release review approves publication.

What FreeBench measures

  • Outcome on real work. Whether the setup produced artifacts that pass protected behavioral checks for a versioned workload. A model's own statement that it succeeded is not evidence.
  • Practical access. Observed request limits, rate-limit refusals, context limits, reset behaviour, and time to complete within the frozen budget.
  • Complete denominators. Every scheduled episode stays in the record, including failures and aborted attempts. A pass rate is reported with the schedule it came from.
  • Reproducibility. The frozen configuration, suite hash, evidence hashes, runtime identity and collection date travel with the result.

What an exact model-and-harness pair includes

A published result names every element below. Changing any one of them creates a different setup, and results do not transfer across harness versions, endpoints, offers or task suites.

Provider and route
The gateway and endpoint origin, with the model identifier exactly as routed. A vendor API and a reseller route to the same model are different setups.
Model revision
The revision the provider exposes, or an explicit unavailable value when none is exposed. A revision is never guessed.
Harness and configuration
The coding agent or CLI, its version, and a hash of the configuration used for the run. Two harness versions are two different pairs.
Access and entitlement
The free tier, subscription or API-key class, the offer identity, and the date access terms were checked.
Runtime
The execution adapter, image digest, resource limits and the budget ceiling for the episode.
Workload
The versioned task package, grading policy and the frozen campaign that scheduled the episodes.

How a setup becomes qualified

  1. The route is confirmed available and free under the recorded access policy. A route that is not classified available is refused for a scored episode.
  2. The exact runtime and harness configuration is qualified against a controlled probe instead of being inferred from documentation.
  3. Identities, weights, repeats and budgets are frozen before execution begins.
  4. Protected graders evaluate collected artifacts. Task stdout, fabricated test output and model self-reports cannot establish success.
  5. A fresh campaign is calibrated and reviewed before any public release, and publication is scoped to the evidence that exists.

What “unmeasured” means

Unmeasured is not zero. When a setup has no qualified result, FreeBench shows no score, no rank and no implied ordering.

  • A free listing, a zero token price or a provider benchmark claim does not establish agent performance on a FreeBench workload.
  • Missing facts stay unknown. If a quota or revision cannot be confirmed from a source, the field says so instead of being estimated.
  • When a dimension cannot yet be measured, such as a confidence interval for a small fixed cohort, the result reports that dimension as unavailable with a reason.

How to read a result and its limits

  1. Read the identity first. Provider, route, model revision, harness, version, offer, runtime and workload. The result is only about that combination.
  2. Read the denominator. How many episodes were scheduled, attempted, administratively valid and successful. A rate without its denominator is not interpretable.
  3. Separate provider conditions from model failures. A rate-limit refusal or unavailable endpoint is recorded as a provider condition and excluded from scoring, not counted as a wrong answer.
  4. Check uncertainty. Small fixed cohorts may have no interval. An unavailable uncertainty field is a limit on the claim, not a sign of confidence.
  5. Check the date and the evidence. Free-tier terms change. Every result names when access was checked and links to the underlying record.

What FreeBench does not claim

FreeBench does not claim that one model is best for everyone, and it does not publish a ranking from unqualified data. Any comparison is workload-specific, tied to the exact pairs described above, and limited by sample size, coverage and date. Where the evidence does not support a comparison, the honest answer is that the comparison is unresolved.