Public benchmark context / external sources

Two boards. Two different evaluations.

These external rankings provide context for coding-agent offers. They are not FreeBench results. A score applies to the source's own tasks, harness, model, and setup. Each board stays separate, and unmatched source model names remain unchanged.

249 source entries loaded · no combined rank

External source · not a FreeBench evaluation

Aider Polyglot

The board reports the source pass_rate_2 percentage. Per-entry test counts, Aider command, edit format, source version, commit, and date are retained when published.

Last successful refresh: 2026-10-23T00:51:08.263Z · Fresh

Source observation: 2026-10-23T00:35:59.243Z

Open official source

69 entries in source order. Position is the source's display order, not a FreeBench rank.

Source position 1 · 2025-02-25

Gemini 2.0 Pro exp-02-05

35.6% · pass_rate_2

Published setup details

command: aider --model gemini/gemini-2.0-pro-exp-02-05
commit hash: 2fccd47
date: 2025-02-25
dirname: 2025-02-25-20-23-07--gemini-pro
edit format: whole
error outputs: 430
exhausted context windows: 0
indentation errors: 0
lazy comments: 0
model: Gemini 2.0 Pro exp-02-05
num malformed responses: 0
num with malformed responses: 0
pass num 1: 46
pass num 2: 80
pass rate 1: 20.4
pass rate 2: 35.6
percent cases well formed: 100
seconds per case: 34.8
syntax errors: 0
test cases: 225
test timeouts: 5
total cost: 0
total tests: 225
user asks: 13
versions: 0.75.2.dev
FreeBench model match: No explicit identity match; source name retained.

Source position 2 · 2024-12-21

gpt-4o-mini-2024-07-18

3.6% · pass_rate_2

Published setup details

command: aider --model gpt-4o-mini-2024-07-18
commit hash: a755079-dirty
date: 2024-12-21
dirname: 2024-12-21-18-41-18--polyglot-gpt-4o-mini
edit format: whole
error outputs: 0
exhausted context windows: 0
indentation errors: 0
lazy comments: 0
model: gpt-4o-mini-2024-07-18
num malformed responses: 0
num with malformed responses: 0
pass num 1: 2
pass num 2: 8
pass rate 1: 0.9
pass rate 2: 3.6
percent cases well formed: 100
seconds per case: 17.3
syntax errors: 0
test cases: 225
test timeouts: 3
total cost: 0.3236
total tests: 225
user asks: 36
versions: 0.69.2.dev
FreeBench model match: No explicit identity match; source name retained.

Source position 3 · 2025-01-17

claude-3-5-sonnet-20241022

51.6% · pass_rate_2

Published setup details

command: aider --model claude-3-5-sonnet-20241022
commit hash: 6451d59
date: 2025-01-17
dirname: 2025-01-17-19-44-33--sonnet-baseline-jan-17
edit format: diff
error outputs: 2
exhausted context windows: 1
indentation errors: 0
lazy comments: 0
model: claude-3-5-sonnet-20241022
num malformed responses: 1
num with malformed responses: 1
pass num 1: 50
pass num 2: 116
pass rate 1: 22.2
pass rate 2: 51.6
percent cases well formed: 99.6
seconds per case: 21.4
syntax errors: 0
test cases: 225
test timeouts: 8
total cost: 14.4063
total tests: 225
user asks: 11
versions: 0.71.2.dev
FreeBench model match: No explicit identity match; source name retained.

Source position 4 · 2024-12-30

gpt-4o-2024-11-20

18.2% · pass_rate_2

Published setup details

command: aider --model gpt-4o-2024-11-20
commit hash: 09ee197-dirty
date: 2024-12-30
dirname: 2024-12-30-20-57-12--gpt-4o-2024-11-20-ex-as-sys
edit format: diff
error outputs: 12
exhausted context windows: 0
indentation errors: 0
lazy comments: 0
model: gpt-4o-2024-11-20
num malformed responses: 12
num with malformed responses: 11
pass num 1: 11
pass num 2: 41
pass rate 1: 4.9
pass rate 2: 18.2
percent cases well formed: 95.1
seconds per case: 12.1
syntax errors: 0
test cases: 225
test timeouts: 12
total cost: 6.7351
total tests: 225
user asks: 53
versions: 0.70.1.dev
FreeBench model match: No explicit identity match; source name retained.

Source position 5 · 2024-12-30

gpt-4o-2024-08-06

23.1% · pass_rate_2

Published setup details

command: aider --model gpt-4o-2024-08-06
commit hash: 09ee197-dirty
date: 2024-12-30
dirname: 2024-12-30-20-44-54--gpt4o-ex-as-sys-clean-prompt
edit format: diff
error outputs: 21
exhausted context windows: 0
indentation errors: 0
lazy comments: 0
model: gpt-4o-2024-08-06
num malformed responses: 21
num with malformed responses: 13
pass num 1: 11
pass num 2: 52
pass rate 1: 4.9
pass rate 2: 23.1
percent cases well formed: 94.2
seconds per case: 16
syntax errors: 0
test cases: 225
test timeouts: 3
total cost: 7.0286
total tests: 225
user asks: 65
versions: 0.70.1.dev
FreeBench model match: No explicit identity match; source name retained.

Source position 6 · 2024-12-21

o1-2024-12-17 (high)

61.7% · pass_rate_2

Published setup details

command: aider --model openrouter/openai/o1
commit hash: a755079-dirty
date: 2024-12-21
dirname: 2024-12-21-19-23-03--polyglot-o1-hard-diff
edit format: diff
error outputs: 25
exhausted context windows: 0
indentation errors: 0
lazy comments: 0
model: o1-2024-12-17 (high)
num malformed responses: 24
num with malformed responses: 19
pass num 1: 53
pass num 2: 139
pass rate 1: 23.7
pass rate 2: 61.7
percent cases well formed: 91.5
seconds per case: 133.2
syntax errors: 0
test cases: 224
test timeouts: 2
total cost: 186.4958
total tests: 225
user asks: 16
versions: 0.69.2.dev
FreeBench model match: No explicit identity match; source name retained.

Source position 7 · 2024-12-21

DeepSeek Chat V2.5

17.8% · pass_rate_2

Published setup details

command: aider --model deepseek/deepseek-chat
commit hash: a755079-dirty
date: 2024-12-21
dirname: 2024-12-21-20-56-21--polyglot-deepseek-diff
edit format: diff
error outputs: 42
exhausted context windows: 5
indentation errors: 0
lazy comments: 0
model: DeepSeek Chat V2.5
num malformed responses: 37
num with malformed responses: 16
pass num 1: 12
pass num 2: 40
pass rate 1: 5.3
pass rate 2: 17.8
percent cases well formed: 92.9
seconds per case: 184
syntax errors: 0
test cases: 225
test timeouts: 5
total cost: 0.5101
total tests: 225
user asks: 23
versions: 0.69.2.dev
FreeBench model match: No explicit identity match; source name retained.

Source position 8 · 2024-12-21

claude-3-5-haiku-20241022

28.0% · pass_rate_2

Published setup details

command: aider --model claude-3-5-haiku-20241022
commit hash: a755079-dirty
date: 2024-12-21
dirname: 2024-12-21-21-46-27--polyglot-haiku-diff
edit format: diff
error outputs: 31
exhausted context windows: 1
indentation errors: 0
lazy comments: 0
model: claude-3-5-haiku-20241022
num malformed responses: 30
num with malformed responses: 20
pass num 1: 16
pass num 2: 63
pass rate 1: 7.1
pass rate 2: 28
percent cases well formed: 91.1
seconds per case: 31.8
syntax errors: 0
test cases: 225
test timeouts: 9
total cost: 6.0583
total tests: 225
user asks: 13
versions: 0.69.2.dev
FreeBench model match: No explicit identity match; source name retained.

Source position 9 · 2024-12-22

Qwen2.5-Coder-32B-Instruct

8.0% · pass_rate_2

Published setup details

command: aider --model openai/Qwen/Qwen2.5-Coder-32B-Instruct # via hyperbolic
commit hash: 6d7e8be-dirty
date: 2024-12-22
dirname: 2024-12-22-13-22-32--polyglot-qwen-diff
edit format: diff
error outputs: 158
exhausted context windows: 1
indentation errors: 0
lazy comments: 0
model: Qwen2.5-Coder-32B-Instruct
num malformed responses: 148
num with malformed responses: 64
pass num 1: 10
pass num 2: 18
pass rate 1: 4.4
pass rate 2: 8
percent cases well formed: 71.6
seconds per case: 84.4
syntax errors: 0
test cases: 225
test timeouts: 2
total cost: 0
total tests: 225
user asks: 132
versions: 0.69.2.dev
FreeBench model match: No explicit identity match; source name retained.

Source position 10 · 2024-12-22

o1-mini-2024-09-12

32.9% · pass_rate_2

Published setup details

command: aider --model o1-mini
commit hash: 37df899
date: 2024-12-22
dirname: 2024-12-22-21-26-35--polyglot-o1mini-whole
edit format: whole
error outputs: 8
exhausted context windows: 0
indentation errors: 0
lazy comments: 0
model: o1-mini-2024-09-12
num malformed responses: 8
num with malformed responses: 7
pass num 1: 13
pass num 2: 74
pass rate 1: 5.8
pass rate 2: 32.9
percent cases well formed: 96.9
seconds per case: 34.7
syntax errors: 0
test cases: 225
test timeouts: 3
total cost: 18.577
total tests: 225
user asks: 27
versions: 0.69.2.dev
FreeBench model match: No explicit identity match; source name retained.

Source position 11 · 2024-12-22

gemini-exp-1206

38.2% · pass_rate_2

Published setup details

command: aider --model gemini/gemini-exp-1206
commit hash: b1bc2f8
date: 2024-12-22
dirname: 2024-12-22-18-43-25--gemini-exp-1206-polyglot-whole-2
edit format: whole
error outputs: 8
exhausted context windows: 0
indentation errors: 0
lazy comments: 0
model: gemini-exp-1206
num malformed responses: 8
num with malformed responses: 4
pass num 1: 44
pass num 2: 86
pass rate 1: 19.6
pass rate 2: 38.2
percent cases well formed: 98.2
seconds per case: 45.5
syntax errors: 0
test cases: 225
test timeouts: 9
total cost: 0
total tests: 225
user asks: 32
versions: 0.69.2.dev
FreeBench model match: No explicit identity match; source name retained.

Source position 12 · 2024-12-22

gemini-2.0-flash-exp

22.2% · pass_rate_2

Published setup details

command: aider --model gemini/gemini-2.0-flash-exp
commit hash: b1bc2f8
date: 2024-12-22
dirname: 2024-12-22-20-08-13--gemini-2.0-flash-exp-polyglot-whole
edit format: whole
error outputs: 1
exhausted context windows: 1
indentation errors: 0
lazy comments: 0
model: gemini-2.0-flash-exp
num malformed responses: 0
num with malformed responses: 0
pass num 1: 26
pass num 2: 50
pass rate 1: 11.6
pass rate 2: 22.2
percent cases well formed: 100
seconds per case: 12.2
syntax errors: 0
test cases: 225
test timeouts: 8
total cost: 0
total tests: 225
user asks: 9
versions: 0.69.2.dev
FreeBench model match: No explicit identity match; source name retained.

Source position 13 · 2024-12-23

yi-lightning

12.9% · pass_rate_2

Published setup details

command: aider --model openai/yi-lightning
commit hash: 2b1625e
date: 2024-12-23
dirname: 2024-12-23-01-11-56--yi-test
edit format: whole
error outputs: 87
exhausted context windows: 1
indentation errors: 0
lazy comments: 0
model: yi-lightning
num malformed responses: 72
num with malformed responses: 16
pass num 1: 13
pass num 2: 29
pass rate 1: 5.8
pass rate 2: 12.9
percent cases well formed: 92.9
seconds per case: 146.7
syntax errors: 0
test cases: 225
test timeouts: 6
total cost: 0
total tests: 225
user asks: 107
versions: 0.69.2.dev
FreeBench model match: No explicit identity match; source name retained.

Source position 14 · 2024-12-25

DeepSeek Chat V3 (prev)

48.4% · pass_rate_2

Published setup details

command: aider --model deepseek/deepseek-chat
commit hash: 0a23c4a-dirty
date: 2024-12-25
dirname: 2024-12-25-13-31-51--deepseekv3preview-diff2
edit format: diff
error outputs: 7
exhausted context windows: 0
indentation errors: 0
lazy comments: 0
model: DeepSeek Chat V3 (prev)
num malformed responses: 7
num with malformed responses: 3
pass num 1: 51
pass num 2: 109
pass rate 1: 22.7
pass rate 2: 48.4
percent cases well formed: 98.7
seconds per case: 34.8
syntax errors: 0
test cases: 225
test timeouts: 8
total cost: 0.3369
total tests: 225
user asks: 19
versions: 0.69.2.dev
FreeBench model match: No explicit identity match; source name retained.

Source position 15 · 2024-12-26

Qwen2.5-Coder-32B-Instruct

16.4% · pass_rate_2

Published setup details

command: aider --model openai/Qwen2.5-Coder-32B-Instruct
commit hash: b51768b0
date: 2024-12-26
dirname: 2024-12-26-00-55-20--Qwen2.5-Coder-32B-Instruct
edit format: whole
error outputs: 1
exhausted context windows: 0
indentation errors: 0
lazy comments: 0
model: Qwen2.5-Coder-32B-Instruct
num malformed responses: 1
num with malformed responses: 1
pass num 1: 11
pass num 2: 37
pass rate 1: 4.9
pass rate 2: 16.4
percent cases well formed: 99.6
seconds per case: 42
syntax errors: 0
test cases: 225
test timeouts: 6
total cost: 0
total tests: 225
user asks: 33
versions: 0.69.2.dev
FreeBench model match: No explicit identity match; source name retained.

Source position 16 · 2025-01-13

Codestral 25.01

11.1% · pass_rate_2

Published setup details

command: aider --model mistral/codestral-latest
commit hash: 0cba898-dirty
date: 2025-01-13
dirname: 2025-01-13-18-17-25--codestral-whole2
edit format: whole
error outputs: 0
exhausted context windows: 0
indentation errors: 0
lazy comments: 0
model: Codestral 25.01
num malformed responses: 0
num with malformed responses: 0
pass num 1: 9
pass num 2: 25
pass rate 1: 4
pass rate 2: 11.1
percent cases well formed: 100
seconds per case: 9.3
syntax errors: 0
test cases: 225
test timeouts: 4
total cost: 1.9834
total tests: 225
user asks: 47
versions: 0.71.2.dev
FreeBench model match: No explicit identity match; source name retained.

Source position 17 · 2025-01-20

DeepSeek R1

56.9% · pass_rate_2

Published setup details

command: aider --model deepseek/deepseek-reasoner
commit hash: 5650697-dirty
date: 2025-01-20
dirname: 2025-01-20-19-11-38--ds-turns-upd-cur-msgs-fix-with-summarizer
edit format: diff
error outputs: 8
exhausted context windows: 1
indentation errors: 0
lazy comments: 0
model: DeepSeek R1
num malformed responses: 7
num with malformed responses: 7
pass num 1: 60
pass num 2: 128
pass rate 1: 26.7
pass rate 2: 56.9
percent cases well formed: 96.9
seconds per case: 113.7
syntax errors: 0
test cases: 225
test timeouts: 5
total cost: 5.4193
total tests: 225
user asks: 15
versions: 0.71.2.dev
FreeBench model match: No explicit identity match; source name retained.

Source position 18 · 2025-01-23

DeepSeek R1 + claude-3-5-sonnet-20241022

64.0% · pass_rate_2

Published setup details

command: aider --architect --model r1 --editor-model sonnet
commit hash: 05a77c7
date: 2025-01-23
dirname: 2025-01-23-19-14-48--r1-architect-sonnet
edit format: architect
editor edit format: editor-diff
editor model: claude-3-5-sonnet-20241022
error outputs: 2
exhausted context windows: 0
indentation errors: 0
lazy comments: 6
model: DeepSeek R1 + claude-3-5-sonnet-20241022
num malformed responses: 0
num with malformed responses: 0
pass num 1: 61
pass num 2: 144
pass rate 1: 27.1
pass rate 2: 64
percent cases well formed: 100
seconds per case: 251.6
syntax errors: 0
test cases: 225
test timeouts: 5
total cost: 13.2933
total tests: 225
user asks: 392
versions: 0.72.3.dev
FreeBench model match: No explicit identity match; source name retained.

Source position 19 · 2025-01-28

qwen-max-2025-01-25

21.8% · pass_rate_2

Published setup details

command: OPENAI_API_BASE=https://dashscope-intl.aliyuncs.com/compatible-mode/v1 aider --model openai/qwen-max-2025-01-25
commit hash: ae7d459
date: 2025-01-28
dirname: 2025-01-28-16-00-03--qwen-max-2025-01-25-polyglot-diff
edit format: diff
error outputs: 46
exhausted context windows: 0
indentation errors: 0
lazy comments: 0
model: qwen-max-2025-01-25
num malformed responses: 44
num with malformed responses: 22
pass num 1: 21
pass num 2: 49
pass rate 1: 9.3
pass rate 2: 21.8
percent cases well formed: 90.2
seconds per case: 39.5
syntax errors: 0
test cases: 225
test timeouts: 9
total tests: 225
user asks: 23
versions: 0.72.4.dev
FreeBench model match: No explicit identity match; source name retained.

Source position 20 · 2025-01-31

o3-mini (medium)

53.8% · pass_rate_2

Published setup details

command: aider --model o3-mini
commit hash: 2fb517b-dirty
date: 2025-01-31
dirname: 2025-01-31-20-27-46--o3-mini-diff2
edit format: diff
error outputs: 28
exhausted context windows: 0
indentation errors: 0
lazy comments: 0
model: o3-mini (medium)
num malformed responses: 28
num with malformed responses: 11
pass num 1: 43
pass num 2: 121
pass rate 1: 19.1
pass rate 2: 53.8
percent cases well formed: 95.1
seconds per case: 47.2
syntax errors: 0
test cases: 225
test timeouts: 2
total cost: 8.8599
total tests: 225
user asks: 17
versions: 0.72.4.dev
FreeBench model match: No explicit identity match; source name retained.

Source position 21 · 2025-01-31

o3-mini (high)

60.4% · pass_rate_2

Published setup details

command: aider --model o3-mini --reasoning-effort high
commit hash: b0d58d1-dirty
date: 2025-01-31
dirname: 2025-01-31-20-42-47--o3-mini-diff-high
edit format: diff
error outputs: 26
exhausted context windows: 1
indentation errors: 0
lazy comments: 0
model: o3-mini (high)
num malformed responses: 24
num with malformed responses: 15
pass num 1: 47
pass num 2: 136
pass rate 1: 21
pass rate 2: 60.4
percent cases well formed: 93.3
seconds per case: 124.6
syntax errors: 0
test cases: 224
test timeouts: 7
total cost: 18.1584
total tests: 225
user asks: 19
versions: 0.72.4.dev
FreeBench model match: No explicit identity match; source name retained.

Source position 22 · 2025-01-21

gemini-2.0-flash-thinking-exp-01-21

18.2% · pass_rate_2

Published setup details

command: aider --model gemini/gemini-2.0-flash-thinking-exp-01-21
commit hash: 843720a
date: 2025-01-21
dirname: 2025-01-21-22-51-49--gemini-2.0-flash-thinking-exp-01-21-polyglot-diff
edit format: diff
error outputs: 182
exhausted context windows: 2
indentation errors: 0
lazy comments: 0
model: gemini-2.0-flash-thinking-exp-01-21
num malformed responses: 180
num with malformed responses: 50
pass num 1: 13
pass num 2: 41
pass rate 1: 5.8
pass rate 2: 18.2
percent cases well formed: 77.8
seconds per case: 24.2
syntax errors: 0
test cases: 225
test timeouts: 7
total cost: 0
total tests: 225
user asks: 26
versions: 0.72.2.dev
FreeBench model match: No explicit identity match; source name retained.

Source position 23 · 2025-02-15

chatgpt-4o-latest (2025-02-15)

27.1% · pass_rate_2

Published setup details

command: aider --model chatgpt-4o-latest
commit hash: 108ce18-dirty
date: 2025-02-15
dirname: 2025-02-15-19-51-22--chatgpt4o-feb15-diff
edit format: diff
error outputs: 66
exhausted context windows: 0
indentation errors: 0
lazy comments: 0
model: chatgpt-4o-latest (2025-02-15)
num malformed responses: 21
num with malformed responses: 15
pass num 1: 20
pass num 2: 61
pass rate 1: 9
pass rate 2: 27.1
percent cases well formed: 93.3
seconds per case: 12.4
syntax errors: 0
test cases: 223
test timeouts: 2
total cost: 14.3703
total tests: 225
user asks: 57
versions: 0.74.3.dev
FreeBench model match: No explicit identity match; source name retained.

Source position 24 · 2025-02-24

claude-3-7-sonnet-20250219 (no thinking)

60.4% · pass_rate_2

Published setup details

command: aider --model sonnet
commit hash: 75e9ee6
date: 2025-02-24
dirname: 2025-02-24-19-54-07--sonnet37-diff
edit format: diff
error outputs: 16
exhausted context windows: 0
indentation errors: 0
lazy comments: 0
model: claude-3-7-sonnet-20250219 (no thinking)
num malformed responses: 16
num with malformed responses: 15
pass num 1: 55
pass num 2: 136
pass rate 1: 24.4
pass rate 2: 60.4
percent cases well formed: 93.3
seconds per case: 28.3
syntax errors: 0
test cases: 225
test timeouts: 0
total cost: 17.7191
total tests: 225
user asks: 12
versions: 0.74.4.dev
FreeBench model match: No explicit identity match; source name retained.

Source position 25 · 2025-02-24

claude-3-7-sonnet-20250219 (32k thinking tokens)

64.9% · pass_rate_2

Published setup details

command: aider --model anthropic/claude-3-7-sonnet-20250219 --thinking-tokens 32k
commit hash: 60d11a6, 93edbda
date: 2025-02-24
dirname: 2025-02-24-21-47-23--sonnet37-diff-think-32k-64k
edit format: diff
error outputs: 66
exhausted context windows: 0
indentation errors: 0
lazy comments: 0
model: claude-3-7-sonnet-20250219 (32k thinking tokens)
num malformed responses: 5
num with malformed responses: 5
pass num 1: 66
pass num 2: 146
pass rate 1: 29.3
pass rate 2: 64.9
percent cases well formed: 97.8
seconds per case: 105.2
syntax errors: 0
test cases: 225
test timeouts: 1
total cost: 36.8343
total tests: 225
user asks: 5
versions: 0.75.1.dev
FreeBench model match: No explicit identity match; source name retained.

Source position 26 · 2025-02-27

gpt-4.5-preview

44.9% · pass_rate_2

Published setup details

command: aider --model openai/gpt-4.5-preview
commit hash: b462e55-dirty
date: 2025-02-27
dirname: 2025-02-27-20-26-15--gpt45-diff3
edit format: diff
error outputs: 10
exhausted context windows: 1
indentation errors: 0
lazy comments: 0
model: gpt-4.5-preview
num malformed responses: 8
num with malformed responses: 6
pass num 1: 50
pass num 2: 101
pass rate 1: 22.3
pass rate 2: 44.9
percent cases well formed: 97.3
seconds per case: 113.5
syntax errors: 0
test cases: 224
test timeouts: 2
total cost: 183.1802
total tests: 225
user asks: 15
versions: 0.75.2.dev
FreeBench model match: No explicit identity match; source name retained.

Source position 27 · 2025-03-06

QwQ-32B

20.9% · pass_rate_2

Published setup details

command: aider --model fireworks_ai/accounts/fireworks/models/qwq-32b
commit hash: 51d118f-dirty
date: 2025-03-06
dirname: 2025-03-06-17-40-24--qwq32b-diff-temp-topp-ex-sys-remind-user-for-real
edit format: diff
error outputs: 145
exhausted context windows: 1
indentation errors: 0
lazy comments: 0
model: QwQ-32B
num malformed responses: 143
num with malformed responses: 73
pass num 1: 18
pass num 2: 47
pass rate 1: 8
pass rate 2: 20.9
percent cases well formed: 67.6
seconds per case: 228.6
syntax errors: 0
test cases: 225
test timeouts: 4
total cost: 0
total tests: 225
user asks: 17
versions: 0.75.3.dev
FreeBench model match: No explicit identity match; source name retained.

Source position 28 · 2025-03-07

QwQ-32B + Qwen 2.5 Coder Instruct

26.2% · pass_rate_2

Published setup details

command: aider --model fireworks_ai/accounts/fireworks/models/qwq-32b --architect
commit hash: 52162a5
date: 2025-03-07
dirname: 2025-03-07-15-11-27--qwq32b-arch-temp-topp-again
edit format: architect
editor edit format: editor-diff
editor model: fireworks_ai/accounts/fireworks/models/qwen2p5-coder-32b-instruct
error outputs: 122
exhausted context windows: 1
indentation errors: 0
lazy comments: 8
model: QwQ-32B + Qwen 2.5 Coder Instruct
num malformed responses: 0
num with malformed responses: 0
pass num 1: 22
pass num 2: 59
pass rate 1: 9.8
pass rate 2: 26.2
percent cases well formed: 100
seconds per case: 137.4
syntax errors: 0
test cases: 225
test timeouts: 2
total cost: 0
total tests: 225
user asks: 489
versions: 0.75.3.dev
FreeBench model match: No explicit identity match; source name retained.

Source position 29 · 2025-03-14

command-a-03-2025-quality

12.0% · pass_rate_2

Published setup details

command: OPENAI_API_BASE=https://api.cohere.ai/compatibility/v1 aider --model openai/command-a-03-2025-quality
commit hash: a1aa63f
date: 2025-03-14
dirname: 2025-03-14-23-40-00--cmda-quality-whole2
edit format: whole
error outputs: 2
exhausted context windows: 1
indentation errors: 0
lazy comments: 0
model: command-a-03-2025-quality
num malformed responses: 1
num with malformed responses: 1
pass num 1: 5
pass num 2: 27
pass rate 1: 2.2
pass rate 2: 12
percent cases well formed: 99.6
seconds per case: 85.1
syntax errors: 0
test cases: 225
test timeouts: 4
total cost: 0
total tests: 225
user asks: 215
versions: 0.77.1.dev
FreeBench model match: No explicit identity match; source name retained.

Source position 30 · 2025-03-15

gemma-3-27b-it

4.9% · pass_rate_2

Published setup details

command: aider --model openrouter/google/gemma-3-27b-it
commit hash: fd21f51-dirty
date: 2025-03-15
dirname: 2025-03-15-01-21-24--gemma3-27b-or
edit format: whole
error outputs: 3
exhausted context windows: 1
indentation errors: 0
lazy comments: 0
model: gemma-3-27b-it
num malformed responses: 0
num with malformed responses: 0
pass num 1: 4
pass num 2: 11
pass rate 1: 1.8
pass rate 2: 4.9
percent cases well formed: 100
seconds per case: 79.7
syntax errors: 0
test cases: 225
test timeouts: 3
total cost: 0
total tests: 225
user asks: 181
versions: 0.77.1.dev
FreeBench model match: No explicit identity match; source name retained.

Source position 31 · 2025-03-24

DeepSeek V3 (0324)

55.1% · pass_rate_2

Published setup details

command: aider --model deepseek/deepseek-chat
commit hash: 502b863
date: 2025-03-24
dirname: 2025-03-24-15-41-33--deepseek-v3-0324-polyglot-diff
edit format: diff
error outputs: 32
exhausted context windows: 2
indentation errors: 0
lazy comments: 0
model: DeepSeek V3 (0324)
num malformed responses: 1
num with malformed responses: 1
pass num 1: 63
pass num 2: 124
pass rate 1: 28
pass rate 2: 55.1
percent cases well formed: 99.6
seconds per case: 290
syntax errors: 0
test cases: 225
test timeouts: 4
total cost: 1.1164
total tests: 225
user asks: 96
versions: 0.78.1.dev
FreeBench model match: No explicit identity match; source name retained.

Source position 32 · 2025-04-12

Gemini 2.5 Pro Preview 03-25

72.9% · pass_rate_2

Published setup details

command: aider --model gemini/gemini-2.5-pro-preview-03-25
commit hash: 282574
date: 2025-04-12
dirname: 2025-04-12-04-55-50--gemini-25-pro-diff-fenced
edit format: diff-fenced
error outputs: 21
exhausted context windows: 0
indentation errors: 0
lazy comments: 0
model: Gemini 2.5 Pro Preview 03-25
num malformed responses: 21
num with malformed responses: 17
pass num 1: 92
pass num 2: 164
pass rate 1: 40.9
pass rate 2: 72.9
percent cases well formed: 92.4
seconds per case: 45.3
syntax errors: 0
test cases: 225
test timeouts: 2
total cost: 0
total tests: 225
user asks: 69
versions: 0.81.3.dev
FreeBench model match: No explicit identity match; source name retained.

Source position 33 · 2025-03-29

chatgpt-4o-latest (2025-03-29)

45.3% · pass_rate_2

Published setup details

command: aider --model chatgpt-4o-latest
commit hash: 0decbad
date: 2025-03-29
dirname: 2025-03-29-05-24-55--chatgpt4o-mar28-diff
edit format: diff
error outputs: 85
exhausted context windows: 0
indentation errors: 0
lazy comments: 0
model: chatgpt-4o-latest (2025-03-29)
num malformed responses: 85
num with malformed responses: 80
pass num 1: 37
pass num 2: 102
pass rate 1: 16.4
pass rate 2: 45.3
percent cases well formed: 64.4
seconds per case: 10.3
syntax errors: 0
test cases: 225
test timeouts: 4
total cost: 19.7416
total tests: 225
user asks: 174
versions: 0.79.3.dev
FreeBench model match: No explicit identity match; source name retained.

Source position 34 · 2025-04-04

Quasar Alpha

54.7% · pass_rate_2

Published setup details

command: aider --model openrouter/openrouter/quasar-alpha
commit hash: 8a34a6c-dirty
date: 2025-04-04
dirname: 2025-04-04-02-57-25--qalpha-diff-exsys
edit format: diff
error outputs: 4
exhausted context windows: 0
indentation errors: 0
lazy comments: 0
model: Quasar Alpha
num malformed responses: 4
num with malformed responses: 4
pass num 1: 49
pass num 2: 123
pass rate 1: 21.8
pass rate 2: 54.7
percent cases well formed: 98.2
seconds per case: 14.8
syntax errors: 0
test cases: 225
test timeouts: 4
total cost: 0
total tests: 225
user asks: 187
versions: 0.80.5.dev
FreeBench model match: No explicit identity match; source name retained.

Source position 35 · 2025-04-06

Llama 4 Maverick

15.6% · pass_rate_2

Published setup details

command: aider --model nvidia_nim/meta/llama-4-maverick-17b-128e-instruct
commit hash: 9445a31
date: 2025-04-06
dirname: 2025-04-06-08-39-52--llama-4-maverick-17b-128e-instruct-polyglot-whole
edit format: whole
error outputs: 12
exhausted context windows: 0
indentation errors: 0
lazy comments: 0
model: Llama 4 Maverick
num malformed responses: 2
num with malformed responses: 2
pass num 1: 10
pass num 2: 35
pass rate 1: 4.4
pass rate 2: 15.6
percent cases well formed: 99.1
seconds per case: 20.5
syntax errors: 0
test cases: 225
test timeouts: 4
total cost: 0
total tests: 225
user asks: 248
versions: 0.81.2.dev
FreeBench model match: No explicit identity match; source name retained.

Source position 36 · 2025-04-10

Grok 3 Beta

53.3% · pass_rate_2

Published setup details

command: aider --model openrouter/x-ai/grok-3-beta
commit hash: 2dd40fc-dirty
date: 2025-04-10
dirname: 2025-04-10-04-21-31--grok3-diff-exuser
edit format: diff
error outputs: 1
exhausted context windows: 0
indentation errors: 0
lazy comments: 0
model: Grok 3 Beta
num malformed responses: 1
num with malformed responses: 1
pass num 1: 50
pass num 2: 120
pass rate 1: 22.2
pass rate 2: 53.3
percent cases well formed: 99.6
seconds per case: 15.3
syntax errors: 0
test cases: 225
test timeouts: 2
total cost: 11.0338
total tests: 225
user asks: 68
versions: 0.81.2.dev
FreeBench model match: No explicit identity match; source name retained.

Source position 37 · 2025-04-10

Grok 3 Mini Beta (low)

34.7% · pass_rate_2

Published setup details

command: aider --model openrouter/x-ai/grok-3-mini-beta
commit hash: 14ffe77-dirty
date: 2025-04-10
dirname: 2025-04-10-18-47-24--grok3-mini-whole-exuser
edit format: whole
error outputs: 3
exhausted context windows: 0
indentation errors: 0
lazy comments: 0
model: Grok 3 Mini Beta (low)
num malformed responses: 0
num with malformed responses: 0
pass num 1: 25
pass num 2: 78
pass rate 1: 11.1
pass rate 2: 34.7
percent cases well formed: 100
seconds per case: 35.1
syntax errors: 0
test cases: 225
test timeouts: 5
total cost: 0.7856
total tests: 225
user asks: 73
versions: 0.81.2.dev
FreeBench model match: No explicit identity match; source name retained.

Source position 38 · 2025-04-10

Grok 3 Mini Beta (high)

49.3% · pass_rate_2

Published setup details

command: aider --model xai/grok-3-mini-beta --reasoning-effort high
commit hash: 8ee33da-dirty
date: 2025-04-10
dirname: 2025-04-10-23-59-02--xai-grok3-mini-whole-high
edit format: whole
error outputs: 1
exhausted context windows: 0
indentation errors: 0
lazy comments: 0
model: Grok 3 Mini Beta (high)
num malformed responses: 1
num with malformed responses: 1
pass num 1: 39
pass num 2: 111
pass rate 1: 17.3
pass rate 2: 49.3
percent cases well formed: 99.6
seconds per case: 79.1
syntax errors: 0
test cases: 225
test timeouts: 0
total cost: 0.7346
total tests: 225
user asks: 64
versions: 0.81.3.dev
FreeBench model match: No explicit identity match; source name retained.

Source position 39 · 2025-04-10

Optimus Alpha

52.9% · pass_rate_2

Published setup details

command: aider --model openrouter/openrouter/optimus-alpha
commit hash: 532bc45-dirty
date: 2025-04-10
dirname: 2025-04-10-19-02-44--oalpha-diff-exsys
edit format: diff
error outputs: 7
exhausted context windows: 0
indentation errors: 0
lazy comments: 0
model: Optimus Alpha
num malformed responses: 6
num with malformed responses: 6
pass num 1: 48
pass num 2: 119
pass rate 1: 21.3
pass rate 2: 52.9
percent cases well formed: 97.3
seconds per case: 18.4
syntax errors: 0
test cases: 225
test timeouts: 3
total cost: 0
total tests: 225
user asks: 182
versions: 0.81.2.dev
FreeBench model match: No explicit identity match; source name retained.

Source position 40 · 2025-04-14

gpt-4.1

52.4% · pass_rate_2

Published setup details

command: aider --model gpt-4.1
commit hash: 7a87db5-dirty
date: 2025-04-14
dirname: 2025-04-14-21-05-54--gpt41-diff-exuser
edit format: diff
error outputs: 6
exhausted context windows: 1
indentation errors: 0
lazy comments: 0
model: gpt-4.1
num malformed responses: 5
num with malformed responses: 4
pass num 1: 45
pass num 2: 118
pass rate 1: 20
pass rate 2: 52.4
percent cases well formed: 98.2
seconds per case: 20.5
syntax errors: 0
test cases: 225
test timeouts: 5
total cost: 9.8556
total tests: 225
user asks: 171
versions: 0.81.4.dev
FreeBench model match: No explicit identity match; source name retained.

Source position 41 · 2025-04-14

gpt-4.1-mini

32.4% · pass_rate_2

Published setup details

command: aider --model gpt-4.1-mini
commit hash: ffb743e-dirty
date: 2025-04-14
dirname: 2025-04-14-21-27-53--gpt41mini-diff
edit format: diff
error outputs: 64
exhausted context windows: 2
indentation errors: 0
lazy comments: 0
model: gpt-4.1-mini
num malformed responses: 62
num with malformed responses: 17
pass num 1: 25
pass num 2: 73
pass rate 1: 11.1
pass rate 2: 32.4
percent cases well formed: 92.4
seconds per case: 19.5
syntax errors: 0
test cases: 225
test timeouts: 2
total cost: 1.9918
total tests: 225
user asks: 159
versions: 0.81.4.dev
FreeBench model match: No explicit identity match; source name retained.

Source position 42 · 2025-04-14

gpt-4.1-nano

8.9% · pass_rate_2

Published setup details

command: aider --model gpt-4.1-nano
commit hash: 71d1591-dirty
date: 2025-04-14
dirname: 2025-04-14-22-46-01--gpt41nano-diff
edit format: whole
error outputs: 20
exhausted context windows: 0
indentation errors: 0
lazy comments: 0
model: gpt-4.1-nano
num malformed responses: 20
num with malformed responses: 13
pass num 1: 7
pass num 2: 20
pass rate 1: 3.1
pass rate 2: 8.9
percent cases well formed: 94.2
seconds per case: 12
syntax errors: 0
test cases: 225
test timeouts: 8
total cost: 0.4281
total tests: 225
user asks: 316
versions: 0.81.4.dev
FreeBench model match: No explicit identity match; source name retained.

Source position 43 · 2025-04-16

o4-mini (high)

72.0% · pass_rate_2

Published setup details

command: aider --model o4-mini
commit hash: b66901f-dirty
date: 2025-04-16
dirname: 2025-04-16-22-01-58--o4-mini-high-diff-exsys
edit format: diff
error outputs: 26
exhausted context windows: 1
indentation errors: 0
lazy comments: 0
model: o4-mini (high)
num malformed responses: 24
num with malformed responses: 21
pass num 1: 44
pass num 2: 162
pass rate 1: 19.6
pass rate 2: 72
percent cases well formed: 90.7
seconds per case: 176.5
syntax errors: 0
test cases: 225
test timeouts: 2
total cost: 19.6399
total tests: 225
user asks: 66
versions: 0.82.1.dev
FreeBench model match: No explicit identity match; source name retained.

Source position 44 · 2025-04-19

openhands-lm-32b-v0.1

10.2% · pass_rate_2

Published setup details

command: aider --model openrouter/all-hands/openhands-lm-32b-v0.1
commit hash: c08336f
date: 2025-04-19
dirname: 2025-04-19-14-43-04--o4-mini-patch
edit format: whole
error outputs: 55
exhausted context windows: 0
indentation errors: 0
lazy comments: 0
model: openhands-lm-32b-v0.1
num malformed responses: 41
num with malformed responses: 11
pass num 1: 9
pass num 2: 23
pass rate 1: 4
pass rate 2: 10.2
percent cases well formed: 95.1
seconds per case: 195.6
syntax errors: 0
test cases: 225
test timeouts: 11
total cost: 0
total tests: 225
user asks: 166
versions: 0.82.2.dev
FreeBench model match: No explicit identity match; source name retained.

Source position 45 · 2025-04-20

gemini-2.5-flash-preview-04-17 (default)

47.1% · pass_rate_2

Published setup details

command: aider --model gemini/gemini-2.5-flash-preview-04-17
commit hash: 7fcce5d-dirty
date: 2025-04-20
dirname: 2025-04-20-19-54-31--flash25-diff-no-think
edit format: diff
error outputs: 60
exhausted context windows: 5
indentation errors: 0
lazy comments: 1
model: gemini-2.5-flash-preview-04-17 (default)
num malformed responses: 55
num with malformed responses: 33
pass num 1: 49
pass num 2: 106
pass rate 1: 21.8
pass rate 2: 47.1
percent cases well formed: 85.3
seconds per case: 50.1
syntax errors: 0
test cases: 225
test timeouts: 4
total cost: 1.8451
total tests: 225
user asks: 82
versions: 0.82.3.dev
FreeBench model match: No explicit identity match; source name retained.

Source position 46 · 2025-05-07

Gemini 2.5 Pro Preview 05-06

76.9% · pass_rate_2

Published setup details

command: aider --model gemini/gemini-2.5-pro-preview-05-06
commit hash: 3b08327-dirty
date: 2025-05-07
dirname: 2025-05-07-19-32-40--gemini0506-diff-fenced-completion_cost
edit format: diff-fenced
error outputs: 15
exhausted context windows: 0
indentation errors: 0
lazy comments: 0
model: Gemini 2.5 Pro Preview 05-06
num malformed responses: 7
num with malformed responses: 6
pass num 1: 82
pass num 2: 173
pass rate 1: 36.4
pass rate 2: 76.9
percent cases well formed: 97.3
seconds per case: 165.3
syntax errors: 0
test cases: 225
test timeouts: 2
total cost: 37.4104
total tests: 225
user asks: 105
versions: 0.82.4.dev
FreeBench model match: No explicit identity match; source name retained.

Source position 47 · 2025-05-08

Qwen3 32B

40.0% · pass_rate_2

Published setup details

command: aider --model openrouter/qwen/qwen3-32b
commit hash: aaacee5-dirty, aeaf259
completion tokens: 120418
date: 2025-05-08
dirname: 2025-05-08-03-20-24--qwen3-32b-default
edit format: diff
error outputs: 119
exhausted context windows: 12
indentation errors: 0
lazy comments: 0
model: Qwen3 32B
num malformed responses: 50
num with malformed responses: 37
pass num 1: 32
pass num 2: 90
pass rate 1: 14.2
pass rate 2: 40
percent cases well formed: 83.6
prompt tokens: 317591
seconds per case: 372.2
syntax errors: 0
test cases: 225
test timeouts: 5
total cost: 0.7603
total tests: 225
user asks: 97
versions: 0.82.4.dev
FreeBench model match: No explicit identity match; source name retained.

Source position 48 · 2025-05-09

Qwen3 235B A22B diff, no think, Alibaba API

59.6% · pass_rate_2

Published setup details

command: aider --model openai/qwen3-235b-a22b
commit hash: 91d7fbd-dirty
completion tokens: 342062
date: 2025-05-09
dirname: 2025-05-09-17-02-02--qwen3-235b-a22b.unthink_16k_diff
edit format: diff
error outputs: 22
exhausted context windows: 0
indentation errors: 0
lazy comments: 0
model: Qwen3 235B A22B diff, no think, Alibaba API
num malformed responses: 22
num with malformed responses: 16
pass num 1: 65
pass num 2: 134
pass rate 1: 28.9
pass rate 2: 59.6
percent cases well formed: 92.9
prompt tokens: 2816192
seconds per case: 45.4
syntax errors: 0
test cases: 225
test timeouts: 1
total cost: 0
total tests: 225
user asks: 111
versions: 0.82.4.dev
FreeBench model match: No explicit identity match; source name retained.

Source position 49 · 2025-05-24

claude-sonnet-4-20250514 (no thinking)

56.4% · pass_rate_2

Published setup details

command: aider --model claude-sonnet-4-20250514
commit hash: ef3f8bb-dirty
completion tokens: 433373
date: 2025-05-24
dirname: 2025-05-24-21-17-54--sonnet4-diff-exuser
edit format: diff
error outputs: 6
exhausted context windows: 1
indentation errors: 0
lazy comments: 0
model: claude-sonnet-4-20250514 (no thinking)
num malformed responses: 4
num with malformed responses: 4
pass num 1: 46
pass num 2: 127
pass rate 1: 20.4
pass rate 2: 56.4
percent cases well formed: 98.2
prompt tokens: 3460663
seconds per case: 29.8
syntax errors: 0
test cases: 225
test timeouts: 7
total cost: 15.8155
total tests: 225
user asks: 129
versions: 0.83.3.dev
FreeBench model match: No explicit identity match; source name retained.

Source position 50 · 2025-05-24

claude-sonnet-4-20250514 (32k thinking)

61.3% · pass_rate_2

Published setup details

command: aider --model claude-sonnet-4-20250514
commit hash: e3cb907
completion tokens: 1271074
date: 2025-05-24
dirname: 2025-05-24-22-10-36--sonnet4-diff-exuser-think32k
edit format: diff
error outputs: 10
exhausted context windows: 0
indentation errors: 0
lazy comments: 0
model: claude-sonnet-4-20250514 (32k thinking)
num malformed responses: 10
num with malformed responses: 6
pass num 1: 58
pass num 2: 138
pass rate 1: 25.8
pass rate 2: 61.3
percent cases well formed: 97.3
prompt tokens: 2863068
seconds per case: 79.9
syntax errors: 0
test cases: 225
test timeouts: 6
thinking tokens: 32000
total cost: 26.5755
total tests: 225
user asks: 111
versions: 0.83.3.dev
FreeBench model match: No explicit identity match; source name retained.

Source position 51 · 2025-05-25

claude-opus-4-20250514 (no think)

70.7% · pass_rate_2

Published setup details

command: aider --model claude-opus-4-20250514
commit hash: 9ef3211
completion tokens: 380717
date: 2025-05-25
dirname: 2025-05-25-19-57-20--opus4-diff-exuser
edit format: diff
error outputs: 3
exhausted context windows: 0
indentation errors: 0
lazy comments: 0
model: claude-opus-4-20250514 (no think)
num malformed responses: 3
num with malformed responses: 3
pass num 1: 74
pass num 2: 159
pass rate 1: 32.9
pass rate 2: 70.7
percent cases well formed: 98.7
prompt tokens: 2671437
seconds per case: 42.5
syntax errors: 0
test cases: 225
test timeouts: 3
total cost: 68.6253
total tests: 225
user asks: 105
versions: 0.83.3.dev
FreeBench model match: No explicit identity match; source name retained.

Source position 52 · 2025-05-25

claude-opus-4-20250514 (32k thinking)

72.0% · pass_rate_2

Published setup details

command: aider --model claude-opus-4-20250514
commit hash: 9ef3211
completion tokens: 363142
date: 2025-05-25
dirname: 2025-05-25-20-40-51--opus4-diff-exuser
edit format: diff
error outputs: 10
exhausted context windows: 0
indentation errors: 0
lazy comments: 0
model: claude-opus-4-20250514 (32k thinking)
num malformed responses: 6
num with malformed responses: 6
pass num 1: 84
pass num 2: 162
pass rate 1: 37.3
pass rate 2: 72
percent cases well formed: 97.3
prompt tokens: 2567514
seconds per case: 44.1
syntax errors: 0
test cases: 225
test timeouts: 4
thinking tokens: 32000
total cost: 65.7484
total tests: 225
user asks: 97
versions: 0.83.3.dev
FreeBench model match: No explicit identity match; source name retained.

Source position 53 · 2025-05-26

gemini-2.5-flash-preview-05-20 (no think)

44.0% · pass_rate_2

Published setup details

command: aider --model gemini/gemini-2.5-flash-preview-05-20
commit hash: 214b811-dirty
completion tokens: 514145
date: 2025-05-26
dirname: 2025-05-26-15-56-31--flash25-05-20-24k-think
edit format: diff
error outputs: 16
exhausted context windows: 0
indentation errors: 0
lazy comments: 0
model: gemini-2.5-flash-preview-05-20 (no think)
num malformed responses: 16
num with malformed responses: 14
pass num 1: 47
pass num 2: 99
pass rate 1: 20.9
pass rate 2: 44
percent cases well formed: 93.8
prompt tokens: 5512458
seconds per case: 12.2
syntax errors: 0
test cases: 225
test timeouts: 4
thinking tokens: 0
total cost: 1.1354
total tests: 225
user asks: 79
versions: 0.83.3.dev
FreeBench model match: No explicit identity match; source name retained.

Source position 54 · 2025-05-25

gemini-2.5-flash-preview-05-20 (24k think)

55.1% · pass_rate_2

Published setup details

command: aider --model gemini/gemini-2.5-flash-preview-05-20
commit hash: a8568c3-dirty
completion tokens: 2703162
date: 2025-05-25
dirname: 2025-05-25-22-58-44--flash25-05-20-24k-think
edit format: diff
error outputs: 15
exhausted context windows: 0
indentation errors: 0
lazy comments: 0
model: gemini-2.5-flash-preview-05-20 (24k think)
num malformed responses: 15
num with malformed responses: 10
pass num 1: 59
pass num 2: 124
pass rate 1: 26.2
pass rate 2: 55.1
percent cases well formed: 95.6
prompt tokens: 3666792
seconds per case: 53.9
syntax errors: 0
test cases: 225
test timeouts: 4
thinking tokens: 24576
total cost: 8.5625
total tests: 225
user asks: 101
versions: 0.83.3.dev
FreeBench model match: No explicit identity match; source name retained.

Source position 55 · 2025-06-06

gemini-2.5-pro-preview-06-05 (default think)

79.1% · pass_rate_2

Published setup details

command: aider --model gemini/gemini-2.5-pro-preview-06-05
commit hash: 4c161f9-dirty
completion tokens: 4142197
date: 2025-06-06
dirname: 2025-06-06-18-38-56--gemini0605-diff-fenced
edit format: diff-fenced
error outputs: 4
exhausted context windows: 4
indentation errors: 0
lazy comments: 0
model: gemini-2.5-pro-preview-06-05 (default think)
num malformed responses: 0
num with malformed responses: 0
pass num 1: 101
pass num 2: 178
pass rate 1: 44.9
pass rate 2: 79.1
percent cases well formed: 100
prompt tokens: 2751296
seconds per case: 175.2
syntax errors: 0
test cases: 225
test timeouts: 1
total cost: 45.5961
total tests: 225
user asks: 105
versions: 0.84.1.dev
FreeBench model match: No explicit identity match; source name retained.

Source position 56 · 2025-06-06

gemini-2.5-pro-preview-06-05 (32k think)

83.1% · pass_rate_2

Published setup details

command: aider --model gemini/gemini-2.5-pro-preview-06-05 --thinking-tokens 32k
commit hash: f827f22
completion tokens: 4648227
date: 2025-06-06
dirname: 2025-06-06-16-36-21--gemini0605-32k-think-diff-fenced
edit format: diff-fenced
error outputs: 1
exhausted context windows: 0
indentation errors: 0
lazy comments: 0
model: gemini-2.5-pro-preview-06-05 (32k think)
num malformed responses: 1
num with malformed responses: 1
pass num 1: 104
pass num 2: 187
pass rate 1: 46.2
pass rate 2: 83.1
percent cases well formed: 99.6
prompt tokens: 2719961
seconds per case: 200.3
syntax errors: 0
test cases: 225
test timeouts: 0
thinking tokens: 32768
total cost: 49.8822
total tests: 225
user asks: 112
versions: 0.84.1.dev
FreeBench model match: No explicit identity match; source name retained.

Source position 57 · 2025-06-06

DeepSeek R1 (0528)

71.4% · pass_rate_2

Published setup details

command: aider --model deepseek/deepseek-reasoner
commit hash: 4c161f9-dirty
completion tokens: 1842168
date: 2025-06-06
dirname: 2025-06-06-16-47-07--r1-diff
edit format: diff
error outputs: 28
exhausted context windows: 0
indentation errors: 0
lazy comments: 0
model: DeepSeek R1 (0528)
num malformed responses: 15
num with malformed responses: 12
pass num 1: 77
pass num 2: 160
pass rate 1: 34.4
pass rate 2: 71.4
percent cases well formed: 94.6
prompt tokens: 2644169
seconds per case: 716.6
syntax errors: 0
test cases: 224
test timeouts: 2
total cost: 4.8016
total tests: 225
user asks: 105
versions: 0.84.1.dev
FreeBench model match: No explicit identity match; source name retained.

Source position 58 · 2025-06-25

o3 (high)

81.3% · pass_rate_2

Published setup details

command: aider --model o3 --reasoning-effort high
commit hash: c48fea6
completion tokens: 2047615
date: 2025-06-25
dirname: 2025-06-25-21-04-24--o3-price-reduction-high
edit format: diff
error outputs: 25
exhausted context windows: 1
indentation errors: 0
lazy comments: 0
model: o3 (high)
num malformed responses: 23
num with malformed responses: 12
pass num 1: 90
pass num 2: 183
pass rate 1: 40
pass rate 2: 81.3
percent cases well formed: 94.7
prompt tokens: 3148932
reasoning effort: high
seconds per case: 197.3
syntax errors: 0
test cases: 225
test timeouts: 2
total cost: 21.2259
total tests: 225
user asks: 116
versions: 0.84.1.dev
FreeBench model match: No explicit identity match; source name retained.

Source position 59 · 2025-06-25

o3

76.9% · pass_rate_2

Published setup details

command: aider --model o3
commit hash: c48fea6
completion tokens: 1154767
date: 2025-06-25
dirname: 2025-06-25-20-30-16--o3-price-reduction
edit format: diff
error outputs: 22
exhausted context windows: 0
indentation errors: 0
lazy comments: 2
model: o3
num malformed responses: 22
num with malformed responses: 14
pass num 1: 92
pass num 2: 173
pass rate 1: 40.9
pass rate 2: 76.9
percent cases well formed: 93.8
prompt tokens: 2893189
seconds per case: 101.7
syntax errors: 0
test cases: 225
test timeouts: 1
total cost: 13.7517
total tests: 225
user asks: 108
versions: 0.84.1.dev
FreeBench model match: No explicit identity match; source name retained.

Source position 60 · 2025-06-27

o3 (high) + gpt-4.1

78.2% · pass_rate_2

Published setup details

command: aider --model o3
commit hash: 4f4f00f-dirty
completion tokens: 1327154
date: 2025-06-27
dirname: 2025-06-27-23-53-57--o3-mini-high-diff-arch
edit format: architect
editor edit format: editor-diff
editor model: gpt-4.1
error outputs: 18
exhausted context windows: 1
indentation errors: 0
lazy comments: 0
model: o3 (high) + gpt-4.1
num malformed responses: 0
num with malformed responses: 0
pass num 1: 78
pass num 2: 176
pass rate 1: 34.8
pass rate 2: 78.2
percent cases well formed: 100
prompt tokens: 1306877
reasoning effort: high
seconds per case: 121.8
syntax errors: 0
test cases: 224
test timeouts: 1
total cost: 17.5518
total tests: 225
user asks: 172
versions: 0.85.1.dev
FreeBench model match: No explicit identity match; source name retained.

Source position 61 · 2025-06-28

o3-pro (high)

84.9% · pass_rate_2

Published setup details

command: aider --model o3-pro
commit hash: 5318380
completion tokens: 1235902
date: 2025-06-28
dirname: 2025-06-28-00-38-18--o3-pro-high
edit format: diff
error outputs: 20
exhausted context windows: 0
indentation errors: 0
lazy comments: 0
model: o3-pro (high)
num malformed responses: 8
num with malformed responses: 5
pass num 1: 98
pass num 2: 191
pass rate 1: 43.6
pass rate 2: 84.9
percent cases well formed: 97.8
prompt tokens: 2372636
reasoning effort: high
seconds per case: 449
syntax errors: 0
test cases: 225
test timeouts: 1
total cost: 146.3249
total tests: 225
user asks: 100
versions: 0.85.1.dev
FreeBench model match: No explicit identity match; source name retained.

Source position 62 · 2025-07-11

grok-4 (high)

79.6% · pass_rate_2

Published setup details

command: aider --model openrouter/x-ai/grok-4
commit hash: f7870b6-dirty
completion tokens: 3411480
date: 2025-07-11
dirname: 2025-07-11-19-37-40--xai-or-grok4-high
edit format: diff
error outputs: 11
exhausted context windows: 0
indentation errors: 0
lazy comments: 0
model: grok-4 (high)
num malformed responses: 8
num with malformed responses: 6
pass num 1: 92
pass num 2: 179
pass rate 1: 40.9
pass rate 2: 79.6
percent cases well formed: 97.3
prompt tokens: 2815347
reasoning effort: high
seconds per case: 403.2
syntax errors: 0
test cases: 225
test timeouts: 0
total cost: 59.6182
total tests: 225
user asks: 133
versions: 0.85.2.dev
FreeBench model match: No explicit identity match; source name retained.

Source position 63 · 2025-07-17

Kimi K2

59.1% · pass_rate_2

Published setup details

command: aider --model openrouter/moonshotai/kimi-k2
commit hash: 915ebff-dirty
completion tokens: 363846
date: 2025-07-17
dirname: 2025-07-17-17-41-54--kimi-k2-diff-or-pricing
edit format: diff
error outputs: 19
exhausted context windows: 0
indentation errors: 0
lazy comments: 0
model: Kimi K2
num malformed responses: 19
num with malformed responses: 16
pass num 1: 46
pass num 2: 133
pass rate 1: 20.4
pass rate 2: 59.1
percent cases well formed: 92.9
prompt tokens: 2355141
seconds per case: 67.6
syntax errors: 0
test cases: 225
test timeouts: 4
total cost: 1.2357
total tests: 225
user asks: 61
versions: 0.85.3.dev
FreeBench model match: No explicit identity match; source name retained.

Source position 64 · 2025-08-06

gpt-oss-120b (high)

41.8% · pass_rate_2

Published setup details

command: aider --model openrouter/openai/gpt-oss-120b --reasoning-effort high
commit hash: 1af0e59
completion tokens: 856495
date: 2025-08-06
dirname: 2025-08-06-04-54-48--gpt-oss-120b-high-polyglot
edit format: diff
error outputs: 95
exhausted context windows: 0
indentation errors: 0
lazy comments: 0
model: gpt-oss-120b (high)
num malformed responses: 77
num with malformed responses: 47
pass num 1: 31
pass num 2: 94
pass rate 1: 13.8
pass rate 2: 41.8
percent cases well formed: 79.1
prompt tokens: 3123768
seconds per case: 35.5
syntax errors: 0
test cases: 225
test timeouts: 4
total cost: 0.7406
total tests: 225
user asks: 142
versions: 0.85.3.dev
FreeBench model match: No explicit identity match; source name retained.

Source position 65 · 2025-08-23

gpt-5 (high)

88.0% · pass_rate_2

Published setup details

command: aider --model openai/gpt-5
commit hash: 32faf82
completion tokens: 2623429
date: 2025-08-23
dirname: 2025-08-23-15-47-21--gpt-5-high
edit format: diff
error outputs: 23
exhausted context windows: 0
indentation errors: 0
lazy comments: 3
model: gpt-5 (high)
num malformed responses: 22
num with malformed responses: 19
pass num 1: 117
pass num 2: 198
pass rate 1: 52
pass rate 2: 88
percent cases well formed: 91.6
prompt tokens: 2675561
reasoning effort: high
seconds per case: 194
syntax errors: 0
test cases: 225
test timeouts: 3
total cost: 29.0829
total tests: 225
user asks: 96
versions: 0.86.2.dev
FreeBench model match: No explicit identity match; source name retained.

Source position 66 · 2025-08-25

gpt-5 (medium)

86.7% · pass_rate_2

Published setup details

command: aider --model openai/gpt-5
commit hash: 32faf82
completion tokens: 1468799
date: 2025-08-25
dirname: 2025-08-25-13-23-27--gpt-5-medium
edit format: diff
error outputs: 40
exhausted context windows: 0
indentation errors: 0
lazy comments: 0
model: gpt-5 (medium)
num malformed responses: 40
num with malformed responses: 26
pass num 1: 112
pass num 2: 195
pass rate 1: 49.8
pass rate 2: 86.7
percent cases well formed: 88.4
prompt tokens: 2827261
reasoning effort: medium
seconds per case: 118.7
syntax errors: 0
test cases: 225
test timeouts: 0
total cost: 17.693
total tests: 225
user asks: 102
versions: 0.86.2.dev
FreeBench model match: No explicit identity match; source name retained.

Source position 67 · 2025-08-25

gpt-5 (low)

81.3% · pass_rate_2

Published setup details

command: aider --model openai/gpt-5
commit hash: 32faf82
completion tokens: 779568
date: 2025-08-25
dirname: 2025-08-25-14-16-37--gpt-5-low
edit format: diff
error outputs: 46
exhausted context windows: 0
indentation errors: 0
lazy comments: 1
model: gpt-5 (low)
num malformed responses: 46
num with malformed responses: 30
pass num 1: 97
pass num 2: 183
pass rate 1: 43.1
pass rate 2: 81.3
percent cases well formed: 86.7
prompt tokens: 2534059
reasoning effort: low
seconds per case: 62.4
syntax errors: 0
test cases: 225
test timeouts: 1
total cost: 10.3713
total tests: 225
user asks: 113
versions: 0.86.2.dev
FreeBench model match: No explicit identity match; source name retained.

Source position 68 · 2025-10-03

DeepSeek-V3.2-Exp (Reasoner)

74.2% · pass_rate_2

Published setup details

command: aider --model deepseek/deepseek-reasoner
commit hash: cbb5376
completion tokens: 1645129
date: 2025-10-03
dirname: 2025-10-03-09-45-34--deepseek-v3.2-reasoner
edit format: diff
error outputs: 8
exhausted context windows: 1
indentation errors: 0
lazy comments: 0
model: DeepSeek-V3.2-Exp (Reasoner)
num malformed responses: 6
num with malformed responses: 6
pass num 1: 89
pass num 2: 167
pass rate 1: 39.6
pass rate 2: 74.2
percent cases well formed: 97.3
prompt tokens: 2191446
seconds per case: 291.2
syntax errors: 0
test cases: 225
test timeouts: 1
total cost: 1.3045
total tests: 225
user asks: 67
versions: 0.86.2.dev
FreeBench model match: No explicit identity match; source name retained.

Source position 69 · 2025-10-03

DeepSeek-V3.2-Exp (Chat)

70.2% · pass_rate_2

Published setup details

command: aider --model deepseek/deepseek-chat
commit hash: cbb5376
completion tokens: 573477
date: 2025-10-03
dirname: 2025-10-03-09-21-36--deepseek-v3.2-chat
edit format: diff
error outputs: 6
exhausted context windows: 1
indentation errors: 0
lazy comments: 0
model: DeepSeek-V3.2-Exp (Chat)
num malformed responses: 4
num with malformed responses: 4
pass num 1: 87
pass num 2: 158
pass rate 1: 38.7
pass rate 2: 70.2
percent cases well formed: 98.2
prompt tokens: 2266868
seconds per case: 104
syntax errors: 0
test cases: 225
test timeouts: 4
total cost: 0.8756
total tests: 225
user asks: 60
versions: 0.86.2.dev
FreeBench model match: No explicit identity match; source name retained.

External source · not a FreeBench evaluation

SWE-bench Verified

The board reports the source resolved percentage for SWE-bench Verified. Per-entry agent/system labels, model labels, checked field, date, tags, and reasoning effort are retained when published.

Last successful refresh: 2026-10-23T00:51:08.877Z · Fresh

Source observation: 2026-10-23T00:35:59.768Z

Open official source

180 entries in source order. Position is the source's display order, not a FreeBench rank.

Source position 1 · 2025-12-05

Claude 4.5 Opus

79.2% · resolved

Published setup details

agent: Sonar Foundation Agent
agent org: Sonar
checked: false
cost: Unknown
date: 2025-12-05
folder: 20251205_sonar-foundation-agent_claude-opus-4-5
instance calls: Unknown
instance cost: Unknown
logo: img/logos/20251205_sonar-foundation-agent_claude-opus-4-5.png
logs: s3://swe-bench-submissions/verified/20251205_sonar-foundation-agent_claude-opus-4-5/logs
model display: Claude 4.5 Opus
model org: Anthropic
model release date: Unknown
name: Sonar Foundation Agent + Claude 4.5 Opus
os model: false
os system: false
reasoning effort: Unknown
resolved: 79.2
site: https://www.sonarsource.com
submission: Sonar Foundation Agent + Claude 4.5 Opus
tags: Model: claude-opus-4-5 · Org: Sonar · System: Attempts - 1
trajs: s3://swe-bench-submissions/verified/20251205_sonar-foundation-agent_claude-opus-4-5/trajs
trajs docent: false
warning: Unknown
FreeBench model match: No explicit identity match; source name retained.

Source position 2 · 2025-12-15

Claude 4.5 Opus

79.2% · resolved

Published setup details

agent: live-SWE-agent
agent org: UIUC
checked: false
cost: Unknown
date: 2025-12-15
folder: 20251215_livesweagent_claude-opus-4-5
instance calls: Unknown
instance cost: Unknown
logo: img/logos/20251215_livesweagent_claude-opus-4-5.png
logs: s3://swe-bench-submissions/verified/20251215_livesweagent_claude-opus-4-5/logs
model display: Claude 4.5 Opus
model org: Anthropic
model release date: Unknown
name: live-SWE-agent + Claude 4.5 Opus medium (20251101)
os model: false
os system: true
reasoning effort: medium
resolved: 79.2
site: https://github.com/OpenAutoCoder/live-swe-agent
submission: live-SWE-agent + Claude 4.5 Opus medium (20251101)
tags: Model: claude-opus-4-5-20251101 · Org: UIUC · System: Attempts - 1
trajs: s3://swe-bench-submissions/verified/20251215_livesweagent_claude-opus-4-5/trajs
trajs docent: false
warning: Unknown
FreeBench model match: No explicit identity match; source name retained.

Source position 3 · 2025-09-28

Doubao-Seed-Code

78.8% · resolved

Published setup details

agent: TRAE
agent org: ByteDance
checked: false
cost: Unknown
date: 2025-09-28
folder: 20250928_trae_doubao_seed_code
instance calls: Unknown
instance cost: Unknown
logo: img/logos/20250928_trae_doubao_seed_code.jpg
logs: s3://swe-bench-submissions/verified/20250928_trae_doubao_seed_code/logs
model display: Doubao-Seed-Code
model org: ByteDance
model release date: Unknown
name: TRAE + Doubao-Seed-Code
os model: false
os system: true
reasoning effort: Unknown
resolved: 78.8
site: https://github.com/bytedance/trae-agent · https://seed.bytedance.com/
submission: TRAE + Doubao-Seed-Code
tags: Model: Doubao-Seed-Code · Model: Doubao-Seed-1.6 · Org: ByteDance · System: Attempts - 2+
trajs: s3://swe-bench-submissions/verified/20250928_trae_doubao_seed_code/trajs
trajs docent: false
warning: Unknown
FreeBench model match: No explicit identity match; source name retained.

Source position 4 · 2025-11-20

Gemini 3 Pro Preview

77.4% · resolved

Published setup details

agent: live-SWE-agent
agent org: UIUC
checked: false (See README.md for info on how to get your results verified)
cost: Unknown
date: 2025-11-20
folder: 20251120_livesweagent_gemini-3-pro-preview
instance calls: Unknown
instance cost: Unknown
logo: img/logos/20251120_livesweagent_gemini-3-pro-preview.png
logs: s3://swe-bench-submissions/verified/20251120_livesweagent_gemini-3-pro-preview/logs
model display: Gemini 3 Pro Preview
model org: Google DeepMind
model release date: Unknown
name: live-SWE-agent + Gemini 3 Pro Preview (2025-11-18)
os model: false
os system: true
reasoning effort: Unknown
resolved: 77.4
site: https://github.com/OpenAutoCoder/live-swe-agent
submission: live-SWE-agent + Gemini 3 Pro Preview (2025-11-18)
tags: Model: gemini-3-pro-preview · Org: UIUC · System: Attempts - 1
trajs: s3://swe-bench-submissions/verified/20251120_livesweagent_gemini-3-pro-preview/trajs
trajs docent: false
warning: Unknown
FreeBench model match: No explicit identity match; source name retained.

Source position 5 · 2025-08-04

Claude 4 Sonnet

76.8% · resolved

Published setup details

agent: EPAM AI/Run Developer Agent
agent org: EPAM Systems, Inc.
checked: false
cost: Unknown
date: 2025-08-04
folder: 20250804_epam-ai-run-claude-4-sonnet
instance calls: Unknown
instance cost: Unknown
logo: img/logos/20250804_epam-ai-run-claude-4-sonnet.jpg
logs: s3://swe-bench-submissions/verified/20250804_epam-ai-run-claude-4-sonnet/logs
model display: Claude 4 Sonnet
model org: Anthropic
model release date: Unknown
name: EPAM AI/Run Developer Agent v20250719 + Claude 4 Sonnet
os model: false
os system: false
reasoning effort: Unknown
resolved: 76.8
site: https://www.epam.com/services/artificial-intelligence
submission: EPAM AI/Run Developer Agent v20250719 + Claude 4 Sonnet
tags: Model: claude-sonnet-4-20250514 · Org: EPAM Systems, Inc. · System: Attempts - 2+
trajs: s3://swe-bench-submissions/verified/20250804_epam-ai-run-claude-4-sonnet/trajs
trajs docent: false
warning: Unknown
FreeBench model match: No explicit identity match; source name retained.

Source position 6 · 2025-09-02

Multiple

76.8% · resolved

Published setup details

agent: Atlassian Rovo Dev
agent org: Atlassian
checked: false
cost: Unknown
date: 2025-09-02
folder: 20250902_atlassian-rovo-dev
instance calls: Unknown
instance cost: Unknown
logo: img/logos/20250902_atlassian-rovo-dev.png
logs: s3://swe-bench-submissions/verified/20250902_atlassian-rovo-dev/logs
model display: Multiple
model org: Anthropic
model release date: Unknown
name: Atlassian Rovo Dev (2025-09-02)
os model: false
os system: false
reasoning effort: Unknown
resolved: 76.8
site: https://www.atlassian.com/rovo-dev
submission: Atlassian Rovo Dev (2025-09-02)
tags: Model: claude-sonnet-4-20250514 · Model: gpt-5 · Org: Atlassian · System: Attempts - 2
trajs: s3://swe-bench-submissions/verified/20250902_atlassian-rovo-dev/trajs
trajs docent: false
warning: Unknown
FreeBench model match: No explicit identity match; source name retained.

Source position 7 · 2026-02-17

Claude 4.5 Opus

76.8% · resolved

Published setup details

agent: mini-SWE-agent
agent org: SWE-agent
checked: Unknown
cost: 376.9539985
date: 2026-02-17
folder: 20260217_mini-v2.0.0_claude-4-5-opus-high
instance calls: 32.896
instance cost: 0.753907997
logo: img/logos/20260217_mini-v2.0.0_claude-4-5-opus-high.jpg · img/mini-icon.svg
logs: Unknown
mini-swe-agent version: 2.0.0
model display: Claude 4.5 Opus
model org: Anthropic
model release date: 20251124
name: Claude 4.5 Opus (high)
os model: false
os system: true
reasoning effort: high
resolved: 76.8
site: https://www.anthropic.com/claude
submission: Claude 4.5 Opus (high)
tags: Model: claude-4-5-opus · Org: Anthropic · System: Attempts - 1 · Mini: 2.0.0
trajs: s3://swe-bench-submissions/bash-only/20260217_mini-v2.0.0_claude-4-5-opus-high/trajs
trajs docent: https://docent.transluce.org/dashboard/b038912e-0133-4594-b093-92806f8ffb17
warning: Unknown
FreeBench model match: No explicit identity match; source name retained.

Source position 8 · 2025-08-19

Multiple

76.4% · resolved

Published setup details

agent: ACoder
agent org: ACoder
checked: false
cost: Unknown
date: 2025-08-19
folder: 20250819_ACoder
instance calls: Unknown
instance cost: Unknown
logo: img/logos/20250819_ACoder.png
logs: s3://swe-bench-submissions/verified/20250819_ACoder/logs
model display: Multiple
model org: Anthropic
model release date: Unknown
name: ACoder
os model: false
os system: false
reasoning effort: Unknown
resolved: 76.4
site: https://github.com/ACoder-AI/ACoder
submission: ACoder
tags: Model: claude-4-sonnet · Model: claude-4.1-opus · Model: gpt-5-0807-global · Model: gemini-2.5-pro-06-17 · Org: ACoder · System: Attempts - 2+
trajs: s3://swe-bench-submissions/verified/20250819_ACoder/trajs
trajs docent: false
warning: Unknown
FreeBench model match: No explicit identity match; source name retained.

Source position 9 · 2026-02-17

Gemini 3 Flash

75.8% · resolved

Published setup details

agent: mini-SWE-agent
agent org: SWE-agent
checked: Unknown
cost: 177.98206539999998
date: 2026-02-17
folder: 20260217_mini-v2.0.0_gemini-3-flash-high
instance calls: 56.126
instance cost: 0.35596413079999994
logo: img/logos/20260217_mini-v2.0.0_gemini-3-flash-high.svg · img/mini-icon.svg
logs: Unknown
mini-swe-agent version: 2.0.0
model display: Gemini 3 Flash
model org: Google DeepMind
model release date: 20251217
name: Gemini 3 Flash (high)
os model: false
os system: true
reasoning effort: high
resolved: 75.8
site: https://ai.google.dev/gemini-api/docs/models
submission: Gemini 3 Flash (high)
tags: Model: gemini-3-flash-preview · Org: Google DeepMind · System: Attempts - 1 · Mini: 2.0.0
trajs: s3://swe-bench-submissions/bash-only/20260217_mini-v2.0.0_gemini-3-flash-high/trajs
trajs docent: https://docent.transluce.org/dashboard/1ebbdd7a-55b3-4015-9b83-5978cc7fb618
warning: Unknown
FreeBench model match: No explicit identity match; source name retained.

Source position 10 · 2026-02-17

MiniMax M2.5

75.8% · resolved

Published setup details

agent: mini-SWE-agent
agent org: SWE-agent
checked: Unknown
cost: 36.644993922
date: 2026-02-17
folder: 20260217_mini-v2.0.0_minimax-2-5-high
instance calls: 60.45
instance cost: 0.073289987844
logo: img/logos/20260217_mini-v2.0.0_minimax-2-5-high.png · img/mini-icon.svg
logs: Unknown
mini-swe-agent version: 2.0.0
model display: MiniMax M2.5
model org: Minimax
model release date: 20260212
name: MiniMax M2.5 (high)
os model: true
os system: true
reasoning effort: high
resolved: 75.8
site: https://www.minimax.io
submission: MiniMax M2.5 (high)
tags: Model: minimax-m2.5 · Org: Minimax · System: Attempts - 1 · Mini: 2.0.0
trajs: s3://swe-bench-submissions/bash-only/20260217_mini-v2.0.0_minimax-2-5-high/trajs
trajs docent: https://docent.transluce.org/dashboard/5b77e003-7328-4003-879e-9b55dd3a0b6f
warning: Unknown
FreeBench model match: No explicit identity match; source name retained.

Source position 11 · 2025-09-01

Multiple

75.6% · resolved

Published setup details

agent: Warp
agent org: Warp
checked: false
cost: Unknown
date: 2025-09-01
folder: 20250901_warp
instance calls: Unknown
instance cost: Unknown
logo: img/logos/20250901_warp.png
logs: s3://swe-bench-submissions/verified/20250901_warp/logs
model display: Multiple
model org: OpenAI
model release date: Unknown
name: Warp
os model: false
os system: false
reasoning effort: Unknown
resolved: 75.6
site: https://www.warp.dev
submission: Warp
tags: Model: gpt-5 · Model: claude-sonnet-4-20250514 · Org: Warp · System: Attempts - 2+
trajs: s3://swe-bench-submissions/verified/20250901_warp/trajs
trajs docent: false
warning: Unknown
FreeBench model match: No explicit identity match; source name retained.

Source position 12 · 2026-02-17

Claude 4.6 Opus

75.6% · resolved

Published setup details

agent: mini-SWE-agent
agent org: SWE-agent
checked: Unknown
cost: 275.76132225
date: 2026-02-17
folder: 20260217_mini-v2.0.0_claude-4-6-opus
instance calls: 28.932
instance cost: 0.5515226445
logo: img/logos/20260217_mini-v2.0.0_claude-4-6-opus.jpg · img/mini-icon.svg
logs: s3://swe-bench-submissions/bash-only/20260217_mini-v2.0.0_claude-4-6-opus/logs
mini-swe-agent version: 2.0.0
model display: Claude 4.6 Opus
model org: Anthropic
model release date: 20260205
name: Claude 4.6 Opus
os model: false
os system: true
reasoning effort: Unknown
resolved: 75.6
site: https://www.anthropic.com/claude
submission: Claude 4.6 Opus
tags: Model: claude-opus-4-6 · Org: Anthropic · System: Attempts - 1 · Mini: 2.0.0
trajs: s3://swe-bench-submissions/bash-only/20260217_mini-v2.0.0_claude-4-6-opus/trajs
trajs docent: https://docent.transluce.org/dashboard/9243cc78-d399-402f-be97-e366ff63282c
warning: Unknown
FreeBench model match: No explicit identity match; source name retained.

Source position 13 · 2025-06-12

Multiple

75.2% · resolved

Published setup details

agent: TRAE
agent org: TRAE
checked: false
cost: Unknown
date: 2025-06-12
folder: 20250612_trae
instance calls: Unknown
instance cost: Unknown
logo: img/logos/20250612_trae.png
logs: s3://swe-bench-submissions/verified/20250612_trae/logs
model display: Multiple
model org: Anthropic
model release date: Unknown
name: TRAE + Claude Sonnet 4 + Opus 4 + Sonnet 3.7 + Gemini 2.5 Pro
os model: false
os system: true
reasoning effort: Unknown
resolved: 75.2
site: https://github.com/bytedance/trae-agent
submission: TRAE + Claude Sonnet 4 + Opus 4 + Sonnet 3.7 + Gemini 2.5 Pro
tags: Model: claude-4-sonnet-20250522 · Model: claude-4-opus-20250522 · Model: claude-3-7-sonnet-20250219 · Model: gemini-2.5-pro-preview-06-05 · Org: TRAE · System: Attempts - 2+
trajs: s3://swe-bench-submissions/verified/20250612_trae/trajs
trajs docent: false
warning: Unknown
FreeBench model match: No explicit identity match; source name retained.

Source position 14 · 2025-07-31

Claude Sonnet 4

74.8% · resolved

Published setup details

agent: Harness AI
agent org: Harness
checked: false
cost: Unknown
date: 2025-07-31
folder: 20250731_harness_ai
instance calls: Unknown
instance cost: Unknown
logo: img/logos/20250731_harness_ai.svg
logs: s3://swe-bench-submissions/verified/20250731_harness_ai/logs
model display: Claude Sonnet 4
model org: Anthropic
model release date: Unknown
name: Harness AI
os model: false
os system: false
reasoning effort: Unknown
resolved: 74.8
site: https://www.harness.io/
submission: Harness AI
tags: Model: claude-sonnet-4-20250514 · Org: Harness · System: Attempts - 1
trajs: s3://swe-bench-submissions/verified/20250731_harness_ai/trajs
trajs docent: false
warning: Unknown
FreeBench model match: No explicit identity match; source name retained.

Source position 15 · 2025-11-03

Claude 4.5 Sonnet

74.8% · resolved

Published setup details

agent: Sonar Foundation Agent
agent org: Sonar
checked: false
cost: Unknown
date: 2025-11-03
folder: 20251103_sonar-foundation-agent_claude-sonnet-4-5
instance calls: Unknown
instance cost: Unknown
logo: img/logos/20251103_sonar-foundation-agent_claude-sonnet-4-5.png
logs: s3://swe-bench-submissions/verified/20251103_sonar-foundation-agent_claude-sonnet-4-5/logs
model display: Claude 4.5 Sonnet
model org: Anthropic
model release date: Unknown
name: Sonar Foundation Agent + Claude 4.5 Sonnet
os model: false
os system: false
reasoning effort: Unknown
resolved: 74.8
site: https://www.sonarsource.com
submission: Sonar Foundation Agent + Claude 4.5 Sonnet
tags: Model: claude-sonnet-4-5 · Org: Sonar · System: Attempts - 1
trajs: s3://swe-bench-submissions/verified/20251103_sonar-foundation-agent_claude-sonnet-4-5/trajs
trajs docent: false
warning: Unknown
FreeBench model match: No explicit identity match; source name retained.

Source position 16 · 2025-07-20

Claude 4 Sonnet

74.6% · resolved

Published setup details

agent: Lingxi-v1.5
agent org: Lingxi
checked: false
cost: Unknown
date: 2025-07-20
folder: 20250720_Lingxi-v1.5_claude-4-sonnet-20250514
instance calls: Unknown
instance cost: Unknown
logo: img/logos/20250720_Lingxi-v1.5_claude-4-sonnet-20250514.png
logs: s3://swe-bench-submissions/verified/20250720_Lingxi-v1.5_claude-4-sonnet-20250514/logs
model display: Claude 4 Sonnet
model org: Anthropic
model release date: Unknown
name: Lingxi-v1.5_claude-4-sonnet-20250514
os model: false
os system: true
reasoning effort: Unknown
resolved: 74.6
site: https://github.com/nimasteryang/Lingxi
submission: Lingxi-v1.5_claude-4-sonnet-20250514
tags: Model: claude-4-sonnet-20250514 · Org: Lingxi · System: Attempts - 1
trajs: s3://swe-bench-submissions/verified/20250720_Lingxi-v1.5_claude-4-sonnet-20250514/trajs
trajs docent: false
warning: Unknown
FreeBench model match: No explicit identity match; source name retained.

Source position 17 · 2025-09-15

Multiple

74.6% · resolved

Published setup details

agent: JoyCode
agent org: JoyCode
checked: false
cost: Unknown
date: 2025-09-15
folder: 20250915_JoyCode
instance calls: Unknown
instance cost: Unknown
logo: img/logos/20250915_JoyCode.png
logs: s3://swe-bench-submissions/verified/20250915_JoyCode/logs
model display: Multiple
model org: Anthropic
model release date: Unknown
name: JoyCode + Claude 4 Sonnet + GPT-4.1
os model: false
os system: true
reasoning effort: Unknown
resolved: 74.6
site: https://github.com/jd-opensource/joycode-agent
submission: JoyCode + Claude 4 Sonnet + GPT-4.1
tags: Model: claude-4-sonnet · Model: gpt-4.1 · Org: JoyCode · System: Attempts - 1
trajs: s3://swe-bench-submissions/verified/20250915_JoyCode/trajs
trajs docent: false
warning: Unknown
FreeBench model match: No explicit identity match; source name retained.

Source position 18 · 2025-06-03

Multiple

74.4% · resolved

Published setup details

agent: Refact.ai Agent
agent org: Refact.ai
checked: false
cost: Unknown
date: 2025-06-03
folder: 20250603_Refact_Agent_claude-4-sonnet
instance calls: Unknown
instance cost: Unknown
logo: img/logos/20250603_Refact_Agent_claude-4-sonnet.png
logs: s3://swe-bench-submissions/verified/20250603_Refact_Agent_claude-4-sonnet/logs
model display: Multiple
model org: Anthropic
model release date: Unknown
name: Refact.ai Agent + Claude 4 Sonnet + o4-mini
os model: false
os system: true
reasoning effort: Unknown
resolved: 74.4
site: https://refact.ai
submission: Refact.ai Agent + Claude 4 Sonnet + o4-mini
tags: Model: claude-4-sonnet · Model: o4-mini · Org: Refact.ai · System: Attempts - 1
trajs: s3://swe-bench-submissions/verified/20250603_Refact_Agent_claude-4-sonnet/trajs
trajs docent: false
warning: Unknown
FreeBench model match: No explicit identity match; source name retained.

Source position 19 · 2025-10-15

GPT-5

74.4% · resolved

Published setup details

agent: Prometheus-v1.2.1
agent org: EuniAI
checked: false
cost: Unknown
date: 2025-10-15
folder: 20251015_Prometheus_v1.2.1_gpt5
instance calls: Unknown
instance cost: Unknown
logo: img/logos/20251015_Prometheus_v1.2.1_gpt5.jpg
logs: s3://swe-bench-submissions/verified/20251015_Prometheus_v1.2.1_gpt5/logs
model display: GPT-5
model org: OpenAI
model release date: Unknown
name: Prometheus-v1.2.1 + GPT-5
os model: false
os system: true
reasoning effort: Unknown
resolved: 74.4
site: https://euni.ai/
submission: Prometheus-v1.2.1 + GPT-5
tags: Model: gpt-5-2025-08-07 · Org: EuniAI · Org: Delysium · System: Attempts - 2+
trajs: s3://swe-bench-submissions/verified/20251015_Prometheus_v1.2.1_gpt5/trajs
trajs docent: false
warning: Unknown
FreeBench model match: No explicit identity match; source name retained.

Source position 20 · 2025-11-24

Claude 4.5 Opus

74.4% · resolved

Published setup details

agent: mini-SWE-agent
agent org: SWE-agent
checked: true
cost: 360.6224985
date: 2025-11-24
folder: 20251124_mini-v1.16.0_claude-opus-4-5-20251101
instance calls: 38.042
instance cost: 0.721244997
logo: img/logos/20251124_mini-v1.16.0_claude-opus-4-5-20251101.jpg · img/mini-icon.svg
logs: s3://swe-bench-submissions/bash-only/20251124_mini-v1.16.0_claude-opus-4-5-20251101/logs
mini-swe-agent version: 1.16.0
model display: Claude 4.5 Opus
model org: Anthropic
model release date: 20251124
name: Claude 4.5 Opus (20251101) (medium)
os model: false
os system: true
reasoning effort: medium
resolved: 74.4
site: https://www.anthropic.com/news/claude-opus-4-5
submission: Claude 4.5 Opus (20251101) (medium)
tags: Model: claude-opus-4-5-20251101 · Org: Anthropic · System: Attempts - 1 · Mini: 1.16.0
trajs: s3://swe-bench-submissions/bash-only/20251124_mini-v1.16.0_claude-opus-4-5-20251101/trajs
trajs docent: https://docent.transluce.org/dashboard/6f6e2390-00c7-479f-9a2a-87b53b7477cb
warning: Unknown
FreeBench model match: No explicit identity match; source name retained.

Source position 21 · 2025-11-18

Gemini 3 Pro Preview

74.2% · resolved

Published setup details

agent: mini-SWE-agent
agent org: SWE-agent
checked: true
cost: 229.978759475
date: 2025-11-18
folder: 20251118_mini-v1.15.0_gemini-3-pro-preview-20251118
instance calls: 40.326
instance cost: 0.45995751895000003
logo: img/logos/20251118_mini-v1.15.0_gemini-3-pro-preview-20251118.svg · img/mini-icon.svg
logs: s3://swe-bench-submissions/bash-only/20251118_mini-v1.15.0_gemini-3-pro-preview-20251118/logs
mini-swe-agent version: 1.15.0
model display: Gemini 3 Pro Preview
model org: Google DeepMind
model release date: 20251118
name: Gemini 3 Pro Preview (2025-11-18)
os model: false
os system: true
reasoning effort: Unknown
resolved: 74.2
site: https://ai.google.dev/gemini-api/docs/models
submission: Gemini 3 Pro Preview (2025-11-18)
tags: Model: gemini-3-pro-preview · Org: Google DeepMind · System: Attempts - 1 · Mini: 1.15.0
trajs: s3://swe-bench-submissions/bash-only/20251118_mini-v1.15.0_gemini-3-pro-preview-20251118/trajs
trajs docent: https://docent.transluce.org/dashboard/3641b17f-034e-4b36-aa66-471dfed837d6
warning: Unknown
FreeBench model match: No explicit identity match; source name retained.

Source position 22 · 2025-11-03

Multiple

73.8% · resolved

Published setup details

agent: Salesforce AI Research SAGE
agent org: Salesforce AI Research
checked: false
cost: Unknown
date: 2025-11-03
folder: 20251103_SalesforceAIResearch_SAGE_OpenHands
instance calls: Unknown
instance cost: Unknown
logo: img/logos/20251103_SalesforceAIResearch_SAGE_OpenHands.png
logs: s3://swe-bench-submissions/verified/20251103_SalesforceAIResearch_SAGE_OpenHands/logs
model display: Multiple
model org: Anthropic
model release date: Unknown
name: Salesforce AI Research SAGE (OpenHands)
os model: false
os system: false
reasoning effort: Unknown
resolved: 73.8
site: https://www.salesforce.com/blog/sage-swe/
submission: Salesforce AI Research SAGE (OpenHands)
tags: Model: claude-sonnet-4.5 · Model: gpt-5 · Org: Salesforce AI Research · System: Attempts - 2+
trajs: s3://swe-bench-submissions/verified/20251103_SalesforceAIResearch_SAGE_OpenHands/trajs
trajs docent: false
warning: Unknown
FreeBench model match: No explicit identity match; source name retained.

Source position 23 · 2025-05-22

Claude 4 Opus

73.2% · resolved

Published setup details

agent: Tools
agent org: Unknown
checked: false
cost: Unknown
date: 2025-05-22
folder: 20250522_tools_claude-4-opus
instance calls: Unknown
instance cost: Unknown
logo: img/logos/20250522_tools_claude-4-opus.jpg
logs: s3://swe-bench-submissions/verified/20250522_tools_claude-4-opus/logs
model display: Claude 4 Opus
model org: Anthropic
model release date: Unknown
name: Tools + Claude 4 Opus (2025-05-22)
os model: false
os system: false
reasoning effort: Unknown
resolved: 73.2
site: https://www.anthropic.com/
submission: Tools + Claude 4 Opus (2025-05-22)
tags: Model: claude-4-opus-20250514 · System: Attempts - 1
trajs: s3://swe-bench-submissions/verified/20250522_tools_claude-4-opus/trajs
trajs docent: false
warning: Unknown
FreeBench model match: No explicit identity match; source name retained.

Source position 24 · 2025-10-21

Multiple

73.0% · resolved

Published setup details

agent: Salesforce AI Research SAGE
agent org: Salesforce AI Research
checked: false
cost: Unknown
date: 2025-10-21
folder: 20251021_SalesforceAIResearch_SAGE_bash_only
instance calls: Unknown
instance cost: Unknown
logo: img/logos/20251021_SalesforceAIResearch_SAGE_bash_only.png
logs: s3://swe-bench-submissions/verified/20251021_SalesforceAIResearch_SAGE_bash_only/logs
model display: Multiple
model org: Anthropic
model release date: Unknown
name: Salesforce AI Research SAGE (bash-only)
os model: false
os system: false
reasoning effort: Unknown
resolved: 73
site: https://www.salesforce.com/blog/sage-swe/
submission: Salesforce AI Research SAGE (bash-only)
tags: Model: claude-sonnet-4.5 · Model: gpt-5 · Org: Salesforce AI Research · System: Attempts - 2+
trajs: s3://swe-bench-submissions/verified/20251021_SalesforceAIResearch_SAGE_bash_only/trajs
trajs docent: false
warning: Unknown
FreeBench model match: No explicit identity match; source name retained.

Source position 25 · 2026-02-17

GLM 5

72.8% · resolved

Published setup details

agent: mini-SWE-agent
agent org: SWE-agent
checked: Unknown
cost: 267.19456811
date: 2026-02-17
folder: 20260217_mini-v2.0.0_glm-5-high
instance calls: 76.176
instance cost: 0.53438913622
logo: img/logos/20260217_mini-v2.0.0_glm-5-high.svg · img/mini-icon.svg
logs: Unknown
mini-swe-agent version: 2.0.0
model display: GLM 5
model org: Z-AI
model release date: 20260211
name: GLM 5 (high)
os model: true
os system: true
reasoning effort: high
resolved: 72.8
site: https://z.ai
submission: GLM 5 (high)
tags: Model: glm-5 · Org: Z-AI · System: Attempts - 1 · Mini: 2.0.0
trajs: s3://swe-bench-submissions/bash-only/20260217_mini-v2.0.0_glm-5-high/trajs
trajs docent: https://docent.transluce.org/dashboard/5fc22eb4-1f44-405e-9dff-40a61f9f6ef6
warning: Unknown
FreeBench model match: No explicit identity match; source name retained.

Source position 26 · 2026-02-17

GPT 5.2

72.8% · resolved

Published setup details

agent: mini-SWE-agent
agent org: SWE-agent
checked: Unknown
cost: 236.78350885
date: 2026-02-17
folder: 20260217_mini-v2.0.0_gpt-5-2-high
instance calls: 35.046
instance cost: 0.4735670177
logo: img/logos/20260217_mini-v2.0.0_gpt-5-2-high.svg · img/mini-icon.svg
logs: Unknown
mini-swe-agent version: 2.0.0
model display: GPT 5.2
model org: OpenAI
model release date: 20251211
name: GPT 5.2 (high)
os model: false
os system: true
reasoning effort: high
resolved: 72.8
site: https://platform.openai.com
submission: GPT 5.2 (high)
tags: Model: gpt-5-2 · Org: OpenAI · System: Attempts - 1 · Mini: 2.0.0
trajs: s3://swe-bench-submissions/bash-only/20260217_mini-v2.0.0_gpt-5-2-high/trajs
trajs docent: https://docent.transluce.org/dashboard/c2c7f5be-2c1a-472f-a34a-92e9e1df4f01
warning: Unknown
FreeBench model match: No explicit identity match; source name retained.

Source position 27 · 2026-02-19

GPT 5.2 Codex

72.8% · resolved

Published setup details

agent: mini-SWE-agent
agent org: SWE-agent
checked: Unknown
cost: 224.71194935
date: 2026-02-19
folder: 20260219_mini-v2.0.0_gpt-5-2-codex
instance calls: 28.072
instance cost: 0.4494238987
logo: img/logos/20260219_mini-v2.0.0_gpt-5-2-codex.svg · img/mini-icon.svg
logs: s3://swe-bench-submissions/bash-only/20260219_mini-v2.0.0_gpt-5-2-codex/logs
mini-swe-agent version: 2.0.0
model display: GPT 5.2 Codex
model org: OpenAI
model release date: 20251211
name: GPT 5.2 Codex
os model: false
os system: true
reasoning effort: Unknown
resolved: 72.8
site: https://platform.openai.com/docs/models/gpt-5
submission: GPT 5.2 Codex
tags: Model: gpt-5-2-codex · Org: OpenAI · System: Attempts - 1 · Mini: 2.0.0
trajs: s3://swe-bench-submissions/bash-only/20260219_mini-v2.0.0_gpt-5-2-codex/trajs
trajs docent: https://docent.transluce.org/dashboard/fb22a2e4-0a41-4d41-8e1e-388d4cb50d80
warning: Unknown
FreeBench model match: No explicit identity match; source name retained.

Source position 28 · 2025-05-22

Claude 4 Sonnet

72.4% · resolved

Published setup details

agent: Tools
agent org: Unknown
checked: false
cost: Unknown
date: 2025-05-22
folder: 20250522_tools_claude-4-sonnet
instance calls: Unknown
instance cost: Unknown
logo: img/logos/20250522_tools_claude-4-sonnet.jpg
logs: s3://swe-bench-submissions/verified/20250522_tools_claude-4-sonnet/logs
model display: Claude 4 Sonnet
model org: Anthropic
model release date: Unknown
name: Tools + Claude 4 Sonnet (2025-05-22)
os model: false
os system: false
reasoning effort: Unknown
resolved: 72.4
site: https://www.anthropic.com/
submission: Tools + Claude 4 Sonnet (2025-05-22)
tags: Model: claude-4-sonnet-20250514 · System: Attempts - 1
trajs: s3://swe-bench-submissions/verified/20250522_tools_claude-4-sonnet/trajs
trajs docent: false
warning: Unknown
FreeBench model match: No explicit identity match; source name retained.

Source position 29 · 2025-08-07

GPT-5

71.8% · resolved

Published setup details

agent: OpenHands
agent org: OpenHands
checked: true
cost: Unknown
date: 2025-08-07
folder: 20250807_openhands_gpt5
instance calls: Unknown
instance cost: Unknown
logo: img/logos/20250807_openhands_gpt5.png
logs: s3://swe-bench-submissions/verified/20250807_openhands_gpt5/logs
model display: GPT-5
model org: OpenAI
model release date: Unknown
name: OpenHands + GPT-5
os model: false
os system: true
reasoning effort: Unknown
resolved: 71.8
site: https://all-hands.dev/
submission: OpenHands + GPT-5
tags: Model: openai/gpt-5-2025-08-07 · Org: OpenHands · System: Attempts - 1
trajs: s3://swe-bench-submissions/verified/20250807_openhands_gpt5/trajs
trajs docent: false
warning: Unknown
FreeBench model match: No explicit identity match; source name retained.

Source position 30 · 2025-12-11

GPT 5.2

71.8% · resolved

Published setup details

agent: mini-SWE-agent
agent org: SWE-agent
checked: Unknown
cost: 260.1295018
date: 2025-12-11
folder: 20251211_mini-v1.17.2_gpt-5.2-2025-12-11-high
instance calls: 19.762
instance cost: 0.5202590036
logo: img/logos/20251211_mini-v1.17.2_gpt-5.2-2025-12-11-high.svg · img/mini-icon.svg
logs: s3://swe-bench-submissions/bash-only/20251211_mini-v1.17.2_gpt-5.2-2025-12-11-high/logs
mini-swe-agent version: 1.17.2
model display: GPT 5.2
model org: OpenAI
model release date: 20251211
name: GPT 5.2 (2025-12-11) (high)
os model: false
os system: true
reasoning effort: high
resolved: 71.8
site: https://platform.openai.com/docs/models/gpt-5
submission: GPT 5.2 (2025-12-11) (high)
tags: Model: gpt-5.2-2025-12-11 · Org: OpenAI · System: Attempts - 1 · Mini: 1.17.2
trajs: s3://swe-bench-submissions/bash-only/20251211_mini-v1.17.2_gpt-5.2-2025-12-11-high/trajs
trajs docent: https://docent.transluce.org/dashboard/93ef4179-a475-4f0b-88d1-46b5aa75e19b
warning: Unknown
FreeBench model match: No explicit identity match; source name retained.

Source position 31 · 2026-02-17

Claude 4.5 Sonnet

71.4% · resolved

Published setup details

agent: mini-SWE-agent
agent org: SWE-agent
checked: Unknown
cost: 328.9482492
date: 2026-02-17
folder: 20260217_mini-v2.0.0_claude-4-5-sonnet-high
instance calls: 48.3
instance cost: 0.6578964984000001
logo: img/logos/20260217_mini-v2.0.0_claude-4-5-sonnet-high.jpg · img/mini-icon.svg
logs: Unknown
mini-swe-agent version: 2.0.0
model display: Claude 4.5 Sonnet
model org: Anthropic
model release date: 20250929
name: Claude 4.5 Sonnet (high)
os model: false
os system: true
reasoning effort: high
resolved: 71.4
site: https://www.anthropic.com/claude
submission: Claude 4.5 Sonnet (high)
tags: Model: claude-sonnet-4-5-20250929 · Org: Anthropic · System: Attempts - 1 · Mini: 2.0.0
trajs: s3://swe-bench-submissions/bash-only/20260217_mini-v2.0.0_claude-4-5-sonnet-high/trajs
trajs docent: https://docent.transluce.org/dashboard/737e5dd2-8555-435a-9fbd-1c6907c972f1
warning: Unknown
FreeBench model match: No explicit identity match; source name retained.

Source position 32 · 2025-07-10

Claude 4 Sonnet

71.2% · resolved

Published setup details

agent: Bloop
agent org: Bloop AI
checked: false
cost: Unknown
date: 2025-07-10
folder: 20250710_bloop
instance calls: Unknown
instance cost: Unknown
logo: Unknown
logs: s3://swe-bench-submissions/verified/20250710_bloop/logs
model display: Claude 4 Sonnet
model org: Anthropic
model release date: Unknown
name: Bloop
os model: false
os system: false
reasoning effort: Unknown
resolved: 71.2
site: https://bloop.ai/
submission: Bloop
tags: Model: claude-4-sonnet-20250514 · Org: Bloop AI · System: Attempts - 1
trajs: s3://swe-bench-submissions/verified/20250710_bloop/trajs
trajs docent: false
warning: Unknown
FreeBench model match: No explicit identity match; source name retained.

Source position 33 · 2025-07-15

Multiple

71.2% · resolved

Published setup details

agent: Qodo Command
agent org: Qodo
checked: false
cost: Unknown
date: 2025-07-15
folder: 20250715_qodo_command
instance calls: Unknown
instance cost: Unknown
logo: img/logos/20250715_qodo_command.png
logs: s3://swe-bench-submissions/verified/20250715_qodo_command/logs
model display: Multiple
model org: Anthropic
model release date: Unknown
name: Qodo Command
os model: false
os system: false
reasoning effort: Unknown
resolved: 71.2
site: https://www.qodo.ai/
submission: Qodo Command
tags: Model: claude-sonnet-4-20250514 · Model: gpt-4o · Org: Qodo · System: Attempts - 1
trajs: s3://swe-bench-submissions/verified/20250715_qodo_command/trajs
trajs docent: false
warning: Unknown
FreeBench model match: No explicit identity match; source name retained.

Source position 34 · 2025-09-29

GPT-5

71.2% · resolved

Published setup details

agent: Prometheus-v1.2
agent org: EuniAI
checked: false
cost: Unknown
date: 2025-09-29
folder: 20250929_Prometheus_v1.2_gpt5
instance calls: Unknown
instance cost: Unknown
logo: img/logos/20250929_Prometheus_v1.2_gpt5.jpg
logs: s3://swe-bench-submissions/verified/20250929_Prometheus_v1.2_gpt5/logs
model display: GPT-5
model org: OpenAI
model release date: Unknown
name: Prometheus-v1.2 + GPT-5
os model: false
os system: true
reasoning effort: Unknown
resolved: 71.2
site: https://euni.ai/
submission: Prometheus-v1.2 + GPT-5
tags: Model: gpt-5-2025-08-07 · Org: EuniAI · Org: Delysium · System: Attempts - 1
trajs: s3://swe-bench-submissions/verified/20250929_Prometheus_v1.2_gpt5/trajs
trajs docent: false
warning: Unknown
FreeBench model match: No explicit identity match; source name retained.

Source position 35 · 2025-10-14

Kimi K2

71.2% · resolved

Published setup details

agent: Lingxi v1.5
agent org: Huawei
checked: false
cost: Unknown
date: 2025-10-14
folder: 20251014_Lingxi_kimi_k2
instance calls: Unknown
instance cost: Unknown
logo: img/logos/20251014_Lingxi_kimi_k2.png
logs: s3://swe-bench-submissions/verified/20251014_Lingxi_kimi_k2/logs
model display: Kimi K2
model org: Moonshot AI
model release date: Unknown
name: Lingxi v1.5 x Kimi K2
os model: true
os system: true
reasoning effort: Unknown
resolved: 71.2
site: https://github.com/lingxi-agent/Lingxi/tree/master
submission: Lingxi v1.5 x Kimi K2
tags: Model: kimi-k2-0905-preview · Org: Huawei · System: Attempts - 1
trajs: s3://swe-bench-submissions/verified/20251014_Lingxi_kimi_k2/trajs
trajs docent: false
warning: Unknown
FreeBench model match: No explicit identity match; source name retained.

Source position 36 · 2025-06-23

Multiple

71.0% · resolved

Published setup details

agent: Warp
agent org: Warp
checked: false
cost: Unknown
date: 2025-06-23
folder: 20250623_warp
instance calls: Unknown
instance cost: Unknown
logo: img/logos/20250623_warp.png
logs: s3://swe-bench-submissions/verified/20250623_warp/logs
model display: Multiple
model org: Anthropic
model release date: Unknown
name: Warp
os model: false
os system: false
reasoning effort: Unknown
resolved: 71
site: https://www.warp.dev
submission: Warp
tags: Model: claude-sonnet-4-20250514 · Model: claude-3-7-sonnet-20250219 · Model: gemini-2.5-pro · Model: gpt-4.1-2025-04-14 · Org: Warp · System: Attempts - 1
trajs: s3://swe-bench-submissions/verified/20250623_warp/trajs
trajs docent: false
warning: Unknown
FreeBench model match: No explicit identity match; source name retained.

Source position 37 · 2025-06-11

Claude 4 Sonnet

70.8% · resolved

Published setup details

agent: Moatless Tools
agent org: Moatless AI
checked: true
cost: Unknown
date: 2025-06-11
folder: 20250611_moatless_claude-4-sonnet-20250514
instance calls: Unknown
instance cost: Unknown
logo: img/logos/20250611_moatless_claude-4-sonnet-20250514.png
logs: s3://swe-bench-submissions/verified/20250611_moatless_claude-4-sonnet-20250514/logs
model display: Claude 4 Sonnet
model org: Anthropic
model release date: Unknown
name: Moatless Tools + Claude 4 Sonnet
os model: false
os system: true
reasoning effort: Unknown
resolved: 70.8
site: https://github.com/aorwall/moatless-tools
submission: Moatless Tools + Claude 4 Sonnet
tags: Model: claude-4-sonnet-20250514 · Org: Moatless AI · System: Attempts - 1
trajs: s3://swe-bench-submissions/verified/20250611_moatless_claude-4-sonnet-20250514/trajs
trajs docent: false
warning: Unknown
FreeBench model match: No explicit identity match; source name retained.

Source position 38 · 2026-02-17

Kimi K2.5

70.8% · resolved

Published setup details

agent: mini-SWE-agent
agent org: SWE-agent
checked: Unknown
cost: 73.276870372
date: 2026-02-17
folder: 20260217_mini-v2.0.0_kimi-k2-5-high
instance calls: 51.178
instance cost: 0.14655374074400002
logo: img/logos/20260217_mini-v2.0.0_kimi-k2-5-high.jpg · img/mini-icon.svg
logs: Unknown
mini-swe-agent version: 2.0.0
model display: Kimi K2.5
model org: Moonshot AI
model release date: 20260127
name: Kimi K2.5 (high)
os model: true
os system: true
reasoning effort: high
resolved: 70.8
site: https://moonshotai.github.io/Kimi-K2/
submission: Kimi K2.5 (high)
tags: Model: kimi-k2.5 · Org: Moonshot AI · System: Attempts - 1 · Mini: 2.0.0
trajs: s3://swe-bench-submissions/bash-only/20260217_mini-v2.0.0_kimi-k2-5-high/trajs
trajs docent: https://docent.transluce.org/dashboard/c4f6e550-6e49-436e-96fa-3aafd9a0c936
warning: Unknown
FreeBench model match: No explicit identity match; source name retained.

Source position 39 · 2025-05-19

Undisclosed

70.6% · resolved

Published setup details

agent: TRAE
agent org: Unknown
checked: false
cost: Unknown
date: 2025-05-19
folder: 20250519_trae
instance calls: Unknown
instance cost: Unknown
logo: img/logos/20250519_trae.png
logs: s3://swe-bench-submissions/verified/20250519_trae/logs
model display: Undisclosed
model org: Unknown
model release date: Unknown
name: TRAE
os model: false
os system: false
reasoning effort: Unknown
resolved: 70.6
site: https://www.trae.ai/
submission: TRAE
tags: System: Attempts - 2+
trajs: s3://swe-bench-submissions/verified/20250519_trae/trajs
trajs docent: false
warning: Unknown
FreeBench model match: No explicit identity match; source name retained.

Source position 40 · 2025-09-29

Claude 4.5 Sonnet

70.6% · resolved

Published setup details

agent: mini-SWE-agent
agent org: SWE-agent
checked: true
cost: 279.16737045
date: 2025-09-29
folder: 20250929_mini-v1.13.3_sonnet-4-5-20250929
instance calls: 50.988
instance cost: 0.5583347409
logo: img/logos/20250929_mini-v1.13.3_sonnet-4-5-20250929.jpg · img/mini-icon.svg
logs: s3://swe-bench-submissions/bash-only/20250929_mini-v1.13.3_sonnet-4-5-20250929/logs
mini-swe-agent version: 1.13.3
model display: Claude 4.5 Sonnet
model org: Anthropic
model release date: 20250929
name: Claude 4.5 Sonnet (20250929)
os model: false
os system: true
reasoning effort: Unknown
resolved: 70.6
site: https://www.anthropic.com/news/claude-sonnet-4-5
submission: Claude 4.5 Sonnet (20250929)
tags: Model: claude-sonnet-4-5-20250929 · Org: Anthropic · System: Attempts - 1 · Mini: 1.13.3
trajs: s3://swe-bench-submissions/bash-only/20250929_mini-v1.13.3_sonnet-4-5-20250929/trajs
trajs docent: https://docent.transluce.org/dashboard/a4844da1-fbb9-4d61-b82c-f46e471f748a
warning: Unknown
FreeBench model match: No explicit identity match; source name retained.

Source position 41 · 2025-05-15

Multiple

70.4% · resolved

Published setup details

agent: Refact.ai Agent
agent org: Refact.ai
checked: false
cost: Unknown
date: 2025-05-15
folder: 20250515_Refact_Agent
instance calls: Unknown
instance cost: Unknown
logo: img/logos/20250515_Refact_Agent.png
logs: s3://swe-bench-submissions/verified/20250515_Refact_Agent/logs
model display: Multiple
model org: Anthropic
model release date: Unknown
name: Refact.ai Agent
os model: false
os system: true
reasoning effort: Unknown
resolved: 70.4
site: https://refact.ai
submission: Refact.ai Agent
tags: Model: claude-3-7-sonnet-20250219 · Model: o3 · Model: o4-mini · Org: Refact.ai · System: Attempts - 1
trajs: s3://swe-bench-submissions/verified/20250515_Refact_Agent/trajs
trajs docent: false
warning: Unknown
FreeBench model match: No explicit identity match; source name retained.

Source position 42 · 2025-05-24

Claude 4 Sonnet

70.4% · resolved

Published setup details

agent: OpenHands
agent org: OpenHands
checked: true
cost: Unknown
date: 2025-05-24
folder: 20250524_openhands_claude_4_sonnet
instance calls: Unknown
instance cost: Unknown
logo: img/logos/20250524_openhands_claude_4_sonnet.png
logs: s3://swe-bench-submissions/verified/20250524_openhands_claude_4_sonnet/logs
model display: Claude 4 Sonnet
model org: Anthropic
model release date: Unknown
name: OpenHands + Claude 4 Sonnet
os model: false
os system: true
reasoning effort: Unknown
resolved: 70.4
site: https://all-hands.dev/
submission: OpenHands + Claude 4 Sonnet
tags: Model: claude-4-sonnet-20250514 · Org: OpenHands · System: Attempts - 1
trajs: s3://swe-bench-submissions/verified/20250524_openhands_claude_4_sonnet/trajs
trajs docent: false
warning: Unknown
FreeBench model match: No explicit identity match; source name retained.

Source position 43 · 2025-06-10

Undisclosed

70.4% · resolved

Published setup details

agent: Augment Agent v1
agent org: Unknown
checked: false
cost: Unknown
date: 2025-06-10
folder: 20250610_augment_agent_v1
instance calls: Unknown
instance cost: Unknown
logo: img/logos/20250610_augment_agent_v1.png
logs: s3://swe-bench-submissions/verified/20250610_augment_agent_v1/logs
model display: Undisclosed
model org: Unknown
model release date: Unknown
name: Augment Agent v1
os model: false
os system: true
reasoning effort: Unknown
resolved: 70.4
site: https://www.augmentcode.com
submission: Augment Agent v1
tags: System: Attempts - 1
trajs: s3://swe-bench-submissions/verified/20250610_augment_agent_v1/trajs
trajs docent: false
warning: Unknown
FreeBench model match: No explicit identity match; source name retained.

Source position 44 · 2025-05-19

Multiple

70.2% · resolved

Published setup details

agent: devlo
agent org: devlo
checked: false
cost: Unknown
date: 2025-05-19
folder: 20250519_devlo
instance calls: Unknown
instance cost: Unknown
logo: img/logos/20250519_devlo.jpg
logs: s3://swe-bench-submissions/verified/20250519_devlo/logs
model display: Multiple
model org: Anthropic
model release date: Unknown
name: devlo
os model: false
os system: false
reasoning effort: Unknown
resolved: 70.2
site: https://devlo.ai/?utm_source=swe-bench
submission: devlo
tags: Model: claude-3-7-sonnet-20250219 · Model: o3 · Model: gemini-2.5-pro · Org: devlo · System: Attempts - 1
trajs: s3://swe-bench-submissions/verified/20250519_devlo/trajs
trajs docent: false
warning: Unknown
FreeBench model match: No explicit identity match; source name retained.

Source position 45 · 2025-04-30

Multiple

70.0% · resolved

Published setup details

agent: Zencoder
agent org: Unknown
checked: false
cost: Unknown
date: 2025-04-30
folder: 20250430_zencoder_ai
instance calls: Unknown
instance cost: Unknown
logo: img/logos/20250430_zencoder_ai.svg
logs: s3://swe-bench-submissions/verified/20250430_zencoder_ai/logs
model display: Multiple
model org: Anthropic
model release date: Unknown
name: Zencoder (2025-04-30)
os model: false
os system: false
reasoning effort: Unknown
resolved: 70
site: https://zencoder.ai
submission: Zencoder (2025-04-30)
tags: Model: claude-3-7-sonnet-20250219 · Model: o4-mini · System: Attempts - 2+
trajs: s3://swe-bench-submissions/verified/20250430_zencoder_ai/trajs
trajs docent: false
warning: Unknown
FreeBench model match: No explicit identity match; source name retained.

Source position 46 · 2026-02-17

DeepSeek V3.2

70.0% · resolved

Published setup details

agent: mini-SWE-agent
agent org: SWE-agent
checked: Unknown
cost: 223.92324825
date: 2026-02-17
folder: 20260217_mini-v2.0.0_deepseek-3-2-high
instance calls: 88.506
instance cost: 0.4478464965
logo: img/logos/20260217_mini-v2.0.0_deepseek-3-2-high.svg · img/mini-icon.svg
logs: Unknown
mini-swe-agent version: 2.0.0
model display: DeepSeek V3.2
model org: DeepSeek
model release date: 20251201
name: DeepSeek V3.2 (high)
os model: true
os system: true
reasoning effort: high
resolved: 70
site: https://api-docs.deepseek.com
submission: DeepSeek V3.2 (high)
tags: Model: deepseek-v3.2 · Org: DeepSeek · System: Attempts - 1 · Mini: 2.0.0
trajs: s3://swe-bench-submissions/bash-only/20260217_mini-v2.0.0_deepseek-3-2-high/trajs
trajs docent: https://docent.transluce.org/dashboard/4b463c14-4776-49a2-accf-b42b1c076bba
warning: Unknown
FreeBench model match: No explicit identity match; source name retained.

Source position 47 · 2025-08-05

Qwen3-Coder-480B-A35B-Instruct

69.6% · resolved

Published setup details

agent: OpenHands
agent org: Qwen
checked: false
cost: Unknown
date: 2025-08-05
folder: 20250805_openhands-Qwen3-Coder-480B-A35B-Instruct
instance calls: Unknown
instance cost: Unknown
logo: img/logos/20250805_openhands-Qwen3-Coder-480B-A35B-Instruct.png
logs: s3://swe-bench-submissions/verified/20250805-openhands-Qwen3-Coder-480B-A35B-Instruct/logs
model display: Qwen3-Coder-480B-A35B-Instruct
model org: Qwen
model release date: Unknown
name: OpenHands + Qwen3-Coder-480B-A35B-Instruct
os model: true
os system: true
reasoning effort: Unknown
resolved: 69.6
site: https://qwenlm.github.io/blog/qwen3-coder/
submission: OpenHands + Qwen3-Coder-480B-A35B-Instruct
tags: Model: https://huggingface.co/Qwen/Qwen3-Coder-480B-A35B-Instruct · Org: Qwen · Org: OpenHands · System: Attempts - 1
trajs: s3://swe-bench-submissions/verified/20250805-openhands-Qwen3-Coder-480B-A35B-Instruct/trajs
trajs docent: false
warning: Unknown
FreeBench model match: No explicit identity match; source name retained.

Source position 48 · 2026-02-26

Gemini 3 Pro

69.6% · resolved

Published setup details

agent: mini-SWE-agent
agent org: SWE-agent
checked: Unknown
cost: 480.0099282
date: 2026-02-26
folder: 20260226_mini-v2.0.0_gemini-3-pro-high
instance calls: 51.282
instance cost: 0.9600198564
logo: img/logos/20260226_mini-v2.0.0_gemini-3-pro-high.svg · img/mini-icon.svg
logs: s3://swe-bench-submissions/bash-only/20260226_mini-v2.0.0_gemini-3-pro-high/logs
mini-swe-agent version: 2.0.0
model display: Gemini 3 Pro
model org: Google DeepMind
model release date: 20251118
name: Gemini 3 Pro
os model: false
os system: true
reasoning effort: high
resolved: 69.6
site: https://ai.google.dev/gemini-api/docs/models
submission: Gemini 3 Pro
tags: Model: gemini-3-pro-preview · Org: Google DeepMind · System: Attempts - 1 · Mini: 2.0.0
trajs: s3://swe-bench-submissions/bash-only/20260226_mini-v2.0.0_gemini-3-pro-high/trajs
trajs docent: https://docent.transluce.org/dashboard/cf47db93-89ae-4f15-984d-e117c77cb96d
warning: Unknown
FreeBench model match: No explicit identity match; source name retained.

Source position 49 · 2025-12-11

GPT 5.2

69.0% · resolved

Published setup details

agent: mini-SWE-agent
agent org: SWE-agent
checked: Unknown
cost: 134.8337487
date: 2025-12-11
folder: 20251211_mini-v1.17.2_gpt-5.2-2025-12-11
instance calls: 16.27
instance cost: 0.2696674974
logo: img/logos/20251211_mini-v1.17.2_gpt-5.2-2025-12-11.svg · img/mini-icon.svg
logs: s3://swe-bench-submissions/bash-only/20251211_mini-v1.17.2_gpt-5.2-2025-12-11/logs
mini-swe-agent version: 1.17.2
model display: GPT 5.2
model org: OpenAI
model release date: 20251211
name: GPT 5.2 (2025-12-11)
os model: false
os system: true
reasoning effort: Unknown
resolved: 69
site: https://platform.openai.com/docs/models/
submission: GPT 5.2 (2025-12-11)
tags: Model: gpt-5.2-2025-12-11 · Org: OpenAI · System: Attempts - 1 · Mini: 1.17.2
trajs: s3://swe-bench-submissions/bash-only/20251211_mini-v1.17.2_gpt-5.2-2025-12-11/trajs
trajs docent: https://docent.transluce.org/dashboard/c0d7b5a6-9d24-45d4-9e4b-3ae87f2c654a
warning: Unknown
FreeBench model match: No explicit identity match; source name retained.

Source position 50 · 2025-05-16

Multiple

68.2% · resolved

Published setup details

agent: Nemotron-CORTEXA
agent org: NVIDIA
checked: false
cost: Unknown
date: 2025-05-16
folder: 20250516_cortexa_o3
instance calls: Unknown
instance cost: Unknown
logo: img/logos/20250516_cortexa_o3.jpg
logs: s3://swe-bench-submissions/verified/20250516_cortexa_o3/logs
model display: Multiple
model org: OpenAI
model release date: Unknown
name: Nemotron-CORTEXA
os model: false
os system: true
reasoning effort: Unknown
resolved: 68.2
site: https://research.nvidia.com/labs/adlr/cortexa/
submission: Nemotron-CORTEXA
tags: Model: NV-EmbedCode · Model: o3 · Model: o3-mini · Model: o1 · Model: gpt-4.1 · Model: gpt-4o-2024-05-13 · Model: gpt-4-turbo-2024-04-09 · Model: claude-3-7-sonnet-20250219 · Model: deepseek-v3 · Model: Qwen/Qwen2.5-72B-Instruct · Model: meta-llama/Llama-3.1-405B-Instruct · Model: meta-llama/Llama-3.3-70B-Instruct · Org: NVIDIA · System: Attempts - 2+
trajs: s3://swe-bench-submissions/verified/20250516_cortexa_o3/trajs
trajs docent: false
warning: Unknown
FreeBench model match: No explicit identity match; source name retained.

Source position 51 · 2025-09-30

GLM-4.6

68.2% · resolved

Published setup details

agent: Undisclosed
agent org: Z.ai
checked: false
cost: Unknown
date: 2025-09-30
folder: 20250930_zai_glm4-6
instance calls: Unknown
instance cost: Unknown
logo: img/logos/20250930_zai_glm4-6.png
logs: s3://swe-bench-submissions/verified/20250930_zai_glm4-6/logs
model display: GLM-4.6
model org: Z.ai
model release date: Unknown
name: GLM-4.6
os model: true
os system: true
reasoning effort: Unknown
resolved: 68.2
site: https://chat.z.ai/
submission: GLM-4.6
tags: Model: https://huggingface.co/zai-org/GLM-4.6 · Org: Z.ai · System: Attempts - 1
trajs: s3://swe-bench-submissions/verified/20250930_zai_glm4-6/trajs
trajs docent: false
warning: Unknown
FreeBench model match: No explicit identity match; source name retained.

Source position 52 · 2025-08-02

Claude 4 Opus

67.6% · resolved

Published setup details

agent: mini-SWE-agent
agent org: SWE-agent
checked: true
cost: 565.6352234999999
date: 2025-08-02
folder: 20250802_mini-v1.0.0_claude-4-opus-20250514
instance calls: 31.076
instance cost: 1.131270447
logo: img/logos/20250802_mini-v1.0.0_claude-4-opus-20250514.jpg · img/mini-icon.svg
logs: s3://swe-bench-submissions/bash-only/20250802_mini-v1.0.0_claude-4-opus-20250514/logs
mini-swe-agent version: 1.0.0
model display: Claude 4 Opus
model org: Anthropic
model release date: 20250522
name: Claude 4 Opus (20250514)
os model: false
os system: true
reasoning effort: Unknown
resolved: 67.6
site: https://www.anthropic.com/news/claude-4
submission: Claude 4 Opus (20250514)
tags: Model: claude-4-opus-20250514 · Org: Anthropic · System: Attempts - 1 · Mini: 1.0.0
trajs: s3://swe-bench-submissions/bash-only/20250802_mini-v1.0.0_claude-4-opus-20250514/trajs
trajs docent: https://docent.transluce.org/dashboard/c93d3381-a539-4cd6-89be-97066d36d906
warning: Unknown
FreeBench model match: No explicit identity match; source name retained.

Source position 53 · 2025-05-22

Claude 4 Sonnet

66.6% · resolved

Published setup details

agent: SWE-agent
agent org: SWE-agent
checked: true
cost: Unknown
date: 2025-05-22
folder: 20250522_sweagent_claude-4-sonnet-20250514
instance calls: Unknown
instance cost: Unknown
logo: img/logos/20250522_sweagent_claude-4-sonnet-20250514.png
logs: s3://swe-bench-submissions/verified/20250522_sweagent_claude-4-sonnet-20250514/logs
model display: Claude 4 Sonnet
model org: Anthropic
model release date: Unknown
name: SWE-agent + Claude 4 Sonnet
os model: false
os system: true
reasoning effort: Unknown
resolved: 66.6
site: https://github.com/SWE-agent/SWE-agent
submission: SWE-agent + Claude 4 Sonnet
tags: Model: claude-4-sonnet-20250514 · Org: SWE-agent · System: Attempts - 1
trajs: s3://swe-bench-submissions/verified/20250522_sweagent_claude-4-sonnet-20250514/trajs
trajs docent: false
warning: Unknown
FreeBench model match: No explicit identity match; source name retained.

Source position 54 · 2026-02-17

Claude 4.5 Haiku

66.6% · resolved

Published setup details

agent: mini-SWE-agent
agent org: SWE-agent
checked: Unknown
cost: 165.46221805000002
date: 2026-02-17
folder: 20260217_mini-v2.0.0_claude-4-5-haiku-high
instance calls: 66.154
instance cost: 0.33092443610000005
logo: img/logos/20260217_mini-v2.0.0_claude-4-5-haiku-high.jpg · img/mini-icon.svg
logs: Unknown
mini-swe-agent version: 2.0.0
model display: Claude 4.5 Haiku
model org: Anthropic
model release date: 20251015
name: Claude 4.5 Haiku (high)
os model: false
os system: true
reasoning effort: high
resolved: 66.6
site: https://www.anthropic.com/claude
submission: Claude 4.5 Haiku (high)
tags: Model: claude-haiku-4-5-20251001 · Org: Anthropic · System: Attempts - 1 · Mini: 2.0.0
trajs: s3://swe-bench-submissions/bash-only/20260217_mini-v2.0.0_claude-4-5-haiku-high/trajs
trajs docent: https://docent.transluce.org/dashboard/a91fc32b-2035-4ec0-950e-433922852071
warning: Unknown
FreeBench model match: No explicit identity match; source name retained.

Source position 55 · 2025-05-14

Claude 3.7 Sonnet

66.4% · resolved

Published setup details

agent: Aime-coder v1
agent org: ByteDance DevInfra
checked: false
cost: Unknown
date: 2025-05-14
folder: 20250514_aime_coder
instance calls: Unknown
instance cost: Unknown
logo: img/logos/20250514_aime_coder.png
logs: s3://swe-bench-submissions/verified/20250514_aime_coder/logs
model display: Claude 3.7 Sonnet
model org: Anthropic
model release date: Unknown
name: Aime-coder v1 + Anthopic Claude 3.7 Sonnet
os model: false
os system: false
reasoning effort: Unknown
resolved: 66.4
site: https://github.com/SWE-bench/experiments/blob/main/evaluation/verified/20250514_aime_coder/README.md
submission: Aime-coder v1 + Anthopic Claude 3.7 Sonnet
tags: Model: claude-3-7-sonnet-20250219 · Org: ByteDance DevInfra · System: Attempts - 2+
trajs: s3://swe-bench-submissions/verified/20250514_aime_coder/trajs
trajs docent: false
warning: Unknown
FreeBench model match: No explicit identity match; source name retained.

Source position 56 · 2025-11-20

GPT 5.1

66.0% · resolved

Published setup details

agent: mini-SWE-agent
agent org: SWE-agent
checked: Unknown
cost: 153.07773625000002
date: 2025-11-20
folder: 20251120_mini-v1.15.0_gpt-5.1-2025-11-13
instance calls: 21
instance cost: 0.3061554725
logo: img/logos/20251120_mini-v1.15.0_gpt-5.1-2025-11-13.svg · img/mini-icon.svg
logs: s3://swe-bench-submissions/bash-only/20251120_mini-v1.15.0_gpt-5.1-2025-11-13/logs
mini-swe-agent version: 1.15.0
model display: GPT 5.1
model org: OpenAI
model release date: 20251112
name: GPT 5.1 (2025-11-13) (medium)
os model: false
os system: true
reasoning effort: medium
resolved: 66
site: https://platform.openai.com/docs/models/gpt-5
submission: GPT 5.1 (2025-11-13) (medium)
tags: Model: gpt-5.1-2025-11-13 · Org: OpenAI · System: Attempts - 1 · Mini: 1.15.0
trajs: s3://swe-bench-submissions/bash-only/20251120_mini-v1.15.0_gpt-5.1-2025-11-13/trajs
trajs docent: https://docent.transluce.org/dashboard/0f6398cb-461e-478a-8030-85b2d005d987
warning: Unknown
FreeBench model match: No explicit identity match; source name retained.

Source position 57 · 2025-11-24

GPT 5.1 Codex

66.0% · resolved

Published setup details

agent: mini-SWE-agent
agent org: SWE-agent
checked: Unknown
cost: 294.44117500000004
date: 2025-11-24
folder: 20251124_mini-v1.16.0_gpt-5.1-codex
instance calls: 23.882
instance cost: 0.5888823500000001
logo: img/logos/20251124_mini-v1.16.0_gpt-5.1-codex.svg · img/mini-icon.svg
logs: s3://swe-bench-submissions/bash-only/20251124_mini-v1.16.0_gpt-5.1-codex/logs
mini-swe-agent version: 1.16.0
model display: GPT 5.1 Codex
model org: OpenAI
model release date: 20251112
name: GPT 5.1 Codex (medium)
os model: false
os system: true
reasoning effort: medium
resolved: 66
site: https://platform.openai.com/docs/models/gpt-5
submission: GPT 5.1 Codex (medium)
tags: Model: gpt-5.1-codex · Org: OpenAI · System: Attempts - 1 · Mini: 1.16.0
trajs: s3://swe-bench-submissions/bash-only/20251124_mini-v1.16.0_gpt-5.1-codex/trajs
trajs docent: https://docent.transluce.org/dashboard/ebeb5896-b0bc-4504-a687-a5bf9824a146
warning: Unknown
FreeBench model match: No explicit identity match; source name retained.

Source position 58 · 2025-04-15

Undisclosed

65.8% · resolved

Published setup details

agent: OpenHands
agent org: OpenHands
checked: true
cost: Unknown
date: 2025-04-15
folder: 20250415_openhands
instance calls: Unknown
instance cost: Unknown
logo: img/logos/20250415_openhands.png
logs: s3://swe-bench-submissions/verified/20250415_openhands/logs
model display: Undisclosed
model org: Unknown
model release date: Unknown
name: OpenHands
os model: false
os system: true
reasoning effort: Unknown
resolved: 65.8
site: https://all-hands.dev/
submission: OpenHands
tags: Org: OpenHands · System: Attempts - 2+
trajs: s3://swe-bench-submissions/verified/20250415_openhands/trajs
trajs docent: false
warning: Unknown
FreeBench model match: No explicit identity match; source name retained.

Source position 59 · 2025-03-16

Undisclosed

65.4% · resolved

Published setup details

agent: Augment Agent v0
agent org: Unknown
checked: false
cost: Unknown
date: 2025-03-16
folder: 20250316_augment_agent_v0
instance calls: Unknown
instance cost: Unknown
logo: img/logos/20250316_augment_agent_v0.png
logs: s3://swe-bench-submissions/verified/20250316_augment_agent_v0/logs
model display: Undisclosed
model org: Unknown
model release date: Unknown
name: Augment Agent v0
os model: false
os system: true
reasoning effort: Unknown
resolved: 65.4
site: https://www.augmentcode.com
submission: Augment Agent v0
tags: System: Attempts - 2+
trajs: s3://swe-bench-submissions/verified/20250316_augment_agent_v0/trajs
trajs docent: false
warning: Unknown
FreeBench model match: No explicit identity match; source name retained.

Source position 60 · 2025-04-05

Undisclosed

65.4% · resolved

Published setup details

agent: Amazon Q Developer Agent
agent org: AWS
checked: false
cost: Unknown
date: 2025-04-05
folder: 20250405_amazon-q-developer-agent-20250405-dev
instance calls: Unknown
instance cost: Unknown
logo: img/logos/20250405_amazon-q-developer-agent-20250405-dev.jpg
logs: s3://swe-bench-submissions/verified/20250405_amazon-q-developer-agent-20250405-dev/logs
model display: Undisclosed
model org: Unknown
model release date: Unknown
name: Amazon Q Developer Agent (v20250405-dev)
os model: false
os system: false
reasoning effort: Unknown
resolved: 65.4
site: https://aws.amazon.com/q/developer/
submission: Amazon Q Developer Agent (v20250405-dev)
tags: Org: AWS · System: Attempts - 2+
trajs: s3://swe-bench-submissions/verified/20250405_amazon-q-developer-agent-20250405-dev/trajs
trajs docent: false
warning: Unknown
FreeBench model match: No explicit identity match; source name retained.

Source position 61 · 2025-07-16

Kimi K2

65.4% · resolved

Published setup details

agent: OpenHands
agent org: OpenHands
checked: true
cost: Unknown
date: 2025-07-16
folder: 20250716_openhands_kimi_k2
instance calls: Unknown
instance cost: Unknown
logo: img/logos/20250716_openhands_kimi_k2.png
logs: s3://swe-bench-submissions/verified/20250716_openhands_kimi_k2/logs
model display: Kimi K2
model org: Moonshot AI
model release date: Unknown
name: OpenHands + Kimi K2
os model: true
os system: true
reasoning effort: Unknown
resolved: 65.4
site: https://all-hands.dev/
submission: OpenHands + Kimi K2
tags: Model: moonshot/kimi-k2-0711-preview · Org: OpenHands · System: Attempts - 1
trajs: s3://swe-bench-submissions/verified/20250716_openhands_kimi_k2/trajs
trajs docent: false
warning: Unknown
FreeBench model match: No explicit identity match; source name retained.

Source position 62 · 2025-08-07

GPT 5

65.0% · resolved

Published setup details

agent: mini-SWE-agent
agent org: SWE-agent
checked: true
cost: 140.19150875
date: 2025-08-07
folder: 20250807_mini-v1.7.0_gpt-5
instance calls: 13.208
instance cost: 0.2803830175
logo: img/logos/20250807_mini-v1.7.0_gpt-5.svg · img/mini-icon.svg
logs: s3://swe-bench-submissions/bash-only/20250807_mini-v1.7.0_gpt-5/logs
mini-swe-agent version: 1.7.0
model display: GPT 5
model org: OpenAI
model release date: 20250807
name: GPT 5 (2025-08-07) (medium)
os model: false
os system: true
reasoning effort: medium
resolved: 65
site: https://platform.openai.com/docs/models/gpt-5
submission: GPT 5 (2025-08-07) (medium)
tags: Model: gpt-5-2025-08-07 · Org: OpenAI · System: Attempts - 1 · Mini: 1.7.0
trajs: s3://swe-bench-submissions/bash-only/20250807_mini-v1.7.0_gpt-5/trajs
trajs docent: https://docent.transluce.org/dashboard/4844cb3c-d5cd-4c75-b64b-973029f2d37c
warning: Unknown
FreeBench model match: No explicit identity match; source name retained.

Source position 63 · 2025-07-26

Claude 4 Sonnet

64.9% · resolved

Published setup details

agent: mini-SWE-agent
agent org: SWE-agent
checked: true
cost: 185.7265839
date: 2025-07-26
folder: 20250726_mini-v1.0.0_claude-sonnet-4-20250514
instance calls: 37.172
instance cost: 0.37145316780000004
logo: img/logos/20250726_mini-v1.0.0_claude-sonnet-4-20250514.jpg · img/mini-icon.svg
logs: s3://swe-bench-submissions/bash-only/20250726_mini-v1.0.0_claude-sonnet-4-20250514/logs
mini-swe-agent version: 1.0.0
model display: Claude 4 Sonnet
model org: Anthropic
model release date: 20250522
name: Claude 4 Sonnet (20250514)
os model: false
os system: true
reasoning effort: Unknown
resolved: 64.93
site: https://www.anthropic.com/news/claude-4
submission: Claude 4 Sonnet (20250514)
tags: Model: claude-4-sonnet-20250514 · Org: Anthropic · System: Attempts - 1 · Mini: 1.0.0
trajs: s3://swe-bench-submissions/bash-only/20250726_mini-v1.0.0_claude-sonnet-4-20250514/trajs
trajs docent: https://docent.transluce.org/dashboard/0cb59666-bca8-476b-bf8e-3b924fafcae7
warning: Unknown
FreeBench model match: No explicit identity match; source name retained.

Source position 64 · 2025-01-17

o1-preview

64.6% · resolved

Published setup details

agent: W&B Programmer O1 crosscheck5
agent org: Unknown
checked: false
cost: Unknown
date: 2025-01-17
folder: 20250117_wandb_programmer_o1_crosscheck5
instance calls: Unknown
instance cost: Unknown
logo: img/logos/20250117_wandb_programmer_o1_crosscheck5.png
logs: s3://swe-bench-submissions/verified/20250117_wandb_programmer_o1_crosscheck5/logs
model display: o1-preview
model org: OpenAI
model release date: Unknown
name: W&B Programmer O1 crosscheck5
os model: false
os system: false
reasoning effort: Unknown
resolved: 64.6
site: https://wandb.ai
submission: W&B Programmer O1 crosscheck5
tags: Model: o1-preview · System: Attempts - 2+
trajs: s3://swe-bench-submissions/verified/20250117_wandb_programmer_o1_crosscheck5/trajs
trajs docent: false
warning: Unknown
FreeBench model match: No explicit identity match; source name retained.

Source position 65 · 2025-05-03

o4-mini

64.6% · resolved

Published setup details

agent: PatchPilot-v1.1
agent org: Unknown
checked: false
cost: Unknown
date: 2025-05-03
folder: 20250503_patchpilot-v1.1-o4-mini
instance calls: Unknown
instance cost: Unknown
logo: img/logos/20250503_patchpilot-v1.1-o4-mini.gif
logs: s3://swe-bench-submissions/verified/20250503_patchpilot-v1.1-o4-mini/logs
model display: o4-mini
model org: OpenAI
model release date: Unknown
name: PatchPilot-v1.1
os model: false
os system: true
reasoning effort: Unknown
resolved: 64.6
site: Unknown
submission: PatchPilot-v1.1
tags: Model: o4-mini · System: Attempts - 2+
trajs: s3://swe-bench-submissions/verified/20250503_patchpilot-v1.1-o4-mini/trajs
trajs docent: false
warning: Unknown
FreeBench model match: No explicit identity match; source name retained.

Source position 66 · 2025-07-28

GLM-4.5

64.2% · resolved

Published setup details

agent: Undisclosed
agent org: Z.ai
checked: false
cost: Unknown
date: 2025-07-28
folder: 20250728_zai_glm4-5
instance calls: Unknown
instance cost: Unknown
logo: img/logos/20250728_zai_glm4-5.png
logs: s3://swe-bench-submissions/verified/20250728_zai_glm4-5/logs
model display: GLM-4.5
model org: Z.ai
model release date: Unknown
name: GLM-4.5
os model: true
os system: true
reasoning effort: Unknown
resolved: 64.2
site: https://chat.z.ai/
submission: GLM-4.5
tags: Model: https://huggingface.co/zai-org/GLM-4.5 · Org: Z.ai · System: Attempts - 1
trajs: s3://swe-bench-submissions/verified/20250728_zai_glm4-5/trajs
trajs docent: false
warning: Unknown
FreeBench model match: No explicit identity match; source name retained.

Source position 67 · 2025-02-06

Claude 3.5 Sonnet

63.4% · resolved

Published setup details

agent: AgentScope
agent org: Alibaba
checked: false
cost: Unknown
date: 2025-02-06
folder: 20250206_agentscope
instance calls: Unknown
instance cost: Unknown
logo: img/logos/20250206_agentscope.jpg
logs: s3://swe-bench-submissions/verified/20250206_agentscope/logs
model display: Claude 3.5 Sonnet
model org: Anthropic
model release date: Unknown
name: AgentScope
os model: false
os system: false
reasoning effort: Unknown
resolved: 63.4
site: https://github.com/modelscope/agentscope
submission: AgentScope
tags: Model: claude-3-5-sonnet-20241022 · Org: Alibaba · System: Attempts - 2+
trajs: s3://swe-bench-submissions/verified/20250206_agentscope/trajs
trajs docent: false
warning: Unknown
FreeBench model match: No explicit identity match; source name retained.

Source position 68 · 2025-12-10

Kimi K2 Thinking

63.4% · resolved

Published setup details

agent: mini-SWE-agent
agent org: SWE-agent
checked: true
cost: 219.15361769999998
date: 2025-12-10
folder: 20251210_mini-v1.17.2_kimi-k2-thinking
instance calls: 46.82
instance cost: 0.4383072354
logo: img/logos/20251210_mini-v1.17.2_kimi-k2-thinking.jpg · img/mini-icon.svg
logs: s3://swe-bench-submissions/bash-only/20251210_mini-v1.17.2_kimi-k2-thinking/logs
mini-swe-agent version: 1.17.2
model display: Kimi K2 Thinking
model org: Moonshot AI
model release date: 20251106
name: Kimi K2 Thinking
os model: true
os system: true
reasoning effort: Unknown
resolved: 63.4
site: https://moonshotai.github.io/Kimi-K2/
submission: Kimi K2 Thinking
tags: Model: Kimi-K2-Thinking · Org: Moonshot AI · System: Attempts - 1 · Mini: 1.17.2
trajs: s3://swe-bench-submissions/bash-only/20251210_mini-v1.17.2_kimi-k2-thinking/trajs
trajs docent: https://docent.transluce.org/dashboard/9dcb7016-5356-4966-a184-ce4b1131f33c
warning: Unknown
FreeBench model match: No explicit identity match; source name retained.

Source position 69 · 2025-02-24

Claude 3.7 Sonnet

63.2% · resolved

Published setup details

agent: Tools
agent org: Anthropic
checked: false
cost: Unknown
date: 2025-02-24
folder: 20250224_tools_claude-3-7-sonnet
instance calls: Unknown
instance cost: Unknown
logo: img/logos/20250224_tools_claude-3-7-sonnet.jpg
logs: s3://swe-bench-submissions/verified/20250224_tools_claude-3-7-sonnet/logs
model display: Claude 3.7 Sonnet
model org: Anthropic
model release date: Unknown
name: Tools + Claude 3.7 Sonnet (2025-02-24)
os model: false
os system: false
reasoning effort: Unknown
resolved: 63.2
site: https://www.anthropic.com/
submission: Tools + Claude 3.7 Sonnet (2025-02-24)
tags: Org: Anthropic · System: Attempts - 2+
trajs: s3://swe-bench-submissions/verified/20250224_tools_claude-3-7-sonnet/trajs
trajs docent: false
warning: Unknown
FreeBench model match: No explicit identity match; source name retained.

Source position 70 · 2025-01-10

Undisclosed

62.8% · resolved

Published setup details

agent: Blackbox AI Agent
agent org: Unknown
checked: false
cost: Unknown
date: 2025-01-10
folder: 20250110_blackboxai_agent_v1.1
instance calls: Unknown
instance cost: Unknown
logo: img/logos/20250110_blackboxai_agent_v1.1.png
logs: s3://swe-bench-submissions/verified/20250110_blackboxai_agent_v1.1/logs
model display: Undisclosed
model org: Unknown
model release date: Unknown
name: Blackbox AI Agent
os model: false
os system: false
reasoning effort: Unknown
resolved: 62.8
site: https://www.blackbox.ai/
submission: Blackbox AI Agent
tags: System: Attempts - 2+
trajs: s3://swe-bench-submissions/verified/20250110_blackboxai_agent_v1.1/trajs
trajs docent: false
warning: Unknown
FreeBench model match: No explicit identity match; source name retained.

Source position 71 · 2025-02-28

Claude 3.5 Sonnet

62.8% · resolved

Published setup details

agent: EPAM AI/Run Developer Agent
agent org: Unknown
checked: false
cost: Unknown
date: 2025-02-28
folder: 20250228_epam-ai-run-claude-3-5-sonnet
instance calls: Unknown
instance cost: Unknown
logo: img/logos/20250228_epam-ai-run-claude-3-5-sonnet.jpg
logs: s3://swe-bench-submissions/verified/20250228_epam-ai-run-claude-3-5-sonnet/logs/
model display: Claude 3.5 Sonnet
model org: Anthropic
model release date: Unknown
name: EPAM AI/Run Developer Agent v20250219 + Anthopic Claude 3.5 Sonnet
os model: false
os system: false
reasoning effort: Unknown
resolved: 62.8
site: https://www.epam.com/services/artificial-intelligence
submission: EPAM AI/Run Developer Agent v20250219 + Anthopic Claude 3.5 Sonnet
tags: Model: claude-3-5-sonnet-20241022 · System: Attempts - 2+
trajs: s3://swe-bench-submissions/verified/20250228_epam-ai-run-claude-3-5-sonnet/trajs
trajs docent: false
warning: Unknown
FreeBench model match: No explicit identity match; source name retained.

Source position 72 · 2025-02-25

Claude 3.7 Sonnet w/ Review Heavy

62.4% · resolved

Published setup details

agent: SWE-agent
agent org: SWE-agent
checked: true
cost: Unknown
date: 2025-02-25
folder: 20250225_sweagent_claude-3-7-sonnet
instance calls: Unknown
instance cost: Unknown
logo: img/logos/20250225_sweagent_claude-3-7-sonnet.png
logs: s3://swe-bench-submissions/verified/20250225_sweagent_claude-3-7-sonnet/logs
model display: Claude 3.7 Sonnet w/ Review Heavy
model org: Anthropic
model release date: Unknown
name: SWE-agent + Claude 3.7 Sonnet w/ Review Heavy
os model: false
os system: true
reasoning effort: Unknown
resolved: 62.4
site: https://github.com/SWE-agent/SWE-agent
submission: SWE-agent + Claude 3.7 Sonnet w/ Review Heavy
tags: Model: claude-3-7-sonnet-20250219 · Org: SWE-agent · System: Attempts - 2+
trajs: s3://swe-bench-submissions/verified/20250225_sweagent_claude-3-7-sonnet/trajs
trajs docent: false
warning: Unknown
FreeBench model match: No explicit identity match; source name retained.

Source position 73 · 2024-12-21

swe-search

62.2% · resolved

Published setup details

agent: CodeStory Midwit Agent
agent org: Unknown
checked: false
cost: Unknown
date: 2024-12-21
folder: 20241221_codestory_midwit_claude-3-5-sonnet_swe-search
instance calls: Unknown
instance cost: Unknown
logo: img/logos/20241221_codestory_midwit_claude-3-5-sonnet_swe-search.png
logs: s3://swe-bench-submissions/verified/20241221_codestory_midwit_claude-3-5-sonnet_swe-search/logs
model display: swe-search
model org: Anthropic
model release date: Unknown
name: CodeStory Midwit Agent + swe-search
os model: false
os system: false
reasoning effort: Unknown
resolved: 62.2
site: https://aide.dev/
submission: CodeStory Midwit Agent + swe-search
tags: Model: claude-3-5-sonnet-20241022 · System: Attempts - 2+
trajs: s3://swe-bench-submissions/verified/20241221_codestory_midwit_claude-3-5-sonnet_swe-search/trajs
trajs docent: false
warning: Unknown
FreeBench model match: No explicit identity match; source name retained.

Source position 74 · 2025-11-24

MiniMax M2

61.0% · resolved

Published setup details

agent: mini-SWE-agent
agent org: SWE-agent
checked: true
cost: 214.128088645
date: 2025-11-24
folder: 20251124_mini-v1.17.0_minimax-m2
instance calls: 74.138
instance cost: 0.42825617729
logo: img/logos/20251124_mini-v1.17.0_minimax-m2.png · img/mini-icon.svg
logs: s3://swe-bench-submissions/bash-only/20251124_mini-v1.17.0_minimax-m2/logs
mini-swe-agent version: 1.17.0
model display: MiniMax M2
model org: Minimax
model release date: 20251027
name: MiniMax M2
os model: true
os system: true
reasoning effort: Unknown
resolved: 61
site: https://www.minimax.io
submission: MiniMax M2
tags: Model: minimax-m2 · Org: Minimax · System: Attempts - 1 · Mini: 1.17.0
trajs: s3://swe-bench-submissions/bash-only/20251124_mini-v1.17.0_minimax-m2/trajs
trajs docent: https://docent.transluce.org/dashboard/dcfba689-d8e6-4372-9e92-dd524b7a0c56
warning: Unknown
FreeBench model match: No explicit identity match; source name retained.

Source position 75 · 2025-02-03

4x Scaled

60.8% · resolved

Published setup details

agent: OpenHands
agent org: OpenHands
checked: true
cost: Unknown
date: 2025-02-03
folder: 20250203_openhands_4x_scaled
instance calls: Unknown
instance cost: Unknown
logo: img/logos/20250203_openhands_4x_scaled.png
logs: s3://swe-bench-submissions/verified/20250203_openhands_4x_scaled/logs
model display: 4x Scaled
model org: Unknown
model release date: Unknown
name: OpenHands + 4x Scaled (2024-02-03)
os model: false
os system: true
reasoning effort: Unknown
resolved: 60.8
site: https://all-hands.dev/
submission: OpenHands + 4x Scaled (2024-02-03)
tags: Org: OpenHands · System: Attempts - 2+
trajs: s3://swe-bench-submissions/verified/20250203_openhands_4x_scaled/trajs
trajs docent: false
warning: Unknown
FreeBench model match: No explicit identity match; source name retained.

Source position 76 · 2025-09-01

Qwen3-Coder-30B-A3B-Instruct

60.4% · resolved

Published setup details

agent: EntroPO + R2E
agent org: 42-b3yond-6ug
checked: false (See README.md for info on how to get your results verified)
cost: Unknown
date: 2025-09-01
folder: 20250901_entroPO_R2E_QwenCoder30BA3B_tts
instance calls: Unknown
instance cost: Unknown
logo: img/logos/20250901_entroPO_R2E_QwenCoder30BA3B_tts.png
logs: s3://swe-bench-submissions/verified/20250901_entroPO_R2E_QwenCoder30BA3B_tts/logs
model display: Qwen3-Coder-30B-A3B-Instruct
model org: Qwen
model release date: Unknown
name: EntroPO + R2E + Qwen3-Coder-30B-A3B-Instruct
os model: true
os system: true
reasoning effort: Unknown
resolved: 60.4
site: https://github.com/sherdencooper/R2E-Gym
submission: EntroPO + R2E + Qwen3-Coder-30B-A3B-Instruct
tags: Model: Qwen3-Coder-30B-A3B-Instruct · Org: 42-b3yond-6ug · System: Attempts - 2+
trajs: s3://swe-bench-submissions/verified/20250901_entroPO_R2E_QwenCoder30BA3B_tts/trajs
trajs docent: false
warning: Unknown
FreeBench model match: No explicit identity match; source name retained.

Source position 77 · 2025-01-10

Claude 3.5 Sonnet

60.2% · resolved

Published setup details

agent: Learn-by-interact
agent org: Google
checked: false
cost: Unknown
date: 2025-01-10
folder: 20250110_learn_by_interact_claude3.5
instance calls: Unknown
instance cost: Unknown
logo: Unknown
logs: s3://swe-bench-submissions/verified/20250110_learn_by_interact_claude3.5/logs
model display: Claude 3.5 Sonnet
model org: Anthropic
model release date: Unknown
name: Learn-by-interact
os model: false
os system: false
reasoning effort: Unknown
resolved: 60.2
site: https://arxiv.org/pdf/2501.10893
submission: Learn-by-interact
tags: Model: claude-3-5-sonnet-20241022 · Org: Google · System: Attempts - 2+
trajs: s3://swe-bench-submissions/verified/20250110_learn_by_interact_claude3.5/trajs
trajs docent: false
warning: Unknown
FreeBench model match: No explicit identity match; source name retained.

Source position 78 · 2025-12-01

DeepSeek V3.2 Reasoner

60.0% · resolved

Published setup details

agent: mini-SWE-agent
agent org: SWE-agent
checked: true
cost: 14.0381514
date: 2025-12-01
folder: 20251201_mini-v1.17.1_deepseek-v3.2-reasoner
instance calls: 46.404
instance cost: 0.0280763028
logo: img/logos/20251201_mini-v1.17.1_deepseek-v3.2-reasoner.svg · img/mini-icon.svg
logs: s3://swe-bench-submissions/bash-only/20251201_mini-v1.17.1_deepseek-v3.2-reasoner/logs
mini-swe-agent version: 1.17.1
model display: DeepSeek V3.2 Reasoner
model org: deepseek
model release date: 20251201
name: DeepSeek V3.2 Reasoner
os model: true
os system: true
reasoning effort: Unknown
resolved: 60
site: Unknown
submission: DeepSeek V3.2 Reasoner
tags: Model: deepseek-v3.2-reasoner · Org: deepseek · System: Attempts - 1 · Mini: 1.17.1
trajs: s3://swe-bench-submissions/bash-only/20251201_mini-v1.17.1_deepseek-v3.2-reasoner/trajs
trajs docent: https://docent.transluce.org/dashboard/7fde5552-6b17-4cb7-ab9c-15fd9fb5b845
warning: Unknown
FreeBench model match: No explicit identity match; source name retained.

Source position 79 · 2025-08-07

GPT 5 mini

59.8% · resolved

Published setup details

agent: mini-SWE-agent
agent org: SWE-agent
checked: true
cost: 17.73853365
date: 2025-08-07
folder: 20250807_mini-v1.7.0_gpt-5-mini
instance calls: 14.466
instance cost: 0.035477067300000005
logo: img/logos/20250807_mini-v1.7.0_gpt-5-mini.svg · img/mini-icon.svg
logs: s3://swe-bench-submissions/bash-only/20250807_mini-v1.7.0_gpt-5-mini/logs
mini-swe-agent version: 1.7.0
model display: GPT 5 mini
model org: OpenAI
model release date: 20250807
name: GPT 5 mini (2025-08-07) (medium)
os model: false
os system: true
reasoning effort: medium
resolved: 59.8
site: https://platform.openai.com/docs/models/gpt-5-mini
submission: GPT 5 mini (2025-08-07) (medium)
tags: Model: gpt-5-mini-2025-08-07 · Org: OpenAI · System: Attempts - 1 · Mini: 1.7.0
trajs: s3://swe-bench-submissions/bash-only/20250807_mini-v1.7.0_gpt-5-mini/trajs
trajs docent: https://docent.transluce.org/dashboard/7f5e3cfc-8e38-4c73-812c-aca21298e7bc
warning: Unknown
FreeBench model match: No explicit identity match; source name retained.

Source position 80 · 2025-06-29

TTS(Bo16)

58.8% · resolved

Published setup details

agent: DeepSWE-Preview
agent org: Agentica
checked: false
cost: Unknown
date: 2025-06-29
folder: 20250629_deepswerl_r2eagent_tts
instance calls: Unknown
instance cost: Unknown
logo: img/logos/20250629_deepswerl_r2eagent_tts.jpg
logs: s3://swe-bench-submissions/verified/20250629_deepswerl_r2eagent_tts/logs
model display: TTS(Bo16)
model org: OpenAI
model release date: Unknown
name: DeepSWE-Preview + TTS(Bo16)
os model: true
os system: true
reasoning effort: Unknown
resolved: 58.8
site: agentica-project.com/
submission: DeepSWE-Preview + TTS(Bo16)
tags: Model: https://huggingface.co/agentica-org/DeepSWE-Preview · Org: Agentica · System: Attempts - 2+
trajs: s3://swe-bench-submissions/verified/20250629_deepswerl_r2eagent_tts/trajs
trajs docent: false
warning: Unknown
FreeBench model match: No explicit identity match; source name retained.

Source position 81 · 2025-07-26

o3

58.4% · resolved

Published setup details

agent: mini-SWE-agent
agent org: SWE-agent
checked: true
cost: 166.826374
date: 2025-07-26
folder: 20250726_mini-v1.0.0_o3-2025-04-16
instance calls: 24.698
instance cost: 0.333652748
logo: img/logos/20250726_mini-v1.0.0_o3-2025-04-16.svg · img/mini-icon.svg
logs: s3://swe-bench-submissions/bash-only/20250726_mini-v1.0.0_o3-2025-04-16/logs
mini-swe-agent version: 1.0.0
model display: o3
model org: OpenAI
model release date: 20250416
name: o3 (2025-04-16)
os model: false
os system: true
reasoning effort: Unknown
resolved: 58.4
site: https://platform.openai.com/docs/models/o3
submission: o3 (2025-04-16)
tags: Model: o3-20250416 · Org: OpenAI · System: Attempts - 1 · Mini: 1.0.0
trajs: s3://swe-bench-submissions/bash-only/20250726_mini-v1.0.0_o3-2025-04-16/trajs
trajs docent: https://docent.transluce.org/dashboard/565e5680-b913-4031-b537-00721a7a619a
warning: Unknown
FreeBench model match: No explicit identity match; source name retained.

Source position 82 · 2024-12-13

Undisclosed

58.2% · resolved

Published setup details

agent: devlo
agent org: Unknown
checked: false
cost: Unknown
date: 2024-12-13
folder: 20241213_devlo
instance calls: Unknown
instance cost: Unknown
logo: img/logos/20241213_devlo.jpg
logs: s3://swe-bench-submissions/verified/20241213_devlo/logs
model display: Undisclosed
model org: Unknown
model release date: Unknown
name: devlo
os model: false
os system: false
reasoning effort: Unknown
resolved: 58.2
site: https://devlo.ai/?utm_source=swe-bench
submission: devlo
tags: System: Attempts - 2+
trajs: s3://swe-bench-submissions/verified/20241213_devlo/trajs
trajs docent: false
warning: Unknown
FreeBench model match: No explicit identity match; source name retained.

Source position 83 · 2025-04-10

Multiple

58.2% · resolved

Published setup details

agent: Nemotron-CORTEXA
agent org: NVIDIA
checked: true
cost: Unknown
date: 2025-04-10
folder: 20250410_cortexa
instance calls: Unknown
instance cost: Unknown
logo: img/logos/20250410_cortexa.jpg
logs: s3://swe-bench-submissions/verified/20250410_cortexa/logs
model display: Multiple
model org: Anthropic
model release date: Unknown
name: Nemotron-CORTEXA
os model: false
os system: false
reasoning effort: Unknown
resolved: 58.2
site: https://research.nvidia.com/labs/adlr/cortexa/
submission: Nemotron-CORTEXA
tags: Model: NV-EmbedCode · Model: claude-3-5-sonnet-20241022 · Model: deepseek-v3 · Model: o3-mini · Model: gpt-4o-2024-05-13 · Model: gpt-4-turbo-2024-04-09 · Model: Qwen/Qwen2.5-72B-Instruct · Model: meta-llama/Llama-3.1-405B-Instruct · Model: meta-llama/Llama-3.3-70B-Instruct · Org: NVIDIA · System: Attempts - 2+
trajs: s3://swe-bench-submissions/verified/20250410_cortexa/trajs
trajs docent: false
warning: Unknown
FreeBench model match: No explicit identity match; source name retained.

Source position 84 · 2024-12-23

Multiple

57.2% · resolved

Published setup details

agent: Emergent E1
agent org: Unknown
checked: false
cost: Unknown
date: 2024-12-23
folder: 20241223_emergent
instance calls: Unknown
instance cost: Unknown
logo: img/logos/20241223_emergent.png
logs: s3://swe-bench-submissions/verified/20241223_emergent/logs
model display: Multiple
model org: Anthropic
model release date: Unknown
name: Emergent E1 (v2024-12-23)
os model: false
os system: false
reasoning effort: Unknown
resolved: 57.2
site: https://emergent.sh
submission: Emergent E1 (v2024-12-23)
tags: Model: claude-3-5-sonnet-20241022 · Model: gpt-4o-2024-08-06 · System: Attempts - 2+
trajs: s3://swe-bench-submissions/verified/20241223_emergent/trajs
trajs docent: false
warning: Unknown
FreeBench model match: No explicit identity match; source name retained.

Source position 85 · 2024-12-08

Undisclosed

57.0% · resolved

Published setup details

agent: Gru
agent org: Unknown
checked: false
cost: Unknown
date: 2024-12-08
folder: 20241208_gru
instance calls: Unknown
instance cost: Unknown
logo: img/logos/20241208_gru.png
logs: s3://swe-bench-submissions/verified/20241208_gru/logs
model display: Undisclosed
model org: Unknown
model release date: Unknown
name: Gru(2024-12-08)
os model: false
os system: false
reasoning effort: Unknown
resolved: 57
site: https://gru.ai
submission: Gru(2024-12-08)
tags: System: Attempts - 2+
trajs: s3://swe-bench-submissions/verified/20241208_gru/trajs
trajs docent: false
warning: Unknown
FreeBench model match: No explicit identity match; source name retained.

Source position 86 · 2025-09-24

Claude Sonnet 4

57.0% · resolved

Published setup details

agent: Artemis Agent v2
agent org: Turintech
checked: false
cost: Unknown
date: 2025-09-24
folder: 20250924_artemis_agent_v2
instance calls: Unknown
instance cost: Unknown
logo: img/logos/20250924_artemis_agent_v2.jpg
logs: s3://swe-bench-submissions/verified/20250924_artemis_agent_v2/logs
model display: Claude Sonnet 4
model org: Anthropic
model release date: Unknown
name: Artemis Agent v2 (2025-09-24)
os model: false
os system: false
reasoning effort: Unknown
resolved: 57
site: https://www.turintech.ai/
submission: Artemis Agent v2 (2025-09-24)
tags: Model: claude-sonnet-4-20250514 · Org: Turintech · System: Attempts - 1
trajs: s3://swe-bench-submissions/verified/20250924_artemis_agent_v2/trajs
trajs docent: false
warning: Unknown
FreeBench model match: No explicit identity match; source name retained.

Source position 87 · 2025-04-05

Undisclosed

56.6% · resolved

Published setup details

agent: SWE-Rizzo
agent org: Unknown
checked: false
cost: Unknown
date: 2025-04-05
folder: 20250405_swe-rizzo_claude37
instance calls: Unknown
instance cost: Unknown
logo: img/logos/20250405_swe-rizzo_claude37.png
logs: s3://swe-bench-submissions/verified/20250405_swe-rizzo_claude37/logs
model display: Undisclosed
model org: Unknown
model release date: Unknown
name: SWE-Rizzo
os model: false
os system: true
reasoning effort: Unknown
resolved: 56.6
site: https://gen42.ai
submission: SWE-Rizzo
tags: System: Attempts - 2+
trajs: s3://swe-bench-submissions/verified/20250405_swe-rizzo_claude37/trajs
trajs docent: false
warning: Unknown
FreeBench model match: No explicit identity match; source name retained.

Source position 88 · 2025-12-09

Devstral Small (2512)

56.4% · resolved

Published setup details

agent: mini-SWE-agent
agent org: SWE-agent
checked: true
cost: Unknown
date: 2025-12-09
folder: 20251209_mini-v1.17.2_devstral-small-2512
instance calls: 86.866
instance cost: Unknown
logo: img/logos/20251209_mini-v1.17.2_devstral-small-2512.svg · img/mini-icon.svg
logs: s3://swe-bench-submissions/bash-only/20251209_mini-v1.17.2_devstral-small-2512/logs
mini-swe-agent version: 1.17.2
model display: Devstral Small (2512)
model org: mistral
model release date: 20251209
name: Devstral Small (2512)
os model: true
os system: true
reasoning effort: Unknown
resolved: 56.4
site: Unknown
submission: Devstral Small (2512)
tags: Model: devstral-small-2512 · Org: mistral · System: Attempts - 1 · Mini: 1.17.2
trajs: s3://swe-bench-submissions/bash-only/20251209_mini-v1.17.2_devstral-small-2512/trajs
trajs docent: false
warning: Unknown
FreeBench model match: No explicit identity match; source name retained.

Source position 89 · 2026-02-17

GPT 5 mini

56.2% · resolved

Published setup details

agent: mini-SWE-agent
agent org: SWE-agent
checked: Unknown
cost: 23.600609549999998
date: 2026-02-17
folder: 20260217_mini-v2.0.0_gpt-5-mini
instance calls: 20.342
instance cost: 0.0472012191
logo: img/logos/20260217_mini-v2.0.0_gpt-5-mini.svg · img/mini-icon.svg
logs: Unknown
mini-swe-agent version: 2.0.0
model display: GPT 5 mini
model org: OpenAI
model release date: 20250807
name: GPT 5 mini
os model: false
os system: true
reasoning effort: Unknown
resolved: 56.2
site: https://platform.openai.com
submission: GPT 5 mini
tags: Model: gpt-5-mini-2025-08-07 · Org: OpenAI · System: Attempts - 1 · Mini: 2.0.0
trajs: s3://swe-bench-submissions/bash-only/20260217_mini-v2.0.0_gpt-5-mini/trajs
trajs docent: https://docent.transluce.org/dashboard/7e2c27bc-5b56-4359-b871-677bcb7bbdf1
warning: Unknown
FreeBench model match: No explicit identity match; source name retained.

Source position 90 · 2024-12-12

Claude 3.5 Sonnet

55.4% · resolved

Published setup details

agent: EPAM AI/Run Developer Agent
agent org: Unknown
checked: false
cost: Unknown
date: 2024-12-12
folder: 20241212_epam-ai-run-claude-3-5-sonnet
instance calls: Unknown
instance cost: Unknown
logo: img/logos/20241212_epam-ai-run-claude-3-5-sonnet.jpg
logs: s3://swe-bench-submissions/verified/20241212_epam-ai-run-claude-3-5-sonnet/logs
model display: Claude 3.5 Sonnet
model org: Anthropic
model release date: Unknown
name: EPAM AI/Run Developer Agent v20241212 + Anthopic Claude 3.5 Sonnet
os model: false
os system: false
reasoning effort: Unknown
resolved: 55.4
site: https://www.epam.com/services/artificial-intelligence
submission: EPAM AI/Run Developer Agent v20241212 + Anthopic Claude 3.5 Sonnet
tags: Model: claude-3-5-sonnet-20241022 · System: Attempts - 2+
trajs: s3://swe-bench-submissions/verified/20241212_epam-ai-run-claude-3-5-sonnet/trajs
trajs docent: false
warning: Unknown
FreeBench model match: No explicit identity match; source name retained.

Source position 91 · 2025-08-02

Qwen3-Coder 480B/A35B Instruct

55.4% · resolved

Published setup details

agent: mini-SWE-agent
agent org: SWE-agent
checked: true
cost: 123.963554
date: 2025-08-02
folder: 20250802_mini-v1.0.0_qwen3-coder-480b-a35b-instruct
instance calls: 64.792
instance cost: 0.247927108
logo: img/logos/20250802_mini-v1.0.0_qwen3-coder-480b-a35b-instruct.png · img/mini-icon.svg
logs: s3://swe-bench-submissions/bash-only/20250802_mini-v1.0.0_Qwen3-Coder-480B-A35B-Instruct/logs
mini-swe-agent version: 1.0.0
model display: Qwen3-Coder 480B/A35B Instruct
model org: Qwen
model release date: 20250722
name: Qwen3-Coder 480B/A35B Instruct
os model: true
os system: true
reasoning effort: Unknown
resolved: 55.4
site: https://qwenlm.github.io/blog/qwen3-coder/
submission: Qwen3-Coder 480B/A35B Instruct
tags: Model: Qwen3-Coder-480B-A35B-Instruct · Org: Qwen · System: Attempts - 1 · Mini: 1.0.0
trajs: s3://swe-bench-submissions/bash-only/20250802_mini-v1.0.0_Qwen3-Coder-480B-A35B-Instruct/trajs
trajs docent: https://docent.transluce.org/dashboard/f39d3041-d9d7-4f1b-b75e-8a13addb9e6e
warning: Unknown
FreeBench model match: No explicit identity match; source name retained.

Source position 92 · 2025-12-01

GLM 4.6

55.4% · resolved

Published setup details

agent: mini-SWE-agent
agent org: SWE-agent
checked: true
cost: 48.3050376
date: 2025-12-01
folder: 20251201_mini-v1.17.1_glm-4.6
instance calls: 49.348
instance cost: 0.09661007519999999
logo: img/logos/20251201_mini-v1.17.1_glm-4.6.svg · img/mini-icon.svg
logs: s3://swe-bench-submissions/bash-only/20251201_mini-v1.17.1_glm-4.6/logs
mini-swe-agent version: 1.17.1
model display: GLM 4.6
model org: Z.ai
model release date: 20250930
name: GLM 4.6 (T=1)
os model: true
os system: true
reasoning effort: Unknown
resolved: 55.4
site: Unknown
submission: GLM 4.6 (T=1)
tags: Model: glm-4.6 · Org: Z.ai · System: Attempts - 1 · Mini: 1.17.1
trajs: s3://swe-bench-submissions/bash-only/20251201_mini-v1.17.1_glm-4.6/trajs
trajs docent: https://docent.transluce.org/dashboard/1fc7d1bf-f715-42a1-8ebb-4dc19ccc8fc0
warning: Unknown
FreeBench model match: No explicit identity match; source name retained.

Source position 93 · 2024-12-02

Undisclosed

55.0% · resolved

Published setup details

agent: Amazon Q Developer Agent
agent org: AWS
checked: false
cost: Unknown
date: 2024-12-02
folder: 20241202_amazon-q-developer-agent-20241202-dev
instance calls: Unknown
instance cost: Unknown
logo: img/logos/20241202_amazon-q-developer-agent-20241202-dev.jpg
logs: s3://swe-bench-submissions/verified/20241202_amazon-q-developer-agent-20241202-dev/logs
model display: Undisclosed
model org: Unknown
model release date: Unknown
name: Amazon Q Developer Agent (v20241202-dev)
os model: false
os system: false
reasoning effort: Unknown
resolved: 55
site: https://aws.amazon.com/q/developer/
submission: Amazon Q Developer Agent (v20241202-dev)
tags: Org: AWS · System: Attempts - 2+
trajs: s3://swe-bench-submissions/verified/20241202_amazon-q-developer-agent-20241202-dev/trajs
trajs docent: false
warning: Unknown
FreeBench model match: No explicit identity match; source name retained.

Source position 94 · 2024-11-08

Claude 3.5 Sonnet

54.2% · resolved

Published setup details

agent: devlo
agent org: Unknown
checked: false
cost: Unknown
date: 2024-11-08
folder: 20241108_devlo
instance calls: Unknown
instance cost: Unknown
logo: img/logos/20241108_devlo.jpg
logs: s3://swe-bench-submissions/verified/20241108_devlo/logs
model display: Claude 3.5 Sonnet
model org: Anthropic
model release date: Unknown
name: devlo
os model: false
os system: false
reasoning effort: Unknown
resolved: 54.2
site: https://devlo.ai/?utm_source=swe-bench
submission: devlo
tags: Model: claude-3-5-sonnet-20241022 · System: Attempts - 2+
trajs: s3://swe-bench-submissions/verified/20241108_devlo/trajs
trajs docent: false
warning: Unknown
FreeBench model match: No explicit identity match; source name retained.

Source position 95 · 2025-08-22

GLM 4.5

54.2% · resolved

Published setup details

agent: mini-SWE-agent
agent org: SWE-agent
checked: true
cost: 148.5951744
date: 2025-08-22
folder: 20250822_mini-v1.9.1_glm-4.5
instance calls: 40.224
instance cost: 0.2971903488
logo: img/logos/20250822_mini-v1.9.1_glm-4.5.svg · img/mini-icon.svg
logs: s3://swe-bench-submissions/bash-only/20250822_mini-v1.9.1_glm-4.5/logs/
mini-swe-agent version: 1.9.1
model display: GLM 4.5
model org: Z.ai
model release date: 20250728
name: GLM 4.5 (2025-08-22)
os model: true
os system: true
reasoning effort: Unknown
resolved: 54.2
site: https://z.ai/blog/glm-4.5
submission: GLM 4.5 (2025-08-22)
tags: Model: GLM-4.5 · Org: Z.ai · System: Attempts - 1 · Mini: 1.9.1
trajs: s3://swe-bench-submissions/bash-only/20250822_mini-v1.9.1_glm-4.5/trajs/
trajs docent: https://docent.transluce.org/dashboard/9ae7f9a3-c9e3-48af-833c-52adcc7f5921
warning: Unknown
FreeBench model match: No explicit identity match; source name retained.

Source position 96 · 2025-12-09

Devstral (2512)

53.8% · resolved

Published setup details

agent: mini-SWE-agent
agent org: SWE-agent
checked: true
cost: Unknown
date: 2025-12-09
folder: 20251209_mini-v1.17.2_devstral-2512
instance calls: 75.086
instance cost: Unknown
logo: img/logos/20251209_mini-v1.17.2_devstral-2512.svg · img/mini-icon.svg
logs: s3://swe-bench-submissions/bash-only/20251209_mini-v1.17.2_devstral-2512/logs
mini-swe-agent version: 1.17.2
model display: Devstral (2512)
model org: mistral
model release date: 20251209
name: Devstral (2512)
os model: true
os system: true
reasoning effort: Unknown
resolved: 53.8
site: Unknown
submission: Devstral (2512)
tags: Model: devstral-2512 · Org: mistral · System: Attempts - 1 · Mini: 1.17.2
trajs: s3://swe-bench-submissions/bash-only/20251209_mini-v1.17.2_devstral-2512/trajs
trajs docent: false
warning: Unknown
FreeBench model match: No explicit identity match; source name retained.

Source position 97 · 2025-11-10

Frogboss 32B 2510

53.6% · resolved

Published setup details

agent: FrogBoss-32B-2510
agent org: Microsoft
checked: false (See README.md for info on how to get your results verified)
cost: Unknown
date: 2025-11-10
folder: 20251110_frogboss-32b
instance calls: Unknown
instance cost: Unknown
logo: img/logos/20251110_frogboss-32b.svg
logs: s3://swe-bench-submissions/verified/20251110_frogboss-32b/logs
model display: Frogboss 32B 2510
model org: Unknown
model release date: Unknown
name: FrogBoss-32B-2510
os model: false
os system: true
reasoning effort: Unknown
resolved: 53.6
site: https://microsoft.github.io/debug-gym/blog/2025/10/bug-pilot/
submission: FrogBoss-32B-2510
tags: Model: FrogBoss-32B-2510 · Org: Microsoft · System: Attempts - 1
trajs: s3://swe-bench-submissions/verified/20251110_frogboss-32b/trajs
trajs docent: false
warning: Unknown
FreeBench model match: No explicit identity match; source name retained.

Source position 98 · 2025-07-26

Gemini 2.5 Pro

53.6% · resolved

Published setup details

agent: mini-SWE-agent
agent org: SWE-agent
checked: true
cost: 144.18604781250002
date: 2025-07-26
folder: 20250726_mini-v1.0.0_gemini-2.5-pro
instance calls: 20.478
instance cost: 0.28837209562500005
logo: img/logos/20250726_mini-v1.0.0_gemini-2.5-pro.svg · img/mini-icon.svg
logs: s3://swe-bench-submissions/bash-only/20250726_mini-v1.0.0_gemini-2.5-pro/logs
mini-swe-agent version: 1.0.0
model display: Gemini 2.5 Pro
model org: Google DeepMind
model release date: 20250506
name: Gemini 2.5 Pro (2025-05-06)
os model: false
os system: true
reasoning effort: Unknown
resolved: 53.6
site: https://ai.google.dev/gemini-api/docs/models
submission: Gemini 2.5 Pro (2025-05-06)
tags: Model: gemini-2.5-pro · Org: Google DeepMind · System: Attempts - 1 · Mini: 1.0.0
trajs: s3://swe-bench-submissions/bash-only/20250726_mini-v1.0.0_gemini-2.5-pro/trajs
trajs docent: https://docent.transluce.org/dashboard/8b22ae97-2cfd-4b05-8438-38550cfaa163
warning: Unknown
FreeBench model match: No explicit identity match; source name retained.

Source position 99 · 2025-08-04

Kimi K2 Instruct

53.4% · resolved

Published setup details

agent: CodeSweep - SWE-agent
agent org: CodeSweep Inc.
checked: false
cost: Unknown
date: 2025-08-04
folder: 20250804_codesweep_sweagent_kimi_k2_instruct
instance calls: Unknown
instance cost: Unknown
logo: img/logos/20250804_codesweep_sweagent_kimi_k2_instruct.svg
logs: s3://swe-bench-submissions/verified/20250804_codesweep_sweagent_kimi_k2_instruct/logs
model display: Kimi K2 Instruct
model org: Moonshot AI
model release date: Unknown
name: CodeSweep - SWE-agent - Kimi K2 Instruct
os model: true
os system: true
reasoning effort: Unknown
resolved: 53.4
site: https://codesweep.ai/
submission: CodeSweep - SWE-agent - Kimi K2 Instruct
tags: Model: kimi-k2-instruct · Org: CodeSweep Inc. · System: Attempts - 1
trajs: s3://swe-bench-submissions/verified/20250804_codesweep_sweagent_kimi_k2_instruct/trajs
trajs docent: false
warning: Unknown
FreeBench model match: No explicit identity match; source name retained.

Source position 100 · 2025-01-20

Undisclosed

53.2% · resolved

Published setup details

agent: Bracket.sh
agent org: Unknown
checked: false
cost: Unknown
date: 2025-01-20
folder: 20250120_Bracket
instance calls: Unknown
instance cost: Unknown
logo: Unknown
logs: s3://swe-bench-submissions/verified/20250120_Bracket/logs
model display: Undisclosed
model org: Unknown
model release date: Unknown
name: Bracket.sh
os model: false
os system: false
reasoning effort: Unknown
resolved: 53.2
site: https://bracket.sh/
submission: Bracket.sh
tags: System: Attempts - 2+
trajs: s3://swe-bench-submissions/verified/20250120_Bracket/trajs
trajs docent: false
warning: Unknown
FreeBench model match: No explicit identity match; source name retained.

Source position 101 · 2024-10-29

CodeAct v2.1 (claude-3-5-sonnet-20241022)

53.0% · resolved

Published setup details

agent: OpenHands
agent org: OpenHands
checked: true
cost: Unknown
date: 2024-10-29
folder: 20241029_OpenHands-CodeAct-2.1-sonnet-20241022
instance calls: Unknown
instance cost: Unknown
logo: img/logos/20241029_OpenHands-CodeAct-2.1-sonnet-20241022.png
logs: s3://swe-bench-submissions/verified/20241029_OpenHands-CodeAct-2.1-sonnet-20241022/logs
model display: CodeAct v2.1 (claude-3-5-sonnet-20241022)
model org: Anthropic
model release date: Unknown
name: OpenHands + CodeAct v2.1 (claude-3-5-sonnet-20241022)
os model: false
os system: true
reasoning effort: Unknown
resolved: 53
site: https://docs.all-hands.dev/
submission: OpenHands + CodeAct v2.1 (claude-3-5-sonnet-20241022)
tags: Org: OpenHands · System: Attempts - 2+
trajs: s3://swe-bench-submissions/verified/20241029_OpenHands-CodeAct-2.1-sonnet-20241022/trajs
trajs docent: false
warning: Unknown
FreeBench model match: No explicit identity match; source name retained.

Source position 102 · 2025-07-20

Claude 3.7 Sonnet

52.8% · resolved

Published setup details

agent: mini-SWE-agent
agent org: SWE-agent
checked: true
cost: 177.3077241
date: 2025-07-20
folder: 20250720_mini-v0.0.0-claude-3-7-sonnet-20250219
instance calls: 34.7
instance cost: 0.3546154482
logo: img/logos/20250720_mini-v0.0.0-claude-3-7-sonnet-20250219.jpg · img/mini-icon.svg
logs: s3://swe-bench-submissions/bash-only/20250720_mini-v0.0.0-claude-3-7-sonnet-20250219/logs
mini-swe-agent version: 0.0.0
model display: Claude 3.7 Sonnet
model org: Anthropic
model release date: 20250224
name: Claude 3.7 Sonnet (20250219)
os model: false
os system: true
reasoning effort: Unknown
resolved: 52.8
site: https://www.anthropic.com/news/claude-3-7-sonnet
submission: Claude 3.7 Sonnet (20250219)
tags: Model: claude-3-7-sonnet-20250219 · Org: Anthropic · System: Attempts - 1 · Mini: 0.0.0
trajs: s3://swe-bench-submissions/bash-only/20250720_mini-v0.0.0-claude-3-7-sonnet-20250219/trajs
trajs docent: false
warning: Unknown
FreeBench model match: No explicit identity match; source name retained.

Source position 103 · 2024-12-12

Gemini 2.0 Flash (v20241212-experimental)

52.2% · resolved

Published setup details

agent: Google Jules
agent org: Google
checked: false
cost: Unknown
date: 2024-12-12
folder: 20241212_google_jules_gemini_2.0_flash_experimental
instance calls: Unknown
instance cost: Unknown
logo: img/logos/20241212_google_jules_gemini_2.0_flash_experimental.png
logs: s3://swe-bench-submissions/verified/20241212_google_jules_gemini_2.0_flash_experimental/logs
model display: Gemini 2.0 Flash (v20241212-experimental)
model org: Google DeepMind
model release date: Unknown
name: Google Jules + Gemini 2.0 Flash (v20241212-experimental)
os model: false
os system: false
reasoning effort: Unknown
resolved: 52.2
site: https://developers.googleblog.com/en/the-next-chapter-of-the-gemini-era-for-developers/
submission: Google Jules + Gemini 2.0 Flash (v20241212-experimental)
tags: Org: Google · System: Attempts - 2+
trajs: s3://swe-bench-submissions/verified/20241212_google_jules_gemini_2.0_flash_experimental/trajs
trajs docent: false
warning: Unknown
FreeBench model match: No explicit identity match; source name retained.

Source position 104 · 2025-09-01

Qwen3-Coder-30B-A3B-Instruct

52.2% · resolved

Published setup details

agent: EntroPO + R2E
agent org: 42-b3yond-6ug
checked: false (See README.md for info on how to get your results verified)
cost: Unknown
date: 2025-09-01
folder: 20250901_entroPO_R2E_QwenCoder30BA3B
instance calls: Unknown
instance cost: Unknown
logo: img/logos/20250901_entroPO_R2E_QwenCoder30BA3B.png
logs: s3://swe-bench-submissions/verified/20250901_entroPO_R2E_QwenCoder30BA3B/logs
model display: Qwen3-Coder-30B-A3B-Instruct
model org: Qwen
model release date: Unknown
name: EntroPO + R2E + Qwen3-Coder-30B-A3B-Instruct
os model: true
os system: true
reasoning effort: Unknown
resolved: 52.2
site: https://github.com/sherdencooper/R2E-Gym
submission: EntroPO + R2E + Qwen3-Coder-30B-A3B-Instruct
tags: Model: Qwen3-Coder-30B-A3B-Instruct · Org: 42-b3yond-6ug · System: Attempts - 1
trajs: s3://swe-bench-submissions/verified/20250901_entroPO_R2E_QwenCoder30BA3B/trajs
trajs docent: false
warning: Unknown
FreeBench model match: No explicit identity match; source name retained.

Source position 105 · 2024-11-25

Claude 3.5 Sonnet

51.8% · resolved

Published setup details

agent: Engine Labs
agent org: Engine Labs
checked: false
cost: Unknown
date: 2024-11-25
folder: 20241125_enginelabs
instance calls: Unknown
instance cost: Unknown
logo: img/logos/20241125_enginelabs.svg
logs: s3://swe-bench-submissions/verified/20241125_enginelabs/logs
model display: Claude 3.5 Sonnet
model org: Anthropic
model release date: Unknown
name: Engine Labs (2024-11-25)
os model: false
os system: false
reasoning effort: Unknown
resolved: 51.8
site: https://www.enginelabs.ai/
submission: Engine Labs (2024-11-25)
tags: Model: claude-3-5-sonnet-20241022 · Org: Engine Labs · System: Attempts - 1
trajs: s3://swe-bench-submissions/verified/20241125_enginelabs/trajs
trajs docent: false
warning: Unknown
FreeBench model match: No explicit identity match; source name retained.

Source position 106 · 2025-01-22

Claude 3.5-Sonnet-20241022

51.6% · resolved

Published setup details

agent: AutoCodeRover-v2.1
agent org: Unknown
checked: false
cost: Unknown
date: 2025-01-22
folder: 20250122_autocoderover-v2.1-claude-3-5-sonnet-20241022
instance calls: Unknown
instance cost: Unknown
logo: img/logos/20250122_autocoderover-v2.1-claude-3-5-sonnet-20241022.png
logs: Unknown
model display: Claude 3.5-Sonnet-20241022
model org: Anthropic
model release date: Unknown
name: AutoCodeRover-v2.1 (Claude-3.5-Sonnet-20241022)
os model: false
os system: false
reasoning effort: Unknown
resolved: 51.6
site: https://www.autocoderover.net/
submission: AutoCodeRover-v2.1 (Claude-3.5-Sonnet-20241022)
tags: Model: claude-3-5-sonnet-20241022 · System: Attempts - 2+
trajs: Unknown
trajs docent: false
warning: Unknown
FreeBench model match: No explicit identity match; source name retained.

Source position 107 · 2025-08-05

Qwen3-Coder-30B-A3B-Instruct

51.6% · resolved

Published setup details

agent: OpenHands
agent org: Qwen
checked: false
cost: Unknown
date: 2025-08-05
folder: 20250805_openhands-Qwen3-Coder-30B-A3B-Instruct
instance calls: Unknown
instance cost: Unknown
logo: img/logos/20250805_openhands-Qwen3-Coder-30B-A3B-Instruct.png
logs: s3://swe-bench-submissions/verified/20250805-openhands-Qwen3-Coder-30B-A3B-Instruct/logs
model display: Qwen3-Coder-30B-A3B-Instruct
model org: Qwen
model release date: Unknown
name: OpenHands + Qwen3-Coder-30B-A3B-Instruct
os model: true
os system: true
reasoning effort: Unknown
resolved: 51.6
site: https://qwenlm.github.io/blog/qwen3-coder/
submission: OpenHands + Qwen3-Coder-30B-A3B-Instruct
tags: Model: https://huggingface.co/Qwen/Qwen3-Coder-30B-A3B-Instruct · Org: Qwen · Org: OpenHands · System: Attempts - 1
trajs: s3://swe-bench-submissions/verified/20250805-openhands-Qwen3-Coder-30B-A3B-Instruct/trajs
trajs docent: false
warning: Unknown
FreeBench model match: No explicit identity match; source name retained.

Source position 108 · 2024-12-02

Claude 3.5 Sonnet

50.8% · resolved

Published setup details

agent: Agentless-1.5
agent org: Agentless
checked: false
cost: Unknown
date: 2024-12-02
folder: 20241202_agentless-1.5_claude-3.5-sonnet-20241022
instance calls: Unknown
instance cost: Unknown
logo: img/logos/20241202_agentless-1.5_claude-3.5-sonnet-20241022.png
logs: s3://swe-bench-submissions/verified/20241202_agentless-1.5_claude-3.5-sonnet-20241022/logs
model display: Claude 3.5 Sonnet
model org: Anthropic
model release date: Unknown
name: Agentless-1.5 + Claude-3.5 Sonnet (20241022)
os model: false
os system: true
reasoning effort: Unknown
resolved: 50.8
site: https://github.com/OpenAutoCoder/Agentless
submission: Agentless-1.5 + Claude-3.5 Sonnet (20241022)
tags: Model: claude-3-5-sonnet-20241022 · Org: Agentless · System: Attempts - 2+
trajs: s3://swe-bench-submissions/verified/20241202_agentless-1.5_claude-3.5-sonnet-20241022/trajs
trajs docent: false
warning: Unknown
FreeBench model match: No explicit identity match; source name retained.

Source position 109 · 2024-10-28

Undisclosed

50.0% · resolved

Published setup details

agent: Solver
agent org: Unknown
checked: false
cost: Unknown
date: 2024-10-28
folder: 20241028_solver
instance calls: Unknown
instance cost: Unknown
logo: img/logos/20241028_solver.png
logs: s3://swe-bench-submissions/verified/20241028_solver/logs
model display: Undisclosed
model org: Unknown
model release date: Unknown
name: Solver (2024-10-28)
os model: false
os system: false
reasoning effort: Unknown
resolved: 50
site: https://solverai.com/
submission: Solver (2024-10-28)
tags: System: Attempts - 2+
trajs: s3://swe-bench-submissions/verified/20241028_solver/trajs
trajs docent: false
warning: Unknown
FreeBench model match: No explicit identity match; source name retained.

Source position 110 · 2024-11-25

Undisclosed

50.0% · resolved

Published setup details

agent: Bytedance MarsCode Agent
agent org: Bytedance
checked: false
cost: Unknown
date: 2024-11-25
folder: 20241125_marscode-agent-dev
instance calls: Unknown
instance cost: Unknown
logo: img/logos/20241125_marscode-agent-dev.png
logs: s3://swe-bench-submissions/verified/20241125_marscode-agent-dev/logs
model display: Undisclosed
model org: Unknown
model release date: Unknown
name: Bytedance MarsCode Agent
os model: false
os system: false
reasoning effort: Unknown
resolved: 50
site: https://www.marscode.com/
submission: Bytedance MarsCode Agent
tags: Org: Bytedance · System: Attempts - 2+
trajs: s3://swe-bench-submissions/verified/20241125_marscode-agent-dev/trajs
trajs docent: false
warning: Unknown
FreeBench model match: No explicit identity match; source name retained.

Source position 111 · 2024-11-05

Multiple

49.2% · resolved

Published setup details

agent: nFactorial
agent org: Unknown
checked: false
cost: Unknown
date: 2024-11-05
folder: 20241105_nfactorial
instance calls: Unknown
instance cost: Unknown
logo: img/logos/20241105_nfactorial.png
logs: s3://swe-bench-submissions/verified/20241105_nfactorial/logs
model display: Multiple
model org: Anthropic
model release date: Unknown
name: nFactorial (2024-11-05)
os model: false
os system: false
reasoning effort: Unknown
resolved: 49.2
site: https://nfactorial.dev/
submission: nFactorial (2024-11-05)
tags: Model: claude-3-5-sonnet-20241022 · Model: gpt-4o-2024-08-06 · System: Attempts - 2+
trajs: s3://swe-bench-submissions/verified/20241105_nfactorial/trajs
trajs docent: false
warning: Unknown
FreeBench model match: No explicit identity match; source name retained.

Source position 112 · 2024-10-22

Claude 3.5 Sonnet

49.0% · resolved

Published setup details

agent: Tools
agent org: Anthropic
checked: false
cost: Unknown
date: 2024-10-22
folder: 20241022_tools_claude-3-5-sonnet-updated
instance calls: Unknown
instance cost: Unknown
logo: img/logos/20241022_tools_claude-3-5-sonnet-updated.jpg
logs: s3://swe-bench-submissions/verified/20241022_tools_claude-3-5-sonnet-updated/logs
model display: Claude 3.5 Sonnet
model org: Anthropic
model release date: Unknown
name: Tools + Claude 3.5 Sonnet (2024-10-22)
os model: false
os system: false
reasoning effort: Unknown
resolved: 49
site: https://www.anthropic.com/
submission: Tools + Claude 3.5 Sonnet (2024-10-22)
tags: Model: claude-3-5-sonnet-20241022 · Model: claude-3.5-sonnet-20241022 · Org: Anthropic · System: Attempts - 2+
trajs: s3://swe-bench-submissions/verified/20241022_tools_claude-3-5-sonnet-updated/trajs
trajs docent: false
warning: Unknown
FreeBench model match: No explicit identity match; source name retained.

Source position 113 · 2024-10-25

Multiple

48.6% · resolved

Published setup details

agent: Composio SWE-Kit
agent org: Unknown
checked: true
cost: Unknown
date: 2024-10-25
folder: 20241025_composio_swekit
instance calls: Unknown
instance cost: Unknown
logo: img/logos/20241025_composio_swekit.png
logs: s3://swe-bench-submissions/verified/20241025_composio_swekit/logs
model display: Multiple
model org: Anthropic
model release date: Unknown
name: Composio SWE-Kit (2024-10-25)
os model: false
os system: true
reasoning effort: Unknown
resolved: 48.6
site: https://github.com/ComposioHQ/composio/tree/master/python/swe/agent
submission: Composio SWE-Kit (2024-10-25)
tags: Model: claude-3-5-sonnet-20241022 · Model: o1-mini · System: Attempts - 2+
trajs: s3://swe-bench-submissions/verified/20241025_composio_swekit/trajs
trajs docent: false
warning: Unknown
FreeBench model match: No explicit identity match; source name retained.

Source position 114 · 2024-11-06

Multiple

47.2% · resolved

Published setup details

agent: AppMap Navie v2
agent org: Unknown
checked: true
cost: Unknown
date: 2024-11-06
folder: 20241106_navie-2-gpt4o-sonnet
instance calls: Unknown
instance cost: Unknown
logo: img/logos/20241106_navie-2-gpt4o-sonnet.png
logs: s3://swe-bench-submissions/verified/20241106_navie-2-gpt4o-sonnet/logs
model display: Multiple
model org: Anthropic
model release date: Unknown
name: AppMap Navie v2
os model: false
os system: true
reasoning effort: Unknown
resolved: 47.2
site: https://appmap.io/navie
submission: AppMap Navie v2
tags: Model: claude-3-5-sonnet-20241022 · Model: gpt-4o-2024-08-06 · System: Attempts - 2+
trajs: s3://swe-bench-submissions/verified/20241106_navie-2-gpt4o-sonnet/trajs
trajs docent: false
warning: Unknown
FreeBench model match: No explicit identity match; source name retained.

Source position 115 · 2025-06-16

TTS(Bo8)

47.0% · resolved

Published setup details

agent: Skywork-SWE-32B
agent org: Skywork AI
checked: true
cost: Unknown
date: 2025-06-16
folder: 20250616_Skywork-SWE-32B+TTS_Bo8
instance calls: Unknown
instance cost: Unknown
logo: img/logos/20250616_Skywork-SWE-32B+TTS_Bo8.jpg
logs: s3://swe-bench-submissions/verified/20250616_Skywork-SWE-32B+TTS_Bo8/logs
model display: TTS(Bo8)
model org: Qwen
model release date: Unknown
name: Skywork-SWE-32B + TTS(Bo8)
os model: true
os system: true
reasoning effort: Unknown
resolved: 47
site: https://quixotic-sting-239.notion.site/eb17f379610040ceb54da5d5d24065bd
submission: Skywork-SWE-32B + TTS(Bo8)
tags: Model: Qwen2.5-Coder-32B-Instruct · Org: Skywork AI · System: Attempts - 2+
trajs: s3://swe-bench-submissions/verified/20250616_Skywork-SWE-32B+TTS_Bo8/trajs
trajs docent: false
warning: Unknown
FreeBench model match: No explicit identity match; source name retained.

Source position 116 · 2025-05-20

DevStral Small 2505

46.8% · resolved

Published setup details

agent: OpenHands
agent org: OpenHands
checked: true
cost: Unknown
date: 2025-05-20
folder: 20250520_openhands_devstral_small
instance calls: Unknown
instance cost: Unknown
logo: img/logos/20250520_openhands_devstral_small.png
logs: s3://swe-bench-submissions/verified/20250520_openhands_devstral_small/logs
model display: DevStral Small 2505
model org: Mistral
model release date: Unknown
name: OpenHands + DevStral Small 2505
os model: true
os system: true
reasoning effort: Unknown
resolved: 46.8
site: https://all-hands.dev/
submission: OpenHands + DevStral Small 2505
tags: Model: mistralai/Devstral-Small-2505 · Org: OpenHands · Org: Mistral · System: Attempts - 1
trajs: s3://swe-bench-submissions/verified/20250520_openhands_devstral_small/trajs
trajs docent: false
warning: Unknown
FreeBench model match: No explicit identity match; source name retained.

Source position 117 · 2024-10-23

Multiple

46.6% · resolved

Published setup details

agent: Emergent E1
agent org: Unknown
checked: false
cost: Unknown
date: 2024-10-23
folder: 20241023_emergent
instance calls: Unknown
instance cost: Unknown
logo: img/logos/20241023_emergent.png
logs: s3://swe-bench-submissions/verified/20241023_emergent/logs
model display: Multiple
model org: Anthropic
model release date: Unknown
name: Emergent E1 (v2024-10-12)
os model: false
os system: false
reasoning effort: Unknown
resolved: 46.6
site: https://emergent.sh
submission: Emergent E1 (v2024-10-12)
tags: Model: claude-3-5-sonnet-20241022 · Model: gpt-4o-2024-08-06 · System: Attempts - 2+
trajs: s3://swe-bench-submissions/verified/20241023_emergent/trajs
trajs docent: false
warning: Unknown
FreeBench model match: No explicit identity match; source name retained.

Source position 118 · 2024-11-08

Claude 3.5-Sonnet-20241022

46.2% · resolved

Published setup details

agent: AutoCodeRover-v2.0
agent org: Unknown
checked: false
cost: Unknown
date: 2024-11-08
folder: 20241108_autocoderover-v2.0-claude-3-5-sonnet-20241022
instance calls: Unknown
instance cost: Unknown
logo: img/logos/20241108_autocoderover-v2.0-claude-3-5-sonnet-20241022.png
logs: s3://swe-bench-submissions/verified/20241108_autocoderover-v2.0-claude-3-5-sonnet-20241022/logs
model display: Claude 3.5-Sonnet-20241022
model org: Anthropic
model release date: Unknown
name: AutoCodeRover-v2.0 (Claude-3.5-Sonnet-20241022)
os model: false
os system: true
reasoning effort: Unknown
resolved: 46.2
site: https://www.autocoderover.net/
submission: AutoCodeRover-v2.0 (Claude-3.5-Sonnet-20241022)
tags: Model: claude-3-5-sonnet-20241022 · System: Attempts - 2+
trajs: s3://swe-bench-submissions/verified/20241108_autocoderover-v2.0-claude-3-5-sonnet-20241022/trajs
trajs docent: false
warning: Unknown
FreeBench model match: No explicit identity match; source name retained.

Source position 119 · 2025-05-28

Co-PatcheR

46.0% · resolved

Published setup details

agent: PatchPilot
agent org: Unknown
checked: false
cost: Unknown
date: 2025-05-28
folder: 20250528_patchpilot_Co-PatcheR
instance calls: Unknown
instance cost: Unknown
logo: img/logos/20250528_patchpilot_Co-PatcheR.gif
logs: s3://swe-bench-submissions/verified/20250528_patchpilot_Co-PatcheR/logs
model display: Co-PatcheR
model org: Unknown
model release date: Unknown
name: PatchPilot + Co-PatcheR
os model: true
os system: true
reasoning effort: Unknown
resolved: 46
site: https://github.com/ucsb-mlsec/Co-PatcheR
submission: PatchPilot + Co-PatcheR
tags: System: Attempts - 2+
trajs: s3://swe-bench-submissions/verified/20250528_patchpilot_Co-PatcheR/trajs
trajs docent: false
warning: Unknown
FreeBench model match: No explicit identity match; source name retained.

Source position 120 · 2024-09-24

Undisclosed

45.4% · resolved

Published setup details

agent: Solver
agent org: Unknown
checked: false
cost: Unknown
date: 2024-09-24
folder: 20240924_solver
instance calls: Unknown
instance cost: Unknown
logo: img/logos/20240924_solver.png
logs: s3://swe-bench-submissions/verified/20240924_solver/logs
model display: Undisclosed
model org: Unknown
model release date: Unknown
name: Solver (2024-09-12)
os model: false
os system: false
reasoning effort: Unknown
resolved: 45.4
site: https://laredolabs.com/
submission: Solver (2024-09-12)
tags: System: Attempts - 2+
trajs: s3://swe-bench-submissions/verified/20240924_solver/trajs
trajs docent: false
warning: Unknown
FreeBench model match: No explicit identity match; source name retained.

Source position 121 · 2024-08-24

Undisclosed

45.2% · resolved

Published setup details

agent: Gru
agent org: Unknown
checked: false
cost: Unknown
date: 2024-08-24
folder: 20240824_gru
instance calls: Unknown
instance cost: Unknown
logo: img/logos/20240824_gru.png
logs: s3://swe-bench-submissions/verified/20240824_gru/logs
model display: Undisclosed
model org: Unknown
model release date: Unknown
name: Gru(2024-08-24)
os model: false
os system: false
reasoning effort: Unknown
resolved: 45.2
site: https://gru.ai
submission: Gru(2024-08-24)
tags: System: Attempts - 2+
trajs: s3://swe-bench-submissions/verified/20240824_gru/trajs
trajs docent: false
warning: Unknown
FreeBench model match: No explicit identity match; source name retained.

Source position 122 · 2025-11-10

Frogmini 14B 2510

45.0% · resolved

Published setup details

agent: FrogMini-14B-2510
agent org: Microsoft
checked: false (See README.md for info on how to get your results verified)
cost: Unknown
date: 2025-11-10
folder: 20251110_frogmini-14b
instance calls: Unknown
instance cost: Unknown
logo: img/logos/20251110_frogmini-14b.svg
logs: s3://swe-bench-submissions/verified/20251110_frogmini-14b/logs
model display: Frogmini 14B 2510
model org: Unknown
model release date: Unknown
name: FrogMini-14B-2510
os model: false
os system: true
reasoning effort: Unknown
resolved: 45
site: https://microsoft.github.io/debug-gym/blog/2025/10/bug-pilot/
submission: FrogMini-14B-2510
tags: Model: FrogMini-14B-2510 · Org: Microsoft · System: Attempts - 1
trajs: s3://swe-bench-submissions/verified/20251110_frogmini-14b/trajs
trajs docent: false
warning: Unknown
FreeBench model match: No explicit identity match; source name retained.

Source position 123 · 2025-07-26

o4-mini

45.0% · resolved

Published setup details

agent: mini-SWE-agent
agent org: SWE-agent
checked: true
cost: 104.9866664
date: 2025-07-26
folder: 20250726_mini-v1.0.0_o4-mini-2025-04-16
instance calls: 23.278
instance cost: 0.2099733328
logo: img/logos/20250726_mini-v1.0.0_o4-mini-2025-04-16.svg · img/mini-icon.svg
logs: s3://swe-bench-submissions/bash-only/20250521_o4-mini-20250416/logs
mini-swe-agent version: 1.0.0
model display: o4-mini
model org: OpenAI
model release date: 20250416
name: o4-mini (2025-04-16)
os model: false
os system: true
reasoning effort: Unknown
resolved: 45
site: https://platform.openai.com/docs/models/o4-mini
submission: o4-mini (2025-04-16)
tags: Model: o4-mini-20250416 · Org: OpenAI · System: Attempts - 1 · Mini: 1.0.0
trajs: s3://swe-bench-submissions/bash-only/20250726_mini-v1.0.0_o4-mini-2025-04-16/trajs
trajs docent: https://docent.transluce.org/dashboard/f46eb5b4-b4b2-478e-bc11-f1eb47eca637
warning: Unknown
FreeBench model match: No explicit identity match; source name retained.

Source position 124 · 2025-01-18

Gemini 2.0 Flash (Experimental)

44.2% · resolved

Published setup details

agent: CodeShellAgent
agent org: Unknown
checked: false
cost: Unknown
date: 2025-01-18
folder: 20250118_codeshellagent_gemini_2.0_flash_experimental
instance calls: Unknown
instance cost: Unknown
logo: img/logos/20250118_codeshellagent_gemini_2.0_flash_experimental.png
logs: evaluation/verified/20250118_codeshellagent_gemini_2.0_flash_experimental/logs
model display: Gemini 2.0 Flash (Experimental)
model org: Google DeepMind
model release date: Unknown
name: CodeShellAgent + Gemini 2.0 Flash (Experimental)
os model: false
os system: false
reasoning effort: Unknown
resolved: 44.2
site: https://github.com/WisdomShell/codeshell
submission: CodeShellAgent + Gemini 2.0 Flash (Experimental)
tags: Model: gemini-2.0-flash-exp · System: Attempts - 2+
trajs: evaluation/verified/20250118_codeshellagent_gemini_2.0_flash_experimental/trajs
trajs docent: false
warning: Unknown
FreeBench model match: No explicit identity match; source name retained.

Source position 125 · 2025-08-07

Kimi K2 Instruct

43.8% · resolved

Published setup details

agent: mini-SWE-agent
agent org: SWE-agent
checked: true
cost: 265.796683
date: 2025-08-07
folder: 20250807_mini-v1.7.0_kimi-k2-instruct
instance calls: 37.52
instance cost: 0.531593366
logo: img/logos/20250807_mini-v1.7.0_kimi-k2-instruct.jpg · img/mini-icon.svg
logs: s3://swe-bench-submissions/bash-only/20250807_mini-v1.7.0_Kimi-K2-Instruct/logs
mini-swe-agent version: 1.7.0
model display: Kimi K2 Instruct
model org: Moonshot AI
model release date: 20250711
name: Kimi K2 Instruct
os model: true
os system: true
reasoning effort: Unknown
resolved: 43.8
site: https://moonshotai.github.io/Kimi-K2/
submission: Kimi K2 Instruct
tags: Model: Kimi-K2-Instruct · Org: Moonshot AI · System: Attempts - 1 · Mini: 1.7.0
trajs: s3://swe-bench-submissions/bash-only/20250807_mini-v1.7.0_Kimi-K2-Instruct/trajs
trajs docent: https://docent.transluce.org/dashboard/b6a6d69b-e69e-4011-aa94-46f44b84202e
warning: Unknown
FreeBench model match: No explicit identity match; source name retained.

Source position 126 · 2024-09-20

Undisclosed

43.6% · resolved

Published setup details

agent: Solver
agent org: Unknown
checked: false
cost: Unknown
date: 2024-09-20
folder: 20240920_solver
instance calls: Unknown
instance cost: Unknown
logo: img/logos/20240920_solver.png
logs: s3://swe-bench-submissions/verified/20240920_solver/logs
model display: Undisclosed
model org: Unknown
model release date: Unknown
name: Solver (2024-09-12)
os model: false
os system: false
reasoning effort: Unknown
resolved: 43.6
site: https://laredolabs.com/
submission: Solver (2024-09-12)
tags: System: Attempts - 2+
trajs: s3://swe-bench-submissions/verified/20240920_solver/trajs
trajs docent: false
warning: Unknown
FreeBench model match: No explicit identity match; source name retained.

Source position 127 · 2025-02-14

O3 Mini

42.4% · resolved

Published setup details

agent: Agentless Lite
agent org: Agentless
checked: true
cost: Unknown
date: 2025-02-14
folder: 20250214_agentless_lite_o3_mini
instance calls: Unknown
instance cost: Unknown
logo: img/logos/20250214_agentless_lite_o3_mini.png
logs: s3://swe-bench-submissions/verified/20250214_agentless_lite_o3_mini/logs
model display: O3 Mini
model org: OpenAI
model release date: Unknown
name: Agentless Lite + O3 Mini (20250214)
os model: false
os system: true
reasoning effort: Unknown
resolved: 42.4
site: https://github.com/sorendunn/Agentless-Lite
submission: Agentless Lite + O3 Mini (20250214)
tags: Model: o3-mini · Org: Agentless · System: Attempts - 2+
trajs: s3://swe-bench-submissions/verified/20250214_agentless_lite_o3_mini/trajs
trajs docent: false
warning: Unknown
FreeBench model match: No explicit identity match; source name retained.

Source position 128 · 2025-05-27

Amazon.nova Premier v1:0

42.4% · resolved

Published setup details

agent: Amazon Nova Premier 1.0
agent org: Amazon Nova
checked: false
cost: Unknown
date: 2025-05-27
folder: 20250527_amazon.nova-premier-v1.0
instance calls: Unknown
instance cost: Unknown
logo: img/logos/20250527_amazon.nova-premier-v1.0.png
logs: s3://swe-bench-submissions/verified/20250527_amazon.nova-premier-v1.0/logs
model display: Amazon.nova Premier v1:0
model org: Unknown
model release date: Unknown
name: Amazon Nova Premier 1.0 (2025-04-30)
os model: false
os system: false
reasoning effort: Unknown
resolved: 42.4
site: aws.amazon.com/nova
submission: Amazon Nova Premier 1.0 (2025-04-30)
tags: Model: amazon.nova-premier-v1:0 · Org: Amazon Nova · System: Attempts - 2+
trajs: s3://swe-bench-submissions/verified/20250527_amazon.nova-premier-v1.0/trajs
trajs docent: false
warning: Unknown
FreeBench model match: No explicit identity match; source name retained.

Source position 129 · 2025-06-29

DeepSWE-Preview

42.2% · resolved

Published setup details

agent: R2E-Gym
agent org: Agentica
checked: false
cost: Unknown
date: 2025-06-29
folder: 20250629_deepswerl_r2eagent
instance calls: Unknown
instance cost: Unknown
logo: img/logos/20250629_deepswerl_r2eagent.jpg
logs: s3://swe-bench-submissions/verified/20250629_deepswerl_r2eagent/logs
model display: DeepSWE-Preview
model org: Unknown
model release date: Unknown
name: DeepSWE-Preview
os model: true
os system: true
reasoning effort: Unknown
resolved: 42.2
site: agentica-project.com/
submission: DeepSWE-Preview
tags: Model: https://huggingface.co/agentica-org/DeepSWE-Preview · Org: Agentica · System: Attempts - 1
trajs: s3://swe-bench-submissions/verified/20250629_deepswerl_r2eagent/trajs
trajs docent: false
warning: Unknown
FreeBench model match: No explicit identity match; source name retained.

Source position 130 · 2025-08-06

DeepSeek V3 0324

42.0% · resolved

Published setup details

agent: SWE-Exp
agent org: SWE-Exp
checked: false (See README.md for info on how to get your results verified)
cost: Unknown
date: 2025-08-06
folder: 20250806_SWE-Exp_DeepSeek-V3
instance calls: Unknown
instance cost: Unknown
logo: img/logos/20250806_SWE-Exp_DeepSeek-V3.png
logs: s3://swe-bench-submissions/verified/20250806_SWE-Exp_DeepSeek-V3/logs
model display: DeepSeek V3 0324
model org: DeepSeek
model release date: Unknown
name: SWE-Exp
os model: true
os system: true
reasoning effort: Unknown
resolved: 42
site: https://github.com/YerbaPage/SWE-Exp
submission: SWE-Exp
tags: Model: DeepSeek-V3-0324 · Org: SWE-Exp · System: Attempts - 2
trajs: s3://swe-bench-submissions/verified/20250806_SWE-Exp_DeepSeek-V3/trajs
trajs docent: false
warning: Unknown
FreeBench model match: No explicit identity match; source name retained.

Source position 131 · 2024-10-30

Multiple

41.6% · resolved

Published setup details

agent: nFactorial
agent org: Unknown
checked: false
cost: Unknown
date: 2024-10-30
folder: 20241030_nfactorial
instance calls: Unknown
instance cost: Unknown
logo: img/logos/20241030_nfactorial.png
logs: s3://swe-bench-submissions/verified/20241030_nfactorial/logs
model display: Multiple
model org: Anthropic
model release date: Unknown
name: nFactorial (2024-10-30)
os model: false
os system: false
reasoning effort: Unknown
resolved: 41.6
site: https://nfactorial.dev/
submission: nFactorial (2024-10-30)
tags: Model: claude-3-5-sonnet-20241022 · Model: gpt-4o-2024-08-06 · Model: o1-preview · System: Attempts - 2+
trajs: s3://swe-bench-submissions/verified/20241030_nfactorial/trajs
trajs docent: false
warning: Unknown
FreeBench model match: No explicit identity match; source name retained.

Source position 132 · 2025-01-12

Undisclosed

41.6% · resolved

Published setup details

agent: ugaiforge
agent org: Unknown
checked: false
cost: Unknown
date: 2025-01-12
folder: 20250112_ugaiforge
instance calls: Unknown
instance cost: Unknown
logo: Unknown
logs: s3://swe-bench-submissions/verified/20250112_ugaiforge/logs
model display: Undisclosed
model org: Unknown
model release date: Unknown
name: ugaiforge
os model: false
os system: false
reasoning effort: Unknown
resolved: 41.6
site: Unknown
submission: ugaiforge
tags: System: Attempts - 2+
trajs: s3://swe-bench-submissions/verified/20250112_ugaiforge/trajs
trajs docent: false
warning: Unknown
FreeBench model match: No explicit identity match; source name retained.

Source position 133 · 2025-02-26

Llama3-SWE-RL-70B

41.2% · resolved

Published setup details

agent: Agentless Mini
agent org: Agentless
checked: false
cost: Unknown
date: 2025-02-26
folder: 20250226_swerl_llama3_70b
instance calls: Unknown
instance cost: Unknown
logo: img/logos/20250226_swerl_llama3_70b.png
logs: s3://swe-bench-submissions/verified/20250226_swerl_llama3_70b/logs
model display: Llama3-SWE-RL-70B
model org: Meta
model release date: Unknown
name: SWE-RL (Llama3-SWE-RL-70B + Agentless Mini) (20250226)
os model: false
os system: true
reasoning effort: Unknown
resolved: 41.2
site: https://github.com/facebookresearch/swe-rl
submission: SWE-RL (Llama3-SWE-RL-70B + Agentless Mini) (20250226)
tags: Model: Llama3-SWE-RL-70B · Model_size: 70 · Org: Agentless · System: Attempts - 2+
trajs: s3://swe-bench-submissions/verified/20250226_swerl_llama3_70b/trajs
trajs docent: false
warning: Unknown
FreeBench model match: No explicit identity match; source name retained.

Source position 134 · 2024-08-20

Undisclosed

40.6% · resolved

Published setup details

agent: Honeycomb
agent org: Unknown
checked: false
cost: Unknown
date: 2024-08-20
folder: 20240820_honeycomb
instance calls: Unknown
instance cost: Unknown
logo: img/logos/20240820_honeycomb.png
logs: s3://swe-bench-submissions/verified/20240820_honeycomb/logs
model display: Undisclosed
model org: Unknown
model release date: Unknown
name: Honeycomb
os model: false
os system: false
reasoning effort: Unknown
resolved: 40.6
site: https://honeycomb.sh/
submission: Honeycomb
tags: System: Attempts - 2+
trajs: s3://swe-bench-submissions/verified/20240820_honeycomb/trajs
trajs docent: false
warning: Unknown
FreeBench model match: No explicit identity match; source name retained.

Source position 135 · 2024-10-16

Claude 3.5 Sonnet

40.6% · resolved

Published setup details

agent: Composio SWEkit
agent org: Unknown
checked: false
cost: Unknown
date: 2024-10-16
folder: 20241016_composio_swekit
instance calls: Unknown
instance cost: Unknown
logo: img/logos/20241016_composio_swekit.png
logs: s3://swe-bench-submissions/verified/20241016_composio_swekit/logs
model display: Claude 3.5 Sonnet
model org: Anthropic
model release date: Unknown
name: Composio SWEkit + Claude 3.5 Sonnet (2024-10-16)
os model: false
os system: true
reasoning effort: Unknown
resolved: 40.6
site: https://github.com/ComposioHQ/composio/tree/master/python/swe/agent
submission: Composio SWEkit + Claude 3.5 Sonnet (2024-10-16)
tags: Model: claude-3-5-sonnet-20241022 · Model: claude-3.5-sonnet-20241022 · System: Attempts - 2+
trajs: s3://swe-bench-submissions/verified/20241016_composio_swekit/trajs
trajs docent: false
warning: Unknown
FreeBench model match: No explicit identity match; source name retained.

Source position 136 · 2024-10-22

Claude 3.5 Haiku

40.6% · resolved

Published setup details

agent: Tools
agent org: Anthropic
checked: false
cost: Unknown
date: 2024-10-22
folder: 20241022_tools_claude-3-5-haiku
instance calls: Unknown
instance cost: Unknown
logo: img/logos/20241022_tools_claude-3-5-haiku.jpg
logs: s3://swe-bench-submissions/verified/20241022_tools_claude-3-5-haiku/logs
model display: Claude 3.5 Haiku
model org: Anthropic
model release date: Unknown
name: Tools + Claude 3.5 Haiku
os model: false
os system: false
reasoning effort: Unknown
resolved: 40.6
site: https://www.anthropic.com/
submission: Tools + Claude 3.5 Haiku
tags: Model: claude-3-haiku-20240307 · Org: Anthropic · System: Attempts - 2+
trajs: s3://swe-bench-submissions/verified/20241022_tools_claude-3-5-haiku/trajs
trajs docent: false
warning: Unknown
FreeBench model match: No explicit identity match; source name retained.

Source position 137 · 2024-11-13

Multiple

40.6% · resolved

Published setup details

agent: Nebius AI
agent org: Unknown
checked: false
cost: Unknown
date: 2024-11-13
folder: 20241113_nebius-search-open-weight-models-11-24
instance calls: Unknown
instance cost: Unknown
logo: img/logos/20241113_nebius-search-open-weight-models-11-24.png
logs: s3://swe-bench-submissions/verified/20241113_nebius-search-open-weight-models-11-24/logs
model display: Multiple
model org: Meta
model release date: Unknown
name: Nebius AI Qwen 2.5 72B Generator + LLama 3.1 70B Critic
os model: false
os system: false
reasoning effort: Unknown
resolved: 40.6
site: https://nebius.com/blog/posts/training-and-search-for-software-engineering-agents
submission: Nebius AI Qwen 2.5 72B Generator + LLama 3.1 70B Critic
tags: Model: Llama 3.1 · Model: Qwen 2.5 · System: Attempts - 2+
trajs: s3://swe-bench-submissions/verified/20241113_nebius-search-open-weight-models-11-24/trajs
trajs docent: false
warning: Unknown
FreeBench model match: No explicit identity match; source name retained.

Source position 138 · 2025-05-11

SWE-agent-LM-32B

40.2% · resolved

Published setup details

agent: SWE-agent
agent org: SWE-agent
checked: true
cost: Unknown
date: 2025-05-11
folder: 20250511_sweagent_lm_32b
instance calls: Unknown
instance cost: Unknown
logo: img/logos/20250511_sweagent_lm_32b.png
logs: s3://swe-bench-submissions/verified/20250511_sweagent_lm_32b/logs
model display: SWE-agent-LM-32B
model org: Qwen
model release date: Unknown
name: SWE-agent + SWE-agent-LM-32B
os model: true
os system: true
reasoning effort: Unknown
resolved: 40.2
site: https://swesmith.com/
submission: SWE-agent + SWE-agent-LM-32B
tags: Model: Qwen 2.5 · Org: SWE-agent · System: Attempts - 1
trajs: s3://swe-bench-submissions/verified/20250511_sweagent_lm_32b/trajs
trajs docent: false
warning: Unknown
FreeBench model match: No explicit identity match; source name retained.

Source position 139 · 2024-10-29

Claude 3.5 Sonnet

39.6% · resolved

Published setup details

agent: EPAM AI/Run Developer Agent
agent org: Unknown
checked: false
cost: Unknown
date: 2024-10-29
folder: 20241029_epam-ai-run-claude-3-5-sonnet
instance calls: Unknown
instance cost: Unknown
logo: img/logos/20241029_epam-ai-run-claude-3-5-sonnet.jpg
logs: s3://swe-bench-submissions/verified/20241029_epam-ai-run-claude-3-5-sonnet/logs
model display: Claude 3.5 Sonnet
model org: Anthropic
model release date: Unknown
name: EPAM AI/Run Developer Agent v20241029 + Anthopic Claude 3.5 Sonnet
os model: false
os system: false
reasoning effort: Unknown
resolved: 39.6
site: https://www.epam.com/services/artificial-intelligence
submission: EPAM AI/Run Developer Agent v20241029 + Anthopic Claude 3.5 Sonnet
tags: Model: claude-3-5-sonnet-20241022 · System: Attempts - 2+
trajs: s3://swe-bench-submissions/verified/20241029_epam-ai-run-claude-3-5-sonnet/trajs
trajs docent: false
warning: Unknown
FreeBench model match: No explicit identity match; source name retained.

Source position 140 · 2025-07-26

GPT 4.1

39.6% · resolved

Published setup details

agent: mini-SWE-agent
agent org: SWE-agent
checked: true
cost: 73.156562
date: 2025-07-26
folder: 20250726_mini-v1.0.0_gpt-4.1-2025-04-14
instance calls: 19.544
instance cost: 0.146313124
logo: img/logos/20250726_mini-v1.0.0_gpt-4.1-2025-04-14.svg · img/mini-icon.svg
logs: s3://swe-bench-submissions/bash-only/20250726_mini-v1.0.0_gpt-4.1-2025-04-14/logs
mini-swe-agent version: 1.0.0
model display: GPT 4.1
model org: OpenAI
model release date: 20250414
name: GPT 4.1 (2025-04-14)
os model: false
os system: true
reasoning effort: Unknown
resolved: 39.58
site: https://platform.openai.com/docs/models/gpt-4.1
submission: GPT 4.1 (2025-04-14)
tags: Model: gpt-4.1-20250414 · Org: OpenAI · System: Attempts - 1 · Mini: 1.0.0
trajs: s3://swe-bench-submissions/bash-only/20250726_mini-v1.0.0_gpt-4.1-2025-04-14/trajs
trajs docent: https://docent.transluce.org/dashboard/cd7a23c5-a2b1-4cab-b851-6e2c42aaf0f3
warning: Unknown
FreeBench model match: No explicit identity match; source name retained.

Source position 141 · 2024-07-21

Undisclosed

38.8% · resolved

Published setup details

agent: Amazon Q Developer Agent
agent org: AWS
checked: false
cost: Unknown
date: 2024-07-21
folder: 20240721_amazon-q-developer-agent-20240719-dev
instance calls: Unknown
instance cost: Unknown
logo: img/logos/20240721_amazon-q-developer-agent-20240719-dev.jpg
logs: s3://swe-bench-submissions/verified/20240721_amazon-q-developer-agent-20240719-dev/logs
model display: Undisclosed
model org: Unknown
model release date: Unknown
name: Amazon Q Developer Agent (v20240719-dev)
os model: false
os system: false
reasoning effort: Unknown
resolved: 38.8
site: https://aws.amazon.com/q/developer/
submission: Amazon Q Developer Agent (v20240719-dev)
tags: Org: AWS · System: Attempts - 2+
trajs: s3://swe-bench-submissions/verified/20240721_amazon-q-developer-agent-20240719-dev/trajs
trajs docent: false
warning: Unknown
FreeBench model match: No explicit identity match; source name retained.

Source position 142 · 2024-10-28

GPT-4o

38.8% · resolved

Published setup details

agent: Agentless-1.5
agent org: Agentless
checked: false
cost: Unknown
date: 2024-10-28
folder: 20241028_agentless-1.5_gpt4o
instance calls: Unknown
instance cost: Unknown
logo: img/logos/20241028_agentless-1.5_gpt4o.png
logs: s3://swe-bench-submissions/verified/20241028_agentless-1.5_gpt4o/logs
model display: GPT-4o
model org: OpenAI
model release date: Unknown
name: Agentless-1.5 + GPT 4o (2024-05-13)
os model: false
os system: true
reasoning effort: Unknown
resolved: 38.8
site: https://github.com/OpenAutoCoder/Agentless
submission: Agentless-1.5 + GPT 4o (2024-05-13)
tags: Model: gpt-4o-2024-05-13 · Org: Agentless · System: Attempts - 1
trajs: s3://swe-bench-submissions/verified/20241028_agentless-1.5_gpt4o/trajs
trajs docent: false
warning: Unknown
FreeBench model match: No explicit identity match; source name retained.

Source position 143 · 2024-06-28

GPT-4o

38.4% · resolved

Published setup details

agent: AutoCodeRover
agent org: Unknown
checked: false
cost: Unknown
date: 2024-06-28
folder: 20240628_autocoderover-v20240620
instance calls: Unknown
instance cost: Unknown
logo: img/logos/20240628_autocoderover-v20240620.jpg
logs: s3://swe-bench-submissions/verified/20240628_autocoderover-v20240620/logs
model display: GPT-4o
model org: OpenAI
model release date: Unknown
name: AutoCodeRover (v20240620) + GPT 4o (2024-05-13)
os model: false
os system: false
reasoning effort: Unknown
resolved: 38.4
site: https://autocoderover.dev/
submission: AutoCodeRover (v20240620) + GPT 4o (2024-05-13)
tags: Model: gpt-4o-2024-05-13 · System: Attempts - 2+
trajs: Unknown
trajs docent: false
warning: Unknown
FreeBench model match: No explicit identity match; source name retained.

Source position 144 · 2025-06-16

Qwen2.5 Coder 32B Instruct

38.0% · resolved

Published setup details

agent: Skywork-SWE-32B
agent org: Skywork AI
checked: true
cost: Unknown
date: 2025-06-16
folder: 20250616_Skywork-SWE-32B
instance calls: Unknown
instance cost: Unknown
logo: img/logos/20250616_Skywork-SWE-32B.jpg
logs: s3://swe-bench-submissions/verified/20250616_Skywork-SWE-32B/logs
model display: Qwen2.5 Coder 32B Instruct
model org: Qwen
model release date: Unknown
name: Skywork-SWE-32B
os model: true
os system: true
reasoning effort: Unknown
resolved: 38
site: https://quixotic-sting-239.notion.site/eb17f379610040ceb54da5d5d24065bd
submission: Skywork-SWE-32B
tags: Model: Qwen2.5-Coder-32B-Instruct · Org: Skywork AI · System: Attempts - 1
trajs: s3://swe-bench-submissions/verified/20250616_Skywork-SWE-32B/trajs
trajs docent: false
warning: Unknown
FreeBench model match: No explicit identity match; source name retained.

Source position 145 · 2025-07-25

DevStral Small 2507

38.0% · resolved

Published setup details

agent: SWE-agent
agent org: SWE-agent
checked: true
cost: Unknown
date: 2025-07-25
folder: 20250725_sweagent_devstral_small_2507
instance calls: Unknown
instance cost: Unknown
logo: img/logos/20250725_sweagent_devstral_small_2507.png
logs: s3://swe-bench-submissions/verified/20250725_sweagent_devstral_small_2507/logs
model display: DevStral Small 2507
model org: Mistral
model release date: Unknown
name: SWE-agent + DevStral Small 2507
os model: true
os system: true
reasoning effort: Unknown
resolved: 38
site: https://swe-agent.com/
submission: SWE-agent + DevStral Small 2507
tags: Model: devstral-small-2507 · Org: SWE-agent · System: Attempts - 1
trajs: s3://swe-bench-submissions/verified/20250725_sweagent_devstral_small_2507/trajs
trajs docent: false
warning: Unknown
FreeBench model match: No explicit identity match; source name retained.

Source position 146 · 2024-06-17

Undisclosed

37.0% · resolved

Published setup details

agent: Factory Code Droid
agent org: Unknown
checked: false
cost: Unknown
date: 2024-06-17
folder: 20240617_factory_code_droid
instance calls: Unknown
instance cost: Unknown
logo: img/logos/20240617_factory_code_droid.jpg
logs: s3://swe-bench-submissions/verified/20240617_factory_code_droid/logs
model display: Undisclosed
model org: Unknown
model release date: Unknown
name: Factory Code Droid
os model: false
os system: false
reasoning effort: Unknown
resolved: 37
site: https://www.factory.ai/
submission: Factory Code Droid
tags: System: Attempts - 2+
trajs: Unknown
trajs docent: false
warning: Unknown
FreeBench model match: No explicit identity match; source name retained.

Source position 147 · 2025-08-07

GPT 5 nano

34.8% · resolved

Published setup details

agent: mini-SWE-agent
agent org: SWE-agent
checked: true
cost: 19.037590390000002
date: 2025-08-07
folder: 20250807_mini-v1.7.0_gpt-5-nano
instance calls: 39.76
instance cost: 0.03807518078
logo: img/logos/20250807_mini-v1.7.0_gpt-5-nano.svg · img/mini-icon.svg
logs: s3://swe-bench-submissions/bash-only/20250807_mini-v1.7.0_gpt-5-nano/logs
mini-swe-agent version: 1.7.0
model display: GPT 5 nano
model org: OpenAI
model release date: 20250807
name: GPT 5 nano (2025-08-07) (medium)
os model: false
os system: true
reasoning effort: medium
resolved: 34.8
site: https://platform.openai.com/docs/models/gpt-5-nano
submission: GPT 5 nano (2025-08-07) (medium)
tags: Model: gpt-5-nano-2025-08-07 · Org: OpenAI · System: Attempts - 1 · Mini: 1.7.0
trajs: s3://swe-bench-submissions/bash-only/20250807_mini-v1.7.0_gpt-5-nano/trajs
trajs docent: https://docent.transluce.org/dashboard/3dc9b424-547c-4bdc-9b72-3b58d316cb0a
warning: Unknown
FreeBench model match: No explicit identity match; source name retained.

Source position 148 · 2024-06-20

Claude 3.5 Sonnet

33.6% · resolved

Published setup details

agent: SWE-agent
agent org: SWE-agent
checked: true
cost: Unknown
date: 2024-06-20
folder: 20240620_sweagent_claude3.5sonnet
instance calls: Unknown
instance cost: Unknown
logo: img/logos/20240620_sweagent_claude3.5sonnet.png
logs: s3://swe-bench-submissions/verified/20240620_sweagent_claude3.5sonnet/logs
model display: Claude 3.5 Sonnet
model org: Anthropic
model release date: Unknown
name: SWE-agent + Claude 3.5 Sonnet
os model: false
os system: true
reasoning effort: Unknown
resolved: 33.6
site: https://github.com/SWE-agent/SWE-agent
submission: SWE-agent + Claude 3.5 Sonnet
tags: Model: claude-3-5-sonnet-20241022 · Org: SWE-agent · System: Attempts - 1
trajs: s3://swe-bench-submissions/verified/20240620_sweagent_claude3.5sonnet/trajs
trajs docent: false
warning: Unknown
FreeBench model match: No explicit identity match; source name retained.

Source position 149 · 2025-03-06

Qwen2.5 (7B + 72B)

32.8% · resolved

Published setup details

agent: SWE-Fixer
agent org: Unknown
checked: true
cost: Unknown
date: 2025-03-06
folder: 20250306_SWE-Fixer_Qwen2.5-7b-retriever_Qwen2.5-72b-editor
instance calls: Unknown
instance cost: Unknown
logo: img/logos/20250306_SWE-Fixer_Qwen2.5-7b-retriever_Qwen2.5-72b-editor.png
logs: s3://swe-bench-submissions/verified/20250306_SWE-Fixer_Qwen2.5-7b-retriever_Qwen2.5-72b-editor/logs
model display: Qwen2.5 (7B + 72B)
model org: Qwen
model release date: Unknown
name: SWE-Fixer (Qwen2.5-7b retriever + Qwen2.5-72b editor)
os model: true
os system: true
reasoning effort: Unknown
resolved: 32.8
site: https://github.com/InternLM/SWE-Fixer
submission: SWE-Fixer (Qwen2.5-7b retriever + Qwen2.5-72b editor)
tags: Model: Qwen 2.5 · System: Attempts - 1
trajs: s3://swe-bench-submissions/verified/20250306_SWE-Fixer_Qwen2.5-7b-retriever_Qwen2.5-72b-editor/trajs
trajs docent: false
warning: Unknown
FreeBench model match: No explicit identity match; source name retained.

Source position 150 · 2024-06-12

GPT-4o

32.6% · resolved

Published setup details

agent: MASAI
agent org: Unknown
checked: false
cost: Unknown
date: 2024-06-12
folder: 20240612_MASAI_gpt4o
instance calls: Unknown
instance cost: Unknown
logo: Unknown
logs: s3://swe-bench-submissions/verified/20240612_MASAI_gpt4o/logs
model display: GPT-4o
model org: OpenAI
model release date: Unknown
name: MASAI + GPT 4o (2024-06-12)
os model: false
os system: false
reasoning effort: Unknown
resolved: 32.6
site: https://masai-dev-agent.github.io/
submission: MASAI + GPT 4o (2024-06-12)
tags: Model: gpt-4o-2024-08-06 · System: Attempts - 2+
trajs: s3://swe-bench-submissions/verified/20240612_MASAI_gpt4o/trajs
trajs docent: false
warning: Unknown
FreeBench model match: No explicit identity match; source name retained.

Source position 151 · 2024-11-20

Undisclosed

32.0% · resolved

Published setup details

agent: Artemis Agent v1
agent org: Unknown
checked: false
cost: Unknown
date: 2024-11-20
folder: 20241120_artemis_agent
instance calls: Unknown
instance cost: Unknown
logo: img/logos/20241120_artemis_agent.jpg
logs: s3://swe-bench-submissions/verified/20241120_artemis_agent/logs
model display: Undisclosed
model org: Unknown
model release date: Unknown
name: Artemis Agent v1 (2024-11-20)
os model: false
os system: false
reasoning effort: Unknown
resolved: 32
site: https://www.turintech.ai/
submission: Artemis Agent v1 (2024-11-20)
tags: System: Attempts - 2+
trajs: s3://swe-bench-submissions/verified/20241120_artemis_agent/trajs
trajs docent: false
warning: Unknown
FreeBench model match: No explicit identity match; source name retained.

Source position 152 · 2024-10-07

Undisclosed

31.6% · resolved

Published setup details

agent: nFactorial
agent org: Unknown
checked: false
cost: Unknown
date: 2024-10-07
folder: 20241007_nfactorial
instance calls: Unknown
instance cost: Unknown
logo: img/logos/20241007_nfactorial.png
logs: s3://swe-bench-submissions/verified/20241007_nfactorial/logs
model display: Undisclosed
model org: Unknown
model release date: Unknown
name: nFactorial (2024-10-07)
os model: false
os system: false
reasoning effort: Unknown
resolved: 31.6
site: https://nfactorial.dev/
submission: nFactorial (2024-10-07)
tags: System: Attempts - 2+
trajs: s3://swe-bench-submissions/verified/20241007_nfactorial/trajs
trajs docent: false
warning: Unknown
FreeBench model match: No explicit identity match; source name retained.

Source position 153 · 2024-11-28

Qwen2.5 (7B + 72B)

30.2% · resolved

Published setup details

agent: SWE-Fixer
agent org: Unknown
checked: false
cost: Unknown
date: 2024-11-28
folder: 20241128_SWE-Fixer_Qwen2.5-7b-retriever_Qwen2.5-72b-editor_20241128
instance calls: Unknown
instance cost: Unknown
logo: img/logos/20241128_SWE-Fixer_Qwen2.5-7b-retriever_Qwen2.5-72b-editor_20241128.png
logs: s3://swe-bench-submissions/verified/20241128_SWE-Fixer_Qwen2.5-7b-retriever_Qwen2.5-72b-editor_20241128/logs
model display: Qwen2.5 (7B + 72B)
model org: Qwen
model release date: Unknown
name: SWE-Fixer (Qwen2.5-7b retriever + Qwen2.5-72b editor) 20241128
os model: false
os system: true
reasoning effort: Unknown
resolved: 30.2
site: https://github.com/InternLM/SWE-Fixer
submission: SWE-Fixer (Qwen2.5-7b retriever + Qwen2.5-72b editor) 20241128
tags: Model: Qwen 2.5 · System: Attempts - 2+
trajs: s3://swe-bench-submissions/verified/20241128_SWE-Fixer_Qwen2.5-7b-retriever_Qwen2.5-72b-editor_20241128/trajs
trajs docent: false
warning: Unknown
FreeBench model match: No explicit identity match; source name retained.

Source position 154 · 2024-10-02

Lingma SWE-GPT 72b (v0925)

28.8% · resolved

Published setup details

agent: Lingma Agent
agent org: Alibaba
checked: false
cost: Unknown
date: 2024-10-02
folder: 20241002_lingma-agent_lingma-swe-gpt-72b
instance calls: Unknown
instance cost: Unknown
logo: img/logos/20241002_lingma-agent_lingma-swe-gpt-72b.jpg
logs: s3://swe-bench-submissions/verified/20241002_lingma-agent_lingma-swe-gpt-72b/logs
model display: Lingma SWE-GPT 72b (v0925)
model org: OpenAI
model release date: Unknown
name: Lingma Agent + Lingma SWE-GPT 72b (v0925)
os model: false
os system: true
reasoning effort: Unknown
resolved: 28.8
site: https://www.modelscope.cn/models/yingwei/Lingma-SWE-GPT (https://www.modelscope.cn/models/yingwei/Lingma-SWE-GPT-v20240925)
submission: Lingma Agent + Lingma SWE-GPT 72b (v0925)
tags: Org: Alibaba · System: Attempts - 2+
trajs: s3://swe-bench-submissions/verified/20241002_lingma-agent_lingma-swe-gpt-72b/trajs
trajs docent: false
warning: Unknown
FreeBench model match: No explicit identity match; source name retained.

Source position 155 · 2025-07-26

Gemini 2.5 Flash

28.7% · resolved

Published setup details

agent: mini-SWE-agent
agent org: SWE-agent
checked: true
cost: 66.942859175
date: 2025-07-26
folder: 20250726_mini-v1.0.0_gemini-2.5-flash
instance calls: 26.646
instance cost: 0.13388571835
logo: img/logos/20250726_mini-v1.0.0_gemini-2.5-flash.svg · img/mini-icon.svg
logs: s3://swe-bench-submissions/bash-only/20250726_mini-v1.0.0_gemini-2.5-flash/logs
mini-swe-agent version: 1.0.0
model display: Gemini 2.5 Flash
model org: Google DeepMind
model release date: 20250417
name: Gemini 2.5 Flash (2025-04-17)
os model: false
os system: true
reasoning effort: Unknown
resolved: 28.73
site: https://ai.google.dev/gemini-api/docs/models
submission: Gemini 2.5 Flash (2025-04-17)
tags: Model: gemini-2.5-flash · Org: Google DeepMind · System: Attempts - 1 · Mini: 1.0.0
trajs: s3://swe-bench-submissions/bash-only/20250726_mini-v1.0.0_gemini-2.5-flash/trajs
trajs docent: https://docent.transluce.org/dashboard/37e5933e-71a1-4011-8340-fbca563124b3
warning: Unknown
FreeBench model match: No explicit identity match; source name retained.

Source position 156 · 2024-10-16

GPT-4o

27.0% · resolved

Published setup details

agent: EPAM AI/Run Developer Agent
agent org: Unknown
checked: false
cost: Unknown
date: 2024-10-16
folder: 20241016_epam-ai-run-gpt-4o
instance calls: Unknown
instance cost: Unknown
logo: img/logos/20241016_epam-ai-run-gpt-4o.jpg
logs: s3://swe-bench-submissions/verified/20241016_epam-ai-run-gpt-4o/logs
model display: GPT-4o
model org: OpenAI
model release date: Unknown
name: EPAM AI/Run Developer Agent + GPT4o
os model: false
os system: false
reasoning effort: Unknown
resolved: 27
site: https://www.epam.com/services/artificial-intelligence
submission: EPAM AI/Run Developer Agent + GPT4o
tags: Model: gpt-4o-2024-08-06 · System: Attempts - 2+
trajs: s3://swe-bench-submissions/verified/20241016_epam-ai-run-gpt-4o/trajs
trajs docent: false
warning: Unknown
FreeBench model match: No explicit identity match; source name retained.

Source position 157 · 2024-06-15

GPT-4o

26.2% · resolved

Published setup details

agent: AppMap Navie
agent org: Unknown
checked: true
cost: Unknown
date: 2024-06-15
folder: 20240615_appmap-navie_gpt4o
instance calls: Unknown
instance cost: Unknown
logo: img/logos/20240615_appmap-navie_gpt4o.png
logs: s3://swe-bench-submissions/verified/20240615_appmap-navie_gpt4o/logs
model display: GPT-4o
model org: OpenAI
model release date: Unknown
name: AppMap Navie + GPT 4o (2024-05-13)
os model: false
os system: true
reasoning effort: Unknown
resolved: 26.2
site: https://appmap.io/navie
submission: AppMap Navie + GPT 4o (2024-05-13)
tags: Model: gpt-4o-2024-05-13 · System: Attempts - 2+
trajs: Unknown
trajs docent: false
warning: Unknown
FreeBench model match: No explicit identity match; source name retained.

Source position 158 · 2025-08-07

gpt-oss-120b

26.0% · resolved

Published setup details

agent: mini-SWE-agent
agent org: SWE-agent
checked: true
cost: 28.559527899999996
date: 2025-08-07
folder: 20250807_mini-v1.7.0_gpt-oss-120b
instance calls: 27.598
instance cost: 0.05711905579999999
logo: img/logos/20250807_mini-v1.7.0_gpt-oss-120b.svg · img/mini-icon.svg
logs: s3://swe-bench-submissions/bash-only/20250807_mini-v1.7.0_gpt-oss-120b/logs
mini-swe-agent version: 1.7.0
model display: gpt-oss-120b
model org: OpenAI
model release date: 20250805
name: gpt-oss-120b
os model: true
os system: true
reasoning effort: Unknown
resolved: 26
site: https://platform.openai.com/docs/models/gpt-oss-120b
submission: gpt-oss-120b
tags: Model: gpt-oss-120b · Org: OpenAI · System: Attempts - 1 · Mini: 1.7.0
trajs: s3://swe-bench-submissions/bash-only/20250807_mini-v1.7.0_gpt-oss-120b/trajs
trajs docent: https://docent.transluce.org/dashboard/4a0a2b2a-a65b-452a-88c7-bb5d99745c75
warning: Unknown
FreeBench model match: No explicit identity match; source name retained.

Source position 159 · 2024-10-01

Undisclosed

25.8% · resolved

Published setup details

agent: nFactorial
agent org: Unknown
checked: false
cost: Unknown
date: 2024-10-01
folder: 20241001_nfactorial
instance calls: Unknown
instance cost: Unknown
logo: img/logos/20241001_nfactorial.png
logs: s3://swe-bench-submissions/verified/20241001_nfactorial/logs
model display: Undisclosed
model org: Unknown
model release date: Unknown
name: nFactorial (2024-10-01)
os model: false
os system: false
reasoning effort: Unknown
resolved: 25.8
site: https://nfactorial.dev/
submission: nFactorial (2024-10-01)
tags: System: Attempts - 2+
trajs: s3://swe-bench-submissions/verified/20241001_nfactorial/trajs
trajs docent: false
warning: Unknown
FreeBench model match: No explicit identity match; source name retained.

Source position 160 · 2024-05-09

Undisclosed

25.6% · resolved

Published setup details

agent: Amazon Q Developer Agent
agent org: AWS
checked: false
cost: Unknown
date: 2024-05-09
folder: 20240509_amazon-q-developer-agent-20240430-dev
instance calls: Unknown
instance cost: Unknown
logo: img/logos/20240509_amazon-q-developer-agent-20240430-dev.jpg
logs: s3://swe-bench-submissions/verified/20240509_amazon-q-developer-agent-20240430-dev/logs
model display: Undisclosed
model org: Unknown
model release date: Unknown
name: Amazon Q Developer Agent (v20240430-dev)
os model: false
os system: false
reasoning effort: Unknown
resolved: 25.6
site: https://aws.amazon.com/q/developer/
submission: Amazon Q Developer Agent (v20240430-dev)
tags: Org: AWS · System: Attempts - 2+
trajs: Unknown
trajs docent: false
warning: Unknown
FreeBench model match: No explicit identity match; source name retained.

Source position 161 · 2024-09-18

Lingma SWE-GPT 72b (v0918)

25.0% · resolved

Published setup details

agent: Lingma Agent
agent org: Alibaba
checked: false
cost: Unknown
date: 2024-09-18
folder: 20240918_lingma-agent_lingma-swe-gpt-72b
instance calls: Unknown
instance cost: Unknown
logo: img/logos/20240918_lingma-agent_lingma-swe-gpt-72b.jpg
logs: s3://swe-bench-submissions/verified/20240918_lingma-agent_lingma-swe-gpt-72b/logs
model display: Lingma SWE-GPT 72b (v0918)
model org: OpenAI
model release date: Unknown
name: Lingma Agent + Lingma SWE-GPT 72b (v0918)
os model: false
os system: true
reasoning effort: Unknown
resolved: 25
site: https://www.modelscope.cn/models/yingwei/Lingma-SWE-GPT
submission: Lingma Agent + Lingma SWE-GPT 72b (v0918)
tags: Org: Alibaba · System: Attempts - 2+
trajs: s3://swe-bench-submissions/verified/20240918_lingma-agent_lingma-swe-gpt-72b/trajs
trajs docent: false
warning: Unknown
FreeBench model match: No explicit identity match; source name retained.

Source position 162 · 2024-08-20

GPT-4o

24.0% · resolved

Published setup details

agent: EPAM AI/Run Developer Agent
agent org: Unknown
checked: false
cost: Unknown
date: 2024-08-20
folder: 20240820_epam-ai-run-gpt-4o
instance calls: Unknown
instance cost: Unknown
logo: img/logos/20240820_epam-ai-run-gpt-4o.jpg
logs: s3://swe-bench-submissions/verified/20240820_epam-ai-run-gpt-4o/logs
model display: GPT-4o
model org: OpenAI
model release date: Unknown
name: EPAM AI/Run Developer Agent + GPT4o
os model: false
os system: false
reasoning effort: Unknown
resolved: 24
site: https://www.epam.com/services/artificial-intelligence
submission: EPAM AI/Run Developer Agent + GPT4o
tags: Model: gpt-4o-2024-08-06 · System: Attempts - 2+
trajs: s3://swe-bench-submissions/verified/20240820_epam-ai-run-gpt-4o/trajs
trajs docent: false
warning: Unknown
FreeBench model match: No explicit identity match; source name retained.

Source position 163 · 2025-07-20

GPT 4.1 mini

23.9% · resolved

Published setup details

agent: mini-SWE-agent
agent org: SWE-agent
checked: true
cost: 217.57707639999998
date: 2025-07-20
folder: 20250720_mini-v0.0.0_gpt-4.1-mini-2025-04-14
instance calls: 89.196
instance cost: 0.43515415279999997
logo: img/logos/20250720_mini-v0.0.0_gpt-4.1-mini-2025-04-14.svg · img/mini-icon.svg
logs: s3://swe-bench-submissions/bash-only/20250720_mini-v0.0.0_gpt-4.1-mini-2025-04-14/logs
mini-swe-agent version: 0.0.0
model display: GPT 4.1 mini
model org: OpenAI
model release date: 20250414
name: GPT 4.1 mini (2025-04-14)
os model: false
os system: true
reasoning effort: Unknown
resolved: 23.94
site: https://platform.openai.com/docs/models/gpt-4.1-mini
submission: GPT 4.1 mini (2025-04-14)
tags: Model: gpt-4.1-mini-20250414 · Org: OpenAI · System: Attempts - 1 · Mini: 0.0.0
trajs: s3://swe-bench-submissions/bash-only/20250720_mini-v0.0.0_gpt-4.1-mini-2025-04-14/trajs
trajs docent: false
warning: Unknown
FreeBench model match: No explicit identity match; source name retained.

Source position 164 · 2024-07-28

GPT-4o

23.2% · resolved

Published setup details

agent: SWE-agent
agent org: SWE-agent
checked: true
cost: Unknown
date: 2024-07-28
folder: 20240728_sweagent_gpt4o
instance calls: Unknown
instance cost: Unknown
logo: img/logos/20240728_sweagent_gpt4o.png
logs: s3://swe-bench-submissions/verified/20240728_sweagent_gpt4o/logs
model display: GPT-4o
model org: OpenAI
model release date: Unknown
name: SWE-agent + GPT 4o (2024-05-13)
os model: false
os system: true
reasoning effort: Unknown
resolved: 23.2
site: https://github.com/SWE-agent/SWE-agent
submission: SWE-agent + GPT 4o (2024-05-13)
tags: Model: gpt-4o-2024-05-13 · Org: SWE-agent · System: Attempts - 1
trajs: s3://swe-bench-submissions/verified/20240728_sweagent_gpt4o/trajs
trajs docent: false
warning: Unknown
FreeBench model match: No explicit identity match; source name retained.

Source position 165 · 2025-06-27

MCTS Refine 7B

23.2% · resolved

Published setup details

agent: MCTS-Refine-7B
agent org: Unknown
checked: false
cost: Unknown
date: 2025-06-27
folder: 20250627_agentless_MCTS-Refine-7B
instance calls: Unknown
instance cost: Unknown
logo: img/logos/20250627_agentless_MCTS-Refine-7B.png
logs: s3://swe-bench-submissions/verified/20250627_agentless_MCTS-Refine-7B/logs
model display: MCTS Refine 7B
model org: Unknown
model release date: Unknown
name: MCTS-Refine-7B
os model: true
os system: true
reasoning effort: Unknown
resolved: 23.2
site: https://mcts-refine.github.io/MCTS-Refine
submission: MCTS-Refine-7B
tags: Model: MCTS-Refine-7B · System: Attempts - 1
trajs: s3://swe-bench-submissions/verified/20250627_agentless_MCTS-Refine-7B/trajs
trajs docent: false
warning: Unknown
FreeBench model match: No explicit identity match; source name retained.

Source position 166 · 2024-04-02

GPT-4 (1106)

22.4% · resolved

Published setup details

agent: SWE-agent
agent org: SWE-agent
checked: true
cost: Unknown
date: 2024-04-02
folder: 20240402_sweagent_gpt4
instance calls: Unknown
instance cost: Unknown
logo: img/logos/20240402_sweagent_gpt4.png
logs: s3://swe-bench-submissions/verified/20240402_sweagent_gpt4/logs
model display: GPT-4 (1106)
model org: OpenAI
model release date: Unknown
name: SWE-agent + GPT 4 (1106)
os model: false
os system: true
reasoning effort: Unknown
resolved: 22.4
site: https://github.com/SWE-agent/SWE-agent
submission: SWE-agent + GPT 4 (1106)
tags: Model: gpt-4-1106-preview · Org: SWE-agent · System: Attempts - 1
trajs: s3://swe-bench-submissions/verified/20240402_sweagent_gpt4/trajs
trajs docent: false
warning: Unknown
FreeBench model match: No explicit identity match; source name retained.

Source position 167 · 2025-07-20

GPT 4o

21.6% · resolved

Published setup details

agent: mini-SWE-agent
agent org: SWE-agent
checked: true
cost: 765.5740625000001
date: 2025-07-20
folder: 20250720_mini-v0.0.0-gpt-4o-2024-11-20
instance calls: 63.066
instance cost: 1.531148125
logo: img/logos/20250720_mini-v0.0.0-gpt-4o-2024-11-20.svg · img/mini-icon.svg
logs: s3://swe-bench-submissions/bash-only/20250720_mini-v0.0.0-gpt-4o-2024-11-20/logs
mini-swe-agent version: 0.0.0
model display: GPT 4o
model org: OpenAI
model release date: 20241120
name: GPT 4o (2024-11-20)
os model: false
os system: true
reasoning effort: Unknown
resolved: 21.62
site: https://platform.openai.com/docs/models/gpt-4o
submission: GPT 4o (2024-11-20)
tags: Model: gpt-4o-20241120 · Org: OpenAI · System: Attempts - 1 · Mini: 0.0.0
trajs: s3://swe-bench-submissions/bash-only/20250720_mini-v0.0.0-gpt-4o-2024-11-20/trajs
trajs docent: false
warning: Unknown
FreeBench model match: No explicit identity match; source name retained.

Source position 168 · 2025-07-20

Llama 4 Maverick Instruct

21.0% · resolved

Published setup details

agent: mini-SWE-agent
agent org: SWE-agent
checked: true
cost: 156.85304879999998
date: 2025-07-20
folder: 20250720_mini-v0.0.0-Llama-4-Maverick-17B-Instruct
instance calls: 48.076
instance cost: 0.31370609759999996
logo: img/logos/20250720_mini-v0.0.0-Llama-4-Maverick-17B-Instruct.svg · img/mini-icon.svg
logs: s3://swe-bench-submissions/bash-only/20250720_mini-v0.0.0-Llama-4-Maverick-17B-Instruct/logs
mini-swe-agent version: 0.0.0
model display: Llama 4 Maverick Instruct
model org: Meta
model release date: 20250405
name: Llama 4 Maverick Instruct
os model: false
os system: true
reasoning effort: Unknown
resolved: 21.04
site: https://www.together.ai/models/llama-4-maverick
submission: Llama 4 Maverick Instruct
tags: Model: llama-4-maverick-instruct · Org: Meta · System: Attempts - 1 · Mini: 0.0.0
trajs: s3://swe-bench-submissions/bash-only/20250720_mini-v0.0.0-Llama-4-Maverick-17B-Instruct/trajs
trajs docent: false
warning: Unknown
FreeBench model match: No explicit identity match; source name retained.

Source position 169 · 2024-10-02

Lingma SWE-GPT 7b (v0925)

18.2% · resolved

Published setup details

agent: Lingma Agent
agent org: Alibaba
checked: false
cost: Unknown
date: 2024-10-02
folder: 20241002_lingma-agent_lingma-swe-gpt-7b
instance calls: Unknown
instance cost: Unknown
logo: img/logos/20241002_lingma-agent_lingma-swe-gpt-7b.jpg
logs: s3://swe-bench-submissions/verified/20241002_lingma-agent_lingma-swe-gpt-7b/logs
model display: Lingma SWE-GPT 7b (v0925)
model org: OpenAI
model release date: Unknown
name: Lingma Agent + Lingma SWE-GPT 7b (v0925)
os model: false
os system: true
reasoning effort: Unknown
resolved: 18.2
site: https://www.modelscope.cn/models/yingwei/Lingma-SWE-GPT (https://www.modelscope.cn/models/yingwei/Lingma-SWE-GPT-v20240925)
submission: Lingma Agent + Lingma SWE-GPT 7b (v0925)
tags: Org: Alibaba · System: Attempts - 2+
trajs: s3://swe-bench-submissions/verified/20241002_lingma-agent_lingma-swe-gpt-7b/trajs
trajs docent: false
warning: Unknown
FreeBench model match: No explicit identity match; source name retained.

Source position 170 · 2024-04-02

Claude 3 Opus

15.8% · resolved

Published setup details

agent: SWE-agent
agent org: SWE-agent
checked: true
cost: Unknown
date: 2024-04-02
folder: 20240402_sweagent_claude3opus
instance calls: Unknown
instance cost: Unknown
logo: img/logos/20240402_sweagent_claude3opus.png
logs: s3://swe-bench-submissions/verified/20240402_sweagent_claude3opus/logs
model display: Claude 3 Opus
model org: Anthropic
model release date: Unknown
name: SWE-agent + Claude 3 Opus
os model: false
os system: true
reasoning effort: Unknown
resolved: 15.8
site: https://github.com/SWE-agent/SWE-agent
submission: SWE-agent + Claude 3 Opus
tags: Model: claude-3-opus-20240229 · Org: SWE-agent · System: Attempts - 1
trajs: s3://swe-bench-submissions/verified/20240402_sweagent_claude3opus/trajs
trajs docent: false
warning: Unknown
FreeBench model match: No explicit identity match; source name retained.

Source position 171 · 2025-07-26

Gemini 2.0 Flash

13.5% · resolved

Published setup details

agent: mini-SWE-agent
agent org: SWE-agent
checked: true
cost: 0
date: 2025-07-26
folder: 20250726_mini-v1.0.0_gemini-2.0-flash
instance calls: 0
instance cost: 0
logo: img/logos/20250726_mini-v1.0.0_gemini-2.0-flash.svg · img/mini-icon.svg
logs: s3://swe-bench-submissions/bash-only/20250726_mini-v1.0.0_gemini-2.0-flash/logs
mini-swe-agent version: 0.0.0
model display: Gemini 2.0 Flash
model org: Google DeepMind
model release date: 20250205
name: Gemini 2.0 Flash
os model: false
os system: true
reasoning effort: Unknown
resolved: 13.52
site: https://ai.google.dev/gemini-api/docs/models
submission: Gemini 2.0 Flash
tags: Model: gemini-2.0-flash · Org: Google DeepMind · System: Attempts - 1 · Mini: 0.0.0
trajs: s3://swe-bench-submissions/bash-only/20250726_mini-v1.0.0_gemini-2.0-flash/trajs
trajs docent: false
warning: Unknown
FreeBench model match: No explicit identity match; source name retained.

Source position 172 · 2024-09-18

Lingma SWE-GPT 7b (v0918)

10.2% · resolved

Published setup details

agent: Lingma Agent
agent org: Alibaba
checked: false
cost: Unknown
date: 2024-09-18
folder: 20240918_lingma-agent_lingma-swe-gpt-7b
instance calls: Unknown
instance cost: Unknown
logo: img/logos/20240918_lingma-agent_lingma-swe-gpt-7b.jpg
logs: s3://swe-bench-submissions/verified/20240918_lingma-agent_lingma-swe-gpt-7b/logs
model display: Lingma SWE-GPT 7b (v0918)
model org: OpenAI
model release date: Unknown
name: Lingma Agent + Lingma SWE-GPT 7b (v0918)
os model: false
os system: true
reasoning effort: Unknown
resolved: 10.2
site: https://www.modelscope.cn/models/yingwei/Lingma-SWE-GPT
submission: Lingma Agent + Lingma SWE-GPT 7b (v0918)
tags: Org: Alibaba · System: Attempts - 2+
trajs: s3://swe-bench-submissions/verified/20240918_lingma-agent_lingma-swe-gpt-7b/trajs
trajs docent: false
warning: Unknown
FreeBench model match: No explicit identity match; source name retained.

Source position 173 · 2025-07-20

Llama 4 Scout Instruct

9.1% · resolved

Published setup details

agent: mini-SWE-agent
agent org: SWE-agent
checked: true
cost: 57.8157615
date: 2025-07-20
folder: 20250720_mini-v0.0.0-Llama-4-Scout-17B-Instruct
instance calls: 29.216
instance cost: 0.115631523
logo: img/logos/20250720_mini-v0.0.0-Llama-4-Scout-17B-Instruct.svg · img/mini-icon.svg
logs: s3://swe-bench-submissions/bash-only/20250720_mini-v0.0.0-Llama-4-Scout-17B-Instruct/logs
mini-swe-agent version: 0.0.0
model display: Llama 4 Scout Instruct
model org: Meta
model release date: 20250405
name: Llama 4 Scout Instruct
os model: false
os system: true
reasoning effort: Unknown
resolved: 9.06
site: https://www.together.ai/models/llama-4-scout
submission: Llama 4 Scout Instruct
tags: Model: llama-4-scout-instruct · Org: Meta · System: Attempts - 1 · Mini: 0.0.0
trajs: s3://swe-bench-submissions/bash-only/20250720_mini-v0.0.0-Llama-4-Scout-17B-Instruct/trajs
trajs docent: false
warning: Unknown
FreeBench model match: No explicit identity match; source name retained.

Source position 174 · 2025-08-03

Qwen2.5-Coder 32B Instruct

9.0% · resolved

Published setup details

agent: mini-SWE-agent
agent org: SWE-agent
checked: true
cost: 34.05807032999999
date: 2025-08-03
folder: 20250803_mini-v1.0.0_qwen2-5-coder-32b-instruct
instance calls: 48.206
instance cost: 0.06811614065999999
logo: img/logos/20250803_mini-v1.0.0_qwen2-5-coder-32b-instruct.png · img/mini-icon.svg
logs: s3://swe-bench-submissions/bash-only/20250803_mini-v1.0.0_qwen2-5-coder-32b-instruct/
mini-swe-agent version: 1.0.0
model display: Qwen2.5-Coder 32B Instruct
model org: Qwen
model release date: 20241112
name: Qwen2.5-Coder 32B Instruct
os model: true
os system: true
reasoning effort: Unknown
resolved: 9
site: https://qwenlm.github.io/blog/qwen2.5-coder/
submission: Qwen2.5-Coder 32B Instruct
tags: Model: Qwen2.5-Coder-32B-Instruct · Org: Qwen · System: Attempts - 1 · Mini: 1.0.0
trajs: s3://swe-bench-submissions/bash-only/20250803_mini-v1.0.0_qwen2-5-coder-32b-instruct/
trajs docent: https://docent.transluce.org/dashboard/f5288cc2-48b7-4df9-8e30-072d96b58606
warning: Unknown
FreeBench model match: No explicit identity match; source name retained.

Source position 175 · 2024-04-02

Claude 3 Opus

7.0% · resolved

Published setup details

agent: RAG baseline
agent org: Unknown
checked: true
cost: Unknown
date: 2024-04-02
folder: 20240402_rag_claude3opus
instance calls: Unknown
instance cost: Unknown
logo: img/logos/20240402_rag_claude3opus.png
logs: s3://swe-bench-submissions/verified/20240402_rag_claude3opus/logs
model display: Claude 3 Opus
model org: Anthropic
model release date: Unknown
name: RAG + Claude 3 Opus
os model: false
os system: true
reasoning effort: Unknown
resolved: 7
site: https://github.com/SWE-bench/SWE-bench/tree/main/swebench/inference
submission: RAG + Claude 3 Opus
tags: Model: claude-3-opus-20240229 · System: Attempts - 2+
trajs: Unknown
trajs docent: false
warning: Unknown
FreeBench model match: No explicit identity match; source name retained.

Source position 176 · 2023-10-10

Claude 2

4.4% · resolved

Published setup details

agent: RAG baseline
agent org: Unknown
checked: true
cost: Unknown
date: 2023-10-10
folder: 20231010_rag_claude2
instance calls: Unknown
instance cost: Unknown
logo: img/logos/20231010_rag_claude2.png
logs: s3://swe-bench-submissions/verified/20231010_rag_claude2/logs
model display: Claude 2
model org: Anthropic
model release date: Unknown
name: RAG + Claude 2
os model: false
os system: true
reasoning effort: Unknown
resolved: 4.4
site: Unknown
submission: RAG + Claude 2
tags: Model: claude-2 · System: Attempts - 1
trajs: Unknown
trajs docent: false
warning: Unknown
FreeBench model match: No explicit identity match; source name retained.

Source position 177 · 2024-04-02

GPT-4 (1106)

2.8% · resolved

Published setup details

agent: RAG baseline
agent org: Unknown
checked: true
cost: Unknown
date: 2024-04-02
folder: 20240402_rag_gpt4
instance calls: Unknown
instance cost: Unknown
logo: img/logos/20240402_rag_gpt4.png
logs: s3://swe-bench-submissions/verified/20240402_rag_gpt4/logs
model display: GPT-4 (1106)
model org: OpenAI
model release date: Unknown
name: RAG + GPT 4 (1106)
os model: false
os system: true
reasoning effort: Unknown
resolved: 2.8
site: Unknown
submission: RAG + GPT 4 (1106)
tags: Model: gpt-4-1106-preview · System: Attempts - 2+
trajs: Unknown
trajs docent: false
warning: Unknown
FreeBench model match: No explicit identity match; source name retained.

Source position 178 · 2023-10-10

SWE-Llama 7B

1.4% · resolved

Published setup details

agent: RAG baseline
agent org: SWE-agent
checked: true
cost: Unknown
date: 2023-10-10
folder: 20231010_rag_swellama7b
instance calls: Unknown
instance cost: Unknown
logo: img/logos/20231010_rag_swellama7b.png
logs: s3://swe-bench-submissions/verified/20231010_rag_swellama7b/logs
model display: SWE-Llama 7B
model org: Meta
model release date: Unknown
name: RAG + SWE-Llama 7B
os model: false
os system: true
reasoning effort: Unknown
resolved: 1.4
site: Unknown
submission: RAG + SWE-Llama 7B
tags: Model: SWE-Llama · Org: SWE-agent · System: Attempts - 1
trajs: Unknown
trajs docent: false
warning: Unknown
FreeBench model match: No explicit identity match; source name retained.

Source position 179 · 2023-10-10

SWE-Llama 13B

1.2% · resolved

Published setup details

agent: RAG baseline
agent org: SWE-agent
checked: true
cost: Unknown
date: 2023-10-10
folder: 20231010_rag_swellama13b
instance calls: Unknown
instance cost: Unknown
logo: img/logos/20231010_rag_swellama13b.png
logs: s3://swe-bench-submissions/verified/20231010_rag_swellama13b/logs
model display: SWE-Llama 13B
model org: Meta
model release date: Unknown
name: RAG + SWE-Llama 13B
os model: false
os system: true
reasoning effort: Unknown
resolved: 1.2
site: Unknown
submission: RAG + SWE-Llama 13B
tags: Org: SWE-agent · System: Attempts - 1
trajs: Unknown
trajs docent: false
warning: Unknown
FreeBench model match: No explicit identity match; source name retained.

Source position 180 · 2023-10-10

GPT-3.5

0.4% · resolved

Published setup details

agent: RAG baseline
agent org: SWE-agent
checked: true
cost: Unknown
date: 2023-10-10
folder: 20231010_rag_gpt35
instance calls: Unknown
instance cost: Unknown
logo: img/logos/20231010_rag_gpt35.png
logs: s3://swe-bench-submissions/verified/20231010_rag_gpt35/logs
model display: GPT-3.5
model org: OpenAI
model release date: Unknown
name: RAG + ChatGPT 3.5
os model: false
os system: true
reasoning effort: Unknown
resolved: 0.4
site: Unknown
submission: RAG + ChatGPT 3.5
tags: Org: SWE-agent · System: Attempts - 1
trajs: Unknown
trajs docent: false
warning: Unknown
FreeBench model match: No explicit identity match; source name retained.