CursorBench vs SWE-bench vs HumanEval: What Each Benchmark Actually Tests

Cluster: CursorBench 3.2 analysis · Open models comparison · Best model by work mode

What do CursorBench, SWE-bench, and HumanEval measure?

CursorBench scores Cursor agent sessions on ambiguous, multi-file tasks and reports cost, tokens, and steps per task. SWE-bench scores repository patch success on real GitHub issues. HumanEval scores single-function Python completion from a docstring.

Who it is for: Anyone comparing AI vendor scorecards — a founder reading a launch post, a student picking tools, an executive approving spend — who needs to know which number answers which question before a default locks in.

What you will learn: a side-by-side harness map in plain English, what each benchmark omits, and a reading checklist so launch headlines do not override session cost and real-world fit.

AI tool decisions now land in budgets and buying decisions, not only in model pickers. The same week Cursor publishes eval pages, vendors ship SWE-bench and HumanEval rows in launch posts. The scores look comparable. The tests are not.

Approving a default from the wrong column is how a model that wins a patch benchmark becomes a ~17 USD-per-task habit in an agent UI it never ran in — with no line item that explains why.

When you set team defaults or approve model spend, start with the benchmark that reports steps and dollars. Patch scores alone will not predict your monthly bill.

SWE-bench and HumanEval answer programming-task strength on vendor harnesses. CursorBench answers what multi-step agent work costs inside Cursor — the line item steering committees actually need.

Why benchmark choice changes the model you pick

Optimizing for HumanEval does not predict agent step count. Optimizing for SWE-bench does not predict Cursor cost per task.

Harness comparison table

CursorBench: session economics

CursorBench 3.2 draws tasks from real agent sessions: codebase understanding, bugfinding, planning, code review, plus instruction following and advanced tool use (new in 3.2), and edit/refactor/bugfix work from earlier versions.

It reports four numbers together:

1. Task success rate (score)
2. Cost per task
3. Tokens per task
4. Steps per task

That quartet is why Fable 5 vs Composer 2.5 analysis belongs on CursorBench, not on SWE-bench alone. Example row: Fable 5 Max 70.5% at $17.32 and 72 steps versus Composer 2.5 56.1% at $0.44 and 33 steps (benchmarked numbers from CursorBench 3.2). Grok 4.5 High* sits at 66.7% / $1.51 / 33 steps with Cursor's training-data caveat.

Caveat: Cursor documents variance; treat small gaps as directional.

SWE-bench: patch success on real repos

SWE-bench evaluates whether a model (with an agent scaffold) can produce a patch that makes a real repository's tests pass for a recorded issue.

Strengths:

• End-to-end software engineering signal
• Real dependencies, real test failures, real merge constraints

Gaps for Cursor users:

• Harness is not Cursor's agent loop
• Tables rarely include dollars per issue or agent steps
• Vendor posts often cite SWE-bench Pro or Multilingual subsets that are not comparable across press releases without reading the split

Open-model posts (for example LongCat 2.0 evaluations) lean on SWE-bench rows. Those rows are useful for coding model quality. They do not replace CursorBench for picker defaults.

HumanEval: function completion baseline

HumanEval tests single-function Python synthesis from a docstring. It is a classic codegen benchmark.

Strengths:

• Cheap to run, easy to compare across years of models
• Isolates raw coding from tooling

Gaps:

• No repository navigation
• No multi-file refactors
• No agent stop/retry behavior

A model can score well on HumanEval and still burn 70+ agent steps on CursorBench tasks.

How vendor tables mix sources (read carefully)

Launch posts often combine:

• In-house SWE-bench runs
• External numbers marked with asterisks
• Chat benchmarks (BrowseComp, GPQA, etc.)

Before you merge a vendor row with CursorBench:

1. Check if the score is in-house or cited external
2. Check which SWE-bench split (Verified, Pro, Multilingual)
3. Check whether the agent scaffold matches your IDE
4. Check if LongCat / GLM / Kimi / Grok even appear in CursorBench yet (open models note)

Practical reading checklist

Limitations

• Benchmarks age quickly; models and prices change monthly.
• Pass rate does not measure human review time (yours or your team's).
• No public benchmark fully captures your private monorepo constraints.

Reader action

1. Bookmark Cursor evals for session economics.
2. When a launch post cites SWE-bench, screenshot the split name and agent scaffold footnote.
3. Pick IDE defaults from CursorBench; use SWE-bench to shortlist coding models for experiments.
4. Read best model by work mode for a Cursor-first map.