MemSWE
The object of study is the memory, not the model. MemSWE is a software-maintenance task benchmark that isolates the memory condition and holds the runtime, model, tools, and evaluation constant.
- My role
- Thesis co-author with Fernando Ramos
- Status
- Research
0 of 120 passed. The harder tasks failed every check the agent could not see.
15 tasks across, six memory conditions down, four repetitions in each block. 66.7% is an exploratory score, not a leaderboard: equal results establish neither equivalence nor a winner.
The test cannot change the test
Contract and execution stay separate: memswe defines what is evaluated, and pi-memswe implements how it runs under a fixed runtime, so harness changes cannot alter the benchmark's identity.
memswe, the contract
- Task descriptors and fixtures
- Sessions, facts and scope
- Schemas, scoring and measurement
- Verifier declarations
pi-memswe, the fixed runtime
- Orchestrates the conditions
- Runs the coding agent
- Captures traces and verifiers
- Emits run records
- Validate
- Isolate
- Execute
- Collect
- Clean
- Evaluate
What we learned
Result
The specify, run, verify, aggregate cycle completed across 360 executions.
Limit
The first matrix was too easy, and the hard tasks failed the hidden verifier uniformly.
Next
Build intermediate difficulty, and report each verification level separately.
Who did what
What I owned
Thesis collaboration
Withheld: detailed individual technical ownership requires Lalo's confirmation.
What the team owned
Benchmark contract
Withheld: task, fixture, verifier, schema, and policy ownership is not approved for individual attribution.
Built with
Bun, YAML task descriptors, JSON Schema, Python verifiers
Receipts
| Source | Kind | Verified |
|---|---|---|
| MemSWE benchmark contract README | Source not public | 2026-08-13 |
| Public project site | live product | 2026-08-20 |
Can’t trust your agent yet?