01
Functional correctness
Software & terminal
DeepSWE
Repository-level tasks for feature implementation and software repair.
Benchmark / task domain
The work
Complete software work in real codebases.
Agents inspect an existing codebase, understand the requested behavior and deliver working changes. Task coverage and evaluation requirements are scoped to the target software-engineering capability.
Delivery can include
- 01
- Harbor-format task packages with Markdown instructions and task.toml configuration
- 02
- Docker environment definitions and initial repository state
- 03
- Reference solutions and executable evaluation code
How we evaluate the task
Define how to evaluate the result.
02
System behavior and integration
03
Completeness of task outputs
WORK WITH ASTRAFLUX
What should your
model be able to do?
Share the capability, task types and delivery requirements. We’ll discuss the right program and representative tasks.
Contact us norman383@astraflux.spaceBenchmark names indicate reference directions. Task scope and delivery specifications are agreed for each project.