swe-bench

Tagline Brand's website · 2026-10-04

Focus on real-world software engineering tasks from actual GitHub issues, the inclusion of a human-filtered 'Verified' dataset for high reliability, and a transparent leaderboard comparing a wide array of leading AI models and agents.

Description Brand's website · 2026-10-04

swe-bench is a benchmark for evaluating the performance of AI agents and models on real-world software engineering tasks. It provides a dataset of GitHub issues from open-source projects, including a human-filtered 'Verified' subset, to measure how well AI agents can resolve these issues. The platform serves as a leaderboard to compare various AI models and agents based on their resolution rates and associated costs.

Core capabilities Brand's website · 2026-10-04

• The swe-bench benchmark
• comprising datasets of real-world software engineering issues (e.g.
• Verified
• Multimodal
• Multilingual
• Full)
• and an online leaderboard for comparing AI agent performance.

Pricing Brand's website · 2026-10-04

The benchmark itself and access to its results appear to be open and free, serving as a public standard for evaluation. The 'Avg. $' column in the data refers to the estimated cost incurred by the *evaluated AI agents* to resolve tasks, not a pricing model for swe-bench.

Target industry Brand's website · 2026-10-04

AI researchers, AI model developers, software engineering teams leveraging AI, academic institutions, and companies developing autonomous agents.

Competitors Brand's website · 2026-10-04

• Hugging Face Leaderboard (for code generation/fix tasks)
• CodeXGLUE
• HumanEval
• Live Code Arena / CodeContests (from Google DeepMind)
• Other academic benchmarks for code intelligence