arxiv:2603.24755

SlopCodeBench: Benchmarking How Coding Agents Degrade Over Long-Horizon Iterative Tasks

Published on Mar 25

· Submitted by

Gabe Orlanski on Mar 27

University of Wisconsin - Madison

Upvote

Authors:

Gabriel Orlanski ,

Albert Ge ,

Abstract

Software development is iterative, yet agentic coding benchmarks overwhelmingly evaluate single-shot solutions against complete specifications. Code can pass the test suite but become progressively harder to extend. Recent iterative benchmarks attempt to close this gap, but constrain the agent's design decisions too tightly to faithfully measure how code quality shapes future extensions. We introduce SlopCodeBench, a language-agnostic benchmark comprising 20 problems and 93 checkpoints, in which agents repeatedly extend their own prior solutions under evolving specifications that force architectural decisions without prescribing internal structure. We track two trajectory-level quality signals: verbosity, the fraction of redundant or duplicated code, and structural erosion, the share of complexity mass concentrated in high-complexity functions. No agent solves any problem end-to-end across 11 models; the highest checkpoint solve rate is 17.2%. Quality degrades steadily: erosion rises in 80% of trajectories and verbosity in 89.8%. Against 48 open-source Python repositories, agent code is 2.2x more verbose and markedly more eroded. Tracking 20 of those repositories over time shows that human code stays flat, while agent code deteriorates with each iteration. A prompt-intervention study shows that initial quality can be improved, but it does not halt degradation. These results demonstrate that pass-rate benchmarks systematically undermeasure extension robustness, and that current agents lack the design discipline iterative software development demands.

Generated by Qwen/Qwen2.5-Coder-32B-Instruct

View arXiv page View PDF Project page GitHub 183 Add to collection

Community

gabeorlanski

Paper author Paper submitter Mar 27

Coding benchmarks tell you if an agent passed the tests. They don't tell you if the code is any good, or if it's going to be workable three features from now. SlopCodeBench makes agents extend their own solutions across multiple steps without prescribing structure. They all fall apart. No model finishes any problem end-to-end. Duplication and complexity blow up nearly every time. Agent code is 2.2x more verbose than comparable open-source projects and degrades way faster than human code does over time. The benchmark ships with 20 problems but we want more. If you write multi-step coding problems or have ideas for ones, we want them.

librarian-bot

Mar 28

This is an automated message from the Librarian Bot. I found the following papers similar to this paper.

The following papers were recommended by the Semantic Scholar API

Please give a thumbs up to this comment if you found it helpful!

If you want recommendations for any Paper on Hugging Face checkout this Space

You can directly ask Librarian Bot for paper recommendations by tagging it in a comment: @librarian-bot recommend

weathon

5 days ago

•

edited 5 days ago

Interesting benchmark, but I think the main conclusion (agents write low-quality code) is not well supported by the setup. A few points:

Between sessions, there is no handoff doc or harness-level memory, so even for human it is hard for them to understand and improve a fresh codebase. The baseline prompt tells the agent "Implement a program that 100% solves the specification. That is all you need to do." At C1 there is no signal that later checkpoints exist. This also does not imply to the model a note/handoff is needed.
The only pressure for the model is correctness (even pushed to 100% in the prompt). That combination naturally pushes toward defensive, edge-case-heavy code. Without tests, model also cannot know what is enough and what other cases there is. Coding without testing also does not mimic real world coding agent usage. Even with anti-slop prompt, the model is still being pushed to finish the spec sheet and it only shows how the model acts at somewhat conflicting prompt.
The quality metrics have two separate problems. First, I doubt that these hard-coded rules can actually capture verbosity and erosion. Secondly, even if they do capture these things, the verbosity might not be a bad thing, they are argubly just a style choice (e.g., list comprehension vs filter).
And lastly, a lot of these make the code taste worse, but not really harming the accuracy. There are many SKILLs out there that can help you refactor your code, just like human code needs refactoring as well. Humans do that by themselves on the way; maybe agents don't: they just want to finish the current task asap (as the prompt stated). The sampled commits from human repo where spanning long time frame, which makes it almost ceterinly a refactor or rewrite happened somewhere.

I think the benchmark is interesting and worth noticing, but I think it tells us more about how to use these tools, instead of how good are these tools.

Also,

problems that did not meaningfully test design decisions, or that frontier agents could solve in a single shot, were removed from the pool.

This is a bit weird, it is like you filtered out the easy questions, and then claiming that they cannot solve them. If you filter out the questions agent cannot solve in one shot, it means they will keep refining it and make it bolt.