Software Engineering has changed. Again.

The craft of software engineering is not new to transitions. We went from punch cards to terminals, from terminals to IDEs and cloud, and many, many more. Now with the rise of AI and agents, we are finding ourselves in the middle of a new transition. One that is very much unlike anything previously seen.

In the pre-AI world, writing code was the bottleneck. It was expensive, time-consuming and required considerable effort to correct if something went wrong in the first place. That is why, in that world, good developers were the prize!

And it made sense. If you hired good developers, the quality of your output went up: reliable software with fewer bugs. That was the unsaid but accepted norm.

That is, however, no longer the case.

Agents produce code faster than teams can ship it

Just a few days ago, I came across a post by the cofounder of CodeRabbit saying that they "shipped 5k PRs last week", something that’s "more than a thousand-person dev team" and that done correctly, "50k PRs a week are totally achievable".

Stripe has a similar story. They merge 1300 PRs a week that contain no human-written code.

That’s a whole new world!

No one can review these thousands of PRs line by line. Because nobody reads them line by line.

The whole domain of code review was built for a world where people wrote code at the speed of people. One developer decided what to build and wrote code, another would read the diff and confirm that it did what it should.

Reading code was slow but writing it was slower. That’s why the read-the-PR-and-review-it process kept up.

Agents completely broke that balance. Code is now a commodity and is cheap to produce.

But even with code generation being solved, what has barely moved is the amount of shipped work.

In the study conducted by Harvard economists Fiona Chen and James Stratton (conducted on 725,938 engineers at 718 firms, from January 2021 to March 2026), it was found that while the lines of code produced went up by 30%, resolved Jira issues and epics showed a statistically insignificant increase.

The extra code went into "review" instead of going into production. Average review time per PR rose by 49%, comments per PR went up by 35%, the number of PRs with changes requested nearly doubled and 14% more engineers spent time reviewing.

And this is not the only study pointing to these trends. Faros AI saw a similar pattern across more than 10,000 developers on 1255 teams, and I can point out several other studies and surveys confirming this (that’s just a Claude session away!).

This has made something else the new bottleneck.

Line-by-line review has a ceiling

Human attention in code review has measured limits. SmartBear’s study on a Cisco team across 2500 reviews of 3.2 million lines of code over 10 months found that above 400 lines per review, reviewers found fewer defects. Above 500 lines, defect detection fell even more.

Take those limits and apply them to the agent volume.

With the assumption of a modest 3k PRs a week (CodeRabbit’s team does 5k PRs a week!), and an assumed 300 changed lines per PR, we have 900k lines of change. In a week.

At 400 lines per hour, the review would take 2250 hours (~56 engineers doing nothing else). And the 300-line figure is a conservative estimate. Faros measured PRs at about 2.5 times their earlier size, so the real number is probably higher. Plus, this is just review. Imagine what happens when changes are requested, then those changes are made, which are then reviewed again.

And, agent-generated code is also harder to approve on a quick read. CodeRabbit’s own report says that AI PRs had 10.83 issues each, compared to 6.45 for human PRs, with logic and correctness issues becoming 75% more common.

What this means is that even though you can spit out code instantly, you still have to verify that it is working as expected. Andrej Karpathy in his June 2025 talk at YC AI Startup School said "Even though the 1,000 lines come out instantly, I have to make sure that this thing is not introducing bugs."

All of this points to a new shift. More time is now spent in verifying that the code produced is correct.

But how do you do that at agent volume? Line-by-line review is clearly not working.

So where has the loop moved to?

From reading code to reading evidence

Writing code has scaled. But reviews just can’t. So teams have changed what the human checks.

Gone are the days of a human reading a diff to decide whether the code is correct. These days, you’re not even sure if whatever is supposed to be read is being read by a reviewer or yet another agent.

A person or an agent now reads evidence that the change behaves as intended: test results, traces, screenshots and failure history. If I were to represent this shift of bottleneck, I’d present it as the bottleneck moving from reading code to reading evidence:

In both loops, a human is deciding what to build. What has changed with agents is that now the slow part isn’t reading the diff, it is getting a result you can trust and acting on it. In fact, we can formally categorise it as three changes in the loop:

1. Human work has moved to the two ends. Before the change, someone has to describe what is to be done. After the change, someone has to decide what evidence is enough to merge.
2. Tests and CI are the main gateways to merging. Human review goes only to those changes where a mistake is expensive, things like authentication, payments, data migrations, infrastructure, to name a few.
3. And finally, evidence can be read by agents as well as humans. An agent can see why a test failed and can fix it without human intervention in between.

Removing the line-by-line review doesn’t remove the work of verification. In fact, it moves that work into CI, into the test suite, and into the four steps between a failure and a fix, each with its own limits:

1. Time to verdict:
   • Before agents: A developer pushed a few times a day and did other work while CI ran. A slow pipeline cost little, because the developer did not sit idle during the wait.
   • With agents: An agent can’t do anything useful until the result comes back. The time between a push and a pass or fail is wasted. Across thousands of agent PRs a week, that time wastage adds up.
   • What it takes: Track the time from a push to a pass or fail you can trust as the metric that limits throughput. Split the suite across machines, stop the run at the first failure, and rerun only the tests that failed.
2. Trust in tests:
   • Before agents: A developer who knew the suite could tell a flaky failure from a real one and move on.
   • With agents: Tests now decide what merges. So a test that passes and fails on the same code blocks good changes and hides real failures. An agent that hits a flaky test rewrites correct code and adds a retry or a timeout. It keeps doing this until the test passes. If you ask it to rerun until green, it reruns real regressions in the hope of a lucky pass.
   • What it takes: A record of which tests are flaky, available before the agent acts on a failure. A known-flaky failure and a new failure need different actions.
3. Evidence an agent can use:
   • Before agents: For a developer familiar with your codebase, a stack trace in CI logs is often enough.
   • With agents: An agent needs more context to fix a failing end-to-end test, and the usual sources give it too much or too little. A Playwright trace, for example, is an archive of several megabytes. If you paste the whole trace in your prompt, you waste tokens. If you paste only the stack trace, you get generic fixes.
   • What it takes: A record of every run that is kept, searchable, and small enough to fit in an agent’s context window. It must answer the questions a developer answers from experience:
     • Did this test fail before this change?
     • Does it fail only in CI?
     • Did other tests fail with the same error?
     • What did the page show at the moment of failure?
4. Hand-offs between the steps:
   • Before agents: A developer opened each failed run and decided what to do. At human volume, that was a few runs a day.
   • With agents: Each failure still needs an action: decide if it is real, file a ticket, quarantine a flaky test, notify the owner, or hand it to an agent. Each hand-off is small, but the count grows with the number of runs, and agents multiply the runs. If a human has to open each failed run, that becomes the new review bottleneck.
   • What it takes: Hand-off rules that run without a human. For example, send a known-flaky failure to quarantine, a new failure to its owner, and a failure with a clear cause to an agent.

And this leads us to what engineering teams can do today (without any additional tools!), to move a decision that earlier used to happen in review, to a place where it can keep up with the agent volume.

Five changes engineering teams can make today

1. Write down which changes still need a human review:
   • Start with authentication, payments, data migrations, infrastructure and public APIs.
   • Everything else merges on evidence.
   • A short, explicit list beats an unwritten rule that every pull request gets a careful read, because at agent volume that rule is already false.
2. Give the test suite an owner and targets:
   • Set a target for time to verdict and a ceiling for the flaky rate.
   • Treat a miss the way you treat a missed latency target on a production API: someone owns it and fixes it.
3. Measure the loop, not the output:
   • In the Harvard and Faros data, lines of code and PR counts went up while shipped work stayed flat.
   • Track time from push to verdict, CI rounds per agent task, flaky rate and rerun rate instead.
4. Quarantine flaky tests with an owner and an expiry date:
   • Google’s tooling quarantines a test automatically when its flakiness is too high and files a bug for the owner.
   • A quarantine with no expiry date turns into a permanent skip.
5. Review test changes separately from code changes:
   • Researchers from Carnegie Mellon and Anthropic built ImpossibleBench, a set of tasks where the tests contradict the spec, so any pass means the agent cheated (arXiv).
   • GPT-5 passed 54% of one such task set. Claude models cheated mostly by modifying the test cases.
   • If the agent that writes the code can also edit the tests, a green run proves nothing.
   • Require a human to approve changes to test files, even when the code itself merges on evidence.

Faster evidence does not fix a wrong spec

Everything above assumes that your tests accurately describe what you want. But some gaps remain even when the evidence is fast and trustworthy:

1. A vague spec. If nobody wrote down what the change must do, no test can confirm it. More of the senior engineer’s time goes to the start of the loop: acceptance criteria, clear tickets with accurate descriptions, and examples that an agent can turn into tests.
2. Missing coverage. Evidence covers only what the tests describe. A behaviour with no test, passes on every run. The more changes merge on evidence alone, the more each untested path matters.
3. Risk management. When engineers stop reading most diffs, they learn less about how their system works. With agents now managing incidents as well, what still remains critical enough to be understood by humans is risk management. So anything you deem critical should have a human review associated so as to manage the risk of it going haywire.

And I wanna emphasise that while it seems that the old ways are lost, I’d argue that not the entirety of it is true. We still need engineers. Probably more of them (Yes I’m aware of the latest models and their capabilities). Good wisdom asks us to never waste a crisis. And if a crisis is what this is, we should definitely not waste it. All we need to do is come to terms with what has changed.

And what has changed isn’t that software engineering is done and dusted, but that the value of humans in the software loop has shifted.

Shifted from writing and reviewing code to crafting accurate descriptions of what needs to be done, shifted from reading diffs and reviewing PRs to ensuring that the evidence matches, shifted from fixing failing tests to building systems that manage hand-offs without a human in most cases.

And while it may seem that this isn’t what software engineering was, the crude reality is that it is what software engineering is now, and done right, we can scale to tens of thousands of PRs a week from the paltry numbers a few months ago. Those teams that will make this leap will reap the benefits of the productivity increase (which is why we’ll need more engineers); those that don’t will become dinosaurs.

That is what the future holds.

And it is about to get very, very real!