What Bun's full rewrite taught me
I’ve mentioned process-based quality control before, and you can actually see something close to it in How Anthropic runs large-scale code migrations with Claude Code | Claude by Anthropic. What they put at the center of how the whole rewrite ran is the “process”.
The core insight is that you don’t fix the code. You fix the process (loop) that produced the code.
Don’t focus on fixing the code, focus on fixing the process, the loop. You could say this bears out what I’m doing right now, or at least I’ve always thought this is the right direction for where things are going. As the amount of code swells, holding the quality of a million-line migration done inside a short window by human effort alone is basically impossible.
They split the migration into these steps.
First, set up verifiable and strict exit criteria, tests and discriminators. All of it is to make sure the code after the migration still matches what it was before, and mechanized or even formalized verifiability testing is the only way to do that. You may even have to build a separate test framework just for the migration.
Then settle on a rulebook you can iterate on, a dependency graph, and clearly defined targets. The key in this step is making sure no drift happens during the migration, especially in the critical architecture decisions and the dependency structure, and knowing how far you still are from the end. This step ties back to the first one.
Set up reviewer agents whose direct purpose is suspicion, adversarial review, which is the same as what I’ve been doing. Where it differs is that they set up two or more independent, crossing reviews, and when the two contradict each other a third reviewer comes out. This step is obviously extremely expensive in tokens, but adversarial pressure is the necessary guarantee of soundness outside of mechanized testing, and it’s the best way to imitate a human. How many to set up depends, I think, on how important and how large the project is, and on the budget.
Stress-test “the rules themselves”. I think this practice is worth learning from. Before a large migration they run a pilot, actually running the rules once and then watching for problems that might come up. There are two cases here though. One is testing in the shape of a structural migration, where the point is that the structure stays the same but the language changes, so the way you watch the rules is to run them and then see whether the structure held, and a person can check that. If it isn’t that kind of migration, the recommended practice is “adversarial review”, with e2e tests to back it up. I think the key here is still whether behavior and characteristics are fixed, because the typical case for the latter is that a migration may take on a large refactor along the way, so you can’t only watch whether the structure changed, which makes pinning behavior and characteristics with e2e tests critical. Unit tests, on the other hand, may be invalidated by the structure changing. The core point is to guarantee it through input and output (if there were side effects before, and the side effect happened to be a key function, then once that badly designed structure changes it will most likely break in e2e).
Then you actually start running the loop, the implement-review-fix cycle. The whole workflow here should be mechanized, purely defined and then set running, which is how you minimize the cost. And plenty of the tasks don’t need the best model, since a small model obviously has the advantage in speed and cost. Depending on the scale, the second half of this step can also be split out on its own, into build, smoke test, review, fix. The interesting thing they did here is build a daemon that runs tests in serialized batches, which keeps a pile of subagents from each running the full suite over and over and paying for it. It gathers the test requests first, settles them in one run, and sends the results back. For large software this really does matter, otherwise every subagent has to sit through thirty minutes of tests after it finishes writing, and the time cost gets stretched by an order of magnitude.
Through the whole thing they keep stressing “process”, meaning don’t over-focus on an individual failure, keep a process that runs soundly. Code review should be adversarial and verification should be mechanized, which also confirms what I said in on reading code, that checkable feedback is critical for quality control in agent development. Especially since current RL-dominated training keeps rewarding the agent for satisfying those verifiable results. That does lead to a problem, which is finishing by any means available, up to all sorts of moves that look finished on the surface but may wreck the overall task result, and that’s when you need the adversarial agent. For the latter, its task turns into checking, which is likewise a checkable goal. One builds, one attacks, and once the two are in a loop, the feedback balance can be established.
And in the end, our assessment and sign-off on the overall result isn’t done by “reading the code” either. For a large system rewrite, going through it line by line is basically impossible and not realistic. But the criterion for signing off can be very direct, which is “look at the end result of the loop running”. Working correctly is only the baseline. The core question is what my original purpose in doing the rewrite was. Bun’s purpose was memory leaks, so bringing the leaks down is the gold standard for whether the rewrite worked. If your purpose is performance, then how much performance went up says directly what this rewrite returned. So the criterion for signing off depends on the original purpose, but either way, making sure it works is the first prerequisite.
A million lines rewritten by one person in eleven days, at a speed that was hard to imagine before AI. That said, I think the safer approach may still be the strangler fig pattern, finishing the whole migration by replacing piece by piece and going live in slow steps. Bun does have an inexhaustible supply of tokens and the most advanced models, and an ordinary company doesn’t have their conditions, so doing a full rewrite too aggressively could lead to problems nobody saw coming.
The times have changed anyway. As the article says, a rewrite is at least no longer a decision about survival, which is clearly also a reminder aimed at decision makers. Even the same action can’t be looked at through the old eyes. If the return from a rewrite is high enough, then a rewrite should be the best choice. What’s left is only which model to use to do it safely and solidly.
Translation note. I wrote this in Chinese. This English version is an LLM translation, so the wording is not mine even though the thinking is. Original: 关于Bun在全量重构中的经验.