When I started building Tough Crowd, I was just manually reviewing the code. I still didn’t have working parallel agents, so the amount of code I could generate was still pretty small.
However, once I got parallel agents working and stabilized, the amount of code increased substantially. To put it in perspective, I shipped 81 PRs in May. In June, I had some vacation time and still managed to ship 54 PRs. When I had the parallel agents working well in July, it rocketed to 501 PRs. I went from reading the code somewhat closely to mostly skimming. Then I noticed a lot of mistakes getting pushed to production.
The good, the bad, and the ugly with Greptile
I knew I had to do something, so I reached for Greptile back in July. It immediately started catching things I might have caught if I did a detailed code review. However, it was also catching legitimate issues that I’m not at all sure I would have caught.
All this was great and gave me confidence to not look so closely at the code. However, there are some things that I liked less. First, Greptile (and for that matter, the other competitors I looked at) really wants to provide findings in GitHub comments. But I wanted it to be a CI check that was pass fail so it would be clear the PR was broken and needed fixing and to make it easier for my agent to consume. Maybe they’ll all add this as an option one day. (Note: Greptile does have a CI check, but at the time of this writing, it doesn’t appear to pass/fail based on the review.)
But the much bigger issue was price. The bill for the first month of usage was $1,255! This is because the pricing for the plan I’m on is $1 per review. However, if you fix something, it triggers another review. Between July 13 and August 13, I did 623 PRs and it generated a total of 1,283 reviews, or 660 follow-up reviews. And I often merged if there was a minor finding still there that I cared less about. (Note: the pricing is slightly more complicated than $1/review - there’s also subscription pricing per user along with a number of included reviews).
In that time period, I was mainly using Codex as the agent and GPT-5.5 (79% of the time) and GPT-5.6 Sol (17% of the time). I never once hit my usage cap and therefore generated all that code for $200. This is a big imbalance to say the least!
Finally, there were still some obvious things that slipped through the cracks. The worst was an N+1 query in a PR whose final review was 5/5: “The PR appears safe to merge”.
Experimenting with a frontier model and a great prompt
So with that in mind, I decided to run an experiment. What if I just manually ran reviews on the same branches as Greptile using Sol High? I was hoping I could catch maybe 80% of what Greptile is catching. So I started off with an intentionally naive prompt (something equivalent to “Review this branch for problems”). Not surprisingly, it missed a lot of stuff that Greptile caught.
But then I cribbed a prompt from Addy Osmani’s agent-skills package. Within a few hours of running reviews using his code-review-and-quality skill with Sol High, I was finding more bugs and more serious bugs than I was with Greptile. Of course this is anecdotal and YMMV, but it’s at least a highly plausible approach and it’s definitely way cheaper.
In the near future, I want to systematically evaluate different models and prompts against the same changes, including past PRs, and publish what happens. If you’re interested in giving this a try, I’d love to hear from you.
Combining review sessions and autofix
I decided to automate the review process by implementing a Review Sessions feature in Tough Crowd. I am continuing to use the code-review-and-quality skill to find the issues. And I’ve got it outputting a CI check that fails when it finds issues. This allows me to use the autofix feature to have the coding session and review session go back and forth before I ever get involved.
So now my process is to fire off a coding session and then let the review session and the coding session duke it out. Most of the time they get to a conclusion after 2 or 3 autofixes. Every now and then, it can go a lot longer. If they are not converging after, say, 7 or 8 times, I’ve found that it often makes sense to have the coding agent take a step back and redesign the problematic area rather than continuing to go back and forth.
After the AI reviews are done, I quickly skim the code if it’s of consequence. For example, a schema change or a change in a complicated area like the session state machine. If it’s a change to a feature that’s released to users, I will also watch the UI tests run to make sure everything looks good. If it’s behind a feature flag, I just rely on the automated checks and then see how it works in production once it’s there.
I’m pretty happy with my current setup. It’s supporting my current volume of work handily and, so far, hasn’t maxed out my ChatGPT subscription. Also, I haven’t seen it miss anything huge yet, knock on wood. But I’m eager to learn more about what it catches and what it misses, so I’ll be digging into that in the near future.