Enjoying this issue?
Get tomorrow's AI & engineering digest in your inbox — hand-picked, summarized, and always spam-free.
TLDR
Qodo's CEO argues that the real bottleneck in AI-assisted software development is not model capability but context: models can already reason well over code changes if given the right organizational context (rules, architecture, tribal knowledge). The path to trusting AI-generated code without human review requires codifying that context into a governance layer that both humans and agents can use, then gradually automating approval and blocking decisions. The company's goal is zero critical production outages by 2027.
Key points
- The speaker claims that model capability is no longer the barrier to automated code review; the barrier is context. Benchmarks for code review have not changed much across recent model releases.
- Organizational context is spread across documents, Slack/Teams messages, and developers' heads. Codifying this tribal knowledge is necessary for trustworthy automated review.
- Qodo's approach involves building a 'context engine' that collects rules, standards, and system architecture (including microservice contracts and historical outage data) and surfaces them during code review.
- The speaker distinguishes between two philosophies: teams that prioritize velocity and fix bugs in production, and teams that require every line to be reviewed. The chosen philosophy determines the tools and milestones needed.
- The speaker describes a future where code review shifts from reviewing individual PRs to reviewing a software graph, where PRs are bubbles with potential contract conflicts across multiple concurrent changes.
- Automated approval and blocking should be introduced gradually, starting with semantic rules that learn from accepted/rejected reviews and production incidents.
- The speaker asserts that if AI-generated code is being shipped faster than humans can review it, the organization already has a problem that requires governance infrastructure.
- Qodo's vision is to reach zero critical outages and bugs in production by 2027 through codified organizational wisdom ('artificial wisdom').
Tools mentioned
Techniques
- context engine for code review
- software graph analysis for contract conflict detection
- gradual automation of PR approval and blocking via semantic rules
- codification of tribal knowledge for both human and agent interfaces
Stop scrolling. Start reading smarter.
Receive the day's most important AI & engineering updates in one concise email. No spam.
Transcript (captions)
Are you all set up with your AI factory? Everything is smooth as you're sitting here. Your code is being deployed. AR growing, right? Is if it's like that, raise your hand. No. Oh, okay. Two
people. Great. um let us know like before and after this talk what's ARR and and do you feel like there's a bottleneck a new bottleneck that is not on writing
code rather somewhere else else in the SDLC is that your biggest thing to tackle if if you are tackling that right now okay so you're in the right place and what about like code review
verifying that the code work according to your intent according to your architecture standards, best practices, etc. Is this something that you're tackling dayto day or week by week?
Okay, so you're you're in the right place. So, I'm Edomar Friedman, the CEO and co-founder of Cotto. Um, I don't share it too much, but Quotto stands for quality of development optimization. Our
mission and and uh is to help you all have a code governance code review platform that understand your codebase, your tribal knowledge, your best practices and that does not come off the
shelf from a model. It requires a system that optimized for you as we go and that's uh why we decided to name our company uh this way. Uh so uh I'm going to talk about the last human code review
and what do we need to do in order to get there. Okay. And um c can you switch instead of seeing myself seeing the presentation here? Thank you. Um so I I think like first of all we need to agree
on why do we have code review at all. So I think basically these are two buckets that we should agree on and I think I think it's quite common. One is we want to validate the code that is in high
quality, safe, maintainable, the right architecture according to our best practice etc. The second re reason is actually alignment and learning right like where senior developers for example
has one last chance a gateway gatekeeper before uh it's being a code is being pushed to production to have that alignment and teaching right so these are the two reasons and then that's what
you need to think if you're trying to automate a code review can human still do that and this if this is the right place the the code review process the pull pull request process is that the
right place to still doing these two things. If you have uh new tools, new processes that will help you unblock this bottleneck but let you still do these two buckets of tasks then you're
on the right path. Uh so we are here to ask yes no is human code review still optional end of 2026 is it becoming optional or is it still mandatory? Okay that's that we're here
to answer. Do you think like every PR or the majority of PRs are going to be reviewed line by line or you know bucket by bucket in the in the chunks and the hunks by your developers? Raise your
hand. Do you think do you think by end of this year your developers are still going to review diff by diff? Okay. So try to think why why is that happening? Now I I I wanted to share
with you that yesterday night we did a drone show and I had to uh opportunity to talk to different people during that drone different uh engineering leaders and what I can tell you is that I saw
two very different groups of school of thoughts. Okay, they both agree that bugs are coming in different shapes. For some this is fine and we're just going to fix
that quickly after it hits to production and for some not. So, so actually we do see two teams. Those that are thinking about the room is split into two. The other are thinking like hey we have to
let make sure that every piece of line is is trusted uh and and the humans must review that or the other group somewhat reckless or so saying let's let's like push those bugs into production and we
quickly fix that and that's how we actually do things because it's much faster. velocity is more important than getting getting things uh right and I think like you need to think like where
do you sit of course I put it in two two sides of the spectrum but there's some somewhere in between and you you need to think what what's your philosophy because that will lead you to different
milestones or different tools that you need to use in order to get that uh uh confidence that you can skip over a a human review in the pull request and the code you. So let's start talking about
what is the process, how do we need to start thinking about it. So I claim that models are not not a barrier anymore like it's not a matter of of the you know a model doing a good good job or
not having the right reasoning or not. Basically the models are improving but I'm telling you I just came from one of the leading labs where we are inspecting how benchmarks for code review did not
change a lot throughout the latest model. The the key here is actually context. Okay, like the models if you give them the right context and what is that right context we're going to talk
about they could already reason pretty well over what is the issues that we what are the issues that we need to surface for a certain change in the code otherwise if you don't have the context
even the best model out there they will give you different types of of uh bugs and issues some of them are really good but in many cases they will simply tell you hey did you consider error handling
or like or not. By the way, error handling could be like a really good thing to handle depends. In some cases, it's critical, in some cases are not. And again, the context is what what
matters. Right now our context is like spread all across like we have agents MDs, cloud MDs, skills MDs and and the thing is that each each one of them has like different standards uh different or
organization and suborganization are dealing with different differently even within a certain team. you might be using that that differently and you're actually maybe using like the same uh
one one team is using uh the same agent to do code review and sorry co coding and code review the other might be using something else and all of that does not bring you the trust and consistency that
you're looking uh towards um by the way like you also might have like coding agents that are running in your IDE but you're probably building the AI factory that running those agents like in
workflows to automate some of the coding. I see teams that are already uh having more lines of code being uh shipped that are not generated from the CLI or or the IDE. So how do you control
all that? like that's that's missing uh like in right now like in our in our uh tools in our infrastructure and let alone if you add those MCPS and and and rag like uh style uh context I I don't
know if you have like great visibility there are ways to tackle that there's great talks out there check how you could have like MCP versioning and have data sets for every like a benchmark for
every MCP change but that's hard to manage we're missing like a governance layer for us to move to the next level. Okay, like where we can actually trust the code without human reviewing it. So
where where is that context? Where is that context? Basically I would say experience tribal knowledge wisdom of your developers is a lot in their heads. There are in some documents
there are documents infrastructure documents. Um but a lot by the way a lot of them is are slacks or or teams or or so um the the data is there but a lot of the information are are in your
developer heads and we need like to to codify them. Now um I think like basically another thing that I'm saying like when I'm saying that we need to codify human knowledge
what we're actually saying is that we're trying to build an interface for agents an interface for humans to collaborate each other on that on that knowledge and that's a very important cont uh um a
very important point when you want to extract that information the tribal knowledge from your uh like senior developers etc. and codify that. Do you codify that only in agents language uh
which is very maybe verbos and structured or you want to codify that in a wiki style uh get started and all that what developers love love doing and the answer is that you probably need to
build your context lake your context engine as I mentioned that's the the the the real like gold mine here to to get the code review uh like auto automated you have to have it fitting for both So
what you're seeing here for example is that Kodto as an example but you can use other tools help you collect all the rules and standards that your team own that your team is using dayto-day and
then it will provide that information during the review for humans. Hey, notice that Cotto used four rules uh sorry used uh uh many rules and four are violated and that includes a link to all
the rules that are were being used that's for human in order to trust okay in order to trust the results that coming from your code review tool etc. You have to build that interface for for
human. You have to accumulate that knowledge and have an interface for human. But you also want to have an interface that is dedicated for agents. What you're seeing here uh for example
is a com comment for example by Cotto that is speaking to another agent. Hey dear agent Cotto just reviewed this uh PR and has found five different issues. Cotto already spend like some back
background task and use cloud code for example harness in order to do fixes and there is a closed PR like you can see here in the top right there are some closed PR with all the fixes and now
when an agent is coming to review this PR once again then it has like a cherrypicking moment uh with everything that all the code that is actually is passing your rules your standard uh and
everything that we're like more architectural decision that we're going to talk about. So what what I said so far is that just to to c like recap so far if you
want to trust and you want to get to a point where you're trusting the code is being shipped you need to have the right context that is being gathered and being used during the code review process.
Then you will see that this code review process have links and information for human and have links and information for for agents. Okay. And when that is in place, you will see that developers are
writing less and less comments in the pull request. And then after 100 of these pull requests or human no more human review, you know that you're ready for for automation. Okay. Now back to
back to the context. So far I talked about relatively simple context. I talked about uh rules and standards and skills but actual the human knowledge the the tribal knowledge in your
organization sits in understanding the system architecture. What are the P zeros the the bugs that actually made an outage outage for for you unfortunately in the last like three
months or so when a micros service uh one changed its contract and broke a microser 2. Right? that is does not exist in most code review and and like if you try to build yourself it's really
hard to build but it is available in some of the those code review uh solutions that is dedicated for that. For example, what you're seeing here is a graph being built for a certain
microser and all the repos and and their connection and in each node an edge there is what is the cont if it's an edge what is the contract between two uh piece of your software but also links to
history of discussions between developers that they had when they fixed an issue because of root code analysis and now you need to codify that. Okay. And when you get to to that level of a
context engine, now you're ready to start approving and blocking PRs automatically. And you want to do that not just by letting AI some like choose by yourself rather giving some semantic
rules that that for example when you when do you guys approve or or or block a PR and that knowledge also needs to be accumulated as part of your your context. What I'm actually saying is
that software development, at least code governance, is going to change from reviewing your pull request to actually reviewing your entire software development from a graph
obstruction where you're seeing your PRs as bubbles with all the issues that might uh happen even if three different PRs are in on the fly when they which which contract
they might bridge. they they might ruin. Okay. And that's how the software development uh uh future is going to look like. What you need to do in order to get there is to codify your
standards. It needs to be built in a way that humans can trust and audit and control. You need to build real time self-learning context. Learning from peer history, learning from accepted and
unaccept learning from discussions between between developers, learning between uh learning like from those cases that broke your production and that context needs to be not just like
thrown into files. It needs to to to to sit and located in a place that agent understand where is that context fitting and then you need the governance infrastructure that gives you that
visibility of what's happening rather the graph I presented but there is more to that okay I'm going to show you very soon another v visualization that helps you understand the overall status of
your PRs and software if you are already shipping AI generated code faster than your human can view. I'm actually saying that you are in the problem. You're not like ahead of the problem. You you if
you put the infrastructure, if you gather the context, if you start accumulating how code review can be automated for you, then that's where you're going to get the 10x velocity
that you're being promising your CEO or yourself or or your developers because otherwise it's a bottleneck. You need to own your rules and standards and codify them and get analytics and statistics
about each one of them. How many times they're being caught which rules and standards and skill is actually being used during their review process and is useful or not or does it need to get an
update? You need to have vis full visualization of your software graph. What is the connection? What are the contracts? What is working? What is not working? whether two PRs are going to
crash very soon because they're touching the same agent and they don't and they don't know and you need to start learning how to auto approve and autoblock that won't happen in like
immediately that needs to gradually being automated for you step by step by adding more rules for blocking and more rules for for for approving over time.
So trustworthy automated review your rules your standards in your software graph placed in the right edge in the right node of your software learning from the tribal knowledge of
discussions of history of your software and codifying that. There's so much tribal knowledge what to do and not to do and that needs to be codified as well both for the agents and for the humans
and basically you need to accumulate that experience and codify that and sorry for the buzz or whatever you want to call it that's moving from artificial intelligence to artificial wisdom
because right now you know better than I that your developer holds the judgment of what's bad and what's good. It's not your your software, not your AI tools. If you want to get to a point where
judgment is moving to your AI tools, that's AI wisdom where that experience needs to be codified on the right way, the right place for agents and humans. And that's why we are here COD to help
you. And our vision and mission is not far away from now. We want to reach in 2027 to a place where you have zero outages, zero bugs in production, at least the critical and
the high ones. That's why we're here. Thank you so much for having me.