AI Writes More PRs. Who Validates Them? — Ali-Reza Adl-Tabatabai, Sonar

0:13 Let's get started. Um, good afternoon everyone. I'm Ali and um, I'm one of the founders of guitar.ai AI and previously the CEO up until about a month ago where we were

0:28 acquired by Sonar and now we are part of Sonar and you've heard a lot in the past few days about the importance of verification in the age of AI and so

0:40 today I'm going to focus my talk on code and verifying code. So in in almost all software teams there is this centralized validation phase that is basically CI plus code review

0:53 and as you scale your engineering this plays this phase plays a very critical part especially in an AI world. Uh first it enforces all your quality gates centrally. So these are security

1:07 compliance code quality gates. It's on the critical path of getting your changes into production. So the faster is always the better. But as you scale engineering, this becomes kind of a

1:19 challenge. As as you have longer running builds and tasks, you have asynchronous team review flows across sides and geographies and things just get slower naturally as you grow. Um, this is also

1:33 a phase where you end up spending a lot of developer infrastructure dollars. So there's a lot of expensive tools here that need to be integrated and scaled up as you um scale your organization and it

1:47 all runs on very expensive infrastructure. Uh for these reasons it's often times the main focus of your platform organization team or your platform team. Um but any kind of

2:00 failure in this phase is very expensive. It introduces failures introduce an outer loop that are typically measured in hours and days. So failures are expensive. They costly. They really slow

2:14 down your velocity. Uh you lose developer time and that's compounded by disruptions to your flow state. It's also very expensive to optimize. So you've got a lot of fragmented systems

2:27 that have been integrated together with bespoke configuration and scripting. and you have a usually headcount restricted, constrained platform organization trying to keep it all up and running for you.

2:39 And trying to address that all with AI is also kind of complex because it requires complex workflow orchestration involving various inter asynchronous interaction, team workflows,

2:51 integrations across tools and so on. In fact, AI kind of makes the problem even worse. uh you've got a lot of PRs now being generated, more PRs than before uh thanks to AI and bigger PRs uh

3:08 all of which can have defects and so your costs in your development life cycle are kind of shifting to the right. You have a lot more code to review. You can have potential for a lot more

3:18 failures in production and and so your choices are kind of tough here. either you slow down and ask developers to review every PR very carefully or you rubber stamp PRs and you risk incidents

3:30 in production. Either way, you've got less happy developers, lower sentiment, and much less productivity than than you'd expect. So, what's the answer here? Of course,

3:43 it's more AI. It's uh an agentic approach to validation which is essentially what we built at guitar um that we're now also shipping uh as part of sonar.

3:56 And basically what we've got here is an agent that fully automates all the validation steps and as outcomes it delivers for you green PRs that are ready to merge or merge automatically.

4:08 Now how does this work? Well first when you create your PR it reviews your PR and posts issues and inline comments. No surprise there. But our goal has been to really make this experience as noisefree

4:20 as possible, both in the way we handle the UX as well as posting issues that are real and really matter. You can customize your review with your own rules and you can add your own

4:32 custom checks and you can trigger your own custom workflow automations. And then you can also configure the system to block merging if there are any unresolved code review findings.

4:44 It also analyzes CI failures and posts a summary of the root causes of those failures. And in particular, it's really good at catching flaky tests. And you can even set up rules that automatically

4:56 retry flaky tests if you want. That's a very popular feature. You can also automatically fix any code review issues or any CI failures. Any kind of uh failures that happen in CI,

5:08 it can fix it for you as well as address any of the issues that it raises. You can either ask it to specifically fix an issue or more interestingly you can set it up to automatically loop until all

5:20 the fish issues have been fixed in your PR and you have a green PR ready to merge. And finally, you can automatically approve and merge

5:31 PRs that are green. you can define rules and conditions that control when uh PRs get automatically approved by the agent and merged automatically. So we're seeing actually a very

5:46 interesting trend in our users. There's a clear trajectory towards fully automated PRs. uh as users use the system and as they build trust in the accuracy of the AI what we see is they

6:00 work towards full automation first they get reviews and they're happy with they build trust in the accuracy of the reviews they see that these reviews are not only accurate but they're raising

6:11 really important issues so then they give teeth to the agent by allowing it to start blocking the PRs and then they start using the autofix capabilities to automatically turn PRs green and fix all

6:24 the issues that were raised by code review or any CI failures that come up. And then where the real interesting leap is beginning to happen now is users setting up rules that define when the

6:36 agent automatically approves and merges PRs. So over time as they build trust in the system they unlock more levels of automation and with the additional levels of automation and with the trust

6:51 they're getting more value out of the system. Now because the system processes every single PR using AI it's also able to uncover new types of insights. For

7:03 example, uh one of the insights that are is very useful to platform teams is a categorization of the top CI failures. Where are you seeing failures in your CI system where your developer is running

7:16 into failures? Are these flakiness issues that you can that you need to address by improving the reliability of the tests or are there infrastructure issues that you need to address or are

7:28 there other kinds of failures that they're running into? Very very useful for platform teams. We also have another example of what what else we have in terms of AI enabled

7:38 insights are categorization of PRs. What exactly is happening across your engineering team in terms of the PRs that are being landed? This is extremely useful for leadership teams.

7:50 For example, what amount of PRs are going to feature development versus fixing issues, chores, refactors, and so on. Now under the hood, if you look at the

8:01 system, there's a number of components that constitute guitar. There are three interesting pieces here. One is the control plane that acts as this orchestration layer for handling PR

8:13 workflows. And then there's an agent runtime or or harness. We built our own harness here that handles multi- aent execution, context management, memory management, um tool calling, all the

8:25 integrations, etc. And having this having our own agent harness has really allowed us to optimize specifically for the validation use case and it's allowed us to optimize for token cost, precision

8:40 and coverage of the agent itself. So that allows us to essentially guarantee outcomes at a fixed PR price while giving really good quality results. We also have an NLM proxy that

8:54 handles the routing across different models as well as things like failover. Now that we're part of Sonar, we've been integrating guitar with sonar cube. There's a ton of opportunity here in

9:08 combining an agentic approach to validation with program analysis. So combining agents with things like u taint analysis, control flow analysis, data flow analysis, software composition

9:23 analysis. So all the traditional approaches combined with agents. We think there's a ton of opportunity here and that together you can get much better precision, much better coverage

9:34 at a really great price point. So the overall results uh the value that we're seeing in in all of this is much better quality, much better security, especially with AI generated code,

9:47 faster delivery times through all the automation going from PRs straight to approved and merged PRs, much higher productivity and sentiment by developers. So less time spent on

9:59 actually reviewing code and worrying about what's happening in production with more confidence as well as sentiment because now you much better sentiment now because you have a lot of

10:07 automation in place and less grunt work. And finally the to the features that we've built for platform teams gives them a lot more leverage to be able to fix issues and to add customizations and

10:20 see insights in this critical part of the software development life cycle. To wrap up, PR validation is really critical, even more so in AI, but AI turns validation

10:35 into a friction point. And guitar, what we've built, allows you to automate this step, fully automate PR validation, go from created PRs all the way to merge, completely eliminating this friction and

10:49 helps you keep up with the rate at which you're generating code from AI. And finally, the combination of using agents uh doing a gentic validation plus an algorithmic program analysis

11:03 will give you something that that I think is going to be much better than either alone. If you want to talk more and you want to see more, if you'd like to see a demo, come on and visit us at

11:11 booth number P7 right down right down the hall there. Thank you.