https://www.qodo.ai/blog/effective-code-suggestions-llms-less-is-more/ We are now Qodo (pronounced koh-doh), formerly Codium. * Platform + Products + Qodo Gen: IDE Plugin + Qodo Merge: Git Agent + Qodo Cover: CLI Agent + AlphaCodium RESEARCH + Qodo API CONTACT US + Solutions + Automated Testing + Refactoring COMING SOON + Code Review COMING SOON * Pricing * Docs * Resources + Blog + Useful Resources + Events * About + Our Mission + Meet the Team + Backed by + Careers + Contact Us ACCOUNT LOG IN BOOK A DEMOGET STARTED Effective AI code suggestions: less is more Code ReviewsTechnology coincoin coincoin coincoin coincoin coincoin coincoin Tal RidnikTal Ridnik Tal Ridnik January 29, 2025 * 5 min read [svg][Effective-Code-Suggestions-with-LLMs_-Less-is-More-2-690x463] TL;DR In our quest to generate effective code suggestions with LLMs at Qodo Merge, an AI-powered tool for automated pull request analysis and feedback, we discovered that prioritization isn't enough. Initially we tried telling the model to prioritize finding bugs and problems in pull requests, while still allowing lower-priority suggestions for style, code smells, and best practices. We discovered that this approach was ineffective, and despite the explicit prioritization, the model got overwhelmed by the easier-to-spot style issues, leading to developers flooded with low-importance suggestions. The breakthrough came when we stopped asking for different priority levels altogether, and instead limited the model to look only for meaningful bugs and problems. This laser-focus approach improved bug detection rates and signal-to-noise ratio, resulting in code suggestions that developers actually wanted to implement, with suggestion acceptance rates jumping 50% and overall impact increasing by 11% across pull requests. The key takeaway - sometimes the best strategy with LLMs isn't to prioritize or add complicated instructions, but to eliminate distractions entirely. Initial approach: casting a wide net Our initial strategy was ambitious. The model was instructed to prioritize critical issues in its review process, looking first for major problems before considering style-like improvements. Each review would contain up to 5 suggestions, prioretized by severity: Primary Concerns: * Critical bugs and problems * Security vulnerabilities Secondary Improvements: * Code style * Best practices * Potential enhancements This hierarchical review system seemed robust and effective. We leveraged modern prompting techniques like YAML-structured output formats to ensure consistency and machine-parseable responses, with each suggestion containing well-defined metadata fields. The unexpected challenge: style suggestion overflow In practice, we encountered an interesting phenomenon. Despite our careful prompt and flow engineering strategy, the model consistently gravitated toward style-related suggestions. For example: * Variable naming conventions * General exception handling * Minor refactoring opportunities * Documentation improvements While these suggestions weren't incorrect, they often overshadowed more critical issues. The abundance of minor suggestions often drowned out genuinely important findings. In addition, the model would sometimes misclassify style improvements as major concerns. The core problem wasn't the model's ability to spot serious bugs - it was that these easier-to-detect style issues dominated the output We tried to combat this overflow by adding explicit instructions like "don't suggest documentation improvements" to our prompt, but it proved futile. The sheer volume of potential style improvements was overwhelming. Even a single mention of 'code enhancement suggestions' in our instructions seemed to hijack the model's attention - there were simply too many possible minor improvements for the model to maintain its focus on truly critical issues. [svg][Screenshot-2025-01-29-at-11]Figure 1: Example of code suggestions using the prioritization method. The high number of categories and large quantity of suggestion, combined with inconsistent priority classification, resulted in 'suggestion fatigue' and poor signal-to-noise ratio. The signal-to-noise problem The 'Casting-wide-net' approach revealed significant challenges: 1. Developers struggled to locate critical issues buried within numerous style suggestions 2. The high volume of suggestions led to "suggestion fatigue," where developers started ignoring the feedback altogether rather than going through numerous style fixes to find the few critical issues. Beyond adding explicit exclusion rules to our prompts, we implemented both importance scoring and suggestion categorization ('bugs', 'enhancements', 'best practices'). But even this proved insufficient - the labels and the importance score weren't accurate enough to truly separate critical issues from nice-to-have improvements. User feedback revealed a fundamental truth: we needed to show only critical 'problem-finding' suggestions by default - everything else was often seen as noise. The breakthrough: laser focus on problems and bugs What ultimately worked was a paradigm shift in our approach. Instead of trying to manage priorities across different types of suggestions, we simplified the model's task to a single focus: identifying only meaningful problems that could result in bugs and issues in production. Our new prompt was ruthlessly focused: "Only give suggestions that address major problems and bugs in the pull request code. If no relevant suggestions are applicable, return an empty list" The results were dramatic: * The acceptance rate of each code suggestion increased by 50% * The impact level, meaning the percentage of pull requests where at least one suggestion was applied, increased by 11% The power of this approach lies in its singular focus. Rather than explicitly excluding certain types of suggestions, we simply directed the model's attention entirely toward finding meaningful code problems. This clarity of purpose had two key benefits: (1) it eliminated the complexity of evaluating and prioritizing different types of suggestions, and (2) it prevented the model from being overwhelmed by the sheer volume of potential style improvements that could otherwise dilute its attention from critical issues. [svg][Screenshot-2025-01-29-at-11]Figure 2: Example of code suggestions using the new focused method, which generates fewer, more targeted suggestions that highlight significant code issues. Code best practices: a parallel solution While maintaining our focused approach to problem detection, we developed a separate solution for style and best practices code suggestions. Rather than attempting to enforce every possible best practice - an almost infinite set, we enabled teams and organizations to define their own standards. Qodo Merge now operates through two distinct routes: one dedicated to finding critical problems, and another for evaluating code against company-specific best practices. Teams can also utilize our learning system that analyzes which suggestions developers actually implement - either to establish an initial set of best practices, or to continuously refine and enhance future recommendations. The success of this dual approach validated our strategy: developers trust and implement our critical bug suggestions because they know each one represents with high probability a genuine problem that needs fixing, while still having the flexibility to maintain their team's coding standards through configurable best practices. Final thoughts: the power of scope constraint This separation reflects a broader insight: sometimes the best strategy isn't to create complex prioritization schemes or scoring systems - it's to ruthlessly eliminate scope until you're left with only what truly matters. This might seem counterintuitive in an era where AI promises to handle increasingly complex tasks. However, our experience suggests that the path to better AI assistance lies not in just adding more complexity, but in being more thoughtful about what we ask these systems to do in the first place. The principle extends beyond just code review: Instead of trying to handle everything at once, we can try to handle important aspects well but separately. When working with LLMs, this focused approach yields better results than attempting to juggle multiple priorities simultaneously. Get Qodo VS Code --------------------------------------------------------------------- JetBrains --------------------------------------------------------------------- GitHub | Qodo Merge Subscribe to the newsletter Share coincoin Tal RidnikTal Ridnik Tal Ridnik January 29, 2025 Subscribe to the newsletter Quick contact always up to date JOIN OUR DISCORD More from our blog [svg][Qodo-Merge-1] Code ReviewsNews & Updates 5 min Qodo Merge 1.0: solving key challenges in AI-assisted code reviews Elana KrasnerElana KrasnerElana Krasner Jan 29, 2025 Read more How to Effectively Use AI Code Reviewers on GitHubHow to Effectively Use AI Code Reviewers on GitHub Code Reviews 6 min How to Effectively Use AI Code Reviewers on GitHub Talia ParnelTalia ParnelTalia Parnel Jan 26, 2025 Read more Next.js vs. Node.js: Comparison for Modern Web DevelopmentNext.js vs. Node.js: Comparison for Modern Web Development Technology 6 min Next.js vs. Node.js: Comparison for Modern Web Development Jonas RezelJonas RezelJonas Rezel Jan 22, 2025 Read more Qodo * Community * Useful Resources * Developer's Tips * Developer's Hub * Qodo vs. ChatGPT 4 * Qodo vs. GitHub Copilot * Qodo vs Tabnine * Qodo vs Amazon Q * Qodo Merge vs. GitHub Copilot for PR * Data Privacy and Security * Qodo's FAQs * General Q&A's * Supported Languages * Status Products * Qodo Gen: IDE Plugin * Qodo Merge: Git Agent * Qodo Cover: CLI Agent * AlphaCodium * Qodo API CONTACT US * Why Code Integrity? * Trust Center * Pricing * Report an issue About * Our Mission * IDE Integrations * Meet the Team * Backed by * Careers Navigation * Blog * Docs * Glossary * Code Review * Contact Qodo (formerly Codium) is a quality-first generative AI coding platform that helps developers write, test, and review code within IDE and Git. Our AI code generation offers automated code reviews, contextual suggestions, and comprehensive test generation, ensuring robust, reliable software. Seamless integration maintains high standards of code quality and integrity throughout development. Available now on VS CodeVS CodeWebStormWebStormIntelliJ IDEAIntelliJ IDEACLionCLion PyCharmPyCharmJetBrainsJetBrainsGOGO We support all major programming languages Python (VS Code & JetBrains)Python (VS Code & JetBrains)TypeScript (VS Code & JetBrains)TypeScript (VS Code & JetBrains)JavaScript (VS Code & JetBrains)JavaScript (VS Code & JetBrains)Java (VS Code & JetBrains)Java (VS Code & JetBrains)Kotlin (VS Code)Kotlin (VS Code) Go Programming (VS Code)Go Programming (VS Code)PHP (VS Code)PHP (VS Code)C++ (VS Code)C++ (VS Code)Swift (VS Code)Swift (VS Code)C# (VS Code)C# (VS Code) @ Qodo. All Rights Reserved. Terms of Use Privacy Policy Data Processing Addendum AWSAWS DeloitteDeloitte x George SoftasGeorge SoftasDedy KredoDedy Kredo AI model selection for developers: Finding the right fit for every coding challenge February 5, 2025 9AM PST / 4PM GMT REGISTER x * Platform + Products + Qodo Gen: IDE Plugin + Qodo Merge: Git Agent + Qodo Cover: CLI Agent + AlphaCodium RESEARCH + Qodo API CONTACT US + Solutions + Automated Testing + Refactoring COMING SOON + Code Review COMING SOON * Pricing * Docs * Resources + Blog + Useful Resources + Events * About + Our Mission + Meet the Team + Backed by + Careers + Contact Us 533.3K533.3K405.6K6.5K LOG INBOOK A DEMOGET STARTED *