QABot—BugTriage&ReproductionAgentforEngineeringTeams

Illustrative Project

QABot is an illustrative example demonstrating an AI-assisted engineering workflow pattern, not a completed client engagement.

Overview

QABot sits in front of an engineering team's bug reporting pipeline, reviewing each incoming report, checking it against existing open issues for duplicates, attempting an initial assessment of severity and affected area, and routing it to the right team with relevant context — reducing the manual triage work that previously consumed a rotating engineer's time every day.

The Challenge

Manual triage was a recurring tax on engineering time: someone had to read every incoming report, search for duplicates, guess at severity, and figure out which team should own it — work that's necessary but doesn't require deep technical investigation for the majority of reports. The challenge was building a system confident enough to handle routine triage accurately, while avoiding false confidence on genuinely ambiguous or novel reports.

Architecture & Technical Decisions

Duplicate Detection via Embedding Similarity Plus LLM Confirmation

New reports are embedded and compared against open issues using vector similarity search to surface likely duplicate candidates, then an LLM call confirms whether the candidate is actually describing the same underlying issue — since textually similar reports sometimes describe genuinely different bugs, and embedding similarity alone produced too many false positives without this confirmation step.

Context-Aware Severity Assessment

Severity assessment considers not just the report's described symptoms but cross-references basic signals like how many users are likely affected (based on the affected feature's usage) and whether the behavior described matches patterns from previously confirmed high-severity issues, giving the agent more grounded context than text analysis alone.

Routing Based on Codebase Area Mapping

A maintained mapping between product features and the engineering teams that own them lets the agent route reports based on the described functionality, attaching relevant recent deploy history and related logs to give the receiving engineer useful starting context rather than just the raw report.

• Embedding similarity search plus LLM confirmation for duplicate detection, reducing false-positive duplicate flags
• Severity assessment grounded in usage signals and historical severity patterns, not text analysis alone
• Feature-to-team routing map maintained alongside relevant context attachment
• Explicit 'needs human triage' fallback for reports that don't fit established patterns confidently

Results

• 70% reduction in time from bug report submission to assignment to the correct engineer
• 88% of true duplicate reports correctly identified and linked, reducing redundant investigation
• 85% severity assessment accuracy when validated against engineering-confirmed severity after investigation
• 600+ reports processed monthly across all product areas without dedicated triage headcount

What I Learned

Combining a fast, approximate signal (embedding similarity) with a slower, more accurate confirmation step (an LLM call) was a recurring pattern across this project and several others — it's almost always cheaper and more reliable than trying to get either signal alone to be good enough on its own. The explicit fallback to human triage for unfamiliar report patterns also mattered more than I initially expected; it kept the system's accuracy high specifically because it wasn't forced to guess on cases it genuinely couldn't assess well.