Expert AI PR Reviewer: Consolidated Capability Model

Synthesized from multi-source deep research (ChatGPT, Gemini, DeepSeek, Perplexity) — May 2026

Preflight Note: What to Watch For in This Synthesis

Before diving in, key cross-cutting observations from comparing all four research outputs:

Confidence scores vary significantly across sources. Gemini is consistently more optimistic (85–95% for most dimensions). DeepSeek and ChatGPT are more conservative (Low–Medium for concurrency, race conditions, business logic). Perplexity aligns with the conservative view. The synthesized view uses the conservative estimate as the realistic floor and the optimistic estimate as a ceiling achievable only with hybrid static + LLM tooling.
One critical dimension is underweighted across all four sources: supply chain and dependency risk. GitHub’s dependency review and SBOM obligations are mentioned briefly but not integrated into the multi-order impact model or the review dimension matrix. This document adds it explicitly as a P0 review dimension.
Perplexity’s unique contributions vs. the other three sources: (a) A sharper four-rule false-positive decision policy; (b) an explicit principle that the AI should not review everything equally — if a major design problem exists, line-by-line review is the wrong first move; (c) a fourth context tier separating “Useful” operational context from “High value for risk” runtime data; (d) an Observability Agent as an explicit named agent in the multi-agent design; (e) the clearest implementation decision rule: block on correctness/security/compatibility/data loss/migration/rollback; comment on maintainability/clarity/tests when evidence is solid; skip pure preference and out-of-scope cleanup.

Executive Summary

An expert AI PR reviewer is not a linter with prose. It is a context-hungry, risk-ranked, evidence-first system that reasons like a senior engineer — understanding intent before inspecting code, tracing downstream consequences before posting comments, and handing final authority to humans.

Six non-negotiable operating principles emerge consistently across all sources:

Principle	In Practice
Code health over perfection	Approve low-risk changes that improve the system; do not block for style preferences
Intent before implementation	Parse PR description, linked issues, and team instructions before reading the diff
Risk first	Correctness, security, compatibility, migrations, and reliability outrank cosmetics
Evidence before assertion	Every finding must be grounded in code behavior, a contract, a test result, or an explicit standard
Explain, don’t accuse	State why the issue matters, what could go wrong, what reduces the risk
Human final say	Never auto-approve or auto-merge; escalate uncertainty rather than bluff

Industry benchmarks worth anchoring to: Anthropic’s internal Claude Code deployment increased substantive PR review coverage from 16% to 54% with fewer than 1% of AI findings marked incorrect. Microsoft’s internal AI reviewer processes 600K+ PRs per month with 10–20% median PR completion time improvement. iCodeReviewer reports 84% comment acceptance rates using mixture-of-prompts routing. Tencent research shows hybrid LLM + static analysis eliminates 94–98% of false positives.

Section 1: Review Scope — All 41 Dimensions

Priority Model

Not all dimensions are reviewed on every PR. The synthesized priority model below (drawing from DeepSeek’s tiered approach and ChatGPT’s risk-first framing) governs when each dimension activates.

Priority	Group	Dimensions	Activation
P0	Security & Safety	Secrets handling, Input validation, Authentication, Authorization, Supply chain / dependencies	Every PR, always
P1	Correctness	Functional correctness, Logic defects, Error handling, Null handling, Edge cases	Every PR
P2	Compatibility	API compatibility, Breaking changes, Backward compatibility	Every PR with interface changes
P3	Data & Infrastructure	Database changes, Migration risks, Infrastructure changes, Configuration changes	Triggered by file type or path
P4	Reliability	Performance, Concurrency, Memory, Race conditions, Scalability, Resilience, Retry behavior	Triggered by path or pattern
P5	Observability	Logging, Telemetry, Tracing, Feature flags	Triggered by telemetry/config changes
P6	Maintainability	Architecture consistency, Maintainability, Readability, Duplication, Naming	Every PR (capped volume)
P7	Completeness	Testing completeness, Documentation, Accessibility, UX impact	Always, lightweight
P8	Governance	Compliance, Privacy, Business rule correctness, Product intent alignment	Where applicable; human escalation likely

Detailed Dimension Matrix

The table below consolidates the strongest claims across all three sources. Confidence is expressed as a realistic range (floor = conservative sources, ceiling = with hybrid static + LLM tooling).

#	Dimension	Why It Matters	Common Failures	Key AI Signals	Confidence
1	Functional Correctness	Core purpose; broken logic ships bugs	Inverted conditions, off-by-one, wrong defaults, partial implementations	Logic path analysis vs. PR intent; changed predicates; mismatched tests	Medium–High
2	Logic Defects	Subtle bugs passing tests but failing in production	Dead code, contradictory conditions, fall-through bugs	Control flow graph analysis, unreachable branch detection	Medium
3	Edge Cases	Production failures at boundaries	Unhandled empty/null/zero inputs, Unicode edge cases, leap second issues	Missing input guards, boundary value pattern gaps	Medium
4	Null Handling	NPE/NullReference is a top production crash cause	Unchecked nullable access, null in collections, null returned unexpectedly	Type system analysis, nullability annotation checking, data flow	High
5	Error Handling	Silent failures cause unrecoverable data corruption	Swallowed exceptions, empty catch blocks, missing cleanup on error	Exception flow tracing, catch-block analysis, resource cleanup	High
6	Input Validation	Untrusted input is the #1 attack vector (OWASP)	Missing API param validation, unsanitized user input, trusting internal services	Taint tracking, input source identification, missing validation decorators	High (with taint analysis)
7	Concurrency	Race conditions cause non-deterministic, hard-to-reproduce failures	Missing locks, double-checked locking bugs, non-atomic read-modify-write	Shared-state access patterns, lock ordering, atomicity boundaries	Low–Medium
8	Race Conditions	Data corruption from unsynchronized access	TOCTOU bugs, lost updates, un-awaited promises, missing async/await	Thread interleaving analysis, async execution paths	Low
9	Memory Concerns	Leaks and corruption cause gradual degradation	Retained event listeners, open resource handles, large object retention	Resource lifecycle tracking, allocation in global closures	Medium
10	Performance	Algorithmic inefficiency, N+1 queries, hot-path waste	DB queries inside loops, unnecessary allocations, blocking I/O on event loops	Complexity analysis, query-inside-loop AST patterns, removed batching	Medium–High (patterns)
11	Security	Breaches are catastrophic; OWASP Top 10	SQL injection, XSS, CSRF, path traversal, insecure deserialization, SSRF	Taint tracking, sanitizer verification, CWE-mapped patterns	Medium (LLM) / High (with SAST)
12	Authentication	Broken auth = full system compromise	Missing auth checks, weak token generation, session fixation	Auth middleware verification, endpoint protection audit	High
13	Authorization	Privilege escalation, IDOR vulnerabilities	Missing ownership checks, role confusion, missing tenant scoping	Permission check presence, object ownership tracing	Medium
14	Secrets Handling	Exposed credentials = instant compromise	Hardcoded secrets, secrets in logs, weak key derivation, secrets in test fixtures	Regex + entropy analysis, log statement scan, config file scan	Very High
15	API Compatibility	Breaking changes cascade through consumers	Renamed fields, stricter validation, removed endpoints, changed response semantics	API diff analysis, signature comparison, semantic version check	High
16	Backward Compatibility	Rolling deployments require old/new coexistence	Schema changes without migration, removed endpoints, changed behavior without deprecation	Protocol/schema evolution analysis, migration path verification	Medium
17	Migration Risks	Failed migrations corrupt data at scale	Missing rollback plan, blocking migrations, untested on large datasets	Migration script analysis, data volume awareness, rollback detection	Medium
18	Breaking Changes	Silent breakage of dependent systems	Unannounced breaking changes, inconsistent API versioning	Dependency graph traversal, consumer code search	High (with dependency graph)
19	Database Changes	Schema changes lock tables, corrupt data, break queries	Missing indexes, locking migrations, type coercion, ORM N+1 introduction	Schema diff analysis, query plan estimation, migration lock detection	Medium–High
20	Infrastructure Changes	IaC bugs take down entire environments	Overly permissive security groups, missing resource limits, hardcoded AZs	IaC static analysis, policy-as-code validation	High (with policy engine)
21	Configuration Changes	Config bugs are a top cause of production incidents	Missing validation, unsafe defaults, feature flag debt	Config schema validation, flag lifecycle tracking, startup validation	High
22	Supply Chain / Dependencies	Vulnerable packages introduce instant known CVEs	Vulnerable packages, license violations, no SBOM update, transitive risk	Lockfile diff analysis, CVE database scan, license compliance check	High (with tooling)
23	Observability	You can’t fix what you can’t see	Missing error logging, unstructured logs, no trace propagation, high-cardinality labels	Log statement analysis, metric emission check, trace context propagation	Medium–High
24	Logging	Debugging without logs is guesswork	Logging inside tight loops, PII leakage in logs, wrong log levels	Log pattern analysis, PII detection, log level appropriateness	High
25	Telemetry / Tracing	Distributed systems require distributed observability	Missing spans, broken trace context, inconsistent span naming	Span creation check, context propagation, metric name stability	Medium
26	Feature Flags	Unmanaged flags become tech debt and risk vectors	No removal plan, untested flag-off path, flag combinatorial explosion	Flag lifecycle tracking, both-branch test coverage	Medium–High
27	Resilience	Distributed systems fail; resilience patterns prevent cascades	Missing timeouts, no circuit breaker, infinite retries, no bulkhead	Resilience pattern detection, timeout presence, retry policy	Medium
28	Retry Behavior	Bad retry logic amplifies load during outages	No jitter, non-idempotent retries, unbounded retries, thundering herd	Retry config analysis, idempotency verification, deadline propagation	Medium–High
29	Scalability	Works at 100 users; fails at 100K	Per-request full table scans, local caching without invalidation, sequential processing	Complexity analysis, query fan-out, contention point detection	Low–Medium
30	Architecture Consistency	Divergent patterns create cognitive load and bugs	Business logic in controllers, circular dependencies, violated layering	Architecture pattern matching, import direction, layer placement	Medium–High
31	Code Maintainability	Code is read far more than written	God classes, deeply nested conditionals, magic numbers, over-clever code	Cyclomatic complexity, nesting depth, naming clarity	Medium
32	Readability	Unreadable code hides bugs	Cryptic variable names, long methods, misleading names	Readability heuristics, method length, naming convention adherence	Medium
33	Duplication	DRY violations lead to divergent bug fixes	Copy-pasted logic, repeated validation, duplicated configuration	Clone detection, AST similarity analysis	Very High
34	Naming	Names are the primary documentation	Vague names (data, temp, info), inconsistent terminology, unexplained abbreviations	Semantic naming analysis, domain terminology consistency	Medium
35	Documentation	Missing docs = tribal knowledge and onboarding friction	Undocumented public APIs, stale comments, missing README updates	Documentation coverage for public symbols, README change detection	Medium
36	Testing Completeness	Tests that cannot fail for the defect are worthless	Missing regression tests, removed assertions, overly mocked critical paths	Coverage shape analysis, test-to-production-code ratio, CI result correlation	High (coverage shape)
37	Accessibility	Legal obligation; excludes users when broken	Missing ARIA labels, non-keyboard-navigable UI, broken contrast logic	ARIA attribute presence, image alt text, form label analysis	Medium
38	UX Impact	Visual regressions degrade user trust	Unintentional layout shifts, broken UI flows, hardcoded locale strings	Structural UI component changes, locale string detection	Medium (code only)
39	Compliance	Regulatory violations carry legal and financial risk	GDPR/CCPA violations, SOC 2 control gaps, audit trail breaks	Compliance rule matching (requires external rule definition)	Low–Medium
40	Privacy	PII exposure causes legal harm and user distrust	PII in logs, over-collection, missing deletion paths, cross-border exposure	New user-data fields, logging of sensitive attributes, analytics export changes	Medium (with data classification)
41	Business Rule Correctness	Code can be clean and still implement the wrong thing	Incorrect discount logic, broken workflows, misaligned product behavior	PR intent comparison, acceptance criteria matching	Medium (with requirements context)

Section 2: Multi-Order Impact Detection

Why This Matters

The defining difference between a senior reviewer and a static checker is multi-order reasoning. An AI reviewer must not stop at “this line changed” — it must trace what that change means for callers, consumers, migration paths, dashboards, support playbooks, and rollback safety.

The Five Impact Paths

For every non-trivial change, the reviewer reasons across five paths:

Path	What the Reviewer Asks
Contract path	Did the public or internal contract change for callers, consumers, schemas, events, flags, or configs?
Data path	Does data now flow to a new sink, get transformed differently, migrate differently, or get logged differently?
Operational path	Could this alter retries, latency, load, cardinality, alerting, failure domains, or rollback?
Release path	Can old and new versions coexist? Is the migration reversible? Is there a flag, canary, or kill switch?
Human path	Does this need a specialist reviewer? Are docs, runbooks, or change notes being updated?

Cascading Impact Examples

API Payload Change
  → First-order:  API structure modified
  → Second-order: Consumer contract breaks; analytics payload distorted
  → Third-order:  Dashboards break; alerting rules become stale
  → Long-tail:    Reporting becomes wrong; support tickets increase

Schema Refactoring
  → First-order:  Database schema updated
  → Second-order: Migration script locks; old readers fail
  → Third-order:  Historical data mismatch or truncation
  → Long-tail:    Backfill required; rollback impossible

Event/Topic Rename
  → First-order:  Message topic modified
  → Second-order: Subscriber failures; real-time integrations break
  → Third-order:  Delayed analytics; alerting silently stops firing
  → Long-tail:    Missed SLAs; no alert that alerts are missing

Retry Config Change
  → First-order:  Altered resilience behavior
  → Second-order: Increased network load; thundering herd
  → Third-order:  Cascade failure across downstream services
  → Long-tail:    Infrastructure cost spike; incident postmortems required

Caching Key Change
  → First-order:  Modified cache lookup behavior
  → Second-order: Cache miss surge under load
  → Third-order:  Stale data reaching users; race conditions
  → Long-tail:    Support tickets; manual cache purge required

Dependency Addition
  → First-order:  New package in dependency graph
  → Second-order: Transitive vulnerability introduced; license conflict
  → Third-order:  SBOM now inaccurate; compliance gap
  → Long-tail:    Supply chain attack surface enlarged

Multi-Order Reasoning Template

Change: [description]

FIRST-ORDER:
├── API surface:    [added/removed/modified endpoints or signatures]
├── Schema:         [DDL changes, type modifications]
├── Configuration:  [new/changed/removed keys]
├── Events:         [new/changed/removed event types]
└── Dependencies:   [new/removed packages, version bumps]

SECOND-ORDER:
├── Direct consumers of changed API: [list]
│   └── Potential behavior change: [description]
├── Migration impact on historical data: [assessment]
├── Dashboard/alerting impact: [affected dashboards and alerts]
└── Integration impact: [affected integrations]

THIRD-ORDER:
├── Infrastructure scaling implications: [assessment]
├── Cost implications: [if detectable]
└── Organizational impact (teams affected): [list]

LONG-TAIL:
├── Technical debt introduced: [assessment]
├── Future constraint implications: [assessment]
└── Maintenance burden change: [assessment]

Detection Methods

AST delta analysis — Parse modified files to isolate changed public interfaces and type structures
Static call graph traversal — Project AST delta onto the global dependency graph to find all callers of changed functions
API surface diffing — Compare OpenAPI/gRPC specs before and after; flag non-additive changes
Schema evolution analysis — Diff database schemas; flag destructive operations (column removal, type narrowing, constraint additions)
Event/message schema validation — Parse queue topics and configurations; identify renamed or restructured event types
Dependency graph traversal — Identify all consumers of changed internal libraries and flag version compatibility
Data lineage mapping — Trace schema changes to downstream analytics transformations and BI queries
Blast radius summary — Before generating comments, produce an explicit list: impacted services, contracts, data stores, migrations, feature flags, observability surfaces, and specialist-review needs

Section 3: Context Awareness

Four-Tier Context Model

Context quality is the single largest differentiator between mediocre and excellent AI reviews. CodeRabbit’s “1:1 code-to-context ratio” principle applies: match context depth to change risk. Perplexity’s research adds a meaningful split in the upper tiers — separating operational “useful” context from high-risk runtime diagnostic data.

Tier	Artefacts	Why Required
Required (cannot review without)	Full PR diff; base branch and changed files; tests changed or affected; CI/check results; PR title and description; linked issue/ticket; repo-wide and path-specific AI instructions (CLAUDE.md, REVIEW.md, copilot-instructions.md); dependency diff; symbol/reference graph for changed code; CODEOWNERS	Minimum needed to understand intent, review local implementation, detect contract breaks, apply team conventions
Strongly Recommended	Design docs / ADRs; acceptance criteria; previous PRs touching the same area; deployment topology; config/environment overlays; feature-flag metadata; architecture diagrams; language/framework version constraints	Enables reasoning about blast radius, architecture fit, rollout risk, and recurrence of known failure modes
Useful for Structural Impact	Dependency graph; API contracts and OpenAPI specs; DB schema; feature flag registry; previous PRs in the same area; service ownership maps	Enables accurate multi-order impact tracing — catches consumer breakage, schema drift, and flag lifecycle issues
High Value for Risk Reasoning	Incident history; production metrics for affected services; logs and alerts; test history; rollout strategy; known SLOs; security review history; unwritten team conventions (inferred from past review comments)	Powers second- and third-order impact detection; identifies recurrence of known failure patterns; reduces speculative comments on high-stakes changes

Critical Implementation Constraint

GitHub Copilot reads only the first 4,000 characters of any custom instruction file. When designing instruction files (REVIEW.md, CLAUDE.md), the most critical rules must appear at the top. This is an easy thing to overlook when building the skill.

Missing Context Protocol

When required context is absent, the reviewer must:

Lower confidence for all affected findings
Convert would-be assertions into questions
Explicitly note what context would resolve the uncertainty

Section 4: Review Workflow

Ten-Phase Review Process

The workflow below synthesizes Google’s reviewer guide sequence (design first, then functionality, complexity, tests, documentation) with Anthropic’s parallel-verification pattern and Gemini’s phased pipeline.

Phase 1: UNDERSTAND INTENT
  ├── Parse PR title, description, linked issues
  ├── Extract stated goal and acceptance criteria
  └── Form hypothesis about what a clean implementation looks like

Phase 2: SCOPE THE CHANGE
  ├── Identify all changed files and their roles in the system
  ├── Classify change type (feature, fix, refactor, config, infra, etc.)
  └── Calibrate review depth proportional to risk and change size

Phase 3: BUILD DEPENDENCY CONTEXT
  ├── Construct dependency graph for changed code
  ├── Identify direct and indirect consumers
  └── Map affected subsystems and team ownership

Phase 4: IDENTIFY RISK AREAS
  ├── Flag high-risk patterns (auth, data mutation, API change, migration)
  ├── Assess test coverage for changed paths
  └── Prioritize review focus before implementation pass begins

Phase 5: ARCHITECTURE & DESIGN REVIEW
  ├── Is the approach appropriate for this system?
  ├── Is the change in the right layer/module?
  ├── Does it follow established patterns?
  └── Any over-engineering or unnecessary complexity?

Phase 6: IMPLEMENTATION REVIEW (LINE-BY-LINE)
  ├── Functional correctness and business rule alignment
  ├── Error handling completeness (all paths, including failures)
  ├── Edge case coverage (nulls, empty, boundary values)
  ├── Security vulnerabilities (OWASP, taint paths, auth)
  ├── Performance issues (complexity, hot paths, query patterns)
  └── Code quality (naming, duplication, readability, maintainability)

Phase 7: TEST REVIEW
  ├── Do tests actually verify the changed behavior?
  ├── Are error paths and edge cases covered?
  ├── Would the tests fail for the bug this PR could introduce?
  └── Are there missing test scenarios?

Phase 8: MULTI-ORDER IMPACT ANALYSIS
  ├── First-order:  direct consumers affected
  ├── Second-order: indirect system effects
  ├── Third-order:  infrastructure and organizational impact
  └── Long-tail:    tech debt, maintenance burden, supply chain

Phase 9: GENERATE & PRIORITIZE FINDINGS
  ├── Assign severity (Critical / High / Medium / Low)
  ├── Assign confidence (High / Medium / Low)
  ├── Deduplicate overlapping findings across review areas
  └── Sort by severity

Phase 10: GENERATE RECOMMENDATIONS & OUTPUT
  ├── Provide fix suggestions where confidence is sufficient
  ├── Explain reasoning and tradeoffs
  ├── Flag areas requiring human judgment
  └── Produce structured review output (summary + inline comments)

Review Depth Modes

Mode	Steps Included	Target Use Case
Quick	1, 2, 5, 6, 9, 10	Trivial changes, docs-only, config tweaks
Standard	All 10	Typical feature or bug-fix PR
Deep	All 10 with expanded multi-agent analysis	High-risk changes: auth, schema, public APIs, infra

Critical Workflow Principle: Do Not Review Everything Equally

A key insight from Perplexity’s synthesis of Google’s reviewer guidance: if a major design problem exists in the PR, performing a line-by-line implementation review is the wrong first move. The AI should spend attention where the risk is highest. When Phase 5 (architecture and design review) surfaces a fundamental problem, the right action is to surface that finding immediately and flag that deeper review is premature until the design question is resolved. Reviewing every line in detail first wastes signal budget and risks burying the design issue under lower-priority noise.

Three-Bucket Implementation Decision Rule

From Perplexity’s synthesis — the clearest practical decision rule for what to surface vs. suppress:

Bucket	Action	Examples
Block or strongly warn	Always surface; Critical or High severity	Correctness bugs, security vulnerabilities, API compatibility breaks, data loss risks, migration/rollback risks
Comment when evidence is solid	Surface at Medium severity with concrete evidence	Maintainability issues, clarity gaps, test coverage gaps, observability gaps
Skip entirely	Suppress	Pure preference, speculative refactors, out-of-scope cleanup, adjacent code the PR didn’t touch

Section 5: Recommendation Generation

Fix Suggestion Decision Tree

Does the issue have a clear, unambiguous fix?
│
├── YES → Is the fix trivial (rename, add null check, add await)?
│   │
│   ├── YES → Provide exact code diff
│   │         Format: ```suggestion``` block for single-click apply
│   │
│   └── NO  → Provide pseudocode with principle-based explanation
│             Let the author implement; provide tradeoff options
│
└── NO (or LOW CONFIDENCE)
      → Do NOT propose a fix
      → State the concern and why it matters
      → Ask clarifying questions
      → Flag for human judgment explicitly

Recommendation Format by Situation

Situation	Preferred Response
Localized, well-understood defect with clear convention	Exact code suggestion in diff format
Cross-cutting change, migration, or architectural trade-off	Pseudocode or stepwise guidance; not a patch-shaped hallucination
Ambiguous business rule or missing requirement	Question — state what evidence is missing
Security-sensitive change with multiple valid remediations	Alternatives with trade-offs — not a single confident fix
Low-confidence suspicion with potentially high impact	Risk flag — not an established fact

Hallucination Prevention

Only suggest fixes for patterns with high-confidence detection
Ground all suggestions in existing codebase patterns (not assumed conventions)
Verify suggestions against the actual symbol table and type system before posting
Run proposed code through local compiler/type checker where possible (tsc, mypy, etc.)
Use “Consider…” and “One approach might be…” rather than “You should…”
Never suggest changes for purely stylistic reasons outside the team’s style guide

Section 6: Finding Structure

Standard Finding Format

### [CLASS]: [One-line summary]
 
**Severity:** Critical | High | Medium | Low
**Confidence:** High | Medium | Low
**Category:** Security | Correctness | Performance | Architecture | Compatibility |
Reliability | Testing | Observability | Maintainability | Privacy | Compliance | Documentation
**Action:** Must Fix | Should Fix | Consider | Question
 
**Location:** `path/to/file.ext:line-start–line-end`
 
**Evidence:**
[What the code currently does and why it is problematic.
Include relevant code snippets with line references.]
 
**Impact:**
[What could go wrong if not addressed — be specific about
consequences: production outage, security breach, data loss,
degraded UX, cascade failure, broken dashboard, etc.]
 
**Affected Systems:**
 
- [Component / service / team / dashboard / table / job / flag]
 
**Recommendation:**
[Exact code fix, pseudocode approach, or question for human judgment.
Provide alternatives with trade-offs for non-trivial issues.]
 
**Reasoning:**
[The engineering principle, past incident pattern, or system property
that makes this important — not just "this is bad".]
 
**Open Questions (if any):**
 
- [Questions requiring human domain knowledge to resolve]

Severity Definitions

Severity	Definition	Examples
Critical	Must fix before merge; would cause production incident, security breach, or data loss	Hardcoded secret, SQL injection, auth bypass, data deletion without safeguard
High	Should fix before merge; high probability of production bug	Null reference on common path, missing error handling, breaking API change, incorrect business logic
Medium	Should address; degrades quality or introduces risk	Missing test coverage, unclear naming, missing error logging, minor performance issue
Low	Consider addressing; improvement opportunity	Style deviation, comment clarity, minor duplication, documentation gap

Low-severity findings should use Google’s “Nit:” prefix convention — clearly marked as optional.

Confidence Definitions

Confidence	Criteria	Suggestion Behavior
High (≥85%)	Deterministic detection; multiple signals converge; no ambiguity	Provide exact fix for trivial issues
Medium (60–85%)	Pattern recognized but context-dependent; some ambiguity	Provide pseudocode or principle-based guidance
Low (<60%)	Heuristic detection; significant ambiguity; domain knowledge required	Ask questions; flag for human; do NOT suggest fixes

Worked Example (Perplexity)

The most compact, complete end-to-end example across all four sources — illustrating the finding structure applied to a real API compatibility scenario:

### Should Fix: Removed response field may silently break downstream clients
 
**Severity:** High
**Confidence:** Medium
**Category:** Compatibility
**Action:** Should Fix
 
**Location:** `api/user-controller.ts:87`
 
**Evidence:**
The endpoint response removes `customer_type`, but two existing call sites
in `billing-service` and `analytics-consumer` still read this field.
 
**Impact:**
Downstream clients may fail silently or misclassify users if they receive
a response without `customer_type`. Billing logic that branches on this
field could produce incorrect invoices.
 
**Affected Systems:**
 
- billing-service (reads customer_type for invoice tier selection)
- analytics-consumer (uses customer_type for segmentation dashboards)
 
**Recommendation:**
Keep the field for at least one release behind a feature flag, or version
the API endpoint (e.g. /v2/user) and deprecate /v1/user with a migration
timeline communicated to consumers.
 
**Open Questions:**
 
- Are all consumers of this endpoint updated in the same rollout?
- Is there a deprecation window agreed with downstream teams?

Section 7: False Positive Reduction

Four-Layer Control Model

LAYER 1: PRE-GENERATION FILTERING
├── Confidence thresholds: Surface only findings above configured minimum
├── Evidence requirements: Require concrete code evidence before flagging
├── Scope limiting: Only flag issues introduced by this PR, not pre-existing
├── Path exclusions: Skip generated files, lock files, binaries, dist/, .gen.* files
└── Pattern exclusions: Suppress known false-positive patterns (test fixtures, etc.)

LAYER 2: CROSS-VALIDATION
├── Multi-agent verification: Cross-check findings across specialized agents
├── Static analysis correlation: Correlate LLM findings with deterministic tool results
├── Symbol table verification: Confirm all referenced symbols exist in the actual diff
└── Codebase pattern matching: Verify against how similar patterns are handled elsewhere

LAYER 3: POST-GENERATION VERIFICATION
├── Deduplication: Merge overlapping findings by file + line + category
├── Severity calibration: Verify severity assignment against actual impact analysis
├── Hallucination check: Confirm every code reference exists in the diff
└── Guideline compliance: Verify finding matches team review guidelines

LAYER 4: BEHAVIORAL CONTROLS
├── Ask instead of assert: For low-confidence findings, use interrogative framing
├── Explicit uncertainty: State confidence level and reasons for doubt
├── Human escalation rules: Clear criteria for when human must decide
├── Comment volume cap: Limit total comments and nit-level annotations per PR
└── Feedback loop: Learn from human corrections to improve future accuracy

Interrogative Framing for Low-Confidence Findings

Instead of asserting findings the reviewer isn’t sure about, phrase them as questions:

“Could this shared state be accessed concurrently? If so, consider…”
“Is there a reason this query doesn’t use an index? For large datasets…”
“Was the decision to use X instead of Y intentional? Y would provide…”

Four-Rule False Positive Decision Policy

From Perplexity’s synthesis — a precise, operationally useful decision rule for every potential finding:

Situation	What to Do
Issue is clear and consequential	State it as a finding with full structure
Issue is plausible but uncertain	Frame it as a question, not an assertion
Issue is preference-based or out of scope	Do not raise it at all
Issue depends on runtime data or production behavior not visible in the repo	Explicitly state that evidence is incomplete; do not guess

This policy — especially the fourth rule — is the most important single guard against hallucinated findings. When the reviewer cannot see the data it needs to make the call, saying so honestly is more valuable than speculating.

Key Proven Techniques

Hybrid LLM + Static Analysis: Tencent research: hybrid approach eliminates 94–98% of false positives while maintaining high recall
Mixture-of-Prompts Routing: iCodeReviewer activates only necessary prompt experts based on code features — 84% acceptance rate in production
Post-generation verification: CodeRabbit and Claude Code both verify suggestions against actual codebase patterns before posting
Nit volume capping: Limit to 5 nit-level inline annotations per PR; consolidate remaining style comments into a single summary table
Pre-existing issue suppression: Only flag issues introduced by this PR; never relitigate existing codebase problems unless significantly worsened

Section 8: AI Reviewer Behavior Model

Core Behavioral Principles

Prioritize high-signal findings. Focus on what matters: correctness, security, compatibility, reliability. Resist the temptation to comment on everything.
Avoid flooding PRs. Produce a single structured summary plus inline comments for specific, high-confidence issues only.
Explain reasoning. Every finding must include the “why” — the engineering principle, potential consequence, or system property at stake. This builds trust and is educational.
State uncertainty explicitly. Confidence levels must be transparent. Saying “I’m not certain this race condition is reachable” is more useful than asserting it is.
Adapt to team conventions. Use instructions files (REVIEW.md, copilot-instructions.md) to encode team-specific preferences for testing rigor, security posture, and style conventions.
Critique code, never the author. Use neutral, professional language. Prefer “this function” over “you wrote.” Use “consider” not “you should.”
Never approve PRs. The AI is advisory. Approval is always a human call.
Auto-resolve when fixed. Track which threads correspond to which findings; resolve them when the author fixes the issue (Claude Code behavior).
Reinforce good patterns. Occasionally note strong implementations — this is educational and builds collaborative tone.

Finding Type Classification

Type	Severity + Confidence	Example
Must Fix	Critical/High severity, High confidence	”Hardcoded AWS secret key found at line 42”
Should Fix	High severity + Medium confidence, or Medium severity + High confidence	”Missing null check in common execution path — could cause NPE”
Consider	Medium severity + lower confidence, or Low severity	”This loop could be refactored for clarity using the existing helper in utils.ts”
Question	Any finding with Low confidence	”Could this shared state be accessed concurrently? If so, a lock may be needed.”

Cognitive Bias Advantages

AI reviewers have structural advantages over humans for certain biases:

No confirmation bias (doesn’t assume the author’s approach is correct based on rapport)
No decision fatigue (reviews the 50th PR with the same rigor as the first)
No anchoring (evaluates each finding independently)
No authority bias (doesn’t defer to senior authors or push back harder on junior ones)

These are genuine advantages — they should be reinforced by design, not eroded by training reviewers to be “agreeable.”

Tone Research Finding

Studies on code review communication show neutral-toned comments are perceived as useful 79% of the time versus 57% for very negative comments. The behavior model targets neutral, collaborative, professional language throughout.

Section 9: Multi-Agent Architecture

Why Multi-Agent

Review tasks are heterogeneous. A security specialist must reason differently from a performance specialist. Parallel specialization produces deeper coverage, enables cross-validation, and reduces false positives. Anthropic’s Claude Code explicitly dispatches a team of agents that look for different classes of problems in parallel, then verifies, deduplicates, and ranks candidate findings.

Recommended Agent Architecture

                    ┌─────────────────────────┐
                    │    ORCHESTRATOR AGENT   │
                    │  (Intent + Triage +     │
                    │   Context Assembly)     │
                    └──────────┬──────────────┘
                               │
        ┌──────────────────────┼────────────────────┐
        ▼                      ▼                    ▼
┌──────────────┐  ┌──────────────────┐  ┌────────────────────┐
│ CORRECTNESS  │  │    SECURITY &    │  │   ARCHITECTURE     │
│   AGENT      │  │    PRIVACY AGENT │  │   AGENT            │
│              │  │                  │  │                    │
│ Logic bugs   │  │ OWASP Top 10     │  │ Design patterns    │
│ Edge cases   │  │ Auth/authz       │  │ Layer violations   │
│ Null safety  │  │ Secrets          │  │ Coupling issues    │
│ Error paths  │  │ Input validation │  │ Consistency        │
│ Concurrency  │  │ PII / Privacy    │  │ ADR compliance     │
└──────┬───────┘  └───────┬──────────┘  └──────────┬─────────┘
        │                  │                        │
        └──────────────────┼────────────────────────┘
                           │
        ┌──────────────────┼────────────────────┐
        ▼                  ▼                    ▼
┌──────────────┐  ┌──────────────────┐  ┌────────────────────┐
│ PERFORMANCE  │  │    TESTING       │  │    IMPACT          │
│   AGENT      │  │    AGENT         │  │    AGENT           │
│              │  │                  │  │                    │
│ Complexity   │  │ Coverage gaps    │  │ API surface diff   │
│ Query perf   │  │ Test quality     │  │ Schema impact      │
│ Memory usage │  │ Edge cases       │  │ Config/infra       │
│ Hot paths    │  │ Assertion depth  │  │ Multi-order traces │
│ Retry/cache  │  │ Mock quality     │  │ Dependency chain   │
└──────┬───────┘  └───────┬──────────┘  └──────────┬─────────┘
        │                  │                        │
        └──────────────────┼────────────────────────┘
                           │
                           ▼
              ┌─────────────────────────┐
              │   VERIFICATION AGENT   │
              │                        │
              │ Cross-check findings   │
              │ Symbol table verify    │
              │ Compiler/type check    │
              │ Deduplicate            │
              │ Severity calibrate     │
              │ False positive filter  │
              └────────────┬───────────┘
                           │
                           ▼
              ┌─────────────────────────┐
              │   SYNTHESIS AGENT      │
              │                        │
              │ Prioritize by severity │
              │ Structure output       │
              │ Generate summary       │
              │ Post inline + overview │
              └─────────────────────────┘

Named Agent Roster (All Sources Combined)

Agent	Primary Responsibilities
Orchestrator	Intent parsing, triage, context assembly, proportional depth calibration
Correctness	Logic bugs, edge cases, null safety, error handling, concurrency
Security & Privacy	OWASP Top 10, auth/authz, secrets, input validation, PII/data exposure
Architecture	Design patterns, layer violations, coupling, consistency, ADR compliance
Performance	Complexity, query patterns, memory, hot paths, retry/cache behavior
Testing	Coverage gaps, test quality, edge cases, assertion depth, mock quality
Impact	API surface diff, schema impact, config/infra changes, multi-order traces, dependency chain
Observability (Perplexity explicit addition)	Logging completeness, metrics emission, trace context propagation, alert impact, event name stability
Verification	Cross-check findings, symbol table verification, compiler/type-check, deduplication, severity calibration
Synthesis	Prioritize by severity, structure output, generate summary, post inline comments + overview

Coordination Model Options

Model	Description	Best For
Parallel Dispatch	All specialized agents run simultaneously; findings merged by aggregator	Standard reviews — minimizes latency
Sequential Pipeline	Agents run in order, passing findings downstream	Deep chain-of-thought analysis; higher latency
Orchestrator-Worker-Validator	Orchestrator assigns, workers analyze, validator verifies before posting	High-reliability or security-critical repositories

Single-Agent Multi-Pass (Cost-Sensitive Alternative)

For cost-sensitive deployments: a single agent performing sequential specialized passes (architecture → correctness → security → performance → impact) approximates multi-agent depth at lower token cost. Trades parallelism for affordability.

Cost Consideration

Running 6 specialized agents + verification + synthesis ≈ 4–6× the token cost of a single-agent review. However, catching issues on the first pass reduces iteration cycles — net cost may be neutral or favorable per merged PR. Start with the multi-pass single-agent model and graduate to full multi-agent for high-risk repositories.

Pros and Cons

Factor	Multi-Agent	Single-Agent Multi-Pass
Coverage depth	Higher — specialist reasoning per domain	Moderate — depends on pass quality
False positive rate	Lower — cross-validation between agents	Higher — no cross-check
Latency	Lower — parallelism	Higher — sequential passes
Token cost	4–6× baseline	1.5–2× baseline
Coordination complexity	Higher	Lower

Section 10: Complete Capability Model

10.1 Responsibilities

The AI PR Reviewer Skill is responsible for:

Detecting issues across all 41 review dimensions (Section 1)
Explaining each finding with evidence, impact, and reasoning
Recommending fixes where confidence is sufficient; asking questions otherwise
Identifying multi-order impacts across the system (Section 2)
Prioritizing findings by severity and confidence
Adapting to team conventions via configuration (REVIEW.md, instructions files)
Learning from human feedback to improve accuracy over time
Remaining advisory — never approving, blocking, or merging PRs

10.2 Complete Review Output Format

# AI Code Review: [PR Title]
 
**PR:** #[number] | **Author:** @[author] | **Changed Files:** [count]
**Review Depth:** Quick | Standard | Deep | **Duration:** [time]
**Blast Radius:** [impacted services, contracts, data stores, teams]
 
---
 
## Summary
 
[2–3 sentences: what changed, overall assessment, biggest risk area.
Explicitly note if critical areas require specialist human review.]
 
---
 
## Critical Findings
 
[Critical-severity issues. Each follows the finding structure from Section 6.
If none: "No critical issues detected."]
 
## High-Priority Findings
 
[High-severity issues.]
 
## Medium-Priority Findings
 
[Medium-severity issues.]
 
## Low-Priority / Suggestions (Nit)
 
[Low-severity findings. Clearly marked as optional and non-blocking.]
 
---
 
## Impact Analysis
 
[Multi-order impact summary from Section 2 reasoning template.]
 
## Test Coverage Assessment
 
[Analysis of test quality and coverage gaps for changed paths.]
 
## Questions for the Author
 
[Open questions requiring human clarification — numbered list.]
 
---
 
_AI-generated review. Human judgment required for final approval decision._
_Confidence: High ≥85% | Medium 60–85% | Low <60%_
_Review only covers issues introduced by this PR, not pre-existing code._

10.3 Decision Tree: When to Flag vs. Stay Silent

AI encounters potential issue
│
├── Introduced by this PR?
│   ├── NO  → Suppress (pre-existing issue)
│   └── YES → Continue
│
├── Confidence ≥ configured threshold?
│   ├── NO  → Ask as question (if High impact) or suppress
│   └── YES → Continue
│
├── Severity ≥ minimum for this path/category?
│   ├── NO  → Suppress or consolidate into nit summary
│   └── YES → Continue
│
├── Already flagged by another agent?
│   ├── YES → Deduplicate (merge into highest-severity finding)
│   └── NO  → Continue
│
├── Matches known false-positive pattern?
│   ├── YES → Suppress
│   └── NO  → Continue
│
├── Evidence is concrete (line references, code paths)?
│   ├── NO  → Downgrade to question; do not assert
│   └── YES → Post finding with full structure
│
└── Is this a nit-level issue?
    ├── YES → Check: have we hit the nit cap (default: 5)?
    │         If cap reached → consolidate into nit summary table
    └── NO  → Post as inline comment

10.4 Implementation Suggestions

1. Start hybrid, not pure LLM. Combine deterministic static analysis (null checks, secrets, duplication, style, CVE scanning) with LLM reasoning (logic, design, impact, business rules). This is the approach used by DeepSource, CodeRabbit, GitHub Copilot, and Anthropic’s Code Review.

2. Deploy progressively.

Phase 1: Instrumentation
  → Basic AI summaries + standard static linters. Advisory only. No blocking.

Phase 2: Contextualization
  → Precompute dependency maps. Link local guidelines (REVIEW.md).
  → Add linked-ticket and ADR context.

Phase 3: Multi-Agent Model
  → Deploy specialized agents with domain routing.
  → Add compiler/type-checker verification.
  → Enable confidence-threshold filtering.

Phase 4: Feedback Loops
  → Track accepted/rejected/false-positive comments.
  → Use data to calibrate severity and confidence thresholds.
  → Consider enabling quality gates for Critical findings only.

3. Invest heavily in context engineering. The single biggest differentiator between mediocre and excellent AI reviews is context quality. Match context depth to change risk.

4. Build feedback loops from day one. Track which findings are accepted, dismissed, or marked false positive. Use this data to tune confidence thresholds and false-positive suppression rules.

5. Use REVIEW.md / instructions files for team-specific rules. Centralize team conventions, severity overrides, path exclusions, and domain-specific requirements. The first 4,000 characters are read by GitHub Copilot; put the most critical rules first.

6. Never block without human review. The AI provides information; humans make decisions. This is a consistent design principle across Anthropic, GitHub, AWS, and Microsoft.

7. Path-activate specialist review. Don’t run security checks against CSS files or accessibility checks against database migration scripts. Route by file extension, directory, and change type to reduce noise and cost.

8. Auto-resolve fixed threads. Detect when a previously flagged issue has been addressed; auto-resolve the thread. This reduces review noise and makes the AI feel like a collaborator rather than an auditor.

10.5 Gaps Requiring Human Reviewers

The following areas remain firmly in the human domain — these are permanent gaps, not temporary tooling deficits:

Gap	Why AI Cannot Resolve It
Business rule correctness	Requires deep domain knowledge and business context not present in the codebase
Product intent alignment	Requires understanding of product strategy, user needs, and commercial tradeoffs
Architectural vision	Requires long-term system evolution perspective and organizational context
Team dynamics & mentorship	Code review is a teaching and relationship-building activity; AI cannot mentor
Complex concurrency under load	Non-deterministic race conditions emerging only under production load require human analysis and stress testing
Creative approach validation	Novel or unconventional approaches may be correct but unfamiliar — requires human engineering judgment
Cross-team coordination	Requires understanding of organizational politics, priorities, and team agreements
Calculated risk acceptance	”Ship it anyway” decisions require human authority
Strategic technical debt	Some tech debt is intentional and strategic — AI cannot distinguish acceptable from harmful debt
Visual UX quality	HTML structure is parseable; whether a layout feels intuitive or matches a design language requires human eyes
Compliance edge cases	Novel regulatory scenarios require legal and compliance expertise

Appendix: What Each Source Contributed (Unique Additions)

Source	Strongest Unique Contribution
ChatGPT	Most rigorous synthesis of public engineering guidance (Google, NIST, OWASP, OpenTelemetry); best framing of the five impact paths; strongest treatment of API and telemetry stability as first-class review concerns
Gemini	Most detailed multi-agent orchestration blueprint; quantitative confidence scoring formula; agent.toml configuration schema; phased implementation roadmap; 10-phase review pipeline with explicit data structures; hallucination control protocols with compiler verification
DeepSeek	Best structured finding format; clearest priority tiering (P0–P8); strongest treatment of cognitive bias advantages; most actionable false-positive multi-layer control model; best cost analysis of single vs. multi-agent tradeoffs; supply chain / SBOM as an explicit review dimension
Perplexity	(1) Four-rule false positive decision policy — the clearest operational guard against hallucinated findings, especially the “say evidence is incomplete” rule for runtime-dependent issues; (2) Explicit “do not review everything equally” workflow principle — if a major design issue exists, line-by-line review is the wrong first move; (3) Four-tier context model — meaningful split between “Useful for structural impact” and “High value for risk reasoning”; (4) Observability Agent as an explicitly named, isolated agent in the multi-agent design; (5) Three-bucket implementation decision rule (block / comment-with-evidence / skip) — the most concise practical filter for what an AI reviewer should and should not surface

This consolidated capability model is designed to translate directly into an AI coding agent skill or system prompt implementation. The most implementation-ready sections are Section 6 (finding structure), Section 7 (false positive controls), Section 9 (agent architecture), and Section 10.2 (output format).

Kenny's Thinking Garden

Explorer

AI PR Reviewer Skill Research