Code Review Benchmarks
How well does Tenki do versus other
AI code reviewers in catching real bugs?
Last updated: May 20, 2026
TL;DR
Tenki is the #1 reviewer based on finding-level scoring.
In this independent 2026 benchmark of 6 leading AI code review tools, Tenki catches 84 of 122 real, production bugs across 50 pull requests from cal.com, Sentry, Grafana, Keycloak, and Discourse, graded per finding by a 3-judge LLM panel. That's 1.9× the next-best reviewer.
84/122
Real Bugs Caught
68.9%
Catch Rate
3
LLM Judges
7
Tools
5
Repositories
350
Code Reviews
Methodology
How we benchmarked AI code reviewers.
Every pull request contains a real, merged bug-fix from an open-source codebase. We replay the bug-introducing diff into a clean fork and let each tool review it with its default configuration. Reviews are then graded by a 3-LLM judge panel using majority vote. No synthetic bugs, no repo-specific tuning.
Sources
50 bug-fix PRs from 5 open-source repositories
Real merged fixes from cal.com (TypeScript), sentry (Python), grafana (Go), keycloak (Java), and discourse (Ruby). All five major server-side languages are represented in the bug set.
Replay
Real bugs reintroduced, not invented
The pre-fix diff is replayed against every tool with default settings: no custom rules, no repo-specific tuning, and full repository context for every reviewer. Every tool sees the same code at the same point in history.
Scoring
Per-finding 3-judge LLM majority vote
Bugs are scored individually, not per-PR (which would over-credit drive-by comments). A bug counts as caught only if a line-level comment pinpoints the faulty code and explains its impact, and at least two of three independent LLM judges agree.
Overall performance
Tenki leads recall and F1 across every real bug.
Each individual bug is scored, not each pull request. Data is sorted by F1 and tools were kept at their default configurations. Higher-precision tools post fewer comments overall; Tenki's higher comment volume drives both higher recall and lower precision. See methodology for how each metric is computed.
Reviewer
Recall
Precision
F1
Tenki
68.9%
29.9%
41.7
CodeRabbit
28.7%
25.0%
26.7
Greptile
36.1%
15.9%
22.1
Copilot
24.6%
18.9%
21.4
Graphite
3.3%
50.0%
6.2
Coding agents
Devin
36.1%
47.3%
40.9
Cursor
32.0%
51.3%
39.4
By Severity
Do AI code reviewers catch the bugs that actually matter?
Catch rate broken down by the severity of the individual finding. Critical bugs cause outages, data loss, or auth bypass. High-severity bugs break major user-facing flows. Medium bugs degrade behavior without breaking it. The severity that matters most for production reliability is the first column.
Tenki69%(84/122 Bugs)
Devin36%(44/122 Bugs)
Greptile36%(44/122 Bugs)
Cursor32%(39/122 Bugs)
CodeRabbit29%(35/122 Bugs)
Copilot25%(30/122 Bugs)
Graphite3%(4/122 Bugs)
By Repository
How each reviewer performs across five production codebases.
Per-repository recall: the share of real bugs each AI code reviewer caught in each codebase.
Tenki91%(30/33 Bugs)
Devin33%(11/33 Bugs)
Cursor30%(10/33 Bugs)
Greptile30%(10/33 Bugs)
CodeRabbit21%(7/33 Bugs)
Copilot21%(7/33 Bugs)
Graphite0%(0/33 Bugs)
Case Library
Every real bug, every reviewer verdict.
One row per real bug, 122 findings across 50 production pull requests. Each cell shows whether that reviewer flagged that specific defect, decided by 3-LLM majority vote. Click any verdict to see the actual review on GitHub.
Caught Missed
deleteCacheHandler throws generic Error → tRPC surfaces as 500 instead of 403/404
Medium
Cal.com
Cal.comCheckbox fires onChange twice per click via redundant onClick + onCheckedChange handlers
LowCal.com
Cal.com`getPendingActions` never shows confirm button for paid payment-enabled bookings
HighCal.com
Cal.comMissing `revalidateTag('team-features')` leaves settings layout cache stale after role update
MediumCal.com
Cal.comPast pending-unconfirmed bookings lose cancel/edit actions
HighCal.com
Cal.comFunctional regression: 'Check for recordings' action replaced with disabled button
MediumCal.com
Cal.comafterNthPaintCycle fires callback after n+1 frames, not n frames
LowCal.com
Cal.comno_show action always included in afterEventActions, shown disabled for upcoming bookings
MediumCal.com
Cal.comCharge card action hidden for recurring bookings in recurring tab
MediumCal.com
Cal.comforEach with async callback fire-and-forgets calendar/video deletions
HighCal.com
Cal.comBackup-code login bypasses password verification
CriticalCal.com
Cal.comUnguarded mainHostDestinationCalendar.integration access crashes booking
HighCal.com
Cal.comSilent validation failure in constructor — invalid options return empty results, not errors
MediumCal.com
Cal.comDead branches in getBaseConditions — `else if (filterConditions)` and `else` unreachable
LowCal.com
Cal.comimmediateDelete path does not update the WorkflowReminder DB row
HighCal.com
Cal.comDefault (non-immediate) path marks DB cancelled but never calls SendGrid
HighCal.com
Cal.comWorking-hours `end` check uses slotStartTime, missing the end-of-slot boundary
HighCal.com
Cal.comDayjs object compared with `===` instead of `.isSame()` — always `false`
HighCal.com
Cal.comUTC offset sign is inverted when checking date-override day match
HighCal.com
Cal.comMissing guard for absent `timeZone` in `getSlots` override offset calculation
HighCal.com
Cal.comWorking-hours check does not reject slots on days with no working-hour entries
HighCal.com
Cal.com`availabilityCheckProps` does not include `dateOverrides` / `workingHours`, creating inconsistent behaviour between fixed and loose hosts
MediumCal.com
Cal.comWorking-hours day-of-week comparison uses UTC day instead of organizer timezone day
MediumCal.com
Cal.comGoogleCalendarService stores safeParse wrapper {success,data} as credential key
HighCal.com
Cal.comMissing `prisma` import in SalesforceCalendarService causes runtime crash
CriticalCal.com
Cal.comComputed property keys in minimumTokenResponseSchema use Zod schema objects, not string values
HighCal.com
Cal.comrefreshOAuthTokens returns incompatible types: Response vs axios response
HighCal.com
Cal.comWebhook secret compared with non-constant-time string equality
HighCal.com
Cal.comZohoCRM token expiry calculation adds seconds instead of milliseconds
MediumCal.com
Cal.comUnhandled `ZodError` from `.parse()` returns 500 instead of 400 for invalid request body
MediumCal.com
Cal.comOutbound credential sync request carries no authentication — sync endpoint response is implicitly trusted
MediumCal.com
Cal.comSMS-reminder cleanup deletes non-SMS reminders with retryCount > 1
HighCal.com
Cal.comisTeamAdminOrOwner uses && instead of || — check is unsatisfiable
HighCal.com
Cal.comDuplicate self.downsize definition silently overrides 4-arg signature; existing callers crash
High
Discourse
DiscourseUnsubscribe endpoint NoMethodError when no TopicUser row exists
HighDiscourse
DiscourseCSRF: GET request permanently mutates topic notification state
HighDiscourse
DiscourseMissing `unsubscribe_url` in `template_args` causes `I18n::MissingInterpolationArgument` for non-notification callers
HighDiscourse
DiscourseDuplicate `%{respond_instructions}` placeholder in email notification template causes text to appear twice
MediumDiscourse
Discourseemail_in_restriction_setting? regex has no end anchor — whitelist bypass via domain suffix
HighDiscourse
Discoursebest.html.erb closes if/else with `<%- end if %>` — invalid Ruby, view crashes
CriticalDiscourse
DiscourseSSRF via user-controlled embed_url triggers open() to arbitrary URL
CriticalDiscourse
DiscourseSSRF and RCE via Kernel#open with attacker-controlled URL from Disqus XML
CriticalDiscourse
DiscourseNon-atomic Redis SETNX + EXPIRE creates permanent throttle lock on crash
HighDiscourse
Discourse`invalid_host?` silently blocks all retrieval when `embeddable_host` is blank
HighDiscourse
DiscourseXSS via unsanitized Referer header injected into JavaScript postMessage targetOrigin
HighDiscourse
Discourse`i.content` in PollFeed may be nil, causing NoMethodError on `.scrub`
HighDiscourse
Discourse`save_reply_relationships` called unconditionally even when `save` fails
MediumDiscourse
DiscoursePort-stripping logic in `absolutize_urls` uses hardcoded ports instead of scheme-relative defaults
MediumDiscourse
DiscourseSynchronous `PollFeed.new.execute({})` blocks caller and polls entire feed for one URL
MediumDiscourse
Discourse.title float removed without flex replacement; child floats now ignored
MediumDiscourse
DiscourseDestructive String#<< on literal in website_name domain check
MediumDiscourse
DiscourseLight-theme lightness values silently changed during scale-color → dark-light-choose conversion
MediumDiscourse
Discourseremove_member spec uses PUT but the route is DELETE
MediumDiscourse
DiscourseI18n Fallbacks included on backend.class instead of I18n::Backend::Simple
MediumDiscourse
Discoursecategory_fabricator.rb and embeddable_host_fabricator.rb contents swapped
MediumDiscourse
DiscourserowsAffected==0 misclassified as ErrDeviceLimitReached
Medium
Grafana
GrafanaTOCTOU race between CountDevices and INSERT
MediumGrafana
GrafanaStale denial cache blocks newly granted permissions
MediumGrafana
GrafanaIn-process authz client loses 30s client-side cache
LowGrafana
GrafanaStream and admission requests lose contextual logging
MediumGrafana
GrafanaEmpty-after-interpolation queries bypass expr filter (shardQuerySplitting)
MediumGrafana
GrafanaFilter precedes interpolation in querySplitting too
MediumGrafana
GrafanaSwitch from interpolateVariablesInQueries to applyTemplateVariables changes filter semantics
LowGrafana
GrafanaMissing React key prop on GrafanaRuleListItem in FilterView
MediumGrafana
GrafanaSilence drawer never opens for Grafana rules in v2 list
MediumGrafana
GrafanaStorage.Create error records legacy duration
MediumGrafana
GrafanaStorage.Update error records legacy duration
MediumGrafana
GrafanaDeleteCollection async legacy goroutine records storage duration
MediumGrafana
GrafanaDelete success records `name` instead of `options.Kind`
LowGrafana
GrafanaDelete passes d.Log instead of contextual log to klog.NewContext
LowGrafana
GrafanaCleanUpService ticker reduced from 10 minutes to 1 minute
HighGrafana
GrafanaRoutine cleanup progress logged at Error level
MediumGrafana
GrafanaMissing double-checked lock in GetWebAssets
MediumGrafana
GrafanaenableSqlExpressions is dead code; FlagSqlExpressions feature flag is non-functional
HighGrafana
GrafanaBuildIndex no longer serializes concurrent builds for the same key
MediumGrafana
GrafanaUsernameForm.authenticate calls isConditionalPasskeysEnabled() with no args; compile error
High
Keycloak
KeycloakCache-hit path of getForLogin skips createOrganizationAwareIdentityProviderModel wrapping
HighKeycloak
KeycloakOrganizationAwareIdentityProviderBean adds defensive isEnabled() re-checks
LowKeycloak
KeycloakCryptoProvider.order() added without default — breaks third-party SPI implementations
HighKeycloak
KeycloakAll production CryptoProviders share the same order value — FIPS provider can be silently bypassed
HighKeycloak
Keycloak`readSequence` silently returns empty list for indefinite-length encoded sequences
MediumKeycloak
KeycloakRECREATE_UPGRADE_EXIT_CODE silently changed from 4 to 3 (breaking external contract)
MediumKeycloak
KeycloakCleanup listener guarded by V1 flag while rest of class uses V2
HighKeycloak
KeycloakGroupPermissionsV2.canManage() accepts VIEW scope — privilege escalation
CriticalKeycloak
KeycloakGroupPermissionsV2 adds authorization resource IDs instead of group UUIDs to view-permission filter, breaking group-scoped admin user visibility
HighKeycloak
KeycloakLithuanian totpStep1/loginTotpStep1 strings replaced with Italian text
MediumKeycloak
KeycloakAccessTokenContext constructor null-checks grantType twice instead of rawTokenId
MediumKeycloak
KeycloakisAccessTokenId test matcher inverts the grant-shortcut check
MediumKeycloak
KeycloakOptional<CredentialModel>.get() called without isPresent check in login bean
MediumKeycloak
KeycloakgetSubGroupsCount returns null instead of delegating like sibling methods
HighKeycloak
KeycloakOptimizedCursorPaginator.get_item_key TypeError on datetime values
High
Sentry
SentryNegative-offset queryset slicing raises AssertionError
HighSentry
SentryAttributeError on organization_context.member when None
MediumSentry
Sentryenable_advanced gate is effectively a no-op (has_global_access defaults True)
MediumSentry
SentryRedirect-chain iteration limit reduced from 10000 to 1000 in add-buffer.lua
HighSentry
Sentrymax_segment_spans safety check removed from flush path
HighSentry
SentryOptimizedCursorPaginator.get_item_key TypeError on datetime values
HighSentry
SentryNegative-offset queryset slicing raises AssertionError
HighSentry
SentryAudit-logs date-range filtering removed; permission gate too permissive
MediumSentry
SentryUpsampling check uses outer `dataset` instead of `scoped_dataset`
HighSentry
Sentry`timestamp` raw_groupby validation silently removed
MediumSentry
SentryMisleading 'cached' comments around upsampling decision
LowSentry
SentryOAuth state parameter is a deterministic pipeline.signature, not a per-session nonce
CriticalSentry
Sentryfetch_error_details mispairs error IDs to payloads via zip(error_ids, events.values())
HighSentry
Sentryvalidate_timestamp/validate_age silently skip mutual-exclusion check when age=0 or timestamp=0
HighSentry
SentryWrong key 'detector_type' in update() prevents type from ever being updated
HighSentry
SentryFeature-flagged TableWidgetVisualization renders with empty data instead of query results
HighSentry
Sentrydrain_mailbox_parallel processing_deadline_duration (120s) is shorter than its internal loop timeout (180s)
HighSentry
SentryOrganization slug missing from queryKey causes cross-org cache hits
HighSentry
SentryRemoved `retry: false` causes unintended request retries for attribute key fetches
MediumSentry
SentryInconsistent optional chaining in tableData?.meta.fields can throw at runtime
MediumSentry
SentryHook returns `isFetching` labeled as `isLoading`, causing misleading loading state during background refetches
MediumSentry
SentryGlobal max-flush-segments cap is multiplied by flusher process count
HighSentry
SentryRedis cluster memory polled redundantly once per flusher process buffer
MediumSentry
SentryAssignmentSource.queued default is evaluated at class-definition time
MediumSentry
Sentryschedule_type display transformation is dropped on return
MediumSentry
SentryFixedQueuePool.shutdown signals workers before draining queues, dropping messages
HighSentry
Sentryresult_processor moved before block-size config in ResultsStrategyFactory.__init__
LowSentry
Sentry