500xEngineer.

All postsSHORT3 min read

CodeRabbit Verify: AI Testing Finds a Demo Checkout Bug

CodeRabbit Verify entered public beta Nov 18, 2026. Its demo catches a simulated charge mismatch. How PR runs work and what to check before setup.

CodeRabbit's demo checkout shows $80 and records a $100 simulated charge. Its AI tester catches the mismatch. On Nov 18, 2026, CodeRabbit announced Verify in public beta. The example uses a demo app with no real payments.

Video not loading? Watch the Short on YouTube

What CodeRabbit launched

CodeRabbit Verify starts your app in an isolated sandbox and uses it the way a user would, mostly through a browser. It can also use the app's CLI or HTTP API when the change calls for that.

A run produces evidence you can inspect: results, screenshots and session recordings. You can watch it work in a live browser view. CodeRabbit's launch post on X had over 1.5 million views when fetched Nov 20, 2026.

The demo bug a shallow test would miss

In CodeRabbit's own demo, applying SAVE20 to a $100 order should reduce the total to $80. The confirmation displays $80 correctly. But the app still records a $100 simulated charge. There are no real payments.

A test that stops after the coupon is accepted or the confirmation appears would miss the bug. The recorded charge is the part that exposes the mismatch.

The coupon demo reports 2 bugs and 1 passed scenario. Those counts describe that demo run.

How a pull request run works

Once the repo and test environment are ready, comment:

@coderabbitai verify

Verify writes test scenarios from the pull request's changes, runs them against the app, and posts a comment with results and links to evidence. The links open Change Stack, where you sign in and your repository access is checked. Expected results come from intended behavior, docs and existing tests, “so a new implementation doesn't get to define its own correctness.”

You can add an instruction after the command. The docs give this example:

@coderabbitai verify check that the login form shows an error after a wrong password

Automatic runs can start when a pull request opens or gets new commits, if you enable them. They skip drafts, bot-opened pull requests, closed and merged pull requests, and public GitHub repositories. Use the comment to start a run on a public GitHub repo.

User journeys and schedules

Verify can inspect the repository and propose user journeys in plain English, such as signup and checkout. Approved journeys become a reusable regression library.

Run selected journeys on demand, group them with tags, or schedule them against a branch and test environment. The journey docs list daily, weekly, monthly and interval schedules. Each repo's library holds up to 50 journeys.

Findings can go to CodeRabbit Agent for a fix, to your preferred coding agent, or become tickets in a connected Jira or Linear tracker. Watch the recording before acting on a finding. Starting a Coding Agent task requires you to review and start it yourself.

What it costs

Verify runs are free during the beta.

The catch

This is a public beta. Check the access requirements and setup docs before preparing an environment.

  • Platforms: GitHub Cloud, GitLab.com and Bitbucket Cloud. GitHub Enterprise Server, self-managed GitLab, Bitbucket Data Center, Azure DevOps and self-hosted CodeRabbit aren't supported.
  • App types: web apps and services. Native mobile apps aren't supported.
  • Setup permissions: preparing the environment needs an org Admin with a Review seat or a trial. Coding Agent and data retention must be enabled for the organization.
  • Startup: write a startup script that applies migrations, seeds test data and starts every service the app needs.
  • Secrets and recordings: during a run, the tester can read the test environment's variables and secrets. Screenshots and session recordings aren't redacted. Use test accounts and non-production values only.
  • Results: Passed means the assessed criteria passed within the tested scope. Other parts of the application may remain untested.

The builder angle: my opinion

I'd offer small SaaS teams a monthly money-flow watch. Map signup, checkout and billing as user journeys, prepare seeded test data, then schedule daily runs with test accounts and non-production values.

Send a short weekly report covering bugs, blockers and fixes. Include links to the evidence so the team can inspect the recording and decide what to do next. A passed run belongs to its tested scope, so keep that scope visible in the report.

This is my opinion, with no income guaranteed.

Daily email

Get each breakdown in your inbox.

One email each evening, right after the Short goes live: the AI tool it covers, the full write-up with sources, and how to get paid for it. Free, and you can unsubscribe anytime.

Read past emails on Substack

Sources

All postsWatch on YouTube