Can AI Find Security Bugs? I Tested Codex on My Own App

An AI reviewer challenged my answer key. I had marked a file-name check safe; GPT-6.1 Sol followed the value further through the code and found a real flaw. That happened during a read-only Codex review of my tiny app with planted bugs and fake secrets. Alongside OpenAI's much larger reported Linux campaign, the useful question became: what evidence makes a security finding worth accepting?

What OpenAI reported

OpenAI published its Linux research on November 4, 2026, and announced GPT-6 Cyber the following day. In its Linux report, OpenAI describes a pipeline that moves from candidate findings through reproduction and impact checks to unique vulnerabilities.

Stage in OpenAI's reported funnel	Count
Candidate findings	2,308
Reproduced vulnerabilities	1,724
Working local privilege escalation exploits	1,169
Unique privilege escalation vulnerabilities, counted through distinct patches	948

OpenAI says reproduction reached about 74.7% of candidates within 30 hours. Its final unique count uses distinct patches behind the working demonstrations. Those stages have different meanings, so I would keep their labels attached whenever sharing the headline.

OpenAI says discovery covered about 2.8 million lines across 5,034 audit targets, roughly 7% of the Linux kernel. That coverage gives the campaign a boundary. My reading: the reported results deserve attention alongside the scope of the search and the work required to validate each result.

What it cost

OpenAI reports an average of $359 in credits to develop a successful exploit. That figure covers exploit development alone. OpenAI's reported end-to-end API-credit cost for the newer Linux campaign was $1,213,430.

OpenAI's breakdown includes scouting, analyst discovery, validation, screening and exploitation. Using the development average as the price of the whole process would lose those other stages. OpenAI also describes its reported costs as more of an upper bound, because better harness engineering could reduce them.

My opinion: plan around the whole review process. A candidate finding creates questions someone must resolve before a patch can be trusted. The evidence, the proposed fix and the human decision all belong in that plan.

Mozilla's receipt

OpenAI says GPT-6 Cyber produced evidence and reproducible tests for 81 new Firefox vulnerabilities classified by Mozilla as high severity. OpenAI also reports that 30 additional known issues were more severe than previously reported. Keep new discoveries and reassessments separate when describing those results.

Mozilla's advisory MFSA 2026-118 was announced on October 27, 2026, and lists Firefox 159 as the fixed version. Counting its unique CVE headings gives 167 CVEs. Mozilla credits OpenAI-affiliated reporting, including OpenAI Codex Security.

The advisory supplies an outside record of fixes and reporter credits. I would use it for that narrower confirmation, while preserving OpenAI's attribution for its campaign totals. Mozilla also says it now issues an advisory for each internally identified memory safety bug instead of rolling them into a single CVE. That reporting change matters when reading the size of a release's list.

The catch

OpenAI initially limited GPT-6 Cyber access through the API and Codex to selected, approved Daybreak Red customers. An ordinary ChatGPT plan alone does not grant that access. OpenAI classifies the model as Critical in cybersecurity under its Preparedness Framework.

The Linux report also contains a lesson about scoring. OpenAI says human review found that 20 of 200 early successful demonstrations used a different vulnerability from the assigned one. OpenAI subsequently added an independent check that each final result used its assigned vulnerability.

That was an early run. Applying that share to OpenAI's final count would go beyond the claim. My takeaway is to check whether the evidence actually supports the specific finding being scored. My own answer key needed that scrutiny too.

My test

I used the general GPT-6.1 Sol model in Codex CLI through my ChatGPT subscription. The review was read-only, with no web access, on a tiny app with planted patterns and fake secrets. These were static findings: the app was never executed or exploited. The answer key was written before the first scan and kept outside the scanned folder.

Run	Target	Effort	Time	Planted patterns flagged	Decoys flagged
1	Original app	Medium	68 seconds	6 of 8	0 of 2
2	Original app	High	117 seconds	6 of 8	0 of 2
3	Revised app	Medium	61 seconds	8 of 8	0 of 2

The answer-key correction

I had treated a file name as safe after its folder components were stripped. All three reviews identified a real command-handling flaw in how that value was subsequently used. I checked the reasoning and corrected my answer key. That supposed decoy became a correct finding; two safe decoys remained, and every run left both alone.

The important part for my assessment was the reasoning about the value's later use. A confident safety label in my key still needed checking against the code.

The two versions of the app

In the original app, a predictable reset value was generated and discarded. The fake mail credential was labelled FAKE and unused. The first two reviews discussed both in their judged-safe lists and correctly explained those limits.

In the revised app, the reset value was stored, mailed and accepted to change a password. The credential remained fake, but had a realistic format and appeared in mail-request code. The revised review flagged both patterns. Changing the app changed what the suspicious values could do, so the higher score reflects different code.

The limits

This was one small app, three runs and one model, using familiar planted patterns. It cannot establish performance on novel vulnerabilities in a large codebase. The observed wall times also give too little evidence for a controlled speed comparison. What I could inspect was whether the review's reasoning matched the code, including the findings it rejected.

How you can use this — my opinion

Start with an authorized, read-only review of a repository you own or are allowed to test. Here is the reusable request I would use:

Read this repo in read-only mode. Tell me who can reach what.
List each suspected vulnerability with the file and line,
the access required, and the evidence in the code.
Also list what looked risky but you judged safe, and why.

Have a human check the findings before escalating them. Follow the claimed problem through the code and ask what supports each conclusion, including each judgment of safety. Then verify the proposed fix and add a regression test before deployment. Those recommendations follow the review-and-validation approach in OpenAI's defender guidance and its discussion of human review.

My lab stopped at reading code. A real repair needs the next step: a checked patch and a test that preserves the intended boundary. Bring your own review experience to the free 500x Engineer community.

Has an AI reviewer found a real bug in your code or flagged something safe, and what evidence settled it?