500xEngineer.

All postsSHORT4 min read

OpenAI's Performance Factory: Codex Agents Investigate ChatGPT Slowdowns

OpenAI says its internal Performance Factory investigates regressions with Codex agents, adversarial review and an engineer deciding what ships.

OpenAI says its Codex agents investigate ChatGPT performance regressions, and a separate session tries to disprove their diagnosis. On Nov 19, 2026, Okay Zed and Cyrus Forbes published the engineering post describing this internal system. An engineer still decides what ships.

Video not loading? Watch the Short on YouTube

What OpenAI described

The OpenAI engineering post is titled "How OpenAI's Performance Factory keeps ChatGPT fast." Its authors, Okay Zed and Cyrus Forbes, are Members of Technical Staff.

OpenAI describes the Performance Factory as an internal system that runs in the cloud and uses Codex agents to investigate performance regressions and propose fixes. Engineers define workflows for recurring jobs and write the instructions the agents follow.

The Regression Workflow starts when a performance alert fires. An agent investigates, records its findings in the task and proposes a fix for an engineer to review. After deployment, an agent checks whether performance has returned to baseline. If there's no improvement, the task goes back for investigation.

OpenAI's numbers

These are separate figures from OpenAI's own account:

  • In just over 3 months, the Regression Workflow tracked over 1,700 investigations and root caused over 600 cases. Root caused means the cause was found. Whether a fix shipped is a separate question.
  • OpenAI puts that at 17 automated investigations every day.
  • In a manual review of 100 root-cause suggestions, 70% graded accurate.

OpenAI also says it accumulated more than 30 performance tools over time. That internal tooling matters when thinking about what another team could reproduce.

How an investigation runs

OpenAI describes four parts: a task tracker with configurable workflows, a dispatch loop that assigns ready work, an API that runs Codex in the cloud, and subscriptions that wake agents when something changes. A review comment or a completed deployment can trigger that wake-up.

Engineers define workflows with configuration files and Markdown prompts. The post's example uses a code-change/ folder with config.toml, work/implementing.md and work/reviewing.md.

The shared investigation skill is called perf-forensics. OpenAI says it captures the performance team's investigation process: which tools are available, when to use them and how to work through a slowdown. It directs the agent to find when the slowdown began, identify affected requests and investigate whether deployments or configuration changes could explain it.

OpenAI tested and refined the skill with evals based on past regressions. The agent runs under a service identity, letting OpenAI scope its permissions and identify actions taken by performance agents.

The second agent tries to disprove the first

OpenAI says agents sometimes stopped at the first plausible explanation, and some diagnoses turned out to be wrong. Its response was adversarial review: a separate Codex session that tries to disprove the proposed diagnosis.

The image-download example makes the check concrete. An agent traced failing ChatGPT conversations to image-download errors and marked the cause as found. The reviewer noticed that the internal service responsible for locating files had already recovered while image requests were still failing. That diagnosis couldn't explain the continuing failures, so the reviewer reopened the investigation.

Once a diagnosis passes adversarial review, the workflow asks Codex to implement a fix and open a pull request. An engineer reviews the PR and decides whether to merge it. After deployment, an agent checks recovery against the earlier baseline. If there's no improvement, it records what happened and sends the regression back for investigation.

The catch

OpenAI writes:

Getting fixes right has been harder than finding the cause.

Some proposed changes were more complicated than necessary or introduced new issues. Architectural decisions also need judgment. OpenAI's example is a cache that could improve performance while introducing stale data. An engineer who knows the service has to weigh that tradeoff.

The Performance Factory is internal. Its API invokes Codex inside OpenAI's network. The post announces no product, release or price. An engineer reviews every fix PR and decides whether to merge. OpenAI describes shipping fixes without a human as something it's exploring.

The agents also need monitoring. OpenAI says a broken profiler or missing permission can stop an agent from verifying the cause. Updating a skill or model can improve some diagnoses while making others worse, so those changes need evaluation.

This is OpenAI's own account of its own system. We haven't seen or tested it, and we haven't independently verified the figures.

What OpenAI says other teams can take from it

OpenAI offers lessons for teams building their own agentic workflows. It emphasizes protecting engineers' attention, evaluating changes to agents and turning recurring failures into repairs the workflow can handle.

For investigations, OpenAI backtests against past production incidents. Those incidents become test cases it can rerun when it changes the skill.

The pattern moves tasks through explicit states. Agents advance a task when more work is ready, separate reviewers challenge conclusions, and engineers step in when the next decision requires judgment.

What you can build with public tools today

The public Codex non-interactive mode docs describe running Codex from scripts with codex exec, without opening the interactive TUI. The docs include pipeline uses such as CI jobs, pre-merge checks and scheduled jobs.

The public Build skills docs describe reusable workflows. A skill is a directory containing a SKILL.md file, plus optional scripts and references.

These docs pages don't describe the Performance Factory. They provide building blocks for your own workflow.

The builder angle: my opinion

I'd sell a small software team an alert-to-diagnosis setup. When a performance alert fires, a coding agent investigates with a written playbook. A second session tries to disprove the diagnosis, and a human reviews the proposed fix.

The deliverables I'd hand over:

  • An investigation playbook written as a skill, tested first on the team's past incidents.
  • A second agent session whose job is to challenge the diagnosis.
  • A pull request plus a short report for a human to review, with nothing merging on its own.

Results will differ from OpenAI's because OpenAI built on its own internal tools and telemetry. Fixes still need a human who knows the service. This is my opinion, with no income guaranteed.

Daily email

Get each breakdown in your inbox.

One email each evening, right after the Short goes live: the AI tool it covers, the full write-up with sources, and how to get paid for it. Free, and you can unsubscribe anytime.

Read past emails on Substack

Sources

All postsWatch on YouTube