I Let AI Agents Run My Content Business for a Month

For about a month I've let AI agents run my content business. They research, write, check and ship the work, a lot of it while I sleep. Several creators posted AI factory videos this week, so this episode is the receipts from mine: what it read, where it broke, and the one thing that decides how far you can scale agents. That thing is trust, and trust comes from a strict playbook the agents follow every time, with checks that prove they did.

What a software factory is

A software factory is a crowd of AI agents taking tasks off a list, doing them, getting checked, and shipping, while a human manages the line. The public examples sit far apart. Stripe says over a thousand pull requests merged each week are written entirely by its agents, with humans reviewing them. Simon Willison wrote that StrongDM's AI team runs on two rules: humans don't write the code and humans don't review it.

That spread is really a trust dial. How much you let agents ship on their own depends on how much you trust them.

Mine makes videos. A queue item is a ticket, a finished Short is a pull request, the fact-check is the code review, and publishing is the merge. Links (or news the AI picks) go into a queue. A producer agent wakes six times a night and makes one Short per run. A fresh checker agent that never saw the draft verifies every claim. Plain code checks the length, the loudness and the files. The Short lands on my phone, and a scheduler agent uploads it through Chrome. Since October 31, 25 items came through the queue: 21 were approved and 4 were stopped at a gate.

The agent that wouldn't stop

I started one agent on October 31 with a one-page brief: add an end bumper and some more animation to one video. Over the next three days more than a hundred of my messages landed in its conversation: questions about other projects, approvals, whole new tasks. It started 35 other agents, made 1,276 model calls and read 675 million tokens.

A token is a word-piece, the unit an AI is billed and limited in. The count gets huge because at every step an agent re-reads its whole history. Caching makes that cheap (about 90% of those tokens were cache reads), but a long run keeps growing and keeps getting cached again.

My plans are flat, so I never got a bill for this. Priced at Opus 5.5 API list prices ($0.20 per million tokens for a cache read, $5 per million to write the cache, $20 per million for output), that one agent comes to about $470. The cache writes were a tenth of the tokens and about 70% of the cost.

Across the whole floor: 338 background agents between Oct 23 and Nov 3 read about 3.66 billion tokens, roughly $2,050 at API list price, in 11 days. The median agent read 3.2 million tokens. Seven agents read 44% of everything.

The fix went into the playbook as a handoff note with four rules: short, focused agents with a cheaper model for watching renders; no polling; small images; five-line reports. Honest status: over 16,000 calls ran on Opus and fewer than a hundred on the cheaper models, so that rule is brand new.

Limits hit everyone at once

On flat plans, waste shows up as limits. I run a separate coding fleet, and on October 3 a usage limit stopped every worker at the same moment, mid-task, for three hours. That night 93% of the usage came from sessions heavy with subagents, and two thirds from sessions past 150,000 tokens of context.

Four more rules: check usage before adding agents and ramp only when the week is under pace; hand a worker off at about 150,000 tokens; start at most three new agents every two minutes (one burst of spawns pushed my machine's load to 155); and never ask GitHub for updates in a loop.

Success that wasn't

On November 1 the Chrome profile my scheduler uses went offline. The scheduler agent tried to upload one Short 22 times over about ten hours. Twenty-one runs finished with a green exit code, and twenty of them uploaded nothing.

The factory caught it anyway, because finishing is a separate step: the agent has to report an outcome (uploaded with the video's ID, or offline), and plain code reads that report. A green exit code only says the process ended. Rule: verify the result, never the exit code.

Facts work the same way. The hook I first chose for episode 4 claimed zero code reviews; a fresh checker showed the agents had reviewed each other, and we changed it. A news Short was stopped after three failed fact-check rounds.

The gate that faded

At first every Short waited for my tap. Then on November 1 the producer skipped three runs in a row because four Shorts were waiting on me. So now, if I don't answer within two hours, the Short is approved. The last eleven Shorts all approved themselves.

That choice has a cost. The human check moved earlier, into the fact-checker and the code checks, and trust got built there. Reject still stops a Short. Three things never approve themselves: long-form videos like this one, anything published to GitHub, and spending money.

The playbook

The five rules on my desk:

1. One agent, one job. Long conversations get expensive.
2. Budget before speed. Limits hit everyone at once.
3. Verify the result, never the exit code.
4. A separate checker for every claim.
5. Gate the expensive stuff.

I built this episode with Claude Sonnet 5.5 as the lead agent, and two fresh Sonnet agents fact-checked the script. The first round caught several numbers I had wrong or overstated, which is the separate checker doing its job.

Build your own

Start smaller than you think: one list, one agent on a timer doing one kind of job, one separate checker, and one place where you say yes or no. That works for a codebase, a newsletter, or a school club's posts. Add stations when the first one gets boring.

For a head start, zStack, the multi-agent setup from episode 4, is free and open source. A premium course on using it is coming to the 500x Skool, and the free community has starter material now.

Join the free community: 500xengineer.com/skool.