Vanilla Claude vs Our Best Skill. Here's What Actually Happened.

Composite — 3 rounds, 3 contestants. Each block = 1 quality point.

🎯 The Question We Had to Answer

We'd been building execution skills for months. Godmode, Godmode+, One-Shot — layers of instructions that tell Claude how to build, test, and verify code.

But we'd never tested the obvious question: does any of this actually help?

Maybe raw Claude Code — no skill, no protocol, just "build this" — produces the same quality. Maybe our elaborate 8-phase execution protocol is just expensive overhead. We had to know.

🔬 The Experiment

Three tasks. Three sizes. Three competitors. Same rubric every time.

The tasks scaled in complexity:

Each output was graded on 8 dimensions: code quality, test coverage, security, completeness, process, documentation, polish, and decision quality. Weighted. Blind where possible.

📊 The Scoreboard

Here's every round, every score, every competitor.

Round 1 — Small

Round 2 — Medium

Round 3 — Large

Tests written per cell. Watch Round 3 Monolith collapse — that pillar is the failure mode.

💥 The Result Nobody Expected

Here are the averages across all three rounds:

Scripts won every single round. That part wasn't surprising — we'd already seen that in the A/B test.

What was surprising: vanilla Claude writes more tests on average than our monolith skill. 22 tests vs 20. On the large task, it's 30 vs zero.

Our skill's phase overhead — the scorecards, delivery reports, process labels — was consuming the "attention budget" that would otherwise go to writing tests. The model spent so much time following protocol that it forgot the most important part.

🔧 Why Scripts Beat Everything

The monolith puts every instruction in one big file. By the time Claude is deep into building, the "write tests" instruction is paragraph 12 in a wall of text. Easy to skip.

The scripts model loads each phase from its own file, when that phase starts. Testing isn't buried — it's a standalone document with its own clear instructions and exit criteria.

Three structural advantages showed up in every round:

🏗 How We Got Here: The Journey

The scripts model didn't appear from nowhere. It emerged from failures.

The conventional wisdom was that splitting a skill into separate files would hurt performance. More handoffs, more file reads, more chances to lose context. That intuition was completely wrong.

What actually happens: each phase gets fresher, more focused context instead of competing with a wall of instructions the model has been staring at for 10 minutes.

💰 What It Costs

Better quality isn't free. Here's the price tag at Opus API pricing ($15/$75 per million input/output tokens):

Scripts costs 3x vanilla. But the quality jump is enormous — from "probably has bugs" to "tested, secure, documented, verified."

The real question isn't "is $16 too much?" It's "how much does rework cost?" If you ship vanilla output and spend 20 minutes manually adding tests and reviewing for security, the skill already paid for itself.

Cost climbs left→right. Quality climbs up. Only Scripts crosses the 0.85 ship-ready plane.

🛡 Where Each Approach Wins

This isn't a simple "scripts is always best" story. Each approach has a use case.

The monolith? Honestly, the data says it's the worst of both worlds. Slower than vanilla but doesn't reliably produce tests. More ceremony but less substance.

🧠 The Deeper Lesson: Architecture Matters More Than Content

The monolith and scripts contain identical instructions. Same phases. Same scoring rubric. Same delivery format. The only difference is how they're packaged.

That packaging difference produces a +0.15 composite quality gap. Consistently. Across all task sizes.

The scripts model also has 7-13x more headroom before hitting its ceiling. The monolith is already near its architectural limit — you can't cram much more into one file before instructions start getting ignored.

Hover any brick in the Scripts tower to read its filename and line count.

✅ What We Shipped

Based on these results, we've adopted the modular scripts architecture as the default for all Godmode execution skills. Here's what changed:

• One-Shot Scripts is now the primary execution skill. 16 files, ~11,600 tokens, 0.92 average composite.
• The monolith stays available as a fallback for edge cases where script loading might fail.
• The A/B grader skill we built to run this experiment is now a permanent tool for testing future skill changes.
• Every future skill improvement gets measured against vanilla baseline. If it doesn't beat raw Claude, it doesn't ship.

The irony isn't lost on us. We built a complex 8-phase protocol, discovered it was barely better than doing nothing on the dimension that matters most (testing), then fixed it by making the architecture more complex (16 files instead of 1). But it's the right kind of complexity — modular, focused, measurable.

We failed our way to something that genuinely outperforms vanilla Claude Code. That's the whole story.