My AI Pipeline Shipped Fast. Then I Measured the Output.

Autonomous AI pipelines fail not because they can't write code, but because they optimize for speed without measuring quality. After 22 pipeline runs with a 10% success rate and 78% correct lines of code, I rebuilt the system from scratch: 9 phases, 10 specialized agents, a 3-dimensional quality framework, and a standalone pipeline engine that works with any AI coding tool. The speed was real. The quality wasn't.

In Part 2 of this series, I ended with a number: 25 minutes to ship a full-stack feature. Two parallel workers, independent reviewers, CI green on first attempt. That number was accurate. What I didn't say, because I didn't know yet: that was one of roughly two successful runs out of twenty-two attempts. The other twenty produced merge conflicts, scope violations, half-implemented features, and code that looked correct until you actually ran it.

This is Part 3. Part 1 covered the infrastructure that keeps AI useful across sessions. Part 2 covered nine failure versions that turned into an autonomous pipeline. This part covers what happened when I stopped celebrating speed and started measuring what came out the other end. And Part 4 will cover the next evolution: teaching the pipeline to improve itself between runs.

The number that changed everything

Across 22 pipeline runs over two weeks, 2 completed successfully, 1 partially completed, and 19 failed at various stages. That's roughly a 10% success rate.

Anthropic's 2026 Agentic Coding Trends Report found that developers use AI in about 60% of their work but can fully delegate only 0 to 20% of tasks. The gap between "using AI" and "letting AI run autonomously" is wider than most people realize. My pipeline was operating in that gap.

The failures clustered into patterns. I ran a deep audit of one specific run where the pipeline had shipped successfully: 8 parallel tracks, all code merged, CI green, deployed to staging. On the surface, a triumph. When I tested the feature on staging the next morning, two of three core acceptance criteria failed. The deployed code had the right functions in the right files. It just didn't work correctly at runtime.

That audit revealed something I should have seen earlier: out of 144 lines of code the pipeline produced, 112 were correct. 78%. Three bugs required manual fixes that took 15 minutes. The pipeline run itself had taken 47 minutes and cost $6.50. An experienced developer could have done the entire feature in 2 to 3 hours with zero bugs.

The pipeline was optimizing for the wrong thing. It was fast at producing code. It was not good at producing correct code.

CodeRabbit's analysis of 470 real-world pull requests found that AI-generated code introduces 1.7 times more issues than human-written code, with 75% more logic and correctness errors. Those aren't typos or formatting issues. They're the kind of bugs where the code looks right, passes a quick review, and breaks under real conditions. That matched my experience exactly.

What QA actually verified

The root cause wasn't that the AI wrote bad code. It was that the quality system verified the wrong things.

My QA phase had a 4-level verification model: Static (does the code exist?), Call Graph (is the code referenced?), Behavior (does the code run under the right conditions?), and Outcome (does it produce the correct result in the live system?). On paper, solid. In practice, the QA agent consistently stopped at level 2 or 3. It would confirm that a pricing function existed and was called from the right place. It never actually ran the function with real data to see if the output was correct.

One bug was a model pricing lookup that used exact string matching against a table of five entries. The AI provider returned model names with suffixes. None of them matched. Every API call returned a cost of zero. The code was syntactically correct, properly typed, cleanly integrated. It just silently failed at runtime because the lookup logic had never been tested against actual API responses.

Cortex's 2026 Engineering Benchmark Report found that while pull requests per author increased 20% year over year, incidents per pull request rose 23.5% and change failure rates climbed roughly 30%. More code shipped, more things broke. The pipeline amplified exactly this pattern.

The Quality Gate, the final checkpoint before deployment, had a clause in its decision logic: ship if the retro quality score is above 70 or if the retro report is not available. That "or" is the kind of escape hatch that sounds reasonable and quietly undermines the entire system. When the Retro Analyst timed out (its 5-minute window was too short for complex runs), the Quality Gate shipped anyway, missing data and all.

Code that exists is not code that works. And a quality gate that accepts missing data is not a quality gate.

From 2 reviews to 10 agents

Part 2 described a system with three quality gates and three models. The system I run now has 10 specialized agent roles across 9 phases, each with a defined scope, defined inputs and outputs, and defined failure modes.

The Anthropic Agentic Coding Trends Report describes this as the dominant pattern for 2026: single agents evolving into coordinated multi-agent systems where an orchestrator coordinates specialists working in parallel. That's the architecture, but the report understandably focuses on the capability. What it can't cover is the operational reality: every additional agent is another failure surface.

Here's what the 9-phase model looks like now:

Phase	Name	Agent	What it does
1	Understand	Product Strategist	Validates the problem, gathers context
2	Define	Product Strategist	Writes spec with testable acceptance criteria
3	Design	Product Strategist + Human	Design brief for UI features
4	Plan	Tech Lead	Codebase analysis, atomic task breakdown with disjoint file scopes
5	Build	Developer (parallel)	Implementation in isolated git worktrees
6a	Verify	QA Engineer	AC verification, scope-violation check, minor fixes
9	Learn	Retro Analyst	Run analysis, 3-dimension quality scoring
6b	Quality Gate	Quality Gate	SHIP / FIX / REJECT decision based on QA + Retro
7	Ship	Release Engineer	Deploy, CI check, smoke test

On failure at Phase 5 or later, a Closer Agent rescues partial work to a separate branch so nothing gets lost.

The phase numbering looks odd because it is. Phase 9 (Learn) runs before Phase 6b (Quality Gate) so the retro report can inform the ship-or-reject decision. The Quality Gate now receives three inputs: the QA report, the retro analysis with quality scores, and the solution specification as business reference. No more shipping with missing data.

Stack Overflow's engineering blog described the core problem with autonomous agents precisely: as context fills up, agents start forgetting their task lists and checking off items that aren't done. The specialized agents address this by keeping each agent's context narrow. The Tech Lead only plans. Developers only implement their assigned files. The QA Engineer only verifies. No agent has to hold the full picture in its context window, because no agent needs to.

The extraction nobody asked for

The more fundamental change was pulling the pipeline out of the product codebase entirely.

In Part 2, the Vibe Machine lived inside the same repository it was building. Agent definitions, rules, hooks, slash commands, all co-located with the application code. That worked fine when there was one product. It stopped working when I started building a second one.

Seventy-three artifacts needed to move. I classified every command, skill, hook, agent definition, and rule as either universal (works in any repository) or repo-specific (tied to a particular product). The universal artifacts migrated to a standalone repository. The repo-specific ones stayed where they were. A CLI tool (vm install) creates symlinks from the central repository to your local Claude configuration, so the pipeline toolchain loads automatically regardless of which project you're working in.

The more consequential decision was making the pipeline engine CLI-agnostic and tracker-agnostic. The engine doesn't know or care whether it's driving Claude Code, Gemini CLI, or OpenAI Codex. It talks to an abstract CLI adapter. It doesn't know whether tasks live in Vibe Kanban, Linear, or Jira. It talks to an abstract tracker adapter.

This wasn't altruism. It was survival. I had been burned by tight coupling to specific tools before, and the AI coding landscape is moving fast enough that locking into one tool's CLI is a liability. Anthropic's report notes that average Claude Code session length has grown from 4 minutes in the autocomplete era to 23 minutes in the agentic era, with 47 tool calls per session. Sessions are getting longer, more complex, and more dependent on the surrounding infrastructure. That infrastructure should be portable.

The pipeline now runs as a FastAPI service with an MCP interface. You start a run by passing a solution issue ID. It handles everything from Phase 4 through Phase 7 autonomously: planning, parallel worker execution in git worktrees, QA, retrospective, quality gate, merge, and deployment. A React dashboard shows real-time status via WebSocket.

Building the system that builds your product is at least as much work as building the product itself. That's been true since Part 1. The difference now is that the builder is becoming its own thing.

Measuring quality in three dimensions

The single most impactful change was giving the system a way to measure itself.

The Retro Analyst now scores every pipeline run across three dimensions: input quality (was the specification clear enough?), throughput quality (did the pipeline execute cleanly?), and output quality (does the code actually work?). Each dimension has weighted sub-metrics. The combined score determines whether the Quality Gate can ship.

Dimension	Weight	What it measures
Input	30%	IST-Zustand present, spec referenced, ACs testable, constraints complete
Throughput	40%	Gate pass rate, merge success, scope violations, worker completion
Output	30%	TypeScript clean, ACs fulfilled, no manual rework needed, business outcome verified

Before this framework, my "quality metric" was binary: did it work when I tested it in the morning? Now there's a number. And numbers create pressure. When a run scores 68.5 out of 100, you can see exactly where the points were lost: QA skipped Phase 4 outcome verification (output score hit), retro timed out (throughput score hit), solution lacked a proper current-state analysis (input score hit).

CodeRabbit's blog post summarizing their research stated it directly: 2025 was the year of AI speed, 2026 will be the year of AI quality. That framing resonated because it described my exact experience. I spent months building a fast pipeline and two weeks rebuilding it into a measured one.

What's still broken

The system isn't done. I know exactly where the gaps are because I audit every run.

No cross-run learning. The Retro Analyst scores each run in isolation. When the Tech Lead plans the next feature, it has no access to patterns from previous runs. No score history, no persistent failure patterns, no accumulated learnings. Every run starts blind. The infrastructure for this exists (the retro reports are stored, the scoring is consistent), but the feedback loop back into planning isn't connected yet.

Workers can find bugs but can't report them. Developers are bound to their file scope. If they discover a pre-existing bug in a file they're editing, they can neither fix it (scope violation) nor report it (no mechanism). The bug stays invisible until QA catches it, which QA often doesn't because it wasn't in the acceptance criteria.

Smoke tests are non-blocking. Phase 7 deploys to staging and runs a smoke test. If the smoke test fails, the pipeline logs a warning and marks the run as done anyway. This is the same "escape hatch" problem as the Quality Gate's missing-retro clause, and it exists for the same reason: I built it for the happy path and haven't hardened it yet.

QA still struggles with live-system verification. The 4-level verification model is correct in theory. In practice, the QA agent runs in a git worktree where the application isn't running. It can verify code existence and structure but can't easily hit an API endpoint or load a page. This is the gap that lets runtime bugs through.

Each of these is a known, documented issue with a planned fix. Transparency here isn't modesty. It's the same principle the entire system is built on: you can't fix what you don't name.

Three takeaways for builders

Speed without measurement is waste. A fast pipeline that produces unreliable output isn't saving time. It's redistributing it from development to debugging. Measure what comes out, not just how fast it comes out. Track your actual success rate across runs, not just the runs you remember.

Quality in autonomous systems requires structural enforcement. Prompts asking agents to be thorough don't survive pressure. Tool restrictions, mandatory phases, and quality gates with explicit requirements work. If your Quality Gate can ship without a retro report, it will. Remove the escape hatches.

Separate the builder from the built. When your development toolchain lives inside your product repository, it's coupled to one context. Extract it. Make it CLI-agnostic, tracker-agnostic, portable. The investment pays off the moment you have a second project, and it forces cleaner abstractions that improve the first one.

Frequently Asked Questions

What's the actual success rate of autonomous AI development pipelines?

My pipeline achieved roughly 10% across 22 runs before the quality rebuild. Individual worker tasks succeed at a much higher rate (near 100% for isolated, well-scoped tasks). The failures compound at integration points: merge conflicts, scope violations, and runtime bugs that pass code review. Smaller features with 2 to 4 parallel tracks succeed roughly 50% of the time.

How do you measure the quality of AI-generated code in an autonomous pipeline?

Three dimensions: input quality (was the specification clear?), throughput quality (did the pipeline execute cleanly?), and output quality (does the code work?). Each has weighted sub-metrics that produce a composite score per run. This replaces the binary "it worked / it didn't" with a number you can track across runs and improve systematically.

Is multi-agent development faster than a single AI coding session?

For anything touching more than 2 to 3 files, yes. Parallel workers in isolated git worktrees avoid the context degradation that happens when a single agent edits many files in one session. But multi-agent adds orchestration overhead. For a one-file fix, a direct coding session is faster. The pipeline is for features, not patches.

Can you build an autonomous pipeline without engineering experience?

Yes, but expect the iteration cycle to be slower. I built everything described here as a non-programmer with AI assistance. The critical skill isn't coding. It's systems thinking: understanding where failure surfaces exist, designing constraints that prevent known failures, and building feedback loops that surface new ones. Every rule in the system exists because I hit a problem I couldn't have predicted.

---

Pedram Shahlaifar is building intentic as a learning project: a complex AI system built by someone from the business side, using AI as the development partner. He writes about the technical decisions, trade-offs, and surprises along the way. Connect on LinkedIn.