AI Economics: The Success Rate Metric That Actually Matters

The landscape of AI pricing is shifting. We've moved from flat subscriptions to token-based billing as AI adoption has scaled. The era of subsidized AI costs is ending — companies are now paying for actual usage through real token consumption. Some claim the same subscription price still covers usage, but I'm skeptical. Token counts have changed significantly, especially for more capable models, sometimes increasing tenfold.

This shift raises an urgent question: Does AI still deliver value when you pay per token? The answer depends entirely on how you measure it. Cost-per-token alone is a misleading metric. You need a framework to understand whether AI investments actually produce a return.

OpenAI recently published a framework that helps answer this. Their research shows that a cheaper model per-token might actually cost more overall if it requires multiple attempts or more human intervention. Conversely, a more expensive model can be cheaper when measured against total effort for complex tasks. This taught me to switch models based on the task:

Haiku for grammar checking
Sonnet for implementation tasks
Opus for planning and design work

The framework OpenAI proposes asks four key questions to determine if you benefit from AI:

Is AI completing work that matters?
What does each successful task cost?
Can people depend on the result?
Does each AI unit produce more value as usage grows?

Measuring AI Value: The Personal Level (Questions #1 and #2)

Before companies can measure AI's impact, individuals need to measure their own. The first two questions from OpenAI's framework apply here: Is AI completing work that matters? and What does each successful task cost?

I track two things, token cost and time saved. Let me share two concrete scenarios where I've measured the return.

Scenario 1: Greenfield Projects

You as an engineer always start with boilerplate code. Imagine you need to create a new service. AI can take over with one good prompt and generate a complete project that usually runs out of the box.

Cost: ~5 EUR in tokens
Time saved: 4–5 hours (project scaffolding, dependencies, Dockerfile, basic tests)
ROI: At a typical European tech salary, this is clearly positive.

Scenario 2: Adding Features to Existing Codebases

This is trickier because AI must understand existing patterns. For a typical minor feature, AI needs to find the right places to apply code, match the existing style, and identify all places the new code touches.

Cost: ~5 EUR in tokens
Time invested: 1 hour of prompting and review
Time saved: 3–4 hours total (reading the ticket, understanding requirements, implementation, and testing)
ROI: Still clearly positive.

In both cases, I spent roughly 5 EUR while saving a minimum of 3 hours. At my hourly rate, the math is straightforward — I'm in surplus. But this is personal measurement. What matters to a company is whether these savings compound.

Note: AI application isn't limited to code generation. The same framework applies to other uses:

Finding context to fix bugs
Refactoring and deduplication
Identifying blind spots in planning
Documentation and knowledge work

The point: measure cost and time saved for each use case. If the math is positive at the individual level, it's worth exploring at scale.

Measuring AI Value: The Company Level (Questions #3 and #4)

Individual ROI is necessary but not sufficient. The harder questions are: Can people depend on the result? and Does each AI unit produce more value as usage grows?

These require a shift in how companies think about AI. You can't just multiply individual metrics by headcount. AI output quality varies — sometimes you get a great result, sometimes you need multiple attempts. This non-determinism means success rates matter more than raw usage.

Dependability isn't binary. You need to measure it. Start tracking:

Success Rate: What percentage of AI-generated tasks were accepted on the first review without major rework? Track this by task type (greenfield, feature addition, refactoring, bug fix).
Rework Cost: When a task fails, how much human time does it take to fix? Is it 30% of the original effort, or 150%? If rework consistently costs more than the original generation, your success rate is too low.
Time to Acceptable: From prompt to merge-ready code, how long does the full cycle take? This includes prompting, generation, review, rework, and testing. If a task that "should" take 1 hour consistently takes 4 hours because of failed attempts, the token savings are illusory.

Example from practice: On my team, greenfield projects now have an 85% success rate (code runs, tests pass, no major refactoring needed). Features in existing codebases sit at 60% since they require more domain knowledge AI doesn't have. Bug fixes are hit or miss at 50%. This means I allocate my review time differently. For greenfield work, I do a quick sanity check. For bug fixes, I do deep investigation before merging. Note: these are rough estimates based on experience.

The company-level question becomes: Is the time saved by AI generation greater than the time spent on failed attempts and rework?

This is where the story gets interesting. As teams improve, each token spent should produce more value.

The improvement lever is success rates climbing when:

Engineers write better prompts (after a few months, people learn what works)
You build shared context files (CLAUDE.md, architecture guides) so every engineer starts from the same baseline
Code review practices improve (reviewers learn the patterns AI gets wrong)
You select the right model for the right task instead of defaulting to the cheapest option

The measurement to track over quarters:

Token cost per successful task (should decline or stay flat as volume grows)
Time to successful outcome (should decline)
Success rate by task type (should improve)

If these trends move in the right direction, you're seeing compounding value. If token costs climb and success rates stagnate, you're throwing money at AI without improving the process.

The critical metric is: How many times did this approach work versus how many times was it attempted? Build a simple dashboard: successful AI tasks versus all attempts within a given time window. As your team gets better at prompting, as you refine your context (shared CLAUDE.md files, documentation), as reviewers learn what to watch for, the success rate should climb.

The Transition: From Individual to Team Measurement

Over the past year, I've shifted my approach to AI. Early on, I spent more time prompting and refining requests. I burned more tokens and spent more cycles correcting the output. The result was cheaper per task — but expensive per successful outcome.

Now I spend less total time developing features. I'm better at writing prompts, and the output is more reliable. Crucially, I spend less time verifying and correcting. The compound effect: fewer tokens, less human time, higher quality.

This personal improvement only scales if the whole team improves together. Shared context (documented patterns, coding standards, architecture decisions) means every engineer starts from the same baseline. Code review practices tuned specifically for AI-generated code catch the failure modes humans might miss. Model selection discipline (Haiku for lightweight tasks, Opus for complex reasoning) prevents wasted tokens on the wrong tool.

The Bottom Line

When AI pricing shifts to tokens, the question isn't "How much does this cost per token?" It's "Does work completed grow faster than costs, with stable or improving quality?"

If yes, AI delivers positive ROI — not just for you, but for your company. If no, you either need to improve your process or AI isn't the right tool for that problem.

Published by...

Jernej Klancic

Visit author page