Turbo AI PM — the fast reference for AI product metrics

Start here

Decision Guide & Starter Kits

Don't read all 33 metrics. Find the question you're actually asking, or the kit that matches what you're building, and jump straight to the relevant cards below — every ID is a link.

by question

What are you trying to answer?

Is the model getting the right answer?	Q-01 Accuracy Q-02 Precision Q-03 Recall Q-04 F1
Is it making things up?	S-01 Hallucination Rate R-05 Faithfulness
Is it finding the right information first?	R-01 Recall@k R-02 Precision@k R-06 Context Relevance
Is it fast enough to feel usable?	P-01 Latency P-02 Time to First Token
Can it handle our traffic without falling over?	P-03 Throughput P-04 Error Rate P-05 Uptime
What does this cost us to run?	C-01 Cost per Inference C-02 Cost per Task C-04 Cache Hit Rate
Are people actually using it, and coming back?	B-01 Adoption Rate B-04 Retention
Is it actually helping them, or just responding?	B-02 Task Completion B-06 Override / Edit Rate B-05 CSAT
Could this get us sued, embarrassed, or fined?	S-02 Toxicity Rate S-04 Jailbreak / ASR S-05 Demographic Parity Gap
Should it even be answering this?	S-03 Refusal Rate

by product

Starter kits by product type

Six or fewer metrics to stand up first for common AI product shapes. Add from the full set as the product matures — don't try to track everything on day one.

RAG / knowledge assistant

Answers grounded in your own documents. The risk is confident-sounding wrong answers, so retrieval and faithfulness come first.

R-01 R-02 R-05 S-01 P-02 C-01

Copilot / drafting tool

Suggests text, code, or edits a person reviews before acting on. Trust and time-saved matter more than raw model scores.

B-06 B-02 B-05 P-01 C-01

Agent (multi-step / tool-using)

Takes actions across steps and tools, largely unsupervised. Completion and cost-per-task compound fast when a chain fails partway.

B-02 B-03 S-04 P-04 C-02

Classifier / triage & moderation

Sorts, flags, or routes items — fraud, support tickets, content review. Precision/recall trade-offs are the whole ballgame.

Q-01 Q-02 Q-03 Q-04 S-05

Customer-facing chatbot / support

Talks directly to customers, often unsupervised in the moment. Containment and safety floors matter as much as being right.

B-03 B-05 S-03 S-02 P-01

Q — 01–07

Model Quality

Does the model get the answer right? These carry over from classic ML and still anchor most eval suites, even for generative systems.

Q-01↑ higher better

Accuracy

Share of all predictions the model got exactly right. Misleading on imbalanced data — a 99%-negative dataset makes "always predict negative" look great.

Accuracy = (TP + TN) / (TP + TN + FP + FN)

TP true positive · TN true negative · FP false positive · FN false negative

Try it

TP TN FP FN

Result: —

Q-02↑ higher better

Precision

Of everything the model flagged positive, how much was actually positive. The metric to protect when false alarms are expensive.

Precision = TP / (TP + FP)

TP true positive · FP false positive

Try it

TP FP

Result: —

Q-03↑ higher better

Recall (Sensitivity)

Of everything that was actually positive, how much the model caught. The metric to protect when missing a case is expensive.

Recall = TP / (TP + FN)

TP true positive · FN false negative

Try it

TP FN

Result: —

Q-04↑ higher better

F1 Score

Harmonic mean of precision and recall. Use when you need one number and neither false positives nor false negatives are clearly worse.

F1 = 2 · (Precision · Recall) / (Precision + Recall)

Weighted variant: Fβ uses β to favor recall (β>1) or precision (β<1)

Try it

Precision % Recall %

Result: —

Q-05↑ higher better

AUC-ROC

Probability the model ranks a random positive above a random negative, across every classification threshold. 0.5 = coin flip, 1.0 = perfect separation.

AUC = ∫₀¹ TPR(f) d[FPR(f)]

TPR true positive rate (recall) · FPR false positive rate = FP/(FP+TN) · f decision threshold

Try it

Needs a full range of thresholds to plot — not a single-number calculator.

Q-06↓ lower better

Perplexity

How "surprised" a language model is by held-out text — the exponentiated average negative log-likelihood it assigns to the correct next tokens. Lower means the model predicts the sequence more confidently.

PPL = exp( −(1/N) Σ log P(w_i | w<i) )

N number of tokens · P(w_i|w<i) model's probability for the true token given prior context

Try it

Needs per-token model probabilities from a real held-out set.

Q-07↑ higher better

BLEU / ROUGE

N-gram overlap between generated text and a reference answer. BLEU weights precision (translation-style tasks); ROUGE weights recall (summarization). Both are weak proxies for meaning — pair with human or LLM-judge eval.

BLEU = BP · exp(Σ w_n log p_n)

BP brevity penalty (penalizes short outputs) · p_n n-gram precision · w_n weight per n-gram order

Try it

Needs full reference and candidate text for n-gram comparison.

R — 01–06

Retrieval / RAG

For retrieval-augmented systems, generation quality is capped by what got retrieved. These measure the search layer on its own terms.

R-01↑ higher better

Recall@k

Of all the relevant documents that exist, what fraction showed up somewhere in the top k retrieved. The core coverage metric for a retriever.

Recall@k = |Relevant ∩ Retrieved_k| / |Relevant|

Retrieved_k top-k results returned · Relevant full set of documents that should have matched

Try it

Relevant retrieved Total relevant

Result: —

R-02↑ higher better

Precision@k

Of the top k results shown, what fraction were actually relevant. Matters most when context windows are small and every slot is expensive.

Precision@k = |Relevant ∩ Retrieved_k| / k

k number of results returned/considered

Try it

Relevant retrieved k (results returned)

Result: —

R-03↑ higher better

MRR (Mean Reciprocal Rank)

How early the first relevant result appears, averaged across queries. Rewards getting one good hit near the top — the metric that tracks "did the top result answer it."

MRR = (1/|Q|) Σ 1/rank_i

Q set of queries · rank_i position of the first relevant result for query i

Try it

Needs a ranked result list across multiple queries.

R-04↑ higher better

NDCG@k

Rewards relevant results for both appearing and appearing near the top, using graded (not just binary) relevance. The standard when "somewhat relevant" and "highly relevant" both matter.

NDCG@k = DCG@k / IDCG@k

DCG@k = Σ (2^rel_i − 1) / log₂(i+1) · IDCG@k DCG of the ideal (perfectly sorted) ranking

Try it

Needs a graded relevance list to compare against the ideal ranking.

R-05↑ higher better

Groundedness / Faithfulness

Fraction of claims in the generated answer that are actually supported by the retrieved context, usually scored by an LLM judge against each retrieved chunk. Directly tracks hallucination risk in RAG.

Faithfulness = Supported claims / Total claims in answer

Typically computed by decomposing the answer into atomic claims, then checking each against the retrieved context

Try it

Supported claims Total claims

Result: —

R-06↑ higher better

Context Relevance

Of the chunks actually retrieved and sent to the model, what fraction were relevant to the query. Distinct from Recall@k — this scores precision of what the generator actually saw.

Context Relevance = Relevant sentences in context / Total sentences in context

Scored per-query, then averaged across the eval set

Try it

Relevant sentences Total sentences

Result: —

P — 01–05

System Performance

Correct answers that arrive too slowly, or too expensively at scale, still fail the product. Latency and throughput are what users and infra teams actually feel.

P-01↓ lower better

Latency (p50 / p95 / p99)

Response time at given percentiles. Never report only the average — tail latency (p95/p99) is what determines whether your worst-case users bounce.

p95 = value below which 95% of requests fall

Compute by sorting all request latencies and reading the value at the target percentile rank

P-02↓ lower better

Time to First Token (TTFT)

Delay between sending a request and the first streamed token appearing. Dominates perceived responsiveness in chat interfaces, independent of total generation time.

TTFT = t(first token emitted) − t(request sent)

t timestamp

Try it

Request sent (ms) First token (ms)

Result: —

P-03↑ higher better

Throughput

Volume of work the system completes per unit time — requests or tokens per second. The capacity-planning counterpart to latency.

Throughput = Completed requests / Time window

Also expressed as tokens/sec: total output tokens generated ÷ wall-clock time

Try it

Completed requests Time window (sec)

Result: —

P-04→ depends on SLA

Error / Timeout Rate

Share of requests that fail outright or exceed the timeout budget. Track separately from latency — a fast wrong answer and a slow failure are different problems.

Error Rate = Failed requests / Total requests

Segment by error type (timeout, 5xx, rate-limited, malformed output) — the mix tells you where to fix

Try it

Failed requests Total requests

Result: —

P-05↑ higher better

Uptime / Availability

Fraction of time the service was able to serve requests within SLA. Standard infra metric that still governs trust in an AI feature.

Availability = Uptime / (Uptime + Downtime)

Often expressed in "nines" — 99.9% ("three nines") ≈ 8.8 hours downtime/year

Try it

Uptime (hrs) Downtime (hrs)

Result: —

C — 01–04

Cost & Efficiency

Unlike traditional software, every AI request has a marginal cost. These turn model choice and prompt design into a P&L conversation.

C-01↓ lower better

Cost per Inference

Fully-loaded cost of a single request — model API spend plus retrieval, orchestration, and infra overhead. The unit economics number that scales (or breaks) a product.

Cost/Inference = (Input tokens · Rate_in) + (Output tokens · Rate_out) + Overhead

Rate_in/Rate_out price per token, input/output · Overhead retrieval, vector DB, orchestration cost allocated per call

Try it

Input tokens Rate in ($/1K tok) Output tokens Rate out ($/1K tok) Overhead ($)

Result: —

C-02↓ lower better

Cost per Resolved Task

Total cost divided by tasks the system actually completed successfully — not just requests served. Surfaces the real cost of retries, escalations, and failed attempts.

Cost/Task = Total spend / Successfully resolved tasks

"Resolved" should match your product's own success definition (see B-02, Task Completion Rate)

Try it

Total spend ($) Resolved tasks

Result: —

C-03↑ higher better

GPU / Compute Utilization

Share of provisioned compute actually doing useful work. Low utilization means you're paying for idle capacity; the lever behind most infra cost-reduction efforts.

Utilization = Active compute time / Provisioned compute time

Tracked per GPU-hour or accelerator-hour, usually averaged over a billing period

Try it

Active compute (hrs) Provisioned (hrs)

Result: —

C-04↑ higher better

Cache Hit Rate

Fraction of requests served from a cache (semantic, prompt, or embedding cache) instead of a full model call. Directly trades off against cost and latency.

Cache Hit Rate = Cache hits / (Cache hits + Cache misses)

Watch for staleness — a high hit rate on outdated cached answers trades cost for accuracy

Try it

Cache hits Cache misses

Result: —

B — 01–06

Product & Adoption

The metrics that answer the question a model score can't: is this actually working for the people using it.

B-01↑ higher better

Adoption Rate

Share of eligible users who tried the AI feature at least once in a period. First gate before any downstream metric matters.

Adoption Rate = Users who engaged / Users with access

Define "engaged" precisely — one click vs. one completed session tell very different stories

Try it

Users engaged Users with access

Result: —

B-02↑ higher better

Task Completion Rate

Share of AI-assisted sessions that end in the user's goal actually being met, not just a response being generated. The single best proxy for "is this useful."

Completion Rate = Sessions with goal met / Total sessions started

Requires an explicit success definition per use case — set it before launch, not after

Try it

Sessions with goal met Total sessions

Result: —

B-03↑ higher better

Deflection / Containment Rate

Share of interactions the AI resolved fully, without a human handoff. Standard for support and agentic products — the metric finance teams ask for first.

Containment Rate = Resolved without escalation / Total interactions

Pair with CSAT (B-05) — high containment with low satisfaction usually means silent deflection, not real resolution

Try it

Resolved w/o escalation Total interactions

Result: —

B-04↑ higher better

Retention (AI feature)

Of users who tried the feature in one period, what share returned to use it again. Distinguishes genuine usefulness from one-time curiosity.

Retention = Returning users in period N / Users active in period N−1

Cohort by first-use date to see retention curves, not just a single blended number

Try it

Returning users Active last period

Result: —

B-05↑ higher better

CSAT / Thumbs-up Rate

Direct user rating of individual AI responses. Noisy at small sample sizes and prone to selection bias (who bothers to rate) — treat as a trend, not a single-session verdict.

CSAT = Positive ratings / Total ratings given

Response rate matters as much as the score — a 95% CSAT from a 2% response rate is a different signal than from 40%

Try it

Positive ratings Total ratings

Result: —

B-06↓ lower better

Human Override / Edit Rate

How often a person edits, rejects, or overrides the AI's output before acting on it. A direct, low-noise trust signal — especially strong in copilot and drafting products.

Override Rate = Outputs edited or rejected / Total outputs shown

Track edit *distance* too, not just yes/no — a one-word fix and a full rewrite shouldn't count the same

Try it

Outputs edited/rejected Total outputs shown

Result: —

S — 01–05

Safety & Trust

Non-negotiable floor metrics. A product can win on every metric above and still be unshippable if these are off.

S-01↓ lower better

Hallucination Rate

Share of responses containing at least one fabricated or unsupported claim, typically scored by a reference-grounded judge (human or LLM) against source material.

Hallucination Rate = Responses with ≥1 unsupported claim / Total responses evaluated

Report alongside Faithfulness (R-05) — same underlying signal, opposite framing

Try it

Responses w/ unsupported claim Total responses

Result: —

S-02↓ lower better

Toxicity / Harmful Content Rate

Share of outputs flagged by a safety classifier or reviewer for toxic, harassing, or unsafe content. Usually tracked continuously in production, not just in pre-launch eval.

Toxicity Rate = Flagged outputs / Total outputs

Threshold sensitivity matters — report the rate at your production classifier's actual cutoff, not an idealized one

Try it

Flagged outputs Total outputs

Result: —

S-03→ depends on use case

Refusal Rate

Share of requests the model declines to answer. Too low can mean unsafe compliance; too high means the product is unusable — this metric needs a target band, not a direction.

Refusal Rate = Declined requests / Total requests

Split into appropriate vs. over-refusal using a labeled eval set — the split matters more than the headline number

Try it

Declined requests Total requests

Result: —

S-04↓ lower better

Jailbreak / Attack Success Rate

Share of adversarial red-team prompts that successfully bypass intended guardrails. The core metric from adversarial evaluation and red-teaming exercises.

ASR = Successful bypasses / Total adversarial attempts

ASR attack success rate · measured against a fixed, versioned red-team prompt set for comparability over time

Try it

Successful bypasses Attack attempts

Result: —

S-05↓ lower better

Demographic Parity Gap

Difference in a key outcome (approval rate, accuracy, response quality) across demographic groups. One of several fairness metrics — pick the definition that matches your legal and ethical obligations, since fairness metrics can conflict with each other.

Gap = |P(outcome | Group A) − P(outcome | Group B)|

Related metrics to know: Equalized Odds (equal TPR/FPR across groups), Equal Opportunity (equal TPR only)

Try it

Outcome rate, Group A (%) Outcome rate, Group B (%)

Result: —

T — stakeholder layer

Talking to Stakeholders

The metrics above are how you'll run the eval. They're rarely how you'll pitch it. This section is the translation layer — plain-English framing and the business stake, for the moments you're in a room without another PM or engineer.

7.1

Metric → Business Translation

Same 33 metrics from above, reframed as what a stakeholder actually needs to hear. A few are flagged internal only — worth tracking, not worth putting in a business deck.

Technical metric	Say this instead	Why they should care
QAccuracy / Precision / Recall / F1	How often it's right, and what kind of mistakes it makes when it's wrong	Precision misses cost review time; recall misses cost real incidents (fraud, defects). Different stakeholders should care about different sides of this trade-off.
QAUC-ROC	How well it separates the "yes" cases from the "no" cases, at any sensitivity setting	Tells you how much room you have to tune the model tighter or looser without retraining it.internal only
QPerplexity	How confidently the model predicts language, in the abstract	Doesn't map to a user-facing outcome on its own — useful for comparing model checkpoints, not for a business case.internal only
QBLEU / ROUGE	How closely the wording matches a reference answer	A rough automated stand-in for quality, used mainly to catch regressions between builds.internal only
RRecall@k / Precision@k	Is it finding the right information, and is most of what it finds actually useful	Sets the ceiling on answer quality — no amount of good writing fixes bad retrieval.
RMRR / NDCG	Does the best evidence show up first, not buried on page two	Determines whether the model actually uses the good source or gets distracted by a mediocre one ranked above it.
RGroundedness / Faithfulness	Is every claim in the answer actually backed by our own source material	Directly maps to trust and liability — this is the number that answers "can we trust what it tells customers."
RContext Relevance	Is it reading the right pages before it answers	A quiet cost driver — irrelevant context wastes tokens and can quietly worsen answers.internal only
PLatency (p50/p95/p99)	How long the user waits — on a typical request, and on the slow ones	Directly drives abandonment; the tail (p95/p99) is what gets compared to every other app people use daily.
PTime to First Token	How long before it feels like it's started responding	Perceived speed, independent of total answer length — often the bigger UX lever than raw latency.
PThroughput / Uptime	Can it handle everyone at once, without falling over	Reliability commitments and SLA risk — the number ops and leadership ask about before a big launch.
CCost per Inference / Task	What it costs us, every single time someone uses this	Unit economics — whether the feature is profitable at scale, and what happens to margin as usage grows.
CGPU Utilization / Cache Hit Rate	Are we paying for compute we're not actually using	The lever behind most cost-reduction work — good to know exists, rarely worth a stakeholder's meeting time.internal only
BAdoption / Retention	Are people trying it, and do they keep coming back	Already speaks business fluently — the two numbers that tell you if this is a feature or a habit.
BTask Completion Rate	Does it actually finish the job, not just produce a response	The single best proxy for "is this useful" — closer to a business outcome than any model score.
BDeflection / Containment	How much it handles without a human stepping in	The number finance asks for first — direct line to support cost savings.
BOverride / Edit Rate	Do people trust it enough to use the output as-is	Distinguishes a real productivity gain from theater — high edit rates mean people are redoing the work anyway.
SHallucination Rate	How often it gives a confident, wrong answer	Brand and legal risk — this is usually the first question a skeptical exec asks.
SToxicity / Jailbreak / Bias	Could this embarrass us, get us sued, or treat people unfairly	Regulatory and PR exposure — the metrics that gate a launch regardless of how good everything else looks.
SRefusal Rate	How often it declines to help	Too low is a safety problem; too high is a usability problem — needs a target range, not just a trend line.

7.2

Concept Glossary

Not metrics — the underlying architecture terms that come up in the same conversations. Useful for the stakeholder who nods at "hallucination" but glazes over at "reranking."

Technical concept	Business translation
Hallucination	AI gives a confident but incorrect answer
Poor retrieval	AI uses the wrong company policy
Low confidence	AI isn't sufficiently certain to answer
Context window	How much information the AI can consider at once
Embeddings	Representing meaning so similar information can be found
Chunking	Breaking large documents into usable pieces
Reranking	Choosing the most relevant evidence
Fine-tuning	Teaching behavior/style, not giving the model a reliable knowledge base
RAG	Giving the AI access to relevant, controlled evidence
Evaluation	Proving whether the AI actually works
Guardrails	Preventing unacceptable behavior
Latency	How long the user waits
Token cost	Cost of operating each AI interaction