AI PM is still PM
The 38 metrics in this guide cover one layer — the ML/model layer. A PM's job is to own the connections between all three. A model metric improving doesn't automatically mean the product got better. A product metric improving doesn't automatically mean the business benefited. Each layer connects to the one above it through a specific, testable hypothesis — and that hypothesis is yours to hold.
Model / Eval
Is the model doing what it's built to do?
The question at this layer: Is the model technically correct, grounded, and well-calibrated? This is what your ML team owns day to day — but you need to be able to read it.
Product / User
Are users getting value from it?
The question at this layer: Does a model improvement actually change user behavior? This is where you'll first see if the model's "better" number translated into something users care about.
Business
Is it driving outcomes that matter to the company?
The question at this layer: Did the product improvement drive a business result? These live in your analytics platform, not your ML pipeline — and this is the layer leadership actually cares about.
How the layers connect — two examples
Each layer links to the next through a hypothesis: "if this model metric improves, then this product behavior should follow." The PM's job is to make that hypothesis explicit before the work starts, and check it after.
Example 1 — Recommendation system: the connection holds
All three layers moved together — the model improvement was real and the hypotheses held.
Example 2 — Recommendation system: the disconnect
The model got "better" at something users don't care about. The PM's job is to catch this disconnect — and own the question of what would actually change user behavior.
Tracing these connections in a live system — baselines, slices, guardrails, and monitoring — is its own discipline. See Production & Business Fit →
Model Quality
Does the model get the answer right? These carry over from classic ML and still anchor most eval suites, even for generative systems.
Accuracy
Share of all predictions the model got exactly right. Misleading on imbalanced data — a 99%-negative dataset makes "always predict negative" look great.
Precision
Of everything the model flagged positive, how much was actually positive. The metric to protect when false alarms are expensive.
Recall (Sensitivity)
Of everything that was actually positive, how much the model caught. The metric to protect when missing a case is expensive.
F1 Score
Harmonic mean of precision and recall. Use when you need one number and neither false positives nor false negatives are clearly worse.
AUC-ROC
Probability the model ranks a random positive above a random negative, across every classification threshold. 0.5 = coin flip, 1.0 = perfect separation.
Needs a full range of thresholds to plot — not a single-number calculator.
Perplexity pre-training / fine-tuning
How "surprised" a language model is by held-out text — the exponentiated average negative log-likelihood it assigns to the correct next tokens. Lower means the model predicts the sequence more confidently. PMs using API-based LLMs (GPT-4, Claude, Gemini) almost never track this directly — it's a foundational training metric, not an application-layer one. You'll encounter it when evaluating fine-tuning runs.
Needs per-token model probabilities from a real held-out set.
BLEU / ROUGE
N-gram overlap between generated text and a reference answer. BLEU weights precision (translation-style tasks); ROUGE weights recall (summarization). Both are weak proxies for meaning — pair with human or LLM-judge eval.
Needs full reference and candidate text for n-gram comparison.
Pass@K code / logic
The probability that at least one of the top K generated samples passes a correctness check (unit tests, validator, or verifier). The standard eval metric for code generation and complex logic tasks — models rarely produce a perfect solution first try, but K samples dramatically increases the odds. Used by OpenAI Codex, AlphaCode, and most coding benchmark leaderboards.
Retrieval / RAG
For retrieval-augmented systems, generation quality is capped by what got retrieved. These measure the search layer on its own terms.
Recall@k
Of all the relevant documents that exist, what fraction showed up somewhere in the top k retrieved. The core coverage metric for a retriever.
Precision@k
Of the top k results shown, what fraction were actually relevant. Matters most when context windows are small and every slot is expensive.
MRR (Mean Reciprocal Rank)
How early the first relevant result appears, averaged across queries. Rewards getting one good hit near the top — the metric that tracks "did the top result answer it."
Needs a ranked result list across multiple queries.
NDCG@k
Rewards relevant results for both appearing and appearing near the top, using graded (not just binary) relevance. The standard when "somewhat relevant" and "highly relevant" both matter.
Needs a graded relevance list to compare against the ideal ranking.
Groundedness / Faithfulness
Fraction of claims in the generated answer that are actually supported by the retrieved context, usually scored by an LLM judge against each retrieved chunk. Directly tracks hallucination risk in RAG.
Context Relevance
Of the chunks actually retrieved and sent to the model, what fraction were relevant to the query. Distinct from Recall@k — this scores precision of what the generator actually saw.
Answer Relevance
Does the answer actually address the user's question — regardless of whether it was grounded in the retrieved context? The third leg of the RAGAS triad: Faithfulness (R-05) checks grounding, Context Relevance (R-06) checks retrieval quality, Answer Relevance checks whether the output is on-topic and responsive. A perfectly grounded answer to the wrong question scores zero.
Requires embedding model — compute cosine similarity between original query and questions reverse-engineered from the answer.
System Performance
Correct answers that arrive too slowly, or too expensively at scale, still fail the product. Latency and throughput are what users and infra teams actually feel.
Latency (p50 / p95 / p99)
Response time at given percentiles. Never report only the average — tail latency (p95/p99) is what determines whether your worst-case users bounce.
Time to First Token (TTFT)
Delay between sending a request and the first streamed token appearing. Dominates perceived responsiveness in chat interfaces, independent of total generation time.
Throughput
Volume of work the system completes per unit time — requests or tokens per second. The capacity-planning counterpart to latency.
Error / Timeout Rate
Share of requests that fail outright or exceed the timeout budget. Track separately from latency — a fast wrong answer and a slow failure are different problems.
Uptime / Availability
Fraction of time the service was able to serve requests within SLA. Standard infra metric that still governs trust in an AI feature.
Cost & Efficiency
Unlike traditional software, every AI request has a marginal cost. These turn model choice and prompt design into a P&L conversation.
Cost per Inference
Fully-loaded cost of a single request — model API spend plus retrieval, orchestration, and infra overhead. The unit economics number that scales (or breaks) a product.
Cost per Resolved Task
Total cost divided by tasks the system actually completed successfully — not just requests served. Surfaces the real cost of retries, escalations, and failed attempts.
GPU / Compute Utilization
Share of provisioned compute actually doing useful work. Low utilization means you're paying for idle capacity; the lever behind most infra cost-reduction efforts.
Cache Hit Rate
Fraction of requests served from a cache (semantic, prompt, or embedding cache) instead of a full model call. Directly trades off against cost and latency.
Context Window Utilization
Of the tokens sent to the model in the context window, what fraction actually contributed to the response? As context windows expand to 1M+ tokens, blindly stuffing them with data wastes money and degrades quality — models have finite attention, and irrelevant context competes with relevant context. Low utilization signals a retrieval or chunking problem, not a model problem.
Product & Adoption
The metrics that answer the question a model score can't: is this actually working for the people using it.
Adoption Rate
Share of eligible users who tried the AI feature at least once in a period. First gate before any downstream metric matters.
Task Completion Rate
Share of AI-assisted sessions that end in the user's goal actually being met, not just a response being generated. The single best proxy for "is this useful."
Deflection / Containment Rate
Share of interactions the AI resolved fully, without a human handoff. Standard for support and agentic products — the metric finance teams ask for first.
Retention (AI feature)
Of users who tried the feature in one period, what share returned to use it again. Distinguishes genuine usefulness from one-time curiosity.
CSAT / Thumbs-up Rate
Direct user rating of individual AI responses. Noisy at small sample sizes and prone to selection bias (who bothers to rate) — treat as a trend, not a single-session verdict.
Human Override / Edit Rate
How often a person edits, rejects, or overrides the AI's output before acting on it. A direct, low-noise trust signal — especially strong in copilot and drafting products.
Acceptance Rate copilot
Percentage of AI suggestions accepted without modification — the gold standard for copilot and drafting products, popularised by GitHub Copilot. Distinct from Override/Edit Rate (B-06): edit rate measures how much users change outputs; acceptance rate measures whether they took the suggestion at all. High acceptance with high edit distance means users say yes but don't trust the output.
Time to Resolution Delta (TTR Δ)
The difference in resolution time between AI-handled and human-handled interactions. Deflection rate (B-03) tells you if the AI resolved it; TTR Δ tells you how much faster. A support bot that deflects 70% of tickets but takes 3× as long as a human isn't saving time — it's just routing differently. Negative = AI faster; positive = AI slower.
Safety & Trust
Non-negotiable floor metrics. A product can win on every metric above and still be unshippable if these are off.
Hallucination Rate
Share of responses containing at least one fabricated or unsupported claim, typically scored by a reference-grounded judge (human or LLM) against source material.
Toxicity / Harmful Content Rate
Share of outputs flagged by a safety classifier or reviewer for toxic, harassing, or unsafe content. Usually tracked continuously in production, not just in pre-launch eval.
Refusal Rate
Share of requests the model declines to answer. Too low can mean unsafe compliance; too high means the product is unusable — this metric needs a target band, not a direction.
Jailbreak / Attack Success Rate
Share of adversarial red-team prompts that successfully bypass intended guardrails. The core metric from adversarial evaluation and red-teaming exercises.
Demographic Parity Gap
Difference in a key outcome (approval rate, accuracy, response quality) across demographic groups. One of several fairness metrics — pick the definition that matches your legal and ethical obligations, since fairness metrics can conflict with each other.
Talking to Stakeholders
The metrics above are how you'll run the eval. They're rarely how you'll pitch it. This section is the translation layer — plain-English framing and the business stake, for the moments you're in a room without another PM or engineer.
Metric → Business Translation
Same 38 metrics from above, reframed as what a stakeholder actually needs to hear. A few are flagged internal only — worth tracking, not worth putting in a business deck.
| Technical metric | Say this instead | Why they should care |
|---|---|---|
| QAccuracy / Precision / Recall / F1 | How often it's right, and what kind of mistakes it makes when it's wrong | Precision misses cost review time; recall misses cost real incidents (fraud, defects). Different stakeholders should care about different sides of this trade-off. |
| QAUC-ROC | How well it separates the "yes" cases from the "no" cases, at any sensitivity setting | Tells you how much room you have to tune the model tighter or looser without retraining it.internal only |
| QPerplexity | How confidently the model predicts language, in the abstract | Doesn't map to a user-facing outcome on its own — useful for comparing model checkpoints, not for a business case.internal only |
| QBLEU / ROUGE | How closely the wording matches a reference answer | A rough automated stand-in for quality, used mainly to catch regressions between builds.internal only |
| RRecall@k / Precision@k | Is it finding the right information, and is most of what it finds actually useful | Sets the ceiling on answer quality — no amount of good writing fixes bad retrieval. |
| RMRR / NDCG | Does the best evidence show up first, not buried on page two | Determines whether the model actually uses the good source or gets distracted by a mediocre one ranked above it. |
| RGroundedness / Faithfulness | Is every claim in the answer actually backed by our own source material | Directly maps to trust and liability — this is the number that answers "can we trust what it tells customers." |
| RContext Relevance | Is it reading the right pages before it answers | A quiet cost driver — irrelevant context wastes tokens and can quietly worsen answers.internal only |
| PLatency (p50/p95/p99) | How long the user waits — on a typical request, and on the slow ones | Directly drives abandonment; the tail (p95/p99) is what gets compared to every other app people use daily. |
| PTime to First Token | How long before it feels like it's started responding | Perceived speed, independent of total answer length — often the bigger UX lever than raw latency. |
| PThroughput / Uptime | Can it handle everyone at once, without falling over | Reliability commitments and SLA risk — the number ops and leadership ask about before a big launch. |
| CCost per Inference / Task | What it costs us, every single time someone uses this | Unit economics — whether the feature is profitable at scale, and what happens to margin as usage grows. |
| CGPU Utilization / Cache Hit Rate | Are we paying for compute we're not actually using | The lever behind most cost-reduction work — good to know exists, rarely worth a stakeholder's meeting time.internal only |
| BAdoption / Retention | Are people trying it, and do they keep coming back | Already speaks business fluently — the two numbers that tell you if this is a feature or a habit. |
| BTask Completion Rate | Does it actually finish the job, not just produce a response | The single best proxy for "is this useful" — closer to a business outcome than any model score. |
| BDeflection / Containment | How much it handles without a human stepping in | The number finance asks for first — direct line to support cost savings. |
| BOverride / Edit Rate | Do people trust it enough to use the output as-is | Distinguishes a real productivity gain from theater — high edit rates mean people are redoing the work anyway. |
| SHallucination Rate | How often it gives a confident, wrong answer | Brand and legal risk — this is usually the first question a skeptical exec asks. |
| SToxicity / Jailbreak / Bias | Could this embarrass us, get us sued, or treat people unfairly | Regulatory and PR exposure — the metrics that gate a launch regardless of how good everything else looks. |
| SRefusal Rate | How often it declines to help | Too low is a safety problem; too high is a usability problem — needs a target range, not just a trend line. |
A short example: one number, three audiences
The table above gives you the phrasing. This shows the translation actually happening — the same measurement, carried up through all three layers to the room where a decision gets made.
An internal HR assistant that answers employee policy questions
Note what didn't happen: nobody exaggerated, and nobody dumbed it down. Each version is true — they differ in which true thing is relevant to that listener.
Concept Glossary
Not metrics — the underlying architecture terms that come up in the same conversations. Useful for the stakeholder who nods at "hallucination" but glazes over at "reranking."
| Technical concept | Business translation |
|---|---|
| Hallucination | AI gives a confident but incorrect answer |
| Poor retrieval | AI uses the wrong company policy |
| Low confidence | AI isn't sufficiently certain to answer |
| Context window | How much information the AI can consider at once |
| Embeddings | Representing meaning so similar information can be found |
| Chunking | Breaking large documents into usable pieces |
| Reranking | Choosing the most relevant evidence |
| Fine-tuning | Teaching behavior/style, not giving the model a reliable knowledge base |
| RAG | Giving the AI access to relevant, controlled evidence |
| Evaluation | Proving whether the AI actually works |
| Guardrails | Preventing unacceptable behavior |
| Latency | How long the user waits |
| Token cost | Cost of operating each AI interaction |