Chapter 3: Requirements, Risk, and Product Fit

1. Chapter Overview

Before a team designs prompts, retrieves context, orchestrates tools, or evaluates model behavior, it must answer a more basic question: should an AI system carry this product responsibility at all?

A plausible model response does not settle that question. Models can summarize, classify, draft, infer, and recommend in ways that look useful in a demonstration. Product fit asks whether that capability belongs in the real workflow, with its constraints, risks, and accountability.

The requirements and risk discipline in this chapter precedes detailed design. It covers AI suitability, product fit, decision surfaces, risk classification, blast radius, and human accountability.

The primary engineering decision is whether the target problem is suitable for AI, what behavior the system may own, which risks the organization accepts, and where human accountability must remain explicit.

Technical architecture, threat models, evaluation plans, and runtime operations remain with their owning chapters, which consume the decision made here. In particular, Chapter 4's Production Prompt Specification must inherit the accepted system role, non-goals, and non-success boundaries. A prompt cannot repair a badly chosen decision surface; a harness cannot preserve accountability the product never assigned; and an evaluation plan cannot make unclear responsibility acceptable.

2. Primary Engineering Problem

A promising demonstration often becomes an implementation project before anyone proves that the problem is suitable, bounded, and worth the risk.

The failure usually begins with a capability statement: the model can answer support questions, write claims summaries, inspect contracts, generate code, or recommend next actions. The team then infers a product direction and builds around what the model can do, rather than what the workflow needs and what the organization will accept when the model is wrong.

The central problem is that AI usefulness and AI acceptability are different decisions.

Usefulness asks whether AI can improve a workflow. Acceptability asks whether the resulting behavior can be controlled, understood, and owned well enough for the consequences it may create.

The core tension is:

AI can increase capability in ambiguous workflows while also increasing the need for explicit boundaries, review, and accountability.

The product team therefore needs to determine whether the problem benefits from AI, which decision surface the system touches, who is affected when it is wrong, how far a failure can propagate, and where human judgment must remain responsible.

Without that discipline, later engineering rests on unstable assumptions. Prompt quality becomes a proxy for product judgment, and human review becomes a vague safety story. Evaluation measures response quality but misses consequence; security review sees data boundaries but not product risk; operations discovers incidents after exposure has already exceeded the design.

3. Core Mental Model

AI product fit is a suitability and risk decision, not a vote of confidence in a capability.

A useful way to reason about fit is to separate seven questions:

1. Problem fit: Does the workflow contain language, ambiguity, synthesis, judgment support, or variation that AI can usefully handle?
2. Decision surface: What exact decision, output, recommendation, or action will the system influence?
3. Autonomy level: Is the system informing, drafting, recommending, deciding, or acting?
4. Consequence class: What happens if the system is wrong, incomplete, biased, stale, or overconfident?
5. Blast radius: How many people, records, systems, dollars, obligations, or decisions can one failure affect?
6. Reversibility: Can the consequence be corrected, and at what cost?
7. Human accountability: Who owns the judgment, approval, exception, and harm response?

Answer these questions before choosing an implementation pattern. A workflow may suit AI assistance but not AI autonomy; a task may be safe for drafting but unsafe for execution; and a recommendation that works internally may be unacceptable when exposed directly to users.

Autonomy Ladder

Use the autonomy ladder to name what authority the system is being given:

Higher autonomy is not inherently better. Each step up the ladder increases the burden of evidence, containment, and accountable ownership.

Consequence Classes

Classify consequence before autonomy:

The team can move into detailed design once it can state the AI-owned behavior, human-owned behavior, credible failure consequences, and conditions for abstention, escalation, or approval.

4. System Boundary

The boundary covers the product and risk decisions that determine whether an AI system should be built and what responsibility it may carry.

It includes:

• the workflow or decision being improved
• the reason AI may be appropriate
• the system's decision surface
• affected users and non-users
• consequence class and blast radius
• reversibility and recovery expectations
• human accountability and approval requirements
• conditions for abstention, escalation, or non-action

Detailed technical architecture, full threat modeling, evaluation methodology, and runtime operations sit outside this boundary. Those disciplines consume the conclusions recorded here.

AI suitability is not model capability

A model's ability to perform a task does not make that task a suitable AI product responsibility. Suitability also depends on workflow value, consequence, control, explainability, and accountability.

Business value is not risk acceptance

A valuable system can still be unacceptable if the downside is too large, the accountable owner is unclear, or the affected parties cannot be protected.

Human-in-the-loop is not human accountability

A reviewer does not automatically create accountability. Someone who cannot understand the evidence, inspect the result, override the system, or meaningfully reject its output is a throughput bottleneck, not a control.

Low frequency is not low risk

A rare event may still be severe, correlated across many cases, or difficult to detect. Classify risk by consequence and propagation as well as expected frequency.

Reversible is not harmless

Undoing a decision does not erase user confusion, compliance exposure, loss of trust, operational load, or temporary harm.

5. Design Principles

Principle 1: Start with the workflow decision

Name the specific workflow, judgment, or action being changed. Requirements such as "add AI" or "use an agent" describe a means, not a bounded responsibility in a real process.

Principle 2: Keep deterministic responsibilities deterministic

When a consequential rule can be expressed deterministically, do not ask a probabilistic model to enforce it alone. AI can handle ambiguity, language, synthesis, prioritization, and judgment support; deterministic controls should enforce boundaries, permissions, arithmetic, eligibility, and irreversible decisions.

Principle 3: Make the decision surface explicit

State what the system influences: information display, draft content, recommendation, routing, approval, execution, or follow-up. Identical model behavior can carry very different risk on different decision surfaces.

Principle 4: Classify consequence before autonomy

Model performance alone cannot justify an autonomy level. The decision also depends on error consequence, detectability, recovery, and accountable review.

Principle 5: Bound blast radius before expanding exposure

Define how far one failure can spread before increasing volume, user population, tool access, or action authority. Expand exposure only as containment, evidence, and accountability improve.

Principle 6: Design abstention as a feature

Non-action is a required system behavior. Uncertainty, missing information, ambiguous authority, high consequence, or conflicting instructions should trigger abstention, escalation, or approval rather than confident completion.

Principle 7: Preserve meaningful human control

Human review works only when the reviewer has context, time, authority, and a clear intervention standard. Specify what that person owns, not merely where a human appears in the flow.

6. Architecture Patterns

These patterns describe product fit and authority, not technical reference architectures.

Pattern 1: Assistive recommendation

The system suggests options, priorities, or next steps while a person remains responsible for the decision. This pattern suits ambiguous work where AI can reduce search or synthesis burden.

Automation bias is the main risk. Users may accept a recommendation because it is convenient or looks authoritative, so they need visible evidence, uncertainty, and alternatives.

Pattern 2: Draft then review

The system produces a message, summary, plan, analysis, or other draft. A human reviews, edits, and approves it before use.

Draft-then-review works well for language-heavy workflows. It fails when review becomes ceremonial, reviewers lack source context, or volume pressure makes acceptance the default.

Pattern 3: Bounded automation

Within explicit rules, the system executes low-consequence, reversible actions such as tagging, formatting, enrichment, deduplication, or simple routing.

Permissions must stay narrow and recovery paths clear. Moving from bounded automation to broader autonomy requires a separate risk decision; project momentum is not evidence.

Pattern 4: Triage and routing

The system classifies work and sends it to a queue, person, or workflow. Triage often creates value before the system is trusted to decide a final outcome.

Silent misrouting is the characteristic risk. Reviewers should be able to see the escalation, sampling, and correction paths.

Pattern 5: Decision support with human ownership

The system gathers evidence, compares options, and identifies implications; a human owns the final judgment.

High-judgment domains often need this pattern. Its value depends on exposing supporting facts and uncertainty, not merely presenting a polished conclusion.

Pattern 6: Progressive autonomy

The system begins in an assistive role and gains authority only after evidence, containment, and accountability improve.

Risk and demonstrated behavior should gate each increase. Delivery pressure or accumulated familiarity does not justify more authority.

7. Failure Modes

Failure mode 1: Building because the demo is impressive

The team confuses capability with product fit and solves the easiest visible task rather than the most important workflow problem.

Failure mode 2: Automating an undefined workflow

The organization has not agreed on a decision process, yet expects the AI system to perform it consistently. The resulting behavior encodes unresolved human ambiguity and becomes harder to challenge.

Failure mode 3: Hiding policy decisions inside model behavior

The product has not decided what should happen, so the model must infer policy from examples or prompt text. Behavior becomes inconsistent, and nobody clearly owns the policy decision.

Failure mode 4: Treating human review as symbolic control

Reviewers lack time, evidence, authority, or criteria. Supervision exists on the diagram, but practical control remains weak.

Failure mode 5: Ignoring affected non-users

The direct user gains the benefit while another party bears mistakes, bias, privacy exposure, or operational burden.

Failure mode 6: Scaling before blast radius is bounded

Exposure expands to more users, tools, or records before the team contains known failures. A defect that was tolerable in a pilot becomes material in production.

Failure mode 7: Measuring value while ignoring consequence

Speed, completion rate, or unit cost improves while downstream correction, appeal, review load, or user harm grows.

Failure mode 8: Treating reversibility as a complete safety argument

The team treats an undo operation as proof that no harm remains, even after the error has changed a user's decision, triggered follow-up work, or damaged trust.

8. Tradeoffs

Automation value versus accountability burden

More autonomy can increase speed and scale, but it also raises the burden of evidence, containment, and ownership. A low-consequence workflow may tolerate lighter controls; a high-consequence decision cannot.

Speed versus reversibility

Fast action matters when delay is costly, but becomes dangerous when errors are hard to detect or undo. The product decision must weigh latency against consequence instead of assuming that faster is better.

Personalization versus consistency

Adaptation can make an experience more relevant to a user's context. The same variation can produce inconsistent treatment, unpredictable outcomes, and evidence gaps around fairness or compliance.

Broad capability versus narrow responsibility

A general model may support many tasks, but the production system still needs a bounded responsibility. Expand scope through an explicit product decision, never because an incidental capability is already available.

Human review quality versus review load

Human oversight can improve accountability. Its value falls quickly when volume, time pressure, or poor evidence makes careful judgment impractical.

Early market learning versus controlled exposure

An early release can reveal real workflow value, but it can also expose users before the risks are understood. A useful pilot preserves enough realism to learn while keeping its blast radius deliberately small.

9. Production Checklist

Before approving product fit, confirm:

• The target workflow or decision is named.
• The business outcome is translated into observable system behavior.
• The reason AI is suitable is explicit.
• Deterministic requirements are not delegated only to probabilistic behavior.
• The system's decision surface is defined.
• The autonomy level is stated using a shared ladder.
• Affected users and affected non-users are identified.
• Consequence class is assigned for credible failures.
• Blast radius is bounded for pilot and production exposure.
• Reversibility and recovery costs are understood.
• Conditions for abstention, escalation, and human approval are explicit.
• The human accountable for final judgment is named where required.
• Human reviewers have enough context, authority, and time to intervene.
• Value metrics are paired with harm, correction, or downstream burden measures.
• A change in autonomy, exposure, or tool authority requires renewed review.
• Residual risk is accepted by an accountable owner before design proceeds.

10. Design Review Questions

1. What workflow or decision is this system changing?
2. Why is AI suitable for this problem instead of deterministic automation or ordinary software?
3. What exact output, recommendation, decision, or action does the AI system influence?
4. Which autonomy level is being requested, and why is that level justified?
5. Who benefits from the system, and who can be harmed by it?
6. What is the credible worst normal failure, not the most extreme imaginable failure?
7. What is the blast radius of one bad output, one bad batch, or one bad policy assumption?
8. What harm remains even if the system's action can be reversed?
9. What must the system refuse, abstain from, escalate, or require approval for?
10. Where is human judgment required, and what does the human actually own?
11. Can the human reviewer detect likely errors with the evidence shown?
12. What value metric could improve while the product gets riskier?
13. What exposure increase would require a new risk decision?
14. What policy decision is currently implicit in the model or prompt?
15. Who accepts the residual risk before this system proceeds?

11. Main Artifact: AI Fit and Risk Assessment Worksheet

Complete this worksheet before detailed design, then reopen it before increasing autonomy, exposure, or tool authority.

1. Problem Fit

• Source AI System Stack Reference Model and version:
• Source engineering requirement or decision IDs:
• Workflow or decision being improved:
• Current pain or opportunity:
• Why AI is suitable:
• Why ordinary software or deterministic automation is insufficient:
• Conditions where AI should not be used:

2. Decision Surface

• System role: inform, draft, recommend, route, decide, or act
• User-visible output:
• Downstream systems affected:
• Decisions influenced:
• Actions initiated:
• Required abstention conditions:

3. Affected Parties

• Direct users:
• People affected by outputs:
• Operational teams affected:
• Business owner:
• Accountable decision owner:

4. Risk Classification

• Autonomy level:
• Consequence class:
• Credible failure examples:
• Bias, fairness, or unequal-treatment concerns:
• Privacy or confidentiality concerns:
• Compliance or contractual boundaries:
• Reputational or trust concerns:

5. Blast Radius and Reversibility

• Maximum records, users, transactions, or actions affected by one failure:
• Maximum exposure during pilot:
• Maximum exposure during production:
• Reversibility:
• Recovery owner:
• Recovery cost:

6. Autonomy and Human Accountability

• Proposed autonomy level:
• Human approval required:
• Human review criteria:
• Evidence shown to reviewer:
• Override path:
• Appeal or correction path:
• Residual risk owner:

7. Product-Fit Decision Gate

• Assessment ID and version:
• Decision: proceed, narrow, pilot, require human approval, defer, or reject
• Required constraints before design:
• Required constraints before launch:
• Exposure limit:
• Autonomy limit:
• Residual risk accepted by:
• Review date:

8. Design Handoff

• Accepted engineering responsibility and requirement IDs:
• Allowed and prohibited AI-owned behavior:
• Required clarification, abstention, escalation, or approval outcomes:
• Maximum autonomy, exposure, and tool-authority boundary:
• Context, evidence, policy, recovery, and operability obligations for owning chapters:
• Assumptions or unresolved risks that downstream artifacts must preserve:
• Conditions that require this assessment to be reopened:

Any receiving chapter may impose a tighter boundary. A Production Prompt Specification, Context Assembly Plan, Harness Control-Loop Diagram, Tool Permission Matrix, Evaluation Plan, AI Threat Model, AI Release Manifest, or AI Operations Runbook must not silently expand the authority or exposure accepted here.

12. Key Takeaways

• AI suitability is a product and risk decision, not a model-capability claim.
• The decision surface determines the system's consequence more than the model call alone.
• Autonomy should increase only when evidence, containment, and accountability increase with it.
• Human-in-the-loop design is meaningful only when the human has context, authority, time, and accountability.
• Blast radius and reversibility should be bounded before exposure expands.
• Abstention, escalation, and approval are product requirements, not afterthoughts.
• The AI Fit and Risk Assessment Worksheet is the source decision for downstream design artifacts; they may narrow its boundary but must not silently broaden accepted authority, exposure, or risk.