Chapter 9: Security, Privacy, and Governance

1. Chapter Overview

AI systems bring untrusted language, sensitive data, probabilistic interpretation, delegated authority, human judgment, and external dependencies into one operating path. Each element is familiar on its own; their composition creates boundary crossings that ordinary application controls may not represent.

The security question is not whether the model is aligned, the prompt is strong, or a filter is enabled. The complete system has to preserve its security, privacy, policy, and accountability properties when inputs are hostile, dependencies change, identities are confused, controls fail, or outputs are reused outside their original setting.

The primary engineering decision in this chapter is:

Which trust claims must hold for this AI system, how are those claims enforced and verified, and who may accept the residual risk for a bounded use?

One living artifact records that decision: the AI Threat Model. It begins with assets, actors, effective principals, data and authority flows, and security objectives. From there, it traces threats and obligations to independent controls, assurance evidence, response requirements, and an expiring governance decision. This is more than a list of generic attacks. It is the engineering argument for why a named system may handle particular data, serve particular parties, and exercise particular authority under declared conditions.

This chapter owns prompt injection, data leakage, privacy, access control, model and supply-chain risk, compliance boundaries, governance review, responsible-AI review, and security-specific response. It consumes product consequence and accountability decisions from Chapter 3; instruction, context, control-loop, and action contracts from Chapters 4 through 7; evaluation evidence from Chapter 8; release identity and enforcement from Chapter 10; and operational mechanisms from Chapter 11. It does not re-teach those disciplines.

2. Primary Engineering Problem

AI systems interpret data and instructions through the same channel. A user request, retrieved document, web page, tool result, image, memory record, or prior output may carry useful content alongside adversarial direction. If the model follows that direction, it does so with authority, context, and credentials supplied by the surrounding system. An attacker who lacks direct access can then turn the system into a confused deputy that discloses, decides, remembers, or acts on the attacker's behalf.

The attack surface extends well beyond the response. An input can poison persistent memory, alter an artifact that another workflow later consumes, manipulate evidence shown to an approver, place sensitive data in a search query, or trigger a tool effect after the initiating session has ended. Transformation does not remove the risk: summaries, embeddings, extracted metadata, and generated files can preserve hostile instructions or sensitive meaning while obscuring their provenance.

Privacy and governance failures follow the same shape. A service may be able to access data that the represented user or approved purpose is not permitted to use. A record authorized for one answer may remain off limits for model improvement, evaluator calibration, debugging, analytics, or long-term retention. Likewise, an approval granted for one population, environment, and autonomy level cannot simply follow the system into a broader exposure.

The engineering problem is therefore:

Prevent untrusted information, excessive authority, impermissible data use, opaque dependencies, and probabilistic behavior from violating enforceable trust boundaries.

The model cannot own this problem. Its instructions, classifications, and output checks may contribute to the control system, but none provides security isolation. Durable protection comes from minimizing reachable assets, binding authority to the initiating purpose, enforcing policy outside model behavior, containing compromise, and requiring evidence before an authorized owner accepts the residual risk.

3. Core Mental Model

Security is a chain of trust claims

A production threat model reconstructs the decision as a chain:

1. Scope and objectives: What workflow, environment, assets, populations, and uses are in scope? Which confidentiality, integrity, availability, provenance, isolation, purpose-limitation, and accountable-use properties must hold?
2. Actors and effective principals: Who initiates, who is represented, which identity executes, who owns the affected resource, and who is accountable for the outcome?
3. Data and authority flows: What crosses each trust boundary, in which direction, through which component, under which identity, and into which persistent or external destination?
4. Threats and obligations: Which attacker capabilities, misuse paths, privacy requirements, compliance interpretations, and impact concerns can invalidate the objectives?
5. Controls and containment: Where is policy decided and independently enforced? If prevention fails, how far can disclosure, corruption, or action propagate?
6. Assurance and evidence: What analysis, test, review, or attestation supports each security claim, for which system identity and validity period?
7. Governance decision: Who may approve, constrain, reject, suspend, or retire the use? What residual risk, exceptions, expiry, and re-review triggers bind that decision?

Every link supports the conclusion. A control detached from a threat has no demonstrated purpose, while a threat detached from an asset and objective has no concrete impact. A test cannot support the deployed system unless it is bound to a release identity. Nor can an approval remain meaningful without constraints, authority, and expiry.

Distinguish actors from effective principals

The human who submits a request is only one identity in the flow. A useful authorization model distinguishes:

• the initiating principal, which starts the workflow;
• the represented principal, on whose behalf data or action is requested;
• the executing principal, whose credential accesses a resource or tool;
• the resource owner, whose policy governs the asset;
• the approving principal, which authorizes a sensitive decision or action; and
• the accountable owner, which accepts the outcome and residual risk.

One person may fill several of these roles in a low-risk workflow; a consequential workflow may deliberately separate them. The system must never infer that they are equivalent. A backend credential's ability to retrieve an object proves neither that the represented user may see it nor that the initiating purpose permits its use or transfer to an external processor.

Authorization remains bound from request through retrieval, context assembly, generation, approval, and action. Depending on the workflow, the binding may include tenant, resource, field, action, purpose, environment, and time. Enforcement must remain independent of model judgment, and the system must recheck policy where state can change. If identity, entitlement, purpose, or policy is indeterminate, the crossing fails closed.

Model assets, flows, and derived state

Trust belongs to a flow and its claims, not to a label attached to a component. Calling a retrieval service or model endpoint “trusted” says little unless the threat model states which claim is inherited, who owns it, what evidence supports it, and how compromise is contained.

The system map covers entry points, application and policy services, model invocations, context stores, indexes, memory, tools, queues, reviewer interfaces, analytics, administrative surfaces, external processors, and egress paths. It follows data as the system collects, transmits, retrieves, assembles, generates, caches, logs, labels, exports, backs up, and deletes it. Prompts, outputs, embeddings, indexes, memory, labels, traces, action receipts, and evaluation assets can all become sensitive derived stores.

For every important boundary, answer:

• What asset or authority crosses?
• In which direction and under whose effective identity?
• Which policy decision authorizes the crossing?
• Where is that decision enforced?
• What persists or leaves the system afterward?
• What happens when identity, provenance, classification, or policy is unknown?
• Which evidence proves that the boundary behaved as claimed?

Treat approval as a bounded authorization

Governance cannot certify that a system is safe in the abstract. It authorizes a particular use under constraints, binding the decision to a system and use-case identity, population, data classes, autonomy, environment, exposure, evidence package, unresolved findings, residual-risk statement, and expiration.

A technically successful deployment may still invalidate its authorization. New data categories, affected populations, tool authority, processors, model behavior, dependency terms, retention uses, or incident evidence can all require re-review. Chapter 9 defines the approval and its invalidation conditions; Chapter 10 enforces that approved state in a release.

4. System Boundary

Chapter 9 owns the trust argument: which security, privacy, compliance, and responsible-AI claims apply; which threats and abuse paths challenge them; which control properties and assurance evidence are required; and which authority may accept the residual risk.

Security consumes product responsibility; it does not redefine product fit

Chapter 3 supplies the intended use, prohibited outcomes, affected parties, consequence class, blast radius, reversibility, accountable owner, and human-responsibility decisions. The threat model consumes these as inputs. It does not decide whether an AI feature is useful or introduce a competing general risk taxonomy.

Security challenges design contracts; it does not reimplement them

Chapter 4 owns instruction structure and output contracts. Chapter 5 owns context selection, source trust, permission-aware assembly, and memory lifecycle. Chapter 6 owns control-loop states, validation, retries, fallbacks, and approval mechanics. Chapter 7 owns action authorization, parameter binding, idempotency, delegation, and receipts.

Here, the review asks how an attacker could abuse those contracts, which security properties must survive, and whether enforcement remains independent of the model. It may require provenance preservation, deny-by-default authorization, credential attenuation, isolation, or approval integrity without teaching those implementation disciplines again.

Security defines assurance claims; evaluation defines repeatable measurement

Chapter 9 supplies the threat, attacker capability, abuse preconditions, affected asset, required safe behavior, severity, and security or privacy claim. Chapter 8 turns suitable claims into versioned cases, datasets, evaluators, uncertainty treatment, regression coverage, and gate evidence. Other assurance work—including architectural analysis, penetration review, supplier assurance, and privacy inspection—may produce evidence that is not a quality score. The governance record preserves each form along with its limitations.

Security constrains release; it does not own deployment

Chapter 9 states which approval, control evidence, dependency identity, data-use condition, or unresolved finding must bind a release and which changes invalidate approval. Chapter 10 owns release manifests, promotion, routing, rollout, rollback, runtime configuration, and CI/CD enforcement.

Security supplies response requirements; operations owns incident execution

Chapter 9 defines security-specific event classes, containment constraints, credential revocation, sensitive-evidence handling, notification decision inputs, recovery authority, and re-review triggers. Chapter 11 owns telemetry platforms, general incident command, communication mechanics, service restoration, and post-incident operations.

Engineering records do not replace authoritative interpretation

Legal, compliance, contractual, privacy, and policy obligations must come from an authorized source and interpreting owner. The threat model translates those approved interpretations into affected flows, controls, owners, and evidence. Where interpretation remains unresolved, it records the uncertainty and conservative behavior rather than inventing jurisdiction-specific legal advice.

5. Design Principles

Principle 1: Begin with assets, objectives, and flows

Start with what requires protection, not a catalog of attacks. The inventory includes data, credentials, policies, instructions, model and data artifacts, indexes, memory, approvals, action receipts, and audit evidence. Each asset receives the relevant confidentiality, integrity, availability, provenance, isolation, purpose-limitation, or accountable-use objectives. Flows and state changes then give each threat a concrete boundary and consequence.

Principle 2: Treat all external content as data, never authority

User input, retrieved material, tool output, web content, attachments, metadata, memory, and generated artifacts may all carry instructions. Their provenance needs to survive transformation, while trusted instruction, untrusted content, and executable action remain in distinct control domains. Content labels and model instructions may reduce confusion; only independent authorization and action enforcement create a security boundary.

Principle 3: Bind least privilege to principal, purpose, and time

Give the workflow only the data, tool, network, and persistence authority required for the represented principal and approved purpose. Attenuated, expiring credentials are safer than ambient service authority. Where entitlements can change, permissions need another check at use. Identifier substitution, cross-tenant access, secondary use, and action expansion are denied by default.

Principle 4: Minimize before exposure; inspect egress as defense in depth

Output filtering cannot repair excessive context. Reusable secrets stay outside model-visible state, and the system minimizes collection, retrieval, context, logging, and human access before applying channel-specific egress controls. Those controls cover responses, citations, files, tool parameters, search queries, errors, memory, analytics, evaluation corpora, and reviewer surfaces. Derived or inferred sensitive information still needs protection even when no source text is reproduced.

Principle 5: Make data use explicit across its lifecycle

Each data category needs a record of its source, affected party, approved purpose, prohibited secondary uses, processors, transfer boundary, storage location, derived stores, retention, deletion, correction, export, and legal-hold behavior. Embedding, summarizing, redacting, or pseudonymizing data does not automatically remove its privacy obligations. Uncertain classification or re-identification risk calls for conservative handling and accountable escalation.

Principle 6: Design controls for independent failure

Effective control layers span prevention, policy enforcement, containment, detection requirements, response, recovery, and assurance. Several checks that all depend on the same model judgment do not constitute defense in depth. Every useful control names its owner, enforcement point, dependencies, failure behavior, evidence, and residual risk. Isolation across tenants, sessions, credentials, tools, stores, and environments keeps one failure from acquiring unbounded reach.

Principle 7: Govern dependencies as part of the trusted system

Models, adapters, datasets, indexes, packages, plugins, evaluators, and external services all inherit a place in the trusted system. For each dependency, capture its identity, source, version or change channel, owner, deployment role, provenance, integrity evidence, privileges, inherited claims, data-use commitments, and assurance age. Update approval, emergency change, substitution, disablement, exit, and material-change semantics need definition before that dependency becomes unavailable or untrusted.

Principle 8: Make governance decisions constrained, traceable, and temporary

As risk rises, the evidence becomes stronger, decision rights become more separated, exceptions narrow, and approvals expire sooner. The record distinguishes approval, constrained approval, remediation, rejection, suspension, and retirement. Every exception has an accountable acceptor, compensating controls, an expiry, and a remediation date. Changed exposure, obligations, evidence, or affected populations must never inherit an old approval silently.

6. Architecture Patterns

Together, these patterns form a layered control strategy. Few systems need every mechanism at maximum strength. Every material threat does, however, need prevention, independent enforcement, containment, assurance, and a failure response proportionate to its consequence.

Pattern 1: Authorization-preserving data path

Carry the initiating and represented principals, tenant, purpose, and policy context through retrieval and context assembly. An authoritative service decides policy, and an independent enforcement point limits the resource set before model exposure. Returned objects remain bound to that decision: semantic similarity, possession of an identifier, and backend access are not proof of authorization. If entitlements, resource state, or purpose can change, check again at use.

This pattern keeps the model out of the policy-engine role and limits confused-deputy paths. Assurance covers negative authorization, cross-tenant isolation, stale entitlements, identifier substitution, and time-of-check/time-of-use conditions. Chapter 5 implements the context behavior; Chapter 9 records the security claim and the evidence it requires.

Pattern 2: Capability-isolated action path

Expose narrow operations through scoped, expiring credentials. When the consequence warrants it, proposal, authorization, and execution remain separate. Approval binds the represented principal, exact action, parameters, target, validity window, and evidence shown to the approver. The model never chooses its own network reach, tool catalog, secret access, or write authority.

Containment matters as much as prevention. A compromised workflow must not inherit organization-wide credentials, cross-environment access, or unrelated tools. Chapter 7 owns the action protocol; the threat model captures attacker paths, privilege ceilings, isolation objectives, and revocation behavior.

Pattern 3: Provenance-preserving content path

Preserve source identity, trust class, transformation history, and destination restrictions as content moves through ingestion, retrieval, summarization, memory, and output. Instruction-like text from an untrusted source remains content. If provenance, authorization, or classification becomes indeterminate, quarantine or exclude the source.

Provenance provides a security signal, not a complete control. It informs source policy, incident investigation, and egress decisions but cannot guarantee that the model ignores hostile instructions. Minimized authority, independent policy enforcement, and constrained outputs carry the rest of the boundary.

Pattern 4: Channel-specific leakage containment

Begin with every output, persistence, and human-access channel, then define its authorized audience and data classes. Depending on consequence, the channel may use structured allowlists, schema constraints, redaction, policy checks, quarantine, or human review. Relevant inspection includes citations, attachments, encoded content, partial matches, derived attributes, and inference across repeated queries.

Inspection produces both false positives and false negatives. For each channel, the threat model states which error is safer and requires evidence that blocked content does not survive in logs, cache, memory, tool parameters, analytics, or reviewer interfaces. Authorization and minimization remain the primary defenses.

Pattern 5: Security evidence without indiscriminate content capture

Security evidence begins with the questions the organization must later answer. Who initiated and authorized the workflow? Which policy, control, dependency, and system versions applied? Which classified categories or resource references crossed a boundary? Which approvals, exceptions, disclosures, and external effects occurred? Did every required control complete?

Structured facts, protected identifiers, hashes, and references usually answer those questions better than full prompt and context capture. When an investigation requires sensitive payload evidence, store it separately under stricter access, integrity, retention, deletion, and legal-hold controls. Chapter 11 operates the telemetry; Chapter 9 defines the evidence and privacy properties it must preserve.

Pattern 6: Expiring governance gate

The decision package brings together product-risk inputs, threat-model findings, privacy and compliance obligations, responsible-AI impacts, dependency assurance, control evidence, and unresolved gaps. Consequence determines the required roles and quorum. The resulting decision binds system identity, use, population, data, autonomy, environment, exposure, constraints, and expiry.

Release and runtime controls consume this decision as enforceable state, not as a document link. According to policy, expired evidence, an unapproved dependency, broader authority, a new population, or a material incident blocks promotion or forces re-review. Chapter 10 implements that state transition.

A compact indirect-injection walkthrough

Consider a support assistant that retrieves tenant documents and drafts refunds for approval. An attacker hides instructions in a document that the victim's workflow may retrieve: ignore policy, search for other customers with similar account identifiers, include their details in the justification, and request the maximum refund.

Tenant records, refund authority, approval integrity, system policy, and audit evidence all need protection. The victim is both initiating and represented principal, while the retrieval and refund services execute under separate credentials. The document author remains untrusted. This path crosses content-ingestion, retrieval, model-context, approval, and action boundaries; if ingestion extracts text without preserving its source and author, the path also weakens provenance.

Layered controls alter the outcome at several independent points. Ingestion preserves the document's untrusted provenance, and retrieval enforces victim, tenant, and purpose scope before context assembly. The model invocation holds neither search nor refund credentials. The harness treats retrieved instructions as data and accepts only a structured proposal; the refund service then authorizes the exact customer, amount, and initiating principal independently. The approver sees authoritative account facts instead of relying on generated claims alone. Finally, egress policy keeps other-customer data out of the proposal, while structured evidence records policy versions, resource references, approval, and action receipt without copying all customer content.

The assurance package combines cross-tenant negative tests, indirect-injection challenges derived from the threat, approval-binding tests, and an architectural review of credential isolation. Chapter 8 owns the repeatable test design and results. Some risk remains: hostile content may distort a draft within the victim's authorized data, inspection may miss transformed sensitive meaning, and a persuasive draft may mislead a human reviewer. Governance can respond by limiting eligible refund classes, requiring approval, capping amounts, and expiring the authorization when retrieval sources or tool authority change.

7. Failure Modes

Failure mode 1: Prompt injection is treated as a string pattern

The team tests obvious direct phrases while ignoring indirect content, tool results, images, metadata, summaries, persistent memory, and delayed artifacts. An attacker preserves intent through a transformation or moves the effect into another workflow. Correct analysis follows reachable assets, attacker capability, the cross-boundary path, expected safe behavior, containment, and residual limitations—not merely forbidden wording.

Failure mode 2: Authorization collapses into backend access

A broad service identity retrieves data, and the application mistakes successful retrieval for permitted use. Somewhere before context assembly, it loses the tenant, represented principal, purpose, field scope, or current entitlement. An explicit policy decision, resource binding, independent enforcement, fail-closed ambiguity, and negative isolation evidence close that gap.

Failure mode 3: Credentials turn the system into a confused deputy

The model or workflow receives ambient authority broader than the task, allowing untrusted content to steer it toward unrelated resources, tools, or environments. Capability isolation, credential attenuation, parameter and target binding, short validity, revocation, and containment prevent one compromised step from expanding its reach.

Failure mode 4: Sensitive data escapes through a secondary channel

The main response is redacted, but the same content survives in citations, generated files, search queries, tool arguments, error text, memory, traces, analytics, evaluation assets, caches, or reviewer surfaces. An egress and persistence inventory applies audience-specific policy and derived-data treatment to every channel. Secondary-path tests then establish that rejected material does not remain elsewhere.

Failure mode 5: Privacy approval covers collection but not reuse

Data collected for a valid task is later reused for debugging, model improvement, evaluation, or analytics without a compatible purpose, notice, retention rule, or deletion behavior. Copies and derivatives outlive the primary record. A complete lifecycle record makes processors, embeddings, indexes, labels, backups, exports, legal holds, and secondary uses visible to the decision.

Failure mode 6: Supply-chain trust is implicit

A model, adapter, dataset, plugin, package, evaluator, or external service changes provenance, behavior, terms, processing location, retention, or isolation without security re-review. The inventory still lists its name, but that is not assurance. Trust depends on claim-based supplier evidence, an owned change channel, pinning or approval policy, containment, disablement, and explicit invalidation triggers.

Failure mode 7: Governance becomes a one-time sign-off

The record names reviewers but omits authority, evidence, constraints, exceptions, expiry, and re-review semantics. An old approval then survives a broader population, new data, increased autonomy, or expired evidence. A bounded decision contract, enforced at release, prevents that inheritance.

Failure mode 8: Responsible-AI review is reduced to generic principles

The review records fairness, transparency, and accountability as aspirations but identifies no affected non-users, differentiated harms, disclosure requirement, accountable human, contestability, correction, prohibited reuse, or recourse. Those concerns become reviewable only when an impact-specific obligation and safeguard record connects them to evidence and decision authority.

Failure mode 9: Compliance participation is not traceability

A compliance owner attends review, but no authoritative interpretation maps to the affected flow, required or prohibited behavior, control, evidence, effective date, or unresolved question. An obligations register creates that traceability. Engineering records can support qualified interpretation; attendance alone cannot establish compliance.

Failure mode 10: Auditability becomes a privacy sink

Full prompts, contexts, outputs, tool results, and reviewer notes are retained because they might be useful later. The evidence corpus becomes a high-value secondary store with weak purpose, access, integrity, and deletion controls. A question-driven schema, structured facts, protected references, separate payload access, and deliberate retention preserve auditability without creating another privacy sink.

Failure mode 11: Several controls share one failure mode

The prompt says not to disclose, the model classifies whether content is sensitive, and the same model validates its own output. One manipulation can defeat all three layers. Independent enforcement, deterministic policy where possible, constrained authority, justified human or external review, and explicit containment provide different failure modes.

Failure mode 12: Security response starts after the incident

The team discovers during an incident that nobody has pre-authorized revocation, secret rotation, source quarantine, tenant isolation, processor suspension, sensitive-evidence preservation, or restoration. Threat-specific safe-state and recovery requirements must reach Chapter 11 before launch.

8. Tradeoffs

A security tradeoff becomes reviewable only when it names the default, conditions for deviation, compensating controls, required evidence, and approving authority. The decision turns on consequence and reversibility, sensitivity and affected population, privilege breadth and exposure duration, attacker leverage, control independence, detectability, and the ability to revoke, delete, correct, compensate, or notify.

Utility versus data minimization

The default is to expose only the data needed for the represented principal and approved purpose. Broader context needs both an authorized data use and evidence that the product value cannot be obtained otherwise. A team accepting that deviation can narrow the population, filter fields, strengthen isolation, shorten retention, restrict processors, and add egress controls. The accountable data and risk authorities need to see the privacy-purpose analysis, Chapter 8 evidence of incremental utility, and leakage assurance before approving it.

Autonomy versus control

By default, no consequential external effect proceeds without independently enforced authority and an appropriate approval boundary. Greater autonomy may be justified for actions that are reversible, bounded, observable, and reliably reconciled. Lower limits, short-lived credentials, narrow capabilities, isolation, circuit breaking, and rapid revocation compensate for the added authority. The action owner and residual-risk authority accept the deviation only after reviewing abuse paths and ambiguous-outcome assurance.

Transparency versus information leakage

The default is to disclose AI involvement, decision responsibility, relevant evidence, limitations, and recourse without exposing secrets, unrelated personal data, or exploitable control detail. Specific details may need to remain restricted to protect people, investigations, or controls. Safe summaries, independent review, and an escalation or appeal path compensate for that restriction. Responsible product, security, and policy owners approve it against user-impact and leakage analysis.

Central governance versus team velocity

Consistent minimum gates are the default, with review depth increasing alongside consequence and exposure. Policy may let a team self-approve a low-consequence change that stays inside a previously reviewed boundary and presents the required evidence. Automated evidence checks, sampled oversight, shorter validity, escalation triggers, and immutable records constrain that delegation. It remains valid only while control maturity is demonstrated and the governance authority explicitly grants it.

Retention versus diagnosability

Retain the minimum structured evidence needed to reconstruct security decisions and effects. A defined investigation, assurance need, or obligation may justify keeping sensitive payloads, but only under separate storage, purpose binding, stricter access, integrity protection, short expiry, deletion verification, and legal-hold handling. Data, security, privacy, and any other required authorities approve the exception against a documented evidentiary need.

Supplier capability versus control

The system should rely only on dependency claims that are evidenced, owned, and either replaceable or containable. An opaque or hard-to-substitute dependency may still be necessary when available alternatives are materially worse. In that case, reduce its data and privilege, secure appropriate commitments, define monitoring and manual fallback, plan an exit, and shorten re-review. Current supplier assurance, concentration analysis, and explicit residual-risk acceptance support the decision.

9. Production Checklist

Before approving security, privacy, and governance readiness, confirm:

• The in-scope workflow, environments, entry points, outputs, administrative surfaces, assumptions, exclusions, and owners are declared.
• Protected assets have explicit security, privacy, provenance, isolation, purpose, and accountable-use objectives where applicable.
• Actors and initiating, represented, executing, approving, and accountable principals are distinguished.
• Component, data, authority, tenancy, processor, persistence, and egress flows are mapped.
• Direct and indirect prompt-injection paths cover persistent state, delayed effects, provenance loss, confused-deputy behavior, and reachable assets.
• Authorization is bound to resource, action, tenant, purpose, environment, and time as required, with independent enforcement and deny-by-default ambiguity.
• Reusable credentials are excluded from model-visible context; tool, network, data, and persistence authority is attenuated and revocable.
• Privacy records cover source, purpose, prohibited reuse, processors, transfers, derived stores, retention, deletion, correction, export, and legal hold.
• Leakage analysis covers responses, citations, files, tool parameters, queries, errors, memory, logs, analytics, evaluation assets, caches, reviewers, and derived inference.
• Dependencies have identities, provenance and integrity evidence, privileges, inherited claims, owners, update policies, containment options, and change triggers.
• Authoritative obligations map to affected flows, controls, evidence, owners, effective dates, and conservative handling of unresolved interpretation.
• Responsible-AI review covers affected parties, differential harm, disclosure, human accountability, contestability, correction, prohibited use, safeguards, evidence, and recourse.
• Each material threat or obligation maps to prevention, independent enforcement, containment, assurance, failure behavior, control owner, and residual risk.
• Repeatable security cases are handed to Chapter 8 with attacker capability, preconditions, required safe behavior, and severity.
• Security evidence is sufficient to reconstruct authorization, policy, system identity, disclosure, approval, exception, and external effect without indiscriminate content retention.
• Security-specific response defines safe-state actions, revocation, evidence handling, notification inputs, recovery authority, and re-review triggers for Chapter 11.
• Governance decision rights, quorum, outcomes, constraints, exceptions, residual-risk authority, expiry, and material-change triggers are explicit.
• Chapter 10 can bind the approval and its invalidation conditions to a named release and exposure state.

10. Design Review Questions

1. Which workflow and release identity does this threat model authorize, and which uses are expressly excluded?
2. Which assets require confidentiality, integrity, isolation, provenance, purpose limitation, or accountable use, and who owns each claim?
3. What crosses every important trust boundary, under which effective principal, policy decision, enforcement point, and failure behavior?
4. Which direct, indirect, persistent, and delayed injection paths can reach data, memory, approvals, tools, outputs, or future workflows?
5. Where can a broad service credential become a confused deputy, and how is its authority attenuated, bound, revoked, and contained?
6. What evidence proves that identifier substitution, stale entitlement, cross-tenant access, and time-of-check/time-of-use failures are denied?
7. Which data uses are permitted at collection, context, processing, review, evaluation, analytics, retention, export, and deletion—and which are prohibited?
8. Through which response, file, citation, query, tool, error, memory, cache, log, analytics, or reviewer channel could source or derived-sensitive information escape?
9. Which dependency claims are inherited, how current is their evidence, and what change, outage, or loss of trust forces containment or re-review?
10. Which authoritative obligations and impact concerns map to which flows, controls, evidence, safeguards, and recourse paths?
11. Which controls fail independently, which share a model or dependency, and what is the maximum reach after one control fails?
12. Which security claims require architectural review, penetration work, supplier evidence, or repeatable Chapter 8 evaluation?
13. What minimum structured evidence can reconstruct authorization, disclosure, approval, exception, and external effect without creating a privacy sink?
14. Which security event forces immediate revocation, isolation, quarantine, notification review, or suspension, and who may restore the capability?
15. Who can approve, constrain, grant an exception, suspend, or retire this use, and when does that authority expire?
16. Which changes in data, population, autonomy, dependency, processor, evidence, obligations, or incidents invalidate the current decision?

11. Main Artifact: AI Threat Model

The AI Threat Model is one living artifact expressed through ten coordinated views. A team may maintain those views in a document, repository, governance system, or linked records, provided their identifiers and traceability remain stable. It references rather than duplicates the Chapter 3 AI Fit and Risk Assessment Worksheet, Chapter 5 Context Assembly Plan, Chapter 6 Harness Control-Loop Diagram, Chapter 7 Tool Permission Matrix, Chapter 8 Evaluation Plan, Chapter 10 AI Release Manifest, and Chapter 11 AI Operations Runbook.

Artifact control and handoffs

• AI Threat Model identity and version:
• Source AI Fit and Risk Assessment Worksheet identity, version, and accepted decision IDs:
• Source Context Assembly Plan identity, version, and permission/source-trust boundary:
• Source Harness Control-Loop Diagram identity, version, and control/approval boundary:
• Source Tool Permission Matrix identity, version, and authority/effect boundary:
• Supporting Evaluation Plan identity, version, and applicable assurance-evidence IDs:
• Governance decision-record identity, constraints, residual-risk acceptor, and expiry:
• AI Release Manifest handoff: governance decision ID, enforceable constraints, and material-change triggers:
• AI Operations Runbook handoff: security event classes, containment constraints, evidence rules, and restoration authority:
• Safe behavior when a source decision is missing, stale, expired, or contradictory:

These handoffs are monotonic. A downstream release or operations record may narrow exposure, suspend capability, or demand stronger evidence, but it cannot silently broaden approved data use, population, principal, authority, autonomy, guarantees, or validity. Any proposed broadening returns to the artifact owner authorized to decide it.

View 1: Scope and security objectives

This view establishes:

• system and use-case identity, owner, environment, population, exposure, and workflow boundary;
• intended and prohibited uses, affected parties, consequence inputs, and accountability inherited from Chapter 3;
• assets: data, credentials, policy, instructions, model and data artifacts, indexes, memory, approvals, receipts, and evidence;
• security and privacy objectives for each asset;
• assumptions, inherited controls, evidence sources, exclusions, approving authority, and review date.

View 2: Architecture, data flows, and trust boundaries

Attach a component and data-flow diagram covering applications, policy services, context sources, indexes, model endpoints, memory, tools, queues, reviewers, analytics, administrative surfaces, external processors, tenancy zones, storage locations, entry points, egress paths, and enforcement points.

View 3: Threat and abuse-case register

The register covers the paths relevant to this scope, including direct and indirect injection, provenance loss, persistent or delayed effects, poisoning, data leakage, inference, cross-tenant exposure, credential misuse, confused-deputy action, approval manipulation, dependency compromise, material denial of service, privacy misuse, governance bypass, and responsible-AI abuse.

View 4: Access and authority model

For every sensitive data or action path, distinguish the initiating, represented, executing, resource-owning, approving, and accountable principals. The view also captures tenant, resource, field, action, purpose, environment, and time scopes; credential source, attenuation, expiry, rotation, and revocation; policy decision and enforcement points; recheck behavior; administrator and break-glass paths; and deny semantics.

View 5: Privacy and compliance obligations

Maintain a data-use record and obligations register. Each category identifies sensitivity, source, subject or affected party, collection context, approved purpose, prohibited secondary use, processors and subprocessors, transfer and residency boundaries, minimization, retention, deletion, correction, access, export, backup, and legal-hold behavior. The scope includes prompts, outputs, embeddings, indexes, memory, caches, labels, traces, evaluation assets, and derived-sensitive data.

Each authoritative obligation links its source and interpreting owner to the affected flow, required, prohibited, or conditional behavior, control, owner, evidence, effective date, evidence freshness, unresolved interpretation, and conservative interim behavior.

View 6: Dependency and supply-chain assurance

This view identifies every model, adapter, dataset, index, package, plugin, evaluator, or external service that affects a trust claim.

View 7: Responsible-AI impact review

The impact review covers intended users and affected non-users, material decisions or representations, plausible differentiated harms, accessibility and language conditions, disclosure requirements, human responsibility, contestability, correction, escalation, appeal, prohibited uses, and repurposing safeguards. It links these concerns to required Chapter 8 evidence, unresolved measurement gaps, Chapter 11 monitoring requirements, and the owner of residual impact.

View 8: Control and assurance matrix

Each material security claim traces back to its originating threat or obligation. The matrix separates prevention, independent enforcement, containment, detection requirements, response or recovery, and assurance. It names the control owner, enforcement point, dependencies, fail-open or fail-closed behavior, evidence identity and freshness, gap, and residual risk.

For assurance evidence to support a claim, it states the test preconditions, attacker capability, expected safe behavior, severity, evidence owner, cadence, acceptable result, limitations, failed-assurance response, and conditions requiring independent review. Repeatable behavioral tests link to Chapter 8 instead of recreating its Evaluation Plan.

View 9: Security response requirements

Before launch, define event classes for suspected disclosure, credential compromise, successful injection abuse, poisoned content or memory, prohibited processing, governance bypass, and untrusted dependency change. Each class specifies its severity inputs, immediate safe state, capability revocation, tenant or environment isolation, source quarantine, secret rotation, sensitive-evidence preservation, notification decision inputs, break-glass authority, recovery conditions, restoration authority, and required threat-model or governance re-review.

Chapter 11 consumes these requirements in the AI Operations Runbook and owns detection, command, communication, restoration procedure, and learning mechanics.

View 10: Governance decision record

The final view names the decision and its authority; required security, privacy, compliance, legal, responsible-AI, product, and operational roles; quorum and separation of proposer, reviewer, and acceptor; evidence package; unresolved findings; control owners; and residual-risk statement.

The outcome is explicit: approve, approve with constraints, request remediation, reject, suspend, or retire. Any approval binds use, release identity, population, data, autonomy, environment, exposure, and validity period. Exceptions carry a rationale, compensating controls, accountable acceptor, expiry, and remediation deadline. Material changes, incidents, new obligations or affected populations, expired evidence, and control failure all define re-review triggers.

Once issued, the decision record is immutable and linked to the release and operational state that consume it. A superseding decision may replace the record; silent editing may not.

12. Key Takeaways

• AI security is an argument about assets, principals, flows, enforceable controls, evidence, and bounded authority—not confidence in model behavior.
• Prompt injection is a family of direct, indirect, persistent, and delayed cross-boundary attack paths. Instruction hierarchy and provenance help, but they are not isolation.
• Access must remain bound to the represented principal, resource, action, purpose, tenant, environment, and time; backend capability is not permission.
• Privacy obligations follow data into prompts, outputs, embeddings, memory, logs, labels, evaluation assets, processors, retention, reuse, and deletion.
• Defense in depth requires prevention, enforcement, containment, assurance, and response controls that do not share a single failure mode.
• Dependencies, responsible-AI impacts, compliance interpretations, and security evidence belong inside the trust decision.
• Governance approval is a constrained, expiring authorization tied to a named use and system identity.
• The AI Threat Model is the living record that connects threat reasoning to controls, assurance, response requirements, and accountable residual-risk acceptance.