Chapter 14: Templates, Checklists, and Runbooks

1. Chapter Overview

Important AI systems decisions must outlive the meetings in which people make them. A useful toolkit records the system and population under review, the evidence available at the time, the authority behind the decision, the unresolved uncertainty, and the changes that would invalidate the result. Another engineer can then reconstruct the argument without relying on memory or inheriting claims from a stale document.

The chapter's primary engineering decision is how to select, tailor, and govern a reusable toolkit that preserves decision quality across design, release, operation, and change without turning delivery into document production.

The Complete AI Systems Engineering Toolkit packages the handbook's main artifacts. Chapter 14 owns the forms, packet assemblies, shared identifiers, lifecycle states, closure records, and maintenance rules that make those artifacts work together. Chapters 1 through 13 remain authoritative for the judgment inside each discipline. This chapter does not compress their methods into another framework; it provides faithful blank structures and a reliable way to connect them.

Completeness does not mean using every module for every change. It means that a bounded decision can move through its lifecycle with explicit applicability, authoritative references, proportionate depth, valid evidence, accountable disposition, closure, and a path for reopening what later evidence affects.

2. Primary Engineering Problem

Production AI decisions rarely happen in one place. Product fit may be settled before an architecture exists, while prompt and context contracts can evolve on separate schedules. Evaluation evidence may describe a different candidate or population from the AI Release Manifest. An incident can invalidate an assumption without reopening the design record that depends on it. Every artifact may look complete even when the system-level decision no longer holds together.

The engineering problem is:

Preserve decision scope, authority, evidence lineage, and change impact across a set of lightweight artifacts, while requiring only the depth justified by consequence and uncertainty.

Six operational objects keep those responsibilities distinct:

• A template is a reusable blank structure. It contains no approved claim.
• A record is a completed, identified instance of a template for an exact scope.
• A checklist is a memory and inspection aid. Its result is a set of findings, not an approval.
• A runbook is an executable decision-and-action path with authority, commands or references, verification, and stopping conditions.
• A packet is a version-pinned assembly of records and evidence prepared for one decision or operating action.
• A receipt records what actually happened, including effective state, actor, time, result, and deviations.

Blurring these objects creates false confidence. A release plan is not a deployment receipt, and a checked box is not a finding disposition. A dashboard link becomes evidence only when its query, population, time window, health, and limits are known. Plausible text in a template does not make the result an approved record.

3. Core Mental Model

Treat the toolkit as a decision system, not a document library.

Every material claim should support a reconstruction chain:

decision scope -> artifact version -> claim -> evidence reference -> limitation -> accountable authority -> disposition -> effective state -> reopening event

Four properties make this chain usable.

Identity

Every artifact and packet carries a stable identifier and version. A completed record binds to the system, release or workflow, population, environment, consequence class, and time horizon it describes. Similar titles never establish compatible scope.

Authority

Artifact owners maintain records; decision authorities approve defined consequences. Editing a file does not allow its owner to accept product, security, release, or operational risk. The record must name the authority for the decision at hand.

Evidence lineage

Records point to authoritative evidence instead of copying its conclusion. Each reference identifies the source, version or query, population, observation window, age, known limits, and invalidation conditions. Those links reveal the dependent decisions when evidence changes.

Lifecycle and closure

Intended, approved, effective, and actual state are not interchangeable. A finding remains open until someone verifies the correction and the authorized closer records the result. Expiry, supersession, incidents, and material changes may reopen a decision without erasing its history.

Shared semantics keep the toolkit lightweight. One common contract carries the header, evidence rules, and disposition vocabulary; each decision adds only the discipline-specific views it needs.

4. System Boundary

Chapter 14 owns reusable packaging and practical use. It does not redefine the AI System Stack, product-fit analysis, prompt design, context assembly, harness control, tool authority, evaluation, security, deployment, operations, architecture selection, or case analysis.

Each module in Section 11 therefore contains:

• its purpose and applicability;
• a compact blank structure preserving the owning chapter's material decisions;
• references to upstream and downstream records; and
• a route back to the owning chapter for method, examples, and judgment.

The forms are deliberately incomplete as explanations. An Evaluation Plan template cannot teach a team to design a regression suite, and an AI Operations Runbook cannot choose containment on the operator's behalf. Chapters 8 and 11 own those methods. As decision records, however, the forms are complete: they retain the identities, claims, authority, failure states, evidence, dispositions, and closure fields those chapters require.

Artifact contents may themselves be sensitive. Prompts, source names, threat scenarios, production queries, personal data, incident evidence, tool parameters, and recovery actions require access, retention, redaction, and audit rules appropriate to their risk. The toolkit manifest references those controls; it does not make the repository safe by declaration.

5. Design Principles

Principle 1: Reference authoritative evidence; do not clone it

Copying a result into several packets breaks invalidation. Keep evidence in one authoritative location where practical, then reference its identity, scope, age, limitations, and change conditions. When audit reconstruction requires a snapshot, label it immutable and record the authoritative source and capture point.

Principle 2: Bind every record to an exact decision scope

Name the system, workflow, release or configuration, population, environment, consequence, and period. Evidence for advisory drafting cannot approve automated action. A threat model for one tenant boundary does not survive a topology change by implication.

Principle 3: Separate authorship, review, decision, and execution authority

Record the maintainer, reviewers, residual-risk authority, transition authority, and executor. One person may hold several roles, but the record must keep their authorities distinct.

Principle 4: Tailor by consequence and uncertainty, not delivery pressure

Every packet begins with a small mandatory core. Greater authority, blast radius, uncertainty, irreversibility, data sensitivity, external effect, operational complexity, or recovery difficulty activates additional modules and views. An omission needs a rationale and an authorized tailoring decision with an expiry.

Principle 5: Preserve state, findings, and dissent

Draft, in review, conditionally approved, approved, effective, expired, superseded, and archived are distinct states. Conditional approval has bounded conditions. Findings carry severity, consequence, owner, due date, verification, and closure authority. Dissent is part of the decision record, not noise to delete.

Principle 6: Make change traverse dependencies

Start a change review from the assumption, contract, component, population, authority, or operating condition that changed. Trace outward to the affected claims, evidence, records, decisions, and releases. A record may remain valid, but someone must document that conclusion rather than assume it.

Principle 7: Test the toolkit by reconstruction and execution

Test the toolkit in use. Another engineer should be able to reconstruct the decision, and an operator should be able to follow the runbook without inventing authority, query semantics, or completion criteria. Automation can check completeness; it cannot supply engineering judgment.

Principle 8: Give the toolkit an owner and retirement path

Templates are production dependencies. Give them versions, compatibility notes, migration guidance, local-extension rules, deprecation dates, staleness review, feedback ownership, and archives. Without that stewardship, teams keep completing obsolete forms after the source chapters or governed system have changed.

6. Architecture Patterns

The toolkit assembles its modules into four packets. Each packet is a decision workflow, not a folder of attachments. It declares entry criteria, required claims, exact record versions, participants and authority, walkthroughs, findings, disposition, validity, and reopening events.

Lifecycle handoff

text product decision -> design packet and architecture disposition -> release packet bound to one composite candidate -> deployment and operating receipts -> operations packet and incident records -> change packet traverses affected claims and decisions -> revised design, release, or operating records

The handoff uses stable references for decisions, evidence, risks, findings, releases, actions, and outcomes. Packets do not copy one another's prose. A release packet cites the approved design decision and current evaluation result. An operations packet cites the effective release receipt. A change packet records which of those references remain valid.

Pattern 1: Design Review Packet

Purpose: Decide whether the proposed responsibility, architecture, authority, and control argument are coherent enough to proceed.

Entry criteria: The problem and non-goals are defined; affected parties and consequence are known well enough to bound review; a design option and a credible simpler alternative exist; open assumptions are visible.

Core modules: Toolkit manifest and tailoring record, AI System Design Brief, AI Fit and Risk Assessment Worksheet, relevant Reference Architecture Catalog record from Chapter 12, evidence index, finding log, and decision record. Prompt, context, harness, tool, evaluation, and threat modules are added when those contracts are material to the decision.

Required walkthroughs: Canonical success, credible failure, denied or missing authority, degraded mode, recovery or correction, and one material change.

Dispositions: Approve, approve with conditions, return for redesign, reject, or defer for missing evidence. Approval is scoped and dated; it does not authorize release.

Reopening events: Responsibility, affected population, consequence, autonomy, authority, source class, pattern composition, material dependency, or recovery assumption changes.

Pattern 2: Release Readiness Packet

Purpose: Decide whether one immutable composite candidate may enter a named environment and exposure stage.

Entry criteria: The governing design disposition is valid; the candidate is identifiable; required evidence and risk reviews apply to that candidate and population; rollout, health decisions, containment, recovery, and operating ownership are ready.

Core modules: AI Release Manifest, Evaluation Plan result and disposition, AI Threat Model decision, applicable prompt/context/harness/tool records, operating-readiness evidence, risk and exception register, and packet decision record.

Required walkthroughs: Initial exposure, hold, advance, indeterminate evidence, rollback or containment, mixed-version behavior, in-flight work, and post-transition verification.

Dispositions: Approve named stage, approve named stage with conditions, hold, reverse, reject candidate, or defer for missing evidence. The deployment receipt records what actually became effective.

Reopening events: Any component identity changes, target population or environment changes, evidence expires, a blocking condition recurs, recovery is unavailable, or effective state diverges from the manifest.

Pattern 3: Operations Packet

Purpose: Operate the effective service, detect behavioral and execution failure, coordinate incidents, verify recovery, and close learning actions.

Entry criteria: Effective release and topology are known; objectives and evidence-health rules exist; operators have authoritative queries and bounded actions; command roles and escalation paths are staffed; recovery verification is possible.

Core modules: AI Operations Runbook, effective release receipt, objective and signal catalog, action and authority references, incident record, evidence index, recovery record, and improvement finding log.

Required walkthroughs or exercises: Missing telemetry, quality regression, dependency failure, unsafe or duplicated action, stale or conflicting context, containment expiry, queue and in-flight work, recovery proof, and communication.

Dispositions: Continue, observe with bounded condition, degrade, contain, declare incident, recover, accept a scoped degraded mode, or retire. Individual actions still require their recorded authority.

Reopening events: Objective or evidence-health failure, incident, authority or topology change, ineffective containment, failed recovery exercise, or overdue critical improvement action.

Pattern 4: Change Review Packet

Purpose: Determine what a proposed or observed change invalidates and what must be re-decided before consequence increases.

Entry criteria: The changed object and intended effective scope are identified. Previous decisions and effective state are discoverable.

Core modules: Change-impact record, affected claim and dependency map, retained and missing evidence register, affected module versions, required re-reviews, owner and due-date plan, and change disposition.

Required walkthroughs: Compatibility, population and authority drift, evidence invalidation, in-flight work, rollback or correction, downstream records, and archive or supersession.

Dispositions: No material impact, bounded update, re-evaluate, re-review design, prepare new release, require operational action, defer, or reject. No material impact requires recorded reasoning and authority.

Reopening events: A supposedly unchanged assumption proves false, the actual effect differs from the proposed effect, or dependent evidence becomes invalid.

Minimum working packet

A bounded, reversible, advisory pilot still needs an AI System Design Brief, an AI Fit and Risk Assessment Worksheet, an identified candidate, applicable Production Prompt Specification, Context Assembly Plan, and Harness Control-Loop Diagram records, an Evaluation Plan disposition, an AI Release Manifest and rollback record, an operations owner, an evidence index, a tailoring record, and a decision receipt. Compact views may reference existing organizational records. Security, tool, action, migration, and incident views become mandatory when the pilot handles sensitive data, takes external action, creates durable state, crosses trust boundaries, or cannot be reversed cleanly.

The minimum does not bypass material decisions; it represents a smaller consequence with less ceremony.

7. Failure Modes

8. Tradeoffs

The toolkit steward decides packaging policy. The authority named in a completed packet decides whether its consequence is acceptable. Neither role can stand in for the other.

9. Production Checklist

Before a team relies on the toolkit, verify:

• [ ] The toolkit manifest names its version, steward, canonical location, module catalog, source-chapter compatibility, extension rules, and deprecation policy.
• [ ] Every live record has a stable identity, exact decision scope, owner, decision authority, state, dates, validity, and change triggers.
• [ ] The applicability and tailoring record lists every module, each omitted section, the rationale, the accepting authority, the compensating control, and the expiry or invalidation event.
• [ ] Packet entry criteria are met and every referenced artifact is pinned to an exact version or immutable result.
• [ ] Claims link to current evidence with population, window, age, limitations, and invalidation conditions; copied evidence is explicitly controlled.
• [ ] Intended, approved, effective, and actual state are distinguishable, and transition receipts can be reconciled with authoritative runtime state.
• [ ] Reviewers and decision authorities have the required domain, consequence, and risk rights; dissent and absent expertise are recorded.
• [ ] Blocking findings are closed with verification or bound to a valid conditional disposition whose scope, authority, due date, and expiry are explicit.
• [ ] The release packet identifies the composite candidate, target population and environment, exposure decision, recovery evidence, and operating owner.
• [ ] Operators can move from a signal through evidence-health validation, declaration, containment, investigation, recovery proof, communication, and learning closure using tested references.
• [ ] A change can traverse affected assumptions, claims, evidence, records, decisions, releases, and owners without relying on document titles or memory.
• [ ] Sensitive artifact content has appropriate access, redaction, retention, deletion, and audit controls.
• [ ] Inline chapter forms and standalone templates declare the same toolkit schema version or a documented compatibility or migration relationship.
• [ ] One design decision, release transition, incident action, and material change have been reconstructed from records by someone other than the author.
• [ ] Obsolete templates and records are expired, superseded, migrated, or archived rather than left silently discoverable as current.

10. Design Review Questions

1. Which material decision cannot be reconstructed from its scope, evidence, authority, disposition, and receipt chain?
2. Which claim remains apparently valid only because evidence was copied rather than referenced?
3. Where could an approval for one candidate, population, environment, or consequence be mistaken for another?
4. Which owner maintains a record but lacks authority to accept the risk described in it?
5. What module or section was omitted, who accepted the omission, and what event invalidates that tailoring?
6. Which finding could remain open while exposure increases, and what control prevents that transition?
7. What change would leave a stale design, evaluation, threat, or release decision appearing current?
8. Can the effective release be reconstructed from receipts rather than intended configuration?
9. What would an operator need to invent during an incident: query meaning, command authority, action scope, verification, reversal, or communication?
10. Which runbook action was most recently exercised, under what conditions, and what limitation remained?
11. What happens when evidence is missing, delayed, unhealthy, conflicting, or outside its approved population?
12. Which field is routinely empty because it adds no value, and which is empty because the team has not made the decision?
13. Can a local extension be migrated without losing authority, evidence, findings, or history?
14. Who can supersede this decision, and how will downstream users discover that it is no longer effective?
15. Can another engineer explain what was decided, by whom, from which evidence, under which accepted limits, and what reopens it?

11. Main Artifact: Complete AI Systems Engineering Toolkit

The toolkit below uses schema version PAISEH-TK-1.0. Teams may store its forms separately, but every standalone form declares the schema version, uses the common header, and stays linked through the manifest. Chapter 14 owns the packaging schema; the cited source chapter owns each module's engineering method.

Artifact control and source compatibility

The Complete AI Systems Engineering Toolkit carries a toolkit ID, schema version, source handbook version, steward, release record, and compatibility statement. Its catalog names each exact source artifact—not merely the chapter—and the source version used to check the blank structure. Every toolkit release records semantic additions, removals, renamed fields, migrations, and source decisions that require teams to revisit completed records.

Packaging is a monotonic handoff: a template may preserve a source decision, request stronger evidence, or impose a narrower operating ceiling. It cannot silently broaden an upstream artifact's scope, population, authority, data use, accepted risk, guarantees, or validity. Any field that permits broader consequence routes back to the authorized owner of the source decision.

The Chapter 13 Case Study Analysis Format hands findings, evidence limits, affected record identities, and reopening requirements into toolkit change packets. The toolkit packages those records but does not absorb case-analysis method or treat an illustrative outcome as reusable approval evidence.

A. Toolkit manifest

The canonical manifest governs template identities and compatibility. Completed records remain authoritative in their named systems of record; the manifest points to them without replacing them.

B. Common artifact header

Apply this header to every completed module, packet, finding register, and receipt.

A completed record may narrow or block an upstream decision. Any proposed broadening of scope, population, authority, data use, accepted risk, guarantee, or validity returns to the authorized owner of the source artifact.

C. Lifecycle states and transition record

Use only states that apply, but do not collapse their meanings.

D. Applicability and tailoring record

Tailoring cannot remove the mandatory core: identity, scope, ownership, authority, evidence, state, disposition, findings, closure, and change triggers. It may reduce representation, never erase a material decision.

E. Packet manifest and decision record

F. Evidence, finding, risk, and change records

Evidence index

Finding and closure log

Risk and exception register

Change-impact record

G. Module 1: AI System Design Brief

Source artifacts: Chapter 1, AI Systems Engineering Definition and Principle Set; Chapter 2, AI System Stack Reference Model; Chapter 12, Reference Architecture Catalog. Purpose: establish responsibility, boundary, system shape, and open design decisions. It does not replace focused discipline records.

H. Module 2: AI Fit and Risk Assessment Worksheet

Source artifact: Chapter 3, AI Fit and Risk Assessment Worksheet. Purpose: decide whether to proceed, narrow, pilot, require human approval, defer, or reject before architecture detail creates momentum.

I. Module 3: Production Prompt Specification

Source artifact: Chapter 4, Production Prompt Specification. Purpose: define a maintainable instruction and outcome contract. Context, harness, evaluation, threat, and release records remain separate references.

J. Module 4: Context Assembly Plan

Source artifact: Chapter 5, Context Assembly Plan. Purpose: define the evidence contract and the authorized, traceable context package supplied to the model.

K. Module 5: Harness Control-Loop Diagram

Source artifact: Chapter 6, Harness Control-Loop Diagram. Purpose: make workflow states, enforceable transitions, bounded recovery, approval, and terminal outcomes inspectable.

L. Module 6: Tool Permission Matrix

Source artifact: Chapter 7, Tool Permission Matrix. Purpose: define who or what may request, approve, execute, reconcile, and verify effects.

Also record:

• workflow and delegation scope, including whether authority may be narrowed or transferred;
• pre-action revalidation of identity, state, eligibility, approval snapshot, and cancellation;
• long-running states, leases, heartbeats, checkpoints, takeover, and terminal ownership;
• retry rules based on effect knowledge rather than transport error alone;
• compensation limits and irreversible effects;
• required audit events and authoritative effect receipts; and
• tool-specific checks, exception authority, expiry, and reopening events.

M. Module 7: Evaluation Plan

Source artifact: Chapter 8, Evaluation Plan. Purpose: bind claims and risks to valid evidence and an accountable decision to ship, hold, narrow, reject, or defer.

N. Module 8: AI Threat Model

Source artifact: Chapter 9, AI Threat Model. Purpose: record trust, privacy, access, misuse, dependency, affected-party, assurance, governance, and residual-risk decisions.

O. Module 9: AI Release Manifest

Source artifact: Chapter 10, AI Release Manifest. Purpose: bind an immutable composite candidate to topology, configuration, exposure, recovery, health decisions, and actual transition receipts.

P. Module 10: AI Operations Runbook

Source artifact: Chapter 11, AI Operations Runbook. Purpose: give operators an executable path from signal to evidence validation, command, containment, investigation, recovery, communication, and learning.

For each alert and action, record the objective, population, window, threshold, evidence completeness, acknowledgement, deduplication, automated authority, last exercise, command or procedure reference, stop condition, and receipt. Links named dashboard or rollback do not make the procedure executable.

Q. Module 11: Principal Design Review Checklist and Record

Source artifact: Chapter 12, Reference Architecture Catalog, especially its design-review record. Purpose: inspect an exact packet, create evidence-indexed findings, and record an accountable design disposition. The checklist does not grant approval.

The completed packet decision record names reviewer rights, walkthroughs, findings, dissent, residual risks, disposition, conditions, closure requirements, decision authority, validity, expiry, and reopening events. Its valid dispositions are approve, approve with conditions, return for redesign, reject, and defer for missing evidence.

R. Compact usage trace

This compact trace reuses Chapter 13's support knowledge assistant to show the toolkit mechanics. It adds no new case analysis.

Packet selection: Design packet DRP-support-draft-v3 includes AI System Design Brief DB-17, AI Fit and Risk Assessment Worksheet FR-09, Production Prompt Specification PS-12, Context Assembly Plan CP-08, Harness Control-Loop Diagram HL-05, Evaluation Plan EP-11, AI Threat Model TM-07, and Principal Design Review Checklist and Record PDR-06. Because the system cannot send, resolve, waive policy, or alter records, the tailoring authority marks the Tool Permission Matrix as not applicable. Any proposal for an external effect expires that omission.

Claim and evidence: Claim CL-24 states that trained agents can detect unsupported material claims in the standard-product pilot. Evidence reference EV-31 identifies the blinded review study, exact population, queue conditions, and result, along with declining detection after sustained review. That limitation caps the assisted queue and preserves ordinary search as the overload mode. The claim does not treat a citation as proof of currency.

Finding and disposition: Finding F-18 records that indexing does not reliably preserve policy-notice effective dates, so it blocks policy-sensitive drafting. The design authority approves only standard-product questions and attaches conditions. Closure requires an effective-date lineage check verified by the context owner; delivery pressure cannot convert the condition into approval.

Release and receipt: Release REL-14 binds the approved prompt, source policy, index, route, and exclusion rules to the pilot population. Receipt DEP-14-02 records which identities became effective and the exposure actually achieved. The operations packet cites that receipt, not the intended manifest.

Change and reopening: A source-lifecycle defect later drops an effective date. Change record CHG-22 invalidates the context source-policy evidence and the release health claim, reopening CP-08 and REL-14. Record FR-09 remains valid because responsibility, consequence, and authority have not changed. The operations record contains policy-sensitive drafting, identifies affected responses from source and release identities, and requires recovery proof before bounded exposure resumes.

The preservation test now has a concrete answer. Another engineer can identify the decision, its evidence and authority, the covered population and limitation, the effective state, and why the later defect reopened two records instead of the entire toolkit.

S. Toolkit stewardship record

12. Key Takeaways

• Govern the toolkit as a decision system, not a collection of forms.
• A template becomes a trustworthy record only when identity, scope, authority, evidence, state, and change triggers are explicit.
• Packets pin exact artifacts and turn design, release, operation, and change work into findings, dispositions, closure, and receipts.
• Scale the toolkit through recorded tailoring and consequence-based triggers; never hide an omitted decision.
• References preserve evidence lineage, receipts preserve what happened, and reopening rules preserve validity as the system changes.
• Chapter 14 packages the handbook's artifacts while Chapters 1 through 13 retain ownership of the engineering judgment inside them.
• The practical test is reconstruction and execution: another engineer can recover the decision, and an operator can act without inventing authority or semantics.