Chapter 5: Context Engineering and Retrieval

1. Chapter Overview

Context engineering decides what information a model sees, when it sees it, under whose authority, and in what form. The result is not a pile of search hits but a bounded, labeled evidence package for one model interaction.

A production system may draw from user input, operational records, documents, policies, workflow state, prior interactions, and completed tool results. None is eligible merely because it is available. The task, actor, purpose, authorization, freshness, authority, and usable evidence budget determine eligibility.

The primary engineering decision in this chapter is the Context Assembly Plan, a reviewable contract for sourcing, authorizing, retrieving, ranking, reconciling, shaping, budgeting, labeling, and expiring evidence. Reviewers should be able to inspect three decisions:

• why an item was allowed into the candidate set
• why it was selected, transformed, or dropped
• whether the resulting package is complete enough for the requested task

This chapter owns context sources, retrieval, ranking, compression, memory, token budgets, source trust, context lifecycle, and permission-aware assembly. Prompt instructions belong to Chapter 4. Workflow control belongs to Chapter 6. Tool execution belongs to Chapter 7. Evaluation methodology, security policy, release mechanics, and operational telemetry belong to their respective chapters.

2. Primary Engineering Problem

AI systems often fail because the interaction contains the wrong evidence, not because the model lacks general capability.

Thousands of plausible documents do not compensate for a missing policy version, current account state, or controlling exception. Semantically close text may carry no authority for the fact at issue. A summary may preserve the topic yet delete a decisive date or negation. Memory may carry forward an intent the user has withdrawn. Even a cached index can disclose that a protected record exists without displaying the record itself.

The context layer must therefore solve a constrained evidence-selection problem:

Supply enough authorized, current, and task-relevant evidence to support the interaction, while excluding noise, protected information, unsupported authority, and content that cannot fit without losing decisive meaning.

The problem starts before retrieval. The owning team first states which facts the task requires. Explaining a general policy may need only the policy currently in force and its scope. Explaining an account-specific outcome may also need the actor's record, the version in force when the event occurred, and any applicable exception. When those requirements remain implicit, search relevance becomes a proxy for product intent.

Retrieval serves two distinct jobs. Candidate generation finds potentially useful material with adequate recall. Evidence selection chooses the eligible candidates that best satisfy the task's fact requirements. Combining the jobs encourages fixed top-result lists, similarity-only ranking, and accidental trust in repeated copies.

After assembly fails, the context layer does not choose the wider workflow's response. It does report a definite outcome: complete, partial, stale, conflicting, unauthorized, or unavailable. The harness can act on that fact instead of asking the model to infer whether its own evidence is adequate.

3. Core Mental Model

Context assembly works as a controlled information supply chain. Each stage receives a defined input, makes one bounded decision, and returns an auditable output or a named failure state.

Stage 1: State the evidence requirements

Translate the task into required fact types, optional supporting facts, freshness constraints, and explicit non-requirements. Describe the information needed rather than the mechanism used to find it. “Current entitlement and the rule effective when the event occurred” is an evidence requirement; “retrieve five chunks from the policy index” is an implementation choice.

Output: an evidence-requirements record with required, optional, and prohibited categories.

Failure state: the task is too ambiguous to determine what evidence is needed.

Stage 2: Establish the assembly scope

Bind the interaction to the actor, tenant or domain, declared purpose, workflow role, relevant time, and inputs to the authorization decision. Chapter 9 owns the policy; context assembly applies the resulting decision before protected information can enter the candidate set.

Output: a scope that every source adapter and cache must preserve.

Failure state: identity, purpose, or effective permission scope cannot be established.

Stage 3: Select candidate sources

Choose sources by authority domain and by their ability to satisfy the evidence requirements. Authenticity alone does not make a source authoritative for every fact. A user's statement governs current intent, for example, but not an account balance. A derived knowledge article may explain a policy without superseding the signed policy record.

Output: a source plan that maps fact types to primary, derived, advisory, or user-asserted sources.

Failure state: no approved source can provide a required fact type.

Stage 4: Retrieve within the eligibility boundary

Derive queries or record lookups from the task and required fact types. Retrieval then stays inside the established permission and source scope. Metadata, tenant, field, effective-time, and record filters are hard eligibility constraints, not soft ranking features. Precomputed indexes and caches preserve that same boundary.

Output: an eligible candidate set with stable source references and a retrieval timestamp.

Failure state: a source is unavailable, a request is denied, or no eligible candidates exist. “No eligible evidence” must not disclose whether inaccessible evidence exists.

Stage 5: Normalize, deduplicate, and bind provenance

Represent candidates in a consistent evidence form while preserving source identifiers, effective dates, retrieval timestamps, authority class, and transformation history. Collapse exact or near-duplicate copies so repetition cannot crowd out distinct evidence. Records that disagree materially remain separate.

Output: provenance-bound evidence units suitable for comparison.

Failure state: provenance, effective time, or authority cannot be established for a required item.

Stage 6: Rank for task usefulness and coverage

Rank eligible evidence against the task rather than textual similarity alone. Useful dimensions include:

• authority for the required fact type
• effective-time and freshness fit
• specificity to the actor, object, or event
• directness of support for a required fact
• coverage of evidence requirements
• redundancy with already selected evidence
• uncertainty or transformation distance from the primary source

Candidate generation may favor recall; final selection favors decisive, non-redundant coverage. Selection stops when every required fact type meets its declared support standard or the package records the remaining gap. A fixed result count is a resource limit, not evidence of sufficiency.

Output: a selected set with selection reasons and identified evidence gaps.

Failure state: eligible candidates exist but cannot satisfy one or more required fact types.

Stage 7: Resolve or expose conflicts

A task-specific source policy decides whether a conflict has a valid winner. Automatic precedence is safe only when authority and effective-time rules establish one. If authoritative records disagree, their effective periods are unclear, or a user assertion conflicts with a system of record, the package preserves the disagreement. Ordering and rank scores must not manufacture certainty.

Output: resolved evidence with the rule used, or a conflict set labeled as unresolved.

Failure state: a required fact remains materially conflicted and the source policy forbids automatic resolution.

Stage 8: Shape evidence and allocate the token budget

Begin with lossless shaping: field projection, deduplication, boilerplate removal, canonical formatting, and bounded passage extraction that leaves meaning intact. Lossy compression is acceptable only with explicit protected-fact invariants. Dates, units, identifiers, qualifiers, exceptions, negations, and provenance anchors commonly need protection.

Allocate capacity before appending content. Fixed interaction overhead, current user input, required evidence, provenance labels, and output headroom receive hard reservations; optional supporting evidence and memory use elastic allocations. When the package is too large, a predetermined shedding order replaces blind truncation from the end.

Output: budgeted evidence units with transformation lineage.

Failure state: the minimum viable evidence package cannot fit without violating a protected reservation or fidelity invariant.

Stage 9: Validate the package

Bounded checks confirm that permission scope remains intact, required evidence is present or marked missing, freshness satisfies policy, conflicts remain visible, provenance is complete, budgeting is deterministic, and protected facts survived shaping. These checks validate the assembly contract. They do not replace the representative scenarios, quality scoring, regression strategy, or launch thresholds owned by Evaluation Engineering.

Output: a validated context package and assembly metadata.

Failure state: the package violates an assembly invariant.

Stage 10: Emit the context package

At minimum, the package exposes:

• task and evidence-requirements identity
• assembly version and permission-scope reference
• supplied evidence units with source, authority, effective time, and transformation labels
• missing or excluded required evidence categories
• unresolved conflicts
• completeness and freshness state
• token allocation and shedding actions

The package reports the condition of the evidence. Retry, approval, escalation, and response behavior remain downstream control decisions.

4. System Boundary

The context layer begins when a task requires information and ends when a labeled evidence package becomes available to one model interaction. Between those points, it owns source selection, eligibility enforcement, retrieval, normalization, ranking, deduplication, conflict representation, shaping, memory selection, token allocation, provenance, freshness, and lifecycle rules.

It does not include:

• writing prompt instructions or outcome contracts
• invoking tools or authorizing actions
• routing retries, approvals, or state transitions
• defining enterprise access-control or privacy policy
• designing full-system evaluation methodology
• bundling releases or operating deployment controls
• building telemetry platforms or incident processes

Completed tool results may become context. This chapter governs their eligibility, provenance, authority, and shaping only after execution. Authorization decisions also constrain context, but Chapter 5 applies those decisions rather than defining the underlying security policy.

Four distinctions keep the boundary practical:

1. Available information is not supplied context. Access to a store creates candidates, not entitlement or relevance.
2. Context is not truth. Evidence can be stale, derived, conflicted, incomplete, or adversarial; those conditions must remain visible.
3. Memory is not authority. Stored preferences and summaries cannot silently override current explicit input or authoritative records.
4. Retrieval is not understanding. Similarity helps find candidates; evidence sufficiency is a task-level judgment.

5. Design Principles

Principle 1: Design backward from evidence requirements

Begin with the facts the task needs and the standard each fact must meet. Only then map those requirements to sources. This order makes missing evidence observable and prevents whatever happens to be available from defining product behavior.

Principle 2: Make eligibility a hard boundary

Actor, purpose, tenant, role, record, field, and time constraints apply before a candidate can be selected or exposed. Broad retrieval followed by an instruction to ignore protected content is not an access boundary. Cached summaries and indexes inherit the authorization scope of their underlying records.

Principle 3: Separate authority, authenticity, and freshness

An authentic document may still lack authority for the fact in question, while an authoritative record may be stale for the relevant effective time. Record these dimensions independently. Precedence follows the fact type and effective period, not the source's general reputation.

Principle 4: Rank for sufficiency, not just similarity

Relevance has several dimensions. Evidence earns selection by being authorized, authoritative, current, specific, and directly supportive; repeated copies add little. Selection stops when required coverage is achieved or returns an explicit gap.

Principle 5: Degrade intentionally under budget pressure

Budget policy needs hard reservations, elastic allocations, and a shedding order. Optional memory or background evidence cannot displace current user input, required evidence, provenance, or output headroom. If the minimum viable package will not fit, assembly reports failure or partiality rather than silently truncating it.

Principle 6: Preserve decisive facts through shaping

Lossless shaping is the default. A lossy transformation names the facts that must survive, labels its output as derived, and retains a recovery reference wherever accountability requires the original evidence.

Principle 7: Give memory explicit semantics

Current-turn state, session summaries, workflow state, durable preferences, and inferred facts are different memory classes, not one conversational history. Each class needs its own purpose, provenance, scope, precedence, correction, retention, expiration, and read-time eligibility.

Principle 8: Version the assembly contract

A context asset identifies its source catalog, eligibility assumptions, retrieval and ranking configuration, shaping behavior, budget policy, memory policy, package schema, owner, and rationale. Chapter 5 makes that asset identifiable and declares compatibility implications; Chapter 10 owns bundling, deployment, and rollback.

6. Architecture Patterns

Pattern 1: Bounded authoritative record projection

When the authoritative entity and required fact types are known, project named fields from the system of record. This pattern gives deterministic provenance but can mistake an incomplete schema projection for the whole truth. The plan therefore states required fields, effective-time semantics, and the result of an absent field.

Pattern 2: Permission-scoped retrieval and reranking

When evidence is distributed across a large corpus, generate candidates inside the actor's effective scope, then rerank for authority, specificity, freshness, coverage, and redundancy. The characteristic failure is a semantically similar passage being mistaken for decisive evidence.

Pattern 3: Provenance-preserving document evidence

Policies, manuals, and contractual material often require bounded passages with stable document identity, section location, effective date, and authority class. The chief risk is severing a passage from a qualifier elsewhere in the document, so selection rules preserve the surrounding conditions that govern meaning.

Pattern 4: Scoped memory overlay

For continuity, workflow resumability, or durable preferences, add eligible prior state only after current input and authoritative records are established. Explicit precedence and correction rules prevent stale or inferred memory from overriding present evidence.

Privacy, consent, and retention policy are defined in Chapter 9. The Context Assembly Plan implements the approved policy by controlling which authorized memory is read, how it is labeled, and what can override it.

Pattern 5: Multi-source evidence composition

When no single source covers the task, combine records, documents, user assertions, memory, and completed tool results under a fact-type source policy. Composition preserves each item's authority domain and exposes disagreements that lack a valid precedence rule; otherwise, source diversity merely hides conflict.

Pattern 6: Budgeted context tiers

Where context size or latency varies materially by interaction, divide evidence into protected and elastic tiers. A sound order protects current input, required evidence, provenance, and output headroom; retains high-value supporting evidence next; and sheds redundant passages, optional background, and low-priority memory last. A shedding rule chosen for implementation convenience rather than evidence requirements defeats the pattern.

A compact assembly decision

Consider a task that needs both the rule effective on an event date and the actor's current status. Retrieval finds a current policy summary, an older signed policy, three copied help articles, and a current structured account record.

The source policy treats the signed policy as authoritative for policy wording, the account record as authoritative for current status, and the help articles as derived explanation. The event predates the current summary, so freshness alone does not make that summary decisive. Deduplication collapses the copied articles. The older signed policy is retained because its effective interval covers the event. If its exception paragraph cannot fit, the system sheds the derived help article before compressing the exception. If two signed policies claim the same effective interval, the package reports an unresolved conflict rather than selecting the one with the higher similarity score.

The decisive choice is not which item ranks first. Evidence requirements, authority, effective time, coverage, conflict, and budget jointly determine what the model sees.

7. Failure Modes

Failure mode 1: Context stuffing

Every plausible item enters the package. Signal weakens, cost and latency rise, and no reviewer can reconstruct a selection decision that never occurred. Evidence requirements, deduplication, and sufficiency-based stopping replace accumulation.

Failure mode 2: Permission-blind candidate generation

The retriever searches beyond the effective actor or purpose scope and filters late. By then, ranking features, snippets, counts, or caches may have disclosed protected metadata. Eligibility begins at source access and survives every derived artifact.

Failure mode 3: Similarity masquerades as usefulness

The highest-scoring passages share the task's vocabulary but do not establish its required facts. Separating candidate recall from final selection allows authority, effective time, specificity, and coverage to govern the final package.

Failure mode 4: Stale authority

The newest retrieved item is assumed to govern an earlier event, or a formerly authoritative record overrides a current fact. Recording both retrieval time and effective time lets the fact requirement choose the applicable evidence.

Failure mode 5: Conflict disappears in ordering

Two sources disagree, yet only the first result is supplied or the lower-ranked item disappears. The package now presents false certainty. A valid precedence rule resolves the conflict; without one, both items remain visible as unresolved.

Failure mode 6: Compression removes the decision fact

A summary keeps the theme but drops a date, unit, exception, negation, or identifier that controls the answer. Lossy output must protect decisive elements, identify itself as derived, and retain a recovery link to the source.

Failure mode 7: Token truncation violates priorities

The packet exceeds its limit, so tail content is cut. Required evidence or provenance vanishes because optional history was appended first. Allocation must precede assembly, with overflow shed by declared evidence priority.

Failure mode 8: Memory becomes shadow state

A session summary or inferred preference quietly overrides current input or an authoritative record. Explicit precedence, correction, expiration, and write ownership keep memory from becoming an invisible second database.

Failure mode 9: Missing evidence becomes hallucination pressure

The package lacks support for a required fact but does not declare the gap. The request then appears answerable, inviting unsupported completion. A partial or unavailable state names the missing category.

Failure mode 10: Context changes cannot be reconstructed

A source, ranker, compression rule, or memory policy changes without a new asset identity. Reviewers can observe changed behavior but cannot explain it. Versioned assembly contracts and preserved selection metadata make the change reconstructable.

8. Tradeoffs

Recall versus precision

Broad candidate recall reduces the chance of missing evidence at the cost of redundancy, conflict, and computation. Strict final precision improves focus but can conceal a coverage gap. Candidate generation can favor recall while eligibility enforcement and coverage-aware selection narrow the final set; one ranking stage cannot serve both goals well.

Freshness versus availability and latency

Live sources improve current-state accuracy while adding latency and another dependency failure path. Cached evidence improves availability only when its age and effective-time semantics remain explicit. The plan must distinguish evidence that is merely older from evidence too stale to use.

Compression versus fidelity

Compression creates room for broader coverage by accepting transformation risk. Lossless projection and deduplication come first. Lossy summarization is justified only when the plan defines protected facts and a recovery path to original evidence.

Personalization versus stable authority

Memory can make an interaction continuous and relevant, but durable or inferred state may be stale. Current explicit input and authoritative records take clear precedence. Personalization remains scoped evidence, never invisible truth.

Source diversity versus conflict complexity

Multiple sources improve coverage and resilience while creating more disagreement and duplicate authority. Each added source needs a clear authority domain or coverage benefit, followed by an explicit rule to resolve or expose conflicts.

Rich provenance versus token cost

Detailed labels consume context capacity, yet unlabeled evidence is difficult to trust or audit. Machine-readable lineage can remain outside the model-visible packet while the interaction receives the labels it needs. The underlying source binding is never discarded.

Graceful degradation versus explicit failure

Shedding optional evidence can preserve service under pressure until required coverage or fidelity is at risk. A predefined minimum viable package marks that limit, after which degradation ends in an explicit partial or failed assembly state.

9. Production Checklist

Before approving a context design, confirm:

• The task has explicit required, optional, and prohibited evidence categories.
• Each required fact type maps to an approved authority domain.
• Actor, purpose, tenant, role, record or field scope, and effective time are applied where relevant.
• Permission filters are eligibility constraints at retrieval time, including for caches and derived indexes.
• Candidate generation and final evidence selection are distinct decisions.
• Ranking considers authority, freshness, specificity, coverage, and redundancy—not similarity alone.
• Evidence sufficiency and no-evidence behavior are defined.
• Sources identify provenance, authenticity expectations, effective time, authority class, and conflict precedence.
• Unresolved disagreement remains visible in the context package.
• Lossless shaping precedes lossy compression.
• Protected facts and recovery links are defined for every lossy transformation.
• Hard token reservations, elastic allocations, output headroom, and shedding order are explicit.
• The minimum viable package and assembly-failure condition are defined.
• Every memory class has purpose, scope, source, confidence, precedence, correction, retention, expiration, and read-time eligibility.
• The package reports completeness, freshness, conflicts, missing evidence, and assembly version.
• Context-local checks cover permissions, provenance, freshness, required evidence, budgets, conflicts, and shaping invariants.
• The context asset has an owner, version identity, rationale, and declared dependency implications.

10. Design Review Questions

1. What facts does the task require, and which facts are optional or prohibited?
2. Which source is authoritative for each fact type and effective time?
3. Where is actor and purpose scope enforced before candidates become visible?
4. Can a cache, summary, index, or metadata feature broaden that scope?
5. How are requests derived from evidence requirements rather than surface wording alone?
6. What separates candidate generation from final selection?
7. How do ranking and stopping rules demonstrate evidence coverage?
8. What does the package report when no eligible evidence exists?
9. Which conflicts may be resolved automatically, and which must remain explicit?
10. Which qualifiers, dates, units, exceptions, negations, and identifiers must survive shaping?
11. What is protected in the token budget, and what is shed first?
12. What happens when the minimum viable evidence package cannot fit?
13. Which memory classes exist, who can write them, and what overrides them?
14. Can a reviewer trace every supplied item to its source and transformation history?
15. Which bounded assembly checks feed the Evaluation Plan in Chapter 8?
16. Which context asset changes create compatibility implications for the AI Release Manifest in Chapter 10?

11. Main Artifact: Context Assembly Plan

Use one plan for each distinct task or evidence contract. A generic plan cannot safely hide materially different authority, permission, or freshness requirements.

A. Purpose and Ownership

• System behavior supported:
• Task or interaction identifier:
• Context owner:
• Source AI System Stack Reference Model identity, version, and context-layer boundary:
• Source AI Fit and Risk Assessment Worksheet identity, version, and accepted decision IDs:
• Source Production Prompt Specification identity, version, and required context inputs:
• Downstream Harness Control-Loop Diagram identity and context-package interface:
• Context Assembly Plan identity and version:
• Change rationale:
• Compatibility implications:
• Evaluation Plan handoff: context invariants, known gaps, and scenario IDs:
• AI Release Manifest handoff: context asset identity and compatibility classification:

B. Evidence Requirements

Evidence-sufficiency rule:

Minimum viable context package:

C. Actor, Purpose, and Permission Scope

• Actor and tenant/domain inputs:
• Declared purpose and workflow role:
• Authorization decision reference:
• Enforcement stage:
• Filtered unit: source / record / field / passage / memory item
• Time-dependent scope:
• Cache and derived-index constraints:
• Denial behavior that does not reveal protected metadata:

D. Source Policy

Conflict precedence rules:

Conflicts that prohibit automatic resolution:

E. Assembly Pipeline

Query or request derivation rules:

Ranking dimensions and tie-breakers:

Evidence-coverage and stopping rule:

No-eligible-evidence representation:

F. Shaping and Token Budget

Lossless shaping methods:

Lossy compression methods and allowed inputs:

Protected-fact invariants:

Original-evidence recovery method:

Ordered shedding policy:

Minimum-package overflow behavior:

G. Memory and Lifecycle

Invalidation triggers:

Rules preventing memory from overriding current input or authoritative records:

H. Context Package Contract

• Package schema or identifier:
• Evidence-requirements identity:
• Supplied-item fields:
• Source and provenance fields:
• Authority and effective-time fields:
• Transformation lineage fields:
• Missing-evidence representation:
• Conflict representation:
• Completeness states: complete / partial / stale / conflicting / unauthorized / unavailable
• Budget allocation and shedding metadata:
• Assembly timestamp and version:

I. Context-Specific Checks

These checks show whether the context asset follows its own contract. Chapter 8 determines representative scenario coverage, quality metrics, regression policy, human review, and launch thresholds.

The Context Assembly Plan applies the context-layer boundary from the AI System Stack Reference Model, the accepted limits in the AI Fit and Risk Assessment Worksheet, and the required inputs in the Production Prompt Specification. The Harness Control-Loop Diagram consumes this authoritative context-design record. Bounded checks and known evidence gaps feed the Evaluation Plan; version and compatibility implications feed the AI Release Manifest. A downstream artifact may narrow eligibility, require stronger evidence, or block use. It must not silently broaden the actor, purpose, source, permission, freshness, or memory scope accepted by this plan.

12. Key Takeaways

• Context is a provenance-bound evidence package, not a copy of everything the system can access.
• Evidence requirements precede source selection and retrieval.
• Permissions establish eligibility; ranking cannot repair an overbroad candidate set.
• Candidate generation, evidence selection, conflict handling, and sufficiency are separate decisions.
• Token budgets require protected allocations, explicit shedding, and a minimum viable package.
• Compression and memory require lifecycle and fidelity contracts, not informal convenience rules.
• The package exposes missing, stale, conflicting, or unauthorized conditions without deciding downstream control flow.
• Versioned context assets and bounded assembly checks make context behavior reviewable while preserving the ownership of evaluation, security, deployment, and operations chapters.