Preserve the failure before proposing the fix.

# CI and Test Failure Triage Scenario Pack

Toolkit schema: `PAISEH-TK-1.0`
Scenario pack: `PAISEH-SC-CITFT-1.0`

Use this pack when AI helps interpret build or test failures, rank diagnoses, recommend a bounded next step, or initiate a separately authorized rerun. It keeps observed evidence, diagnosis, remediation, execution, and repository decisions distinct.

This pack does not replace the AI-Assisted Code Change pack for source changes, the Tool-Using Workflow or Agent pack for effecting actions, or source-chapter methods for harness, evaluation, tool authority, release, and operations.

## When to Use This Pack

Use it for one identified candidate and CI or test execution whose logs and artifacts can be preserved. Do not use it to let a model waive required checks, classify a failure as flaky without evidence, make an unreviewed code change, or restart work under inferred authority.

## Roles and Authority

| Role | Responsibility | Authority retained |
| --- | --- | --- |
| Requester | Identifies the failed candidate and desired triage outcome | May request diagnosis; cannot waive repository policy |
| Triage owner | Owns scope, evidence, hypotheses, and handoff | Accepts the triage record, not the code or release |
| CI owner | Owns pipeline semantics, runners, and rerun controls | Authorizes infrastructure action and failure classification |
| Test owner | Owns test intent, validity, fixtures, and quarantine policy | Decides whether a test or result is valid |
| Remediation owner | Owns an accepted correction path | Must use the code-change pack for source edits |
| Verifier | Inspects evidence and attempted remediation | Produces evidence-indexed findings |
| Decision authority | Controls merge, release, waiver, quarantine, or retry decisions | Records the final disposition |

## Minimum Viable Pack

For advisory triage, complete:

1. Common Artifact Header.
2. Failure Triage Brief.
3. CI Evidence Snapshot Manifest.
4. Triage Verification Checklist.
5. Triage Completion Receipt.

Use the Harness Control-Loop Diagram when retries, fallbacks, or multiple states matter. Add the Tool Permission Matrix and Tool-Using Workflow pack before any rerun, cancellation, job restart, quarantine, or artifact mutation. Use the AI-Assisted Code Change pack for a proposed source change. Use the Evaluation Plan when a check's validity or regression claim must be established.

## Entry Criteria

- Candidate revision, pipeline definition, run, job, test, and environment identities are known.
- Logs, reports, exit status, timing, dependencies, and relevant artifacts are retrievable.
- Repository and CI policies identify required checks and decision authorities.
- The task states whether output is diagnosis only, a remediation proposal, or a request for separately authorized action.
- Sensitive logs and artifacts have an access and redaction path.

## Working Sequence

1. Bound the failed candidate, run, and permitted outcome.
2. Freeze the evidence snapshot before reruns or mutable logs change.
3. Separate observations from hypotheses and rank disconfirming checks.
4. Run the checklist against evidence validity, reproduction, diagnosis, and action boundaries.
5. Hand source changes to the AI-Assisted Code Change pack and tool effects to the Tool-Using Workflow pack.
6. Preserve the final result, uncertainty, and decision in the completion receipt.

## Exit and Acceptance Criteria

- Every diagnosis points to evidence and names competing hypotheses.
- Candidate, pipeline, environment, dependency, and test identities are reproducible.
- Missing, truncated, sampled, flaky, skipped, retried, or quarantined evidence remains visible.
- No rerun result overwrites the first failure or silently converts an unknown cause into success.
- Any code change, waiver, quarantine, restart, or release decision has separate authority and receipt.
- The completion receipt records what was learned, what remains unknown, and the exact handoff.

## Failure Paths

- Use `runbooks/incomplete-or-misleading-ci-evidence.md` when logs, reports, test semantics, or environment identity cannot support the diagnosis.
- Use `runbooks/flaky-or-nonreproducible-failure.md` when outcomes vary and recurrence has not been explained.

## Material Change and Reopening Triggers

Reopen triage when the candidate revision, pipeline definition, runner image, dependency lock, environment, test, fixture, seed, shard, clock, retry policy, quarantine state, evidence snapshot, diagnosis, remediation, or decision use changes. A passing rerun does not close the original failure unless the recorded authority accepts the causal explanation and evidence.

# CI and Test Failure Triage Brief

Toolkit schema: `PAISEH-TK-1.0`
Scenario pack: `PAISEH-SC-CITFT-1.0`

Complete `../../common-artifact-header.md` first. Use one brief for one bounded candidate and related failure set.

## 1. Failure Identity

| Field | Record |
| --- | --- |
| Repository, branch, and candidate revision | |
| Pipeline definition and configuration version | |
| Run, job, step, shard, and attempt IDs | |
| Runner image, environment, region, and architecture | |
| Dependency lock, cache, fixture, and seed identities | |
| Requester, triage owner, CI owner, and test owner | |

## 2. Scope and Permitted Outcome

- Failure summary:
- Required checks and repository policy:
- In-scope jobs, tests, files, and dependencies:
- Explicit exclusions:
- Requested output: `diagnosis`, `remediation proposal`, or `authorized action request`
- Actions that require a separate Tool Permission Matrix and approval:
- Source changes that require the AI-Assisted Code Change pack:

## 3. Observations

| Observation ID | Evidence ID | Direct observation | First seen / recurrence | Known limitation |
| --- | --- | --- | --- | --- |
| OB- | EV- | | | |

Do not write a diagnosis in this table. Preserve exit codes, messages, timing, resource state, failing assertions, and surrounding events as observations.

## 4. Hypotheses and Disconfirmation

| Hypothesis ID | Proposed cause | Supporting observation | Contradicting observation | Lowest-cost discriminating check | Check authority | State |
| --- | --- | --- | --- | --- | --- | --- |
| HY- | | | | | | Open / supported / rejected / indeterminate |

## 5. Consequence and Priority

- Blocks merge, release, deployment, or developer feedback:
- Potential product, security, data, or infrastructure consequence:
- Affected candidates, branches, teams, or users:
- Cost of delay versus cost of an incorrect fix:
- Required escalation time:

## 6. Entry and Stop Decision

- Evidence sufficient to begin: `yes`, `no`, or `indeterminate`
- Missing inputs and owner:
- Safe immediate checks:
- Stop conditions:
- Triage owner decision and timestamp:
- Expiry and reopening triggers:

# CI Evidence Snapshot Manifest

Toolkit schema: `PAISEH-TK-1.0`
Scenario pack: `PAISEH-SC-CITFT-1.0`

Complete `../../common-artifact-header.md` first. Freeze the evidence used for triage before mutable logs, caches, reruns, or retention windows change it.

## 1. Snapshot Identity

- Snapshot ID and immutable location:
- Candidate, pipeline, run, job, and attempt identities:
- Capture actor, method, and timestamp:
- Source retention and expected expiry:
- Access, redaction, and sharing constraints:

## 2. Evidence Inventory

| Evidence ID | Artifact or event | Source identity | Time range | Integrity/provenance | Completeness | Access | Limitation |
| --- | --- | --- | --- | --- | --- | --- | --- |
| EV- | Log / report / trace / dump / diff / configuration / receipt | | | | Complete / partial / truncated / sampled / unavailable | | |

## 3. Execution Context

| Dimension | Effective value and evidence |
| --- | --- |
| Source revision and uncommitted state | |
| Pipeline and test configuration | |
| Runner image, platform, capacity, and clock | |
| Dependency graph and resolved versions | |
| Cache keys, hits, misses, and restored artifacts | |
| Fixtures, datasets, seeds, shards, and ordering | |
| Network and external dependency state | |
| Secrets and protected variables, recorded without values | |

## 4. Result Semantics

- Authoritative completion and exit-status source:
- Passed, failed, skipped, retried, cancelled, timed out, or unknown steps:
- Aggregation rule for job and pipeline status:
- Expected reports that are missing:
- Logs or artifacts produced after the terminal status:
- Difference between transport, runner, test, and assertion failure:

## 5. Reproduction Record

| Attempt | Candidate and environment | Controlled differences | Result | Evidence ID | Interpretation limit |
| --- | --- | --- | --- | --- | --- |
| | | | | | |

## 6. Snapshot Closure

- Evidence gaps:
- Suspect or invalid evidence:
- Snapshot accepted by triage owner:
- Supersedes / superseded by:
- Expiry and reopening triggers:

# CI Triage Verification Checklist

Toolkit schema: `PAISEH-TK-1.0`
Scenario pack: `PAISEH-SC-CITFT-1.0`

Complete `../../common-artifact-header.md` first. This checklist produces evidence-indexed findings. It does not waive a required check, classify a test as flaky, approve a code change, authorize a rerun, or grant release approval.

## 1. Review Basis

| Field | Record |
| --- | --- |
| Failure Triage Brief version | |
| CI Evidence Snapshot Manifest version | |
| Candidate, pipeline, run, and environment | |
| Diagnosis or remediation proposal | |
| Verifier, CI owner, test owner, and decision authority | |

## 2. Verification Checks

| Check | Evidence ID | Result | Finding ID |
| --- | --- | --- | --- |
| Candidate, pipeline, environment, dependencies, and tests are exactly identified | | Pass / fail / indeterminate / not applicable | |
| Logs and reports are complete enough for the claimed diagnosis | | | |
| Observations remain separate from hypotheses | | | |
| Competing hypotheses and disconfirming evidence were considered | | | |
| Reproduction controlled material differences | | | |
| Passing reruns do not erase or mislabel the original failure | | | |
| Flaky, skipped, retried, quarantined, sampled, or timed-out states are explicit | | | |
| Proposed remediation is bounded to the supported cause | | | |
| Source edits have a separate code-change record and review | | | |
| Reruns, cancellations, restarts, or mutations have separate action authority and receipts | | | |
| Required checks and release policy remain externally enforced | | | |
| Residual uncertainty and recurrence signals are owned | | | |

## 3. Findings Register

| Finding ID | Evidence ID(s) | Observation and consequence | Severity | Required correction or decision | Owner / due date | State |
| --- | --- | --- | --- | --- | --- | --- |
| F- | | | | | | |

Allowed states are `open`, `corrected`, `verified`, `accepted risk`, `deferred`, and `not reproducible`. Only the recorded authority may waive, quarantine, accept risk, or close a finding.

## 4. Outcome

- Supported diagnosis:
- Rejected or untested hypotheses:
- Evidence gaps:
- Reviewer recommendation: `correct`, `rerun under authority`, `return`, `escalate`, `defer`, or `no recommendation`
- Required handoff:
- Decision authority, disposition, conditions, and expiry:

## 5. Reopening Triggers

Reopen when any candidate, pipeline, runner, dependency, cache, fixture, test, seed, shard, retry, evidence, diagnosis, remediation, waiver, quarantine, or required-check state changes.

# CI Triage Completion Receipt

Toolkit schema: `PAISEH-TK-1.0`
Scenario pack: `PAISEH-SC-CITFT-1.0`

Complete `../../common-artifact-header.md` first. Preserve this receipt after the triage disposition. It records what the evidence supports and the separately authorized actions that actually occurred.

## 1. Triage Identity

| Field | Actual result |
| --- | --- |
| Brief and evidence snapshot versions | |
| Repository, candidate, pipeline, run, and environment | |
| Triage owner, CI owner, test owner, verifier, and decision authority | |
| Start, decision, completion, and expiry times | |

## 2. Diagnosis and Evidence

| Hypothesis ID | Final state | Evidence ID(s) | Causal claim supported | Limitation |
| --- | --- | --- | --- | --- |
| HY- | Supported / rejected / indeterminate | EV- | | |

- Primary diagnosis:
- Evidence gaps:
- Recurrence likelihood and signal:
- Why a passing rerun does or does not change the diagnosis:

## 3. Actual Actions and Handoffs

| Action or handoff | Contract / record / approval | Effect or result receipt | Deviation | Owner |
| --- | --- | --- | --- | --- |
| Source change | AI-Assisted Code Change pack | | | |
| Rerun, restart, cancellation, or mutation | Tool-Using Workflow pack | | | |
| Waiver or quarantine | Repository or CI policy decision | | | |
| Release impact | AI Release and Change Review pack | | | |

## 4. Findings and Disposition

| Finding ID | Final state | Correction, waiver, quarantine, or accepted risk | Authority and evidence | Reopening trigger |
| --- | --- | --- | --- | --- |
| F- | | | | |

- Final disposition: `diagnosed`, `corrected`, `rerun`, `quarantined`, `waived`, `deferred`, `rejected`, or `superseded`
- Required-check status:
- Merge or release implication:
- Residual uncertainty and owner:

## 5. Closure

- Original failure evidence remains retrievable:
- Follow-up work and due dates:
- Affected teams notified:
- Authoritative receipt location:
- Closure authority and timestamp:
- Supersedes / superseded by:

# Runbook: Incomplete or Misleading CI Evidence

Toolkit schema: `PAISEH-TK-1.0`
Scenario pack: `PAISEH-SC-CITFT-1.0`

Complete `../../../common-artifact-header.md` first. Use this runbook when logs are truncated, reports are absent, status aggregation is wrong, environment identity is missing, or a test cannot detect the failure it claims to cover.

## Entry Conditions and Authority

- Candidate, pipeline, run, job, test, and evidence identities:
- Suspect claim, missing evidence, and affected decision:
- Triage owner, CI owner, test owner, verifier, and decision authority:
- Authority to preserve artifacts, mark results indeterminate, rerun, repair instrumentation, or block a decision:

## Immediate Action

1. Mark affected diagnoses and check results `indeterminate`.
2. Preserve available logs, reports, status events, environment metadata, and retention settings.
3. Stop merge, waiver, quarantine, or remediation decisions that rely on the invalid claim.
4. Record the missing evidence and assign an owner before any rerun.

## Diagnosis and Response

| Step | Action | Expected evidence | Stop / escalate when |
| --- | --- | --- | --- |
| Reconstruct | Join runner, job, step, test, report, and terminal events | Complete execution timeline | Terminal state is contradictory |
| Validate | Confirm assertions can detect the claimed failure and reports cover executed tests | Test-validity evidence | Test semantics are owner-disputed |
| Compare | Check expected versus retained artifacts and status aggregation | Gap inventory | Retention destroyed required evidence |
| Repair | Correct capture, reporting, aggregation, or test behavior under owner authority | Controlled candidate | Repair changes source without review |
| Re-execute | Rerun only under recorded authority and controlled differences | New immutable snapshot | Rerun would overwrite original evidence |

## Verification and Stopping Conditions

Verify artifact completeness, environment identity, test discovery, executed versus reported counts, assertion behavior, exit-status propagation, retention, and reproducibility.

Stop and escalate when the original result cannot be reconstructed, required-check status is unreliable, a protected artifact is exposed, or a decision already used the invalid evidence.

## Reversal

- Restore the last verified reporting or pipeline configuration.
- Withdraw unsupported diagnoses, waivers, quarantines, or remediations.
- Reopen dependent merge or release decisions.
- Preserve the original evidence gap; do not relabel it as a pass.

## Evidence and Closure

Record the gap, affected claims and decisions, corrected capture or test, rerun authority, new evidence, withdrawn dispositions, and remaining limitations.

Close when decision evidence is valid or the decision remains explicitly blocked. Reopen on another missing report, contradictory status, or invalid assertion.

# Runbook: Flaky or Nonreproducible Failure

Toolkit schema: `PAISEH-TK-1.0`
Scenario pack: `PAISEH-SC-CITFT-1.0`

Complete `../../../common-artifact-header.md` first. Use this runbook when equivalent-looking executions disagree and the cause has not been established.

## Entry Conditions and Authority

- Candidate, pipeline, test, attempts, and evidence snapshot identities:
- Observed pass/fail sequence and suspected varying dimensions:
- Triage owner, CI owner, test owner, repository authority, and release authority:
- Authority to rerun, isolate, quarantine, block, repair, or accept residual risk:

A passing rerun does not prove the first failure was harmless, and the assistant cannot label a test flaky by frequency alone.

## Immediate Action

1. Preserve every attempt and mark the causal claim `indeterminate`.
2. Stop blind reruns once the recorded investigation budget is exhausted.
3. Keep the required-check policy in force unless the named authority records a waiver or quarantine.
4. Bound affected candidates, branches, environments, and release decisions.

## Diagnosis and Response

| Step | Action | Expected evidence | Stop / escalate when |
| --- | --- | --- | --- |
| Align | Compare revision, pipeline, runner, dependencies, caches, fixtures, seeds, shards, ordering, time, and external services | Difference matrix | Execution identity is incomplete |
| Reproduce | Change one suspected dimension at a time within budget | Controlled attempt set | Attempts create unsafe load or state |
| Classify | Distinguish test defect, product race, infrastructure fault, environmental dependency, and unknown cause | Evidence-supported class | Multiple classes remain plausible |
| Contain | Block, isolate, or quarantine only under policy authority | Enforced disposition | Quarantine hides product risk |
| Correct | Repair the owned cause through the proper change path | Verified candidate | Correction broadens task scope |

## Verification and Stopping Conditions

Verify recurrence rate with denominators, controlled dimensions, test sensitivity, product behavior, infrastructure health, quarantine scope and expiry, and downstream required-check state.

Stop and escalate when the failure could represent a product race, security boundary, data corruption, or release risk; when the varying dimension cannot be controlled; or when investigation exceeds the accepted budget.

## Reversal

- Remove temporary retries or quarantine when their authority expires.
- Restore the last verified pipeline or test configuration.
- Reopen merges or releases that relied on an unsupported flaky classification.
- Undo source changes only through the accepted code-change and release paths.

## Evidence and Closure

Record all attempts, controlled differences, classification evidence, containment, correction, policy decisions, expiry, and recurrence signals.

Close when the cause is verified and corrected, or an authorized time-bounded disposition owns the remaining uncertainty. Reopen on recurrence outside the accepted envelope.