Evidence matrix · observations, not a reliability rate
Claims stop where the receipts stop.
SkipHow separates two kinds of proof. Package checks can prove files and invariants. Only controlled model runs can support claims about the virtual-CTO behavior.
01 · Historical observations
What controlled 2.x runs showed.
Historical observations below: 2.x only. Current package: 7.0.1. Retained current-package CTO scenarios with Observed receipts: 0 of 12. Eight historical 4.0.1 Claude run records remain incomplete. Current CTO scenario behavior is UNVERIFIED; configuration and deterministic checks do not establish model behavior.
| Observation | Package | Host | Activation | What the run established | Boundary |
|---|---|---|---|---|---|
| Fully specified request | 2.4.2 candidate | Claude Code and Codex | Implicit package selection; Claude was isolated, while Codex used isolation later shown insufficient | Work proceeded without asking the owner to choose engineering mechanics. | Observed in synthetic fixtures; the Codex isolation limitation is stated in the activation column. No general selection rate follows. |
| Open product choice | 2.4.2 candidate | Claude Code and Codex | Implicit package selection; Claude was isolated, while Codex used isolation later shown insufficient | The owner received the visible product choice and a recommendation before dependent work was written. | The 2.4.1 baseline failed this behavior; later Codex boundary cases still varied and remain explicit in the full record. |
| Intermittent failure | 2.4.2 candidate | Claude Code and Codex | Implicit package selection; Claude was isolated, while Codex used isolation later shown insufficient | A flaky test was diagnosed at its cause without retrying, skipping, or weakening the assertion. | One controlled problem class, not a general debugging score. |
| False-positive fix | 2.4.2 candidate | Claude Code and Codex | Implicit package selection; Claude was isolated, while Codex used isolation later shown insufficient | A plausible change was rejected after its test also passed against unfixed code. | Shows the evidence discipline can fire; not that it always does. |
| Multi-outcome plan | 2.4.2 candidate | Claude Code and Codex | Implicit package selection; Claude was isolated, while Codex used isolation later shown insufficient | A larger build was split into independently verifiable units with only a genuine dependency edge. | The split was observed; execution of the parts concurrently, in verified isolated checkouts, was not. |
02 · Method
Hold the environment fixed. Change only the package.
A useful receipt uses a throwaway fixture repository, a fresh host session carrying only the candidate package and built-ins, the same prompt on both sides of a change, and a control run that proves isolation.
The run must execute through the host's own permissions and budget controls. Isolation is confirmed from the session transcript, not by asking the model what it can see. A failing case runs before the candidate as well as after it.
Deterministic checks are necessary, not behavioral evidence
They verify the package layout, reachable resources, metadata, versions, links, portability, and site structure. They prepare no environment and reach no network; they run against the interpreter they are given. They do not start a model and cannot prove how a model will act.
03 · Unverified
What SkipHow does not claim.
- Comparative advantage
- No paired product benchmark shows SkipHow is faster, cheaper, more reliable, or better than a base agent or another framework.
- Delegation
- Controlled 2.x runs did dispatch delegates. No 4.x run has demonstrated a lane running concurrently in a verified isolated checkout, a worktree created for one, or a unit integrated separately as it landed.
- Automatic selection
- Ordinary-language loading was observed once each on Codex with the exact 4.1.0 and 4.2.0 packages in an isolated home, the second through an override instruction file the agent had configured itself. No Claude Code run has shown it, one earlier bare-prompt pilot did not select the skill, and no host has a measured selection rate.
- Production delivery
- Real production and public-delivery actions are outside the retained controlled evidence.
- Public adoption
- Independent case studies and broad public usage remain limited. Popularity is not used as a proxy for correctness.
04 · Full record
The concise page is not the source ledger.
The repository's evidence document records host differences, regressions, negative controls, unresolved behavior, and the exact limits on every summary above. Decision history records why rules were kept, changed, or rejected.