221 lines
10 KiB
Markdown
221 lines
10 KiB
Markdown
---
|
||
name: reflect
|
||
description: >
|
||
Review findings from your own self-diagnosis and apply approved fixes, one at a time.
|
||
Triggers ONLY on the explicit `/reflect` command. Never infer this skill from
|
||
conversational mentions of reflection, findings, self-improvement, or "what did you
|
||
learn" — it can edit your own configuration, so it must never start by accident.
|
||
---
|
||
|
||
# Reflect
|
||
|
||
Interactive review of findings produced by the unattended analysis run
|
||
(`scripts/reflect_auto.py`, daily via cron). You show the user one finding at a time,
|
||
wait for a decision, and apply only what they approve.
|
||
|
||
Answer in the language the user writes in.
|
||
|
||
## STOP gates — read before anything else
|
||
|
||
1. **This skill never runs the analysis.** Analysis is a cron script. If the user asks
|
||
for a fresh analysis, tell them to run
|
||
`~/.local/bin/uv run --script skills/reflect/scripts/reflect_auto.py` — do not
|
||
distil sessions or diagnose patterns yourself in this turn.
|
||
2. **Never edit anything without approval of that specific finding.** Not "the user
|
||
seemed positive earlier", not "finding 3 is obviously right". One `ok` approves
|
||
exactly one finding.
|
||
3. **One finding per step.** Never present two findings at once, never apply a batch,
|
||
even if the user says "apply everything" — in that case apply the first, show the
|
||
result, and continue to the next.
|
||
4. **Never edit a target file yourself — always through `scripts/reflect_apply.py`.**
|
||
That script refuses a patch whose original text is missing or ambiguous, and commits
|
||
only the file it touched. If it refuses, relay the reason; never hand-edit around it.
|
||
**`reflect/findings.jsonl` is never edited by hand either — not with a one-off script,
|
||
not with a heredoc, never.** It is the audit trail; every write to it belongs to
|
||
`reflect_apply.py`. Drafting a patch during the review goes through `--set-patch`.
|
||
|
||
## Data
|
||
|
||
| Path | What |
|
||
|---|---|
|
||
| `reflect/findings.jsonl` | one JSON object per line — the findings store |
|
||
| `reflect/state.json` | cursor and per-run statistics (owned by the script) |
|
||
| `results/<date>_reflect.md` | full report of a run — read it when the user asks what the last run found |
|
||
| `log/reflect.log` | append-only audit of every decision — applied, rejected, skipped |
|
||
|
||
Statuses: `watch` (seen once, not yet worth the user's attention) · `open` (waiting for
|
||
review) · `applied` · `rejected`. Beyond the obvious fields a finding carries:
|
||
|
||
- `last_seen` — the newest date in the **evidence**, not when the record was rewritten;
|
||
older records lack it, fall back to `created`.
|
||
- `history` — superseded records of this pattern as `"<created>:<id>"`; the date in
|
||
`history[0]` is when it was first filed.
|
||
- `regression_of` — the `id` of the applied fix this pattern came back after.
|
||
- `patch` (`file` / `old_text` / `new_text`) and `patch_drafted_at` — see step 2.
|
||
- `applied` / `rejected` once decided, `skipped` while the finding is still open — see
|
||
[Decision history](#decision-history) for what is in them.
|
||
|
||
Do not overstate the numbers: a run analyses a **window** of recent sessions, not the whole
|
||
history (`state.json` holds the window, the report repeats it), and `occurrences` /
|
||
`sessions_affected` are **cumulative across runs** — what the model counted in each slice,
|
||
not a measurement over the corpus.
|
||
|
||
## Procedure
|
||
|
||
### 1. Load
|
||
|
||
Read `reflect/findings.jsonl`. Take the records with `status: open` and sort them:
|
||
**regressions first** (`regression_of` is set), then by severity (`high`, `medium`, `low`),
|
||
then `last_seen`, then `occurrences`, all descending.
|
||
|
||
A finding whose `last_seen` is older than the window of the latest run (`state.json`,
|
||
`window_from`) is **stale**: no run looks that far back any more, so nothing will refresh
|
||
it. Say so when you present it — it is a candidate for rejection rather than a patch.
|
||
|
||
Assign **display IDs 1..N** over that sorted list, computed fresh each time. The user
|
||
refers to findings by these short numbers; the internal `id` stays the key in the store
|
||
and in the audit log, and is never what you ask the user to type.
|
||
|
||
If there are none: say so, mention how many `watch` findings are being tracked, and stop.
|
||
|
||
### 2. Prepare the patch
|
||
|
||
The user decides on a diff, not on a sentence describing one, so have the patch ready
|
||
*before* you present the finding.
|
||
|
||
- **It already has a `patch`** — the analysis wrote one, or `patch_drafted_at` says an
|
||
earlier review drafted it and ended without deciding; say so when you present it. Either
|
||
way, verify it still applies:
|
||
|
||
```bash
|
||
uv run --script skills/reflect/scripts/reflect_apply.py --id <internal id> --check
|
||
```
|
||
|
||
It changes nothing and prints the diff. Exit code 2 means the patch no longer applies —
|
||
relay the printed reason and offer only *skip* / *reject*.
|
||
|
||
- **It has none** — most findings. Draft one now, before anything is shown. Read the target
|
||
file for one thing only: copying `old_text` out of it character-for-character (never
|
||
retype it, watch the quotation marks); it must occur in the file exactly once. The diff
|
||
always comes from the script, never from your own reading. File the draft with
|
||
`--set-patch`:
|
||
|
||
```bash
|
||
# tmp/patch.json: {"file": "SOUL.md", "old_text": "<copied character-for-character>", "new_text": "<the fix>"}
|
||
uv run --script skills/reflect/scripts/reflect_apply.py --id <internal id> --set-patch tmp/patch.json
|
||
```
|
||
|
||
A refusal (exit 2) changes nothing — relay the reason and draft again. On success it
|
||
prints the diff, touches no file and decides nothing; applying still needs step 4.
|
||
|
||
- **No file edit can fix it** — "think before answering", "be less eager to agree". Say that
|
||
plainly, offer only *skip* / *reject*, and give no odds. Never invent a patch just to have
|
||
something to show.
|
||
|
||
### 3. Present one finding
|
||
|
||
Show exactly one, in this shape — labels in the user's language, the odds line as
|
||
[Estimating the odds](#estimating-the-odds) prescribes:
|
||
|
||
```text
|
||
[1/4] retry-without-diagnosis · medium · 7× in 4 sessions
|
||
first seen 2026-09-01, last seen 2026-09-02 · deferred 2× already
|
||
<diagnosis>
|
||
|
||
Evidence:
|
||
- websocket_e5a6aa… (2026-07-11) — web_fetch → ERROR 403 ×4
|
||
- …
|
||
|
||
Proposal: <proposal>
|
||
|
||
<the diff, exactly as the script printed it>
|
||
|
||
Odds it works: ~40 % — rewords guidance in a file that is in context, but this pattern
|
||
already came back once after a fix.
|
||
```
|
||
|
||
The second line comes from `history[0]` (first filed), `last_seen` (when the pattern last
|
||
actually occurred) and `skipped.count`; drop the skip part when there is none, and do say
|
||
when a finding has been deferred several times — rejecting it is cleaner than a list that
|
||
keeps re-presenting it.
|
||
|
||
Mark a finding with `regression_of` clearly as a **regression**, and a stale one (step 1)
|
||
as stale.
|
||
|
||
Then ask for a decision and wait.
|
||
|
||
### 4. Decide and apply
|
||
|
||
Every row runs `uv run --script skills/reflect/scripts/reflect_apply.py --id <internal id>
|
||
…` — `--id` is the internal `id`, never the display number you showed.
|
||
|
||
| Input | Meaning | Flags |
|
||
|---|---|---|
|
||
| `ok`, `apply` | apply this finding's patch | *(none)* |
|
||
| `edit: <text>` | the user rewrites `new_text` | write their version to a temp file; `--check --new-text-file <path>` shows the new diff and changes nothing, then on `ok` the same file **without** `--check` |
|
||
| `skip` | record the deferral, leave it `open`, move to the next | `--skip` |
|
||
| `reject` | **ask why first**, then close it for good — it never opens again | `--reject --reason "<user's words>"` |
|
||
| `stop` | end the review | *(nothing to run)* |
|
||
|
||
Keep that temp file until the finding is decided — the user's wording never enters `patch`.
|
||
|
||
Accept equivalents in the language the user writes in — the words above are meanings, not a
|
||
required vocabulary.
|
||
|
||
Rejection needs a reason and the script will not take it without one: ask in one short
|
||
question, pass the user's own words through, invent nothing, and do not talk them out of it.
|
||
|
||
Exit code 2 means refused: relay the printed reason and move on. On success the script
|
||
prints the commit SHA and the revert command — pass those on, then take the next finding.
|
||
|
||
### 5. Close
|
||
|
||
When the user stops or the list is exhausted, summarise: how many applied, skipped,
|
||
rejected, and how many remain `open`. If anything was applied, remind the user that the
|
||
change lives only on the server and should be pulled back into the tracking repo.
|
||
|
||
## Estimating the odds
|
||
|
||
Read the odds off the record, not off a feeling — otherwise two reviews of the same finding
|
||
say 40 % and 75 %. Start from what the patch changes:
|
||
|
||
| The patch … | Band |
|
||
|---|---|
|
||
| changes mechanism — a script, a gate the agent cannot talk itself past | ~80 % |
|
||
| adds a hard prohibition to a file that is in context when the pattern occurs (`SOUL.md`, `AGENTS.md`, the evidence's own `SKILL.md`) | ~60 % |
|
||
| rewords or softens existing guidance in such a file | ~40 % |
|
||
| edits a file that is not in context at that moment, or leaves the call to the agent's judgement | ~20 % |
|
||
|
||
Move one band down for each of these — ~20 % is the floor — and name the reason in the line:
|
||
|
||
- `regression_of` — this kind of instruction has already failed on this very pattern.
|
||
- the evidence spans unrelated contexts — the trigger is not in the file being patched.
|
||
|
||
Two rules:
|
||
|
||
- it is an estimate from the record, **not a measurement** — say so the first time you
|
||
give one;
|
||
- keep to the bands; `63 %` claims a precision that is not there.
|
||
|
||
A stale finding (step 1) gets no percentage — say the pattern may already be gone.
|
||
|
||
## Decision history
|
||
|
||
Asked what was already decided, or what a past decision changed: read `log/reflect.log`
|
||
— it is the source of truth for the list, one line per decision, oldest first. Then fill
|
||
each line in from `findings.jsonl`:
|
||
|
||
- applied → the `patch` (the model's proposal), `applied.new_text` when the user rewrote
|
||
it, and `revert: git revert <applied.sha>`. Use `git show <sha>` for the real diff.
|
||
- rejected → `rejected.reason`. Records decided before reasons existed carry a flat
|
||
`rejected_at` and no reason; say the reason is missing rather than guessing one.
|
||
- skipped → `skipped.count`, still open.
|
||
|
||
Do not reconstruct this list from memory or from the findings store alone.
|
||
|
||
## When a finding is wrong
|
||
|
||
Findings come from an LLM reading its own logs and can be plain wrong; rejection is a normal
|
||
outcome, not a failure. If several in a row are noise, say so plainly instead of working
|
||
through the list — the analysis prompt needs tuning.
|