206 lines
9.7 KiB
Markdown
206 lines
9.7 KiB
Markdown
---
|
||
name: reflect
|
||
description: >
|
||
Review findings from your own self-diagnosis and apply approved fixes, one at a time.
|
||
Triggers ONLY on the explicit `/reflect` command. Never infer this skill from
|
||
conversational mentions of reflection, findings, self-improvement, or "what did you
|
||
learn" — it can edit your own configuration, so it must never start by accident.
|
||
---
|
||
|
||
# Reflect
|
||
|
||
Interactive review of findings produced by the unattended analysis run
|
||
(`scripts/reflect_auto.py`, daily via cron). You show the user one finding at a time,
|
||
wait for a decision, and apply only what they approve.
|
||
|
||
Answer in the language the user writes in.
|
||
|
||
## STOP gates — read before anything else
|
||
|
||
1. **This skill never runs the analysis.** Analysis is a cron script. If the user asks
|
||
for a fresh analysis, tell them to run
|
||
`~/.local/bin/uv run --script skills/reflect/scripts/reflect_auto.py` — do not
|
||
distil sessions or diagnose patterns yourself in this turn.
|
||
2. **Never edit anything without approval of that specific finding.** Not "the user
|
||
seemed positive earlier", not "finding 3 is obviously right". One `ok` approves
|
||
exactly one finding.
|
||
3. **One finding per step.** Never present two findings at once, never apply a batch,
|
||
even if the user says "apply everything" — in that case apply the first, show the
|
||
result, and continue to the next.
|
||
4. **Never edit a target file yourself — always through `scripts/reflect_apply.py`.**
|
||
That script refuses a patch whose original text is missing or ambiguous, and commits
|
||
only the file it touched. If it refuses, relay the reason; never hand-edit around it.
|
||
**`reflect/findings.jsonl` is never edited by hand either — not with a one-off script,
|
||
not with a heredoc, never.** It is the audit trail; every write to it belongs to
|
||
`reflect_apply.py`. Drafting a patch during the review goes through `--set-patch`.
|
||
|
||
## Data
|
||
|
||
| Path | What |
|
||
|---|---|
|
||
| `reflect/findings.jsonl` | one JSON object per line — the findings store |
|
||
| `reflect/state.json` | cursor and per-run statistics (owned by the script) |
|
||
| `results/<date>_reflect.md` | full report of a run |
|
||
| `log/reflect.log` | append-only audit of every decision — applied, rejected, skipped |
|
||
|
||
A finding has: `id`, `status`, `created`, `pattern`, `severity`, `diagnosis`,
|
||
`last_seen`, `evidence`, `occurrences`, `sessions_affected`, `proposal`, optional `patch`
|
||
(`file` / `old_text` / `new_text`), optional `patch_drafted_at` (the patch was drafted
|
||
during a review, not proposed by the analysis), optional `regression_of`, optional `history`,
|
||
and once decided one of `applied` (`at` / `sha` / `file`, plus `new_text` and
|
||
`edited_by_user` when the user rewrote it), `rejected` (`at` / `reason`), `skipped`
|
||
(`count` / `last`).
|
||
|
||
Statuses: `watch` (seen once, not worth the user's attention yet) · `open` (waiting for
|
||
review) · `applied` · `rejected`.
|
||
|
||
Two things about the numbers, both of which you must not overstate to the user:
|
||
|
||
- A run analyses a **window** of recent sessions, not the whole history. `state.json`
|
||
holds the window it used; the dated report repeats it in its header.
|
||
- `occurrences` and `sessions_affected` are **cumulative across runs** — the sum of what
|
||
the model counted in each slice, not a figure anyone measured over the whole corpus.
|
||
`last_seen` is the newest date in the evidence, `history[0]` when the pattern was first
|
||
filed, `created` only when the record was last rewritten. Older records may lack
|
||
`last_seen`; fall back to `created` for those.
|
||
|
||
## Procedure
|
||
|
||
### 1. Load
|
||
|
||
Read `reflect/findings.jsonl`. Take the records with `status: open`, sorted by severity
|
||
(`high`, `medium`, `low`), then by `last_seen` and then by `occurrences`, all descending.
|
||
`occurrences` alone would let a stale pattern with a large cumulative count outrank a
|
||
fresh one.
|
||
|
||
A finding whose `last_seen` is older than the window of the latest run (`state.json`,
|
||
`window_from`) is **stale**: no run looks that far back any more, so nothing will refresh
|
||
it. Say so when you present it — it is a candidate for rejection rather than a patch.
|
||
|
||
Assign **display IDs 1..N** over that sorted list, computed fresh each time. The user
|
||
refers to findings by these short numbers; the internal `id` stays the key in the store
|
||
and in the audit log, and is never what you ask the user to type.
|
||
|
||
If there are none: say so, mention how many `watch` findings are being tracked, and stop.
|
||
|
||
### 2. Present one finding
|
||
|
||
Show exactly one, in this shape:
|
||
|
||
```text
|
||
[1/4] retry-without-diagnosis · medium · 7× in 4 sessions
|
||
first seen 2026-09-01, last seen 2026-09-02 · deferred 2× already
|
||
<diagnosis>
|
||
|
||
Evidence:
|
||
- websocket_e5a6aa… (2026-07-11) — web_fetch → ERROR 403 ×4
|
||
- …
|
||
|
||
Proposal: <proposal>
|
||
```
|
||
|
||
Write the labels in the user's language, not necessarily as shown here.
|
||
|
||
The second line carries what the count alone hides: `history[0]` for when the pattern
|
||
was first filed, `last_seen` for when it last actually occurred, and `skipped.count` when
|
||
the user has already deferred it. Drop the skip part when there is none; a finding deferred
|
||
several times is worth saying so about, because rejecting it is cleaner than a list that
|
||
keeps re-presenting it.
|
||
|
||
Mark a finding with `regression_of` clearly as a **regression** — this pattern was fixed
|
||
before and came back after the fix. Mark a stale finding (see step 1) as such too, and
|
||
say what it means: the evidence predates the current window, so the pattern may well be
|
||
gone already.
|
||
|
||
If the finding has a `patch`, get the diff from the script — never by reading the file and
|
||
judging for yourself, that is the duplication gate 4 exists to prevent:
|
||
|
||
```bash
|
||
uv run --script skills/reflect/scripts/reflect_apply.py --id <internal id> --check
|
||
```
|
||
|
||
It changes nothing and prints the diff; show the user what it printed. Exit code 2 means the
|
||
patch no longer applies — relay the printed reason and offer only *skip* / *reject*.
|
||
|
||
Then ask for a decision and wait.
|
||
|
||
### 3. Accept a decision
|
||
|
||
| Input | Meaning |
|
||
|---|---|
|
||
| `ok`, `apply` | apply this finding's patch |
|
||
| `edit: <text>` | the user rewrites `new_text`; show the new diff and ask again |
|
||
| `skip` | record the deferral, leave it `open`, move to the next |
|
||
| `reject` | **ask why first**, then close it for good — it never opens again |
|
||
| `stop` | end the review |
|
||
|
||
Accept the equivalents in whatever language the user writes in — the words above are the
|
||
meanings, not a required vocabulary.
|
||
|
||
Rejection needs a reason and the script will not take it without one. Ask for it in one
|
||
short question and pass the user's own words through — half of the first findings were
|
||
rejected, and that number only says something about the analysis if the reasons are on
|
||
the record. Do not invent a reason, and do not talk the user out of rejecting.
|
||
|
||
A finding without a `patch` cannot be applied. Offer to draft one, and file it with
|
||
`--set-patch` — **never by writing to `reflect/findings.jsonl` yourself** (gate 4):
|
||
|
||
```bash
|
||
# {"file": "SOUL.md", "old_text": "<copied character-for-character>", "new_text": "<the fix>"}
|
||
uv run --script skills/reflect/scripts/reflect_apply.py --id <internal id> --set-patch tmp/patch.json
|
||
```
|
||
|
||
`old_text` must be copied from the current file and occur in it exactly once — copy it,
|
||
never retype it, and watch the quotation marks. The script verifies the patch *before*
|
||
storing it, so a refusal (exit 2) leaves the finding exactly as it was; relay the reason
|
||
and draft again. On success it prints the diff — show that, then ask for the `ok`. It
|
||
changes no file and decides nothing, so applying still needs step 4.
|
||
|
||
### 4. Apply (only after `ok`)
|
||
|
||
**Never edit the file yourself.** `scripts/reflect_apply.py` does it, and it is what
|
||
enforces the gates — it refuses a stale or ambiguous patch, commits only the touched
|
||
file, writes the audit line and updates the store. Editing by hand would bypass all of it.
|
||
|
||
```bash
|
||
uv run --script skills/reflect/scripts/reflect_apply.py --id <internal id>
|
||
```
|
||
|
||
- The `--id` is the finding's internal `id`, not the display number you showed the user.
|
||
- User rewrote the text (`edit:`)? Write their version to a temp file and pass
|
||
`--new-text-file <path>`.
|
||
- *skip* → `--skip`. Changes no file and no status; it only counts the deferral.
|
||
- *reject* → `--reject --reason "<user's words>"`. It changes no file.
|
||
- Exit code 2 means refused: relay the printed reason and move on. Do not work around it,
|
||
do not edit the file to make the patch fit.
|
||
|
||
On success the script prints the commit SHA and the revert command. Pass that on, then
|
||
continue with the next finding.
|
||
|
||
### 5. Close
|
||
|
||
When the user stops or the list is exhausted, summarise: how many applied, skipped,
|
||
rejected, and how many remain `open`. If anything was applied, remind the user that the
|
||
change lives only on the server and should be pulled back into the tracking repo.
|
||
|
||
## Decision history
|
||
|
||
Asked what was already decided, or what a past decision changed: read `log/reflect.log`
|
||
— it is the source of truth for the list, one line per decision, oldest first. Then fill
|
||
each line in from `findings.jsonl`:
|
||
|
||
- applied → the `patch` (the model's proposal), `applied.new_text` when the user rewrote
|
||
it, and `revert: git revert <applied.sha>`. Use `git show <sha>` for the real diff.
|
||
- rejected → `rejected.reason`. Records decided before reasons existed carry a flat
|
||
`rejected_at` and no reason; say the reason is missing rather than guessing one.
|
||
- skipped → `skipped.count`, still open.
|
||
|
||
Do not reconstruct this list from memory or from the findings store alone.
|
||
|
||
## When a finding is wrong
|
||
|
||
Findings come from an LLM reading its own logs and can be plain wrong. That is expected —
|
||
rejection is a normal outcome, not a failure. If several findings in a row are noise, say
|
||
so plainly; that is a signal the analysis prompt needs tuning, and it is worth telling the
|
||
user rather than working through a list of nonsense.
|