diff --git a/skills/reflect/README.md b/skills/reflect/README.md index fb2ee80..764e297 100644 --- a/skills/reflect/README.md +++ b/skills/reflect/README.md @@ -9,7 +9,7 @@ opravu. Nálezy pak procházíš ty, jeden po druhém, a rozhoduješ, co se pou ```text ANALÝZA — denně 03:30, bez tebe REVIEW — jen když napíšeš /reflect cron → reflect_auto.py ├─ vezme 1 otevřený nález - ├─ destiluje session z okna ├─ ukáže diagnózu, důkazy, návrh, diff + ├─ destiluje session z okna ├─ ukáže diagnózu, důkazy, diff + šanci ├─ LLM tah → nálezy ├─ čeká na tvoje rozhodnutí ├─ zapíše do findings.jsonl ├─ aplikuje → git commit → audit └─ Telegram (jen když je co) └─ další nález @@ -57,13 +57,36 @@ Zamítnutí je rozhodnutí, ne odklad — zamítnutý vzor se znovu neotevře. | `zamítni` | nález se zavře natrvalo | | `konec` | konec review | -Nález, který od analýzy patch nedostal (většina), si ho složí až při review: agent napíše -návrh do JSON a nechá ho ověřit (`--set-patch`). Skript ho **nejdřív ověří proti souboru -a teprve pak uloží**, takže nepoužitelný pokus ve `findings.jsonl` nezůstane — a agent do -store nesahá vůbec. Pak ti ukáže diff a čeká na `ok` jako u každého jiného patche. +Rozhoduješ vždycky nad **hotovým diffem, ne nad větou o něm**. Nález, který od analýzy patch +nedostal (většina), si ho složí ještě předtím, než ti ho předloží: agent napíše návrh do JSON +a nechá ho ověřit (`--set-patch`). Skript ho **nejdřív ověří proti souboru a teprve pak +uloží**, takže nepoužitelný pokus ve `findings.jsonl` nezůstane — a agent do store nesahá +vůbec. Nález, který se editací souboru opravit nedá („míň se překotně přizvukovat"), diff +nedostane — to ti agent řekne rovnou a nabídne jen přeskočení nebo zamítnutí. Vždy jen **jeden** nález najednou. Nálezy se číslují `1..N` podle pořadí, ne podle -interního id. +interního id. **Nahoře jsou regrese**, teprve pak se řadí podle závažnosti: závažnost +odhaduje model znovu v každém běhu a u téhož vzoru kolísá, kdežto „tohle už jednou +opravené bylo a vrátilo se" je fakt z auditu. + +## Šance, že oprava zabere + +Pod diffem je odhad typu `Šance, že zabere: ~40 %`. Nejde o změřenou úspěšnost, ale +o zařazení do jednoho ze **čtyř pásem** (~80 / ~60 / ~40 / ~20 %) podle toho, co patch dělá: + +| Co patch mění | Pásmo | +|---|---| +| mechaniku — skript nebo hradlo, které nejde ukecat | ~80 % | +| tvrdý zákaz do souboru, který je v kontextu ve chvíli, kdy chyba vzniká (`SOUL.md`, `AGENTS.md`, `SKILL.md` dotčeného skillu) | ~60 % | +| přeformulování existujícího pokynu v takovém souboru | ~40 % | +| soubor, který v tu chvíli v kontextu není, nebo rozhodnutí nechává na úvaze agenta | ~20 % | + +Hlavní osa je **jestli je opravovaný text vůbec v kontextu, když chyba nastává** — +sebelíp formulovaná věta v souboru, který se v tu chvíli nenačítá, chování změnit nemůže. +O pásmo dolů jde nález, který už jednou opravený byl a vrátil se (regrese), a nález, jehož +důkazy pocházejí z nesouvisejících situací. **Zastaralý nález odhad nedostane vůbec** — vzor +už možná dávno zmizel, takže není co předpovídat. Ke každému číslu patří věta, co ho tam +zařadilo. ## Kde co leží @@ -88,7 +111,7 @@ nastal, ani víc, než kolik jich v dávce vůbec bylo. | Rozhodnutí | Do `findings.jsonl` | Do `log/reflect.log` | |---|---|---| -| složení patche | `patch` + `patch_drafted_at` (= složeno při review, ne modelem) | `DRAFTED [vzor] ` | +| složení patche (u každého nálezu bez patche, ještě před předložením) | `patch` + `patch_drafted_at` (= složeno při review, ne modelem) | `DRAFTED [vzor] ` | | `ok` | `applied: {at, sha, file}`, `patch` = návrh modelu | `APPLIED [vzor] — ` | | `uprav:` | navíc `applied.new_text` = tvoje verze (návrh modelu zůstává v `patch`) | `APPLIED-EDITED …` | | `zamítni` | `rejected: {at, reason}` — **důvod je povinný** | `REJECTED [vzor] — ` | diff --git a/skills/reflect/SKILL.md b/skills/reflect/SKILL.md index 071cbd9..38f98da 100644 --- a/skills/reflect/SKILL.md +++ b/skills/reflect/SKILL.md @@ -40,38 +40,33 @@ Answer in the language the user writes in. |---|---| | `reflect/findings.jsonl` | one JSON object per line — the findings store | | `reflect/state.json` | cursor and per-run statistics (owned by the script) | -| `results/_reflect.md` | full report of a run | +| `results/_reflect.md` | full report of a run — read it when the user asks what the last run found | | `log/reflect.log` | append-only audit of every decision — applied, rejected, skipped | -A finding has: `id`, `status`, `created`, `pattern`, `severity`, `diagnosis`, -`last_seen`, `evidence`, `occurrences`, `sessions_affected`, `proposal`, optional `patch` -(`file` / `old_text` / `new_text`), optional `patch_drafted_at` (the patch was drafted -during a review, not proposed by the analysis), optional `regression_of`, optional `history`, -and once decided one of `applied` (`at` / `sha` / `file`, plus `new_text` and -`edited_by_user` when the user rewrote it), `rejected` (`at` / `reason`), `skipped` -(`count` / `last`). +Statuses: `watch` (seen once, not yet worth the user's attention) · `open` (waiting for +review) · `applied` · `rejected`. Beyond the obvious fields a finding carries: -Statuses: `watch` (seen once, not worth the user's attention yet) · `open` (waiting for -review) · `applied` · `rejected`. +- `last_seen` — the newest date in the **evidence**, not when the record was rewritten; + older records lack it, fall back to `created`. +- `history` — superseded records of this pattern as `":"`; the date in + `history[0]` is when it was first filed. +- `regression_of` — the `id` of the applied fix this pattern came back after. +- `patch` (`file` / `old_text` / `new_text`) and `patch_drafted_at` — see step 2. +- `applied` / `rejected` once decided, `skipped` while the finding is still open — see + [Decision history](#decision-history) for what is in them. -Two things about the numbers, both of which you must not overstate to the user: - -- A run analyses a **window** of recent sessions, not the whole history. `state.json` - holds the window it used; the dated report repeats it in its header. -- `occurrences` and `sessions_affected` are **cumulative across runs** — the sum of what - the model counted in each slice, not a figure anyone measured over the whole corpus. - `last_seen` is the newest date in the evidence, `history[0]` when the pattern was first - filed, `created` only when the record was last rewritten. Older records may lack - `last_seen`; fall back to `created` for those. +Do not overstate the numbers: a run analyses a **window** of recent sessions, not the whole +history (`state.json` holds the window, the report repeats it), and `occurrences` / +`sessions_affected` are **cumulative across runs** — what the model counted in each slice, +not a measurement over the corpus. ## Procedure ### 1. Load -Read `reflect/findings.jsonl`. Take the records with `status: open`, sorted by severity -(`high`, `medium`, `low`), then by `last_seen` and then by `occurrences`, all descending. -`occurrences` alone would let a stale pattern with a large cumulative count outrank a -fresh one. +Read `reflect/findings.jsonl`. Take the records with `status: open` and sort them: +**regressions first** (`regression_of` is set), then by severity (`high`, `medium`, `low`), +then `last_seen`, then `occurrences`, all descending. A finding whose `last_seen` is older than the window of the latest run (`state.json`, `window_from`) is **stale**: no run looks that far back any more, so nothing will refresh @@ -83,9 +78,44 @@ and in the audit log, and is never what you ask the user to type. If there are none: say so, mention how many `watch` findings are being tracked, and stop. -### 2. Present one finding +### 2. Prepare the patch -Show exactly one, in this shape: +The user decides on a diff, not on a sentence describing one, so have the patch ready +*before* you present the finding. + +- **It already has a `patch`** — the analysis wrote one, or `patch_drafted_at` says an + earlier review drafted it and ended without deciding; say so when you present it. Either + way, verify it still applies: + + ```bash + uv run --script skills/reflect/scripts/reflect_apply.py --id --check + ``` + + It changes nothing and prints the diff. Exit code 2 means the patch no longer applies — + relay the printed reason and offer only *skip* / *reject*. + +- **It has none** — most findings. Draft one now, before anything is shown. Read the target + file for one thing only: copying `old_text` out of it character-for-character (never + retype it, watch the quotation marks); it must occur in the file exactly once. The diff + always comes from the script, never from your own reading. File the draft with + `--set-patch`: + + ```bash + # tmp/patch.json: {"file": "SOUL.md", "old_text": "", "new_text": ""} + uv run --script skills/reflect/scripts/reflect_apply.py --id --set-patch tmp/patch.json + ``` + + A refusal (exit 2) changes nothing — relay the reason and draft again. On success it + prints the diff, touches no file and decides nothing; applying still needs step 4. + +- **No file edit can fix it** — "think before answering", "be less eager to agree". Say that + plainly, offer only *skip* / *reject*, and give no odds. Never invent a patch just to have + something to show. + +### 3. Present one finding + +Show exactly one, in this shape — labels in the user's language, the odds line as +[Estimating the odds](#estimating-the-odds) prescribes: ```text [1/4] retry-without-diagnosis · medium · 7× in 4 sessions @@ -97,85 +127,46 @@ Evidence: - … Proposal: + + + +Odds it works: ~40 % — rewords guidance in a file that is in context, but this pattern +already came back once after a fix. ``` -Write the labels in the user's language, not necessarily as shown here. - -The second line carries what the count alone hides: `history[0]` for when the pattern -was first filed, `last_seen` for when it last actually occurred, and `skipped.count` when -the user has already deferred it. Drop the skip part when there is none; a finding deferred -several times is worth saying so about, because rejecting it is cleaner than a list that +The second line comes from `history[0]` (first filed), `last_seen` (when the pattern last +actually occurred) and `skipped.count`; drop the skip part when there is none, and do say +when a finding has been deferred several times — rejecting it is cleaner than a list that keeps re-presenting it. -Mark a finding with `regression_of` clearly as a **regression** — this pattern was fixed -before and came back after the fix. Mark a stale finding (see step 1) as such too, and -say what it means: the evidence predates the current window, so the pattern may well be -gone already. - -If the finding has a `patch`, get the diff from the script — never by reading the file and -judging for yourself, that is the duplication gate 4 exists to prevent: - -```bash -uv run --script skills/reflect/scripts/reflect_apply.py --id --check -``` - -It changes nothing and prints the diff; show the user what it printed. Exit code 2 means the -patch no longer applies — relay the printed reason and offer only *skip* / *reject*. +Mark a finding with `regression_of` clearly as a **regression**, and a stale one (step 1) +as stale. Then ask for a decision and wait. -### 3. Accept a decision +### 4. Decide and apply -| Input | Meaning | -|---|---| -| `ok`, `apply` | apply this finding's patch | -| `edit: ` | the user rewrites `new_text`; show the new diff and ask again | -| `skip` | record the deferral, leave it `open`, move to the next | -| `reject` | **ask why first**, then close it for good — it never opens again | -| `stop` | end the review | +Every row runs `uv run --script skills/reflect/scripts/reflect_apply.py --id +…` — `--id` is the internal `id`, never the display number you showed. -Accept the equivalents in whatever language the user writes in — the words above are the -meanings, not a required vocabulary. +| Input | Meaning | Flags | +|---|---|---| +| `ok`, `apply` | apply this finding's patch | *(none)* | +| `edit: ` | the user rewrites `new_text` | write their version to a temp file; `--check --new-text-file ` shows the new diff and changes nothing, then on `ok` the same file **without** `--check` | +| `skip` | record the deferral, leave it `open`, move to the next | `--skip` | +| `reject` | **ask why first**, then close it for good — it never opens again | `--reject --reason ""` | +| `stop` | end the review | *(nothing to run)* | -Rejection needs a reason and the script will not take it without one. Ask for it in one -short question and pass the user's own words through — half of the first findings were -rejected, and that number only says something about the analysis if the reasons are on -the record. Do not invent a reason, and do not talk the user out of rejecting. +Keep that temp file until the finding is decided — the user's wording never enters `patch`. -A finding without a `patch` cannot be applied. Offer to draft one, and file it with -`--set-patch` — **never by writing to `reflect/findings.jsonl` yourself** (gate 4): +Accept equivalents in the language the user writes in — the words above are meanings, not a +required vocabulary. -```bash -# {"file": "SOUL.md", "old_text": "", "new_text": ""} -uv run --script skills/reflect/scripts/reflect_apply.py --id --set-patch tmp/patch.json -``` +Rejection needs a reason and the script will not take it without one: ask in one short +question, pass the user's own words through, invent nothing, and do not talk them out of it. -`old_text` must be copied from the current file and occur in it exactly once — copy it, -never retype it, and watch the quotation marks. The script verifies the patch *before* -storing it, so a refusal (exit 2) leaves the finding exactly as it was; relay the reason -and draft again. On success it prints the diff — show that, then ask for the `ok`. It -changes no file and decides nothing, so applying still needs step 4. - -### 4. Apply (only after `ok`) - -**Never edit the file yourself.** `scripts/reflect_apply.py` does it, and it is what -enforces the gates — it refuses a stale or ambiguous patch, commits only the touched -file, writes the audit line and updates the store. Editing by hand would bypass all of it. - -```bash -uv run --script skills/reflect/scripts/reflect_apply.py --id -``` - -- The `--id` is the finding's internal `id`, not the display number you showed the user. -- User rewrote the text (`edit:`)? Write their version to a temp file and pass - `--new-text-file `. -- *skip* → `--skip`. Changes no file and no status; it only counts the deferral. -- *reject* → `--reject --reason ""`. It changes no file. -- Exit code 2 means refused: relay the printed reason and move on. Do not work around it, - do not edit the file to make the patch fit. - -On success the script prints the commit SHA and the revert command. Pass that on, then -continue with the next finding. +Exit code 2 means refused: relay the printed reason and move on. On success the script +prints the commit SHA and the revert command — pass those on, then take the next finding. ### 5. Close @@ -183,6 +174,31 @@ When the user stops or the list is exhausted, summarise: how many applied, skipp rejected, and how many remain `open`. If anything was applied, remind the user that the change lives only on the server and should be pulled back into the tracking repo. +## Estimating the odds + +Read the odds off the record, not off a feeling — otherwise two reviews of the same finding +say 40 % and 75 %. Start from what the patch changes: + +| The patch … | Band | +|---|---| +| changes mechanism — a script, a gate the agent cannot talk itself past | ~80 % | +| adds a hard prohibition to a file that is in context when the pattern occurs (`SOUL.md`, `AGENTS.md`, the evidence's own `SKILL.md`) | ~60 % | +| rewords or softens existing guidance in such a file | ~40 % | +| edits a file that is not in context at that moment, or leaves the call to the agent's judgement | ~20 % | + +Move one band down for each of these — ~20 % is the floor — and name the reason in the line: + +- `regression_of` — this kind of instruction has already failed on this very pattern. +- the evidence spans unrelated contexts — the trigger is not in the file being patched. + +Two rules: + +- it is an estimate from the record, **not a measurement** — say so the first time you + give one; +- keep to the bands; `63 %` claims a precision that is not there. + +A stale finding (step 1) gets no percentage — say the pattern may already be gone. + ## Decision history Asked what was already decided, or what a past decision changed: read `log/reflect.log` @@ -199,7 +215,6 @@ Do not reconstruct this list from memory or from the findings store alone. ## When a finding is wrong -Findings come from an LLM reading its own logs and can be plain wrong. That is expected — -rejection is a normal outcome, not a failure. If several findings in a row are noise, say -so plainly; that is a signal the analysis prompt needs tuning, and it is worth telling the -user rather than working through a list of nonsense. +Findings come from an LLM reading its own logs and can be plain wrong; rejection is a normal +outcome, not a failure. If several in a row are noise, say so plainly instead of working +through the list — the analysis prompt needs tuning. diff --git a/skills/reflect/scripts/reflect_auto.py b/skills/reflect/scripts/reflect_auto.py index 3700bcf..7033a4a 100644 --- a/skills/reflect/scripts/reflect_auto.py +++ b/skills/reflect/scripts/reflect_auto.py @@ -604,8 +604,18 @@ def _is_stale(finding: Finding, stats: dict[str, Any]) -> bool: return bool(window_from) and finding.last_seen < window_from +def _report_order(finding: Finding) -> tuple[bool, bool, int]: + """Open first, then regressions, then severity — see render_report for why regressions win.""" + return finding.status != STATUS_OPEN, finding.regression_of is None, SEVERITIES.index(finding.severity) * -1 + + def render_report(merged: list[Finding], stats: dict[str, Any], previous_rate: float | None, when: datetime) -> str: - """The dated results/ report — the human-readable record behind the Telegram one-liner.""" + """The dated results/ report — the human-readable record behind the Telegram one-liner. + + Regressions come before severity: the model re-guesses `severity` every run and it drifts on + the same pattern, while `regression_of` is a fact from the audit — a fix that already failed + once belongs at the top whatever today's guess says. + """ lines = [ f"# Self-reflection {when:%Y-%m-%d}", "", @@ -621,7 +631,7 @@ def render_report(merged: list[Finding], stats: dict[str, Any], previous_rate: f lines.append("Nothing to report.") return "\n".join(lines) + "\n" - for finding in sorted(merged, key=lambda f: (f.status != STATUS_OPEN, SEVERITIES.index(f.severity) * -1)): + for finding in sorted(merged, key=_report_order): flag = " — REGRESSION" if finding.regression_of else "" if _is_stale(finding, stats): flag += " — STALE" diff --git a/skills/reflect/tests/test_reflect_apply.py b/skills/reflect/tests/test_reflect_apply.py index e8d4772..015a2f0 100644 --- a/skills/reflect/tests/test_reflect_apply.py +++ b/skills/reflect/tests/test_reflect_apply.py @@ -189,6 +189,20 @@ class TestCheckAndReject: assert "-If the fetch fails, try again." in printed assert "+If the fetch fails, STOP and diagnose." in printed + def test_check_previews_the_users_own_rewrite(self, workspace, tmp_path, capsys): + """`edit:` shows the user's wording as a diff first, so --check has to take it too — + without --check the same flag applies and commits straight away.""" + replacement = tmp_path / "new.txt" + replacement.write_text("If the fetch fails, ask the user.", encoding="utf-8") + exit_code = reflect_apply.main( + ["--id", "f7a2", "--check", "--new-text-file", str(replacement), "--workspace", str(workspace)] + ) + + assert exit_code == 0 + assert "+If the fetch fails, ask the user." in capsys.readouterr().out + assert (workspace / TARGET_REL).read_text(encoding="utf-8") == ORIGINAL + assert _findings(workspace)[0]["status"] == "open" + def test_check_fails_on_a_stale_patch(self, workspace): _write_findings(workspace, _record(patch={**_record()["patch"], "old_text": "absent"})) assert reflect_apply.main(["--id", "f7a2", "--check", "--workspace", str(workspace)]) == 2 diff --git a/skills/reflect/tests/test_reflect_auto.py b/skills/reflect/tests/test_reflect_auto.py index 512290b..8490396 100644 --- a/skills/reflect/tests/test_reflect_auto.py +++ b/skills/reflect/tests/test_reflect_auto.py @@ -394,6 +394,17 @@ class TestRendering: report = reflect_auto.render_report(merged, _stats(), None, WHEN) assert "REGRESSION" in report + def test_a_regression_outranks_a_higher_severity_finding(self): + """Severity is the model's per-run guess and drifts; `regression_of` is a fact from the audit.""" + applied = _filed(status="applied", applied={"at": "2026-07-01", "sha": "abc1234", "file": "SOUL.md"}) + regression = _raw_finding(severity="low") + louder = _raw_finding(pattern="speculation-presented-as-fact", severity="high", occurrences=3, sessions_affected=2) + merged = reflect_auto.merge_findings([applied], _parsed(_answer(louder, regression)), TODAY) + report = reflect_auto.render_report(merged, _stats(), None, WHEN) + + assert [f.status for f in merged] == [reflect_auto.STATUS_OPEN] * 2, "both must be open for the order to matter" + assert report.index("`retry-without-diagnosis`") < report.index("`speculation-presented-as-fact`") + def test_report_renders_a_patch_as_a_diff(self): raw = _raw_finding(patch={"file": "a.md", "old_text": "try again", "new_text": "STOP"}) merged = reflect_auto.merge_findings([], _parsed(_answer(raw)), TODAY)