nanobot: 2026-09-10 12:33:37
This commit is contained in:
lachtan
215
skills/wiki/README.md
Normal file
215
skills/wiki/README.md
Normal file
@@ -0,0 +1,215 @@
|
||||
# wiki — jak to funguje
|
||||
|
||||
Hybrid hledání ve tvých vlastních poznámkách: dva git repozitáře (`index`, `travel`) plus živý
|
||||
workspace nanobota. Skill je **výhradně čtecí** — do tvých repů nikdy nezapisuje a klony drží
|
||||
jen jako pracovní kopii. Index je derived artifact: kdykoli se dá smazat a postavit od nuly
|
||||
(u dnešního objemu jsou to desítky sekund).
|
||||
|
||||
## Tři vrstvy, od nejlevnější
|
||||
|
||||
```text
|
||||
grep ripgrep přes soubory na disku žádný index, nikdy zastaralé, vidí i kód
|
||||
↓
|
||||
toc katalog `files` z databáze adresář → soubor → titulek + tagy
|
||||
↓
|
||||
search FTS5 (BM25) + vec0 (KNN) → RRF parafráze, když neznáš slova
|
||||
```
|
||||
|
||||
Není to fallback řetěz, kde se jde dolů, když horní vrstva zklame — je to volba podle **tvaru
|
||||
otázky**: přesný string, hostname nebo identifikátor patří do `grep`, orientace („co o tom
|
||||
vůbec mám?") do `toc`, parafráze do `search`.
|
||||
|
||||
`grep` je tedy základ, ne poslední záchrana. Soubory na disku beztak leží, takže nestojí nic
|
||||
navíc, a nad kódem je exact match to hlavní — proto se zdrojáky vůbec neindexují (viz níž).
|
||||
|
||||
## Index nikdy nevzniká v tahu agenta
|
||||
|
||||
Indexuje cron každou minutu. Agent v tahu jen **čte** hotový index.
|
||||
|
||||
Proč takhle: `exec` tool má timeout 60 s a plný index je minuty. Indexace v odpovědi by
|
||||
navíc při nedostupné Ollamě dělala z čerstvého dokumentu FTS-only výsledek.
|
||||
|
||||
Drtivá většina tiků neudělá nic a **nezapíše ani řádek**. Když je `log/wiki_sync_cron.log`
|
||||
prázdný, je to správný stav, ne že cron neběží.
|
||||
|
||||
## Co se v tom tiku vlastně děje
|
||||
|
||||
1. Vezme lock `wiki/.sync.lock`. Když ho drží živý běh, skončí bez výpisu. Lock po mrtvém
|
||||
procesu (`pid` neexistuje nebo je starší než 30 min) si vezme zpátky a zaloguje
|
||||
`WARN stale lock, reclaiming`.
|
||||
2. **Levná detekce změn, bez indexace.** U git zdrojů `git ls-remote` — zjistí remote `HEAD`
|
||||
**bez** `fetch`, což je na minutovou kadenci ten správný nástroj; `fetch` teprve když se
|
||||
revize liší. U workspace zdroje walk a porovnání `path` + `size` + `mtime`; sha256 se
|
||||
počítá jen při neshodě.
|
||||
3. Nic se nezměnilo a nic nečeká na vektor → konec.
|
||||
4. Změněné soubory se rozřežou na chunky a pošlou do Ollamy po 32. Pak se **doberou všechny
|
||||
chunky bez vektoru**, i když se žádný soubor nezměnil — tohle je cesta zpátky
|
||||
z degradovaného režimu, když byla Ollama chvíli mimo.
|
||||
5. Zapíše `indexed_rev` a `last_sync_at`, přidá coverage řádek a shrnutí do
|
||||
`log/wiki_sync.log`.
|
||||
|
||||
Selhání sítě je **per zdroj**: timeout nebo chyba gitu zaloguje `WARN`, ten zdroj přeskočí
|
||||
s nezměněnou `indexed_rev` a ostatní dojedou. Bez timeoutů by visící `ls-remote` držel lock
|
||||
a zablokoval i workspace zdroj, který se sítí nemá nic společného.
|
||||
|
||||
## Proč hybrid
|
||||
|
||||
Každá polovina umí něco jiného a ani jedna neumí obojí:
|
||||
|
||||
| | Umí | Neumí |
|
||||
|---|---|---|
|
||||
| **BM25** (FTS5) | exact match, čísla, jména, zkratky | parafrázi, kde se slova nepřekrývají |
|
||||
| **vektory** (`vec0`) | „elektroodpad ze stavebnic" → „Mindstorms po ukončení podpory" | přesné řetězce a čísla |
|
||||
|
||||
Výsledky slučuje **RRF** — sečte převrácené ranky z obou seznamů. Aby to mělo definovaný
|
||||
význam, musí obě poloviny řadit **tutéž množinu**: proto je jedinou retrieval jednotkou chunk,
|
||||
ne soubor. Kdyby BM25 řadil soubory a vektory chunky, merge by nešlo interpretovat.
|
||||
|
||||
Naměřeno na reálných poznámkách: RRF dává MRR 0,367 proti 0,257 (BM25 sám) a 0,261 (vektory
|
||||
samy), takže merge opravdu vyhrává. Plná čísla, včetně srovnání čtyř modelů, jsou
|
||||
v `.claude/tracking/plans/final-wiki-hybrid-rag-result.md` v trackovacím repu.
|
||||
|
||||
Ve výstupu `search` u každého hitu stojí, která polovina ho našla (`bm25 #2, vec #1`). Hit,
|
||||
na kterém se poloviny **shodnou**, má výrazně větší cenu než ten, který vytáhly jen vektory.
|
||||
|
||||
## Česká flexe — na co narazíš
|
||||
|
||||
Dotazová vrstva lepí `*` na slova od tří znaků, aby pokryla české koncovky. **Funguje to jen
|
||||
napůl** a je dobré vědět jak: wildcard je **prefixový**, takže pomůže jen tehdy, když je tvoje
|
||||
slovo prefixem tvaru v dokumentu. Česká flexe ale mění koncovku, ne začátek.
|
||||
|
||||
Naměřeno nad `travel/packaging-list.md`:
|
||||
|
||||
| Dotaz | Chunků v tom souboru |
|
||||
|---|---|
|
||||
| `cestu*` | **0** |
|
||||
| `cesty*` | 3 |
|
||||
| `cest*` | 4 |
|
||||
|
||||
Prakticky: na *„co si vzít na cestu do zahraničí"* se doslovný seznam věcí na cestu nedostal
|
||||
ani do top 10. Když víš, že něco máš, a `search` to nenajde, zkus **kmen slova** (`cest`,
|
||||
`záloh`) nebo rovnou `grep`. Rozhodnutí, jestli to řešit systémově, je otevřená položka
|
||||
v `todo.md` trackovacího repa — každá varianta rozšiřuje kandidátní pool, takže to není
|
||||
zdarma.
|
||||
|
||||
## Index je nápověda, ne odpověď
|
||||
|
||||
Dvě věci, které se snadno přehlédnou:
|
||||
|
||||
- **Výřez v `search` není zdroj pravdy.** Je zkrácený a index může být o minutu starší než
|
||||
disk. Než se z nálezu odpovídá, má se ten soubor přečíst čerstvý — `SKILL.md` to agentovi
|
||||
říká, ale platí to i pro tebe.
|
||||
- **Vektorová polovina vždycky něco vrátí.** KNN nemá dolní mez podobnosti, takže na jakýkoli
|
||||
dotaz nabídne svých k nejbližších chunků. Pro retrieval je to záměr (posoudit relevanci je
|
||||
na čtenáři), ale znamená to, že prázdný výsledek nikdy neznamená „nic nesedí" — znamená
|
||||
prázdný index.
|
||||
|
||||
## Co se indexuje a co ne
|
||||
|
||||
Rozsah řídí `wiki/config.yaml` ve třech krocích: `paths` (whitelist — co v něm není, pro index
|
||||
neexistuje) → `include` (whitelist přípon, `*.md`) → `exclude` (skalpel, vyhrává nad oběma).
|
||||
|
||||
Proč whitelist a ne blacklist: rozhoduje **směr selhání**. U blacklistu nový adresář *tiše
|
||||
vstoupí* do indexu, u whitelistu *tiše chybí*. Index je komprimovaná kopie obsahu včetně
|
||||
osobních věcí, takže „tiše zaindexováno" je horší porucha — a workspace se mění i bez tebe,
|
||||
Dream do něj zapisuje sám.
|
||||
|
||||
Proti té jediné slabině whitelistu stojí záchranná síť: sync na konci zaloguje řádek
|
||||
`coverage <zdroj>: markdown outside paths in …` s top-level adresáři, které markdown mají
|
||||
a žádný zdroj je nepokrývá. Adresáře, které tam vidíš pořád (`backup`, `tmp`, `skills`, …),
|
||||
jsou vědomě vynechané; zajímavý je **nový** přírůstek v tom seznamu.
|
||||
|
||||
**Indexuje se výhradně markdown**, a to bez jakékoli detekce typu obsahu — `main.py` se
|
||||
netrefí do `include: ["*.md"]` a k chunkeru se nedostane. Důvod je měřený: markdown parser
|
||||
dělá z Python komentáře `# TODO: …` nadpis a `unicode61` neumí identifikátory (`send` uvnitř
|
||||
`SendAsync` je miss). Kód pokrývá `grep` — jede přes celý klon včetně `.txt` a `.mhtml`.
|
||||
|
||||
## Kde co leží
|
||||
|
||||
Relativně ke workspace (`~/.nanobot/workspace`):
|
||||
|
||||
| Cesta | Role |
|
||||
|---|---|
|
||||
| `wiki/config.yaml` | zdroje a model — **jediný verzovaný soubor** pod `wiki/` |
|
||||
| `wiki/index.sqlite` | `chunks` + FTS5 + vektory; derived, gitignorovaný |
|
||||
| `wiki/remote/<id>/` | non-bare klony git zdrojů (working tree, aby měl `grep` co číst) |
|
||||
| `wiki/.sync.lock` | JSON `{pid, started_at}`, jen když sync běží |
|
||||
| `log/wiki_sync.log` | jeden řádek na událost — co se naindexovalo, `WARN`, coverage |
|
||||
| `log/wiki_sync_cron.log` | stdout cronu; **prázdný je správně** |
|
||||
| `skills/wiki/scripts/` | `wiki_sync.py` (indexace) a `wiki_search.py` (dotazy) |
|
||||
|
||||
Klon je záměrně non-bare: `--mirror` nemá working tree, takže by `grep` neměl co číst.
|
||||
Cena je dvojnásobek místa na disku, což je na těchhle repech nic.
|
||||
|
||||
## Ruční spuštění
|
||||
|
||||
```bash
|
||||
cd ~/.nanobot/workspace
|
||||
|
||||
# to, co dělá cron
|
||||
~/.local/bin/uv run skills/wiki/scripts/wiki_sync.py
|
||||
|
||||
# jen jeden zdroj
|
||||
~/.local/bin/uv run skills/wiki/scripts/wiki_sync.py --source travel
|
||||
|
||||
# smazat index a postavit od nuly
|
||||
~/.local/bin/uv run skills/wiki/scripts/wiki_sync.py --full
|
||||
|
||||
# dotazy
|
||||
~/.local/bin/uv run skills/wiki/scripts/wiki_search.py search "zálohování" --limit 5
|
||||
~/.local/bin/uv run skills/wiki/scripts/wiki_search.py toc --source travel
|
||||
~/.local/bin/uv run skills/wiki/scripts/wiki_search.py toc --tag caj
|
||||
~/.local/bin/uv run skills/wiki/scripts/wiki_search.py grep "WireGuard" --source index
|
||||
```
|
||||
|
||||
`--full` potřebuješ jen po změně embedding modelu, jeho dimenzí, `query_prefix` nebo chunkeru
|
||||
— tedy přesně tehdy, když dotaz začne hlásit `reindex needed`. Přeembeduje všechno, takže to
|
||||
nepouštěj v tahu, kde na to někdo čeká.
|
||||
|
||||
Plná cesta k `uv` je tu proto, že v neinteraktivním SSH není v `PATH`. Crontab si `PATH`
|
||||
nastavuje sám, takže tam stačí `uv run …`.
|
||||
|
||||
## Když něco nefunguje
|
||||
|
||||
| Hláška | Co to je |
|
||||
|---|---|
|
||||
| `note: embeddings unavailable … FTS-only results` | Ollama na `nvidia.hell` neodpovídá. Sync mezitím indexuje dál a vektory dosadí sám, až se vrátí. |
|
||||
| `note: N chunks still awaiting vectors` | Sync je rozjezdu nebo byl degradovaný. Sémantická polovina je zatím neúplná. |
|
||||
| `reindex needed: …` (exit 1) | Index byl postavený pod jiným embedding kontraktem než má config. Poznámky jsou v pořádku, řeší to `--full`. |
|
||||
| `WARN <zdroj>: git … failed` | Ten zdroj se přeskočil, ostatní dojely. `indexed_rev` zůstala, takže příští tik to zkusí znovu. |
|
||||
| `(no matches)` | Prázdný index — ne „nic nesedí" (viz výš). |
|
||||
|
||||
Všechno ostatní hledej v `log/wiki_sync.log`. Když je prázdný i po změně poznámek, problém je
|
||||
v cronu, ne v hledání.
|
||||
|
||||
## Jak ověřit, že to funguje
|
||||
|
||||
**Smoke loop v konzoli** — projde celou smyčku od zápisu po nalezení:
|
||||
|
||||
```bash
|
||||
cd ~/.nanobot/workspace
|
||||
printf '# Smoke\n\nZrzavý jednorožec kontroluje wiki index.\n' > notes/_smoke.md
|
||||
sleep 90
|
||||
~/.local/bin/uv run skills/wiki/scripts/wiki_search.py search "zrzavý jednorožec" --limit 3
|
||||
rm notes/_smoke.md # cron ho z indexu odklidí sám do minuty
|
||||
```
|
||||
|
||||
**Dotaz do WebUI na každý zdroj.** Princip: vyber fakt, který model **nemůže** znát
|
||||
z obecných znalostí — pak je konkrétní správná odpověď sama důkazem, že přišla z poznámek.
|
||||
Když odpoví obecně a čísla vynechá nebo si je vymyslí, neprošel.
|
||||
|
||||
Příklady platné k 2026-09-09 (poznámky se mění, princip ne):
|
||||
|
||||
| Zdroj | Dotaz | Musí padnout |
|
||||
|---|---|---|
|
||||
| `workspace` | Kolik vody a čaje mám v receptu na Teh Tarik? | 700–800 ml, dvě polévkové lžíce čaje |
|
||||
| `travel` | Co si mám podle poznámek vzít na Sněžku? | rozdvojka do zásuvky, power banka, Sony ANC sluchátka |
|
||||
| `index` | Co je projekt Sauter Bus? | RS485 proxy s odposlechem a modifikací provozu, micro:bit |
|
||||
|
||||
**Negativní kontrola** je nejcennější z celé sady: zeptej se na téma, které v poznámkách
|
||||
prokazatelně není (např. pěstování bonsají). Správná odpověď je „nic o tom nemáš". Když
|
||||
začne vyprávět, doplňuje si výsledky z obecných znalostí — a to je zrádnější vada než
|
||||
nefunkční index, protože se tváří jako odpověď z poznámek.
|
||||
|
||||
Když si nejsi jistý, odkud odpověď je, dopiš **„ze kterého souboru to je?"**. Má přijít
|
||||
konkrétní cesta, ne „z tvých poznámek".
|
||||
106
skills/wiki/SKILL.md
Normal file
106
skills/wiki/SKILL.md
Normal file
@@ -0,0 +1,106 @@
|
||||
---
|
||||
name: wiki
|
||||
description: >
|
||||
Search the user's own notes — personal git note repos plus the live workspace — by
|
||||
keyword, topic or meaning. Use to answer questions about what the user has written
|
||||
down, recorded or decided, and to find which file covers a topic.
|
||||
Triggers on: "/wiki", "find in my notes", "what did I write about", "do I have notes on".
|
||||
Read-only: it never writes, edits or captures notes, and it does not answer from
|
||||
general knowledge — only from what is indexed on disk.
|
||||
---
|
||||
|
||||
# /wiki
|
||||
|
||||
Hybrid search over the user's notes. `chunks` is the single retrieval unit: FTS5 supplies a
|
||||
BM25 rank, `sqlite-vec` a vector rank, and RRF merges the two.
|
||||
|
||||
Reply to the user in their own language.
|
||||
|
||||
## Pick a layer
|
||||
|
||||
Three layers, cheapest first. Start with the one that matches the question.
|
||||
|
||||
| Question shape | Command |
|
||||
|---|---|
|
||||
| exact string, filename, hostname, IP, identifier, code | `grep <pattern>` |
|
||||
| "what do I even have about X", orientation, browsing | `toc [--source X] [--tag Y]` |
|
||||
| a topic or a paraphrase, wording unknown | `search "<query>"` |
|
||||
|
||||
```sh
|
||||
uv run skills/wiki/scripts/wiki_search.py grep "rotate_snapshots"
|
||||
uv run skills/wiki/scripts/wiki_search.py toc --source travel
|
||||
uv run skills/wiki/scripts/wiki_search.py search "jak snížit elektroodpad" --limit 5
|
||||
```
|
||||
|
||||
For the full flag reference of any subcommand:
|
||||
|
||||
```sh
|
||||
uv run skills/wiki/scripts/wiki_search.py --help
|
||||
uv run skills/wiki/scripts/wiki_search.py <subcommand> --help
|
||||
```
|
||||
|
||||
## Behavioral contract
|
||||
|
||||
**The index is a retrieval hint, never the answer.** `search` returns an excerpt to tell you
|
||||
*which file* is relevant. Before answering, read that file fresh from disk — the index can be
|
||||
up to a minute behind, and the excerpt is truncated. Never quote figures, commands or
|
||||
decisions straight from a `search` excerpt.
|
||||
|
||||
**Only markdown is indexed; `grep` sees everything.** Source code, configs and data files never
|
||||
reach `search` or `toc`. For anything code-shaped — an identifier, a flag, a function name —
|
||||
`grep` is the right layer, not a fallback. Exact match is what matters there anyway.
|
||||
|
||||
**`grep` is a regex.** Escape regex metacharacters when the user means them literally
|
||||
(`.`, `(`, `[`, `*`, `+`, `?`, `|`). It searches the files on disk, so it is always current.
|
||||
|
||||
**`search` results always look plausible.** The vector half has no distance floor: it returns
|
||||
its nearest chunks whatever you ask, so an empty result set means an empty index, not "nothing
|
||||
matches". Judge each hit against the question and say so when nothing genuinely fits — do not
|
||||
present a weak nearest neighbour as an answer.
|
||||
|
||||
**Read the notes the output prints.** Each hit shows `source:path`, the breadcrumb, and which
|
||||
half found it (`bm25 #2, vec #1`). A hit both halves rank highly is worth more than one only
|
||||
the vectors found.
|
||||
|
||||
## Notes the output may print
|
||||
|
||||
- `embeddings unavailable … FTS-only results` — the semantic half is down, so only keyword
|
||||
matching ran. Paraphrase queries will do badly; say so rather than concluding the notes are
|
||||
silent on the topic. Try `grep` with the user's own wording.
|
||||
- `N chunks still awaiting vectors` — a sync is mid-flight or was degraded. Recent notes may
|
||||
be missing from the semantic half.
|
||||
- `reindex needed: …` on stderr with exit 1 — the index was built under a different embedding
|
||||
contract. Nothing is wrong with the notes; the index has to be rebuilt (see below). Do not
|
||||
retry the query.
|
||||
|
||||
## Sources
|
||||
|
||||
Defined in `wiki/config.yaml`. Each has a stable **source id** used in output and in
|
||||
`--source`. Git sources are read-only clones under `wiki/remote/<id>/`; the `workspace` source
|
||||
indexes the live workspace. The user's repos are the canonical data — nothing here ever writes
|
||||
to them.
|
||||
|
||||
`toc --tag` filters on frontmatter tags, which only exist where the user wrote them.
|
||||
|
||||
## Indexing
|
||||
|
||||
`wiki_sync.py` runs every minute from the crontab and does all the work offline. Never index
|
||||
inside a turn: a full rebuild takes minutes and the `exec` tool times out at 60 s.
|
||||
|
||||
```sh
|
||||
uv run skills/wiki/scripts/wiki_sync.py # what cron runs; a quiet tick is free
|
||||
uv run skills/wiki/scripts/wiki_sync.py --full # wipe and rebuild
|
||||
```
|
||||
|
||||
`--full` is needed only after the embedding model, its dimensions, the query prefix or the
|
||||
chunker change — that is what a `reindex needed` message means. It re-embeds everything, so
|
||||
run it in the background and tell the user it is running, never inside a turn where they wait.
|
||||
|
||||
`log/wiki_sync.log` holds one line per event: what was indexed, `WARN` for a source that was
|
||||
skipped, and a `coverage` line naming top-level directories that hold markdown no source
|
||||
covers. Read it when `search` cannot find something the user is sure they wrote.
|
||||
|
||||
## Environment
|
||||
|
||||
- `WIKI_DB` — override the index path (tests).
|
||||
- `WIKI_CONFIG` — override the config path (tests).
|
||||
260
skills/wiki/scripts/wiki_chunker.py
Normal file
260
skills/wiki/scripts/wiki_chunker.py
Normal file
@@ -0,0 +1,260 @@
|
||||
#!/usr/bin/env python3
|
||||
# /// script
|
||||
# requires-python = ">=3.11"
|
||||
# dependencies = ["markdown-it-py", "pyyaml"]
|
||||
# ///
|
||||
"""Structure-aware markdown chunker for the wiki skill.
|
||||
|
||||
Splits a document along its H1-H3 heading hierarchy, then merges small siblings and
|
||||
splits oversized sections along block boundaries. Every chunk carries a breadcrumb
|
||||
(`path > title > section > subsection`) which goes into the embedded text as well as
|
||||
the metadata, so a vector represents a passage in context rather than in isolation.
|
||||
|
||||
markdown-it-py supplies the AST (it knows the CommonMark edge cases); the chunking
|
||||
policy below is ours. Block boundaries come from token line maps, so the emitted text
|
||||
is the original markdown — code fences and tables stay byte-for-byte intact.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import re
|
||||
from dataclasses import dataclass, field
|
||||
|
||||
import yaml
|
||||
from markdown_it import MarkdownIt # ty: ignore[unresolved-import]
|
||||
|
||||
CHUNKER_VERSION = "1"
|
||||
|
||||
MERGE_BELOW = 200
|
||||
SPLIT_ABOVE = 800
|
||||
OVERLAP = 64
|
||||
CHARS_PER_TOKEN = 4
|
||||
|
||||
SECTION_LEVELS = (1, 2, 3)
|
||||
BREADCRUMB_SEP = " > "
|
||||
|
||||
_FRONTMATTER_RE = re.compile(r"\A---[ \t]*\r?\n(.*?)\r?\n---[ \t]*(?:\r?\n|\Z)", re.DOTALL)
|
||||
|
||||
_md = MarkdownIt("commonmark")
|
||||
|
||||
|
||||
@dataclass(frozen=True)
|
||||
class Chunk:
|
||||
breadcrumb: str
|
||||
text: str
|
||||
|
||||
|
||||
@dataclass(frozen=True)
|
||||
class ParsedFile:
|
||||
title: str | None = None
|
||||
tags: list[str] = field(default_factory=list)
|
||||
headings: list[str] = field(default_factory=list)
|
||||
chunks: list[Chunk] = field(default_factory=list)
|
||||
|
||||
|
||||
@dataclass(frozen=True)
|
||||
class _Heading:
|
||||
start: int
|
||||
end: int
|
||||
level: int
|
||||
title: str
|
||||
|
||||
|
||||
@dataclass(frozen=True)
|
||||
class _Section:
|
||||
breadcrumb: tuple[str, ...]
|
||||
text: str
|
||||
|
||||
@property
|
||||
def parent(self) -> tuple[str, ...]:
|
||||
return self.breadcrumb[:-1]
|
||||
|
||||
|
||||
def token_estimate(text: str) -> int:
|
||||
"""Approximate token count. Sizing does not need a real tokenizer (plan: 4 chars/token)."""
|
||||
return max(1, len(text) // CHARS_PER_TOKEN)
|
||||
|
||||
|
||||
def parse_markdown(text: str, path: str) -> ParsedFile:
|
||||
"""Chunk one markdown document. `path` is the source-relative path, the breadcrumb root."""
|
||||
frontmatter, body = _split_frontmatter(text)
|
||||
frontmatter_title = _frontmatter_title(frontmatter)
|
||||
tags = _frontmatter_tags(frontmatter)
|
||||
|
||||
root: tuple[str, ...] = (path,) if frontmatter_title is None else (path, frontmatter_title)
|
||||
sections, headings = _split_sections(body, root)
|
||||
merged = _merge_small_siblings(sections)
|
||||
chunks = [chunk for section in merged for chunk in _split_oversized(section)]
|
||||
|
||||
# The catalog title falls back to the first heading, because notes carry their title as
|
||||
# `# H1` far more often than as frontmatter. It deliberately does NOT feed the breadcrumb
|
||||
# root: the H1 already reaches the breadcrumb through the heading stack, and changing the
|
||||
# root would rewrite every chunk's text and force a full reindex.
|
||||
title = frontmatter_title or (headings[0] if headings else None)
|
||||
return ParsedFile(title=title, tags=tags, headings=headings, chunks=chunks)
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# frontmatter
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
|
||||
def _split_frontmatter(text: str) -> tuple[dict, str]:
|
||||
"""Peel off a leading YAML frontmatter block. Malformed frontmatter stays body text."""
|
||||
match = _FRONTMATTER_RE.match(text)
|
||||
if not match:
|
||||
return {}, text
|
||||
try:
|
||||
data = yaml.safe_load(match.group(1))
|
||||
except yaml.YAMLError:
|
||||
return {}, text
|
||||
if not isinstance(data, dict):
|
||||
return {}, text
|
||||
return data, text[match.end() :]
|
||||
|
||||
|
||||
def _frontmatter_title(frontmatter: dict) -> str | None:
|
||||
title = frontmatter.get("title")
|
||||
if isinstance(title, str) and title.strip():
|
||||
return title.strip()
|
||||
return None
|
||||
|
||||
|
||||
def _frontmatter_tags(frontmatter: dict) -> list[str]:
|
||||
raw = frontmatter.get("tags")
|
||||
if isinstance(raw, str):
|
||||
values = raw.split(",")
|
||||
elif isinstance(raw, list):
|
||||
values = [str(item) for item in raw]
|
||||
else:
|
||||
return []
|
||||
return [tag.strip() for tag in values if tag.strip()]
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# sectioning
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
|
||||
def _split_sections(body: str, root: tuple[str, ...]) -> tuple[list[_Section], list[str]]:
|
||||
"""Cut the body at H1-H3 headings. H4+ stay inside their parent section."""
|
||||
lines = body.split("\n")
|
||||
tokens = _md.parse(body)
|
||||
|
||||
starts: list[_Heading] = []
|
||||
for index, token in enumerate(tokens):
|
||||
if token.type != "heading_open" or token.map is None:
|
||||
continue
|
||||
level = int(token.tag[1:])
|
||||
if level not in SECTION_LEVELS:
|
||||
continue
|
||||
inline = tokens[index + 1] if index + 1 < len(tokens) else None
|
||||
title = inline.content.strip() if inline is not None else ""
|
||||
starts.append(_Heading(start=token.map[0], end=token.map[1], level=level, title=title))
|
||||
|
||||
headings = [heading.title for heading in starts]
|
||||
boundaries = [heading.start for heading in starts] + [len(lines)]
|
||||
|
||||
sections: list[_Section] = []
|
||||
preamble = "\n".join(lines[: boundaries[0]]).strip()
|
||||
if preamble:
|
||||
sections.append(_Section(root, preamble))
|
||||
|
||||
stack: list[tuple[int, str]] = []
|
||||
for position, heading in enumerate(starts):
|
||||
while stack and stack[-1][0] >= heading.level:
|
||||
stack.pop()
|
||||
stack.append((heading.level, heading.title))
|
||||
section_end = boundaries[position + 1]
|
||||
# A heading with no prose of its own would embed as a bare title; the heading
|
||||
# still reaches the index through its children's breadcrumbs, so drop it.
|
||||
if not "\n".join(lines[heading.end : section_end]).strip():
|
||||
continue
|
||||
text = "\n".join(lines[heading.start : section_end]).strip()
|
||||
sections.append(_Section(_dedupe(root + tuple(title for _, title in stack)), text))
|
||||
return sections, headings
|
||||
|
||||
|
||||
def _dedupe(parts: tuple[str, ...]) -> tuple[str, ...]:
|
||||
"""Drop consecutive repeats so a frontmatter title matching the H1 shows up once."""
|
||||
out: list[str] = []
|
||||
for part in parts:
|
||||
if not out or out[-1] != part:
|
||||
out.append(part)
|
||||
return tuple(out)
|
||||
|
||||
|
||||
def _merge_small_siblings(sections: list[_Section]) -> list[_Section]:
|
||||
"""Glue a too-small section onto the following sibling under the same parent."""
|
||||
merged: list[_Section] = []
|
||||
for section in sections:
|
||||
if not merged:
|
||||
merged.append(section)
|
||||
continue
|
||||
previous = merged[-1]
|
||||
combined = f"{previous.text}\n\n{section.text}"
|
||||
if (
|
||||
token_estimate(previous.text) < MERGE_BELOW
|
||||
and previous.parent == section.parent
|
||||
and token_estimate(combined) <= SPLIT_ABOVE
|
||||
):
|
||||
merged[-1] = _Section(previous.breadcrumb, combined)
|
||||
else:
|
||||
merged.append(section)
|
||||
return merged
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# splitting
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
|
||||
def _split_oversized(section: _Section) -> list[Chunk]:
|
||||
"""Break a section over SPLIT_ABOVE into block-aligned pieces with OVERLAP carry-over."""
|
||||
breadcrumb = BREADCRUMB_SEP.join(section.breadcrumb)
|
||||
if token_estimate(section.text) <= SPLIT_ABOVE:
|
||||
return [Chunk(breadcrumb, _embed_text(breadcrumb, section.text))]
|
||||
|
||||
blocks = _top_level_blocks(section.text)
|
||||
chunks: list[Chunk] = []
|
||||
current: list[str] = []
|
||||
for block in blocks:
|
||||
candidate = [*current, block]
|
||||
if current and token_estimate("\n\n".join(candidate)) > SPLIT_ABOVE:
|
||||
chunks.append(Chunk(breadcrumb, _embed_text(breadcrumb, "\n\n".join(current))))
|
||||
current = [*_overlap_tail(current), block]
|
||||
else:
|
||||
current = candidate
|
||||
if current:
|
||||
chunks.append(Chunk(breadcrumb, _embed_text(breadcrumb, "\n\n".join(current))))
|
||||
return chunks
|
||||
|
||||
|
||||
def _top_level_blocks(text: str) -> list[str]:
|
||||
"""Top-level markdown blocks, taken from token line maps so fences stay whole."""
|
||||
lines = text.split("\n")
|
||||
tokens = _md.parse(text)
|
||||
starts = sorted({token.map[0] for token in tokens if token.level == 0 and token.map})
|
||||
if not starts:
|
||||
return [text]
|
||||
bounds = [*starts, len(lines)]
|
||||
blocks = ["\n".join(lines[bounds[i] : bounds[i + 1]]).strip() for i in range(len(starts))]
|
||||
return [block for block in blocks if block]
|
||||
|
||||
|
||||
def _overlap_tail(blocks: list[str]) -> list[str]:
|
||||
"""Trailing whole blocks of the emitted chunk, up to OVERLAP tokens."""
|
||||
tail: list[str] = []
|
||||
budget = OVERLAP
|
||||
for block in reversed(blocks):
|
||||
cost = token_estimate(block)
|
||||
if cost > budget:
|
||||
break
|
||||
tail.insert(0, block)
|
||||
budget -= cost
|
||||
return tail
|
||||
|
||||
|
||||
def _embed_text(breadcrumb: str, text: str) -> str:
|
||||
"""The stored chunk text carries its breadcrumb, so the vector sees the context."""
|
||||
return f"{breadcrumb}\n\n{text}"
|
||||
225
skills/wiki/scripts/wiki_config.py
Normal file
225
skills/wiki/scripts/wiki_config.py
Normal file
@@ -0,0 +1,225 @@
|
||||
#!/usr/bin/env python3
|
||||
# /// script
|
||||
# requires-python = ">=3.11"
|
||||
# dependencies = ["pyyaml"]
|
||||
# ///
|
||||
"""Layout and configuration for the wiki skill.
|
||||
|
||||
All runtime data lives under `workspace/wiki/`; only `config.yaml` is versioned.
|
||||
|
||||
The source id is the stable key — the catalog and the vectors hang off it, while a
|
||||
URL or a path may change. Renaming an id is therefore an explicit invalidation of
|
||||
that source's index, not a rename.
|
||||
|
||||
Scope precedence is `paths` (whitelist — what is not in it does not exist for the
|
||||
index) then `include` (extension whitelist) then `exclude` (scalpel, wins over both).
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import os
|
||||
import re
|
||||
from dataclasses import dataclass
|
||||
from pathlib import Path
|
||||
|
||||
import yaml
|
||||
|
||||
# workspace/skills/wiki/scripts/wiki_config.py -> parents[3] = workspace root.
|
||||
WORKSPACE = Path(__file__).resolve().parents[3]
|
||||
WIKI_DIR = WORKSPACE / "wiki"
|
||||
DEFAULT_DB_PATH = WIKI_DIR / "index.sqlite"
|
||||
DEFAULT_CONFIG_PATH = WIKI_DIR / "config.yaml"
|
||||
REMOTE_DIR = WIKI_DIR / "remote"
|
||||
LOCK_PATH = WIKI_DIR / ".sync.lock"
|
||||
SYNC_LOG_PATH = WORKSPACE / "log" / "wiki_sync.log"
|
||||
|
||||
GIT_KIND = "git"
|
||||
WORKSPACE_KIND = "workspace"
|
||||
VALID_KINDS = (GIT_KIND, WORKSPACE_KIND)
|
||||
|
||||
|
||||
def db_path() -> Path:
|
||||
"""Index location, overridable with WIKI_DB for tests."""
|
||||
return Path(os.environ.get("WIKI_DB", str(DEFAULT_DB_PATH)))
|
||||
|
||||
|
||||
def config_path() -> Path:
|
||||
"""Config location, overridable with WIKI_CONFIG for tests."""
|
||||
return Path(os.environ.get("WIKI_CONFIG", str(DEFAULT_CONFIG_PATH)))
|
||||
|
||||
|
||||
class ConfigError(ValueError):
|
||||
"""Malformed or incomplete wiki/config.yaml."""
|
||||
|
||||
|
||||
@dataclass(frozen=True)
|
||||
class EmbeddingConfig:
|
||||
endpoint: str
|
||||
model: str
|
||||
dims: int
|
||||
batch: int
|
||||
keep_alive: int
|
||||
query_prefix: str
|
||||
|
||||
|
||||
@dataclass(frozen=True)
|
||||
class SourceConfig:
|
||||
source_id: str
|
||||
kind: str
|
||||
url: str | None
|
||||
paths: tuple[str, ...]
|
||||
include: tuple[str, ...]
|
||||
exclude: tuple[str, ...]
|
||||
|
||||
def covers(self, rel_path: str) -> bool:
|
||||
"""True when a source-relative path belongs in the index."""
|
||||
if not any(_matches(pattern, rel_path) for pattern in self.paths):
|
||||
return False
|
||||
name = rel_path.rsplit("/", 1)[-1]
|
||||
if not any(_matches(pattern, name) for pattern in self.include):
|
||||
return False
|
||||
return not any(_matches(pattern, rel_path) for pattern in self.exclude)
|
||||
|
||||
def covers_dir(self, rel_dir: str) -> bool:
|
||||
"""Cheap walk prune: could anything under this directory ever be covered?"""
|
||||
if not rel_dir:
|
||||
return True
|
||||
probe = f"{rel_dir}/"
|
||||
if any(_matches(pattern, probe) for pattern in self.exclude):
|
||||
return False
|
||||
return any(_prefix_could_match(pattern, probe) for pattern in self.paths)
|
||||
|
||||
|
||||
@dataclass(frozen=True)
|
||||
class WikiConfig:
|
||||
embedding: EmbeddingConfig
|
||||
sources: tuple[SourceConfig, ...]
|
||||
|
||||
def source(self, source_id: str) -> SourceConfig | None:
|
||||
return next((s for s in self.sources if s.source_id == source_id), None)
|
||||
|
||||
|
||||
def source_root(source: SourceConfig) -> Path:
|
||||
"""Where a source's files live. Git clones are derived from the id, never configured."""
|
||||
if source.kind == GIT_KIND:
|
||||
return REMOTE_DIR / source.source_id
|
||||
return WORKSPACE
|
||||
|
||||
|
||||
def load_config(path: Path | None = None) -> WikiConfig:
|
||||
path = path or config_path()
|
||||
if not path.exists():
|
||||
raise ConfigError(f"missing config: {path}")
|
||||
try:
|
||||
raw = yaml.safe_load(path.read_text(encoding="utf-8"))
|
||||
except yaml.YAMLError as exc:
|
||||
raise ConfigError(f"unparseable config {path}: {exc}") from exc
|
||||
if not isinstance(raw, dict):
|
||||
raise ConfigError(f"config {path} must be a mapping")
|
||||
return WikiConfig(
|
||||
embedding=_parse_embedding(raw.get("embedding")),
|
||||
sources=_parse_sources(raw.get("sources")),
|
||||
)
|
||||
|
||||
|
||||
def _as_mapping(raw: object, what: str) -> dict[str, object]:
|
||||
if not isinstance(raw, dict):
|
||||
raise ConfigError(f"{what} must be a mapping")
|
||||
return {str(key): value for key, value in raw.items()}
|
||||
|
||||
|
||||
def _parse_embedding(raw: object) -> EmbeddingConfig:
|
||||
values = _as_mapping(raw, "`embedding`")
|
||||
missing = [key for key in ("endpoint", "model", "dims") if key not in values]
|
||||
if missing:
|
||||
raise ConfigError(f"embedding is missing {', '.join(missing)}")
|
||||
keep_alive = values.get("keep_alive", -1)
|
||||
if not isinstance(keep_alive, int):
|
||||
# Ollama rejects a string keep_alive of "-1" with HTTP 400.
|
||||
raise ConfigError("embedding.keep_alive must be a number, not a string")
|
||||
return EmbeddingConfig(
|
||||
endpoint=str(values["endpoint"]).rstrip("/"),
|
||||
model=str(values["model"]),
|
||||
dims=int(str(values["dims"])),
|
||||
batch=int(str(values.get("batch", 32))),
|
||||
keep_alive=keep_alive,
|
||||
query_prefix=str(values.get("query_prefix", "")),
|
||||
)
|
||||
|
||||
|
||||
def _parse_sources(raw: object) -> tuple[SourceConfig, ...]:
|
||||
entries = _as_mapping(raw, "`sources`")
|
||||
if not entries:
|
||||
raise ConfigError("config needs a non-empty `sources` mapping")
|
||||
sources = []
|
||||
for source_id, raw_body in entries.items():
|
||||
body = _as_mapping(raw_body, f"source {source_id}")
|
||||
kind = str(body.get("kind"))
|
||||
if kind not in VALID_KINDS:
|
||||
raise ConfigError(f"source {source_id}: kind must be one of {VALID_KINDS}, got {kind!r}")
|
||||
url = body.get("url")
|
||||
if kind == GIT_KIND and not url:
|
||||
raise ConfigError(f"source {source_id}: git sources need a url")
|
||||
sources.append(
|
||||
SourceConfig(
|
||||
source_id=source_id,
|
||||
kind=kind,
|
||||
url=str(url) if url else None,
|
||||
paths=_as_patterns(body.get("paths"), source_id, "paths"),
|
||||
include=_as_patterns(body.get("include") or ["*.md"], source_id, "include"),
|
||||
exclude=_as_patterns(body.get("exclude") or [], source_id, "exclude"),
|
||||
)
|
||||
)
|
||||
return tuple(sources)
|
||||
|
||||
|
||||
def _as_patterns(raw: object, source_id: str, key: str) -> tuple[str, ...]:
|
||||
if raw is None:
|
||||
raise ConfigError(f"source {source_id}: `{key}` is required")
|
||||
if not isinstance(raw, list):
|
||||
raise ConfigError(f"source {source_id}: `{key}` must be a list")
|
||||
return tuple(str(item) for item in raw)
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# glob matching
|
||||
#
|
||||
# fnmatch lets `*` cross a `/` and PurePath.match has no recursive `**` before
|
||||
# Python 3.13, so the patterns are translated by hand.
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
_SEGMENT_ANY = "[^/]*"
|
||||
|
||||
|
||||
def _glob_to_regex(pattern: str) -> str:
|
||||
parts = pattern.split("/")
|
||||
out = []
|
||||
for index, part in enumerate(parts):
|
||||
is_last = index == len(parts) - 1
|
||||
if part == "**":
|
||||
out.append(".*" if is_last else "(?:[^/]+/)*")
|
||||
else:
|
||||
segment = re.escape(part).replace(r"\*", _SEGMENT_ANY).replace(r"\?", "[^/]")
|
||||
out.append(segment if is_last else segment + "/")
|
||||
return "^" + "".join(out) + "$"
|
||||
|
||||
|
||||
def _matches(pattern: str, value: str) -> bool:
|
||||
return re.match(_glob_to_regex(pattern), value) is not None
|
||||
|
||||
|
||||
def _prefix_could_match(pattern: str, directory: str) -> bool:
|
||||
"""True when `pattern` can still match something below `directory`."""
|
||||
if pattern.startswith("**"):
|
||||
return True
|
||||
pattern_parts = pattern.split("/")
|
||||
dir_parts = [part for part in directory.split("/") if part]
|
||||
for depth, dir_part in enumerate(dir_parts):
|
||||
if depth >= len(pattern_parts):
|
||||
return False
|
||||
pattern_part = pattern_parts[depth]
|
||||
if pattern_part == "**":
|
||||
return True
|
||||
if not _matches(pattern_part, dir_part):
|
||||
return False
|
||||
return True
|
||||
129
skills/wiki/scripts/wiki_db.py
Normal file
129
skills/wiki/scripts/wiki_db.py
Normal file
@@ -0,0 +1,129 @@
|
||||
#!/usr/bin/env python3
|
||||
# /// script
|
||||
# requires-python = ">=3.11"
|
||||
# dependencies = ["sqlite-vec==0.1.6"]
|
||||
# ///
|
||||
"""SQLite storage layer for the wiki skill.
|
||||
|
||||
Schema, connection factory and the sqlite-vec extension load.
|
||||
|
||||
`chunks` is the canonical retrieval unit: `chunks_fts` gives it a BM25 rank and
|
||||
`vec_chunks` a KNN rank, both keyed on `chunks.id`, so RRF merges two rankings of
|
||||
the *same* set.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import sqlite3
|
||||
from pathlib import Path
|
||||
|
||||
import sqlite_vec # ty: ignore[unresolved-import]
|
||||
|
||||
# Compiled into the vec0 table definition. `wiki_embed` guards config.dims against it —
|
||||
# a mismatch means the index was built for a different model.
|
||||
EMBEDDING_DIMS = 1024
|
||||
|
||||
SCHEMA = f"""
|
||||
PRAGMA journal_mode = WAL;
|
||||
PRAGMA foreign_keys = ON;
|
||||
|
||||
CREATE TABLE IF NOT EXISTS meta (
|
||||
key TEXT PRIMARY KEY,
|
||||
value TEXT NOT NULL
|
||||
);
|
||||
|
||||
CREATE TABLE IF NOT EXISTS sources (
|
||||
source_id TEXT PRIMARY KEY,
|
||||
kind TEXT NOT NULL CHECK(kind IN ('git', 'workspace')),
|
||||
indexed_rev TEXT,
|
||||
last_sync_at TEXT
|
||||
);
|
||||
|
||||
CREATE TABLE IF NOT EXISTS files (
|
||||
source_id TEXT NOT NULL REFERENCES sources(source_id),
|
||||
path TEXT NOT NULL,
|
||||
title TEXT,
|
||||
tags TEXT,
|
||||
headings TEXT,
|
||||
sha256 TEXT NOT NULL,
|
||||
size INTEGER NOT NULL,
|
||||
mtime REAL,
|
||||
indexed_at TEXT NOT NULL,
|
||||
PRIMARY KEY (source_id, path)
|
||||
);
|
||||
|
||||
CREATE TABLE IF NOT EXISTS chunks (
|
||||
id INTEGER PRIMARY KEY AUTOINCREMENT,
|
||||
source_id TEXT NOT NULL,
|
||||
path TEXT NOT NULL,
|
||||
chunk_idx INTEGER NOT NULL,
|
||||
breadcrumb TEXT NOT NULL,
|
||||
text TEXT NOT NULL,
|
||||
embedded_at TEXT,
|
||||
UNIQUE (source_id, path, chunk_idx),
|
||||
FOREIGN KEY (source_id, path) REFERENCES files(source_id, path) ON DELETE CASCADE
|
||||
);
|
||||
|
||||
CREATE INDEX IF NOT EXISTS idx_chunks_pending ON chunks(id) WHERE embedded_at IS NULL;
|
||||
|
||||
CREATE VIRTUAL TABLE IF NOT EXISTS chunks_fts USING fts5(
|
||||
breadcrumb, text,
|
||||
content='chunks', content_rowid='id',
|
||||
tokenize='unicode61 remove_diacritics 2'
|
||||
);
|
||||
|
||||
CREATE TRIGGER IF NOT EXISTS chunks_ai AFTER INSERT ON chunks BEGIN
|
||||
INSERT INTO chunks_fts(rowid, breadcrumb, text) VALUES (new.id, new.breadcrumb, new.text);
|
||||
END;
|
||||
|
||||
CREATE TRIGGER IF NOT EXISTS chunks_ad AFTER DELETE ON chunks BEGIN
|
||||
INSERT INTO chunks_fts(chunks_fts, rowid, breadcrumb, text)
|
||||
VALUES('delete', old.id, old.breadcrumb, old.text);
|
||||
END;
|
||||
|
||||
-- The WHEN guard keeps `UPDATE chunks SET embedded_at` (sync step 4b) from rewriting
|
||||
-- an FTS row whose indexed text did not change.
|
||||
CREATE TRIGGER IF NOT EXISTS chunks_au AFTER UPDATE ON chunks
|
||||
WHEN old.text IS NOT new.text OR old.breadcrumb IS NOT new.breadcrumb
|
||||
BEGIN
|
||||
INSERT INTO chunks_fts(chunks_fts, rowid, breadcrumb, text)
|
||||
VALUES('delete', old.id, old.breadcrumb, old.text);
|
||||
INSERT INTO chunks_fts(rowid, breadcrumb, text) VALUES (new.id, new.breadcrumb, new.text);
|
||||
END;
|
||||
|
||||
CREATE VIRTUAL TABLE IF NOT EXISTS vec_chunks USING vec0(
|
||||
embedding float[{EMBEDDING_DIMS}] distance_metric=cosine
|
||||
);
|
||||
"""
|
||||
|
||||
|
||||
def get_db(path: Path) -> sqlite3.Connection:
|
||||
"""Return an autocommit connection with sqlite-vec loaded and foreign keys on."""
|
||||
conn = sqlite3.connect(path, isolation_level=None)
|
||||
conn.enable_load_extension(True)
|
||||
sqlite_vec.load(conn)
|
||||
conn.enable_load_extension(False)
|
||||
conn.execute("PRAGMA journal_mode = WAL")
|
||||
conn.execute("PRAGMA foreign_keys = ON")
|
||||
conn.row_factory = sqlite3.Row
|
||||
_migrate(conn)
|
||||
return conn
|
||||
|
||||
|
||||
def _migrate(conn: sqlite3.Connection) -> None:
|
||||
"""Idempotently bring an existing DB up to the current schema.
|
||||
|
||||
init_db only runs the full SCHEMA on a missing file, so live DBs never see
|
||||
later additions. Each step must be a no-op once applied.
|
||||
"""
|
||||
# No migrations yet — schema_version 1 is the initial shape.
|
||||
|
||||
|
||||
def init_db(path: Path) -> None:
|
||||
"""Create tables, indexes and triggers if they don't exist."""
|
||||
path.parent.mkdir(parents=True, exist_ok=True)
|
||||
conn = get_db(path)
|
||||
try:
|
||||
conn.executescript(SCHEMA)
|
||||
finally:
|
||||
conn.close()
|
||||
127
skills/wiki/scripts/wiki_embed.py
Normal file
127
skills/wiki/scripts/wiki_embed.py
Normal file
@@ -0,0 +1,127 @@
|
||||
#!/usr/bin/env python3
|
||||
# /// script
|
||||
# requires-python = ">=3.11"
|
||||
# dependencies = ["requests", "pyyaml", "markdown-it-py", "sqlite-vec==0.1.6"]
|
||||
# ///
|
||||
"""Embedding client and the identity guard over the vector space.
|
||||
|
||||
Two invariants live here:
|
||||
|
||||
* Vectors are stored L2-normalized, so cosine distance equals the dot product.
|
||||
* The query prefix must be bit-identical at index and at query time. That is the real
|
||||
reason it is persisted in `meta` rather than only read from the config — mixing
|
||||
vectors produced under two different contracts degrades results silently, which is
|
||||
the most expensive kind of bug.
|
||||
|
||||
An unreachable Ollama is not an error here: the caller stores chunks with
|
||||
`embedded_at IS NULL` and the query side says out loud that it is FTS-only.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import math
|
||||
from collections.abc import Sequence
|
||||
from dataclasses import dataclass
|
||||
|
||||
import requests
|
||||
from wiki_chunker import CHUNKER_VERSION
|
||||
from wiki_config import EmbeddingConfig
|
||||
from wiki_db import EMBEDDING_DIMS
|
||||
|
||||
SCHEMA_VERSION = "1"
|
||||
SQLITE_VEC_VERSION = "0.1.6"
|
||||
REQUEST_TIMEOUT_SECONDS = 60
|
||||
|
||||
# Only these keys make two vectors comparable; the rest of `meta` is informational.
|
||||
GUARDED_META_KEYS = (
|
||||
"embedding_model",
|
||||
"embedding_dims",
|
||||
"normalized",
|
||||
"query_prefix",
|
||||
"chunker_version",
|
||||
)
|
||||
|
||||
|
||||
class EmbeddingUnavailable(RuntimeError):
|
||||
"""Ollama could not be reached or refused the request."""
|
||||
|
||||
|
||||
class IndexIdentityMismatch(RuntimeError):
|
||||
"""The index was built under a different embedding contract — a reindex is needed."""
|
||||
|
||||
|
||||
def expected_meta(config: EmbeddingConfig) -> dict[str, str]:
|
||||
return {
|
||||
"embedding_model": config.model,
|
||||
"embedding_dims": str(config.dims),
|
||||
"normalized": "l2",
|
||||
"query_prefix": config.query_prefix,
|
||||
"chunker_version": CHUNKER_VERSION,
|
||||
"schema_version": SCHEMA_VERSION,
|
||||
"sqlite_vec_version": SQLITE_VEC_VERSION,
|
||||
}
|
||||
|
||||
|
||||
def meta_mismatches(stored: dict[str, str], config: EmbeddingConfig) -> list[str]:
|
||||
"""Guarded meta keys that disagree with the config. Empty on a fresh (unwritten) index."""
|
||||
if not stored:
|
||||
return []
|
||||
expected = expected_meta(config)
|
||||
return [key for key in GUARDED_META_KEYS if stored.get(key) != expected[key]]
|
||||
|
||||
|
||||
def require_matching_index(stored: dict[str, str], config: EmbeddingConfig) -> None:
|
||||
"""Refuse to query an index built under a different contract."""
|
||||
if config.dims != EMBEDDING_DIMS:
|
||||
raise IndexIdentityMismatch(f"reindex needed: config dims {config.dims} != schema dims {EMBEDDING_DIMS}")
|
||||
mismatches = meta_mismatches(stored, config)
|
||||
if mismatches:
|
||||
detail = ", ".join(
|
||||
f"{key}: index={stored.get(key)!r} config={expected_meta(config)[key]!r}" for key in mismatches
|
||||
)
|
||||
raise IndexIdentityMismatch(f"reindex needed: {detail}")
|
||||
|
||||
|
||||
def l2_normalize(vector: Sequence[float]) -> list[float]:
|
||||
norm = math.sqrt(sum(value * value for value in vector))
|
||||
if norm == 0.0:
|
||||
return list(vector)
|
||||
return [value / norm for value in vector]
|
||||
|
||||
|
||||
@dataclass(frozen=True)
|
||||
class OllamaEmbedder:
|
||||
config: EmbeddingConfig
|
||||
|
||||
def embed_documents(self, texts: Sequence[str]) -> list[list[float]]:
|
||||
"""Documents are embedded without the instruct prefix (the model is asymmetric)."""
|
||||
return self._embed(list(texts))
|
||||
|
||||
def embed_query(self, text: str) -> list[float]:
|
||||
return self._embed([self.config.query_prefix + text])[0]
|
||||
|
||||
def probe(self) -> None:
|
||||
"""Raise EmbeddingUnavailable unless the endpoint answers an embed request."""
|
||||
self._embed(["ping"])
|
||||
|
||||
def _embed(self, inputs: list[str]) -> list[list[float]]:
|
||||
if not inputs:
|
||||
return []
|
||||
payload = {
|
||||
"model": self.config.model,
|
||||
"input": inputs,
|
||||
"keep_alive": self.config.keep_alive,
|
||||
}
|
||||
try:
|
||||
response = requests.post(f"{self.config.endpoint}/api/embed", json=payload, timeout=REQUEST_TIMEOUT_SECONDS)
|
||||
response.raise_for_status()
|
||||
embeddings = response.json()["embeddings"]
|
||||
except (requests.RequestException, KeyError, ValueError) as exc:
|
||||
raise EmbeddingUnavailable(f"{self.config.endpoint}: {exc}") from exc
|
||||
|
||||
if len(embeddings) != len(inputs):
|
||||
raise EmbeddingUnavailable(f"asked for {len(inputs)} vectors, got {len(embeddings)}")
|
||||
for vector in embeddings:
|
||||
if len(vector) != self.config.dims:
|
||||
raise EmbeddingUnavailable(f"model returned {len(vector)} dims, config says {self.config.dims}")
|
||||
return [l2_normalize(vector) for vector in embeddings]
|
||||
308
skills/wiki/scripts/wiki_search.py
Normal file
308
skills/wiki/scripts/wiki_search.py
Normal file
@@ -0,0 +1,308 @@
|
||||
#!/usr/bin/env -S uv run --script
|
||||
# /// script
|
||||
# requires-python = ">=3.11"
|
||||
# dependencies = ["requests", "pyyaml", "markdown-it-py", "sqlite-vec==0.1.6"]
|
||||
# ///
|
||||
"""Query side of the wiki skill — three layers, cheapest first.
|
||||
|
||||
grep live ripgrep over the files on disk. No index, never stale, and not limited
|
||||
to the indexed extensions, so it is the layer that covers source code.
|
||||
toc directory -> file -> title + tags, read straight from the `files` catalog.
|
||||
search FTS5 (BM25) and vec0 (KNN) over the same `chunks` rows, merged with RRF.
|
||||
|
||||
Both halves rank the same unit, which is what makes the merge meaningful. The
|
||||
parameters below are module constants on purpose: `--limit` is the only knob worth
|
||||
exposing, and a config key that never changes is a config key that rots.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import argparse
|
||||
import json
|
||||
import shutil
|
||||
import sqlite3
|
||||
import subprocess
|
||||
import sys
|
||||
from pathlib import Path
|
||||
|
||||
import wiki_store as store
|
||||
from wiki_config import (
|
||||
GIT_KIND,
|
||||
ConfigError,
|
||||
SourceConfig,
|
||||
WikiConfig,
|
||||
db_path,
|
||||
load_config,
|
||||
source_root,
|
||||
)
|
||||
from wiki_embed import (
|
||||
EmbeddingUnavailable,
|
||||
IndexIdentityMismatch,
|
||||
OllamaEmbedder,
|
||||
require_matching_index,
|
||||
)
|
||||
|
||||
CANDIDATE_LIMIT = 50 # KNN k, and the LIMIT for BM25
|
||||
RRF_K = 60 # Cormack et al. 2009
|
||||
DEFAULT_LIMIT = 10
|
||||
PREFIX_MIN_LENGTH = 3 # a one- or two-character prefix matches too widely to carry signal
|
||||
EXCERPT_CHARS = 320
|
||||
GREP_TIMEOUT = 30
|
||||
GREP_MAX_PER_FILE = 5
|
||||
|
||||
|
||||
def rrf_merge(ranked_lists: list[list[int]]) -> list[tuple[int, float]]:
|
||||
"""score(id) = sum over lists of 1 / (RRF_K + rank). No weights: both halves count equally."""
|
||||
scores: dict[int, float] = {}
|
||||
for ids in ranked_lists:
|
||||
for rank, chunk_id in enumerate(ids, start=1):
|
||||
scores[chunk_id] = scores.get(chunk_id, 0.0) + 1.0 / (RRF_K + rank)
|
||||
return sorted(scores.items(), key=lambda item: (-item[1], item[0]))
|
||||
|
||||
|
||||
def fts_match_expression(query: str) -> str:
|
||||
"""Turn free text into an FTS5 OR-query of quoted prefix terms.
|
||||
|
||||
Terms are quoted so reserved words (`and`, `not`, `near`) and punctuation cannot be
|
||||
read as operators. The trailing `*` covers Czech inflection, which `unicode61` does
|
||||
not stem — `záloh*` finds záloha/zálohování/zálohy.
|
||||
"""
|
||||
terms = []
|
||||
for word in _tokenize(query):
|
||||
wildcard = "*" if len(word) >= PREFIX_MIN_LENGTH else ""
|
||||
terms.append(f'"{word}"{wildcard}')
|
||||
return " OR ".join(terms)
|
||||
|
||||
|
||||
def _tokenize(text: str) -> list[str]:
|
||||
return "".join(char if char.isalnum() else " " for char in text).split()
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# search
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
|
||||
def run_search(config: WikiConfig, query: str, limit: int) -> int:
|
||||
expression = fts_match_expression(query)
|
||||
if not expression:
|
||||
print("(empty query)")
|
||||
return 0
|
||||
|
||||
with store.connection(db_path()) as conn:
|
||||
try:
|
||||
require_matching_index(store.read_meta(conn), config.embedding)
|
||||
except IndexIdentityMismatch as exc:
|
||||
print(str(exc), file=sys.stderr)
|
||||
return 1
|
||||
|
||||
bm25_ids = store.bm25_ranked_ids(conn, expression, CANDIDATE_LIMIT)
|
||||
|
||||
vector_ids: list[int] = []
|
||||
degraded: str | None = None
|
||||
try:
|
||||
vector = OllamaEmbedder(config.embedding).embed_query(query)
|
||||
vector_ids = store.knn_ranked_ids(conn, vector, CANDIDATE_LIMIT)
|
||||
except EmbeddingUnavailable as exc:
|
||||
degraded = f"note: embeddings unavailable ({exc}) — FTS-only results"
|
||||
|
||||
pending = store.index_stats(conn)["pending"]
|
||||
merged = rrf_merge([ids for ids in (bm25_ids, vector_ids) if ids])[:limit]
|
||||
rows = store.fetch_chunks(conn, [chunk_id for chunk_id, _ in merged])
|
||||
|
||||
if degraded:
|
||||
print(degraded)
|
||||
elif pending:
|
||||
print(f"note: {pending} chunks still awaiting vectors — semantic half is incomplete")
|
||||
|
||||
if not merged:
|
||||
print("(no matches)")
|
||||
return 0
|
||||
|
||||
bm25_rank = {chunk_id: rank for rank, chunk_id in enumerate(bm25_ids, start=1)}
|
||||
vector_rank = {chunk_id: rank for rank, chunk_id in enumerate(vector_ids, start=1)}
|
||||
for position, (chunk_id, score) in enumerate(merged, start=1):
|
||||
row = rows.get(chunk_id)
|
||||
if row is None:
|
||||
continue
|
||||
origin = _origin_label(bm25_rank.get(chunk_id), vector_rank.get(chunk_id))
|
||||
print(f"{position}. {row['source_id']}:{row['path']} (rrf {score:.4f}, {origin})")
|
||||
print(f" {row['breadcrumb']}")
|
||||
print(_indent(_excerpt(row["text"], row["breadcrumb"])))
|
||||
print()
|
||||
return 0
|
||||
|
||||
|
||||
def _origin_label(bm25: int | None, vector: int | None) -> str:
|
||||
parts = []
|
||||
if bm25 is not None:
|
||||
parts.append(f"bm25 #{bm25}")
|
||||
if vector is not None:
|
||||
parts.append(f"vec #{vector}")
|
||||
return ", ".join(parts)
|
||||
|
||||
|
||||
def _excerpt(text: str, breadcrumb: str) -> str:
|
||||
body = text[len(breadcrumb) :].lstrip("\n") if text.startswith(breadcrumb) else text
|
||||
body = " ".join(body.split())
|
||||
return body[:EXCERPT_CHARS] + ("…" if len(body) > EXCERPT_CHARS else "")
|
||||
|
||||
|
||||
def _indent(text: str) -> str:
|
||||
return "\n".join(f" {line}" for line in text.splitlines())
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# toc
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
|
||||
def run_toc(source_id: str | None, tag: str | None) -> int:
|
||||
with store.connection(db_path()) as conn:
|
||||
rows = store.list_toc_files(conn, source_id=source_id, tag=tag)
|
||||
if not rows:
|
||||
print("(no indexed files match)")
|
||||
return 0
|
||||
|
||||
by_source: dict[str, list[sqlite3.Row]] = {}
|
||||
for row in rows:
|
||||
by_source.setdefault(row["source_id"], []).append(row)
|
||||
|
||||
for source, files in by_source.items():
|
||||
print(f"{source} ({len(files)} files)")
|
||||
print()
|
||||
directory = None
|
||||
for row in files:
|
||||
path = row["path"]
|
||||
parent, _, name = path.rpartition("/")
|
||||
if parent != directory:
|
||||
directory = parent
|
||||
print(f"{parent}/" if parent else "./")
|
||||
print(f" {name:<28} {row['title'] or '':<32} {_tag_label(row['tags'])}".rstrip())
|
||||
print()
|
||||
return 0
|
||||
|
||||
|
||||
def _tag_label(raw: str | None) -> str:
|
||||
try:
|
||||
tags = json.loads(raw) if raw else []
|
||||
except json.JSONDecodeError:
|
||||
return ""
|
||||
return f"[{', '.join(tags)}]" if tags else ""
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# grep
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
|
||||
def grep_roots(source: SourceConfig) -> list[Path]:
|
||||
"""Directories ripgrep should walk for one source.
|
||||
|
||||
A git clone is searched whole. A workspace source is bounded by the literal prefix
|
||||
of each `paths` glob, so grep stays inside what the source claims without being
|
||||
narrowed to the indexed extensions.
|
||||
"""
|
||||
root = source_root(source)
|
||||
if source.kind == GIT_KIND:
|
||||
return [root] if root.is_dir() else []
|
||||
roots = []
|
||||
for pattern in source.paths:
|
||||
prefix = _literal_prefix(pattern)
|
||||
candidate = root / prefix if prefix else root
|
||||
if candidate.is_dir() and candidate not in roots:
|
||||
roots.append(candidate)
|
||||
return roots
|
||||
|
||||
|
||||
def _literal_prefix(pattern: str) -> str:
|
||||
parts = []
|
||||
for segment in pattern.split("/"):
|
||||
if any(char in segment for char in "*?["):
|
||||
break
|
||||
parts.append(segment)
|
||||
return "/".join(parts)
|
||||
|
||||
|
||||
def run_grep(config: WikiConfig, pattern: str, source_id: str | None) -> int:
|
||||
if shutil.which("rg") is None:
|
||||
print("ripgrep (rg) not found", file=sys.stderr)
|
||||
return 1
|
||||
sources = [s for s in config.sources if not source_id or s.source_id == source_id]
|
||||
if not sources:
|
||||
print(f"unknown source {source_id!r}", file=sys.stderr)
|
||||
return 1
|
||||
|
||||
roots = [path for source in sources for path in grep_roots(source)]
|
||||
if not roots:
|
||||
print("(nothing on disk to grep — has wiki_sync.py run?)")
|
||||
return 0
|
||||
|
||||
command = [
|
||||
"rg",
|
||||
"--line-number",
|
||||
"--no-heading",
|
||||
"--color",
|
||||
"never",
|
||||
"--smart-case",
|
||||
"--max-count",
|
||||
str(GREP_MAX_PER_FILE),
|
||||
"--glob",
|
||||
"!.git",
|
||||
pattern,
|
||||
*[str(path) for path in roots],
|
||||
]
|
||||
try:
|
||||
result = subprocess.run(command, capture_output=True, text=True, timeout=GREP_TIMEOUT, check=False)
|
||||
except subprocess.TimeoutExpired:
|
||||
print(f"grep timed out after {GREP_TIMEOUT}s", file=sys.stderr)
|
||||
return 1
|
||||
if result.returncode not in (0, 1):
|
||||
print(result.stderr.strip(), file=sys.stderr)
|
||||
return 1
|
||||
output = result.stdout.strip()
|
||||
print(output if output else "(no matches)")
|
||||
return 0
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# cli
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
|
||||
def parse_args(argv: list[str] | None = None) -> argparse.Namespace:
|
||||
parser = argparse.ArgumentParser(description="Search the wiki index over your notes.")
|
||||
subparsers = parser.add_subparsers(dest="command", required=True)
|
||||
|
||||
search = subparsers.add_parser("search", help="hybrid BM25 + vector search over chunks")
|
||||
search.add_argument("query", help="free text; Czech inflection is covered by prefix matching")
|
||||
search.add_argument("--limit", type=int, default=DEFAULT_LIMIT, help="chunks to return")
|
||||
|
||||
toc = subparsers.add_parser("toc", help="directory -> file -> title + tags from the catalog")
|
||||
toc.add_argument("--source", help="limit to one source id")
|
||||
toc.add_argument("--tag", help="only files carrying this frontmatter tag")
|
||||
|
||||
grep = subparsers.add_parser("grep", help="live ripgrep over the files on disk, index-free")
|
||||
grep.add_argument("pattern", help="ripgrep regex")
|
||||
grep.add_argument("--source", help="limit to one source id")
|
||||
|
||||
return parser.parse_args(argv)
|
||||
|
||||
|
||||
def main(argv: list[str] | None = None) -> int:
|
||||
args = parse_args(argv)
|
||||
try:
|
||||
config = load_config()
|
||||
except ConfigError as exc:
|
||||
print(f"config error: {exc}", file=sys.stderr)
|
||||
return 1
|
||||
|
||||
if args.command == "search":
|
||||
return run_search(config, args.query, args.limit)
|
||||
if args.command == "toc":
|
||||
return run_toc(args.source, args.tag)
|
||||
return run_grep(config, args.pattern, args.source)
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
sys.exit(main())
|
||||
280
skills/wiki/scripts/wiki_store.py
Normal file
280
skills/wiki/scripts/wiki_store.py
Normal file
@@ -0,0 +1,280 @@
|
||||
#!/usr/bin/env python3
|
||||
# /// script
|
||||
# requires-python = ">=3.11"
|
||||
# dependencies = ["sqlite-vec==0.1.6"]
|
||||
# ///
|
||||
"""Data-access layer for the wiki skill.
|
||||
|
||||
Pure SQL plus lifecycle helpers. No printing, no argparse, no sys.exit.
|
||||
|
||||
`vec_chunks` is a vec0 virtual table and therefore NOT reachable by the foreign key
|
||||
cascade that cleans up `chunks` and `chunks_fts`. Every path that removes chunks must
|
||||
delete their vectors explicitly — that is why the deletes here go through
|
||||
`_delete_vectors_for_file`.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import json
|
||||
import sqlite3
|
||||
from collections.abc import Iterator, Sequence
|
||||
from contextlib import contextmanager
|
||||
from pathlib import Path
|
||||
|
||||
from sqlite_vec import serialize_float32 # ty: ignore[unresolved-import]
|
||||
from wiki_db import get_db, init_db
|
||||
|
||||
|
||||
@contextmanager
|
||||
def connection(db_path: Path) -> Iterator[sqlite3.Connection]:
|
||||
"""Open a connection, initialising the DB if missing."""
|
||||
if not db_path.exists():
|
||||
init_db(db_path)
|
||||
conn = get_db(db_path)
|
||||
try:
|
||||
yield conn
|
||||
finally:
|
||||
conn.close()
|
||||
|
||||
|
||||
@contextmanager
|
||||
def tx(conn: sqlite3.Connection) -> Iterator[sqlite3.Connection]:
|
||||
"""Wrap an already-open connection in an explicit transaction.
|
||||
|
||||
The connection is in autocommit mode (`isolation_level=None`), so transactions are
|
||||
ours to open — sqlite3's implicit handling would otherwise fail a nested BEGIN.
|
||||
"""
|
||||
conn.execute("BEGIN")
|
||||
try:
|
||||
yield conn
|
||||
conn.execute("COMMIT")
|
||||
except Exception:
|
||||
conn.execute("ROLLBACK")
|
||||
raise
|
||||
|
||||
|
||||
@contextmanager
|
||||
def transaction(db_path: Path) -> Iterator[sqlite3.Connection]:
|
||||
"""Open a connection wrapped in an explicit transaction."""
|
||||
with connection(db_path) as conn, tx(conn):
|
||||
yield conn
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# meta — identity of the embedding space
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
|
||||
def read_meta(conn: sqlite3.Connection) -> dict[str, str]:
|
||||
return {row["key"]: row["value"] for row in conn.execute("SELECT key, value FROM meta")}
|
||||
|
||||
|
||||
def write_meta(conn: sqlite3.Connection, values: dict[str, str]) -> None:
|
||||
conn.executemany(
|
||||
"INSERT INTO meta (key, value) VALUES (?, ?) ON CONFLICT(key) DO UPDATE SET value = excluded.value",
|
||||
sorted(values.items()),
|
||||
)
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# sources
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
|
||||
def upsert_source(conn: sqlite3.Connection, source_id: str, kind: str) -> None:
|
||||
conn.execute(
|
||||
"INSERT INTO sources (source_id, kind) VALUES (?, ?) ON CONFLICT(source_id) DO UPDATE SET kind = excluded.kind",
|
||||
(source_id, kind),
|
||||
)
|
||||
|
||||
|
||||
def get_source(conn: sqlite3.Connection, source_id: str) -> sqlite3.Row | None:
|
||||
return conn.execute("SELECT * FROM sources WHERE source_id = ?", (source_id,)).fetchone()
|
||||
|
||||
|
||||
def mark_synced(conn: sqlite3.Connection, source_id: str, indexed_rev: str | None, now: str) -> None:
|
||||
conn.execute(
|
||||
"UPDATE sources SET indexed_rev = ?, last_sync_at = ? WHERE source_id = ?",
|
||||
(indexed_rev, now, source_id),
|
||||
)
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# files
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
|
||||
def get_file(conn: sqlite3.Connection, source_id: str, path: str) -> sqlite3.Row | None:
|
||||
return conn.execute("SELECT * FROM files WHERE source_id = ? AND path = ?", (source_id, path)).fetchone()
|
||||
|
||||
|
||||
def list_source_files(conn: sqlite3.Connection, source_id: str) -> dict[str, sqlite3.Row]:
|
||||
"""Indexed files of one source, keyed by path — the basis for deletion detection."""
|
||||
rows = conn.execute("SELECT * FROM files WHERE source_id = ?", (source_id,))
|
||||
return {row["path"]: row for row in rows}
|
||||
|
||||
|
||||
def upsert_file(
|
||||
conn: sqlite3.Connection,
|
||||
source_id: str,
|
||||
path: str,
|
||||
title: str | None,
|
||||
tags: list[str],
|
||||
headings: list[str],
|
||||
sha256: str,
|
||||
size: int,
|
||||
mtime: float | None,
|
||||
now: str,
|
||||
) -> None:
|
||||
conn.execute(
|
||||
"INSERT INTO files (source_id, path, title, tags, headings, sha256, size, mtime, indexed_at) "
|
||||
"VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?) "
|
||||
"ON CONFLICT(source_id, path) DO UPDATE SET "
|
||||
"title = excluded.title, tags = excluded.tags, headings = excluded.headings, "
|
||||
"sha256 = excluded.sha256, size = excluded.size, mtime = excluded.mtime, "
|
||||
"indexed_at = excluded.indexed_at",
|
||||
(
|
||||
source_id,
|
||||
path,
|
||||
title,
|
||||
json.dumps(tags, ensure_ascii=False),
|
||||
json.dumps(headings, ensure_ascii=False),
|
||||
sha256,
|
||||
size,
|
||||
mtime,
|
||||
now,
|
||||
),
|
||||
)
|
||||
|
||||
|
||||
def touch_file_stat(conn: sqlite3.Connection, source_id: str, path: str, size: int, mtime: float | None) -> None:
|
||||
"""Refresh the cheap change-detection stats after a sha256 match (content unchanged)."""
|
||||
conn.execute(
|
||||
"UPDATE files SET size = ?, mtime = ? WHERE source_id = ? AND path = ?",
|
||||
(size, mtime, source_id, path),
|
||||
)
|
||||
|
||||
|
||||
def delete_file(conn: sqlite3.Connection, source_id: str, path: str) -> None:
|
||||
"""Drop a file and everything derived from it, vec0 rows included."""
|
||||
_delete_vectors_for_file(conn, source_id, path)
|
||||
conn.execute("DELETE FROM files WHERE source_id = ? AND path = ?", (source_id, path))
|
||||
|
||||
|
||||
def list_toc_files(conn: sqlite3.Connection, source_id: str | None = None, tag: str | None = None) -> list[sqlite3.Row]:
|
||||
sql = "SELECT source_id, path, title, tags FROM files"
|
||||
clauses: list[str] = []
|
||||
params: list[str] = []
|
||||
if source_id:
|
||||
clauses.append("source_id = ?")
|
||||
params.append(source_id)
|
||||
if tag:
|
||||
clauses.append("EXISTS (SELECT 1 FROM json_each(files.tags) WHERE value = ?)")
|
||||
params.append(tag)
|
||||
if clauses:
|
||||
sql += " WHERE " + " AND ".join(clauses)
|
||||
sql += " ORDER BY source_id, path"
|
||||
return list(conn.execute(sql, params))
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# chunks
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
|
||||
def _delete_vectors_for_file(conn: sqlite3.Connection, source_id: str, path: str) -> None:
|
||||
ids = [
|
||||
row["id"] for row in conn.execute("SELECT id FROM chunks WHERE source_id = ? AND path = ?", (source_id, path))
|
||||
]
|
||||
conn.executemany("DELETE FROM vec_chunks WHERE rowid = ?", [(i,) for i in ids])
|
||||
|
||||
|
||||
def replace_chunks(conn: sqlite3.Connection, source_id: str, path: str, chunks: Sequence[tuple[str, str]]) -> None:
|
||||
"""Swap a file's chunks for a freshly built set, as (breadcrumb, text) in order.
|
||||
|
||||
New chunks land with `embedded_at IS NULL`; the embed pass picks them up, so an
|
||||
unreachable Ollama degrades to FTS-only instead of failing the sync.
|
||||
"""
|
||||
_delete_vectors_for_file(conn, source_id, path)
|
||||
conn.execute("DELETE FROM chunks WHERE source_id = ? AND path = ?", (source_id, path))
|
||||
conn.executemany(
|
||||
"INSERT INTO chunks (source_id, path, chunk_idx, breadcrumb, text) VALUES (?, ?, ?, ?, ?)",
|
||||
[(source_id, path, idx, breadcrumb, text) for idx, (breadcrumb, text) in enumerate(chunks)],
|
||||
)
|
||||
|
||||
|
||||
def pending_chunks(conn: sqlite3.Connection, limit: int) -> list[sqlite3.Row]:
|
||||
return list(
|
||||
conn.execute(
|
||||
"SELECT id, breadcrumb, text FROM chunks WHERE embedded_at IS NULL ORDER BY id LIMIT ?",
|
||||
(limit,),
|
||||
)
|
||||
)
|
||||
|
||||
|
||||
def has_pending(conn: sqlite3.Connection) -> bool:
|
||||
return conn.execute("SELECT 1 FROM chunks WHERE embedded_at IS NULL LIMIT 1").fetchone() is not None
|
||||
|
||||
|
||||
def store_embedding(conn: sqlite3.Connection, chunk_id: int, vector: Sequence[float], now: str) -> None:
|
||||
conn.execute("DELETE FROM vec_chunks WHERE rowid = ?", (chunk_id,))
|
||||
conn.execute(
|
||||
"INSERT INTO vec_chunks (rowid, embedding) VALUES (?, ?)",
|
||||
(chunk_id, serialize_float32(list(vector))),
|
||||
)
|
||||
conn.execute("UPDATE chunks SET embedded_at = ? WHERE id = ?", (now, chunk_id))
|
||||
|
||||
|
||||
def reset_index(conn: sqlite3.Connection) -> None:
|
||||
"""Drop every indexed artifact, keeping the source rows. Used by `--full`."""
|
||||
ids = [row["id"] for row in conn.execute("SELECT id FROM chunks")]
|
||||
conn.executemany("DELETE FROM vec_chunks WHERE rowid = ?", [(i,) for i in ids])
|
||||
conn.execute("DELETE FROM files")
|
||||
conn.execute("UPDATE sources SET indexed_rev = NULL")
|
||||
|
||||
|
||||
def index_stats(conn: sqlite3.Connection) -> dict[str, int]:
|
||||
return {
|
||||
"files": conn.execute("SELECT count(*) AS n FROM files").fetchone()["n"],
|
||||
"chunks": conn.execute("SELECT count(*) AS n FROM chunks").fetchone()["n"],
|
||||
"vectors": conn.execute("SELECT count(*) AS n FROM vec_chunks").fetchone()["n"],
|
||||
"pending": conn.execute("SELECT count(*) AS n FROM chunks WHERE embedded_at IS NULL").fetchone()["n"],
|
||||
}
|
||||
|
||||
|
||||
def orphan_vector_ids(conn: sqlite3.Connection) -> list[int]:
|
||||
"""Vector rowids with no surviving chunk — must always be empty (regression guard)."""
|
||||
rows = conn.execute("SELECT rowid AS rid FROM vec_chunks WHERE rowid NOT IN (SELECT id FROM chunks)")
|
||||
return [row["rid"] for row in rows]
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# retrieval
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
|
||||
def bm25_ranked_ids(conn: sqlite3.Connection, match_expr: str, limit: int) -> list[int]:
|
||||
rows = conn.execute(
|
||||
"SELECT rowid AS rid FROM chunks_fts WHERE chunks_fts MATCH ? ORDER BY rank LIMIT ?",
|
||||
(match_expr, limit),
|
||||
)
|
||||
return [row["rid"] for row in rows]
|
||||
|
||||
|
||||
def knn_ranked_ids(conn: sqlite3.Connection, vector: Sequence[float], k: int) -> list[int]:
|
||||
rows = conn.execute(
|
||||
"SELECT rowid AS rid FROM vec_chunks WHERE embedding MATCH ? AND k = ? ORDER BY distance",
|
||||
(serialize_float32(list(vector)), k),
|
||||
)
|
||||
return [row["rid"] for row in rows]
|
||||
|
||||
|
||||
def fetch_chunks(conn: sqlite3.Connection, ids: Sequence[int]) -> dict[int, sqlite3.Row]:
|
||||
if not ids:
|
||||
return {}
|
||||
placeholders = ",".join("?" * len(ids))
|
||||
rows = conn.execute(
|
||||
f"SELECT id, source_id, path, chunk_idx, breadcrumb, text FROM chunks WHERE id IN ({placeholders})",
|
||||
list(ids),
|
||||
)
|
||||
return {row["id"]: row for row in rows}
|
||||
529
skills/wiki/scripts/wiki_sync.py
Normal file
529
skills/wiki/scripts/wiki_sync.py
Normal file
@@ -0,0 +1,529 @@
|
||||
#!/usr/bin/env -S uv run --script
|
||||
# /// script
|
||||
# requires-python = ">=3.11"
|
||||
# dependencies = ["requests", "pyyaml", "markdown-it-py", "sqlite-vec==0.1.6"]
|
||||
# ///
|
||||
"""Offline indexer for the wiki skill. Runs every minute from the nanobot crontab.
|
||||
|
||||
Indexing never happens inside an agent turn: the `exec` tool times out at 60 s while a
|
||||
full index of 10^4 chunks takes ~93 s. The agent only ever reads a finished index.
|
||||
|
||||
Per tick:
|
||||
|
||||
1. take `wiki/.sync.lock`; a live holder means exit 0 in silence
|
||||
2. detect changes cheaply — `git ls-remote` (no fetch) for git sources, a
|
||||
path+size+mtime walk for workspace sources, sha256 only on a stat mismatch
|
||||
3. nothing changed and nothing pending -> exit 0 (the overwhelming majority of ticks)
|
||||
4. re-chunk changed files, then drain every chunk with `embedded_at IS NULL` — which
|
||||
is also the way back out of degraded mode after Ollama returns
|
||||
5. record indexed_rev / last_sync_at
|
||||
6. log a coverage line naming top-level directories no source covers
|
||||
7. append the run summary to log/wiki_sync.log
|
||||
|
||||
Network failure is per-source: a timeout or non-zero git exit logs WARN, skips that
|
||||
source with its `indexed_rev` untouched, and lets the others finish. Without the
|
||||
timeouts a hanging `ls-remote` would hold the lock and block workspace sources that
|
||||
have nothing to do with the network.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import argparse
|
||||
import hashlib
|
||||
import json
|
||||
import os
|
||||
import subprocess
|
||||
import sys
|
||||
from dataclasses import dataclass, field
|
||||
from datetime import UTC, datetime
|
||||
from pathlib import Path
|
||||
|
||||
import wiki_store as store
|
||||
from wiki_chunker import parse_markdown
|
||||
from wiki_config import (
|
||||
GIT_KIND,
|
||||
LOCK_PATH,
|
||||
SYNC_LOG_PATH,
|
||||
WIKI_DIR,
|
||||
ConfigError,
|
||||
SourceConfig,
|
||||
WikiConfig,
|
||||
db_path,
|
||||
load_config,
|
||||
source_root,
|
||||
)
|
||||
from wiki_embed import (
|
||||
EmbeddingUnavailable,
|
||||
OllamaEmbedder,
|
||||
expected_meta,
|
||||
meta_mismatches,
|
||||
)
|
||||
|
||||
STALE_SECONDS = 30 * 60
|
||||
LS_REMOTE_TIMEOUT = 20
|
||||
GIT_TIMEOUT = 300
|
||||
GIT_ERROR_CHARS = 300
|
||||
SKIP_DIRS = frozenset({".git"})
|
||||
|
||||
|
||||
class GitError(RuntimeError):
|
||||
"""A git subprocess failed or timed out."""
|
||||
|
||||
|
||||
@dataclass
|
||||
class SourcePlan:
|
||||
"""What one source needs done this tick."""
|
||||
|
||||
changed: list[str] = field(default_factory=list)
|
||||
deleted: list[str] = field(default_factory=list)
|
||||
stat_refresh: list[tuple[str, int, float]] = field(default_factory=list)
|
||||
new_rev: str | None = None
|
||||
|
||||
@property
|
||||
def has_work(self) -> bool:
|
||||
return bool(self.changed or self.deleted or self.stat_refresh)
|
||||
|
||||
|
||||
def log(message: str) -> None:
|
||||
SYNC_LOG_PATH.parent.mkdir(parents=True, exist_ok=True)
|
||||
stamp = datetime.now().astimezone().isoformat(timespec="seconds")
|
||||
with SYNC_LOG_PATH.open("a", encoding="utf-8") as handle:
|
||||
handle.write(f"{stamp} {message}\n")
|
||||
|
||||
|
||||
def _now() -> str:
|
||||
return datetime.now(UTC).isoformat(timespec="seconds")
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# lock
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
|
||||
def _pid_alive(pid: int) -> bool:
|
||||
try:
|
||||
os.kill(pid, 0)
|
||||
except ProcessLookupError:
|
||||
return False
|
||||
except PermissionError:
|
||||
return True
|
||||
return True
|
||||
|
||||
|
||||
def _lock_is_stale() -> bool:
|
||||
"""A lock is dead if unreadable, its pid is gone, or it is older than STALE_SECONDS."""
|
||||
try:
|
||||
data = json.loads(LOCK_PATH.read_text(encoding="utf-8"))
|
||||
pid = int(data["pid"])
|
||||
started = datetime.fromisoformat(data["started_at"])
|
||||
except (OSError, ValueError, KeyError):
|
||||
return True
|
||||
if not _pid_alive(pid):
|
||||
return True
|
||||
return (datetime.now().astimezone() - started).total_seconds() > STALE_SECONDS
|
||||
|
||||
|
||||
def acquire_lock() -> bool:
|
||||
"""Atomically create the lock. False when a live sync already runs.
|
||||
|
||||
Reclaiming a stale lock is not optional: without it a crashed run (OOM, reboot)
|
||||
would make every later cron tick exit 0 in silence, forever.
|
||||
"""
|
||||
LOCK_PATH.parent.mkdir(parents=True, exist_ok=True)
|
||||
for _ in range(2):
|
||||
try:
|
||||
handle = os.open(LOCK_PATH, os.O_CREAT | os.O_EXCL | os.O_WRONLY, 0o644)
|
||||
except FileExistsError:
|
||||
if not _lock_is_stale():
|
||||
return False
|
||||
log("WARN stale lock, reclaiming")
|
||||
LOCK_PATH.unlink(missing_ok=True)
|
||||
continue
|
||||
payload = {"pid": os.getpid(), "started_at": datetime.now().astimezone().isoformat()}
|
||||
with os.fdopen(handle, "w", encoding="utf-8") as file:
|
||||
json.dump(payload, file)
|
||||
return True
|
||||
return False
|
||||
|
||||
|
||||
def release_lock() -> None:
|
||||
LOCK_PATH.unlink(missing_ok=True)
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# filesystem
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
|
||||
def _sha256(path: Path) -> str:
|
||||
digest = hashlib.sha256()
|
||||
with path.open("rb") as handle:
|
||||
for block in iter(lambda: handle.read(65536), b""):
|
||||
digest.update(block)
|
||||
return digest.hexdigest()
|
||||
|
||||
|
||||
def walk_source(source: SourceConfig, root: Path) -> dict[str, tuple[int, float]]:
|
||||
"""Covered files under `root`, as rel_path -> (size, mtime). Prunes uncovered dirs."""
|
||||
found: dict[str, tuple[int, float]] = {}
|
||||
if not root.is_dir():
|
||||
return found
|
||||
for dirpath, dirnames, filenames in os.walk(root):
|
||||
rel_dir = os.path.relpath(dirpath, root)
|
||||
rel_dir = "" if rel_dir == "." else rel_dir
|
||||
dirnames[:] = [
|
||||
name for name in dirnames if name not in SKIP_DIRS and source.covers_dir(f"{rel_dir}/{name}".lstrip("/"))
|
||||
]
|
||||
for name in filenames:
|
||||
rel_path = f"{rel_dir}/{name}".lstrip("/")
|
||||
if not source.covers(rel_path):
|
||||
continue
|
||||
file_path = Path(dirpath) / name
|
||||
if not file_path.is_file():
|
||||
continue
|
||||
stat = file_path.stat()
|
||||
found[rel_path] = (stat.st_size, stat.st_mtime)
|
||||
return found
|
||||
|
||||
|
||||
def uncovered_top_level_dirs(source: SourceConfig, root: Path) -> list[str]:
|
||||
"""Top-level directories holding markdown that no source path reaches.
|
||||
|
||||
This is the safety net under the whitelist: a directory that silently fails to be
|
||||
indexed shows up here instead of nowhere. `wiki/` is skipped because it holds this
|
||||
skill's own clones and index — never a candidate, so reporting it is pure noise.
|
||||
"""
|
||||
if not root.is_dir():
|
||||
return []
|
||||
uncovered = []
|
||||
for entry in sorted(root.iterdir()):
|
||||
if not entry.is_dir() or entry.name in SKIP_DIRS or entry == WIKI_DIR:
|
||||
continue
|
||||
if source.covers_dir(entry.name):
|
||||
continue
|
||||
if next(entry.rglob("*.md"), None) is not None:
|
||||
uncovered.append(entry.name)
|
||||
return uncovered
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# git driver
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
|
||||
def _git(args: list[str], cwd: Path | None = None, timeout: int = GIT_TIMEOUT) -> str:
|
||||
try:
|
||||
result = subprocess.run(
|
||||
["git", *args],
|
||||
cwd=cwd,
|
||||
capture_output=True,
|
||||
text=True,
|
||||
timeout=timeout,
|
||||
check=False,
|
||||
)
|
||||
except subprocess.TimeoutExpired as exc:
|
||||
raise GitError(f"git {' '.join(args)} timed out after {timeout}s") from exc
|
||||
if result.returncode != 0:
|
||||
raise GitError(f"git {' '.join(args)} failed: {_one_line(result.stderr)}")
|
||||
return result.stdout
|
||||
|
||||
|
||||
def _one_line(text: str) -> str:
|
||||
"""Squash git's multi-line stderr into one capped line.
|
||||
|
||||
A permanently unreachable source warns on every tick, so a six-line stderr would
|
||||
put thousands of lines a day into the log and drown everything else.
|
||||
"""
|
||||
collapsed = " ".join(text.split())
|
||||
return collapsed[:GIT_ERROR_CHARS] + ("…" if len(collapsed) > GIT_ERROR_CHARS else "")
|
||||
|
||||
|
||||
def remote_head(url: str) -> str:
|
||||
"""Remote HEAD without fetching — the right tool for a per-minute cadence."""
|
||||
output = _git(["ls-remote", "--symref", url, "HEAD"], timeout=LS_REMOTE_TIMEOUT)
|
||||
for line in output.splitlines():
|
||||
parts = line.split()
|
||||
if len(parts) == 2 and parts[1] == "HEAD":
|
||||
return parts[0]
|
||||
raise GitError(f"no HEAD in ls-remote output for {url}")
|
||||
|
||||
|
||||
def _diff_names(root: Path, base_rev: str) -> tuple[list[str], list[str]] | None:
|
||||
"""(changed, deleted) between base_rev and FETCH_HEAD, or None if base_rev is gone."""
|
||||
try:
|
||||
output = _git(["diff", "--name-status", f"{base_rev}..FETCH_HEAD"], cwd=root)
|
||||
except GitError:
|
||||
return None
|
||||
changed, deleted = [], []
|
||||
for line in output.splitlines():
|
||||
fields = line.split("\t")
|
||||
if len(fields) < 2:
|
||||
continue
|
||||
status = fields[0]
|
||||
if status.startswith("R") and len(fields) >= 3:
|
||||
deleted.append(fields[1])
|
||||
changed.append(fields[2])
|
||||
elif status.startswith("D"):
|
||||
deleted.append(fields[1])
|
||||
else:
|
||||
changed.append(fields[1])
|
||||
return changed, deleted
|
||||
|
||||
|
||||
def plan_git_source(conn, source: SourceConfig, root: Path, full: bool) -> SourcePlan:
|
||||
rev = remote_head(str(source.url))
|
||||
row = store.get_source(conn, source.source_id)
|
||||
indexed_rev = row["indexed_rev"] if row else None
|
||||
|
||||
if not root.exists():
|
||||
_git(["clone", str(source.url), str(root)])
|
||||
indexed_rev = None
|
||||
if not full and indexed_rev == rev:
|
||||
return SourcePlan(new_rev=rev)
|
||||
|
||||
_git(["fetch", "--prune"], cwd=root)
|
||||
diff = None if (full or not indexed_rev) else _diff_names(root, indexed_rev)
|
||||
_git(["reset", "--hard", "FETCH_HEAD"], cwd=root)
|
||||
|
||||
indexed = store.list_source_files(conn, source.source_id)
|
||||
on_disk = walk_source(source, root)
|
||||
if diff is None:
|
||||
# Fresh clone, --full, or an indexed_rev the repo no longer has: reindex it all.
|
||||
return SourcePlan(
|
||||
changed=sorted(on_disk),
|
||||
deleted=sorted(path for path in indexed if path not in on_disk),
|
||||
new_rev=rev,
|
||||
)
|
||||
raw_changed, raw_deleted = diff
|
||||
changed = sorted({path for path in raw_changed if path in on_disk})
|
||||
deleted = sorted(
|
||||
{path for path in raw_deleted if path in indexed} | {path for path in indexed if path not in on_disk}
|
||||
)
|
||||
return SourcePlan(changed=changed, deleted=[p for p in deleted if p not in changed], new_rev=rev)
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# workspace driver
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
|
||||
def plan_workspace_source(conn, source: SourceConfig, root: Path, full: bool) -> SourcePlan:
|
||||
on_disk = walk_source(source, root)
|
||||
indexed = store.list_source_files(conn, source.source_id)
|
||||
plan = SourcePlan()
|
||||
for rel_path, (size, mtime) in sorted(on_disk.items()):
|
||||
row = indexed.get(rel_path)
|
||||
if full or row is None:
|
||||
plan.changed.append(rel_path)
|
||||
continue
|
||||
if row["size"] == size and row["mtime"] == mtime:
|
||||
continue
|
||||
if _sha256(root / rel_path) == row["sha256"]:
|
||||
# Content is identical; refresh the stats so the next tick stays cheap.
|
||||
plan.stat_refresh.append((rel_path, size, mtime))
|
||||
else:
|
||||
plan.changed.append(rel_path)
|
||||
plan.deleted = sorted(path for path in indexed if path not in on_disk)
|
||||
return plan
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# indexing
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
|
||||
def index_file(conn, source_id: str, root: Path, rel_path: str, now: str) -> int:
|
||||
"""Re-chunk one file. Returns the chunk count, or -1 when it could not be read."""
|
||||
file_path = root / rel_path
|
||||
try:
|
||||
text = file_path.read_text(encoding="utf-8")
|
||||
stat = file_path.stat()
|
||||
except (OSError, UnicodeDecodeError):
|
||||
return -1
|
||||
parsed = parse_markdown(text, rel_path)
|
||||
store.upsert_file(
|
||||
conn,
|
||||
source_id,
|
||||
rel_path,
|
||||
parsed.title,
|
||||
parsed.tags,
|
||||
parsed.headings,
|
||||
_sha256(file_path),
|
||||
stat.st_size,
|
||||
stat.st_mtime,
|
||||
now,
|
||||
)
|
||||
store.replace_chunks(conn, source_id, rel_path, [(c.breadcrumb, c.text) for c in parsed.chunks])
|
||||
return len(parsed.chunks)
|
||||
|
||||
|
||||
def apply_plan(conn, source: SourceConfig, root: Path, plan: SourcePlan) -> dict[str, int]:
|
||||
counts = {"indexed": 0, "deleted": 0, "unreadable": 0, "chunks": 0}
|
||||
now = _now()
|
||||
for rel_path in plan.deleted:
|
||||
with store.tx(conn):
|
||||
store.delete_file(conn, source.source_id, rel_path)
|
||||
counts["deleted"] += 1
|
||||
for rel_path in plan.changed:
|
||||
with store.tx(conn):
|
||||
chunks = index_file(conn, source.source_id, root, rel_path, now)
|
||||
if chunks < 0:
|
||||
counts["unreadable"] += 1
|
||||
log(f"WARN {source.source_id}: unreadable {rel_path}")
|
||||
continue
|
||||
counts["indexed"] += 1
|
||||
counts["chunks"] += chunks
|
||||
if plan.stat_refresh:
|
||||
with store.tx(conn):
|
||||
for rel_path, size, mtime in plan.stat_refresh:
|
||||
store.touch_file_stat(conn, source.source_id, rel_path, size, mtime)
|
||||
with store.tx(conn):
|
||||
store.mark_synced(conn, source.source_id, plan.new_rev, now)
|
||||
return counts
|
||||
|
||||
|
||||
def drain_pending(conn, config: WikiConfig) -> tuple[int, str | None]:
|
||||
"""Embed every chunk still missing a vector. Returns (embedded, warning)."""
|
||||
if not store.has_pending(conn):
|
||||
return 0, None
|
||||
embedder = OllamaEmbedder(config.embedding)
|
||||
embedded = 0
|
||||
while True:
|
||||
batch = store.pending_chunks(conn, config.embedding.batch)
|
||||
if not batch:
|
||||
return embedded, None
|
||||
try:
|
||||
vectors = embedder.embed_documents([row["text"] for row in batch])
|
||||
except EmbeddingUnavailable as exc:
|
||||
return embedded, f"embeddings unavailable, staying FTS-only: {exc}"
|
||||
now = _now()
|
||||
with store.tx(conn):
|
||||
for row, vector in zip(batch, vectors, strict=True):
|
||||
store.store_embedding(conn, row["id"], vector, now)
|
||||
embedded += len(batch)
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# run
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
|
||||
def ensure_meta(conn, config: WikiConfig, full: bool) -> str | None:
|
||||
"""Write the index identity, or report a mismatch that only --full can resolve."""
|
||||
stored = store.read_meta(conn)
|
||||
mismatches = meta_mismatches(stored, config.embedding)
|
||||
if mismatches and not full:
|
||||
return f"index identity mismatch on {', '.join(mismatches)} — run wiki_sync.py --full"
|
||||
with store.tx(conn):
|
||||
store.write_meta(conn, expected_meta(config.embedding))
|
||||
return None
|
||||
|
||||
|
||||
def run(config: WikiConfig, args: argparse.Namespace) -> int:
|
||||
sources = [s for s in config.sources if not args.source or s.source_id == args.source]
|
||||
if not sources:
|
||||
log(f"WARN unknown source {args.source!r}")
|
||||
return 1
|
||||
|
||||
with store.connection(db_path()) as conn:
|
||||
problem = ensure_meta(conn, config, args.full)
|
||||
if problem:
|
||||
log(f"WARN {problem}")
|
||||
return 1
|
||||
|
||||
for source in sources:
|
||||
with store.tx(conn):
|
||||
store.upsert_source(conn, source.source_id, source.kind)
|
||||
|
||||
warnings: list[str] = []
|
||||
did_work = args.full
|
||||
for source in sources:
|
||||
root = source_root(source)
|
||||
try:
|
||||
if source.kind == GIT_KIND:
|
||||
plan = plan_git_source(conn, source, root, args.full)
|
||||
else:
|
||||
plan = plan_workspace_source(conn, source, root, args.full)
|
||||
except GitError as exc:
|
||||
# indexed_rev stays untouched, so the next tick retries this source.
|
||||
warnings.append(f"{source.source_id}: {exc}")
|
||||
log(f"WARN {source.source_id}: {exc}")
|
||||
continue
|
||||
|
||||
row = store.get_source(conn, source.source_id)
|
||||
indexed_rev = row["indexed_rev"] if row else None
|
||||
rev_moved = plan.new_rev is not None and plan.new_rev != indexed_rev
|
||||
if not plan.has_work and not rev_moved:
|
||||
continue
|
||||
|
||||
did_work = True
|
||||
if not plan.has_work:
|
||||
# Upstream moved but touched nothing we index — record the rev so the
|
||||
# next tick stops fetching.
|
||||
with store.tx(conn):
|
||||
store.mark_synced(conn, source.source_id, plan.new_rev, _now())
|
||||
log(f"{source.source_id}: rev moved to {plan.new_rev}, no indexed file changed")
|
||||
continue
|
||||
|
||||
counts = apply_plan(conn, source, root, plan)
|
||||
log(
|
||||
f"{source.source_id}: indexed {counts['indexed']} files "
|
||||
f"({counts['chunks']} chunks), deleted {counts['deleted']}"
|
||||
)
|
||||
|
||||
embedded, embed_warning = drain_pending(conn, config)
|
||||
if embed_warning:
|
||||
warnings.append(embed_warning)
|
||||
log(f"WARN {embed_warning}")
|
||||
|
||||
if not did_work and not embedded and not warnings:
|
||||
return 0
|
||||
|
||||
for source in sources:
|
||||
uncovered = uncovered_top_level_dirs(source, source_root(source))
|
||||
if uncovered:
|
||||
log(f"coverage {source.source_id}: markdown outside paths in {', '.join(uncovered)}")
|
||||
|
||||
stats = store.index_stats(conn)
|
||||
log(
|
||||
f"done files={stats['files']} chunks={stats['chunks']} vectors={stats['vectors']} "
|
||||
f"pending={stats['pending']} embedded={embedded}"
|
||||
)
|
||||
return 1 if warnings else 0
|
||||
|
||||
|
||||
def parse_args(argv: list[str] | None = None) -> argparse.Namespace:
|
||||
parser = argparse.ArgumentParser(description="Index the wiki sources into wiki/index.sqlite.")
|
||||
parser.add_argument(
|
||||
"--full",
|
||||
action="store_true",
|
||||
help="wipe and rebuild the index (needed after a model or chunker change)",
|
||||
)
|
||||
parser.add_argument("--source", help="limit the run to one source id")
|
||||
return parser.parse_args(argv)
|
||||
|
||||
|
||||
def main(argv: list[str] | None = None) -> int:
|
||||
args = parse_args(argv)
|
||||
try:
|
||||
config = load_config()
|
||||
except ConfigError as exc:
|
||||
log(f"WARN {exc}")
|
||||
print(f"config error: {exc}", file=sys.stderr)
|
||||
return 1
|
||||
|
||||
WIKI_DIR.mkdir(parents=True, exist_ok=True)
|
||||
if not acquire_lock():
|
||||
return 0
|
||||
try:
|
||||
if args.full:
|
||||
log("full reindex requested")
|
||||
with store.transaction(db_path()) as conn:
|
||||
store.reset_index(conn)
|
||||
return run(config, args)
|
||||
finally:
|
||||
release_lock()
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
sys.exit(main())
|
||||
141
skills/wiki/tests/conftest.py
Normal file
141
skills/wiki/tests/conftest.py
Normal file
@@ -0,0 +1,141 @@
|
||||
import hashlib
|
||||
import math
|
||||
import sys
|
||||
from dataclasses import dataclass
|
||||
from pathlib import Path
|
||||
|
||||
import pytest
|
||||
|
||||
# The modules under test live in the sibling scripts/ directory.
|
||||
sys.path.insert(0, str(Path(__file__).parent.parent / "scripts"))
|
||||
|
||||
import wiki_config
|
||||
import wiki_search
|
||||
import wiki_sync
|
||||
from wiki_db import EMBEDDING_DIMS
|
||||
|
||||
CONFIG_TEMPLATE = """
|
||||
embedding:
|
||||
endpoint: http://embed.invalid:11434
|
||||
model: qwen3-embedding:0.6b
|
||||
dims: 1024
|
||||
batch: 4
|
||||
keep_alive: -1
|
||||
query_prefix: "Instruct: task\\nQuery: "
|
||||
|
||||
sources:
|
||||
{sources}
|
||||
"""
|
||||
|
||||
WORKSPACE_SOURCE = """ workspace:
|
||||
kind: workspace
|
||||
paths:
|
||||
- "notes/**"
|
||||
- "develop/**"
|
||||
include: ["*.md"]
|
||||
exclude:
|
||||
- "**/inbox/**"
|
||||
- "develop/history.md"
|
||||
"""
|
||||
|
||||
|
||||
@dataclass
|
||||
class WikiEnv:
|
||||
workspace: Path
|
||||
wiki_dir: Path
|
||||
config_path: Path
|
||||
db_path: Path
|
||||
log_path: Path
|
||||
|
||||
def write_config(self, sources: str = WORKSPACE_SOURCE) -> None:
|
||||
self.config_path.write_text(CONFIG_TEMPLATE.format(sources=sources), encoding="utf-8")
|
||||
|
||||
def write_file(self, rel_path: str, text: str) -> Path:
|
||||
path = self.workspace / rel_path
|
||||
path.parent.mkdir(parents=True, exist_ok=True)
|
||||
path.write_text(text, encoding="utf-8")
|
||||
return path
|
||||
|
||||
def log_text(self) -> str:
|
||||
return self.log_path.read_text(encoding="utf-8") if self.log_path.exists() else ""
|
||||
|
||||
|
||||
@pytest.fixture
|
||||
def wiki_env(tmp_path, monkeypatch) -> WikiEnv:
|
||||
"""Redirect every wiki path at a throwaway workspace."""
|
||||
workspace = tmp_path / "workspace"
|
||||
wiki_dir = workspace / "wiki"
|
||||
wiki_dir.mkdir(parents=True)
|
||||
|
||||
env = WikiEnv(
|
||||
workspace=workspace,
|
||||
wiki_dir=wiki_dir,
|
||||
config_path=wiki_dir / "config.yaml",
|
||||
db_path=wiki_dir / "index.sqlite",
|
||||
log_path=workspace / "log" / "wiki_sync.log",
|
||||
)
|
||||
|
||||
monkeypatch.setenv("WIKI_DB", str(env.db_path))
|
||||
monkeypatch.setenv("WIKI_CONFIG", str(env.config_path))
|
||||
monkeypatch.setattr(wiki_config, "WORKSPACE", workspace)
|
||||
monkeypatch.setattr(wiki_config, "WIKI_DIR", wiki_dir)
|
||||
monkeypatch.setattr(wiki_config, "REMOTE_DIR", wiki_dir / "remote")
|
||||
monkeypatch.setattr(wiki_config, "LOCK_PATH", wiki_dir / ".sync.lock")
|
||||
monkeypatch.setattr(wiki_sync, "WIKI_DIR", wiki_dir)
|
||||
monkeypatch.setattr(wiki_sync, "LOCK_PATH", wiki_dir / ".sync.lock")
|
||||
monkeypatch.setattr(wiki_sync, "SYNC_LOG_PATH", env.log_path)
|
||||
|
||||
env.write_config()
|
||||
return env
|
||||
|
||||
|
||||
class FakeEmbedder:
|
||||
"""Bag-of-words vectors: cosine tracks lexical overlap, so ranks are predictable.
|
||||
|
||||
Enough to exercise the KNN and RRF plumbing without a live model. Semantic quality
|
||||
is measured against the real endpoint, not here.
|
||||
"""
|
||||
|
||||
def __init__(self, config):
|
||||
self.config = config
|
||||
self.calls: list[list[str]] = []
|
||||
|
||||
def embed_documents(self, texts):
|
||||
self.calls.append(list(texts))
|
||||
return [self._vector(text) for text in texts]
|
||||
|
||||
def embed_query(self, text):
|
||||
return self._vector(self.config.query_prefix + text)
|
||||
|
||||
def probe(self):
|
||||
return None
|
||||
|
||||
def _vector(self, text: str) -> list[float]:
|
||||
vector = [0.0] * EMBEDDING_DIMS
|
||||
for word in _words(text):
|
||||
digest = hashlib.sha256(word.encode("utf-8")).digest()
|
||||
vector[int.from_bytes(digest[:4], "big") % EMBEDDING_DIMS] += 1.0
|
||||
norm = math.sqrt(sum(value * value for value in vector))
|
||||
if norm == 0.0:
|
||||
vector[0] = 1.0
|
||||
return vector
|
||||
return [value / norm for value in vector]
|
||||
|
||||
|
||||
def _words(text: str) -> list[str]:
|
||||
return [word for word in "".join(c.lower() if c.isalnum() else " " for c in text).split() if len(word) > 2]
|
||||
|
||||
|
||||
@pytest.fixture
|
||||
def fake_embedder(monkeypatch):
|
||||
"""Swap the Ollama client for the deterministic fake in both entry points."""
|
||||
created: list[FakeEmbedder] = []
|
||||
|
||||
def factory(config):
|
||||
embedder = FakeEmbedder(config)
|
||||
created.append(embedder)
|
||||
return embedder
|
||||
|
||||
monkeypatch.setattr(wiki_sync, "OllamaEmbedder", factory)
|
||||
monkeypatch.setattr(wiki_search, "OllamaEmbedder", factory)
|
||||
return created
|
||||
209
skills/wiki/tests/test_wiki_chunker.py
Normal file
209
skills/wiki/tests/test_wiki_chunker.py
Normal file
@@ -0,0 +1,209 @@
|
||||
"""Chunker policy: heading sections, breadcrumbs, sibling merge, block-aligned split."""
|
||||
|
||||
import sys
|
||||
from pathlib import Path
|
||||
|
||||
SCRIPTS = Path(__file__).parent.parent / "scripts"
|
||||
sys.path.insert(0, str(SCRIPTS))
|
||||
|
||||
from wiki_chunker import MERGE_BELOW, OVERLAP, SPLIT_ABOVE, parse_markdown, token_estimate
|
||||
|
||||
FILLER = "Tohle je odstavec s dostatkem textu na to, aby se do velikosti chunku počítal. "
|
||||
|
||||
|
||||
def _paragraph(tokens: int) -> str:
|
||||
return (FILLER * (1 + tokens * 4 // len(FILLER)))[: tokens * 4]
|
||||
|
||||
|
||||
def _crumbs(parsed):
|
||||
return [chunk.breadcrumb for chunk in parsed.chunks]
|
||||
|
||||
|
||||
def test_heading_hierarchy_builds_breadcrumbs():
|
||||
doc = "\n".join(
|
||||
[
|
||||
"# Kořen",
|
||||
"",
|
||||
_paragraph(MERGE_BELOW),
|
||||
"",
|
||||
"## Sekce A",
|
||||
"",
|
||||
_paragraph(MERGE_BELOW),
|
||||
"",
|
||||
"### Podsekce",
|
||||
"",
|
||||
_paragraph(MERGE_BELOW),
|
||||
"",
|
||||
"## Sekce B",
|
||||
"",
|
||||
_paragraph(MERGE_BELOW),
|
||||
]
|
||||
)
|
||||
parsed = parse_markdown(doc, "notes/doc.md")
|
||||
assert _crumbs(parsed) == [
|
||||
"notes/doc.md > Kořen",
|
||||
"notes/doc.md > Kořen > Sekce A",
|
||||
"notes/doc.md > Kořen > Sekce A > Podsekce",
|
||||
"notes/doc.md > Kořen > Sekce B",
|
||||
]
|
||||
assert parsed.headings == ["Kořen", "Sekce A", "Podsekce", "Sekce B"]
|
||||
|
||||
|
||||
def test_frontmatter_title_and_tags():
|
||||
doc = "---\ntitle: Tokyo metro tips\ntags: [transit, jr-pass]\n---\n\n" + _paragraph(MERGE_BELOW)
|
||||
parsed = parse_markdown(doc, "japan/metro.md")
|
||||
assert parsed.title == "Tokyo metro tips"
|
||||
assert parsed.tags == ["transit", "jr-pass"]
|
||||
# Only the title reaches the embedded text; tags stay metadata for filtering.
|
||||
assert _crumbs(parsed) == ["japan/metro.md > Tokyo metro tips"]
|
||||
assert "transit" not in parsed.chunks[0].text
|
||||
|
||||
|
||||
def test_frontmatter_title_matching_h1_is_not_duplicated():
|
||||
doc = "---\ntitle: Zálohy\n---\n\n# Zálohy\n\n" + _paragraph(MERGE_BELOW)
|
||||
parsed = parse_markdown(doc, "a.md")
|
||||
assert _crumbs(parsed) == ["a.md > Zálohy"]
|
||||
|
||||
|
||||
def test_comma_separated_tags():
|
||||
doc = "---\ntags: gear, safety\n---\n\ntext"
|
||||
assert parse_markdown(doc, "a.md").tags == ["gear", "safety"]
|
||||
|
||||
|
||||
def test_malformed_frontmatter_stays_body():
|
||||
"""Unparseable frontmatter is not consumed — it stays body text, tags included."""
|
||||
doc = "---\ntitle: [unclosed\ntags: [a\n---\n\ntext"
|
||||
parsed = parse_markdown(doc, "a.md")
|
||||
assert parsed.tags == []
|
||||
assert "title: [unclosed" in parsed.chunks[-1].text # the raw block survived as prose
|
||||
|
||||
|
||||
def test_small_adjacent_siblings_merge():
|
||||
doc = "\n".join(["# Root", "", "## A", "", "krátké A", "", "## B", "", "krátké B"])
|
||||
parsed = parse_markdown(doc, "a.md")
|
||||
# A and B share the parent `a.md > Root` and are both under MERGE_BELOW.
|
||||
# `# Root` carries no prose of its own, so it contributes no chunk.
|
||||
assert _crumbs(parsed) == ["a.md > Root > A"]
|
||||
assert "krátké A" in parsed.chunks[0].text
|
||||
assert "krátké B" in parsed.chunks[0].text
|
||||
|
||||
|
||||
def test_heading_without_own_prose_yields_no_chunk():
|
||||
"""`# Titul` immediately followed by `## Sekce` must not embed a bare title."""
|
||||
doc = "\n".join(["# Titul", "", "## Sekce", "", _paragraph(MERGE_BELOW)])
|
||||
parsed = parse_markdown(doc, "a.md")
|
||||
assert _crumbs(parsed) == ["a.md > Titul > Sekce"]
|
||||
# The dropped heading still reaches the index through the breadcrumb and headings list.
|
||||
assert parsed.headings == ["Titul", "Sekce"]
|
||||
|
||||
|
||||
def test_merge_does_not_cross_parents():
|
||||
"""A small H3 must not be glued onto the next H2 — different parents."""
|
||||
doc = "\n".join(["# Root", "", "## A", "", "### A1", "", "malé", "", "## B", "", "malé B"])
|
||||
parsed = parse_markdown(doc, "a.md")
|
||||
assert "a.md > Root > A > A1" in _crumbs(parsed)
|
||||
a1 = next(c for c in parsed.chunks if c.breadcrumb.endswith("A1"))
|
||||
assert "malé B" not in a1.text
|
||||
|
||||
|
||||
def test_merge_never_exceeds_split_threshold():
|
||||
big = _paragraph(SPLIT_ABOVE - 100)
|
||||
doc = "\n".join(["# Root", "", "## A", "", "malé", "", "## B", "", big])
|
||||
parsed = parse_markdown(doc, "a.md")
|
||||
for chunk in parsed.chunks:
|
||||
assert token_estimate(chunk.text) <= SPLIT_ABOVE + token_estimate(chunk.breadcrumb) + 1
|
||||
|
||||
|
||||
def test_h4_stays_inside_its_h3_section():
|
||||
doc = "\n".join(["# Root", "", "### Trojka", "", "text", "", "#### Čtyřka", "", "hluboký text"])
|
||||
parsed = parse_markdown(doc, "a.md")
|
||||
assert "a.md > Root > Trojka" in _crumbs(parsed)
|
||||
assert not any("Čtyřka" in crumb for crumb in _crumbs(parsed))
|
||||
section = next(c for c in parsed.chunks if c.breadcrumb.endswith("Trojka"))
|
||||
assert "hluboký text" in section.text
|
||||
|
||||
|
||||
def test_preamble_before_first_heading_is_kept():
|
||||
doc = "úvodní odstavec bez nadpisu\n\n" + "# Root\n\n" + _paragraph(MERGE_BELOW)
|
||||
parsed = parse_markdown(doc, "a.md")
|
||||
assert "úvodní odstavec bez nadpisu" in parsed.chunks[0].text
|
||||
|
||||
|
||||
def test_document_without_headings_is_one_chunk():
|
||||
parsed = parse_markdown(_paragraph(MERGE_BELOW), "a.md")
|
||||
assert _crumbs(parsed) == ["a.md"]
|
||||
|
||||
|
||||
def test_empty_document_yields_no_chunks():
|
||||
assert parse_markdown("", "a.md").chunks == []
|
||||
assert parse_markdown(" \n\n \n", "a.md").chunks == []
|
||||
|
||||
|
||||
def test_setext_heading_is_a_section():
|
||||
doc = "Nadpis setext\n=============\n\n" + _paragraph(MERGE_BELOW)
|
||||
parsed = parse_markdown(doc, "a.md")
|
||||
assert _crumbs(parsed) == ["a.md > Nadpis setext"]
|
||||
|
||||
|
||||
def test_oversized_section_splits_and_keeps_code_fence_whole():
|
||||
fence = "```python\n" + "\n".join(f"value_{i} = {i} # a comment long enough to count" for i in range(60)) + "\n```"
|
||||
doc = "# Velká sekce\n\n" + "\n\n".join([_paragraph(120)] * 6) + "\n\n" + fence + "\n\n" + _paragraph(120)
|
||||
parsed = parse_markdown(doc, "big.md")
|
||||
|
||||
assert len(parsed.chunks) > 1
|
||||
assert {c.breadcrumb for c in parsed.chunks} == {"big.md > Velká sekce"}
|
||||
# The fence is larger than SPLIT_ABOVE on its own, so it must sit alone and unbroken.
|
||||
assert sum(fence in chunk.text for chunk in parsed.chunks) == 1
|
||||
|
||||
|
||||
def test_split_carries_block_aligned_overlap():
|
||||
blocks = [_paragraph(40) + f" marker{i:03d}" for i in range(40)]
|
||||
doc = "# Root\n\n" + "\n\n".join(blocks)
|
||||
parsed = parse_markdown(doc, "a.md")
|
||||
|
||||
assert len(parsed.chunks) > 1
|
||||
# Each boundary repeats at least one whole block, capped at OVERLAP tokens.
|
||||
for earlier, later in zip(parsed.chunks, parsed.chunks[1:], strict=False):
|
||||
shared = [b for b in blocks if b in earlier.text and b in later.text]
|
||||
assert shared, "expected overlap blocks between consecutive chunks"
|
||||
assert token_estimate("".join(shared)) <= OVERLAP
|
||||
|
||||
|
||||
def test_table_is_not_broken():
|
||||
table = "\n".join(["| a | b |", "|---|---|"] + [f"| {i} | {i * 2} |" for i in range(80)])
|
||||
doc = "# Root\n\n" + "\n\n".join([_paragraph(150)] * 5) + "\n\n" + table
|
||||
parsed = parse_markdown(doc, "a.md")
|
||||
assert sum(table in chunk.text for chunk in parsed.chunks) == 1
|
||||
|
||||
|
||||
def test_breadcrumb_is_prefixed_to_chunk_text():
|
||||
parsed = parse_markdown("# Root\n\n" + _paragraph(MERGE_BELOW), "notes/a.md")
|
||||
chunk = parsed.chunks[0]
|
||||
assert chunk.text.startswith(chunk.breadcrumb + "\n\n")
|
||||
|
||||
|
||||
def test_catalog_title_falls_back_to_first_heading():
|
||||
"""Notes carry their title as `# H1` far more often than as frontmatter."""
|
||||
doc = "# Zálohování dat\n\n" + _paragraph(MERGE_BELOW) + "\n\n## Retence\n\n" + _paragraph(MERGE_BELOW)
|
||||
parsed = parse_markdown(doc, "notes/zalohy.md")
|
||||
assert parsed.title == "Zálohování dat"
|
||||
|
||||
|
||||
def test_frontmatter_title_wins_over_first_heading():
|
||||
doc = "---\ntitle: Z frontmatteru\n---\n\n# Z nadpisu\n\n" + _paragraph(MERGE_BELOW)
|
||||
assert parse_markdown(doc, "a.md").title == "Z frontmatteru"
|
||||
|
||||
|
||||
def test_document_without_headings_has_no_title():
|
||||
assert parse_markdown(_paragraph(MERGE_BELOW), "a.md").title is None
|
||||
assert parse_markdown("", "a.md").title is None
|
||||
|
||||
|
||||
def test_heading_title_fallback_does_not_change_any_chunk():
|
||||
"""The fallback feeds only the catalog — touching the breadcrumb root would force --full."""
|
||||
doc = "úvod bez nadpisu\n\n# Root\n\n" + _paragraph(MERGE_BELOW) + "\n\n## Sekce\n\n" + _paragraph(MERGE_BELOW)
|
||||
parsed = parse_markdown(doc, "notes/a.md")
|
||||
|
||||
assert parsed.title == "Root"
|
||||
# The root stays the bare path, so no chunk text is rewritten by the fallback.
|
||||
assert _crumbs(parsed) == ["notes/a.md", "notes/a.md > Root", "notes/a.md > Root > Sekce"]
|
||||
165
skills/wiki/tests/test_wiki_config.py
Normal file
165
skills/wiki/tests/test_wiki_config.py
Normal file
@@ -0,0 +1,165 @@
|
||||
"""Scope precedence (paths -> include -> exclude) and config validation."""
|
||||
|
||||
import sys
|
||||
from pathlib import Path
|
||||
|
||||
import pytest
|
||||
|
||||
SCRIPTS = Path(__file__).parent.parent / "scripts"
|
||||
sys.path.insert(0, str(SCRIPTS))
|
||||
|
||||
import wiki_config
|
||||
from wiki_config import ConfigError, SourceConfig, load_config, source_root
|
||||
|
||||
WORKSPACE_YAML = """
|
||||
embedding:
|
||||
endpoint: http://nvidia.hell:11434/
|
||||
model: qwen3-embedding:0.6b
|
||||
dims: 1024
|
||||
batch: 32
|
||||
keep_alive: -1
|
||||
query_prefix: "Instruct: task\\nQuery: "
|
||||
|
||||
sources:
|
||||
index:
|
||||
kind: git
|
||||
url: git@git.fnet.cz:lachtan/index.git
|
||||
paths: ["**"]
|
||||
include: ["*.md"]
|
||||
exclude:
|
||||
- "**/node_modules/**"
|
||||
- "**/vendor/**"
|
||||
|
||||
workspace:
|
||||
kind: workspace
|
||||
paths:
|
||||
- "notes/**"
|
||||
- "develop/**"
|
||||
include: ["*.md"]
|
||||
exclude:
|
||||
- "**/inbox/**"
|
||||
- "develop/history.md"
|
||||
"""
|
||||
|
||||
|
||||
def _write(tmp_path, text):
|
||||
path = tmp_path / "config.yaml"
|
||||
path.write_text(text, encoding="utf-8")
|
||||
return path
|
||||
|
||||
|
||||
def _source(
|
||||
source_id: str = "ws",
|
||||
kind: str = "workspace",
|
||||
url: str | None = None,
|
||||
paths: tuple[str, ...] = ("**",),
|
||||
include: tuple[str, ...] = ("*.md",),
|
||||
exclude: tuple[str, ...] = (),
|
||||
) -> SourceConfig:
|
||||
return SourceConfig(source_id=source_id, kind=kind, url=url, paths=paths, include=include, exclude=exclude)
|
||||
|
||||
|
||||
def test_loads_embedding_and_sources(tmp_path):
|
||||
config = load_config(_write(tmp_path, WORKSPACE_YAML))
|
||||
assert config.embedding.endpoint == "http://nvidia.hell:11434" # trailing slash trimmed
|
||||
assert config.embedding.model == "qwen3-embedding:0.6b"
|
||||
assert config.embedding.dims == 1024
|
||||
assert config.embedding.keep_alive == -1
|
||||
assert config.embedding.query_prefix.endswith("Query: ")
|
||||
assert [s.source_id for s in config.sources] == ["index", "workspace"]
|
||||
index = config.source("index")
|
||||
assert index is not None and index.kind == "git"
|
||||
assert config.source("nope") is None
|
||||
|
||||
|
||||
def test_string_keep_alive_is_rejected():
|
||||
"""Ollama answers HTTP 400 to a string "-1" — catch it in config, not at runtime."""
|
||||
with pytest.raises(ConfigError, match="keep_alive"):
|
||||
wiki_config._parse_embedding({"endpoint": "http://x", "model": "m", "dims": 1024, "keep_alive": "-1"})
|
||||
|
||||
|
||||
def test_missing_config_file(tmp_path):
|
||||
with pytest.raises(ConfigError, match="missing config"):
|
||||
load_config(tmp_path / "nope.yaml")
|
||||
|
||||
|
||||
def test_git_source_needs_url(tmp_path):
|
||||
yaml_text = WORKSPACE_YAML.replace(" url: git@git.fnet.cz:lachtan/index.git\n", "")
|
||||
with pytest.raises(ConfigError, match="need a url"):
|
||||
load_config(_write(tmp_path, yaml_text))
|
||||
|
||||
|
||||
def test_unknown_kind_is_rejected(tmp_path):
|
||||
yaml_text = WORKSPACE_YAML.replace(" kind: workspace", " kind: mirror")
|
||||
with pytest.raises(ConfigError, match="kind must be one of"):
|
||||
load_config(_write(tmp_path, yaml_text))
|
||||
|
||||
|
||||
def test_paths_is_required(tmp_path):
|
||||
yaml_text = WORKSPACE_YAML.replace(' paths: ["**"]\n', "")
|
||||
with pytest.raises(ConfigError, match="`paths` is required"):
|
||||
load_config(_write(tmp_path, yaml_text))
|
||||
|
||||
|
||||
def test_paths_whitelist_gates_everything():
|
||||
source = _source(paths=("notes/**", "develop/**"))
|
||||
assert source.covers("notes/a.md")
|
||||
assert source.covers("notes/deep/nested/a.md")
|
||||
assert source.covers("develop/knowledge.md")
|
||||
# Not on the whitelist -> does not exist for the index.
|
||||
assert not source.covers("tmp/a.md")
|
||||
assert not source.covers("skills/wiki/SKILL.md")
|
||||
assert not source.covers("AGENTS.md")
|
||||
|
||||
|
||||
def test_include_filters_extensions():
|
||||
source = _source(paths=("**",))
|
||||
assert source.covers("notes/a.md")
|
||||
assert not source.covers("notes/main.py")
|
||||
assert not source.covers("memory/history.jsonl")
|
||||
assert not source.covers("assets/photo.png")
|
||||
|
||||
|
||||
def test_exclude_wins_over_paths_and_include():
|
||||
source = _source(paths=("notes/**", "develop/**"), exclude=("**/inbox/**", "develop/history.md"))
|
||||
assert not source.covers("notes/inbox/raw.md")
|
||||
assert not source.covers("develop/history.md")
|
||||
assert source.covers("develop/knowledge.md")
|
||||
assert source.covers("notes/notes.md")
|
||||
|
||||
|
||||
def test_double_star_matches_whole_repo():
|
||||
source = _source(paths=("**",), exclude=("**/node_modules/**", "**/vendor/**"))
|
||||
assert source.covers("README.md")
|
||||
assert source.covers("japan/tokyo/metro.md")
|
||||
assert not source.covers("node_modules/pkg/README.md")
|
||||
assert not source.covers("web/vendor/lib/CHANGELOG.md")
|
||||
|
||||
|
||||
def test_glob_star_does_not_cross_a_slash():
|
||||
source = _source(paths=("notes/*",))
|
||||
assert source.covers("notes/a.md")
|
||||
assert not source.covers("notes/deep/a.md")
|
||||
|
||||
|
||||
def test_covers_dir_prunes_the_walk():
|
||||
source = _source(paths=("notes/**", "develop/**"), exclude=("**/inbox/**",))
|
||||
assert source.covers_dir("")
|
||||
assert source.covers_dir("notes")
|
||||
assert source.covers_dir("notes/deep")
|
||||
assert source.covers_dir("develop")
|
||||
assert not source.covers_dir("tmp")
|
||||
assert not source.covers_dir("notes/inbox")
|
||||
|
||||
|
||||
def test_covers_dir_never_prunes_a_double_star_source():
|
||||
source = _source(paths=("**",), exclude=("**/node_modules/**",))
|
||||
assert source.covers_dir("anything/deep")
|
||||
assert not source.covers_dir("app/node_modules")
|
||||
|
||||
|
||||
def test_source_root_derives_clone_path_from_id():
|
||||
assert source_root(_source(source_id="travel", kind="git", url="git@x:y.git")) == (
|
||||
wiki_config.REMOTE_DIR / "travel"
|
||||
)
|
||||
assert source_root(_source(kind="workspace")) == wiki_config.WORKSPACE
|
||||
136
skills/wiki/tests/test_wiki_db.py
Normal file
136
skills/wiki/tests/test_wiki_db.py
Normal file
@@ -0,0 +1,136 @@
|
||||
"""Schema-level guarantees: the cascade, and the one hole the cascade leaves."""
|
||||
|
||||
import sys
|
||||
from pathlib import Path
|
||||
|
||||
SCRIPTS = Path(__file__).parent.parent / "scripts"
|
||||
sys.path.insert(0, str(SCRIPTS))
|
||||
|
||||
import wiki_store as store
|
||||
from wiki_db import EMBEDDING_DIMS
|
||||
|
||||
NOW = "2026-09-09T12:00:00+00:00"
|
||||
|
||||
|
||||
def _seed_file(conn, source_id, path, chunk_texts):
|
||||
store.upsert_source(conn, source_id, "workspace")
|
||||
store.upsert_file(conn, source_id, path, "T", ["t"], ["H"], "sha", 10, 1.0, NOW)
|
||||
store.replace_chunks(conn, source_id, path, [(f"{path} > s", t) for t in chunk_texts])
|
||||
for row in store.pending_chunks(conn, 100):
|
||||
store.store_embedding(conn, row["id"], [0.1] * EMBEDDING_DIMS, NOW)
|
||||
|
||||
|
||||
def test_delete_file_leaves_no_orphan_vectors(tmp_path):
|
||||
"""The FK cascade reaches chunks and chunks_fts but never vec0 — deletes must be explicit."""
|
||||
db_path = tmp_path / "index.sqlite"
|
||||
with store.transaction(db_path) as conn:
|
||||
_seed_file(conn, "ws", "a.md", ["alpha text", "beta text"])
|
||||
_seed_file(conn, "ws", "b.md", ["gamma text"])
|
||||
|
||||
with store.connection(db_path) as conn:
|
||||
assert store.index_stats(conn) == {"files": 2, "chunks": 3, "vectors": 3, "pending": 0}
|
||||
|
||||
with store.transaction(db_path) as conn:
|
||||
store.delete_file(conn, "ws", "a.md")
|
||||
|
||||
with store.connection(db_path) as conn:
|
||||
assert store.index_stats(conn) == {"files": 1, "chunks": 1, "vectors": 1, "pending": 0}
|
||||
assert store.orphan_vector_ids(conn) == []
|
||||
assert store.bm25_ranked_ids(conn, "alpha", 10) == []
|
||||
assert len(store.bm25_ranked_ids(conn, "gamma", 10)) == 1
|
||||
|
||||
|
||||
def test_replace_chunks_drops_old_vectors(tmp_path):
|
||||
db_path = tmp_path / "index.sqlite"
|
||||
with store.transaction(db_path) as conn:
|
||||
_seed_file(conn, "ws", "a.md", ["alpha text", "beta text", "delta text"])
|
||||
|
||||
with store.transaction(db_path) as conn:
|
||||
store.replace_chunks(conn, "ws", "a.md", [("a.md > s", "only one now")])
|
||||
|
||||
with store.connection(db_path) as conn:
|
||||
assert store.index_stats(conn) == {"files": 1, "chunks": 1, "vectors": 0, "pending": 1}
|
||||
assert store.orphan_vector_ids(conn) == []
|
||||
assert store.bm25_ranked_ids(conn, "alpha", 10) == []
|
||||
|
||||
|
||||
def test_delete_by_path_does_not_touch_other_source(tmp_path):
|
||||
"""The key is (source_id, path); a path-only delete would eat a foreign source's chunks."""
|
||||
db_path = tmp_path / "index.sqlite"
|
||||
with store.transaction(db_path) as conn:
|
||||
_seed_file(conn, "ws", "notes.md", ["shared name one"])
|
||||
_seed_file(conn, "git", "notes.md", ["shared name two"])
|
||||
|
||||
with store.transaction(db_path) as conn:
|
||||
store.delete_file(conn, "ws", "notes.md")
|
||||
|
||||
with store.connection(db_path) as conn:
|
||||
assert store.index_stats(conn)["chunks"] == 1
|
||||
assert store.orphan_vector_ids(conn) == []
|
||||
remaining = list(conn.execute("SELECT source_id FROM chunks"))
|
||||
assert remaining[0]["source_id"] == "git"
|
||||
|
||||
|
||||
def test_embedded_at_update_keeps_fts_row(tmp_path):
|
||||
"""Step 4b flips embedded_at on unchanged text; the WHEN guard must keep FTS intact."""
|
||||
db_path = tmp_path / "index.sqlite"
|
||||
with store.transaction(db_path) as conn:
|
||||
store.upsert_source(conn, "ws", "workspace")
|
||||
store.upsert_file(conn, "ws", "a.md", "T", [], [], "sha", 10, 1.0, NOW)
|
||||
store.replace_chunks(conn, "ws", "a.md", [("a.md", "zaloha dat na disk")])
|
||||
|
||||
with store.connection(db_path) as conn:
|
||||
assert len(store.bm25_ranked_ids(conn, "zaloha", 10)) == 1
|
||||
|
||||
with store.transaction(db_path) as conn:
|
||||
pending = store.pending_chunks(conn, 10)
|
||||
store.store_embedding(conn, pending[0]["id"], [0.2] * EMBEDDING_DIMS, NOW)
|
||||
|
||||
with store.connection(db_path) as conn:
|
||||
assert len(store.bm25_ranked_ids(conn, "zaloha", 10)) == 1
|
||||
assert store.index_stats(conn) == {"files": 1, "chunks": 1, "vectors": 1, "pending": 0}
|
||||
|
||||
|
||||
def test_fts_folds_czech_diacritics(tmp_path):
|
||||
"""remove_diacritics 2 is what makes `zaloha` find `záloha`."""
|
||||
db_path = tmp_path / "index.sqlite"
|
||||
with store.transaction(db_path) as conn:
|
||||
store.upsert_source(conn, "ws", "workspace")
|
||||
store.upsert_file(conn, "ws", "a.md", "T", [], [], "sha", 10, 1.0, NOW)
|
||||
store.replace_chunks(conn, "ws", "a.md", [("a.md", "zálohování dat probíhá denně")])
|
||||
|
||||
with store.connection(db_path) as conn:
|
||||
assert len(store.bm25_ranked_ids(conn, "zaloh*", 10)) == 1
|
||||
assert len(store.bm25_ranked_ids(conn, "záloh*", 10)) == 1
|
||||
|
||||
|
||||
def test_meta_roundtrip_and_rollback(tmp_path):
|
||||
db_path = tmp_path / "index.sqlite"
|
||||
with store.transaction(db_path) as conn:
|
||||
store.write_meta(conn, {"embedding_model": "qwen3-embedding:0.6b", "embedding_dims": "1024"})
|
||||
|
||||
with store.connection(db_path) as conn:
|
||||
assert store.read_meta(conn)["embedding_dims"] == "1024"
|
||||
|
||||
try:
|
||||
with store.transaction(db_path) as conn:
|
||||
store.write_meta(conn, {"embedding_dims": "768"})
|
||||
raise RuntimeError("boom")
|
||||
except RuntimeError:
|
||||
pass
|
||||
|
||||
with store.connection(db_path) as conn:
|
||||
assert store.read_meta(conn)["embedding_dims"] == "1024"
|
||||
|
||||
|
||||
def test_toc_filters_by_tag(tmp_path):
|
||||
db_path = tmp_path / "index.sqlite"
|
||||
with store.transaction(db_path) as conn:
|
||||
store.upsert_source(conn, "travel", "git")
|
||||
store.upsert_file(conn, "travel", "japan/metro.md", "Metro", ["transit"], [], "s", 1, 1.0, NOW)
|
||||
store.upsert_file(conn, "travel", "alps/gear.md", "Gear", ["gear", "safety"], [], "s", 1, 1.0, NOW)
|
||||
|
||||
with store.connection(db_path) as conn:
|
||||
assert [r["path"] for r in store.list_toc_files(conn, tag="gear")] == ["alps/gear.md"]
|
||||
assert len(store.list_toc_files(conn, source_id="travel")) == 2
|
||||
assert store.list_toc_files(conn, source_id="nope") == []
|
||||
266
skills/wiki/tests/test_wiki_git.py
Normal file
266
skills/wiki/tests/test_wiki_git.py
Normal file
@@ -0,0 +1,266 @@
|
||||
"""Git driver: clone, ls-remote change detection, diff, deletions, per-source failure."""
|
||||
|
||||
import subprocess
|
||||
import sys
|
||||
from pathlib import Path
|
||||
|
||||
SCRIPTS = Path(__file__).parent.parent / "scripts"
|
||||
sys.path.insert(0, str(SCRIPTS))
|
||||
|
||||
import wiki_store as store
|
||||
import wiki_sync
|
||||
from conftest import WORKSPACE_SOURCE
|
||||
|
||||
DOC = """# Zálohování
|
||||
|
||||
Záloha běží každou noc přes rsync na druhý disk.
|
||||
"""
|
||||
|
||||
TRAVEL_DOC = """# Tokijské metro
|
||||
|
||||
Z Narity do centra jede Skyliner za osmatřicet minut.
|
||||
"""
|
||||
|
||||
GIT_SOURCE = """ notes:
|
||||
kind: git
|
||||
url: {url}
|
||||
paths: ["**"]
|
||||
include: ["*.md"]
|
||||
exclude:
|
||||
- "**/node_modules/**"
|
||||
- "**/vendor/**"
|
||||
"""
|
||||
|
||||
|
||||
def _git(args, cwd=None):
|
||||
result = subprocess.run(
|
||||
["git", "-c", "user.email=t@t", "-c", "user.name=t", *args],
|
||||
cwd=cwd,
|
||||
capture_output=True,
|
||||
text=True,
|
||||
check=True,
|
||||
)
|
||||
return result.stdout
|
||||
|
||||
|
||||
class Remote:
|
||||
"""A bare repo plus a working clone to push commits from."""
|
||||
|
||||
def __init__(self, root: Path):
|
||||
self.bare = root / "origin.git"
|
||||
self.work = root / "origin-work"
|
||||
_git(["init", "--bare", "--initial-branch=master", str(self.bare)])
|
||||
_git(["clone", str(self.bare), str(self.work)])
|
||||
|
||||
@property
|
||||
def url(self) -> str:
|
||||
return str(self.bare)
|
||||
|
||||
def commit(self, files: dict[str, str | None], message: str = "change") -> str:
|
||||
for rel_path, text in files.items():
|
||||
path = self.work / rel_path
|
||||
if text is None:
|
||||
_git(["rm", "-q", rel_path], cwd=self.work)
|
||||
continue
|
||||
path.parent.mkdir(parents=True, exist_ok=True)
|
||||
path.write_text(text, encoding="utf-8")
|
||||
_git(["add", rel_path], cwd=self.work)
|
||||
_git(["commit", "-m", message], cwd=self.work)
|
||||
_git(["push", "-q", "origin", "master"], cwd=self.work)
|
||||
return _git(["rev-parse", "HEAD"], cwd=self.work).strip()
|
||||
|
||||
def move(self, old: str, new: str) -> str:
|
||||
(self.work / new).parent.mkdir(parents=True, exist_ok=True)
|
||||
_git(["mv", old, new], cwd=self.work)
|
||||
_git(["commit", "-m", "rename"], cwd=self.work)
|
||||
_git(["push", "-q", "origin", "master"], cwd=self.work)
|
||||
return _git(["rev-parse", "HEAD"], cwd=self.work).strip()
|
||||
|
||||
|
||||
def _use_git_source(env, url, with_workspace=False):
|
||||
sources = GIT_SOURCE.format(url=url)
|
||||
if with_workspace:
|
||||
sources += WORKSPACE_SOURCE
|
||||
env.write_config(sources)
|
||||
|
||||
|
||||
def _stats(env):
|
||||
with store.connection(env.db_path) as conn:
|
||||
return store.index_stats(conn)
|
||||
|
||||
|
||||
def test_first_run_clones_and_indexes(wiki_env, fake_embedder, tmp_path):
|
||||
remote = Remote(tmp_path)
|
||||
head = remote.commit({"zalohy.md": DOC, "japan/metro.md": TRAVEL_DOC})
|
||||
_use_git_source(wiki_env, remote.url)
|
||||
|
||||
assert wiki_sync.main([]) == 0
|
||||
|
||||
clone = wiki_env.wiki_dir / "remote" / "notes"
|
||||
assert (clone / "zalohy.md").is_file() # non-bare: a working tree grep can read
|
||||
assert (clone / ".git").is_dir()
|
||||
|
||||
with store.connection(wiki_env.db_path) as conn:
|
||||
assert sorted(store.list_source_files(conn, "notes")) == ["japan/metro.md", "zalohy.md"]
|
||||
source = store.get_source(conn, "notes")
|
||||
assert source is not None
|
||||
assert source["indexed_rev"] == head
|
||||
assert source["kind"] == "git"
|
||||
stats = _stats(wiki_env)
|
||||
assert stats["vectors"] == stats["chunks"] > 0
|
||||
|
||||
|
||||
def test_second_run_over_the_same_revision_changes_nothing(wiki_env, fake_embedder, tmp_path):
|
||||
"""Verification 2 for the git driver."""
|
||||
remote = Remote(tmp_path)
|
||||
remote.commit({"zalohy.md": DOC})
|
||||
_use_git_source(wiki_env, remote.url)
|
||||
assert wiki_sync.main([]) == 0
|
||||
|
||||
before = _stats(wiki_env)
|
||||
log_before = wiki_env.log_text()
|
||||
|
||||
assert wiki_sync.main([]) == 0
|
||||
|
||||
assert _stats(wiki_env) == before
|
||||
assert wiki_env.log_text() == log_before # ls-remote matched, so not even a fetch
|
||||
|
||||
|
||||
def test_new_commit_reindexes_only_the_changed_file(wiki_env, fake_embedder, tmp_path):
|
||||
remote = Remote(tmp_path)
|
||||
remote.commit({"zalohy.md": DOC, "japan/metro.md": TRAVEL_DOC})
|
||||
_use_git_source(wiki_env, remote.url)
|
||||
assert wiki_sync.main([]) == 0
|
||||
|
||||
with store.connection(wiki_env.db_path) as conn:
|
||||
metro_indexed_at = store.list_source_files(conn, "notes")["japan/metro.md"]["indexed_at"]
|
||||
|
||||
head = remote.commit({"zalohy.md": DOC + "\n## Offsite\n\nKopie jede do S3.\n"})
|
||||
assert wiki_sync.main([]) == 0
|
||||
|
||||
with store.connection(wiki_env.db_path) as conn:
|
||||
files = store.list_source_files(conn, "notes")
|
||||
assert files["japan/metro.md"]["indexed_at"] == metro_indexed_at # untouched
|
||||
source = store.get_source(conn, "notes")
|
||||
assert source is not None and source["indexed_rev"] == head
|
||||
assert len(store.bm25_ranked_ids(conn, "offsite", 10)) == 1
|
||||
assert store.orphan_vector_ids(conn) == []
|
||||
|
||||
|
||||
def test_deleted_file_upstream_drops_its_chunks_and_vectors(wiki_env, fake_embedder, tmp_path):
|
||||
remote = Remote(tmp_path)
|
||||
remote.commit({"zalohy.md": DOC, "japan/metro.md": TRAVEL_DOC})
|
||||
_use_git_source(wiki_env, remote.url)
|
||||
assert wiki_sync.main([]) == 0
|
||||
|
||||
remote.commit({"japan/metro.md": None}, message="drop metro")
|
||||
assert wiki_sync.main([]) == 0
|
||||
|
||||
with store.connection(wiki_env.db_path) as conn:
|
||||
assert sorted(store.list_source_files(conn, "notes")) == ["zalohy.md"]
|
||||
assert store.bm25_ranked_ids(conn, "skyliner", 10) == []
|
||||
assert store.orphan_vector_ids(conn) == []
|
||||
stats = _stats(wiki_env)
|
||||
assert stats["vectors"] == stats["chunks"]
|
||||
|
||||
|
||||
def test_renamed_file_moves_its_chunks(wiki_env, fake_embedder, tmp_path):
|
||||
remote = Remote(tmp_path)
|
||||
remote.commit({"japan/metro.md": TRAVEL_DOC})
|
||||
_use_git_source(wiki_env, remote.url)
|
||||
assert wiki_sync.main([]) == 0
|
||||
|
||||
remote.move("japan/metro.md", "japan/tokyo-metro.md")
|
||||
assert wiki_sync.main([]) == 0
|
||||
|
||||
with store.connection(wiki_env.db_path) as conn:
|
||||
assert sorted(store.list_source_files(conn, "notes")) == ["japan/tokyo-metro.md"]
|
||||
assert len(store.bm25_ranked_ids(conn, "skyliner", 10)) == 1
|
||||
assert store.orphan_vector_ids(conn) == []
|
||||
|
||||
|
||||
def test_commit_touching_nothing_indexed_only_records_the_rev(wiki_env, fake_embedder, tmp_path):
|
||||
remote = Remote(tmp_path)
|
||||
remote.commit({"zalohy.md": DOC})
|
||||
_use_git_source(wiki_env, remote.url)
|
||||
assert wiki_sync.main([]) == 0
|
||||
before = _stats(wiki_env)
|
||||
|
||||
head = remote.commit({"tool.py": "print('hello')\n"})
|
||||
assert wiki_sync.main([]) == 0
|
||||
|
||||
assert _stats(wiki_env) == before
|
||||
with store.connection(wiki_env.db_path) as conn:
|
||||
source = store.get_source(conn, "notes")
|
||||
assert source is not None and source["indexed_rev"] == head
|
||||
assert "no indexed file changed" in wiki_env.log_text()
|
||||
|
||||
|
||||
def test_vendored_markdown_is_excluded(wiki_env, fake_embedder, tmp_path):
|
||||
remote = Remote(tmp_path)
|
||||
remote.commit(
|
||||
{
|
||||
"zalohy.md": DOC,
|
||||
"node_modules/pkg/README.md": "# Foreign readme\n\nnenasazovat\n",
|
||||
"web/vendor/lib/CHANGELOG.md": "# Changelog\n\nnenasazovat\n",
|
||||
}
|
||||
)
|
||||
_use_git_source(wiki_env, remote.url)
|
||||
assert wiki_sync.main([]) == 0
|
||||
|
||||
with store.connection(wiki_env.db_path) as conn:
|
||||
assert sorted(store.list_source_files(conn, "notes")) == ["zalohy.md"]
|
||||
|
||||
|
||||
def test_unreachable_remote_warns_skips_and_lets_other_sources_finish(wiki_env, fake_embedder, tmp_path):
|
||||
"""Verification 8: per-source failure, indexed_rev untouched, workspace completes."""
|
||||
_use_git_source(wiki_env, str(tmp_path / "does-not-exist.git"), with_workspace=True)
|
||||
wiki_env.write_file("notes/local.md", DOC)
|
||||
|
||||
assert wiki_sync.main([]) == 1
|
||||
|
||||
log = wiki_env.log_text()
|
||||
assert "WARN notes:" in log
|
||||
# A permanent failure warns every tick, so each warning must stay a single line.
|
||||
assert len([line for line in log.splitlines() if "WARN notes:" in line]) == 1
|
||||
assert all(line.strip() for line in log.splitlines())
|
||||
with store.connection(wiki_env.db_path) as conn:
|
||||
assert store.list_source_files(conn, "notes") == {}
|
||||
git_source = store.get_source(conn, "notes")
|
||||
assert git_source is not None and git_source["indexed_rev"] is None
|
||||
# The workspace source, which has nothing to do with the network, finished.
|
||||
assert sorted(store.list_source_files(conn, "workspace")) == ["notes/local.md"]
|
||||
|
||||
|
||||
def test_remote_that_recovers_is_picked_up_on_the_next_tick(wiki_env, fake_embedder, tmp_path):
|
||||
missing = tmp_path / "later.git"
|
||||
_use_git_source(wiki_env, str(missing))
|
||||
assert wiki_sync.main([]) == 1
|
||||
|
||||
remote = Remote(tmp_path / "real")
|
||||
(tmp_path / "real").mkdir(exist_ok=True)
|
||||
remote_bare = remote.bare
|
||||
remote.commit({"zalohy.md": DOC})
|
||||
_use_git_source(wiki_env, str(remote_bare))
|
||||
|
||||
assert wiki_sync.main([]) == 0
|
||||
with store.connection(wiki_env.db_path) as conn:
|
||||
assert sorted(store.list_source_files(conn, "notes")) == ["zalohy.md"]
|
||||
|
||||
|
||||
def test_missing_indexed_rev_falls_back_to_a_full_reindex(wiki_env, fake_embedder, tmp_path):
|
||||
"""A force-pushed or gc'd base revision must degrade to a full pass, not crash."""
|
||||
remote = Remote(tmp_path)
|
||||
remote.commit({"zalohy.md": DOC})
|
||||
_use_git_source(wiki_env, remote.url)
|
||||
assert wiki_sync.main([]) == 0
|
||||
|
||||
with store.connection(wiki_env.db_path) as conn, store.tx(conn):
|
||||
store.mark_synced(conn, "notes", "0" * 40, "2026-01-01T00:00:00+00:00")
|
||||
|
||||
remote.commit({"japan/metro.md": TRAVEL_DOC})
|
||||
assert wiki_sync.main([]) == 0
|
||||
|
||||
with store.connection(wiki_env.db_path) as conn:
|
||||
assert sorted(store.list_source_files(conn, "notes")) == ["japan/metro.md", "zalohy.md"]
|
||||
assert store.orphan_vector_ids(conn) == []
|
||||
306
skills/wiki/tests/test_wiki_search.py
Normal file
306
skills/wiki/tests/test_wiki_search.py
Normal file
@@ -0,0 +1,306 @@
|
||||
"""Query layers: RRF merge, the FTS expression, degraded mode, toc and grep."""
|
||||
|
||||
import shutil
|
||||
import sys
|
||||
from pathlib import Path
|
||||
|
||||
import pytest
|
||||
|
||||
SCRIPTS = Path(__file__).parent.parent / "scripts"
|
||||
sys.path.insert(0, str(SCRIPTS))
|
||||
|
||||
import wiki_search
|
||||
import wiki_store as store
|
||||
import wiki_sync
|
||||
from wiki_config import SourceConfig, load_config
|
||||
|
||||
BACKUP_DOC = """---
|
||||
title: Zálohování dat
|
||||
tags: [devops, backup]
|
||||
---
|
||||
|
||||
# Zálohování dat
|
||||
|
||||
Záloha běží každou noc přes rsync na druhý disk.
|
||||
|
||||
## Retence snapshotů
|
||||
|
||||
Držíme třicet denních snapshotů a dvanáct měsíčních.
|
||||
"""
|
||||
|
||||
TRAVEL_DOC = """---
|
||||
title: Tokijské metro
|
||||
tags: [transit]
|
||||
---
|
||||
|
||||
# Tokijské metro
|
||||
|
||||
Z Narity do centra jede Skyliner za osmatřicet minut.
|
||||
"""
|
||||
|
||||
|
||||
def _index(env):
|
||||
assert wiki_sync.main([]) == 0
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# pure functions
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
|
||||
def test_rrf_merge_over_known_ranks():
|
||||
merged = wiki_search.rrf_merge([[1, 2, 3], [3, 1, 4]])
|
||||
assert [chunk_id for chunk_id, _ in merged] == [1, 3, 2, 4]
|
||||
scores = dict(merged)
|
||||
assert scores[1] == pytest.approx(1 / 61 + 1 / 62)
|
||||
assert scores[3] == pytest.approx(1 / 63 + 1 / 61)
|
||||
assert scores[4] == pytest.approx(1 / 63)
|
||||
|
||||
|
||||
def test_rrf_merge_of_a_single_list_keeps_its_order():
|
||||
assert [i for i, _ in wiki_search.rrf_merge([[7, 8, 9]])] == [7, 8, 9]
|
||||
|
||||
|
||||
def test_fts_expression_adds_prefix_wildcards():
|
||||
"""Czech inflection is covered by the wildcard; a 2-char prefix is too broad to keep."""
|
||||
assert wiki_search.fts_match_expression("záloha dat") == '"záloha"* OR "dat"*'
|
||||
assert wiki_search.fts_match_expression("v ok dva") == '"v" OR "ok" OR "dva"*'
|
||||
|
||||
|
||||
def test_fts_expression_neutralises_operators_and_punctuation():
|
||||
expression = wiki_search.fts_match_expression("NOT (a AND b) -c*")
|
||||
assert expression == '"NOT"* OR "a" OR "AND"* OR "b" OR "c"'
|
||||
|
||||
|
||||
def test_fts_expression_of_empty_query_is_empty():
|
||||
assert wiki_search.fts_match_expression("!!! ") == ""
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# search
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
|
||||
def test_search_reports_which_half_found_each_hit(wiki_env, fake_embedder, capsys):
|
||||
wiki_env.write_file("notes/zalohy.md", BACKUP_DOC)
|
||||
wiki_env.write_file("notes/tokio.md", TRAVEL_DOC)
|
||||
_index(wiki_env)
|
||||
|
||||
config = load_config(wiki_env.config_path)
|
||||
assert wiki_search.run_search(config, "retence snapshotů", limit=5) == 0
|
||||
|
||||
out = capsys.readouterr().out
|
||||
assert "notes/zalohy.md" in out
|
||||
assert "Retence snapshotů" in out
|
||||
assert "bm25 #" in out and "vec #" in out
|
||||
|
||||
|
||||
def test_search_prefix_match_finds_inflected_form(wiki_env, fake_embedder, capsys):
|
||||
wiki_env.write_file("notes/zalohy.md", BACKUP_DOC)
|
||||
_index(wiki_env)
|
||||
|
||||
config = load_config(wiki_env.config_path)
|
||||
assert wiki_search.run_search(config, "zaloh", limit=5) == 0
|
||||
assert "notes/zalohy.md" in capsys.readouterr().out
|
||||
|
||||
|
||||
def test_search_says_out_loud_when_embeddings_are_unavailable(wiki_env, fake_embedder, capsys):
|
||||
"""Verification 4: degrade to FTS-only, never silently."""
|
||||
wiki_env.write_file("notes/zalohy.md", BACKUP_DOC)
|
||||
_index(wiki_env)
|
||||
capsys.readouterr()
|
||||
|
||||
# Drop the fake and let the config's unreachable endpoint take over.
|
||||
wiki_search.OllamaEmbedder = wiki_search.__dict__["OllamaEmbedder"]
|
||||
from wiki_embed import OllamaEmbedder as RealEmbedder
|
||||
|
||||
wiki_search.OllamaEmbedder = RealEmbedder
|
||||
|
||||
config = load_config(wiki_env.config_path)
|
||||
assert wiki_search.run_search(config, "záloha", limit=5) == 0
|
||||
|
||||
out = capsys.readouterr().out
|
||||
assert "embeddings unavailable" in out
|
||||
assert "FTS-only" in out
|
||||
assert "notes/zalohy.md" in out # the lexical half still answers
|
||||
|
||||
|
||||
def test_search_refuses_an_index_from_another_contract(wiki_env, fake_embedder, capsys):
|
||||
"""Verification 5: a mismatch must say `reindex needed`, not return results."""
|
||||
wiki_env.write_file("notes/zalohy.md", BACKUP_DOC)
|
||||
_index(wiki_env)
|
||||
|
||||
wiki_env.config_path.write_text(
|
||||
wiki_env.config_path.read_text(encoding="utf-8").replace(
|
||||
"model: qwen3-embedding:0.6b", "model: qwen3-embedding:8b"
|
||||
),
|
||||
encoding="utf-8",
|
||||
)
|
||||
config = load_config(wiki_env.config_path)
|
||||
assert wiki_search.run_search(config, "záloha", limit=5) == 1
|
||||
|
||||
captured = capsys.readouterr()
|
||||
assert "reindex needed" in captured.err
|
||||
assert "notes/zalohy.md" not in captured.out
|
||||
|
||||
|
||||
def test_search_warns_while_vectors_are_still_pending(wiki_env, monkeypatch, capsys):
|
||||
wiki_env.write_file("notes/zalohy.md", BACKUP_DOC)
|
||||
assert wiki_sync.main([]) == 1 # unreachable endpoint -> chunks stay pending
|
||||
|
||||
from conftest import FakeEmbedder
|
||||
|
||||
monkeypatch.setattr(wiki_search, "OllamaEmbedder", FakeEmbedder)
|
||||
config = load_config(wiki_env.config_path)
|
||||
assert wiki_search.run_search(config, "záloha", limit=5) == 0
|
||||
|
||||
out = capsys.readouterr().out
|
||||
assert "awaiting vectors" in out
|
||||
|
||||
|
||||
def test_search_over_an_empty_index_says_no_matches(wiki_env, fake_embedder, capsys):
|
||||
config = load_config(wiki_env.config_path)
|
||||
assert wiki_search.run_search(config, "záloha", limit=5) == 0
|
||||
assert "(no matches)" in capsys.readouterr().out
|
||||
|
||||
|
||||
def test_an_unrelated_query_still_returns_nearest_neighbours(wiki_env, fake_embedder, capsys):
|
||||
"""KNN has no distance floor: the semantic half always offers its k closest chunks.
|
||||
|
||||
That is deliberate for retrieval — the agent reads the chunk and judges it — but it
|
||||
means an empty result set only ever means an empty index.
|
||||
"""
|
||||
wiki_env.write_file("notes/zalohy.md", BACKUP_DOC)
|
||||
_index(wiki_env)
|
||||
capsys.readouterr()
|
||||
|
||||
config = load_config(wiki_env.config_path)
|
||||
assert wiki_search.run_search(config, "kajakářství", limit=5) == 0
|
||||
out = capsys.readouterr().out
|
||||
assert "notes/zalohy.md" in out
|
||||
assert "bm25 #" not in out # nothing lexical matched; every hit came from the vectors
|
||||
|
||||
|
||||
def test_search_with_an_empty_query(wiki_env, fake_embedder, capsys):
|
||||
wiki_env.write_file("notes/zalohy.md", BACKUP_DOC)
|
||||
_index(wiki_env)
|
||||
capsys.readouterr()
|
||||
|
||||
config = load_config(wiki_env.config_path)
|
||||
assert wiki_search.run_search(config, "???", limit=5) == 0
|
||||
assert "(empty query)" in capsys.readouterr().out
|
||||
|
||||
|
||||
def test_search_limit_caps_the_result_count(wiki_env, fake_embedder, capsys):
|
||||
for index in range(8):
|
||||
wiki_env.write_file(f"notes/doc{index}.md", f"# Dokument {index}\n\nZáloha dat číslo {index}.\n")
|
||||
_index(wiki_env)
|
||||
capsys.readouterr()
|
||||
|
||||
config = load_config(wiki_env.config_path)
|
||||
assert wiki_search.run_search(config, "záloha dat", limit=3) == 0
|
||||
out = capsys.readouterr().out
|
||||
assert len([line for line in out.splitlines() if line.startswith(("1. ", "2. ", "3. ", "4. "))]) == 3
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# toc
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
|
||||
def test_toc_groups_by_directory_and_shows_titles_and_tags(wiki_env, fake_embedder, capsys):
|
||||
wiki_env.write_file("notes/devops/zalohy.md", BACKUP_DOC)
|
||||
wiki_env.write_file("notes/travel/tokio.md", TRAVEL_DOC)
|
||||
_index(wiki_env)
|
||||
capsys.readouterr()
|
||||
|
||||
assert wiki_search.run_toc(None, None) == 0
|
||||
out = capsys.readouterr().out
|
||||
assert "workspace (2 files)" in out
|
||||
assert "notes/devops/" in out
|
||||
assert "zalohy.md" in out
|
||||
assert "Zálohování dat" in out
|
||||
assert "[devops, backup]" in out
|
||||
|
||||
|
||||
def test_toc_filters_by_tag(wiki_env, fake_embedder, capsys):
|
||||
wiki_env.write_file("notes/zalohy.md", BACKUP_DOC)
|
||||
wiki_env.write_file("notes/tokio.md", TRAVEL_DOC)
|
||||
_index(wiki_env)
|
||||
capsys.readouterr()
|
||||
|
||||
assert wiki_search.run_toc(None, "transit") == 0
|
||||
out = capsys.readouterr().out
|
||||
assert "tokio.md" in out
|
||||
assert "zalohy.md" not in out
|
||||
|
||||
|
||||
def test_toc_on_an_empty_index(wiki_env, fake_embedder, capsys):
|
||||
assert wiki_search.run_toc(None, None) == 0
|
||||
assert "(no indexed files match)" in capsys.readouterr().out
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# grep
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
|
||||
@pytest.mark.skipif(shutil.which("rg") is None, reason="ripgrep not installed")
|
||||
def test_grep_sees_source_code_the_index_never_touches(wiki_env, fake_embedder, capsys):
|
||||
"""Layer 1 is the substitute for indexing code, and it costs nothing extra."""
|
||||
wiki_env.write_file("notes/zalohy.md", BACKUP_DOC)
|
||||
wiki_env.write_file("notes/tools/backup.py", "def rotate_snapshots(keep=30):\n return keep\n")
|
||||
_index(wiki_env)
|
||||
capsys.readouterr()
|
||||
|
||||
config = load_config(wiki_env.config_path)
|
||||
assert wiki_search.run_grep(config, "rotate_snapshots", None) == 0
|
||||
out = capsys.readouterr().out
|
||||
assert "backup.py" in out
|
||||
assert "rotate_snapshots" in out
|
||||
|
||||
# The same identifier is absent from the index — grep is the only layer that has it.
|
||||
with store.connection(wiki_env.db_path) as conn:
|
||||
assert store.bm25_ranked_ids(conn, "rotate_snapshots", 10) == []
|
||||
|
||||
|
||||
@pytest.mark.skipif(shutil.which("rg") is None, reason="ripgrep not installed")
|
||||
def test_grep_stays_inside_the_source_paths(wiki_env, fake_embedder, capsys):
|
||||
wiki_env.write_file("notes/zalohy.md", BACKUP_DOC)
|
||||
wiki_env.write_file("tmp/dump/leak.md", "tajnastruna v tmp\n")
|
||||
|
||||
config = load_config(wiki_env.config_path)
|
||||
assert wiki_search.run_grep(config, "tajnastruna", None) == 0
|
||||
assert "(no matches)" in capsys.readouterr().out
|
||||
|
||||
|
||||
@pytest.mark.skipif(shutil.which("rg") is None, reason="ripgrep not installed")
|
||||
def test_grep_reports_an_unknown_source(wiki_env, fake_embedder, capsys):
|
||||
config = load_config(wiki_env.config_path)
|
||||
assert wiki_search.run_grep(config, "cokoli", "nope") == 1
|
||||
assert "unknown source" in capsys.readouterr().err
|
||||
|
||||
|
||||
def test_grep_roots_use_the_literal_prefix_of_each_glob(wiki_env):
|
||||
wiki_env.write_file("notes/a.md", "x")
|
||||
wiki_env.write_file("develop/b.md", "x")
|
||||
source = load_config(wiki_env.config_path).source("workspace")
|
||||
assert source is not None
|
||||
assert wiki_search.grep_roots(source) == [
|
||||
wiki_env.workspace / "notes",
|
||||
wiki_env.workspace / "develop",
|
||||
]
|
||||
|
||||
|
||||
def test_grep_roots_of_a_git_source_is_the_whole_clone(wiki_env):
|
||||
clone = wiki_env.wiki_dir / "remote" / "travel"
|
||||
clone.mkdir(parents=True)
|
||||
source = SourceConfig(
|
||||
source_id="travel",
|
||||
kind="git",
|
||||
url="git@example:travel.git",
|
||||
paths=("**",),
|
||||
include=("*.md",),
|
||||
exclude=(),
|
||||
)
|
||||
assert wiki_search.grep_roots(source) == [clone]
|
||||
279
skills/wiki/tests/test_wiki_sync.py
Normal file
279
skills/wiki/tests/test_wiki_sync.py
Normal file
@@ -0,0 +1,279 @@
|
||||
"""Sync driver: idempotence, the lock, degraded mode, the meta guard, coverage."""
|
||||
|
||||
import json
|
||||
import os
|
||||
import sys
|
||||
from pathlib import Path
|
||||
|
||||
SCRIPTS = Path(__file__).parent.parent / "scripts"
|
||||
sys.path.insert(0, str(SCRIPTS))
|
||||
|
||||
import wiki_store as store
|
||||
import wiki_sync
|
||||
from wiki_config import load_config
|
||||
from wiki_embed import expected_meta
|
||||
|
||||
DOC = """# Zálohování
|
||||
|
||||
Záloha dat na disk probíhá každý den v noci pomocí rsyncu.
|
||||
|
||||
## Retence
|
||||
|
||||
Držíme třicet denních snapshotů a dvanáct měsíčních.
|
||||
"""
|
||||
|
||||
OTHER_DOC = """# Cestování
|
||||
|
||||
Tokio metro je nejrychlejší cesta z Narity do centra.
|
||||
"""
|
||||
|
||||
|
||||
def _stats(env):
|
||||
with store.connection(env.db_path) as conn:
|
||||
return store.index_stats(conn)
|
||||
|
||||
|
||||
def _run(argv=None):
|
||||
return wiki_sync.main(argv or [])
|
||||
|
||||
|
||||
def _file_row(conn, path):
|
||||
row = store.get_file(conn, "workspace", path)
|
||||
assert row is not None, f"{path} is not indexed"
|
||||
return row
|
||||
|
||||
|
||||
def test_indexes_covered_workspace_files(wiki_env, fake_embedder):
|
||||
wiki_env.write_file("notes/zalohy.md", DOC)
|
||||
wiki_env.write_file("develop/knowledge.md", OTHER_DOC)
|
||||
|
||||
assert _run() == 0
|
||||
|
||||
stats = _stats(wiki_env)
|
||||
assert stats["files"] == 2
|
||||
assert stats["chunks"] > 0
|
||||
assert stats["vectors"] == stats["chunks"]
|
||||
assert stats["pending"] == 0
|
||||
assert "indexed 2 files" in wiki_env.log_text() or "indexed 1 files" in wiki_env.log_text()
|
||||
|
||||
|
||||
def test_paths_whitelist_and_exclude_are_honoured(wiki_env, fake_embedder):
|
||||
wiki_env.write_file("notes/keep.md", DOC)
|
||||
wiki_env.write_file("notes/inbox/raw.md", DOC) # excluded
|
||||
wiki_env.write_file("develop/history.md", DOC) # excluded
|
||||
wiki_env.write_file("tmp/clone/README.md", DOC) # outside paths
|
||||
wiki_env.write_file("notes/script.py", "# TODO: fix this\n") # outside include
|
||||
|
||||
assert _run() == 0
|
||||
|
||||
with store.connection(wiki_env.db_path) as conn:
|
||||
paths = sorted(store.list_source_files(conn, "workspace"))
|
||||
assert paths == ["notes/keep.md"]
|
||||
|
||||
|
||||
def test_second_run_over_unchanged_tree_is_a_no_op(wiki_env, fake_embedder):
|
||||
"""Verification 2: counts identical and nothing appended to the log."""
|
||||
wiki_env.write_file("notes/zalohy.md", DOC)
|
||||
assert _run() == 0
|
||||
|
||||
first_stats = _stats(wiki_env)
|
||||
with store.connection(wiki_env.db_path) as conn:
|
||||
first_indexed_at = _file_row(conn, "notes/zalohy.md")["indexed_at"]
|
||||
log_after_first = wiki_env.log_text()
|
||||
|
||||
assert _run() == 0
|
||||
|
||||
assert _stats(wiki_env) == first_stats
|
||||
assert wiki_env.log_text() == log_after_first
|
||||
with store.connection(wiki_env.db_path) as conn:
|
||||
assert _file_row(conn, "notes/zalohy.md")["indexed_at"] == first_indexed_at
|
||||
|
||||
|
||||
def test_touching_a_file_without_changing_it_only_refreshes_stats(wiki_env, fake_embedder):
|
||||
wiki_env.write_file("notes/zalohy.md", DOC)
|
||||
assert _run() == 0
|
||||
before = _stats(wiki_env)
|
||||
with store.connection(wiki_env.db_path) as conn:
|
||||
indexed_at = _file_row(conn, "notes/zalohy.md")["indexed_at"]
|
||||
|
||||
os.utime(wiki_env.workspace / "notes/zalohy.md", (1_600_000_000, 1_600_000_000))
|
||||
assert _run() == 0
|
||||
|
||||
assert _stats(wiki_env) == before
|
||||
with store.connection(wiki_env.db_path) as conn:
|
||||
row = _file_row(conn, "notes/zalohy.md")
|
||||
assert row["indexed_at"] == indexed_at # content untouched -> no re-chunk
|
||||
assert row["mtime"] == 1_600_000_000
|
||||
|
||||
|
||||
def test_edited_file_is_rechunked_and_reembedded(wiki_env, fake_embedder):
|
||||
wiki_env.write_file("notes/zalohy.md", DOC)
|
||||
assert _run() == 0
|
||||
|
||||
wiki_env.write_file("notes/zalohy.md", DOC + "\n## Offsite\n\nKopie jede do S3 každý týden.\n")
|
||||
assert _run() == 0
|
||||
|
||||
stats = _stats(wiki_env)
|
||||
assert stats["pending"] == 0
|
||||
assert stats["vectors"] == stats["chunks"]
|
||||
with store.connection(wiki_env.db_path) as conn:
|
||||
assert store.orphan_vector_ids(conn) == []
|
||||
assert len(store.bm25_ranked_ids(conn, "offsite", 10)) == 1
|
||||
|
||||
|
||||
def test_deleted_file_leaves_no_chunks_or_vectors(wiki_env, fake_embedder):
|
||||
wiki_env.write_file("notes/zalohy.md", DOC)
|
||||
wiki_env.write_file("notes/travel.md", OTHER_DOC)
|
||||
assert _run() == 0
|
||||
|
||||
(wiki_env.workspace / "notes/travel.md").unlink()
|
||||
assert _run() == 0
|
||||
|
||||
with store.connection(wiki_env.db_path) as conn:
|
||||
assert sorted(store.list_source_files(conn, "workspace")) == ["notes/zalohy.md"]
|
||||
assert store.orphan_vector_ids(conn) == []
|
||||
assert store.bm25_ranked_ids(conn, "tokio", 10) == []
|
||||
stats = _stats(wiki_env)
|
||||
assert stats["vectors"] == stats["chunks"]
|
||||
|
||||
|
||||
def test_live_lock_makes_the_run_a_silent_no_op(wiki_env, fake_embedder):
|
||||
"""Verification 3: a second concurrent sync exits 0 without writing."""
|
||||
wiki_env.write_file("notes/zalohy.md", DOC)
|
||||
wiki_sync.LOCK_PATH.write_text(
|
||||
json.dumps({"pid": os.getpid(), "started_at": wiki_sync.datetime.now().astimezone().isoformat()}),
|
||||
encoding="utf-8",
|
||||
)
|
||||
|
||||
assert _run() == 0
|
||||
|
||||
assert not wiki_env.db_path.exists()
|
||||
assert wiki_env.log_text() == ""
|
||||
|
||||
|
||||
def test_stale_lock_is_reclaimed_and_logged(wiki_env, fake_embedder):
|
||||
wiki_env.write_file("notes/zalohy.md", DOC)
|
||||
dead_pid = 999_999
|
||||
wiki_sync.LOCK_PATH.write_text(
|
||||
json.dumps({"pid": dead_pid, "started_at": wiki_sync.datetime.now().astimezone().isoformat()}),
|
||||
encoding="utf-8",
|
||||
)
|
||||
|
||||
assert _run() == 0
|
||||
|
||||
assert "stale lock, reclaiming" in wiki_env.log_text()
|
||||
assert _stats(wiki_env)["files"] == 1
|
||||
assert not wiki_sync.LOCK_PATH.exists()
|
||||
|
||||
|
||||
def test_unreadable_lock_is_treated_as_stale(wiki_env, fake_embedder):
|
||||
wiki_env.write_file("notes/zalohy.md", DOC)
|
||||
wiki_sync.LOCK_PATH.write_text("not json", encoding="utf-8")
|
||||
|
||||
assert _run() == 0
|
||||
assert _stats(wiki_env)["files"] == 1
|
||||
|
||||
|
||||
def test_coverage_report_names_uncovered_directories(wiki_env, fake_embedder):
|
||||
"""Verification 6: markdown outside `paths` must surface as a log line."""
|
||||
wiki_env.write_file("notes/zalohy.md", DOC)
|
||||
wiki_env.write_file("recepty/gulas.md", "# Guláš\n\nCibule na dva kusy hovězího.\n")
|
||||
# The skill's own clone storage holds markdown but is never a candidate.
|
||||
(wiki_env.wiki_dir / "remote" / "index").mkdir(parents=True)
|
||||
(wiki_env.wiki_dir / "remote" / "index" / "foreign.md").write_text("# Cizí\n", encoding="utf-8")
|
||||
|
||||
assert _run() == 0
|
||||
|
||||
coverage = [line for line in wiki_env.log_text().splitlines() if "coverage" in line]
|
||||
assert len(coverage) == 1
|
||||
assert "recepty" in coverage[0]
|
||||
assert "wiki" not in coverage[0]
|
||||
|
||||
|
||||
def test_degraded_mode_then_recovery_without_file_change(wiki_env, monkeypatch):
|
||||
"""Verification 4: no Ollama -> FTS-only pending; once back, step 4b fills the vectors."""
|
||||
wiki_env.write_file("notes/zalohy.md", DOC)
|
||||
|
||||
# The fixture config points at embed.invalid, so no embedder is reachable here.
|
||||
assert _run() == 1
|
||||
stats = _stats(wiki_env)
|
||||
assert stats["chunks"] > 0
|
||||
assert stats["vectors"] == 0
|
||||
assert stats["pending"] == stats["chunks"]
|
||||
assert "embeddings unavailable" in wiki_env.log_text()
|
||||
|
||||
with store.connection(wiki_env.db_path) as conn:
|
||||
assert len(store.bm25_ranked_ids(conn, "zaloh*", 10)) >= 1 # FTS works meanwhile
|
||||
|
||||
from conftest import FakeEmbedder
|
||||
|
||||
monkeypatch.setattr(wiki_sync, "OllamaEmbedder", FakeEmbedder)
|
||||
assert _run() == 0 # no file changed, yet the pending chunks get vectors
|
||||
|
||||
stats = _stats(wiki_env)
|
||||
assert stats["pending"] == 0
|
||||
assert stats["vectors"] == stats["chunks"]
|
||||
|
||||
|
||||
def test_meta_mismatch_blocks_indexing_until_full(wiki_env, fake_embedder):
|
||||
"""Verification 5: never mix vectors from two contracts."""
|
||||
wiki_env.write_file("notes/zalohy.md", DOC)
|
||||
assert _run() == 0
|
||||
before = _stats(wiki_env)
|
||||
|
||||
wiki_env.config_path.write_text(
|
||||
wiki_env.config_path.read_text(encoding="utf-8").replace(
|
||||
"model: qwen3-embedding:0.6b", "model: qwen3-embedding:4b"
|
||||
),
|
||||
encoding="utf-8",
|
||||
)
|
||||
wiki_env.write_file("notes/new.md", OTHER_DOC)
|
||||
|
||||
assert _run() == 1
|
||||
assert "index identity mismatch" in wiki_env.log_text()
|
||||
assert _stats(wiki_env) == before # nothing indexed under the new contract
|
||||
|
||||
assert _run(["--full"]) == 0
|
||||
with store.connection(wiki_env.db_path) as conn:
|
||||
assert store.read_meta(conn)["embedding_model"] == "qwen3-embedding:4b"
|
||||
assert sorted(store.list_source_files(conn, "workspace")) == ["notes/new.md", "notes/zalohy.md"]
|
||||
assert store.orphan_vector_ids(conn) == []
|
||||
|
||||
|
||||
def test_full_rebuild_wipes_and_reindexes(wiki_env, fake_embedder):
|
||||
wiki_env.write_file("notes/zalohy.md", DOC)
|
||||
assert _run() == 0
|
||||
before = _stats(wiki_env)
|
||||
|
||||
assert _run(["--full"]) == 0
|
||||
|
||||
assert _stats(wiki_env) == before
|
||||
with store.connection(wiki_env.db_path) as conn:
|
||||
assert store.orphan_vector_ids(conn) == []
|
||||
|
||||
|
||||
def test_meta_is_written_on_a_fresh_index(wiki_env, fake_embedder):
|
||||
wiki_env.write_file("notes/zalohy.md", DOC)
|
||||
assert _run() == 0
|
||||
|
||||
config = load_config(wiki_env.config_path)
|
||||
with store.connection(wiki_env.db_path) as conn:
|
||||
assert store.read_meta(conn) == expected_meta(config.embedding)
|
||||
|
||||
|
||||
def test_unknown_source_argument_is_reported(wiki_env, fake_embedder):
|
||||
wiki_env.write_file("notes/zalohy.md", DOC)
|
||||
assert _run(["--source", "nope"]) == 1
|
||||
assert "unknown source" in wiki_env.log_text()
|
||||
|
||||
|
||||
def test_batching_respects_the_configured_size(wiki_env, fake_embedder):
|
||||
"""batch: 4 in the fixture config — the embedder must be called in chunks of 4."""
|
||||
for index in range(6):
|
||||
wiki_env.write_file(f"notes/doc{index}.md", f"# Dokument {index}\n\n{DOC}\n")
|
||||
|
||||
assert _run() == 0
|
||||
|
||||
calls = [len(call) for embedder in fake_embedder for call in embedder.calls]
|
||||
assert calls, "the embedder was never called"
|
||||
assert max(calls) <= 4
|
||||
Reference in New Issue
Block a user