nanobot: 2026-09-10 12:33:37
This commit is contained in:
lachtan
@@ -1,171 +0,0 @@
|
||||
---
|
||||
name: llm-wiki
|
||||
description: >
|
||||
Build and maintain an LLM-curated personal knowledge base — the "LLM Wiki" pattern. Use whenever
|
||||
the user wants to ingest a source (paper, article, transcript, PDF, notes) into a persistent,
|
||||
compounding knowledge base, ask a question against the accumulated notes, lint or audit such a
|
||||
base, or initialize a new one. Applies even when the user doesn't say "wiki" — any time they
|
||||
accumulate textual sources over time and want them organized. A deliberate, standalone store,
|
||||
distinct from agent memory (note / keep / MEMORY.md).
|
||||
---
|
||||
|
||||
# LLM Wiki
|
||||
|
||||
A skill for building and maintaining an LLM-curated knowledge base inside a project, following the pattern Andrej Karpathy described in his April 2026 gist. The wiki is a directory of markdown files that the LLM owns and maintains; the user curates sources and asks questions, and the LLM does the bookkeeping.
|
||||
|
||||
## Nanobot adaptation — read this first
|
||||
|
||||
This skill is ported to run on this nanobot. The generic docs below describe a project-local wiki; the rules here pin it to this nanobot and override anything that conflicts.
|
||||
|
||||
**⚠️ On any add/save/ingest request: capture only.** Write the source to `cml/raw/<slug>.md`, confirm in one short line, and **STOP** — no reads, no scripts, no compiles. Full rules and the escape hatch in "Capture vs compile" below — read it before acting on any such request.
|
||||
|
||||
- **One wiki, fixed location.** This nanobot has exactly one wiki, at `cml/wiki/`, with raw sources at `cml/raw/` — both relative to your working directory (the workspace). Wherever the docs below say `wiki/` or `<project-root>/wiki/`, read `cml/wiki/`; `raw/` means `cml/raw/`.
|
||||
- **Run scripts with `uv run`, never bare `python`.** Always `uv run skills/llm-wiki/scripts/<script>.py …`. The scripts default to `cml/wiki`, so for most you can omit the path argument.
|
||||
- **Bootstrap once:** `uv run skills/llm-wiki/scripts/init_wiki.py . --wiki-dir cml/wiki --raw-dir cml/raw` creates `cml/wiki/` + `cml/raw/`. Idempotent — safe to re-run.
|
||||
- **Separate store — not agent memory.** The wiki is a deliberate, standalone knowledge base of curated sources. It is **not** the agent's memory: keep it distinct from `note`, `keep`, and `MEMORY.md`, and do not fold wiki content into them (or vice versa). The Dream processor must **not** touch `cml/` — it is outside the memory and skills Dream curates. Do not wire the wiki into `MEMORY.md`; its location is documented here.
|
||||
- **Lint is report-only — a lint turn never mutates the wiki.** A lint request ("lint", "what's broken", "clean up the wiki", or a HEARTBEAT lint) means exactly: run `wiki_lint.py` and — if the graph layer exists — `wiki_graph_lint.py`, present the findings as a summary of *proposed* edits, and **STOP the turn**. Forbidden during a lint turn (all of it is fixing, done later in a separate approved turn): creating or editing any page under `cml/wiki/`, writing debug/throwaway scripts, regenerating the graph (`wiki_graph_extract.py`), re-running lint in a loop, updating `index.md` / `log.md`. If you catch yourself editing a page or re-running lint to check your own fix, you are fixing inline — stop. Fixes happen only after the user approves, one category at a time (see the lint workflow).
|
||||
- **Language.** This skill body is English; reply to the user in the user's own language.
|
||||
- **Scope: local PoC.** Single machine, versioned by the git repo running over the workspace. No shared remote, no multi-client sync.
|
||||
|
||||
## Capture vs compile — the background pipeline
|
||||
|
||||
Ingest is split into two phases so the interactive turn stays instant. Full agentic compile takes a minute or more; doing it inline made capture unusable.
|
||||
|
||||
- **Capture (interactive default — instant).** When the user wants to add a source ("save this", "ingest this", "add X to the wiki"), do exactly three things and nothing more:
|
||||
1. Write the source into `cml/raw/<slug>.md` (pick a descriptive `<slug>`). Inline text → write as-is. URL → write the URL as-is (the compile step will fetch it).
|
||||
2. Confirm in **one short line** ("zachyceno — zkompiluju na pozadí").
|
||||
3. **STOP the turn.**
|
||||
Forbidden during a capture turn (all of this is compile, done later in the background): reading `SCHEMA.md` / `index.md` / any wiki page, creating or editing pages under `cml/wiki/`, running `init_wiki.py` / `wiki_lint.py` / `wiki_graph_*` / any script, rebuilding the graph, updating `index.md` or `log.md`. If you catch yourself about to read the schema or write a page, you are doing compile inline — stop and just capture.
|
||||
- **Escape hatch.** Only if the user *explicitly* says "compile now" / "synchronously" / "do it now" / "hned" do you run the full Compile workflow inline in this turn. A normal "add this to my wiki" is **not** an escape hatch — it is capture.
|
||||
- **Compile (drain — background, batched).** A system cron runs `scripts/wiki_compile.py` every minute; when `cml/raw/` has pending sources it invokes this skill with a drain goal. Compile processes **every** pending source in one batch (one index/graph update for many sources), then moves each processed source into `cml/raw/_done/`. This is the existing ingest workflow (below) applied per pending source. **Idempotency:** if `cml/wiki/sources/<slug>.md` already exists for a source, treat it as already compiled — skip re-processing and move the raw file to `cml/raw/_done/`. Always move a source out of `cml/raw/` once handled so the next cron tick doesn't re-process it.
|
||||
- **`cml/raw/` layout.** Regular files directly in `cml/raw/` = the **pending inbox**. `cml/raw/_done/` = processed sources (move here after a successful ingest). `cml/raw/_hard/` = sources held back as ambiguous/conflicting (don't force-compile these; record why in `log.md`). `cml/raw/assets/` = downloaded images, never a source. The pre-check and compile both ignore `_done/`, `_hard/`, and `assets/`.
|
||||
|
||||
## Architecture: three layers, three operations
|
||||
|
||||
The wiki has three layers and three operations. Internalize this vocabulary because the rest of the skill assumes it.
|
||||
|
||||
The three layers are **raw sources** (the user's curated source material — articles, papers, PDFs, transcripts; immutable, the LLM reads but never modifies them), **the wiki** (a directory of LLM-generated markdown pages — entity pages, concept pages, comparisons, summaries; the LLM owns this layer entirely), and **the schema** (a `SCHEMA.md` file at the wiki root that documents the conventions for this particular wiki — page types, naming rules, tag taxonomy, ingest workflow customizations; co-evolved with the user).
|
||||
|
||||
The three operations are **ingest** (a new source arrives; the LLM reads it, writes a summary page, updates relevant entity and concept pages, appends to the log), **query** (the user asks a question; the LLM navigates the wiki via the index, reads the relevant pages, and synthesizes an answer — often filing the answer back as a new page so the exploration compounds), and **lint** (a periodic health check; the LLM scans for contradictions, stale claims, orphan pages, missing concepts, broken links).
|
||||
|
||||
For the canonical write-up of these operations, read `references/architecture.md`. For the step-by-step procedures, read `references/ingest-workflow.md`, `references/query-workflow.md`, and `references/lint-workflow.md` as needed.
|
||||
|
||||
## Graph layer (compiled, optional)
|
||||
|
||||
Pages can carry typed `graph:` metadata in frontmatter. A bundled extractor compiles every page into `wiki/graph/`: `nodes.jsonl`, `edges.jsonl`, `graph.sqlite`, `graph.graphml`. **Markdown is canonical**; the graph is a regenerable index. Pages without `graph:` still appear as nodes (derived from their `type`/`kind`) and contribute low-confidence `mentions` edges from body wikilinks. Typed semantic edges (e.g. `founded`, `proposed`, `depends_on`) require an explicit source and evidence quote — never emit one inferred from training data.
|
||||
|
||||
The conventions for the graph layer (predicate vocabulary, node id format, required fields) live in `wiki/graph/ontology.yaml`. The full reference is `references/graph-workflow.md`. Run the bundled scripts after substantive ingests:
|
||||
|
||||
```bash
|
||||
uv run skills/llm-wiki/scripts/wiki_graph_lint.py cml/wiki/ # check ontology + evidence + alias collisions
|
||||
uv run skills/llm-wiki/scripts/wiki_graph_extract.py cml/wiki/ # rebuild nodes.jsonl, edges.jsonl, graph.sqlite, graph.graphml
|
||||
uv run skills/llm-wiki/scripts/wiki_graph_query.py cml/wiki/ neighbors --node product:konvy
|
||||
```
|
||||
|
||||
If `wiki/graph/ontology.yaml` does not exist, the wiki is pre-graph and you should treat the graph step as a no-op — don't fabricate it.
|
||||
|
||||
## Default project layout
|
||||
|
||||
The wiki is at a fixed location on this nanobot (`cml/wiki/`, `cml/raw/`):
|
||||
|
||||
```
|
||||
<workspace>/
|
||||
├── cml/
|
||||
│ ├── wiki/
|
||||
│ │ ├── SCHEMA.md ← conventions, the "config file" — read this FIRST
|
||||
│ │ ├── index.md ← entry point: catalog of all pages with one-line summaries
|
||||
│ │ ├── log.md ← append-only chronological log of ingests/queries/lints
|
||||
│ │ ├── indexes/ ← (appears once index.md shards) per-category indexes
|
||||
│ │ ├── entities/ ← pages about specific things (people, products, papers, places)
|
||||
│ │ ├── concepts/ ← pages about ideas, methods, frameworks
|
||||
│ │ ├── sources/ ← per-source summary pages (one per ingested source)
|
||||
│ │ └── synthesis/ ← cross-cutting analyses, comparisons, query results filed back
|
||||
│ └── raw/ ← the user's source material (PDFs, .md clippings, images)
|
||||
│ └── assets/ ← downloaded images referenced by raw clippings
|
||||
└── ...
|
||||
```
|
||||
|
||||
## The scalability discipline
|
||||
|
||||
The single biggest failure mode of the LLM Wiki pattern is the wiki itself becoming a context bottleneck. Naive implementations break around a few hundred pages: the LLM either reads too many pages per query or starts hallucinating because it skipped the relevant ones. This skill's design is shaped almost entirely by avoiding that failure. The principles below are non-negotiable; ignoring them is what makes the pattern collapse at scale.
|
||||
|
||||
**Atomic pages.** Every wiki page is about one concept and stays small — soft cap 400 lines or roughly 2,000 words, hard cap 800 lines. When a page outgrows this, split it: extract sub-concepts into their own pages and have the parent link to them. A page that takes up 30% of the context window on its own is a design smell.
|
||||
|
||||
**Index-first navigation.** Never grep or glob the wiki blindly when answering a query. Always read `index.md` (or the relevant sharded index under `indexes/`) first to identify candidate pages, then drill into only those. The index is engineered to be cheap to read — one line per page, no bodies — and it is the cache that makes the whole pattern scalable.
|
||||
|
||||
**Sharded indexes.** When `index.md` itself exceeds ~300 lines or the wiki passes ~150 pages, shard it: move category-specific entries into `indexes/<category>.md` files (e.g. `indexes/entities.md`, `indexes/concepts.md`, `indexes/sources.md`, or finer domain shards), and have the top-level `index.md` become a directory of those shards. Now reading the index is a two-step lookup but each step is bounded.
|
||||
|
||||
**YAML frontmatter on every page.** Every wiki page begins with frontmatter that includes at minimum `type`, `tags`, `sources`, and `updated`. The bundled `wiki_search.py` script can filter on these without reading page bodies. See `references/page-conventions.md`.
|
||||
|
||||
**Surgical edits, not rewrites.** When updating a page (e.g. adding a new cross-reference because a freshly ingested source mentions an existing entity), use `str_replace` to touch only the relevant section. Rewriting whole pages is slow, expensive in tokens, and risks losing prior nuance.
|
||||
|
||||
**Backlink discovery via grep.** To find every page that references a given entity, run `grep -rl "\[\[entity-name\]\]" cml/wiki/` rather than reading pages to look for mentions. The bundled scripts make this easy.
|
||||
|
||||
**Chunked source ingestion.** Large raw sources (long PDFs, book chapters, lengthy transcripts) should be read in chunks during ingest, not loaded whole. The ingest workflow handles this — see `references/ingest-workflow.md`.
|
||||
|
||||
**Search script for large wikis.** Once the wiki passes ~300 pages, plain index lookup may not surface the right pages for fuzzy queries. Use `scripts/wiki_search.py` for BM25-ranked retrieval with optional frontmatter filters. It's a fallback, not the default — index-first is still cheaper when it works.
|
||||
|
||||
**Stats.** `uv run skills/llm-wiki/scripts/wiki_stats.py` gives a quick summary of page count by type and link density — useful for deciding when to shard the index.
|
||||
|
||||
For the full scaling playbook including thresholds and migration steps, read `references/scaling-playbook.md`.
|
||||
|
||||
## Initializing a new wiki
|
||||
|
||||
If the project does not contain a `wiki/` directory (or whatever the user calls theirs), run the bootstrap script:
|
||||
|
||||
```bash
|
||||
uv run skills/llm-wiki/scripts/init_wiki.py . --wiki-dir cml/wiki --raw-dir cml/raw
|
||||
```
|
||||
|
||||
This creates the directory structure, drops in templates for `SCHEMA.md`, `index.md`, and `log.md`, and seeds a starter page convention document. After bootstrapping, briefly walk the user through the schema and ask whether they want to customize anything (e.g. domain-specific page types, custom tags) before the first ingest. The schema is meant to evolve — encourage editing it.
|
||||
|
||||
Do **not** wire the wiki into an agent-memory file (`MEMORY.md` / `AGENTS.md`) on this nanobot — see the Nanobot adaptation rules: the wiki is a separate store and its location is documented in this SKILL.md, which the skill description already surfaces.
|
||||
|
||||
## The ingest workflow (summary)
|
||||
|
||||
**STOP — do not run this for a plain "add this to my wiki" request.** This is the **compile (drain)** step. It runs only from the background cron (the drain goal) or the explicit "compile now" escape hatch. If the user just asked to add/save/ingest a source, you are in *capture* — write to `cml/raw/` and stop (see "Capture vs compile"). The source is already captured in `cml/raw/` when compile runs, so don't re-write it; read it from there.
|
||||
|
||||
The full workflow is in `references/ingest-workflow.md`; what follows is the shape of it. Read the source — chunked if large — and write a single source-summary page in `cml/wiki/sources/`, named after the source slug, with full frontmatter and citations back to the raw file. Then identify which existing entity and concept pages this source touches; for each, surgically update the relevant section using `str_replace` rather than rewriting. Identify any new entities or concepts the source introduces and create new pages for them, linking from related existing pages so they don't become orphans. Update `index.md` (or the relevant shard) with the new pages. Append a single line to `log.md` with the date, operation type, and source title. After the source is fully ingested, move its raw file into `cml/raw/_done/`. When run interactively, discuss the takeaways with the user as a final step — what surprised them, what's worth following up on — and offer to file that discussion back as a synthesis page.
|
||||
|
||||
**Page naming — avoid slug collisions.** A source page and a concept/entity page must not claim the same slug, or their `[[wikilinks]]` collide (e.g. a paper *and* the concept it introduces both wanting `rotary-position-embedding.md`). Name **concept/entity pages after the short name of the idea** (`rope.md`, `transformer.md`), and **source pages after the source's own slug** (the title/filename of the raw source). If two pages still resolve to the same slug, suffix one to disambiguate.
|
||||
|
||||
## The query workflow (summary)
|
||||
|
||||
Full version in `references/query-workflow.md`. To answer a query against the wiki: read `index.md` (or the relevant shard) first; identify candidate pages from one-line summaries; read those pages (and any backlinks they list that look relevant); synthesize the answer with `[[wikilink]]` citations to the pages you used; offer to file the synthesized answer back into `wiki/synthesis/` so future queries benefit. If the index doesn't surface good candidates, fall back to `uv run skills/llm-wiki/scripts/wiki_search.py "query terms"` for ranked retrieval. If the wiki appears to lack coverage of the topic, say so plainly rather than confabulating — flag it as a candidate ingest target.
|
||||
|
||||
## The lint workflow (summary)
|
||||
|
||||
**STOP — a lint turn reports, it does not fix.** See the report-only gate in "Nanobot adaptation". Run the scripts, present the findings, stop. The scripts are fast (sub-second on a small wiki); if a lint turn runs long, you have wrongly slipped into fixing.
|
||||
|
||||
Full version in `references/lint-workflow.md`. Lint is best run on a cadence (after every N ingests or weekly), not on every operation. Run `uv run skills/llm-wiki/scripts/wiki_lint.py` for structural issues (orphan pages, broken `[[wikilinks]]`, oversized pages, missing/malformed frontmatter, stale `updated` dates) and — if the graph layer exists — `uv run skills/llm-wiki/scripts/wiki_graph_lint.py` for typed-edge issues. For the semantic checks that need an LLM (contradictions with older claims, concepts mentioned but lacking a page, coverage gaps) read at most the ~10 most-recently-updated pages — no blind globbing. Present everything as proposed edits for the user to approve; never apply them in the lint turn — the wiki is the user's, and silent rewrites erode trust.
|
||||
|
||||
**When the user approves fixes (a later turn):** fix in bounded batches, one category at a time. Do **not** re-read the lint script to reverse-engineer it, and do **not** loop edit↔lint — run lint once at the end to confirm, and if findings remain, report them and ask rather than continuing blind. Graph-lint findings are interdependent (fixing one edge can create an orphan); clearing a large backlog is its own task, not part of a lint turn.
|
||||
|
||||
## Failure modes to guard against
|
||||
|
||||
- **Silent corruption:** every wiki claim must carry a `sources:` frontmatter entry pointing back to the raw file. When in doubt during ingest, hedge ("the source claims X") rather than asserting.
|
||||
- **Wiki-reads-its-own-output drift:** during ingest, when updating an existing page, re-read the relevant raw source for the existing claim before merging — don't take the wiki's word for what the source said.
|
||||
|
||||
## Reference files
|
||||
|
||||
The reference files are the source of truth for the detailed procedures. Read them when the relevant operation is happening, not preemptively.
|
||||
|
||||
- `references/architecture.md` — the three layers and three operations explained in depth, with examples of page formats and the rationale behind each design choice
|
||||
- `references/ingest-workflow.md` — the step-by-step ingest procedure including chunked reading for large sources and the per-page-type templates
|
||||
- `references/query-workflow.md` — navigation patterns from index → page → backlinks, when to fall back to the search script, and how to file answers back as synthesis pages
|
||||
- `references/lint-workflow.md` — what to check, how to present findings, and the cadence
|
||||
- `references/page-conventions.md` — frontmatter schema, page naming, link syntax, page-type definitions, sizing rules
|
||||
- `references/scaling-playbook.md` — thresholds at which to shard the index, when to introduce the search script, signals that the wiki has outgrown its current conventions
|
||||
- `references/graph-workflow.md` — the optional graph layer: ontology, frontmatter schema, when to add typed edges vs plain wikilinks, and the extract/lint/query flow
|
||||
|
||||
## Templates
|
||||
|
||||
The templates in `assets/` are starting points — they get copied into the user's wiki on bootstrap and then evolve under the user's editing.
|
||||
|
||||
- `assets/SCHEMA.md.template` — the canonical schema document for a new wiki
|
||||
- `assets/index.md.template` — the empty index file
|
||||
- `assets/log.md.template` — the empty log file
|
||||
- `assets/page.md.template` — a generic wiki page with the frontmatter scaffold
|
||||
- `assets/ontology.yaml.template` — starter graph ontology copied to `wiki/graph/ontology.yaml`
|
||||
- `assets/graph_README.md.template` — explainer for `wiki/graph/` (canonical vs generated files)
|
||||
- `assets/graph_gitignore.template` — `.gitignore` for `wiki/graph/` (ignores `graph.sqlite` and `graph.graphml` by default)
|
||||
@@ -1,113 +0,0 @@
|
||||
# Wiki Schema
|
||||
|
||||
This file is the configuration for this wiki. It documents the conventions, page types, tag taxonomy, and any workflow customizations. The LLM reads this first when entering the wiki, and its conventions override the defaults documented in the `llm-wiki` skill.
|
||||
|
||||
This file is **co-evolved with the user**. When the LLM notices a recurring pattern in your edits or feedback that isn't here, it will propose adding it. When something here stops fitting, prune it.
|
||||
|
||||
## Wiki location
|
||||
|
||||
- Wiki root: `wiki/`
|
||||
- Raw sources: `raw/`
|
||||
- Asset/image storage: `raw/assets/`
|
||||
|
||||
## Page types
|
||||
|
||||
This wiki uses these page types, each with a dedicated subdirectory:
|
||||
|
||||
- `source` (in `wiki/sources/`) — one summary page per ingested source.
|
||||
- `entity` (in `wiki/entities/`) — pages about specific things: people, papers, products, places, organizations.
|
||||
- `concept` (in `wiki/concepts/`) — pages about ideas, methods, frameworks, abstractions.
|
||||
- `synthesis` (in `wiki/synthesis/`) — cross-cutting analyses, comparisons, query answers filed back.
|
||||
|
||||
Add additional types here as the wiki evolves.
|
||||
|
||||
## Tag taxonomy
|
||||
|
||||
(Empty initially. Add tags here as you adopt them, with one-line descriptions. Keep this list small and disciplined — a wiki with 200 tags has effectively no tags.)
|
||||
|
||||
Example structure:
|
||||
- `methodology` — pages about research or analytical methods.
|
||||
- `open-question` — pages or sections that flag unresolved questions.
|
||||
- `contested` — pages where sources contradict.
|
||||
|
||||
## Page sizing
|
||||
|
||||
- Soft cap: 400 lines / ~2,000 words. Consider splitting beyond this.
|
||||
- Hard cap: 800 lines. Must split.
|
||||
|
||||
## Frontmatter requirements
|
||||
|
||||
Every page must have:
|
||||
- `type`
|
||||
- `title`
|
||||
- `tags`
|
||||
- `created`
|
||||
- `updated`
|
||||
|
||||
Plus type-specific:
|
||||
- `source` pages: `authors`, `url` (if applicable), `raw`, `ingested`
|
||||
- Non-source pages: `sources` listing the source-summary pages drawn from
|
||||
|
||||
## Optional graph metadata
|
||||
|
||||
Pages may declare typed graph metadata under a top-level `graph:` key. This is the source of truth for the compiled knowledge graph under `wiki/graph/`. Markdown remains canonical; the graph is a regenerable index. Pages without `graph:` still appear as nodes (derived from `type`/`kind`) and still contribute `mentions` edges from body `[[wikilinks]]`.
|
||||
|
||||
```yaml
|
||||
graph:
|
||||
node_id: person:praney-behl # optional; default <node_type>:<slug>
|
||||
node_type: person # optional; default mapped from type/kind via ontology
|
||||
canonical: true # mark as canonical when multiple slugs alias the same entity
|
||||
aliases: [Praney, praney@example.com]
|
||||
relationships:
|
||||
- predicate: founded
|
||||
object: company:seedblocks
|
||||
source: praney-founder-context-dump # source-page slug
|
||||
evidence: "Solo technical founder and sole director..."
|
||||
confidence: high # high | medium | low
|
||||
status: current # current | historical | proposed | disputed | superseded
|
||||
# optional:
|
||||
# valid_from: 2025-01-15
|
||||
# valid_to: 2026-03-01
|
||||
# notes: "..."
|
||||
# raw_ref: "raw/founder-dump.md#L42"
|
||||
# contradicts: edge-id-or-source-slug
|
||||
# supersedes: edge-id-or-source-slug
|
||||
```
|
||||
|
||||
Required fields on every relationship: `predicate`, `object`, `source`, `evidence`, `confidence`, `status`. Predicates and the subject/object types they accept are declared in `wiki/graph/ontology.yaml`. Typed semantic edges must be supported by an explicit source — never emit one inferred from training data alone.
|
||||
|
||||
## Index structure
|
||||
|
||||
(Update this section when sharding.)
|
||||
|
||||
Currently flat: a single `wiki/index.md` listing all pages.
|
||||
|
||||
When the wiki passes ~150 pages or `index.md` exceeds 300 lines, shard into `wiki/indexes/<type>.md` and update this section.
|
||||
|
||||
## Graph layer
|
||||
|
||||
The wiki has an optional compiled graph layer under `wiki/graph/`:
|
||||
|
||||
- `wiki/graph/ontology.yaml` — declares node types and predicates. **Tracked.** Edit this when you introduce new predicates or domain types.
|
||||
- `wiki/graph/nodes.jsonl`, `wiki/graph/edges.jsonl` — generated. Track in git only if you want graph diffs in PRs.
|
||||
- `wiki/graph/graph.sqlite` — generated. Gitignored by default.
|
||||
- `wiki/graph/graph.graphml` — generated. Track only if you want to diff it.
|
||||
|
||||
Generation is reproducible from markdown via `scripts/wiki_graph_extract.py`. The graph can be deleted at any time and rebuilt without losing knowledge — markdown is canonical.
|
||||
|
||||
## Workflow customizations
|
||||
|
||||
(Empty initially. Document any deviations from the default ingest/query/lint workflows here.)
|
||||
|
||||
## User preferences
|
||||
|
||||
(Empty initially. As the user expresses style preferences — "always include a 'Why this matters' section on concept pages", "never use bullet lists in summaries", "prefer comparative tables for synthesis pages" — capture them here so they persist across sessions.)
|
||||
|
||||
## Lint cadence
|
||||
|
||||
- Structural lint: after every 5 ingests.
|
||||
- Semantic lint: weekly or after every 20 ingests.
|
||||
- Gap-finding: monthly.
|
||||
- Graph lint + extract: after every ingest that adds typed `graph.relationships`.
|
||||
|
||||
Adjust based on the wiki's growth rate.
|
||||
@@ -1,32 +0,0 @@
|
||||
# Wiki Graph Layer
|
||||
|
||||
This directory holds the compiled knowledge graph derived from the markdown
|
||||
wiki. **Markdown is canonical.** Everything here can be deleted and rebuilt
|
||||
without losing knowledge:
|
||||
|
||||
```bash
|
||||
python scripts/wiki_graph_extract.py wiki/ --out wiki/graph
|
||||
```
|
||||
|
||||
## Files
|
||||
|
||||
| File | Purpose | Tracking |
|
||||
|------|---------|----------|
|
||||
| `ontology.yaml` | Declares node types and predicates the graph recognises. The contract `wiki_graph_lint.py` validates against. | **Tracked. Edit by hand.** |
|
||||
| `nodes.jsonl` | One JSON object per node, sorted by id. | Generated. Track if you want graph diffs in PRs; otherwise gitignore. |
|
||||
| `edges.jsonl` | One JSON object per edge, sorted by id. Includes typed semantic edges, `mentions`, `sourced_from`, and `summarizes_raw`. | Generated. Same trade-off as `nodes.jsonl`. |
|
||||
| `graph.sqlite` | Queryable index used by `wiki_graph_query.py`. Schema: `nodes`, `aliases`, `edges`. | Generated. **Gitignored** — rebuild on demand. |
|
||||
| `graph.graphml` | GraphML export for tools like Gephi or yEd. | Generated. Gitignored by default. |
|
||||
|
||||
## Workflow
|
||||
|
||||
1. Author or edit a wiki page. Add typed `graph.relationships` only when an explicit source supports them.
|
||||
2. Run `python scripts/wiki_graph_lint.py wiki/` — catches unknown predicates, broken object references, missing evidence, alias collisions.
|
||||
3. Run `python scripts/wiki_graph_extract.py wiki/ --out wiki/graph` — regenerates the artifacts above.
|
||||
4. Query with `python scripts/wiki_graph_query.py wiki/ neighbors --node product:konvy` (or `edges`, `path`, `facts`).
|
||||
|
||||
## Anti-patterns
|
||||
|
||||
- **Hand-editing `nodes.jsonl` / `edges.jsonl` / `graph.sqlite`.** Edit the markdown; regenerate.
|
||||
- **Treating graph rows as evidence.** They accelerate navigation. For high-stakes claims, follow the edge's `source` and `evidence` fields back to the wiki page and the raw source.
|
||||
- **Adding typed edges the source doesn't support.** Use a normal `[[wikilink]]` instead — the `mentions` edge captures the connection without overclaiming.
|
||||
@@ -1,2 +0,0 @@
|
||||
graph.sqlite
|
||||
graph.graphml
|
||||
@@ -1,25 +0,0 @@
|
||||
# Wiki Index
|
||||
|
||||
The catalog of all pages in this wiki. Each entry: a wikilink to the page and a one-line summary. The LLM reads this first when answering queries to identify candidate pages.
|
||||
|
||||
Keep summaries tight — one line each. The index is engineered to be cheap to read; a fat index defeats its purpose.
|
||||
|
||||
When this file exceeds ~300 lines or the wiki passes ~150 pages, shard into `wiki/indexes/<type>.md` and replace this file with a directory of shards. See the `scaling-playbook.md` reference in the `llm-wiki` skill for the migration procedure.
|
||||
|
||||
---
|
||||
|
||||
## Sources
|
||||
|
||||
(populated as sources are ingested)
|
||||
|
||||
## Entities
|
||||
|
||||
(populated as entity pages are created)
|
||||
|
||||
## Concepts
|
||||
|
||||
(populated as concept pages are created)
|
||||
|
||||
## Synthesis
|
||||
|
||||
(populated as query answers are filed back)
|
||||
@@ -1,12 +0,0 @@
|
||||
# Wiki Log
|
||||
|
||||
Append-only chronological record of operations on the wiki. Each entry begins with `## [YYYY-MM-DD] <op> | <description>` so it's parseable with `grep "^## \[" log.md | tail -N`.
|
||||
|
||||
Operations:
|
||||
- `ingest` — a source was processed into the wiki.
|
||||
- `query` — a question was answered against the wiki (typically only logged when the answer was filed back as synthesis).
|
||||
- `lint` — a health check was run.
|
||||
- `schema` — the schema was modified.
|
||||
- `shard` — an index was sharded.
|
||||
|
||||
---
|
||||
@@ -1,123 +0,0 @@
|
||||
# Wiki Graph Ontology
|
||||
#
|
||||
# Declares the node types and predicates that the compiled graph layer
|
||||
# (wiki/graph/) recognises. Edit this file when you introduce a new
|
||||
# domain-specific predicate or node type — wiki_graph_lint.py reads it
|
||||
# to validate every typed edge declared in page frontmatter.
|
||||
#
|
||||
# Markdown remains canonical. This file is just the contract that makes
|
||||
# the graph layer machine-checkable.
|
||||
|
||||
node_types:
|
||||
person:
|
||||
maps_from:
|
||||
type: entity
|
||||
kind: person
|
||||
company:
|
||||
maps_from:
|
||||
type: entity
|
||||
kind: company
|
||||
product:
|
||||
maps_from:
|
||||
type: entity
|
||||
kind: product
|
||||
paper:
|
||||
maps_from:
|
||||
type: entity
|
||||
kind: paper
|
||||
place:
|
||||
maps_from:
|
||||
type: entity
|
||||
kind: place
|
||||
organization:
|
||||
maps_from:
|
||||
type: entity
|
||||
kind: organization
|
||||
concept:
|
||||
maps_from:
|
||||
type: concept
|
||||
source:
|
||||
maps_from:
|
||||
type: source
|
||||
synthesis:
|
||||
maps_from:
|
||||
type: synthesis
|
||||
decision:
|
||||
explicit_only: true
|
||||
claim:
|
||||
explicit_only: true
|
||||
raw:
|
||||
explicit_only: true
|
||||
|
||||
predicates:
|
||||
# --- Implicit predicates emitted by the extractor. ---
|
||||
mentions:
|
||||
subject_types: ["*"]
|
||||
object_types: ["*"]
|
||||
requires_evidence: false
|
||||
description: |
|
||||
Low-specificity edge derived from body wikilinks. Use it for
|
||||
navigation, not as evidence of a typed relationship.
|
||||
sourced_from:
|
||||
subject_types: ["*"]
|
||||
object_types: [source]
|
||||
requires_evidence: false
|
||||
description: |
|
||||
Derived from each non-source page's frontmatter `sources:` list.
|
||||
summarizes_raw:
|
||||
subject_types: [source]
|
||||
object_types: ["*"]
|
||||
requires_evidence: false
|
||||
description: |
|
||||
Derived from a source page's frontmatter `raw:` field. Object is
|
||||
the raw file path string, not a wiki node id.
|
||||
|
||||
# --- Typed semantic predicates. Add domain-specific ones below. ---
|
||||
founded:
|
||||
subject_types: [person]
|
||||
object_types: [company, organization]
|
||||
requires_evidence: true
|
||||
owns:
|
||||
subject_types: [person, company, organization]
|
||||
object_types: [company, product, organization]
|
||||
requires_evidence: true
|
||||
contains_product:
|
||||
subject_types: [company, organization]
|
||||
object_types: [product]
|
||||
requires_evidence: true
|
||||
works_on:
|
||||
subject_types: [person]
|
||||
object_types: [product, concept]
|
||||
requires_evidence: true
|
||||
chose:
|
||||
subject_types: [person, company, organization]
|
||||
object_types: [product, concept]
|
||||
requires_evidence: true
|
||||
proposed:
|
||||
subject_types: [person]
|
||||
object_types: [decision, claim]
|
||||
requires_evidence: true
|
||||
competes_with:
|
||||
subject_types: [product, company, organization]
|
||||
object_types: [product, company, organization]
|
||||
requires_evidence: true
|
||||
depends_on:
|
||||
subject_types: [product, concept]
|
||||
object_types: [product, concept]
|
||||
requires_evidence: true
|
||||
authored:
|
||||
subject_types: [person, organization]
|
||||
object_types: [paper, source]
|
||||
requires_evidence: true
|
||||
cites:
|
||||
subject_types: [paper, source, synthesis]
|
||||
object_types: [paper, source]
|
||||
requires_evidence: true
|
||||
contradicts:
|
||||
subject_types: [claim, source, synthesis]
|
||||
object_types: [claim, source, synthesis]
|
||||
requires_evidence: true
|
||||
supersedes:
|
||||
subject_types: [claim, source, decision]
|
||||
object_types: [claim, source, decision]
|
||||
requires_evidence: true
|
||||
@@ -1,26 +0,0 @@
|
||||
---
|
||||
type: <source|entity|concept|synthesis>
|
||||
title: ""
|
||||
tags: []
|
||||
sources: []
|
||||
created: YYYY-MM-DD
|
||||
updated: YYYY-MM-DD
|
||||
---
|
||||
|
||||
# Title
|
||||
|
||||
Lead paragraph: a clear, encyclopedic definition or framing of what this page is about. Should answer "what is this and why does it matter" in one or two sentences.
|
||||
|
||||
## Section 1
|
||||
|
||||
Body content. Use `[[wikilinks]]` liberally to cross-reference other pages. (Frontmatter `sources:` list above uses bare slugs; only the body uses double-bracket wikilinks.)
|
||||
|
||||
## Section 2
|
||||
|
||||
More body content. Hedge claims that aren't yet corroborated by multiple sources ("Source X claims Y, though this is not yet corroborated by other sources in the wiki").
|
||||
|
||||
## Where this fits
|
||||
|
||||
(For source pages.) List the entity and concept pages this source touches:
|
||||
- [[entity-page-1]]
|
||||
- [[concept-page-1]]
|
||||
@@ -1,76 +0,0 @@
|
||||
# Architecture
|
||||
|
||||
This document explains the three-layer / three-operation architecture in detail. The main `SKILL.md` summarises it; this is the reference you reach for when you need to know *why* a design decision exists or how to handle an edge case.
|
||||
|
||||
## The three layers
|
||||
|
||||
### Raw sources
|
||||
|
||||
Raw sources are the user's curated input material. They live in `cml/raw/` (or wherever the project's `SCHEMA.md` declares). They are **immutable** — the LLM reads from them but never modifies them. This immutability is load-bearing: it means the wiki can always be re-derived from the raw sources if it gets corrupted, and it gives the user a stable ground truth they can audit independently.
|
||||
|
||||
What goes in `cml/raw/`: PDFs, web articles converted to markdown (the Obsidian Web Clipper is one popular path), transcripts, code repos, dataset descriptions, screenshots, hand-typed notes the user wants the wiki to incorporate. What does *not* go in `cml/raw/`: anything the LLM generated — that all belongs in the wiki layer.
|
||||
|
||||
A useful convention is one source per file (or per directory if the source has multiple pieces, e.g. a paper plus its appendix), with a slugified filename that ends up matching the wiki's source-summary page name. This makes the back-pointer from the wiki to the raw source trivial.
|
||||
|
||||
### The wiki
|
||||
|
||||
The wiki is a directory of LLM-generated markdown files. The LLM owns this layer entirely — it creates pages, updates them when new sources arrive, maintains cross-references, and keeps everything consistent. The user reads it (typically through Obsidian, but any markdown viewer works); the LLM writes it.
|
||||
|
||||
The wiki directory is conventionally split into subdirectories by **page type**:
|
||||
|
||||
- `cml/wiki/sources/` — one summary page per ingested source. Captures what the source said, in the LLM's words, with a citation back to the raw file. These are append-mostly: you write one when you ingest a source and rarely modify it after.
|
||||
- `cml/wiki/entities/` — pages about specific things: people, products, papers, places, companies, events. Anything that has a proper noun or could plausibly be a Wikipedia article subject. These accumulate updates as new sources mention the entity.
|
||||
- `cml/wiki/concepts/` — pages about ideas, methods, frameworks, abstractions. These are the most heavily cross-referenced pages and the ones most likely to evolve as understanding deepens.
|
||||
- `cml/wiki/synthesis/` — cross-cutting analyses, comparisons, query answers filed back. This is where exploration compounds: a comparison the user asked for becomes a page the next query can build on.
|
||||
|
||||
`SCHEMA.md` may declare additional types (e.g. `cml/wiki/decisions/` for an engineering team, `cml/wiki/characters/` for a fan wiki, `cml/wiki/experiments/` for a research lab). Adding a new type costs nothing — make the directory, document it in the schema, update the index template.
|
||||
|
||||
### The schema
|
||||
|
||||
`SCHEMA.md` lives at the wiki root and is **the configuration file** that turns a generic LLM into a disciplined wiki maintainer for *this specific* knowledge base. It documents the page types in use, the tag taxonomy, the naming conventions, any custom workflow steps, and the user's stylistic preferences (e.g. "always include a 'Why this matters' section on concept pages", "never use bullet lists in summaries").
|
||||
|
||||
The schema is **co-evolved** with the user. On bootstrap it starts from the default template in `assets/SCHEMA.md.template`, but every user will customize it as they discover what fits their domain. When you notice a recurring pattern in the user's edits or feedback that isn't in the schema, propose adding it. When the schema starts contradicting itself or growing unwieldy, propose pruning it.
|
||||
|
||||
The schema is the first file you read when entering an existing wiki. Its conventions override the defaults documented in the skill.
|
||||
|
||||
### The graph (optional, compiled)
|
||||
|
||||
The wiki may carry a fourth layer at `cml/wiki/graph/`: a compiled, queryable view of the typed `graph:` metadata in page frontmatter and the body wikilinks. It contains a hand-edited `ontology.yaml` (the contract: which node types and predicates exist) plus generated artifacts (`nodes.jsonl`, `edges.jsonl`, `graph.sqlite`, `graph.graphml`) produced by `wiki_graph_extract.py`. **Markdown is canonical**; the graph can be deleted and regenerated from the markdown without losing knowledge.
|
||||
|
||||
Its purpose is to make typed, provenance-backed relationships machine-queryable — "who founded what", "what does Konvy depend on", "shortest path from A to B" — without giving up the editability and human-legibility of markdown. Typed edges require an explicit `source` (a source-page slug) and `evidence` quote; the extractor never invents them. Plain `[[wikilinks]]` in the body produce low-confidence `mentions` edges, which are useful for navigation but not for evidence.
|
||||
|
||||
Use the graph layer when the user's questions are predominantly relational and the cost of maintaining typed metadata is paying for itself. Skip it for purely textual wikis. Full reference: `graph-workflow.md`.
|
||||
|
||||
## The three operations
|
||||
|
||||
### Ingest
|
||||
|
||||
A new source has arrived. The user has dropped a file in `cml/raw/` (or pasted content and asked you to file it). The job is to integrate this source into the wiki such that future queries can benefit from it.
|
||||
|
||||
The shape of an ingest: read the source (chunked if large), discuss the key takeaways with the user briefly, write a summary page in `cml/wiki/sources/`, identify which existing pages are touched, surgically update those pages, create new pages for any new entities or concepts, update the relevant index, and append to the log.
|
||||
|
||||
The temptation to skip the discussion step is strong, but resist it — the user's reaction to the takeaways often reveals what should be emphasized in the wiki versus left out. Ingest is not a batch import; it's a collaborative reading.
|
||||
|
||||
For the full procedure, see `ingest-workflow.md`.
|
||||
|
||||
### Query
|
||||
|
||||
The user asks a question. The job is to answer it from the wiki, with citations, and to file the answer back if it represents new synthesis.
|
||||
|
||||
The shape of a query: read the index to identify candidate pages; read those pages; if needed, follow `[[wikilinks]]` from those pages or grep for backlinks; synthesize the answer with `[[wikilink]]` citations; offer to file the answer into `cml/wiki/synthesis/` if it's substantive enough to be worth keeping.
|
||||
|
||||
The compounding effect of the wiki only works if good answers get filed back. A comparison you generated, a connection you discovered, an analysis you produced — these should not evaporate into chat history. Default to offering to file; let the user decline if the answer was too trivial or too transient.
|
||||
|
||||
For the full procedure, see `query-workflow.md`.
|
||||
|
||||
### Lint
|
||||
|
||||
Periodic health check. The job is to find structural and semantic problems before they compound.
|
||||
|
||||
Structural problems are mechanical and the bundled `wiki_lint.py` script catches them: orphan pages with no inbound links, broken `[[wikilinks]]` to nonexistent pages, oversized pages that need splitting, missing or malformed frontmatter, suspicious staleness (a page that hasn't been updated despite many recent ingests touching its topic).
|
||||
|
||||
Semantic problems need the LLM: contradictions between pages, claims that newer sources have superseded, concepts mentioned in many pages but lacking their own page, cross-references that should exist but don't, knowledge gaps the user might want to fill.
|
||||
|
||||
Lint findings are presented as proposed edits, not silent rewrites. The user approves changes. This is essential for trust — a wiki that mutates under the user is not a wiki the user can rely on.
|
||||
|
||||
For the full procedure, see `lint-workflow.md`.
|
||||
@@ -1,126 +0,0 @@
|
||||
# Graph Workflow
|
||||
|
||||
The graph layer is the optional **compiled index** over the markdown wiki. It does not replace the wiki — it sits alongside it under `cml/wiki/graph/` and is reproducible from the markdown at any time. The point is to make typed, provenance-backed relationships machine-queryable while keeping markdown canonical.
|
||||
|
||||
If `cml/wiki/graph/ontology.yaml` is absent, the wiki is pre-graph: don't run extract/lint/query and don't fabricate ontology files. Either propose adding the layer, or proceed without it.
|
||||
|
||||
## What the graph captures
|
||||
|
||||
Three classes of edge come out of an extract:
|
||||
|
||||
1. **Typed semantic edges** declared in a page's `graph.relationships[]` frontmatter. Examples: `founded`, `proposed`, `depends_on`. Each one carries an explicit `source` (a source-page slug), an `evidence` quote, a `confidence` (high/medium/low), and a `status` (current/historical/proposed/disputed/superseded). The extractor never invents these.
|
||||
2. **`mentions` edges** — one per body `[[wikilink]]` (deduplicated per page). Confidence is `low`; they accelerate navigation but should not be cited as evidence of a typed relationship.
|
||||
3. **`sourced_from`** edges — one per slug in a non-source page's frontmatter `sources:` list, pointing at the source page. **`summarizes_raw`** edges — one per source page's `raw:` field, with the raw file path as the (string-literal) object.
|
||||
|
||||
## Frontmatter schema
|
||||
|
||||
```yaml
|
||||
graph:
|
||||
node_id: person:praney-behl # optional; default <node_type>:<slug>
|
||||
node_type: person # optional; default mapped from type/kind via ontology
|
||||
canonical: true # mark canonical when multiple slugs alias the same entity
|
||||
aliases: [Praney, praney@example.com]
|
||||
relationships:
|
||||
- predicate: founded
|
||||
object: company:seedblocks
|
||||
source: praney-founder-context-dump
|
||||
evidence: "Solo technical founder and sole director..."
|
||||
confidence: high
|
||||
status: current
|
||||
# optional:
|
||||
# valid_from: 2025-01-15
|
||||
# valid_to: 2026-03-01
|
||||
# notes: "..."
|
||||
# raw_ref: "cml/raw/founder-dump.md#L42"
|
||||
# contradicts: <node-id-or-edge-id>
|
||||
# supersedes: <node-id-or-edge-id>
|
||||
```
|
||||
|
||||
Required relationship fields: `predicate`, `object`, `source`, `evidence`, `confidence`, `status`.
|
||||
|
||||
`node_id` format is `<node_type>:<slug>`. The default is derived from the page's `type`/`kind` via the ontology's `maps_from` block. `decision`, `claim`, and `raw` are explicit-only — they don't have wiki pages, so they only show up as edge objects. If a typed edge points at one of these, the `wiki_graph_lint.py` flag for "broken object reference" will fire until either (a) you create a page for it, or (b) you add it to the ontology with `explicit_only: true` and accept that lint will continue to flag the reference.
|
||||
|
||||
## The ontology
|
||||
|
||||
`cml/wiki/graph/ontology.yaml` is the contract. It declares:
|
||||
|
||||
- `node_types[*].maps_from` — how page `type`/`kind` projects onto a node type.
|
||||
- `predicates[*]` — the allowed predicates, each with `subject_types`, `object_types`, and `requires_evidence`. `"*"` is a wildcard for either side.
|
||||
|
||||
Edit the ontology when you need a new domain predicate. Re-run `wiki_graph_lint.py` to validate; existing typed edges will be caught if they no longer match.
|
||||
|
||||
## When to add a typed edge vs a plain `[[wikilink]]`
|
||||
|
||||
Add a typed edge when:
|
||||
|
||||
- A specific source explicitly states the relationship.
|
||||
- You can quote a snippet of evidence.
|
||||
- The predicate is meaningful for downstream queries ("who founded what", "what did Stephanie propose").
|
||||
|
||||
Use a plain `[[wikilink]]` when:
|
||||
|
||||
- The relationship is implicit, atmospheric, or you're hedging.
|
||||
- You cannot pin the claim to a single source quote.
|
||||
- The predicate would be `mentions` anyway.
|
||||
|
||||
When in doubt, write the wikilink and skip the typed edge. The lint surfaces missing evidence; it does not punish under-claiming.
|
||||
|
||||
## Extract / lint / query loop
|
||||
|
||||
```bash
|
||||
# Validate the typed metadata first; lint is conservative, never edits.
|
||||
uv run skills/llm-wiki/scripts/wiki_graph_lint.py cml/wiki/
|
||||
|
||||
# Compile to nodes.jsonl, edges.jsonl, graph.sqlite, graph.graphml.
|
||||
uv run skills/llm-wiki/scripts/wiki_graph_extract.py cml/wiki/
|
||||
|
||||
# Navigate.
|
||||
uv run skills/llm-wiki/scripts/wiki_graph_query.py cml/wiki/ neighbors --node product:konvy
|
||||
uv run skills/llm-wiki/scripts/wiki_graph_query.py cml/wiki/ edges --subject person:stephanie-emmanouel
|
||||
uv run skills/llm-wiki/scripts/wiki_graph_query.py cml/wiki/ path --from person:praney-behl --to product:konvy
|
||||
uv run skills/llm-wiki/scripts/wiki_graph_query.py cml/wiki/ facts --about product:konvy
|
||||
```
|
||||
|
||||
`--json` works on both lint and query commands.
|
||||
|
||||
## Ingest workflow integration
|
||||
|
||||
After Step 6 of the standard ingest workflow (after surgical updates and source page creation), run:
|
||||
|
||||
1. If new typed edges were added on the page being ingested, run `wiki_graph_lint.py`. **Interactive:** triage findings with the user before extract. **Drain (headless):** if lint is clean, proceed to extract; if lint reports errors, record them in `log.md` and skip extract for this batch — never silently rewrite typed edges, never block waiting for a user.
|
||||
2. Run `wiki_graph_extract.py` to refresh the compiled artifacts.
|
||||
3. Append a sub-line under the ingest's `log.md` entry:
|
||||
` graph: +N nodes, +M typed edges (predicates: founded, contains_product, ...)`
|
||||
|
||||
Skip extract if this ingest added no `graph:` metadata and created no new pages — the compiled artifacts are unchanged.
|
||||
|
||||
## Query workflow integration
|
||||
|
||||
When the user asks a question that smells relational ("what's connected to X", "who proposed Y", "trace the path from A to B"):
|
||||
|
||||
1. Read the index as usual.
|
||||
2. If `cml/wiki/graph/graph.sqlite` exists and is fresher than the latest log entry, query it for typed edges around the candidate pages — `neighbors`, `edges`, `facts` are the most useful.
|
||||
3. Read the wiki pages behind the relevant nodes/edges. Don't answer from graph rows alone for high-stakes claims; the `evidence` field is a hint, not the source of truth.
|
||||
4. Cite with `[[wikilinks]]` to wiki pages, not graph rows.
|
||||
|
||||
If `graph.sqlite` is stale (older than the most recent ingest in `log.md`), use it as-is and note the staleness — do **not** regenerate inline. Extract is a compile/drain step; the query turn stays read-only, and the background drain refreshes the graph after each ingest.
|
||||
|
||||
## Generated artifact policy
|
||||
|
||||
| File | Canonical? | Default tracking |
|
||||
|------|-----------|------------------|
|
||||
| `cml/wiki/graph/ontology.yaml` | Yes — edit by hand | Tracked |
|
||||
| `cml/wiki/graph/nodes.jsonl` | Generated | Optional — `cml/wiki/graph/.gitignore` does not ignore it; track if you want graph diffs in PRs |
|
||||
| `cml/wiki/graph/edges.jsonl` | Generated | Same as above |
|
||||
| `cml/wiki/graph/graph.sqlite` | Generated | **Gitignored** by default (large, binary) |
|
||||
| `cml/wiki/graph/graph.graphml` | Generated | **Gitignored** by default |
|
||||
|
||||
The bootstrapped `cml/wiki/graph/.gitignore` ignores `graph.sqlite` and `graph.graphml`. Edit it if your team prefers different policy.
|
||||
|
||||
## Anti-patterns
|
||||
|
||||
- **Typed edges without evidence.** Defeats the entire point. Lint will flag them; do not silence.
|
||||
- **Editing `nodes.jsonl` / `edges.jsonl` / `graph.sqlite` by hand.** Edit the markdown; regenerate.
|
||||
- **Inventing ontology entries to make a typed edge "fit".** The ontology should reflect domain reality, not paper over a too-eager edge. Either add the predicate with proper `subject_types`/`object_types`, or use `mentions`.
|
||||
- **Treating graph rows as evidence in answers.** Always cite the wiki page; the graph just told you which wiki page to read.
|
||||
- **Forgetting to regenerate after an ingest.** The graph diverges silently. Tie extract to ingest in muscle memory.
|
||||
@@ -1,131 +0,0 @@
|
||||
# Ingest Workflow
|
||||
|
||||
When a new source arrives, this is the procedure. The order matters — each step builds context for the next.
|
||||
|
||||
## Step 0: Check the schema
|
||||
|
||||
Before anything else, read `cml/wiki/SCHEMA.md`. The user may have customized the page-type structure, the tag taxonomy, the naming conventions, or the ingest workflow itself. Schema overrides everything documented here.
|
||||
|
||||
## Step 1: Place the raw source
|
||||
|
||||
If the source isn't already in `cml/raw/`, place it there. Use a slugified filename: lowercase, hyphens for spaces, no special characters, with the original extension. For web articles, save as `.md` (Obsidian Web Clipper output is ideal). For PDFs, keep the `.pdf`. For transcripts, save as `.md` or `.txt`.
|
||||
|
||||
The slug you pick here will become the slug of the source-summary page in `cml/wiki/sources/`, so make it descriptive and stable.
|
||||
|
||||
## Step 2: Read the source
|
||||
|
||||
For short sources (under ~5,000 words / ~25,000 tokens), read the whole thing in one pass.
|
||||
|
||||
For long sources (papers over ~30 pages, book chapters, multi-hour transcripts), **chunk-read**: read the table of contents or section headers first to build a mental map, then read sections sequentially, summarizing each section in working memory before moving to the next. Do not load the entire raw source into context at once if it would consume more than ~25% of your context window — that leaves no room for the rest of the operation.
|
||||
|
||||
For PDFs specifically, prefer the `pdf-reading` skill if available (it handles the chunking automatically). Otherwise extract text first with `pdftotext` or `pdfminer` and then chunk-read the extracted text.
|
||||
|
||||
For images embedded in the source: read the surrounding text first, then view only the images that the text suggests are load-bearing (a chart referenced in an argument, a diagram of a system, a figure the source explicitly walks through). Don't blindly load every image — many are decorative.
|
||||
|
||||
## Step 3: Discuss the takeaways with the user (interactive only)
|
||||
|
||||
**Skip this step entirely in the background drain** — the compile cron runs headless with no user present, so there is no one to discuss with, and the remaining steps must not depend on a user reaction.
|
||||
|
||||
When run interactively, do this briefly — three or four sentences. Surface what struck you as important, what was surprising, what connects to existing wiki content, and what's worth flagging. The user's reaction shapes the next steps. They might say "skip the methodology section, only the results matter for me" or "we already have a page on this — just update it" or "this contradicts the page on X, flag that prominently".
|
||||
|
||||
If there is no user to react (the background drain, or a hands-off "just process these 20 papers" batch), skip the discussion and be more conservative on the wiki edits — make smaller, safer updates and surface anything ambiguous in the log.
|
||||
|
||||
## Step 4: Identify what's touched
|
||||
|
||||
Before writing anything, do a survey pass against the wiki to determine the impact:
|
||||
|
||||
1. Read `cml/wiki/index.md` (or the relevant shard under `cml/wiki/indexes/`) to identify existing pages this source touches. Look for entity names, concept names, and topical overlap.
|
||||
2. For each potentially-touched page, read the page to confirm. (Read, don't grep — the index summaries can mislead.)
|
||||
3. List, in working memory: existing pages to update, new pages to create, contradictions to flag.
|
||||
|
||||
This survey is what prevents duplication. Without it, you'll create a new page on a topic that already has one under a slightly different name.
|
||||
|
||||
## Step 5: Write the source-summary page
|
||||
|
||||
Create `cml/wiki/sources/<source-slug>.md`. Use the page template (`assets/page.md.template`) as a starting point. Frontmatter should include at minimum:
|
||||
|
||||
```yaml
|
||||
---
|
||||
type: source
|
||||
title: "Original title of the source"
|
||||
authors: ["Author Name"]
|
||||
url: "https://..." # if applicable
|
||||
raw: "cml/raw/<source-slug>.<ext>"
|
||||
ingested: 2026-04-15
|
||||
tags: [tag1, tag2]
|
||||
entities: [entity-page-1, entity-page-2]
|
||||
concepts: [concept-page-1, concept-page-2]
|
||||
---
|
||||
```
|
||||
|
||||
Frontmatter list values are bare slugs — the `[[wikilink]]` syntax goes in the body, not in YAML.
|
||||
|
||||
The body should be the LLM's summary of the source — the key claims, the methodology if relevant, the conclusions, the open questions. Do not paraphrase the entire source; that defeats the purpose. Aim for a summary that captures what a future query would need to know without re-reading the raw file.
|
||||
|
||||
Keep the page atomic — under the 400-line soft cap. If the source is so dense that a single summary page can't capture it, split: one page per major section, each linking to the others, with a parent page that gives the overview.
|
||||
|
||||
End the body with a "Where this fits" section listing `[[wikilinks]]` to the entity and concept pages this source touches. This is the bidirectional link from source → existing structure.
|
||||
|
||||
## Step 6: Update touched pages
|
||||
|
||||
For each existing page identified in Step 4, surgically edit it to incorporate what the new source adds. Use `str_replace`, not full rewrites. The goal is to add a sentence or paragraph in the relevant section, with a `[[wikilinks]]` citation to the new source-summary page.
|
||||
|
||||
Common update patterns:
|
||||
|
||||
- A new source corroborates an existing claim → add a citation: `..., as established in [[paper-X]] and now corroborated by [[paper-Y]]`.
|
||||
- A new source contradicts an existing claim → flag prominently: add a "## Contradictions" section if it doesn't exist, and document the contradiction with both sources cited. Do not silently overwrite the older claim.
|
||||
- A new source adds a new dimension to an existing topic → add a new sub-section, don't dilute the existing prose.
|
||||
- A new source mentions an entity or concept the page already discusses → update the `sources:` frontmatter and add the cross-reference where relevant in the body.
|
||||
|
||||
If updating a page would push it over the 800-line hard cap, that's the signal to split the page (extract the new dimension into its own page, link from the parent). Do that as part of the ingest, not as a deferred lint task.
|
||||
|
||||
## Step 7: Create new pages for new entities and concepts
|
||||
|
||||
For each new entity or concept the source introduces that doesn't have a page yet, decide whether it warrants its own page. Heuristic: if the source mentions it in passing and no other source is likely to expand on it, just mention it inline on a related page. If the source treats it as a first-class topic or you can foresee future sources building on it, create a page.
|
||||
|
||||
New pages need:
|
||||
- Frontmatter with `type`, `tags`, `sources: [[[<source-slug>]]]`, `created`, `updated`.
|
||||
- A body that introduces the entity/concept with what this source said about it.
|
||||
- Inbound links — at least one existing page should `[[wikilink]]` to the new page, otherwise it's an instant orphan. Update the existing page to add the link.
|
||||
|
||||
A new page that nothing links to is a bug in the ingest, not just a lint finding.
|
||||
|
||||
## Step 8: Update the index
|
||||
|
||||
Add entries for any new pages to `cml/wiki/index.md` (or the appropriate shard). Each entry is one line: a wikilink to the page and a one-sentence summary. Keep the summary tight — the index is engineered to be cheap to read, and a fat index defeats the index-first navigation principle.
|
||||
|
||||
If `index.md` exceeds 300 lines after this update, that's the signal to shard. Do it now while the structure is fresh — see `scaling-playbook.md` for the procedure.
|
||||
|
||||
## Step 8b: Refresh the graph layer (only if `cml/wiki/graph/ontology.yaml` exists)
|
||||
|
||||
If the wiki has the optional graph layer:
|
||||
|
||||
1. Add typed `graph.relationships[]` only when the source explicitly supports them (predicate, source-page slug, evidence quote, confidence, status). When uncertain, prefer a plain `[[wikilink]]` in the body — the body wikilink already produces a `mentions` edge.
|
||||
2. Run `uv run skills/llm-wiki/scripts/wiki_graph_lint.py cml/wiki/`. **Interactive:** triage findings with the user before extracting. **Drain (headless):** if lint is clean, proceed to extract; if lint reports errors, record them in `log.md` and skip extract for this batch. Never silently rewrite typed edges, and never block waiting for a user.
|
||||
3. Run `uv run skills/llm-wiki/scripts/wiki_graph_extract.py cml/wiki/` to regenerate `nodes.jsonl`, `edges.jsonl`, `graph.sqlite`, `graph.graphml`.
|
||||
|
||||
Skip extract if this ingest added no `graph:` metadata and created no new pages — the compiled artifacts are unchanged. Full reference: `references/graph-workflow.md`.
|
||||
|
||||
## Step 9: Append to the log
|
||||
|
||||
One line in `cml/wiki/log.md`, with the prefix `## [YYYY-MM-DD] ingest | <source-title>`. Optionally add a sub-line listing the pages touched. If the graph layer was refreshed, add a second sub-line: ` graph: +N nodes, +M typed edges`. The log is parsed by simple unix tools (`grep "^## \[" log.md | tail -10`), so the prefix matters.
|
||||
|
||||
## Step 10: Close the loop with the user
|
||||
|
||||
Tell the user what you did, briefly: "Ingested. Created the source page and a new entity page for X; updated the concept pages for Y and Z. Flagged a contradiction with [[paper-A]] regarding the claim about W."
|
||||
|
||||
If the source revealed something worth following up on (an obvious gap, a question the source raised but didn't answer, a candidate next source to ingest), say so — this is where the wiki's compounding effect comes from.
|
||||
|
||||
## Anti-patterns to avoid
|
||||
|
||||
**Loading the whole source into context at once when it's large.** This is the most common scaling failure. Chunk-read.
|
||||
|
||||
**Rewriting whole pages instead of surgical edits.** This burns tokens, risks losing nuance, and erodes diff quality if the wiki is in git.
|
||||
|
||||
**Creating pages with no inbound links.** Orphans accumulate fast and become invisible. Always link.
|
||||
|
||||
**Ingesting silently in batch mode without surfacing surprises.** Batch ingest is fine, but a one-line "ingested 5 sources, 2 new entity pages, 1 contradiction with [[X]]" summary is the minimum.
|
||||
|
||||
**Treating prior wiki pages as ground truth instead of the raw sources.** When updating an existing claim, re-read the raw source for that claim before merging the new one. Don't compound on the wiki's own paraphrase.
|
||||
|
||||
**Letting the page split decision drift to a future lint pass.** If a page crossed the size cap during this ingest, split it during this ingest.
|
||||
@@ -1,99 +0,0 @@
|
||||
# Lint Workflow
|
||||
|
||||
A health check on the wiki. Best run on a cadence — after every N ingests, weekly, or when the user explicitly requests it — not on every operation. Lint is split into a structural pass (handled by `wiki_lint.py`) and a semantic pass (handled by the LLM directly).
|
||||
|
||||
> **Report-only gate (this nanobot).** A lint turn runs the read steps (0–1, 3–4), presents findings, and **stops**. The mutation steps below — Step 2 (apply fixes), Step 5 (index), Step 6 (log) — happen **only in a separate turn after the user approves**, one category at a time. Inside a lint turn, never create/edit pages, write debug scripts, regenerate the graph, or loop re-lint. When fixing later: bounded batches, do **not** re-read the lint script to reverse-engineer it, re-lint once to confirm — if issues remain, report and ask, don't continue blind. The scripts are sub-second; a long lint turn means you slipped into fixing.
|
||||
|
||||
## Step 0: Check the schema
|
||||
|
||||
`cml/wiki/SCHEMA.md` may declare additional lint rules specific to this wiki (e.g. "every entity page must have an `aliases:` field", "no concept page without at least 2 sources"). Read it.
|
||||
|
||||
## Step 1: Run the structural lint script
|
||||
|
||||
```bash
|
||||
uv run skills/llm-wiki/scripts/wiki_lint.py cml/wiki/
|
||||
```
|
||||
|
||||
If the wiki has the optional graph layer (`cml/wiki/graph/ontology.yaml` exists), also run:
|
||||
|
||||
```bash
|
||||
uv run skills/llm-wiki/scripts/wiki_graph_lint.py cml/wiki/
|
||||
```
|
||||
|
||||
This catches typed-edge problems independently of the structural lint: unknown predicates, missing evidence, broken object references, alias collisions, invalid `confidence`/`status` values, broken `contradicts`/`supersedes` references. Triage findings the same way as structural lint — propose fixes, don't apply them silently. After approved fixes, run `wiki_graph_extract.py` to refresh the compiled artifacts.
|
||||
|
||||
This produces a report covering:
|
||||
|
||||
- **Orphan pages** — pages with no inbound `[[wikilinks]]` from anywhere else in the wiki. Orphans are usually a sign that an ingest forgot to update a parent page. They become invisible because the index-first navigation can't surface them.
|
||||
- **Broken wikilinks** — `[[page-name]]` references pointing to nonexistent pages. Usually a typo or a page that got renamed without updating its referrers.
|
||||
- **Oversized pages** — anything over the 800-line hard cap (or 400-line soft cap, with a warning).
|
||||
- **Frontmatter issues** — pages missing required fields (`type`, `tags`, `sources`, `updated`), or with malformed YAML.
|
||||
- **Stale pages** — pages whose `updated:` date is much older than the most recent ingest that touched their topic. (The script approximates this using the page's tags and the log.)
|
||||
- **Duplicate slugs** — two pages with the same slug in different subdirectories (a sign of an ingest collision that wasn't resolved).
|
||||
|
||||
The script is conservative — it reports findings but doesn't fix them. Present the report to the user.
|
||||
|
||||
## Step 2: Triage the structural findings (propose only; apply in a later approved turn)
|
||||
|
||||
Walk through the findings with the user and propose a fix for each. Proposing is part of the lint turn; **applying** the edits is not — that waits for approval (see the report-only gate above).
|
||||
|
||||
- Orphans: either link them from a sensible parent page (preferred), or determine that the page is genuinely useless and delete it (rare). If many orphans pile up, the index is probably out of date — re-derive the index entries.
|
||||
- Broken links: rename the link to match the actual page, or create the missing page if it should exist, or remove the link if the concept turned out not to warrant a page.
|
||||
- Oversized pages: split. Extract sub-concepts into their own pages, link from the parent, update the index.
|
||||
- Frontmatter issues: add the missing fields. If many pages have the same gap, consider whether the schema should be relaxed or the bootstrap template improved.
|
||||
- Stale pages: read the recently-touched related pages and the relevant raw sources, update the stale page surgically.
|
||||
- Duplicate slugs: one is canonical, the other should be merged in and deleted. Pick the better-named one as canonical and migrate inbound links with `grep` + `str_replace`.
|
||||
|
||||
Present each proposed fix as an edit, not a fait accompli. The user approves.
|
||||
|
||||
## Step 3: Run the semantic pass
|
||||
|
||||
The semantic pass is what the script can't do — it requires reading pages and reasoning about content. The good news is that you don't have to read the whole wiki: focus on the pages most likely to have semantic issues.
|
||||
|
||||
**Recently updated pages.** Read the last ~10 pages that were modified (sorted by `updated:` frontmatter or by the log). Look for:
|
||||
|
||||
- Contradictions with older pages on the same topic. The new claim may have superseded the old one (in which case mark the old as superseded with a note), or both may be true but in different contexts (in which case clarify each), or the new ingest may have been wrong (in which case revert).
|
||||
- Internal contradictions within a page (an ingest layered on a new claim without reconciling against existing prose).
|
||||
- Repeated mentions of an entity or concept that doesn't have its own page yet — candidate for promotion.
|
||||
|
||||
**Highly-linked pages (hubs).** These are the most likely to drift because every ingest touches them. Use `wiki_search.py --top-linked 10` (or grep `[[wikilinks]]` and count) to find them. Read each hub and check that the prose still hangs together and the cross-references still make sense.
|
||||
|
||||
**Pages flagged with explicit uncertainty.** During ingest, the convention is to hedge ("the source claims X, though this is not yet corroborated") rather than assert. The lint pass is when you check whether subsequent ingests have corroborated or contradicted, and update accordingly.
|
||||
|
||||
## Step 4: Surface gaps
|
||||
|
||||
Look at what's referenced but not pageified. Run:
|
||||
|
||||
```bash
|
||||
uv run skills/llm-wiki/scripts/wiki_lint.py cml/wiki/ --suggest-pages
|
||||
```
|
||||
|
||||
This finds entity-like and concept-like names that appear in many pages but lack their own page. The user can decide which to promote and which to leave as inline mentions.
|
||||
|
||||
Also look for what's not covered at all — topics the user has expressed interest in but that haven't been ingested yet. The log can help here ("you ingested 5 sources on diffusion models in March but nothing since; want to refresh?").
|
||||
|
||||
## Step 5: Update the index (fix turn — only after approved fixes)
|
||||
|
||||
After lint fixes, the index almost certainly needs updates: new pages, renamed pages, deleted pages, changed summaries. Re-derive the affected index entries surgically.
|
||||
|
||||
If `index.md` is over 300 lines and hasn't been sharded yet, this is a good moment to do it. See `scaling-playbook.md`.
|
||||
|
||||
## Step 6: Append to the log (fix turn)
|
||||
|
||||
One line: `## [YYYY-MM-DD] lint | <N> structural fixes, <M> semantic fixes, <K> proposed gaps`. Optionally a sub-line listing the highest-impact changes.
|
||||
|
||||
## Cadence
|
||||
|
||||
A reasonable default: structural lint after every 5 ingests, semantic lint weekly or after every 20 ingests, gap-finding monthly. Adjust based on the user's pace and the wiki's volatility. A wiki that's growing fast needs more frequent lint; a stable mature wiki needs less.
|
||||
|
||||
The user may also trigger lint explicitly ("clean up the wiki", "what's broken", "lint pass please"). Treat these as priority — the user is asking because they noticed something.
|
||||
|
||||
## Anti-patterns to avoid
|
||||
|
||||
**Silent rewrites.** Lint findings are proposals. The user approves changes. A wiki that mutates without the user's knowledge stops being trustworthy.
|
||||
|
||||
**Treating lint as cleanup-only.** The semantic pass is also where new connections get made. If you read 10 pages and notice that two of them point at the same underlying idea, that's a synthesis-page candidate, not just a lint finding.
|
||||
|
||||
**Letting the lint report grow until it's overwhelming.** If the report is too long for the user to triage, the cadence is wrong (lint more often) or the wiki has outgrown its conventions (revisit the schema).
|
||||
|
||||
**Skipping lint because everything seems fine.** Silent corruption is the failure mode that's hardest to detect and most damaging. Lint catches it.
|
||||
@@ -1,89 +0,0 @@
|
||||
# Page Conventions
|
||||
|
||||
The structural rules every wiki page follows. The schema may extend or override these for a particular wiki, but these are the defaults.
|
||||
|
||||
## Frontmatter
|
||||
|
||||
Every wiki page begins with YAML frontmatter. The required fields:
|
||||
|
||||
```yaml
|
||||
---
|
||||
type: <source|entity|concept|synthesis|...>
|
||||
title: "Human-readable title"
|
||||
tags: [tag1, tag2, tag3]
|
||||
sources: [source-slug-1, source-slug-2]
|
||||
created: 2026-04-15
|
||||
updated: 2026-04-15
|
||||
---
|
||||
```
|
||||
|
||||
Note that frontmatter list values are **bare slugs**, not `[[wikilinks]]`. The double-bracket syntax is only used in the page body. The bundled scripts treat frontmatter `sources:`, `entities:`, and `concepts:` lists as slug references and resolve them the same way as body wikilinks.
|
||||
|
||||
`type` determines the page-type and which subdirectory the page lives in. Standard types are `source`, `entity`, `concept`, `synthesis`. The schema can declare additional types.
|
||||
|
||||
`title` is the human-readable name. The filename slug is separate and may be a short version of the title. For pages about people, the convention is `Last, First` for sortability; for everything else, natural casing.
|
||||
|
||||
`tags` is a flat list. Tags are how `wiki_search.py` filters and how the user navigates topically in Obsidian. Keep the tag taxonomy small and disciplined — a wiki with 200 tags has effectively no tags. The schema should declare the canonical tag set and the lint script will warn on tags outside it.
|
||||
|
||||
`sources` is the list of source-summary pages this page draws from. Always populated for entity/concept/synthesis pages; for source pages themselves, this field is omitted (the source page is the source).
|
||||
|
||||
`created` and `updated` are dates in ISO format. `updated` is the load-bearing one — it powers the staleness check in lint and the "what's new" view.
|
||||
|
||||
Type-specific fields:
|
||||
- Source pages add `authors`, `url`, `raw` (path to the raw file), `ingested`.
|
||||
- Entity pages may add `aliases` (other names for the same entity), `kind` (person, paper, product, etc.).
|
||||
- Synthesis pages add `question` (the original question, if it was filed back from a query) and `sources_consulted`.
|
||||
|
||||
## Wikilinks
|
||||
|
||||
Cross-references use the `[[page-slug]]` syntax. The slug is the filename without the `.md` extension and without the directory prefix — Obsidian-style. Aliasing is supported: `[[page-slug|display text]]`.
|
||||
|
||||
Every page should have at least one inbound link. New pages without inbound links are orphans and the lint pass will flag them.
|
||||
|
||||
The bundled `wiki_search.py --backlinks <slug>` returns inbound links. To find them manually:
|
||||
|
||||
```bash
|
||||
grep -rln "\[\[<slug>\]\]" cml/wiki/
|
||||
```
|
||||
|
||||
## Page sizing
|
||||
|
||||
**Soft cap: 400 lines or ~2,000 words.** When a page approaches this, consider whether it should split.
|
||||
|
||||
**Hard cap: 800 lines.** Pages over this must split. The lint script flags violations.
|
||||
|
||||
Atomicity heuristic: a page is about *one* thing. A page on "Diffusion Models" should not also be the page on "Stable Diffusion" — those are two pages with cross-references. If you find yourself writing "## Variants" with five sub-sections of substantive prose, those sub-sections are probably their own pages.
|
||||
|
||||
The reason for the size cap is the context-bottleneck principle: any single page read is bounded, so the LLM can confidently read several pages without exhausting context.
|
||||
|
||||
## Naming
|
||||
|
||||
Page slugs are lowercase, hyphenated, no special characters. Match the directory: a concept page lives at `cml/wiki/concepts/<slug>.md`.
|
||||
|
||||
For entity pages about people: prefer the full name slugified (`andrej-karpathy.md`), not just the surname. Add `aliases:` to frontmatter for common short names.
|
||||
|
||||
For source pages: use a slug derived from the source title, possibly with the year for disambiguation (`attention-is-all-you-need-2017.md`). The source page slug should match the raw file slug if possible — that makes the back-pointer trivial.
|
||||
|
||||
For synthesis pages: derive from the question or the topic, not the date. `comparing-rag-vs-llm-wiki.md`, not `2026-04-15-question.md`. Date-based names don't surface usefully in the index.
|
||||
|
||||
## Body structure
|
||||
|
||||
The body has no rigid template — the schema may declare one for specific page types — but a few defaults work well:
|
||||
|
||||
**Source pages**: lead paragraph summarizing the source's main contribution. Sections for key claims, methodology (if relevant), conclusions, open questions. End with a "Where this fits" section listing the entity and concept pages this source touches.
|
||||
|
||||
**Entity pages**: lead paragraph defining the entity. Sections for relevant attributes (for a paper: authors, venue, key claims; for a person: affiliation, notable work; for a product: what it does, who makes it). A "Mentioned in" section is unnecessary — the backlink discovery handles that.
|
||||
|
||||
**Concept pages**: lead paragraph defining the concept clearly. Sections for the key formulation, variants, contested aspects, related concepts. Heavy use of `[[wikilinks]]` is expected — concept pages are the connective tissue of the wiki.
|
||||
|
||||
**Synthesis pages**: lead with the question (if filed from a query) or the framing. The body is the answer/analysis. End with the sources consulted as wikilinks.
|
||||
|
||||
## Hedging language
|
||||
|
||||
When a source claims something that hasn't been corroborated by other sources in the wiki, hedge: "Source X claims Y, though this is not yet corroborated by other sources in the wiki." The lint pass will revisit hedged claims as the wiki accumulates more sources, either upgrading them to confirmed or flagging them as contested.
|
||||
|
||||
When two sources contradict, document both: "Source X claims Y; source Z claims not-Y. The contradiction is unresolved." Do not silently pick a side.
|
||||
|
||||
## Keeping pages voice-neutral
|
||||
|
||||
The wiki is the LLM's voice, not the source author's voice and not the user's voice. Paraphrase rather than quote, except for short load-bearing phrases where the exact wording matters. Maintain a consistent, neutral, encyclopedic tone — close to a Wikipedia article in register, not a chat reply.
|
||||
@@ -1,102 +0,0 @@
|
||||
# Query Workflow
|
||||
|
||||
The user is asking a question against the wiki. The job is to answer it from the wiki, with citations, scaling navigation to the wiki's size, and to file the answer back as a synthesis page when warranted.
|
||||
|
||||
## Step 0: Check the schema
|
||||
|
||||
Read `cml/wiki/SCHEMA.md` if you haven't this session. Some wikis declare query-specific conventions (e.g. "always answer with a comparison table when the question is comparative", "answers go in `cml/wiki/synthesis/qa/` not `cml/wiki/synthesis/`"). Schema overrides defaults.
|
||||
|
||||
## Step 1: Read the index
|
||||
|
||||
Always start at `cml/wiki/index.md`. If the index has been sharded into `cml/wiki/indexes/`, read the top-level `index.md` first to identify which shard(s) are relevant, then read those.
|
||||
|
||||
The index is engineered for this — one line per page with a tight summary. You should be able to identify candidate pages from the index alone in most cases. If a query touches multiple shards (e.g. "compare the methodologies in papers A and B"), read all relevant shards.
|
||||
|
||||
## Step 2: Identify candidate pages
|
||||
|
||||
From the index, build a short list of pages that look relevant to the query. Be selective — reading 30 pages to answer a question is a sign you've fallen back to brute-force search, which doesn't scale. If the index summaries don't disambiguate well, that's a signal that the index entries are too terse — note it for the next lint pass.
|
||||
|
||||
If the index doesn't surface good candidates (the query uses fuzzy or domain-specific language that doesn't match the index summaries), fall back to the search script:
|
||||
|
||||
```bash
|
||||
uv run skills/llm-wiki/scripts/wiki_search.py "your query terms" --top 10
|
||||
```
|
||||
|
||||
This returns the top-N pages by BM25 score, with optional filters on frontmatter (`--type concept`, `--tag llms`, `--since 2026-01-01`). Use the search script *as a fallback*, not as the default — index-first is cheaper and produces more interpretable results when it works.
|
||||
|
||||
## Step 2b: Graph-assisted lookup (only if `cml/wiki/graph/graph.sqlite` exists)
|
||||
|
||||
For relational questions ("what's connected to X", "who proposed Y", "trace the path from A to B"), query the compiled graph after the index pass and before reading pages:
|
||||
|
||||
```bash
|
||||
uv run skills/llm-wiki/scripts/wiki_graph_query.py cml/wiki/ neighbors --node <node-id>
|
||||
uv run skills/llm-wiki/scripts/wiki_graph_query.py cml/wiki/ facts --about <node-id>
|
||||
```
|
||||
|
||||
Use the structured neighbors/facts to pick the right wiki pages to read — but never answer from graph rows alone for high-stakes claims. The graph accelerates navigation; the wiki page and its raw source remain the evidence. If `graph.sqlite` is older than the most recent `## [YYYY-MM-DD] ingest |` entry in `log.md`, use it as-is and note the staleness — do **not** run `wiki_graph_extract.py` inline. Extract is a compile-phase step; a query turn stays read-only and the background drain refreshes the graph after each ingest. Full reference: `references/graph-workflow.md`.
|
||||
|
||||
## Step 3: Read the candidate pages
|
||||
|
||||
Read each candidate page in full. While reading, note any `[[wikilinks]]` to other pages that look relevant — those are pre-curated leads. Follow the most promising ones, but don't recursively chase every link or you'll exhaust your context window on tangentially-relevant pages.
|
||||
|
||||
If a page references a source-summary page in its frontmatter and the answer hinges on what that source actually said, read the source-summary page too. Avoid going all the way back to the raw source unless the wiki summaries are clearly insufficient — the whole point of the wiki is that the synthesis is already done.
|
||||
|
||||
## Step 4: Find backlinks if needed
|
||||
|
||||
If the query is "what does my wiki say about X" or "where is X mentioned", and X has its own page, the inbound links are often more interesting than the page itself. Find them with:
|
||||
|
||||
```bash
|
||||
grep -rl "\[\[<page-slug>\]\]" cml/wiki/
|
||||
```
|
||||
|
||||
This is faster than reading pages to look for mentions. Note that the bundled `wiki_search.py` has a `--backlinks <slug>` mode that does this and returns ranked results.
|
||||
|
||||
## Step 5: Synthesize the answer
|
||||
|
||||
Write the answer in your own words, with `[[wikilink]]` citations to the wiki pages you used and (where helpful) `cml/raw/<file>` references to specific raw sources. The citations matter — they let the user verify the answer and follow up.
|
||||
|
||||
If the wiki contains contradicting claims, surface the contradiction explicitly rather than picking one and presenting it as settled. The wiki's value is partly in tracking what's known versus what's contested.
|
||||
|
||||
If the wiki has no relevant content for the query, say so plainly. Do not confabulate — that's the surest way to corrupt the wiki when the answer gets filed back. Instead, suggest sources the user could ingest to fill the gap.
|
||||
|
||||
## Step 6: Offer to file the answer back
|
||||
|
||||
If the synthesized answer represents new connection-making — a comparison the wiki didn't already contain, an analysis that pulls together threads from multiple pages, an answer to a recurring question — offer to file it as a synthesis page. The user will say no for trivial answers and yes for substantive ones. Default to offering.
|
||||
|
||||
The file-back creates `cml/wiki/synthesis/<answer-slug>.md` with frontmatter:
|
||||
|
||||
```yaml
|
||||
---
|
||||
type: synthesis
|
||||
question: "the original question, verbatim or lightly cleaned"
|
||||
asked: 2026-04-15
|
||||
sources_consulted: [page-1, page-2, page-3]
|
||||
tags: [...]
|
||||
---
|
||||
```
|
||||
|
||||
Body: the answer as you gave it to the user, possibly lightly edited for the wiki's voice. Add the new synthesis page to the relevant index. Append to `log.md` with prefix `## [YYYY-MM-DD] query | <question-summary>`.
|
||||
|
||||
A filed synthesis page is itself queryable — the next time the user asks an adjacent question, the synthesis page may be the most relevant candidate. This is how exploration compounds.
|
||||
|
||||
## Special query types
|
||||
|
||||
**"What's missing on topic X?"** — Read existing pages on X, identify open questions or unstated assumptions, and propose ingest candidates. This is essentially a per-topic micro-lint.
|
||||
|
||||
**"Compare X and Y"** — Read both pages, look for explicit comparison pages already in `synthesis/`, generate a comparison table or contrast prose. Strong file-back candidate.
|
||||
|
||||
**"Show me the timeline of X"** — Use `log.md` to reconstruct chronology of ingests touching X, supplement with `created`/`updated` frontmatter on relevant pages.
|
||||
|
||||
**"What did source X say about Y?"** — Read `cml/wiki/sources/<x>.md` for the source's summary; if it doesn't directly answer, read `cml/raw/<x>` (chunk-read if large).
|
||||
|
||||
**"Lint check on this answer"** — Before filing back, ask the user to verify a key claim by pointing to its raw source. Especially valuable for high-stakes wikis (medical, legal, financial).
|
||||
|
||||
## Anti-patterns to avoid
|
||||
|
||||
**Reading every page in the wiki to be safe.** This doesn't scale and produces vague answers. Trust the index; if the index fails, fix the index, don't bypass it.
|
||||
|
||||
**Citing the wiki without citing the underlying sources for hard claims.** The wiki page is a paraphrase; for any claim the user might need to verify, the citation should chain back to the raw source.
|
||||
|
||||
**Filing back trivial answers.** Not every Q&A is worth a permanent page. If the answer is a one-line lookup or restates an existing page, don't pollute synthesis/. The threshold is "would I want to find this when I ask a similar question in three months?"
|
||||
|
||||
**Confabulating when the wiki is silent.** Better to say "the wiki doesn't cover this" than to invent an answer that gets filed back as authoritative.
|
||||
@@ -1,91 +0,0 @@
|
||||
# Scaling Playbook
|
||||
|
||||
Thresholds at which the wiki's structure needs to evolve, and the migration steps. The goal is to keep the wiki's context cost roughly constant per query as the wiki grows — the LLM should never need to read more pages or larger pages just because the wiki got bigger.
|
||||
|
||||
## The bottleneck
|
||||
|
||||
Naive LLM Wiki implementations break at scale because of a few specific failure modes, each of which has a structural fix:
|
||||
|
||||
- **The index file becomes too large to read cheaply.** Fix: shard the index by category.
|
||||
- **Pages grow unboundedly as more sources mention them.** Fix: enforce the page size cap; split when violated.
|
||||
- **Index summaries become too vague to disambiguate candidates.** Fix: tighten summaries; introduce frontmatter filtering via the search script.
|
||||
- **Brute-force grep over the wiki replaces index-first navigation.** Fix: make the index actually useful and use the search script as the explicit fallback.
|
||||
|
||||
## Threshold 1: ~50 pages
|
||||
|
||||
Below this scale, you need almost no structure. A flat `cml/wiki/` directory with `index.md` and `log.md` is enough. The categorical subdirectories (`entities/`, `concepts/`, etc.) are still worth using from the start because they're free, but the index doesn't need sharding and the search script is overkill.
|
||||
|
||||
## Threshold 2: ~150 pages OR `index.md` over 300 lines
|
||||
|
||||
Time to **shard the index**. The migration:
|
||||
|
||||
1. Create `cml/wiki/indexes/` directory.
|
||||
2. Split `index.md` by category into `indexes/sources.md`, `indexes/entities.md`, `indexes/concepts.md`, `indexes/synthesis.md` (and any custom types from the schema). Each shard is a list of pages of that type, with the same one-line summaries.
|
||||
3. Rewrite the top-level `index.md` to be a directory of shards: each shard linked, with a one-line description of what's in it and a count.
|
||||
4. Update the schema to document the sharded structure.
|
||||
5. Update the ingest workflow in your working memory: now you update `indexes/<type>.md`, not `index.md` directly.
|
||||
|
||||
The top-level `index.md` should now be tiny — under 50 lines — and the shards are each bounded by the type-specific volume.
|
||||
|
||||
If a single shard later exceeds 300 lines (most likely `entities.md` or `concepts.md` if the wiki has a strong topical focus), shard *that* by sub-category: `indexes/entities-people.md`, `indexes/entities-papers.md`, etc. The principle generalizes.
|
||||
|
||||
## Threshold 3: ~300 pages
|
||||
|
||||
Time to introduce the **search script as a routine fallback**. Index navigation still works for direct lookups ("the page on diffusion models"), but fuzzy queries ("which papers discuss training stability") benefit from BM25 ranking.
|
||||
|
||||
`scripts/wiki_search.py` provides:
|
||||
|
||||
- `uv run skills/llm-wiki/scripts/wiki_search.py "query terms"` — top-N pages by BM25 score.
|
||||
- `--type concept` — filter by frontmatter type.
|
||||
- `--tag <tag>` — filter by tag.
|
||||
- `--since 2026-01-01` — filter by `updated` date.
|
||||
- `--backlinks <slug>` — find pages that link to a given page.
|
||||
- `--top-linked N` — find the N most-linked-to pages (hubs).
|
||||
|
||||
Update the schema to declare the search script as a sanctioned fallback, so that future LLM sessions know to reach for it rather than degenerating into recursive grep.
|
||||
|
||||
## Threshold 4: ~500 pages
|
||||
|
||||
At this scale, two things start to matter:
|
||||
|
||||
**Structural lint cadence becomes weekly or per-N-ingests.** Manual oversight stops scaling. Rely on `wiki_lint.py` to surface structural drift and triage with the user.
|
||||
|
||||
**The search script may want a real index.** The default `wiki_search.py` rebuilds its BM25 index on every run, which is fine up to a few thousand pages. Beyond that, persist the index to disk (the script supports `--cache .wiki-search-cache.json`).
|
||||
|
||||
Also consider whether the wiki has organically split into distinct topic clusters that don't really cross-reference each other. If so, a single wiki may be the wrong shape — splitting into per-topic wikis (each with its own `SCHEMA.md`, `index.md`, etc.) may be cleaner. The user should make this call.
|
||||
|
||||
## Threshold 5: ~1,000+ pages
|
||||
|
||||
At this scale, the question is whether the LLM Wiki pattern is still the right tool. Markdown + grep + frontmatter scales further than people expect (the gist author reports a wiki of ~100 articles and ~400K words working fine), but at some point a real database with structured queries beats markdown. Signals it's time to consider migrating:
|
||||
|
||||
- The user's queries are predominantly relational ("show me all papers from author X cited by papers in topic Y published after date Z"). A graph database serves this better.
|
||||
- Lint reports are too long to triage even at high cadence.
|
||||
- The schema has grown to specify dozens of types and hundreds of tags — at that point you've manually built a database schema in markdown.
|
||||
|
||||
If the user wants to migrate, the markdown wiki is excellent input for the migration: every page has frontmatter and `[[wikilinks]]` that map cleanly to a property graph.
|
||||
|
||||
## When to *not* shard
|
||||
|
||||
Sharding is irreversible-ish (you can un-shard, but it's annoying), so don't do it preemptively. Wait for the actual threshold. A wiki of 80 pages with a sharded index has worse usability than the same wiki with a flat index, because the shards add a navigation step without enough volume to justify it.
|
||||
|
||||
## When to introduce custom page types
|
||||
|
||||
The default types (`source`, `entity`, `concept`, `synthesis`) cover most use cases. Add a custom type only when the user has a clear category of pages that don't fit any default and that benefits from being a distinct subdirectory (queryable, distinct lint rules, distinct templates). Examples that justify it:
|
||||
|
||||
- `decision` pages for a team that documents architectural decisions
|
||||
- `experiment` pages for a research lab logging trial results
|
||||
- `character` pages for a fan wiki tracking a fictional cast
|
||||
- `meeting` pages for a team logging meetings
|
||||
|
||||
Don't add a type for a one-off — use tags instead. Adding a type is a schema change that affects the index, the lint script, and every future ingest.
|
||||
|
||||
## Detecting "we've outgrown our conventions"
|
||||
|
||||
Some signals that the schema needs revision rather than just more lint:
|
||||
|
||||
- The same kind of frontmatter issue keeps reappearing — the field needs to be optional, or the bootstrap template needs an example.
|
||||
- Pages keep getting created that don't fit cleanly into any existing type — a new type is wanted.
|
||||
- The tag list has grown unmanageable — prune, consolidate, or formalize the taxonomy.
|
||||
- The user keeps overriding the LLM on a particular kind of decision — encode their preference in the schema so it persists across sessions.
|
||||
|
||||
Schema revision is healthy. A schema that doesn't change after the first few weeks of a wiki's life is probably not being used.
|
||||
@@ -1,206 +0,0 @@
|
||||
#!/usr/bin/env -S uv run --script
|
||||
# /// script
|
||||
# requires-python = ">=3.11"
|
||||
# dependencies = []
|
||||
# ///
|
||||
"""
|
||||
init_wiki.py — Bootstrap or upgrade an LLM Wiki structure in a project.
|
||||
|
||||
Plain init creates the directory layout and drops in templates for SCHEMA.md,
|
||||
index.md, log.md, the page template, and the optional graph layer
|
||||
(graph/ontology.yaml, graph/README.md, graph/.gitignore). It is idempotent:
|
||||
re-running won't clobber existing files.
|
||||
|
||||
`--upgrade` mode is for wikis bootstrapped under an older plugin version. It
|
||||
does the same idempotent file creation, then inspects the existing SCHEMA.md
|
||||
for sections introduced in newer versions and prints clear instructions for
|
||||
what to merge by hand. It never overwrites SCHEMA.md — the schema is
|
||||
co-evolved with the user.
|
||||
|
||||
Usage:
|
||||
python init_wiki.py <project-root> [--wiki-dir wiki] [--raw-dir raw] [--upgrade]
|
||||
|
||||
Examples:
|
||||
python init_wiki.py .
|
||||
python init_wiki.py . --upgrade
|
||||
python init_wiki.py ~/research --wiki-dir kb --raw-dir sources
|
||||
"""
|
||||
|
||||
import argparse
|
||||
import sys
|
||||
from pathlib import Path
|
||||
from datetime import date
|
||||
|
||||
|
||||
SKILL_ROOT = Path(__file__).resolve().parent.parent
|
||||
TEMPLATES = SKILL_ROOT / "assets"
|
||||
|
||||
|
||||
SUBDIRS = ["sources", "entities", "concepts", "synthesis", "graph"]
|
||||
|
||||
# Markers used by --upgrade to detect SCHEMA.md sections introduced in
|
||||
# specific plugin versions. Each entry: (heading_marker, version_label,
|
||||
# template_anchor, blurb).
|
||||
SCHEMA_SECTION_MARKERS = [
|
||||
{
|
||||
"marker": "## Optional graph metadata",
|
||||
"version": "0.3.0",
|
||||
"anchor": "## Optional graph metadata",
|
||||
"label": "Optional graph metadata (Frontmatter section)",
|
||||
},
|
||||
{
|
||||
"marker": "## Graph layer",
|
||||
"version": "0.3.0",
|
||||
"anchor": "## Graph layer",
|
||||
"label": "Graph layer (canonical-vs-generated artifact policy)",
|
||||
},
|
||||
{
|
||||
"marker": "Graph lint + extract",
|
||||
"version": "0.3.0",
|
||||
"anchor": "- Graph lint + extract: after every ingest that adds typed `graph.relationships`.",
|
||||
"label": "Graph lint + extract cadence (Lint cadence section)",
|
||||
},
|
||||
]
|
||||
|
||||
|
||||
def copy_template(src: Path, dst: Path, substitutions: dict | None = None) -> bool:
|
||||
"""Copy a template file to dst. Returns True if file was created, False if it already existed."""
|
||||
if dst.exists():
|
||||
return False
|
||||
text = src.read_text()
|
||||
if substitutions:
|
||||
for key, value in substitutions.items():
|
||||
text = text.replace(key, value)
|
||||
dst.parent.mkdir(parents=True, exist_ok=True)
|
||||
dst.write_text(text)
|
||||
return True
|
||||
|
||||
|
||||
def detect_schema_gaps(schema_path: Path) -> list[dict]:
|
||||
"""Return the SCHEMA_SECTION_MARKERS entries missing from the user's SCHEMA.md."""
|
||||
if not schema_path.exists():
|
||||
return []
|
||||
text = schema_path.read_text(encoding="utf-8")
|
||||
return [m for m in SCHEMA_SECTION_MARKERS if m["marker"] not in text]
|
||||
|
||||
|
||||
def print_schema_upgrade_guidance(schema_path: Path, gaps: list[dict]) -> None:
|
||||
template_path = TEMPLATES / "SCHEMA.md.template"
|
||||
print()
|
||||
print("=" * 64)
|
||||
print(f"Upgrade required: {schema_path}")
|
||||
print("=" * 64)
|
||||
print(
|
||||
"Your SCHEMA.md predates one or more sections introduced by newer\n"
|
||||
"plugin versions. The graph layer itself is opt-in, but to make Claude\n"
|
||||
"aware of it, merge the sections below by hand. SCHEMA.md is co-evolved\n"
|
||||
"with you — this script never overwrites it."
|
||||
)
|
||||
print()
|
||||
print("Missing sections:")
|
||||
for m in gaps:
|
||||
print(f" - [{m['version']}] {m['label']}")
|
||||
print()
|
||||
print(f"Reference template: {template_path}")
|
||||
print(
|
||||
"Diff your SCHEMA.md against the template and copy the missing\n"
|
||||
"sections in. Or run /wiki:upgrade and Claude will propose the edits\n"
|
||||
"interactively (one section at a time, never silent)."
|
||||
)
|
||||
|
||||
|
||||
def init_wiki(project_root: Path, wiki_dir: str, raw_dir: str, upgrade: bool = False) -> None:
|
||||
project_root = project_root.resolve()
|
||||
if not project_root.exists():
|
||||
print(f"Error: project root does not exist: {project_root}", file=sys.stderr)
|
||||
sys.exit(1)
|
||||
|
||||
wiki = project_root / wiki_dir
|
||||
raw = project_root / raw_dir
|
||||
|
||||
mode = "Upgrading" if upgrade else "Initializing"
|
||||
print(f"{mode} LLM Wiki in: {project_root}")
|
||||
print(f" Wiki directory: {wiki}")
|
||||
print(f" Raw directory: {raw}")
|
||||
print()
|
||||
|
||||
created = []
|
||||
skipped = []
|
||||
|
||||
# Create wiki subdirs
|
||||
for subdir in SUBDIRS:
|
||||
d = wiki / subdir
|
||||
if not d.exists():
|
||||
d.mkdir(parents=True)
|
||||
created.append(f"{wiki_dir}/{subdir}/")
|
||||
else:
|
||||
skipped.append(f"{wiki_dir}/{subdir}/")
|
||||
|
||||
# Create raw + raw/assets
|
||||
for d, label in [(raw, raw_dir), (raw / "assets", f"{raw_dir}/assets")]:
|
||||
if not d.exists():
|
||||
d.mkdir(parents=True)
|
||||
created.append(f"{label}/")
|
||||
else:
|
||||
skipped.append(f"{label}/")
|
||||
|
||||
# Copy templates
|
||||
template_map = [
|
||||
("SCHEMA.md.template", wiki / "SCHEMA.md"),
|
||||
("index.md.template", wiki / "index.md"),
|
||||
("log.md.template", wiki / "log.md"),
|
||||
("page.md.template", wiki / ".page-template.md"),
|
||||
("ontology.yaml.template", wiki / "graph" / "ontology.yaml"),
|
||||
("graph_README.md.template", wiki / "graph" / "README.md"),
|
||||
("graph_gitignore.template", wiki / "graph" / ".gitignore"),
|
||||
]
|
||||
for src_name, dst in template_map:
|
||||
src = TEMPLATES / src_name
|
||||
if not src.exists():
|
||||
print(f"Warning: template missing: {src}", file=sys.stderr)
|
||||
continue
|
||||
if copy_template(src, dst):
|
||||
created.append(str(dst.relative_to(project_root)))
|
||||
else:
|
||||
skipped.append(str(dst.relative_to(project_root)))
|
||||
|
||||
# Report
|
||||
if created:
|
||||
print("Created:")
|
||||
for path in created:
|
||||
print(f" + {path}")
|
||||
if skipped:
|
||||
print("Already existed (skipped):")
|
||||
for path in skipped:
|
||||
print(f" = {path}")
|
||||
|
||||
if upgrade:
|
||||
gaps = detect_schema_gaps(wiki / "SCHEMA.md")
|
||||
if gaps:
|
||||
print_schema_upgrade_guidance(wiki / "SCHEMA.md", gaps)
|
||||
else:
|
||||
print()
|
||||
print("SCHEMA.md is up to date with the current template — no manual merge needed.")
|
||||
return
|
||||
|
||||
print()
|
||||
print("Next steps:")
|
||||
print(f" 1. Read {wiki_dir}/SCHEMA.md and customize it for your domain.")
|
||||
print(f" 2. (Optional) Edit {wiki_dir}/graph/ontology.yaml to add domain-specific predicates.")
|
||||
print(f" 3. Drop your first source into {raw_dir}/.")
|
||||
print(f" 4. Ask Claude to ingest it.")
|
||||
|
||||
|
||||
def main():
|
||||
parser = argparse.ArgumentParser(description=__doc__, formatter_class=argparse.RawDescriptionHelpFormatter)
|
||||
parser.add_argument("project_root", type=Path, help="Project root directory.")
|
||||
parser.add_argument("--wiki-dir", default="wiki", help="Name of the wiki subdirectory (default: wiki).")
|
||||
parser.add_argument("--raw-dir", default="raw", help="Name of the raw sources subdirectory (default: raw).")
|
||||
parser.add_argument("--upgrade", action="store_true",
|
||||
help="Upgrade an existing wiki: add missing files idempotently and surface SCHEMA.md sections to merge by hand.")
|
||||
args = parser.parse_args()
|
||||
init_wiki(args.project_root, args.wiki_dir, args.raw_dir, upgrade=args.upgrade)
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
main()
|
||||
@@ -1,169 +0,0 @@
|
||||
#!/usr/bin/env -S uv run --script
|
||||
# /// script
|
||||
# requires-python = ">=3.11"
|
||||
# dependencies = ["nanobot-ai"]
|
||||
# ///
|
||||
"""wiki_compile.py — dávkový compile nasbíraných zdrojů z cml/raw/ do cml/wiki/.
|
||||
|
||||
Spouštěn systémovým cronem každou minutu. Capture (interaktivní) hází zdroje do
|
||||
cml/raw/ a hned potvrdí; těžký raw→wiki compile (čtení zdrojů, psaní stránek,
|
||||
rozhodování) běží mimo interaktivní tah přes LLM agenta — tady, na pozadí.
|
||||
|
||||
Tok:
|
||||
1. Levná pre-kontrola (BEZ LLM): jsou v cml/raw/ nezpracované zdroje
|
||||
(regulérní soubory mimo _done/, _hard/, assets/)? Žádné → exit 0, agenta
|
||||
vůbec neinstancuj.
|
||||
2. Lockfile (cml/.compile.lock, PID + start-timestamp): běží jiný compile?
|
||||
→ exit 0 (neduplikovat). Stale lock (mrtvý proces / > STALE_SECONDS) se
|
||||
přebere, ať se to nezasekne po pádu.
|
||||
3. Jinak Nanobot.from_config() + bot.run(<drain goal>) — vyprázdní VŠECHNO
|
||||
nasbírané v jednom dávkovém běhu (jeden index/graph update pro víc zdrojů).
|
||||
4. Tiše: jen append do log/wiki_compile_cron.log; žádný Telegram.
|
||||
|
||||
Vzor = skills/detach/scripts/tasks-daemon.py (shebang uv run, deps nanobot-ai,
|
||||
Nanobot.from_config + asyncio.wait_for(bot.run(...), timeout)).
|
||||
"""
|
||||
|
||||
import asyncio
|
||||
import json
|
||||
import os
|
||||
import sys
|
||||
import traceback
|
||||
from datetime import datetime
|
||||
from pathlib import Path
|
||||
|
||||
# Skript žije v workspace/skills/llm-wiki/scripts/ → parents[3] = workspace.
|
||||
WORKSPACE = Path(__file__).resolve().parents[3]
|
||||
CML = WORKSPACE / "cml"
|
||||
RAW = CML / "raw"
|
||||
LOCK = CML / ".compile.lock"
|
||||
LOG = WORKSPACE / "log" / "wiki_compile_cron.log"
|
||||
|
||||
# Podadresáře v raw/, které NEjsou pending zdroje.
|
||||
RESERVED_DIRS = {"_done", "_hard", "assets"}
|
||||
|
||||
TIMEOUT_SECONDS = 25 * 60
|
||||
STALE_SECONDS = 30 * 60
|
||||
|
||||
DRAIN_GOAL = (
|
||||
"Pomocí skillu llm-wiki (operace Compile/drain) zkompiluj VŠECHNY nezpracované zdroje "
|
||||
"v `cml/raw/` (regulérní soubory přímo v `cml/raw/`, mimo `_done/`, `_hard/`, `assets/`) "
|
||||
"do wiki v `cml/wiki/`. Pro každý zdroj proveď plný ingest podle "
|
||||
"references/ingest-workflow.md: source/entity/concept stránky s frontmatterem a `[[odkazy]]`, "
|
||||
"aktualizuj `cml/wiki/index.md` a `cml/wiki/log.md`. Po zpracování všech zdrojů regeneruj graph "
|
||||
"(`wiki_graph_lint.py` + `wiki_graph_extract.py` na `cml/wiki/`). Každý úspěšně zpracovaný "
|
||||
"zdroj přesuň do `cml/raw/_done/`. Ambiguózní/konfliktní zdroj NEcompiluj natvrdo — nech ho "
|
||||
"v `cml/raw/` (nebo přesuň do `cml/raw/_hard/`) a důvod zaznamenej do `cml/wiki/log.md`. "
|
||||
"Lint je report-only: žádné destruktivní úpravy existujících stránek bez potvrzení. "
|
||||
"Idempotence: pokud pro zdroj už stránky existují (byl zkompilován dřív, jen nepřesunut), "
|
||||
"NEcykluj reconciliací — ber ho jako hotový, přesuň raw soubor do `cml/raw/_done/` a pokračuj. "
|
||||
"Každý vyřízený zdroj VŽDY přesuň z `cml/raw/` pryč, ať ho příští cron tik nezpracovává znovu. "
|
||||
"Běžíš v izolované session na pozadí, bez interakce s uživatelem."
|
||||
)
|
||||
|
||||
|
||||
def log(message: str) -> None:
|
||||
LOG.parent.mkdir(parents=True, exist_ok=True)
|
||||
stamp = datetime.now().astimezone().isoformat(timespec="seconds")
|
||||
with LOG.open("a", encoding="utf-8") as handle:
|
||||
handle.write(f"{stamp} {message}\n")
|
||||
|
||||
|
||||
def pending_sources() -> list[Path]:
|
||||
"""Regulérní soubory přímo v cml/raw/ (mimo skryté a rezervované podadresáře)."""
|
||||
if not RAW.exists():
|
||||
return []
|
||||
return [p for p in sorted(RAW.iterdir()) if p.is_file() and not p.name.startswith(".")]
|
||||
|
||||
|
||||
def _pid_alive(pid: int) -> bool:
|
||||
try:
|
||||
os.kill(pid, 0)
|
||||
except ProcessLookupError:
|
||||
return False
|
||||
except PermissionError:
|
||||
return True
|
||||
return True
|
||||
|
||||
|
||||
def _lock_is_stale() -> bool:
|
||||
"""Lock je mrtvý, když ho nelze přečíst, proces neběží, nebo je starší než STALE_SECONDS."""
|
||||
try:
|
||||
data = json.loads(LOCK.read_text())
|
||||
pid = int(data["pid"])
|
||||
started = datetime.fromisoformat(data["started"])
|
||||
except (OSError, ValueError, KeyError):
|
||||
return True
|
||||
if not _pid_alive(pid):
|
||||
return True
|
||||
age = (datetime.now().astimezone() - started).total_seconds()
|
||||
return age > STALE_SECONDS
|
||||
|
||||
|
||||
def acquire_lock() -> bool:
|
||||
"""Atomicky vytvoř lock. Vrať False, když už běží živý compile."""
|
||||
for _ in range(2):
|
||||
try:
|
||||
fd = os.open(LOCK, os.O_CREAT | os.O_EXCL | os.O_WRONLY, 0o644)
|
||||
except FileExistsError:
|
||||
if not _lock_is_stale():
|
||||
return False
|
||||
log("stale lock, reclaiming")
|
||||
LOCK.unlink(missing_ok=True)
|
||||
continue
|
||||
payload = {"pid": os.getpid(), "started": datetime.now().astimezone().isoformat()}
|
||||
with os.fdopen(fd, "w", encoding="utf-8") as handle:
|
||||
json.dump(payload, handle)
|
||||
return True
|
||||
return False
|
||||
|
||||
|
||||
async def run_compile(goal: str) -> str:
|
||||
# Heavy import deferred: the per-minute pre-check (no pending work) must not
|
||||
# pay the nanobot import cost — only an actual compile run needs it.
|
||||
from nanobot import Nanobot
|
||||
|
||||
bot = Nanobot.from_config()
|
||||
result = await bot.run(goal, session_key="wiki-compile")
|
||||
return result.content or ""
|
||||
|
||||
|
||||
def main() -> int:
|
||||
dry_run = "--dry-run" in sys.argv[1:]
|
||||
|
||||
pending = pending_sources()
|
||||
if not pending:
|
||||
return 0
|
||||
|
||||
if not acquire_lock():
|
||||
log(f"SKIP compile already running ({len(pending)} pending)")
|
||||
return 0
|
||||
|
||||
if dry_run:
|
||||
names = ", ".join(p.name for p in pending)
|
||||
log(f"DRY-RUN would compile {len(pending)} pending: {names}")
|
||||
LOCK.unlink(missing_ok=True)
|
||||
return 0
|
||||
|
||||
started = datetime.now().astimezone()
|
||||
log(f"START compile {len(pending)} pending: {', '.join(p.name for p in pending)}")
|
||||
try:
|
||||
result_text = asyncio.run(
|
||||
asyncio.wait_for(run_compile(DRAIN_GOAL), timeout=TIMEOUT_SECONDS)
|
||||
)
|
||||
summary = result_text.strip().splitlines()[0][:200] if result_text.strip() else "(prázdný výstup)"
|
||||
duration = int((datetime.now().astimezone() - started).total_seconds())
|
||||
log(f"END compile duration={duration}s remaining={len(pending_sources())} :: {summary}")
|
||||
return 0
|
||||
except asyncio.TimeoutError:
|
||||
log(f"TIMEOUT compile po {TIMEOUT_SECONDS // 60} min")
|
||||
return 1
|
||||
except Exception as error:
|
||||
log(f"EXCEPTION compile: {error}\n{traceback.format_exc()}")
|
||||
return 1
|
||||
finally:
|
||||
LOCK.unlink(missing_ok=True)
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
sys.exit(main())
|
||||
@@ -1,541 +0,0 @@
|
||||
#!/usr/bin/env -S uv run --script
|
||||
# /// script
|
||||
# requires-python = ">=3.11"
|
||||
# dependencies = ["pyyaml"]
|
||||
# ///
|
||||
"""
|
||||
wiki_graph_extract.py — Compile the markdown wiki into a queryable graph.
|
||||
|
||||
Markdown remains canonical. This script reads every wiki page, derives nodes
|
||||
and edges (typed semantic edges from `graph.relationships`, plus implicit
|
||||
`mentions`, `sourced_from`, `summarizes_raw` edges), and emits artifacts under
|
||||
`<wiki>/graph/` that can be deleted and rebuilt at any time.
|
||||
|
||||
Requires PyYAML (`pip install pyyaml`) — the new graph layer uses real YAML
|
||||
parsing for its nested frontmatter, unlike the stdlib-only lint/search/stats
|
||||
scripts.
|
||||
|
||||
Usage:
|
||||
python wiki_graph_extract.py <wiki-dir> [options]
|
||||
|
||||
Options:
|
||||
--out <dir> Output directory (default: <wiki-dir>/graph)
|
||||
--formats jsonl,sqlite,... Comma-list of formats to emit
|
||||
(jsonl, sqlite, graphml; default: all three)
|
||||
--ontology <path> Override ontology path
|
||||
(default: <wiki-dir>/graph/ontology.yaml)
|
||||
|
||||
Examples:
|
||||
python wiki_graph_extract.py wiki/
|
||||
python wiki_graph_extract.py wiki/ --out wiki/graph --formats jsonl,sqlite
|
||||
"""
|
||||
|
||||
import argparse
|
||||
import hashlib
|
||||
import json
|
||||
import re
|
||||
import sqlite3
|
||||
import sys
|
||||
import xml.etree.ElementTree as ET
|
||||
from collections import defaultdict
|
||||
from pathlib import Path
|
||||
|
||||
try:
|
||||
import yaml
|
||||
except ImportError:
|
||||
print(
|
||||
"wiki_graph_extract.py requires PyYAML.\n"
|
||||
"Install with: pip install pyyaml",
|
||||
file=sys.stderr,
|
||||
)
|
||||
sys.exit(2)
|
||||
|
||||
|
||||
WIKILINK_RE = re.compile(r"\[\[([^\]|]+)(?:\|[^\]]+)?\]\]")
|
||||
FRONTMATTER_RE = re.compile(r"^---\s*\n(.*?)\n---\s*\n", re.DOTALL)
|
||||
|
||||
SKIP_TOP_LEVEL_FILES = {"SCHEMA.md", "index.md", "log.md", "README.md"}
|
||||
SKIP_TOP_LEVEL_DIRS = {"indexes", "graph"}
|
||||
|
||||
DEFAULT_FORMATS = ["jsonl", "sqlite", "graphml"]
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Page collection
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
|
||||
def parse_frontmatter(text: str) -> tuple[dict, str]:
|
||||
"""Extract YAML frontmatter using PyYAML. Returns (meta, body)."""
|
||||
m = FRONTMATTER_RE.match(text)
|
||||
if not m:
|
||||
return {}, text
|
||||
fm_text = m.group(1)
|
||||
body = text[m.end():]
|
||||
try:
|
||||
meta = yaml.safe_load(fm_text) or {}
|
||||
except yaml.YAMLError:
|
||||
meta = {}
|
||||
if not isinstance(meta, dict):
|
||||
meta = {}
|
||||
return meta, body
|
||||
|
||||
|
||||
def collect_pages(wiki_root: Path) -> list[dict]:
|
||||
pages = []
|
||||
for md_path in sorted(wiki_root.rglob("*.md")):
|
||||
rel = md_path.relative_to(wiki_root)
|
||||
if rel.parts[0] in SKIP_TOP_LEVEL_FILES or rel.parts[0] in SKIP_TOP_LEVEL_DIRS:
|
||||
continue
|
||||
if rel.name.startswith("."):
|
||||
continue
|
||||
try:
|
||||
text = md_path.read_text(encoding="utf-8")
|
||||
except (UnicodeDecodeError, OSError):
|
||||
continue
|
||||
meta, body = parse_frontmatter(text)
|
||||
links = [m.group(1).strip() for m in WIKILINK_RE.finditer(body)]
|
||||
pages.append({
|
||||
"path": str(md_path),
|
||||
"rel_path": str(rel).replace("\\", "/"),
|
||||
"slug": md_path.stem,
|
||||
"meta": meta,
|
||||
"body": body,
|
||||
"links": links,
|
||||
})
|
||||
return pages
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Ontology
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
|
||||
def load_ontology(path: Path) -> dict:
|
||||
if not path.exists():
|
||||
return {"node_types": {}, "predicates": {}}
|
||||
try:
|
||||
data = yaml.safe_load(path.read_text(encoding="utf-8")) or {}
|
||||
except yaml.YAMLError as e:
|
||||
print(f"Ontology parse error ({path}): {e}", file=sys.stderr)
|
||||
sys.exit(2)
|
||||
data.setdefault("node_types", {})
|
||||
data.setdefault("predicates", {})
|
||||
return data
|
||||
|
||||
|
||||
def derive_node_type(meta: dict, ontology: dict) -> str | None:
|
||||
"""Map a page's frontmatter to a node_type using ontology[node_types][*].maps_from."""
|
||||
page_type = meta.get("type")
|
||||
page_kind = meta.get("kind")
|
||||
explicit = (meta.get("graph") or {}).get("node_type") if isinstance(meta.get("graph"), dict) else None
|
||||
if explicit:
|
||||
return explicit
|
||||
# Try (type, kind) match first, then type-only.
|
||||
type_kind_match = None
|
||||
type_only_match = None
|
||||
for nt_name, nt_def in ontology["node_types"].items():
|
||||
maps = (nt_def or {}).get("maps_from") or {}
|
||||
m_type = maps.get("type")
|
||||
m_kind = maps.get("kind")
|
||||
if m_type and m_type == page_type:
|
||||
if m_kind and m_kind == page_kind:
|
||||
type_kind_match = nt_name
|
||||
break
|
||||
if not m_kind and type_only_match is None:
|
||||
type_only_match = nt_name
|
||||
return type_kind_match or type_only_match
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Node + edge construction
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
|
||||
def build_nodes(pages: list[dict], ontology: dict) -> tuple[list[dict], dict, list[dict]]:
|
||||
"""Build the node list + slug→node_id index + alias rows. Returns (nodes, slug_to_id, aliases)."""
|
||||
nodes: list[dict] = []
|
||||
slug_to_id: dict[str, str] = {}
|
||||
aliases: list[dict] = []
|
||||
seen_ids: set[str] = set()
|
||||
|
||||
for p in pages:
|
||||
meta = p["meta"]
|
||||
graph_meta = meta.get("graph") if isinstance(meta.get("graph"), dict) else {}
|
||||
node_type = derive_node_type(meta, ontology) or "concept"
|
||||
explicit_id = graph_meta.get("node_id")
|
||||
node_id = explicit_id or f"{node_type}:{p['slug']}"
|
||||
|
||||
# Skip duplicates — first one wins; lint will flag this.
|
||||
if node_id in seen_ids:
|
||||
continue
|
||||
seen_ids.add(node_id)
|
||||
|
||||
node = {
|
||||
"id": node_id,
|
||||
"slug": p["slug"],
|
||||
"title": meta.get("title") or p["slug"],
|
||||
"page_type": meta.get("type") or "",
|
||||
"node_type": node_type,
|
||||
"kind": meta.get("kind") or "",
|
||||
"tags": list(meta.get("tags") or []),
|
||||
"aliases": list(graph_meta.get("aliases") or []),
|
||||
"path": p["rel_path"],
|
||||
"created": meta.get("created") or "",
|
||||
"updated": meta.get("updated") or "",
|
||||
"canonical": bool(graph_meta.get("canonical", False)),
|
||||
}
|
||||
nodes.append(node)
|
||||
slug_to_id[p["slug"]] = node_id
|
||||
for alias in node["aliases"]:
|
||||
aliases.append({"alias": str(alias), "node_id": node_id})
|
||||
|
||||
return nodes, slug_to_id, aliases
|
||||
|
||||
|
||||
def edge_id(subject: str, predicate: str, obj: str, source: str | None, evidence: str | None) -> str:
|
||||
# Truncated to 96 bits — collision risk is negligible at any plausible
|
||||
# wiki scale and shorter ids keep the JSONL/sqlite/graphml outputs readable.
|
||||
h = hashlib.sha256()
|
||||
parts = [subject or "", predicate or "", obj or "", source or "", evidence or ""]
|
||||
h.update("\x1f".join(parts).encode("utf-8"))
|
||||
return h.hexdigest()[:24]
|
||||
|
||||
|
||||
def make_edge(*, subject, predicate, obj, source, evidence, confidence, status,
|
||||
extraction_method, page, extras: dict | None = None) -> dict:
|
||||
return {
|
||||
"id": edge_id(subject, predicate, obj, source, evidence),
|
||||
"subject": subject,
|
||||
"predicate": predicate,
|
||||
"object": obj,
|
||||
"source": source or "",
|
||||
"evidence": evidence or "",
|
||||
"confidence": confidence or "",
|
||||
"status": status or "",
|
||||
"extraction_method": extraction_method,
|
||||
"page": page,
|
||||
"extras": extras or {},
|
||||
}
|
||||
|
||||
|
||||
def build_edges(pages: list[dict], slug_to_id: dict[str, str]) -> list[dict]:
|
||||
edges: list[dict] = []
|
||||
seen_ids: set[str] = set()
|
||||
|
||||
def push(edge: dict) -> None:
|
||||
if edge["id"] in seen_ids:
|
||||
return
|
||||
seen_ids.add(edge["id"])
|
||||
edges.append(edge)
|
||||
|
||||
for p in pages:
|
||||
slug = p["slug"]
|
||||
subject_id = slug_to_id.get(slug)
|
||||
if not subject_id:
|
||||
continue
|
||||
meta = p["meta"]
|
||||
graph_meta = meta.get("graph") if isinstance(meta.get("graph"), dict) else {}
|
||||
|
||||
# 1. Typed semantic edges from graph.relationships[].
|
||||
for rel in graph_meta.get("relationships") or []:
|
||||
if not isinstance(rel, dict):
|
||||
continue
|
||||
obj = rel.get("object")
|
||||
predicate = rel.get("predicate")
|
||||
if not (obj and predicate):
|
||||
continue
|
||||
extras = {
|
||||
k: rel[k] for k in ("valid_from", "valid_to", "notes", "raw_ref",
|
||||
"contradicts", "supersedes")
|
||||
if k in rel and rel[k] is not None
|
||||
}
|
||||
push(make_edge(
|
||||
subject=subject_id,
|
||||
predicate=str(predicate),
|
||||
obj=str(obj),
|
||||
source=rel.get("source"),
|
||||
evidence=rel.get("evidence"),
|
||||
confidence=rel.get("confidence"),
|
||||
status=rel.get("status"),
|
||||
extraction_method="explicit_graph_frontmatter",
|
||||
page=p["rel_path"],
|
||||
extras=extras,
|
||||
))
|
||||
|
||||
# 2. Mentions edges from body wikilinks.
|
||||
seen_targets: set[str] = set()
|
||||
for link in p["links"]:
|
||||
target_slug = link.split("#")[0].strip()
|
||||
if not target_slug or target_slug == slug:
|
||||
continue
|
||||
target_id = slug_to_id.get(target_slug)
|
||||
if not target_id or target_id in seen_targets:
|
||||
continue
|
||||
seen_targets.add(target_id)
|
||||
push(make_edge(
|
||||
subject=subject_id,
|
||||
predicate="mentions",
|
||||
obj=target_id,
|
||||
source=None,
|
||||
evidence=None,
|
||||
confidence="low",
|
||||
status="current",
|
||||
extraction_method="body_wikilink",
|
||||
page=p["rel_path"],
|
||||
))
|
||||
|
||||
# 3. sourced_from edges from frontmatter `sources:` (skip on source pages themselves).
|
||||
if meta.get("type") != "source":
|
||||
for src_slug in meta.get("sources") or []:
|
||||
src_id = slug_to_id.get(str(src_slug))
|
||||
if not src_id:
|
||||
continue
|
||||
push(make_edge(
|
||||
subject=subject_id,
|
||||
predicate="sourced_from",
|
||||
obj=src_id,
|
||||
source=str(src_slug),
|
||||
evidence=None,
|
||||
confidence="high",
|
||||
status="current",
|
||||
extraction_method="frontmatter_sources",
|
||||
page=p["rel_path"],
|
||||
))
|
||||
|
||||
# 4. summarizes_raw edges from source pages' raw: field.
|
||||
if meta.get("type") == "source":
|
||||
raw_path = meta.get("raw")
|
||||
if raw_path:
|
||||
push(make_edge(
|
||||
subject=subject_id,
|
||||
predicate="summarizes_raw",
|
||||
obj=f"raw:{raw_path}",
|
||||
source=None,
|
||||
evidence=None,
|
||||
confidence="high",
|
||||
status="current",
|
||||
extraction_method="frontmatter_raw",
|
||||
page=p["rel_path"],
|
||||
))
|
||||
|
||||
return edges
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Output writers
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
|
||||
def _normalize_for_json(value):
|
||||
if hasattr(value, "isoformat"):
|
||||
return value.isoformat()
|
||||
if isinstance(value, list):
|
||||
return [_normalize_for_json(v) for v in value]
|
||||
if isinstance(value, dict):
|
||||
return {k: _normalize_for_json(v) for k, v in value.items()}
|
||||
return value
|
||||
|
||||
|
||||
def write_jsonl(out_dir: Path, nodes: list[dict], edges: list[dict]) -> None:
|
||||
nodes_sorted = sorted(nodes, key=lambda n: n["id"])
|
||||
edges_sorted = sorted(edges, key=lambda e: e["id"])
|
||||
with (out_dir / "nodes.jsonl").open("w", encoding="utf-8") as f:
|
||||
for n in nodes_sorted:
|
||||
f.write(json.dumps(_normalize_for_json(n), sort_keys=True, ensure_ascii=False))
|
||||
f.write("\n")
|
||||
with (out_dir / "edges.jsonl").open("w", encoding="utf-8") as f:
|
||||
for e in edges_sorted:
|
||||
f.write(json.dumps(_normalize_for_json(e), sort_keys=True, ensure_ascii=False))
|
||||
f.write("\n")
|
||||
|
||||
|
||||
def write_sqlite(out_dir: Path, nodes: list[dict], aliases: list[dict], edges: list[dict]) -> None:
|
||||
db_path = out_dir / "graph.sqlite"
|
||||
if db_path.exists():
|
||||
db_path.unlink()
|
||||
conn = sqlite3.connect(db_path)
|
||||
try:
|
||||
conn.executescript("""
|
||||
CREATE TABLE nodes (
|
||||
id TEXT PRIMARY KEY,
|
||||
slug TEXT NOT NULL UNIQUE,
|
||||
title TEXT NOT NULL,
|
||||
page_type TEXT NOT NULL,
|
||||
node_type TEXT NOT NULL,
|
||||
kind TEXT,
|
||||
path TEXT NOT NULL,
|
||||
created TEXT,
|
||||
updated TEXT,
|
||||
metadata_json TEXT NOT NULL
|
||||
);
|
||||
CREATE TABLE aliases (
|
||||
alias TEXT NOT NULL,
|
||||
node_id TEXT NOT NULL,
|
||||
PRIMARY KEY (alias, node_id),
|
||||
FOREIGN KEY (node_id) REFERENCES nodes(id)
|
||||
);
|
||||
CREATE TABLE edges (
|
||||
id TEXT PRIMARY KEY,
|
||||
subject TEXT NOT NULL,
|
||||
predicate TEXT NOT NULL,
|
||||
object TEXT NOT NULL,
|
||||
source TEXT,
|
||||
evidence TEXT,
|
||||
confidence TEXT,
|
||||
status TEXT,
|
||||
extraction_method TEXT NOT NULL,
|
||||
page TEXT NOT NULL,
|
||||
metadata_json TEXT NOT NULL
|
||||
);
|
||||
CREATE INDEX idx_edges_subject ON edges(subject);
|
||||
CREATE INDEX idx_edges_object ON edges(object);
|
||||
CREATE INDEX idx_edges_predicate ON edges(predicate);
|
||||
CREATE INDEX idx_edges_source ON edges(source);
|
||||
""")
|
||||
|
||||
for n in sorted(nodes, key=lambda n: n["id"]):
|
||||
metadata_json = json.dumps(_normalize_for_json({
|
||||
"tags": n.get("tags", []),
|
||||
"aliases": n.get("aliases", []),
|
||||
"canonical": n.get("canonical", False),
|
||||
}), sort_keys=True, ensure_ascii=False)
|
||||
conn.execute(
|
||||
"INSERT INTO nodes (id, slug, title, page_type, node_type, kind, path, created, updated, metadata_json) "
|
||||
"VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, ?)",
|
||||
(
|
||||
n["id"], n["slug"], n["title"], n["page_type"], n["node_type"],
|
||||
n.get("kind") or None, n["path"],
|
||||
str(n.get("created") or "") or None,
|
||||
str(n.get("updated") or "") or None,
|
||||
metadata_json,
|
||||
),
|
||||
)
|
||||
|
||||
for a in sorted(aliases, key=lambda a: (a["alias"], a["node_id"])):
|
||||
conn.execute(
|
||||
"INSERT OR IGNORE INTO aliases (alias, node_id) VALUES (?, ?)",
|
||||
(a["alias"], a["node_id"]),
|
||||
)
|
||||
|
||||
for e in sorted(edges, key=lambda e: e["id"]):
|
||||
metadata_json = json.dumps(_normalize_for_json(e.get("extras") or {}),
|
||||
sort_keys=True, ensure_ascii=False)
|
||||
conn.execute(
|
||||
"INSERT INTO edges (id, subject, predicate, object, source, evidence, "
|
||||
"confidence, status, extraction_method, page, metadata_json) "
|
||||
"VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?)",
|
||||
(
|
||||
e["id"], e["subject"], e["predicate"], e["object"],
|
||||
e.get("source") or None, e.get("evidence") or None,
|
||||
e.get("confidence") or None, e.get("status") or None,
|
||||
e["extraction_method"], e["page"], metadata_json,
|
||||
),
|
||||
)
|
||||
conn.commit()
|
||||
finally:
|
||||
conn.close()
|
||||
|
||||
|
||||
def write_graphml(out_dir: Path, nodes: list[dict], edges: list[dict]) -> None:
|
||||
ns = "http://graphml.graphdrawing.org/xmlns"
|
||||
ET.register_namespace("", ns)
|
||||
root = ET.Element(f"{{{ns}}}graphml")
|
||||
|
||||
keys = [
|
||||
("d_title", "node", "title", "string"),
|
||||
("d_node_type", "node", "node_type", "string"),
|
||||
("d_page_type", "node", "page_type", "string"),
|
||||
("d_path", "node", "path", "string"),
|
||||
("d_predicate", "edge", "predicate", "string"),
|
||||
("d_confidence", "edge", "confidence", "string"),
|
||||
("d_status", "edge", "status", "string"),
|
||||
("d_source", "edge", "source", "string"),
|
||||
]
|
||||
for kid, kfor, kname, ktype in keys:
|
||||
k = ET.SubElement(root, f"{{{ns}}}key")
|
||||
k.set("id", kid)
|
||||
k.set("for", kfor)
|
||||
k.set("attr.name", kname)
|
||||
k.set("attr.type", ktype)
|
||||
|
||||
graph = ET.SubElement(root, f"{{{ns}}}graph")
|
||||
graph.set("id", "wiki")
|
||||
graph.set("edgedefault", "directed")
|
||||
|
||||
for n in sorted(nodes, key=lambda n: n["id"]):
|
||||
node_el = ET.SubElement(graph, f"{{{ns}}}node")
|
||||
node_el.set("id", n["id"])
|
||||
for kid, kfor, kname, _ in keys:
|
||||
if kfor != "node":
|
||||
continue
|
||||
data = ET.SubElement(node_el, f"{{{ns}}}data")
|
||||
data.set("key", kid)
|
||||
data.text = str(n.get(kname) or "")
|
||||
|
||||
for e in sorted(edges, key=lambda e: e["id"]):
|
||||
edge_el = ET.SubElement(graph, f"{{{ns}}}edge")
|
||||
edge_el.set("id", e["id"])
|
||||
edge_el.set("source", e["subject"])
|
||||
edge_el.set("target", e["object"])
|
||||
for kid, kfor, kname, _ in keys:
|
||||
if kfor != "edge":
|
||||
continue
|
||||
data = ET.SubElement(edge_el, f"{{{ns}}}data")
|
||||
data.set("key", kid)
|
||||
data.text = str(e.get(kname) or "")
|
||||
|
||||
tree = ET.ElementTree(root)
|
||||
ET.indent(tree, space=" ")
|
||||
tree.write(out_dir / "graph.graphml", encoding="utf-8", xml_declaration=True)
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# CLI
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
|
||||
def main():
|
||||
parser = argparse.ArgumentParser(description=__doc__, formatter_class=argparse.RawDescriptionHelpFormatter)
|
||||
parser.add_argument("wiki", type=Path, help="Wiki directory.")
|
||||
parser.add_argument("--out", type=Path, help="Output directory (default: <wiki>/graph)")
|
||||
parser.add_argument("--formats", default=",".join(DEFAULT_FORMATS),
|
||||
help="Comma-list: jsonl, sqlite, graphml")
|
||||
parser.add_argument("--ontology", type=Path, help="Ontology file (default: <wiki>/graph/ontology.yaml)")
|
||||
args = parser.parse_args()
|
||||
|
||||
if not args.wiki.exists():
|
||||
print(f"Wiki directory not found: {args.wiki}", file=sys.stderr)
|
||||
sys.exit(1)
|
||||
|
||||
out_dir = args.out or (args.wiki / "graph")
|
||||
out_dir.mkdir(parents=True, exist_ok=True)
|
||||
ontology_path = args.ontology or (args.wiki / "graph" / "ontology.yaml")
|
||||
ontology = load_ontology(ontology_path)
|
||||
formats = [f.strip().lower() for f in args.formats.split(",") if f.strip()]
|
||||
unknown = [f for f in formats if f not in DEFAULT_FORMATS]
|
||||
if unknown:
|
||||
print(f"Unknown formats: {unknown}. Allowed: {DEFAULT_FORMATS}", file=sys.stderr)
|
||||
sys.exit(1)
|
||||
|
||||
pages = collect_pages(args.wiki)
|
||||
nodes, slug_to_id, aliases = build_nodes(pages, ontology)
|
||||
edges = build_edges(pages, slug_to_id)
|
||||
|
||||
if "jsonl" in formats:
|
||||
write_jsonl(out_dir, nodes, edges)
|
||||
if "sqlite" in formats:
|
||||
write_sqlite(out_dir, nodes, aliases, edges)
|
||||
if "graphml" in formats:
|
||||
write_graphml(out_dir, nodes, edges)
|
||||
|
||||
print(f"Extracted {len(nodes)} nodes, {len(edges)} edges → {out_dir}")
|
||||
breakdown = defaultdict(int)
|
||||
for e in edges:
|
||||
breakdown[e["predicate"]] += 1
|
||||
for pred, count in sorted(breakdown.items(), key=lambda x: (-x[1], x[0])):
|
||||
print(f" {pred:20s} {count}")
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
main()
|
||||
@@ -1,418 +0,0 @@
|
||||
#!/usr/bin/env -S uv run --script
|
||||
# /// script
|
||||
# requires-python = ">=3.11"
|
||||
# dependencies = ["pyyaml"]
|
||||
# ///
|
||||
"""
|
||||
wiki_graph_lint.py — Validate the typed graph metadata in a wiki.
|
||||
|
||||
Reads every page's `graph:` frontmatter, cross-checks against the ontology
|
||||
(`wiki/graph/ontology.yaml`), and reports problems. Conservative by design:
|
||||
reports only, never edits.
|
||||
|
||||
Requires PyYAML (`pip install pyyaml`).
|
||||
|
||||
Checks:
|
||||
- Unique `graph.node_id` values across pages.
|
||||
- All relationship `object` ids resolve to known nodes (or are allowed
|
||||
string-literal targets for predicates whose object_types include "*").
|
||||
- All predicates exist in `graph/ontology.yaml`.
|
||||
- Predicate subject/object types match ontology.
|
||||
- Typed semantic edges (anything except mentions/sourced_from/summarizes_raw
|
||||
and predicates with `requires_evidence: false`) carry `source` and
|
||||
`evidence`.
|
||||
- `source` references resolve to an existing source page.
|
||||
- `confidence` is one of high|medium|low; `status` is one of
|
||||
current|historical|proposed|disputed|superseded.
|
||||
- No duplicate canonical nodes for the same node id.
|
||||
- Aliases do not collide across distinct canonical nodes.
|
||||
- `contradicts` / `supersedes` references resolve to known node/edge ids.
|
||||
- Generated graph has no orphan typed nodes (nodes with no inbound or
|
||||
outbound typed edges) except for `source` nodes (allowed source-only).
|
||||
|
||||
Usage:
|
||||
python wiki_graph_lint.py [<wiki-dir>] [--json]
|
||||
"""
|
||||
|
||||
import argparse
|
||||
import json
|
||||
import re
|
||||
import sys
|
||||
from collections import defaultdict
|
||||
from pathlib import Path
|
||||
|
||||
try:
|
||||
import yaml
|
||||
except ImportError:
|
||||
print(
|
||||
"wiki_graph_lint.py requires PyYAML.\n"
|
||||
"Install with: pip install pyyaml",
|
||||
file=sys.stderr,
|
||||
)
|
||||
sys.exit(2)
|
||||
|
||||
# Same module is imported by extract; we re-use its build_nodes/build_edges to
|
||||
# guarantee lint sees exactly what extract would emit.
|
||||
SCRIPT_DIR = Path(__file__).resolve().parent
|
||||
sys.path.insert(0, str(SCRIPT_DIR))
|
||||
import wiki_graph_extract as _extract # noqa: E402
|
||||
|
||||
|
||||
FRONTMATTER_RE = re.compile(r"^---\s*\n(.*?)\n---\s*\n", re.DOTALL)
|
||||
WIKILINK_RE = re.compile(r"\[\[([^\]|]+)(?:\|[^\]]+)?\]\]")
|
||||
|
||||
SKIP_TOP_LEVEL_FILES = {"SCHEMA.md", "index.md", "log.md", "README.md"}
|
||||
SKIP_TOP_LEVEL_DIRS = {"indexes", "graph"}
|
||||
|
||||
ALLOWED_CONFIDENCE = {"high", "medium", "low"}
|
||||
ALLOWED_STATUS = {"current", "historical", "proposed", "disputed", "superseded"}
|
||||
IMPLICIT_PREDICATES = {"mentions", "sourced_from", "summarizes_raw"}
|
||||
|
||||
|
||||
def parse_frontmatter(text: str) -> tuple[dict, str]:
|
||||
m = FRONTMATTER_RE.match(text)
|
||||
if not m:
|
||||
return {}, text
|
||||
fm_text = m.group(1)
|
||||
body = text[m.end():]
|
||||
try:
|
||||
meta = yaml.safe_load(fm_text) or {}
|
||||
except yaml.YAMLError:
|
||||
meta = {}
|
||||
if not isinstance(meta, dict):
|
||||
meta = {}
|
||||
return meta, body
|
||||
|
||||
|
||||
def collect_pages(wiki_root: Path) -> list[dict]:
|
||||
pages = []
|
||||
for md_path in sorted(wiki_root.rglob("*.md")):
|
||||
rel = md_path.relative_to(wiki_root)
|
||||
if rel.parts[0] in SKIP_TOP_LEVEL_FILES or rel.parts[0] in SKIP_TOP_LEVEL_DIRS:
|
||||
continue
|
||||
if rel.name.startswith("."):
|
||||
continue
|
||||
try:
|
||||
text = md_path.read_text(encoding="utf-8")
|
||||
except (UnicodeDecodeError, OSError):
|
||||
continue
|
||||
meta, body = parse_frontmatter(text)
|
||||
pages.append({
|
||||
"path": str(md_path),
|
||||
"rel_path": str(rel).replace("\\", "/"),
|
||||
"slug": md_path.stem,
|
||||
"meta": meta,
|
||||
"body": body,
|
||||
"links": [m.group(1).strip() for m in WIKILINK_RE.finditer(body)],
|
||||
})
|
||||
return pages
|
||||
|
||||
|
||||
def derive_node_type(meta: dict, ontology: dict) -> str | None:
|
||||
page_type = meta.get("type")
|
||||
page_kind = meta.get("kind")
|
||||
explicit = (meta.get("graph") or {}).get("node_type") if isinstance(meta.get("graph"), dict) else None
|
||||
if explicit:
|
||||
return explicit
|
||||
type_kind_match = None
|
||||
type_only_match = None
|
||||
for nt_name, nt_def in ontology.get("node_types", {}).items():
|
||||
maps = (nt_def or {}).get("maps_from") or {}
|
||||
m_type = maps.get("type")
|
||||
m_kind = maps.get("kind")
|
||||
if m_type and m_type == page_type:
|
||||
if m_kind and m_kind == page_kind:
|
||||
type_kind_match = nt_name
|
||||
break
|
||||
if not m_kind and type_only_match is None:
|
||||
type_only_match = nt_name
|
||||
return type_kind_match or type_only_match
|
||||
|
||||
|
||||
def derive_node_id(meta: dict, slug: str, ontology: dict) -> str:
|
||||
graph_meta = meta.get("graph") if isinstance(meta.get("graph"), dict) else {}
|
||||
explicit = graph_meta.get("node_id")
|
||||
if explicit:
|
||||
return str(explicit)
|
||||
node_type = derive_node_type(meta, ontology) or "concept"
|
||||
return f"{node_type}:{slug}"
|
||||
|
||||
|
||||
def types_match(allowed: list[str] | None, actual: str | None) -> bool:
|
||||
if not allowed:
|
||||
return True
|
||||
if "*" in allowed:
|
||||
return True
|
||||
return actual in allowed
|
||||
|
||||
|
||||
def lint(pages: list[dict], ontology: dict) -> dict:
|
||||
findings = {
|
||||
"duplicate_node_ids": [],
|
||||
"unknown_predicates": [],
|
||||
"broken_object_refs": [],
|
||||
"subject_type_mismatch": [],
|
||||
"object_type_mismatch": [],
|
||||
"missing_evidence": [],
|
||||
"missing_source_field": [],
|
||||
"broken_source_refs": [],
|
||||
"invalid_confidence": [],
|
||||
"invalid_status": [],
|
||||
"duplicate_canonical": [],
|
||||
"alias_collisions": [],
|
||||
"broken_contradicts": [],
|
||||
"broken_supersedes": [],
|
||||
"orphan_typed_nodes": [],
|
||||
"summary": {},
|
||||
}
|
||||
|
||||
predicates = ontology.get("predicates", {})
|
||||
node_types = ontology.get("node_types", {})
|
||||
|
||||
# Build node index
|
||||
node_by_id: dict[str, dict] = {}
|
||||
duplicates: dict[str, list[str]] = defaultdict(list)
|
||||
for p in pages:
|
||||
nid = derive_node_id(p["meta"], p["slug"], ontology)
|
||||
if nid in node_by_id:
|
||||
duplicates[nid].append(p["rel_path"])
|
||||
duplicates[nid].append(node_by_id[nid]["rel_path"])
|
||||
continue
|
||||
node_type = derive_node_type(p["meta"], ontology) or "concept"
|
||||
graph_meta = p["meta"].get("graph") if isinstance(p["meta"].get("graph"), dict) else {}
|
||||
node_by_id[nid] = {
|
||||
"id": nid,
|
||||
"node_type": node_type,
|
||||
"rel_path": p["rel_path"],
|
||||
"slug": p["slug"],
|
||||
"page_type": p["meta"].get("type"),
|
||||
"canonical": bool(graph_meta.get("canonical", False)),
|
||||
"aliases": list(graph_meta.get("aliases") or []),
|
||||
"graph": graph_meta,
|
||||
}
|
||||
|
||||
for nid, paths in duplicates.items():
|
||||
findings["duplicate_node_ids"].append({"node_id": nid, "paths": sorted(set(paths))})
|
||||
|
||||
# Source pages by slug — used to validate `source:` refs on edges.
|
||||
source_slugs = {p["slug"] for p in pages if p["meta"].get("type") == "source"}
|
||||
|
||||
# Aliases
|
||||
alias_to_canonicals: dict[str, set[str]] = defaultdict(set)
|
||||
canonical_by_id: dict[str, list[str]] = defaultdict(list)
|
||||
for n in node_by_id.values():
|
||||
if n["canonical"]:
|
||||
canonical_by_id[n["id"]].append(n["rel_path"])
|
||||
for alias in n["aliases"]:
|
||||
alias_to_canonicals[str(alias)].add(n["id"])
|
||||
|
||||
for nid, paths in canonical_by_id.items():
|
||||
if len(paths) > 1:
|
||||
findings["duplicate_canonical"].append({"node_id": nid, "paths": paths})
|
||||
|
||||
for alias, owners in alias_to_canonicals.items():
|
||||
if len(owners) > 1:
|
||||
findings["alias_collisions"].append({"alias": alias, "owners": sorted(owners)})
|
||||
|
||||
# Walk relationships
|
||||
for p in pages:
|
||||
graph_meta = p["meta"].get("graph") if isinstance(p["meta"].get("graph"), dict) else {}
|
||||
subject_id = derive_node_id(p["meta"], p["slug"], ontology)
|
||||
subject_type = node_by_id.get(subject_id, {}).get("node_type")
|
||||
|
||||
for idx, rel in enumerate(graph_meta.get("relationships") or []):
|
||||
if not isinstance(rel, dict):
|
||||
continue
|
||||
predicate = rel.get("predicate")
|
||||
obj = rel.get("object")
|
||||
here = {"page": p["rel_path"], "predicate": predicate,
|
||||
"object": obj, "index": idx}
|
||||
|
||||
if not predicate or predicate not in predicates:
|
||||
findings["unknown_predicates"].append({**here})
|
||||
continue
|
||||
pdef = predicates[predicate] or {}
|
||||
|
||||
# Object resolution. Allow string-literal objects only when
|
||||
# ontology lists "*" in object_types (e.g. summarizes_raw).
|
||||
object_types = pdef.get("object_types") or []
|
||||
allows_wildcard_obj = "*" in object_types
|
||||
if obj and obj not in node_by_id:
|
||||
if not allows_wildcard_obj:
|
||||
findings["broken_object_refs"].append({**here})
|
||||
|
||||
# Subject type check
|
||||
if not types_match(pdef.get("subject_types"), subject_type):
|
||||
findings["subject_type_mismatch"].append({
|
||||
**here,
|
||||
"subject": subject_id,
|
||||
"subject_type": subject_type,
|
||||
"allowed": pdef.get("subject_types"),
|
||||
})
|
||||
# Object type check (only if object resolves to a node)
|
||||
obj_node = node_by_id.get(obj) if obj else None
|
||||
obj_type = obj_node["node_type"] if obj_node else None
|
||||
if obj_node and not types_match(pdef.get("object_types"), obj_type):
|
||||
findings["object_type_mismatch"].append({
|
||||
**here,
|
||||
"object_type": obj_type,
|
||||
"allowed": pdef.get("object_types"),
|
||||
})
|
||||
|
||||
requires_evidence = pdef.get("requires_evidence", True)
|
||||
if requires_evidence:
|
||||
if not rel.get("evidence"):
|
||||
findings["missing_evidence"].append({**here})
|
||||
if not rel.get("source"):
|
||||
findings["missing_source_field"].append({**here})
|
||||
|
||||
# source field must reference an existing source page slug
|
||||
src = rel.get("source")
|
||||
if src and str(src) not in source_slugs:
|
||||
findings["broken_source_refs"].append({**here, "source": src})
|
||||
|
||||
confidence = rel.get("confidence")
|
||||
if confidence and confidence not in ALLOWED_CONFIDENCE:
|
||||
findings["invalid_confidence"].append({**here, "confidence": confidence})
|
||||
|
||||
status = rel.get("status")
|
||||
if status and status not in ALLOWED_STATUS:
|
||||
findings["invalid_status"].append({**here, "status": status})
|
||||
|
||||
# contradicts / supersedes resolution
|
||||
for ref_field, bucket in (("contradicts", "broken_contradicts"),
|
||||
("supersedes", "broken_supersedes")):
|
||||
ref = rel.get(ref_field)
|
||||
if ref:
|
||||
ref_str = str(ref)
|
||||
if ref_str not in node_by_id and ref_str not in source_slugs:
|
||||
findings[bucket].append({**here, ref_field: ref_str})
|
||||
|
||||
# Orphan typed nodes — pages that declared `graph:` frontmatter but end
|
||||
# up with no typed (non-implicit) edge touching them after extraction.
|
||||
# Source nodes are exempt (they participate via implicit edges).
|
||||
extracted_edges = _extract.build_edges(pages, {n["slug"]: n["id"] for n in node_by_id.values()})
|
||||
typed_node_refs: set[str] = set()
|
||||
for e in extracted_edges:
|
||||
if e["predicate"] in IMPLICIT_PREDICATES:
|
||||
continue
|
||||
typed_node_refs.add(e["subject"])
|
||||
if e["object"] in node_by_id:
|
||||
typed_node_refs.add(e["object"])
|
||||
|
||||
for n in node_by_id.values():
|
||||
if n["node_type"] == "source":
|
||||
continue
|
||||
graph_meta = n.get("graph") or {}
|
||||
if not graph_meta:
|
||||
continue # Pages without graph metadata are valid; they're text-only nodes.
|
||||
if n["id"] in typed_node_refs:
|
||||
continue
|
||||
findings["orphan_typed_nodes"].append({
|
||||
"node_id": n["id"],
|
||||
"path": n["rel_path"],
|
||||
})
|
||||
|
||||
# Summary
|
||||
findings["summary"] = {
|
||||
"pages_scanned": len(pages),
|
||||
"nodes": len(node_by_id),
|
||||
**{k: len(v) for k, v in findings.items() if isinstance(v, list)},
|
||||
}
|
||||
return findings
|
||||
|
||||
|
||||
def render_text(findings: dict) -> str:
|
||||
out = []
|
||||
s = findings["summary"]
|
||||
out.append("=" * 60)
|
||||
out.append("Wiki Graph Lint Report")
|
||||
out.append("=" * 60)
|
||||
out.append(f"Pages scanned: {s['pages_scanned']} Nodes: {s['nodes']}")
|
||||
out.append("")
|
||||
|
||||
sections = [
|
||||
("duplicate_node_ids", "Duplicate node ids",
|
||||
lambda f: f" - {f['node_id']}: {', '.join(f['paths'])}"),
|
||||
("unknown_predicates", "Unknown predicates (not in ontology)",
|
||||
lambda f: f" - {f['page']}#rel[{f['index']}] predicate={f['predicate']!r}"),
|
||||
("broken_object_refs", "Broken object references",
|
||||
lambda f: f" - {f['page']}#rel[{f['index']}] {f['predicate']} → {f['object']!r}"),
|
||||
("subject_type_mismatch", "Subject type does not match ontology",
|
||||
lambda f: f" - {f['page']}#rel[{f['index']}] {f['predicate']}: subject={f['subject_type']} (allowed: {f['allowed']})"),
|
||||
("object_type_mismatch", "Object type does not match ontology",
|
||||
lambda f: f" - {f['page']}#rel[{f['index']}] {f['predicate']}: object={f['object_type']} (allowed: {f['allowed']})"),
|
||||
("missing_evidence", "Missing evidence on typed edge",
|
||||
lambda f: f" - {f['page']}#rel[{f['index']}] {f['predicate']} → {f['object']}"),
|
||||
("missing_source_field", "Missing source on typed edge",
|
||||
lambda f: f" - {f['page']}#rel[{f['index']}] {f['predicate']} → {f['object']}"),
|
||||
("broken_source_refs", "source: does not match any source page",
|
||||
lambda f: f" - {f['page']}#rel[{f['index']}] source={f['source']!r}"),
|
||||
("invalid_confidence", "Invalid confidence value",
|
||||
lambda f: f" - {f['page']}#rel[{f['index']}] confidence={f['confidence']!r}"),
|
||||
("invalid_status", "Invalid status value",
|
||||
lambda f: f" - {f['page']}#rel[{f['index']}] status={f['status']!r}"),
|
||||
("duplicate_canonical", "Duplicate canonical nodes",
|
||||
lambda f: f" - {f['node_id']}: {', '.join(f['paths'])}"),
|
||||
("alias_collisions", "Alias used by multiple canonical nodes",
|
||||
lambda f: f" - {f['alias']!r}: {', '.join(f['owners'])}"),
|
||||
("broken_contradicts", "Broken contradicts reference",
|
||||
lambda f: f" - {f['page']}#rel[{f['index']}] contradicts={f.get('contradicts')}"),
|
||||
("broken_supersedes", "Broken supersedes reference",
|
||||
lambda f: f" - {f['page']}#rel[{f['index']}] supersedes={f.get('supersedes')}"),
|
||||
("orphan_typed_nodes", "Orphan typed nodes (no inbound or outbound typed edges)",
|
||||
lambda f: f" - {f['node_id']} ({f['path']})"),
|
||||
]
|
||||
|
||||
healthy = True
|
||||
for key, label, formatter in sections:
|
||||
items = findings[key]
|
||||
if not items:
|
||||
continue
|
||||
healthy = False
|
||||
out.append(f"{label} ({len(items)}):")
|
||||
for item in items[:50]:
|
||||
out.append(formatter(item))
|
||||
if len(items) > 50:
|
||||
out.append(f" ... and {len(items) - 50} more")
|
||||
out.append("")
|
||||
|
||||
if healthy:
|
||||
out.append("No graph issues found.")
|
||||
return "\n".join(out)
|
||||
|
||||
|
||||
def main():
|
||||
parser = argparse.ArgumentParser(description=__doc__, formatter_class=argparse.RawDescriptionHelpFormatter)
|
||||
parser.add_argument("wiki", nargs="?", type=Path, default=Path("cml/wiki"))
|
||||
parser.add_argument("--ontology", type=Path, help="Ontology file (default: <wiki>/graph/ontology.yaml)")
|
||||
parser.add_argument("--json", action="store_true")
|
||||
args = parser.parse_args()
|
||||
|
||||
if not args.wiki.exists():
|
||||
print(f"Wiki directory not found: {args.wiki}", file=sys.stderr)
|
||||
sys.exit(1)
|
||||
|
||||
ontology_path = args.ontology or (args.wiki / "graph" / "ontology.yaml")
|
||||
if not ontology_path.exists():
|
||||
print(f"Ontology not found: {ontology_path}", file=sys.stderr)
|
||||
print("Did you forget to seed wiki/graph/ontology.yaml? See assets/ontology.yaml.template.",
|
||||
file=sys.stderr)
|
||||
sys.exit(1)
|
||||
try:
|
||||
ontology = yaml.safe_load(ontology_path.read_text(encoding="utf-8")) or {}
|
||||
except yaml.YAMLError as e:
|
||||
print(f"Ontology parse error: {e}", file=sys.stderr)
|
||||
sys.exit(2)
|
||||
|
||||
pages = collect_pages(args.wiki)
|
||||
findings = lint(pages, ontology)
|
||||
|
||||
if args.json:
|
||||
print(json.dumps(findings, indent=2, default=str))
|
||||
else:
|
||||
print(render_text(findings))
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
main()
|
||||
@@ -1,267 +0,0 @@
|
||||
#!/usr/bin/env -S uv run --script
|
||||
# /// script
|
||||
# requires-python = ">=3.11"
|
||||
# dependencies = []
|
||||
# ///
|
||||
"""
|
||||
wiki_graph_query.py — Query the compiled wiki graph (graph.sqlite).
|
||||
|
||||
Use this to accelerate navigation: find what's connected to a node, list
|
||||
typed edges around a subject, find a path between two nodes, or dump every
|
||||
fact about a node. The graph is a navigation index — for high-stakes
|
||||
claims, follow the `source` field back to the wiki page and the raw file.
|
||||
|
||||
Subcommands:
|
||||
neighbors --node <id> List nodes one hop away from <id>
|
||||
edges --subject <id> List all outbound edges from <id>
|
||||
[--predicate <p>] Filter by predicate
|
||||
path --from <id> --to <id> Shortest directed path (BFS, max depth 6)
|
||||
[--max-depth N]
|
||||
facts --about <id> Outbound + inbound edges for <id>
|
||||
|
||||
Common options:
|
||||
--db <path> Path to graph.sqlite (default: <wiki>/graph/graph.sqlite)
|
||||
--json Emit JSON instead of text
|
||||
|
||||
Examples:
|
||||
python wiki_graph_query.py wiki/ neighbors --node product:konvy
|
||||
python wiki_graph_query.py wiki/ edges --subject person:stephanie-emmanouel
|
||||
python wiki_graph_query.py wiki/ path --from person:praney-behl --to product:konvy
|
||||
python wiki_graph_query.py wiki/ facts --about product:konvy
|
||||
"""
|
||||
|
||||
import argparse
|
||||
import json
|
||||
import sqlite3
|
||||
import sys
|
||||
from collections import deque
|
||||
from pathlib import Path
|
||||
|
||||
|
||||
EVIDENCE_SNIPPET_LEN = 140
|
||||
|
||||
|
||||
def open_db(path: Path) -> sqlite3.Connection:
|
||||
if not path.exists():
|
||||
print(f"graph.sqlite not found at {path}.", file=sys.stderr)
|
||||
print("Run wiki_graph_extract.py first.", file=sys.stderr)
|
||||
sys.exit(1)
|
||||
conn = sqlite3.connect(path)
|
||||
conn.row_factory = sqlite3.Row
|
||||
return conn
|
||||
|
||||
|
||||
def fetch_node(conn: sqlite3.Connection, node_id: str) -> dict | None:
|
||||
row = conn.execute("SELECT * FROM nodes WHERE id = ?", (node_id,)).fetchone()
|
||||
return dict(row) if row else None
|
||||
|
||||
|
||||
def edges_from(conn: sqlite3.Connection, subject: str, predicate: str | None = None) -> list[dict]:
|
||||
q = "SELECT * FROM edges WHERE subject = ?"
|
||||
params: list = [subject]
|
||||
if predicate:
|
||||
q += " AND predicate = ?"
|
||||
params.append(predicate)
|
||||
q += " ORDER BY predicate, object"
|
||||
return [dict(r) for r in conn.execute(q, params).fetchall()]
|
||||
|
||||
|
||||
def edges_to(conn: sqlite3.Connection, obj: str, predicate: str | None = None) -> list[dict]:
|
||||
q = "SELECT * FROM edges WHERE object = ?"
|
||||
params: list = [obj]
|
||||
if predicate:
|
||||
q += " AND predicate = ?"
|
||||
params.append(predicate)
|
||||
q += " ORDER BY predicate, subject"
|
||||
return [dict(r) for r in conn.execute(q, params).fetchall()]
|
||||
|
||||
|
||||
def truncate(text: str | None) -> str:
|
||||
if not text:
|
||||
return ""
|
||||
if len(text) <= EVIDENCE_SNIPPET_LEN:
|
||||
return text
|
||||
return text[: EVIDENCE_SNIPPET_LEN - 1].rstrip() + "…"
|
||||
|
||||
|
||||
def render_edge_row(e: dict) -> str:
|
||||
pieces = [
|
||||
f" {e['subject']} --[{e['predicate']}]--> {e['object']}",
|
||||
]
|
||||
confidence = e.get("confidence") or "-"
|
||||
status = e.get("status") or "-"
|
||||
src = e.get("source") or "-"
|
||||
pieces.append(f" via {src} conf={confidence} status={status}")
|
||||
if e.get("evidence"):
|
||||
pieces.append(f" evidence: {truncate(e['evidence'])}")
|
||||
pieces.append(f" (page: {e['page']})")
|
||||
return "\n".join(pieces)
|
||||
|
||||
|
||||
def cmd_neighbors(conn: sqlite3.Connection, args) -> dict:
|
||||
node = fetch_node(conn, args.node)
|
||||
if not node:
|
||||
print(f"node not found: {args.node}", file=sys.stderr)
|
||||
sys.exit(1)
|
||||
out_edges = edges_from(conn, args.node)
|
||||
in_edges = edges_to(conn, args.node)
|
||||
|
||||
neighbors: dict[str, dict] = {}
|
||||
for e in out_edges:
|
||||
neighbors.setdefault(e["object"], {"node_id": e["object"], "out": [], "in": []})
|
||||
neighbors[e["object"]]["out"].append(e)
|
||||
for e in in_edges:
|
||||
neighbors.setdefault(e["subject"], {"node_id": e["subject"], "out": [], "in": []})
|
||||
neighbors[e["subject"]]["in"].append(e)
|
||||
|
||||
# Resolve neighbor titles where possible
|
||||
for nid, slot in neighbors.items():
|
||||
target = fetch_node(conn, nid)
|
||||
slot["title"] = target["title"] if target else nid
|
||||
slot["path"] = target["path"] if target else None
|
||||
|
||||
return {
|
||||
"node": node,
|
||||
"neighbors": sorted(neighbors.values(), key=lambda n: n["node_id"]),
|
||||
}
|
||||
|
||||
|
||||
def cmd_edges(conn: sqlite3.Connection, args) -> dict:
|
||||
es = edges_from(conn, args.subject, args.predicate)
|
||||
return {"subject": args.subject, "predicate": args.predicate, "edges": es}
|
||||
|
||||
|
||||
def cmd_facts(conn: sqlite3.Connection, args) -> dict:
|
||||
node = fetch_node(conn, args.about)
|
||||
if not node:
|
||||
print(f"node not found: {args.about}", file=sys.stderr)
|
||||
sys.exit(1)
|
||||
return {
|
||||
"node": node,
|
||||
"outbound": edges_from(conn, args.about),
|
||||
"inbound": edges_to(conn, args.about),
|
||||
}
|
||||
|
||||
|
||||
def cmd_path(conn: sqlite3.Connection, args) -> dict:
|
||||
src = fetch_node(conn, getattr(args, "from"))
|
||||
dst = fetch_node(conn, args.to)
|
||||
if not src:
|
||||
print(f"from-node not found: {getattr(args, 'from')}", file=sys.stderr)
|
||||
sys.exit(1)
|
||||
if not dst:
|
||||
print(f"to-node not found: {args.to}", file=sys.stderr)
|
||||
sys.exit(1)
|
||||
|
||||
start, goal = getattr(args, "from"), args.to
|
||||
queue = deque([(start, [start], [])])
|
||||
visited = {start}
|
||||
while queue:
|
||||
node, node_path, edge_path = queue.popleft()
|
||||
if node == goal:
|
||||
return {"from": start, "to": goal, "path_nodes": node_path, "path_edges": edge_path}
|
||||
if len(node_path) - 1 >= args.max_depth:
|
||||
continue
|
||||
for e in edges_from(conn, node):
|
||||
nxt = e["object"]
|
||||
if nxt in visited:
|
||||
continue
|
||||
visited.add(nxt)
|
||||
queue.append((nxt, node_path + [nxt], edge_path + [e]))
|
||||
return {"from": start, "to": goal, "path_nodes": [], "path_edges": []}
|
||||
|
||||
|
||||
def render(result: dict, command: str) -> str:
|
||||
out: list[str] = []
|
||||
if command == "neighbors":
|
||||
n = result["node"]
|
||||
out.append(f"Node: {n['id']} ({n['title']}) {n['node_type']} [{n['path']}]")
|
||||
out.append(f"Neighbors: {len(result['neighbors'])}")
|
||||
for nb in result["neighbors"]:
|
||||
out.append("")
|
||||
out.append(f" → {nb['node_id']} ({nb['title']})")
|
||||
for e in nb.get("out", []):
|
||||
out.append(f" out [{e['predicate']}] conf={e.get('confidence') or '-'} src={e.get('source') or '-'}")
|
||||
for e in nb.get("in", []):
|
||||
out.append(f" in [{e['predicate']}] from {e['subject']} src={e.get('source') or '-'}")
|
||||
elif command == "edges":
|
||||
out.append(f"Edges from {result['subject']}"
|
||||
+ (f" with predicate {result['predicate']}" if result['predicate'] else ""))
|
||||
for e in result["edges"]:
|
||||
out.append("")
|
||||
out.append(render_edge_row(e))
|
||||
elif command == "facts":
|
||||
n = result["node"]
|
||||
out.append(f"Facts about {n['id']} ({n['title']}) [{n['path']}]")
|
||||
out.append("")
|
||||
out.append(f"Outbound ({len(result['outbound'])}):")
|
||||
for e in result["outbound"]:
|
||||
out.append(render_edge_row(e))
|
||||
out.append("")
|
||||
out.append(f"Inbound ({len(result['inbound'])}):")
|
||||
for e in result["inbound"]:
|
||||
out.append(render_edge_row(e))
|
||||
elif command == "path":
|
||||
if not result["path_nodes"]:
|
||||
out.append(f"No path found from {result['from']} to {result['to']} within depth limit.")
|
||||
else:
|
||||
out.append(f"Path from {result['from']} to {result['to']} ({len(result['path_edges'])} hops):")
|
||||
for e in result["path_edges"]:
|
||||
out.append("")
|
||||
out.append(render_edge_row(e))
|
||||
return "\n".join(out)
|
||||
|
||||
|
||||
def main():
|
||||
parser = argparse.ArgumentParser(description=__doc__, formatter_class=argparse.RawDescriptionHelpFormatter)
|
||||
parser.add_argument("wiki", type=Path, help="Wiki directory.")
|
||||
parser.add_argument("--db", type=Path, help="Path to graph.sqlite (default: <wiki>/graph/graph.sqlite)")
|
||||
parser.add_argument("--json", action="store_true")
|
||||
|
||||
sub = parser.add_subparsers(dest="command", required=True)
|
||||
|
||||
p_n = sub.add_parser("neighbors")
|
||||
p_n.add_argument("--node", required=True)
|
||||
|
||||
p_e = sub.add_parser("edges")
|
||||
p_e.add_argument("--subject", required=True)
|
||||
p_e.add_argument("--predicate")
|
||||
|
||||
p_p = sub.add_parser("path")
|
||||
p_p.add_argument("--from", dest="from", required=True)
|
||||
p_p.add_argument("--to", required=True)
|
||||
p_p.add_argument("--max-depth", type=int, default=6)
|
||||
|
||||
p_f = sub.add_parser("facts")
|
||||
p_f.add_argument("--about", required=True)
|
||||
|
||||
args = parser.parse_args()
|
||||
|
||||
if not args.wiki.exists():
|
||||
print(f"Wiki directory not found: {args.wiki}", file=sys.stderr)
|
||||
sys.exit(1)
|
||||
db_path = args.db or (args.wiki / "graph" / "graph.sqlite")
|
||||
conn = open_db(db_path)
|
||||
try:
|
||||
if args.command == "neighbors":
|
||||
result = cmd_neighbors(conn, args)
|
||||
elif args.command == "edges":
|
||||
result = cmd_edges(conn, args)
|
||||
elif args.command == "path":
|
||||
result = cmd_path(conn, args)
|
||||
elif args.command == "facts":
|
||||
result = cmd_facts(conn, args)
|
||||
else:
|
||||
parser.print_help()
|
||||
sys.exit(1)
|
||||
finally:
|
||||
conn.close()
|
||||
|
||||
if args.json:
|
||||
print(json.dumps(result, indent=2, default=str))
|
||||
else:
|
||||
print(render(result, args.command))
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
main()
|
||||
@@ -1,319 +0,0 @@
|
||||
#!/usr/bin/env -S uv run --script
|
||||
# /// script
|
||||
# requires-python = ">=3.11"
|
||||
# dependencies = []
|
||||
# ///
|
||||
"""
|
||||
wiki_lint.py — Structural health check for an LLM Wiki.
|
||||
|
||||
Reports orphan pages, broken wikilinks, oversized pages, frontmatter issues,
|
||||
stale pages, duplicate slugs, and (with --suggest-pages) terms that appear in
|
||||
many pages without their own page.
|
||||
|
||||
Conservative by design: reports findings, never edits.
|
||||
|
||||
Usage:
|
||||
python wiki_lint.py [<wiki-dir>] [options]
|
||||
|
||||
Options:
|
||||
--soft-cap N Page-size soft cap in lines (default: 400)
|
||||
--hard-cap N Page-size hard cap in lines (default: 800)
|
||||
--required-fm a,b Required frontmatter fields (default: type,title,tags,created,updated)
|
||||
--suggest-pages Surface terms appearing in many pages without a page
|
||||
--suggest-min N Minimum occurrences for --suggest-pages (default: 5)
|
||||
--json Emit JSON instead of text
|
||||
|
||||
Examples:
|
||||
python wiki_lint.py wiki/
|
||||
python wiki_lint.py wiki/ --suggest-pages
|
||||
python wiki_lint.py wiki/ --json > lint.json
|
||||
"""
|
||||
|
||||
import argparse
|
||||
import json
|
||||
import re
|
||||
import sys
|
||||
from collections import Counter, defaultdict
|
||||
from datetime import date, datetime
|
||||
from pathlib import Path
|
||||
|
||||
|
||||
WIKILINK_RE = re.compile(r"\[\[([^\]|]+)(?:\|[^\]]+)?\]\]")
|
||||
FRONTMATTER_RE = re.compile(r"^---\s*\n(.*?)\n---\s*\n", re.DOTALL)
|
||||
CAPITALIZED_PHRASE_RE = re.compile(r"\b([A-Z][a-zA-Z0-9]+(?:\s+[A-Z][a-zA-Z0-9]+){0,3})\b")
|
||||
|
||||
|
||||
SKIP_TOP_LEVEL_FILES = {"SCHEMA.md", "index.md", "log.md", "README.md"}
|
||||
SKIP_TOP_LEVEL_DIRS = {"indexes", "graph"}
|
||||
|
||||
|
||||
def parse_frontmatter(text: str) -> tuple[dict, str, bool]:
|
||||
"""Returns (metadata, body, malformed). malformed=True if frontmatter was attempted but unparseable."""
|
||||
if not text.startswith("---"):
|
||||
return {}, text, False
|
||||
m = FRONTMATTER_RE.match(text)
|
||||
if not m:
|
||||
return {}, text, True
|
||||
fm_text = m.group(1)
|
||||
body = text[m.end():]
|
||||
meta = {}
|
||||
current_key = None
|
||||
for line in fm_text.split("\n"):
|
||||
if not line.strip():
|
||||
continue
|
||||
kv = re.match(r"^([a-zA-Z_]+):\s*(.*)$", line)
|
||||
if kv:
|
||||
key, value = kv.group(1), kv.group(2).strip()
|
||||
if value.startswith("[") and value.endswith("]"):
|
||||
items = [x.strip().strip('"').strip("'") for x in value[1:-1].split(",") if x.strip()]
|
||||
meta[key] = items
|
||||
elif value:
|
||||
meta[key] = value.strip('"').strip("'")
|
||||
else:
|
||||
meta[key] = []
|
||||
current_key = key
|
||||
elif line.startswith(" - ") and current_key:
|
||||
meta[current_key].append(line[4:].strip().strip('"').strip("'"))
|
||||
return meta, body, False
|
||||
|
||||
|
||||
def collect_pages(wiki_root: Path) -> list[dict]:
|
||||
pages = []
|
||||
for md_path in wiki_root.rglob("*.md"):
|
||||
rel = md_path.relative_to(wiki_root)
|
||||
if rel.parts[0] in SKIP_TOP_LEVEL_FILES or rel.parts[0] in SKIP_TOP_LEVEL_DIRS:
|
||||
continue
|
||||
if rel.name.startswith("."):
|
||||
continue
|
||||
try:
|
||||
text = md_path.read_text(encoding="utf-8")
|
||||
except (UnicodeDecodeError, OSError) as e:
|
||||
pages.append({
|
||||
"path": str(md_path),
|
||||
"rel_path": str(rel),
|
||||
"slug": md_path.stem,
|
||||
"read_error": str(e),
|
||||
})
|
||||
continue
|
||||
meta, body, malformed = parse_frontmatter(text)
|
||||
line_count = text.count("\n") + 1
|
||||
links = [m.group(1).strip() for m in WIKILINK_RE.finditer(body)]
|
||||
pages.append({
|
||||
"path": str(md_path),
|
||||
"rel_path": str(rel),
|
||||
"slug": md_path.stem,
|
||||
"meta": meta,
|
||||
"body": body,
|
||||
"line_count": line_count,
|
||||
"links": links,
|
||||
"malformed_fm": malformed,
|
||||
})
|
||||
return pages
|
||||
|
||||
|
||||
def parse_date(s):
|
||||
if not s or not isinstance(s, str):
|
||||
return None
|
||||
try:
|
||||
return datetime.strptime(s[:10], "%Y-%m-%d").date()
|
||||
except (ValueError, TypeError):
|
||||
return None
|
||||
|
||||
|
||||
def lint(pages: list[dict], soft_cap: int, hard_cap: int, required_fm: list[str], suggest_pages: bool, suggest_min: int) -> dict:
|
||||
findings = {
|
||||
"orphans": [],
|
||||
"broken_links": [],
|
||||
"oversized_hard": [],
|
||||
"oversized_soft": [],
|
||||
"missing_frontmatter": [],
|
||||
"malformed_frontmatter": [],
|
||||
"duplicate_slugs": [],
|
||||
"stale_pages": [],
|
||||
"read_errors": [],
|
||||
"suggested_pages": [],
|
||||
"summary": {},
|
||||
}
|
||||
|
||||
# Read errors
|
||||
for p in pages:
|
||||
if "read_error" in p:
|
||||
findings["read_errors"].append({"path": p["rel_path"], "error": p["read_error"]})
|
||||
|
||||
pages = [p for p in pages if "read_error" not in p]
|
||||
|
||||
# Slugs
|
||||
slug_to_pages = defaultdict(list)
|
||||
for p in pages:
|
||||
slug_to_pages[p["slug"]].append(p["rel_path"])
|
||||
for slug, paths in slug_to_pages.items():
|
||||
if len(paths) > 1:
|
||||
findings["duplicate_slugs"].append({"slug": slug, "paths": paths})
|
||||
|
||||
# Inbound link map
|
||||
inbound = defaultdict(set)
|
||||
all_slugs = set(slug_to_pages.keys())
|
||||
for p in pages:
|
||||
for link in p["links"]:
|
||||
inbound[link].add(p["slug"])
|
||||
|
||||
# Orphans, broken links, oversize, frontmatter, staleness
|
||||
for p in pages:
|
||||
# Orphans
|
||||
if not inbound.get(p["slug"]):
|
||||
findings["orphans"].append({"slug": p["slug"], "path": p["rel_path"]})
|
||||
|
||||
# Broken links
|
||||
for link in p["links"]:
|
||||
if link not in all_slugs:
|
||||
findings["broken_links"].append({
|
||||
"from": p["slug"],
|
||||
"from_path": p["rel_path"],
|
||||
"to": link,
|
||||
})
|
||||
|
||||
# Oversize
|
||||
if p["line_count"] > hard_cap:
|
||||
findings["oversized_hard"].append({"path": p["rel_path"], "lines": p["line_count"]})
|
||||
elif p["line_count"] > soft_cap:
|
||||
findings["oversized_soft"].append({"path": p["rel_path"], "lines": p["line_count"]})
|
||||
|
||||
# Frontmatter
|
||||
if p["malformed_fm"]:
|
||||
findings["malformed_frontmatter"].append({"path": p["rel_path"]})
|
||||
else:
|
||||
missing = [field for field in required_fm if field not in p["meta"] or p["meta"].get(field) in ("", None, [])]
|
||||
if missing:
|
||||
findings["missing_frontmatter"].append({"path": p["rel_path"], "missing": missing})
|
||||
|
||||
# Staleness: heuristic — page hasn't been updated in 90 days AND has been touched by recent ingests.
|
||||
# Approximate: if updated > 90d ago and the page is well-linked (a hub), flag it.
|
||||
updated = parse_date(p["meta"].get("updated"))
|
||||
if updated:
|
||||
age_days = (date.today() - updated).days
|
||||
if age_days > 90 and len(inbound.get(p["slug"], [])) >= 3:
|
||||
findings["stale_pages"].append({
|
||||
"path": p["rel_path"],
|
||||
"updated": p["meta"].get("updated"),
|
||||
"age_days": age_days,
|
||||
"inbound_count": len(inbound.get(p["slug"], [])),
|
||||
})
|
||||
|
||||
# Suggested pages: capitalized multi-word phrases appearing in many pages without a page
|
||||
if suggest_pages:
|
||||
phrase_pages = defaultdict(set)
|
||||
for p in pages:
|
||||
seen = set()
|
||||
for m in CAPITALIZED_PHRASE_RE.finditer(p["body"]):
|
||||
phrase = m.group(1).strip()
|
||||
seen.add(phrase)
|
||||
for phrase in seen:
|
||||
phrase_pages[phrase].add(p["slug"])
|
||||
|
||||
# Title set for filtering
|
||||
existing_titles = {p["meta"].get("title", "").lower() for p in pages}
|
||||
existing_slugs_normalized = {s.lower().replace("-", " ") for s in all_slugs}
|
||||
|
||||
candidates = []
|
||||
for phrase, page_set in phrase_pages.items():
|
||||
if len(page_set) < suggest_min:
|
||||
continue
|
||||
if phrase.lower() in existing_titles:
|
||||
continue
|
||||
if phrase.lower() in existing_slugs_normalized:
|
||||
continue
|
||||
# Filter out section header garbage
|
||||
if phrase.split()[0] in {"Section", "Where", "Sources", "Tags", "Type", "Title"}:
|
||||
continue
|
||||
candidates.append({"phrase": phrase, "page_count": len(page_set), "pages": sorted(page_set)[:5]})
|
||||
candidates.sort(key=lambda x: -x["page_count"])
|
||||
findings["suggested_pages"] = candidates[:30]
|
||||
|
||||
findings["summary"] = {
|
||||
"total_pages": len(pages),
|
||||
"orphans": len(findings["orphans"]),
|
||||
"broken_links": len(findings["broken_links"]),
|
||||
"oversized_hard": len(findings["oversized_hard"]),
|
||||
"oversized_soft": len(findings["oversized_soft"]),
|
||||
"missing_frontmatter": len(findings["missing_frontmatter"]),
|
||||
"malformed_frontmatter": len(findings["malformed_frontmatter"]),
|
||||
"duplicate_slugs": len(findings["duplicate_slugs"]),
|
||||
"stale_pages": len(findings["stale_pages"]),
|
||||
"read_errors": len(findings["read_errors"]),
|
||||
"suggested_pages": len(findings["suggested_pages"]),
|
||||
}
|
||||
return findings
|
||||
|
||||
|
||||
def render_text(findings: dict) -> str:
|
||||
out = []
|
||||
s = findings["summary"]
|
||||
out.append("=" * 60)
|
||||
out.append("Wiki Lint Report")
|
||||
out.append("=" * 60)
|
||||
out.append(f"Total pages scanned: {s['total_pages']}")
|
||||
out.append("")
|
||||
|
||||
sections = [
|
||||
("orphans", "Orphan pages (no inbound links)", lambda f: f" - {f['slug']} ({f['path']})"),
|
||||
("broken_links", "Broken wikilinks", lambda f: f" - [[{f['to']}]] referenced from {f['from_path']}"),
|
||||
("oversized_hard", "OVERSIZE (over hard cap — must split)", lambda f: f" - {f['path']} ({f['lines']} lines)"),
|
||||
("oversized_soft", "Oversize (over soft cap — consider splitting)", lambda f: f" - {f['path']} ({f['lines']} lines)"),
|
||||
("missing_frontmatter", "Missing frontmatter fields", lambda f: f" - {f['path']} missing: {', '.join(f['missing'])}"),
|
||||
("malformed_frontmatter", "Malformed frontmatter", lambda f: f" - {f['path']}"),
|
||||
("duplicate_slugs", "Duplicate slugs", lambda f: f" - {f['slug']}: {', '.join(f['paths'])}"),
|
||||
("stale_pages", "Stale pages (well-linked but not updated in 90+ days)", lambda f: f" - {f['path']} (updated {f['updated']}, {f['age_days']}d ago, {f['inbound_count']} inbound)"),
|
||||
("read_errors", "Read errors", lambda f: f" - {f['path']}: {f['error']}"),
|
||||
]
|
||||
|
||||
for key, label, formatter in sections:
|
||||
items = findings[key]
|
||||
if not items:
|
||||
continue
|
||||
out.append(f"{label} ({len(items)}):")
|
||||
for item in items[:50]:
|
||||
out.append(formatter(item))
|
||||
if len(items) > 50:
|
||||
out.append(f" ... and {len(items) - 50} more")
|
||||
out.append("")
|
||||
|
||||
if findings["suggested_pages"]:
|
||||
out.append(f"Suggested page candidates ({len(findings['suggested_pages'])}):")
|
||||
out.append(" Phrases appearing in many pages without a dedicated page:")
|
||||
for item in findings["suggested_pages"]:
|
||||
out.append(f" - \"{item['phrase']}\" ({item['page_count']} pages)")
|
||||
out.append("")
|
||||
|
||||
if all(v == 0 for k, v in s.items() if k != "total_pages"):
|
||||
out.append("No issues found. Wiki is healthy.")
|
||||
|
||||
return "\n".join(out)
|
||||
|
||||
|
||||
def main():
|
||||
parser = argparse.ArgumentParser(description=__doc__, formatter_class=argparse.RawDescriptionHelpFormatter)
|
||||
parser.add_argument("wiki", nargs="?", type=Path, default=Path("cml/wiki"), help="Wiki directory (default: cml/wiki).")
|
||||
parser.add_argument("--soft-cap", type=int, default=400, help="Page-size soft cap (lines).")
|
||||
parser.add_argument("--hard-cap", type=int, default=800, help="Page-size hard cap (lines).")
|
||||
parser.add_argument("--required-fm", default="type,title,tags,created,updated", help="Required frontmatter fields, comma-separated.")
|
||||
parser.add_argument("--suggest-pages", action="store_true", help="Surface page candidates.")
|
||||
parser.add_argument("--suggest-min", type=int, default=5, help="Minimum page count for suggestions.")
|
||||
parser.add_argument("--json", action="store_true", help="Emit JSON.")
|
||||
args = parser.parse_args()
|
||||
|
||||
if not args.wiki.exists():
|
||||
print(f"Wiki directory not found: {args.wiki}", file=sys.stderr)
|
||||
sys.exit(1)
|
||||
|
||||
pages = collect_pages(args.wiki)
|
||||
required_fm = [f.strip() for f in args.required_fm.split(",") if f.strip()]
|
||||
findings = lint(pages, args.soft_cap, args.hard_cap, required_fm, args.suggest_pages, args.suggest_min)
|
||||
|
||||
if args.json:
|
||||
print(json.dumps(findings, indent=2, default=str))
|
||||
else:
|
||||
print(render_text(findings))
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
main()
|
||||
@@ -1,270 +0,0 @@
|
||||
#!/usr/bin/env -S uv run --script
|
||||
# /// script
|
||||
# requires-python = ">=3.11"
|
||||
# dependencies = []
|
||||
# ///
|
||||
"""
|
||||
wiki_search.py — BM25 search over wiki pages with frontmatter filters.
|
||||
|
||||
Fallback for when index-first navigation doesn't surface the right pages.
|
||||
Pure-Python implementation (no dependencies beyond stdlib) so it runs anywhere.
|
||||
|
||||
Usage:
|
||||
python wiki_search.py "query terms" [options]
|
||||
|
||||
Options:
|
||||
--wiki <dir> Wiki directory (default: cml/wiki)
|
||||
--top N Return top N results (default: 10)
|
||||
--type <type> Filter by frontmatter type (source|entity|concept|synthesis|...)
|
||||
--tag <tag> Filter by tag (repeatable)
|
||||
--since YYYY-MM-DD Only pages updated on or after this date
|
||||
--backlinks <slug> Find pages that link to <slug>; ignores the query
|
||||
--top-linked N Show the N most-linked-to pages (hubs); ignores the query
|
||||
--cache <path> Persist the BM25 index to disk for faster reruns
|
||||
|
||||
Examples:
|
||||
python wiki_search.py "diffusion training stability" --top 5
|
||||
python wiki_search.py "alignment" --type concept --tag safety
|
||||
python wiki_search.py "" --backlinks transformer
|
||||
python wiki_search.py "" --top-linked 10
|
||||
"""
|
||||
|
||||
import argparse
|
||||
import json
|
||||
import math
|
||||
import re
|
||||
import sys
|
||||
from collections import Counter, defaultdict
|
||||
from datetime import date, datetime
|
||||
from pathlib import Path
|
||||
|
||||
|
||||
WIKILINK_RE = re.compile(r"\[\[([^\]|]+)(?:\|[^\]]+)?\]\]")
|
||||
FRONTMATTER_RE = re.compile(r"^---\s*\n(.*?)\n---\s*\n", re.DOTALL)
|
||||
TOKEN_RE = re.compile(r"[a-z0-9]+")
|
||||
|
||||
|
||||
def parse_frontmatter(text: str) -> tuple[dict, str]:
|
||||
"""Lightweight YAML-ish frontmatter parser. Returns (metadata, body)."""
|
||||
m = FRONTMATTER_RE.match(text)
|
||||
if not m:
|
||||
return {}, text
|
||||
fm_text = m.group(1)
|
||||
body = text[m.end():]
|
||||
meta = {}
|
||||
current_key = None
|
||||
for line in fm_text.split("\n"):
|
||||
if not line.strip():
|
||||
continue
|
||||
# Inline list: tags: [a, b, c]
|
||||
kv = re.match(r"^([a-zA-Z_]+):\s*(.*)$", line)
|
||||
if kv:
|
||||
key, value = kv.group(1), kv.group(2).strip()
|
||||
if value.startswith("[") and value.endswith("]"):
|
||||
items = [x.strip().strip('"').strip("'") for x in value[1:-1].split(",") if x.strip()]
|
||||
meta[key] = items
|
||||
elif value:
|
||||
meta[key] = value.strip('"').strip("'")
|
||||
else:
|
||||
meta[key] = []
|
||||
current_key = key
|
||||
elif line.startswith(" - ") and current_key:
|
||||
meta[current_key].append(line[4:].strip().strip('"').strip("'"))
|
||||
return meta, body
|
||||
|
||||
|
||||
def tokenize(text: str) -> list[str]:
|
||||
return TOKEN_RE.findall(text.lower())
|
||||
|
||||
|
||||
def slug_from_path(path: Path, wiki_root: Path) -> str:
|
||||
return path.stem
|
||||
|
||||
|
||||
def extract_wikilinks(body: str) -> list[str]:
|
||||
return [m.group(1).strip() for m in WIKILINK_RE.finditer(body)]
|
||||
|
||||
|
||||
def collect_pages(wiki_root: Path) -> list[dict]:
|
||||
"""Walk the wiki and return [{path, slug, meta, body, tokens, links}]."""
|
||||
pages = []
|
||||
for md_path in wiki_root.rglob("*.md"):
|
||||
# Skip the schema, index, log, and template files
|
||||
rel = md_path.relative_to(wiki_root)
|
||||
if rel.parts[0] in {"SCHEMA.md", "index.md", "log.md"} or rel.name.startswith("."):
|
||||
continue
|
||||
if rel.parts[0] in {"indexes", "graph"}:
|
||||
continue
|
||||
try:
|
||||
text = md_path.read_text(encoding="utf-8")
|
||||
except (UnicodeDecodeError, OSError):
|
||||
continue
|
||||
meta, body = parse_frontmatter(text)
|
||||
pages.append({
|
||||
"path": str(md_path),
|
||||
"rel_path": str(rel),
|
||||
"slug": slug_from_path(md_path, wiki_root),
|
||||
"meta": meta,
|
||||
"body": body,
|
||||
"tokens": tokenize(body + " " + meta.get("title", "")),
|
||||
"links": extract_wikilinks(body),
|
||||
})
|
||||
return pages
|
||||
|
||||
|
||||
def build_bm25(pages: list[dict]) -> dict:
|
||||
"""Build a BM25 index. Returns {df, avgdl, N, doc_lens, term_freqs}."""
|
||||
N = len(pages)
|
||||
df = Counter()
|
||||
doc_lens = []
|
||||
term_freqs = []
|
||||
for page in pages:
|
||||
tokens = page["tokens"]
|
||||
doc_lens.append(len(tokens))
|
||||
tf = Counter(tokens)
|
||||
term_freqs.append(tf)
|
||||
for term in tf:
|
||||
df[term] += 1
|
||||
avgdl = sum(doc_lens) / N if N else 0
|
||||
return {"N": N, "df": df, "avgdl": avgdl, "doc_lens": doc_lens, "term_freqs": term_freqs}
|
||||
|
||||
|
||||
def bm25_score(query_tokens: list[str], doc_idx: int, idx: dict, k1: float = 1.5, b: float = 0.75) -> float:
|
||||
score = 0.0
|
||||
N = idx["N"]
|
||||
df = idx["df"]
|
||||
avgdl = idx["avgdl"]
|
||||
dl = idx["doc_lens"][doc_idx]
|
||||
tf = idx["term_freqs"][doc_idx]
|
||||
for term in query_tokens:
|
||||
if term not in df:
|
||||
continue
|
||||
idf = math.log(1 + (N - df[term] + 0.5) / (df[term] + 0.5))
|
||||
f = tf.get(term, 0)
|
||||
if f == 0:
|
||||
continue
|
||||
denom = f + k1 * (1 - b + b * (dl / avgdl if avgdl else 1))
|
||||
score += idf * (f * (k1 + 1)) / denom
|
||||
return score
|
||||
|
||||
|
||||
def parse_date(s: str | None) -> date | None:
|
||||
if not s:
|
||||
return None
|
||||
try:
|
||||
return datetime.strptime(s[:10], "%Y-%m-%d").date()
|
||||
except (ValueError, TypeError):
|
||||
return None
|
||||
|
||||
|
||||
def passes_filters(page: dict, args) -> bool:
|
||||
meta = page["meta"]
|
||||
if args.type and meta.get("type") != args.type:
|
||||
return False
|
||||
if args.tag:
|
||||
page_tags = set(meta.get("tags", []) or [])
|
||||
if not all(t in page_tags for t in args.tag):
|
||||
return False
|
||||
if args.since:
|
||||
since = parse_date(args.since)
|
||||
updated = parse_date(meta.get("updated"))
|
||||
if since and updated and updated < since:
|
||||
return False
|
||||
if since and not updated:
|
||||
return False
|
||||
return True
|
||||
|
||||
|
||||
def cmd_search(args, pages: list[dict]) -> None:
|
||||
filtered = [p for p in pages if passes_filters(p, args)]
|
||||
if not filtered:
|
||||
print("No pages matched the filters.", file=sys.stderr)
|
||||
return
|
||||
idx = build_bm25(filtered)
|
||||
query_tokens = tokenize(args.query)
|
||||
if not query_tokens:
|
||||
print("Empty query.", file=sys.stderr)
|
||||
return
|
||||
scored = [(bm25_score(query_tokens, i, idx), i) for i in range(len(filtered))]
|
||||
scored.sort(key=lambda x: -x[0])
|
||||
top = [(s, filtered[i]) for s, i in scored[:args.top] if s > 0]
|
||||
if not top:
|
||||
print("No matches.", file=sys.stderr)
|
||||
return
|
||||
print(f"Top {len(top)} results for: {args.query!r}")
|
||||
print()
|
||||
for score, page in top:
|
||||
title = page["meta"].get("title") or page["slug"]
|
||||
page_type = page["meta"].get("type", "?")
|
||||
print(f" [{score:6.2f}] [{page_type:9}] {title}")
|
||||
print(f" {page['rel_path']}")
|
||||
|
||||
|
||||
def cmd_backlinks(args, pages: list[dict]) -> None:
|
||||
target = args.backlinks
|
||||
inbound = []
|
||||
for page in pages:
|
||||
if target in page["links"]:
|
||||
inbound.append(page)
|
||||
if not inbound:
|
||||
print(f"No pages link to [[{target}]].", file=sys.stderr)
|
||||
return
|
||||
print(f"Pages linking to [[{target}]] ({len(inbound)}):")
|
||||
for page in inbound:
|
||||
title = page["meta"].get("title") or page["slug"]
|
||||
print(f" - {title} ({page['rel_path']})")
|
||||
|
||||
|
||||
def cmd_top_linked(args, pages: list[dict]) -> None:
|
||||
inbound_count = Counter()
|
||||
for page in pages:
|
||||
for link in page["links"]:
|
||||
inbound_count[link] += 1
|
||||
top = inbound_count.most_common(args.top_linked)
|
||||
if not top:
|
||||
print("No links found in the wiki.", file=sys.stderr)
|
||||
return
|
||||
print(f"Top {len(top)} most-linked-to pages (hubs):")
|
||||
for slug, count in top:
|
||||
# Try to find the page for the title
|
||||
match = next((p for p in pages if p["slug"] == slug), None)
|
||||
title = (match["meta"].get("title") if match else None) or slug
|
||||
marker = "" if match else " [BROKEN LINK]"
|
||||
print(f" {count:4d} {title} ({slug}){marker}")
|
||||
|
||||
|
||||
def main():
|
||||
parser = argparse.ArgumentParser(description=__doc__, formatter_class=argparse.RawDescriptionHelpFormatter)
|
||||
parser.add_argument("query", nargs="?", default="", help="Query terms.")
|
||||
parser.add_argument("--wiki", type=Path, default=Path("cml/wiki"), help="Wiki directory (default: cml/wiki).")
|
||||
parser.add_argument("--top", type=int, default=10, help="Top N results (default: 10).")
|
||||
parser.add_argument("--type", help="Filter by frontmatter type.")
|
||||
parser.add_argument("--tag", action="append", default=[], help="Filter by tag (repeatable).")
|
||||
parser.add_argument("--since", help="Only pages updated on or after YYYY-MM-DD.")
|
||||
parser.add_argument("--backlinks", help="Find pages linking to this slug.")
|
||||
parser.add_argument("--top-linked", type=int, help="Show the N most-linked-to pages.")
|
||||
parser.add_argument("--cache", type=Path, help="(reserved) Cache path for BM25 index.")
|
||||
args = parser.parse_args()
|
||||
|
||||
if not args.wiki.exists():
|
||||
print(f"Wiki directory not found: {args.wiki}", file=sys.stderr)
|
||||
sys.exit(1)
|
||||
|
||||
pages = collect_pages(args.wiki)
|
||||
if not pages:
|
||||
print(f"No wiki pages found under {args.wiki}", file=sys.stderr)
|
||||
sys.exit(0)
|
||||
|
||||
if args.backlinks:
|
||||
cmd_backlinks(args, pages)
|
||||
elif args.top_linked:
|
||||
cmd_top_linked(args, pages)
|
||||
elif args.query:
|
||||
cmd_search(args, pages)
|
||||
else:
|
||||
parser.print_help()
|
||||
sys.exit(1)
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
main()
|
||||
@@ -1,159 +0,0 @@
|
||||
#!/usr/bin/env -S uv run --script
|
||||
# /// script
|
||||
# requires-python = ">=3.11"
|
||||
# dependencies = []
|
||||
# ///
|
||||
"""
|
||||
wiki_stats.py — Quick summary of wiki size, shape, and link density.
|
||||
|
||||
Useful for deciding when to shard the index or split pages.
|
||||
|
||||
Usage:
|
||||
python wiki_stats.py [<wiki-dir>]
|
||||
|
||||
Example:
|
||||
python wiki_stats.py wiki/
|
||||
"""
|
||||
|
||||
import argparse
|
||||
import re
|
||||
import sys
|
||||
from collections import Counter
|
||||
from pathlib import Path
|
||||
|
||||
|
||||
WIKILINK_RE = re.compile(r"\[\[([^\]|]+)(?:\|[^\]]+)?\]\]")
|
||||
FRONTMATTER_RE = re.compile(r"^---\s*\n(.*?)\n---\s*\n", re.DOTALL)
|
||||
|
||||
|
||||
SKIP_TOP_LEVEL_FILES = {"SCHEMA.md", "log.md", "README.md"}
|
||||
SKIP_TOP_LEVEL_DIRS = {"indexes", "graph"}
|
||||
|
||||
|
||||
def parse_type(text: str) -> str | None:
|
||||
m = FRONTMATTER_RE.match(text)
|
||||
if not m:
|
||||
return None
|
||||
fm = m.group(1)
|
||||
for line in fm.split("\n"):
|
||||
kv = re.match(r"^type:\s*(.*)$", line)
|
||||
if kv:
|
||||
return kv.group(1).strip().strip('"').strip("'")
|
||||
return None
|
||||
|
||||
|
||||
def main():
|
||||
parser = argparse.ArgumentParser(description=__doc__, formatter_class=argparse.RawDescriptionHelpFormatter)
|
||||
parser.add_argument("wiki", nargs="?", type=Path, default=Path("cml/wiki"))
|
||||
args = parser.parse_args()
|
||||
|
||||
if not args.wiki.exists():
|
||||
print(f"Wiki directory not found: {args.wiki}", file=sys.stderr)
|
||||
sys.exit(1)
|
||||
|
||||
total_pages = 0
|
||||
total_lines = 0
|
||||
total_words = 0
|
||||
total_links = 0
|
||||
pages_by_type = Counter()
|
||||
pages_by_dir = Counter()
|
||||
largest = []
|
||||
most_linked_in = Counter()
|
||||
index_lines = 0
|
||||
|
||||
for md_path in args.wiki.rglob("*.md"):
|
||||
rel = md_path.relative_to(args.wiki)
|
||||
try:
|
||||
text = md_path.read_text(encoding="utf-8")
|
||||
except (UnicodeDecodeError, OSError):
|
||||
continue
|
||||
|
||||
if rel.name == "index.md" and len(rel.parts) == 1:
|
||||
index_lines = text.count("\n") + 1
|
||||
continue
|
||||
if rel.parts[0] in SKIP_TOP_LEVEL_FILES:
|
||||
continue
|
||||
if rel.parts[0] in SKIP_TOP_LEVEL_DIRS:
|
||||
continue
|
||||
if rel.name.startswith("."):
|
||||
continue
|
||||
|
||||
total_pages += 1
|
||||
line_count = text.count("\n") + 1
|
||||
word_count = len(text.split())
|
||||
total_lines += line_count
|
||||
total_words += word_count
|
||||
# Strip frontmatter before counting wikilinks; frontmatter uses bare slugs.
|
||||
body = FRONTMATTER_RE.sub("", text, count=1) if text.startswith("---") else text
|
||||
links = WIKILINK_RE.findall(body)
|
||||
total_links += len(links)
|
||||
for link in links:
|
||||
target = link.split("|")[0].strip()
|
||||
most_linked_in[target] += 1
|
||||
page_type = parse_type(text) or "(none)"
|
||||
pages_by_type[page_type] += 1
|
||||
if len(rel.parts) > 1:
|
||||
pages_by_dir[rel.parts[0]] += 1
|
||||
else:
|
||||
pages_by_dir["(root)"] += 1
|
||||
largest.append((line_count, str(rel)))
|
||||
|
||||
largest.sort(reverse=True)
|
||||
|
||||
print("=" * 60)
|
||||
print(f"Wiki Stats: {args.wiki}")
|
||||
print("=" * 60)
|
||||
print(f"Pages: {total_pages}")
|
||||
print(f"Total lines: {total_lines:,}")
|
||||
print(f"Total words: {total_words:,}")
|
||||
print(f"Total links: {total_links:,}")
|
||||
if total_pages:
|
||||
print(f"Avg page: {total_lines // total_pages} lines / {total_words // total_pages} words")
|
||||
print(f"Link density: {total_links / total_pages:.1f} links per page")
|
||||
print(f"index.md: {index_lines} lines" + (" ← shard recommended (>300)" if index_lines > 300 else ""))
|
||||
print()
|
||||
|
||||
print("Pages by type:")
|
||||
for t, n in pages_by_type.most_common():
|
||||
print(f" {t:15s} {n}")
|
||||
print()
|
||||
|
||||
print("Pages by directory:")
|
||||
for d, n in pages_by_dir.most_common():
|
||||
print(f" {d:15s} {n}")
|
||||
print()
|
||||
|
||||
if largest:
|
||||
print("Largest pages:")
|
||||
for lines, path in largest[:10]:
|
||||
warn = ""
|
||||
if lines > 800:
|
||||
warn = " ← OVER HARD CAP"
|
||||
elif lines > 400:
|
||||
warn = " ← over soft cap"
|
||||
print(f" {lines:5d} {path}{warn}")
|
||||
print()
|
||||
|
||||
if most_linked_in:
|
||||
print("Most-linked-to pages (hubs):")
|
||||
for slug, count in most_linked_in.most_common(10):
|
||||
print(f" {count:4d} [[{slug}]]")
|
||||
print()
|
||||
|
||||
# Scaling recommendations
|
||||
print("Scaling thresholds:")
|
||||
if total_pages < 50:
|
||||
print(" → Below first threshold. Flat structure is fine.")
|
||||
elif total_pages < 150 and index_lines < 300:
|
||||
print(" → Below shard threshold. Continue with single index.md.")
|
||||
elif (total_pages >= 150 or index_lines >= 300) and not (args.wiki / "indexes").exists():
|
||||
print(" → AT SHARD THRESHOLD. Consider sharding index.md into wiki/indexes/<type>.md.")
|
||||
print(" See references/scaling-playbook.md.")
|
||||
elif total_pages >= 300:
|
||||
print(" → Past 300 pages. Use scripts/wiki_search.py as a routine fallback.")
|
||||
if total_pages >= 500:
|
||||
print(" → Past 500 pages. Run lint weekly or per-N-ingests.")
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
main()
|
||||
215
skills/wiki/README.md
Normal file
215
skills/wiki/README.md
Normal file
@@ -0,0 +1,215 @@
|
||||
# wiki — jak to funguje
|
||||
|
||||
Hybrid hledání ve tvých vlastních poznámkách: dva git repozitáře (`index`, `travel`) plus živý
|
||||
workspace nanobota. Skill je **výhradně čtecí** — do tvých repů nikdy nezapisuje a klony drží
|
||||
jen jako pracovní kopii. Index je derived artifact: kdykoli se dá smazat a postavit od nuly
|
||||
(u dnešního objemu jsou to desítky sekund).
|
||||
|
||||
## Tři vrstvy, od nejlevnější
|
||||
|
||||
```text
|
||||
grep ripgrep přes soubory na disku žádný index, nikdy zastaralé, vidí i kód
|
||||
↓
|
||||
toc katalog `files` z databáze adresář → soubor → titulek + tagy
|
||||
↓
|
||||
search FTS5 (BM25) + vec0 (KNN) → RRF parafráze, když neznáš slova
|
||||
```
|
||||
|
||||
Není to fallback řetěz, kde se jde dolů, když horní vrstva zklame — je to volba podle **tvaru
|
||||
otázky**: přesný string, hostname nebo identifikátor patří do `grep`, orientace („co o tom
|
||||
vůbec mám?") do `toc`, parafráze do `search`.
|
||||
|
||||
`grep` je tedy základ, ne poslední záchrana. Soubory na disku beztak leží, takže nestojí nic
|
||||
navíc, a nad kódem je exact match to hlavní — proto se zdrojáky vůbec neindexují (viz níž).
|
||||
|
||||
## Index nikdy nevzniká v tahu agenta
|
||||
|
||||
Indexuje cron každou minutu. Agent v tahu jen **čte** hotový index.
|
||||
|
||||
Proč takhle: `exec` tool má timeout 60 s a plný index je minuty. Indexace v odpovědi by
|
||||
navíc při nedostupné Ollamě dělala z čerstvého dokumentu FTS-only výsledek.
|
||||
|
||||
Drtivá většina tiků neudělá nic a **nezapíše ani řádek**. Když je `log/wiki_sync_cron.log`
|
||||
prázdný, je to správný stav, ne že cron neběží.
|
||||
|
||||
## Co se v tom tiku vlastně děje
|
||||
|
||||
1. Vezme lock `wiki/.sync.lock`. Když ho drží živý běh, skončí bez výpisu. Lock po mrtvém
|
||||
procesu (`pid` neexistuje nebo je starší než 30 min) si vezme zpátky a zaloguje
|
||||
`WARN stale lock, reclaiming`.
|
||||
2. **Levná detekce změn, bez indexace.** U git zdrojů `git ls-remote` — zjistí remote `HEAD`
|
||||
**bez** `fetch`, což je na minutovou kadenci ten správný nástroj; `fetch` teprve když se
|
||||
revize liší. U workspace zdroje walk a porovnání `path` + `size` + `mtime`; sha256 se
|
||||
počítá jen při neshodě.
|
||||
3. Nic se nezměnilo a nic nečeká na vektor → konec.
|
||||
4. Změněné soubory se rozřežou na chunky a pošlou do Ollamy po 32. Pak se **doberou všechny
|
||||
chunky bez vektoru**, i když se žádný soubor nezměnil — tohle je cesta zpátky
|
||||
z degradovaného režimu, když byla Ollama chvíli mimo.
|
||||
5. Zapíše `indexed_rev` a `last_sync_at`, přidá coverage řádek a shrnutí do
|
||||
`log/wiki_sync.log`.
|
||||
|
||||
Selhání sítě je **per zdroj**: timeout nebo chyba gitu zaloguje `WARN`, ten zdroj přeskočí
|
||||
s nezměněnou `indexed_rev` a ostatní dojedou. Bez timeoutů by visící `ls-remote` držel lock
|
||||
a zablokoval i workspace zdroj, který se sítí nemá nic společného.
|
||||
|
||||
## Proč hybrid
|
||||
|
||||
Každá polovina umí něco jiného a ani jedna neumí obojí:
|
||||
|
||||
| | Umí | Neumí |
|
||||
|---|---|---|
|
||||
| **BM25** (FTS5) | exact match, čísla, jména, zkratky | parafrázi, kde se slova nepřekrývají |
|
||||
| **vektory** (`vec0`) | „elektroodpad ze stavebnic" → „Mindstorms po ukončení podpory" | přesné řetězce a čísla |
|
||||
|
||||
Výsledky slučuje **RRF** — sečte převrácené ranky z obou seznamů. Aby to mělo definovaný
|
||||
význam, musí obě poloviny řadit **tutéž množinu**: proto je jedinou retrieval jednotkou chunk,
|
||||
ne soubor. Kdyby BM25 řadil soubory a vektory chunky, merge by nešlo interpretovat.
|
||||
|
||||
Naměřeno na reálných poznámkách: RRF dává MRR 0,367 proti 0,257 (BM25 sám) a 0,261 (vektory
|
||||
samy), takže merge opravdu vyhrává. Plná čísla, včetně srovnání čtyř modelů, jsou
|
||||
v `.claude/tracking/plans/final-wiki-hybrid-rag-result.md` v trackovacím repu.
|
||||
|
||||
Ve výstupu `search` u každého hitu stojí, která polovina ho našla (`bm25 #2, vec #1`). Hit,
|
||||
na kterém se poloviny **shodnou**, má výrazně větší cenu než ten, který vytáhly jen vektory.
|
||||
|
||||
## Česká flexe — na co narazíš
|
||||
|
||||
Dotazová vrstva lepí `*` na slova od tří znaků, aby pokryla české koncovky. **Funguje to jen
|
||||
napůl** a je dobré vědět jak: wildcard je **prefixový**, takže pomůže jen tehdy, když je tvoje
|
||||
slovo prefixem tvaru v dokumentu. Česká flexe ale mění koncovku, ne začátek.
|
||||
|
||||
Naměřeno nad `travel/packaging-list.md`:
|
||||
|
||||
| Dotaz | Chunků v tom souboru |
|
||||
|---|---|
|
||||
| `cestu*` | **0** |
|
||||
| `cesty*` | 3 |
|
||||
| `cest*` | 4 |
|
||||
|
||||
Prakticky: na *„co si vzít na cestu do zahraničí"* se doslovný seznam věcí na cestu nedostal
|
||||
ani do top 10. Když víš, že něco máš, a `search` to nenajde, zkus **kmen slova** (`cest`,
|
||||
`záloh`) nebo rovnou `grep`. Rozhodnutí, jestli to řešit systémově, je otevřená položka
|
||||
v `todo.md` trackovacího repa — každá varianta rozšiřuje kandidátní pool, takže to není
|
||||
zdarma.
|
||||
|
||||
## Index je nápověda, ne odpověď
|
||||
|
||||
Dvě věci, které se snadno přehlédnou:
|
||||
|
||||
- **Výřez v `search` není zdroj pravdy.** Je zkrácený a index může být o minutu starší než
|
||||
disk. Než se z nálezu odpovídá, má se ten soubor přečíst čerstvý — `SKILL.md` to agentovi
|
||||
říká, ale platí to i pro tebe.
|
||||
- **Vektorová polovina vždycky něco vrátí.** KNN nemá dolní mez podobnosti, takže na jakýkoli
|
||||
dotaz nabídne svých k nejbližších chunků. Pro retrieval je to záměr (posoudit relevanci je
|
||||
na čtenáři), ale znamená to, že prázdný výsledek nikdy neznamená „nic nesedí" — znamená
|
||||
prázdný index.
|
||||
|
||||
## Co se indexuje a co ne
|
||||
|
||||
Rozsah řídí `wiki/config.yaml` ve třech krocích: `paths` (whitelist — co v něm není, pro index
|
||||
neexistuje) → `include` (whitelist přípon, `*.md`) → `exclude` (skalpel, vyhrává nad oběma).
|
||||
|
||||
Proč whitelist a ne blacklist: rozhoduje **směr selhání**. U blacklistu nový adresář *tiše
|
||||
vstoupí* do indexu, u whitelistu *tiše chybí*. Index je komprimovaná kopie obsahu včetně
|
||||
osobních věcí, takže „tiše zaindexováno" je horší porucha — a workspace se mění i bez tebe,
|
||||
Dream do něj zapisuje sám.
|
||||
|
||||
Proti té jediné slabině whitelistu stojí záchranná síť: sync na konci zaloguje řádek
|
||||
`coverage <zdroj>: markdown outside paths in …` s top-level adresáři, které markdown mají
|
||||
a žádný zdroj je nepokrývá. Adresáře, které tam vidíš pořád (`backup`, `tmp`, `skills`, …),
|
||||
jsou vědomě vynechané; zajímavý je **nový** přírůstek v tom seznamu.
|
||||
|
||||
**Indexuje se výhradně markdown**, a to bez jakékoli detekce typu obsahu — `main.py` se
|
||||
netrefí do `include: ["*.md"]` a k chunkeru se nedostane. Důvod je měřený: markdown parser
|
||||
dělá z Python komentáře `# TODO: …` nadpis a `unicode61` neumí identifikátory (`send` uvnitř
|
||||
`SendAsync` je miss). Kód pokrývá `grep` — jede přes celý klon včetně `.txt` a `.mhtml`.
|
||||
|
||||
## Kde co leží
|
||||
|
||||
Relativně ke workspace (`~/.nanobot/workspace`):
|
||||
|
||||
| Cesta | Role |
|
||||
|---|---|
|
||||
| `wiki/config.yaml` | zdroje a model — **jediný verzovaný soubor** pod `wiki/` |
|
||||
| `wiki/index.sqlite` | `chunks` + FTS5 + vektory; derived, gitignorovaný |
|
||||
| `wiki/remote/<id>/` | non-bare klony git zdrojů (working tree, aby měl `grep` co číst) |
|
||||
| `wiki/.sync.lock` | JSON `{pid, started_at}`, jen když sync běží |
|
||||
| `log/wiki_sync.log` | jeden řádek na událost — co se naindexovalo, `WARN`, coverage |
|
||||
| `log/wiki_sync_cron.log` | stdout cronu; **prázdný je správně** |
|
||||
| `skills/wiki/scripts/` | `wiki_sync.py` (indexace) a `wiki_search.py` (dotazy) |
|
||||
|
||||
Klon je záměrně non-bare: `--mirror` nemá working tree, takže by `grep` neměl co číst.
|
||||
Cena je dvojnásobek místa na disku, což je na těchhle repech nic.
|
||||
|
||||
## Ruční spuštění
|
||||
|
||||
```bash
|
||||
cd ~/.nanobot/workspace
|
||||
|
||||
# to, co dělá cron
|
||||
~/.local/bin/uv run skills/wiki/scripts/wiki_sync.py
|
||||
|
||||
# jen jeden zdroj
|
||||
~/.local/bin/uv run skills/wiki/scripts/wiki_sync.py --source travel
|
||||
|
||||
# smazat index a postavit od nuly
|
||||
~/.local/bin/uv run skills/wiki/scripts/wiki_sync.py --full
|
||||
|
||||
# dotazy
|
||||
~/.local/bin/uv run skills/wiki/scripts/wiki_search.py search "zálohování" --limit 5
|
||||
~/.local/bin/uv run skills/wiki/scripts/wiki_search.py toc --source travel
|
||||
~/.local/bin/uv run skills/wiki/scripts/wiki_search.py toc --tag caj
|
||||
~/.local/bin/uv run skills/wiki/scripts/wiki_search.py grep "WireGuard" --source index
|
||||
```
|
||||
|
||||
`--full` potřebuješ jen po změně embedding modelu, jeho dimenzí, `query_prefix` nebo chunkeru
|
||||
— tedy přesně tehdy, když dotaz začne hlásit `reindex needed`. Přeembeduje všechno, takže to
|
||||
nepouštěj v tahu, kde na to někdo čeká.
|
||||
|
||||
Plná cesta k `uv` je tu proto, že v neinteraktivním SSH není v `PATH`. Crontab si `PATH`
|
||||
nastavuje sám, takže tam stačí `uv run …`.
|
||||
|
||||
## Když něco nefunguje
|
||||
|
||||
| Hláška | Co to je |
|
||||
|---|---|
|
||||
| `note: embeddings unavailable … FTS-only results` | Ollama na `nvidia.hell` neodpovídá. Sync mezitím indexuje dál a vektory dosadí sám, až se vrátí. |
|
||||
| `note: N chunks still awaiting vectors` | Sync je rozjezdu nebo byl degradovaný. Sémantická polovina je zatím neúplná. |
|
||||
| `reindex needed: …` (exit 1) | Index byl postavený pod jiným embedding kontraktem než má config. Poznámky jsou v pořádku, řeší to `--full`. |
|
||||
| `WARN <zdroj>: git … failed` | Ten zdroj se přeskočil, ostatní dojely. `indexed_rev` zůstala, takže příští tik to zkusí znovu. |
|
||||
| `(no matches)` | Prázdný index — ne „nic nesedí" (viz výš). |
|
||||
|
||||
Všechno ostatní hledej v `log/wiki_sync.log`. Když je prázdný i po změně poznámek, problém je
|
||||
v cronu, ne v hledání.
|
||||
|
||||
## Jak ověřit, že to funguje
|
||||
|
||||
**Smoke loop v konzoli** — projde celou smyčku od zápisu po nalezení:
|
||||
|
||||
```bash
|
||||
cd ~/.nanobot/workspace
|
||||
printf '# Smoke\n\nZrzavý jednorožec kontroluje wiki index.\n' > notes/_smoke.md
|
||||
sleep 90
|
||||
~/.local/bin/uv run skills/wiki/scripts/wiki_search.py search "zrzavý jednorožec" --limit 3
|
||||
rm notes/_smoke.md # cron ho z indexu odklidí sám do minuty
|
||||
```
|
||||
|
||||
**Dotaz do WebUI na každý zdroj.** Princip: vyber fakt, který model **nemůže** znát
|
||||
z obecných znalostí — pak je konkrétní správná odpověď sama důkazem, že přišla z poznámek.
|
||||
Když odpoví obecně a čísla vynechá nebo si je vymyslí, neprošel.
|
||||
|
||||
Příklady platné k 2026-09-09 (poznámky se mění, princip ne):
|
||||
|
||||
| Zdroj | Dotaz | Musí padnout |
|
||||
|---|---|---|
|
||||
| `workspace` | Kolik vody a čaje mám v receptu na Teh Tarik? | 700–800 ml, dvě polévkové lžíce čaje |
|
||||
| `travel` | Co si mám podle poznámek vzít na Sněžku? | rozdvojka do zásuvky, power banka, Sony ANC sluchátka |
|
||||
| `index` | Co je projekt Sauter Bus? | RS485 proxy s odposlechem a modifikací provozu, micro:bit |
|
||||
|
||||
**Negativní kontrola** je nejcennější z celé sady: zeptej se na téma, které v poznámkách
|
||||
prokazatelně není (např. pěstování bonsají). Správná odpověď je „nic o tom nemáš". Když
|
||||
začne vyprávět, doplňuje si výsledky z obecných znalostí — a to je zrádnější vada než
|
||||
nefunkční index, protože se tváří jako odpověď z poznámek.
|
||||
|
||||
Když si nejsi jistý, odkud odpověď je, dopiš **„ze kterého souboru to je?"**. Má přijít
|
||||
konkrétní cesta, ne „z tvých poznámek".
|
||||
106
skills/wiki/SKILL.md
Normal file
106
skills/wiki/SKILL.md
Normal file
@@ -0,0 +1,106 @@
|
||||
---
|
||||
name: wiki
|
||||
description: >
|
||||
Search the user's own notes — personal git note repos plus the live workspace — by
|
||||
keyword, topic or meaning. Use to answer questions about what the user has written
|
||||
down, recorded or decided, and to find which file covers a topic.
|
||||
Triggers on: "/wiki", "find in my notes", "what did I write about", "do I have notes on".
|
||||
Read-only: it never writes, edits or captures notes, and it does not answer from
|
||||
general knowledge — only from what is indexed on disk.
|
||||
---
|
||||
|
||||
# /wiki
|
||||
|
||||
Hybrid search over the user's notes. `chunks` is the single retrieval unit: FTS5 supplies a
|
||||
BM25 rank, `sqlite-vec` a vector rank, and RRF merges the two.
|
||||
|
||||
Reply to the user in their own language.
|
||||
|
||||
## Pick a layer
|
||||
|
||||
Three layers, cheapest first. Start with the one that matches the question.
|
||||
|
||||
| Question shape | Command |
|
||||
|---|---|
|
||||
| exact string, filename, hostname, IP, identifier, code | `grep <pattern>` |
|
||||
| "what do I even have about X", orientation, browsing | `toc [--source X] [--tag Y]` |
|
||||
| a topic or a paraphrase, wording unknown | `search "<query>"` |
|
||||
|
||||
```sh
|
||||
uv run skills/wiki/scripts/wiki_search.py grep "rotate_snapshots"
|
||||
uv run skills/wiki/scripts/wiki_search.py toc --source travel
|
||||
uv run skills/wiki/scripts/wiki_search.py search "jak snížit elektroodpad" --limit 5
|
||||
```
|
||||
|
||||
For the full flag reference of any subcommand:
|
||||
|
||||
```sh
|
||||
uv run skills/wiki/scripts/wiki_search.py --help
|
||||
uv run skills/wiki/scripts/wiki_search.py <subcommand> --help
|
||||
```
|
||||
|
||||
## Behavioral contract
|
||||
|
||||
**The index is a retrieval hint, never the answer.** `search` returns an excerpt to tell you
|
||||
*which file* is relevant. Before answering, read that file fresh from disk — the index can be
|
||||
up to a minute behind, and the excerpt is truncated. Never quote figures, commands or
|
||||
decisions straight from a `search` excerpt.
|
||||
|
||||
**Only markdown is indexed; `grep` sees everything.** Source code, configs and data files never
|
||||
reach `search` or `toc`. For anything code-shaped — an identifier, a flag, a function name —
|
||||
`grep` is the right layer, not a fallback. Exact match is what matters there anyway.
|
||||
|
||||
**`grep` is a regex.** Escape regex metacharacters when the user means them literally
|
||||
(`.`, `(`, `[`, `*`, `+`, `?`, `|`). It searches the files on disk, so it is always current.
|
||||
|
||||
**`search` results always look plausible.** The vector half has no distance floor: it returns
|
||||
its nearest chunks whatever you ask, so an empty result set means an empty index, not "nothing
|
||||
matches". Judge each hit against the question and say so when nothing genuinely fits — do not
|
||||
present a weak nearest neighbour as an answer.
|
||||
|
||||
**Read the notes the output prints.** Each hit shows `source:path`, the breadcrumb, and which
|
||||
half found it (`bm25 #2, vec #1`). A hit both halves rank highly is worth more than one only
|
||||
the vectors found.
|
||||
|
||||
## Notes the output may print
|
||||
|
||||
- `embeddings unavailable … FTS-only results` — the semantic half is down, so only keyword
|
||||
matching ran. Paraphrase queries will do badly; say so rather than concluding the notes are
|
||||
silent on the topic. Try `grep` with the user's own wording.
|
||||
- `N chunks still awaiting vectors` — a sync is mid-flight or was degraded. Recent notes may
|
||||
be missing from the semantic half.
|
||||
- `reindex needed: …` on stderr with exit 1 — the index was built under a different embedding
|
||||
contract. Nothing is wrong with the notes; the index has to be rebuilt (see below). Do not
|
||||
retry the query.
|
||||
|
||||
## Sources
|
||||
|
||||
Defined in `wiki/config.yaml`. Each has a stable **source id** used in output and in
|
||||
`--source`. Git sources are read-only clones under `wiki/remote/<id>/`; the `workspace` source
|
||||
indexes the live workspace. The user's repos are the canonical data — nothing here ever writes
|
||||
to them.
|
||||
|
||||
`toc --tag` filters on frontmatter tags, which only exist where the user wrote them.
|
||||
|
||||
## Indexing
|
||||
|
||||
`wiki_sync.py` runs every minute from the crontab and does all the work offline. Never index
|
||||
inside a turn: a full rebuild takes minutes and the `exec` tool times out at 60 s.
|
||||
|
||||
```sh
|
||||
uv run skills/wiki/scripts/wiki_sync.py # what cron runs; a quiet tick is free
|
||||
uv run skills/wiki/scripts/wiki_sync.py --full # wipe and rebuild
|
||||
```
|
||||
|
||||
`--full` is needed only after the embedding model, its dimensions, the query prefix or the
|
||||
chunker change — that is what a `reindex needed` message means. It re-embeds everything, so
|
||||
run it in the background and tell the user it is running, never inside a turn where they wait.
|
||||
|
||||
`log/wiki_sync.log` holds one line per event: what was indexed, `WARN` for a source that was
|
||||
skipped, and a `coverage` line naming top-level directories that hold markdown no source
|
||||
covers. Read it when `search` cannot find something the user is sure they wrote.
|
||||
|
||||
## Environment
|
||||
|
||||
- `WIKI_DB` — override the index path (tests).
|
||||
- `WIKI_CONFIG` — override the config path (tests).
|
||||
260
skills/wiki/scripts/wiki_chunker.py
Normal file
260
skills/wiki/scripts/wiki_chunker.py
Normal file
@@ -0,0 +1,260 @@
|
||||
#!/usr/bin/env python3
|
||||
# /// script
|
||||
# requires-python = ">=3.11"
|
||||
# dependencies = ["markdown-it-py", "pyyaml"]
|
||||
# ///
|
||||
"""Structure-aware markdown chunker for the wiki skill.
|
||||
|
||||
Splits a document along its H1-H3 heading hierarchy, then merges small siblings and
|
||||
splits oversized sections along block boundaries. Every chunk carries a breadcrumb
|
||||
(`path > title > section > subsection`) which goes into the embedded text as well as
|
||||
the metadata, so a vector represents a passage in context rather than in isolation.
|
||||
|
||||
markdown-it-py supplies the AST (it knows the CommonMark edge cases); the chunking
|
||||
policy below is ours. Block boundaries come from token line maps, so the emitted text
|
||||
is the original markdown — code fences and tables stay byte-for-byte intact.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import re
|
||||
from dataclasses import dataclass, field
|
||||
|
||||
import yaml
|
||||
from markdown_it import MarkdownIt # ty: ignore[unresolved-import]
|
||||
|
||||
CHUNKER_VERSION = "1"
|
||||
|
||||
MERGE_BELOW = 200
|
||||
SPLIT_ABOVE = 800
|
||||
OVERLAP = 64
|
||||
CHARS_PER_TOKEN = 4
|
||||
|
||||
SECTION_LEVELS = (1, 2, 3)
|
||||
BREADCRUMB_SEP = " > "
|
||||
|
||||
_FRONTMATTER_RE = re.compile(r"\A---[ \t]*\r?\n(.*?)\r?\n---[ \t]*(?:\r?\n|\Z)", re.DOTALL)
|
||||
|
||||
_md = MarkdownIt("commonmark")
|
||||
|
||||
|
||||
@dataclass(frozen=True)
|
||||
class Chunk:
|
||||
breadcrumb: str
|
||||
text: str
|
||||
|
||||
|
||||
@dataclass(frozen=True)
|
||||
class ParsedFile:
|
||||
title: str | None = None
|
||||
tags: list[str] = field(default_factory=list)
|
||||
headings: list[str] = field(default_factory=list)
|
||||
chunks: list[Chunk] = field(default_factory=list)
|
||||
|
||||
|
||||
@dataclass(frozen=True)
|
||||
class _Heading:
|
||||
start: int
|
||||
end: int
|
||||
level: int
|
||||
title: str
|
||||
|
||||
|
||||
@dataclass(frozen=True)
|
||||
class _Section:
|
||||
breadcrumb: tuple[str, ...]
|
||||
text: str
|
||||
|
||||
@property
|
||||
def parent(self) -> tuple[str, ...]:
|
||||
return self.breadcrumb[:-1]
|
||||
|
||||
|
||||
def token_estimate(text: str) -> int:
|
||||
"""Approximate token count. Sizing does not need a real tokenizer (plan: 4 chars/token)."""
|
||||
return max(1, len(text) // CHARS_PER_TOKEN)
|
||||
|
||||
|
||||
def parse_markdown(text: str, path: str) -> ParsedFile:
|
||||
"""Chunk one markdown document. `path` is the source-relative path, the breadcrumb root."""
|
||||
frontmatter, body = _split_frontmatter(text)
|
||||
frontmatter_title = _frontmatter_title(frontmatter)
|
||||
tags = _frontmatter_tags(frontmatter)
|
||||
|
||||
root: tuple[str, ...] = (path,) if frontmatter_title is None else (path, frontmatter_title)
|
||||
sections, headings = _split_sections(body, root)
|
||||
merged = _merge_small_siblings(sections)
|
||||
chunks = [chunk for section in merged for chunk in _split_oversized(section)]
|
||||
|
||||
# The catalog title falls back to the first heading, because notes carry their title as
|
||||
# `# H1` far more often than as frontmatter. It deliberately does NOT feed the breadcrumb
|
||||
# root: the H1 already reaches the breadcrumb through the heading stack, and changing the
|
||||
# root would rewrite every chunk's text and force a full reindex.
|
||||
title = frontmatter_title or (headings[0] if headings else None)
|
||||
return ParsedFile(title=title, tags=tags, headings=headings, chunks=chunks)
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# frontmatter
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
|
||||
def _split_frontmatter(text: str) -> tuple[dict, str]:
|
||||
"""Peel off a leading YAML frontmatter block. Malformed frontmatter stays body text."""
|
||||
match = _FRONTMATTER_RE.match(text)
|
||||
if not match:
|
||||
return {}, text
|
||||
try:
|
||||
data = yaml.safe_load(match.group(1))
|
||||
except yaml.YAMLError:
|
||||
return {}, text
|
||||
if not isinstance(data, dict):
|
||||
return {}, text
|
||||
return data, text[match.end() :]
|
||||
|
||||
|
||||
def _frontmatter_title(frontmatter: dict) -> str | None:
|
||||
title = frontmatter.get("title")
|
||||
if isinstance(title, str) and title.strip():
|
||||
return title.strip()
|
||||
return None
|
||||
|
||||
|
||||
def _frontmatter_tags(frontmatter: dict) -> list[str]:
|
||||
raw = frontmatter.get("tags")
|
||||
if isinstance(raw, str):
|
||||
values = raw.split(",")
|
||||
elif isinstance(raw, list):
|
||||
values = [str(item) for item in raw]
|
||||
else:
|
||||
return []
|
||||
return [tag.strip() for tag in values if tag.strip()]
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# sectioning
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
|
||||
def _split_sections(body: str, root: tuple[str, ...]) -> tuple[list[_Section], list[str]]:
|
||||
"""Cut the body at H1-H3 headings. H4+ stay inside their parent section."""
|
||||
lines = body.split("\n")
|
||||
tokens = _md.parse(body)
|
||||
|
||||
starts: list[_Heading] = []
|
||||
for index, token in enumerate(tokens):
|
||||
if token.type != "heading_open" or token.map is None:
|
||||
continue
|
||||
level = int(token.tag[1:])
|
||||
if level not in SECTION_LEVELS:
|
||||
continue
|
||||
inline = tokens[index + 1] if index + 1 < len(tokens) else None
|
||||
title = inline.content.strip() if inline is not None else ""
|
||||
starts.append(_Heading(start=token.map[0], end=token.map[1], level=level, title=title))
|
||||
|
||||
headings = [heading.title for heading in starts]
|
||||
boundaries = [heading.start for heading in starts] + [len(lines)]
|
||||
|
||||
sections: list[_Section] = []
|
||||
preamble = "\n".join(lines[: boundaries[0]]).strip()
|
||||
if preamble:
|
||||
sections.append(_Section(root, preamble))
|
||||
|
||||
stack: list[tuple[int, str]] = []
|
||||
for position, heading in enumerate(starts):
|
||||
while stack and stack[-1][0] >= heading.level:
|
||||
stack.pop()
|
||||
stack.append((heading.level, heading.title))
|
||||
section_end = boundaries[position + 1]
|
||||
# A heading with no prose of its own would embed as a bare title; the heading
|
||||
# still reaches the index through its children's breadcrumbs, so drop it.
|
||||
if not "\n".join(lines[heading.end : section_end]).strip():
|
||||
continue
|
||||
text = "\n".join(lines[heading.start : section_end]).strip()
|
||||
sections.append(_Section(_dedupe(root + tuple(title for _, title in stack)), text))
|
||||
return sections, headings
|
||||
|
||||
|
||||
def _dedupe(parts: tuple[str, ...]) -> tuple[str, ...]:
|
||||
"""Drop consecutive repeats so a frontmatter title matching the H1 shows up once."""
|
||||
out: list[str] = []
|
||||
for part in parts:
|
||||
if not out or out[-1] != part:
|
||||
out.append(part)
|
||||
return tuple(out)
|
||||
|
||||
|
||||
def _merge_small_siblings(sections: list[_Section]) -> list[_Section]:
|
||||
"""Glue a too-small section onto the following sibling under the same parent."""
|
||||
merged: list[_Section] = []
|
||||
for section in sections:
|
||||
if not merged:
|
||||
merged.append(section)
|
||||
continue
|
||||
previous = merged[-1]
|
||||
combined = f"{previous.text}\n\n{section.text}"
|
||||
if (
|
||||
token_estimate(previous.text) < MERGE_BELOW
|
||||
and previous.parent == section.parent
|
||||
and token_estimate(combined) <= SPLIT_ABOVE
|
||||
):
|
||||
merged[-1] = _Section(previous.breadcrumb, combined)
|
||||
else:
|
||||
merged.append(section)
|
||||
return merged
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# splitting
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
|
||||
def _split_oversized(section: _Section) -> list[Chunk]:
|
||||
"""Break a section over SPLIT_ABOVE into block-aligned pieces with OVERLAP carry-over."""
|
||||
breadcrumb = BREADCRUMB_SEP.join(section.breadcrumb)
|
||||
if token_estimate(section.text) <= SPLIT_ABOVE:
|
||||
return [Chunk(breadcrumb, _embed_text(breadcrumb, section.text))]
|
||||
|
||||
blocks = _top_level_blocks(section.text)
|
||||
chunks: list[Chunk] = []
|
||||
current: list[str] = []
|
||||
for block in blocks:
|
||||
candidate = [*current, block]
|
||||
if current and token_estimate("\n\n".join(candidate)) > SPLIT_ABOVE:
|
||||
chunks.append(Chunk(breadcrumb, _embed_text(breadcrumb, "\n\n".join(current))))
|
||||
current = [*_overlap_tail(current), block]
|
||||
else:
|
||||
current = candidate
|
||||
if current:
|
||||
chunks.append(Chunk(breadcrumb, _embed_text(breadcrumb, "\n\n".join(current))))
|
||||
return chunks
|
||||
|
||||
|
||||
def _top_level_blocks(text: str) -> list[str]:
|
||||
"""Top-level markdown blocks, taken from token line maps so fences stay whole."""
|
||||
lines = text.split("\n")
|
||||
tokens = _md.parse(text)
|
||||
starts = sorted({token.map[0] for token in tokens if token.level == 0 and token.map})
|
||||
if not starts:
|
||||
return [text]
|
||||
bounds = [*starts, len(lines)]
|
||||
blocks = ["\n".join(lines[bounds[i] : bounds[i + 1]]).strip() for i in range(len(starts))]
|
||||
return [block for block in blocks if block]
|
||||
|
||||
|
||||
def _overlap_tail(blocks: list[str]) -> list[str]:
|
||||
"""Trailing whole blocks of the emitted chunk, up to OVERLAP tokens."""
|
||||
tail: list[str] = []
|
||||
budget = OVERLAP
|
||||
for block in reversed(blocks):
|
||||
cost = token_estimate(block)
|
||||
if cost > budget:
|
||||
break
|
||||
tail.insert(0, block)
|
||||
budget -= cost
|
||||
return tail
|
||||
|
||||
|
||||
def _embed_text(breadcrumb: str, text: str) -> str:
|
||||
"""The stored chunk text carries its breadcrumb, so the vector sees the context."""
|
||||
return f"{breadcrumb}\n\n{text}"
|
||||
225
skills/wiki/scripts/wiki_config.py
Normal file
225
skills/wiki/scripts/wiki_config.py
Normal file
@@ -0,0 +1,225 @@
|
||||
#!/usr/bin/env python3
|
||||
# /// script
|
||||
# requires-python = ">=3.11"
|
||||
# dependencies = ["pyyaml"]
|
||||
# ///
|
||||
"""Layout and configuration for the wiki skill.
|
||||
|
||||
All runtime data lives under `workspace/wiki/`; only `config.yaml` is versioned.
|
||||
|
||||
The source id is the stable key — the catalog and the vectors hang off it, while a
|
||||
URL or a path may change. Renaming an id is therefore an explicit invalidation of
|
||||
that source's index, not a rename.
|
||||
|
||||
Scope precedence is `paths` (whitelist — what is not in it does not exist for the
|
||||
index) then `include` (extension whitelist) then `exclude` (scalpel, wins over both).
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import os
|
||||
import re
|
||||
from dataclasses import dataclass
|
||||
from pathlib import Path
|
||||
|
||||
import yaml
|
||||
|
||||
# workspace/skills/wiki/scripts/wiki_config.py -> parents[3] = workspace root.
|
||||
WORKSPACE = Path(__file__).resolve().parents[3]
|
||||
WIKI_DIR = WORKSPACE / "wiki"
|
||||
DEFAULT_DB_PATH = WIKI_DIR / "index.sqlite"
|
||||
DEFAULT_CONFIG_PATH = WIKI_DIR / "config.yaml"
|
||||
REMOTE_DIR = WIKI_DIR / "remote"
|
||||
LOCK_PATH = WIKI_DIR / ".sync.lock"
|
||||
SYNC_LOG_PATH = WORKSPACE / "log" / "wiki_sync.log"
|
||||
|
||||
GIT_KIND = "git"
|
||||
WORKSPACE_KIND = "workspace"
|
||||
VALID_KINDS = (GIT_KIND, WORKSPACE_KIND)
|
||||
|
||||
|
||||
def db_path() -> Path:
|
||||
"""Index location, overridable with WIKI_DB for tests."""
|
||||
return Path(os.environ.get("WIKI_DB", str(DEFAULT_DB_PATH)))
|
||||
|
||||
|
||||
def config_path() -> Path:
|
||||
"""Config location, overridable with WIKI_CONFIG for tests."""
|
||||
return Path(os.environ.get("WIKI_CONFIG", str(DEFAULT_CONFIG_PATH)))
|
||||
|
||||
|
||||
class ConfigError(ValueError):
|
||||
"""Malformed or incomplete wiki/config.yaml."""
|
||||
|
||||
|
||||
@dataclass(frozen=True)
|
||||
class EmbeddingConfig:
|
||||
endpoint: str
|
||||
model: str
|
||||
dims: int
|
||||
batch: int
|
||||
keep_alive: int
|
||||
query_prefix: str
|
||||
|
||||
|
||||
@dataclass(frozen=True)
|
||||
class SourceConfig:
|
||||
source_id: str
|
||||
kind: str
|
||||
url: str | None
|
||||
paths: tuple[str, ...]
|
||||
include: tuple[str, ...]
|
||||
exclude: tuple[str, ...]
|
||||
|
||||
def covers(self, rel_path: str) -> bool:
|
||||
"""True when a source-relative path belongs in the index."""
|
||||
if not any(_matches(pattern, rel_path) for pattern in self.paths):
|
||||
return False
|
||||
name = rel_path.rsplit("/", 1)[-1]
|
||||
if not any(_matches(pattern, name) for pattern in self.include):
|
||||
return False
|
||||
return not any(_matches(pattern, rel_path) for pattern in self.exclude)
|
||||
|
||||
def covers_dir(self, rel_dir: str) -> bool:
|
||||
"""Cheap walk prune: could anything under this directory ever be covered?"""
|
||||
if not rel_dir:
|
||||
return True
|
||||
probe = f"{rel_dir}/"
|
||||
if any(_matches(pattern, probe) for pattern in self.exclude):
|
||||
return False
|
||||
return any(_prefix_could_match(pattern, probe) for pattern in self.paths)
|
||||
|
||||
|
||||
@dataclass(frozen=True)
|
||||
class WikiConfig:
|
||||
embedding: EmbeddingConfig
|
||||
sources: tuple[SourceConfig, ...]
|
||||
|
||||
def source(self, source_id: str) -> SourceConfig | None:
|
||||
return next((s for s in self.sources if s.source_id == source_id), None)
|
||||
|
||||
|
||||
def source_root(source: SourceConfig) -> Path:
|
||||
"""Where a source's files live. Git clones are derived from the id, never configured."""
|
||||
if source.kind == GIT_KIND:
|
||||
return REMOTE_DIR / source.source_id
|
||||
return WORKSPACE
|
||||
|
||||
|
||||
def load_config(path: Path | None = None) -> WikiConfig:
|
||||
path = path or config_path()
|
||||
if not path.exists():
|
||||
raise ConfigError(f"missing config: {path}")
|
||||
try:
|
||||
raw = yaml.safe_load(path.read_text(encoding="utf-8"))
|
||||
except yaml.YAMLError as exc:
|
||||
raise ConfigError(f"unparseable config {path}: {exc}") from exc
|
||||
if not isinstance(raw, dict):
|
||||
raise ConfigError(f"config {path} must be a mapping")
|
||||
return WikiConfig(
|
||||
embedding=_parse_embedding(raw.get("embedding")),
|
||||
sources=_parse_sources(raw.get("sources")),
|
||||
)
|
||||
|
||||
|
||||
def _as_mapping(raw: object, what: str) -> dict[str, object]:
|
||||
if not isinstance(raw, dict):
|
||||
raise ConfigError(f"{what} must be a mapping")
|
||||
return {str(key): value for key, value in raw.items()}
|
||||
|
||||
|
||||
def _parse_embedding(raw: object) -> EmbeddingConfig:
|
||||
values = _as_mapping(raw, "`embedding`")
|
||||
missing = [key for key in ("endpoint", "model", "dims") if key not in values]
|
||||
if missing:
|
||||
raise ConfigError(f"embedding is missing {', '.join(missing)}")
|
||||
keep_alive = values.get("keep_alive", -1)
|
||||
if not isinstance(keep_alive, int):
|
||||
# Ollama rejects a string keep_alive of "-1" with HTTP 400.
|
||||
raise ConfigError("embedding.keep_alive must be a number, not a string")
|
||||
return EmbeddingConfig(
|
||||
endpoint=str(values["endpoint"]).rstrip("/"),
|
||||
model=str(values["model"]),
|
||||
dims=int(str(values["dims"])),
|
||||
batch=int(str(values.get("batch", 32))),
|
||||
keep_alive=keep_alive,
|
||||
query_prefix=str(values.get("query_prefix", "")),
|
||||
)
|
||||
|
||||
|
||||
def _parse_sources(raw: object) -> tuple[SourceConfig, ...]:
|
||||
entries = _as_mapping(raw, "`sources`")
|
||||
if not entries:
|
||||
raise ConfigError("config needs a non-empty `sources` mapping")
|
||||
sources = []
|
||||
for source_id, raw_body in entries.items():
|
||||
body = _as_mapping(raw_body, f"source {source_id}")
|
||||
kind = str(body.get("kind"))
|
||||
if kind not in VALID_KINDS:
|
||||
raise ConfigError(f"source {source_id}: kind must be one of {VALID_KINDS}, got {kind!r}")
|
||||
url = body.get("url")
|
||||
if kind == GIT_KIND and not url:
|
||||
raise ConfigError(f"source {source_id}: git sources need a url")
|
||||
sources.append(
|
||||
SourceConfig(
|
||||
source_id=source_id,
|
||||
kind=kind,
|
||||
url=str(url) if url else None,
|
||||
paths=_as_patterns(body.get("paths"), source_id, "paths"),
|
||||
include=_as_patterns(body.get("include") or ["*.md"], source_id, "include"),
|
||||
exclude=_as_patterns(body.get("exclude") or [], source_id, "exclude"),
|
||||
)
|
||||
)
|
||||
return tuple(sources)
|
||||
|
||||
|
||||
def _as_patterns(raw: object, source_id: str, key: str) -> tuple[str, ...]:
|
||||
if raw is None:
|
||||
raise ConfigError(f"source {source_id}: `{key}` is required")
|
||||
if not isinstance(raw, list):
|
||||
raise ConfigError(f"source {source_id}: `{key}` must be a list")
|
||||
return tuple(str(item) for item in raw)
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# glob matching
|
||||
#
|
||||
# fnmatch lets `*` cross a `/` and PurePath.match has no recursive `**` before
|
||||
# Python 3.13, so the patterns are translated by hand.
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
_SEGMENT_ANY = "[^/]*"
|
||||
|
||||
|
||||
def _glob_to_regex(pattern: str) -> str:
|
||||
parts = pattern.split("/")
|
||||
out = []
|
||||
for index, part in enumerate(parts):
|
||||
is_last = index == len(parts) - 1
|
||||
if part == "**":
|
||||
out.append(".*" if is_last else "(?:[^/]+/)*")
|
||||
else:
|
||||
segment = re.escape(part).replace(r"\*", _SEGMENT_ANY).replace(r"\?", "[^/]")
|
||||
out.append(segment if is_last else segment + "/")
|
||||
return "^" + "".join(out) + "$"
|
||||
|
||||
|
||||
def _matches(pattern: str, value: str) -> bool:
|
||||
return re.match(_glob_to_regex(pattern), value) is not None
|
||||
|
||||
|
||||
def _prefix_could_match(pattern: str, directory: str) -> bool:
|
||||
"""True when `pattern` can still match something below `directory`."""
|
||||
if pattern.startswith("**"):
|
||||
return True
|
||||
pattern_parts = pattern.split("/")
|
||||
dir_parts = [part for part in directory.split("/") if part]
|
||||
for depth, dir_part in enumerate(dir_parts):
|
||||
if depth >= len(pattern_parts):
|
||||
return False
|
||||
pattern_part = pattern_parts[depth]
|
||||
if pattern_part == "**":
|
||||
return True
|
||||
if not _matches(pattern_part, dir_part):
|
||||
return False
|
||||
return True
|
||||
129
skills/wiki/scripts/wiki_db.py
Normal file
129
skills/wiki/scripts/wiki_db.py
Normal file
@@ -0,0 +1,129 @@
|
||||
#!/usr/bin/env python3
|
||||
# /// script
|
||||
# requires-python = ">=3.11"
|
||||
# dependencies = ["sqlite-vec==0.1.6"]
|
||||
# ///
|
||||
"""SQLite storage layer for the wiki skill.
|
||||
|
||||
Schema, connection factory and the sqlite-vec extension load.
|
||||
|
||||
`chunks` is the canonical retrieval unit: `chunks_fts` gives it a BM25 rank and
|
||||
`vec_chunks` a KNN rank, both keyed on `chunks.id`, so RRF merges two rankings of
|
||||
the *same* set.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import sqlite3
|
||||
from pathlib import Path
|
||||
|
||||
import sqlite_vec # ty: ignore[unresolved-import]
|
||||
|
||||
# Compiled into the vec0 table definition. `wiki_embed` guards config.dims against it —
|
||||
# a mismatch means the index was built for a different model.
|
||||
EMBEDDING_DIMS = 1024
|
||||
|
||||
SCHEMA = f"""
|
||||
PRAGMA journal_mode = WAL;
|
||||
PRAGMA foreign_keys = ON;
|
||||
|
||||
CREATE TABLE IF NOT EXISTS meta (
|
||||
key TEXT PRIMARY KEY,
|
||||
value TEXT NOT NULL
|
||||
);
|
||||
|
||||
CREATE TABLE IF NOT EXISTS sources (
|
||||
source_id TEXT PRIMARY KEY,
|
||||
kind TEXT NOT NULL CHECK(kind IN ('git', 'workspace')),
|
||||
indexed_rev TEXT,
|
||||
last_sync_at TEXT
|
||||
);
|
||||
|
||||
CREATE TABLE IF NOT EXISTS files (
|
||||
source_id TEXT NOT NULL REFERENCES sources(source_id),
|
||||
path TEXT NOT NULL,
|
||||
title TEXT,
|
||||
tags TEXT,
|
||||
headings TEXT,
|
||||
sha256 TEXT NOT NULL,
|
||||
size INTEGER NOT NULL,
|
||||
mtime REAL,
|
||||
indexed_at TEXT NOT NULL,
|
||||
PRIMARY KEY (source_id, path)
|
||||
);
|
||||
|
||||
CREATE TABLE IF NOT EXISTS chunks (
|
||||
id INTEGER PRIMARY KEY AUTOINCREMENT,
|
||||
source_id TEXT NOT NULL,
|
||||
path TEXT NOT NULL,
|
||||
chunk_idx INTEGER NOT NULL,
|
||||
breadcrumb TEXT NOT NULL,
|
||||
text TEXT NOT NULL,
|
||||
embedded_at TEXT,
|
||||
UNIQUE (source_id, path, chunk_idx),
|
||||
FOREIGN KEY (source_id, path) REFERENCES files(source_id, path) ON DELETE CASCADE
|
||||
);
|
||||
|
||||
CREATE INDEX IF NOT EXISTS idx_chunks_pending ON chunks(id) WHERE embedded_at IS NULL;
|
||||
|
||||
CREATE VIRTUAL TABLE IF NOT EXISTS chunks_fts USING fts5(
|
||||
breadcrumb, text,
|
||||
content='chunks', content_rowid='id',
|
||||
tokenize='unicode61 remove_diacritics 2'
|
||||
);
|
||||
|
||||
CREATE TRIGGER IF NOT EXISTS chunks_ai AFTER INSERT ON chunks BEGIN
|
||||
INSERT INTO chunks_fts(rowid, breadcrumb, text) VALUES (new.id, new.breadcrumb, new.text);
|
||||
END;
|
||||
|
||||
CREATE TRIGGER IF NOT EXISTS chunks_ad AFTER DELETE ON chunks BEGIN
|
||||
INSERT INTO chunks_fts(chunks_fts, rowid, breadcrumb, text)
|
||||
VALUES('delete', old.id, old.breadcrumb, old.text);
|
||||
END;
|
||||
|
||||
-- The WHEN guard keeps `UPDATE chunks SET embedded_at` (sync step 4b) from rewriting
|
||||
-- an FTS row whose indexed text did not change.
|
||||
CREATE TRIGGER IF NOT EXISTS chunks_au AFTER UPDATE ON chunks
|
||||
WHEN old.text IS NOT new.text OR old.breadcrumb IS NOT new.breadcrumb
|
||||
BEGIN
|
||||
INSERT INTO chunks_fts(chunks_fts, rowid, breadcrumb, text)
|
||||
VALUES('delete', old.id, old.breadcrumb, old.text);
|
||||
INSERT INTO chunks_fts(rowid, breadcrumb, text) VALUES (new.id, new.breadcrumb, new.text);
|
||||
END;
|
||||
|
||||
CREATE VIRTUAL TABLE IF NOT EXISTS vec_chunks USING vec0(
|
||||
embedding float[{EMBEDDING_DIMS}] distance_metric=cosine
|
||||
);
|
||||
"""
|
||||
|
||||
|
||||
def get_db(path: Path) -> sqlite3.Connection:
|
||||
"""Return an autocommit connection with sqlite-vec loaded and foreign keys on."""
|
||||
conn = sqlite3.connect(path, isolation_level=None)
|
||||
conn.enable_load_extension(True)
|
||||
sqlite_vec.load(conn)
|
||||
conn.enable_load_extension(False)
|
||||
conn.execute("PRAGMA journal_mode = WAL")
|
||||
conn.execute("PRAGMA foreign_keys = ON")
|
||||
conn.row_factory = sqlite3.Row
|
||||
_migrate(conn)
|
||||
return conn
|
||||
|
||||
|
||||
def _migrate(conn: sqlite3.Connection) -> None:
|
||||
"""Idempotently bring an existing DB up to the current schema.
|
||||
|
||||
init_db only runs the full SCHEMA on a missing file, so live DBs never see
|
||||
later additions. Each step must be a no-op once applied.
|
||||
"""
|
||||
# No migrations yet — schema_version 1 is the initial shape.
|
||||
|
||||
|
||||
def init_db(path: Path) -> None:
|
||||
"""Create tables, indexes and triggers if they don't exist."""
|
||||
path.parent.mkdir(parents=True, exist_ok=True)
|
||||
conn = get_db(path)
|
||||
try:
|
||||
conn.executescript(SCHEMA)
|
||||
finally:
|
||||
conn.close()
|
||||
127
skills/wiki/scripts/wiki_embed.py
Normal file
127
skills/wiki/scripts/wiki_embed.py
Normal file
@@ -0,0 +1,127 @@
|
||||
#!/usr/bin/env python3
|
||||
# /// script
|
||||
# requires-python = ">=3.11"
|
||||
# dependencies = ["requests", "pyyaml", "markdown-it-py", "sqlite-vec==0.1.6"]
|
||||
# ///
|
||||
"""Embedding client and the identity guard over the vector space.
|
||||
|
||||
Two invariants live here:
|
||||
|
||||
* Vectors are stored L2-normalized, so cosine distance equals the dot product.
|
||||
* The query prefix must be bit-identical at index and at query time. That is the real
|
||||
reason it is persisted in `meta` rather than only read from the config — mixing
|
||||
vectors produced under two different contracts degrades results silently, which is
|
||||
the most expensive kind of bug.
|
||||
|
||||
An unreachable Ollama is not an error here: the caller stores chunks with
|
||||
`embedded_at IS NULL` and the query side says out loud that it is FTS-only.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import math
|
||||
from collections.abc import Sequence
|
||||
from dataclasses import dataclass
|
||||
|
||||
import requests
|
||||
from wiki_chunker import CHUNKER_VERSION
|
||||
from wiki_config import EmbeddingConfig
|
||||
from wiki_db import EMBEDDING_DIMS
|
||||
|
||||
SCHEMA_VERSION = "1"
|
||||
SQLITE_VEC_VERSION = "0.1.6"
|
||||
REQUEST_TIMEOUT_SECONDS = 60
|
||||
|
||||
# Only these keys make two vectors comparable; the rest of `meta` is informational.
|
||||
GUARDED_META_KEYS = (
|
||||
"embedding_model",
|
||||
"embedding_dims",
|
||||
"normalized",
|
||||
"query_prefix",
|
||||
"chunker_version",
|
||||
)
|
||||
|
||||
|
||||
class EmbeddingUnavailable(RuntimeError):
|
||||
"""Ollama could not be reached or refused the request."""
|
||||
|
||||
|
||||
class IndexIdentityMismatch(RuntimeError):
|
||||
"""The index was built under a different embedding contract — a reindex is needed."""
|
||||
|
||||
|
||||
def expected_meta(config: EmbeddingConfig) -> dict[str, str]:
|
||||
return {
|
||||
"embedding_model": config.model,
|
||||
"embedding_dims": str(config.dims),
|
||||
"normalized": "l2",
|
||||
"query_prefix": config.query_prefix,
|
||||
"chunker_version": CHUNKER_VERSION,
|
||||
"schema_version": SCHEMA_VERSION,
|
||||
"sqlite_vec_version": SQLITE_VEC_VERSION,
|
||||
}
|
||||
|
||||
|
||||
def meta_mismatches(stored: dict[str, str], config: EmbeddingConfig) -> list[str]:
|
||||
"""Guarded meta keys that disagree with the config. Empty on a fresh (unwritten) index."""
|
||||
if not stored:
|
||||
return []
|
||||
expected = expected_meta(config)
|
||||
return [key for key in GUARDED_META_KEYS if stored.get(key) != expected[key]]
|
||||
|
||||
|
||||
def require_matching_index(stored: dict[str, str], config: EmbeddingConfig) -> None:
|
||||
"""Refuse to query an index built under a different contract."""
|
||||
if config.dims != EMBEDDING_DIMS:
|
||||
raise IndexIdentityMismatch(f"reindex needed: config dims {config.dims} != schema dims {EMBEDDING_DIMS}")
|
||||
mismatches = meta_mismatches(stored, config)
|
||||
if mismatches:
|
||||
detail = ", ".join(
|
||||
f"{key}: index={stored.get(key)!r} config={expected_meta(config)[key]!r}" for key in mismatches
|
||||
)
|
||||
raise IndexIdentityMismatch(f"reindex needed: {detail}")
|
||||
|
||||
|
||||
def l2_normalize(vector: Sequence[float]) -> list[float]:
|
||||
norm = math.sqrt(sum(value * value for value in vector))
|
||||
if norm == 0.0:
|
||||
return list(vector)
|
||||
return [value / norm for value in vector]
|
||||
|
||||
|
||||
@dataclass(frozen=True)
|
||||
class OllamaEmbedder:
|
||||
config: EmbeddingConfig
|
||||
|
||||
def embed_documents(self, texts: Sequence[str]) -> list[list[float]]:
|
||||
"""Documents are embedded without the instruct prefix (the model is asymmetric)."""
|
||||
return self._embed(list(texts))
|
||||
|
||||
def embed_query(self, text: str) -> list[float]:
|
||||
return self._embed([self.config.query_prefix + text])[0]
|
||||
|
||||
def probe(self) -> None:
|
||||
"""Raise EmbeddingUnavailable unless the endpoint answers an embed request."""
|
||||
self._embed(["ping"])
|
||||
|
||||
def _embed(self, inputs: list[str]) -> list[list[float]]:
|
||||
if not inputs:
|
||||
return []
|
||||
payload = {
|
||||
"model": self.config.model,
|
||||
"input": inputs,
|
||||
"keep_alive": self.config.keep_alive,
|
||||
}
|
||||
try:
|
||||
response = requests.post(f"{self.config.endpoint}/api/embed", json=payload, timeout=REQUEST_TIMEOUT_SECONDS)
|
||||
response.raise_for_status()
|
||||
embeddings = response.json()["embeddings"]
|
||||
except (requests.RequestException, KeyError, ValueError) as exc:
|
||||
raise EmbeddingUnavailable(f"{self.config.endpoint}: {exc}") from exc
|
||||
|
||||
if len(embeddings) != len(inputs):
|
||||
raise EmbeddingUnavailable(f"asked for {len(inputs)} vectors, got {len(embeddings)}")
|
||||
for vector in embeddings:
|
||||
if len(vector) != self.config.dims:
|
||||
raise EmbeddingUnavailable(f"model returned {len(vector)} dims, config says {self.config.dims}")
|
||||
return [l2_normalize(vector) for vector in embeddings]
|
||||
308
skills/wiki/scripts/wiki_search.py
Normal file
308
skills/wiki/scripts/wiki_search.py
Normal file
@@ -0,0 +1,308 @@
|
||||
#!/usr/bin/env -S uv run --script
|
||||
# /// script
|
||||
# requires-python = ">=3.11"
|
||||
# dependencies = ["requests", "pyyaml", "markdown-it-py", "sqlite-vec==0.1.6"]
|
||||
# ///
|
||||
"""Query side of the wiki skill — three layers, cheapest first.
|
||||
|
||||
grep live ripgrep over the files on disk. No index, never stale, and not limited
|
||||
to the indexed extensions, so it is the layer that covers source code.
|
||||
toc directory -> file -> title + tags, read straight from the `files` catalog.
|
||||
search FTS5 (BM25) and vec0 (KNN) over the same `chunks` rows, merged with RRF.
|
||||
|
||||
Both halves rank the same unit, which is what makes the merge meaningful. The
|
||||
parameters below are module constants on purpose: `--limit` is the only knob worth
|
||||
exposing, and a config key that never changes is a config key that rots.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import argparse
|
||||
import json
|
||||
import shutil
|
||||
import sqlite3
|
||||
import subprocess
|
||||
import sys
|
||||
from pathlib import Path
|
||||
|
||||
import wiki_store as store
|
||||
from wiki_config import (
|
||||
GIT_KIND,
|
||||
ConfigError,
|
||||
SourceConfig,
|
||||
WikiConfig,
|
||||
db_path,
|
||||
load_config,
|
||||
source_root,
|
||||
)
|
||||
from wiki_embed import (
|
||||
EmbeddingUnavailable,
|
||||
IndexIdentityMismatch,
|
||||
OllamaEmbedder,
|
||||
require_matching_index,
|
||||
)
|
||||
|
||||
CANDIDATE_LIMIT = 50 # KNN k, and the LIMIT for BM25
|
||||
RRF_K = 60 # Cormack et al. 2009
|
||||
DEFAULT_LIMIT = 10
|
||||
PREFIX_MIN_LENGTH = 3 # a one- or two-character prefix matches too widely to carry signal
|
||||
EXCERPT_CHARS = 320
|
||||
GREP_TIMEOUT = 30
|
||||
GREP_MAX_PER_FILE = 5
|
||||
|
||||
|
||||
def rrf_merge(ranked_lists: list[list[int]]) -> list[tuple[int, float]]:
|
||||
"""score(id) = sum over lists of 1 / (RRF_K + rank). No weights: both halves count equally."""
|
||||
scores: dict[int, float] = {}
|
||||
for ids in ranked_lists:
|
||||
for rank, chunk_id in enumerate(ids, start=1):
|
||||
scores[chunk_id] = scores.get(chunk_id, 0.0) + 1.0 / (RRF_K + rank)
|
||||
return sorted(scores.items(), key=lambda item: (-item[1], item[0]))
|
||||
|
||||
|
||||
def fts_match_expression(query: str) -> str:
|
||||
"""Turn free text into an FTS5 OR-query of quoted prefix terms.
|
||||
|
||||
Terms are quoted so reserved words (`and`, `not`, `near`) and punctuation cannot be
|
||||
read as operators. The trailing `*` covers Czech inflection, which `unicode61` does
|
||||
not stem — `záloh*` finds záloha/zálohování/zálohy.
|
||||
"""
|
||||
terms = []
|
||||
for word in _tokenize(query):
|
||||
wildcard = "*" if len(word) >= PREFIX_MIN_LENGTH else ""
|
||||
terms.append(f'"{word}"{wildcard}')
|
||||
return " OR ".join(terms)
|
||||
|
||||
|
||||
def _tokenize(text: str) -> list[str]:
|
||||
return "".join(char if char.isalnum() else " " for char in text).split()
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# search
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
|
||||
def run_search(config: WikiConfig, query: str, limit: int) -> int:
|
||||
expression = fts_match_expression(query)
|
||||
if not expression:
|
||||
print("(empty query)")
|
||||
return 0
|
||||
|
||||
with store.connection(db_path()) as conn:
|
||||
try:
|
||||
require_matching_index(store.read_meta(conn), config.embedding)
|
||||
except IndexIdentityMismatch as exc:
|
||||
print(str(exc), file=sys.stderr)
|
||||
return 1
|
||||
|
||||
bm25_ids = store.bm25_ranked_ids(conn, expression, CANDIDATE_LIMIT)
|
||||
|
||||
vector_ids: list[int] = []
|
||||
degraded: str | None = None
|
||||
try:
|
||||
vector = OllamaEmbedder(config.embedding).embed_query(query)
|
||||
vector_ids = store.knn_ranked_ids(conn, vector, CANDIDATE_LIMIT)
|
||||
except EmbeddingUnavailable as exc:
|
||||
degraded = f"note: embeddings unavailable ({exc}) — FTS-only results"
|
||||
|
||||
pending = store.index_stats(conn)["pending"]
|
||||
merged = rrf_merge([ids for ids in (bm25_ids, vector_ids) if ids])[:limit]
|
||||
rows = store.fetch_chunks(conn, [chunk_id for chunk_id, _ in merged])
|
||||
|
||||
if degraded:
|
||||
print(degraded)
|
||||
elif pending:
|
||||
print(f"note: {pending} chunks still awaiting vectors — semantic half is incomplete")
|
||||
|
||||
if not merged:
|
||||
print("(no matches)")
|
||||
return 0
|
||||
|
||||
bm25_rank = {chunk_id: rank for rank, chunk_id in enumerate(bm25_ids, start=1)}
|
||||
vector_rank = {chunk_id: rank for rank, chunk_id in enumerate(vector_ids, start=1)}
|
||||
for position, (chunk_id, score) in enumerate(merged, start=1):
|
||||
row = rows.get(chunk_id)
|
||||
if row is None:
|
||||
continue
|
||||
origin = _origin_label(bm25_rank.get(chunk_id), vector_rank.get(chunk_id))
|
||||
print(f"{position}. {row['source_id']}:{row['path']} (rrf {score:.4f}, {origin})")
|
||||
print(f" {row['breadcrumb']}")
|
||||
print(_indent(_excerpt(row["text"], row["breadcrumb"])))
|
||||
print()
|
||||
return 0
|
||||
|
||||
|
||||
def _origin_label(bm25: int | None, vector: int | None) -> str:
|
||||
parts = []
|
||||
if bm25 is not None:
|
||||
parts.append(f"bm25 #{bm25}")
|
||||
if vector is not None:
|
||||
parts.append(f"vec #{vector}")
|
||||
return ", ".join(parts)
|
||||
|
||||
|
||||
def _excerpt(text: str, breadcrumb: str) -> str:
|
||||
body = text[len(breadcrumb) :].lstrip("\n") if text.startswith(breadcrumb) else text
|
||||
body = " ".join(body.split())
|
||||
return body[:EXCERPT_CHARS] + ("…" if len(body) > EXCERPT_CHARS else "")
|
||||
|
||||
|
||||
def _indent(text: str) -> str:
|
||||
return "\n".join(f" {line}" for line in text.splitlines())
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# toc
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
|
||||
def run_toc(source_id: str | None, tag: str | None) -> int:
|
||||
with store.connection(db_path()) as conn:
|
||||
rows = store.list_toc_files(conn, source_id=source_id, tag=tag)
|
||||
if not rows:
|
||||
print("(no indexed files match)")
|
||||
return 0
|
||||
|
||||
by_source: dict[str, list[sqlite3.Row]] = {}
|
||||
for row in rows:
|
||||
by_source.setdefault(row["source_id"], []).append(row)
|
||||
|
||||
for source, files in by_source.items():
|
||||
print(f"{source} ({len(files)} files)")
|
||||
print()
|
||||
directory = None
|
||||
for row in files:
|
||||
path = row["path"]
|
||||
parent, _, name = path.rpartition("/")
|
||||
if parent != directory:
|
||||
directory = parent
|
||||
print(f"{parent}/" if parent else "./")
|
||||
print(f" {name:<28} {row['title'] or '':<32} {_tag_label(row['tags'])}".rstrip())
|
||||
print()
|
||||
return 0
|
||||
|
||||
|
||||
def _tag_label(raw: str | None) -> str:
|
||||
try:
|
||||
tags = json.loads(raw) if raw else []
|
||||
except json.JSONDecodeError:
|
||||
return ""
|
||||
return f"[{', '.join(tags)}]" if tags else ""
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# grep
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
|
||||
def grep_roots(source: SourceConfig) -> list[Path]:
|
||||
"""Directories ripgrep should walk for one source.
|
||||
|
||||
A git clone is searched whole. A workspace source is bounded by the literal prefix
|
||||
of each `paths` glob, so grep stays inside what the source claims without being
|
||||
narrowed to the indexed extensions.
|
||||
"""
|
||||
root = source_root(source)
|
||||
if source.kind == GIT_KIND:
|
||||
return [root] if root.is_dir() else []
|
||||
roots = []
|
||||
for pattern in source.paths:
|
||||
prefix = _literal_prefix(pattern)
|
||||
candidate = root / prefix if prefix else root
|
||||
if candidate.is_dir() and candidate not in roots:
|
||||
roots.append(candidate)
|
||||
return roots
|
||||
|
||||
|
||||
def _literal_prefix(pattern: str) -> str:
|
||||
parts = []
|
||||
for segment in pattern.split("/"):
|
||||
if any(char in segment for char in "*?["):
|
||||
break
|
||||
parts.append(segment)
|
||||
return "/".join(parts)
|
||||
|
||||
|
||||
def run_grep(config: WikiConfig, pattern: str, source_id: str | None) -> int:
|
||||
if shutil.which("rg") is None:
|
||||
print("ripgrep (rg) not found", file=sys.stderr)
|
||||
return 1
|
||||
sources = [s for s in config.sources if not source_id or s.source_id == source_id]
|
||||
if not sources:
|
||||
print(f"unknown source {source_id!r}", file=sys.stderr)
|
||||
return 1
|
||||
|
||||
roots = [path for source in sources for path in grep_roots(source)]
|
||||
if not roots:
|
||||
print("(nothing on disk to grep — has wiki_sync.py run?)")
|
||||
return 0
|
||||
|
||||
command = [
|
||||
"rg",
|
||||
"--line-number",
|
||||
"--no-heading",
|
||||
"--color",
|
||||
"never",
|
||||
"--smart-case",
|
||||
"--max-count",
|
||||
str(GREP_MAX_PER_FILE),
|
||||
"--glob",
|
||||
"!.git",
|
||||
pattern,
|
||||
*[str(path) for path in roots],
|
||||
]
|
||||
try:
|
||||
result = subprocess.run(command, capture_output=True, text=True, timeout=GREP_TIMEOUT, check=False)
|
||||
except subprocess.TimeoutExpired:
|
||||
print(f"grep timed out after {GREP_TIMEOUT}s", file=sys.stderr)
|
||||
return 1
|
||||
if result.returncode not in (0, 1):
|
||||
print(result.stderr.strip(), file=sys.stderr)
|
||||
return 1
|
||||
output = result.stdout.strip()
|
||||
print(output if output else "(no matches)")
|
||||
return 0
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# cli
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
|
||||
def parse_args(argv: list[str] | None = None) -> argparse.Namespace:
|
||||
parser = argparse.ArgumentParser(description="Search the wiki index over your notes.")
|
||||
subparsers = parser.add_subparsers(dest="command", required=True)
|
||||
|
||||
search = subparsers.add_parser("search", help="hybrid BM25 + vector search over chunks")
|
||||
search.add_argument("query", help="free text; Czech inflection is covered by prefix matching")
|
||||
search.add_argument("--limit", type=int, default=DEFAULT_LIMIT, help="chunks to return")
|
||||
|
||||
toc = subparsers.add_parser("toc", help="directory -> file -> title + tags from the catalog")
|
||||
toc.add_argument("--source", help="limit to one source id")
|
||||
toc.add_argument("--tag", help="only files carrying this frontmatter tag")
|
||||
|
||||
grep = subparsers.add_parser("grep", help="live ripgrep over the files on disk, index-free")
|
||||
grep.add_argument("pattern", help="ripgrep regex")
|
||||
grep.add_argument("--source", help="limit to one source id")
|
||||
|
||||
return parser.parse_args(argv)
|
||||
|
||||
|
||||
def main(argv: list[str] | None = None) -> int:
|
||||
args = parse_args(argv)
|
||||
try:
|
||||
config = load_config()
|
||||
except ConfigError as exc:
|
||||
print(f"config error: {exc}", file=sys.stderr)
|
||||
return 1
|
||||
|
||||
if args.command == "search":
|
||||
return run_search(config, args.query, args.limit)
|
||||
if args.command == "toc":
|
||||
return run_toc(args.source, args.tag)
|
||||
return run_grep(config, args.pattern, args.source)
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
sys.exit(main())
|
||||
280
skills/wiki/scripts/wiki_store.py
Normal file
280
skills/wiki/scripts/wiki_store.py
Normal file
@@ -0,0 +1,280 @@
|
||||
#!/usr/bin/env python3
|
||||
# /// script
|
||||
# requires-python = ">=3.11"
|
||||
# dependencies = ["sqlite-vec==0.1.6"]
|
||||
# ///
|
||||
"""Data-access layer for the wiki skill.
|
||||
|
||||
Pure SQL plus lifecycle helpers. No printing, no argparse, no sys.exit.
|
||||
|
||||
`vec_chunks` is a vec0 virtual table and therefore NOT reachable by the foreign key
|
||||
cascade that cleans up `chunks` and `chunks_fts`. Every path that removes chunks must
|
||||
delete their vectors explicitly — that is why the deletes here go through
|
||||
`_delete_vectors_for_file`.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import json
|
||||
import sqlite3
|
||||
from collections.abc import Iterator, Sequence
|
||||
from contextlib import contextmanager
|
||||
from pathlib import Path
|
||||
|
||||
from sqlite_vec import serialize_float32 # ty: ignore[unresolved-import]
|
||||
from wiki_db import get_db, init_db
|
||||
|
||||
|
||||
@contextmanager
|
||||
def connection(db_path: Path) -> Iterator[sqlite3.Connection]:
|
||||
"""Open a connection, initialising the DB if missing."""
|
||||
if not db_path.exists():
|
||||
init_db(db_path)
|
||||
conn = get_db(db_path)
|
||||
try:
|
||||
yield conn
|
||||
finally:
|
||||
conn.close()
|
||||
|
||||
|
||||
@contextmanager
|
||||
def tx(conn: sqlite3.Connection) -> Iterator[sqlite3.Connection]:
|
||||
"""Wrap an already-open connection in an explicit transaction.
|
||||
|
||||
The connection is in autocommit mode (`isolation_level=None`), so transactions are
|
||||
ours to open — sqlite3's implicit handling would otherwise fail a nested BEGIN.
|
||||
"""
|
||||
conn.execute("BEGIN")
|
||||
try:
|
||||
yield conn
|
||||
conn.execute("COMMIT")
|
||||
except Exception:
|
||||
conn.execute("ROLLBACK")
|
||||
raise
|
||||
|
||||
|
||||
@contextmanager
|
||||
def transaction(db_path: Path) -> Iterator[sqlite3.Connection]:
|
||||
"""Open a connection wrapped in an explicit transaction."""
|
||||
with connection(db_path) as conn, tx(conn):
|
||||
yield conn
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# meta — identity of the embedding space
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
|
||||
def read_meta(conn: sqlite3.Connection) -> dict[str, str]:
|
||||
return {row["key"]: row["value"] for row in conn.execute("SELECT key, value FROM meta")}
|
||||
|
||||
|
||||
def write_meta(conn: sqlite3.Connection, values: dict[str, str]) -> None:
|
||||
conn.executemany(
|
||||
"INSERT INTO meta (key, value) VALUES (?, ?) ON CONFLICT(key) DO UPDATE SET value = excluded.value",
|
||||
sorted(values.items()),
|
||||
)
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# sources
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
|
||||
def upsert_source(conn: sqlite3.Connection, source_id: str, kind: str) -> None:
|
||||
conn.execute(
|
||||
"INSERT INTO sources (source_id, kind) VALUES (?, ?) ON CONFLICT(source_id) DO UPDATE SET kind = excluded.kind",
|
||||
(source_id, kind),
|
||||
)
|
||||
|
||||
|
||||
def get_source(conn: sqlite3.Connection, source_id: str) -> sqlite3.Row | None:
|
||||
return conn.execute("SELECT * FROM sources WHERE source_id = ?", (source_id,)).fetchone()
|
||||
|
||||
|
||||
def mark_synced(conn: sqlite3.Connection, source_id: str, indexed_rev: str | None, now: str) -> None:
|
||||
conn.execute(
|
||||
"UPDATE sources SET indexed_rev = ?, last_sync_at = ? WHERE source_id = ?",
|
||||
(indexed_rev, now, source_id),
|
||||
)
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# files
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
|
||||
def get_file(conn: sqlite3.Connection, source_id: str, path: str) -> sqlite3.Row | None:
|
||||
return conn.execute("SELECT * FROM files WHERE source_id = ? AND path = ?", (source_id, path)).fetchone()
|
||||
|
||||
|
||||
def list_source_files(conn: sqlite3.Connection, source_id: str) -> dict[str, sqlite3.Row]:
|
||||
"""Indexed files of one source, keyed by path — the basis for deletion detection."""
|
||||
rows = conn.execute("SELECT * FROM files WHERE source_id = ?", (source_id,))
|
||||
return {row["path"]: row for row in rows}
|
||||
|
||||
|
||||
def upsert_file(
|
||||
conn: sqlite3.Connection,
|
||||
source_id: str,
|
||||
path: str,
|
||||
title: str | None,
|
||||
tags: list[str],
|
||||
headings: list[str],
|
||||
sha256: str,
|
||||
size: int,
|
||||
mtime: float | None,
|
||||
now: str,
|
||||
) -> None:
|
||||
conn.execute(
|
||||
"INSERT INTO files (source_id, path, title, tags, headings, sha256, size, mtime, indexed_at) "
|
||||
"VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?) "
|
||||
"ON CONFLICT(source_id, path) DO UPDATE SET "
|
||||
"title = excluded.title, tags = excluded.tags, headings = excluded.headings, "
|
||||
"sha256 = excluded.sha256, size = excluded.size, mtime = excluded.mtime, "
|
||||
"indexed_at = excluded.indexed_at",
|
||||
(
|
||||
source_id,
|
||||
path,
|
||||
title,
|
||||
json.dumps(tags, ensure_ascii=False),
|
||||
json.dumps(headings, ensure_ascii=False),
|
||||
sha256,
|
||||
size,
|
||||
mtime,
|
||||
now,
|
||||
),
|
||||
)
|
||||
|
||||
|
||||
def touch_file_stat(conn: sqlite3.Connection, source_id: str, path: str, size: int, mtime: float | None) -> None:
|
||||
"""Refresh the cheap change-detection stats after a sha256 match (content unchanged)."""
|
||||
conn.execute(
|
||||
"UPDATE files SET size = ?, mtime = ? WHERE source_id = ? AND path = ?",
|
||||
(size, mtime, source_id, path),
|
||||
)
|
||||
|
||||
|
||||
def delete_file(conn: sqlite3.Connection, source_id: str, path: str) -> None:
|
||||
"""Drop a file and everything derived from it, vec0 rows included."""
|
||||
_delete_vectors_for_file(conn, source_id, path)
|
||||
conn.execute("DELETE FROM files WHERE source_id = ? AND path = ?", (source_id, path))
|
||||
|
||||
|
||||
def list_toc_files(conn: sqlite3.Connection, source_id: str | None = None, tag: str | None = None) -> list[sqlite3.Row]:
|
||||
sql = "SELECT source_id, path, title, tags FROM files"
|
||||
clauses: list[str] = []
|
||||
params: list[str] = []
|
||||
if source_id:
|
||||
clauses.append("source_id = ?")
|
||||
params.append(source_id)
|
||||
if tag:
|
||||
clauses.append("EXISTS (SELECT 1 FROM json_each(files.tags) WHERE value = ?)")
|
||||
params.append(tag)
|
||||
if clauses:
|
||||
sql += " WHERE " + " AND ".join(clauses)
|
||||
sql += " ORDER BY source_id, path"
|
||||
return list(conn.execute(sql, params))
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# chunks
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
|
||||
def _delete_vectors_for_file(conn: sqlite3.Connection, source_id: str, path: str) -> None:
|
||||
ids = [
|
||||
row["id"] for row in conn.execute("SELECT id FROM chunks WHERE source_id = ? AND path = ?", (source_id, path))
|
||||
]
|
||||
conn.executemany("DELETE FROM vec_chunks WHERE rowid = ?", [(i,) for i in ids])
|
||||
|
||||
|
||||
def replace_chunks(conn: sqlite3.Connection, source_id: str, path: str, chunks: Sequence[tuple[str, str]]) -> None:
|
||||
"""Swap a file's chunks for a freshly built set, as (breadcrumb, text) in order.
|
||||
|
||||
New chunks land with `embedded_at IS NULL`; the embed pass picks them up, so an
|
||||
unreachable Ollama degrades to FTS-only instead of failing the sync.
|
||||
"""
|
||||
_delete_vectors_for_file(conn, source_id, path)
|
||||
conn.execute("DELETE FROM chunks WHERE source_id = ? AND path = ?", (source_id, path))
|
||||
conn.executemany(
|
||||
"INSERT INTO chunks (source_id, path, chunk_idx, breadcrumb, text) VALUES (?, ?, ?, ?, ?)",
|
||||
[(source_id, path, idx, breadcrumb, text) for idx, (breadcrumb, text) in enumerate(chunks)],
|
||||
)
|
||||
|
||||
|
||||
def pending_chunks(conn: sqlite3.Connection, limit: int) -> list[sqlite3.Row]:
|
||||
return list(
|
||||
conn.execute(
|
||||
"SELECT id, breadcrumb, text FROM chunks WHERE embedded_at IS NULL ORDER BY id LIMIT ?",
|
||||
(limit,),
|
||||
)
|
||||
)
|
||||
|
||||
|
||||
def has_pending(conn: sqlite3.Connection) -> bool:
|
||||
return conn.execute("SELECT 1 FROM chunks WHERE embedded_at IS NULL LIMIT 1").fetchone() is not None
|
||||
|
||||
|
||||
def store_embedding(conn: sqlite3.Connection, chunk_id: int, vector: Sequence[float], now: str) -> None:
|
||||
conn.execute("DELETE FROM vec_chunks WHERE rowid = ?", (chunk_id,))
|
||||
conn.execute(
|
||||
"INSERT INTO vec_chunks (rowid, embedding) VALUES (?, ?)",
|
||||
(chunk_id, serialize_float32(list(vector))),
|
||||
)
|
||||
conn.execute("UPDATE chunks SET embedded_at = ? WHERE id = ?", (now, chunk_id))
|
||||
|
||||
|
||||
def reset_index(conn: sqlite3.Connection) -> None:
|
||||
"""Drop every indexed artifact, keeping the source rows. Used by `--full`."""
|
||||
ids = [row["id"] for row in conn.execute("SELECT id FROM chunks")]
|
||||
conn.executemany("DELETE FROM vec_chunks WHERE rowid = ?", [(i,) for i in ids])
|
||||
conn.execute("DELETE FROM files")
|
||||
conn.execute("UPDATE sources SET indexed_rev = NULL")
|
||||
|
||||
|
||||
def index_stats(conn: sqlite3.Connection) -> dict[str, int]:
|
||||
return {
|
||||
"files": conn.execute("SELECT count(*) AS n FROM files").fetchone()["n"],
|
||||
"chunks": conn.execute("SELECT count(*) AS n FROM chunks").fetchone()["n"],
|
||||
"vectors": conn.execute("SELECT count(*) AS n FROM vec_chunks").fetchone()["n"],
|
||||
"pending": conn.execute("SELECT count(*) AS n FROM chunks WHERE embedded_at IS NULL").fetchone()["n"],
|
||||
}
|
||||
|
||||
|
||||
def orphan_vector_ids(conn: sqlite3.Connection) -> list[int]:
|
||||
"""Vector rowids with no surviving chunk — must always be empty (regression guard)."""
|
||||
rows = conn.execute("SELECT rowid AS rid FROM vec_chunks WHERE rowid NOT IN (SELECT id FROM chunks)")
|
||||
return [row["rid"] for row in rows]
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# retrieval
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
|
||||
def bm25_ranked_ids(conn: sqlite3.Connection, match_expr: str, limit: int) -> list[int]:
|
||||
rows = conn.execute(
|
||||
"SELECT rowid AS rid FROM chunks_fts WHERE chunks_fts MATCH ? ORDER BY rank LIMIT ?",
|
||||
(match_expr, limit),
|
||||
)
|
||||
return [row["rid"] for row in rows]
|
||||
|
||||
|
||||
def knn_ranked_ids(conn: sqlite3.Connection, vector: Sequence[float], k: int) -> list[int]:
|
||||
rows = conn.execute(
|
||||
"SELECT rowid AS rid FROM vec_chunks WHERE embedding MATCH ? AND k = ? ORDER BY distance",
|
||||
(serialize_float32(list(vector)), k),
|
||||
)
|
||||
return [row["rid"] for row in rows]
|
||||
|
||||
|
||||
def fetch_chunks(conn: sqlite3.Connection, ids: Sequence[int]) -> dict[int, sqlite3.Row]:
|
||||
if not ids:
|
||||
return {}
|
||||
placeholders = ",".join("?" * len(ids))
|
||||
rows = conn.execute(
|
||||
f"SELECT id, source_id, path, chunk_idx, breadcrumb, text FROM chunks WHERE id IN ({placeholders})",
|
||||
list(ids),
|
||||
)
|
||||
return {row["id"]: row for row in rows}
|
||||
529
skills/wiki/scripts/wiki_sync.py
Normal file
529
skills/wiki/scripts/wiki_sync.py
Normal file
@@ -0,0 +1,529 @@
|
||||
#!/usr/bin/env -S uv run --script
|
||||
# /// script
|
||||
# requires-python = ">=3.11"
|
||||
# dependencies = ["requests", "pyyaml", "markdown-it-py", "sqlite-vec==0.1.6"]
|
||||
# ///
|
||||
"""Offline indexer for the wiki skill. Runs every minute from the nanobot crontab.
|
||||
|
||||
Indexing never happens inside an agent turn: the `exec` tool times out at 60 s while a
|
||||
full index of 10^4 chunks takes ~93 s. The agent only ever reads a finished index.
|
||||
|
||||
Per tick:
|
||||
|
||||
1. take `wiki/.sync.lock`; a live holder means exit 0 in silence
|
||||
2. detect changes cheaply — `git ls-remote` (no fetch) for git sources, a
|
||||
path+size+mtime walk for workspace sources, sha256 only on a stat mismatch
|
||||
3. nothing changed and nothing pending -> exit 0 (the overwhelming majority of ticks)
|
||||
4. re-chunk changed files, then drain every chunk with `embedded_at IS NULL` — which
|
||||
is also the way back out of degraded mode after Ollama returns
|
||||
5. record indexed_rev / last_sync_at
|
||||
6. log a coverage line naming top-level directories no source covers
|
||||
7. append the run summary to log/wiki_sync.log
|
||||
|
||||
Network failure is per-source: a timeout or non-zero git exit logs WARN, skips that
|
||||
source with its `indexed_rev` untouched, and lets the others finish. Without the
|
||||
timeouts a hanging `ls-remote` would hold the lock and block workspace sources that
|
||||
have nothing to do with the network.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import argparse
|
||||
import hashlib
|
||||
import json
|
||||
import os
|
||||
import subprocess
|
||||
import sys
|
||||
from dataclasses import dataclass, field
|
||||
from datetime import UTC, datetime
|
||||
from pathlib import Path
|
||||
|
||||
import wiki_store as store
|
||||
from wiki_chunker import parse_markdown
|
||||
from wiki_config import (
|
||||
GIT_KIND,
|
||||
LOCK_PATH,
|
||||
SYNC_LOG_PATH,
|
||||
WIKI_DIR,
|
||||
ConfigError,
|
||||
SourceConfig,
|
||||
WikiConfig,
|
||||
db_path,
|
||||
load_config,
|
||||
source_root,
|
||||
)
|
||||
from wiki_embed import (
|
||||
EmbeddingUnavailable,
|
||||
OllamaEmbedder,
|
||||
expected_meta,
|
||||
meta_mismatches,
|
||||
)
|
||||
|
||||
STALE_SECONDS = 30 * 60
|
||||
LS_REMOTE_TIMEOUT = 20
|
||||
GIT_TIMEOUT = 300
|
||||
GIT_ERROR_CHARS = 300
|
||||
SKIP_DIRS = frozenset({".git"})
|
||||
|
||||
|
||||
class GitError(RuntimeError):
|
||||
"""A git subprocess failed or timed out."""
|
||||
|
||||
|
||||
@dataclass
|
||||
class SourcePlan:
|
||||
"""What one source needs done this tick."""
|
||||
|
||||
changed: list[str] = field(default_factory=list)
|
||||
deleted: list[str] = field(default_factory=list)
|
||||
stat_refresh: list[tuple[str, int, float]] = field(default_factory=list)
|
||||
new_rev: str | None = None
|
||||
|
||||
@property
|
||||
def has_work(self) -> bool:
|
||||
return bool(self.changed or self.deleted or self.stat_refresh)
|
||||
|
||||
|
||||
def log(message: str) -> None:
|
||||
SYNC_LOG_PATH.parent.mkdir(parents=True, exist_ok=True)
|
||||
stamp = datetime.now().astimezone().isoformat(timespec="seconds")
|
||||
with SYNC_LOG_PATH.open("a", encoding="utf-8") as handle:
|
||||
handle.write(f"{stamp} {message}\n")
|
||||
|
||||
|
||||
def _now() -> str:
|
||||
return datetime.now(UTC).isoformat(timespec="seconds")
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# lock
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
|
||||
def _pid_alive(pid: int) -> bool:
|
||||
try:
|
||||
os.kill(pid, 0)
|
||||
except ProcessLookupError:
|
||||
return False
|
||||
except PermissionError:
|
||||
return True
|
||||
return True
|
||||
|
||||
|
||||
def _lock_is_stale() -> bool:
|
||||
"""A lock is dead if unreadable, its pid is gone, or it is older than STALE_SECONDS."""
|
||||
try:
|
||||
data = json.loads(LOCK_PATH.read_text(encoding="utf-8"))
|
||||
pid = int(data["pid"])
|
||||
started = datetime.fromisoformat(data["started_at"])
|
||||
except (OSError, ValueError, KeyError):
|
||||
return True
|
||||
if not _pid_alive(pid):
|
||||
return True
|
||||
return (datetime.now().astimezone() - started).total_seconds() > STALE_SECONDS
|
||||
|
||||
|
||||
def acquire_lock() -> bool:
|
||||
"""Atomically create the lock. False when a live sync already runs.
|
||||
|
||||
Reclaiming a stale lock is not optional: without it a crashed run (OOM, reboot)
|
||||
would make every later cron tick exit 0 in silence, forever.
|
||||
"""
|
||||
LOCK_PATH.parent.mkdir(parents=True, exist_ok=True)
|
||||
for _ in range(2):
|
||||
try:
|
||||
handle = os.open(LOCK_PATH, os.O_CREAT | os.O_EXCL | os.O_WRONLY, 0o644)
|
||||
except FileExistsError:
|
||||
if not _lock_is_stale():
|
||||
return False
|
||||
log("WARN stale lock, reclaiming")
|
||||
LOCK_PATH.unlink(missing_ok=True)
|
||||
continue
|
||||
payload = {"pid": os.getpid(), "started_at": datetime.now().astimezone().isoformat()}
|
||||
with os.fdopen(handle, "w", encoding="utf-8") as file:
|
||||
json.dump(payload, file)
|
||||
return True
|
||||
return False
|
||||
|
||||
|
||||
def release_lock() -> None:
|
||||
LOCK_PATH.unlink(missing_ok=True)
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# filesystem
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
|
||||
def _sha256(path: Path) -> str:
|
||||
digest = hashlib.sha256()
|
||||
with path.open("rb") as handle:
|
||||
for block in iter(lambda: handle.read(65536), b""):
|
||||
digest.update(block)
|
||||
return digest.hexdigest()
|
||||
|
||||
|
||||
def walk_source(source: SourceConfig, root: Path) -> dict[str, tuple[int, float]]:
|
||||
"""Covered files under `root`, as rel_path -> (size, mtime). Prunes uncovered dirs."""
|
||||
found: dict[str, tuple[int, float]] = {}
|
||||
if not root.is_dir():
|
||||
return found
|
||||
for dirpath, dirnames, filenames in os.walk(root):
|
||||
rel_dir = os.path.relpath(dirpath, root)
|
||||
rel_dir = "" if rel_dir == "." else rel_dir
|
||||
dirnames[:] = [
|
||||
name for name in dirnames if name not in SKIP_DIRS and source.covers_dir(f"{rel_dir}/{name}".lstrip("/"))
|
||||
]
|
||||
for name in filenames:
|
||||
rel_path = f"{rel_dir}/{name}".lstrip("/")
|
||||
if not source.covers(rel_path):
|
||||
continue
|
||||
file_path = Path(dirpath) / name
|
||||
if not file_path.is_file():
|
||||
continue
|
||||
stat = file_path.stat()
|
||||
found[rel_path] = (stat.st_size, stat.st_mtime)
|
||||
return found
|
||||
|
||||
|
||||
def uncovered_top_level_dirs(source: SourceConfig, root: Path) -> list[str]:
|
||||
"""Top-level directories holding markdown that no source path reaches.
|
||||
|
||||
This is the safety net under the whitelist: a directory that silently fails to be
|
||||
indexed shows up here instead of nowhere. `wiki/` is skipped because it holds this
|
||||
skill's own clones and index — never a candidate, so reporting it is pure noise.
|
||||
"""
|
||||
if not root.is_dir():
|
||||
return []
|
||||
uncovered = []
|
||||
for entry in sorted(root.iterdir()):
|
||||
if not entry.is_dir() or entry.name in SKIP_DIRS or entry == WIKI_DIR:
|
||||
continue
|
||||
if source.covers_dir(entry.name):
|
||||
continue
|
||||
if next(entry.rglob("*.md"), None) is not None:
|
||||
uncovered.append(entry.name)
|
||||
return uncovered
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# git driver
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
|
||||
def _git(args: list[str], cwd: Path | None = None, timeout: int = GIT_TIMEOUT) -> str:
|
||||
try:
|
||||
result = subprocess.run(
|
||||
["git", *args],
|
||||
cwd=cwd,
|
||||
capture_output=True,
|
||||
text=True,
|
||||
timeout=timeout,
|
||||
check=False,
|
||||
)
|
||||
except subprocess.TimeoutExpired as exc:
|
||||
raise GitError(f"git {' '.join(args)} timed out after {timeout}s") from exc
|
||||
if result.returncode != 0:
|
||||
raise GitError(f"git {' '.join(args)} failed: {_one_line(result.stderr)}")
|
||||
return result.stdout
|
||||
|
||||
|
||||
def _one_line(text: str) -> str:
|
||||
"""Squash git's multi-line stderr into one capped line.
|
||||
|
||||
A permanently unreachable source warns on every tick, so a six-line stderr would
|
||||
put thousands of lines a day into the log and drown everything else.
|
||||
"""
|
||||
collapsed = " ".join(text.split())
|
||||
return collapsed[:GIT_ERROR_CHARS] + ("…" if len(collapsed) > GIT_ERROR_CHARS else "")
|
||||
|
||||
|
||||
def remote_head(url: str) -> str:
|
||||
"""Remote HEAD without fetching — the right tool for a per-minute cadence."""
|
||||
output = _git(["ls-remote", "--symref", url, "HEAD"], timeout=LS_REMOTE_TIMEOUT)
|
||||
for line in output.splitlines():
|
||||
parts = line.split()
|
||||
if len(parts) == 2 and parts[1] == "HEAD":
|
||||
return parts[0]
|
||||
raise GitError(f"no HEAD in ls-remote output for {url}")
|
||||
|
||||
|
||||
def _diff_names(root: Path, base_rev: str) -> tuple[list[str], list[str]] | None:
|
||||
"""(changed, deleted) between base_rev and FETCH_HEAD, or None if base_rev is gone."""
|
||||
try:
|
||||
output = _git(["diff", "--name-status", f"{base_rev}..FETCH_HEAD"], cwd=root)
|
||||
except GitError:
|
||||
return None
|
||||
changed, deleted = [], []
|
||||
for line in output.splitlines():
|
||||
fields = line.split("\t")
|
||||
if len(fields) < 2:
|
||||
continue
|
||||
status = fields[0]
|
||||
if status.startswith("R") and len(fields) >= 3:
|
||||
deleted.append(fields[1])
|
||||
changed.append(fields[2])
|
||||
elif status.startswith("D"):
|
||||
deleted.append(fields[1])
|
||||
else:
|
||||
changed.append(fields[1])
|
||||
return changed, deleted
|
||||
|
||||
|
||||
def plan_git_source(conn, source: SourceConfig, root: Path, full: bool) -> SourcePlan:
|
||||
rev = remote_head(str(source.url))
|
||||
row = store.get_source(conn, source.source_id)
|
||||
indexed_rev = row["indexed_rev"] if row else None
|
||||
|
||||
if not root.exists():
|
||||
_git(["clone", str(source.url), str(root)])
|
||||
indexed_rev = None
|
||||
if not full and indexed_rev == rev:
|
||||
return SourcePlan(new_rev=rev)
|
||||
|
||||
_git(["fetch", "--prune"], cwd=root)
|
||||
diff = None if (full or not indexed_rev) else _diff_names(root, indexed_rev)
|
||||
_git(["reset", "--hard", "FETCH_HEAD"], cwd=root)
|
||||
|
||||
indexed = store.list_source_files(conn, source.source_id)
|
||||
on_disk = walk_source(source, root)
|
||||
if diff is None:
|
||||
# Fresh clone, --full, or an indexed_rev the repo no longer has: reindex it all.
|
||||
return SourcePlan(
|
||||
changed=sorted(on_disk),
|
||||
deleted=sorted(path for path in indexed if path not in on_disk),
|
||||
new_rev=rev,
|
||||
)
|
||||
raw_changed, raw_deleted = diff
|
||||
changed = sorted({path for path in raw_changed if path in on_disk})
|
||||
deleted = sorted(
|
||||
{path for path in raw_deleted if path in indexed} | {path for path in indexed if path not in on_disk}
|
||||
)
|
||||
return SourcePlan(changed=changed, deleted=[p for p in deleted if p not in changed], new_rev=rev)
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# workspace driver
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
|
||||
def plan_workspace_source(conn, source: SourceConfig, root: Path, full: bool) -> SourcePlan:
|
||||
on_disk = walk_source(source, root)
|
||||
indexed = store.list_source_files(conn, source.source_id)
|
||||
plan = SourcePlan()
|
||||
for rel_path, (size, mtime) in sorted(on_disk.items()):
|
||||
row = indexed.get(rel_path)
|
||||
if full or row is None:
|
||||
plan.changed.append(rel_path)
|
||||
continue
|
||||
if row["size"] == size and row["mtime"] == mtime:
|
||||
continue
|
||||
if _sha256(root / rel_path) == row["sha256"]:
|
||||
# Content is identical; refresh the stats so the next tick stays cheap.
|
||||
plan.stat_refresh.append((rel_path, size, mtime))
|
||||
else:
|
||||
plan.changed.append(rel_path)
|
||||
plan.deleted = sorted(path for path in indexed if path not in on_disk)
|
||||
return plan
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# indexing
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
|
||||
def index_file(conn, source_id: str, root: Path, rel_path: str, now: str) -> int:
|
||||
"""Re-chunk one file. Returns the chunk count, or -1 when it could not be read."""
|
||||
file_path = root / rel_path
|
||||
try:
|
||||
text = file_path.read_text(encoding="utf-8")
|
||||
stat = file_path.stat()
|
||||
except (OSError, UnicodeDecodeError):
|
||||
return -1
|
||||
parsed = parse_markdown(text, rel_path)
|
||||
store.upsert_file(
|
||||
conn,
|
||||
source_id,
|
||||
rel_path,
|
||||
parsed.title,
|
||||
parsed.tags,
|
||||
parsed.headings,
|
||||
_sha256(file_path),
|
||||
stat.st_size,
|
||||
stat.st_mtime,
|
||||
now,
|
||||
)
|
||||
store.replace_chunks(conn, source_id, rel_path, [(c.breadcrumb, c.text) for c in parsed.chunks])
|
||||
return len(parsed.chunks)
|
||||
|
||||
|
||||
def apply_plan(conn, source: SourceConfig, root: Path, plan: SourcePlan) -> dict[str, int]:
|
||||
counts = {"indexed": 0, "deleted": 0, "unreadable": 0, "chunks": 0}
|
||||
now = _now()
|
||||
for rel_path in plan.deleted:
|
||||
with store.tx(conn):
|
||||
store.delete_file(conn, source.source_id, rel_path)
|
||||
counts["deleted"] += 1
|
||||
for rel_path in plan.changed:
|
||||
with store.tx(conn):
|
||||
chunks = index_file(conn, source.source_id, root, rel_path, now)
|
||||
if chunks < 0:
|
||||
counts["unreadable"] += 1
|
||||
log(f"WARN {source.source_id}: unreadable {rel_path}")
|
||||
continue
|
||||
counts["indexed"] += 1
|
||||
counts["chunks"] += chunks
|
||||
if plan.stat_refresh:
|
||||
with store.tx(conn):
|
||||
for rel_path, size, mtime in plan.stat_refresh:
|
||||
store.touch_file_stat(conn, source.source_id, rel_path, size, mtime)
|
||||
with store.tx(conn):
|
||||
store.mark_synced(conn, source.source_id, plan.new_rev, now)
|
||||
return counts
|
||||
|
||||
|
||||
def drain_pending(conn, config: WikiConfig) -> tuple[int, str | None]:
|
||||
"""Embed every chunk still missing a vector. Returns (embedded, warning)."""
|
||||
if not store.has_pending(conn):
|
||||
return 0, None
|
||||
embedder = OllamaEmbedder(config.embedding)
|
||||
embedded = 0
|
||||
while True:
|
||||
batch = store.pending_chunks(conn, config.embedding.batch)
|
||||
if not batch:
|
||||
return embedded, None
|
||||
try:
|
||||
vectors = embedder.embed_documents([row["text"] for row in batch])
|
||||
except EmbeddingUnavailable as exc:
|
||||
return embedded, f"embeddings unavailable, staying FTS-only: {exc}"
|
||||
now = _now()
|
||||
with store.tx(conn):
|
||||
for row, vector in zip(batch, vectors, strict=True):
|
||||
store.store_embedding(conn, row["id"], vector, now)
|
||||
embedded += len(batch)
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# run
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
|
||||
def ensure_meta(conn, config: WikiConfig, full: bool) -> str | None:
|
||||
"""Write the index identity, or report a mismatch that only --full can resolve."""
|
||||
stored = store.read_meta(conn)
|
||||
mismatches = meta_mismatches(stored, config.embedding)
|
||||
if mismatches and not full:
|
||||
return f"index identity mismatch on {', '.join(mismatches)} — run wiki_sync.py --full"
|
||||
with store.tx(conn):
|
||||
store.write_meta(conn, expected_meta(config.embedding))
|
||||
return None
|
||||
|
||||
|
||||
def run(config: WikiConfig, args: argparse.Namespace) -> int:
|
||||
sources = [s for s in config.sources if not args.source or s.source_id == args.source]
|
||||
if not sources:
|
||||
log(f"WARN unknown source {args.source!r}")
|
||||
return 1
|
||||
|
||||
with store.connection(db_path()) as conn:
|
||||
problem = ensure_meta(conn, config, args.full)
|
||||
if problem:
|
||||
log(f"WARN {problem}")
|
||||
return 1
|
||||
|
||||
for source in sources:
|
||||
with store.tx(conn):
|
||||
store.upsert_source(conn, source.source_id, source.kind)
|
||||
|
||||
warnings: list[str] = []
|
||||
did_work = args.full
|
||||
for source in sources:
|
||||
root = source_root(source)
|
||||
try:
|
||||
if source.kind == GIT_KIND:
|
||||
plan = plan_git_source(conn, source, root, args.full)
|
||||
else:
|
||||
plan = plan_workspace_source(conn, source, root, args.full)
|
||||
except GitError as exc:
|
||||
# indexed_rev stays untouched, so the next tick retries this source.
|
||||
warnings.append(f"{source.source_id}: {exc}")
|
||||
log(f"WARN {source.source_id}: {exc}")
|
||||
continue
|
||||
|
||||
row = store.get_source(conn, source.source_id)
|
||||
indexed_rev = row["indexed_rev"] if row else None
|
||||
rev_moved = plan.new_rev is not None and plan.new_rev != indexed_rev
|
||||
if not plan.has_work and not rev_moved:
|
||||
continue
|
||||
|
||||
did_work = True
|
||||
if not plan.has_work:
|
||||
# Upstream moved but touched nothing we index — record the rev so the
|
||||
# next tick stops fetching.
|
||||
with store.tx(conn):
|
||||
store.mark_synced(conn, source.source_id, plan.new_rev, _now())
|
||||
log(f"{source.source_id}: rev moved to {plan.new_rev}, no indexed file changed")
|
||||
continue
|
||||
|
||||
counts = apply_plan(conn, source, root, plan)
|
||||
log(
|
||||
f"{source.source_id}: indexed {counts['indexed']} files "
|
||||
f"({counts['chunks']} chunks), deleted {counts['deleted']}"
|
||||
)
|
||||
|
||||
embedded, embed_warning = drain_pending(conn, config)
|
||||
if embed_warning:
|
||||
warnings.append(embed_warning)
|
||||
log(f"WARN {embed_warning}")
|
||||
|
||||
if not did_work and not embedded and not warnings:
|
||||
return 0
|
||||
|
||||
for source in sources:
|
||||
uncovered = uncovered_top_level_dirs(source, source_root(source))
|
||||
if uncovered:
|
||||
log(f"coverage {source.source_id}: markdown outside paths in {', '.join(uncovered)}")
|
||||
|
||||
stats = store.index_stats(conn)
|
||||
log(
|
||||
f"done files={stats['files']} chunks={stats['chunks']} vectors={stats['vectors']} "
|
||||
f"pending={stats['pending']} embedded={embedded}"
|
||||
)
|
||||
return 1 if warnings else 0
|
||||
|
||||
|
||||
def parse_args(argv: list[str] | None = None) -> argparse.Namespace:
|
||||
parser = argparse.ArgumentParser(description="Index the wiki sources into wiki/index.sqlite.")
|
||||
parser.add_argument(
|
||||
"--full",
|
||||
action="store_true",
|
||||
help="wipe and rebuild the index (needed after a model or chunker change)",
|
||||
)
|
||||
parser.add_argument("--source", help="limit the run to one source id")
|
||||
return parser.parse_args(argv)
|
||||
|
||||
|
||||
def main(argv: list[str] | None = None) -> int:
|
||||
args = parse_args(argv)
|
||||
try:
|
||||
config = load_config()
|
||||
except ConfigError as exc:
|
||||
log(f"WARN {exc}")
|
||||
print(f"config error: {exc}", file=sys.stderr)
|
||||
return 1
|
||||
|
||||
WIKI_DIR.mkdir(parents=True, exist_ok=True)
|
||||
if not acquire_lock():
|
||||
return 0
|
||||
try:
|
||||
if args.full:
|
||||
log("full reindex requested")
|
||||
with store.transaction(db_path()) as conn:
|
||||
store.reset_index(conn)
|
||||
return run(config, args)
|
||||
finally:
|
||||
release_lock()
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
sys.exit(main())
|
||||
141
skills/wiki/tests/conftest.py
Normal file
141
skills/wiki/tests/conftest.py
Normal file
@@ -0,0 +1,141 @@
|
||||
import hashlib
|
||||
import math
|
||||
import sys
|
||||
from dataclasses import dataclass
|
||||
from pathlib import Path
|
||||
|
||||
import pytest
|
||||
|
||||
# The modules under test live in the sibling scripts/ directory.
|
||||
sys.path.insert(0, str(Path(__file__).parent.parent / "scripts"))
|
||||
|
||||
import wiki_config
|
||||
import wiki_search
|
||||
import wiki_sync
|
||||
from wiki_db import EMBEDDING_DIMS
|
||||
|
||||
CONFIG_TEMPLATE = """
|
||||
embedding:
|
||||
endpoint: http://embed.invalid:11434
|
||||
model: qwen3-embedding:0.6b
|
||||
dims: 1024
|
||||
batch: 4
|
||||
keep_alive: -1
|
||||
query_prefix: "Instruct: task\\nQuery: "
|
||||
|
||||
sources:
|
||||
{sources}
|
||||
"""
|
||||
|
||||
WORKSPACE_SOURCE = """ workspace:
|
||||
kind: workspace
|
||||
paths:
|
||||
- "notes/**"
|
||||
- "develop/**"
|
||||
include: ["*.md"]
|
||||
exclude:
|
||||
- "**/inbox/**"
|
||||
- "develop/history.md"
|
||||
"""
|
||||
|
||||
|
||||
@dataclass
|
||||
class WikiEnv:
|
||||
workspace: Path
|
||||
wiki_dir: Path
|
||||
config_path: Path
|
||||
db_path: Path
|
||||
log_path: Path
|
||||
|
||||
def write_config(self, sources: str = WORKSPACE_SOURCE) -> None:
|
||||
self.config_path.write_text(CONFIG_TEMPLATE.format(sources=sources), encoding="utf-8")
|
||||
|
||||
def write_file(self, rel_path: str, text: str) -> Path:
|
||||
path = self.workspace / rel_path
|
||||
path.parent.mkdir(parents=True, exist_ok=True)
|
||||
path.write_text(text, encoding="utf-8")
|
||||
return path
|
||||
|
||||
def log_text(self) -> str:
|
||||
return self.log_path.read_text(encoding="utf-8") if self.log_path.exists() else ""
|
||||
|
||||
|
||||
@pytest.fixture
|
||||
def wiki_env(tmp_path, monkeypatch) -> WikiEnv:
|
||||
"""Redirect every wiki path at a throwaway workspace."""
|
||||
workspace = tmp_path / "workspace"
|
||||
wiki_dir = workspace / "wiki"
|
||||
wiki_dir.mkdir(parents=True)
|
||||
|
||||
env = WikiEnv(
|
||||
workspace=workspace,
|
||||
wiki_dir=wiki_dir,
|
||||
config_path=wiki_dir / "config.yaml",
|
||||
db_path=wiki_dir / "index.sqlite",
|
||||
log_path=workspace / "log" / "wiki_sync.log",
|
||||
)
|
||||
|
||||
monkeypatch.setenv("WIKI_DB", str(env.db_path))
|
||||
monkeypatch.setenv("WIKI_CONFIG", str(env.config_path))
|
||||
monkeypatch.setattr(wiki_config, "WORKSPACE", workspace)
|
||||
monkeypatch.setattr(wiki_config, "WIKI_DIR", wiki_dir)
|
||||
monkeypatch.setattr(wiki_config, "REMOTE_DIR", wiki_dir / "remote")
|
||||
monkeypatch.setattr(wiki_config, "LOCK_PATH", wiki_dir / ".sync.lock")
|
||||
monkeypatch.setattr(wiki_sync, "WIKI_DIR", wiki_dir)
|
||||
monkeypatch.setattr(wiki_sync, "LOCK_PATH", wiki_dir / ".sync.lock")
|
||||
monkeypatch.setattr(wiki_sync, "SYNC_LOG_PATH", env.log_path)
|
||||
|
||||
env.write_config()
|
||||
return env
|
||||
|
||||
|
||||
class FakeEmbedder:
|
||||
"""Bag-of-words vectors: cosine tracks lexical overlap, so ranks are predictable.
|
||||
|
||||
Enough to exercise the KNN and RRF plumbing without a live model. Semantic quality
|
||||
is measured against the real endpoint, not here.
|
||||
"""
|
||||
|
||||
def __init__(self, config):
|
||||
self.config = config
|
||||
self.calls: list[list[str]] = []
|
||||
|
||||
def embed_documents(self, texts):
|
||||
self.calls.append(list(texts))
|
||||
return [self._vector(text) for text in texts]
|
||||
|
||||
def embed_query(self, text):
|
||||
return self._vector(self.config.query_prefix + text)
|
||||
|
||||
def probe(self):
|
||||
return None
|
||||
|
||||
def _vector(self, text: str) -> list[float]:
|
||||
vector = [0.0] * EMBEDDING_DIMS
|
||||
for word in _words(text):
|
||||
digest = hashlib.sha256(word.encode("utf-8")).digest()
|
||||
vector[int.from_bytes(digest[:4], "big") % EMBEDDING_DIMS] += 1.0
|
||||
norm = math.sqrt(sum(value * value for value in vector))
|
||||
if norm == 0.0:
|
||||
vector[0] = 1.0
|
||||
return vector
|
||||
return [value / norm for value in vector]
|
||||
|
||||
|
||||
def _words(text: str) -> list[str]:
|
||||
return [word for word in "".join(c.lower() if c.isalnum() else " " for c in text).split() if len(word) > 2]
|
||||
|
||||
|
||||
@pytest.fixture
|
||||
def fake_embedder(monkeypatch):
|
||||
"""Swap the Ollama client for the deterministic fake in both entry points."""
|
||||
created: list[FakeEmbedder] = []
|
||||
|
||||
def factory(config):
|
||||
embedder = FakeEmbedder(config)
|
||||
created.append(embedder)
|
||||
return embedder
|
||||
|
||||
monkeypatch.setattr(wiki_sync, "OllamaEmbedder", factory)
|
||||
monkeypatch.setattr(wiki_search, "OllamaEmbedder", factory)
|
||||
return created
|
||||
209
skills/wiki/tests/test_wiki_chunker.py
Normal file
209
skills/wiki/tests/test_wiki_chunker.py
Normal file
@@ -0,0 +1,209 @@
|
||||
"""Chunker policy: heading sections, breadcrumbs, sibling merge, block-aligned split."""
|
||||
|
||||
import sys
|
||||
from pathlib import Path
|
||||
|
||||
SCRIPTS = Path(__file__).parent.parent / "scripts"
|
||||
sys.path.insert(0, str(SCRIPTS))
|
||||
|
||||
from wiki_chunker import MERGE_BELOW, OVERLAP, SPLIT_ABOVE, parse_markdown, token_estimate
|
||||
|
||||
FILLER = "Tohle je odstavec s dostatkem textu na to, aby se do velikosti chunku počítal. "
|
||||
|
||||
|
||||
def _paragraph(tokens: int) -> str:
|
||||
return (FILLER * (1 + tokens * 4 // len(FILLER)))[: tokens * 4]
|
||||
|
||||
|
||||
def _crumbs(parsed):
|
||||
return [chunk.breadcrumb for chunk in parsed.chunks]
|
||||
|
||||
|
||||
def test_heading_hierarchy_builds_breadcrumbs():
|
||||
doc = "\n".join(
|
||||
[
|
||||
"# Kořen",
|
||||
"",
|
||||
_paragraph(MERGE_BELOW),
|
||||
"",
|
||||
"## Sekce A",
|
||||
"",
|
||||
_paragraph(MERGE_BELOW),
|
||||
"",
|
||||
"### Podsekce",
|
||||
"",
|
||||
_paragraph(MERGE_BELOW),
|
||||
"",
|
||||
"## Sekce B",
|
||||
"",
|
||||
_paragraph(MERGE_BELOW),
|
||||
]
|
||||
)
|
||||
parsed = parse_markdown(doc, "notes/doc.md")
|
||||
assert _crumbs(parsed) == [
|
||||
"notes/doc.md > Kořen",
|
||||
"notes/doc.md > Kořen > Sekce A",
|
||||
"notes/doc.md > Kořen > Sekce A > Podsekce",
|
||||
"notes/doc.md > Kořen > Sekce B",
|
||||
]
|
||||
assert parsed.headings == ["Kořen", "Sekce A", "Podsekce", "Sekce B"]
|
||||
|
||||
|
||||
def test_frontmatter_title_and_tags():
|
||||
doc = "---\ntitle: Tokyo metro tips\ntags: [transit, jr-pass]\n---\n\n" + _paragraph(MERGE_BELOW)
|
||||
parsed = parse_markdown(doc, "japan/metro.md")
|
||||
assert parsed.title == "Tokyo metro tips"
|
||||
assert parsed.tags == ["transit", "jr-pass"]
|
||||
# Only the title reaches the embedded text; tags stay metadata for filtering.
|
||||
assert _crumbs(parsed) == ["japan/metro.md > Tokyo metro tips"]
|
||||
assert "transit" not in parsed.chunks[0].text
|
||||
|
||||
|
||||
def test_frontmatter_title_matching_h1_is_not_duplicated():
|
||||
doc = "---\ntitle: Zálohy\n---\n\n# Zálohy\n\n" + _paragraph(MERGE_BELOW)
|
||||
parsed = parse_markdown(doc, "a.md")
|
||||
assert _crumbs(parsed) == ["a.md > Zálohy"]
|
||||
|
||||
|
||||
def test_comma_separated_tags():
|
||||
doc = "---\ntags: gear, safety\n---\n\ntext"
|
||||
assert parse_markdown(doc, "a.md").tags == ["gear", "safety"]
|
||||
|
||||
|
||||
def test_malformed_frontmatter_stays_body():
|
||||
"""Unparseable frontmatter is not consumed — it stays body text, tags included."""
|
||||
doc = "---\ntitle: [unclosed\ntags: [a\n---\n\ntext"
|
||||
parsed = parse_markdown(doc, "a.md")
|
||||
assert parsed.tags == []
|
||||
assert "title: [unclosed" in parsed.chunks[-1].text # the raw block survived as prose
|
||||
|
||||
|
||||
def test_small_adjacent_siblings_merge():
|
||||
doc = "\n".join(["# Root", "", "## A", "", "krátké A", "", "## B", "", "krátké B"])
|
||||
parsed = parse_markdown(doc, "a.md")
|
||||
# A and B share the parent `a.md > Root` and are both under MERGE_BELOW.
|
||||
# `# Root` carries no prose of its own, so it contributes no chunk.
|
||||
assert _crumbs(parsed) == ["a.md > Root > A"]
|
||||
assert "krátké A" in parsed.chunks[0].text
|
||||
assert "krátké B" in parsed.chunks[0].text
|
||||
|
||||
|
||||
def test_heading_without_own_prose_yields_no_chunk():
|
||||
"""`# Titul` immediately followed by `## Sekce` must not embed a bare title."""
|
||||
doc = "\n".join(["# Titul", "", "## Sekce", "", _paragraph(MERGE_BELOW)])
|
||||
parsed = parse_markdown(doc, "a.md")
|
||||
assert _crumbs(parsed) == ["a.md > Titul > Sekce"]
|
||||
# The dropped heading still reaches the index through the breadcrumb and headings list.
|
||||
assert parsed.headings == ["Titul", "Sekce"]
|
||||
|
||||
|
||||
def test_merge_does_not_cross_parents():
|
||||
"""A small H3 must not be glued onto the next H2 — different parents."""
|
||||
doc = "\n".join(["# Root", "", "## A", "", "### A1", "", "malé", "", "## B", "", "malé B"])
|
||||
parsed = parse_markdown(doc, "a.md")
|
||||
assert "a.md > Root > A > A1" in _crumbs(parsed)
|
||||
a1 = next(c for c in parsed.chunks if c.breadcrumb.endswith("A1"))
|
||||
assert "malé B" not in a1.text
|
||||
|
||||
|
||||
def test_merge_never_exceeds_split_threshold():
|
||||
big = _paragraph(SPLIT_ABOVE - 100)
|
||||
doc = "\n".join(["# Root", "", "## A", "", "malé", "", "## B", "", big])
|
||||
parsed = parse_markdown(doc, "a.md")
|
||||
for chunk in parsed.chunks:
|
||||
assert token_estimate(chunk.text) <= SPLIT_ABOVE + token_estimate(chunk.breadcrumb) + 1
|
||||
|
||||
|
||||
def test_h4_stays_inside_its_h3_section():
|
||||
doc = "\n".join(["# Root", "", "### Trojka", "", "text", "", "#### Čtyřka", "", "hluboký text"])
|
||||
parsed = parse_markdown(doc, "a.md")
|
||||
assert "a.md > Root > Trojka" in _crumbs(parsed)
|
||||
assert not any("Čtyřka" in crumb for crumb in _crumbs(parsed))
|
||||
section = next(c for c in parsed.chunks if c.breadcrumb.endswith("Trojka"))
|
||||
assert "hluboký text" in section.text
|
||||
|
||||
|
||||
def test_preamble_before_first_heading_is_kept():
|
||||
doc = "úvodní odstavec bez nadpisu\n\n" + "# Root\n\n" + _paragraph(MERGE_BELOW)
|
||||
parsed = parse_markdown(doc, "a.md")
|
||||
assert "úvodní odstavec bez nadpisu" in parsed.chunks[0].text
|
||||
|
||||
|
||||
def test_document_without_headings_is_one_chunk():
|
||||
parsed = parse_markdown(_paragraph(MERGE_BELOW), "a.md")
|
||||
assert _crumbs(parsed) == ["a.md"]
|
||||
|
||||
|
||||
def test_empty_document_yields_no_chunks():
|
||||
assert parse_markdown("", "a.md").chunks == []
|
||||
assert parse_markdown(" \n\n \n", "a.md").chunks == []
|
||||
|
||||
|
||||
def test_setext_heading_is_a_section():
|
||||
doc = "Nadpis setext\n=============\n\n" + _paragraph(MERGE_BELOW)
|
||||
parsed = parse_markdown(doc, "a.md")
|
||||
assert _crumbs(parsed) == ["a.md > Nadpis setext"]
|
||||
|
||||
|
||||
def test_oversized_section_splits_and_keeps_code_fence_whole():
|
||||
fence = "```python\n" + "\n".join(f"value_{i} = {i} # a comment long enough to count" for i in range(60)) + "\n```"
|
||||
doc = "# Velká sekce\n\n" + "\n\n".join([_paragraph(120)] * 6) + "\n\n" + fence + "\n\n" + _paragraph(120)
|
||||
parsed = parse_markdown(doc, "big.md")
|
||||
|
||||
assert len(parsed.chunks) > 1
|
||||
assert {c.breadcrumb for c in parsed.chunks} == {"big.md > Velká sekce"}
|
||||
# The fence is larger than SPLIT_ABOVE on its own, so it must sit alone and unbroken.
|
||||
assert sum(fence in chunk.text for chunk in parsed.chunks) == 1
|
||||
|
||||
|
||||
def test_split_carries_block_aligned_overlap():
|
||||
blocks = [_paragraph(40) + f" marker{i:03d}" for i in range(40)]
|
||||
doc = "# Root\n\n" + "\n\n".join(blocks)
|
||||
parsed = parse_markdown(doc, "a.md")
|
||||
|
||||
assert len(parsed.chunks) > 1
|
||||
# Each boundary repeats at least one whole block, capped at OVERLAP tokens.
|
||||
for earlier, later in zip(parsed.chunks, parsed.chunks[1:], strict=False):
|
||||
shared = [b for b in blocks if b in earlier.text and b in later.text]
|
||||
assert shared, "expected overlap blocks between consecutive chunks"
|
||||
assert token_estimate("".join(shared)) <= OVERLAP
|
||||
|
||||
|
||||
def test_table_is_not_broken():
|
||||
table = "\n".join(["| a | b |", "|---|---|"] + [f"| {i} | {i * 2} |" for i in range(80)])
|
||||
doc = "# Root\n\n" + "\n\n".join([_paragraph(150)] * 5) + "\n\n" + table
|
||||
parsed = parse_markdown(doc, "a.md")
|
||||
assert sum(table in chunk.text for chunk in parsed.chunks) == 1
|
||||
|
||||
|
||||
def test_breadcrumb_is_prefixed_to_chunk_text():
|
||||
parsed = parse_markdown("# Root\n\n" + _paragraph(MERGE_BELOW), "notes/a.md")
|
||||
chunk = parsed.chunks[0]
|
||||
assert chunk.text.startswith(chunk.breadcrumb + "\n\n")
|
||||
|
||||
|
||||
def test_catalog_title_falls_back_to_first_heading():
|
||||
"""Notes carry their title as `# H1` far more often than as frontmatter."""
|
||||
doc = "# Zálohování dat\n\n" + _paragraph(MERGE_BELOW) + "\n\n## Retence\n\n" + _paragraph(MERGE_BELOW)
|
||||
parsed = parse_markdown(doc, "notes/zalohy.md")
|
||||
assert parsed.title == "Zálohování dat"
|
||||
|
||||
|
||||
def test_frontmatter_title_wins_over_first_heading():
|
||||
doc = "---\ntitle: Z frontmatteru\n---\n\n# Z nadpisu\n\n" + _paragraph(MERGE_BELOW)
|
||||
assert parse_markdown(doc, "a.md").title == "Z frontmatteru"
|
||||
|
||||
|
||||
def test_document_without_headings_has_no_title():
|
||||
assert parse_markdown(_paragraph(MERGE_BELOW), "a.md").title is None
|
||||
assert parse_markdown("", "a.md").title is None
|
||||
|
||||
|
||||
def test_heading_title_fallback_does_not_change_any_chunk():
|
||||
"""The fallback feeds only the catalog — touching the breadcrumb root would force --full."""
|
||||
doc = "úvod bez nadpisu\n\n# Root\n\n" + _paragraph(MERGE_BELOW) + "\n\n## Sekce\n\n" + _paragraph(MERGE_BELOW)
|
||||
parsed = parse_markdown(doc, "notes/a.md")
|
||||
|
||||
assert parsed.title == "Root"
|
||||
# The root stays the bare path, so no chunk text is rewritten by the fallback.
|
||||
assert _crumbs(parsed) == ["notes/a.md", "notes/a.md > Root", "notes/a.md > Root > Sekce"]
|
||||
165
skills/wiki/tests/test_wiki_config.py
Normal file
165
skills/wiki/tests/test_wiki_config.py
Normal file
@@ -0,0 +1,165 @@
|
||||
"""Scope precedence (paths -> include -> exclude) and config validation."""
|
||||
|
||||
import sys
|
||||
from pathlib import Path
|
||||
|
||||
import pytest
|
||||
|
||||
SCRIPTS = Path(__file__).parent.parent / "scripts"
|
||||
sys.path.insert(0, str(SCRIPTS))
|
||||
|
||||
import wiki_config
|
||||
from wiki_config import ConfigError, SourceConfig, load_config, source_root
|
||||
|
||||
WORKSPACE_YAML = """
|
||||
embedding:
|
||||
endpoint: http://nvidia.hell:11434/
|
||||
model: qwen3-embedding:0.6b
|
||||
dims: 1024
|
||||
batch: 32
|
||||
keep_alive: -1
|
||||
query_prefix: "Instruct: task\\nQuery: "
|
||||
|
||||
sources:
|
||||
index:
|
||||
kind: git
|
||||
url: git@git.fnet.cz:lachtan/index.git
|
||||
paths: ["**"]
|
||||
include: ["*.md"]
|
||||
exclude:
|
||||
- "**/node_modules/**"
|
||||
- "**/vendor/**"
|
||||
|
||||
workspace:
|
||||
kind: workspace
|
||||
paths:
|
||||
- "notes/**"
|
||||
- "develop/**"
|
||||
include: ["*.md"]
|
||||
exclude:
|
||||
- "**/inbox/**"
|
||||
- "develop/history.md"
|
||||
"""
|
||||
|
||||
|
||||
def _write(tmp_path, text):
|
||||
path = tmp_path / "config.yaml"
|
||||
path.write_text(text, encoding="utf-8")
|
||||
return path
|
||||
|
||||
|
||||
def _source(
|
||||
source_id: str = "ws",
|
||||
kind: str = "workspace",
|
||||
url: str | None = None,
|
||||
paths: tuple[str, ...] = ("**",),
|
||||
include: tuple[str, ...] = ("*.md",),
|
||||
exclude: tuple[str, ...] = (),
|
||||
) -> SourceConfig:
|
||||
return SourceConfig(source_id=source_id, kind=kind, url=url, paths=paths, include=include, exclude=exclude)
|
||||
|
||||
|
||||
def test_loads_embedding_and_sources(tmp_path):
|
||||
config = load_config(_write(tmp_path, WORKSPACE_YAML))
|
||||
assert config.embedding.endpoint == "http://nvidia.hell:11434" # trailing slash trimmed
|
||||
assert config.embedding.model == "qwen3-embedding:0.6b"
|
||||
assert config.embedding.dims == 1024
|
||||
assert config.embedding.keep_alive == -1
|
||||
assert config.embedding.query_prefix.endswith("Query: ")
|
||||
assert [s.source_id for s in config.sources] == ["index", "workspace"]
|
||||
index = config.source("index")
|
||||
assert index is not None and index.kind == "git"
|
||||
assert config.source("nope") is None
|
||||
|
||||
|
||||
def test_string_keep_alive_is_rejected():
|
||||
"""Ollama answers HTTP 400 to a string "-1" — catch it in config, not at runtime."""
|
||||
with pytest.raises(ConfigError, match="keep_alive"):
|
||||
wiki_config._parse_embedding({"endpoint": "http://x", "model": "m", "dims": 1024, "keep_alive": "-1"})
|
||||
|
||||
|
||||
def test_missing_config_file(tmp_path):
|
||||
with pytest.raises(ConfigError, match="missing config"):
|
||||
load_config(tmp_path / "nope.yaml")
|
||||
|
||||
|
||||
def test_git_source_needs_url(tmp_path):
|
||||
yaml_text = WORKSPACE_YAML.replace(" url: git@git.fnet.cz:lachtan/index.git\n", "")
|
||||
with pytest.raises(ConfigError, match="need a url"):
|
||||
load_config(_write(tmp_path, yaml_text))
|
||||
|
||||
|
||||
def test_unknown_kind_is_rejected(tmp_path):
|
||||
yaml_text = WORKSPACE_YAML.replace(" kind: workspace", " kind: mirror")
|
||||
with pytest.raises(ConfigError, match="kind must be one of"):
|
||||
load_config(_write(tmp_path, yaml_text))
|
||||
|
||||
|
||||
def test_paths_is_required(tmp_path):
|
||||
yaml_text = WORKSPACE_YAML.replace(' paths: ["**"]\n', "")
|
||||
with pytest.raises(ConfigError, match="`paths` is required"):
|
||||
load_config(_write(tmp_path, yaml_text))
|
||||
|
||||
|
||||
def test_paths_whitelist_gates_everything():
|
||||
source = _source(paths=("notes/**", "develop/**"))
|
||||
assert source.covers("notes/a.md")
|
||||
assert source.covers("notes/deep/nested/a.md")
|
||||
assert source.covers("develop/knowledge.md")
|
||||
# Not on the whitelist -> does not exist for the index.
|
||||
assert not source.covers("tmp/a.md")
|
||||
assert not source.covers("skills/wiki/SKILL.md")
|
||||
assert not source.covers("AGENTS.md")
|
||||
|
||||
|
||||
def test_include_filters_extensions():
|
||||
source = _source(paths=("**",))
|
||||
assert source.covers("notes/a.md")
|
||||
assert not source.covers("notes/main.py")
|
||||
assert not source.covers("memory/history.jsonl")
|
||||
assert not source.covers("assets/photo.png")
|
||||
|
||||
|
||||
def test_exclude_wins_over_paths_and_include():
|
||||
source = _source(paths=("notes/**", "develop/**"), exclude=("**/inbox/**", "develop/history.md"))
|
||||
assert not source.covers("notes/inbox/raw.md")
|
||||
assert not source.covers("develop/history.md")
|
||||
assert source.covers("develop/knowledge.md")
|
||||
assert source.covers("notes/notes.md")
|
||||
|
||||
|
||||
def test_double_star_matches_whole_repo():
|
||||
source = _source(paths=("**",), exclude=("**/node_modules/**", "**/vendor/**"))
|
||||
assert source.covers("README.md")
|
||||
assert source.covers("japan/tokyo/metro.md")
|
||||
assert not source.covers("node_modules/pkg/README.md")
|
||||
assert not source.covers("web/vendor/lib/CHANGELOG.md")
|
||||
|
||||
|
||||
def test_glob_star_does_not_cross_a_slash():
|
||||
source = _source(paths=("notes/*",))
|
||||
assert source.covers("notes/a.md")
|
||||
assert not source.covers("notes/deep/a.md")
|
||||
|
||||
|
||||
def test_covers_dir_prunes_the_walk():
|
||||
source = _source(paths=("notes/**", "develop/**"), exclude=("**/inbox/**",))
|
||||
assert source.covers_dir("")
|
||||
assert source.covers_dir("notes")
|
||||
assert source.covers_dir("notes/deep")
|
||||
assert source.covers_dir("develop")
|
||||
assert not source.covers_dir("tmp")
|
||||
assert not source.covers_dir("notes/inbox")
|
||||
|
||||
|
||||
def test_covers_dir_never_prunes_a_double_star_source():
|
||||
source = _source(paths=("**",), exclude=("**/node_modules/**",))
|
||||
assert source.covers_dir("anything/deep")
|
||||
assert not source.covers_dir("app/node_modules")
|
||||
|
||||
|
||||
def test_source_root_derives_clone_path_from_id():
|
||||
assert source_root(_source(source_id="travel", kind="git", url="git@x:y.git")) == (
|
||||
wiki_config.REMOTE_DIR / "travel"
|
||||
)
|
||||
assert source_root(_source(kind="workspace")) == wiki_config.WORKSPACE
|
||||
136
skills/wiki/tests/test_wiki_db.py
Normal file
136
skills/wiki/tests/test_wiki_db.py
Normal file
@@ -0,0 +1,136 @@
|
||||
"""Schema-level guarantees: the cascade, and the one hole the cascade leaves."""
|
||||
|
||||
import sys
|
||||
from pathlib import Path
|
||||
|
||||
SCRIPTS = Path(__file__).parent.parent / "scripts"
|
||||
sys.path.insert(0, str(SCRIPTS))
|
||||
|
||||
import wiki_store as store
|
||||
from wiki_db import EMBEDDING_DIMS
|
||||
|
||||
NOW = "2026-09-09T12:00:00+00:00"
|
||||
|
||||
|
||||
def _seed_file(conn, source_id, path, chunk_texts):
|
||||
store.upsert_source(conn, source_id, "workspace")
|
||||
store.upsert_file(conn, source_id, path, "T", ["t"], ["H"], "sha", 10, 1.0, NOW)
|
||||
store.replace_chunks(conn, source_id, path, [(f"{path} > s", t) for t in chunk_texts])
|
||||
for row in store.pending_chunks(conn, 100):
|
||||
store.store_embedding(conn, row["id"], [0.1] * EMBEDDING_DIMS, NOW)
|
||||
|
||||
|
||||
def test_delete_file_leaves_no_orphan_vectors(tmp_path):
|
||||
"""The FK cascade reaches chunks and chunks_fts but never vec0 — deletes must be explicit."""
|
||||
db_path = tmp_path / "index.sqlite"
|
||||
with store.transaction(db_path) as conn:
|
||||
_seed_file(conn, "ws", "a.md", ["alpha text", "beta text"])
|
||||
_seed_file(conn, "ws", "b.md", ["gamma text"])
|
||||
|
||||
with store.connection(db_path) as conn:
|
||||
assert store.index_stats(conn) == {"files": 2, "chunks": 3, "vectors": 3, "pending": 0}
|
||||
|
||||
with store.transaction(db_path) as conn:
|
||||
store.delete_file(conn, "ws", "a.md")
|
||||
|
||||
with store.connection(db_path) as conn:
|
||||
assert store.index_stats(conn) == {"files": 1, "chunks": 1, "vectors": 1, "pending": 0}
|
||||
assert store.orphan_vector_ids(conn) == []
|
||||
assert store.bm25_ranked_ids(conn, "alpha", 10) == []
|
||||
assert len(store.bm25_ranked_ids(conn, "gamma", 10)) == 1
|
||||
|
||||
|
||||
def test_replace_chunks_drops_old_vectors(tmp_path):
|
||||
db_path = tmp_path / "index.sqlite"
|
||||
with store.transaction(db_path) as conn:
|
||||
_seed_file(conn, "ws", "a.md", ["alpha text", "beta text", "delta text"])
|
||||
|
||||
with store.transaction(db_path) as conn:
|
||||
store.replace_chunks(conn, "ws", "a.md", [("a.md > s", "only one now")])
|
||||
|
||||
with store.connection(db_path) as conn:
|
||||
assert store.index_stats(conn) == {"files": 1, "chunks": 1, "vectors": 0, "pending": 1}
|
||||
assert store.orphan_vector_ids(conn) == []
|
||||
assert store.bm25_ranked_ids(conn, "alpha", 10) == []
|
||||
|
||||
|
||||
def test_delete_by_path_does_not_touch_other_source(tmp_path):
|
||||
"""The key is (source_id, path); a path-only delete would eat a foreign source's chunks."""
|
||||
db_path = tmp_path / "index.sqlite"
|
||||
with store.transaction(db_path) as conn:
|
||||
_seed_file(conn, "ws", "notes.md", ["shared name one"])
|
||||
_seed_file(conn, "git", "notes.md", ["shared name two"])
|
||||
|
||||
with store.transaction(db_path) as conn:
|
||||
store.delete_file(conn, "ws", "notes.md")
|
||||
|
||||
with store.connection(db_path) as conn:
|
||||
assert store.index_stats(conn)["chunks"] == 1
|
||||
assert store.orphan_vector_ids(conn) == []
|
||||
remaining = list(conn.execute("SELECT source_id FROM chunks"))
|
||||
assert remaining[0]["source_id"] == "git"
|
||||
|
||||
|
||||
def test_embedded_at_update_keeps_fts_row(tmp_path):
|
||||
"""Step 4b flips embedded_at on unchanged text; the WHEN guard must keep FTS intact."""
|
||||
db_path = tmp_path / "index.sqlite"
|
||||
with store.transaction(db_path) as conn:
|
||||
store.upsert_source(conn, "ws", "workspace")
|
||||
store.upsert_file(conn, "ws", "a.md", "T", [], [], "sha", 10, 1.0, NOW)
|
||||
store.replace_chunks(conn, "ws", "a.md", [("a.md", "zaloha dat na disk")])
|
||||
|
||||
with store.connection(db_path) as conn:
|
||||
assert len(store.bm25_ranked_ids(conn, "zaloha", 10)) == 1
|
||||
|
||||
with store.transaction(db_path) as conn:
|
||||
pending = store.pending_chunks(conn, 10)
|
||||
store.store_embedding(conn, pending[0]["id"], [0.2] * EMBEDDING_DIMS, NOW)
|
||||
|
||||
with store.connection(db_path) as conn:
|
||||
assert len(store.bm25_ranked_ids(conn, "zaloha", 10)) == 1
|
||||
assert store.index_stats(conn) == {"files": 1, "chunks": 1, "vectors": 1, "pending": 0}
|
||||
|
||||
|
||||
def test_fts_folds_czech_diacritics(tmp_path):
|
||||
"""remove_diacritics 2 is what makes `zaloha` find `záloha`."""
|
||||
db_path = tmp_path / "index.sqlite"
|
||||
with store.transaction(db_path) as conn:
|
||||
store.upsert_source(conn, "ws", "workspace")
|
||||
store.upsert_file(conn, "ws", "a.md", "T", [], [], "sha", 10, 1.0, NOW)
|
||||
store.replace_chunks(conn, "ws", "a.md", [("a.md", "zálohování dat probíhá denně")])
|
||||
|
||||
with store.connection(db_path) as conn:
|
||||
assert len(store.bm25_ranked_ids(conn, "zaloh*", 10)) == 1
|
||||
assert len(store.bm25_ranked_ids(conn, "záloh*", 10)) == 1
|
||||
|
||||
|
||||
def test_meta_roundtrip_and_rollback(tmp_path):
|
||||
db_path = tmp_path / "index.sqlite"
|
||||
with store.transaction(db_path) as conn:
|
||||
store.write_meta(conn, {"embedding_model": "qwen3-embedding:0.6b", "embedding_dims": "1024"})
|
||||
|
||||
with store.connection(db_path) as conn:
|
||||
assert store.read_meta(conn)["embedding_dims"] == "1024"
|
||||
|
||||
try:
|
||||
with store.transaction(db_path) as conn:
|
||||
store.write_meta(conn, {"embedding_dims": "768"})
|
||||
raise RuntimeError("boom")
|
||||
except RuntimeError:
|
||||
pass
|
||||
|
||||
with store.connection(db_path) as conn:
|
||||
assert store.read_meta(conn)["embedding_dims"] == "1024"
|
||||
|
||||
|
||||
def test_toc_filters_by_tag(tmp_path):
|
||||
db_path = tmp_path / "index.sqlite"
|
||||
with store.transaction(db_path) as conn:
|
||||
store.upsert_source(conn, "travel", "git")
|
||||
store.upsert_file(conn, "travel", "japan/metro.md", "Metro", ["transit"], [], "s", 1, 1.0, NOW)
|
||||
store.upsert_file(conn, "travel", "alps/gear.md", "Gear", ["gear", "safety"], [], "s", 1, 1.0, NOW)
|
||||
|
||||
with store.connection(db_path) as conn:
|
||||
assert [r["path"] for r in store.list_toc_files(conn, tag="gear")] == ["alps/gear.md"]
|
||||
assert len(store.list_toc_files(conn, source_id="travel")) == 2
|
||||
assert store.list_toc_files(conn, source_id="nope") == []
|
||||
266
skills/wiki/tests/test_wiki_git.py
Normal file
266
skills/wiki/tests/test_wiki_git.py
Normal file
@@ -0,0 +1,266 @@
|
||||
"""Git driver: clone, ls-remote change detection, diff, deletions, per-source failure."""
|
||||
|
||||
import subprocess
|
||||
import sys
|
||||
from pathlib import Path
|
||||
|
||||
SCRIPTS = Path(__file__).parent.parent / "scripts"
|
||||
sys.path.insert(0, str(SCRIPTS))
|
||||
|
||||
import wiki_store as store
|
||||
import wiki_sync
|
||||
from conftest import WORKSPACE_SOURCE
|
||||
|
||||
DOC = """# Zálohování
|
||||
|
||||
Záloha běží každou noc přes rsync na druhý disk.
|
||||
"""
|
||||
|
||||
TRAVEL_DOC = """# Tokijské metro
|
||||
|
||||
Z Narity do centra jede Skyliner za osmatřicet minut.
|
||||
"""
|
||||
|
||||
GIT_SOURCE = """ notes:
|
||||
kind: git
|
||||
url: {url}
|
||||
paths: ["**"]
|
||||
include: ["*.md"]
|
||||
exclude:
|
||||
- "**/node_modules/**"
|
||||
- "**/vendor/**"
|
||||
"""
|
||||
|
||||
|
||||
def _git(args, cwd=None):
|
||||
result = subprocess.run(
|
||||
["git", "-c", "user.email=t@t", "-c", "user.name=t", *args],
|
||||
cwd=cwd,
|
||||
capture_output=True,
|
||||
text=True,
|
||||
check=True,
|
||||
)
|
||||
return result.stdout
|
||||
|
||||
|
||||
class Remote:
|
||||
"""A bare repo plus a working clone to push commits from."""
|
||||
|
||||
def __init__(self, root: Path):
|
||||
self.bare = root / "origin.git"
|
||||
self.work = root / "origin-work"
|
||||
_git(["init", "--bare", "--initial-branch=master", str(self.bare)])
|
||||
_git(["clone", str(self.bare), str(self.work)])
|
||||
|
||||
@property
|
||||
def url(self) -> str:
|
||||
return str(self.bare)
|
||||
|
||||
def commit(self, files: dict[str, str | None], message: str = "change") -> str:
|
||||
for rel_path, text in files.items():
|
||||
path = self.work / rel_path
|
||||
if text is None:
|
||||
_git(["rm", "-q", rel_path], cwd=self.work)
|
||||
continue
|
||||
path.parent.mkdir(parents=True, exist_ok=True)
|
||||
path.write_text(text, encoding="utf-8")
|
||||
_git(["add", rel_path], cwd=self.work)
|
||||
_git(["commit", "-m", message], cwd=self.work)
|
||||
_git(["push", "-q", "origin", "master"], cwd=self.work)
|
||||
return _git(["rev-parse", "HEAD"], cwd=self.work).strip()
|
||||
|
||||
def move(self, old: str, new: str) -> str:
|
||||
(self.work / new).parent.mkdir(parents=True, exist_ok=True)
|
||||
_git(["mv", old, new], cwd=self.work)
|
||||
_git(["commit", "-m", "rename"], cwd=self.work)
|
||||
_git(["push", "-q", "origin", "master"], cwd=self.work)
|
||||
return _git(["rev-parse", "HEAD"], cwd=self.work).strip()
|
||||
|
||||
|
||||
def _use_git_source(env, url, with_workspace=False):
|
||||
sources = GIT_SOURCE.format(url=url)
|
||||
if with_workspace:
|
||||
sources += WORKSPACE_SOURCE
|
||||
env.write_config(sources)
|
||||
|
||||
|
||||
def _stats(env):
|
||||
with store.connection(env.db_path) as conn:
|
||||
return store.index_stats(conn)
|
||||
|
||||
|
||||
def test_first_run_clones_and_indexes(wiki_env, fake_embedder, tmp_path):
|
||||
remote = Remote(tmp_path)
|
||||
head = remote.commit({"zalohy.md": DOC, "japan/metro.md": TRAVEL_DOC})
|
||||
_use_git_source(wiki_env, remote.url)
|
||||
|
||||
assert wiki_sync.main([]) == 0
|
||||
|
||||
clone = wiki_env.wiki_dir / "remote" / "notes"
|
||||
assert (clone / "zalohy.md").is_file() # non-bare: a working tree grep can read
|
||||
assert (clone / ".git").is_dir()
|
||||
|
||||
with store.connection(wiki_env.db_path) as conn:
|
||||
assert sorted(store.list_source_files(conn, "notes")) == ["japan/metro.md", "zalohy.md"]
|
||||
source = store.get_source(conn, "notes")
|
||||
assert source is not None
|
||||
assert source["indexed_rev"] == head
|
||||
assert source["kind"] == "git"
|
||||
stats = _stats(wiki_env)
|
||||
assert stats["vectors"] == stats["chunks"] > 0
|
||||
|
||||
|
||||
def test_second_run_over_the_same_revision_changes_nothing(wiki_env, fake_embedder, tmp_path):
|
||||
"""Verification 2 for the git driver."""
|
||||
remote = Remote(tmp_path)
|
||||
remote.commit({"zalohy.md": DOC})
|
||||
_use_git_source(wiki_env, remote.url)
|
||||
assert wiki_sync.main([]) == 0
|
||||
|
||||
before = _stats(wiki_env)
|
||||
log_before = wiki_env.log_text()
|
||||
|
||||
assert wiki_sync.main([]) == 0
|
||||
|
||||
assert _stats(wiki_env) == before
|
||||
assert wiki_env.log_text() == log_before # ls-remote matched, so not even a fetch
|
||||
|
||||
|
||||
def test_new_commit_reindexes_only_the_changed_file(wiki_env, fake_embedder, tmp_path):
|
||||
remote = Remote(tmp_path)
|
||||
remote.commit({"zalohy.md": DOC, "japan/metro.md": TRAVEL_DOC})
|
||||
_use_git_source(wiki_env, remote.url)
|
||||
assert wiki_sync.main([]) == 0
|
||||
|
||||
with store.connection(wiki_env.db_path) as conn:
|
||||
metro_indexed_at = store.list_source_files(conn, "notes")["japan/metro.md"]["indexed_at"]
|
||||
|
||||
head = remote.commit({"zalohy.md": DOC + "\n## Offsite\n\nKopie jede do S3.\n"})
|
||||
assert wiki_sync.main([]) == 0
|
||||
|
||||
with store.connection(wiki_env.db_path) as conn:
|
||||
files = store.list_source_files(conn, "notes")
|
||||
assert files["japan/metro.md"]["indexed_at"] == metro_indexed_at # untouched
|
||||
source = store.get_source(conn, "notes")
|
||||
assert source is not None and source["indexed_rev"] == head
|
||||
assert len(store.bm25_ranked_ids(conn, "offsite", 10)) == 1
|
||||
assert store.orphan_vector_ids(conn) == []
|
||||
|
||||
|
||||
def test_deleted_file_upstream_drops_its_chunks_and_vectors(wiki_env, fake_embedder, tmp_path):
|
||||
remote = Remote(tmp_path)
|
||||
remote.commit({"zalohy.md": DOC, "japan/metro.md": TRAVEL_DOC})
|
||||
_use_git_source(wiki_env, remote.url)
|
||||
assert wiki_sync.main([]) == 0
|
||||
|
||||
remote.commit({"japan/metro.md": None}, message="drop metro")
|
||||
assert wiki_sync.main([]) == 0
|
||||
|
||||
with store.connection(wiki_env.db_path) as conn:
|
||||
assert sorted(store.list_source_files(conn, "notes")) == ["zalohy.md"]
|
||||
assert store.bm25_ranked_ids(conn, "skyliner", 10) == []
|
||||
assert store.orphan_vector_ids(conn) == []
|
||||
stats = _stats(wiki_env)
|
||||
assert stats["vectors"] == stats["chunks"]
|
||||
|
||||
|
||||
def test_renamed_file_moves_its_chunks(wiki_env, fake_embedder, tmp_path):
|
||||
remote = Remote(tmp_path)
|
||||
remote.commit({"japan/metro.md": TRAVEL_DOC})
|
||||
_use_git_source(wiki_env, remote.url)
|
||||
assert wiki_sync.main([]) == 0
|
||||
|
||||
remote.move("japan/metro.md", "japan/tokyo-metro.md")
|
||||
assert wiki_sync.main([]) == 0
|
||||
|
||||
with store.connection(wiki_env.db_path) as conn:
|
||||
assert sorted(store.list_source_files(conn, "notes")) == ["japan/tokyo-metro.md"]
|
||||
assert len(store.bm25_ranked_ids(conn, "skyliner", 10)) == 1
|
||||
assert store.orphan_vector_ids(conn) == []
|
||||
|
||||
|
||||
def test_commit_touching_nothing_indexed_only_records_the_rev(wiki_env, fake_embedder, tmp_path):
|
||||
remote = Remote(tmp_path)
|
||||
remote.commit({"zalohy.md": DOC})
|
||||
_use_git_source(wiki_env, remote.url)
|
||||
assert wiki_sync.main([]) == 0
|
||||
before = _stats(wiki_env)
|
||||
|
||||
head = remote.commit({"tool.py": "print('hello')\n"})
|
||||
assert wiki_sync.main([]) == 0
|
||||
|
||||
assert _stats(wiki_env) == before
|
||||
with store.connection(wiki_env.db_path) as conn:
|
||||
source = store.get_source(conn, "notes")
|
||||
assert source is not None and source["indexed_rev"] == head
|
||||
assert "no indexed file changed" in wiki_env.log_text()
|
||||
|
||||
|
||||
def test_vendored_markdown_is_excluded(wiki_env, fake_embedder, tmp_path):
|
||||
remote = Remote(tmp_path)
|
||||
remote.commit(
|
||||
{
|
||||
"zalohy.md": DOC,
|
||||
"node_modules/pkg/README.md": "# Foreign readme\n\nnenasazovat\n",
|
||||
"web/vendor/lib/CHANGELOG.md": "# Changelog\n\nnenasazovat\n",
|
||||
}
|
||||
)
|
||||
_use_git_source(wiki_env, remote.url)
|
||||
assert wiki_sync.main([]) == 0
|
||||
|
||||
with store.connection(wiki_env.db_path) as conn:
|
||||
assert sorted(store.list_source_files(conn, "notes")) == ["zalohy.md"]
|
||||
|
||||
|
||||
def test_unreachable_remote_warns_skips_and_lets_other_sources_finish(wiki_env, fake_embedder, tmp_path):
|
||||
"""Verification 8: per-source failure, indexed_rev untouched, workspace completes."""
|
||||
_use_git_source(wiki_env, str(tmp_path / "does-not-exist.git"), with_workspace=True)
|
||||
wiki_env.write_file("notes/local.md", DOC)
|
||||
|
||||
assert wiki_sync.main([]) == 1
|
||||
|
||||
log = wiki_env.log_text()
|
||||
assert "WARN notes:" in log
|
||||
# A permanent failure warns every tick, so each warning must stay a single line.
|
||||
assert len([line for line in log.splitlines() if "WARN notes:" in line]) == 1
|
||||
assert all(line.strip() for line in log.splitlines())
|
||||
with store.connection(wiki_env.db_path) as conn:
|
||||
assert store.list_source_files(conn, "notes") == {}
|
||||
git_source = store.get_source(conn, "notes")
|
||||
assert git_source is not None and git_source["indexed_rev"] is None
|
||||
# The workspace source, which has nothing to do with the network, finished.
|
||||
assert sorted(store.list_source_files(conn, "workspace")) == ["notes/local.md"]
|
||||
|
||||
|
||||
def test_remote_that_recovers_is_picked_up_on_the_next_tick(wiki_env, fake_embedder, tmp_path):
|
||||
missing = tmp_path / "later.git"
|
||||
_use_git_source(wiki_env, str(missing))
|
||||
assert wiki_sync.main([]) == 1
|
||||
|
||||
remote = Remote(tmp_path / "real")
|
||||
(tmp_path / "real").mkdir(exist_ok=True)
|
||||
remote_bare = remote.bare
|
||||
remote.commit({"zalohy.md": DOC})
|
||||
_use_git_source(wiki_env, str(remote_bare))
|
||||
|
||||
assert wiki_sync.main([]) == 0
|
||||
with store.connection(wiki_env.db_path) as conn:
|
||||
assert sorted(store.list_source_files(conn, "notes")) == ["zalohy.md"]
|
||||
|
||||
|
||||
def test_missing_indexed_rev_falls_back_to_a_full_reindex(wiki_env, fake_embedder, tmp_path):
|
||||
"""A force-pushed or gc'd base revision must degrade to a full pass, not crash."""
|
||||
remote = Remote(tmp_path)
|
||||
remote.commit({"zalohy.md": DOC})
|
||||
_use_git_source(wiki_env, remote.url)
|
||||
assert wiki_sync.main([]) == 0
|
||||
|
||||
with store.connection(wiki_env.db_path) as conn, store.tx(conn):
|
||||
store.mark_synced(conn, "notes", "0" * 40, "2026-01-01T00:00:00+00:00")
|
||||
|
||||
remote.commit({"japan/metro.md": TRAVEL_DOC})
|
||||
assert wiki_sync.main([]) == 0
|
||||
|
||||
with store.connection(wiki_env.db_path) as conn:
|
||||
assert sorted(store.list_source_files(conn, "notes")) == ["japan/metro.md", "zalohy.md"]
|
||||
assert store.orphan_vector_ids(conn) == []
|
||||
306
skills/wiki/tests/test_wiki_search.py
Normal file
306
skills/wiki/tests/test_wiki_search.py
Normal file
@@ -0,0 +1,306 @@
|
||||
"""Query layers: RRF merge, the FTS expression, degraded mode, toc and grep."""
|
||||
|
||||
import shutil
|
||||
import sys
|
||||
from pathlib import Path
|
||||
|
||||
import pytest
|
||||
|
||||
SCRIPTS = Path(__file__).parent.parent / "scripts"
|
||||
sys.path.insert(0, str(SCRIPTS))
|
||||
|
||||
import wiki_search
|
||||
import wiki_store as store
|
||||
import wiki_sync
|
||||
from wiki_config import SourceConfig, load_config
|
||||
|
||||
BACKUP_DOC = """---
|
||||
title: Zálohování dat
|
||||
tags: [devops, backup]
|
||||
---
|
||||
|
||||
# Zálohování dat
|
||||
|
||||
Záloha běží každou noc přes rsync na druhý disk.
|
||||
|
||||
## Retence snapshotů
|
||||
|
||||
Držíme třicet denních snapshotů a dvanáct měsíčních.
|
||||
"""
|
||||
|
||||
TRAVEL_DOC = """---
|
||||
title: Tokijské metro
|
||||
tags: [transit]
|
||||
---
|
||||
|
||||
# Tokijské metro
|
||||
|
||||
Z Narity do centra jede Skyliner za osmatřicet minut.
|
||||
"""
|
||||
|
||||
|
||||
def _index(env):
|
||||
assert wiki_sync.main([]) == 0
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# pure functions
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
|
||||
def test_rrf_merge_over_known_ranks():
|
||||
merged = wiki_search.rrf_merge([[1, 2, 3], [3, 1, 4]])
|
||||
assert [chunk_id for chunk_id, _ in merged] == [1, 3, 2, 4]
|
||||
scores = dict(merged)
|
||||
assert scores[1] == pytest.approx(1 / 61 + 1 / 62)
|
||||
assert scores[3] == pytest.approx(1 / 63 + 1 / 61)
|
||||
assert scores[4] == pytest.approx(1 / 63)
|
||||
|
||||
|
||||
def test_rrf_merge_of_a_single_list_keeps_its_order():
|
||||
assert [i for i, _ in wiki_search.rrf_merge([[7, 8, 9]])] == [7, 8, 9]
|
||||
|
||||
|
||||
def test_fts_expression_adds_prefix_wildcards():
|
||||
"""Czech inflection is covered by the wildcard; a 2-char prefix is too broad to keep."""
|
||||
assert wiki_search.fts_match_expression("záloha dat") == '"záloha"* OR "dat"*'
|
||||
assert wiki_search.fts_match_expression("v ok dva") == '"v" OR "ok" OR "dva"*'
|
||||
|
||||
|
||||
def test_fts_expression_neutralises_operators_and_punctuation():
|
||||
expression = wiki_search.fts_match_expression("NOT (a AND b) -c*")
|
||||
assert expression == '"NOT"* OR "a" OR "AND"* OR "b" OR "c"'
|
||||
|
||||
|
||||
def test_fts_expression_of_empty_query_is_empty():
|
||||
assert wiki_search.fts_match_expression("!!! ") == ""
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# search
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
|
||||
def test_search_reports_which_half_found_each_hit(wiki_env, fake_embedder, capsys):
|
||||
wiki_env.write_file("notes/zalohy.md", BACKUP_DOC)
|
||||
wiki_env.write_file("notes/tokio.md", TRAVEL_DOC)
|
||||
_index(wiki_env)
|
||||
|
||||
config = load_config(wiki_env.config_path)
|
||||
assert wiki_search.run_search(config, "retence snapshotů", limit=5) == 0
|
||||
|
||||
out = capsys.readouterr().out
|
||||
assert "notes/zalohy.md" in out
|
||||
assert "Retence snapshotů" in out
|
||||
assert "bm25 #" in out and "vec #" in out
|
||||
|
||||
|
||||
def test_search_prefix_match_finds_inflected_form(wiki_env, fake_embedder, capsys):
|
||||
wiki_env.write_file("notes/zalohy.md", BACKUP_DOC)
|
||||
_index(wiki_env)
|
||||
|
||||
config = load_config(wiki_env.config_path)
|
||||
assert wiki_search.run_search(config, "zaloh", limit=5) == 0
|
||||
assert "notes/zalohy.md" in capsys.readouterr().out
|
||||
|
||||
|
||||
def test_search_says_out_loud_when_embeddings_are_unavailable(wiki_env, fake_embedder, capsys):
|
||||
"""Verification 4: degrade to FTS-only, never silently."""
|
||||
wiki_env.write_file("notes/zalohy.md", BACKUP_DOC)
|
||||
_index(wiki_env)
|
||||
capsys.readouterr()
|
||||
|
||||
# Drop the fake and let the config's unreachable endpoint take over.
|
||||
wiki_search.OllamaEmbedder = wiki_search.__dict__["OllamaEmbedder"]
|
||||
from wiki_embed import OllamaEmbedder as RealEmbedder
|
||||
|
||||
wiki_search.OllamaEmbedder = RealEmbedder
|
||||
|
||||
config = load_config(wiki_env.config_path)
|
||||
assert wiki_search.run_search(config, "záloha", limit=5) == 0
|
||||
|
||||
out = capsys.readouterr().out
|
||||
assert "embeddings unavailable" in out
|
||||
assert "FTS-only" in out
|
||||
assert "notes/zalohy.md" in out # the lexical half still answers
|
||||
|
||||
|
||||
def test_search_refuses_an_index_from_another_contract(wiki_env, fake_embedder, capsys):
|
||||
"""Verification 5: a mismatch must say `reindex needed`, not return results."""
|
||||
wiki_env.write_file("notes/zalohy.md", BACKUP_DOC)
|
||||
_index(wiki_env)
|
||||
|
||||
wiki_env.config_path.write_text(
|
||||
wiki_env.config_path.read_text(encoding="utf-8").replace(
|
||||
"model: qwen3-embedding:0.6b", "model: qwen3-embedding:8b"
|
||||
),
|
||||
encoding="utf-8",
|
||||
)
|
||||
config = load_config(wiki_env.config_path)
|
||||
assert wiki_search.run_search(config, "záloha", limit=5) == 1
|
||||
|
||||
captured = capsys.readouterr()
|
||||
assert "reindex needed" in captured.err
|
||||
assert "notes/zalohy.md" not in captured.out
|
||||
|
||||
|
||||
def test_search_warns_while_vectors_are_still_pending(wiki_env, monkeypatch, capsys):
|
||||
wiki_env.write_file("notes/zalohy.md", BACKUP_DOC)
|
||||
assert wiki_sync.main([]) == 1 # unreachable endpoint -> chunks stay pending
|
||||
|
||||
from conftest import FakeEmbedder
|
||||
|
||||
monkeypatch.setattr(wiki_search, "OllamaEmbedder", FakeEmbedder)
|
||||
config = load_config(wiki_env.config_path)
|
||||
assert wiki_search.run_search(config, "záloha", limit=5) == 0
|
||||
|
||||
out = capsys.readouterr().out
|
||||
assert "awaiting vectors" in out
|
||||
|
||||
|
||||
def test_search_over_an_empty_index_says_no_matches(wiki_env, fake_embedder, capsys):
|
||||
config = load_config(wiki_env.config_path)
|
||||
assert wiki_search.run_search(config, "záloha", limit=5) == 0
|
||||
assert "(no matches)" in capsys.readouterr().out
|
||||
|
||||
|
||||
def test_an_unrelated_query_still_returns_nearest_neighbours(wiki_env, fake_embedder, capsys):
|
||||
"""KNN has no distance floor: the semantic half always offers its k closest chunks.
|
||||
|
||||
That is deliberate for retrieval — the agent reads the chunk and judges it — but it
|
||||
means an empty result set only ever means an empty index.
|
||||
"""
|
||||
wiki_env.write_file("notes/zalohy.md", BACKUP_DOC)
|
||||
_index(wiki_env)
|
||||
capsys.readouterr()
|
||||
|
||||
config = load_config(wiki_env.config_path)
|
||||
assert wiki_search.run_search(config, "kajakářství", limit=5) == 0
|
||||
out = capsys.readouterr().out
|
||||
assert "notes/zalohy.md" in out
|
||||
assert "bm25 #" not in out # nothing lexical matched; every hit came from the vectors
|
||||
|
||||
|
||||
def test_search_with_an_empty_query(wiki_env, fake_embedder, capsys):
|
||||
wiki_env.write_file("notes/zalohy.md", BACKUP_DOC)
|
||||
_index(wiki_env)
|
||||
capsys.readouterr()
|
||||
|
||||
config = load_config(wiki_env.config_path)
|
||||
assert wiki_search.run_search(config, "???", limit=5) == 0
|
||||
assert "(empty query)" in capsys.readouterr().out
|
||||
|
||||
|
||||
def test_search_limit_caps_the_result_count(wiki_env, fake_embedder, capsys):
|
||||
for index in range(8):
|
||||
wiki_env.write_file(f"notes/doc{index}.md", f"# Dokument {index}\n\nZáloha dat číslo {index}.\n")
|
||||
_index(wiki_env)
|
||||
capsys.readouterr()
|
||||
|
||||
config = load_config(wiki_env.config_path)
|
||||
assert wiki_search.run_search(config, "záloha dat", limit=3) == 0
|
||||
out = capsys.readouterr().out
|
||||
assert len([line for line in out.splitlines() if line.startswith(("1. ", "2. ", "3. ", "4. "))]) == 3
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# toc
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
|
||||
def test_toc_groups_by_directory_and_shows_titles_and_tags(wiki_env, fake_embedder, capsys):
|
||||
wiki_env.write_file("notes/devops/zalohy.md", BACKUP_DOC)
|
||||
wiki_env.write_file("notes/travel/tokio.md", TRAVEL_DOC)
|
||||
_index(wiki_env)
|
||||
capsys.readouterr()
|
||||
|
||||
assert wiki_search.run_toc(None, None) == 0
|
||||
out = capsys.readouterr().out
|
||||
assert "workspace (2 files)" in out
|
||||
assert "notes/devops/" in out
|
||||
assert "zalohy.md" in out
|
||||
assert "Zálohování dat" in out
|
||||
assert "[devops, backup]" in out
|
||||
|
||||
|
||||
def test_toc_filters_by_tag(wiki_env, fake_embedder, capsys):
|
||||
wiki_env.write_file("notes/zalohy.md", BACKUP_DOC)
|
||||
wiki_env.write_file("notes/tokio.md", TRAVEL_DOC)
|
||||
_index(wiki_env)
|
||||
capsys.readouterr()
|
||||
|
||||
assert wiki_search.run_toc(None, "transit") == 0
|
||||
out = capsys.readouterr().out
|
||||
assert "tokio.md" in out
|
||||
assert "zalohy.md" not in out
|
||||
|
||||
|
||||
def test_toc_on_an_empty_index(wiki_env, fake_embedder, capsys):
|
||||
assert wiki_search.run_toc(None, None) == 0
|
||||
assert "(no indexed files match)" in capsys.readouterr().out
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# grep
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
|
||||
@pytest.mark.skipif(shutil.which("rg") is None, reason="ripgrep not installed")
|
||||
def test_grep_sees_source_code_the_index_never_touches(wiki_env, fake_embedder, capsys):
|
||||
"""Layer 1 is the substitute for indexing code, and it costs nothing extra."""
|
||||
wiki_env.write_file("notes/zalohy.md", BACKUP_DOC)
|
||||
wiki_env.write_file("notes/tools/backup.py", "def rotate_snapshots(keep=30):\n return keep\n")
|
||||
_index(wiki_env)
|
||||
capsys.readouterr()
|
||||
|
||||
config = load_config(wiki_env.config_path)
|
||||
assert wiki_search.run_grep(config, "rotate_snapshots", None) == 0
|
||||
out = capsys.readouterr().out
|
||||
assert "backup.py" in out
|
||||
assert "rotate_snapshots" in out
|
||||
|
||||
# The same identifier is absent from the index — grep is the only layer that has it.
|
||||
with store.connection(wiki_env.db_path) as conn:
|
||||
assert store.bm25_ranked_ids(conn, "rotate_snapshots", 10) == []
|
||||
|
||||
|
||||
@pytest.mark.skipif(shutil.which("rg") is None, reason="ripgrep not installed")
|
||||
def test_grep_stays_inside_the_source_paths(wiki_env, fake_embedder, capsys):
|
||||
wiki_env.write_file("notes/zalohy.md", BACKUP_DOC)
|
||||
wiki_env.write_file("tmp/dump/leak.md", "tajnastruna v tmp\n")
|
||||
|
||||
config = load_config(wiki_env.config_path)
|
||||
assert wiki_search.run_grep(config, "tajnastruna", None) == 0
|
||||
assert "(no matches)" in capsys.readouterr().out
|
||||
|
||||
|
||||
@pytest.mark.skipif(shutil.which("rg") is None, reason="ripgrep not installed")
|
||||
def test_grep_reports_an_unknown_source(wiki_env, fake_embedder, capsys):
|
||||
config = load_config(wiki_env.config_path)
|
||||
assert wiki_search.run_grep(config, "cokoli", "nope") == 1
|
||||
assert "unknown source" in capsys.readouterr().err
|
||||
|
||||
|
||||
def test_grep_roots_use_the_literal_prefix_of_each_glob(wiki_env):
|
||||
wiki_env.write_file("notes/a.md", "x")
|
||||
wiki_env.write_file("develop/b.md", "x")
|
||||
source = load_config(wiki_env.config_path).source("workspace")
|
||||
assert source is not None
|
||||
assert wiki_search.grep_roots(source) == [
|
||||
wiki_env.workspace / "notes",
|
||||
wiki_env.workspace / "develop",
|
||||
]
|
||||
|
||||
|
||||
def test_grep_roots_of_a_git_source_is_the_whole_clone(wiki_env):
|
||||
clone = wiki_env.wiki_dir / "remote" / "travel"
|
||||
clone.mkdir(parents=True)
|
||||
source = SourceConfig(
|
||||
source_id="travel",
|
||||
kind="git",
|
||||
url="git@example:travel.git",
|
||||
paths=("**",),
|
||||
include=("*.md",),
|
||||
exclude=(),
|
||||
)
|
||||
assert wiki_search.grep_roots(source) == [clone]
|
||||
279
skills/wiki/tests/test_wiki_sync.py
Normal file
279
skills/wiki/tests/test_wiki_sync.py
Normal file
@@ -0,0 +1,279 @@
|
||||
"""Sync driver: idempotence, the lock, degraded mode, the meta guard, coverage."""
|
||||
|
||||
import json
|
||||
import os
|
||||
import sys
|
||||
from pathlib import Path
|
||||
|
||||
SCRIPTS = Path(__file__).parent.parent / "scripts"
|
||||
sys.path.insert(0, str(SCRIPTS))
|
||||
|
||||
import wiki_store as store
|
||||
import wiki_sync
|
||||
from wiki_config import load_config
|
||||
from wiki_embed import expected_meta
|
||||
|
||||
DOC = """# Zálohování
|
||||
|
||||
Záloha dat na disk probíhá každý den v noci pomocí rsyncu.
|
||||
|
||||
## Retence
|
||||
|
||||
Držíme třicet denních snapshotů a dvanáct měsíčních.
|
||||
"""
|
||||
|
||||
OTHER_DOC = """# Cestování
|
||||
|
||||
Tokio metro je nejrychlejší cesta z Narity do centra.
|
||||
"""
|
||||
|
||||
|
||||
def _stats(env):
|
||||
with store.connection(env.db_path) as conn:
|
||||
return store.index_stats(conn)
|
||||
|
||||
|
||||
def _run(argv=None):
|
||||
return wiki_sync.main(argv or [])
|
||||
|
||||
|
||||
def _file_row(conn, path):
|
||||
row = store.get_file(conn, "workspace", path)
|
||||
assert row is not None, f"{path} is not indexed"
|
||||
return row
|
||||
|
||||
|
||||
def test_indexes_covered_workspace_files(wiki_env, fake_embedder):
|
||||
wiki_env.write_file("notes/zalohy.md", DOC)
|
||||
wiki_env.write_file("develop/knowledge.md", OTHER_DOC)
|
||||
|
||||
assert _run() == 0
|
||||
|
||||
stats = _stats(wiki_env)
|
||||
assert stats["files"] == 2
|
||||
assert stats["chunks"] > 0
|
||||
assert stats["vectors"] == stats["chunks"]
|
||||
assert stats["pending"] == 0
|
||||
assert "indexed 2 files" in wiki_env.log_text() or "indexed 1 files" in wiki_env.log_text()
|
||||
|
||||
|
||||
def test_paths_whitelist_and_exclude_are_honoured(wiki_env, fake_embedder):
|
||||
wiki_env.write_file("notes/keep.md", DOC)
|
||||
wiki_env.write_file("notes/inbox/raw.md", DOC) # excluded
|
||||
wiki_env.write_file("develop/history.md", DOC) # excluded
|
||||
wiki_env.write_file("tmp/clone/README.md", DOC) # outside paths
|
||||
wiki_env.write_file("notes/script.py", "# TODO: fix this\n") # outside include
|
||||
|
||||
assert _run() == 0
|
||||
|
||||
with store.connection(wiki_env.db_path) as conn:
|
||||
paths = sorted(store.list_source_files(conn, "workspace"))
|
||||
assert paths == ["notes/keep.md"]
|
||||
|
||||
|
||||
def test_second_run_over_unchanged_tree_is_a_no_op(wiki_env, fake_embedder):
|
||||
"""Verification 2: counts identical and nothing appended to the log."""
|
||||
wiki_env.write_file("notes/zalohy.md", DOC)
|
||||
assert _run() == 0
|
||||
|
||||
first_stats = _stats(wiki_env)
|
||||
with store.connection(wiki_env.db_path) as conn:
|
||||
first_indexed_at = _file_row(conn, "notes/zalohy.md")["indexed_at"]
|
||||
log_after_first = wiki_env.log_text()
|
||||
|
||||
assert _run() == 0
|
||||
|
||||
assert _stats(wiki_env) == first_stats
|
||||
assert wiki_env.log_text() == log_after_first
|
||||
with store.connection(wiki_env.db_path) as conn:
|
||||
assert _file_row(conn, "notes/zalohy.md")["indexed_at"] == first_indexed_at
|
||||
|
||||
|
||||
def test_touching_a_file_without_changing_it_only_refreshes_stats(wiki_env, fake_embedder):
|
||||
wiki_env.write_file("notes/zalohy.md", DOC)
|
||||
assert _run() == 0
|
||||
before = _stats(wiki_env)
|
||||
with store.connection(wiki_env.db_path) as conn:
|
||||
indexed_at = _file_row(conn, "notes/zalohy.md")["indexed_at"]
|
||||
|
||||
os.utime(wiki_env.workspace / "notes/zalohy.md", (1_600_000_000, 1_600_000_000))
|
||||
assert _run() == 0
|
||||
|
||||
assert _stats(wiki_env) == before
|
||||
with store.connection(wiki_env.db_path) as conn:
|
||||
row = _file_row(conn, "notes/zalohy.md")
|
||||
assert row["indexed_at"] == indexed_at # content untouched -> no re-chunk
|
||||
assert row["mtime"] == 1_600_000_000
|
||||
|
||||
|
||||
def test_edited_file_is_rechunked_and_reembedded(wiki_env, fake_embedder):
|
||||
wiki_env.write_file("notes/zalohy.md", DOC)
|
||||
assert _run() == 0
|
||||
|
||||
wiki_env.write_file("notes/zalohy.md", DOC + "\n## Offsite\n\nKopie jede do S3 každý týden.\n")
|
||||
assert _run() == 0
|
||||
|
||||
stats = _stats(wiki_env)
|
||||
assert stats["pending"] == 0
|
||||
assert stats["vectors"] == stats["chunks"]
|
||||
with store.connection(wiki_env.db_path) as conn:
|
||||
assert store.orphan_vector_ids(conn) == []
|
||||
assert len(store.bm25_ranked_ids(conn, "offsite", 10)) == 1
|
||||
|
||||
|
||||
def test_deleted_file_leaves_no_chunks_or_vectors(wiki_env, fake_embedder):
|
||||
wiki_env.write_file("notes/zalohy.md", DOC)
|
||||
wiki_env.write_file("notes/travel.md", OTHER_DOC)
|
||||
assert _run() == 0
|
||||
|
||||
(wiki_env.workspace / "notes/travel.md").unlink()
|
||||
assert _run() == 0
|
||||
|
||||
with store.connection(wiki_env.db_path) as conn:
|
||||
assert sorted(store.list_source_files(conn, "workspace")) == ["notes/zalohy.md"]
|
||||
assert store.orphan_vector_ids(conn) == []
|
||||
assert store.bm25_ranked_ids(conn, "tokio", 10) == []
|
||||
stats = _stats(wiki_env)
|
||||
assert stats["vectors"] == stats["chunks"]
|
||||
|
||||
|
||||
def test_live_lock_makes_the_run_a_silent_no_op(wiki_env, fake_embedder):
|
||||
"""Verification 3: a second concurrent sync exits 0 without writing."""
|
||||
wiki_env.write_file("notes/zalohy.md", DOC)
|
||||
wiki_sync.LOCK_PATH.write_text(
|
||||
json.dumps({"pid": os.getpid(), "started_at": wiki_sync.datetime.now().astimezone().isoformat()}),
|
||||
encoding="utf-8",
|
||||
)
|
||||
|
||||
assert _run() == 0
|
||||
|
||||
assert not wiki_env.db_path.exists()
|
||||
assert wiki_env.log_text() == ""
|
||||
|
||||
|
||||
def test_stale_lock_is_reclaimed_and_logged(wiki_env, fake_embedder):
|
||||
wiki_env.write_file("notes/zalohy.md", DOC)
|
||||
dead_pid = 999_999
|
||||
wiki_sync.LOCK_PATH.write_text(
|
||||
json.dumps({"pid": dead_pid, "started_at": wiki_sync.datetime.now().astimezone().isoformat()}),
|
||||
encoding="utf-8",
|
||||
)
|
||||
|
||||
assert _run() == 0
|
||||
|
||||
assert "stale lock, reclaiming" in wiki_env.log_text()
|
||||
assert _stats(wiki_env)["files"] == 1
|
||||
assert not wiki_sync.LOCK_PATH.exists()
|
||||
|
||||
|
||||
def test_unreadable_lock_is_treated_as_stale(wiki_env, fake_embedder):
|
||||
wiki_env.write_file("notes/zalohy.md", DOC)
|
||||
wiki_sync.LOCK_PATH.write_text("not json", encoding="utf-8")
|
||||
|
||||
assert _run() == 0
|
||||
assert _stats(wiki_env)["files"] == 1
|
||||
|
||||
|
||||
def test_coverage_report_names_uncovered_directories(wiki_env, fake_embedder):
|
||||
"""Verification 6: markdown outside `paths` must surface as a log line."""
|
||||
wiki_env.write_file("notes/zalohy.md", DOC)
|
||||
wiki_env.write_file("recepty/gulas.md", "# Guláš\n\nCibule na dva kusy hovězího.\n")
|
||||
# The skill's own clone storage holds markdown but is never a candidate.
|
||||
(wiki_env.wiki_dir / "remote" / "index").mkdir(parents=True)
|
||||
(wiki_env.wiki_dir / "remote" / "index" / "foreign.md").write_text("# Cizí\n", encoding="utf-8")
|
||||
|
||||
assert _run() == 0
|
||||
|
||||
coverage = [line for line in wiki_env.log_text().splitlines() if "coverage" in line]
|
||||
assert len(coverage) == 1
|
||||
assert "recepty" in coverage[0]
|
||||
assert "wiki" not in coverage[0]
|
||||
|
||||
|
||||
def test_degraded_mode_then_recovery_without_file_change(wiki_env, monkeypatch):
|
||||
"""Verification 4: no Ollama -> FTS-only pending; once back, step 4b fills the vectors."""
|
||||
wiki_env.write_file("notes/zalohy.md", DOC)
|
||||
|
||||
# The fixture config points at embed.invalid, so no embedder is reachable here.
|
||||
assert _run() == 1
|
||||
stats = _stats(wiki_env)
|
||||
assert stats["chunks"] > 0
|
||||
assert stats["vectors"] == 0
|
||||
assert stats["pending"] == stats["chunks"]
|
||||
assert "embeddings unavailable" in wiki_env.log_text()
|
||||
|
||||
with store.connection(wiki_env.db_path) as conn:
|
||||
assert len(store.bm25_ranked_ids(conn, "zaloh*", 10)) >= 1 # FTS works meanwhile
|
||||
|
||||
from conftest import FakeEmbedder
|
||||
|
||||
monkeypatch.setattr(wiki_sync, "OllamaEmbedder", FakeEmbedder)
|
||||
assert _run() == 0 # no file changed, yet the pending chunks get vectors
|
||||
|
||||
stats = _stats(wiki_env)
|
||||
assert stats["pending"] == 0
|
||||
assert stats["vectors"] == stats["chunks"]
|
||||
|
||||
|
||||
def test_meta_mismatch_blocks_indexing_until_full(wiki_env, fake_embedder):
|
||||
"""Verification 5: never mix vectors from two contracts."""
|
||||
wiki_env.write_file("notes/zalohy.md", DOC)
|
||||
assert _run() == 0
|
||||
before = _stats(wiki_env)
|
||||
|
||||
wiki_env.config_path.write_text(
|
||||
wiki_env.config_path.read_text(encoding="utf-8").replace(
|
||||
"model: qwen3-embedding:0.6b", "model: qwen3-embedding:4b"
|
||||
),
|
||||
encoding="utf-8",
|
||||
)
|
||||
wiki_env.write_file("notes/new.md", OTHER_DOC)
|
||||
|
||||
assert _run() == 1
|
||||
assert "index identity mismatch" in wiki_env.log_text()
|
||||
assert _stats(wiki_env) == before # nothing indexed under the new contract
|
||||
|
||||
assert _run(["--full"]) == 0
|
||||
with store.connection(wiki_env.db_path) as conn:
|
||||
assert store.read_meta(conn)["embedding_model"] == "qwen3-embedding:4b"
|
||||
assert sorted(store.list_source_files(conn, "workspace")) == ["notes/new.md", "notes/zalohy.md"]
|
||||
assert store.orphan_vector_ids(conn) == []
|
||||
|
||||
|
||||
def test_full_rebuild_wipes_and_reindexes(wiki_env, fake_embedder):
|
||||
wiki_env.write_file("notes/zalohy.md", DOC)
|
||||
assert _run() == 0
|
||||
before = _stats(wiki_env)
|
||||
|
||||
assert _run(["--full"]) == 0
|
||||
|
||||
assert _stats(wiki_env) == before
|
||||
with store.connection(wiki_env.db_path) as conn:
|
||||
assert store.orphan_vector_ids(conn) == []
|
||||
|
||||
|
||||
def test_meta_is_written_on_a_fresh_index(wiki_env, fake_embedder):
|
||||
wiki_env.write_file("notes/zalohy.md", DOC)
|
||||
assert _run() == 0
|
||||
|
||||
config = load_config(wiki_env.config_path)
|
||||
with store.connection(wiki_env.db_path) as conn:
|
||||
assert store.read_meta(conn) == expected_meta(config.embedding)
|
||||
|
||||
|
||||
def test_unknown_source_argument_is_reported(wiki_env, fake_embedder):
|
||||
wiki_env.write_file("notes/zalohy.md", DOC)
|
||||
assert _run(["--source", "nope"]) == 1
|
||||
assert "unknown source" in wiki_env.log_text()
|
||||
|
||||
|
||||
def test_batching_respects_the_configured_size(wiki_env, fake_embedder):
|
||||
"""batch: 4 in the fixture config — the embedder must be called in chunks of 4."""
|
||||
for index in range(6):
|
||||
wiki_env.write_file(f"notes/doc{index}.md", f"# Dokument {index}\n\n{DOC}\n")
|
||||
|
||||
assert _run() == 0
|
||||
|
||||
calls = [len(call) for embedder in fake_embedder for call in embedder.calls]
|
||||
assert calls, "the embedder was never called"
|
||||
assert max(calls) <= 4
|
||||
Reference in New Issue
Block a user