lachtan/nanobot-runtime
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

provozni zaloha

Browse Source
This commit is contained in:
lachtan
2026-06-24 08:11:12 +02:00
parent 9295dba19f
commit 1db3ec4756
97 changed files with 7698 additions and 817 deletions
Download Patch File Download Diff File Expand all files Collapse all files

171
skills/llm-wiki/SKILL.md Normal file
View File

@@ -0,0 +1,171 @@
---
name: llm-wiki
description: >
Build and maintain an LLM-curated personal knowledge base — the "LLM Wiki" pattern. Use whenever
the user wants to ingest a source (paper, article, transcript, PDF, notes) into a persistent,
compounding knowledge base, ask a question against the accumulated notes, lint or audit such a
base, or initialize a new one. Applies even when the user doesn't say "wiki" — any time they
accumulate textual sources over time and want them organized. A deliberate, standalone store,
distinct from agent memory (note / keep / MEMORY.md).
---
# LLM Wiki
A skill for building and maintaining an LLM-curated knowledge base inside a project, following the pattern Andrej Karpathy described in his April 2026 gist. The wiki is a directory of markdown files that the LLM owns and maintains; the user curates sources and asks questions, and the LLM does the bookkeeping.
## Nanobot adaptation — read this first
This skill is ported to run on this nanobot. The generic docs below describe a project-local wiki; the rules here pin it to this nanobot and override anything that conflicts.
**⚠️ On any add/save/ingest request: capture only.** Write the source to `cml/raw/<slug>.md`, confirm in one short line, and **STOP** — no reads, no scripts, no compiles. Full rules and the escape hatch in "Capture vs compile" below — read it before acting on any such request.
- **One wiki, fixed location.** This nanobot has exactly one wiki, at `cml/wiki/`, with raw sources at `cml/raw/` — both relative to your working directory (the workspace). Wherever the docs below say `wiki/` or `<project-root>/wiki/`, read `cml/wiki/`; `raw/` means `cml/raw/`.
- **Run scripts with `uv run`, never bare `python`.** Always `uv run skills/llm-wiki/scripts/<script>.py …`. The scripts default to `cml/wiki`, so for most you can omit the path argument.
- **Bootstrap once:** `uv run skills/llm-wiki/scripts/init_wiki.py . --wiki-dir cml/wiki --raw-dir cml/raw` creates `cml/wiki/` + `cml/raw/`. Idempotent — safe to re-run.
- **Separate store — not agent memory.** The wiki is a deliberate, standalone knowledge base of curated sources. It is **not** the agent's memory: keep it distinct from `note`, `keep`, and `MEMORY.md`, and do not fold wiki content into them (or vice versa). The Dream processor must **not** touch `cml/` — it is outside the memory and skills Dream curates. Do not wire the wiki into `MEMORY.md`; its location is documented here.
- **Lint is report-only — a lint turn never mutates the wiki.** A lint request ("lint", "what's broken", "clean up the wiki", or a HEARTBEAT lint) means exactly: run `wiki_lint.py` and — if the graph layer exists — `wiki_graph_lint.py`, present the findings as a summary of *proposed* edits, and **STOP the turn**. Forbidden during a lint turn (all of it is fixing, done later in a separate approved turn): creating or editing any page under `cml/wiki/`, writing debug/throwaway scripts, regenerating the graph (`wiki_graph_extract.py`), re-running lint in a loop, updating `index.md` / `log.md`. If you catch yourself editing a page or re-running lint to check your own fix, you are fixing inline — stop. Fixes happen only after the user approves, one category at a time (see the lint workflow).
- **Language.** This skill body is English; reply to the user in the user's own language.
- **Scope: local PoC.** Single machine, versioned by the git repo running over the workspace. No shared remote, no multi-client sync.
## Capture vs compile — the background pipeline
Ingest is split into two phases so the interactive turn stays instant. Full agentic compile takes a minute or more; doing it inline made capture unusable.
- **Capture (interactive default — instant).** When the user wants to add a source ("save this", "ingest this", "add X to the wiki"), do exactly three things and nothing more:
1. Write the source into `cml/raw/<slug>.md` (pick a descriptive `<slug>`). Inline text → write as-is. URL → write the URL as-is (the compile step will fetch it).
2. Confirm in **one short line** ("zachyceno — zkompiluju na pozadí").
3. **STOP the turn.**
Forbidden during a capture turn (all of this is compile, done later in the background): reading `SCHEMA.md` / `index.md` / any wiki page, creating or editing pages under `cml/wiki/`, running `init_wiki.py` / `wiki_lint.py` / `wiki_graph_*` / any script, rebuilding the graph, updating `index.md` or `log.md`. If you catch yourself about to read the schema or write a page, you are doing compile inline — stop and just capture.
- **Escape hatch.** Only if the user *explicitly* says "compile now" / "synchronously" / "do it now" / "hned" do you run the full Compile workflow inline in this turn. A normal "add this to my wiki" is **not** an escape hatch — it is capture.
- **Compile (drain — background, batched).** A system cron runs `scripts/wiki_compile.py` every minute; when `cml/raw/` has pending sources it invokes this skill with a drain goal. Compile processes **every** pending source in one batch (one index/graph update for many sources), then moves each processed source into `cml/raw/_done/`. This is the existing ingest workflow (below) applied per pending source. **Idempotency:** if `cml/wiki/sources/<slug>.md` already exists for a source, treat it as already compiled — skip re-processing and move the raw file to `cml/raw/_done/`. Always move a source out of `cml/raw/` once handled so the next cron tick doesn't re-process it.
- **`cml/raw/` layout.** Regular files directly in `cml/raw/` = the **pending inbox**. `cml/raw/_done/` = processed sources (move here after a successful ingest). `cml/raw/_hard/` = sources held back as ambiguous/conflicting (don't force-compile these; record why in `log.md`). `cml/raw/assets/` = downloaded images, never a source. The pre-check and compile both ignore `_done/`, `_hard/`, and `assets/`.
## Architecture: three layers, three operations
The wiki has three layers and three operations. Internalize this vocabulary because the rest of the skill assumes it.
The three layers are **raw sources** (the user's curated source material — articles, papers, PDFs, transcripts; immutable, the LLM reads but never modifies them), **the wiki** (a directory of LLM-generated markdown pages — entity pages, concept pages, comparisons, summaries; the LLM owns this layer entirely), and **the schema** (a `SCHEMA.md` file at the wiki root that documents the conventions for this particular wiki — page types, naming rules, tag taxonomy, ingest workflow customizations; co-evolved with the user).
The three operations are **ingest** (a new source arrives; the LLM reads it, writes a summary page, updates relevant entity and concept pages, appends to the log), **query** (the user asks a question; the LLM navigates the wiki via the index, reads the relevant pages, and synthesizes an answer — often filing the answer back as a new page so the exploration compounds), and **lint** (a periodic health check; the LLM scans for contradictions, stale claims, orphan pages, missing concepts, broken links).
For the canonical write-up of these operations, read `references/architecture.md`. For the step-by-step procedures, read `references/ingest-workflow.md`, `references/query-workflow.md`, and `references/lint-workflow.md` as needed.
## Graph layer (compiled, optional)
Pages can carry typed `graph:` metadata in frontmatter. A bundled extractor compiles every page into `wiki/graph/`: `nodes.jsonl`, `edges.jsonl`, `graph.sqlite`, `graph.graphml`. **Markdown is canonical**; the graph is a regenerable index. Pages without `graph:` still appear as nodes (derived from their `type`/`kind`) and contribute low-confidence `mentions` edges from body wikilinks. Typed semantic edges (e.g. `founded`, `proposed`, `depends_on`) require an explicit source and evidence quote — never emit one inferred from training data.
The conventions for the graph layer (predicate vocabulary, node id format, required fields) live in `wiki/graph/ontology.yaml`. The full reference is `references/graph-workflow.md`. Run the bundled scripts after substantive ingests:
```bash
uv run skills/llm-wiki/scripts/wiki_graph_lint.py cml/wiki/ # check ontology + evidence + alias collisions
uv run skills/llm-wiki/scripts/wiki_graph_extract.py cml/wiki/ # rebuild nodes.jsonl, edges.jsonl, graph.sqlite, graph.graphml
uv run skills/llm-wiki/scripts/wiki_graph_query.py cml/wiki/ neighbors --node product:konvy
```
If `wiki/graph/ontology.yaml` does not exist, the wiki is pre-graph and you should treat the graph step as a no-op — don't fabricate it.
## Default project layout
The wiki is at a fixed location on this nanobot (`cml/wiki/`, `cml/raw/`):
```
<workspace>/
├── cml/
│ ├── wiki/
│ │ ├── SCHEMA.md ← conventions, the "config file" — read this FIRST
│ │ ├── index.md ← entry point: catalog of all pages with one-line summaries
│ │ ├── log.md ← append-only chronological log of ingests/queries/lints
│ │ ├── indexes/ ← (appears once index.md shards) per-category indexes
│ │ ├── entities/ ← pages about specific things (people, products, papers, places)
│ │ ├── concepts/ ← pages about ideas, methods, frameworks
│ │ ├── sources/ ← per-source summary pages (one per ingested source)
│ │ └── synthesis/ ← cross-cutting analyses, comparisons, query results filed back
│ └── raw/ ← the user's source material (PDFs, .md clippings, images)
│ └── assets/ ← downloaded images referenced by raw clippings
└── ...
```
## The scalability discipline
The single biggest failure mode of the LLM Wiki pattern is the wiki itself becoming a context bottleneck. Naive implementations break around a few hundred pages: the LLM either reads too many pages per query or starts hallucinating because it skipped the relevant ones. This skill's design is shaped almost entirely by avoiding that failure. The principles below are non-negotiable; ignoring them is what makes the pattern collapse at scale.
**Atomic pages.** Every wiki page is about one concept and stays small — soft cap 400 lines or roughly 2,000 words, hard cap 800 lines. When a page outgrows this, split it: extract sub-concepts into their own pages and have the parent link to them. A page that takes up 30% of the context window on its own is a design smell.
**Index-first navigation.** Never grep or glob the wiki blindly when answering a query. Always read `index.md` (or the relevant sharded index under `indexes/`) first to identify candidate pages, then drill into only those. The index is engineered to be cheap to read — one line per page, no bodies — and it is the cache that makes the whole pattern scalable.
**Sharded indexes.** When `index.md` itself exceeds ~300 lines or the wiki passes ~150 pages, shard it: move category-specific entries into `indexes/<category>.md` files (e.g. `indexes/entities.md`, `indexes/concepts.md`, `indexes/sources.md`, or finer domain shards), and have the top-level `index.md` become a directory of those shards. Now reading the index is a two-step lookup but each step is bounded.
**YAML frontmatter on every page.** Every wiki page begins with frontmatter that includes at minimum `type`, `tags`, `sources`, and `updated`. The bundled `wiki_search.py` script can filter on these without reading page bodies. See `references/page-conventions.md`.
**Surgical edits, not rewrites.** When updating a page (e.g. adding a new cross-reference because a freshly ingested source mentions an existing entity), use `str_replace` to touch only the relevant section. Rewriting whole pages is slow, expensive in tokens, and risks losing prior nuance.
**Backlink discovery via grep.** To find every page that references a given entity, run `grep -rl "\[\[entity-name\]\]" cml/wiki/` rather than reading pages to look for mentions. The bundled scripts make this easy.
**Chunked source ingestion.** Large raw sources (long PDFs, book chapters, lengthy transcripts) should be read in chunks during ingest, not loaded whole. The ingest workflow handles this — see `references/ingest-workflow.md`.
**Search script for large wikis.** Once the wiki passes ~300 pages, plain index lookup may not surface the right pages for fuzzy queries. Use `scripts/wiki_search.py` for BM25-ranked retrieval with optional frontmatter filters. It's a fallback, not the default — index-first is still cheaper when it works.
**Stats.** `uv run skills/llm-wiki/scripts/wiki_stats.py` gives a quick summary of page count by type and link density — useful for deciding when to shard the index.
For the full scaling playbook including thresholds and migration steps, read `references/scaling-playbook.md`.
## Initializing a new wiki
If the project does not contain a `wiki/` directory (or whatever the user calls theirs), run the bootstrap script:
```bash
uv run skills/llm-wiki/scripts/init_wiki.py . --wiki-dir cml/wiki --raw-dir cml/raw
```
This creates the directory structure, drops in templates for `SCHEMA.md`, `index.md`, and `log.md`, and seeds a starter page convention document. After bootstrapping, briefly walk the user through the schema and ask whether they want to customize anything (e.g. domain-specific page types, custom tags) before the first ingest. The schema is meant to evolve — encourage editing it.
Do **not** wire the wiki into an agent-memory file (`MEMORY.md` / `AGENTS.md`) on this nanobot — see the Nanobot adaptation rules: the wiki is a separate store and its location is documented in this SKILL.md, which the skill description already surfaces.
## The ingest workflow (summary)
**STOP — do not run this for a plain "add this to my wiki" request.** This is the **compile (drain)** step. It runs only from the background cron (the drain goal) or the explicit "compile now" escape hatch. If the user just asked to add/save/ingest a source, you are in *capture* — write to `cml/raw/` and stop (see "Capture vs compile"). The source is already captured in `cml/raw/` when compile runs, so don't re-write it; read it from there.
The full workflow is in `references/ingest-workflow.md`; what follows is the shape of it. Read the source — chunked if large — and write a single source-summary page in `cml/wiki/sources/`, named after the source slug, with full frontmatter and citations back to the raw file. Then identify which existing entity and concept pages this source touches; for each, surgically update the relevant section using `str_replace` rather than rewriting. Identify any new entities or concepts the source introduces and create new pages for them, linking from related existing pages so they don't become orphans. Update `index.md` (or the relevant shard) with the new pages. Append a single line to `log.md` with the date, operation type, and source title. After the source is fully ingested, move its raw file into `cml/raw/_done/`. When run interactively, discuss the takeaways with the user as a final step — what surprised them, what's worth following up on — and offer to file that discussion back as a synthesis page.
**Page naming — avoid slug collisions.** A source page and a concept/entity page must not claim the same slug, or their `[[wikilinks]]` collide (e.g. a paper *and* the concept it introduces both wanting `rotary-position-embedding.md`). Name **concept/entity pages after the short name of the idea** (`rope.md`, `transformer.md`), and **source pages after the source's own slug** (the title/filename of the raw source). If two pages still resolve to the same slug, suffix one to disambiguate.
## The query workflow (summary)
Full version in `references/query-workflow.md`. To answer a query against the wiki: read `index.md` (or the relevant shard) first; identify candidate pages from one-line summaries; read those pages (and any backlinks they list that look relevant); synthesize the answer with `[[wikilink]]` citations to the pages you used; offer to file the synthesized answer back into `wiki/synthesis/` so future queries benefit. If the index doesn't surface good candidates, fall back to `uv run skills/llm-wiki/scripts/wiki_search.py "query terms"` for ranked retrieval. If the wiki appears to lack coverage of the topic, say so plainly rather than confabulating — flag it as a candidate ingest target.
## The lint workflow (summary)
**STOP — a lint turn reports, it does not fix.** See the report-only gate in "Nanobot adaptation". Run the scripts, present the findings, stop. The scripts are fast (sub-second on a small wiki); if a lint turn runs long, you have wrongly slipped into fixing.
Full version in `references/lint-workflow.md`. Lint is best run on a cadence (after every N ingests or weekly), not on every operation. Run `uv run skills/llm-wiki/scripts/wiki_lint.py` for structural issues (orphan pages, broken `[[wikilinks]]`, oversized pages, missing/malformed frontmatter, stale `updated` dates) and — if the graph layer exists — `uv run skills/llm-wiki/scripts/wiki_graph_lint.py` for typed-edge issues. For the semantic checks that need an LLM (contradictions with older claims, concepts mentioned but lacking a page, coverage gaps) read at most the ~10 most-recently-updated pages — no blind globbing. Present everything as proposed edits for the user to approve; never apply them in the lint turn — the wiki is the user's, and silent rewrites erode trust.
**When the user approves fixes (a later turn):** fix in bounded batches, one category at a time. Do **not** re-read the lint script to reverse-engineer it, and do **not** loop edit↔lint — run lint once at the end to confirm, and if findings remain, report them and ask rather than continuing blind. Graph-lint findings are interdependent (fixing one edge can create an orphan); clearing a large backlog is its own task, not part of a lint turn.
## Failure modes to guard against
- **Silent corruption:** every wiki claim must carry a `sources:` frontmatter entry pointing back to the raw file. When in doubt during ingest, hedge ("the source claims X") rather than asserting.
- **Wiki-reads-its-own-output drift:** during ingest, when updating an existing page, re-read the relevant raw source for the existing claim before merging — don't take the wiki's word for what the source said.
## Reference files
The reference files are the source of truth for the detailed procedures. Read them when the relevant operation is happening, not preemptively.
- `references/architecture.md` — the three layers and three operations explained in depth, with examples of page formats and the rationale behind each design choice
- `references/ingest-workflow.md` — the step-by-step ingest procedure including chunked reading for large sources and the per-page-type templates
- `references/query-workflow.md` — navigation patterns from index → page → backlinks, when to fall back to the search script, and how to file answers back as synthesis pages
- `references/lint-workflow.md` — what to check, how to present findings, and the cadence
- `references/page-conventions.md` — frontmatter schema, page naming, link syntax, page-type definitions, sizing rules
- `references/scaling-playbook.md` — thresholds at which to shard the index, when to introduce the search script, signals that the wiki has outgrown its current conventions
- `references/graph-workflow.md` — the optional graph layer: ontology, frontmatter schema, when to add typed edges vs plain wikilinks, and the extract/lint/query flow
## Templates
The templates in `assets/` are starting points — they get copied into the user's wiki on bootstrap and then evolve under the user's editing.
- `assets/SCHEMA.md.template` — the canonical schema document for a new wiki
- `assets/index.md.template` — the empty index file
- `assets/log.md.template` — the empty log file
- `assets/page.md.template` — a generic wiki page with the frontmatter scaffold
- `assets/ontology.yaml.template` — starter graph ontology copied to `wiki/graph/ontology.yaml`
- `assets/graph_README.md.template` — explainer for `wiki/graph/` (canonical vs generated files)
- `assets/graph_gitignore.template` — `.gitignore` for `wiki/graph/` (ignores `graph.sqlite` and `graph.graphml` by default)

113
skills/llm-wiki/assets/SCHEMA.md.template Normal file
View File

@@ -0,0 +1,113 @@
# Wiki Schema
This file is the configuration for this wiki. It documents the conventions, page types, tag taxonomy, and any workflow customizations. The LLM reads this first when entering the wiki, and its conventions override the defaults documented in the `llm-wiki` skill.
This file is **co-evolved with the user**. When the LLM notices a recurring pattern in your edits or feedback that isn't here, it will propose adding it. When something here stops fitting, prune it.
## Wiki location
- Wiki root: `wiki/`
- Raw sources: `raw/`
- Asset/image storage: `raw/assets/`
## Page types
This wiki uses these page types, each with a dedicated subdirectory:
- `source` (in `wiki/sources/`) — one summary page per ingested source.
- `entity` (in `wiki/entities/`) — pages about specific things: people, papers, products, places, organizations.
- `concept` (in `wiki/concepts/`) — pages about ideas, methods, frameworks, abstractions.
- `synthesis` (in `wiki/synthesis/`) — cross-cutting analyses, comparisons, query answers filed back.
Add additional types here as the wiki evolves.
## Tag taxonomy
(Empty initially. Add tags here as you adopt them, with one-line descriptions. Keep this list small and disciplined — a wiki with 200 tags has effectively no tags.)
Example structure:
- `methodology` — pages about research or analytical methods.
- `open-question` — pages or sections that flag unresolved questions.
- `contested` — pages where sources contradict.
## Page sizing
- Soft cap: 400 lines / ~2,000 words. Consider splitting beyond this.
- Hard cap: 800 lines. Must split.
## Frontmatter requirements
Every page must have:
- `type`
- `title`
- `tags`
- `created`
- `updated`
Plus type-specific:
- `source` pages: `authors`, `url` (if applicable), `raw`, `ingested`
- Non-source pages: `sources` listing the source-summary pages drawn from
## Optional graph metadata
Pages may declare typed graph metadata under a top-level `graph:` key. This is the source of truth for the compiled knowledge graph under `wiki/graph/`. Markdown remains canonical; the graph is a regenerable index. Pages without `graph:` still appear as nodes (derived from `type`/`kind`) and still contribute `mentions` edges from body `[[wikilinks]]`.
```yaml
graph:
node_id: person:praney-behl # optional; default <node_type>:<slug>
node_type: person # optional; default mapped from type/kind via ontology
canonical: true # mark as canonical when multiple slugs alias the same entity
aliases: [Praney, praney@example.com]
relationships:
- predicate: founded
object: company:seedblocks
source: praney-founder-context-dump # source-page slug
evidence: "Solo technical founder and sole director..."
confidence: high # high | medium | low
status: current # current | historical | proposed | disputed | superseded
# optional:
# valid_from: 2025-01-15
# valid_to: 2026-03-01
# notes: "..."
# raw_ref: "raw/founder-dump.md#L42"
# contradicts: edge-id-or-source-slug
# supersedes: edge-id-or-source-slug
```
Required fields on every relationship: `predicate`, `object`, `source`, `evidence`, `confidence`, `status`. Predicates and the subject/object types they accept are declared in `wiki/graph/ontology.yaml`. Typed semantic edges must be supported by an explicit source — never emit one inferred from training data alone.
## Index structure
(Update this section when sharding.)
Currently flat: a single `wiki/index.md` listing all pages.
When the wiki passes ~150 pages or `index.md` exceeds 300 lines, shard into `wiki/indexes/<type>.md` and update this section.
## Graph layer
The wiki has an optional compiled graph layer under `wiki/graph/`:
- `wiki/graph/ontology.yaml` — declares node types and predicates. **Tracked.** Edit this when you introduce new predicates or domain types.
- `wiki/graph/nodes.jsonl`, `wiki/graph/edges.jsonl` — generated. Track in git only if you want graph diffs in PRs.
- `wiki/graph/graph.sqlite` — generated. Gitignored by default.
- `wiki/graph/graph.graphml` — generated. Track only if you want to diff it.
Generation is reproducible from markdown via `scripts/wiki_graph_extract.py`. The graph can be deleted at any time and rebuilt without losing knowledge — markdown is canonical.
## Workflow customizations
(Empty initially. Document any deviations from the default ingest/query/lint workflows here.)
## User preferences
(Empty initially. As the user expresses style preferences — "always include a 'Why this matters' section on concept pages", "never use bullet lists in summaries", "prefer comparative tables for synthesis pages" — capture them here so they persist across sessions.)
## Lint cadence
- Structural lint: after every 5 ingests.
- Semantic lint: weekly or after every 20 ingests.
- Gap-finding: monthly.
- Graph lint + extract: after every ingest that adds typed `graph.relationships`.
Adjust based on the wiki's growth rate.

32
skills/llm-wiki/assets/graph_README.md.template Normal file
View File

@@ -0,0 +1,32 @@
# Wiki Graph Layer
This directory holds the compiled knowledge graph derived from the markdown
wiki. **Markdown is canonical.** Everything here can be deleted and rebuilt
without losing knowledge:
```bash
python scripts/wiki_graph_extract.py wiki/ --out wiki/graph
```
## Files
| File | Purpose | Tracking |
|------|---------|----------|
| `ontology.yaml` | Declares node types and predicates the graph recognises. The contract `wiki_graph_lint.py` validates against. | **Tracked. Edit by hand.** |
| `nodes.jsonl` | One JSON object per node, sorted by id. | Generated. Track if you want graph diffs in PRs; otherwise gitignore. |
| `edges.jsonl` | One JSON object per edge, sorted by id. Includes typed semantic edges, `mentions`, `sourced_from`, and `summarizes_raw`. | Generated. Same trade-off as `nodes.jsonl`. |
| `graph.sqlite` | Queryable index used by `wiki_graph_query.py`. Schema: `nodes`, `aliases`, `edges`. | Generated. **Gitignored** — rebuild on demand. |
| `graph.graphml` | GraphML export for tools like Gephi or yEd. | Generated. Gitignored by default. |
## Workflow
1. Author or edit a wiki page. Add typed `graph.relationships` only when an explicit source supports them.
2. Run `python scripts/wiki_graph_lint.py wiki/` — catches unknown predicates, broken object references, missing evidence, alias collisions.
3. Run `python scripts/wiki_graph_extract.py wiki/ --out wiki/graph` — regenerates the artifacts above.
4. Query with `python scripts/wiki_graph_query.py wiki/ neighbors --node product:konvy` (or `edges`, `path`, `facts`).
## Anti-patterns
- **Hand-editing `nodes.jsonl` / `edges.jsonl` / `graph.sqlite`.** Edit the markdown; regenerate.
- **Treating graph rows as evidence.** They accelerate navigation. For high-stakes claims, follow the edge's `source` and `evidence` fields back to the wiki page and the raw source.
- **Adding typed edges the source doesn't support.** Use a normal `[[wikilink]]` instead — the `mentions` edge captures the connection without overclaiming.

2
skills/llm-wiki/assets/graph_gitignore.template Normal file
View File

@@ -0,0 +1,2 @@
graph.sqlite
graph.graphml

25
skills/llm-wiki/assets/index.md.template Normal file
View File

@@ -0,0 +1,25 @@
# Wiki Index
The catalog of all pages in this wiki. Each entry: a wikilink to the page and a one-line summary. The LLM reads this first when answering queries to identify candidate pages.
Keep summaries tight — one line each. The index is engineered to be cheap to read; a fat index defeats its purpose.
When this file exceeds ~300 lines or the wiki passes ~150 pages, shard into `wiki/indexes/<type>.md` and replace this file with a directory of shards. See the `scaling-playbook.md` reference in the `llm-wiki` skill for the migration procedure.
---
## Sources
(populated as sources are ingested)
## Entities
(populated as entity pages are created)
## Concepts
(populated as concept pages are created)
## Synthesis
(populated as query answers are filed back)

12
skills/llm-wiki/assets/log.md.template Normal file
View File

@@ -0,0 +1,12 @@
# Wiki Log
Append-only chronological record of operations on the wiki. Each entry begins with `## [YYYY-MM-DD] <op> | <description>` so it's parseable with `grep "^## \[" log.md | tail -N`.
Operations:
- `ingest` — a source was processed into the wiki.
- `query` — a question was answered against the wiki (typically only logged when the answer was filed back as synthesis).
- `lint` — a health check was run.
- `schema` — the schema was modified.
- `shard` — an index was sharded.
---

123
skills/llm-wiki/assets/ontology.yaml.template Normal file
View File

@@ -0,0 +1,123 @@
# Wiki Graph Ontology
#
# Declares the node types and predicates that the compiled graph layer
# (wiki/graph/) recognises. Edit this file when you introduce a new
# domain-specific predicate or node type — wiki_graph_lint.py reads it
# to validate every typed edge declared in page frontmatter.
#
# Markdown remains canonical. This file is just the contract that makes
# the graph layer machine-checkable.
node_types:
person:
maps_from:
type: entity
kind: person
company:
maps_from:
type: entity
kind: company
product:
maps_from:
type: entity
kind: product
paper:
maps_from:
type: entity
kind: paper
place:
maps_from:
type: entity
kind: place
organization:
maps_from:
type: entity
kind: organization
concept:
maps_from:
type: concept
source:
maps_from:
type: source
synthesis:
maps_from:
type: synthesis
decision:
explicit_only: true
claim:
explicit_only: true
raw:
explicit_only: true
predicates:
# --- Implicit predicates emitted by the extractor. ---
mentions:
subject_types: ["*"]
object_types: ["*"]
requires_evidence: false
description: |
Low-specificity edge derived from body wikilinks. Use it for
navigation, not as evidence of a typed relationship.
sourced_from:
subject_types: ["*"]
object_types: [source]
requires_evidence: false
description: |
Derived from each non-source page's frontmatter `sources:` list.
summarizes_raw:
subject_types: [source]
object_types: ["*"]
requires_evidence: false
description: |
Derived from a source page's frontmatter `raw:` field. Object is
the raw file path string, not a wiki node id.
# --- Typed semantic predicates. Add domain-specific ones below. ---
founded:
subject_types: [person]
object_types: [company, organization]
requires_evidence: true
owns:
subject_types: [person, company, organization]
object_types: [company, product, organization]
requires_evidence: true
contains_product:
subject_types: [company, organization]
object_types: [product]
requires_evidence: true
works_on:
subject_types: [person]
object_types: [product, concept]
requires_evidence: true
chose:
subject_types: [person, company, organization]
object_types: [product, concept]
requires_evidence: true
proposed:
subject_types: [person]
object_types: [decision, claim]
requires_evidence: true
competes_with:
subject_types: [product, company, organization]
object_types: [product, company, organization]
requires_evidence: true
depends_on:
subject_types: [product, concept]
object_types: [product, concept]
requires_evidence: true
authored:
subject_types: [person, organization]
object_types: [paper, source]
requires_evidence: true
cites:
subject_types: [paper, source, synthesis]
object_types: [paper, source]
requires_evidence: true
contradicts:
subject_types: [claim, source, synthesis]
object_types: [claim, source, synthesis]
requires_evidence: true
supersedes:
subject_types: [claim, source, decision]
object_types: [claim, source, decision]
requires_evidence: true

26
skills/llm-wiki/assets/page.md.template Normal file
View File

@@ -0,0 +1,26 @@
---
type: <source|entity|concept|synthesis>
title: ""
tags: []
sources: []
created: YYYY-MM-DD
updated: YYYY-MM-DD
---
# Title
Lead paragraph: a clear, encyclopedic definition or framing of what this page is about. Should answer "what is this and why does it matter" in one or two sentences.
## Section 1
Body content. Use `[[wikilinks]]` liberally to cross-reference other pages. (Frontmatter `sources:` list above uses bare slugs; only the body uses double-bracket wikilinks.)
## Section 2
More body content. Hedge claims that aren't yet corroborated by multiple sources ("Source X claims Y, though this is not yet corroborated by other sources in the wiki").
## Where this fits
(For source pages.) List the entity and concept pages this source touches:
- [[entity-page-1]]
- [[concept-page-1]]

76
skills/llm-wiki/references/architecture.md Normal file
View File

@@ -0,0 +1,76 @@
# Architecture
This document explains the three-layer / three-operation architecture in detail. The main `SKILL.md` summarises it; this is the reference you reach for when you need to know *why* a design decision exists or how to handle an edge case.
## The three layers
### Raw sources
Raw sources are the user's curated input material. They live in `cml/raw/` (or wherever the project's `SCHEMA.md` declares). They are **immutable** — the LLM reads from them but never modifies them. This immutability is load-bearing: it means the wiki can always be re-derived from the raw sources if it gets corrupted, and it gives the user a stable ground truth they can audit independently.
What goes in `cml/raw/`: PDFs, web articles converted to markdown (the Obsidian Web Clipper is one popular path), transcripts, code repos, dataset descriptions, screenshots, hand-typed notes the user wants the wiki to incorporate. What does *not* go in `cml/raw/`: anything the LLM generated — that all belongs in the wiki layer.
A useful convention is one source per file (or per directory if the source has multiple pieces, e.g. a paper plus its appendix), with a slugified filename that ends up matching the wiki's source-summary page name. This makes the back-pointer from the wiki to the raw source trivial.
### The wiki
The wiki is a directory of LLM-generated markdown files. The LLM owns this layer entirely — it creates pages, updates them when new sources arrive, maintains cross-references, and keeps everything consistent. The user reads it (typically through Obsidian, but any markdown viewer works); the LLM writes it.
The wiki directory is conventionally split into subdirectories by **page type**:
- `cml/wiki/sources/` — one summary page per ingested source. Captures what the source said, in the LLM's words, with a citation back to the raw file. These are append-mostly: you write one when you ingest a source and rarely modify it after.
- `cml/wiki/entities/` — pages about specific things: people, products, papers, places, companies, events. Anything that has a proper noun or could plausibly be a Wikipedia article subject. These accumulate updates as new sources mention the entity.
- `cml/wiki/concepts/` — pages about ideas, methods, frameworks, abstractions. These are the most heavily cross-referenced pages and the ones most likely to evolve as understanding deepens.
- `cml/wiki/synthesis/` — cross-cutting analyses, comparisons, query answers filed back. This is where exploration compounds: a comparison the user asked for becomes a page the next query can build on.
`SCHEMA.md` may declare additional types (e.g. `cml/wiki/decisions/` for an engineering team, `cml/wiki/characters/` for a fan wiki, `cml/wiki/experiments/` for a research lab). Adding a new type costs nothing — make the directory, document it in the schema, update the index template.
### The schema
`SCHEMA.md` lives at the wiki root and is **the configuration file** that turns a generic LLM into a disciplined wiki maintainer for *this specific* knowledge base. It documents the page types in use, the tag taxonomy, the naming conventions, any custom workflow steps, and the user's stylistic preferences (e.g. "always include a 'Why this matters' section on concept pages", "never use bullet lists in summaries").
The schema is **co-evolved** with the user. On bootstrap it starts from the default template in `assets/SCHEMA.md.template`, but every user will customize it as they discover what fits their domain. When you notice a recurring pattern in the user's edits or feedback that isn't in the schema, propose adding it. When the schema starts contradicting itself or growing unwieldy, propose pruning it.
The schema is the first file you read when entering an existing wiki. Its conventions override the defaults documented in the skill.
### The graph (optional, compiled)
The wiki may carry a fourth layer at `cml/wiki/graph/`: a compiled, queryable view of the typed `graph:` metadata in page frontmatter and the body wikilinks. It contains a hand-edited `ontology.yaml` (the contract: which node types and predicates exist) plus generated artifacts (`nodes.jsonl`, `edges.jsonl`, `graph.sqlite`, `graph.graphml`) produced by `wiki_graph_extract.py`. **Markdown is canonical**; the graph can be deleted and regenerated from the markdown without losing knowledge.
Its purpose is to make typed, provenance-backed relationships machine-queryable — "who founded what", "what does Konvy depend on", "shortest path from A to B" — without giving up the editability and human-legibility of markdown. Typed edges require an explicit `source` (a source-page slug) and `evidence` quote; the extractor never invents them. Plain `[[wikilinks]]` in the body produce low-confidence `mentions` edges, which are useful for navigation but not for evidence.
Use the graph layer when the user's questions are predominantly relational and the cost of maintaining typed metadata is paying for itself. Skip it for purely textual wikis. Full reference: `graph-workflow.md`.
## The three operations
### Ingest
A new source has arrived. The user has dropped a file in `cml/raw/` (or pasted content and asked you to file it). The job is to integrate this source into the wiki such that future queries can benefit from it.
The shape of an ingest: read the source (chunked if large), discuss the key takeaways with the user briefly, write a summary page in `cml/wiki/sources/`, identify which existing pages are touched, surgically update those pages, create new pages for any new entities or concepts, update the relevant index, and append to the log.
The temptation to skip the discussion step is strong, but resist it — the user's reaction to the takeaways often reveals what should be emphasized in the wiki versus left out. Ingest is not a batch import; it's a collaborative reading.
For the full procedure, see `ingest-workflow.md`.
### Query
The user asks a question. The job is to answer it from the wiki, with citations, and to file the answer back if it represents new synthesis.
The shape of a query: read the index to identify candidate pages; read those pages; if needed, follow `[[wikilinks]]` from those pages or grep for backlinks; synthesize the answer with `[[wikilink]]` citations; offer to file the answer into `cml/wiki/synthesis/` if it's substantive enough to be worth keeping.
The compounding effect of the wiki only works if good answers get filed back. A comparison you generated, a connection you discovered, an analysis you produced — these should not evaporate into chat history. Default to offering to file; let the user decline if the answer was too trivial or too transient.
For the full procedure, see `query-workflow.md`.
### Lint
Periodic health check. The job is to find structural and semantic problems before they compound.
Structural problems are mechanical and the bundled `wiki_lint.py` script catches them: orphan pages with no inbound links, broken `[[wikilinks]]` to nonexistent pages, oversized pages that need splitting, missing or malformed frontmatter, suspicious staleness (a page that hasn't been updated despite many recent ingests touching its topic).
Semantic problems need the LLM: contradictions between pages, claims that newer sources have superseded, concepts mentioned in many pages but lacking their own page, cross-references that should exist but don't, knowledge gaps the user might want to fill.
Lint findings are presented as proposed edits, not silent rewrites. The user approves changes. This is essential for trust — a wiki that mutates under the user is not a wiki the user can rely on.
For the full procedure, see `lint-workflow.md`.

126
skills/llm-wiki/references/graph-workflow.md Normal file
View File

@@ -0,0 +1,126 @@
# Graph Workflow
The graph layer is the optional **compiled index** over the markdown wiki. It does not replace the wiki — it sits alongside it under `cml/wiki/graph/` and is reproducible from the markdown at any time. The point is to make typed, provenance-backed relationships machine-queryable while keeping markdown canonical.
If `cml/wiki/graph/ontology.yaml` is absent, the wiki is pre-graph: don't run extract/lint/query and don't fabricate ontology files. Either propose adding the layer, or proceed without it.
## What the graph captures
Three classes of edge come out of an extract:
1. **Typed semantic edges** declared in a page's `graph.relationships[]` frontmatter. Examples: `founded`, `proposed`, `depends_on`. Each one carries an explicit `source` (a source-page slug), an `evidence` quote, a `confidence` (high/medium/low), and a `status` (current/historical/proposed/disputed/superseded). The extractor never invents these.
2. **`mentions` edges** — one per body `[[wikilink]]` (deduplicated per page). Confidence is `low`; they accelerate navigation but should not be cited as evidence of a typed relationship.
3. **`sourced_from`** edges — one per slug in a non-source page's frontmatter `sources:` list, pointing at the source page. **`summarizes_raw`** edges — one per source page's `raw:` field, with the raw file path as the (string-literal) object.
## Frontmatter schema
```yaml
graph:
node_id: person:praney-behl # optional; default <node_type>:<slug>
node_type: person # optional; default mapped from type/kind via ontology
canonical: true # mark canonical when multiple slugs alias the same entity
aliases: [Praney, praney@example.com]
relationships:
- predicate: founded
object: company:seedblocks
source: praney-founder-context-dump
evidence: "Solo technical founder and sole director..."
confidence: high
status: current
# optional:
# valid_from: 2025-01-15
# valid_to: 2026-03-01
# notes: "..."
# raw_ref: "cml/raw/founder-dump.md#L42"
# contradicts: <node-id-or-edge-id>
# supersedes: <node-id-or-edge-id>
```
Required relationship fields: `predicate`, `object`, `source`, `evidence`, `confidence`, `status`.
`node_id` format is `<node_type>:<slug>`. The default is derived from the page's `type`/`kind` via the ontology's `maps_from` block. `decision`, `claim`, and `raw` are explicit-only — they don't have wiki pages, so they only show up as edge objects. If a typed edge points at one of these, the `wiki_graph_lint.py` flag for "broken object reference" will fire until either (a) you create a page for it, or (b) you add it to the ontology with `explicit_only: true` and accept that lint will continue to flag the reference.
## The ontology
`cml/wiki/graph/ontology.yaml` is the contract. It declares:
- `node_types[*].maps_from` — how page `type`/`kind` projects onto a node type.
- `predicates[*]` — the allowed predicates, each with `subject_types`, `object_types`, and `requires_evidence`. `"*"` is a wildcard for either side.
Edit the ontology when you need a new domain predicate. Re-run `wiki_graph_lint.py` to validate; existing typed edges will be caught if they no longer match.
## When to add a typed edge vs a plain `[[wikilink]]`
Add a typed edge when:
- A specific source explicitly states the relationship.
- You can quote a snippet of evidence.
- The predicate is meaningful for downstream queries ("who founded what", "what did Stephanie propose").
Use a plain `[[wikilink]]` when:
- The relationship is implicit, atmospheric, or you're hedging.
- You cannot pin the claim to a single source quote.
- The predicate would be `mentions` anyway.
When in doubt, write the wikilink and skip the typed edge. The lint surfaces missing evidence; it does not punish under-claiming.
## Extract / lint / query loop
```bash
# Validate the typed metadata first; lint is conservative, never edits.
uv run skills/llm-wiki/scripts/wiki_graph_lint.py cml/wiki/
# Compile to nodes.jsonl, edges.jsonl, graph.sqlite, graph.graphml.
uv run skills/llm-wiki/scripts/wiki_graph_extract.py cml/wiki/
# Navigate.
uv run skills/llm-wiki/scripts/wiki_graph_query.py cml/wiki/ neighbors --node product:konvy
uv run skills/llm-wiki/scripts/wiki_graph_query.py cml/wiki/ edges --subject person:stephanie-emmanouel
uv run skills/llm-wiki/scripts/wiki_graph_query.py cml/wiki/ path --from person:praney-behl --to product:konvy
uv run skills/llm-wiki/scripts/wiki_graph_query.py cml/wiki/ facts --about product:konvy
```
`--json` works on both lint and query commands.
## Ingest workflow integration
After Step 6 of the standard ingest workflow (after surgical updates and source page creation), run:
1. If new typed edges were added on the page being ingested, run `wiki_graph_lint.py`. **Interactive:** triage findings with the user before extract. **Drain (headless):** if lint is clean, proceed to extract; if lint reports errors, record them in `log.md` and skip extract for this batch — never silently rewrite typed edges, never block waiting for a user.
2. Run `wiki_graph_extract.py` to refresh the compiled artifacts.
3. Append a sub-line under the ingest's `log.md` entry:
` graph: +N nodes, +M typed edges (predicates: founded, contains_product, ...)`
Skip extract if this ingest added no `graph:` metadata and created no new pages — the compiled artifacts are unchanged.
## Query workflow integration
When the user asks a question that smells relational ("what's connected to X", "who proposed Y", "trace the path from A to B"):
1. Read the index as usual.
2. If `cml/wiki/graph/graph.sqlite` exists and is fresher than the latest log entry, query it for typed edges around the candidate pages — `neighbors`, `edges`, `facts` are the most useful.
3. Read the wiki pages behind the relevant nodes/edges. Don't answer from graph rows alone for high-stakes claims; the `evidence` field is a hint, not the source of truth.
4. Cite with `[[wikilinks]]` to wiki pages, not graph rows.
If `graph.sqlite` is stale (older than the most recent ingest in `log.md`), use it as-is and note the staleness — do **not** regenerate inline. Extract is a compile/drain step; the query turn stays read-only, and the background drain refreshes the graph after each ingest.
## Generated artifact policy
| File | Canonical? | Default tracking |
|------|-----------|------------------|
| `cml/wiki/graph/ontology.yaml` | Yes — edit by hand | Tracked |
| `cml/wiki/graph/nodes.jsonl` | Generated | Optional — `cml/wiki/graph/.gitignore` does not ignore it; track if you want graph diffs in PRs |
| `cml/wiki/graph/edges.jsonl` | Generated | Same as above |
| `cml/wiki/graph/graph.sqlite` | Generated | **Gitignored** by default (large, binary) |
| `cml/wiki/graph/graph.graphml` | Generated | **Gitignored** by default |
The bootstrapped `cml/wiki/graph/.gitignore` ignores `graph.sqlite` and `graph.graphml`. Edit it if your team prefers different policy.
## Anti-patterns
- **Typed edges without evidence.** Defeats the entire point. Lint will flag them; do not silence.
- **Editing `nodes.jsonl` / `edges.jsonl` / `graph.sqlite` by hand.** Edit the markdown; regenerate.
- **Inventing ontology entries to make a typed edge "fit".** The ontology should reflect domain reality, not paper over a too-eager edge. Either add the predicate with proper `subject_types`/`object_types`, or use `mentions`.
- **Treating graph rows as evidence in answers.** Always cite the wiki page; the graph just told you which wiki page to read.
- **Forgetting to regenerate after an ingest.** The graph diverges silently. Tie extract to ingest in muscle memory.

131
skills/llm-wiki/references/ingest-workflow.md Normal file
View File

@@ -0,0 +1,131 @@
# Ingest Workflow
When a new source arrives, this is the procedure. The order matters — each step builds context for the next.
## Step 0: Check the schema
Before anything else, read `cml/wiki/SCHEMA.md`. The user may have customized the page-type structure, the tag taxonomy, the naming conventions, or the ingest workflow itself. Schema overrides everything documented here.
## Step 1: Place the raw source
If the source isn't already in `cml/raw/`, place it there. Use a slugified filename: lowercase, hyphens for spaces, no special characters, with the original extension. For web articles, save as `.md` (Obsidian Web Clipper output is ideal). For PDFs, keep the `.pdf`. For transcripts, save as `.md` or `.txt`.
The slug you pick here will become the slug of the source-summary page in `cml/wiki/sources/`, so make it descriptive and stable.
## Step 2: Read the source
For short sources (under ~5,000 words / ~25,000 tokens), read the whole thing in one pass.
For long sources (papers over ~30 pages, book chapters, multi-hour transcripts), **chunk-read**: read the table of contents or section headers first to build a mental map, then read sections sequentially, summarizing each section in working memory before moving to the next. Do not load the entire raw source into context at once if it would consume more than ~25% of your context window — that leaves no room for the rest of the operation.
For PDFs specifically, prefer the `pdf-reading` skill if available (it handles the chunking automatically). Otherwise extract text first with `pdftotext` or `pdfminer` and then chunk-read the extracted text.
For images embedded in the source: read the surrounding text first, then view only the images that the text suggests are load-bearing (a chart referenced in an argument, a diagram of a system, a figure the source explicitly walks through). Don't blindly load every image — many are decorative.
## Step 3: Discuss the takeaways with the user (interactive only)
**Skip this step entirely in the background drain** — the compile cron runs headless with no user present, so there is no one to discuss with, and the remaining steps must not depend on a user reaction.
When run interactively, do this briefly — three or four sentences. Surface what struck you as important, what was surprising, what connects to existing wiki content, and what's worth flagging. The user's reaction shapes the next steps. They might say "skip the methodology section, only the results matter for me" or "we already have a page on this — just update it" or "this contradicts the page on X, flag that prominently".
If there is no user to react (the background drain, or a hands-off "just process these 20 papers" batch), skip the discussion and be more conservative on the wiki edits — make smaller, safer updates and surface anything ambiguous in the log.
## Step 4: Identify what's touched
Before writing anything, do a survey pass against the wiki to determine the impact:
1. Read `cml/wiki/index.md` (or the relevant shard under `cml/wiki/indexes/`) to identify existing pages this source touches. Look for entity names, concept names, and topical overlap.
2. For each potentially-touched page, read the page to confirm. (Read, don't grep — the index summaries can mislead.)
3. List, in working memory: existing pages to update, new pages to create, contradictions to flag.
This survey is what prevents duplication. Without it, you'll create a new page on a topic that already has one under a slightly different name.
## Step 5: Write the source-summary page
Create `cml/wiki/sources/<source-slug>.md`. Use the page template (`assets/page.md.template`) as a starting point. Frontmatter should include at minimum:
```yaml
---
type: source
title: "Original title of the source"
authors: ["Author Name"]
url: "https://..." # if applicable
raw: "cml/raw/<source-slug>.<ext>"
ingested: 2026-04-15
tags: [tag1, tag2]
entities: [entity-page-1, entity-page-2]
concepts: [concept-page-1, concept-page-2]
---
```
Frontmatter list values are bare slugs — the `[[wikilink]]` syntax goes in the body, not in YAML.
The body should be the LLM's summary of the source — the key claims, the methodology if relevant, the conclusions, the open questions. Do not paraphrase the entire source; that defeats the purpose. Aim for a summary that captures what a future query would need to know without re-reading the raw file.
Keep the page atomic — under the 400-line soft cap. If the source is so dense that a single summary page can't capture it, split: one page per major section, each linking to the others, with a parent page that gives the overview.
End the body with a "Where this fits" section listing `[[wikilinks]]` to the entity and concept pages this source touches. This is the bidirectional link from source → existing structure.
## Step 6: Update touched pages
For each existing page identified in Step 4, surgically edit it to incorporate what the new source adds. Use `str_replace`, not full rewrites. The goal is to add a sentence or paragraph in the relevant section, with a `[[wikilinks]]` citation to the new source-summary page.
Common update patterns:
- A new source corroborates an existing claim → add a citation: `..., as established in [[paper-X]] and now corroborated by [[paper-Y]]`.
- A new source contradicts an existing claim → flag prominently: add a "## Contradictions" section if it doesn't exist, and document the contradiction with both sources cited. Do not silently overwrite the older claim.
- A new source adds a new dimension to an existing topic → add a new sub-section, don't dilute the existing prose.
- A new source mentions an entity or concept the page already discusses → update the `sources:` frontmatter and add the cross-reference where relevant in the body.
If updating a page would push it over the 800-line hard cap, that's the signal to split the page (extract the new dimension into its own page, link from the parent). Do that as part of the ingest, not as a deferred lint task.
## Step 7: Create new pages for new entities and concepts
For each new entity or concept the source introduces that doesn't have a page yet, decide whether it warrants its own page. Heuristic: if the source mentions it in passing and no other source is likely to expand on it, just mention it inline on a related page. If the source treats it as a first-class topic or you can foresee future sources building on it, create a page.
New pages need:
- Frontmatter with `type`, `tags`, `sources: [[[<source-slug>]]]`, `created`, `updated`.
- A body that introduces the entity/concept with what this source said about it.
- Inbound links — at least one existing page should `[[wikilink]]` to the new page, otherwise it's an instant orphan. Update the existing page to add the link.
A new page that nothing links to is a bug in the ingest, not just a lint finding.
## Step 8: Update the index
Add entries for any new pages to `cml/wiki/index.md` (or the appropriate shard). Each entry is one line: a wikilink to the page and a one-sentence summary. Keep the summary tight — the index is engineered to be cheap to read, and a fat index defeats the index-first navigation principle.
If `index.md` exceeds 300 lines after this update, that's the signal to shard. Do it now while the structure is fresh — see `scaling-playbook.md` for the procedure.
## Step 8b: Refresh the graph layer (only if `cml/wiki/graph/ontology.yaml` exists)
If the wiki has the optional graph layer:
1. Add typed `graph.relationships[]` only when the source explicitly supports them (predicate, source-page slug, evidence quote, confidence, status). When uncertain, prefer a plain `[[wikilink]]` in the body — the body wikilink already produces a `mentions` edge.
2. Run `uv run skills/llm-wiki/scripts/wiki_graph_lint.py cml/wiki/`. **Interactive:** triage findings with the user before extracting. **Drain (headless):** if lint is clean, proceed to extract; if lint reports errors, record them in `log.md` and skip extract for this batch. Never silently rewrite typed edges, and never block waiting for a user.
3. Run `uv run skills/llm-wiki/scripts/wiki_graph_extract.py cml/wiki/` to regenerate `nodes.jsonl`, `edges.jsonl`, `graph.sqlite`, `graph.graphml`.
Skip extract if this ingest added no `graph:` metadata and created no new pages — the compiled artifacts are unchanged. Full reference: `references/graph-workflow.md`.
## Step 9: Append to the log
One line in `cml/wiki/log.md`, with the prefix `## [YYYY-MM-DD] ingest | <source-title>`. Optionally add a sub-line listing the pages touched. If the graph layer was refreshed, add a second sub-line: ` graph: +N nodes, +M typed edges`. The log is parsed by simple unix tools (`grep "^## \[" log.md | tail -10`), so the prefix matters.
## Step 10: Close the loop with the user
Tell the user what you did, briefly: "Ingested. Created the source page and a new entity page for X; updated the concept pages for Y and Z. Flagged a contradiction with [[paper-A]] regarding the claim about W."
If the source revealed something worth following up on (an obvious gap, a question the source raised but didn't answer, a candidate next source to ingest), say so — this is where the wiki's compounding effect comes from.
## Anti-patterns to avoid
**Loading the whole source into context at once when it's large.** This is the most common scaling failure. Chunk-read.
**Rewriting whole pages instead of surgical edits.** This burns tokens, risks losing nuance, and erodes diff quality if the wiki is in git.
**Creating pages with no inbound links.** Orphans accumulate fast and become invisible. Always link.
**Ingesting silently in batch mode without surfacing surprises.** Batch ingest is fine, but a one-line "ingested 5 sources, 2 new entity pages, 1 contradiction with [[X]]" summary is the minimum.
**Treating prior wiki pages as ground truth instead of the raw sources.** When updating an existing claim, re-read the raw source for that claim before merging the new one. Don't compound on the wiki's own paraphrase.
**Letting the page split decision drift to a future lint pass.** If a page crossed the size cap during this ingest, split it during this ingest.

99
skills/llm-wiki/references/lint-workflow.md Normal file
View File

@@ -0,0 +1,99 @@
# Lint Workflow
A health check on the wiki. Best run on a cadence — after every N ingests, weekly, or when the user explicitly requests it — not on every operation. Lint is split into a structural pass (handled by `wiki_lint.py`) and a semantic pass (handled by the LLM directly).
> **Report-only gate (this nanobot).** A lint turn runs the read steps (0–1, 3–4), presents findings, and **stops**. The mutation steps below — Step 2 (apply fixes), Step 5 (index), Step 6 (log) — happen **only in a separate turn after the user approves**, one category at a time. Inside a lint turn, never create/edit pages, write debug scripts, regenerate the graph, or loop re-lint. When fixing later: bounded batches, do **not** re-read the lint script to reverse-engineer it, re-lint once to confirm — if issues remain, report and ask, don't continue blind. The scripts are sub-second; a long lint turn means you slipped into fixing.
## Step 0: Check the schema
`cml/wiki/SCHEMA.md` may declare additional lint rules specific to this wiki (e.g. "every entity page must have an `aliases:` field", "no concept page without at least 2 sources"). Read it.
## Step 1: Run the structural lint script
```bash
uv run skills/llm-wiki/scripts/wiki_lint.py cml/wiki/
```
If the wiki has the optional graph layer (`cml/wiki/graph/ontology.yaml` exists), also run:
```bash
uv run skills/llm-wiki/scripts/wiki_graph_lint.py cml/wiki/
```
This catches typed-edge problems independently of the structural lint: unknown predicates, missing evidence, broken object references, alias collisions, invalid `confidence`/`status` values, broken `contradicts`/`supersedes` references. Triage findings the same way as structural lint — propose fixes, don't apply them silently. After approved fixes, run `wiki_graph_extract.py` to refresh the compiled artifacts.
This produces a report covering:
- **Orphan pages** — pages with no inbound `[[wikilinks]]` from anywhere else in the wiki. Orphans are usually a sign that an ingest forgot to update a parent page. They become invisible because the index-first navigation can't surface them.
- **Broken wikilinks** — `[[page-name]]` references pointing to nonexistent pages. Usually a typo or a page that got renamed without updating its referrers.
- **Oversized pages** — anything over the 800-line hard cap (or 400-line soft cap, with a warning).
- **Frontmatter issues** — pages missing required fields (`type`, `tags`, `sources`, `updated`), or with malformed YAML.
- **Stale pages** — pages whose `updated:` date is much older than the most recent ingest that touched their topic. (The script approximates this using the page's tags and the log.)
- **Duplicate slugs** — two pages with the same slug in different subdirectories (a sign of an ingest collision that wasn't resolved).
The script is conservative — it reports findings but doesn't fix them. Present the report to the user.
## Step 2: Triage the structural findings (propose only; apply in a later approved turn)
Walk through the findings with the user and propose a fix for each. Proposing is part of the lint turn; **applying** the edits is not — that waits for approval (see the report-only gate above).
- Orphans: either link them from a sensible parent page (preferred), or determine that the page is genuinely useless and delete it (rare). If many orphans pile up, the index is probably out of date — re-derive the index entries.
- Broken links: rename the link to match the actual page, or create the missing page if it should exist, or remove the link if the concept turned out not to warrant a page.
- Oversized pages: split. Extract sub-concepts into their own pages, link from the parent, update the index.
- Frontmatter issues: add the missing fields. If many pages have the same gap, consider whether the schema should be relaxed or the bootstrap template improved.
- Stale pages: read the recently-touched related pages and the relevant raw sources, update the stale page surgically.
- Duplicate slugs: one is canonical, the other should be merged in and deleted. Pick the better-named one as canonical and migrate inbound links with `grep` + `str_replace`.
Present each proposed fix as an edit, not a fait accompli. The user approves.
## Step 3: Run the semantic pass
The semantic pass is what the script can't do — it requires reading pages and reasoning about content. The good news is that you don't have to read the whole wiki: focus on the pages most likely to have semantic issues.
**Recently updated pages.** Read the last ~10 pages that were modified (sorted by `updated:` frontmatter or by the log). Look for:
- Contradictions with older pages on the same topic. The new claim may have superseded the old one (in which case mark the old as superseded with a note), or both may be true but in different contexts (in which case clarify each), or the new ingest may have been wrong (in which case revert).
- Internal contradictions within a page (an ingest layered on a new claim without reconciling against existing prose).
- Repeated mentions of an entity or concept that doesn't have its own page yet — candidate for promotion.
**Highly-linked pages (hubs).** These are the most likely to drift because every ingest touches them. Use `wiki_search.py --top-linked 10` (or grep `[[wikilinks]]` and count) to find them. Read each hub and check that the prose still hangs together and the cross-references still make sense.
**Pages flagged with explicit uncertainty.** During ingest, the convention is to hedge ("the source claims X, though this is not yet corroborated") rather than assert. The lint pass is when you check whether subsequent ingests have corroborated or contradicted, and update accordingly.
## Step 4: Surface gaps
Look at what's referenced but not pageified. Run:
```bash
uv run skills/llm-wiki/scripts/wiki_lint.py cml/wiki/ --suggest-pages
```
This finds entity-like and concept-like names that appear in many pages but lack their own page. The user can decide which to promote and which to leave as inline mentions.
Also look for what's not covered at all — topics the user has expressed interest in but that haven't been ingested yet. The log can help here ("you ingested 5 sources on diffusion models in March but nothing since; want to refresh?").
## Step 5: Update the index (fix turn — only after approved fixes)
After lint fixes, the index almost certainly needs updates: new pages, renamed pages, deleted pages, changed summaries. Re-derive the affected index entries surgically.
If `index.md` is over 300 lines and hasn't been sharded yet, this is a good moment to do it. See `scaling-playbook.md`.
## Step 6: Append to the log (fix turn)
One line: `## [YYYY-MM-DD] lint | <N> structural fixes, <M> semantic fixes, <K> proposed gaps`. Optionally a sub-line listing the highest-impact changes.
## Cadence
A reasonable default: structural lint after every 5 ingests, semantic lint weekly or after every 20 ingests, gap-finding monthly. Adjust based on the user's pace and the wiki's volatility. A wiki that's growing fast needs more frequent lint; a stable mature wiki needs less.
The user may also trigger lint explicitly ("clean up the wiki", "what's broken", "lint pass please"). Treat these as priority — the user is asking because they noticed something.
## Anti-patterns to avoid
**Silent rewrites.** Lint findings are proposals. The user approves changes. A wiki that mutates without the user's knowledge stops being trustworthy.
**Treating lint as cleanup-only.** The semantic pass is also where new connections get made. If you read 10 pages and notice that two of them point at the same underlying idea, that's a synthesis-page candidate, not just a lint finding.
**Letting the lint report grow until it's overwhelming.** If the report is too long for the user to triage, the cadence is wrong (lint more often) or the wiki has outgrown its conventions (revisit the schema).
**Skipping lint because everything seems fine.** Silent corruption is the failure mode that's hardest to detect and most damaging. Lint catches it.

89
skills/llm-wiki/references/page-conventions.md Normal file
View File

@@ -0,0 +1,89 @@
# Page Conventions
The structural rules every wiki page follows. The schema may extend or override these for a particular wiki, but these are the defaults.
## Frontmatter
Every wiki page begins with YAML frontmatter. The required fields:
```yaml
---
type: <source|entity|concept|synthesis|...>
title: "Human-readable title"
tags: [tag1, tag2, tag3]
sources: [source-slug-1, source-slug-2]
created: 2026-04-15
updated: 2026-04-15
---
```
Note that frontmatter list values are **bare slugs**, not `[[wikilinks]]`. The double-bracket syntax is only used in the page body. The bundled scripts treat frontmatter `sources:`, `entities:`, and `concepts:` lists as slug references and resolve them the same way as body wikilinks.
`type` determines the page-type and which subdirectory the page lives in. Standard types are `source`, `entity`, `concept`, `synthesis`. The schema can declare additional types.
`title` is the human-readable name. The filename slug is separate and may be a short version of the title. For pages about people, the convention is `Last, First` for sortability; for everything else, natural casing.
`tags` is a flat list. Tags are how `wiki_search.py` filters and how the user navigates topically in Obsidian. Keep the tag taxonomy small and disciplined — a wiki with 200 tags has effectively no tags. The schema should declare the canonical tag set and the lint script will warn on tags outside it.
`sources` is the list of source-summary pages this page draws from. Always populated for entity/concept/synthesis pages; for source pages themselves, this field is omitted (the source page is the source).
`created` and `updated` are dates in ISO format. `updated` is the load-bearing one — it powers the staleness check in lint and the "what's new" view.
Type-specific fields:
- Source pages add `authors`, `url`, `raw` (path to the raw file), `ingested`.
- Entity pages may add `aliases` (other names for the same entity), `kind` (person, paper, product, etc.).
- Synthesis pages add `question` (the original question, if it was filed back from a query) and `sources_consulted`.
## Wikilinks
Cross-references use the `[[page-slug]]` syntax. The slug is the filename without the `.md` extension and without the directory prefix — Obsidian-style. Aliasing is supported: `[[page-slug|display text]]`.
Every page should have at least one inbound link. New pages without inbound links are orphans and the lint pass will flag them.
The bundled `wiki_search.py --backlinks <slug>` returns inbound links. To find them manually:
```bash
grep -rln "\[\[<slug>\]\]" cml/wiki/
```
## Page sizing
**Soft cap: 400 lines or ~2,000 words.** When a page approaches this, consider whether it should split.
**Hard cap: 800 lines.** Pages over this must split. The lint script flags violations.
Atomicity heuristic: a page is about *one* thing. A page on "Diffusion Models" should not also be the page on "Stable Diffusion" — those are two pages with cross-references. If you find yourself writing "## Variants" with five sub-sections of substantive prose, those sub-sections are probably their own pages.
The reason for the size cap is the context-bottleneck principle: any single page read is bounded, so the LLM can confidently read several pages without exhausting context.
## Naming
Page slugs are lowercase, hyphenated, no special characters. Match the directory: a concept page lives at `cml/wiki/concepts/<slug>.md`.
For entity pages about people: prefer the full name slugified (`andrej-karpathy.md`), not just the surname. Add `aliases:` to frontmatter for common short names.
For source pages: use a slug derived from the source title, possibly with the year for disambiguation (`attention-is-all-you-need-2017.md`). The source page slug should match the raw file slug if possible — that makes the back-pointer trivial.
For synthesis pages: derive from the question or the topic, not the date. `comparing-rag-vs-llm-wiki.md`, not `2026-04-15-question.md`. Date-based names don't surface usefully in the index.
## Body structure
The body has no rigid template — the schema may declare one for specific page types — but a few defaults work well:
**Source pages**: lead paragraph summarizing the source's main contribution. Sections for key claims, methodology (if relevant), conclusions, open questions. End with a "Where this fits" section listing the entity and concept pages this source touches.
**Entity pages**: lead paragraph defining the entity. Sections for relevant attributes (for a paper: authors, venue, key claims; for a person: affiliation, notable work; for a product: what it does, who makes it). A "Mentioned in" section is unnecessary — the backlink discovery handles that.
**Concept pages**: lead paragraph defining the concept clearly. Sections for the key formulation, variants, contested aspects, related concepts. Heavy use of `[[wikilinks]]` is expected — concept pages are the connective tissue of the wiki.
**Synthesis pages**: lead with the question (if filed from a query) or the framing. The body is the answer/analysis. End with the sources consulted as wikilinks.
## Hedging language
When a source claims something that hasn't been corroborated by other sources in the wiki, hedge: "Source X claims Y, though this is not yet corroborated by other sources in the wiki." The lint pass will revisit hedged claims as the wiki accumulates more sources, either upgrading them to confirmed or flagging them as contested.
When two sources contradict, document both: "Source X claims Y; source Z claims not-Y. The contradiction is unresolved." Do not silently pick a side.
## Keeping pages voice-neutral
The wiki is the LLM's voice, not the source author's voice and not the user's voice. Paraphrase rather than quote, except for short load-bearing phrases where the exact wording matters. Maintain a consistent, neutral, encyclopedic tone — close to a Wikipedia article in register, not a chat reply.

102
skills/llm-wiki/references/query-workflow.md Normal file
View File

@@ -0,0 +1,102 @@
# Query Workflow
The user is asking a question against the wiki. The job is to answer it from the wiki, with citations, scaling navigation to the wiki's size, and to file the answer back as a synthesis page when warranted.
## Step 0: Check the schema
Read `cml/wiki/SCHEMA.md` if you haven't this session. Some wikis declare query-specific conventions (e.g. "always answer with a comparison table when the question is comparative", "answers go in `cml/wiki/synthesis/qa/` not `cml/wiki/synthesis/`"). Schema overrides defaults.
## Step 1: Read the index
Always start at `cml/wiki/index.md`. If the index has been sharded into `cml/wiki/indexes/`, read the top-level `index.md` first to identify which shard(s) are relevant, then read those.
The index is engineered for this — one line per page with a tight summary. You should be able to identify candidate pages from the index alone in most cases. If a query touches multiple shards (e.g. "compare the methodologies in papers A and B"), read all relevant shards.
## Step 2: Identify candidate pages
From the index, build a short list of pages that look relevant to the query. Be selective — reading 30 pages to answer a question is a sign you've fallen back to brute-force search, which doesn't scale. If the index summaries don't disambiguate well, that's a signal that the index entries are too terse — note it for the next lint pass.
If the index doesn't surface good candidates (the query uses fuzzy or domain-specific language that doesn't match the index summaries), fall back to the search script:
```bash
uv run skills/llm-wiki/scripts/wiki_search.py "your query terms" --top 10
```
This returns the top-N pages by BM25 score, with optional filters on frontmatter (`--type concept`, `--tag llms`, `--since 2026-01-01`). Use the search script *as a fallback*, not as the default — index-first is cheaper and produces more interpretable results when it works.
## Step 2b: Graph-assisted lookup (only if `cml/wiki/graph/graph.sqlite` exists)
For relational questions ("what's connected to X", "who proposed Y", "trace the path from A to B"), query the compiled graph after the index pass and before reading pages:
```bash
uv run skills/llm-wiki/scripts/wiki_graph_query.py cml/wiki/ neighbors --node <node-id>
uv run skills/llm-wiki/scripts/wiki_graph_query.py cml/wiki/ facts --about <node-id>
```
Use the structured neighbors/facts to pick the right wiki pages to read — but never answer from graph rows alone for high-stakes claims. The graph accelerates navigation; the wiki page and its raw source remain the evidence. If `graph.sqlite` is older than the most recent `## [YYYY-MM-DD] ingest |` entry in `log.md`, use it as-is and note the staleness — do **not** run `wiki_graph_extract.py` inline. Extract is a compile-phase step; a query turn stays read-only and the background drain refreshes the graph after each ingest. Full reference: `references/graph-workflow.md`.
## Step 3: Read the candidate pages
Read each candidate page in full. While reading, note any `[[wikilinks]]` to other pages that look relevant — those are pre-curated leads. Follow the most promising ones, but don't recursively chase every link or you'll exhaust your context window on tangentially-relevant pages.
If a page references a source-summary page in its frontmatter and the answer hinges on what that source actually said, read the source-summary page too. Avoid going all the way back to the raw source unless the wiki summaries are clearly insufficient — the whole point of the wiki is that the synthesis is already done.
## Step 4: Find backlinks if needed
If the query is "what does my wiki say about X" or "where is X mentioned", and X has its own page, the inbound links are often more interesting than the page itself. Find them with:
```bash
grep -rl "\[\[<page-slug>\]\]" cml/wiki/
```
This is faster than reading pages to look for mentions. Note that the bundled `wiki_search.py` has a `--backlinks <slug>` mode that does this and returns ranked results.
## Step 5: Synthesize the answer
Write the answer in your own words, with `[[wikilink]]` citations to the wiki pages you used and (where helpful) `cml/raw/<file>` references to specific raw sources. The citations matter — they let the user verify the answer and follow up.
If the wiki contains contradicting claims, surface the contradiction explicitly rather than picking one and presenting it as settled. The wiki's value is partly in tracking what's known versus what's contested.
If the wiki has no relevant content for the query, say so plainly. Do not confabulate — that's the surest way to corrupt the wiki when the answer gets filed back. Instead, suggest sources the user could ingest to fill the gap.
## Step 6: Offer to file the answer back
If the synthesized answer represents new connection-making — a comparison the wiki didn't already contain, an analysis that pulls together threads from multiple pages, an answer to a recurring question — offer to file it as a synthesis page. The user will say no for trivial answers and yes for substantive ones. Default to offering.
The file-back creates `cml/wiki/synthesis/<answer-slug>.md` with frontmatter:
```yaml
---
type: synthesis
question: "the original question, verbatim or lightly cleaned"
asked: 2026-04-15
sources_consulted: [page-1, page-2, page-3]
tags: [...]
---
```
Body: the answer as you gave it to the user, possibly lightly edited for the wiki's voice. Add the new synthesis page to the relevant index. Append to `log.md` with prefix `## [YYYY-MM-DD] query | <question-summary>`.
A filed synthesis page is itself queryable — the next time the user asks an adjacent question, the synthesis page may be the most relevant candidate. This is how exploration compounds.
## Special query types
**"What's missing on topic X?"** — Read existing pages on X, identify open questions or unstated assumptions, and propose ingest candidates. This is essentially a per-topic micro-lint.
**"Compare X and Y"** — Read both pages, look for explicit comparison pages already in `synthesis/`, generate a comparison table or contrast prose. Strong file-back candidate.
**"Show me the timeline of X"** — Use `log.md` to reconstruct chronology of ingests touching X, supplement with `created`/`updated` frontmatter on relevant pages.
**"What did source X say about Y?"** — Read `cml/wiki/sources/<x>.md` for the source's summary; if it doesn't directly answer, read `cml/raw/<x>` (chunk-read if large).
**"Lint check on this answer"** — Before filing back, ask the user to verify a key claim by pointing to its raw source. Especially valuable for high-stakes wikis (medical, legal, financial).
## Anti-patterns to avoid
**Reading every page in the wiki to be safe.** This doesn't scale and produces vague answers. Trust the index; if the index fails, fix the index, don't bypass it.
**Citing the wiki without citing the underlying sources for hard claims.** The wiki page is a paraphrase; for any claim the user might need to verify, the citation should chain back to the raw source.
**Filing back trivial answers.** Not every Q&A is worth a permanent page. If the answer is a one-line lookup or restates an existing page, don't pollute synthesis/. The threshold is "would I want to find this when I ask a similar question in three months?"
**Confabulating when the wiki is silent.** Better to say "the wiki doesn't cover this" than to invent an answer that gets filed back as authoritative.

91
skills/llm-wiki/references/scaling-playbook.md Normal file
View File

@@ -0,0 +1,91 @@
# Scaling Playbook
Thresholds at which the wiki's structure needs to evolve, and the migration steps. The goal is to keep the wiki's context cost roughly constant per query as the wiki grows — the LLM should never need to read more pages or larger pages just because the wiki got bigger.
## The bottleneck
Naive LLM Wiki implementations break at scale because of a few specific failure modes, each of which has a structural fix:
- **The index file becomes too large to read cheaply.** Fix: shard the index by category.
- **Pages grow unboundedly as more sources mention them.** Fix: enforce the page size cap; split when violated.
- **Index summaries become too vague to disambiguate candidates.** Fix: tighten summaries; introduce frontmatter filtering via the search script.
- **Brute-force grep over the wiki replaces index-first navigation.** Fix: make the index actually useful and use the search script as the explicit fallback.
## Threshold 1: ~50 pages
Below this scale, you need almost no structure. A flat `cml/wiki/` directory with `index.md` and `log.md` is enough. The categorical subdirectories (`entities/`, `concepts/`, etc.) are still worth using from the start because they're free, but the index doesn't need sharding and the search script is overkill.
## Threshold 2: ~150 pages OR `index.md` over 300 lines
Time to **shard the index**. The migration:
1. Create `cml/wiki/indexes/` directory.
2. Split `index.md` by category into `indexes/sources.md`, `indexes/entities.md`, `indexes/concepts.md`, `indexes/synthesis.md` (and any custom types from the schema). Each shard is a list of pages of that type, with the same one-line summaries.
3. Rewrite the top-level `index.md` to be a directory of shards: each shard linked, with a one-line description of what's in it and a count.
4. Update the schema to document the sharded structure.
5. Update the ingest workflow in your working memory: now you update `indexes/<type>.md`, not `index.md` directly.
The top-level `index.md` should now be tiny — under 50 lines — and the shards are each bounded by the type-specific volume.
If a single shard later exceeds 300 lines (most likely `entities.md` or `concepts.md` if the wiki has a strong topical focus), shard *that* by sub-category: `indexes/entities-people.md`, `indexes/entities-papers.md`, etc. The principle generalizes.
## Threshold 3: ~300 pages
Time to introduce the **search script as a routine fallback**. Index navigation still works for direct lookups ("the page on diffusion models"), but fuzzy queries ("which papers discuss training stability") benefit from BM25 ranking.
`scripts/wiki_search.py` provides:
- `uv run skills/llm-wiki/scripts/wiki_search.py "query terms"` — top-N pages by BM25 score.
- `--type concept` — filter by frontmatter type.
- `--tag <tag>` — filter by tag.
- `--since 2026-01-01` — filter by `updated` date.
- `--backlinks <slug>` — find pages that link to a given page.
- `--top-linked N` — find the N most-linked-to pages (hubs).
Update the schema to declare the search script as a sanctioned fallback, so that future LLM sessions know to reach for it rather than degenerating into recursive grep.
## Threshold 4: ~500 pages
At this scale, two things start to matter:
**Structural lint cadence becomes weekly or per-N-ingests.** Manual oversight stops scaling. Rely on `wiki_lint.py` to surface structural drift and triage with the user.
**The search script may want a real index.** The default `wiki_search.py` rebuilds its BM25 index on every run, which is fine up to a few thousand pages. Beyond that, persist the index to disk (the script supports `--cache .wiki-search-cache.json`).
Also consider whether the wiki has organically split into distinct topic clusters that don't really cross-reference each other. If so, a single wiki may be the wrong shape — splitting into per-topic wikis (each with its own `SCHEMA.md`, `index.md`, etc.) may be cleaner. The user should make this call.
## Threshold 5: ~1,000+ pages
At this scale, the question is whether the LLM Wiki pattern is still the right tool. Markdown + grep + frontmatter scales further than people expect (the gist author reports a wiki of ~100 articles and ~400K words working fine), but at some point a real database with structured queries beats markdown. Signals it's time to consider migrating:
- The user's queries are predominantly relational ("show me all papers from author X cited by papers in topic Y published after date Z"). A graph database serves this better.
- Lint reports are too long to triage even at high cadence.
- The schema has grown to specify dozens of types and hundreds of tags — at that point you've manually built a database schema in markdown.
If the user wants to migrate, the markdown wiki is excellent input for the migration: every page has frontmatter and `[[wikilinks]]` that map cleanly to a property graph.
## When to *not* shard
Sharding is irreversible-ish (you can un-shard, but it's annoying), so don't do it preemptively. Wait for the actual threshold. A wiki of 80 pages with a sharded index has worse usability than the same wiki with a flat index, because the shards add a navigation step without enough volume to justify it.
## When to introduce custom page types
The default types (`source`, `entity`, `concept`, `synthesis`) cover most use cases. Add a custom type only when the user has a clear category of pages that don't fit any default and that benefits from being a distinct subdirectory (queryable, distinct lint rules, distinct templates). Examples that justify it:
- `decision` pages for a team that documents architectural decisions
- `experiment` pages for a research lab logging trial results
- `character` pages for a fan wiki tracking a fictional cast
- `meeting` pages for a team logging meetings
Don't add a type for a one-off — use tags instead. Adding a type is a schema change that affects the index, the lint script, and every future ingest.
## Detecting "we've outgrown our conventions"
Some signals that the schema needs revision rather than just more lint:
- The same kind of frontmatter issue keeps reappearing — the field needs to be optional, or the bootstrap template needs an example.
- Pages keep getting created that don't fit cleanly into any existing type — a new type is wanted.
- The tag list has grown unmanageable — prune, consolidate, or formalize the taxonomy.
- The user keeps overriding the LLM on a particular kind of decision — encode their preference in the schema so it persists across sessions.
Schema revision is healthy. A schema that doesn't change after the first few weeks of a wiki's life is probably not being used.

206
skills/llm-wiki/scripts/init_wiki.py Normal file
View File

@@ -0,0 +1,206 @@
#!/usr/bin/env -S uv run --script
# /// script
# requires-python = ">=3.11"
# dependencies = []
# ///
"""
init_wiki.py — Bootstrap or upgrade an LLM Wiki structure in a project.
Plain init creates the directory layout and drops in templates for SCHEMA.md,
index.md, log.md, the page template, and the optional graph layer
(graph/ontology.yaml, graph/README.md, graph/.gitignore). It is idempotent:
re-running won't clobber existing files.
`--upgrade` mode is for wikis bootstrapped under an older plugin version. It
does the same idempotent file creation, then inspects the existing SCHEMA.md
for sections introduced in newer versions and prints clear instructions for
what to merge by hand. It never overwrites SCHEMA.md — the schema is
co-evolved with the user.
Usage:
python init_wiki.py <project-root> [--wiki-dir wiki] [--raw-dir raw] [--upgrade]
Examples:
python init_wiki.py .
python init_wiki.py . --upgrade
python init_wiki.py ~/research --wiki-dir kb --raw-dir sources
"""
import argparse
import sys
from pathlib import Path
from datetime import date
SKILL_ROOT = Path(__file__).resolve().parent.parent
TEMPLATES = SKILL_ROOT / "assets"
SUBDIRS = ["sources", "entities", "concepts", "synthesis", "graph"]
# Markers used by --upgrade to detect SCHEMA.md sections introduced in
# specific plugin versions. Each entry: (heading_marker, version_label,
# template_anchor, blurb).
SCHEMA_SECTION_MARKERS = [
{
"marker": "## Optional graph metadata",
"version": "0.3.0",
"anchor": "## Optional graph metadata",
"label": "Optional graph metadata (Frontmatter section)",
},
{
"marker": "## Graph layer",
"version": "0.3.0",
"anchor": "## Graph layer",
"label": "Graph layer (canonical-vs-generated artifact policy)",
},
{
"marker": "Graph lint + extract",
"version": "0.3.0",
"anchor": "- Graph lint + extract: after every ingest that adds typed `graph.relationships`.",
"label": "Graph lint + extract cadence (Lint cadence section)",
},
]
def copy_template(src: Path, dst: Path, substitutions: dict | None = None) -> bool:
"""Copy a template file to dst. Returns True if file was created, False if it already existed."""
if dst.exists():
return False
text = src.read_text()
if substitutions:
for key, value in substitutions.items():
text = text.replace(key, value)
dst.parent.mkdir(parents=True, exist_ok=True)
dst.write_text(text)
return True
def detect_schema_gaps(schema_path: Path) -> list[dict]:
"""Return the SCHEMA_SECTION_MARKERS entries missing from the user's SCHEMA.md."""
if not schema_path.exists():
return []
text = schema_path.read_text(encoding="utf-8")
return [m for m in SCHEMA_SECTION_MARKERS if m["marker"] not in text]
def print_schema_upgrade_guidance(schema_path: Path, gaps: list[dict]) -> None:
template_path = TEMPLATES / "SCHEMA.md.template"
print()
print("=" * 64)
print(f"Upgrade required: {schema_path}")
print("=" * 64)
print(
"Your SCHEMA.md predates one or more sections introduced by newer\n"
"plugin versions. The graph layer itself is opt-in, but to make Claude\n"
"aware of it, merge the sections below by hand. SCHEMA.md is co-evolved\n"
"with you — this script never overwrites it."
)
print()
print("Missing sections:")
for m in gaps:
print(f" - [{m['version']}] {m['label']}")
print()
print(f"Reference template: {template_path}")
print(
"Diff your SCHEMA.md against the template and copy the missing\n"
"sections in. Or run /wiki:upgrade and Claude will propose the edits\n"
"interactively (one section at a time, never silent)."
)
def init_wiki(project_root: Path, wiki_dir: str, raw_dir: str, upgrade: bool = False) -> None:
project_root = project_root.resolve()
if not project_root.exists():
print(f"Error: project root does not exist: {project_root}", file=sys.stderr)
sys.exit(1)
wiki = project_root / wiki_dir
raw = project_root / raw_dir
mode = "Upgrading" if upgrade else "Initializing"
print(f"{mode} LLM Wiki in: {project_root}")
print(f" Wiki directory: {wiki}")
print(f" Raw directory: {raw}")
print()
created = []
skipped = []
# Create wiki subdirs
for subdir in SUBDIRS:
d = wiki / subdir
if not d.exists():
d.mkdir(parents=True)
created.append(f"{wiki_dir}/{subdir}/")
else:
skipped.append(f"{wiki_dir}/{subdir}/")
# Create raw + raw/assets
for d, label in [(raw, raw_dir), (raw / "assets", f"{raw_dir}/assets")]:
if not d.exists():
d.mkdir(parents=True)
created.append(f"{label}/")
else:
skipped.append(f"{label}/")
# Copy templates
template_map = [
("SCHEMA.md.template", wiki / "SCHEMA.md"),
("index.md.template", wiki / "index.md"),
("log.md.template", wiki / "log.md"),
("page.md.template", wiki / ".page-template.md"),
("ontology.yaml.template", wiki / "graph" / "ontology.yaml"),
("graph_README.md.template", wiki / "graph" / "README.md"),
("graph_gitignore.template", wiki / "graph" / ".gitignore"),
]
for src_name, dst in template_map:
src = TEMPLATES / src_name
if not src.exists():
print(f"Warning: template missing: {src}", file=sys.stderr)
continue
if copy_template(src, dst):
created.append(str(dst.relative_to(project_root)))
else:
skipped.append(str(dst.relative_to(project_root)))
# Report
if created:
print("Created:")
for path in created:
print(f" + {path}")
if skipped:
print("Already existed (skipped):")
for path in skipped:
print(f" = {path}")
if upgrade:
gaps = detect_schema_gaps(wiki / "SCHEMA.md")
if gaps:
print_schema_upgrade_guidance(wiki / "SCHEMA.md", gaps)
else:
print()
print("SCHEMA.md is up to date with the current template — no manual merge needed.")
return
print()
print("Next steps:")
print(f" 1. Read {wiki_dir}/SCHEMA.md and customize it for your domain.")
print(f" 2. (Optional) Edit {wiki_dir}/graph/ontology.yaml to add domain-specific predicates.")
print(f" 3. Drop your first source into {raw_dir}/.")
print(f" 4. Ask Claude to ingest it.")
def main():
parser = argparse.ArgumentParser(description=__doc__, formatter_class=argparse.RawDescriptionHelpFormatter)
parser.add_argument("project_root", type=Path, help="Project root directory.")
parser.add_argument("--wiki-dir", default="wiki", help="Name of the wiki subdirectory (default: wiki).")
parser.add_argument("--raw-dir", default="raw", help="Name of the raw sources subdirectory (default: raw).")
parser.add_argument("--upgrade", action="store_true",
help="Upgrade an existing wiki: add missing files idempotently and surface SCHEMA.md sections to merge by hand.")
args = parser.parse_args()
init_wiki(args.project_root, args.wiki_dir, args.raw_dir, upgrade=args.upgrade)
if __name__ == "__main__":
main()

169
skills/llm-wiki/scripts/wiki_compile.py Executable file
View File

@@ -0,0 +1,169 @@
#!/usr/bin/env -S uv run --script
# /// script
# requires-python = ">=3.11"
# dependencies = ["nanobot-ai"]
# ///
"""wiki_compile.py — dávkový compile nasbíraných zdrojů z cml/raw/ do cml/wiki/.
Spouštěn systémovým cronem každou minutu. Capture (interaktivní) hází zdroje do
cml/raw/ a hned potvrdí; těžký raw→wiki compile (čtení zdrojů, psaní stránek,
rozhodování) běží mimo interaktivní tah přes LLM agenta — tady, na pozadí.
Tok:
1. Levná pre-kontrola (BEZ LLM): jsou v cml/raw/ nezpracované zdroje
(regulérní soubory mimo _done/, _hard/, assets/)? Žádné → exit 0, agenta
vůbec neinstancuj.
2. Lockfile (cml/.compile.lock, PID + start-timestamp): běží jiný compile?
→ exit 0 (neduplikovat). Stale lock (mrtvý proces / > STALE_SECONDS) se
přebere, ať se to nezasekne po pádu.
3. Jinak Nanobot.from_config() + bot.run(<drain goal>) — vyprázdní VŠECHNO
nasbírané v jednom dávkovém běhu (jeden index/graph update pro víc zdrojů).
4. Tiše: jen append do log/wiki_compile_cron.log; žádný Telegram.
Vzor = skills/detach/scripts/tasks-daemon.py (shebang uv run, deps nanobot-ai,
Nanobot.from_config + asyncio.wait_for(bot.run(...), timeout)).
"""
import asyncio
import json
import os
import sys
import traceback
from datetime import datetime
from pathlib import Path
# Skript žije v workspace/skills/llm-wiki/scripts/ → parents[3] = workspace.
WORKSPACE = Path(__file__).resolve().parents[3]
CML = WORKSPACE / "cml"
RAW = CML / "raw"
LOCK = CML / ".compile.lock"
LOG = WORKSPACE / "log" / "wiki_compile_cron.log"
# Podadresáře v raw/, které NEjsou pending zdroje.
RESERVED_DIRS = {"_done", "_hard", "assets"}
TIMEOUT_SECONDS = 25 * 60
STALE_SECONDS = 30 * 60
DRAIN_GOAL = (
"Pomocí skillu llm-wiki (operace Compile/drain) zkompiluj VŠECHNY nezpracované zdroje "
"v `cml/raw/` (regulérní soubory přímo v `cml/raw/`, mimo `_done/`, `_hard/`, `assets/`) "
"do wiki v `cml/wiki/`. Pro každý zdroj proveď plný ingest podle "
"references/ingest-workflow.md: source/entity/concept stránky s frontmatterem a `[[odkazy]]`, "
"aktualizuj `cml/wiki/index.md` a `cml/wiki/log.md`. Po zpracování všech zdrojů regeneruj graph "
"(`wiki_graph_lint.py` + `wiki_graph_extract.py` na `cml/wiki/`). Každý úspěšně zpracovaný "
"zdroj přesuň do `cml/raw/_done/`. Ambiguózní/konfliktní zdroj NEcompiluj natvrdo — nech ho "
"v `cml/raw/` (nebo přesuň do `cml/raw/_hard/`) a důvod zaznamenej do `cml/wiki/log.md`. "
"Lint je report-only: žádné destruktivní úpravy existujících stránek bez potvrzení. "
"Idempotence: pokud pro zdroj už stránky existují (byl zkompilován dřív, jen nepřesunut), "
"NEcykluj reconciliací — ber ho jako hotový, přesuň raw soubor do `cml/raw/_done/` a pokračuj. "
"Každý vyřízený zdroj VŽDY přesuň z `cml/raw/` pryč, ať ho příští cron tik nezpracovává znovu. "
"Běžíš v izolované session na pozadí, bez interakce s uživatelem."
)
def log(message: str) -> None:
LOG.parent.mkdir(parents=True, exist_ok=True)
stamp = datetime.now().astimezone().isoformat(timespec="seconds")
with LOG.open("a", encoding="utf-8") as handle:
handle.write(f"{stamp} {message}\n")
def pending_sources() -> list[Path]:
"""Regulérní soubory přímo v cml/raw/ (mimo skryté a rezervované podadresáře)."""
if not RAW.exists():
return []
return [p for p in sorted(RAW.iterdir()) if p.is_file() and not p.name.startswith(".")]
def _pid_alive(pid: int) -> bool:
try:
os.kill(pid, 0)
except ProcessLookupError:
return False
except PermissionError:
return True
return True
def _lock_is_stale() -> bool:
"""Lock je mrtvý, když ho nelze přečíst, proces neběží, nebo je starší než STALE_SECONDS."""
try:
data = json.loads(LOCK.read_text())
pid = int(data["pid"])
started = datetime.fromisoformat(data["started"])
except (OSError, ValueError, KeyError):
return True
if not _pid_alive(pid):
return True
age = (datetime.now().astimezone() - started).total_seconds()
return age > STALE_SECONDS
def acquire_lock() -> bool:
"""Atomicky vytvoř lock. Vrať False, když už běží živý compile."""
for _ in range(2):
try:
fd = os.open(LOCK, os.O_CREAT | os.O_EXCL | os.O_WRONLY, 0o644)
except FileExistsError:
if not _lock_is_stale():
return False
log("stale lock, reclaiming")
LOCK.unlink(missing_ok=True)
continue
payload = {"pid": os.getpid(), "started": datetime.now().astimezone().isoformat()}
with os.fdopen(fd, "w", encoding="utf-8") as handle:
json.dump(payload, handle)
return True
return False
async def run_compile(goal: str) -> str:
# Heavy import deferred: the per-minute pre-check (no pending work) must not
# pay the nanobot import cost — only an actual compile run needs it.
from nanobot import Nanobot
bot = Nanobot.from_config()
result = await bot.run(goal, session_key="wiki-compile")
return result.content or ""
def main() -> int:
dry_run = "--dry-run" in sys.argv[1:]
pending = pending_sources()
if not pending:
return 0
if not acquire_lock():
log(f"SKIP compile already running ({len(pending)} pending)")
return 0
if dry_run:
names = ", ".join(p.name for p in pending)
log(f"DRY-RUN would compile {len(pending)} pending: {names}")
LOCK.unlink(missing_ok=True)
return 0
started = datetime.now().astimezone()
log(f"START compile {len(pending)} pending: {', '.join(p.name for p in pending)}")
try:
result_text = asyncio.run(
asyncio.wait_for(run_compile(DRAIN_GOAL), timeout=TIMEOUT_SECONDS)
)
summary = result_text.strip().splitlines()[0][:200] if result_text.strip() else "(prázdný výstup)"
duration = int((datetime.now().astimezone() - started).total_seconds())
log(f"END compile duration={duration}s remaining={len(pending_sources())} :: {summary}")
return 0
except asyncio.TimeoutError:
log(f"TIMEOUT compile po {TIMEOUT_SECONDS // 60} min")
return 1
except Exception as error:
log(f"EXCEPTION compile: {error}\n{traceback.format_exc()}")
return 1
finally:
LOCK.unlink(missing_ok=True)
if __name__ == "__main__":
sys.exit(main())

541
skills/llm-wiki/scripts/wiki_graph_extract.py Normal file
View File

@@ -0,0 +1,541 @@
#!/usr/bin/env -S uv run --script
# /// script
# requires-python = ">=3.11"
# dependencies = ["pyyaml"]
# ///
"""
wiki_graph_extract.py — Compile the markdown wiki into a queryable graph.
Markdown remains canonical. This script reads every wiki page, derives nodes
and edges (typed semantic edges from `graph.relationships`, plus implicit
`mentions`, `sourced_from`, `summarizes_raw` edges), and emits artifacts under
`<wiki>/graph/` that can be deleted and rebuilt at any time.
Requires PyYAML (`pip install pyyaml`) — the new graph layer uses real YAML
parsing for its nested frontmatter, unlike the stdlib-only lint/search/stats
scripts.
Usage:
python wiki_graph_extract.py <wiki-dir> [options]
Options:
--out <dir> Output directory (default: <wiki-dir>/graph)
--formats jsonl,sqlite,... Comma-list of formats to emit
(jsonl, sqlite, graphml; default: all three)
--ontology <path> Override ontology path
(default: <wiki-dir>/graph/ontology.yaml)
Examples:
python wiki_graph_extract.py wiki/
python wiki_graph_extract.py wiki/ --out wiki/graph --formats jsonl,sqlite
"""
import argparse
import hashlib
import json
import re
import sqlite3
import sys
import xml.etree.ElementTree as ET
from collections import defaultdict
from pathlib import Path
try:
import yaml
except ImportError:
print(
"wiki_graph_extract.py requires PyYAML.\n"
"Install with: pip install pyyaml",
file=sys.stderr,
)
sys.exit(2)
WIKILINK_RE = re.compile(r"\[\[([^\]|]+)(?:\|[^\]]+)?\]\]")
FRONTMATTER_RE = re.compile(r"^---\s*\n(.*?)\n---\s*\n", re.DOTALL)
SKIP_TOP_LEVEL_FILES = {"SCHEMA.md", "index.md", "log.md", "README.md"}
SKIP_TOP_LEVEL_DIRS = {"indexes", "graph"}
DEFAULT_FORMATS = ["jsonl", "sqlite", "graphml"]
# ---------------------------------------------------------------------------
# Page collection
# ---------------------------------------------------------------------------
def parse_frontmatter(text: str) -> tuple[dict, str]:
"""Extract YAML frontmatter using PyYAML. Returns (meta, body)."""
m = FRONTMATTER_RE.match(text)
if not m:
return {}, text
fm_text = m.group(1)
body = text[m.end():]
try:
meta = yaml.safe_load(fm_text) or {}
except yaml.YAMLError:
meta = {}
if not isinstance(meta, dict):
meta = {}
return meta, body
def collect_pages(wiki_root: Path) -> list[dict]:
pages = []
for md_path in sorted(wiki_root.rglob("*.md")):
rel = md_path.relative_to(wiki_root)
if rel.parts[0] in SKIP_TOP_LEVEL_FILES or rel.parts[0] in SKIP_TOP_LEVEL_DIRS:
continue
if rel.name.startswith("."):
continue
try:
text = md_path.read_text(encoding="utf-8")
except (UnicodeDecodeError, OSError):
continue
meta, body = parse_frontmatter(text)
links = [m.group(1).strip() for m in WIKILINK_RE.finditer(body)]
pages.append({
"path": str(md_path),
"rel_path": str(rel).replace("\\", "/"),
"slug": md_path.stem,
"meta": meta,
"body": body,
"links": links,
})
return pages
# ---------------------------------------------------------------------------
# Ontology
# ---------------------------------------------------------------------------
def load_ontology(path: Path) -> dict:
if not path.exists():
return {"node_types": {}, "predicates": {}}
try:
data = yaml.safe_load(path.read_text(encoding="utf-8")) or {}
except yaml.YAMLError as e:
print(f"Ontology parse error ({path}): {e}", file=sys.stderr)
sys.exit(2)
data.setdefault("node_types", {})
data.setdefault("predicates", {})
return data
def derive_node_type(meta: dict, ontology: dict) -> str | None:
"""Map a page's frontmatter to a node_type using ontology[node_types][*].maps_from."""
page_type = meta.get("type")
page_kind = meta.get("kind")
explicit = (meta.get("graph") or {}).get("node_type") if isinstance(meta.get("graph"), dict) else None
if explicit:
return explicit
# Try (type, kind) match first, then type-only.
type_kind_match = None
type_only_match = None
for nt_name, nt_def in ontology["node_types"].items():
maps = (nt_def or {}).get("maps_from") or {}
m_type = maps.get("type")
m_kind = maps.get("kind")
if m_type and m_type == page_type:
if m_kind and m_kind == page_kind:
type_kind_match = nt_name
break
if not m_kind and type_only_match is None:
type_only_match = nt_name
return type_kind_match or type_only_match
# ---------------------------------------------------------------------------
# Node + edge construction
# ---------------------------------------------------------------------------
def build_nodes(pages: list[dict], ontology: dict) -> tuple[list[dict], dict, list[dict]]:
"""Build the node list + slug→node_id index + alias rows. Returns (nodes, slug_to_id, aliases)."""
nodes: list[dict] = []
slug_to_id: dict[str, str] = {}
aliases: list[dict] = []
seen_ids: set[str] = set()
for p in pages:
meta = p["meta"]
graph_meta = meta.get("graph") if isinstance(meta.get("graph"), dict) else {}
node_type = derive_node_type(meta, ontology) or "concept"
explicit_id = graph_meta.get("node_id")
node_id = explicit_id or f"{node_type}:{p['slug']}"
# Skip duplicates — first one wins; lint will flag this.
if node_id in seen_ids:
continue
seen_ids.add(node_id)
node = {
"id": node_id,
"slug": p["slug"],
"title": meta.get("title") or p["slug"],
"page_type": meta.get("type") or "",
"node_type": node_type,
"kind": meta.get("kind") or "",
"tags": list(meta.get("tags") or []),
"aliases": list(graph_meta.get("aliases") or []),
"path": p["rel_path"],
"created": meta.get("created") or "",
"updated": meta.get("updated") or "",
"canonical": bool(graph_meta.get("canonical", False)),
}
nodes.append(node)
slug_to_id[p["slug"]] = node_id
for alias in node["aliases"]:
aliases.append({"alias": str(alias), "node_id": node_id})
return nodes, slug_to_id, aliases
def edge_id(subject: str, predicate: str, obj: str, source: str | None, evidence: str | None) -> str:
# Truncated to 96 bits — collision risk is negligible at any plausible
# wiki scale and shorter ids keep the JSONL/sqlite/graphml outputs readable.
h = hashlib.sha256()
parts = [subject or "", predicate or "", obj or "", source or "", evidence or ""]
h.update("\x1f".join(parts).encode("utf-8"))
return h.hexdigest()[:24]
def make_edge(*, subject, predicate, obj, source, evidence, confidence, status,
extraction_method, page, extras: dict | None = None) -> dict:
return {
"id": edge_id(subject, predicate, obj, source, evidence),
"subject": subject,
"predicate": predicate,
"object": obj,
"source": source or "",
"evidence": evidence or "",
"confidence": confidence or "",
"status": status or "",
"extraction_method": extraction_method,
"page": page,
"extras": extras or {},
}
def build_edges(pages: list[dict], slug_to_id: dict[str, str]) -> list[dict]:
edges: list[dict] = []
seen_ids: set[str] = set()
def push(edge: dict) -> None:
if edge["id"] in seen_ids:
return
seen_ids.add(edge["id"])
edges.append(edge)
for p in pages:
slug = p["slug"]
subject_id = slug_to_id.get(slug)
if not subject_id:
continue
meta = p["meta"]
graph_meta = meta.get("graph") if isinstance(meta.get("graph"), dict) else {}
# 1. Typed semantic edges from graph.relationships[].
for rel in graph_meta.get("relationships") or []:
if not isinstance(rel, dict):
continue
obj = rel.get("object")
predicate = rel.get("predicate")
if not (obj and predicate):
continue
extras = {
k: rel[k] for k in ("valid_from", "valid_to", "notes", "raw_ref",
"contradicts", "supersedes")
if k in rel and rel[k] is not None
}
push(make_edge(
subject=subject_id,
predicate=str(predicate),
obj=str(obj),
source=rel.get("source"),
evidence=rel.get("evidence"),
confidence=rel.get("confidence"),
status=rel.get("status"),
extraction_method="explicit_graph_frontmatter",
page=p["rel_path"],
extras=extras,
))
# 2. Mentions edges from body wikilinks.
seen_targets: set[str] = set()
for link in p["links"]:
target_slug = link.split("#")[0].strip()
if not target_slug or target_slug == slug:
continue
target_id = slug_to_id.get(target_slug)
if not target_id or target_id in seen_targets:
continue
seen_targets.add(target_id)
push(make_edge(
subject=subject_id,
predicate="mentions",
obj=target_id,
source=None,
evidence=None,
confidence="low",
status="current",
extraction_method="body_wikilink",
page=p["rel_path"],
))
# 3. sourced_from edges from frontmatter `sources:` (skip on source pages themselves).
if meta.get("type") != "source":
for src_slug in meta.get("sources") or []:
src_id = slug_to_id.get(str(src_slug))
if not src_id:
continue
push(make_edge(
subject=subject_id,
predicate="sourced_from",
obj=src_id,
source=str(src_slug),
evidence=None,
confidence="high",
status="current",
extraction_method="frontmatter_sources",
page=p["rel_path"],
))
# 4. summarizes_raw edges from source pages' raw: field.
if meta.get("type") == "source":
raw_path = meta.get("raw")
if raw_path:
push(make_edge(
subject=subject_id,
predicate="summarizes_raw",
obj=f"raw:{raw_path}",
source=None,
evidence=None,
confidence="high",
status="current",
extraction_method="frontmatter_raw",
page=p["rel_path"],
))
return edges
# ---------------------------------------------------------------------------
# Output writers
# ---------------------------------------------------------------------------
def _normalize_for_json(value):
if hasattr(value, "isoformat"):
return value.isoformat()
if isinstance(value, list):
return [_normalize_for_json(v) for v in value]
if isinstance(value, dict):
return {k: _normalize_for_json(v) for k, v in value.items()}
return value
def write_jsonl(out_dir: Path, nodes: list[dict], edges: list[dict]) -> None:
nodes_sorted = sorted(nodes, key=lambda n: n["id"])
edges_sorted = sorted(edges, key=lambda e: e["id"])
with (out_dir / "nodes.jsonl").open("w", encoding="utf-8") as f:
for n in nodes_sorted:
f.write(json.dumps(_normalize_for_json(n), sort_keys=True, ensure_ascii=False))
f.write("\n")
with (out_dir / "edges.jsonl").open("w", encoding="utf-8") as f:
for e in edges_sorted:
f.write(json.dumps(_normalize_for_json(e), sort_keys=True, ensure_ascii=False))
f.write("\n")
def write_sqlite(out_dir: Path, nodes: list[dict], aliases: list[dict], edges: list[dict]) -> None:
db_path = out_dir / "graph.sqlite"
if db_path.exists():
db_path.unlink()
conn = sqlite3.connect(db_path)
try:
conn.executescript("""
CREATE TABLE nodes (
id TEXT PRIMARY KEY,
slug TEXT NOT NULL UNIQUE,
title TEXT NOT NULL,
page_type TEXT NOT NULL,
node_type TEXT NOT NULL,
kind TEXT,
path TEXT NOT NULL,
created TEXT,
updated TEXT,
metadata_json TEXT NOT NULL
);
CREATE TABLE aliases (
alias TEXT NOT NULL,
node_id TEXT NOT NULL,
PRIMARY KEY (alias, node_id),
FOREIGN KEY (node_id) REFERENCES nodes(id)
);
CREATE TABLE edges (
id TEXT PRIMARY KEY,
subject TEXT NOT NULL,
predicate TEXT NOT NULL,
object TEXT NOT NULL,
source TEXT,
evidence TEXT,
confidence TEXT,
status TEXT,
extraction_method TEXT NOT NULL,
page TEXT NOT NULL,
metadata_json TEXT NOT NULL
);
CREATE INDEX idx_edges_subject ON edges(subject);
CREATE INDEX idx_edges_object ON edges(object);
CREATE INDEX idx_edges_predicate ON edges(predicate);
CREATE INDEX idx_edges_source ON edges(source);
""")
for n in sorted(nodes, key=lambda n: n["id"]):
metadata_json = json.dumps(_normalize_for_json({
"tags": n.get("tags", []),
"aliases": n.get("aliases", []),
"canonical": n.get("canonical", False),
}), sort_keys=True, ensure_ascii=False)
conn.execute(
"INSERT INTO nodes (id, slug, title, page_type, node_type, kind, path, created, updated, metadata_json) "
"VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, ?)",
(
n["id"], n["slug"], n["title"], n["page_type"], n["node_type"],
n.get("kind") or None, n["path"],
str(n.get("created") or "") or None,
str(n.get("updated") or "") or None,
metadata_json,
),
)
for a in sorted(aliases, key=lambda a: (a["alias"], a["node_id"])):
conn.execute(
"INSERT OR IGNORE INTO aliases (alias, node_id) VALUES (?, ?)",
(a["alias"], a["node_id"]),
)
for e in sorted(edges, key=lambda e: e["id"]):
metadata_json = json.dumps(_normalize_for_json(e.get("extras") or {}),
sort_keys=True, ensure_ascii=False)
conn.execute(
"INSERT INTO edges (id, subject, predicate, object, source, evidence, "
"confidence, status, extraction_method, page, metadata_json) "
"VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?)",
(
e["id"], e["subject"], e["predicate"], e["object"],
e.get("source") or None, e.get("evidence") or None,
e.get("confidence") or None, e.get("status") or None,
e["extraction_method"], e["page"], metadata_json,
),
)
conn.commit()
finally:
conn.close()
def write_graphml(out_dir: Path, nodes: list[dict], edges: list[dict]) -> None:
ns = "http://graphml.graphdrawing.org/xmlns"
ET.register_namespace("", ns)
root = ET.Element(f"{{{ns}}}graphml")
keys = [
("d_title", "node", "title", "string"),
("d_node_type", "node", "node_type", "string"),
("d_page_type", "node", "page_type", "string"),
("d_path", "node", "path", "string"),
("d_predicate", "edge", "predicate", "string"),
("d_confidence", "edge", "confidence", "string"),
("d_status", "edge", "status", "string"),
("d_source", "edge", "source", "string"),
]
for kid, kfor, kname, ktype in keys:
k = ET.SubElement(root, f"{{{ns}}}key")
k.set("id", kid)
k.set("for", kfor)
k.set("attr.name", kname)
k.set("attr.type", ktype)
graph = ET.SubElement(root, f"{{{ns}}}graph")
graph.set("id", "wiki")
graph.set("edgedefault", "directed")
for n in sorted(nodes, key=lambda n: n["id"]):
node_el = ET.SubElement(graph, f"{{{ns}}}node")
node_el.set("id", n["id"])
for kid, kfor, kname, _ in keys:
if kfor != "node":
continue
data = ET.SubElement(node_el, f"{{{ns}}}data")
data.set("key", kid)
data.text = str(n.get(kname) or "")
for e in sorted(edges, key=lambda e: e["id"]):
edge_el = ET.SubElement(graph, f"{{{ns}}}edge")
edge_el.set("id", e["id"])
edge_el.set("source", e["subject"])
edge_el.set("target", e["object"])
for kid, kfor, kname, _ in keys:
if kfor != "edge":
continue
data = ET.SubElement(edge_el, f"{{{ns}}}data")
data.set("key", kid)
data.text = str(e.get(kname) or "")
tree = ET.ElementTree(root)
ET.indent(tree, space=" ")
tree.write(out_dir / "graph.graphml", encoding="utf-8", xml_declaration=True)
# ---------------------------------------------------------------------------
# CLI
# ---------------------------------------------------------------------------
def main():
parser = argparse.ArgumentParser(description=__doc__, formatter_class=argparse.RawDescriptionHelpFormatter)
parser.add_argument("wiki", type=Path, help="Wiki directory.")
parser.add_argument("--out", type=Path, help="Output directory (default: <wiki>/graph)")
parser.add_argument("--formats", default=",".join(DEFAULT_FORMATS),
help="Comma-list: jsonl, sqlite, graphml")
parser.add_argument("--ontology", type=Path, help="Ontology file (default: <wiki>/graph/ontology.yaml)")
args = parser.parse_args()
if not args.wiki.exists():
print(f"Wiki directory not found: {args.wiki}", file=sys.stderr)
sys.exit(1)
out_dir = args.out or (args.wiki / "graph")
out_dir.mkdir(parents=True, exist_ok=True)
ontology_path = args.ontology or (args.wiki / "graph" / "ontology.yaml")
ontology = load_ontology(ontology_path)
formats = [f.strip().lower() for f in args.formats.split(",") if f.strip()]
unknown = [f for f in formats if f not in DEFAULT_FORMATS]
if unknown:
print(f"Unknown formats: {unknown}. Allowed: {DEFAULT_FORMATS}", file=sys.stderr)
sys.exit(1)
pages = collect_pages(args.wiki)
nodes, slug_to_id, aliases = build_nodes(pages, ontology)
edges = build_edges(pages, slug_to_id)
if "jsonl" in formats:
write_jsonl(out_dir, nodes, edges)
if "sqlite" in formats:
write_sqlite(out_dir, nodes, aliases, edges)
if "graphml" in formats:
write_graphml(out_dir, nodes, edges)
print(f"Extracted {len(nodes)} nodes, {len(edges)} edges → {out_dir}")
breakdown = defaultdict(int)
for e in edges:
breakdown[e["predicate"]] += 1
for pred, count in sorted(breakdown.items(), key=lambda x: (-x[1], x[0])):
print(f" {pred:20s} {count}")
if __name__ == "__main__":
main()

418
skills/llm-wiki/scripts/wiki_graph_lint.py Normal file
View File

@@ -0,0 +1,418 @@
#!/usr/bin/env -S uv run --script
# /// script
# requires-python = ">=3.11"
# dependencies = ["pyyaml"]
# ///
"""
wiki_graph_lint.py — Validate the typed graph metadata in a wiki.
Reads every page's `graph:` frontmatter, cross-checks against the ontology
(`wiki/graph/ontology.yaml`), and reports problems. Conservative by design:
reports only, never edits.
Requires PyYAML (`pip install pyyaml`).
Checks:
- Unique `graph.node_id` values across pages.
- All relationship `object` ids resolve to known nodes (or are allowed
string-literal targets for predicates whose object_types include "*").
- All predicates exist in `graph/ontology.yaml`.
- Predicate subject/object types match ontology.
- Typed semantic edges (anything except mentions/sourced_from/summarizes_raw
and predicates with `requires_evidence: false`) carry `source` and
`evidence`.
- `source` references resolve to an existing source page.
- `confidence` is one of high|medium|low; `status` is one of
current|historical|proposed|disputed|superseded.
- No duplicate canonical nodes for the same node id.
- Aliases do not collide across distinct canonical nodes.
- `contradicts` / `supersedes` references resolve to known node/edge ids.
- Generated graph has no orphan typed nodes (nodes with no inbound or
outbound typed edges) except for `source` nodes (allowed source-only).
Usage:
python wiki_graph_lint.py [<wiki-dir>] [--json]
"""
import argparse
import json
import re
import sys
from collections import defaultdict
from pathlib import Path
try:
import yaml
except ImportError:
print(
"wiki_graph_lint.py requires PyYAML.\n"
"Install with: pip install pyyaml",
file=sys.stderr,
)
sys.exit(2)
# Same module is imported by extract; we re-use its build_nodes/build_edges to
# guarantee lint sees exactly what extract would emit.
SCRIPT_DIR = Path(__file__).resolve().parent
sys.path.insert(0, str(SCRIPT_DIR))
import wiki_graph_extract as _extract # noqa: E402
FRONTMATTER_RE = re.compile(r"^---\s*\n(.*?)\n---\s*\n", re.DOTALL)
WIKILINK_RE = re.compile(r"\[\[([^\]|]+)(?:\|[^\]]+)?\]\]")
SKIP_TOP_LEVEL_FILES = {"SCHEMA.md", "index.md", "log.md", "README.md"}
SKIP_TOP_LEVEL_DIRS = {"indexes", "graph"}
ALLOWED_CONFIDENCE = {"high", "medium", "low"}
ALLOWED_STATUS = {"current", "historical", "proposed", "disputed", "superseded"}
IMPLICIT_PREDICATES = {"mentions", "sourced_from", "summarizes_raw"}
def parse_frontmatter(text: str) -> tuple[dict, str]:
m = FRONTMATTER_RE.match(text)
if not m:
return {}, text
fm_text = m.group(1)
body = text[m.end():]
try:
meta = yaml.safe_load(fm_text) or {}
except yaml.YAMLError:
meta = {}
if not isinstance(meta, dict):
meta = {}
return meta, body
def collect_pages(wiki_root: Path) -> list[dict]:
pages = []
for md_path in sorted(wiki_root.rglob("*.md")):
rel = md_path.relative_to(wiki_root)
if rel.parts[0] in SKIP_TOP_LEVEL_FILES or rel.parts[0] in SKIP_TOP_LEVEL_DIRS:
continue
if rel.name.startswith("."):
continue
try:
text = md_path.read_text(encoding="utf-8")
except (UnicodeDecodeError, OSError):
continue
meta, body = parse_frontmatter(text)
pages.append({
"path": str(md_path),
"rel_path": str(rel).replace("\\", "/"),
"slug": md_path.stem,
"meta": meta,
"body": body,
"links": [m.group(1).strip() for m in WIKILINK_RE.finditer(body)],
})
return pages
def derive_node_type(meta: dict, ontology: dict) -> str | None:
page_type = meta.get("type")
page_kind = meta.get("kind")
explicit = (meta.get("graph") or {}).get("node_type") if isinstance(meta.get("graph"), dict) else None
if explicit:
return explicit
type_kind_match = None
type_only_match = None
for nt_name, nt_def in ontology.get("node_types", {}).items():
maps = (nt_def or {}).get("maps_from") or {}
m_type = maps.get("type")
m_kind = maps.get("kind")
if m_type and m_type == page_type:
if m_kind and m_kind == page_kind:
type_kind_match = nt_name
break
if not m_kind and type_only_match is None:
type_only_match = nt_name
return type_kind_match or type_only_match
def derive_node_id(meta: dict, slug: str, ontology: dict) -> str:
graph_meta = meta.get("graph") if isinstance(meta.get("graph"), dict) else {}
explicit = graph_meta.get("node_id")
if explicit:
return str(explicit)
node_type = derive_node_type(meta, ontology) or "concept"
return f"{node_type}:{slug}"
def types_match(allowed: list[str] | None, actual: str | None) -> bool:
if not allowed:
return True
if "*" in allowed:
return True
return actual in allowed
def lint(pages: list[dict], ontology: dict) -> dict:
findings = {
"duplicate_node_ids": [],
"unknown_predicates": [],
"broken_object_refs": [],
"subject_type_mismatch": [],
"object_type_mismatch": [],
"missing_evidence": [],
"missing_source_field": [],
"broken_source_refs": [],
"invalid_confidence": [],
"invalid_status": [],
"duplicate_canonical": [],
"alias_collisions": [],
"broken_contradicts": [],
"broken_supersedes": [],
"orphan_typed_nodes": [],
"summary": {},
}
predicates = ontology.get("predicates", {})
node_types = ontology.get("node_types", {})
# Build node index
node_by_id: dict[str, dict] = {}
duplicates: dict[str, list[str]] = defaultdict(list)
for p in pages:
nid = derive_node_id(p["meta"], p["slug"], ontology)
if nid in node_by_id:
duplicates[nid].append(p["rel_path"])
duplicates[nid].append(node_by_id[nid]["rel_path"])
continue
node_type = derive_node_type(p["meta"], ontology) or "concept"
graph_meta = p["meta"].get("graph") if isinstance(p["meta"].get("graph"), dict) else {}
node_by_id[nid] = {
"id": nid,
"node_type": node_type,
"rel_path": p["rel_path"],
"slug": p["slug"],
"page_type": p["meta"].get("type"),
"canonical": bool(graph_meta.get("canonical", False)),
"aliases": list(graph_meta.get("aliases") or []),
"graph": graph_meta,
}
for nid, paths in duplicates.items():
findings["duplicate_node_ids"].append({"node_id": nid, "paths": sorted(set(paths))})
# Source pages by slug — used to validate `source:` refs on edges.
source_slugs = {p["slug"] for p in pages if p["meta"].get("type") == "source"}
# Aliases
alias_to_canonicals: dict[str, set[str]] = defaultdict(set)
canonical_by_id: dict[str, list[str]] = defaultdict(list)
for n in node_by_id.values():
if n["canonical"]:
canonical_by_id[n["id"]].append(n["rel_path"])
for alias in n["aliases"]:
alias_to_canonicals[str(alias)].add(n["id"])
for nid, paths in canonical_by_id.items():
if len(paths) > 1:
findings["duplicate_canonical"].append({"node_id": nid, "paths": paths})
for alias, owners in alias_to_canonicals.items():
if len(owners) > 1:
findings["alias_collisions"].append({"alias": alias, "owners": sorted(owners)})
# Walk relationships
for p in pages:
graph_meta = p["meta"].get("graph") if isinstance(p["meta"].get("graph"), dict) else {}
subject_id = derive_node_id(p["meta"], p["slug"], ontology)
subject_type = node_by_id.get(subject_id, {}).get("node_type")
for idx, rel in enumerate(graph_meta.get("relationships") or []):
if not isinstance(rel, dict):
continue
predicate = rel.get("predicate")
obj = rel.get("object")
here = {"page": p["rel_path"], "predicate": predicate,
"object": obj, "index": idx}
if not predicate or predicate not in predicates:
findings["unknown_predicates"].append({**here})
continue
pdef = predicates[predicate] or {}
# Object resolution. Allow string-literal objects only when
# ontology lists "*" in object_types (e.g. summarizes_raw).
object_types = pdef.get("object_types") or []
allows_wildcard_obj = "*" in object_types
if obj and obj not in node_by_id:
if not allows_wildcard_obj:
findings["broken_object_refs"].append({**here})
# Subject type check
if not types_match(pdef.get("subject_types"), subject_type):
findings["subject_type_mismatch"].append({
**here,
"subject": subject_id,
"subject_type": subject_type,
"allowed": pdef.get("subject_types"),
})
# Object type check (only if object resolves to a node)
obj_node = node_by_id.get(obj) if obj else None
obj_type = obj_node["node_type"] if obj_node else None
if obj_node and not types_match(pdef.get("object_types"), obj_type):
findings["object_type_mismatch"].append({
**here,
"object_type": obj_type,
"allowed": pdef.get("object_types"),
})
requires_evidence = pdef.get("requires_evidence", True)
if requires_evidence:
if not rel.get("evidence"):
findings["missing_evidence"].append({**here})
if not rel.get("source"):
findings["missing_source_field"].append({**here})
# source field must reference an existing source page slug
src = rel.get("source")
if src and str(src) not in source_slugs:
findings["broken_source_refs"].append({**here, "source": src})
confidence = rel.get("confidence")
if confidence and confidence not in ALLOWED_CONFIDENCE:
findings["invalid_confidence"].append({**here, "confidence": confidence})
status = rel.get("status")
if status and status not in ALLOWED_STATUS:
findings["invalid_status"].append({**here, "status": status})
# contradicts / supersedes resolution
for ref_field, bucket in (("contradicts", "broken_contradicts"),
("supersedes", "broken_supersedes")):
ref = rel.get(ref_field)
if ref:
ref_str = str(ref)
if ref_str not in node_by_id and ref_str not in source_slugs:
findings[bucket].append({**here, ref_field: ref_str})
# Orphan typed nodes — pages that declared `graph:` frontmatter but end
# up with no typed (non-implicit) edge touching them after extraction.
# Source nodes are exempt (they participate via implicit edges).
extracted_edges = _extract.build_edges(pages, {n["slug"]: n["id"] for n in node_by_id.values()})
typed_node_refs: set[str] = set()
for e in extracted_edges:
if e["predicate"] in IMPLICIT_PREDICATES:
continue
typed_node_refs.add(e["subject"])
if e["object"] in node_by_id:
typed_node_refs.add(e["object"])
for n in node_by_id.values():
if n["node_type"] == "source":
continue
graph_meta = n.get("graph") or {}
if not graph_meta:
continue # Pages without graph metadata are valid; they're text-only nodes.
if n["id"] in typed_node_refs:
continue
findings["orphan_typed_nodes"].append({
"node_id": n["id"],
"path": n["rel_path"],
})
# Summary
findings["summary"] = {
"pages_scanned": len(pages),
"nodes": len(node_by_id),
**{k: len(v) for k, v in findings.items() if isinstance(v, list)},
}
return findings
def render_text(findings: dict) -> str:
out = []
s = findings["summary"]
out.append("=" * 60)
out.append("Wiki Graph Lint Report")
out.append("=" * 60)
out.append(f"Pages scanned: {s['pages_scanned']} Nodes: {s['nodes']}")
out.append("")
sections = [
("duplicate_node_ids", "Duplicate node ids",
lambda f: f" - {f['node_id']}: {', '.join(f['paths'])}"),
("unknown_predicates", "Unknown predicates (not in ontology)",
lambda f: f" - {f['page']}#rel[{f['index']}] predicate={f['predicate']!r}"),
("broken_object_refs", "Broken object references",
lambda f: f" - {f['page']}#rel[{f['index']}] {f['predicate']} → {f['object']!r}"),
("subject_type_mismatch", "Subject type does not match ontology",
lambda f: f" - {f['page']}#rel[{f['index']}] {f['predicate']}: subject={f['subject_type']} (allowed: {f['allowed']})"),
("object_type_mismatch", "Object type does not match ontology",
lambda f: f" - {f['page']}#rel[{f['index']}] {f['predicate']}: object={f['object_type']} (allowed: {f['allowed']})"),
("missing_evidence", "Missing evidence on typed edge",
lambda f: f" - {f['page']}#rel[{f['index']}] {f['predicate']} → {f['object']}"),
("missing_source_field", "Missing source on typed edge",
lambda f: f" - {f['page']}#rel[{f['index']}] {f['predicate']} → {f['object']}"),
("broken_source_refs", "source: does not match any source page",
lambda f: f" - {f['page']}#rel[{f['index']}] source={f['source']!r}"),
("invalid_confidence", "Invalid confidence value",
lambda f: f" - {f['page']}#rel[{f['index']}] confidence={f['confidence']!r}"),
("invalid_status", "Invalid status value",
lambda f: f" - {f['page']}#rel[{f['index']}] status={f['status']!r}"),
("duplicate_canonical", "Duplicate canonical nodes",
lambda f: f" - {f['node_id']}: {', '.join(f['paths'])}"),
("alias_collisions", "Alias used by multiple canonical nodes",
lambda f: f" - {f['alias']!r}: {', '.join(f['owners'])}"),
("broken_contradicts", "Broken contradicts reference",
lambda f: f" - {f['page']}#rel[{f['index']}] contradicts={f.get('contradicts')}"),
("broken_supersedes", "Broken supersedes reference",
lambda f: f" - {f['page']}#rel[{f['index']}] supersedes={f.get('supersedes')}"),
("orphan_typed_nodes", "Orphan typed nodes (no inbound or outbound typed edges)",
lambda f: f" - {f['node_id']} ({f['path']})"),
]
healthy = True
for key, label, formatter in sections:
items = findings[key]
if not items:
continue
healthy = False
out.append(f"{label} ({len(items)}):")
for item in items[:50]:
out.append(formatter(item))
if len(items) > 50:
out.append(f" ... and {len(items) - 50} more")
out.append("")
if healthy:
out.append("No graph issues found.")
return "\n".join(out)
def main():
parser = argparse.ArgumentParser(description=__doc__, formatter_class=argparse.RawDescriptionHelpFormatter)
parser.add_argument("wiki", nargs="?", type=Path, default=Path("cml/wiki"))
parser.add_argument("--ontology", type=Path, help="Ontology file (default: <wiki>/graph/ontology.yaml)")
parser.add_argument("--json", action="store_true")
args = parser.parse_args()
if not args.wiki.exists():
print(f"Wiki directory not found: {args.wiki}", file=sys.stderr)
sys.exit(1)
ontology_path = args.ontology or (args.wiki / "graph" / "ontology.yaml")
if not ontology_path.exists():
print(f"Ontology not found: {ontology_path}", file=sys.stderr)
print("Did you forget to seed wiki/graph/ontology.yaml? See assets/ontology.yaml.template.",
file=sys.stderr)
sys.exit(1)
try:
ontology = yaml.safe_load(ontology_path.read_text(encoding="utf-8")) or {}
except yaml.YAMLError as e:
print(f"Ontology parse error: {e}", file=sys.stderr)
sys.exit(2)
pages = collect_pages(args.wiki)
findings = lint(pages, ontology)
if args.json:
print(json.dumps(findings, indent=2, default=str))
else:
print(render_text(findings))
if __name__ == "__main__":
main()

267
skills/llm-wiki/scripts/wiki_graph_query.py Normal file
View File

@@ -0,0 +1,267 @@
#!/usr/bin/env -S uv run --script
# /// script
# requires-python = ">=3.11"
# dependencies = []
# ///
"""
wiki_graph_query.py — Query the compiled wiki graph (graph.sqlite).
Use this to accelerate navigation: find what's connected to a node, list
typed edges around a subject, find a path between two nodes, or dump every
fact about a node. The graph is a navigation index — for high-stakes
claims, follow the `source` field back to the wiki page and the raw file.
Subcommands:
neighbors --node <id> List nodes one hop away from <id>
edges --subject <id> List all outbound edges from <id>
[--predicate <p>] Filter by predicate
path --from <id> --to <id> Shortest directed path (BFS, max depth 6)
[--max-depth N]
facts --about <id> Outbound + inbound edges for <id>
Common options:
--db <path> Path to graph.sqlite (default: <wiki>/graph/graph.sqlite)
--json Emit JSON instead of text
Examples:
python wiki_graph_query.py wiki/ neighbors --node product:konvy
python wiki_graph_query.py wiki/ edges --subject person:stephanie-emmanouel
python wiki_graph_query.py wiki/ path --from person:praney-behl --to product:konvy
python wiki_graph_query.py wiki/ facts --about product:konvy
"""
import argparse
import json
import sqlite3
import sys
from collections import deque
from pathlib import Path
EVIDENCE_SNIPPET_LEN = 140
def open_db(path: Path) -> sqlite3.Connection:
if not path.exists():
print(f"graph.sqlite not found at {path}.", file=sys.stderr)
print("Run wiki_graph_extract.py first.", file=sys.stderr)
sys.exit(1)
conn = sqlite3.connect(path)
conn.row_factory = sqlite3.Row
return conn
def fetch_node(conn: sqlite3.Connection, node_id: str) -> dict | None:
row = conn.execute("SELECT * FROM nodes WHERE id = ?", (node_id,)).fetchone()
return dict(row) if row else None
def edges_from(conn: sqlite3.Connection, subject: str, predicate: str | None = None) -> list[dict]:
q = "SELECT * FROM edges WHERE subject = ?"
params: list = [subject]
if predicate:
q += " AND predicate = ?"
params.append(predicate)
q += " ORDER BY predicate, object"
return [dict(r) for r in conn.execute(q, params).fetchall()]
def edges_to(conn: sqlite3.Connection, obj: str, predicate: str | None = None) -> list[dict]:
q = "SELECT * FROM edges WHERE object = ?"
params: list = [obj]
if predicate:
q += " AND predicate = ?"
params.append(predicate)
q += " ORDER BY predicate, subject"
return [dict(r) for r in conn.execute(q, params).fetchall()]
def truncate(text: str | None) -> str:
if not text:
return ""
if len(text) <= EVIDENCE_SNIPPET_LEN:
return text
return text[: EVIDENCE_SNIPPET_LEN - 1].rstrip() + "…"
def render_edge_row(e: dict) -> str:
pieces = [
f" {e['subject']} --[{e['predicate']}]--> {e['object']}",
]
confidence = e.get("confidence") or "-"
status = e.get("status") or "-"
src = e.get("source") or "-"
pieces.append(f" via {src} conf={confidence} status={status}")
if e.get("evidence"):
pieces.append(f" evidence: {truncate(e['evidence'])}")
pieces.append(f" (page: {e['page']})")
return "\n".join(pieces)
def cmd_neighbors(conn: sqlite3.Connection, args) -> dict:
node = fetch_node(conn, args.node)
if not node:
print(f"node not found: {args.node}", file=sys.stderr)
sys.exit(1)
out_edges = edges_from(conn, args.node)
in_edges = edges_to(conn, args.node)
neighbors: dict[str, dict] = {}
for e in out_edges:
neighbors.setdefault(e["object"], {"node_id": e["object"], "out": [], "in": []})
neighbors[e["object"]]["out"].append(e)
for e in in_edges:
neighbors.setdefault(e["subject"], {"node_id": e["subject"], "out": [], "in": []})
neighbors[e["subject"]]["in"].append(e)
# Resolve neighbor titles where possible
for nid, slot in neighbors.items():
target = fetch_node(conn, nid)
slot["title"] = target["title"] if target else nid
slot["path"] = target["path"] if target else None
return {
"node": node,
"neighbors": sorted(neighbors.values(), key=lambda n: n["node_id"]),
}
def cmd_edges(conn: sqlite3.Connection, args) -> dict:
es = edges_from(conn, args.subject, args.predicate)
return {"subject": args.subject, "predicate": args.predicate, "edges": es}
def cmd_facts(conn: sqlite3.Connection, args) -> dict:
node = fetch_node(conn, args.about)
if not node:
print(f"node not found: {args.about}", file=sys.stderr)
sys.exit(1)
return {
"node": node,
"outbound": edges_from(conn, args.about),
"inbound": edges_to(conn, args.about),
}
def cmd_path(conn: sqlite3.Connection, args) -> dict:
src = fetch_node(conn, getattr(args, "from"))
dst = fetch_node(conn, args.to)
if not src:
print(f"from-node not found: {getattr(args, 'from')}", file=sys.stderr)
sys.exit(1)
if not dst:
print(f"to-node not found: {args.to}", file=sys.stderr)
sys.exit(1)
start, goal = getattr(args, "from"), args.to
queue = deque([(start, [start], [])])
visited = {start}
while queue:
node, node_path, edge_path = queue.popleft()
if node == goal:
return {"from": start, "to": goal, "path_nodes": node_path, "path_edges": edge_path}
if len(node_path) - 1 >= args.max_depth:
continue
for e in edges_from(conn, node):
nxt = e["object"]
if nxt in visited:
continue
visited.add(nxt)
queue.append((nxt, node_path + [nxt], edge_path + [e]))
return {"from": start, "to": goal, "path_nodes": [], "path_edges": []}
def render(result: dict, command: str) -> str:
out: list[str] = []
if command == "neighbors":
n = result["node"]
out.append(f"Node: {n['id']} ({n['title']}) {n['node_type']} [{n['path']}]")
out.append(f"Neighbors: {len(result['neighbors'])}")
for nb in result["neighbors"]:
out.append("")
out.append(f" → {nb['node_id']} ({nb['title']})")
for e in nb.get("out", []):
out.append(f" out [{e['predicate']}] conf={e.get('confidence') or '-'} src={e.get('source') or '-'}")
for e in nb.get("in", []):
out.append(f" in [{e['predicate']}] from {e['subject']} src={e.get('source') or '-'}")
elif command == "edges":
out.append(f"Edges from {result['subject']}"
+ (f" with predicate {result['predicate']}" if result['predicate'] else ""))
for e in result["edges"]:
out.append("")
out.append(render_edge_row(e))
elif command == "facts":
n = result["node"]
out.append(f"Facts about {n['id']} ({n['title']}) [{n['path']}]")
out.append("")
out.append(f"Outbound ({len(result['outbound'])}):")
for e in result["outbound"]:
out.append(render_edge_row(e))
out.append("")
out.append(f"Inbound ({len(result['inbound'])}):")
for e in result["inbound"]:
out.append(render_edge_row(e))
elif command == "path":
if not result["path_nodes"]:
out.append(f"No path found from {result['from']} to {result['to']} within depth limit.")
else:
out.append(f"Path from {result['from']} to {result['to']} ({len(result['path_edges'])} hops):")
for e in result["path_edges"]:
out.append("")
out.append(render_edge_row(e))
return "\n".join(out)
def main():
parser = argparse.ArgumentParser(description=__doc__, formatter_class=argparse.RawDescriptionHelpFormatter)
parser.add_argument("wiki", type=Path, help="Wiki directory.")
parser.add_argument("--db", type=Path, help="Path to graph.sqlite (default: <wiki>/graph/graph.sqlite)")
parser.add_argument("--json", action="store_true")
sub = parser.add_subparsers(dest="command", required=True)
p_n = sub.add_parser("neighbors")
p_n.add_argument("--node", required=True)
p_e = sub.add_parser("edges")
p_e.add_argument("--subject", required=True)
p_e.add_argument("--predicate")
p_p = sub.add_parser("path")
p_p.add_argument("--from", dest="from", required=True)
p_p.add_argument("--to", required=True)
p_p.add_argument("--max-depth", type=int, default=6)
p_f = sub.add_parser("facts")
p_f.add_argument("--about", required=True)
args = parser.parse_args()
if not args.wiki.exists():
print(f"Wiki directory not found: {args.wiki}", file=sys.stderr)
sys.exit(1)
db_path = args.db or (args.wiki / "graph" / "graph.sqlite")
conn = open_db(db_path)
try:
if args.command == "neighbors":
result = cmd_neighbors(conn, args)
elif args.command == "edges":
result = cmd_edges(conn, args)
elif args.command == "path":
result = cmd_path(conn, args)
elif args.command == "facts":
result = cmd_facts(conn, args)
else:
parser.print_help()
sys.exit(1)
finally:
conn.close()
if args.json:
print(json.dumps(result, indent=2, default=str))
else:
print(render(result, args.command))
if __name__ == "__main__":
main()

319
skills/llm-wiki/scripts/wiki_lint.py Normal file
View File

@@ -0,0 +1,319 @@
#!/usr/bin/env -S uv run --script
# /// script
# requires-python = ">=3.11"
# dependencies = []
# ///
"""
wiki_lint.py — Structural health check for an LLM Wiki.
Reports orphan pages, broken wikilinks, oversized pages, frontmatter issues,
stale pages, duplicate slugs, and (with --suggest-pages) terms that appear in
many pages without their own page.
Conservative by design: reports findings, never edits.
Usage:
python wiki_lint.py [<wiki-dir>] [options]
Options:
--soft-cap N Page-size soft cap in lines (default: 400)
--hard-cap N Page-size hard cap in lines (default: 800)
--required-fm a,b Required frontmatter fields (default: type,title,tags,created,updated)
--suggest-pages Surface terms appearing in many pages without a page
--suggest-min N Minimum occurrences for --suggest-pages (default: 5)
--json Emit JSON instead of text
Examples:
python wiki_lint.py wiki/
python wiki_lint.py wiki/ --suggest-pages
python wiki_lint.py wiki/ --json > lint.json
"""
import argparse
import json
import re
import sys
from collections import Counter, defaultdict
from datetime import date, datetime
from pathlib import Path
WIKILINK_RE = re.compile(r"\[\[([^\]|]+)(?:\|[^\]]+)?\]\]")
FRONTMATTER_RE = re.compile(r"^---\s*\n(.*?)\n---\s*\n", re.DOTALL)
CAPITALIZED_PHRASE_RE = re.compile(r"\b([A-Z][a-zA-Z0-9]+(?:\s+[A-Z][a-zA-Z0-9]+){0,3})\b")
SKIP_TOP_LEVEL_FILES = {"SCHEMA.md", "index.md", "log.md", "README.md"}
SKIP_TOP_LEVEL_DIRS = {"indexes", "graph"}
def parse_frontmatter(text: str) -> tuple[dict, str, bool]:
"""Returns (metadata, body, malformed). malformed=True if frontmatter was attempted but unparseable."""
if not text.startswith("---"):
return {}, text, False
m = FRONTMATTER_RE.match(text)
if not m:
return {}, text, True
fm_text = m.group(1)
body = text[m.end():]
meta = {}
current_key = None
for line in fm_text.split("\n"):
if not line.strip():
continue
kv = re.match(r"^([a-zA-Z_]+):\s*(.*)$", line)
if kv:
key, value = kv.group(1), kv.group(2).strip()
if value.startswith("[") and value.endswith("]"):
items = [x.strip().strip('"').strip("'") for x in value[1:-1].split(",") if x.strip()]
meta[key] = items
elif value:
meta[key] = value.strip('"').strip("'")
else:
meta[key] = []
current_key = key
elif line.startswith(" - ") and current_key:
meta[current_key].append(line[4:].strip().strip('"').strip("'"))
return meta, body, False
def collect_pages(wiki_root: Path) -> list[dict]:
pages = []
for md_path in wiki_root.rglob("*.md"):
rel = md_path.relative_to(wiki_root)
if rel.parts[0] in SKIP_TOP_LEVEL_FILES or rel.parts[0] in SKIP_TOP_LEVEL_DIRS:
continue
if rel.name.startswith("."):
continue
try:
text = md_path.read_text(encoding="utf-8")
except (UnicodeDecodeError, OSError) as e:
pages.append({
"path": str(md_path),
"rel_path": str(rel),
"slug": md_path.stem,
"read_error": str(e),
})
continue
meta, body, malformed = parse_frontmatter(text)
line_count = text.count("\n") + 1
links = [m.group(1).strip() for m in WIKILINK_RE.finditer(body)]
pages.append({
"path": str(md_path),
"rel_path": str(rel),
"slug": md_path.stem,
"meta": meta,
"body": body,
"line_count": line_count,
"links": links,
"malformed_fm": malformed,
})
return pages
def parse_date(s):
if not s or not isinstance(s, str):
return None
try:
return datetime.strptime(s[:10], "%Y-%m-%d").date()
except (ValueError, TypeError):
return None
def lint(pages: list[dict], soft_cap: int, hard_cap: int, required_fm: list[str], suggest_pages: bool, suggest_min: int) -> dict:
findings = {
"orphans": [],
"broken_links": [],
"oversized_hard": [],
"oversized_soft": [],
"missing_frontmatter": [],
"malformed_frontmatter": [],
"duplicate_slugs": [],
"stale_pages": [],
"read_errors": [],
"suggested_pages": [],
"summary": {},
}
# Read errors
for p in pages:
if "read_error" in p:
findings["read_errors"].append({"path": p["rel_path"], "error": p["read_error"]})
pages = [p for p in pages if "read_error" not in p]
# Slugs
slug_to_pages = defaultdict(list)
for p in pages:
slug_to_pages[p["slug"]].append(p["rel_path"])
for slug, paths in slug_to_pages.items():
if len(paths) > 1:
findings["duplicate_slugs"].append({"slug": slug, "paths": paths})
# Inbound link map
inbound = defaultdict(set)
all_slugs = set(slug_to_pages.keys())
for p in pages:
for link in p["links"]:
inbound[link].add(p["slug"])
# Orphans, broken links, oversize, frontmatter, staleness
for p in pages:
# Orphans
if not inbound.get(p["slug"]):
findings["orphans"].append({"slug": p["slug"], "path": p["rel_path"]})
# Broken links
for link in p["links"]:
if link not in all_slugs:
findings["broken_links"].append({
"from": p["slug"],
"from_path": p["rel_path"],
"to": link,
})
# Oversize
if p["line_count"] > hard_cap:
findings["oversized_hard"].append({"path": p["rel_path"], "lines": p["line_count"]})
elif p["line_count"] > soft_cap:
findings["oversized_soft"].append({"path": p["rel_path"], "lines": p["line_count"]})
# Frontmatter
if p["malformed_fm"]:
findings["malformed_frontmatter"].append({"path": p["rel_path"]})
else:
missing = [field for field in required_fm if field not in p["meta"] or p["meta"].get(field) in ("", None, [])]
if missing:
findings["missing_frontmatter"].append({"path": p["rel_path"], "missing": missing})
# Staleness: heuristic — page hasn't been updated in 90 days AND has been touched by recent ingests.
# Approximate: if updated > 90d ago and the page is well-linked (a hub), flag it.
updated = parse_date(p["meta"].get("updated"))
if updated:
age_days = (date.today() - updated).days
if age_days > 90 and len(inbound.get(p["slug"], [])) >= 3:
findings["stale_pages"].append({
"path": p["rel_path"],
"updated": p["meta"].get("updated"),
"age_days": age_days,
"inbound_count": len(inbound.get(p["slug"], [])),
})
# Suggested pages: capitalized multi-word phrases appearing in many pages without a page
if suggest_pages:
phrase_pages = defaultdict(set)
for p in pages:
seen = set()
for m in CAPITALIZED_PHRASE_RE.finditer(p["body"]):
phrase = m.group(1).strip()
seen.add(phrase)
for phrase in seen:
phrase_pages[phrase].add(p["slug"])
# Title set for filtering
existing_titles = {p["meta"].get("title", "").lower() for p in pages}
existing_slugs_normalized = {s.lower().replace("-", " ") for s in all_slugs}
candidates = []
for phrase, page_set in phrase_pages.items():
if len(page_set) < suggest_min:
continue
if phrase.lower() in existing_titles:
continue
if phrase.lower() in existing_slugs_normalized:
continue
# Filter out section header garbage
if phrase.split()[0] in {"Section", "Where", "Sources", "Tags", "Type", "Title"}:
continue
candidates.append({"phrase": phrase, "page_count": len(page_set), "pages": sorted(page_set)[:5]})
candidates.sort(key=lambda x: -x["page_count"])
findings["suggested_pages"] = candidates[:30]
findings["summary"] = {
"total_pages": len(pages),
"orphans": len(findings["orphans"]),
"broken_links": len(findings["broken_links"]),
"oversized_hard": len(findings["oversized_hard"]),
"oversized_soft": len(findings["oversized_soft"]),
"missing_frontmatter": len(findings["missing_frontmatter"]),
"malformed_frontmatter": len(findings["malformed_frontmatter"]),
"duplicate_slugs": len(findings["duplicate_slugs"]),
"stale_pages": len(findings["stale_pages"]),
"read_errors": len(findings["read_errors"]),
"suggested_pages": len(findings["suggested_pages"]),
}
return findings
def render_text(findings: dict) -> str:
out = []
s = findings["summary"]
out.append("=" * 60)
out.append("Wiki Lint Report")
out.append("=" * 60)
out.append(f"Total pages scanned: {s['total_pages']}")
out.append("")
sections = [
("orphans", "Orphan pages (no inbound links)", lambda f: f" - {f['slug']} ({f['path']})"),
("broken_links", "Broken wikilinks", lambda f: f" - [[{f['to']}]] referenced from {f['from_path']}"),
("oversized_hard", "OVERSIZE (over hard cap — must split)", lambda f: f" - {f['path']} ({f['lines']} lines)"),
("oversized_soft", "Oversize (over soft cap — consider splitting)", lambda f: f" - {f['path']} ({f['lines']} lines)"),
("missing_frontmatter", "Missing frontmatter fields", lambda f: f" - {f['path']} missing: {', '.join(f['missing'])}"),
("malformed_frontmatter", "Malformed frontmatter", lambda f: f" - {f['path']}"),
("duplicate_slugs", "Duplicate slugs", lambda f: f" - {f['slug']}: {', '.join(f['paths'])}"),
("stale_pages", "Stale pages (well-linked but not updated in 90+ days)", lambda f: f" - {f['path']} (updated {f['updated']}, {f['age_days']}d ago, {f['inbound_count']} inbound)"),
("read_errors", "Read errors", lambda f: f" - {f['path']}: {f['error']}"),
]
for key, label, formatter in sections:
items = findings[key]
if not items:
continue
out.append(f"{label} ({len(items)}):")
for item in items[:50]:
out.append(formatter(item))
if len(items) > 50:
out.append(f" ... and {len(items) - 50} more")
out.append("")
if findings["suggested_pages"]:
out.append(f"Suggested page candidates ({len(findings['suggested_pages'])}):")
out.append(" Phrases appearing in many pages without a dedicated page:")
for item in findings["suggested_pages"]:
out.append(f" - \"{item['phrase']}\" ({item['page_count']} pages)")
out.append("")
if all(v == 0 for k, v in s.items() if k != "total_pages"):
out.append("No issues found. Wiki is healthy.")
return "\n".join(out)
def main():
parser = argparse.ArgumentParser(description=__doc__, formatter_class=argparse.RawDescriptionHelpFormatter)
parser.add_argument("wiki", nargs="?", type=Path, default=Path("cml/wiki"), help="Wiki directory (default: cml/wiki).")
parser.add_argument("--soft-cap", type=int, default=400, help="Page-size soft cap (lines).")
parser.add_argument("--hard-cap", type=int, default=800, help="Page-size hard cap (lines).")
parser.add_argument("--required-fm", default="type,title,tags,created,updated", help="Required frontmatter fields, comma-separated.")
parser.add_argument("--suggest-pages", action="store_true", help="Surface page candidates.")
parser.add_argument("--suggest-min", type=int, default=5, help="Minimum page count for suggestions.")
parser.add_argument("--json", action="store_true", help="Emit JSON.")
args = parser.parse_args()
if not args.wiki.exists():
print(f"Wiki directory not found: {args.wiki}", file=sys.stderr)
sys.exit(1)
pages = collect_pages(args.wiki)
required_fm = [f.strip() for f in args.required_fm.split(",") if f.strip()]
findings = lint(pages, args.soft_cap, args.hard_cap, required_fm, args.suggest_pages, args.suggest_min)
if args.json:
print(json.dumps(findings, indent=2, default=str))
else:
print(render_text(findings))
if __name__ == "__main__":
main()

270
skills/llm-wiki/scripts/wiki_search.py Normal file
View File

@@ -0,0 +1,270 @@
#!/usr/bin/env -S uv run --script
# /// script
# requires-python = ">=3.11"
# dependencies = []
# ///
"""
wiki_search.py — BM25 search over wiki pages with frontmatter filters.
Fallback for when index-first navigation doesn't surface the right pages.
Pure-Python implementation (no dependencies beyond stdlib) so it runs anywhere.
Usage:
python wiki_search.py "query terms" [options]
Options:
--wiki <dir> Wiki directory (default: cml/wiki)
--top N Return top N results (default: 10)
--type <type> Filter by frontmatter type (source|entity|concept|synthesis|...)
--tag <tag> Filter by tag (repeatable)
--since YYYY-MM-DD Only pages updated on or after this date
--backlinks <slug> Find pages that link to <slug>; ignores the query
--top-linked N Show the N most-linked-to pages (hubs); ignores the query
--cache <path> Persist the BM25 index to disk for faster reruns
Examples:
python wiki_search.py "diffusion training stability" --top 5
python wiki_search.py "alignment" --type concept --tag safety
python wiki_search.py "" --backlinks transformer
python wiki_search.py "" --top-linked 10
"""
import argparse
import json
import math
import re
import sys
from collections import Counter, defaultdict
from datetime import date, datetime
from pathlib import Path
WIKILINK_RE = re.compile(r"\[\[([^\]|]+)(?:\|[^\]]+)?\]\]")
FRONTMATTER_RE = re.compile(r"^---\s*\n(.*?)\n---\s*\n", re.DOTALL)
TOKEN_RE = re.compile(r"[a-z0-9]+")
def parse_frontmatter(text: str) -> tuple[dict, str]:
"""Lightweight YAML-ish frontmatter parser. Returns (metadata, body)."""
m = FRONTMATTER_RE.match(text)
if not m:
return {}, text
fm_text = m.group(1)
body = text[m.end():]
meta = {}
current_key = None
for line in fm_text.split("\n"):
if not line.strip():
continue
# Inline list: tags: [a, b, c]
kv = re.match(r"^([a-zA-Z_]+):\s*(.*)$", line)
if kv:
key, value = kv.group(1), kv.group(2).strip()
if value.startswith("[") and value.endswith("]"):
items = [x.strip().strip('"').strip("'") for x in value[1:-1].split(",") if x.strip()]
meta[key] = items
elif value:
meta[key] = value.strip('"').strip("'")
else:
meta[key] = []
current_key = key
elif line.startswith(" - ") and current_key:
meta[current_key].append(line[4:].strip().strip('"').strip("'"))
return meta, body
def tokenize(text: str) -> list[str]:
return TOKEN_RE.findall(text.lower())
def slug_from_path(path: Path, wiki_root: Path) -> str:
return path.stem
def extract_wikilinks(body: str) -> list[str]:
return [m.group(1).strip() for m in WIKILINK_RE.finditer(body)]
def collect_pages(wiki_root: Path) -> list[dict]:
"""Walk the wiki and return [{path, slug, meta, body, tokens, links}]."""
pages = []
for md_path in wiki_root.rglob("*.md"):
# Skip the schema, index, log, and template files
rel = md_path.relative_to(wiki_root)
if rel.parts[0] in {"SCHEMA.md", "index.md", "log.md"} or rel.name.startswith("."):
continue
if rel.parts[0] in {"indexes", "graph"}:
continue
try:
text = md_path.read_text(encoding="utf-8")
except (UnicodeDecodeError, OSError):
continue
meta, body = parse_frontmatter(text)
pages.append({
"path": str(md_path),
"rel_path": str(rel),
"slug": slug_from_path(md_path, wiki_root),
"meta": meta,
"body": body,
"tokens": tokenize(body + " " + meta.get("title", "")),
"links": extract_wikilinks(body),
})
return pages
def build_bm25(pages: list[dict]) -> dict:
"""Build a BM25 index. Returns {df, avgdl, N, doc_lens, term_freqs}."""
N = len(pages)
df = Counter()
doc_lens = []
term_freqs = []
for page in pages:
tokens = page["tokens"]
doc_lens.append(len(tokens))
tf = Counter(tokens)
term_freqs.append(tf)
for term in tf:
df[term] += 1
avgdl = sum(doc_lens) / N if N else 0
return {"N": N, "df": df, "avgdl": avgdl, "doc_lens": doc_lens, "term_freqs": term_freqs}
def bm25_score(query_tokens: list[str], doc_idx: int, idx: dict, k1: float = 1.5, b: float = 0.75) -> float:
score = 0.0
N = idx["N"]
df = idx["df"]
avgdl = idx["avgdl"]
dl = idx["doc_lens"][doc_idx]
tf = idx["term_freqs"][doc_idx]
for term in query_tokens:
if term not in df:
continue
idf = math.log(1 + (N - df[term] + 0.5) / (df[term] + 0.5))
f = tf.get(term, 0)
if f == 0:
continue
denom = f + k1 * (1 - b + b * (dl / avgdl if avgdl else 1))
score += idf * (f * (k1 + 1)) / denom
return score
def parse_date(s: str | None) -> date | None:
if not s:
return None
try:
return datetime.strptime(s[:10], "%Y-%m-%d").date()
except (ValueError, TypeError):
return None
def passes_filters(page: dict, args) -> bool:
meta = page["meta"]
if args.type and meta.get("type") != args.type:
return False
if args.tag:
page_tags = set(meta.get("tags", []) or [])
if not all(t in page_tags for t in args.tag):
return False
if args.since:
since = parse_date(args.since)
updated = parse_date(meta.get("updated"))
if since and updated and updated < since:
return False
if since and not updated:
return False
return True
def cmd_search(args, pages: list[dict]) -> None:
filtered = [p for p in pages if passes_filters(p, args)]
if not filtered:
print("No pages matched the filters.", file=sys.stderr)
return
idx = build_bm25(filtered)
query_tokens = tokenize(args.query)
if not query_tokens:
print("Empty query.", file=sys.stderr)
return
scored = [(bm25_score(query_tokens, i, idx), i) for i in range(len(filtered))]
scored.sort(key=lambda x: -x[0])
top = [(s, filtered[i]) for s, i in scored[:args.top] if s > 0]
if not top:
print("No matches.", file=sys.stderr)
return
print(f"Top {len(top)} results for: {args.query!r}")
print()
for score, page in top:
title = page["meta"].get("title") or page["slug"]
page_type = page["meta"].get("type", "?")
print(f" [{score:6.2f}] [{page_type:9}] {title}")
print(f" {page['rel_path']}")
def cmd_backlinks(args, pages: list[dict]) -> None:
target = args.backlinks
inbound = []
for page in pages:
if target in page["links"]:
inbound.append(page)
if not inbound:
print(f"No pages link to [[{target}]].", file=sys.stderr)
return
print(f"Pages linking to [[{target}]] ({len(inbound)}):")
for page in inbound:
title = page["meta"].get("title") or page["slug"]
print(f" - {title} ({page['rel_path']})")
def cmd_top_linked(args, pages: list[dict]) -> None:
inbound_count = Counter()
for page in pages:
for link in page["links"]:
inbound_count[link] += 1
top = inbound_count.most_common(args.top_linked)
if not top:
print("No links found in the wiki.", file=sys.stderr)
return
print(f"Top {len(top)} most-linked-to pages (hubs):")
for slug, count in top:
# Try to find the page for the title
match = next((p for p in pages if p["slug"] == slug), None)
title = (match["meta"].get("title") if match else None) or slug
marker = "" if match else " [BROKEN LINK]"
print(f" {count:4d} {title} ({slug}){marker}")
def main():
parser = argparse.ArgumentParser(description=__doc__, formatter_class=argparse.RawDescriptionHelpFormatter)
parser.add_argument("query", nargs="?", default="", help="Query terms.")
parser.add_argument("--wiki", type=Path, default=Path("cml/wiki"), help="Wiki directory (default: cml/wiki).")
parser.add_argument("--top", type=int, default=10, help="Top N results (default: 10).")
parser.add_argument("--type", help="Filter by frontmatter type.")
parser.add_argument("--tag", action="append", default=[], help="Filter by tag (repeatable).")
parser.add_argument("--since", help="Only pages updated on or after YYYY-MM-DD.")
parser.add_argument("--backlinks", help="Find pages linking to this slug.")
parser.add_argument("--top-linked", type=int, help="Show the N most-linked-to pages.")
parser.add_argument("--cache", type=Path, help="(reserved) Cache path for BM25 index.")
args = parser.parse_args()
if not args.wiki.exists():
print(f"Wiki directory not found: {args.wiki}", file=sys.stderr)
sys.exit(1)
pages = collect_pages(args.wiki)
if not pages:
print(f"No wiki pages found under {args.wiki}", file=sys.stderr)
sys.exit(0)
if args.backlinks:
cmd_backlinks(args, pages)
elif args.top_linked:
cmd_top_linked(args, pages)
elif args.query:
cmd_search(args, pages)
else:
parser.print_help()
sys.exit(1)
if __name__ == "__main__":
main()

159
skills/llm-wiki/scripts/wiki_stats.py Normal file
View File

@@ -0,0 +1,159 @@
#!/usr/bin/env -S uv run --script
# /// script
# requires-python = ">=3.11"
# dependencies = []
# ///
"""
wiki_stats.py — Quick summary of wiki size, shape, and link density.
Useful for deciding when to shard the index or split pages.
Usage:
python wiki_stats.py [<wiki-dir>]
Example:
python wiki_stats.py wiki/
"""
import argparse
import re
import sys
from collections import Counter
from pathlib import Path
WIKILINK_RE = re.compile(r"\[\[([^\]|]+)(?:\|[^\]]+)?\]\]")
FRONTMATTER_RE = re.compile(r"^---\s*\n(.*?)\n---\s*\n", re.DOTALL)
SKIP_TOP_LEVEL_FILES = {"SCHEMA.md", "log.md", "README.md"}
SKIP_TOP_LEVEL_DIRS = {"indexes", "graph"}
def parse_type(text: str) -> str | None:
m = FRONTMATTER_RE.match(text)
if not m:
return None
fm = m.group(1)
for line in fm.split("\n"):
kv = re.match(r"^type:\s*(.*)$", line)
if kv:
return kv.group(1).strip().strip('"').strip("'")
return None
def main():
parser = argparse.ArgumentParser(description=__doc__, formatter_class=argparse.RawDescriptionHelpFormatter)
parser.add_argument("wiki", nargs="?", type=Path, default=Path("cml/wiki"))
args = parser.parse_args()
if not args.wiki.exists():
print(f"Wiki directory not found: {args.wiki}", file=sys.stderr)
sys.exit(1)
total_pages = 0
total_lines = 0
total_words = 0
total_links = 0
pages_by_type = Counter()
pages_by_dir = Counter()
largest = []
most_linked_in = Counter()
index_lines = 0
for md_path in args.wiki.rglob("*.md"):
rel = md_path.relative_to(args.wiki)
try:
text = md_path.read_text(encoding="utf-8")
except (UnicodeDecodeError, OSError):
continue
if rel.name == "index.md" and len(rel.parts) == 1:
index_lines = text.count("\n") + 1
continue
if rel.parts[0] in SKIP_TOP_LEVEL_FILES:
continue
if rel.parts[0] in SKIP_TOP_LEVEL_DIRS:
continue
if rel.name.startswith("."):
continue
total_pages += 1
line_count = text.count("\n") + 1
word_count = len(text.split())
total_lines += line_count
total_words += word_count
# Strip frontmatter before counting wikilinks; frontmatter uses bare slugs.
body = FRONTMATTER_RE.sub("", text, count=1) if text.startswith("---") else text
links = WIKILINK_RE.findall(body)
total_links += len(links)
for link in links:
target = link.split("|")[0].strip()
most_linked_in[target] += 1
page_type = parse_type(text) or "(none)"
pages_by_type[page_type] += 1
if len(rel.parts) > 1:
pages_by_dir[rel.parts[0]] += 1
else:
pages_by_dir["(root)"] += 1
largest.append((line_count, str(rel)))
largest.sort(reverse=True)
print("=" * 60)
print(f"Wiki Stats: {args.wiki}")
print("=" * 60)
print(f"Pages: {total_pages}")
print(f"Total lines: {total_lines:,}")
print(f"Total words: {total_words:,}")
print(f"Total links: {total_links:,}")
if total_pages:
print(f"Avg page: {total_lines // total_pages} lines / {total_words // total_pages} words")
print(f"Link density: {total_links / total_pages:.1f} links per page")
print(f"index.md: {index_lines} lines" + (" ← shard recommended (>300)" if index_lines > 300 else ""))
print()
print("Pages by type:")
for t, n in pages_by_type.most_common():
print(f" {t:15s} {n}")
print()
print("Pages by directory:")
for d, n in pages_by_dir.most_common():
print(f" {d:15s} {n}")
print()
if largest:
print("Largest pages:")
for lines, path in largest[:10]:
warn = ""
if lines > 800:
warn = " ← OVER HARD CAP"
elif lines > 400:
warn = " ← over soft cap"
print(f" {lines:5d} {path}{warn}")
print()
if most_linked_in:
print("Most-linked-to pages (hubs):")
for slug, count in most_linked_in.most_common(10):
print(f" {count:4d} [[{slug}]]")
print()
# Scaling recommendations
print("Scaling thresholds:")
if total_pages < 50:
print(" → Below first threshold. Flat structure is fine.")
elif total_pages < 150 and index_lines < 300:
print(" → Below shard threshold. Continue with single index.md.")
elif (total_pages >= 150 or index_lines >= 300) and not (args.wiki / "indexes").exists():
print(" → AT SHARD THRESHOLD. Consider sharding index.md into wiki/indexes/<type>.md.")
print(" See references/scaling-playbook.md.")
elif total_pages >= 300:
print(" → Past 300 pages. Use scripts/wiki_search.py as a routine fallback.")
if total_pages >= 500:
print(" → Past 500 pages. Run lint weekly or per-N-ingests.")
if __name__ == "__main__":
main()