nanobot: 2026-09-10 12:33:37
This commit is contained in:
lachtan
@@ -1,115 +0,0 @@
|
|||||||
# Keep Claude working toward a goal - Claude Code Docs
|
|
||||||
|
|
||||||
Source: https://code.claude.com/docs/en/goal
|
|
||||||
|
|
||||||
The `/goal` command sets a completion condition and Claude keeps working toward it without you prompting each step. After each turn, a small fast model checks whether the condition holds. If not, Claude starts another turn instead of returning control to you. The goal clears automatically once the condition is met.
|
|
||||||
|
|
||||||
Use a goal for substantial work with a verifiable end state:
|
|
||||||
|
|
||||||
- Migrating a module to a new API until every call site compiles and tests pass
|
|
||||||
- Implementing a design doc until all acceptance criteria hold
|
|
||||||
- Splitting a large file into focused modules until each is under a size budget
|
|
||||||
- Working through a labeled issue backlog until the queue is empty
|
|
||||||
|
|
||||||
## Compare ways to keep a session running
|
|
||||||
|
|
||||||
Three approaches keep the current session running between prompts:
|
|
||||||
|
|
||||||
| Approach | Next turn starts when | Stops when |
|
|
||||||
| --- | --- | --- |
|
|
||||||
| `/goal` | The previous turn finishes | A model confirms the condition is met |
|
|
||||||
| `/loop` | A time interval elapses | You stop it, or Claude decides the work is done |
|
|
||||||
| Stop hook | The previous turn finishes | Your own script or prompt decides |
|
|
||||||
|
|
||||||
`/goal` and a Stop hook both fire after every turn. `/goal` is a session-scoped shortcut: you type a condition and it's active for the current session only. A Stop hook lives in your settings file, applies to every session in its scope, and can run a script for deterministic checks or a prompt for model-evaluated ones.
|
|
||||||
|
|
||||||
Auto mode on its own approves tool calls within a single turn but doesn't start a new one. Claude stops when it judges the work done. `/goal` adds a separate evaluator that checks your condition after every turn, so completion is decided by a fresh model rather than the one doing the work. The two are complementary: auto mode removes per-tool prompts, and `/goal` removes per-turn prompts.
|
|
||||||
|
|
||||||
## Use `/goal`
|
|
||||||
|
|
||||||
One goal can be active per session. The same command sets, checks, and clears it depending on the argument.
|
|
||||||
|
|
||||||
### Set a goal
|
|
||||||
|
|
||||||
Run `/goal` followed by the condition you want satisfied. If a goal is already active, the new one replaces it.
|
|
||||||
|
|
||||||
```
|
|
||||||
/goal all tests in test/auth pass and the lint step is clean
|
|
||||||
```
|
|
||||||
|
|
||||||
Setting a goal starts a turn immediately, with the condition itself as the directive. You don't need to send a separate prompt. While the goal is active, a `◎ /goal active` indicator shows how long the goal has been running.
|
|
||||||
|
|
||||||
After each turn, the evaluator returns a short reason explaining why the condition is or isn't met. The most recent reason appears in the status view and in the transcript so you can see what Claude is working toward next.
|
|
||||||
|
|
||||||
### Write an effective condition
|
|
||||||
|
|
||||||
The evaluator judges your condition against what Claude has surfaced in the conversation. It doesn't run commands or read files independently, so write the condition as something Claude's own output can demonstrate. "All tests in `test/auth` pass" works because Claude runs the tests and the result lands in the transcript for the evaluator to read.
|
|
||||||
|
|
||||||
A condition that holds up across many turns usually has:
|
|
||||||
|
|
||||||
- **One measurable end state**: a test result, a build exit code, a file count, an empty queue
|
|
||||||
- **A stated check**: how Claude should prove it, such as "`npm test` exits 0" or "`git status` is clean"
|
|
||||||
- **Constraints that matter**: anything that must not change on the way there, such as "no other test file is modified"
|
|
||||||
|
|
||||||
The condition can be up to 4,000 characters.
|
|
||||||
|
|
||||||
To bound how long a goal runs, include a turn or time clause in the condition, such as `or stop after 20 turns`. Claude reports progress against that clause each turn and the evaluator judges it from the conversation.
|
|
||||||
|
|
||||||
### Check status
|
|
||||||
|
|
||||||
Run `/goal` with no arguments to see the current state.
|
|
||||||
|
|
||||||
```
|
|
||||||
/goal
|
|
||||||
```
|
|
||||||
|
|
||||||
If a goal is active, the status shows:
|
|
||||||
|
|
||||||
- The condition
|
|
||||||
- How long it has been running
|
|
||||||
- How many turns have been evaluated
|
|
||||||
- The current token spend
|
|
||||||
- The evaluator's most recent reason
|
|
||||||
|
|
||||||
If no goal is active but one was achieved earlier in the session, the status shows the achieved condition along with its duration, turn count, and token spend.
|
|
||||||
|
|
||||||
### Clear a goal
|
|
||||||
|
|
||||||
Run `/goal clear` to remove an active goal before its condition is met.
|
|
||||||
|
|
||||||
```
|
|
||||||
/goal clear
|
|
||||||
```
|
|
||||||
|
|
||||||
`stop`, `off`, `reset`, `none`, and `cancel` are accepted as aliases for `clear`. Running `/clear` to start a new conversation also removes any active goal.
|
|
||||||
|
|
||||||
### Resume with an active goal
|
|
||||||
|
|
||||||
A goal that was still active when a session ended is restored when you resume that session with `--resume` or `--continue`. The condition carries over, but the turn count, timer, and token-spend baseline all reset on resume. A goal that was already achieved or cleared is not restored.
|
|
||||||
|
|
||||||
### Run non-interactively
|
|
||||||
|
|
||||||
`/goal` works in non-interactive mode, in the desktop app, and through Remote Control. Setting a goal with `-p` runs the loop to completion in a single invocation:
|
|
||||||
|
|
||||||
```
|
|
||||||
claude -p "/goal CHANGELOG.md has an entry for every PR merged this week"
|
|
||||||
```
|
|
||||||
|
|
||||||
Interrupt the process with Ctrl+C to stop a non-interactive goal before the condition is met.
|
|
||||||
|
|
||||||
## How evaluation works
|
|
||||||
|
|
||||||
`/goal` is a wrapper around a session-scoped prompt-based Stop hook. Each time Claude finishes a turn, the condition and the conversation so far are sent to your configured small fast model, which defaults to Haiku. The model returns a yes-or-no decision and a short reason. A "no" tells Claude to keep working and includes the reason as guidance for the next turn. A "yes" clears the goal and records an achieved entry in the transcript.
|
|
||||||
|
|
||||||
The evaluator runs on whichever provider your session is configured for. It does not call tools, so it can only judge what Claude has already surfaced in the conversation.
|
|
||||||
|
|
||||||
## Requirements
|
|
||||||
|
|
||||||
`/goal` runs only in workspaces where you have accepted the trust dialog, because the evaluator is part of the hooks system. `/goal` is also unavailable when `disableAllHooks` is set at any settings level or when `allowManagedHooksOnly` is set in managed settings. In each case, the command tells you why instead of silently doing nothing.
|
|
||||||
|
|
||||||
## See also
|
|
||||||
|
|
||||||
- Run a prompt repeatedly with `/loop`: re-run on a time interval instead of until a condition holds
|
|
||||||
- Prompt-based hooks: write your own Stop hook when you need custom evaluation logic
|
|
||||||
- Auto mode: approve tool calls automatically so each goal turn runs unattended
|
|
||||||
- Scheduling comparison: run work on a schedule independent of any open session
|
|
||||||
@@ -1,94 +0,0 @@
|
|||||||
# Run Claude Code with Local & Cloud Models in 5 Minutes (Ollama, LM Studio, llama.cpp, OpenRouter)
|
|
||||||
|
|
||||||
**Autor:** Luong NGUYEN
|
|
||||||
**URL:** https://medium.com/@luongnv89/run-claude-code-on-local-cloud-models-in-5-minutes-ollama-openrouter-llama-cpp-6dfeaee03cda
|
|
||||||
**Datum:** Jan 31, 2026
|
|
||||||
|
|
||||||
---
|
|
||||||
|
|
||||||
Průvodce nastavením Claude Code s alternativními modely — Ollama (lokální i cloud), LM Studio, llama.cpp, OpenRouter a další. Většina konfigurace se dělá přes env vars `ANTHROPIC_BASE_URL`, `ANTHROPIC_AUTH_TOKEN`, `ANTHROPIC_API_KEY`, `ANTHROPIC_MODEL`.
|
|
||||||
|
|
||||||
## Doporučené modely pro coding
|
|
||||||
|
|
||||||
- **devstral-small-2 (24B)** — dobrý start pro coding quality
|
|
||||||
- **qwen3-coder:30b** — lepší coding ability, stále praktický na 32GB RAM
|
|
||||||
- **GLM4.7-flash:q8_0** — silný poměr cena/výkon (kvantizovaný)
|
|
||||||
|
|
||||||
Minimální spec: 32GB RAM, model 24B+ parametrů. Na 16GB to jde, ale experience je rough.
|
|
||||||
|
|
||||||
## Option 1: Ollama Local
|
|
||||||
|
|
||||||
```bash
|
|
||||||
ollama pull devstral-small-2
|
|
||||||
ollama launch claude --model devstral-small-2
|
|
||||||
```
|
|
||||||
|
|
||||||
Nebo manuálně přes env vars:
|
|
||||||
```bash
|
|
||||||
export ANTHROPIC_AUTH_TOKEN="ollama"
|
|
||||||
export ANTHROPIC_API_KEY=""
|
|
||||||
export ANTHROPIC_BASE_URL="http://localhost:11434"
|
|
||||||
claude --model devstral-small-2
|
|
||||||
```
|
|
||||||
|
|
||||||
## Option 2: llama.cpp + HuggingFace
|
|
||||||
|
|
||||||
Build llama.cpp s Metal (macOS) nebo CUDA (Linux), spusť server s `--jinja` flag (nutný pro tool calling), připoj Claude Code přes `ANTHROPIC_BASE_URL=http://localhost:8000`.
|
|
||||||
|
|
||||||
```bash
|
|
||||||
llama-server -hf bartowski/cerebras_Qwen3-Coder-REAP-25B-A3B-GGUF:Q4_K_M \
|
|
||||||
--alias "Qwen3-Coder-REAP-25B-A3B-GGUF" \
|
|
||||||
--port 8000 --jinja --kv-unified \
|
|
||||||
--cache-type-k q8_0 --cache-type-v q8_0 \
|
|
||||||
--flash-attn on --batch-size 4096 --ubatch-size 1024 --ctx-size 64000
|
|
||||||
```
|
|
||||||
|
|
||||||
## Option 3: LM Studio
|
|
||||||
|
|
||||||
GUI i CLI varianta (`llmster`). Server na portu 1234, env vars `ANTHROPIC_BASE_URL=http://localhost:1234`, `ANTHROPIC_AUTH_TOKEN=lmstudio`.
|
|
||||||
|
|
||||||
## Option 4: Ollama Cloud Models
|
|
||||||
|
|
||||||
```bash
|
|
||||||
ollama pull kimi-k2.5:cloud
|
|
||||||
ollama pull minimax-m2.1:cloud
|
|
||||||
claude --model kimi-k2.5:cloud
|
|
||||||
```
|
|
||||||
|
|
||||||
Stejný workflow jako lokální, compute v cloudu. Free tier má omezený usage.
|
|
||||||
|
|
||||||
## Option 5: Cloud Provider APIs (OpenRouter atd.)
|
|
||||||
|
|
||||||
```bash
|
|
||||||
export ANTHROPIC_BASE_URL=https://openrouter.ai/api
|
|
||||||
export ANTHROPIC_AUTH_TOKEN=YOUR_OPENROUTER_KEY
|
|
||||||
export ANTHROPIC_API_KEY=
|
|
||||||
export ANTHROPIC_MODEL="openai/gpt-oss-120b:free"
|
|
||||||
```
|
|
||||||
|
|
||||||
Prázdný `ANTHROPIC_API_KEY` je záměr — zabraňuje autentikaci přes Anthropic API přímo.
|
|
||||||
|
|
||||||
Minimax přes OpenRouter: ~98% levnější než Opus 4.5. Podobně GLM, DeepSeek, Kimi.
|
|
||||||
|
|
||||||
## Klíčové env vars
|
|
||||||
|
|
||||||
| Var | Purpose |
|
|
||||||
|-----|---------|
|
|
||||||
| `ANTHROPIC_BASE_URL` | API endpoint |
|
|
||||||
| `ANTHROPIC_AUTH_TOKEN` | API key pro provider |
|
|
||||||
| `ANTHROPIC_API_KEY` | Prázdný = žádný Anthropic fallback |
|
|
||||||
| `ANTHROPIC_MODEL` | Model identifier |
|
|
||||||
|
|
||||||
## Závěr
|
|
||||||
|
|
||||||
- Lokální na M1 32GB: devstral-small-2 (24B) OK, větší modely pomalé
|
|
||||||
- Nvidia DGX Spark: široký výběr modelů
|
|
||||||
- Cloud: nejrychlejší cesta, Ollama Cloud free tier pro emergency, jinak Kimi/Minimax/DeepSeek/GLM přes OpenRouter
|
|
||||||
- Opus 4.5 stále nejlepší quality+speed, ale drahý
|
|
||||||
|
|
||||||
## Zdroje
|
|
||||||
|
|
||||||
- [Ollama Claude Code Integration](https://docs.ollama.com/integrations/claude-code)
|
|
||||||
- [OpenRouter Integration](https://openrouter.ai/docs/guides/guides/claude-code-integration)
|
|
||||||
- [cc-compatible-models](https://github.com/Alorse/cc-compatible-models)
|
|
||||||
- [claude-flow wiki](https://github.com/ruvnet/claude-flow/wiki/Using-Claude-Code-with-Open-Models)
|
|
||||||
@@ -1,43 +0,0 @@
|
|||||||
# I Tried New Claude Code Ollama Workflow (It's Wild & Free)
|
|
||||||
|
|
||||||
**Autor:** Joe Njenga
|
|
||||||
**URL:** https://medium.com/@joe.njenga/i-tried-new-claude-code-ollama-workflow-its-wild-free-cb7a12b733b5
|
|
||||||
**Datum:** Jan 19, 2026
|
|
||||||
**Status:** 🔒 Member-only (paywall) — pouze preview dostupný
|
|
||||||
|
|
||||||
---
|
|
||||||
|
|
||||||
## Dostupný obsah (preview)
|
|
||||||
|
|
||||||
Claude Code nyní funguje s Ollama — lokální i cloud modely. Ollama v0.14.0+ je kompatibilní s Anthropic Messages API, takže Claude Code může komunikovat přímo s Ollama modely.
|
|
||||||
|
|
||||||
### Klíčové body z preview
|
|
||||||
|
|
||||||
- Ollama v0.14.0+ podporuje Anthropic Messages API → Claude Code kompatibilita
|
|
||||||
- Ideální pro privacy-conscious projekty, air-gapped systémy, nebo vyhnutí se API costům
|
|
||||||
- Autor testoval integraci od oznámení a dokumentoval chyby/pastýřky
|
|
||||||
- Workflow: lokální modely bez odesílání každého requestu do cloudu
|
|
||||||
|
|
||||||
### Nastavení Ollama s Claude Code
|
|
||||||
|
|
||||||
```bash
|
|
||||||
# Lokální Ollama
|
|
||||||
export ANTHROPIC_AUTH_TOKEN="ollama"
|
|
||||||
export ANTHROPIC_API_KEY=""
|
|
||||||
export ANTHROPIC_BASE_URL="http://localhost:11434"
|
|
||||||
claude --model devstral-small-2
|
|
||||||
|
|
||||||
# Cloud modely přes Ollama
|
|
||||||
ollama pull kimi-k2.5:cloud
|
|
||||||
claude --model kimi-k2.5:cloud
|
|
||||||
```
|
|
||||||
|
|
||||||
### Varování z preview
|
|
||||||
|
|
||||||
- Autor zmiňuje "všechny chyby, které tě budou stát čas" — konkrétní detaily za paywallem
|
|
||||||
- Free tier Ollama Cloud má omezený usage
|
|
||||||
|
|
||||||
## Zdroje
|
|
||||||
|
|
||||||
- [Ollama Claude Code Integration](https://docs.ollama.com/integrations/claude-code)
|
|
||||||
- [cc-compatible-models](https://github.com/Alorse/cc-compatible-models)
|
|
||||||
@@ -1,40 +0,0 @@
|
|||||||
# How I'm Using Claude Code Like Cline With OpenRouter (To Go Beast Mode at Low Cost)
|
|
||||||
|
|
||||||
**Autor:** Joe Njenga
|
|
||||||
**URL:** https://medium.com/@joe.njenga/how-im-using-claude-code-like-cline-with-openrouter-to-go-beast-mode-at-low-cost-8c78e0bdcb67
|
|
||||||
**Datum:** Jan 18, 2026
|
|
||||||
**Status:** 🔒 Member-only (paywall) — pouze preview dostupný
|
|
||||||
|
|
||||||
---
|
|
||||||
|
|
||||||
## Dostupný obsah (preview)
|
|
||||||
|
|
||||||
Claude Code + OpenRouter integrace pro low-cost coding. OpenRouter nedávno přidal Claude Code do své unified API platformy. Článek ukazuje, jak nastavit Claude Code s OpenRouter podobně jako Cline (VS Code extension) — svoboda volby modelu bez lock-in na jednoho providera.
|
|
||||||
|
|
||||||
### Klíčové body z preview
|
|
||||||
|
|
||||||
- OpenRouter integroval Claude Code do unified API
|
|
||||||
- Cline-like workflow: volit jakýkoliv model (GPT-4, Claude, nové modely) bez provider lock-in
|
|
||||||
- Nastavení přes env vars: `ANTHROPIC_BASE_URL`, `ANTHROPIC_AUTH_TOKEN`, `ANTHROPIC_API_KEY`, `ANTHROPIC_MODEL`
|
|
||||||
- Cílem: x10 budget efficiency oproti nativnímu Claude API
|
|
||||||
|
|
||||||
### Nastavení OpenRouter s Claude Code
|
|
||||||
|
|
||||||
```bash
|
|
||||||
export ANTHROPIC_BASE_URL=https://openrouter.ai/api
|
|
||||||
export ANTHROPIC_AUTH_TOKEN=YOUR_OPENROUTER_KEY
|
|
||||||
export ANTHROPIC_API_KEY=
|
|
||||||
export ANTHROPIC_MODEL="openai/gpt-oss-120b:free"
|
|
||||||
```
|
|
||||||
|
|
||||||
Prázdný `ANTHROPIC_API_KEY` zabraňuje fallback na Anthropic API.
|
|
||||||
|
|
||||||
### Modely zmíněné v článku
|
|
||||||
|
|
||||||
- Minimax přes OpenRouter: ~98% levnější než Opus 4.5
|
|
||||||
- GLM, DeepSeek, Kimi — další low-cost alternativy přes OpenRouter
|
|
||||||
|
|
||||||
## Zdroje
|
|
||||||
|
|
||||||
- [OpenRouter Integration](https://openrouter.ai/docs/guides/guides/claude-code-integration)
|
|
||||||
- [cc-compatible-models](https://github.com/Alorse/cc-compatible-models)
|
|
||||||
@@ -1,3 +0,0 @@
|
|||||||
https://blog.robotmak3rs.com
|
|
||||||
|
|
||||||
Topic: How to continue using LEGO Mindstorms products (after discontinuation / in alternative ways).
|
|
||||||
@@ -1,49 +0,0 @@
|
|||||||
# I Hated Every Coding Agent, So I Built My Own — Mario Zechner (Pi)
|
|
||||||
|
|
||||||
**Source URL:** https://www.youtube.com/watch?v=Dli5slNaJu0
|
|
||||||
**Type:** Video / Talk
|
|
||||||
**Date:** 2026 (approx)
|
|
||||||
**Speaker:** Mario Zechner (creator of Pi coding agent, also known as badlogic — libGDX author)
|
|
||||||
|
|
||||||
## Why Pi was created
|
|
||||||
|
|
||||||
Mario was frustrated by existing coding agents (Claude Code, OpenCode, Codex CLI, AMP) for several reasons:
|
|
||||||
|
|
||||||
1. **Feature bloat** — agents pile on features (built-in to-dos, complex tool suites) that aren't needed and add hidden context injection
|
|
||||||
2. **Hidden behaviors** — vendors change things under the hood (system prompts, context injection) that make LLMs behave unpredictably with existing workflows
|
|
||||||
3. **Poor observability** — hard to see what the agent is actually doing, what context it's using, how much it costs
|
|
||||||
4. **Lack of extensibility** — no way for power users to add custom tools or modify behavior without forking
|
|
||||||
5. **Approval fatigue** — agents offer either full autonomy or approval for every action; both are bad UX
|
|
||||||
6. **Poor context management** — agents like OpenCode rely on session compaction but lose important context
|
|
||||||
|
|
||||||
Key quote: *"So obviously they're doing things right, but not for me."*
|
|
||||||
|
|
||||||
## Pi's design philosophy
|
|
||||||
|
|
||||||
- **Minimal core** — only 4 tools: read file, write file, edit file, bash. That's all you need.
|
|
||||||
- **Tiny system prompt** — frontier RL-trained models don't need massive system prompts
|
|
||||||
- **Tree-structured sessions** — not linear chat history; sub-agents can branch and read files independently while preserving context/lineage
|
|
||||||
- **Full cost tracking** — built-in, not an afterthought
|
|
||||||
- **Hot-reloadable TypeScript extensions** — users can define custom tools, UIs, multi-agent setups without modifying core
|
|
||||||
- **No hidden context injection** — what you see is what the model gets
|
|
||||||
|
|
||||||
## Community extensions
|
|
||||||
|
|
||||||
- **pi-annotate** — visual feedback on live websites
|
|
||||||
- **pi-messenger** — multi-agent chatroom with custom UI
|
|
||||||
- Custom UIs, tool integrations — all as hot-reloadable TS modules
|
|
||||||
|
|
||||||
## Performance
|
|
||||||
|
|
||||||
On TerminalBench, Pi (with Claude Opus 4.5) scored close to Terminus even before advanced optimizations like compaction.
|
|
||||||
|
|
||||||
## Key insight
|
|
||||||
|
|
||||||
*"We are in the messing around and finding out stage, and nobody has any idea what the perfect coding agent should look like."* — simplification can lead to effective performance without unnecessary complexity.
|
|
||||||
|
|
||||||
## Related
|
|
||||||
|
|
||||||
- Pi website: https://pi.dev/
|
|
||||||
- Pi GitHub: https://github.com/earendil-works/pi
|
|
||||||
- Pi is part of OpenClaw ecosystem
|
|
||||||
- Mario Zechner is also the author of libGDX (Java game dev framework)
|
|
||||||
@@ -1,5 +0,0 @@
|
|||||||
# pi.dev — terminálová limitace
|
|
||||||
|
|
||||||
pi.dev je pěkný projekt, ale limitace na terminal je až moc přísná a omezující. Bez IDE to ztrácí všechny výhodné vlastnosti — podobně jako opencode.
|
|
||||||
|
|
||||||
Terminal-only přístup výrazně omezuje uživatelskou zkušenost a produktivitu oproti plnohodnotnému IDE integrovanému řešení.
|
|
||||||
@@ -1,3 +0,0 @@
|
|||||||
# Pi.dev — zajímavé video
|
|
||||||
|
|
||||||
Video k pi.dev: https://www.youtube.com/watch?v=Dli5slNaJu0
|
|
||||||
@@ -1,26 +0,0 @@
|
|||||||
---
|
|
||||||
type: <source|entity|concept|synthesis>
|
|
||||||
title: ""
|
|
||||||
tags: []
|
|
||||||
sources: []
|
|
||||||
created: YYYY-MM-DD
|
|
||||||
updated: YYYY-MM-DD
|
|
||||||
---
|
|
||||||
|
|
||||||
# Title
|
|
||||||
|
|
||||||
Lead paragraph: a clear, encyclopedic definition or framing of what this page is about. Should answer "what is this and why does it matter" in one or two sentences.
|
|
||||||
|
|
||||||
## Section 1
|
|
||||||
|
|
||||||
Body content. Use `[[wikilinks]]` liberally to cross-reference other pages. (Frontmatter `sources:` list above uses bare slugs; only the body uses double-bracket wikilinks.)
|
|
||||||
|
|
||||||
## Section 2
|
|
||||||
|
|
||||||
More body content. Hedge claims that aren't yet corroborated by multiple sources ("Source X claims Y, though this is not yet corroborated by other sources in the wiki").
|
|
||||||
|
|
||||||
## Where this fits
|
|
||||||
|
|
||||||
(For source pages.) List the entity and concept pages this source touches:
|
|
||||||
- [[entity-page-1]]
|
|
||||||
- [[concept-page-1]]
|
|
||||||
@@ -1,121 +0,0 @@
|
|||||||
# Wiki Schema
|
|
||||||
|
|
||||||
This file is the configuration for this wiki. It documents the conventions, page types, tag taxonomy, and any workflow customizations. The LLM reads this first when entering the wiki, and its conventions override the defaults documented in the `llm-wiki` skill.
|
|
||||||
|
|
||||||
This file is **co-evolved with the user**. When the LLM notices a recurring pattern in your edits or feedback that isn't here, it will propose adding it. When something here stops fitting, prune it.
|
|
||||||
|
|
||||||
## Wiki location
|
|
||||||
|
|
||||||
- Wiki root: `wiki/`
|
|
||||||
- Raw sources: `raw/`
|
|
||||||
- Asset/image storage: `raw/assets/`
|
|
||||||
|
|
||||||
## Page types
|
|
||||||
|
|
||||||
This wiki uses these page types, each with a dedicated subdirectory:
|
|
||||||
|
|
||||||
- `source` (in `wiki/sources/`) — one summary page per ingested source.
|
|
||||||
- `entity` (in `wiki/entities/`) — pages about specific things: people, papers, products, places, organizations.
|
|
||||||
- `concept` (in `wiki/concepts/`) — pages about ideas, methods, frameworks, abstractions.
|
|
||||||
- `synthesis` (in `wiki/synthesis/`) — cross-cutting analyses, comparisons, query answers filed back.
|
|
||||||
|
|
||||||
Add additional types here as the wiki evolves.
|
|
||||||
|
|
||||||
## Tag taxonomy
|
|
||||||
|
|
||||||
(Empty initially. Add tags here as you adopt them, with one-line descriptions. Keep this list small and disciplined — a wiki with 200 tags has effectively no tags.)
|
|
||||||
|
|
||||||
Example structure:
|
|
||||||
- `methodology` — pages about research or analytical methods.
|
|
||||||
- `open-question` — pages or sections that flag unresolved questions.
|
|
||||||
- `contested` — pages where sources contradict.
|
|
||||||
|
|
||||||
## Page sizing
|
|
||||||
|
|
||||||
- Soft cap: 400 lines / ~2,000 words. Consider splitting beyond this.
|
|
||||||
- Hard cap: 800 lines. Must split.
|
|
||||||
|
|
||||||
## Frontmatter requirements
|
|
||||||
|
|
||||||
Every page must have:
|
|
||||||
- `type`
|
|
||||||
- `title`
|
|
||||||
- `tags`
|
|
||||||
- `created`
|
|
||||||
- `updated`
|
|
||||||
|
|
||||||
Plus type-specific:
|
|
||||||
- `source` pages: `authors`, `url` (if applicable), `raw`, `ingested`
|
|
||||||
- Non-source pages: `sources` listing the source-summary pages drawn from
|
|
||||||
|
|
||||||
## Optional graph metadata
|
|
||||||
|
|
||||||
Pages may declare typed graph metadata under a top-level `graph:` key. This is the source of truth for the compiled knowledge graph under `wiki/graph/`. Markdown remains canonical; the graph is a regenerable index. Pages without `graph:` still appear as nodes (derived from `type`/`kind`) and still contribute `mentions` edges from body `[[wikilinks]]`.
|
|
||||||
|
|
||||||
```yaml
|
|
||||||
graph:
|
|
||||||
node_id: person:praney-behl # optional; default <node_type>:<slug>
|
|
||||||
node_type: person # optional; default mapped from type/kind via ontology
|
|
||||||
canonical: true # mark as canonical when multiple slugs alias the same entity
|
|
||||||
aliases: [Praney, praney@example.com]
|
|
||||||
relationships:
|
|
||||||
- predicate: founded
|
|
||||||
object: company:seedblocks
|
|
||||||
source: praney-founder-context-dump # source-page slug
|
|
||||||
evidence: "Solo technical founder and sole director..."
|
|
||||||
confidence: high # high | medium | low
|
|
||||||
status: current # current | historical | proposed | disputed | superseded
|
|
||||||
# optional:
|
|
||||||
# valid_from: 2025-01-15
|
|
||||||
# valid_to: 2026-03-01
|
|
||||||
# notes: "..."
|
|
||||||
# raw_ref: "raw/founder-dump.md#L42"
|
|
||||||
# contradicts: edge-id-or-source-slug
|
|
||||||
# supersedes: edge-id-or-source-slug
|
|
||||||
```
|
|
||||||
|
|
||||||
Required fields on every relationship: `predicate`, `object`, `source`, `evidence`, `confidence`, `status`. Predicates and the subject/object types they accept are declared in `wiki/graph/ontology.yaml`. Typed semantic edges must be supported by an explicit source — never emit one inferred from training data alone.
|
|
||||||
|
|
||||||
## Index structure
|
|
||||||
|
|
||||||
(Update this section when sharding.)
|
|
||||||
|
|
||||||
Currently flat: a single `wiki/index.md` listing all pages.
|
|
||||||
|
|
||||||
When the wiki passes ~150 pages or `index.md` exceeds 300 lines, shard into `wiki/indexes/<type>.md` and update this section.
|
|
||||||
|
|
||||||
## Graph layer
|
|
||||||
|
|
||||||
The wiki has an optional compiled graph layer under `wiki/graph/`:
|
|
||||||
|
|
||||||
- `wiki/graph/ontology.yaml` — declares node types and predicates. **Tracked.** Edit this when you introduce new predicates or domain types.
|
|
||||||
- `wiki/graph/nodes.jsonl`, `wiki/graph/edges.jsonl` — generated. Track in git only if you want graph diffs in PRs.
|
|
||||||
- `wiki/graph/graph.sqlite` — generated. Gitignored by default.
|
|
||||||
- `wiki/graph/graph.graphml` — generated. Track only if you want to diff it.
|
|
||||||
|
|
||||||
Generation is reproducible from markdown via `scripts/wiki_graph_extract.py`. The graph can be deleted at any time and rebuilt without losing knowledge — markdown is canonical.
|
|
||||||
|
|
||||||
## Workflow customizations
|
|
||||||
|
|
||||||
### Paywalled sources
|
|
||||||
|
|
||||||
Sites like `medium.com` (member-only stories) often return only a preview when fetched. When a source is paywalled:
|
|
||||||
|
|
||||||
1. **Capture what's available.** Fetch the URL, extract whatever preview/abstract is accessible, and write it into `cml/raw/<slug>.md` with a `🔒 paywall` marker and the original URL.
|
|
||||||
2. **Never fabricate.** Do not infer or hallucinate content behind the paywall. If only the title and first paragraph came through, that's all the source page gets.
|
|
||||||
3. **Flag in source page frontmatter.** Add `paywall: true` to the frontmatter of the corresponding `wiki/sources/<slug>.md` page so future queries know the coverage is partial.
|
|
||||||
4. **Compile normally.** A paywalled source still gets a source-summary page — just with limited content. The wiki should reflect what we actually have, not what we wish we had.
|
|
||||||
5. **Known paywall domains** (non-exhaustive): `medium.com`, `substack.com` (paid posts), `ft.com`, `wsj.com`, `nytimes.com` (soft paywall), `bloomberg.com`. When fetching from these, expect partial content and handle accordingly.
|
|
||||||
|
|
||||||
## User preferences
|
|
||||||
|
|
||||||
(Empty initially. As the user expresses style preferences — "always include a 'Why this matters' section on concept pages", "never use bullet lists in summaries", "prefer comparative tables for synthesis pages" — capture them here so they persist across sessions.)
|
|
||||||
|
|
||||||
## Lint cadence
|
|
||||||
|
|
||||||
- Structural lint: after every 5 ingests.
|
|
||||||
- Semantic lint: weekly or after every 20 ingests.
|
|
||||||
- Gap-finding: monthly.
|
|
||||||
- Graph lint + extract: after every ingest that adds typed `graph.relationships`.
|
|
||||||
|
|
||||||
Adjust based on the wiki's growth rate.
|
|
||||||
@@ -1,43 +0,0 @@
|
|||||||
---
|
|
||||||
type: concept
|
|
||||||
title: "Coding agent setup"
|
|
||||||
tags: [coding-agent, setup, configuration, workflow]
|
|
||||||
sources: [claude-code-local-cloud-models, claude-code-ollama-workflow, claude-code-openrouter-beast-mode]
|
|
||||||
created: 2026-06-18
|
|
||||||
updated: 2026-06-18
|
|
||||||
graph:
|
|
||||||
node_id: concept:coding-agent-setup
|
|
||||||
canonical: true
|
|
||||||
relationships:
|
|
||||||
- predicate: depends_on
|
|
||||||
object: concept:coding-agent
|
|
||||||
source: claude-code-local-cloud-models
|
|
||||||
evidence: "Setup je krok před použitím coding agenta"
|
|
||||||
confidence: high
|
|
||||||
status: current
|
|
||||||
---
|
|
||||||
|
|
||||||
# Coding agent setup
|
|
||||||
|
|
||||||
Koncept konfigurace a nastavení coding agentů pro praktické použití. Zahrnuje volbu modelu, API endpointu, nákladovou optimalizaci a workflow.
|
|
||||||
|
|
||||||
## Klíčové aspekty
|
|
||||||
|
|
||||||
- **Volba modelu** — lokální (Ollama) vs. cloud (OpenRouter, nativní API)
|
|
||||||
- **Konfigurace** — `.claude/settings.json`, env vars (`ANTHROPIC_MODEL`, `OPENAI_API_BASE`, `OPENAI_API_KEY`)
|
|
||||||
- **Nákladová optimalizace** — OpenRouter pro beast mode, Ollama pro zdarma
|
|
||||||
- **Workflow** — jak efektivně pracovat s agentem v terminálu
|
|
||||||
|
|
||||||
## Konfigurace Claude Code
|
|
||||||
|
|
||||||
1. **Ollama**: `OPENAI_API_BASE=http://localhost:11434/v1`, `OPENAI_API_KEY=ollama`
|
|
||||||
2. **OpenRouter**: `OPENAI_API_BASE=https://openrouter.ai/api/v1`, `OPENAI_API_KEY=<klíč>`
|
|
||||||
3. **Nativní API**: defaultní konfigurace, `ANTHROPIC_API_KEY`
|
|
||||||
|
|
||||||
## Související
|
|
||||||
|
|
||||||
- [[product-claude-code]] — hlavní coding agent
|
|
||||||
- [[product-ollama]] — lokální inference
|
|
||||||
- [[product-openrouter]] — cloud proxy
|
|
||||||
- [[local-vs-cloud-models]] — trade-offy
|
|
||||||
- [[cost-optimization]] — optimalizace nákladů
|
|
||||||
@@ -1,57 +0,0 @@
|
|||||||
---
|
|
||||||
type: concept
|
|
||||||
title: "Coding agent"
|
|
||||||
tags: [coding-agent, llm, tool-design, agent-architecture]
|
|
||||||
sources: [pi-coding-agent-mario-zechner, pi-dev-terminal-limitation, claude-code-local-cloud-models, claude-code-ollama-workflow, claude-code-openrouter-beast-mode, claude-code-goal-command]
|
|
||||||
created: 2026-06-16
|
|
||||||
updated: 2026-06-22
|
|
||||||
graph:
|
|
||||||
node_id: concept:coding-agent
|
|
||||||
canonical: true
|
|
||||||
relationships:
|
|
||||||
- predicate: depends_on
|
|
||||||
object: concept:llm
|
|
||||||
source: pi-coding-agent-mario-zechner
|
|
||||||
evidence: "Coding agenti využívají LLM jako jádro"
|
|
||||||
confidence: high
|
|
||||||
status: current
|
|
||||||
---
|
|
||||||
|
|
||||||
# Coding agent
|
|
||||||
|
|
||||||
Software nástroj, který využívá LLM k autonomnímu nebo poloautonomnímu psaní, úpravě a správě kódu. Typicky nabízí schopnosti jako čtení/zápis souborů, spouštění příkazů, vyhledávání v codebase a správu kontextu.
|
|
||||||
|
|
||||||
## Běžné problémy (podle Maria Zechnera)
|
|
||||||
|
|
||||||
- **Feature bloat** — agenti nabírají funkce, které nejsou potřeba a přidávají skrytou kontextovou injekci.
|
|
||||||
- **Skryté chování** — vendoři mění system prompty a kontext bez transparentnosti.
|
|
||||||
- **Špatná pozorovatelnost** — těžké vidět, co agent dělá a kolik stojí.
|
|
||||||
- **Chybějící rozšiřitelnost** — power user nemůže přidat vlastní nástroje bez forku.
|
|
||||||
- **Approval fatigue** — buď plná autonomie, nebo approval pro každou akci.
|
|
||||||
- **Špatná správa kontextu** — session compaction ztrácí důležitý kontext.
|
|
||||||
|
|
||||||
## Příklady
|
|
||||||
|
|
||||||
- [[product-pi]] — minimalistický agent (4 nástroje, tree-structured sessions)
|
|
||||||
- Claude Code, OpenCode, Codex CLI, AMP, Cline — zmínění konkurenti
|
|
||||||
|
|
||||||
## Modely a konfigurace
|
|
||||||
|
|
||||||
- [[local-vs-cloud-models]] — trade-offy mezi lokálními a cloud modely
|
|
||||||
- [[coding-agent-setup]] — konfigurace a nastavení
|
|
||||||
- [[cost-optimization]] — optimalizace nákladů na API
|
|
||||||
|
|
||||||
## Související
|
|
||||||
|
|
||||||
- [[tree-structured-sessions]] — Piův přístup ke správě kontextu
|
|
||||||
- [[terminal-limitation]] — společná limitace terminal-only agentů
|
|
||||||
- [[pi-coding-agent-mario-zechner]] — zdroj (talk)
|
|
||||||
- [[pi-dev-terminal-limitation]] — zdroj (poznámka o terminálové limitaci)
|
|
||||||
- [[claude-code-local-cloud-models]] — zdroj (Ollama/OpenRouter/llama.cpp setup)
|
|
||||||
- [[claude-code-ollama-workflow]] — zdroj (Ollama workflow návod)
|
|
||||||
- [[claude-code-openrouter-beast-mode]] — zdroj (OpenRouter beast mode)
|
|
||||||
- [[claude-code-goal-command]] — zdroj (/goal příkaz dokumentace)
|
|
||||||
|
|
||||||
## Autonomní běh
|
|
||||||
|
|
||||||
- [[goal-driven-agent-loop]] — koncept autonomního agenta s verifikovatelnou koncovou podmínkou
|
|
||||||
@@ -1,42 +0,0 @@
|
|||||||
---
|
|
||||||
type: concept
|
|
||||||
title: "Cost optimization"
|
|
||||||
tags: [llm, cost, api, coding-agent]
|
|
||||||
sources: [claude-code-openrouter-beast-mode]
|
|
||||||
created: 2026-06-18
|
|
||||||
updated: 2026-06-18
|
|
||||||
graph:
|
|
||||||
node_id: concept:cost-optimization
|
|
||||||
canonical: true
|
|
||||||
relationships:
|
|
||||||
- predicate: depends_on
|
|
||||||
object: concept:coding-agent
|
|
||||||
source: claude-code-openrouter-beast-mode
|
|
||||||
evidence: "Cost optimization je relevantní v kontextu coding agentů"
|
|
||||||
confidence: medium
|
|
||||||
status: current
|
|
||||||
---
|
|
||||||
|
|
||||||
# Cost optimization
|
|
||||||
|
|
||||||
Koncept optimalizace nákladů na LLM API při používání coding agentů. Klíčové pro dlouhodobě udržitelné používání.
|
|
||||||
|
|
||||||
## Strategie
|
|
||||||
|
|
||||||
- **OpenRouter proxy** — přístup k modelům za zlomek ceny nativního API
|
|
||||||
- **Lokální modely** — nulové API náklady, ale nižší kvalita
|
|
||||||
- **Model switching** — použití levnějších modelů pro jednoduché úkoly, výkonných pro komplexní
|
|
||||||
- **Beast mode** — cílené použití nejvýkonnějších modelů přes nízkonákladový proxy
|
|
||||||
|
|
||||||
## Praktické poznatky
|
|
||||||
|
|
||||||
- OpenRouter ceny jsou výrazně nižší než přímé API přístupy
|
|
||||||
- Lokální modely (3B–8B) jsou zdarma, ale kvalita stačí jen pro jednoduché úkoly
|
|
||||||
- Hybridní přístup (lokální pro rutinu, cloud pro komplexní úkoly) je nejefektivnější
|
|
||||||
|
|
||||||
## Související
|
|
||||||
|
|
||||||
- [[product-openrouter]] — klíčový nástroj pro cost optimization
|
|
||||||
- [[product-ollama]] — lokální alternativa
|
|
||||||
- [[local-vs-cloud-models]] — trade-offy
|
|
||||||
- [[coding-agent-setup]] — konfigurace
|
|
||||||
@@ -1,29 +0,0 @@
|
|||||||
---
|
|
||||||
type: concept
|
|
||||||
title: "E-waste reduction"
|
|
||||||
tags: [e-waste, sustainability, longevity, hardware]
|
|
||||||
sources: [source-lego-mindstorms-continued-use]
|
|
||||||
created: 2026-06-20
|
|
||||||
updated: 2026-06-20
|
|
||||||
graph:
|
|
||||||
node_id: concept-e-waste-reduction
|
|
||||||
canonical: true
|
|
||||||
edges:
|
|
||||||
- predicate: improves_on
|
|
||||||
object: concept-software-preservation
|
|
||||||
---
|
|
||||||
|
|
||||||
# E-waste reduction
|
|
||||||
|
|
||||||
Koncept prodloužení životnosti elektronických produktů — snižování množství e-waste tím, že hardware zůstává funkční i po ukončení oficiální softwarové podpory.
|
|
||||||
|
|
||||||
## Příklad: LEGO Mindstorms
|
|
||||||
|
|
||||||
Stovky tisíc až miliony sad Mindstorms po celém světě by se bez softwarové alternativy staly e-waste. [[product-pybricks]] tento trend obrací — open-source firmware dává EV3 brickům nový život s moderním programováním a okamžitým bootem.
|
|
||||||
|
|
||||||
Pybricks argumentuje, že technologie LEGO robotiky se za 20 let fundamentálně nezměnila — všechny sady mají smart hub, motory, senzory. Rozdíl je v softwarové zkušenosti, kterou lze obnovit.
|
|
||||||
|
|
||||||
## Odkazy
|
|
||||||
|
|
||||||
- [[source-lego-mindstorms-continued-use]] — zdroj o Mindstorms a Pybricks
|
|
||||||
- [[concept-software-preservation]] — softwarová stránka zachování produktů
|
|
||||||
@@ -1,38 +0,0 @@
|
|||||||
---
|
|
||||||
type: concept
|
|
||||||
title: "Goal-driven agent loop"
|
|
||||||
tags: [coding-agent, agent-loop, evaluation, autonomy, goal]
|
|
||||||
sources: [source-claude-code-goal-command]
|
|
||||||
created: 2026-06-22
|
|
||||||
updated: 2026-06-22
|
|
||||||
graph:
|
|
||||||
node_id: concept-goal-driven-agent-loop
|
|
||||||
canonical: true
|
|
||||||
---
|
|
||||||
|
|
||||||
# Goal-driven agent loop
|
|
||||||
|
|
||||||
Koncept autonomního agenta, který pracuje dokud není splněna verifikovatelná koncová podmínka. Po každém turnu nezávislý evaluátor (menší model) posoudí, zda cíl byl dosažen.
|
|
||||||
|
|
||||||
## Klíčové vlastnosti
|
|
||||||
|
|
||||||
- **Verifikovatelná podmínka** — cíl musí být měřitelný z výstupu agenta (test result, build exit code, file count)
|
|
||||||
- **Separátní evaluátor** — jiný model než agent sám posuzuje dokončení, čímž se eliminuje konflikt zájmů
|
|
||||||
- **Autonomní iterace** — agent pokračuje bez dalšího promptu uživatele, evaluátor poskytuje guidance pro další turn
|
|
||||||
- **Omezení běhu** — turn/time klauzule (např. "or stop after 20 turns") brání nekonečnému běhu
|
|
||||||
|
|
||||||
## Implementace v [[product-claude-code]]
|
|
||||||
|
|
||||||
Claude Code `/goal` příkaz: podmínka až 4000 znaků, evaluátor defaultně Haiku, funguje v interaktivním i non-interactive režimu. Komplementární s auto mode (schvaluje tool calls) — dohromady umožňují plně autonomní běh.
|
|
||||||
|
|
||||||
## Porovnání s jinými přístupy
|
|
||||||
|
|
||||||
- **`/loop`** — časový interval místo podmínky; vhodné pro opakující se úlohy
|
|
||||||
- **Stop hook** — vlastní skript/prompt pro evaluaci; flexibilnější ale složitější
|
|
||||||
- **Auto mode** — schvaluje tool calls v rámci turnu, ale nezačíná další turn
|
|
||||||
|
|
||||||
## Související
|
|
||||||
|
|
||||||
- [[source-claude-code-goal-command]] — zdroj (oficiální dokumentace)
|
|
||||||
- [[product-claude-code]] — implementace /goal příkazu
|
|
||||||
- [[coding-agent]] — obecný koncept
|
|
||||||
@@ -1,48 +0,0 @@
|
|||||||
---
|
|
||||||
type: concept
|
|
||||||
title: "Local vs. cloud models"
|
|
||||||
tags: [llm, local-models, cloud-models, cost, privacy]
|
|
||||||
sources: [claude-code-local-cloud-models, claude-code-ollama-workflow, claude-code-openrouter-beast-mode]
|
|
||||||
created: 2026-06-18
|
|
||||||
updated: 2026-06-18
|
|
||||||
graph:
|
|
||||||
node_id: concept:local-vs-cloud-models
|
|
||||||
canonical: true
|
|
||||||
relationships:
|
|
||||||
- predicate: depends_on
|
|
||||||
object: concept:coding-agent
|
|
||||||
source: claude-code-local-cloud-models
|
|
||||||
evidence: "Lokální vs. cloud modely jsou relevantní primárně v kontextu coding agentů"
|
|
||||||
confidence: medium
|
|
||||||
status: current
|
|
||||||
---
|
|
||||||
|
|
||||||
# Local vs. cloud models
|
|
||||||
|
|
||||||
Koncept volby mezi lokální inference (Ollama, llama.cpp) a cloud API (OpenRouter, nativní API) pro běh LLM modelů, zejména v kontextu coding agentů.
|
|
||||||
|
|
||||||
## Trade-offy
|
|
||||||
|
|
||||||
| Aspekt | Lokální (Ollama) | Cloud (OpenRouter) |
|
|
||||||
|--------|-------------------|-------------------|
|
|
||||||
| Náklady | Nulové (vlastní HW) | Pay-per-token |
|
|
||||||
| Soukromí | Plné | Omezené |
|
|
||||||
| Kvalita | Nižší (menší modely) | Vyšší (nejlepší modely) |
|
|
||||||
| Latence | Nízká (lokální) | Vyšší (síť) |
|
|
||||||
| Dostupnost | Závislá na HW | Vždy dostupné |
|
|
||||||
| Flexibilita | Omezená na lokální modely | Široký výběr |
|
|
||||||
|
|
||||||
## Praktické poznatky
|
|
||||||
|
|
||||||
- Pro jednoduché úkoly stačí lokální modely (3B–8B parametrů)
|
|
||||||
- Pro komplexní úkoly je cloud s výkonnými modely nezbytný
|
|
||||||
- OpenRouter umožňuje hybridní přístup — snadné přepínání mezi lokálními a cloud modely
|
|
||||||
- Beast mode = cloud s nejvýkonnějšími modely za nízkonákladový proxy
|
|
||||||
|
|
||||||
## Související
|
|
||||||
|
|
||||||
- [[product-ollama]] — lokální inference server
|
|
||||||
- [[product-openrouter]] — cloud proxy
|
|
||||||
- [[product-claude-code]] — coding agent podporující oba přístupy
|
|
||||||
- [[coding-agent-setup]] — koncept nastavení coding agentů
|
|
||||||
- [[cost-optimization]] — optimalizace nákladů
|
|
||||||
@@ -1,23 +0,0 @@
|
|||||||
---
|
|
||||||
type: concept
|
|
||||||
title: "MicroPython"
|
|
||||||
tags: [python, firmware, embedded, robotics]
|
|
||||||
sources: [source-lego-mindstorms-continued-use]
|
|
||||||
created: 2026-06-20
|
|
||||||
updated: 2026-06-20
|
|
||||||
graph:
|
|
||||||
node_id: concept-micropython
|
|
||||||
canonical: true
|
|
||||||
---
|
|
||||||
|
|
||||||
# MicroPython
|
|
||||||
|
|
||||||
Lehká implementace Pythonu 3 optimalizovaná pro mikrokontroléry a embedded zařízení. Používá [[product-pybricks]] jako programovací jazyk pro LEGO robotiku — nahrazuje proprietární LEGO software otevřenou alternativou s plnohodnotným Python API.
|
|
||||||
|
|
||||||
## V kontextu LEGO robotiky
|
|
||||||
|
|
||||||
Pybricks běží MicroPython přímo na LEGO hubech (EV3, Robot Inventor, SPIKE Prime). Uživatelé píší standardní Python kód, který se spouští na bricku v reálném čase — žádná závislost na cloudových službách nebo proprietárních aplikacích.
|
|
||||||
|
|
||||||
## Odkazy
|
|
||||||
|
|
||||||
- [[product-pybricks]] — implementace MicroPython pro LEGO huby
|
|
||||||
@@ -1,33 +0,0 @@
|
|||||||
---
|
|
||||||
type: concept
|
|
||||||
title: "Software preservation"
|
|
||||||
tags: [preservation, software, e-waste, longevity]
|
|
||||||
sources: [source-lego-mindstorms-continued-use]
|
|
||||||
created: 2026-06-20
|
|
||||||
updated: 2026-06-20
|
|
||||||
graph:
|
|
||||||
node_id: concept-software-preservation
|
|
||||||
canonical: true
|
|
||||||
edges:
|
|
||||||
- predicate: improves_on
|
|
||||||
object: concept-e-waste-reduction
|
|
||||||
---
|
|
||||||
|
|
||||||
# Software preservation
|
|
||||||
|
|
||||||
Koncept zachování softwaru a jeho funkčnosti po ukončení oficiální podpory. Kritický pro produkty, které závisí na aplikacích nebo serverech — bez softwaru se hardware stává e-waste.
|
|
||||||
|
|
||||||
## Problém
|
|
||||||
|
|
||||||
Jakýkoli gadget vyžadující počítač nebo telefon se rychle stává zastaralým, když původní aplikace přestanou fungovat na nových zařízeních. To platí i pro elektronické LEGO — Mindstorms aplikace mizí z app store a in-app content (tutoriály, build instrukce) je uložen v privátním app storage, který nelze zálohovat bez root přístupu.
|
|
||||||
|
|
||||||
## Řešení
|
|
||||||
|
|
||||||
- **Archivace instalátorů i app dat** — samotný APK/exe nestačí, potřebné jsou i in-app resources
|
|
||||||
- **Komunitní firmware** — [[product-pybricks]] nahrazuje oficiální aplikace open-source alternativou
|
|
||||||
- **Root přístup** — na Androidu nutný pro obnovu privátních app dat (blog robotmak3rs.com dokumentuje postupy)
|
|
||||||
|
|
||||||
## Odkazy
|
|
||||||
|
|
||||||
- [[source-lego-mindstorms-continued-use]] — případová studie Mindstorms
|
|
||||||
- [[concept-e-waste-reduction]] — širší kontext snižování e-waste
|
|
||||||
@@ -1,44 +0,0 @@
|
|||||||
---
|
|
||||||
type: concept
|
|
||||||
title: "Terminal limitation"
|
|
||||||
tags: [coding-agent, terminal, ide, ux, limitation]
|
|
||||||
sources: [pi-dev-terminal-limitation]
|
|
||||||
created: 2026-06-16
|
|
||||||
updated: 2026-06-16
|
|
||||||
graph:
|
|
||||||
node_id: concept:terminal-limitation
|
|
||||||
canonical: true
|
|
||||||
relationships:
|
|
||||||
- predicate: depends_on
|
|
||||||
object: concept:coding-agent
|
|
||||||
source: pi-dev-terminal-limitation
|
|
||||||
evidence: "Terminal limitation je problém specifický pro coding agenty"
|
|
||||||
confidence: high
|
|
||||||
status: current
|
|
||||||
---
|
|
||||||
|
|
||||||
# Terminal limitation
|
|
||||||
|
|
||||||
Koncept omezení coding agentů, kteří fungují pouze v terminálovém prostředí (terminal-only), bez plnohodnotné IDE integrace.
|
|
||||||
|
|
||||||
## Problém
|
|
||||||
|
|
||||||
Terminal-only přístup u coding agentů:
|
|
||||||
- **Ztrácí výhodné vlastnosti** — bez IDE chybí vizuální kontext, navigace v kódu, integrace s debuggerem atd.
|
|
||||||
- **Omezuje UX a produktivitu** — terminál není dostatečný pro komplexní interakci s kódem.
|
|
||||||
- **Je společný pro více agentů** — např. [[product-pi|Pi]] i OpenCode sdílejí tuto limitaci.
|
|
||||||
|
|
||||||
## Srovnání
|
|
||||||
|
|
||||||
| Aspekt | Terminal-only | IDE integrovaný |
|
|
||||||
|--------|--------------|-----------------|
|
|
||||||
| Vizuální kontext | Omezený | Plný |
|
|
||||||
| Navigace v kódu | Textová | Grafická |
|
|
||||||
| Debugging | Omezený | Plný |
|
|
||||||
| Rozšiřitelnost UI | Minimální | Plná |
|
|
||||||
|
|
||||||
## Související
|
|
||||||
|
|
||||||
- [[product-pi]] — Pi coding agent (terminal-only)
|
|
||||||
- [[coding-agent]] — obecný koncept
|
|
||||||
- [[pi-dev-terminal-limitation]] — zdroj (poznámka)
|
|
||||||
@@ -1,41 +0,0 @@
|
|||||||
---
|
|
||||||
type: concept
|
|
||||||
title: "Tree-structured sessions"
|
|
||||||
tags: [coding-agent, agent-architecture, context-management]
|
|
||||||
sources: [pi-coding-agent-mario-zechner]
|
|
||||||
created: 2026-06-16
|
|
||||||
updated: 2026-06-16
|
|
||||||
graph:
|
|
||||||
node_id: concept:tree-structured-sessions
|
|
||||||
relationships:
|
|
||||||
- predicate: depends_on
|
|
||||||
object: concept:coding-agent
|
|
||||||
source: pi-coding-agent-mario-zechner
|
|
||||||
evidence: "Tree-structured sessions jsou designový vzor pro coding agenty"
|
|
||||||
confidence: high
|
|
||||||
status: current
|
|
||||||
---
|
|
||||||
|
|
||||||
# Tree-structured sessions
|
|
||||||
|
|
||||||
Designový vzor pro správu kontextu v coding agentech, zavedený v [[product-pi|Pi]]. Místo lineární chat historie (kde se kontext komprimuje a ztrácí) se session větví jako strom — sub-agenti mohou nezávisle číst soubory a pracovat, přičemž zachovávají lineage a kontext rodičovské session.
|
|
||||||
|
|
||||||
## Problém, který řeší
|
|
||||||
|
|
||||||
Lineární chat historie v coding agentech vede k:
|
|
||||||
- Ztrátě důležitého kontextu při compaction
|
|
||||||
- Nemožnosti paralelně zkoumat různé větve řešení
|
|
||||||
- Nepružnému řízení — buď vše v jedné session, nebo nová session od nuly
|
|
||||||
|
|
||||||
## Princip
|
|
||||||
|
|
||||||
- Session je strom (tree), ne seznam (list).
|
|
||||||
- Sub-agent se může odvětvit od libovolného bodu v konverzaci.
|
|
||||||
- Každý uzel má přístup k souborům a může číst nezávisle.
|
|
||||||
- Lineage (původ) je zachována — lze sledovat, odkud sub-agent vznikl.
|
|
||||||
|
|
||||||
## Související
|
|
||||||
|
|
||||||
- [[coding-agent]] — obecný koncept
|
|
||||||
- [[product-pi]] — agent, který tento vzor implementuje
|
|
||||||
- [[pi-coding-agent-mario-zechner]] — zdroj (talk)
|
|
||||||
@@ -1,29 +0,0 @@
|
|||||||
---
|
|
||||||
type: entity
|
|
||||||
kind: person
|
|
||||||
title: "Mario Zechner"
|
|
||||||
tags: [person, developer, coding-agent, game-dev]
|
|
||||||
sources: [pi-coding-agent-mario-zechner]
|
|
||||||
created: 2026-06-16
|
|
||||||
updated: 2026-06-16
|
|
||||||
graph:
|
|
||||||
node_id: person:mario-zechner
|
|
||||||
canonical: true
|
|
||||||
relationships:
|
|
||||||
- predicate: works_on
|
|
||||||
object: product:pi
|
|
||||||
source: pi-coding-agent-mario-zechner
|
|
||||||
evidence: "Mario Zechner je tvůrce Pi coding agenta"
|
|
||||||
confidence: high
|
|
||||||
status: current
|
|
||||||
---
|
|
||||||
|
|
||||||
# Mario Zechner
|
|
||||||
|
|
||||||
Mario Zechner (aka badlogic) je vývojář a tvůrce Pi coding agenta. Je také autorem libGDX, populárního Java frameworku pro vývoj her.
|
|
||||||
|
|
||||||
## Související
|
|
||||||
|
|
||||||
- [[product-pi]] — coding agent, který vytvořil
|
|
||||||
- [[coding-agent]] — obecný koncept
|
|
||||||
- [[pi-coding-agent-mario-zechner]] — zdroj (talk)
|
|
||||||
@@ -1,42 +0,0 @@
|
|||||||
---
|
|
||||||
type: entity
|
|
||||||
kind: product
|
|
||||||
title: "Claude Code"
|
|
||||||
tags: [coding-agent, llm, tool, anthropic]
|
|
||||||
sources: [pi-coding-agent-mario-zechner, claude-code-local-cloud-models, claude-code-ollama-workflow, claude-code-openrouter-beast-mode, claude-code-goal-command]
|
|
||||||
created: 2026-06-17
|
|
||||||
updated: 2026-06-22
|
|
||||||
graph:
|
|
||||||
node_id: product:claude-code
|
|
||||||
canonical: true
|
|
||||||
---
|
|
||||||
|
|
||||||
# Claude Code
|
|
||||||
|
|
||||||
Coding agent od Anthropic, běžící v terminálu. Podporuje lokální modely (Ollama, llama.cpp) i cloud proxy (OpenRouter) pro cost optimization. Nabízí `/goal` příkaz pro autonomní práci s verifikovatelnou koncovou podmínkou.
|
|
||||||
|
|
||||||
## Klíčové funkce
|
|
||||||
|
|
||||||
- **`/goal` příkaz** — nastaví verifikovatelnou koncovou podmínku, agent pracuje autonomně dokud není splněna; evaluátor (defaultně Haiku) posuzuje dokončení po každém turnu
|
|
||||||
- **Auto mode** — automatické schvalování tool calls v rámci turnu
|
|
||||||
- **`/loop`** — opakované spouštění promptu v časovém intervalu
|
|
||||||
- **Non-interactive** — běh s `-p` flagou, desktop app, Remote Control
|
|
||||||
|
|
||||||
## Konfigurace modelů
|
|
||||||
|
|
||||||
- **Ollama** — lokální inference, nulové náklady, `OPENAI_API_BASE=http://localhost:11434/v1`
|
|
||||||
- **OpenRouter** — cloud proxy, pay-per-token, beast mode, `OPENAI_API_BASE=https://openrouter.ai/api/v1`
|
|
||||||
- **Nativní API** — defaultní, nejvyšší kvalita, nejvyšší náklady
|
|
||||||
|
|
||||||
## Související
|
|
||||||
|
|
||||||
- [[product-pi]] — konkurenční coding agent
|
|
||||||
- [[product-ollama]] — lokální inference server
|
|
||||||
- [[product-openrouter]] — cloud proxy
|
|
||||||
- [[product-cline]] — konkurenční VS Code agent
|
|
||||||
- [[coding-agent]] — obecný koncept
|
|
||||||
- [[local-vs-cloud-models]] — trade-offy
|
|
||||||
- [[coding-agent-setup]] — konfigurace
|
|
||||||
- [[cost-optimization]] — optimalizace nákladů
|
|
||||||
- [[goal-driven-agent-loop]] — koncept autonomního agenta s koncovou podmínkou
|
|
||||||
- [[claude-code-goal-command]] — zdroj (/goal dokumentace)
|
|
||||||
@@ -1,40 +0,0 @@
|
|||||||
---
|
|
||||||
type: entity
|
|
||||||
kind: product
|
|
||||||
title: "Cline"
|
|
||||||
tags: [coding-agent, llm, tool, vscode-extension]
|
|
||||||
sources: [claude-code-openrouter-beast-mode]
|
|
||||||
created: 2026-06-18
|
|
||||||
updated: 2026-06-18
|
|
||||||
graph:
|
|
||||||
node_id: product:cline
|
|
||||||
canonical: true
|
|
||||||
relationships:
|
|
||||||
- predicate: competes_with
|
|
||||||
object: product:claude-code
|
|
||||||
source: claude-code-openrouter-beast-mode
|
|
||||||
evidence: "Autor srovnává Claude Code s Cline — 'like Cline' v titulku článku"
|
|
||||||
confidence: medium
|
|
||||||
status: current
|
|
||||||
---
|
|
||||||
|
|
||||||
# Cline
|
|
||||||
|
|
||||||
VS Code rozšíření fungující jako coding agent. Podobný koncept jako Claude Code, ale integrovaný přímo do IDE (VS Code).
|
|
||||||
|
|
||||||
## Klíčové vlastnosti
|
|
||||||
|
|
||||||
- **VS Code integrace** — běží přímo v editoru, ne v terminálu
|
|
||||||
- **Multi-model** — podporuje různé LLM backendy přes API
|
|
||||||
- **Autonomní akce** — čte, píše a spouští kód v kontextu projektu
|
|
||||||
|
|
||||||
## Srovnání s Claude Code
|
|
||||||
|
|
||||||
- Cline = IDE integrovaný, Claude Code = terminálový
|
|
||||||
- Cline má vizuální kontext editoru, Claude Code má větší flexibilitu modelů
|
|
||||||
- Oba podporují OpenRouter pro cost optimization
|
|
||||||
|
|
||||||
## Související
|
|
||||||
|
|
||||||
- [[product-claude-code]] — konkurenční coding agent
|
|
||||||
- [[source-claude-code-openrouter-beast-mode]] — zdroj srovnání
|
|
||||||
@@ -1,43 +0,0 @@
|
|||||||
---
|
|
||||||
type: entity
|
|
||||||
title: "LEGO Mindstorms"
|
|
||||||
kind: product
|
|
||||||
tags: [lego, mindstorms, robotics, education, discontinued]
|
|
||||||
sources: [source-lego-mindstorms-continued-use]
|
|
||||||
created: 2026-06-20
|
|
||||||
updated: 2026-06-20
|
|
||||||
graph:
|
|
||||||
node_id: product-lego-mindstorms
|
|
||||||
canonical: true
|
|
||||||
edges:
|
|
||||||
- predicate: competes_with
|
|
||||||
object: product-spike-prime
|
|
||||||
- predicate: extended_by
|
|
||||||
object: product-pybricks
|
|
||||||
---
|
|
||||||
|
|
||||||
# LEGO Mindstorms
|
|
||||||
|
|
||||||
Sada robotických stavebnic od LEGO, oficiálně ukončená v říjnu 2022. Existuje ve verzích RCX (1998), NXT (2006), EV3 (2013) a Robot Inventor (2020). Celkem prodáno stovky tisíc až miliony sad po celém světě.
|
|
||||||
|
|
||||||
## Ukončení a důsledky
|
|
||||||
|
|
||||||
LEGO Group přesunul zdroje na SPIKE Prime a další produkty LEGO Education. Robot Inventor app měl zůstat dostupný do konce 2024, ale postupně přestává fungovat na novějších zařízeních. Oficiální aplikace mizí z app store.
|
|
||||||
|
|
||||||
Problém: elektronické LEGO má mnohem kratší životnost než klasické cihly, protože závisí na softwaru. Mnoho škol a FLL týmů stále závisí na EV3 — asi 60 % týmů v roce 2023.
|
|
||||||
|
|
||||||
## Nadále použitelné s [[product-pybricks]]
|
|
||||||
|
|
||||||
Pybricks nahrazuje oficiální aplikace a umožňuje nadále používat Mindstorms hardware moderním způsobem — sjednocuje programování napříč všemi generacemi.
|
|
||||||
|
|
||||||
## Hardware kompatibilita
|
|
||||||
|
|
||||||
- Motory a senzory jsou cross-kompatibilní mezi Mindstorms a SPIKE Prime
|
|
||||||
- Robot Inventor hub má stejný tvar jako SPIKE hub, ale SPIKE3 firmware na něj nejde nainstalovat
|
|
||||||
- EV3 brick s Pybricks bootuje okamžitě (místo desítek sekund s původním Linuxem)
|
|
||||||
|
|
||||||
## Odkazy
|
|
||||||
|
|
||||||
- [[source-lego-mindstorms-continued-use]] — zdroj o pokračování používání po ukončení
|
|
||||||
- [[product-pybricks]] — open-source firmware alternativa
|
|
||||||
- [[product-spike-prime]] — nástupce od LEGO Education
|
|
||||||
@@ -1,43 +0,0 @@
|
|||||||
---
|
|
||||||
type: entity
|
|
||||||
kind: product
|
|
||||||
title: "Ollama"
|
|
||||||
tags: [llm, inference, local-models, open-source]
|
|
||||||
sources: [claude-code-local-cloud-models, claude-code-ollama-workflow]
|
|
||||||
created: 2026-06-18
|
|
||||||
updated: 2026-06-18
|
|
||||||
graph:
|
|
||||||
node_id: product:ollama
|
|
||||||
canonical: true
|
|
||||||
relationships:
|
|
||||||
- predicate: competes_with
|
|
||||||
object: product:openrouter
|
|
||||||
source: claude-code-local-cloud-models
|
|
||||||
evidence: "Ollama a OpenRouter jsou alternativní způsoby připojení modelů k Claude Code"
|
|
||||||
confidence: medium
|
|
||||||
status: current
|
|
||||||
---
|
|
||||||
|
|
||||||
# Ollama
|
|
||||||
|
|
||||||
Lokální inference server pro běh LLM modelů. Podporuje širokou škálu modelů (Llama, Qwen, Mistral, Gemma aj.) a poskytuje OpenAI-compatible API endpoint.
|
|
||||||
|
|
||||||
## Klíčové vlastnosti
|
|
||||||
|
|
||||||
- **Lokální běh** — modely běží na vlastním hardware, žádné API náklady
|
|
||||||
- **OpenAI-compatible API** — snadné připojení z Claude Code a dalších nástrojů
|
|
||||||
- **Model management** — `ollama pull`, `ollama list`, `ollama run`
|
|
||||||
- **Široká podpora modelů** — Llama, Qwen, Mistral, Gemma, Phi a další
|
|
||||||
|
|
||||||
## Použití s Claude Code
|
|
||||||
|
|
||||||
- Nastav `OPENAI_API_BASE=http://localhost:11434/v1` a `OPENAI_API_KEY=ollama`
|
|
||||||
- Vyber model v `.claude/settings.json` nebo přes env var `ANTHROPIC_MODEL`
|
|
||||||
- Výhoda: nulové náklady, soukromí. Nevýhoda: nižší kvalita než cloud modely.
|
|
||||||
|
|
||||||
## Související
|
|
||||||
|
|
||||||
- [[product-claude-code]] — coding agent, který se připojuje k Ollama
|
|
||||||
- [[product-openrouter]] — cloud alternativa
|
|
||||||
- [[source-claude-code-local-cloud-models]] — přehledový článek
|
|
||||||
- [[source-claude-code-ollama-workflow]] — Ollama workflow návod
|
|
||||||
@@ -1,43 +0,0 @@
|
|||||||
---
|
|
||||||
type: entity
|
|
||||||
kind: product
|
|
||||||
title: "OpenRouter"
|
|
||||||
tags: [llm, api-proxy, cloud-models, cost-optimization]
|
|
||||||
sources: [claude-code-local-cloud-models, claude-code-openrouter-beast-mode]
|
|
||||||
created: 2026-06-18
|
|
||||||
updated: 2026-06-18
|
|
||||||
graph:
|
|
||||||
node_id: product:openrouter
|
|
||||||
canonical: true
|
|
||||||
relationships:
|
|
||||||
- predicate: competes_with
|
|
||||||
object: product:ollama
|
|
||||||
source: claude-code-local-cloud-models
|
|
||||||
evidence: "OpenRouter a Ollama jsou alternativní způsoby připojení modelů k Claude Code"
|
|
||||||
confidence: medium
|
|
||||||
status: current
|
|
||||||
---
|
|
||||||
|
|
||||||
# OpenRouter
|
|
||||||
|
|
||||||
Cloudový API proxy poskytující přístup k mnoha LLM modelům přes jednotné API. Umožňuje snadné přepínání mezi modely bez změny kódu.
|
|
||||||
|
|
||||||
## Klíčové vlastnosti
|
|
||||||
|
|
||||||
- **Jednotné API** — jeden endpoint pro Claude, GPT-4, Gemini, Mistral, Qwen a další
|
|
||||||
- **Pay-per-token** — platíš jen za spotřebované tokeny, žádné měsíční poplatky
|
|
||||||
- **Model switching** — snadné přepínání modelů v konfiguraci
|
|
||||||
- **Beast mode** — přístup k nejvýkonnějším modelům za zlomek ceny nativního API
|
|
||||||
|
|
||||||
## Použití s Claude Code
|
|
||||||
|
|
||||||
- Nastav `OPENAI_API_BASE=https://openrouter.ai/api/v1` a `OPENAI_API_KEY=<klíč>`
|
|
||||||
- Vyber model přes `model` v settings
|
|
||||||
- Výhoda: nízké náklady, široký výběr modelů. Nevýhoda: vyšší latence, rate limity.
|
|
||||||
|
|
||||||
## Související
|
|
||||||
|
|
||||||
- [[product-claude-code]] — coding agent, který se připojuje přes OpenRouter
|
|
||||||
- [[product-ollama]] — lokální alternativa
|
|
||||||
- [[source-claude-code-local-cloud-models]] — přehledový článek
|
|
||||||
- [[source-claude-code-openrouter-beast-mode]] — beast mode návod
|
|
||||||
@@ -1,49 +0,0 @@
|
|||||||
---
|
|
||||||
type: entity
|
|
||||||
kind: product
|
|
||||||
title: "Pi (coding agent)"
|
|
||||||
tags: [coding-agent, llm, tool, open-source]
|
|
||||||
sources: [pi-coding-agent-mario-zechner, pi-dev-terminal-limitation]
|
|
||||||
created: 2026-06-16
|
|
||||||
updated: 2026-06-16
|
|
||||||
graph:
|
|
||||||
node_id: product:pi
|
|
||||||
canonical: true
|
|
||||||
relationships:
|
|
||||||
- predicate: competes_with
|
|
||||||
object: product:claude-code
|
|
||||||
source: pi-coding-agent-mario-zechner
|
|
||||||
evidence: "Mario byl frustrován Claude Code a dalšími agenty"
|
|
||||||
confidence: medium
|
|
||||||
status: current
|
|
||||||
---
|
|
||||||
|
|
||||||
# Pi (coding agent)
|
|
||||||
|
|
||||||
Pi je minimalistický coding agent vytvořený Mariem Zechnerem (badlogic). Součást ekosystému OpenClaw.
|
|
||||||
|
|
||||||
## Design
|
|
||||||
|
|
||||||
- **4 nástroje**: read file, write file, edit file, bash — minimální jádro.
|
|
||||||
- **Malý system prompt** — frontier modely nepotřebují masivní prompty.
|
|
||||||
- **[[tree-structured-sessions]]** — větvení místo lineární historie; sub-agenti se mohou větvit a číst soubory nezávisle.
|
|
||||||
- **Full cost tracking** — vestavěný.
|
|
||||||
- **Hot-reloadable TypeScript extensions** — vlastní nástroje, UI, multi-agent setupy bez forku.
|
|
||||||
- **Žádná skrytá kontextová injekce** — transparentní.
|
|
||||||
- **Terminal-only** — limitace na terminálové prostředí bez IDE integrace; viz [[terminal-limitation]].
|
|
||||||
|
|
||||||
## Výkon
|
|
||||||
|
|
||||||
Na TerminalBench dosáhlo Pi (s Claude Opus 4.5) blízko Terminus i před pokročilými optimalizacemi.
|
|
||||||
|
|
||||||
## Odkazy
|
|
||||||
|
|
||||||
- Web: https://pi.dev/
|
|
||||||
- GitHub: https://github.com/earendil-works/pi
|
|
||||||
|
|
||||||
## Související
|
|
||||||
|
|
||||||
- [[person-mario-zechner]] — tvůrce
|
|
||||||
- [[coding-agent]] — obecný koncept
|
|
||||||
- [[pi-coding-agent-mario-zechner]] — zdroj (talk)
|
|
||||||
- [[pi-dev-terminal-limitation]] — zdroj (poznámka o terminálové limitaci)
|
|
||||||
@@ -1,49 +0,0 @@
|
|||||||
---
|
|
||||||
type: entity
|
|
||||||
title: "Pybricks"
|
|
||||||
kind: product
|
|
||||||
tags: [lego, mindstorms, firmware, open-source, robotics, python]
|
|
||||||
sources: [source-lego-mindstorms-continued-use]
|
|
||||||
created: 2026-06-20
|
|
||||||
updated: 2026-06-20
|
|
||||||
graph:
|
|
||||||
node_id: product-pybricks
|
|
||||||
canonical: true
|
|
||||||
edges:
|
|
||||||
- predicate: competes_with
|
|
||||||
object: product-spike-prime
|
|
||||||
- predicate: improves_on
|
|
||||||
object: concept-lego-mindstorms
|
|
||||||
- predicate: depends_on
|
|
||||||
object: concept-micropython
|
|
||||||
---
|
|
||||||
|
|
||||||
# Pybricks
|
|
||||||
|
|
||||||
Open-source firmware a vývojové prostředí pro LEGO robotiku — nahrazuje oficiální LEGO aplikace stabilnějším a lepším API. Podporuje všechny generace Mindstorms (NXT, EV3, Robot Inventor), SPIKE Prime, SPIKE Essential, BOOST, Powered Up a další LEGO huby.
|
|
||||||
|
|
||||||
## Klíčové vlastnosti
|
|
||||||
|
|
||||||
- **MicroPython + blokové programování** v prohlížeči — žádné instalace
|
|
||||||
- **Okamžitý boot** — na rozdíl od původního EV3 Linuxu (desítky sekund)
|
|
||||||
- **Univerzální API** napříč všemi LEGO huby
|
|
||||||
- **Bezplatný firmware**, volitelné placené doplňky (blokové programování)
|
|
||||||
- **Konverze bloků → Python** — na rozdíl od LEGO aplikace
|
|
||||||
- Běží na Chromeboocích, nepotřebuje instalaci
|
|
||||||
|
|
||||||
## Stav projektu EV3 (prosinec 2025)
|
|
||||||
|
|
||||||
Pybricks pro EV3 je v aktivním vývoji. Dosud implementováno: instant power on/off, MicroPython firmware bez microSD karty, program storage, download přes Pybricksdev, všechny EV3 motory a senzory, NXT senzory na EV3, custom UART/I2C/analog zařízení. Zbývá: USB/Bluetooth konektivita, browser-based firmware instalace.
|
|
||||||
|
|
||||||
## Vztah k [[product-lego-mindstorms]]
|
|
||||||
|
|
||||||
Pybricks je hlavní komunitní alternativa k ukončenému Mindstorms softwaru. Umožňuje nadále používat Mindstorms hardware moderním způsobem — sjednocuje programování napříč generacemi a eliminuje závislost na oficiálních aplikacích, které mizí z app store.
|
|
||||||
|
|
||||||
## Vztah k [[product-spike-prime]]
|
|
||||||
|
|
||||||
SPIKE3 firmware nelze nainstalovat na Mindstorms hub. Pybricks naopak funguje na obou — je to univerzální alternativa, která sjednocuje ekosystém LEGO robotiky.
|
|
||||||
|
|
||||||
## Odkazy
|
|
||||||
|
|
||||||
- [[source-lego-mindstorms-continued-use]] — zdroj o pokračování používání Mindstorms po ukončení
|
|
||||||
- [[concept-software-preservation]] — obecný koncept zachování softwaru
|
|
||||||
@@ -1,33 +0,0 @@
|
|||||||
---
|
|
||||||
type: entity
|
|
||||||
title: "SPIKE Prime"
|
|
||||||
kind: product
|
|
||||||
tags: [lego, spike, education, robotics]
|
|
||||||
sources: [source-lego-mindstorms-continued-use]
|
|
||||||
created: 2026-06-20
|
|
||||||
updated: 2026-06-20
|
|
||||||
graph:
|
|
||||||
node_id: product-spike-prime
|
|
||||||
canonical: true
|
|
||||||
edges:
|
|
||||||
- predicate: competes_with
|
|
||||||
object: product-lego-mindstorms
|
|
||||||
- predicate: extended_by
|
|
||||||
object: product-pybricks
|
|
||||||
---
|
|
||||||
|
|
||||||
# SPIKE Prime
|
|
||||||
|
|
||||||
Robotická vzdělávací sada od LEGO Education — oficiální nástupce [[product-lego-mindstorms]]. LEGO přesunulo zdroje z Mindstorms na SPIKE Prime po ukončení Mindstorms v říjnu 2022.
|
|
||||||
|
|
||||||
## Vztah k Mindstorms
|
|
||||||
|
|
||||||
- SPIKE Prime hub má stejný tvar jako Mindstorms Robot Inventor hub
|
|
||||||
- SPIKE2 firmware fungoval na Mindstorms hubu, ale SPIKE3 už ne — chyba při připojení
|
|
||||||
- Motory a senzory jsou cross-kompatibilní
|
|
||||||
- [[product-pybricks]] funguje na obou platformách a sjednocuje ekosystém
|
|
||||||
|
|
||||||
## Odkazy
|
|
||||||
|
|
||||||
- [[source-lego-mindstorms-continued-use]] — zdroj o ukončení Mindstorms a alternativách
|
|
||||||
- [[product-pybricks]] — univerzální alternativa pro obě platformy
|
|
||||||
2
cml/wiki/graph/.gitignore
vendored
2
cml/wiki/graph/.gitignore
vendored
@@ -1,2 +0,0 @@
|
|||||||
graph.sqlite
|
|
||||||
graph.graphml
|
|
||||||
@@ -1,32 +0,0 @@
|
|||||||
# Wiki Graph Layer
|
|
||||||
|
|
||||||
This directory holds the compiled knowledge graph derived from the markdown
|
|
||||||
wiki. **Markdown is canonical.** Everything here can be deleted and rebuilt
|
|
||||||
without losing knowledge:
|
|
||||||
|
|
||||||
```bash
|
|
||||||
python scripts/wiki_graph_extract.py wiki/ --out wiki/graph
|
|
||||||
```
|
|
||||||
|
|
||||||
## Files
|
|
||||||
|
|
||||||
| File | Purpose | Tracking |
|
|
||||||
|------|---------|----------|
|
|
||||||
| `ontology.yaml` | Declares node types and predicates the graph recognises. The contract `wiki_graph_lint.py` validates against. | **Tracked. Edit by hand.** |
|
|
||||||
| `nodes.jsonl` | One JSON object per node, sorted by id. | Generated. Track if you want graph diffs in PRs; otherwise gitignore. |
|
|
||||||
| `edges.jsonl` | One JSON object per edge, sorted by id. Includes typed semantic edges, `mentions`, `sourced_from`, and `summarizes_raw`. | Generated. Same trade-off as `nodes.jsonl`. |
|
|
||||||
| `graph.sqlite` | Queryable index used by `wiki_graph_query.py`. Schema: `nodes`, `aliases`, `edges`. | Generated. **Gitignored** — rebuild on demand. |
|
|
||||||
| `graph.graphml` | GraphML export for tools like Gephi or yEd. | Generated. Gitignored by default. |
|
|
||||||
|
|
||||||
## Workflow
|
|
||||||
|
|
||||||
1. Author or edit a wiki page. Add typed `graph.relationships` only when an explicit source supports them.
|
|
||||||
2. Run `python scripts/wiki_graph_lint.py wiki/` — catches unknown predicates, broken object references, missing evidence, alias collisions.
|
|
||||||
3. Run `python scripts/wiki_graph_extract.py wiki/ --out wiki/graph` — regenerates the artifacts above.
|
|
||||||
4. Query with `python scripts/wiki_graph_query.py wiki/ neighbors --node product:konvy` (or `edges`, `path`, `facts`).
|
|
||||||
|
|
||||||
## Anti-patterns
|
|
||||||
|
|
||||||
- **Hand-editing `nodes.jsonl` / `edges.jsonl` / `graph.sqlite`.** Edit the markdown; regenerate.
|
|
||||||
- **Treating graph rows as evidence.** They accelerate navigation. For high-stakes claims, follow the edge's `source` and `evidence` fields back to the wiki page and the raw source.
|
|
||||||
- **Adding typed edges the source doesn't support.** Use a normal `[[wikilink]]` instead — the `mentions` edge captures the connection without overclaiming.
|
|
||||||
@@ -1,148 +0,0 @@
|
|||||||
{"confidence": "high", "evidence": "", "extraction_method": "frontmatter_sources", "extras": {}, "id": "005b7b4456e37a361b1b4806", "object": "source:claude-code-openrouter-beast-mode", "page": "concepts/coding-agent-setup.md", "predicate": "sourced_from", "source": "claude-code-openrouter-beast-mode", "status": "current", "subject": "concept:coding-agent-setup"}
|
|
||||||
{"confidence": "high", "evidence": "", "extraction_method": "frontmatter_sources", "extras": {}, "id": "02295972088c9f5c8e30b3a8", "object": "source:claude-code-openrouter-beast-mode", "page": "entities/product-claude-code.md", "predicate": "sourced_from", "source": "claude-code-openrouter-beast-mode", "status": "current", "subject": "product:claude-code"}
|
|
||||||
{"confidence": "high", "evidence": "Článek testuje nový Ollama workflow v Claude Code", "extraction_method": "explicit_graph_frontmatter", "extras": {}, "id": "03072619e5187b95b6ea91f5", "object": "product:claude-code", "page": "sources/claude-code-ollama-workflow.md", "predicate": "mentions", "source": "claude-code-ollama-workflow", "status": "current", "subject": "source:claude-code-ollama-workflow"}
|
|
||||||
{"confidence": "high", "evidence": "", "extraction_method": "frontmatter_sources", "extras": {}, "id": "0473b080e02853af2e5d89c0", "object": "source:claude-code-ollama-workflow", "page": "entities/product-ollama.md", "predicate": "sourced_from", "source": "claude-code-ollama-workflow", "status": "current", "subject": "product:ollama"}
|
|
||||||
{"confidence": "low", "evidence": "", "extraction_method": "body_wikilink", "extras": {}, "id": "08037ca34f81ca2ac93364f8", "object": "concept:coding-agent", "page": "concepts/terminal-limitation.md", "predicate": "mentions", "source": "", "status": "current", "subject": "concept:terminal-limitation"}
|
|
||||||
{"confidence": "low", "evidence": "", "extraction_method": "body_wikilink", "extras": {}, "id": "09031c5b37673cdcf3f3bc1e", "object": "concept:local-vs-cloud-models", "page": "sources/claude-code-openrouter-beast-mode.md", "predicate": "mentions", "source": "", "status": "current", "subject": "source:claude-code-openrouter-beast-mode"}
|
|
||||||
{"confidence": "low", "evidence": "", "extraction_method": "body_wikilink", "extras": {}, "id": "09ba44d58a77188279ed87f3", "object": "concept:local-vs-cloud-models", "page": "entities/product-claude-code.md", "predicate": "mentions", "source": "", "status": "current", "subject": "product:claude-code"}
|
|
||||||
{"confidence": "low", "evidence": "", "extraction_method": "body_wikilink", "extras": {}, "id": "124b065e1f070573c072f0ee", "object": "product:ollama", "page": "sources/claude-code-ollama-workflow.md", "predicate": "mentions", "source": "", "status": "current", "subject": "source:claude-code-ollama-workflow"}
|
|
||||||
{"confidence": "low", "evidence": "", "extraction_method": "body_wikilink", "extras": {}, "id": "12db9977d67a261ecacec3af", "object": "product:ollama", "page": "sources/claude-code-local-cloud-models.md", "predicate": "mentions", "source": "", "status": "current", "subject": "source:claude-code-local-cloud-models"}
|
|
||||||
{"confidence": "low", "evidence": "", "extraction_method": "body_wikilink", "extras": {}, "id": "135e60cb41acf2f7be01a1f7", "object": "product-pybricks", "page": "concepts/micropython.md", "predicate": "mentions", "source": "", "status": "current", "subject": "concept-micropython"}
|
|
||||||
{"confidence": "low", "evidence": "", "extraction_method": "body_wikilink", "extras": {}, "id": "13aa325a2d4c93894e4b92cd", "object": "product:pi", "page": "sources/pi-coding-agent-mario-zechner.md", "predicate": "mentions", "source": "", "status": "current", "subject": "source:pi-coding-agent"}
|
|
||||||
{"confidence": "high", "evidence": "", "extraction_method": "frontmatter_sources", "extras": {}, "id": "14d0badac17898a4cbddc9c2", "object": "source:pi-coding-agent", "page": "concepts/tree-structured-sessions.md", "predicate": "sourced_from", "source": "pi-coding-agent-mario-zechner", "status": "current", "subject": "concept:tree-structured-sessions"}
|
|
||||||
{"confidence": "low", "evidence": "", "extraction_method": "body_wikilink", "extras": {}, "id": "159a0776ce947905f7c1f90b", "object": "product:claude-code", "page": "sources/claude-code-ollama-workflow.md", "predicate": "mentions", "source": "", "status": "current", "subject": "source:claude-code-ollama-workflow"}
|
|
||||||
{"confidence": "low", "evidence": "", "extraction_method": "body_wikilink", "extras": {}, "id": "18d439bccd8f98e6ad73391d", "object": "product:ollama", "page": "concepts/cost-optimization.md", "predicate": "mentions", "source": "", "status": "current", "subject": "concept:cost-optimization"}
|
|
||||||
{"confidence": "low", "evidence": "", "extraction_method": "body_wikilink", "extras": {}, "id": "191cc6717313156474e2f7a7", "object": "concept:coding-agent", "page": "concepts/tree-structured-sessions.md", "predicate": "mentions", "source": "", "status": "current", "subject": "concept:tree-structured-sessions"}
|
|
||||||
{"confidence": "low", "evidence": "", "extraction_method": "body_wikilink", "extras": {}, "id": "1e1d0ddc4cfe11ff12dc3b8a", "object": "product:cline", "page": "sources/claude-code-openrouter-beast-mode.md", "predicate": "mentions", "source": "", "status": "current", "subject": "source:claude-code-openrouter-beast-mode"}
|
|
||||||
{"confidence": "high", "evidence": "Coding agenti využívají LLM jako jádro", "extraction_method": "explicit_graph_frontmatter", "extras": {}, "id": "2400d909b4dd0cb5a5e3276c", "object": "concept:llm", "page": "concepts/coding-agent.md", "predicate": "depends_on", "source": "pi-coding-agent-mario-zechner", "status": "current", "subject": "concept:coding-agent"}
|
|
||||||
{"confidence": "low", "evidence": "", "extraction_method": "body_wikilink", "extras": {}, "id": "25260d522577cdaea172cdba", "object": "source:claude-code-local-cloud-models", "page": "concepts/coding-agent.md", "predicate": "mentions", "source": "", "status": "current", "subject": "concept:coding-agent"}
|
|
||||||
{"confidence": "low", "evidence": "", "extraction_method": "body_wikilink", "extras": {}, "id": "275303ea8901a8aaff1af046", "object": "concept-goal-driven-agent-loop", "page": "concepts/coding-agent.md", "predicate": "mentions", "source": "", "status": "current", "subject": "concept:coding-agent"}
|
|
||||||
{"confidence": "high", "evidence": "", "extraction_method": "frontmatter_sources", "extras": {}, "id": "2a7d8a0b7acdf510dfb3dbba", "object": "source:claude-code-local-cloud-models", "page": "entities/product-ollama.md", "predicate": "sourced_from", "source": "claude-code-local-cloud-models", "status": "current", "subject": "product:ollama"}
|
|
||||||
{"confidence": "low", "evidence": "", "extraction_method": "body_wikilink", "extras": {}, "id": "2a94474900b255b7a29e7b15", "object": "concept:tree-structured-sessions", "page": "sources/pi-coding-agent-mario-zechner.md", "predicate": "mentions", "source": "", "status": "current", "subject": "source:pi-coding-agent"}
|
|
||||||
{"confidence": "high", "evidence": "", "extraction_method": "frontmatter_raw", "extras": {}, "id": "2ac884ca271d51e826665a25", "object": "raw:raw/claude-code-openrouter-beast-mode-low-cost.md", "page": "sources/claude-code-openrouter-beast-mode.md", "predicate": "summarizes_raw", "source": "", "status": "current", "subject": "source:claude-code-openrouter-beast-mode"}
|
|
||||||
{"confidence": "low", "evidence": "", "extraction_method": "body_wikilink", "extras": {}, "id": "2c2accb65865d17b083a3a21", "object": "product:ollama", "page": "entities/product-openrouter.md", "predicate": "mentions", "source": "", "status": "current", "subject": "product:openrouter"}
|
|
||||||
{"confidence": "high", "evidence": "", "extraction_method": "frontmatter_sources", "extras": {}, "id": "2cd26a265d0666b7a0b9de36", "object": "source-claude-code-goal-command", "page": "entities/product-claude-code.md", "predicate": "sourced_from", "source": "claude-code-goal-command", "status": "current", "subject": "product:claude-code"}
|
|
||||||
{"confidence": "high", "evidence": "", "extraction_method": "frontmatter_sources", "extras": {}, "id": "2ee54ce8c92556a5f73655c2", "object": "source:pi-dev-terminal-limitation", "page": "entities/product-pi.md", "predicate": "sourced_from", "source": "pi-dev-terminal-limitation", "status": "current", "subject": "product:pi"}
|
|
||||||
{"confidence": "high", "evidence": "", "extraction_method": "frontmatter_sources", "extras": {}, "id": "327579315c7c5f6284938e40", "object": "source:claude-code-local-cloud-models", "page": "concepts/local-vs-cloud-models.md", "predicate": "sourced_from", "source": "claude-code-local-cloud-models", "status": "current", "subject": "concept:local-vs-cloud-models"}
|
|
||||||
{"confidence": "low", "evidence": "", "extraction_method": "body_wikilink", "extras": {}, "id": "32c73de25bd306c2f1dcf0d9", "object": "concept:cost-optimization", "page": "entities/product-claude-code.md", "predicate": "mentions", "source": "", "status": "current", "subject": "product:claude-code"}
|
|
||||||
{"confidence": "low", "evidence": "", "extraction_method": "body_wikilink", "extras": {}, "id": "362b7dc12de188b16d6e5e3e", "object": "concept:coding-agent", "page": "entities/product-claude-code.md", "predicate": "mentions", "source": "", "status": "current", "subject": "product:claude-code"}
|
|
||||||
{"confidence": "high", "evidence": "", "extraction_method": "frontmatter_sources", "extras": {}, "id": "3a43f524911111085adf6f60", "object": "source:claude-code-openrouter-beast-mode", "page": "concepts/cost-optimization.md", "predicate": "sourced_from", "source": "claude-code-openrouter-beast-mode", "status": "current", "subject": "concept:cost-optimization"}
|
|
||||||
{"confidence": "low", "evidence": "", "extraction_method": "body_wikilink", "extras": {}, "id": "3cd0d9cc13e931d9ed3e0d20", "object": "source:pi-coding-agent", "page": "entities/person-mario-zechner.md", "predicate": "mentions", "source": "", "status": "current", "subject": "person:mario-zechner"}
|
|
||||||
{"confidence": "low", "evidence": "", "extraction_method": "body_wikilink", "extras": {}, "id": "3d4aac4a18a108906501f410", "object": "product:pi", "page": "concepts/terminal-limitation.md", "predicate": "mentions", "source": "", "status": "current", "subject": "concept:terminal-limitation"}
|
|
||||||
{"confidence": "high", "evidence": "", "extraction_method": "frontmatter_sources", "extras": {}, "id": "42302f758d8d7576e768e918", "object": "source:claude-code-openrouter-beast-mode", "page": "entities/product-openrouter.md", "predicate": "sourced_from", "source": "claude-code-openrouter-beast-mode", "status": "current", "subject": "product:openrouter"}
|
|
||||||
{"confidence": "low", "evidence": "", "extraction_method": "body_wikilink", "extras": {}, "id": "42a621e62f258ae38c837785", "object": "concept:coding-agent-setup", "page": "concepts/local-vs-cloud-models.md", "predicate": "mentions", "source": "", "status": "current", "subject": "concept:local-vs-cloud-models"}
|
|
||||||
{"confidence": "low", "evidence": "", "extraction_method": "body_wikilink", "extras": {}, "id": "43612f5e8cdbd96c9eeeb93e", "object": "product:claude-code", "page": "entities/product-openrouter.md", "predicate": "mentions", "source": "", "status": "current", "subject": "product:openrouter"}
|
|
||||||
{"confidence": "low", "evidence": "", "extraction_method": "body_wikilink", "extras": {}, "id": "4513d0bea30cb68c9934bfbf", "object": "concept:cost-optimization", "page": "concepts/coding-agent-setup.md", "predicate": "mentions", "source": "", "status": "current", "subject": "concept:coding-agent-setup"}
|
|
||||||
{"confidence": "low", "evidence": "", "extraction_method": "body_wikilink", "extras": {}, "id": "49984558b8c8dcf6b3fff7ab", "object": "concept:local-vs-cloud-models", "page": "sources/claude-code-ollama-workflow.md", "predicate": "mentions", "source": "", "status": "current", "subject": "source:claude-code-ollama-workflow"}
|
|
||||||
{"confidence": "high", "evidence": "Mario Zechner je tvůrce Pi coding agenta", "extraction_method": "explicit_graph_frontmatter", "extras": {}, "id": "4a0a06df47d6853455a2c8d8", "object": "product:pi", "page": "entities/person-mario-zechner.md", "predicate": "works_on", "source": "pi-coding-agent-mario-zechner", "status": "current", "subject": "person:mario-zechner"}
|
|
||||||
{"confidence": "low", "evidence": "", "extraction_method": "body_wikilink", "extras": {}, "id": "4a209994f6e70117bfbf4db9", "object": "concept:local-vs-cloud-models", "page": "concepts/cost-optimization.md", "predicate": "mentions", "source": "", "status": "current", "subject": "concept:cost-optimization"}
|
|
||||||
{"confidence": "low", "evidence": "", "extraction_method": "body_wikilink", "extras": {}, "id": "4d07a788abafd2fcea1494a6", "object": "concept:coding-agent-setup", "page": "sources/claude-code-openrouter-beast-mode.md", "predicate": "mentions", "source": "", "status": "current", "subject": "source:claude-code-openrouter-beast-mode"}
|
|
||||||
{"confidence": "high", "evidence": "", "extraction_method": "frontmatter_sources", "extras": {}, "id": "4f17d5255cac0b991f04036d", "object": "source-claude-code-goal-command", "page": "concepts/coding-agent.md", "predicate": "sourced_from", "source": "claude-code-goal-command", "status": "current", "subject": "concept:coding-agent"}
|
|
||||||
{"confidence": "high", "evidence": "", "extraction_method": "frontmatter_sources", "extras": {}, "id": "4fd3af685ec3b1e84dd2ae6f", "object": "source:claude-code-ollama-workflow", "page": "concepts/local-vs-cloud-models.md", "predicate": "sourced_from", "source": "claude-code-ollama-workflow", "status": "current", "subject": "concept:local-vs-cloud-models"}
|
|
||||||
{"confidence": "low", "evidence": "", "extraction_method": "body_wikilink", "extras": {}, "id": "50d6477ee285ded1cd0fa6be", "object": "product-pybricks", "page": "entities/product-lego-mindstorms.md", "predicate": "mentions", "source": "", "status": "current", "subject": "product-lego-mindstorms"}
|
|
||||||
{"confidence": "high", "evidence": "", "extraction_method": "frontmatter_sources", "extras": {}, "id": "50e622a2c2bc2520041fc009", "object": "source:pi-dev-terminal-limitation", "page": "concepts/coding-agent.md", "predicate": "sourced_from", "source": "pi-dev-terminal-limitation", "status": "current", "subject": "concept:coding-agent"}
|
|
||||||
{"confidence": "low", "evidence": "", "extraction_method": "body_wikilink", "extras": {}, "id": "512096fc452b6bc938440709", "object": "concept:coding-agent-setup", "page": "entities/product-claude-code.md", "predicate": "mentions", "source": "", "status": "current", "subject": "product:claude-code"}
|
|
||||||
{"confidence": "low", "evidence": "", "extraction_method": "body_wikilink", "extras": {}, "id": "5190e9d269ce879b1abf1b07", "object": "product:openrouter", "page": "concepts/cost-optimization.md", "predicate": "mentions", "source": "", "status": "current", "subject": "concept:cost-optimization"}
|
|
||||||
{"confidence": "low", "evidence": "", "extraction_method": "body_wikilink", "extras": {}, "id": "52003666339d725a209b15c8", "object": "concept-goal-driven-agent-loop", "page": "entities/product-claude-code.md", "predicate": "mentions", "source": "", "status": "current", "subject": "product:claude-code"}
|
|
||||||
{"confidence": "low", "evidence": "", "extraction_method": "body_wikilink", "extras": {}, "id": "52eb9ba9326643e195b2957d", "object": "source:claude-code-openrouter-beast-mode", "page": "concepts/coding-agent.md", "predicate": "mentions", "source": "", "status": "current", "subject": "concept:coding-agent"}
|
|
||||||
{"confidence": "low", "evidence": "", "extraction_method": "body_wikilink", "extras": {}, "id": "531e12fbb0a73a0af2ec0373", "object": "product-pybricks", "page": "concepts/e-waste-reduction.md", "predicate": "mentions", "source": "", "status": "current", "subject": "concept-e-waste-reduction"}
|
|
||||||
{"confidence": "low", "evidence": "", "extraction_method": "body_wikilink", "extras": {}, "id": "546b7227bad05cd9b89c6cef", "object": "product:openrouter", "page": "sources/claude-code-local-cloud-models.md", "predicate": "mentions", "source": "", "status": "current", "subject": "source:claude-code-local-cloud-models"}
|
|
||||||
{"confidence": "high", "evidence": "", "extraction_method": "frontmatter_sources", "extras": {}, "id": "5646bcd4c869d872ea59aac0", "object": "source:claude-code-local-cloud-models", "page": "entities/product-claude-code.md", "predicate": "sourced_from", "source": "claude-code-local-cloud-models", "status": "current", "subject": "product:claude-code"}
|
|
||||||
{"confidence": "low", "evidence": "", "extraction_method": "body_wikilink", "extras": {}, "id": "567275b4ad011c88209bf785", "object": "source:pi-dev-terminal-limitation", "page": "entities/product-pi.md", "predicate": "mentions", "source": "", "status": "current", "subject": "product:pi"}
|
|
||||||
{"confidence": "high", "evidence": "", "extraction_method": "frontmatter_raw", "extras": {}, "id": "57e637043e0ada5ff5f1e9f5", "object": "raw:raw/pi-coding-agent-mario-zechner.md", "page": "sources/pi-coding-agent-mario-zechner.md", "predicate": "summarizes_raw", "source": "", "status": "current", "subject": "source:pi-coding-agent"}
|
|
||||||
{"confidence": "medium", "evidence": "OpenRouter a Ollama jsou alternativní způsoby připojení modelů k Claude Code", "extraction_method": "explicit_graph_frontmatter", "extras": {}, "id": "58e0d7ebed1a03b6a6dce2bb", "object": "product:ollama", "page": "entities/product-openrouter.md", "predicate": "competes_with", "source": "claude-code-local-cloud-models", "status": "current", "subject": "product:openrouter"}
|
|
||||||
{"confidence": "low", "evidence": "", "extraction_method": "body_wikilink", "extras": {}, "id": "592eb6b0c67596b642c57391", "object": "product:claude-code", "page": "concepts/local-vs-cloud-models.md", "predicate": "mentions", "source": "", "status": "current", "subject": "concept:local-vs-cloud-models"}
|
|
||||||
{"confidence": "low", "evidence": "", "extraction_method": "body_wikilink", "extras": {}, "id": "59ab4f4f0311802e87ee8c0b", "object": "concept:local-vs-cloud-models", "page": "sources/claude-code-local-cloud-models.md", "predicate": "mentions", "source": "", "status": "current", "subject": "source:claude-code-local-cloud-models"}
|
|
||||||
{"confidence": "high", "evidence": "Autor srovnává Claude Code s Cline — 'like Cline' v titulku", "extraction_method": "explicit_graph_frontmatter", "extras": {}, "id": "5aac916c2af1e38f2886d48a", "object": "product:cline", "page": "sources/claude-code-openrouter-beast-mode.md", "predicate": "mentions", "source": "claude-code-openrouter-beast-mode", "status": "current", "subject": "source:claude-code-openrouter-beast-mode"}
|
|
||||||
{"confidence": "medium", "evidence": "Mario byl frustrován Claude Code a dalšími agenty", "extraction_method": "explicit_graph_frontmatter", "extras": {}, "id": "5b91a8575682860e61f221bc", "object": "product:claude-code", "page": "entities/product-pi.md", "predicate": "competes_with", "source": "pi-coding-agent-mario-zechner", "status": "current", "subject": "product:pi"}
|
|
||||||
{"confidence": "high", "evidence": "", "extraction_method": "frontmatter_raw", "extras": {}, "id": "5e6aba6ee6e7c8a7dc775df5", "object": "raw:raw/claude-code-local-cloud-models-ollama-openrouter.md", "page": "sources/claude-code-local-cloud-models.md", "predicate": "summarizes_raw", "source": "", "status": "current", "subject": "source:claude-code-local-cloud-models"}
|
|
||||||
{"confidence": "high", "evidence": "", "extraction_method": "frontmatter_sources", "extras": {}, "id": "5ec9acf9e195faa93736e322", "object": "source:claude-code-local-cloud-models", "page": "concepts/coding-agent-setup.md", "predicate": "sourced_from", "source": "claude-code-local-cloud-models", "status": "current", "subject": "concept:coding-agent-setup"}
|
|
||||||
{"confidence": "low", "evidence": "", "extraction_method": "body_wikilink", "extras": {}, "id": "5edd59d4acd4bcdd28e6a22a", "object": "product:openrouter", "page": "concepts/coding-agent-setup.md", "predicate": "mentions", "source": "", "status": "current", "subject": "concept:coding-agent-setup"}
|
|
||||||
{"confidence": "low", "evidence": "", "extraction_method": "body_wikilink", "extras": {}, "id": "5f368dafb427845520bda202", "object": "concept:coding-agent-setup", "page": "sources/claude-code-ollama-workflow.md", "predicate": "mentions", "source": "", "status": "current", "subject": "source:claude-code-ollama-workflow"}
|
|
||||||
{"confidence": "low", "evidence": "", "extraction_method": "body_wikilink", "extras": {}, "id": "5f4eb5459b1557f8f36ac2d6", "object": "product:pi", "page": "concepts/tree-structured-sessions.md", "predicate": "mentions", "source": "", "status": "current", "subject": "concept:tree-structured-sessions"}
|
|
||||||
{"confidence": "high", "evidence": "", "extraction_method": "frontmatter_sources", "extras": {}, "id": "60b30be978df0475543ca366", "object": "source:claude-code-ollama-workflow", "page": "concepts/coding-agent-setup.md", "predicate": "sourced_from", "source": "claude-code-ollama-workflow", "status": "current", "subject": "concept:coding-agent-setup"}
|
|
||||||
{"confidence": "low", "evidence": "", "extraction_method": "body_wikilink", "extras": {}, "id": "60d34ac81a92bc21875c6542", "object": "concept:coding-agent", "page": "entities/person-mario-zechner.md", "predicate": "mentions", "source": "", "status": "current", "subject": "person:mario-zechner"}
|
|
||||||
{"confidence": "low", "evidence": "", "extraction_method": "body_wikilink", "extras": {}, "id": "6264a1240c1076a953e7cf24", "object": "concept:cost-optimization", "page": "sources/claude-code-openrouter-beast-mode.md", "predicate": "mentions", "source": "", "status": "current", "subject": "source:claude-code-openrouter-beast-mode"}
|
|
||||||
{"confidence": "high", "evidence": "Setup je krok před použitím coding agenta", "extraction_method": "explicit_graph_frontmatter", "extras": {}, "id": "645122ef8f8d9f56b40faab0", "object": "concept:coding-agent", "page": "concepts/coding-agent-setup.md", "predicate": "depends_on", "source": "claude-code-local-cloud-models", "status": "current", "subject": "concept:coding-agent-setup"}
|
|
||||||
{"confidence": "low", "evidence": "", "extraction_method": "body_wikilink", "extras": {}, "id": "6fc0769cff7ba42e1168ae4a", "object": "product-pybricks", "page": "entities/product-spike-prime.md", "predicate": "mentions", "source": "", "status": "current", "subject": "product-spike-prime"}
|
|
||||||
{"confidence": "low", "evidence": "", "extraction_method": "body_wikilink", "extras": {}, "id": "729c525e2e1e09f87bf24848", "object": "concept:coding-agent-setup", "page": "concepts/coding-agent.md", "predicate": "mentions", "source": "", "status": "current", "subject": "concept:coding-agent"}
|
|
||||||
{"confidence": "low", "evidence": "", "extraction_method": "body_wikilink", "extras": {}, "id": "72f5d24df5a450de6a318cd8", "object": "concept:coding-agent-setup", "page": "concepts/cost-optimization.md", "predicate": "mentions", "source": "", "status": "current", "subject": "concept:cost-optimization"}
|
|
||||||
{"confidence": "high", "evidence": "", "extraction_method": "frontmatter_sources", "extras": {}, "id": "7360fca40798521a714c42fc", "object": "source:claude-code-ollama-workflow", "page": "entities/product-claude-code.md", "predicate": "sourced_from", "source": "claude-code-ollama-workflow", "status": "current", "subject": "product:claude-code"}
|
|
||||||
{"confidence": "low", "evidence": "", "extraction_method": "body_wikilink", "extras": {}, "id": "7566d21d3ccc128d61c5cc1b", "object": "concept:terminal-limitation", "page": "concepts/coding-agent.md", "predicate": "mentions", "source": "", "status": "current", "subject": "concept:coding-agent"}
|
|
||||||
{"confidence": "low", "evidence": "", "extraction_method": "body_wikilink", "extras": {}, "id": "75fb4ae1c4b16e4c4ee0ee78", "object": "product-lego-mindstorms", "page": "entities/product-spike-prime.md", "predicate": "mentions", "source": "", "status": "current", "subject": "product-spike-prime"}
|
|
||||||
{"confidence": "low", "evidence": "", "extraction_method": "body_wikilink", "extras": {}, "id": "76237e017f821d894b1a6116", "object": "concept:cost-optimization", "page": "concepts/coding-agent.md", "predicate": "mentions", "source": "", "status": "current", "subject": "concept:coding-agent"}
|
|
||||||
{"confidence": "low", "evidence": "", "extraction_method": "body_wikilink", "extras": {}, "id": "787e5c76a93cbc50d6484511", "object": "product-spike-prime", "page": "entities/product-lego-mindstorms.md", "predicate": "mentions", "source": "", "status": "current", "subject": "product-lego-mindstorms"}
|
|
||||||
{"confidence": "low", "evidence": "", "extraction_method": "body_wikilink", "extras": {}, "id": "79a2cb96dacb70b421660a8c", "object": "product:openrouter", "page": "entities/product-claude-code.md", "predicate": "mentions", "source": "", "status": "current", "subject": "product:claude-code"}
|
|
||||||
{"confidence": "high", "evidence": "OpenRouter jako cloud provider pro Claude Code", "extraction_method": "explicit_graph_frontmatter", "extras": {}, "id": "7c89b56db95a048b011a0547", "object": "product:openrouter", "page": "sources/claude-code-local-cloud-models.md", "predicate": "mentions", "source": "claude-code-local-cloud-models", "status": "current", "subject": "source:claude-code-local-cloud-models"}
|
|
||||||
{"confidence": "low", "evidence": "", "extraction_method": "body_wikilink", "extras": {}, "id": "809e0e0d9a89f244543c2bca", "object": "concept:local-vs-cloud-models", "page": "concepts/coding-agent.md", "predicate": "mentions", "source": "", "status": "current", "subject": "concept:coding-agent"}
|
|
||||||
{"confidence": "high", "evidence": "OpenRouter jako klíčový enabler nízkonákladového beast mode", "extraction_method": "explicit_graph_frontmatter", "extras": {}, "id": "82e79892095d86a8d18987b8", "object": "product:openrouter", "page": "sources/claude-code-openrouter-beast-mode.md", "predicate": "mentions", "source": "claude-code-openrouter-beast-mode", "status": "current", "subject": "source:claude-code-openrouter-beast-mode"}
|
|
||||||
{"confidence": "low", "evidence": "", "extraction_method": "body_wikilink", "extras": {}, "id": "83320df3257a802400610e54", "object": "product:claude-code", "page": "concepts/coding-agent-setup.md", "predicate": "mentions", "source": "", "status": "current", "subject": "concept:coding-agent-setup"}
|
|
||||||
{"confidence": "high", "evidence": "", "extraction_method": "frontmatter_raw", "extras": {}, "id": "85635f92c9f1da522a6d9b59", "object": "raw:claude-code-goal-command.md", "page": "sources/claude-code-goal-command.md", "predicate": "summarizes_raw", "source": "", "status": "current", "subject": "source-claude-code-goal-command"}
|
|
||||||
{"confidence": "low", "evidence": "", "extraction_method": "body_wikilink", "extras": {}, "id": "86cf09b124c3e57966d68a65", "object": "person:mario-zechner", "page": "sources/pi-coding-agent-mario-zechner.md", "predicate": "mentions", "source": "", "status": "current", "subject": "source:pi-coding-agent"}
|
|
||||||
{"confidence": "low", "evidence": "", "extraction_method": "body_wikilink", "extras": {}, "id": "87f30dbadb049e7e9b3d0d0c", "object": "concept:coding-agent", "page": "sources/pi-coding-agent-mario-zechner.md", "predicate": "mentions", "source": "", "status": "current", "subject": "source:pi-coding-agent"}
|
|
||||||
{"confidence": "high", "evidence": "", "extraction_method": "frontmatter_sources", "extras": {}, "id": "89aa3247f6a7f12a65d9bd18", "object": "source:claude-code-openrouter-beast-mode", "page": "entities/product-cline.md", "predicate": "sourced_from", "source": "claude-code-openrouter-beast-mode", "status": "current", "subject": "product:cline"}
|
|
||||||
{"confidence": "low", "evidence": "", "extraction_method": "body_wikilink", "extras": {}, "id": "8b9cfec70b91547fd914badb", "object": "product:claude-code", "page": "entities/product-ollama.md", "predicate": "mentions", "source": "", "status": "current", "subject": "product:ollama"}
|
|
||||||
{"confidence": "low", "evidence": "", "extraction_method": "body_wikilink", "extras": {}, "id": "8ff7ad666e9f79012252dd6d", "object": "concept:tree-structured-sessions", "page": "concepts/coding-agent.md", "predicate": "mentions", "source": "", "status": "current", "subject": "concept:coding-agent"}
|
|
||||||
{"confidence": "medium", "evidence": "Cost optimization je relevantní v kontextu coding agentů", "extraction_method": "explicit_graph_frontmatter", "extras": {}, "id": "932df23980dd2aee2e1a7fb6", "object": "concept:coding-agent", "page": "concepts/cost-optimization.md", "predicate": "depends_on", "source": "claude-code-openrouter-beast-mode", "status": "current", "subject": "concept:cost-optimization"}
|
|
||||||
{"confidence": "low", "evidence": "", "extraction_method": "body_wikilink", "extras": {}, "id": "9659093034cbdbc62aba0b5a", "object": "product:pi", "page": "sources/pi-dev-terminal-limitation.md", "predicate": "mentions", "source": "", "status": "current", "subject": "source:pi-dev-terminal-limitation"}
|
|
||||||
{"confidence": "high", "evidence": "", "extraction_method": "frontmatter_raw", "extras": {}, "id": "97505665bd7fc6ba98aca3fd", "object": "raw:raw/claude-code-ollama-workflow-free.md", "page": "sources/claude-code-ollama-workflow.md", "predicate": "summarizes_raw", "source": "", "status": "current", "subject": "source:claude-code-ollama-workflow"}
|
|
||||||
{"confidence": "low", "evidence": "", "extraction_method": "body_wikilink", "extras": {}, "id": "979dedf44033e061301b9348", "object": "product:ollama", "page": "concepts/coding-agent-setup.md", "predicate": "mentions", "source": "", "status": "current", "subject": "concept:coding-agent-setup"}
|
|
||||||
{"confidence": "high", "evidence": "", "extraction_method": "frontmatter_sources", "extras": {}, "id": "98114d8ce66ff65616642270", "object": "source:pi-coding-agent", "page": "entities/product-pi.md", "predicate": "sourced_from", "source": "pi-coding-agent-mario-zechner", "status": "current", "subject": "product:pi"}
|
|
||||||
{"confidence": "low", "evidence": "", "extraction_method": "body_wikilink", "extras": {}, "id": "993e772405af6fb11991025b", "object": "product:ollama", "page": "concepts/local-vs-cloud-models.md", "predicate": "mentions", "source": "", "status": "current", "subject": "concept:local-vs-cloud-models"}
|
|
||||||
{"confidence": "high", "evidence": "Ollama jako lokální backend pro Claude Code", "extraction_method": "explicit_graph_frontmatter", "extras": {}, "id": "9998b438a162a6984d497e42", "object": "product:ollama", "page": "sources/claude-code-ollama-workflow.md", "predicate": "mentions", "source": "claude-code-ollama-workflow", "status": "current", "subject": "source:claude-code-ollama-workflow"}
|
|
||||||
{"confidence": "low", "evidence": "", "extraction_method": "body_wikilink", "extras": {}, "id": "9ae5839b2d57e5c1fe7d6af9", "object": "source:pi-dev-terminal-limitation", "page": "concepts/terminal-limitation.md", "predicate": "mentions", "source": "", "status": "current", "subject": "concept:terminal-limitation"}
|
|
||||||
{"confidence": "low", "evidence": "", "extraction_method": "body_wikilink", "extras": {}, "id": "9b50c578bae478c95541f820", "object": "product-spike-prime", "page": "entities/product-pybricks.md", "predicate": "mentions", "source": "", "status": "current", "subject": "product-pybricks"}
|
|
||||||
{"confidence": "high", "evidence": "", "extraction_method": "frontmatter_sources", "extras": {}, "id": "9ba7613061a708db9dbe8014", "object": "source:claude-code-local-cloud-models", "page": "concepts/coding-agent.md", "predicate": "sourced_from", "source": "claude-code-local-cloud-models", "status": "current", "subject": "concept:coding-agent"}
|
|
||||||
{"confidence": "low", "evidence": "", "extraction_method": "body_wikilink", "extras": {}, "id": "9d7aafb4c5d0e759c353e8ca", "object": "concept:coding-agent-setup", "page": "sources/claude-code-local-cloud-models.md", "predicate": "mentions", "source": "", "status": "current", "subject": "source:claude-code-local-cloud-models"}
|
|
||||||
{"confidence": "low", "evidence": "", "extraction_method": "body_wikilink", "extras": {}, "id": "9fd872a4975bc78d8c415957", "object": "product-pybricks", "page": "sources/lego-mindstorms-continued-use.md", "predicate": "mentions", "source": "", "status": "current", "subject": "source-lego-mindstorms-continued-use"}
|
|
||||||
{"confidence": "high", "evidence": "", "extraction_method": "frontmatter_sources", "extras": {}, "id": "a2b050146f72b027db4d4bdd", "object": "source:claude-code-local-cloud-models", "page": "entities/product-openrouter.md", "predicate": "sourced_from", "source": "claude-code-local-cloud-models", "status": "current", "subject": "product:openrouter"}
|
|
||||||
{"confidence": "high", "evidence": "", "extraction_method": "frontmatter_sources", "extras": {}, "id": "a573cd42f850a2f3cef5e6be", "object": "source:claude-code-openrouter-beast-mode", "page": "concepts/local-vs-cloud-models.md", "predicate": "sourced_from", "source": "claude-code-openrouter-beast-mode", "status": "current", "subject": "concept:local-vs-cloud-models"}
|
|
||||||
{"confidence": "high", "evidence": "", "extraction_method": "frontmatter_raw", "extras": {}, "id": "a6fbf9102cefbeb68aa8c00b", "object": "raw:lego-mindstorms-continued-use.md", "page": "sources/lego-mindstorms-continued-use.md", "predicate": "summarizes_raw", "source": "", "status": "current", "subject": "source-lego-mindstorms-continued-use"}
|
|
||||||
{"confidence": "low", "evidence": "", "extraction_method": "body_wikilink", "extras": {}, "id": "a76d8980968571164c707072", "object": "product:openrouter", "page": "sources/claude-code-openrouter-beast-mode.md", "predicate": "mentions", "source": "", "status": "current", "subject": "source:claude-code-openrouter-beast-mode"}
|
|
||||||
{"confidence": "low", "evidence": "", "extraction_method": "body_wikilink", "extras": {}, "id": "ad08b7552f80104c5ab23689", "object": "source:pi-dev-terminal-limitation", "page": "concepts/coding-agent.md", "predicate": "mentions", "source": "", "status": "current", "subject": "concept:coding-agent"}
|
|
||||||
{"confidence": "high", "evidence": "Článek popisuje konfiguraci Claude Code s lokálními a cloud modely", "extraction_method": "explicit_graph_frontmatter", "extras": {}, "id": "ad4b7b7470e8707aca5cfccd", "object": "product:claude-code", "page": "sources/claude-code-local-cloud-models.md", "predicate": "mentions", "source": "claude-code-local-cloud-models", "status": "current", "subject": "source:claude-code-local-cloud-models"}
|
|
||||||
{"confidence": "low", "evidence": "", "extraction_method": "body_wikilink", "extras": {}, "id": "ae31e15a7ae5193a2ba72275", "object": "product-lego-mindstorms", "page": "entities/product-pybricks.md", "predicate": "mentions", "source": "", "status": "current", "subject": "product-pybricks"}
|
|
||||||
{"confidence": "medium", "evidence": "Lokální vs. cloud modely jsou relevantní primárně v kontextu coding agentů", "extraction_method": "explicit_graph_frontmatter", "extras": {}, "id": "ae629e701056053be56434e6", "object": "concept:coding-agent", "page": "concepts/local-vs-cloud-models.md", "predicate": "depends_on", "source": "claude-code-local-cloud-models", "status": "current", "subject": "concept:local-vs-cloud-models"}
|
|
||||||
{"confidence": "low", "evidence": "", "extraction_method": "body_wikilink", "extras": {}, "id": "afb0d7f03ddbd4fd8d06f3b8", "object": "source:pi-coding-agent", "page": "entities/product-pi.md", "predicate": "mentions", "source": "", "status": "current", "subject": "product:pi"}
|
|
||||||
{"confidence": "low", "evidence": "", "extraction_method": "body_wikilink", "extras": {}, "id": "b26a3d7ad08193493ae62a44", "object": "source:claude-code-ollama-workflow", "page": "concepts/coding-agent.md", "predicate": "mentions", "source": "", "status": "current", "subject": "concept:coding-agent"}
|
|
||||||
{"confidence": "low", "evidence": "", "extraction_method": "body_wikilink", "extras": {}, "id": "b79dcf9de688c772d9dba6af", "object": "product:openrouter", "page": "entities/product-ollama.md", "predicate": "mentions", "source": "", "status": "current", "subject": "product:ollama"}
|
|
||||||
{"confidence": "low", "evidence": "", "extraction_method": "body_wikilink", "extras": {}, "id": "b7c72e20f108ecd9ec441ae3", "object": "product:pi", "page": "entities/person-mario-zechner.md", "predicate": "mentions", "source": "", "status": "current", "subject": "person:mario-zechner"}
|
|
||||||
{"confidence": "low", "evidence": "", "extraction_method": "body_wikilink", "extras": {}, "id": "bba333169c20cbea02f15d17", "object": "product:openrouter", "page": "concepts/local-vs-cloud-models.md", "predicate": "mentions", "source": "", "status": "current", "subject": "concept:local-vs-cloud-models"}
|
|
||||||
{"confidence": "low", "evidence": "", "extraction_method": "body_wikilink", "extras": {}, "id": "c06d1064205f533cfff2bdf9", "object": "concept:coding-agent-setup", "page": "sources/claude-code-goal-command.md", "predicate": "mentions", "source": "", "status": "current", "subject": "source-claude-code-goal-command"}
|
|
||||||
{"confidence": "low", "evidence": "", "extraction_method": "body_wikilink", "extras": {}, "id": "c1e3fc44723a7cadd70cac90", "object": "product:pi", "page": "entities/product-claude-code.md", "predicate": "mentions", "source": "", "status": "current", "subject": "product:claude-code"}
|
|
||||||
{"confidence": "low", "evidence": "", "extraction_method": "body_wikilink", "extras": {}, "id": "c23a69828bcd8f1f997fdf05", "object": "product:claude-code", "page": "entities/product-cline.md", "predicate": "mentions", "source": "", "status": "current", "subject": "product:cline"}
|
|
||||||
{"confidence": "low", "evidence": "", "extraction_method": "body_wikilink", "extras": {}, "id": "c314fe03504dd239c8fe4abe", "object": "source:pi-coding-agent", "page": "concepts/tree-structured-sessions.md", "predicate": "mentions", "source": "", "status": "current", "subject": "concept:tree-structured-sessions"}
|
|
||||||
{"confidence": "high", "evidence": "", "extraction_method": "frontmatter_sources", "extras": {}, "id": "c3bf9880f6571955c3d16e40", "object": "source:claude-code-ollama-workflow", "page": "concepts/coding-agent.md", "predicate": "sourced_from", "source": "claude-code-ollama-workflow", "status": "current", "subject": "concept:coding-agent"}
|
|
||||||
{"confidence": "low", "evidence": "", "extraction_method": "body_wikilink", "extras": {}, "id": "c50987e9426ceca82a18c140", "object": "product:pi", "page": "concepts/coding-agent.md", "predicate": "mentions", "source": "", "status": "current", "subject": "concept:coding-agent"}
|
|
||||||
{"confidence": "low", "evidence": "", "extraction_method": "body_wikilink", "extras": {}, "id": "c92c0cc40dbb685cc064afe8", "object": "person:mario-zechner", "page": "entities/product-pi.md", "predicate": "mentions", "source": "", "status": "current", "subject": "product:pi"}
|
|
||||||
{"confidence": "high", "evidence": "", "extraction_method": "frontmatter_sources", "extras": {}, "id": "d0f04257b2ce534883ae4237", "object": "source:pi-coding-agent", "page": "entities/person-mario-zechner.md", "predicate": "sourced_from", "source": "pi-coding-agent-mario-zechner", "status": "current", "subject": "person:mario-zechner"}
|
|
||||||
{"confidence": "high", "evidence": "", "extraction_method": "frontmatter_sources", "extras": {}, "id": "d1b0835488e15a18e90fa40e", "object": "source:pi-dev-terminal-limitation", "page": "concepts/terminal-limitation.md", "predicate": "sourced_from", "source": "pi-dev-terminal-limitation", "status": "current", "subject": "concept:terminal-limitation"}
|
|
||||||
{"confidence": "low", "evidence": "", "extraction_method": "body_wikilink", "extras": {}, "id": "d5ca6775fe73d816cd1c77eb", "object": "source-claude-code-goal-command", "page": "entities/product-claude-code.md", "predicate": "mentions", "source": "", "status": "current", "subject": "product:claude-code"}
|
|
||||||
{"confidence": "high", "evidence": "", "extraction_method": "frontmatter_sources", "extras": {}, "id": "d6e6f6572fb2bcad75fc0ecf", "object": "source:claude-code-openrouter-beast-mode", "page": "concepts/coding-agent.md", "predicate": "sourced_from", "source": "claude-code-openrouter-beast-mode", "status": "current", "subject": "concept:coding-agent"}
|
|
||||||
{"confidence": "medium", "evidence": "Ollama a OpenRouter jsou alternativní způsoby připojení modelů k Claude Code", "extraction_method": "explicit_graph_frontmatter", "extras": {}, "id": "d7cc7dac9bfc2d347b8ca3b8", "object": "product:openrouter", "page": "entities/product-ollama.md", "predicate": "competes_with", "source": "claude-code-local-cloud-models", "status": "current", "subject": "product:ollama"}
|
|
||||||
{"confidence": "low", "evidence": "", "extraction_method": "body_wikilink", "extras": {}, "id": "d92cde592d3e830e65055ca4", "object": "product:claude-code", "page": "sources/claude-code-local-cloud-models.md", "predicate": "mentions", "source": "", "status": "current", "subject": "source:claude-code-local-cloud-models"}
|
|
||||||
{"confidence": "low", "evidence": "", "extraction_method": "body_wikilink", "extras": {}, "id": "da1043334a481222e994ecde", "object": "concept:coding-agent", "page": "sources/pi-dev-terminal-limitation.md", "predicate": "mentions", "source": "", "status": "current", "subject": "source:pi-dev-terminal-limitation"}
|
|
||||||
{"confidence": "low", "evidence": "", "extraction_method": "body_wikilink", "extras": {}, "id": "db84f36f891a321aa001d16d", "object": "product-pybricks", "page": "concepts/software-preservation.md", "predicate": "mentions", "source": "", "status": "current", "subject": "concept-software-preservation"}
|
|
||||||
{"confidence": "low", "evidence": "", "extraction_method": "body_wikilink", "extras": {}, "id": "dcbb7d15a5ceceeb312b3acd", "object": "source:pi-coding-agent", "page": "concepts/coding-agent.md", "predicate": "mentions", "source": "", "status": "current", "subject": "concept:coding-agent"}
|
|
||||||
{"confidence": "low", "evidence": "", "extraction_method": "body_wikilink", "extras": {}, "id": "de0c3489258daf8043e93a66", "object": "product:claude-code", "page": "sources/claude-code-goal-command.md", "predicate": "mentions", "source": "", "status": "current", "subject": "source-claude-code-goal-command"}
|
|
||||||
{"confidence": "high", "evidence": "Tree-structured sessions jsou designový vzor pro coding agenty", "extraction_method": "explicit_graph_frontmatter", "extras": {}, "id": "df1c02ee6d035b348e0214e6", "object": "concept:coding-agent", "page": "concepts/tree-structured-sessions.md", "predicate": "depends_on", "source": "pi-coding-agent-mario-zechner", "status": "current", "subject": "concept:tree-structured-sessions"}
|
|
||||||
{"confidence": "high", "evidence": "", "extraction_method": "frontmatter_raw", "extras": {}, "id": "e026f2c43224691cf89df5a5", "object": "raw:raw/pi-dev-terminal-limitation.md", "page": "sources/pi-dev-terminal-limitation.md", "predicate": "summarizes_raw", "source": "", "status": "current", "subject": "source:pi-dev-terminal-limitation"}
|
|
||||||
{"confidence": "low", "evidence": "", "extraction_method": "body_wikilink", "extras": {}, "id": "e09eeb3421649feadf4e683e", "object": "concept:coding-agent", "page": "entities/product-pi.md", "predicate": "mentions", "source": "", "status": "current", "subject": "product:pi"}
|
|
||||||
{"confidence": "high", "evidence": "Terminal limitation je problém specifický pro coding agenty", "extraction_method": "explicit_graph_frontmatter", "extras": {}, "id": "e0ad6fc772543c636dfc1d61", "object": "concept:coding-agent", "page": "concepts/terminal-limitation.md", "predicate": "depends_on", "source": "pi-dev-terminal-limitation", "status": "current", "subject": "concept:terminal-limitation"}
|
|
||||||
{"confidence": "medium", "evidence": "Autor srovnává Claude Code s Cline — 'like Cline' v titulku článku", "extraction_method": "explicit_graph_frontmatter", "extras": {}, "id": "e1608a8914da09512210b3c3", "object": "product:claude-code", "page": "entities/product-cline.md", "predicate": "competes_with", "source": "claude-code-openrouter-beast-mode", "status": "current", "subject": "product:cline"}
|
|
||||||
{"confidence": "low", "evidence": "", "extraction_method": "body_wikilink", "extras": {}, "id": "e203de988033d0c526a205e5", "object": "concept:terminal-limitation", "page": "sources/pi-dev-terminal-limitation.md", "predicate": "mentions", "source": "", "status": "current", "subject": "source:pi-dev-terminal-limitation"}
|
|
||||||
{"confidence": "low", "evidence": "", "extraction_method": "body_wikilink", "extras": {}, "id": "e2f1bef6c7e27adb0a77c153", "object": "concept:terminal-limitation", "page": "entities/product-pi.md", "predicate": "mentions", "source": "", "status": "current", "subject": "product:pi"}
|
|
||||||
{"confidence": "high", "evidence": "", "extraction_method": "frontmatter_sources", "extras": {}, "id": "e61be465dee933fa6a894d05", "object": "source:pi-coding-agent", "page": "entities/product-claude-code.md", "predicate": "sourced_from", "source": "pi-coding-agent-mario-zechner", "status": "current", "subject": "product:claude-code"}
|
|
||||||
{"confidence": "high", "evidence": "Ollama jako jeden ze tří způsobů spuštění lokálních modelů", "extraction_method": "explicit_graph_frontmatter", "extras": {}, "id": "e999dfe34129fd0647dbeee0", "object": "product:ollama", "page": "sources/claude-code-local-cloud-models.md", "predicate": "mentions", "source": "claude-code-local-cloud-models", "status": "current", "subject": "source:claude-code-local-cloud-models"}
|
|
||||||
{"confidence": "low", "evidence": "", "extraction_method": "body_wikilink", "extras": {}, "id": "ea0867af72d27dba5b072a44", "object": "product:cline", "page": "entities/product-claude-code.md", "predicate": "mentions", "source": "", "status": "current", "subject": "product:claude-code"}
|
|
||||||
{"confidence": "low", "evidence": "", "extraction_method": "body_wikilink", "extras": {}, "id": "ea4a8353699665fea1dafca2", "object": "product:ollama", "page": "entities/product-claude-code.md", "predicate": "mentions", "source": "", "status": "current", "subject": "product:claude-code"}
|
|
||||||
{"confidence": "low", "evidence": "", "extraction_method": "body_wikilink", "extras": {}, "id": "eb6860e090815adfd902fccd", "object": "concept:coding-agent", "page": "sources/claude-code-goal-command.md", "predicate": "mentions", "source": "", "status": "current", "subject": "source-claude-code-goal-command"}
|
|
||||||
{"confidence": "low", "evidence": "", "extraction_method": "body_wikilink", "extras": {}, "id": "ec168c80d7ba7908d52ff348", "object": "source-claude-code-goal-command", "page": "concepts/coding-agent.md", "predicate": "mentions", "source": "", "status": "current", "subject": "concept:coding-agent"}
|
|
||||||
{"confidence": "high", "evidence": "", "extraction_method": "frontmatter_sources", "extras": {}, "id": "ed6693155ca18076d2f5a1ad", "object": "source:pi-coding-agent", "page": "concepts/coding-agent.md", "predicate": "sourced_from", "source": "pi-coding-agent-mario-zechner", "status": "current", "subject": "concept:coding-agent"}
|
|
||||||
{"confidence": "low", "evidence": "", "extraction_method": "body_wikilink", "extras": {}, "id": "f06ef238e753bec11f28a0ea", "object": "concept:coding-agent", "page": "concepts/goal-driven-agent-loop.md", "predicate": "mentions", "source": "", "status": "current", "subject": "concept-goal-driven-agent-loop"}
|
|
||||||
{"confidence": "low", "evidence": "", "extraction_method": "body_wikilink", "extras": {}, "id": "f0755eba6ae33e45cdf9959e", "object": "concept:tree-structured-sessions", "page": "entities/product-pi.md", "predicate": "mentions", "source": "", "status": "current", "subject": "product:pi"}
|
|
||||||
{"confidence": "high", "evidence": "Článek popisuje konfiguraci Claude Code s OpenRouter pro nízkonákladový beast mode", "extraction_method": "explicit_graph_frontmatter", "extras": {}, "id": "f30fc26cb5073c8e9ad3191d", "object": "product:claude-code", "page": "sources/claude-code-openrouter-beast-mode.md", "predicate": "mentions", "source": "claude-code-openrouter-beast-mode", "status": "current", "subject": "source:claude-code-openrouter-beast-mode"}
|
|
||||||
{"confidence": "low", "evidence": "", "extraction_method": "body_wikilink", "extras": {}, "id": "f33992aec3bc47e87845af67", "object": "concept:local-vs-cloud-models", "page": "concepts/coding-agent-setup.md", "predicate": "mentions", "source": "", "status": "current", "subject": "concept:coding-agent-setup"}
|
|
||||||
{"confidence": "low", "evidence": "", "extraction_method": "body_wikilink", "extras": {}, "id": "f773bf3fcae8749d1c8576f4", "object": "product:claude-code", "page": "sources/claude-code-openrouter-beast-mode.md", "predicate": "mentions", "source": "", "status": "current", "subject": "source:claude-code-openrouter-beast-mode"}
|
|
||||||
{"confidence": "low", "evidence": "", "extraction_method": "body_wikilink", "extras": {}, "id": "fc8418c79e357b10ea793919", "object": "product:claude-code", "page": "concepts/goal-driven-agent-loop.md", "predicate": "mentions", "source": "", "status": "current", "subject": "concept-goal-driven-agent-loop"}
|
|
||||||
{"confidence": "low", "evidence": "", "extraction_method": "body_wikilink", "extras": {}, "id": "ff771180caa2dc1b3c163c1a", "object": "concept:cost-optimization", "page": "concepts/local-vs-cloud-models.md", "predicate": "mentions", "source": "", "status": "current", "subject": "concept:local-vs-cloud-models"}
|
|
||||||
@@ -1,26 +0,0 @@
|
|||||||
{"aliases": [], "canonical": true, "created": "2026-06-20", "id": "concept-e-waste-reduction", "kind": "", "node_type": "concept", "page_type": "concept", "path": "concepts/e-waste-reduction.md", "slug": "e-waste-reduction", "tags": ["e-waste", "sustainability", "longevity", "hardware"], "title": "E-waste reduction", "updated": "2026-06-20"}
|
|
||||||
{"aliases": [], "canonical": true, "created": "2026-06-22", "id": "concept-goal-driven-agent-loop", "kind": "", "node_type": "concept", "page_type": "concept", "path": "concepts/goal-driven-agent-loop.md", "slug": "goal-driven-agent-loop", "tags": ["coding-agent", "agent-loop", "evaluation", "autonomy", "goal"], "title": "Goal-driven agent loop", "updated": "2026-06-22"}
|
|
||||||
{"aliases": [], "canonical": true, "created": "2026-06-20", "id": "concept-micropython", "kind": "", "node_type": "concept", "page_type": "concept", "path": "concepts/micropython.md", "slug": "micropython", "tags": ["python", "firmware", "embedded", "robotics"], "title": "MicroPython", "updated": "2026-06-20"}
|
|
||||||
{"aliases": [], "canonical": true, "created": "2026-06-20", "id": "concept-software-preservation", "kind": "", "node_type": "concept", "page_type": "concept", "path": "concepts/software-preservation.md", "slug": "software-preservation", "tags": ["preservation", "software", "e-waste", "longevity"], "title": "Software preservation", "updated": "2026-06-20"}
|
|
||||||
{"aliases": [], "canonical": true, "created": "2026-06-16", "id": "concept:coding-agent", "kind": "", "node_type": "concept", "page_type": "concept", "path": "concepts/coding-agent.md", "slug": "coding-agent", "tags": ["coding-agent", "llm", "tool-design", "agent-architecture"], "title": "Coding agent", "updated": "2026-06-22"}
|
|
||||||
{"aliases": [], "canonical": true, "created": "2026-06-18", "id": "concept:coding-agent-setup", "kind": "", "node_type": "concept", "page_type": "concept", "path": "concepts/coding-agent-setup.md", "slug": "coding-agent-setup", "tags": ["coding-agent", "setup", "configuration", "workflow"], "title": "Coding agent setup", "updated": "2026-06-18"}
|
|
||||||
{"aliases": [], "canonical": true, "created": "2026-06-18", "id": "concept:cost-optimization", "kind": "", "node_type": "concept", "page_type": "concept", "path": "concepts/cost-optimization.md", "slug": "cost-optimization", "tags": ["llm", "cost", "api", "coding-agent"], "title": "Cost optimization", "updated": "2026-06-18"}
|
|
||||||
{"aliases": [], "canonical": true, "created": "2026-06-18", "id": "concept:local-vs-cloud-models", "kind": "", "node_type": "concept", "page_type": "concept", "path": "concepts/local-vs-cloud-models.md", "slug": "local-vs-cloud-models", "tags": ["llm", "local-models", "cloud-models", "cost", "privacy"], "title": "Local vs. cloud models", "updated": "2026-06-18"}
|
|
||||||
{"aliases": [], "canonical": true, "created": "2026-06-16", "id": "concept:terminal-limitation", "kind": "", "node_type": "concept", "page_type": "concept", "path": "concepts/terminal-limitation.md", "slug": "terminal-limitation", "tags": ["coding-agent", "terminal", "ide", "ux", "limitation"], "title": "Terminal limitation", "updated": "2026-06-16"}
|
|
||||||
{"aliases": [], "canonical": false, "created": "2026-06-16", "id": "concept:tree-structured-sessions", "kind": "", "node_type": "concept", "page_type": "concept", "path": "concepts/tree-structured-sessions.md", "slug": "tree-structured-sessions", "tags": ["coding-agent", "agent-architecture", "context-management"], "title": "Tree-structured sessions", "updated": "2026-06-16"}
|
|
||||||
{"aliases": [], "canonical": true, "created": "2026-06-16", "id": "person:mario-zechner", "kind": "person", "node_type": "person", "page_type": "entity", "path": "entities/person-mario-zechner.md", "slug": "person-mario-zechner", "tags": ["person", "developer", "coding-agent", "game-dev"], "title": "Mario Zechner", "updated": "2026-06-16"}
|
|
||||||
{"aliases": [], "canonical": true, "created": "2026-06-20", "id": "product-lego-mindstorms", "kind": "product", "node_type": "product", "page_type": "entity", "path": "entities/product-lego-mindstorms.md", "slug": "product-lego-mindstorms", "tags": ["lego", "mindstorms", "robotics", "education", "discontinued"], "title": "LEGO Mindstorms", "updated": "2026-06-20"}
|
|
||||||
{"aliases": [], "canonical": true, "created": "2026-06-20", "id": "product-pybricks", "kind": "product", "node_type": "product", "page_type": "entity", "path": "entities/product-pybricks.md", "slug": "product-pybricks", "tags": ["lego", "mindstorms", "firmware", "open-source", "robotics", "python"], "title": "Pybricks", "updated": "2026-06-20"}
|
|
||||||
{"aliases": [], "canonical": true, "created": "2026-06-20", "id": "product-spike-prime", "kind": "product", "node_type": "product", "page_type": "entity", "path": "entities/product-spike-prime.md", "slug": "product-spike-prime", "tags": ["lego", "spike", "education", "robotics"], "title": "SPIKE Prime", "updated": "2026-06-20"}
|
|
||||||
{"aliases": [], "canonical": true, "created": "2026-06-17", "id": "product:claude-code", "kind": "product", "node_type": "product", "page_type": "entity", "path": "entities/product-claude-code.md", "slug": "product-claude-code", "tags": ["coding-agent", "llm", "tool", "anthropic"], "title": "Claude Code", "updated": "2026-06-22"}
|
|
||||||
{"aliases": [], "canonical": true, "created": "2026-06-18", "id": "product:cline", "kind": "product", "node_type": "product", "page_type": "entity", "path": "entities/product-cline.md", "slug": "product-cline", "tags": ["coding-agent", "llm", "tool", "vscode-extension"], "title": "Cline", "updated": "2026-06-18"}
|
|
||||||
{"aliases": [], "canonical": true, "created": "2026-06-18", "id": "product:ollama", "kind": "product", "node_type": "product", "page_type": "entity", "path": "entities/product-ollama.md", "slug": "product-ollama", "tags": ["llm", "inference", "local-models", "open-source"], "title": "Ollama", "updated": "2026-06-18"}
|
|
||||||
{"aliases": [], "canonical": true, "created": "2026-06-18", "id": "product:openrouter", "kind": "product", "node_type": "product", "page_type": "entity", "path": "entities/product-openrouter.md", "slug": "product-openrouter", "tags": ["llm", "api-proxy", "cloud-models", "cost-optimization"], "title": "OpenRouter", "updated": "2026-06-18"}
|
|
||||||
{"aliases": [], "canonical": true, "created": "2026-06-16", "id": "product:pi", "kind": "product", "node_type": "product", "page_type": "entity", "path": "entities/product-pi.md", "slug": "product-pi", "tags": ["coding-agent", "llm", "tool", "open-source"], "title": "Pi (coding agent)", "updated": "2026-06-16"}
|
|
||||||
{"aliases": [], "canonical": true, "created": "2026-06-22", "id": "source-claude-code-goal-command", "kind": "", "node_type": "source", "page_type": "source", "path": "sources/claude-code-goal-command.md", "slug": "claude-code-goal-command", "tags": ["claude-code", "coding-agent", "goal", "agent-loop", "evaluation", "anthropic"], "title": "Keep Claude working toward a goal — Claude Code /goal command", "updated": "2026-06-22"}
|
|
||||||
{"aliases": [], "canonical": true, "created": "2026-06-20", "id": "source-lego-mindstorms-continued-use", "kind": "", "node_type": "source", "page_type": "source", "path": "sources/lego-mindstorms-continued-use.md", "slug": "lego-mindstorms-continued-use", "tags": ["lego", "mindstorms", "robotics", "pybricks", "preservation", "firmware"], "title": "LEGO Mindstorms: continued use after discontinuation", "updated": "2026-06-20"}
|
|
||||||
{"aliases": [], "canonical": true, "created": "", "id": "source:claude-code-local-cloud-models", "kind": "", "node_type": "source", "page_type": "source", "path": "sources/claude-code-local-cloud-models.md", "slug": "claude-code-local-cloud-models", "tags": ["claude-code", "ollama", "openrouter", "local-models", "coding-agent", "setup"], "title": "Run Claude Code on Local & Cloud Models in 5 Minutes (Ollama, OpenRouter, llama.cpp)", "updated": ""}
|
|
||||||
{"aliases": [], "canonical": true, "created": "", "id": "source:claude-code-ollama-workflow", "kind": "", "node_type": "source", "page_type": "source", "path": "sources/claude-code-ollama-workflow.md", "slug": "claude-code-ollama-workflow", "tags": ["claude-code", "ollama", "local-models", "coding-agent", "workflow", "free-tier"], "title": "I Tried New Claude Code Ollama Workflow — It's Wild (Free)", "updated": ""}
|
|
||||||
{"aliases": [], "canonical": true, "created": "", "id": "source:claude-code-openrouter-beast-mode", "kind": "", "node_type": "source", "page_type": "source", "path": "sources/claude-code-openrouter-beast-mode.md", "slug": "claude-code-openrouter-beast-mode", "tags": ["claude-code", "openrouter", "cline", "coding-agent", "cost-optimization", "beast-mode"], "title": "How I'm Using Claude Code Like Cline with OpenRouter (Beast Mode, Low Cost)", "updated": ""}
|
|
||||||
{"aliases": [], "canonical": true, "created": "2026-06-16", "id": "source:pi-coding-agent", "kind": "", "node_type": "source", "page_type": "source", "path": "sources/pi-coding-agent-mario-zechner.md", "slug": "pi-coding-agent-mario-zechner", "tags": ["coding-agent", "llm", "tool-design", "agent-architecture", "video"], "title": "I Hated Every Coding Agent, So I Built My Own — Mario Zechner (Pi)", "updated": "2026-06-16"}
|
|
||||||
{"aliases": [], "canonical": true, "created": "2026-06-16", "id": "source:pi-dev-terminal-limitation", "kind": "", "node_type": "source", "page_type": "source", "path": "sources/pi-dev-terminal-limitation.md", "slug": "pi-dev-terminal-limitation", "tags": ["coding-agent", "terminal", "ide", "ux", "limitation"], "title": "pi.dev — terminálová limitace", "updated": "2026-06-16"}
|
|
||||||
@@ -1,135 +0,0 @@
|
|||||||
# Wiki Graph Ontology
|
|
||||||
#
|
|
||||||
# Declares the node types and predicates that the compiled graph layer
|
|
||||||
# (wiki/graph/) recognises. Edit this file when you introduce a new
|
|
||||||
# domain-specific predicate or node type — wiki_graph_lint.py reads it
|
|
||||||
# to validate every typed edge declared in page frontmatter.
|
|
||||||
#
|
|
||||||
# Markdown remains canonical. This file is just the contract that makes
|
|
||||||
# the graph layer machine-checkable.
|
|
||||||
|
|
||||||
node_types:
|
|
||||||
person:
|
|
||||||
maps_from:
|
|
||||||
type: entity
|
|
||||||
kind: person
|
|
||||||
company:
|
|
||||||
maps_from:
|
|
||||||
type: entity
|
|
||||||
kind: company
|
|
||||||
product:
|
|
||||||
maps_from:
|
|
||||||
type: entity
|
|
||||||
kind: product
|
|
||||||
paper:
|
|
||||||
maps_from:
|
|
||||||
type: entity
|
|
||||||
kind: paper
|
|
||||||
place:
|
|
||||||
maps_from:
|
|
||||||
type: entity
|
|
||||||
kind: place
|
|
||||||
organization:
|
|
||||||
maps_from:
|
|
||||||
type: entity
|
|
||||||
kind: organization
|
|
||||||
concept:
|
|
||||||
maps_from:
|
|
||||||
type: concept
|
|
||||||
source:
|
|
||||||
maps_from:
|
|
||||||
type: source
|
|
||||||
synthesis:
|
|
||||||
maps_from:
|
|
||||||
type: synthesis
|
|
||||||
decision:
|
|
||||||
explicit_only: true
|
|
||||||
claim:
|
|
||||||
explicit_only: true
|
|
||||||
raw:
|
|
||||||
explicit_only: true
|
|
||||||
|
|
||||||
predicates:
|
|
||||||
# --- Implicit predicates emitted by the extractor. ---
|
|
||||||
mentions:
|
|
||||||
subject_types: ["*"]
|
|
||||||
object_types: ["*"]
|
|
||||||
requires_evidence: false
|
|
||||||
description: |
|
|
||||||
Low-specificity edge derived from body wikilinks. Use it for
|
|
||||||
navigation, not as evidence of a typed relationship.
|
|
||||||
sourced_from:
|
|
||||||
subject_types: ["*"]
|
|
||||||
object_types: [source]
|
|
||||||
requires_evidence: false
|
|
||||||
description: |
|
|
||||||
Derived from each non-source page's frontmatter `sources:` list.
|
|
||||||
summarizes_raw:
|
|
||||||
subject_types: [source]
|
|
||||||
object_types: ["*"]
|
|
||||||
requires_evidence: false
|
|
||||||
description: |
|
|
||||||
Derived from a source page's frontmatter `raw:` field. Object is
|
|
||||||
the raw file path string, not a wiki node id.
|
|
||||||
|
|
||||||
# --- Typed semantic predicates. Add domain-specific ones below. ---
|
|
||||||
founded:
|
|
||||||
subject_types: [person]
|
|
||||||
object_types: [company, organization]
|
|
||||||
requires_evidence: true
|
|
||||||
owns:
|
|
||||||
subject_types: [person, company, organization]
|
|
||||||
object_types: [company, product, organization]
|
|
||||||
requires_evidence: true
|
|
||||||
contains_product:
|
|
||||||
subject_types: [company, organization]
|
|
||||||
object_types: [product]
|
|
||||||
requires_evidence: true
|
|
||||||
works_on:
|
|
||||||
subject_types: [person]
|
|
||||||
object_types: [product, concept]
|
|
||||||
requires_evidence: true
|
|
||||||
chose:
|
|
||||||
subject_types: [person, company, organization]
|
|
||||||
object_types: [product, concept]
|
|
||||||
requires_evidence: true
|
|
||||||
proposed:
|
|
||||||
subject_types: [person]
|
|
||||||
object_types: [decision, claim]
|
|
||||||
requires_evidence: true
|
|
||||||
competes_with:
|
|
||||||
subject_types: [product, company, organization]
|
|
||||||
object_types: [product, company, organization]
|
|
||||||
requires_evidence: true
|
|
||||||
depends_on:
|
|
||||||
subject_types: [product, concept]
|
|
||||||
object_types: [product, concept]
|
|
||||||
requires_evidence: true
|
|
||||||
authored:
|
|
||||||
subject_types: [person, organization]
|
|
||||||
object_types: [paper, source]
|
|
||||||
requires_evidence: true
|
|
||||||
cites:
|
|
||||||
subject_types: [paper, source, synthesis]
|
|
||||||
object_types: [paper, source]
|
|
||||||
requires_evidence: true
|
|
||||||
contradicts:
|
|
||||||
subject_types: [claim, source, synthesis]
|
|
||||||
object_types: [claim, source, synthesis]
|
|
||||||
requires_evidence: true
|
|
||||||
supersedes:
|
|
||||||
subject_types: [claim, source, decision]
|
|
||||||
object_types: [claim, source, decision]
|
|
||||||
requires_evidence: true
|
|
||||||
improves_on:
|
|
||||||
subject_types: [concept, product, claim, paper]
|
|
||||||
object_types: [concept, product, claim, paper]
|
|
||||||
requires_evidence: true
|
|
||||||
extended_by:
|
|
||||||
subject_types: [concept, product]
|
|
||||||
object_types: [concept, product]
|
|
||||||
requires_evidence: true
|
|
||||||
proposes:
|
|
||||||
subject_types: [paper, person]
|
|
||||||
object_types: [concept, product, claim]
|
|
||||||
requires_evidence: true
|
|
||||||
@@ -1,48 +0,0 @@
|
|||||||
# Wiki Index
|
|
||||||
|
|
||||||
The catalog of all pages in this wiki. Each entry: a wikilink to the page and a one-line summary. The LLM reads this first when answering queries to identify candidate pages.
|
|
||||||
|
|
||||||
Keep summaries tight — one line each. The index is engineered to be cheap to read; a fat index defeats its purpose.
|
|
||||||
|
|
||||||
When this file exceeds ~300 lines or the wiki passes ~150 pages, shard into `wiki/indexes/<type>.md` and replace this file with a directory of shards. See the `scaling-playbook.md` reference in the `llm-wiki` skill for the migration procedure.
|
|
||||||
|
|
||||||
---
|
|
||||||
|
|
||||||
## Sources
|
|
||||||
|
|
||||||
- [[pi-coding-agent-mario-zechner]] — talk Maria Zechnera o motivaci a designu Pi coding agenta
|
|
||||||
- [[pi-dev-terminal-limitation]] — poznámka o terminálové limitaci Pi a podobných agentů
|
|
||||||
- [[claude-code-local-cloud-models]] — průvodce konfigurací Claude Code s lokálními a cloud modely
|
|
||||||
- [[claude-code-ollama-workflow]] — test nového Ollama workflow v Claude Code (zdarma)
|
|
||||||
- [[claude-code-openrouter-beast-mode]] — Claude Code s OpenRouter pro beast mode při nízkých nákladech
|
|
||||||
- [[claude-code-goal-command]] — dokumentace /goal příkazu v Claude Code pro autonomní práci s koncovou podmínkou
|
|
||||||
- [[lego-mindstorms-continued-use]] — pokračování používání LEGO Mindstorms po ukončení — alternativy, firmware, zachování aplikací
|
|
||||||
|
|
||||||
## Entities
|
|
||||||
|
|
||||||
- [[person-mario-zechner]] — tvůrce Pi coding agenta, autor libGDX
|
|
||||||
- [[product-pi]] — minimalistický coding agent (4 nástroje, tree-structured sessions)
|
|
||||||
- [[product-claude-code]] — coding agent od Anthropic, podporuje lokální i cloud modely
|
|
||||||
- [[product-ollama]] — lokální inference server pro LLM modely
|
|
||||||
- [[product-openrouter]] — cloud API proxy pro přístup k mnoha LLM modelům
|
|
||||||
- [[product-cline]] — VS Code coding agent rozšíření
|
|
||||||
- [[product-pybricks]] — open-source firmware a vývojové prostředí pro LEGO robotiku (nahrazuje oficiální aplikace)
|
|
||||||
- [[product-lego-mindstorms]] — robotická stavebnice od LEGO, ukončená 2022, nadále použitelná s Pybricks
|
|
||||||
- [[product-spike-prime]] — robotická vzdělávací sada od LEGO Education, nástupce Mindstorms
|
|
||||||
|
|
||||||
## Concepts
|
|
||||||
|
|
||||||
- [[coding-agent]] — software nástroj využívající LLM k autonomnímu/poloautonomnímu psaní kódu
|
|
||||||
- [[tree-structured-sessions]] — designový vzor pro správu kontextu v coding agentech (větvení místo lineární historie)
|
|
||||||
- [[terminal-limitation]] — koncept omezení coding agentů na terminálové prostředí bez IDE integrace
|
|
||||||
- [[local-vs-cloud-models]] — trade-offy mezi lokálními a cloud LLM modely
|
|
||||||
- [[coding-agent-setup]] — konfigurace a nastavení coding agentů
|
|
||||||
- [[cost-optimization]] — optimalizace nákladů na LLM API
|
|
||||||
- [[goal-driven-agent-loop]] — koncept autonomního agenta s verifikovatelnou koncovou podmínkou
|
|
||||||
- [[software-preservation]] — zachování softwaru a funkčnosti po ukončení oficiální podpory
|
|
||||||
- [[e-waste-reduction]] — prodloužení životnosti elektronických produktů snížením e-waste
|
|
||||||
- [[micropython]] — lehká implementace Pythonu 3 pro mikrokontroléry, používá Pybricks
|
|
||||||
|
|
||||||
## Synthesis
|
|
||||||
|
|
||||||
(populated as query answers are filed back)
|
|
||||||
@@ -1,71 +0,0 @@
|
|||||||
# Wiki Log
|
|
||||||
|
|
||||||
## 2026-06-16 — Pi coding agent (talk Mario Zechner)
|
|
||||||
|
|
||||||
- Zdroj: `raw/pi-coding-agent-mario-zechner.md` (YouTube talk)
|
|
||||||
- Vytvořeny stránky: `sources/pi-coding-agent-mario-zechner.md`, `concepts/coding-agent.md`, `concepts/tree-structured-sessions.md`, `entities/person-mario-zechner.md`, `entities/product-pi.md`
|
|
||||||
|
|
||||||
## 2026-06-16 — Pi dev: terminal limitation
|
|
||||||
|
|
||||||
- Zdroj: `raw/pi-dev-terminal-limitation.md` (poznámka z pi.dev)
|
|
||||||
- Vytvořeny stránky: `sources/pi-dev-terminal-limitation.md`, `concepts/terminal-limitation.md`
|
|
||||||
|
|
||||||
## 2026-06-17 — Claude Code
|
|
||||||
|
|
||||||
- Vytvořena stránka: `entities/product-claude-code.md`
|
|
||||||
|
|
||||||
## 2026-06-17 — Cleanup
|
|
||||||
|
|
||||||
- Smazány testovací data o pozicových embeddingech (RoPE, ALiBi, YaRN, Flash Attention) — 18 souborů, 58 edges, 4 raw zdroje v _done
|
|
||||||
- Obnoveny nodes.jsonl a edges.jsonl na aktuální stav (8 nodes, 34 edges)
|
|
||||||
|
|
||||||
## [2026-06-18] ingest | Claude Code — lokální a cloud modely (3 články)
|
|
||||||
|
|
||||||
- Zdroje: `raw/claude-code-local-cloud-models-ollama-openrouter.md`, `raw/claude-code-ollama-workflow-free.md`, `raw/claude-code-openrouter-beast-mode-low-cost.md`
|
|
||||||
- Vytvořeny source stránky: `sources/claude-code-local-cloud-models.md`, `sources/claude-code-ollama-workflow.md`, `sources/claude-code-openrouter-beast-mode.md`
|
|
||||||
- Vytvořeny entity stránky: `entities/product-ollama.md`, `entities/product-openrouter.md`, `entities/product-cline.md`
|
|
||||||
- Vytvořeny concept stránky: `concepts/local-vs-cloud-models.md`, `concepts/coding-agent-setup.md`, `concepts/cost-optimization.md`
|
|
||||||
- Aktualizovány stránky: `entities/product-claude-code.md` (rozšířeno o modely/konfiguraci), `concepts/coding-agent.md` (přidány zdroje, odkazy)
|
|
||||||
- Aktualizován `index.md`
|
|
||||||
- Všechny 3 zdroje přesunuty do `_done/`
|
|
||||||
|
|
||||||
## [2026-06-18] lint + graph | After Claude Code sources ingest
|
|
||||||
|
|
||||||
Graph regenerated: 17 nodes, 125 edges
|
|
||||||
Lint report-only (no destructive edits):
|
|
||||||
- 1 broken object reference: concept:llm (page not yet created)
|
|
||||||
- No destructive edits applied — report-only per policy
|
|
||||||
## [2026-06-20] ingest | LEGO Mindstorms — continued use after discontinuation
|
|
||||||
|
|
||||||
- Zdroj: `raw/lego-mindstorms-continued-use.md` (blog robotmak3rs.com + Pybricks + Anton's Mindstorms + ToyBrands)
|
|
||||||
- Vytvořeny source stránky: `sources/lego-mindstorms-continued-use.md`
|
|
||||||
- Vytvořeny entity stránky: `entities/product-pybricks.md`, `entities/product-lego-mindstorms.md`, `entities/product-spike-prime.md`
|
|
||||||
- Vytvořeny concept stránky: `concepts/software-preservation.md`, `concepts/e-waste-reduction.md`, `concepts/micropython.md`
|
|
||||||
- Aktualizován `index.md` (přidány nové entity a koncepty)
|
|
||||||
- Zdroj přesunut do `_done/`
|
|
||||||
|
|
||||||
## [2026-06-22] ingest | Claude Code /goal command
|
|
||||||
|
|
||||||
- Zdroj: `raw/claude-code-goal-command.md` (oficiální dokumentace code.claude.com/docs/en/goal)
|
|
||||||
- Vytvořena source stránka: `sources/claude-code-goal-command.md`
|
|
||||||
- Vytvořena concept stránka: `concepts/goal-driven-agent-loop.md`
|
|
||||||
- Aktualizovány stránky: `entities/product-claude-code.md` (přidán /goal příkaz, nový zdroj), `concepts/coding-agent.md` (přidán zdroj, sekce o autonomním běhu)
|
|
||||||
- Aktualizován `index.md` (přidán zdroj a koncept)
|
|
||||||
- Zdroj přesunut do `_done/`
|
|
||||||
|
|
||||||
## [2026-06-20] graph | After LEGO Mindstorms ingest
|
|
||||||
|
|
||||||
Graph regenerated: 24 nodes, 136 edges
|
|
||||||
Lint report-only (no destructive edits):
|
|
||||||
- 1 broken object reference: concept:llm (page not yet created) — pre-existing
|
|
||||||
- 6 orphan typed nodes (new LEGO/Pybricks pages — expected, will gain edges as related sources are added)
|
|
||||||
- No destructive edits applied — report-only per policy
|
|
||||||
- No destructive edits applied — report-only per policy
|
|
||||||
|
|
||||||
## [2026-06-22] graph | After Claude Code /goal ingest
|
|
||||||
|
|
||||||
Graph regenerated: 26 nodes, 148 edges
|
|
||||||
Lint report-only (no destructive edits):
|
|
||||||
- 1 broken object reference: concept:llm (pre-existing, page not yet created)
|
|
||||||
- 7 orphan typed nodes (3 LEGO/Pybricks pages pre-existing, 1 new goal-driven-agent-loop, 3 expected — will gain edges as related sources are added)
|
|
||||||
- No destructive edits applied — report-only per policy
|
|
||||||
@@ -1,44 +0,0 @@
|
|||||||
---
|
|
||||||
type: source
|
|
||||||
title: "Keep Claude working toward a goal — Claude Code /goal command"
|
|
||||||
slug: source-claude-code-goal-command
|
|
||||||
tags: [claude-code, coding-agent, goal, agent-loop, evaluation, anthropic]
|
|
||||||
sources: []
|
|
||||||
raw: "claude-code-goal-command.md"
|
|
||||||
url: "https://code.claude.com/docs/en/goal"
|
|
||||||
created: 2026-06-22
|
|
||||||
updated: 2026-06-22
|
|
||||||
graph:
|
|
||||||
node_id: source-claude-code-goal-command
|
|
||||||
canonical: true
|
|
||||||
---
|
|
||||||
|
|
||||||
# Keep Claude working toward a goal — Claude Code /goal command
|
|
||||||
|
|
||||||
Oficiální dokumentace Claude Code k příkazu `/goal`, který nastavuje dokončovací podmínku a nechá agenta pracovat autonomně, dokud není splněna. Po každém turnu samostatný menší model (defaultně Haiku) vyhodnocuje, zda podmínka platí.
|
|
||||||
|
|
||||||
## Klíčové body
|
|
||||||
|
|
||||||
- **`/goal` příkaz** — nastaví verifikovatelnou koncovou podmínku; agent pokračuje v práci bez nutnosti dalšího promptu uživatele
|
|
||||||
- **Separátní evaluátor** — po každém turnu se podmínka a konverzace pošlou menšímu rychlému modelu (default Haiku), který vrací yes/no + krátký důvod. "No" znamená pokračuj, "yes" znamená cíl splněn
|
|
||||||
- **Efektivní podmínka** — měřitelný koncový stav (test result, build exit code, file count), explicitní způsob ověření, omezení co se nesmí změnit
|
|
||||||
- **Porovnání přístupů**: `/goal` (podmínka), `/loop` (časový interval), Stop hook (vlastní skript/prompt)
|
|
||||||
- **Komplementární s auto mode** — auto mode schvaluje tool calls v rámci jednoho turnu, `/goal` odstraňuje nutnost promptovat mezi turny
|
|
||||||
- **Non-interactive** — funguje s `-p` flagem, v desktop app, přes Remote Control
|
|
||||||
- **Resume** — aktivní goal se obnoví při `--resume` nebo `--continue`, s resetem turn count/timer/token spend
|
|
||||||
- **Omezení** — max 4000 znaků pro podmínku, vyžaduje accepted trust dialog, nefunguje s `disableAllHooks` nebo `allowManagedHooksOnly`
|
|
||||||
|
|
||||||
## Tři přístupy k udržení session
|
|
||||||
|
|
||||||
| Přístup | Další turn začíná když | Zastaví když |
|
|
||||||
|---------|----------------------|-------------|
|
|
||||||
| `/goal` | Předchozí turn skončí | Model potvrdí podmínku |
|
|
||||||
| `/loop` | Uplyne časový interval | Uživatel zastaví nebo agent rozhodne |
|
|
||||||
| Stop hook | Předchozí turn skončí | Vlastní skript/prompt rozhodne |
|
|
||||||
|
|
||||||
## Související
|
|
||||||
|
|
||||||
- [[product-claude-code]] — produkt, kde /goal funguje
|
|
||||||
- [[concept-goal-driven-agent-loop]] — koncept autonomního agenta s verifikovatelnou koncovou podmínkou
|
|
||||||
- [[coding-agent]] — obecný koncept coding agentů
|
|
||||||
- [[coding-agent-setup]] — konfigurace coding agentů
|
|
||||||
@@ -1,54 +0,0 @@
|
|||||||
---
|
|
||||||
type: source
|
|
||||||
title: "Run Claude Code on Local & Cloud Models in 5 Minutes (Ollama, OpenRouter, llama.cpp)"
|
|
||||||
authors: ["Luong Nguyen"]
|
|
||||||
url: "https://medium.com/@luongnv89/run-claude-code-on-local-cloud-models-in-5-minutes-ollama-openrouter-llama-cpp-6dfeaee03cda"
|
|
||||||
raw: "raw/claude-code-local-cloud-models-ollama-openrouter.md"
|
|
||||||
ingested: 2026-06-18
|
|
||||||
tags: [claude-code, ollama, openrouter, local-models, coding-agent, setup]
|
|
||||||
entities: [product-claude-code, product-ollama, product-openrouter]
|
|
||||||
concepts: [local-vs-cloud-models, coding-agent-setup]
|
|
||||||
slug: source-claude-code-local-cloud-models
|
|
||||||
graph:
|
|
||||||
node_id: source:claude-code-local-cloud-models
|
|
||||||
canonical: true
|
|
||||||
relationships:
|
|
||||||
- predicate: mentions
|
|
||||||
object: product:claude-code
|
|
||||||
source: claude-code-local-cloud-models
|
|
||||||
evidence: "Článek popisuje konfiguraci Claude Code s lokálními a cloud modely"
|
|
||||||
confidence: high
|
|
||||||
status: current
|
|
||||||
- predicate: mentions
|
|
||||||
object: product:ollama
|
|
||||||
source: claude-code-local-cloud-models
|
|
||||||
evidence: "Ollama jako jeden ze tří způsobů spuštění lokálních modelů"
|
|
||||||
confidence: high
|
|
||||||
status: current
|
|
||||||
- predicate: mentions
|
|
||||||
object: product:openrouter
|
|
||||||
source: claude-code-local-cloud-models
|
|
||||||
evidence: "OpenRouter jako cloud provider pro Claude Code"
|
|
||||||
confidence: high
|
|
||||||
status: current
|
|
||||||
---
|
|
||||||
|
|
||||||
# Run Claude Code on Local & Cloud Models in 5 Minutes
|
|
||||||
|
|
||||||
Průvodce konfigurací Claude Code s lokálními i cloud modely. Autor popisuje tři cesty: Ollama, OpenRouter a llama.cpp.
|
|
||||||
|
|
||||||
## Klíčové body
|
|
||||||
|
|
||||||
- **Ollama** — lokální inference server, podporuje širokou škálu modelů. Claude Code se připojí přes OpenAI-compatible API endpoint.
|
|
||||||
- **OpenRouter** — cloudový proxy poskytující přístup k mnoha modelům (včetně Claude) přes jednotné API. Umožňuje snadné přepínání modelů.
|
|
||||||
- **llama.cpp** — lightweight lokální inference, vhodná pro jednoduché setupy bez závislostí.
|
|
||||||
- **Konfigurace** — Claude Code podporuje `model` v `.claude/settings.json` nebo env var `ANTHROPIC_MODEL`. Pro lokální modely se nastavuje `OPENAI_API_BASE` a `OPENAI_API_KEY`.
|
|
||||||
- **Trade-offy** — lokální modely = soukromí a nulové náklady, ale nižší kvalita; cloud = vyšší kvalita, ale náklady a latence.
|
|
||||||
|
|
||||||
## Související
|
|
||||||
|
|
||||||
- [[product-claude-code]] — hlavní subjekt článku
|
|
||||||
- [[product-ollama]] — lokální inference server
|
|
||||||
- [[product-openrouter]] — cloud proxy
|
|
||||||
- [[local-vs-cloud-models]] — koncept lokálních vs. cloud modelů
|
|
||||||
- [[coding-agent-setup]] — koncept nastavení coding agentů
|
|
||||||
@@ -1,48 +0,0 @@
|
|||||||
---
|
|
||||||
type: source
|
|
||||||
title: "I Tried New Claude Code Ollama Workflow — It's Wild (Free)"
|
|
||||||
authors: ["Joe Njenga"]
|
|
||||||
url: "https://medium.com/@joe.njenga/i-tried-new-claude-code-ollama-workflow-its-wild-free-cb7a12b733b5"
|
|
||||||
raw: "raw/claude-code-ollama-workflow-free.md"
|
|
||||||
ingested: 2026-06-18
|
|
||||||
tags: [claude-code, ollama, local-models, coding-agent, workflow, free-tier]
|
|
||||||
entities: [product-claude-code, product-ollama]
|
|
||||||
concepts: [coding-agent-setup, local-vs-cloud-models]
|
|
||||||
slug: source-claude-code-ollama-workflow
|
|
||||||
graph:
|
|
||||||
node_id: source:claude-code-ollama-workflow
|
|
||||||
canonical: true
|
|
||||||
relationships:
|
|
||||||
- predicate: mentions
|
|
||||||
object: product:claude-code
|
|
||||||
source: claude-code-ollama-workflow
|
|
||||||
evidence: "Článek testuje nový Ollama workflow v Claude Code"
|
|
||||||
confidence: high
|
|
||||||
status: current
|
|
||||||
- predicate: mentions
|
|
||||||
object: product:ollama
|
|
||||||
source: claude-code-ollama-workflow
|
|
||||||
evidence: "Ollama jako lokální backend pro Claude Code"
|
|
||||||
confidence: high
|
|
||||||
status: current
|
|
||||||
---
|
|
||||||
|
|
||||||
# I Tried New Claude Code Ollama Workflow — It's Wild (Free)
|
|
||||||
|
|
||||||
Autor testuje nový Ollama workflow v Claude Code a popisuje, jak lze zdarma spouštět lokální modely přímo z Claude Code terminálu.
|
|
||||||
|
|
||||||
## Klíčové body
|
|
||||||
|
|
||||||
- **Ollama integration** — Claude Code nově podporuje nativní Ollama workflow. Stačí `ollama serve` a nastavit model.
|
|
||||||
- **Zdarma** — lokální modely přes Ollama = nulové API náklady. Autor zdůrazňuje "wild" fakt, že jde o plně funkční coding agent zdarma.
|
|
||||||
- **Workflow** — autor popisuje konkrétní kroky: instalace Ollama, pull modelu, konfigurace Claude Code, spuštění.
|
|
||||||
- **Omezení** — lokální modely (Qwen, Llama) mají nižší kvalitu než Claude, ale pro jednoduché úkoly dostačující.
|
|
||||||
- **Praktické tipy** — doporučuje začít s menšími modely (3B–8B) pro rychlost, větší (70B+) pro kvalitu.
|
|
||||||
|
|
||||||
## Související
|
|
||||||
|
|
||||||
- [[product-claude-code]] — hlavní subjekt
|
|
||||||
- [[product-ollama]] — lokální inference
|
|
||||||
- [[coding-agent-setup]] — koncept nastavení
|
|
||||||
- [[local-vs-cloud-models]] — lokální vs. cloud
|
|
||||||
- [[source-claude-code-local-cloud-models]] — související článek (širší přehled)
|
|
||||||
@@ -1,58 +0,0 @@
|
|||||||
---
|
|
||||||
type: source
|
|
||||||
title: "How I'm Using Claude Code Like Cline with OpenRouter (Beast Mode, Low Cost)"
|
|
||||||
authors: ["Joe Njenga"]
|
|
||||||
url: "https://medium.com/@joe.njenga/how-im-using-claude-code-like-cline-with-openrouter-to-go-beast-mode-at-low-cost-8c78e0bdcb67"
|
|
||||||
raw: "raw/claude-code-openrouter-beast-mode-low-cost.md"
|
|
||||||
ingested: 2026-06-18
|
|
||||||
tags: [claude-code, openrouter, cline, coding-agent, cost-optimization, beast-mode]
|
|
||||||
entities: [product-claude-code, product-openrouter, product-cline]
|
|
||||||
concepts: [coding-agent-setup, cost-optimization, local-vs-cloud-models]
|
|
||||||
slug: source-claude-code-openrouter-beast-mode
|
|
||||||
graph:
|
|
||||||
node_id: source:claude-code-openrouter-beast-mode
|
|
||||||
canonical: true
|
|
||||||
relationships:
|
|
||||||
- predicate: mentions
|
|
||||||
object: product:claude-code
|
|
||||||
source: claude-code-openrouter-beast-mode
|
|
||||||
evidence: "Článek popisuje konfiguraci Claude Code s OpenRouter pro nízkonákladový beast mode"
|
|
||||||
confidence: high
|
|
||||||
status: current
|
|
||||||
- predicate: mentions
|
|
||||||
object: product:openrouter
|
|
||||||
source: claude-code-openrouter-beast-mode
|
|
||||||
evidence: "OpenRouter jako klíčový enabler nízkonákladového beast mode"
|
|
||||||
confidence: high
|
|
||||||
status: current
|
|
||||||
- predicate: mentions
|
|
||||||
object: product:cline
|
|
||||||
source: claude-code-openrouter-beast-mode
|
|
||||||
evidence: "Autor srovnává Claude Code s Cline — 'like Cline' v titulku"
|
|
||||||
confidence: high
|
|
||||||
status: current
|
|
||||||
---
|
|
||||||
|
|
||||||
# How I'm Using Claude Code Like Cline with OpenRouter (Beast Mode, Low Cost)
|
|
||||||
|
|
||||||
Autor popisuje, jak nakonfigurovat Claude Code s OpenRouter pro "beast mode" — přístup k výkonným modelům za zlomek ceny nativního Claude API. Srovnává přístup s Cline.
|
|
||||||
|
|
||||||
## Klíčové body
|
|
||||||
|
|
||||||
- **OpenRouter jako proxy** — umožňuje přístup k mnoha modelům (Claude, GPT-4, Gemini, Mistral) přes jednotné API. Klíčové pro cost optimization.
|
|
||||||
- **Beast mode** — autor volí nejvýkonnější dostupné modely (Claude Opus, GPT-4) přes OpenRouter, ale platí jen za tokeny, které spotřebuje.
|
|
||||||
- **Cline-like workflow** — Claude Code v terminálu funguje podobně jako Cline (VS Code extension), ale s větší flexibilitou modelů.
|
|
||||||
- **Cost comparison** — OpenRouter ceny jsou výrazně nižší než přímé API přístupy. Autor uvádí konkrétní úspory.
|
|
||||||
- **Konfigurace** — nastavení `OPENAI_API_BASE` na OpenRouter endpoint, výběr modelu přes `model` v settings.
|
|
||||||
- **Trade-offy** — vyšší latence oproti nativnímu API, občasné rate limity, ale výrazně nižší náklady.
|
|
||||||
|
|
||||||
## Související
|
|
||||||
|
|
||||||
- [[product-claude-code]] — hlavní subjekt
|
|
||||||
- [[product-openrouter]] — cloud proxy
|
|
||||||
- [[product-cline]] — srovnávaný nástroj
|
|
||||||
- [[coding-agent-setup]] — koncept nastavení
|
|
||||||
- [[cost-optimization]] — optimalizace nákladů
|
|
||||||
- [[local-vs-cloud-models]] — lokální vs. cloud
|
|
||||||
- [[source-claude-code-local-cloud-models]] — širší přehled modelů
|
|
||||||
- [[source-claude-code-ollama-workflow]] — Ollama workflow
|
|
||||||
@@ -1,68 +0,0 @@
|
|||||||
---
|
|
||||||
type: source
|
|
||||||
title: "LEGO Mindstorms: continued use after discontinuation"
|
|
||||||
slug: source-lego-mindstorms-continued-use
|
|
||||||
tags: [lego, mindstorms, robotics, pybricks, preservation, firmware]
|
|
||||||
sources: []
|
|
||||||
raw: "lego-mindstorms-continued-use.md"
|
|
||||||
created: 2026-06-20
|
|
||||||
updated: 2026-06-20
|
|
||||||
graph:
|
|
||||||
node_id: source-lego-mindstorms-continued-use
|
|
||||||
canonical: true
|
|
||||||
---
|
|
||||||
|
|
||||||
# LEGO Mindstorms: continued use after discontinuation
|
|
||||||
|
|
||||||
Zdrojový materiál o tom, jak nadále používat produkty LEGO Mindstorms po jejich ukončení — včetně komunitních alternativ, firmware obnov a zachování aplikací.
|
|
||||||
|
|
||||||
## Kontext ukončení
|
|
||||||
|
|
||||||
LEGO Mindstorms byl oficiálně ukončen v říjnu 2022. LEGO Group přesunul zdroje na SPIKE Prime a další produkty LEGO Education. Robot Inventor app měl zůstat dostupný minimálně do konce 2024, ale postupně přestává fungovat na novějších zařízeních a platformách.
|
|
||||||
|
|
||||||
Problém je zásadní: existují stovky tisíc až miliony sady Mindstorms po celém světě. Mnoho škol a FIRST LEGO League týmů stále používá EV3 — asi 60 % týmů FLL v roce 2023 podle jednoho průzkumu. Elektronické LEGO má mnohem kratší životnost než klasické plastové cihly, protože závisí na softwaru a aplikacích, které rychle zastarávají.
|
|
||||||
|
|
||||||
## Alternativy a pokračování používání
|
|
||||||
|
|
||||||
### Pybricks — hlavní komunitní alternativa
|
|
||||||
|
|
||||||
[[product-pybricks]] je open-source firmware a vývojové prostředí, které nahrazuje oficiální LEGO aplikace. Klíčové vlastnosti:
|
|
||||||
|
|
||||||
- Funguje na všech generacích Mindstorms (NXT, EV3, Robot Inventor) i na SPIKE Prime a dalších Powered Up hubech
|
|
||||||
- MicroPython a blokové programování v prohlížeči — žádné instalace
|
|
||||||
- Okamžitý boot (na rozdíl od původního EV3 Linuxu, který startoval desítky sekund)
|
|
||||||
- Stabilnější a lepší API než oficiální aplikace
|
|
||||||
- Bezplatný firmware, volitelné placené doplňky (blokové programování)
|
|
||||||
- Podporuje všechny oficiální EV3 motory a senzory, plus NXT senzory na EV3 bricku
|
|
||||||
- Uživatelé mohou přispívat na Patreon a dostat své jméno do credits při vypínání EV3
|
|
||||||
|
|
||||||
Stav projektu Pybricks pro EV3 (k prosinci 2025): instant power on/off, MicroPython firmware bez microSD karty, program storage, download přes Pybricksdev, podpora všech EV3 motorů a senzorů, NXT senzory na EV3, custom UART/I2C/analog zařízení. Zbývá implementovat USB/Bluetooth konektivitu a browser-based firmware instalaci.
|
|
||||||
|
|
||||||
### Zachování oficiálních aplikací
|
|
||||||
|
|
||||||
Blog [[source-lego-mindstorms-continued-use]] (robotmak3rs.com) dokumentuje postupy obnovy oficiálních Mindstorms aplikací z archivovaných záloh:
|
|
||||||
|
|
||||||
- **Android**: Split APK instalace + obnova privátních app dat (vyžaduje root). Záloha obsahuje jak APK, tak stažený in-app content. ARM64 only.
|
|
||||||
- **macOS** a **Windows**: Obnova z archivovaných instalátorů (samostatné články na blog.robotmak3rs.com).
|
|
||||||
|
|
||||||
Klíčové poznání: samotný instalátor nestačí — in-app content (tutoriály, build instrukce) je uložen v privátním app storage a bez jeho zálohy nelze plně obnovit funkční stav aplikace.
|
|
||||||
|
|
||||||
### Kompatibilita hardware
|
|
||||||
|
|
||||||
- Mindstorms Robot Inventor hub má stejný tvar jako SPIKE Prime hub, ale SPIKE3 firmware na něj nejde nainstalovat (chyba při připojení)
|
|
||||||
- SPIKE2 firmware fungoval na Mindstorms hubu, ale aktuální SPIKE3 už ne
|
|
||||||
- Motory a senzory jsou cross-kompatibilní mezi Mindstorms a SPIKE Prime
|
|
||||||
- Robot Inventor set (51515) má stále dobrou play value — Anton's Mindstorms doporučuje koupit, pokud je dostupný za dobrou cenu
|
|
||||||
|
|
||||||
## Důsledky pro uživatele
|
|
||||||
|
|
||||||
- Školy a FLL týmy závislé na EV3 potřebují alternativu — Pybricks je nejlepší volba
|
|
||||||
- Druhový trh: zapečetěné EV3 sety se prodávají za dvojnásobek původní ceny
|
|
||||||
- Oficiální aplikace postupně mizí z app store — komunitní archivy jsou jediná záchrana
|
|
||||||
- Pybricks sjednocuje programování napříč všemi generacemi LEGO robotiky
|
|
||||||
|
|
||||||
## Where this fits
|
|
||||||
|
|
||||||
- [[product-pybricks]] — open-source firmware alternativa
|
|
||||||
- [[concept-software-preservation]] — obecný koncept zachování softwaru po ukončení podpory
|
|
||||||
- [[concept-e-waste-reduction]] — prodloužení životnosti elektronických produktů
|
|
||||||
@@ -1,67 +0,0 @@
|
|||||||
---
|
|
||||||
type: source
|
|
||||||
title: "I Hated Every Coding Agent, So I Built My Own — Mario Zechner (Pi)"
|
|
||||||
authors: ["Mario Zechner"]
|
|
||||||
url: "https://www.youtube.com/watch?v=Dli5slNaJu0"
|
|
||||||
raw: "raw/pi-coding-agent-mario-zechner.md"
|
|
||||||
ingested: 2026-06-16
|
|
||||||
created: 2026-06-16
|
|
||||||
updated: 2026-06-16
|
|
||||||
tags: [coding-agent, llm, tool-design, agent-architecture, video]
|
|
||||||
entities: [person-mario-zechner, product-pi]
|
|
||||||
concepts: [coding-agent, tree-structured-sessions]
|
|
||||||
slug: pi-coding-agent
|
|
||||||
graph:
|
|
||||||
node_id: source:pi-coding-agent
|
|
||||||
canonical: true
|
|
||||||
canonical: true
|
|
||||||
---
|
|
||||||
|
|
||||||
# I Hated Every Coding Agent, So I Built My Own — Mario Zechner (Pi)
|
|
||||||
|
|
||||||
Shrnutí talku Maria Zechnera (tvůrce Pi coding agenta, aka badlogic — autor libGDX) o motivaci a designu Pi.
|
|
||||||
|
|
||||||
## Klíčové body
|
|
||||||
|
|
||||||
### Proč Pi vzniklo
|
|
||||||
|
|
||||||
Mario byl frustrován existujícími coding agenty (Claude Code, OpenCode, Codex CLI, AMP) z několika důvodů:
|
|
||||||
|
|
||||||
1. **Feature bloat** — agenti nabírají funkce (to-dos, komplexní tool suites), které nejsou potřeba a přidávají skrytou kontextovou injekci.
|
|
||||||
2. **Skryté chování** — vendoři mění věci pod pokličkou (system prompty, kontextová injekce), což způsobuje nepředvídatelné chování LLM.
|
|
||||||
3. **Špatná pozorovatelnost** — těžké vidět, co agent dělá, jaký kontext používá, kolik to stojí.
|
|
||||||
4. **Chybějící rozšiřitelnost** — power user nemůže přidat vlastní nástroje bez forku.
|
|
||||||
5. **Approval fatigue** — buď plná autonomie, nebo approval pro každou akci; oboje je špatné UX.
|
|
||||||
6. **Špatná správa kontextu** — agenti jako OpenCode spoléhají na session compaction, ale ztrácí důležitý kontext.
|
|
||||||
|
|
||||||
Klíčový citát: *"So obviously they're doing things right, but not for me."*
|
|
||||||
|
|
||||||
### Designová filozofie Pi
|
|
||||||
|
|
||||||
- **Minimální jádro** — pouze 4 nástroje: read file, write file, edit file, bash. To stačí.
|
|
||||||
- **Malý system prompt** — frontier RL-trénované modely nepotřebují masivní system prompty.
|
|
||||||
- **[[tree-structured-sessions]]** — ne lineární chat history; sub-agenti se mohou větvit a číst soubory nezávisle při zachování kontextu/lineage.
|
|
||||||
- **Full cost tracking** — vestavěný, ne dodatečný.
|
|
||||||
- **Hot-reloadable TypeScript extensions** — uživatelé mohou definovat vlastní nástroje, UI, multi-agent setupy bez úpravy jádra.
|
|
||||||
- **Žádná skrytá kontextová injekce** — co vidíš, to model dostává.
|
|
||||||
|
|
||||||
### Komunitní rozšíření
|
|
||||||
|
|
||||||
- **pi-annotate** — vizuální feedback na živé weby
|
|
||||||
- **pi-messenger** — multi-agent chatroom s vlastním UI
|
|
||||||
- Vlastní UI, tool integrace — vše jako hot-reloadable TS moduly
|
|
||||||
|
|
||||||
### Výkon
|
|
||||||
|
|
||||||
Na TerminalBench dosáhlo Pi (s Claude Opus 4.5) blízko Terminus i před pokročilými optimalizacemi jako compaction.
|
|
||||||
|
|
||||||
### Klíčový insight
|
|
||||||
|
|
||||||
*"We are in the messing around and finding out stage, and nobody has any idea what the perfect coding agent should look like."* — zjednodušení může vést k efektivnímu výkonu bez zbytečné komplexity.
|
|
||||||
|
|
||||||
## Kde to zapadá
|
|
||||||
|
|
||||||
- [[person-mario-zechner]] — řečník a tvůrce Pi
|
|
||||||
- [[product-pi]] — coding agent
|
|
||||||
- [[coding-agent]] — obecný koncept
|
|
||||||
- [[tree-structured-sessions]] — Piův designový přístup ke správě kontextu
|
|
||||||
@@ -1,29 +0,0 @@
|
|||||||
---
|
|
||||||
type: source
|
|
||||||
title: "pi.dev — terminálová limitace"
|
|
||||||
authors: []
|
|
||||||
url: "https://pi.dev/"
|
|
||||||
raw: "raw/pi-dev-terminal-limitation.md"
|
|
||||||
ingested: 2026-06-16
|
|
||||||
created: 2026-06-16
|
|
||||||
updated: 2026-06-16
|
|
||||||
tags: [coding-agent, terminal, ide, ux, limitation]
|
|
||||||
entities: [product-pi]
|
|
||||||
concepts: [terminal-limitation, coding-agent]
|
|
||||||
slug: pi-dev-terminal-limitation
|
|
||||||
graph:
|
|
||||||
node_id: source:pi-dev-terminal-limitation
|
|
||||||
canonical: true
|
|
||||||
---
|
|
||||||
|
|
||||||
# pi.dev — terminálová limitace
|
|
||||||
|
|
||||||
Osobní poznámka: pi.dev je pěkný projekt, ale limitace na terminal je až moc přísná a omezující. Bez IDE to ztrácí všechny výhodné vlastnosti — podobně jako opencode.
|
|
||||||
|
|
||||||
Terminal-only přístup výrazně omezuje uživatelskou zkušenost a produktivitu oproti plnohodnotnému IDE integrovanému řešení.
|
|
||||||
|
|
||||||
## Kontext
|
|
||||||
|
|
||||||
- [[product-pi]] — Pi coding agent, jehož se tato limitace týká
|
|
||||||
- [[coding-agent]] — obecný koncept, kde je terminal-only vs. IDE debata relevantní
|
|
||||||
- [[terminal-limitation]] — koncept terminálové limitace coding agentů
|
|
||||||
392
cron/jobs.json
392
cron/jobs.json
@@ -2,7 +2,7 @@
|
|||||||
"version": 1,
|
"version": 1,
|
||||||
"jobs": [
|
"jobs": [
|
||||||
{
|
{
|
||||||
"id": "e81cda77",
|
"id": "c1e268c8",
|
||||||
"name": "nanobot-version-check",
|
"name": "nanobot-version-check",
|
||||||
"enabled": true,
|
"enabled": true,
|
||||||
"schedule": {
|
"schedule": {
|
||||||
@@ -14,7 +14,7 @@
|
|||||||
},
|
},
|
||||||
"payload": {
|
"payload": {
|
||||||
"kind": "agent_turn",
|
"kind": "agent_turn",
|
||||||
"message": "Check if a new nanobot-ai version is available on PyPI. Compare the latest PyPI version with the currently installed version (run `nanobot --version`). If a new version is found, notify the user via Telegram with: the current version, the new version, and the upgrade command: `uv tool upgrade nanobot-ai`. If already up to date, do nothing.",
|
"message": "Nanobot version check: porovnej nejnovější verzi nanobot-ai na PyPI s nainstalovanou (`nanobot --version`). Pokud je na PyPI novější verze, pošli notifikaci přes Telegram (message tool, chat_id 8826147089) s aktuální verzí, novou verzí a příkazem `uv tool upgrade nanobot-ai`. Pokud jsou verze stejné, nedělej nic a neodepisuj do web session — výstup jen pro poznámku, hlavní kanál je Telegram.",
|
||||||
"deliver": false,
|
"deliver": false,
|
||||||
"channel": null,
|
"channel": null,
|
||||||
"to": null,
|
"to": null,
|
||||||
@@ -24,10 +24,11 @@
|
|||||||
"originChatId": "52d0f338-7a06-42e7-aad2-86a4cabbfb9f",
|
"originChatId": "52d0f338-7a06-42e7-aad2-86a4cabbfb9f",
|
||||||
"originMetadata": {
|
"originMetadata": {
|
||||||
"remote": [
|
"remote": [
|
||||||
"10.20.30.8",
|
"192.168.40.104",
|
||||||
51076
|
40926
|
||||||
],
|
],
|
||||||
"webui": true,
|
"webui": true,
|
||||||
|
"webui_turn_id": "64d0dc00-72a6-45fb-9e16-c4b8d106a1f4",
|
||||||
"workspace_scope": {
|
"workspace_scope": {
|
||||||
"project_path": "/home/nanobot/.nanobot/workspace",
|
"project_path": "/home/nanobot/.nanobot/workspace",
|
||||||
"access_mode": "restricted"
|
"access_mode": "restricted"
|
||||||
@@ -36,135 +37,21 @@
|
|||||||
}
|
}
|
||||||
},
|
},
|
||||||
"state": {
|
"state": {
|
||||||
"nextRunAtMs": 1788415200000,
|
"nextRunAtMs": 1789106400000,
|
||||||
"lastRunAtMs": 1788328800001,
|
"lastRunAtMs": 1789020000002,
|
||||||
"lastStatus": "ok",
|
"lastStatus": "ok",
|
||||||
"lastError": null,
|
"lastError": null,
|
||||||
"runHistory": [
|
"runHistory": [
|
||||||
{
|
{
|
||||||
"runAtMs": 1786687200001,
|
"runAtMs": 1789020000002,
|
||||||
"status": "ok",
|
"status": "ok",
|
||||||
"durationMs": 6221,
|
"durationMs": 6281,
|
||||||
"error": null
|
|
||||||
},
|
|
||||||
{
|
|
||||||
"runAtMs": 1786773600001,
|
|
||||||
"status": "ok",
|
|
||||||
"durationMs": 9316,
|
|
||||||
"error": null
|
|
||||||
},
|
|
||||||
{
|
|
||||||
"runAtMs": 1786860000001,
|
|
||||||
"status": "ok",
|
|
||||||
"durationMs": 7565,
|
|
||||||
"error": null
|
|
||||||
},
|
|
||||||
{
|
|
||||||
"runAtMs": 1786946400001,
|
|
||||||
"status": "ok",
|
|
||||||
"durationMs": 8013,
|
|
||||||
"error": null
|
|
||||||
},
|
|
||||||
{
|
|
||||||
"runAtMs": 1787032800001,
|
|
||||||
"status": "ok",
|
|
||||||
"durationMs": 8457,
|
|
||||||
"error": null
|
|
||||||
},
|
|
||||||
{
|
|
||||||
"runAtMs": 1787119200002,
|
|
||||||
"status": "ok",
|
|
||||||
"durationMs": 8879,
|
|
||||||
"error": null
|
|
||||||
},
|
|
||||||
{
|
|
||||||
"runAtMs": 1787205600002,
|
|
||||||
"status": "ok",
|
|
||||||
"durationMs": 7420,
|
|
||||||
"error": null
|
|
||||||
},
|
|
||||||
{
|
|
||||||
"runAtMs": 1787292000002,
|
|
||||||
"status": "ok",
|
|
||||||
"durationMs": 7307,
|
|
||||||
"error": null
|
|
||||||
},
|
|
||||||
{
|
|
||||||
"runAtMs": 1787378400001,
|
|
||||||
"status": "ok",
|
|
||||||
"durationMs": 7402,
|
|
||||||
"error": null
|
|
||||||
},
|
|
||||||
{
|
|
||||||
"runAtMs": 1787464800001,
|
|
||||||
"status": "ok",
|
|
||||||
"durationMs": 8817,
|
|
||||||
"error": null
|
|
||||||
},
|
|
||||||
{
|
|
||||||
"runAtMs": 1787551200002,
|
|
||||||
"status": "ok",
|
|
||||||
"durationMs": 9626,
|
|
||||||
"error": null
|
|
||||||
},
|
|
||||||
{
|
|
||||||
"runAtMs": 1787637600002,
|
|
||||||
"status": "ok",
|
|
||||||
"durationMs": 12925,
|
|
||||||
"error": null
|
|
||||||
},
|
|
||||||
{
|
|
||||||
"runAtMs": 1787724000002,
|
|
||||||
"status": "ok",
|
|
||||||
"durationMs": 10899,
|
|
||||||
"error": null
|
|
||||||
},
|
|
||||||
{
|
|
||||||
"runAtMs": 1787810400002,
|
|
||||||
"status": "ok",
|
|
||||||
"durationMs": 5000,
|
|
||||||
"error": null
|
|
||||||
},
|
|
||||||
{
|
|
||||||
"runAtMs": 1787896800002,
|
|
||||||
"status": "ok",
|
|
||||||
"durationMs": 4464,
|
|
||||||
"error": null
|
|
||||||
},
|
|
||||||
{
|
|
||||||
"runAtMs": 1787983200001,
|
|
||||||
"status": "ok",
|
|
||||||
"durationMs": 5148,
|
|
||||||
"error": null
|
|
||||||
},
|
|
||||||
{
|
|
||||||
"runAtMs": 1788069600002,
|
|
||||||
"status": "ok",
|
|
||||||
"durationMs": 5748,
|
|
||||||
"error": null
|
|
||||||
},
|
|
||||||
{
|
|
||||||
"runAtMs": 1788156000001,
|
|
||||||
"status": "ok",
|
|
||||||
"durationMs": 95646,
|
|
||||||
"error": null
|
|
||||||
},
|
|
||||||
{
|
|
||||||
"runAtMs": 1788242400002,
|
|
||||||
"status": "ok",
|
|
||||||
"durationMs": 4653,
|
|
||||||
"error": null
|
|
||||||
},
|
|
||||||
{
|
|
||||||
"runAtMs": 1788328800001,
|
|
||||||
"status": "ok",
|
|
||||||
"durationMs": 3988,
|
|
||||||
"error": null
|
"error": null
|
||||||
}
|
}
|
||||||
]
|
]
|
||||||
},
|
},
|
||||||
"createdAtMs": 1782281393634,
|
"createdAtMs": 1788948314679,
|
||||||
"updatedAtMs": 1788328803989,
|
"updatedAtMs": 1789020006283,
|
||||||
"deleteAfterRun": false
|
"deleteAfterRun": false
|
||||||
},
|
},
|
||||||
{
|
{
|
||||||
@@ -191,135 +78,14 @@
|
|||||||
"originMetadata": {}
|
"originMetadata": {}
|
||||||
},
|
},
|
||||||
"state": {
|
"state": {
|
||||||
"nextRunAtMs": 1788360482419,
|
"nextRunAtMs": 1789040180439,
|
||||||
"lastRunAtMs": 1788353282400,
|
"lastRunAtMs": null,
|
||||||
"lastStatus": "ok",
|
"lastStatus": null,
|
||||||
"lastError": null,
|
"lastError": null,
|
||||||
"runHistory": [
|
"runHistory": []
|
||||||
{
|
|
||||||
"runAtMs": 1788216481686,
|
|
||||||
"status": "ok",
|
|
||||||
"durationMs": 10,
|
|
||||||
"error": null
|
|
||||||
},
|
},
|
||||||
{
|
"createdAtMs": 1789032980434,
|
||||||
"runAtMs": 1788223681698,
|
"updatedAtMs": 1789032980434,
|
||||||
"status": "ok",
|
|
||||||
"durationMs": 10,
|
|
||||||
"error": null
|
|
||||||
},
|
|
||||||
{
|
|
||||||
"runAtMs": 1788230881709,
|
|
||||||
"status": "ok",
|
|
||||||
"durationMs": 10,
|
|
||||||
"error": null
|
|
||||||
},
|
|
||||||
{
|
|
||||||
"runAtMs": 1788238081720,
|
|
||||||
"status": "ok",
|
|
||||||
"durationMs": 9,
|
|
||||||
"error": null
|
|
||||||
},
|
|
||||||
{
|
|
||||||
"runAtMs": 1788245281730,
|
|
||||||
"status": "ok",
|
|
||||||
"durationMs": 10,
|
|
||||||
"error": null
|
|
||||||
},
|
|
||||||
{
|
|
||||||
"runAtMs": 1788252481740,
|
|
||||||
"status": "ok",
|
|
||||||
"durationMs": 10,
|
|
||||||
"error": null
|
|
||||||
},
|
|
||||||
{
|
|
||||||
"runAtMs": 1788259681751,
|
|
||||||
"status": "ok",
|
|
||||||
"durationMs": 11,
|
|
||||||
"error": null
|
|
||||||
},
|
|
||||||
{
|
|
||||||
"runAtMs": 1788266881764,
|
|
||||||
"status": "ok",
|
|
||||||
"durationMs": 10,
|
|
||||||
"error": null
|
|
||||||
},
|
|
||||||
{
|
|
||||||
"runAtMs": 1788274081775,
|
|
||||||
"status": "ok",
|
|
||||||
"durationMs": 9,
|
|
||||||
"error": null
|
|
||||||
},
|
|
||||||
{
|
|
||||||
"runAtMs": 1788281281786,
|
|
||||||
"status": "ok",
|
|
||||||
"durationMs": 10,
|
|
||||||
"error": null
|
|
||||||
},
|
|
||||||
{
|
|
||||||
"runAtMs": 1788288481797,
|
|
||||||
"status": "ok",
|
|
||||||
"durationMs": 11,
|
|
||||||
"error": null
|
|
||||||
},
|
|
||||||
{
|
|
||||||
"runAtMs": 1788295681809,
|
|
||||||
"status": "ok",
|
|
||||||
"durationMs": 17,
|
|
||||||
"error": null
|
|
||||||
},
|
|
||||||
{
|
|
||||||
"runAtMs": 1788302881827,
|
|
||||||
"status": "ok",
|
|
||||||
"durationMs": 12,
|
|
||||||
"error": null
|
|
||||||
},
|
|
||||||
{
|
|
||||||
"runAtMs": 1788310081840,
|
|
||||||
"status": "ok",
|
|
||||||
"durationMs": 13,
|
|
||||||
"error": null
|
|
||||||
},
|
|
||||||
{
|
|
||||||
"runAtMs": 1788317282062,
|
|
||||||
"status": "ok",
|
|
||||||
"durationMs": 12,
|
|
||||||
"error": null
|
|
||||||
},
|
|
||||||
{
|
|
||||||
"runAtMs": 1788324482170,
|
|
||||||
"status": "ok",
|
|
||||||
"durationMs": 9,
|
|
||||||
"error": null
|
|
||||||
},
|
|
||||||
{
|
|
||||||
"runAtMs": 1788331682180,
|
|
||||||
"status": "ok",
|
|
||||||
"durationMs": 15,
|
|
||||||
"error": null
|
|
||||||
},
|
|
||||||
{
|
|
||||||
"runAtMs": 1788338882198,
|
|
||||||
"status": "ok",
|
|
||||||
"durationMs": 10,
|
|
||||||
"error": null
|
|
||||||
},
|
|
||||||
{
|
|
||||||
"runAtMs": 1788346082210,
|
|
||||||
"status": "ok",
|
|
||||||
"durationMs": 12,
|
|
||||||
"error": null
|
|
||||||
},
|
|
||||||
{
|
|
||||||
"runAtMs": 1788353282400,
|
|
||||||
"status": "ok",
|
|
||||||
"durationMs": 19,
|
|
||||||
"error": null
|
|
||||||
}
|
|
||||||
]
|
|
||||||
},
|
|
||||||
"createdAtMs": 1788151596627,
|
|
||||||
"updatedAtMs": 1788353282419,
|
|
||||||
"deleteAfterRun": false
|
"deleteAfterRun": false
|
||||||
},
|
},
|
||||||
{
|
{
|
||||||
@@ -346,135 +112,21 @@
|
|||||||
"originMetadata": {}
|
"originMetadata": {}
|
||||||
},
|
},
|
||||||
"state": {
|
"state": {
|
||||||
"nextRunAtMs": 1788356882542,
|
"nextRunAtMs": 1789036580440,
|
||||||
"lastRunAtMs": 1788355082541,
|
"lastRunAtMs": 1789034780440,
|
||||||
"lastStatus": "ok",
|
"lastStatus": "ok",
|
||||||
"lastError": null,
|
"lastError": null,
|
||||||
"runHistory": [
|
"runHistory": [
|
||||||
{
|
{
|
||||||
"runAtMs": 1788320882077,
|
"runAtMs": 1789034780440,
|
||||||
"status": "ok",
|
"status": "ok",
|
||||||
"durationMs": 0,
|
"durationMs": 0,
|
||||||
"error": null
|
"error": null
|
||||||
},
|
|
||||||
{
|
|
||||||
"runAtMs": 1788322682078,
|
|
||||||
"status": "ok",
|
|
||||||
"durationMs": 0,
|
|
||||||
"error": null
|
|
||||||
},
|
|
||||||
{
|
|
||||||
"runAtMs": 1788324482179,
|
|
||||||
"status": "ok",
|
|
||||||
"durationMs": 1,
|
|
||||||
"error": null
|
|
||||||
},
|
|
||||||
{
|
|
||||||
"runAtMs": 1788326282305,
|
|
||||||
"status": "ok",
|
|
||||||
"durationMs": 0,
|
|
||||||
"error": null
|
|
||||||
},
|
|
||||||
{
|
|
||||||
"runAtMs": 1788328082308,
|
|
||||||
"status": "ok",
|
|
||||||
"durationMs": 0,
|
|
||||||
"error": null
|
|
||||||
},
|
|
||||||
{
|
|
||||||
"runAtMs": 1788329882515,
|
|
||||||
"status": "ok",
|
|
||||||
"durationMs": 1,
|
|
||||||
"error": null
|
|
||||||
},
|
|
||||||
{
|
|
||||||
"runAtMs": 1788331682517,
|
|
||||||
"status": "ok",
|
|
||||||
"durationMs": 1,
|
|
||||||
"error": null
|
|
||||||
},
|
|
||||||
{
|
|
||||||
"runAtMs": 1788333482519,
|
|
||||||
"status": "ok",
|
|
||||||
"durationMs": 0,
|
|
||||||
"error": null
|
|
||||||
},
|
|
||||||
{
|
|
||||||
"runAtMs": 1788335282521,
|
|
||||||
"status": "ok",
|
|
||||||
"durationMs": 0,
|
|
||||||
"error": null
|
|
||||||
},
|
|
||||||
{
|
|
||||||
"runAtMs": 1788337082522,
|
|
||||||
"status": "ok",
|
|
||||||
"durationMs": 0,
|
|
||||||
"error": null
|
|
||||||
},
|
|
||||||
{
|
|
||||||
"runAtMs": 1788338882524,
|
|
||||||
"status": "ok",
|
|
||||||
"durationMs": 0,
|
|
||||||
"error": null
|
|
||||||
},
|
|
||||||
{
|
|
||||||
"runAtMs": 1788340682525,
|
|
||||||
"status": "ok",
|
|
||||||
"durationMs": 1,
|
|
||||||
"error": null
|
|
||||||
},
|
|
||||||
{
|
|
||||||
"runAtMs": 1788342482527,
|
|
||||||
"status": "ok",
|
|
||||||
"durationMs": 1,
|
|
||||||
"error": null
|
|
||||||
},
|
|
||||||
{
|
|
||||||
"runAtMs": 1788344282529,
|
|
||||||
"status": "ok",
|
|
||||||
"durationMs": 1,
|
|
||||||
"error": null
|
|
||||||
},
|
|
||||||
{
|
|
||||||
"runAtMs": 1788346082530,
|
|
||||||
"status": "ok",
|
|
||||||
"durationMs": 1,
|
|
||||||
"error": null
|
|
||||||
},
|
|
||||||
{
|
|
||||||
"runAtMs": 1788347882532,
|
|
||||||
"status": "ok",
|
|
||||||
"durationMs": 1,
|
|
||||||
"error": null
|
|
||||||
},
|
|
||||||
{
|
|
||||||
"runAtMs": 1788349682534,
|
|
||||||
"status": "ok",
|
|
||||||
"durationMs": 1,
|
|
||||||
"error": null
|
|
||||||
},
|
|
||||||
{
|
|
||||||
"runAtMs": 1788351482536,
|
|
||||||
"status": "ok",
|
|
||||||
"durationMs": 1,
|
|
||||||
"error": null
|
|
||||||
},
|
|
||||||
{
|
|
||||||
"runAtMs": 1788353282539,
|
|
||||||
"status": "ok",
|
|
||||||
"durationMs": 0,
|
|
||||||
"error": null
|
|
||||||
},
|
|
||||||
{
|
|
||||||
"runAtMs": 1788355082541,
|
|
||||||
"status": "ok",
|
|
||||||
"durationMs": 1,
|
|
||||||
"error": null
|
|
||||||
}
|
}
|
||||||
]
|
]
|
||||||
},
|
},
|
||||||
"createdAtMs": 1788151596637,
|
"createdAtMs": 1789032980437,
|
||||||
"updatedAtMs": 1788355082542,
|
"updatedAtMs": 1789034780440,
|
||||||
"deleteAfterRun": false
|
"deleteAfterRun": false
|
||||||
}
|
}
|
||||||
]
|
]
|
||||||
|
|||||||
@@ -3126,3 +3126,332 @@ a zabránit opakování driftu.
|
|||||||
- `notes.md`: `ssh nanobot@nanobot.hell 'cd ~/.nanobot/workspace && git revert f136f50'`
|
- `notes.md`: `ssh nanobot@nanobot.hell 'cd ~/.nanobot/workspace && git revert f136f50'`
|
||||||
- `AGENTS.md` / `projects/proxmox/state.md`: `git checkout -- AGENTS.md projects/proxmox/state.md` v serverovém workspace repu (pozor: vrátí i případné cizí necommitnuté změny).
|
- `AGENTS.md` / `projects/proxmox/state.md`: `git checkout -- AGENTS.md projects/proxmox/state.md` v serverovém workspace repu (pozor: vrátí i případné cizí necommitnuté změny).
|
||||||
- `skills/keep/`: `git revert a96a5c6` v tomto repu + rsync `skills/keep/` na server.
|
- `skills/keep/`: `git revert a96a5c6` v tomto repu + rsync `skills/keep/` na server.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 2026-09-04 07:20 — `keep` skill: description podle oficiálního formátu + přenositelnost
|
||||||
|
|
||||||
|
**Cíl:** Uživatel reklamoval `description` u `skills/keep/SKILL.md` — čeština uvnitř a špatně
|
||||||
|
napsaná proti tomu, jak má `description` vypadat. V průběhu doplnil druhou vadu: absolutní
|
||||||
|
cesta v těle skillu ho dělá nepřenositelným.
|
||||||
|
|
||||||
|
**Co jsem zkusil:**
|
||||||
|
|
||||||
|
1. Porovnal serverovou verzi s repem (`ssh nanobot@nanobot.hell cat …`) — **identické**,
|
||||||
|
žádný drift od Dreamu k dotažení.
|
||||||
|
2. Dostudoval dokumentaci. `nanobot.wiki` (0.3.0) frontmatter skillů **nedokumentuje** —
|
||||||
|
`/docs/0.3.0/` nemá stránku o skillech, `use-nanobot/concepts` je zmiňuje jen jako obsah
|
||||||
|
workspace. WebFetch na wiki vrací 403, curl s UA projde. Autoritou je proto
|
||||||
|
[Anthropic Skill authoring best practices](https://platform.claude.com/docs/en/agents-and-tools/agent-skills/best-practices)
|
||||||
|
(formát je identický) + naše zjištění ze zdrojáku v `knowledge.md`.
|
||||||
|
3. Změřil současnou verzi proti pravidlům: ✗ CZ triggery (EN-only + ověřeno, že
|
||||||
|
dvojjazyčné nic nepřidají), ✗ otevírá noun-phrase definicí místo slovesné fráze,
|
||||||
|
✗ obsahuje *jak* funguje (dedup/compaction/cesta), ✗ interní architektura
|
||||||
|
(MEMORY.md / Dream), ✗ duplicita, ~ chybí odlišení od `/project`.
|
||||||
|
Třetí osoba porušená nebyla.
|
||||||
|
4. Přepsal `description` do oficiálního tvaru `<co dělá>. Use when <triggery>.` (385 znaků,
|
||||||
|
limit 1 024), opravil poslední češtinu v těle (`"udělej compaction"` → `the user asks
|
||||||
|
for compaction`) a absolutní cestu → `workspace/keep.md`.
|
||||||
|
5. Nasadil rsyncem, ověřil vlastníka.
|
||||||
|
|
||||||
|
**Co fungovalo a proč:**
|
||||||
|
|
||||||
|
- Grepy čisté: žádná čeština, žádné `/home/nanobot` v `skills/keep/SKILL.md`.
|
||||||
|
- **Funkční test routingu:** `nanobot agent -m "zapamatuj si, že testovací server pro
|
||||||
|
staging je stag-07"` → odpověď `Kept: …`, záznam v `keep.md`. Skill se tedy trefil
|
||||||
|
i bez českých triggerů — potvrzuje ověřený předpoklad z `/plan` (2026-05-31), že
|
||||||
|
dvojjazyčné triggery jsou zbytečný token cost.
|
||||||
|
- Testovací záznam z `keep.md` smazán (záloha `/tmp/keep.md.bak` na serveru).
|
||||||
|
- `nanobot` binárka není v PATH non-login shellu — volat `~/.local/bin/nanobot`
|
||||||
|
(stejná gotcha jako u `uv`).
|
||||||
|
|
||||||
|
**Co zbývá:**
|
||||||
|
|
||||||
|
- `skills/plan/SKILL.md:45` má stejnou absolutní cestu, přitom řádky 69 a 75 v témže
|
||||||
|
souboru už jsou relativní. `bookmark` má absolutní cesty ve spouštěcích příkazech —
|
||||||
|
tam opodstatněné (PATH gotcha u `uv`). Neřešeno, mimo scope zadání.
|
||||||
|
- Ostatní skilly nebyly proti oficiálnímu tvaru `description` proměřeny.
|
||||||
|
|
||||||
|
**Jak vrátit zpět:** `git revert 18ab382` + rsync `skills/keep/` na server.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 2026-09-09 20:20 — Final wiki: zhodnocení draftu hybrid RAG a přepis do `final-wiki-hybrid-rag.md`
|
||||||
|
|
||||||
|
**Cíl:** Zhodnotit serverový draft `plans/notes-search-hybrid-rag.md` (hybrid RAG index nad
|
||||||
|
uživatelovými git poznámkami), rozhodnout, jestli by to tak šlo, a přepsat do finálního plánu.
|
||||||
|
|
||||||
|
**Co jsem zkusil:**
|
||||||
|
|
||||||
|
1. Přečetl draft a proměřil prostředí místo věření odhadům v něm:
|
||||||
|
- Ollama na `nvidia.hell` dosažitelná; `qwen3-embedding` tam zpočátku nebyl (uživatel dodal).
|
||||||
|
- Cold load modelu 1,81 s vs. warm 0,043 s; throughput 8,9 chunk/s po jednom vs.
|
||||||
|
**108 chunk/s v batchi 32**.
|
||||||
|
- Cosine nad 10⁵×1024 v BLOB+numpy: **594 ms** = read 382 + pack 190 + **dot jen 22**.
|
||||||
|
Přes `.npy` + `mmap_mode='r'` 35 ms.
|
||||||
|
- FTS5: `remove_diacritics 1`+ foldí diakritiku, `záloh*` pokryje českou flexi.
|
||||||
|
- Korpus workspace: 234 md mimo `tmp/` → ~800 chunků (draft počítal 10⁴–10⁵).
|
||||||
|
2. Ověřil `sqlite-vec` smoke testem celého navrženého schématu (`chunks` + `files` +
|
||||||
|
`chunks_fts` + triggery + `vec_chunks`) — včetně toho, na čem D3 stálo.
|
||||||
|
3. Ověřil model na reálném obsahu: cross-jazyk retrieval test + vliv instruct prefixu.
|
||||||
|
4. Přepsal plán do `final-wiki-hybrid-rag.md`, nasadil na server i do repa, starý draft
|
||||||
|
označil `> Superseded by`.
|
||||||
|
|
||||||
|
**Co fungovalo a proč:**
|
||||||
|
|
||||||
|
- **Díra v návrhu draftu**: RRF slučoval file-level BM25 s chunk-level cosine. Rank-merge dvou
|
||||||
|
různých jednotek nemá definovaný význam. Fix: `chunks` je jediná retrieval jednotka, FTS5 je
|
||||||
|
external-content nad ní, `vec0` má `rowid = chunks.id`, RRF slučuje `chunks.id`. Ověřeno.
|
||||||
|
- **`vec0` obstálo ve všech kritických testech**: `DELETE`/re-`INSERT`/`UPDATE` po `rowid`,
|
||||||
|
transakční `ROLLBACK`, `distance_metric=cosine`, `vec0` v témže souboru jako běžné tabulky.
|
||||||
|
KNN k=20: **19 ms @ 10k, 212 ms @ 100k**. Kdyby `DELETE` nešel, D3 by padlo a vracel by se BLOB.
|
||||||
|
- **Zdůvodnění D3 v draftu bylo špatné** — stavěl `sqlite-vec` jako „indexovaný KNN" (ANN).
|
||||||
|
Není; dokumentovaná cesta je průchod (lineární 19 → 212 ms to potvrzuje). Skutečná výhoda je,
|
||||||
|
že skenuje v C a odřízne těch 96 % Python režie.
|
||||||
|
- **Ollama Cloud embeddingy neexistují** — 18 cloud modelů, žádný s capability `embedding`;
|
||||||
|
`/api/embed` na cloud modelu vrací `unauthorized`, zatímco `/api/generate` na tomtéž projde
|
||||||
|
(takže to není o autorizaci). Katalog na ollama.com potvrzuje: všech 12 embedding modelů
|
||||||
|
je jen ke stažení.
|
||||||
|
- **D1 potvrzeno měřením**, ne argumentem: dotaz „jak snížit elektroodpad ze stavebnic" →
|
||||||
|
BM25 **1/5**, embeddingy **3/5**; lexikální dotaz naopak BM25 4/5. Obě poloviny si vydělávají.
|
||||||
|
- **`exec` timeout 60 s** vs. plný index ~77 s → indexace nesmí běžet v tahu agenta. V draftu to
|
||||||
|
nebylo; uživatel zvolil offline cron s lockem, čímž `fetch-on-query` z draftu vypadl.
|
||||||
|
- **Whitelist místo blacklistu** (na dotaz uživatele): ve workspace je 234 md mimo `tmp/` a
|
||||||
|
použitelných ~99 — zbytek `.venv/` 27, `cml/` 39, `skills/` 32, `backup/` 20, `tasks/` 16;
|
||||||
|
`tmp/` drží dalších 135. Blacklist by musel trvale pokrývat právě adresář určený k balastu.
|
||||||
|
Rozhodující je ale směr selhání: u blacklistu nový adresář *tiše vstoupí* do indexu.
|
||||||
|
- DDL vytažené přímo z hotového dokumentu spuštěno proti `:memory:` — prošlo, a FTS5 i `vec0`
|
||||||
|
vrátily shodné `rowid`.
|
||||||
|
|
||||||
|
**Co jsem měl špatně (korekce vlastních tvrzení):**
|
||||||
|
|
||||||
|
- Tvrdil jsem, že vynechání **instruct prefixu** stojí víc než přechod na menší model. Test na
|
||||||
|
178 chunkách to vyvrátil: recall@5 s prefixem i bez něj **identický** (5/5 vs 5/5, 3/5 vs 3/5),
|
||||||
|
similarita s prefixem dokonce nižší. Prefix zlepšil jen top-1 nejtěžšího dotazu. V plánu
|
||||||
|
zůstává (je zdarma), ale zdůvodnění je teď „bitová identita indexace a dotazu", ne kritičnost.
|
||||||
|
- Cold load jsem extrapoloval na 4–6 s; naměřeno **1,81 s**. `keep_alive: -1` je tedy výhodný,
|
||||||
|
ne nutný.
|
||||||
|
- Dvě chyby v mých vlastních test skriptech, obě mě zdržely: chybějící
|
||||||
|
`isolation_level=None` (→ „cannot start a transaction within a transaction") a bufferovaný
|
||||||
|
stdout Pythonu, kvůli kterému 9 minut běžící skript neukázal, že visí na síťovém čtení.
|
||||||
|
|
||||||
|
**Nálezy pro implementaci:**
|
||||||
|
|
||||||
|
- **`ON DELETE CASCADE` z `files` uklidí `chunks` i FTS5 (trigger se na kaskádě spustí), ale
|
||||||
|
řádek ve `vec_chunks` osiří** — `vec0` není cílem foreign key. Indexer musí mazat `vec_chunks`
|
||||||
|
explicitně. Je to v plánu jako regresní test; jinak by index při mazání souborů tiše hnil.
|
||||||
|
- `keep_alive` musí být **číslo** `-1` (nebo `"24h"`); string `"-1"` vrací HTTP 400.
|
||||||
|
|
||||||
|
**Co zbývá:**
|
||||||
|
|
||||||
|
- **Seznam rep** (URL + kam mirrorovat) — jediná věc, kterou musí doplnit autor; bez toho
|
||||||
|
nelze naplnit `config/notes-search.yaml`.
|
||||||
|
- Zopakovat cross-jazyk benchmark na reálných repech (proběhl na workspace obsahu).
|
||||||
|
- Návrh 6 vět do `decisions.md` je v plánu — **čeká na souhlas autora**, nezapsáno.
|
||||||
|
- Mimo scope a jen hlášeno: `llm-wiki` řeší autor sám (zazálohuje a smaže);
|
||||||
|
`wiki-compile` existuje jen na serveru a v žádném commitu (drift, pravděpodobně Dream).
|
||||||
|
|
||||||
|
**Jak vrátit zpět:** `git revert 36259d7`; na serveru smazat
|
||||||
|
`plans/final-wiki-hybrid-rag.md` a odstranit vložený `> **Superseded by` řádek (druhý řádek)
|
||||||
|
z `plans/notes-search-hybrid-rag.md`.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 2026-09-09 — Oprava plánu final-wiki-hybrid-rag po plan-review
|
||||||
|
|
||||||
|
**Cíl:** Projet `final-wiki-hybrid-rag.md` skrz `/lab:plan-review` a opravit, co review najde,
|
||||||
|
aby byl plán implementovatelný bez hádání.
|
||||||
|
|
||||||
|
**Co jsem zkusil:**
|
||||||
|
|
||||||
|
Review vrátila verdikt „vrátit k přepracování": 5 blokujících nálezů, 7 k doplnění, 3 poznámky.
|
||||||
|
Všechny kotvy, které plán uváděl jako existující, jsem ověřil a všechny sedí
|
||||||
|
(`remind/scripts/db.py` symboly, `note_capture.py:25`, `_ascii_fold`, `.compile.lock`,
|
||||||
|
`llm-wiki/scripts/wiki_search.py`, `develop/history.md` + `knowledge.md` na serveru, crontab
|
||||||
|
vzor, `db/` v `.gitignore`). Plán nebyl superseded — skill neexistoval.
|
||||||
|
|
||||||
|
Pět blokujících nálezů:
|
||||||
|
|
||||||
|
1. Vrstva 2 (TOC) byla vyhlášená, ale sync ji negeneroval, schéma pro ni nemělo tabulku
|
||||||
|
a Verifikace ji netestovala.
|
||||||
|
2. `git clone --mirror` je **bare** repo → vrstva 1 (`rg`) nemá working tree, co grepovat,
|
||||||
|
a chunker neměl řečeno, jak se z mirroru dostane obsah souboru.
|
||||||
|
3. Chunky s `embedded_at IS NULL` se nikdy nedobraly: krok 3 syncu končí „nic se nezměnilo
|
||||||
|
→ exit 0", jenže po degradovaném běhu se soubory nezměnily. Verifikace bod 4 přitom
|
||||||
|
tvrdila opak a partial index `idx_chunks_pending` nikdo nečetl.
|
||||||
|
4. Skupinové klíče v YAML (`sources.git.*`, `sources.nanobot.*`) nešly namapovat na
|
||||||
|
`CHECK(kind IN ('git','workspace'))` — `nanobot` by CHECK odmítl.
|
||||||
|
5. RRF neměl jediný parametr: chybělo `k` pro KNN, počet kandidátů z BM25, RRF konstanta,
|
||||||
|
výstupní top-k i váhy.
|
||||||
|
|
||||||
|
**Co fungovalo a proč:**
|
||||||
|
|
||||||
|
Před opravou jsem si nechal od autora rozhodnout adresářovou strukturu a jméno skillu, protože
|
||||||
|
na tom visí deploy cesta, cron řádek i testy — a pak dvě věci **naměřil místo hádání**, když
|
||||||
|
padla otázka „jak se to bude chovat, když do `include` dám i `*.py` a `*.cs`":
|
||||||
|
|
||||||
|
- markdown-it-py nad Python souborem: komentář `# TODO: fix this` se parsuje jako **ATX
|
||||||
|
heading h1**. Chunker by řezal soubor v komentářích a stavěl breadcrumb
|
||||||
|
`foo.py > TODO: fix this` — ne horší kvalita, ale nesmysl.
|
||||||
|
- FTS5 `unicode61`: `send`/`async` uvnitř `SendAsync` → **miss** (camelCase se nedělí);
|
||||||
|
`migrate` i `_migrate` → oba hit, nerozlišitelné. `tokenchars '_'` problém jen převrátí
|
||||||
|
(`get_db` hit, `get` miss). Jediný použitelný tokenizer je `trigram` — a ten je vlastnost
|
||||||
|
**tabulky**, ne řádku, takže smíšený korpus = druhá FTS tabulka a dvojí dotaz.
|
||||||
|
|
||||||
|
Tohle rozhodlo D15 (index zůstává md-only) měřením, ne názorem. Náhradou je, že grep vrstva
|
||||||
|
jede přes celý klon včetně kódu — soubory už na disku leží, takže to stojí nula.
|
||||||
|
|
||||||
|
Dvě zjednodušení nálezy zrušila místo záplatování: **plochá konfigurace** s explicitním `kind`
|
||||||
|
(padá nález 4) a **sloučení `workspace` + `develop`** do jednoho zdroje (stejný kořen, stejný
|
||||||
|
kind, nebyl důvod je mít dva).
|
||||||
|
|
||||||
|
Autor dodal URL obou rep, ověřeno z `nanobot.hell`: obě dostupná klíčem uživatele `nanobot`,
|
||||||
|
obě `HEAD → refs/heads/master`. Otevřená otázka 1 tím padla.
|
||||||
|
|
||||||
|
**Nálezy pro implementaci:**
|
||||||
|
|
||||||
|
- Workspace na serveru **je git repo** a `wiki/` v jeho `.gitignore` chybí. Nasazení musí
|
||||||
|
přidat `wiki/*` + `!wiki/config.yaml`, jinak se do gitu commitne index (komprimovaná kopie
|
||||||
|
osobního obsahu) i klony. Původní plán se opíral o to, že `db/` je gitignorované —
|
||||||
|
přesunem pod `wiki/` ta záruka zmizela.
|
||||||
|
- `note_compile.py:85-108` má stale-lock reclaim (mrtvý PID nebo stáří > 30 min). Plán měl
|
||||||
|
jen holý `O_EXCL`, což by po pádu procesu navždy umlčelo minutový cron bez jediného řádku
|
||||||
|
v logu. Převzato doslova.
|
||||||
|
- Aritmetika v plánu byla špatně: 10⁴ chunků / 108 chunk/s je ~93 s, ne 77 s. Závěr
|
||||||
|
(> 60 s `exec` timeout) drží.
|
||||||
|
|
||||||
|
**Co zbývá:**
|
||||||
|
|
||||||
|
- **Pustit `/lab:plan-review` znovu** — skill se nedá vyvolat přes `Skill` tool
|
||||||
|
(`disable-model-invocation`), musí ho spustit autor. Cíl: 0 blokujících nálezů.
|
||||||
|
- Návrh 4 vět do `decisions.md` — čeká na souhlas autora, nezapsáno: kořen runtime dat
|
||||||
|
`workspace/wiki/`, non-bare klon místo `--mirror`, plochá konfigurace s explicitním `kind`,
|
||||||
|
index výhradně nad markdownem.
|
||||||
|
- Benchmark modelu na uživatelských datech zůstává jako výstup etapy 6.
|
||||||
|
|
||||||
|
**Jak vrátit zpět:** `git revert 43688c5`. Na server se nesahalo.
|
||||||
|
|
||||||
|
## 2026-09-09 22:10 — implementace a nasazení skillu `wiki` (hybrid RAG index)
|
||||||
|
|
||||||
|
**Cíl:** Vykonat plán [plans/final-wiki-hybrid-rag.md](plans/final-wiki-hybrid-rag.md),
|
||||||
|
etapy 1–7 včetně nasazení na server.
|
||||||
|
|
||||||
|
**Co jsem zkusil / co fungovalo a proč:**
|
||||||
|
|
||||||
|
- **Etapy 1–6 implementované a otestované lokálně** (`skills/wiki/`, commit `aecf597`):
|
||||||
|
87 testů, `ruff` i `ty` čisté. Pokrývají body 1–6 a 8 z Verifikace v plánu.
|
||||||
|
- **Etapa 7 nasazena** na `nanobot.hell`: rsync skillu, `wiki/config.yaml`,
|
||||||
|
`.gitignore` (`wiki/*` + `!wiki/config.yaml`), první plný index a minutový cron.
|
||||||
|
Plný index: **152 souborů, 601 chunků, 601 vektorů, 0 pending za 45 s**
|
||||||
|
(index 55/123, travel 5/9, workspace 92/469).
|
||||||
|
- **Dotaz end-to-end 53–62 ms** (cíl plánu byl pod 1 s). Rozpad odpovídá plánu:
|
||||||
|
embed dotazu ~55 ms, KNN 2–5 ms, BM25 < 1 ms, RRF zanedbatelné.
|
||||||
|
- **Odchylky od plánu (mechanické, ne věcné):** moduly nesou prefix `wiki_`, protože
|
||||||
|
`db.py`/`store.py` kolidují s `remind` a `wiki_search.py` s `llm-wiki` ve `ty.toml`
|
||||||
|
`extra-paths` (ploché jmenné prostory, vyhrává první cesta). Přidány dva moduly nad
|
||||||
|
pětici z plánu — `wiki_config.py` (layout + rozsah) a `wiki_embed.py` (Ollama klient
|
||||||
|
+ `meta` guard), protože obě vstupní body je potřebují a guard **musí** být identický
|
||||||
|
na obou stranách. `numpy` z dependency setu vypuštěno — `sqlite_vec.serialize_float32`
|
||||||
|
stačí, normalizace je jeden `math.sqrt`.
|
||||||
|
- **Reálné selhání sítě ověřeno neplánovaně:** z mého stroje nemá SSH klíč právo na
|
||||||
|
`travel-notes.git`, takže první lokální běh doslova předvedl bod 8 — `WARN`, zdroj
|
||||||
|
přeskočen, `index` dojel. Na serveru (klíč uživatele `nanobot`) projde.
|
||||||
|
|
||||||
|
**Nalezené a opravené vady (moje, ne plánu):**
|
||||||
|
|
||||||
|
1. **Sekce obsahující jen svůj nadpis vytvářela prázdný chunk.** U běžného tvaru
|
||||||
|
`# Titul` → `## Sekce` by šum lezl skoro do každého dokumentu. Fix: taková sekce se
|
||||||
|
zahodí, nadpis se do indexu dostane přes breadcrumb potomků a `files.headings`.
|
||||||
|
2. **Víceřádkový git stderr v logu.** Trvale nedostupný zdroj sype WARN každou minutu;
|
||||||
|
šestiřádkový stderr = tisíce řádků denně. Fix: `_one_line()` + strop 300 znaků.
|
||||||
|
Plán mluví o „řádku WARN" — teď to řádek skutečně je.
|
||||||
|
3. **Coverage report hlásil i `wiki/`** (klony v `wiki/remote/` obsahují md).
|
||||||
|
Strukturálně nikdy nekandidát, ne rozhodnutí k revizi → z reportu vyřazen.
|
||||||
|
|
||||||
|
**Otevřená otázka č. 1 plánu (benchmark modelu) — uzavřena.** 12 parafrázových dotazů
|
||||||
|
s jednoznačným cílovým souborem nad reálnými 123 chunky repa `index`:
|
||||||
|
|
||||||
|
| Varianta | recall@5 | recall@10 | MRR |
|
||||||
|
|---|---|---|---|
|
||||||
|
| BM25 sám | 5/12 | 5/12 | 0,257 |
|
||||||
|
| qwen3-embedding:0.6b + instruct prefix | 4/12 | **8/12** | 0,261 |
|
||||||
|
| qwen3-embedding:0.6b bez prefixu | 2/12 | 3/12 | 0,204 |
|
||||||
|
| mxbai-embed-large | 6/12 | 6/12 | 0,261 |
|
||||||
|
| nomic-embed-text | 3/12 | 4/12 | 0,170 |
|
||||||
|
| **RRF hybrid** | 6/12 | 6/12 | **0,367** |
|
||||||
|
|
||||||
|
D1 i D2 platí: hybrid má lepší MRR než obě poloviny samostatně, `nomic` je nejhorší
|
||||||
|
(plán ho zamítal správně). V top-10 najde cíl **jen vektory u 3 dotazů, jen BM25 u 0** —
|
||||||
|
vektorová polovina na tomhle korpusu lexikální pokrytím dominuje, BM25 ale dává lepší
|
||||||
|
top-1 rank když trefí. Čtyři dotazy nenašel nikdo, protože cílové soubory jsou stuby
|
||||||
|
(50–483 B, většinou shell příkazy) — chyba mých labelů, ne retrievalu; na odpověďorelných
|
||||||
|
dotazech je hybrid 6/8.
|
||||||
|
|
||||||
|
**Co nesouhlasí s plánem:**
|
||||||
|
|
||||||
|
- **Instruct prefix NENÍ „nekritická vlastnost"** (plán, Query kontrakt). Naměřeno
|
||||||
|
recall@10 **8/12 s prefixem vs. 3/12 bez** na uživatelských datech. Plán měřil
|
||||||
|
5/5 vs 5/5 na 178 chuncích jiného korpusu. Prefix je load-bearing; je zapsán
|
||||||
|
v `meta.query_prefix` a jeho změna vyžaduje `--full`.
|
||||||
|
- **D6 („prefix wildcard pokryje českou flexi") platí jen na půl.** Prokázáno na
|
||||||
|
serveru: `cestu*` → 0 chunků v `travel/packaging-list.md`, `cesty*` → 3, `cest*` → 4.
|
||||||
|
Wildcard je **prefixový**, takže pomůže jen když je dotazové slovo prefixem tvaru
|
||||||
|
v dokumentu; „cestu" prefixem „cesty" není. Plán to dokládal dotazem `záloh*`, což
|
||||||
|
je už samotný **kmen**, ne skloněné slovo. Důsledek: na dotaz „co si vzít na cestu
|
||||||
|
do zahraničí" se doslovný seznam věcí na cestu nedostal ani do top 10.
|
||||||
|
|
||||||
|
**Co zbývá:** viz [todo.md](todo.md) — rozhodnout o dotazové vrstvě u české flexe
|
||||||
|
a návrh 4 vět do `decisions.md` (čeká na souhlas autora, nezapsáno).
|
||||||
|
|
||||||
|
**Jak vrátit zpět:** na serveru `crontab ~/.nanobot/backup/crontab.bak-20260909-220814`,
|
||||||
|
pak `rm -rf ~/.nanobot/workspace/wiki ~/.nanobot/workspace/skills/wiki` a vrátit řádky
|
||||||
|
`wiki/*` + `!wiki/config.yaml` z `.gitignore`. Lokálně `git revert aecf597`.
|
||||||
|
|
||||||
|
## 2026-09-09 23:25 — README ke skillu `wiki` + oprava prázdných titulků
|
||||||
|
|
||||||
|
**Cíl:** Doplnit ke skillu `wiki` lidsky psaný `README.md` (mentální model, proč je co tak,
|
||||||
|
operátorský runbook), protože vysvětlení bylo rozeseté po `history.md`, result dokumentu
|
||||||
|
a konverzaci. `SKILL.md` je záměrně EN instrukce pro agenta — tohle do něj nepatří.
|
||||||
|
|
||||||
|
**Co jsem zkusil / co fungovalo a proč:**
|
||||||
|
|
||||||
|
- **`skills/wiki/README.md`** (214 řádků), česky, druhá osoba, ve stylu existujících
|
||||||
|
`skills/reflect/README.md` a `skills/compact-memory/README.md` — což jsou jediné dva
|
||||||
|
README v repu, takže konvence se dala odečíst přímo z nich. Sekce: tři vrstvy, offline
|
||||||
|
indexace a proč, co se v tiku děje, proč hybrid, česká flexe, index jako nápověda,
|
||||||
|
rozsah indexace, kde co leží, ruční spuštění, hlášky, ověření.
|
||||||
|
- **Ze `SKILL.md` na README záměrně nevede odkaz.** Ani `reflect`, ani `compact-memory` ho
|
||||||
|
nemají; README je pro člověka a nesmí stát agenta kontext. (`detach/architecture.md`
|
||||||
|
odkaz má, ale to je anglický dokument jiného žánru.)
|
||||||
|
- **Každé tvrzení v README ověřeno proti kódu**, ne napsáno z hlavy: hlášky doslova
|
||||||
|
grepem z `wiki_search.py`/`wiki_embed.py`, cesty vyhodnocením konstant z `wiki_config.py`,
|
||||||
|
flagy z `parse_args()`. Příkazy ze sekce „Ruční spuštění" spuštěny naostro na serveru
|
||||||
|
včetně `--source` s neexistujícím id (exit 1, `WARN unknown source`).
|
||||||
|
|
||||||
|
**Nalezená a opravená vada — `files.title` byl NULL u všech 152 souborů.**
|
||||||
|
|
||||||
|
Vyšlo to při ověřování příkladu z README: `toc` tiskl prázdný sloupec s titulkem pro
|
||||||
|
**celý korpus**, přitom plán ho ukazuje jako podstatnou část výstupu. Příčina: `title` se
|
||||||
|
bral výhradně z frontmatteru `title:`, ale uživatelovy poznámky nesou titulek jako `# H1`.
|
||||||
|
|
||||||
|
Fix: fallback na první nadpis. Po `--full` je pokrytí **117/152** (`index` 55/55,
|
||||||
|
`travel` 5/5, `workspace` 57/92); zbylých 35 jsou raw zachyty `/note` v `notes/done/`
|
||||||
|
bez jediného H1–H3 nadpisu — tam není z čeho titulek vzít a slug v názvu souboru ho nese.
|
||||||
|
|
||||||
|
Podstatné rozhodnutí u toho fixu: fallback plní **jen katalog, ne breadcrumb root**.
|
||||||
|
Kdyby šel do rootu, přepsal by se text všech chunků (breadcrumb je součástí embedovaného
|
||||||
|
textu) a musel by se bumpnout `chunker_version` s vynuceným `--full` u každého uživatele.
|
||||||
|
Takhle je to změna jednoho sloupce v `files`.
|
||||||
|
|
||||||
|
**Vedlejší nález:** padl existující test `test_malformed_frontmatter_stays_body`. Nebyla to
|
||||||
|
regrese — u rozbitého frontmatteru (`---\ntitle: [unclosed\n---`) udělá markdown-it z toho
|
||||||
|
řádku **setext nadpis**, takže ho fallback vezme jako titulek. Frontmatter se opravdu
|
||||||
|
nespotřeboval (tagy zůstaly prázdné, text zůstal v těle), jen test tvrdil něco jiného,
|
||||||
|
než byl jeho záměr. Přepsán na to, co má tvrdit.
|
||||||
|
|
||||||
|
**Co zbývá:** nic z tohoto zásahu. Otevřené položky skillu `wiki` v `todo.md` (česká flexe,
|
||||||
|
návrh do `decisions.md`) se nemění.
|
||||||
|
|
||||||
|
**Jak vrátit zpět:** `git revert 3adc2ef`, pak na serveru rsync skillu a
|
||||||
|
`wiki_sync.py --full` (titulky se vrátí na NULL). Index se tím nerozbije — `title` je
|
||||||
|
jen sloupec v katalogu, retrieval na něm nestojí.
|
||||||
|
|||||||
@@ -347,6 +347,16 @@ Pole `description` ve frontmatteru non-always skillu je **routing signál**, ne
|
|||||||
|
|
||||||
Zdroj: `nanobot/agent/skills.py:111-159` (`build_skills_summary`, `_get_skill_description`), `skills.py:94-109` (`load_skills_for_context`, always skilly), `nanobot/agent/context.py:87-95`.
|
Zdroj: `nanobot/agent/skills.py:111-159` (`build_skills_summary`, `_get_skill_description`), `skills.py:94-109` (`load_skills_for_context`, always skilly), `nanobot/agent/context.py:87-95`.
|
||||||
|
|
||||||
|
**Oficiální tvar `description`** ([Anthropic — Skill authoring best practices](https://platform.claude.com/docs/en/agents-and-tools/agent-skills/best-practices), platí i pro nanobot, formát je identický). Nanobot wiki frontmatter skillů nedokumentuje — autoritou je tenhle dokument plus zjištění ze zdrojáku výše.
|
||||||
|
|
||||||
|
- Tvar: `<co skill dělá, slovesná fráze>. Use when <triggery/kontexty>.` Např. `Extract text and tables from PDF files, fill forms, merge documents. Use when working with PDF files or when the user mentions PDFs, forms, or document extraction.`
|
||||||
|
- **Vždy třetí osoba** — `Processes Excel files…`, nikdy `I can help you…` ani `You can use this to…`; nekonzistentní osoba zhoršuje discovery.
|
||||||
|
- Musí obsahovat **co dělá i kdy použít**, klíčový use case první, konkrétní klíčové termíny. Vágní (`Helps with documents`) je anti-pattern.
|
||||||
|
- Limit **1 024 znaků** (Agent Skills spec); Claude Code listing ořezává na 1 536.
|
||||||
|
- Nanobotí dodatek k tomu: „co dělá" = **schopnost**, ne postup. `Adds, deduplicates, and compacts entries in keep.md` je *jak* → do těla.
|
||||||
|
|
||||||
|
Aplikováno na `keep` (history 2026-09-04).
|
||||||
|
|
||||||
## Dream procesor — automatické self-improvement
|
## Dream procesor — automatické self-improvement
|
||||||
|
|
||||||
Nanobot má vestavěný Dream procesor (`agent/memory.py:Dream`) který běží každé 2 hodiny. Jde o **dvou-fázový LLM pipeline** nad `history.jsonl`:
|
Nanobot má vestavěný Dream procesor (`agent/memory.py:Dream`) který běží každé 2 hodiny. Jde o **dvou-fázový LLM pipeline** nad `history.jsonl`:
|
||||||
@@ -1163,3 +1173,86 @@ Dream). Zbytek serverového workspace repa běžně stojí s necommitnutými zm
|
|||||||
(`keep.md`, `AGENTS.md`, `cron/jobs.json`, `reflect/*`) — to je normální stav, ne rozbité repo.
|
(`keep.md`, `AGENTS.md`, `cron/jobs.json`, `reflect/*`) — to je normální stav, ne rozbité repo.
|
||||||
Při ruční editaci `notes.md` proto commitni s prefixem `note:` a stage jen `notes/`.
|
Při ruční editaci `notes.md` proto commitni s prefixem `note:` a stage jen `notes/`.
|
||||||
Zdroj: history 2026-09-04.
|
Zdroj: history 2026-09-04.
|
||||||
|
|
||||||
|
## Ollama Cloud embeddingy neexistují — cloud tier je completion-only (2026-09-09)
|
||||||
|
|
||||||
|
Žádný z 18 cloud modelů nemá capability `embedding`; `/api/embed` na cloud modelu vrací
|
||||||
|
`unauthorized`, **zatímco `/api/generate` na tomtéž modelu projde** — takže to není problém
|
||||||
|
autorizace, cloud tier ten endpoint prostě neobsluhuje a hláška je zavádějící. Katalog na
|
||||||
|
ollama.com to potvrzuje: filtr `c=embedding` vrací 12 modelů, všechny jen ke stažení, žádný
|
||||||
|
s cloud tagem. Embeddingy je tedy nutné hostovat lokálně. Zdroj: history 2026-09-09.
|
||||||
|
|
||||||
|
## Cena brute-force cosine v SQLite je z 96 % Python režie, ne matmul (2026-09-09)
|
||||||
|
|
||||||
|
Nad 10⁵×1024 float32: **594 ms = read 382 + pack 190 + dot 22**. Samotný numpy matmul je
|
||||||
|
zanedbatelný; platí se za extrakci BLOBů a jejich složení do matice. Proto `sqlite-vec`
|
||||||
|
(`vec0`) dá **19 ms @ 10k a 212 ms @ 100k** — skenuje v C. `.npy` + `mmap_mode='r'` je
|
||||||
|
ještě rychlejší (35 ms), ale za cenu druhého souboru mimo DB. Pozor: `vec0` **není ANN
|
||||||
|
index** — lineární škálování 19 → 212 ms to potvrzuje, výhoda je konstanta, ne asymptotika.
|
||||||
|
Zdroj: history 2026-09-09.
|
||||||
|
|
||||||
|
## FTS5 a čeština: `remove_diacritics` + prefix wildcard stačí, trigram netřeba (2026-09-09)
|
||||||
|
|
||||||
|
`unicode61 remove_diacritics 1` a výš foldí diakritiku (dotaz `zaloha` najde `záloha`;
|
||||||
|
hodnota `0` ne). Stemming `unicode61` neumí, ale **prefix wildcard to pokryje**: `záloh*`
|
||||||
|
i `zaloh*` najdou *záloha, zálohování, zálohy*. Dotazová vrstva tedy lepí `*` na termy —
|
||||||
|
trigram tokenizer není potřeba. Souvisí s varováním v `remind/scripts/store.py`
|
||||||
|
(`find_active_by_exact_text()`), že SQLite `lower()` foldí jen ASCII. Zdroj: history 2026-09-09.
|
||||||
|
|
||||||
|
## `vec0` není cílem foreign key — kaskáda ho nevyčistí (2026-09-09)
|
||||||
|
|
||||||
|
`ON DELETE CASCADE` z nadřazené tabulky uklidí navázané řádky **i FTS5 external-content
|
||||||
|
tabulku** (AFTER DELETE trigger se na kaskádě spustí), ale řádek ve `vec0` virtuální tabulce
|
||||||
|
**osiří** — indexer ho musí mazat explicitně. Jinak index při mazání souborů tiše plní mrtvými
|
||||||
|
vektory. Ověřeno smoke testem; `DELETE`/re-`INSERT`/`UPDATE` po `rowid` a `ROLLBACK` ve `vec0`
|
||||||
|
jinak fungují normálně (sqlite-vec v0.1.6). Zdroj: history 2026-09-09.
|
||||||
|
|
||||||
|
## Ollama `keep_alive` musí být číslo, ne string (2026-09-09)
|
||||||
|
|
||||||
|
`{"keep_alive": -1}` i `{"keep_alive": "24h"}` → HTTP 200; **`{"keep_alive": "-1"}` → HTTP 400**.
|
||||||
|
Cold load `qwen3-embedding:0.6b` (639 MB) je 1,81 s vs. warm 0,043 s, takže připíchnutí modelu
|
||||||
|
se vyplatí, ale není kritické. Batching je naopak podstatný: 8,9 chunk/s po jednom vs.
|
||||||
|
**108 chunk/s v batchi 32**. Zdroj: history 2026-09-09.
|
||||||
|
|
||||||
|
## Instruct prefix je u `qwen3-embedding` load-bearing, ne kosmetika (2026-09-09)
|
||||||
|
|
||||||
|
Na 12 parafrázových dotazech nad 123 reálnými chunky: **recall@10 8/12 s prefixem
|
||||||
|
vs. 3/12 bez něj** (MRR 0,261 vs 0,204). Plán `final-wiki-hybrid-rag` ho označoval za
|
||||||
|
„nekritickou vlastnost" na základě 5/5 vs 5/5 na jiném korpusu — na uživatelských datech
|
||||||
|
to neplatí. Prefix musí být **bitově identický** při indexaci i dotazu, proto žije
|
||||||
|
v `meta.query_prefix` a jeho změna vyžaduje `wiki_sync.py --full`. Zdroj: history 2026-09-09.
|
||||||
|
|
||||||
|
## FTS5 prefix wildcard NEfolduje českou flexi — jen prefixy (2026-09-09)
|
||||||
|
|
||||||
|
`záloh*` najde *záloha/zálohování/zálohy* jen proto, že `záloh` je **kmen**. Se skutečně
|
||||||
|
skloněným dotazovým slovem to selže: naměřeno `cestu*` → **0** chunků v
|
||||||
|
`travel/packaging-list.md`, `cesty*` → 3, `cest*` → 4. Wildcard je prefixový, takže pomůže
|
||||||
|
jen když je dotazové slovo prefixem tvaru v dokumentu — a česká flexe mění koncovku, ne
|
||||||
|
začátek. Důsledek: dotaz „co si vzít na cestu do zahraničí" nedostal doslovný seznam věcí
|
||||||
|
na cestu ani do top 10. Tohle je polovičnost D6 v plánu, ne chyba implementace.
|
||||||
|
Zdroj: history 2026-09-09.
|
||||||
|
|
||||||
|
## Hybrid RRF zlepšuje rank, ne pokrytí (2026-09-09)
|
||||||
|
|
||||||
|
Nad reálnými poznámkami: MRR **0,367 u RRF** vs 0,257 (BM25 sám) a 0,261 (vektory samy) —
|
||||||
|
merge dvou ranků téže množiny opravdu vyhrává. Ale `recall@10` u RRF je **6/12**, zatímco
|
||||||
|
vektory samotné 8/12: RRF řadí podle **shody** obou polovin, takže chunk, který našla jen
|
||||||
|
vektorová polovina na ranku #9, vytlačí z top-10 chunky, na kterých se poloviny shodnou.
|
||||||
|
Očekávaný kompromis, ne vada. Zdroj: history 2026-09-09.
|
||||||
|
|
||||||
|
## Bare `python3` neotevře `vec_chunks` — potřebuje `sqlite_vec.load()` (2026-09-09)
|
||||||
|
|
||||||
|
Dotaz na běžné tabulky (`chunks`, `files`) přes systémový `python3` projde, ale jakmile se
|
||||||
|
sáhne na `vec0` virtuální tabulku, přijde `sqlite3.OperationalError: no such module: vec0`.
|
||||||
|
Extension se musí načíst explicitně (`enable_load_extension(True)` + `sqlite_vec.load(conn)`),
|
||||||
|
což skill dělá v `wiki_db.get_db()`. Při ruční inspekci indexu na serveru je proto nutné
|
||||||
|
`uv run --with "sqlite-vec==0.1.6"`. Zdroj: history 2026-09-09.
|
||||||
|
|
||||||
|
## `ty.toml` `extra-paths` je plochý jmenný prostor — kolize modulů mezi skilly (2026-09-09)
|
||||||
|
|
||||||
|
Typechecker `ty` řeší `import store` proti seznamu `extra-paths` a **vyhrává první cesta**,
|
||||||
|
takže dva skilly se stejným jménem modulu si navzájem rozbijí kontrolu (testy `wiki`
|
||||||
|
dostávaly `store` z `remind`). Proto všechny skilly kromě `remind` prefixují moduly jménem
|
||||||
|
skillu (`note_capture.py`, `wiki_sync.py`) — je to nutnost, ne estetika. U kolize, které se
|
||||||
|
nelze vyhnout (`wiki_search.py` je i v retired `llm-wiki`), rozhoduje **pořadí** v
|
||||||
|
`extra-paths`. Zdroj: history 2026-09-09.
|
||||||
|
|||||||
@@ -1 +1 @@
|
|||||||
424
|
425
|
||||||
@@ -422,3 +422,4 @@
|
|||||||
{"cursor": 422, "timestamp": "2026-08-25 20:06", "content": "- [durable] `uv` is unavailable and system Python lacks `croniter`; the remind skill CLI cannot be executed via `uv run` and requires direct SQLite database access.\n- [permanent] Safety guard enforces a hard working-directory boundary on shell commands; blocked commands must not be retried with symlinks, base64 piping, alternative tools, or working_dir overrides.\n- [ephemeral] Model preset for this session is `kimi-k2.6:cloud` with a 262144-token context window and 16384 max output tokens.", "session_key": "telegram:8826147089"}
|
{"cursor": 422, "timestamp": "2026-08-25 20:06", "content": "- [durable] `uv` is unavailable and system Python lacks `croniter`; the remind skill CLI cannot be executed via `uv run` and requires direct SQLite database access.\n- [permanent] Safety guard enforces a hard working-directory boundary on shell commands; blocked commands must not be retried with symlinks, base64 piping, alternative tools, or working_dir overrides.\n- [ephemeral] Model preset for this session is `kimi-k2.6:cloud` with a 262144-token context window and 16384 max output tokens.", "session_key": "telegram:8826147089"}
|
||||||
{"cursor": 423, "timestamp": "2026-08-28 13:26", "content": "- [ephemeral] User switched model preset to `kimi27` (`kimi-k2.7-code:cloud`) for the current session.", "session_key": "websocket:f6e1e265-7a37-451c-ad7a-f601c40fdc5a"}
|
{"cursor": 423, "timestamp": "2026-08-28 13:26", "content": "- [ephemeral] User switched model preset to `kimi27` (`kimi-k2.7-code:cloud`) for the current session.", "session_key": "websocket:f6e1e265-7a37-451c-ad7a-f601c40fdc5a"}
|
||||||
{"cursor": 424, "timestamp": "2026-08-31 09:17", "content": "- [skip] Background cron drain task: compiled 17 inbox files from notes/inbox/ into notes/notes.md (created new, 7 thematic sections), moved all sources to notes/done/\n- [skip] Per note SKILL.md, the notes/ store is separate from agent memory (keep/MEMORY.md) — Dream must not touch notes/, so the compiled note contents (chata shopping list, DT Glass product details, ZOT work items, 25.5 cm chair-to-table measurement, etc.) live only in notes/notes.md and are not mirrored here\n- [skip] Two DT Glass product URLs (UNIVERSAL and Amber wine-bottle glasses, 69 Kč) fetched successfully — details filed under \"## DT Glass\" section in notes/notes.md\n- [skip] notes/notes.md did not exist before this compile; it was created fresh with sections: DevOps/Infra, Nanobot, Chata, DT Glass, Měření, 3D tisk, Work/ZOT", "session_key": "note-compile"}
|
{"cursor": 424, "timestamp": "2026-08-31 09:17", "content": "- [skip] Background cron drain task: compiled 17 inbox files from notes/inbox/ into notes/notes.md (created new, 7 thematic sections), moved all sources to notes/done/\n- [skip] Per note SKILL.md, the notes/ store is separate from agent memory (keep/MEMORY.md) — Dream must not touch notes/, so the compiled note contents (chata shopping list, DT Glass product details, ZOT work items, 25.5 cm chair-to-table measurement, etc.) live only in notes/notes.md and are not mirrored here\n- [skip] Two DT Glass product URLs (UNIVERSAL and Amber wine-bottle glasses, 69 Kč) fetched successfully — details filed under \"## DT Glass\" section in notes/notes.md\n- [skip] notes/notes.md did not exist before this compile; it was created fresh with sections: DevOps/Infra, Nanobot, Chata, DT Glass, Měření, 3D tisk, Work/ZOT", "session_key": "note-compile"}
|
||||||
|
{"cursor": 425, "timestamp": "2026-09-08 20:54", "content": "(nothing)", "session_key": "telegram:8826147089"}
|
||||||
|
|||||||
493
plans/final-wiki-hybrid-rag.md
Normal file
493
plans/final-wiki-hybrid-rag.md
Normal file
@@ -0,0 +1,493 @@
|
|||||||
|
# Final Wiki — hybrid RAG index nad poznámkami
|
||||||
|
|
||||||
|
**Stav:** připraveno k implementaci. **Vznik:** 2026-09-09.
|
||||||
|
**Nahrazuje** draft `notes-search-hybrid-rag.md` (server, 2026-09-09) — ten je superseded.
|
||||||
|
|
||||||
|
Veškerá čísla v tomto dokumentu jsou **naměřená na reálném prostředí**
|
||||||
|
(`nanobot.hell` → `nvidia.hell`, 2026-09-09), ne odhadnutá. Kde jde o odhad, je to napsané.
|
||||||
|
|
||||||
|
## Cíl
|
||||||
|
|
||||||
|
Uživatel vede poznámky ve vlastních git repozitářích (adresářová struktura, md soubory; témata
|
||||||
|
devops, traveling, mix). Nanobot dostane **read-only přístup** k mirrorům těchto repů a postaví
|
||||||
|
nad nimi index pro rychlé hledání — tematické i syntaktické. Druhým zdrojem je **nanobot
|
||||||
|
workspace** (živá data).
|
||||||
|
|
||||||
|
Poznámky zůstávají kanonickými daty v gitu; **index je derived artifact, regenerovatelný**.
|
||||||
|
Do uživatelových repů se nikdy nezapisuje.
|
||||||
|
|
||||||
|
Skill je **plně samostatný** — žádná závislost na jiném nanobot skillu.
|
||||||
|
|
||||||
|
## Ověřená fakta o prostředí
|
||||||
|
|
||||||
|
| Co | Naměřeno |
|
||||||
|
|---|---|
|
||||||
|
| Ollama endpoint `http://nvidia.hell:11434` | dosažitelný z nanobota |
|
||||||
|
| `qwen3-embedding:0.6b` | nainstalován; 595,78M params, **1024 dims**, capability `embedding` |
|
||||||
|
| Cold load modelu / warm | **1,81 s** / **0,043 s** |
|
||||||
|
| Embed throughput | 8,9 chunk/s po jednom → **108 chunk/s** v batchi 32 (12×) |
|
||||||
|
| `keep_alive` tvar | číslo `-1` i `"24h"` → 200; **string `"-1"` → HTTP 400** |
|
||||||
|
| Ollama **Cloud** embeddingy | **neexistují** — 18 cloud modelů, žádný s capability `embedding`; `/api/embed` na cloud modelu vrací `unauthorized`, zatímco `/api/generate` na tomtéž projde |
|
||||||
|
| SQLite | 3.46.1; FTS5 `unicode61`/`trigram`/`porter`; **`enable_load_extension` funguje** |
|
||||||
|
| `sqlite-vec` | **v0.1.6**, `vec0(float[1024] distance_metric=cosine)` v témže souboru jako `chunks`+FTS5 |
|
||||||
|
| `vec0` KNN k=20 | **19 ms @ 10k** chunků, **212 ms @ 100k** (BLOB+numpy: 594 ms) |
|
||||||
|
| `vec0` zápis | 10k = 2,1 s; 100k = 25,3 s; soubor 466 MB (BLOB 461 MB — bez režie) |
|
||||||
|
| `vec0` mutace | `DELETE`/re-`INSERT`/`UPDATE` po `rowid` **fungují**; `ROLLBACK` je transakční |
|
||||||
|
| BLOB+numpy rozpad @ 100k | read 382 ms + pack 190 ms + **dot 22 ms** → 96 % je Python režie |
|
||||||
|
| FTS5 čeština | `remove_diacritics 1`+ foldí (`zaloha` najde `záloha`); `záloh*` najde *záloha/zálohování/zálohy* |
|
||||||
|
| Reálný korpus workspace | 234 md mimo `tmp/` (1,28 MB) → **~800 chunků**; z toho použitelných ~99 souborů |
|
||||||
|
| `exec` tool timeout | **60 s** |
|
||||||
|
| `db/` ve workspace | gitignorováno → index je automaticky mimo git |
|
||||||
|
|
||||||
|
**Retrieval kvalita — měřeno na 178 chunkách reálného obsahu:**
|
||||||
|
|
||||||
|
| Dotaz | BM25 | embeddingy |
|
||||||
|
|---|---|---|
|
||||||
|
| „prodloužení životnosti LEGO" | 4/5 | **5/5** |
|
||||||
|
| „jak snížit elektroodpad ze stavebnic" | **1/5** | **3/5** |
|
||||||
|
|
||||||
|
Druhý řádek je přesně ten parafrázový/cross-jazyk případ, pro který tu hybrid je: BM25 selže,
|
||||||
|
vektory najdou. První řádek je opačný — lexikálně snadný dotaz zvládne BM25. **Obě poloviny
|
||||||
|
si vydělávají**, což potvrzuje D1 měřením, ne argumentem.
|
||||||
|
|
||||||
|
## Architektura
|
||||||
|
|
||||||
|
Klíčová vlastnost: **`chunks` je jediná retrieval jednotka.** RRF slučuje dva ranky *téže*
|
||||||
|
množiny — kdyby BM25 řadil soubory a vektory chunky, merge by neměl definovaný význam.
|
||||||
|
|
||||||
|
```
|
||||||
|
Zdroje:
|
||||||
|
git mirrory (RO clone/fetch, keyed na indexed-rev)
|
||||||
|
nanobot workspace (walk path+size+mtime)
|
||||||
|
↓ ingest driver (git-diff / fs-scan — dva drivery, jeden indexer)
|
||||||
|
↓ chunker (md → chunky po heading struktuře, markdown-it-py)
|
||||||
|
↓ embeddings: Ollama /api/embed (qwen3-embedding:0.6b), batch 32, jen změněné chunky
|
||||||
|
↓
|
||||||
|
db/notes-index.sqlite:
|
||||||
|
|
||||||
|
chunks (kanonické: metadata + text + stav embeddingu)
|
||||||
|
├── chunks_fts FTS5 external-content → BM25 rank nad chunks.id
|
||||||
|
└── vec_chunks vec0 virtual table → KNN rank nad chunks.id (rowid = chunks.id)
|
||||||
|
↓
|
||||||
|
RRF merge nad chunks.id
|
||||||
|
```
|
||||||
|
|
||||||
|
Tenhle tvar drží `vec0` **vyměnitelné**: kdyby pre-v1 breaking change zabolel, přidá se
|
||||||
|
`embedding BLOB` zpět do `chunks` a nic jiného se nemění.
|
||||||
|
|
||||||
|
### Schéma
|
||||||
|
|
||||||
|
```sql
|
||||||
|
CREATE TABLE IF NOT EXISTS meta (key TEXT PRIMARY KEY, value TEXT NOT NULL);
|
||||||
|
-- embedding_model, embedding_dims, normalized, query_prefix,
|
||||||
|
-- chunker_version, schema_version, sqlite_vec_version
|
||||||
|
|
||||||
|
CREATE TABLE IF NOT EXISTS sources (
|
||||||
|
source_id TEXT PRIMARY KEY, -- klíč z YAML, stabilní
|
||||||
|
kind TEXT NOT NULL CHECK(kind IN ('git','workspace')),
|
||||||
|
indexed_rev TEXT, -- jen git driver
|
||||||
|
last_sync_at TEXT
|
||||||
|
);
|
||||||
|
|
||||||
|
CREATE TABLE IF NOT EXISTS files (
|
||||||
|
source_id TEXT NOT NULL REFERENCES sources(source_id),
|
||||||
|
path TEXT NOT NULL, -- relativní ke zdroji
|
||||||
|
title TEXT, tags TEXT, headings TEXT, -- tags/headings jako JSON array
|
||||||
|
sha256 TEXT NOT NULL,
|
||||||
|
size INTEGER NOT NULL,
|
||||||
|
mtime REAL, -- rychlý pre-filter workspace driveru
|
||||||
|
indexed_at TEXT NOT NULL,
|
||||||
|
PRIMARY KEY (source_id, path)
|
||||||
|
);
|
||||||
|
|
||||||
|
CREATE TABLE IF NOT EXISTS chunks (
|
||||||
|
id INTEGER PRIMARY KEY AUTOINCREMENT,
|
||||||
|
source_id TEXT NOT NULL,
|
||||||
|
path TEXT NOT NULL,
|
||||||
|
chunk_idx INTEGER NOT NULL,
|
||||||
|
breadcrumb TEXT NOT NULL, -- "soubor > sekce > podsekce"
|
||||||
|
text TEXT NOT NULL,
|
||||||
|
embedded_at TEXT, -- NULL = čeká na vektor (degradovaný režim)
|
||||||
|
UNIQUE (source_id, path, chunk_idx),
|
||||||
|
FOREIGN KEY (source_id, path) REFERENCES files(source_id, path) ON DELETE CASCADE
|
||||||
|
);
|
||||||
|
CREATE INDEX IF NOT EXISTS idx_chunks_pending ON chunks(id) WHERE embedded_at IS NULL;
|
||||||
|
|
||||||
|
CREATE VIRTUAL TABLE IF NOT EXISTS chunks_fts USING fts5(
|
||||||
|
breadcrumb, text,
|
||||||
|
content='chunks', content_rowid='id',
|
||||||
|
tokenize='unicode61 remove_diacritics 2'
|
||||||
|
);
|
||||||
|
|
||||||
|
CREATE TRIGGER IF NOT EXISTS chunks_ai AFTER INSERT ON chunks BEGIN
|
||||||
|
INSERT INTO chunks_fts(rowid, breadcrumb, text) VALUES (new.id, new.breadcrumb, new.text);
|
||||||
|
END;
|
||||||
|
CREATE TRIGGER IF NOT EXISTS chunks_ad AFTER DELETE ON chunks BEGIN
|
||||||
|
INSERT INTO chunks_fts(chunks_fts, rowid, breadcrumb, text)
|
||||||
|
VALUES('delete', old.id, old.breadcrumb, old.text);
|
||||||
|
END;
|
||||||
|
CREATE TRIGGER IF NOT EXISTS chunks_au AFTER UPDATE ON chunks BEGIN
|
||||||
|
INSERT INTO chunks_fts(chunks_fts, rowid, breadcrumb, text)
|
||||||
|
VALUES('delete', old.id, old.breadcrumb, old.text);
|
||||||
|
INSERT INTO chunks_fts(rowid, breadcrumb, text) VALUES (new.id, new.breadcrumb, new.text);
|
||||||
|
END;
|
||||||
|
|
||||||
|
CREATE VIRTUAL TABLE IF NOT EXISTS vec_chunks USING vec0(
|
||||||
|
embedding float[1024] distance_metric=cosine
|
||||||
|
);
|
||||||
|
```
|
||||||
|
|
||||||
|
Poznámky ke schématu, všechny ověřené smoke testem:
|
||||||
|
|
||||||
|
- **Klíč je `(source_id, path, chunk_idx)`**, nikdy jen `path` — `DELETE WHERE path = ?` by
|
||||||
|
mazalo chunky cizího zdroje.
|
||||||
|
- **Stav embeddingu je sloupec `chunks.embedded_at`**, ne absence řádku ve `vec_chunks`.
|
||||||
|
U `vec0` se „chybějící vektor" dotazuje blbě, a degradovaný režim potřebuje levný
|
||||||
|
`WHERE embedded_at IS NULL` (proto ten partial index).
|
||||||
|
- **PAST: `ON DELETE CASCADE` uklidí `chunks` i `chunks_fts` (trigger se na kaskádě spustí),
|
||||||
|
ale řádek ve `vec_chunks` osiří** — `vec0` není cílem foreign key. Indexer **musí** mazat
|
||||||
|
`vec_chunks` explicitně. Patří to do regresního testu.
|
||||||
|
- Vektory se ukládají **L2-normalizované** (`meta.normalized`), takže cosine == dot product.
|
||||||
|
- `sqlite3.connect(path, isolation_level=None)` — autocommit, transakce řízené explicitně.
|
||||||
|
Bez toho `BEGIN` spadne na „cannot start a transaction within a transaction".
|
||||||
|
|
||||||
|
## Chunking
|
||||||
|
|
||||||
|
Strukturně vědomý, po heading hierarchii — uživatelovy poznámky jsou md se smysluplnou
|
||||||
|
strukturou (adresáře = témata, soubory, H1–H3 sekce), takže slepování přes hranice sekcí by
|
||||||
|
embeddingy kazilo.
|
||||||
|
|
||||||
|
- Rozdělení po heading sekcích (H1–H3); každý chunk nese **breadcrumb** (`soubor > sekce >
|
||||||
|
podsekce`) v metadatech **i v embedding textu** — vektor tak nese kontext, ne izolovaný odstavec.
|
||||||
|
- Merge malých sousedních sekcí pod stejným rodičem (< ~200 tokenů).
|
||||||
|
- Split velkých sekcí (> ~800 tokenů) po odstavcích s overlap ~50–80 tokenů.
|
||||||
|
Cílová granularita ~200–800 tokenů/chunk.
|
||||||
|
- Kód bloky a tabulky se nerozbíjejí; checklistové soubory chunkujeme po bullet blocích.
|
||||||
|
- Frontmatter tagy jdou do `files.tags` (na filtrování), do embedding textu jde jen title.
|
||||||
|
- Token counting: aproximace ~4 znaky/token (přesný tokenizer pro sizing netřeba).
|
||||||
|
|
||||||
|
Parser je **markdown-it-py** (zná CommonMark edge cases — setext headings, nested listy,
|
||||||
|
HTML bloky), **chunkovací politika je vlastní** (~50 řádků nad tokeny).
|
||||||
|
|
||||||
|
## Hledání — tři vrstvy, od nejlevnější
|
||||||
|
|
||||||
|
1. **Syntakticky — live grep** (ripgrep přes mirror): exact match, názvy souborů, hostname,
|
||||||
|
tagy, čísla. Žádný index, vždy aktuální. Základ, ne fallback.
|
||||||
|
2. **Tematicky — generovaný TOC** (`toc.md` per zdroj: kategorie → soubor → jedna řádka;
|
||||||
|
title + tagy + headings). Index-first navigace, generovaná syncem.
|
||||||
|
3. **Fuzzy/sémanticky — hybrid**: FTS5 (BM25) + `vec0` (KNN), merge přes RRF.
|
||||||
|
|
||||||
|
### Query kontrakt
|
||||||
|
|
||||||
|
- **Instruct prefix**: dotaz embedovat jako `Instruct: <task>\nQuery: <text>`, dokumenty bez
|
||||||
|
prefixu (Qwen3-Embedding je asymetrický instruct-tuned model; Ollama prefix nepřidá).
|
||||||
|
**Měření ale ukázalo, že to není kritická vlastnost**: recall@5 byl s prefixem i bez něj
|
||||||
|
identický (5/5 vs 5/5, 3/5 vs 3/5) a absolutní similarita s prefixem dokonce nižší.
|
||||||
|
Prefix zlepšil jen **top-1 na nejtěžším dotazu** (`e-waste-reduction.md` místo
|
||||||
|
`software-preservation.md`). Zavádíme ho, protože je zdarma a na hraničním dotazu pomohl —
|
||||||
|
a hlavně proto, že **musí být bitově identický při indexaci i dotazu**, což je skutečný
|
||||||
|
důvod, proč hodnota žije v `meta.query_prefix`.
|
||||||
|
- **FTS5 termy s prefix wildcardem** (`záloh*`) — pokrývá českou flexi, kterou `unicode61`
|
||||||
|
nestemuje. Ověřeno: `záloh*` i `zaloh*` najdou *záloha/zálohování/zálohy*.
|
||||||
|
- **RRF** se dělá v Pythonu nad dvěma seznamy `chunks.id` (`vec0` KNN vyžaduje `k`).
|
||||||
|
- **Mismatch `meta` vs. config** (jiný model/dims/prefix/chunker_version) → **dotaz odmítnout**
|
||||||
|
s „reindex needed". Nikdy tiše nemíchat vektory ze dvou modelů.
|
||||||
|
- **Ollama nedostupná** → FTS-only a **říct to ve výstupu**, ne tiše degradovat.
|
||||||
|
|
||||||
|
## Provozní model — offline sync
|
||||||
|
|
||||||
|
**Indexace nikdy neběží v tahu agenta.** `exec` má timeout 60 s a plný index ~10⁴ chunků při
|
||||||
|
108 chunk/s je ~77 s. Agent v tahu jen **čte** hotový index.
|
||||||
|
|
||||||
|
```
|
||||||
|
cron: * * * * * (vzor: existující remind_send / note_compile řádky; PATH v hlavičce crontabu)
|
||||||
|
↓
|
||||||
|
notes_sync.py
|
||||||
|
1. lock: db/.notes-sync.lock (O_EXCL). Držený → exit 0 bez výpisu. Žádný souběh.
|
||||||
|
2. levná detekce změn, BEZ indexace:
|
||||||
|
git zdroje: git ls-remote <url> HEAD vs. sources.indexed_rev (síť, ne fetch)
|
||||||
|
workspace zdroje: walk + (path, size, mtime) vs. files (sha256 jen na mismatch)
|
||||||
|
3. nic se nezměnilo → exit 0 (běžný případ, drtivá většina tiků)
|
||||||
|
4. změněné soubory: chunk → embed (batch 32, keep_alive -1) → upsert v transakci
|
||||||
|
smazaný/přejmenovaný soubor: DELETE z files (kaskáda uklidí chunks+FTS)
|
||||||
|
+ EXPLICITNÍ DELETE z vec_chunks
|
||||||
|
5. update sources.indexed_rev / last_sync_at
|
||||||
|
6. coverage report: zaloguj top-level adresáře s *.md, které nepokrývá žádný source
|
||||||
|
7. log do log/notes_sync.log
|
||||||
|
```
|
||||||
|
|
||||||
|
- `git ls-remote` je pro minutovou kadenci správný nástroj — zjistí remote HEAD **bez** `fetch`.
|
||||||
|
`fetch` teprve když se rev liší.
|
||||||
|
- **Idempotentní**: re-run nad stejnou revizí = no-op.
|
||||||
|
- Lock řeší souběh sám, žádná externí orchestrace.
|
||||||
|
- **Plný re-index** = tentýž skript s `--full`. Vzácná operace (změna modelu nebo chunkeru).
|
||||||
|
- `keep_alive: -1` na embed requestech drží model resident. Není kritické (cold load je 1,81 s),
|
||||||
|
ale je to zdarma. Pozn.: pod tlakem na VRAM od velkých chat modelů (gemma4:12b má 7,5 GB)
|
||||||
|
může scheduler potřebovat místo — při 639 MB je to nepravděpodobné, ale garance to není.
|
||||||
|
|
||||||
|
## Rozsah indexace — whitelist primárně, blacklist jako skalpel
|
||||||
|
|
||||||
|
**Precedence:** `paths` (whitelist — co v něm není, pro index neexistuje) → `include`
|
||||||
|
(whitelist přípon, default `*.md`) → `exclude` (skalpel, vyhrává nad oběma).
|
||||||
|
|
||||||
|
**Proč whitelist ve workspace — směr selhání.** U blacklistu nový adresář *tiše vstoupí* do
|
||||||
|
indexu, u whitelistu *tiše chybí*. Embedding tabulka je de facto komprimovaná kopie obsahu
|
||||||
|
(včetně osobních věcí), takže „tiše zaindexováno" je horší porucha. A workspace se mění
|
||||||
|
autonomně — Dream přepisuje soubory, skilly appendují, cron zapisuje.
|
||||||
|
|
||||||
|
**Doloženo měřením:** ve workspace je 234 md mimo `tmp/` a použitelných je ~99. Zbytek je
|
||||||
|
`cml/` 39, `skills/` 32, `.venv/` 27, `backup/` 20, `tasks/` 16, `.pytest_cache` 1.
|
||||||
|
A `tmp/` drží **dalších 135 md** (git klony, 5× reflect dump po ~200 kB). Blacklist by musel
|
||||||
|
hned první den správně pokrýt pět a půl adresáře a zůstat správný navždy — a nejhorší z nich
|
||||||
|
je právě ten, který je určený k tomu, aby se v něm hromadil balast.
|
||||||
|
|
||||||
|
**Proč opt-out u git rep.** Uživatelova poznámková repa jsou kurátorovaná a homogenní;
|
||||||
|
vyjmenovávat v nich podadresáře je zbytečná friction. Tam `paths: ["**"]` a malý `exclude`.
|
||||||
|
|
||||||
|
### Co se indexuje
|
||||||
|
|
||||||
|
| Zdroj | Cesty |
|
||||||
|
|---|---|
|
||||||
|
| `workspace` | `notes/**`, `projects/**`, `plans/**`, `knowledge/**`, `results/**`, `cook/**` |
|
||||||
|
| `develop` | `develop/**` mimo `develop/history.md` |
|
||||||
|
| git zdroje | celé repo, `*.md` |
|
||||||
|
|
||||||
|
`results/` (21 souborů, 256 kB výstupů deep-research) a `develop/knowledge.md` (113 kB hutných
|
||||||
|
ověřených faktů o instanci) jsou vědomé **přírůstky** — whitelist z nich dělá rozhodnutí.
|
||||||
|
|
||||||
|
### Co se neindexuje a proč
|
||||||
|
|
||||||
|
| Cesta | Důvod |
|
||||||
|
|---|---|
|
||||||
|
| `tmp/` (135 md) | git klony + reflect dumpy po 200 kB; adresář určený k balastu |
|
||||||
|
| `.venv/` (27), `.pytest_cache/`, `.ruff_cache/` | dokumentace balíčků a cache |
|
||||||
|
| `backup/` (20) | **near-duplicate kopie indexovaného obsahu** — otrávily by top-k redundantními hity; to je horší porucha než chybějící dokument |
|
||||||
|
| `develop/history.md` (331 kB) | append-only deník; ~200 chunků repetitivní narativy = ~25 % indexu při nízké hustotě signálu. `develop/knowledge.md` vedle něj zůstává |
|
||||||
|
| `skills/**` (32) | instrukce pro agenta, ne znalosti; agent si skilly načítá sám |
|
||||||
|
| `cml/` (39) | llm-wiki, ruší se mimo tento plán |
|
||||||
|
| root `AGENTS.md`/`SOUL.md`/`USER.md`/`keep.md`/`HEARTBEAT.md`, `memory/MEMORY.md` | vždy v kontextu nebo triviálně krátké → čistý šum |
|
||||||
|
| `log/`, `sessions/`, `db/`, `cron/`, `tasks/` | provozní stav, ne obsah |
|
||||||
|
| `memory/history.jsonl`, binárky | vyřazuje už `include: ["*.md"]` — do `exclude` psát netřeba |
|
||||||
|
|
||||||
|
**Záchranná síť proti jediné slabině whitelistu:** sync na konci zaloguje top-level adresáře,
|
||||||
|
které obsahují `*.md` a nepokrývá je žádný source. Tím se „tiše chybí" změní z neviditelné
|
||||||
|
poruchy na řádek v `log/notes_sync.log`.
|
||||||
|
|
||||||
|
## Konfigurace
|
||||||
|
|
||||||
|
`config/notes-search.yaml`. **Klíč sekce = source id** (stabilní; visí na něm katalog i vektory;
|
||||||
|
URL/path se můžou změnit, klíč ne; rename = explicitní invalidace indexu daného zdroje).
|
||||||
|
Source id nesmí kolidovat mezi sekcemi — katalog je sdílený přes `source_id`.
|
||||||
|
|
||||||
|
Formát je **YAML** (ne TOML — zamítnuto uživatelem, zapsáno v `USER.md`): config se edituje
|
||||||
|
ručně a komentáře v něm mají hodnotu, což JSON neumí.
|
||||||
|
|
||||||
|
```yaml
|
||||||
|
embedding:
|
||||||
|
endpoint: http://nvidia.hell:11434
|
||||||
|
model: qwen3-embedding:0.6b # tag psát VŽDY explicitně (latest = 8b, 4,7 GB)
|
||||||
|
dims: 1024
|
||||||
|
batch: 32
|
||||||
|
keep_alive: -1 # číslo, ne string ("-1" vrací HTTP 400)
|
||||||
|
query_prefix: "Instruct: Given a web search query, retrieve relevant passages that answer the query\nQuery: "
|
||||||
|
|
||||||
|
sources:
|
||||||
|
git:
|
||||||
|
travel:
|
||||||
|
url: git@host:travel-notes.git
|
||||||
|
mirror: tmp/mirrors/travel.git
|
||||||
|
paths: ["**"]
|
||||||
|
include: ["*.md"]
|
||||||
|
exclude: []
|
||||||
|
devops:
|
||||||
|
url: https://host/devops-notes.git
|
||||||
|
mirror: tmp/mirrors/devops.git
|
||||||
|
paths: ["**"]
|
||||||
|
include: ["*.md"]
|
||||||
|
|
||||||
|
nanobot:
|
||||||
|
workspace:
|
||||||
|
paths: ["notes/**", "projects/**", "plans/**", "knowledge/**", "results/**", "cook/**"]
|
||||||
|
include: ["*.md"]
|
||||||
|
exclude: ["**/inbox/**"] # rozpracované zachyty před compile
|
||||||
|
develop:
|
||||||
|
paths: ["develop/**"]
|
||||||
|
include: ["*.md"]
|
||||||
|
exclude: ["develop/history.md"]
|
||||||
|
```
|
||||||
|
|
||||||
|
## Rozhodnutí
|
||||||
|
|
||||||
|
### D1 — Embeddings jako vrstva nad BM25, ne místo něj
|
||||||
|
|
||||||
|
Embeddingy přidávají parafrázi a cross-jazyk („prodloužení životnosti LEGO" najde „e-waste,
|
||||||
|
Mindstorms po ukončení podpory" — žádná slova se nepřekrývají) a chunk-level relevanci.
|
||||||
|
Nevyřeší exact match a čísla. Proto **hybrid**: FTS5 lexikálně, vektory sémanticky, RRF merge.
|
||||||
|
|
||||||
|
**Potvrzeno měřením**, ne argumentem — viz tabulka retrieval kvality výše: na těžkém dotazu
|
||||||
|
BM25 1/5 vs. embeddingy 3/5, na lexikálním naopak BM25 4/5.
|
||||||
|
|
||||||
|
### D2 — Model `qwen3-embedding:0.6b`
|
||||||
|
|
||||||
|
1024 dims, 32k ctx, 639 MB (q8_0), multilingual s češtinou jako first-class. Naměřeno:
|
||||||
|
cold 1,81 s, warm 0,043 s, 108 chunk/s v batchi 32. Na parafrázových dotazech 5/5 — **0,6B stačí**.
|
||||||
|
|
||||||
|
- **Tag psát vždy explicitně**: `latest` je 8b (4,7 GB), ne šestistovka.
|
||||||
|
- `keep_alive: -1` (číslo).
|
||||||
|
- **Volba modelu NENÍ silně vážící rozhodnutí.** Při 10³–10⁴ chunků je plný re-embed jednotky
|
||||||
|
minut, takže přechod na `4b-q8_0` je odpoledne, ne rewrite. Signál pro upgrade: parafrázový
|
||||||
|
dotaz, kde správný chunk existuje a keyword dotaz ho najde, ale sémantická polovina ho
|
||||||
|
nevrátí ani v top-10.
|
||||||
|
- Storage není omezení: 10⁴ × 1024 × 4 B ≈ 40 MB. Matryoshka truncation netřeba.
|
||||||
|
|
||||||
|
### D3 — Vektory v `sqlite-vec` (`vec0`), ne BLOB + numpy
|
||||||
|
|
||||||
|
Rozhodující je rozpad nákladu, ne teorie: u BLOB+numpy je při 100k chunků 594 ms celkem, z toho
|
||||||
|
**read 382 ms + pack 190 ms a samotný dot jen 22 ms** — 96 % je Python režie na extrakci
|
||||||
|
a packování. `vec0` ji odřízne skenem v C: **19 ms @ 10k, 212 ms @ 100k**.
|
||||||
|
|
||||||
|
Pozor na zdůvodnění: **`vec0` není ANN index.** Dokumentovaná cesta dotazu je průchod
|
||||||
|
(lineární škálování 19 → 212 ms to potvrzuje). Výhoda je konstanta, ne asymptotika.
|
||||||
|
|
||||||
|
**Cena:** `sqlite-vec` je pre-v1 a README píše „expect breaking changes"; užší dotazovací plocha
|
||||||
|
(max 16 metadata sloupců, 4 partition keys, `IS NULL`/`LIKE` na metadatech nefunguje, auxiliary
|
||||||
|
sloupce nesmí do KNN `WHERE`); ztráta volné numpy matematiky (MMR re-ranking, truncation za běhu).
|
||||||
|
|
||||||
|
**Mitigace:** verze připíchnutá (`sqlite-vec==0.1.6`) a zapsaná v `meta.sqlite_vec_version`;
|
||||||
|
index je derived, `db/` gitignorovaná a rebuild jsou jednotky minut; `chunks` zůstává kanonická,
|
||||||
|
takže **návrat k BLOBu je přidání jednoho sloupce**. Blast radius breaking changu je „zůstaň na
|
||||||
|
staré verzi, nebo přizpůsob a přeindexuj", ne ztráta dat.
|
||||||
|
|
||||||
|
**Ověřeno smoke testem** (jinak by D3 padlo): `DELETE`/re-`INSERT`/`UPDATE` po `rowid`,
|
||||||
|
transakční `ROLLBACK`, `distance_metric=cosine`, `vec0` v témže souboru jako běžné tabulky,
|
||||||
|
join `vec_chunks.rowid = chunks.id`.
|
||||||
|
|
||||||
|
### D4 — Chunking strukturně vědomý
|
||||||
|
|
||||||
|
Viz sekce Chunking. Fixed-size sliding window zamítnuto.
|
||||||
|
|
||||||
|
### D5 — markdown-it-py, ne vlastní parser a ne frameworky
|
||||||
|
|
||||||
|
Dva různé případy:
|
||||||
|
|
||||||
|
- **Parser patří do knihovny.** Vlastní regex zná CommonMark edge cases jen do té míry, do jaké
|
||||||
|
si je ošetříš. markdown-it-py je malá, stabilní, zero-bloat (jen `mdurl`), dává proper AST.
|
||||||
|
Chunkovací **politika** ale zůstává vlastní.
|
||||||
|
- **Glue kód neobírat frameworkem.** Sync orchestrace, katalog, hybrid search — tady neexistuje
|
||||||
|
„malá dobrá knihovna", nabídka je binární: langchain/llama-index (stovky MB, abstrakce nad
|
||||||
|
sqlite3/subprocess/HTTP, měnící se API) nebo vlastních ~500 řádků. Tady „psát si sám" není
|
||||||
|
NIH, je to jediná racionální volba, protože alternativa obaluje čtyři stdlib volání.
|
||||||
|
|
||||||
|
Dependency set: `markdown-it-py`, `requests`, `numpy`, `pyyaml`, `sqlite-vec==0.1.6`.
|
||||||
|
|
||||||
|
### D6 — FTS5 tokenizer `unicode61 remove_diacritics 2` + prefix wildcardy
|
||||||
|
|
||||||
|
**Zavřeno měřením, ne odloženo na testování.** `remove_diacritics 1` a výš foldí diakritiku
|
||||||
|
(dotaz `zaloha` najde `záloha`; `0` ne). `unicode61` nestemuje, ale prefix wildcard to pokryje:
|
||||||
|
`záloh*` najde *záloha, zálohování, zálohy*. **Trigram tokenizer zamítnut** — netřeba.
|
||||||
|
Dotazová vrstva lepí `*` na termy delší než 2 znaky.
|
||||||
|
|
||||||
|
### D7 — Skill je samostatný, bez závislostí na jiné skilly
|
||||||
|
|
||||||
|
Vzory z `remind`/`note` se **kopírují, neimportují**. Vlastní `db.py`/`store.py`, vlastní
|
||||||
|
lockfile, **žádný `detach`**, nepřebírat `wiki_search.py` ani `note_capture._ascii_fold()`
|
||||||
|
(FTS5 `remove_diacritics` folding stejně řeší). Cena je duplikace kódu; hodnota je, že skill
|
||||||
|
nespadne s ničím jiným a dá se přenést.
|
||||||
|
|
||||||
|
### D8 — Mirror jako zdroj, sync offline
|
||||||
|
|
||||||
|
Git repa jsou kanonická data, index je derived. Mirror přes `git clone --mirror` / `fetch`.
|
||||||
|
|
||||||
|
**`fetch-on-query` zamítnut**: přidával `git fetch` *i embed nových chunků* do latence dotazu,
|
||||||
|
a při nedostupné Ollamě dělal z čerstvého dokumentu FTS-only výsledek. Místo toho offline cron
|
||||||
|
+ lock (viz Provozní model).
|
||||||
|
|
||||||
|
**Staleness není problém**: index je retrieval hint, ne source of truth — před odpovědí se
|
||||||
|
soubor vždy přečte čerstvý z disku. Dotaz může vidět index o minutu starší, což je přijatelné.
|
||||||
|
|
||||||
|
### D9 — `chunks` je jediná retrieval jednotka
|
||||||
|
|
||||||
|
`chunks_fts` i `vec_chunks` vracejí `chunks.id`, RRF slučuje je. Klíč `(source_id, path,
|
||||||
|
chunk_idx)`. Rank-merge dvou různých jednotek nemá definovaný význam → file-level BM25 vyloučen.
|
||||||
|
|
||||||
|
### D10 — Identita embedding prostoru v `meta`
|
||||||
|
|
||||||
|
`embedding_model`, `embedding_dims`, `normalized`, `query_prefix`, `chunker_version`.
|
||||||
|
Mismatch proti configu **odmítne dotaz** s „reindex needed". Bez toho by se smíchané vektory
|
||||||
|
ze dvou modelů projevily jako **tiché zhoršení výsledků, ne jako chyba** — nejdražší druh bugu.
|
||||||
|
|
||||||
|
### D11 — Indexace výhradně offline
|
||||||
|
|
||||||
|
Cron + lockfile, nikdy v tahu agenta. Důvod: `exec` timeout 60 s vs. plný index ~77 s.
|
||||||
|
|
||||||
|
### D12 — Rozsah indexace whitelistem
|
||||||
|
|
||||||
|
Viz sekce Rozsah indexace. `paths` je hradlo, `exclude` skalpel; workspace opt-in, git repa opt-out.
|
||||||
|
|
||||||
|
## Zamítnuté varianty
|
||||||
|
|
||||||
|
| Varianta | Proč ne |
|
||||||
|
|---|---|
|
||||||
|
| **Ollama Cloud embeddingy** | Cloud tier embedding endpoint **neservíruje** — 18 cloud modelů, žádný s capability `embedding`; `/api/embed` vrací `unauthorized`, zatímco completion na tomtéž modelu projde. Katalog na ollama.com to potvrzuje: všech 12 embedding modelů je jen ke stažení. Navíc by to porušilo lokalitu dat (celý korpus osobních poznámek do cizí služby) a plný re-index = ~10⁴ requestů na externí API |
|
||||||
|
| Generativní cloud model + pooling hidden states | Generativní modely nejsou kontrastivně trénované na retrieval; proto je capability oddělená |
|
||||||
|
| `nomic-embed-text` (137M) | EN-centric, pro češtinu nevýhodný |
|
||||||
|
| `bge-m3` (567M) | Umí dense+sparse hybrid z jednoho modelu, ale přes Ollama embeddings API jde dostat jen dense — výhoda mizí, FTS5 dělá tutéž roli levněji |
|
||||||
|
| `qwen3-embedding` 4b/8b jako start | Lepší multilingual skóre, ale 2,5–15 GB na kartě sdílené s chat modely. Upgrade je odpoledne (re-embed = minuty), tak začít malým |
|
||||||
|
| `*-q4_K_M` kvantizace | 4bit u embeddingu; šestistovka q4 v nabídce ani není |
|
||||||
|
| BLOB + numpy cosine | 96 % nákladu je Python režie; viz D3 |
|
||||||
|
| `.npy` + `mmap_mode='r'` (35 ms @ 100k) | Nejrychlejší, ale druhý soubor mimo DB, který se musí držet v sync s `chunks` — složitost bez přínosu, když `vec0` dává 19 ms na reálné škále |
|
||||||
|
| Externí vektorová DB (Chroma/Qdrant/pgvector) | Řeší problém, který na této škále neexistuje (server, jiná backup story, >10⁶ vektorů) |
|
||||||
|
| Trigram tokenizer | Netřeba, `remove_diacritics 2` + prefix wildcardy stačí (D6) |
|
||||||
|
| Entity/concept pages jako v llm-wiki | Uživatel poznámky strukturoval sám; LLM-kurátorovaná druhá vrstva by byla duplikace. Adresářová struktura repů JE primární tematický index — zpřístupňujeme ji (TOC), nereimplementujeme |
|
||||||
|
| `fetch-on-query` / `scan-on-query` | Viz D8 |
|
||||||
|
| Blacklist jako primární gate | Viz D12 |
|
||||||
|
| Embeddingy až když fuzzy dotazy selžou | Zamítnuto — jsou součástí od začátku |
|
||||||
|
|
||||||
|
## Konvence implementace
|
||||||
|
|
||||||
|
Vzory z existujících skillů (kopírovat, neimportovat — D7):
|
||||||
|
|
||||||
|
| Věc | Vzor | Zdroj |
|
||||||
|
|---|---|---|
|
||||||
|
| Rozdělení kódu | `db.py` (SCHEMA, `get_db()`, `_migrate()`, `init_db()`) + `store.py` (`connection()`/`transaction()` + veškeré SQL) + CLI bez inline SQL | `skills/remind/scripts/` |
|
||||||
|
| Migrace | idempotentní `CREATE ... IF NOT EXISTS` v jednom `SCHEMA` přes `executescript()` + `_migrate()` s `PRAGMA table_info` a `ALTER TABLE ADD COLUMN`. Žádná `schema_version` tabulka, žádný framework | `remind/scripts/db.py` |
|
||||||
|
| Connection | `sqlite3.connect(path, isolation_level=None)` + `PRAGMA journal_mode=WAL`, `foreign_keys=ON`, `row_factory=sqlite3.Row` | `remind/scripts/db.py` |
|
||||||
|
| Cesta k DB | `WORKSPACE = Path(__file__).resolve().parents[3]`, `WORKSPACE/"db"/"notes-index.sqlite"`, env override pro testy | `note/scripts/note_capture.py:25` |
|
||||||
|
| Shebang | entry point `#!/usr/bin/env -S uv run --script` + PEP 723; importovaný modul `#!/usr/bin/env python3` + PEP 723 | `remind/scripts/*` |
|
||||||
|
| Cron řádek | `* * * * * uv run .../scripts/notes_sync.py >> log/notes_sync_cron.log 2>&1` (`PATH` je v hlavičce crontabu) | server `crontab -l` |
|
||||||
|
| Lockfile | `notes/.compile.lock` je precedens → `db/.notes-sync.lock` | `note/scripts/note_compile.py` |
|
||||||
|
| SKILL.md | frontmatter jen `name` + `description` (folded `>`, EN, s `Triggers on:` a **funkční** negativní delimitací — nikdy jménem jiného skillu); tělo ~100 řádků; workspace-relativní `uv run skills/<name>/scripts/x.py` | `remind/SKILL.md` |
|
||||||
|
| Testy | `skills/<name>/tests/`, `uv run --with pytest pytest ...`, izolace přes `tmp_path` + monkeypatch modulového `DB_PATH` | `remind/tests/` |
|
||||||
|
| Deploy | `rsync -av --exclude '__pycache__' --exclude '.pytest_cache' skills/<name>/ nanobot@nanobot.hell:/home/nanobot/.nanobot/workspace/skills/<name>/` | `CLAUDE.md` |
|
||||||
|
|
||||||
|
Query interface: skill s CLI skriptem `notes_search.py` (vzor `remind_cli.py` — argparse,
|
||||||
|
subcommandy, `--help` místo plné flag reference v SKILL.md).
|
||||||
|
|
||||||
|
## Otevřené otázky
|
||||||
|
|
||||||
|
1. **Seznam rep** — která repa, jejich URL, kam mirrorovat, velikost/historie (vliv na fetch čas).
|
||||||
|
Musí doplnit autor; bez toho nelze naplnit `config/notes-search.yaml`.
|
||||||
|
2. **Benchmark modelu na uživatelských datech** — cross-jazyk test výše proběhl na workspace
|
||||||
|
obsahu (`cml/wiki`, `plans/`). Po přidání reálných rep ho zopakovat na nich.
|
||||||
|
|
||||||
|
## Verifikace
|
||||||
|
|
||||||
|
**Hotovo (2026-09-09)** — schéma i model ověřené smoke testem, viz tabulky faktů:
|
||||||
|
`vec0` mutace a transakčnost, KNN latence na dvou škálách, FTS5 external-content triggery
|
||||||
|
s češtinou, RRF merge nad `chunks.id`, cold load / throughput modelu, retrieval kvalita.
|
||||||
|
|
||||||
|
**Při implementaci:**
|
||||||
|
|
||||||
|
1. **Regresní test na osiřelé vektory** — smaž soubor → `vec_chunks` nesmí obsahovat jeho
|
||||||
|
rowidy. Tohle je jediná past, kterou schéma samo neochrání (kaskáda na `vec0` nedosáhne).
|
||||||
|
2. **Idempotence syncu** — dvakrát za sebou nad stejnou revizí: druhý běh nesmí nic změnit
|
||||||
|
(počty v `chunks`/`vec_chunks`/`chunks_fts` shodné, `indexed_rev` stejná).
|
||||||
|
3. **Lock** — spustit dva syncy současně; druhý musí skončit exit 0 bez zápisu.
|
||||||
|
4. **Degradovaný režim** — s vypnutou/nedosažitelnou Ollamou: sync uloží chunky
|
||||||
|
s `embedded_at IS NULL`, dotaz vrátí FTS-only výsledek a **řekne to**; po obnovení
|
||||||
|
Ollamy další sync vektory dosadí.
|
||||||
|
5. **`meta` guard** — podvrhni v configu jiný `model`/`dims` → dotaz musí skončit
|
||||||
|
„reindex needed", ne vrátit výsledky.
|
||||||
|
6. **Coverage report** — přidej md soubor do adresáře mimo `paths` → sync ho musí ohlásit v logu.
|
||||||
|
7. **Latence dotazu end-to-end** — cíl: pod 1 s při teplém modelu (embed dotazu 0,043 s
|
||||||
|
+ KNN ~19 ms + BM25 + RRF).
|
||||||
@@ -1,5 +1,7 @@
|
|||||||
# Notes Search — RO git repa → hybrid RAG index
|
# Notes Search — RO git repa → hybrid RAG index
|
||||||
|
|
||||||
|
> **Superseded by `final-wiki-hybrid-rag.md`** (2026-09-09) — tento draft je historie.
|
||||||
|
|
||||||
Stav: DRAFT — budeme ještě opracovávat, než se pustíme do realizace.
|
Stav: DRAFT — budeme ještě opracovávat, než se pustíme do realizace.
|
||||||
Vznik: diskuze 2026-09-XX (nahradit přesným datem při finalizaci).
|
Vznik: diskuze 2026-09-XX (nahradit přesným datem při finalizaci).
|
||||||
|
|
||||||
|
|||||||
65
projects/ai/artifacts/session-lifecycle-guide.md
Normal file
65
projects/ai/artifacts/session-lifecycle-guide.md
Normal file
@@ -0,0 +1,65 @@
|
|||||||
|
# Session lifecycle — kdy zůstat ve stejné session, kdy začít novou
|
||||||
|
|
||||||
|
Návod pro práci s LLM agenty (Claude Code, nanobot, …). Vychází z lessons learned
|
||||||
|
v tomto projektu (review smyčka, Solarflare, file-based komunikace) a z obecných
|
||||||
|
mechanik kontextu.
|
||||||
|
|
||||||
|
## Rychlý rozhodovací test
|
||||||
|
|
||||||
|
Na každou další operaci si polož jednu otázku:
|
||||||
|
|
||||||
|
**Co další krok víc potřebuje — historii rozhodnutí, nebo svěží pohled?**
|
||||||
|
|
||||||
|
| Další krok potřebuje | Kam | Proč |
|
||||||
|
|---|---|---|
|
||||||
|
| Vědět, co už padlo a proč | **stejná session** | decision history je v kontextu, agent drží odsouhlasená rozhodnutí |
|
||||||
|
| Nezávazný, „čisté oči" | **nová session** | žádný anchoring na vlastní předchozí práci |
|
||||||
|
| Obojí (typicky opravy po review) | **hybrid** | viz níž |
|
||||||
|
|
||||||
|
A dva přepínače, které mají přednost:
|
||||||
|
|
||||||
|
- **Session je „zkažená"** (2+ selhání stejné věci, chybová smyčka) → nová session. Selhané pokusy v kontextu táhnou model k jejich opakování.
|
||||||
|
- **Proběhla kompakce / historie už není kompletní** → radši nová session + přečtené soubory (state, spec, diff) než práce z faded recollection. Půlzapamatovaná historie je horší než žádná.
|
||||||
|
|
||||||
|
## Kdy zůstat ve stejné session
|
||||||
|
|
||||||
|
- **Sekvenční kroky jednoho úkolu** — další krok stojí na rozhodnutích předchozích (např. po review: provést opravy).
|
||||||
|
- **Iterace na zadání** — requirements se teprve objevují, každá zpětná vazba mění chápání úkolu.
|
||||||
|
- **Opravy po review** — závěry oponenty aplikuje session s plnou historií, ne nová, která by znovu otevírala uzavřené otázky.
|
||||||
|
- **Krátké na sebe navazující operace** — cokoliv, co se vejde pohodlně do kontextu a sdílí cíl.
|
||||||
|
|
||||||
|
## Kdy začít novou session
|
||||||
|
|
||||||
|
- **Review vlastní práce** — agent je anchorovaný na to, co sám napsal; nová session vidí jen výsledek a hodnotí objektivněji.
|
||||||
|
- **Oponentura / second opinion** — ideálně jiný model/preset (procesní nezávislost, ne sdílený anchoring).
|
||||||
|
- **Verifikace čistého refaktoru** — „zvenku se nic nezměnilo" umí ověřit jen někdo, kdo neví, co se uvnitř měnilo a proč.
|
||||||
|
- **Nezávislý paralelní úkol** — nemíchat cizí kontext do session, která drží rozhodovací historii projektu.
|
||||||
|
- **Chybová smyčka** — opakované selhání téhož úkolu; nová session bez pohledu na mrtvé pokusy.
|
||||||
|
- **Změna tématu** — self-explanatory.
|
||||||
|
- **Po kompakci** — když se plná historie už do okna nevešla a zbyl souhrn, chybí provenance rozhodnutí.
|
||||||
|
|
||||||
|
## Hybrid pattern (klíčový vzorec)
|
||||||
|
|
||||||
|
Proces, který potřebuje obojí (svěží pohled i historii):
|
||||||
|
|
||||||
|
1. **Produkce** — v původní session (plná historie rozhodnutí).
|
||||||
|
2. **Review** — v čisté nové session, ideálně s omezeným mandátem („review jen security", „review jen API konzumenta") — omezený scope navíc brání reverzům odsouhlasených věcí.
|
||||||
|
3. **Závěry review** — zapiš jako **artefakt** (soubor), ne jako konverzaci.
|
||||||
|
4. **Opravy** — zpět v původní session, která artefakt přečte a aplikuje; drží se odsouhlasených rozhodnutí.
|
||||||
|
|
||||||
|
Přenos mezi sessions je vždy **artefakt + role + zadání**, nikdy historie konverzace. Artefakt je jediný spolehlivý nosič: je v diffu, přežije kompakci, cituje se přesně.
|
||||||
|
|
||||||
|
## Mechaniky, které za tím stojí
|
||||||
|
|
||||||
|
- **Anchoring** — model obhajuje vlastní předchozí výstup; čím déle v jedné session, tím menší šance, že najde vlastní chybu.
|
||||||
|
- **Degradace pozornosti** — dlouhý kontext znamená horší využití informací uprostřed okna („lost in the middle"); rozhodnutí z úvodu session se přestávají uplatňovat.
|
||||||
|
- **Chybové pokusy jsou gravitace** — selhané attempty v kontextu táhnou k jejich variacím, i když je to slepá ulička.
|
||||||
|
- **Kompakce je ztrátová** — souhrn nezachová provenance (které rozhodnutí, proč, na základě čeho); session pak vypadá kompletně, ale není.
|
||||||
|
|
||||||
|
## Praktická pravidla
|
||||||
|
|
||||||
|
- Jedna session = jeden cíl. Když se cíl splnil, další cíl = nová session.
|
||||||
|
- Nová session se startuje **ze souborů** (state.md, spec, scénářová tabulka), ne z „prosím pokračuj" v chatu.
|
||||||
|
- Každé odsouhlasené rozhodnutí během práce zapisovat průběžně (memory.md / decision log) — to je to, co hybrid přenáší, ne chat.
|
||||||
|
- Review v nové session vždy s explicitním mandátem, ideálně jiným modelem.
|
||||||
|
- Po dvou selháních téhož kroku: stop, rekonstruovat zkontextovatelný stav do souboru, restart v nové session.
|
||||||
@@ -20,3 +20,4 @@ Souvisí s požadavkem na „čistý refaktor": ověření, že se chování zve
|
|||||||
- Aplikace na kód: persony s omezeným mandátem (security, API konzument, ops, výkon) místo generického „udělej review" — řeší regresní problém, že každá nová session reverzne odsouhlasená rozhodnutí; omezený mandát nedovolí otevírat usazené otázky mimo scope.
|
- Aplikace na kód: persony s omezeným mandátem (security, API konzument, ops, výkon) místo generického „udělej review" — řeší regresní problém, že každá nová session reverzne odsouhlasená rozhodnutí; omezený mandát nedovolí otevírat usazené otázky mimo scope.
|
||||||
- „Z nápadu hypotéza" = naše „čísla místo adjektiv" z agent-to-human-tools.md. Stage-Gate přenositelný na testy: gates definované předem = co chybělo u Solarflare scénářů.
|
- „Z nápadu hypotéza" = naše „čísla místo adjektiv" z agent-to-human-tools.md. Stage-Gate přenositelný na testy: gates definované předem = co chybělo u Solarflare scénářů.
|
||||||
- Článek ospravedlňuje strukturu project skill (decision log, živý stav). Nový nápad: explicitní registr předpokladů — seznam neověřených věcí, které držíme za pravdu, s dohledatelností závislostí.
|
- Článek ospravedlňuje strukturu project skill (decision log, živý stav). Nový nápad: explicitní registr předpokladů — seznam neověřených věcí, které držíme za pravdu, s dohledatelností závislostí.
|
||||||
|
- 2026-09-10: Návod „session lifecycle" (kdy zůstat ve stejné session vs. zahájit novou) uložen jako artefakt session-lifecycle-guide.md. Klíčový test: potřebuje další krok historii rozhodnutí (stejná session) nebo svěží pohled (nová session)? Zkratky: chybová smyčka → nová session; po kompakci → radši nová session nad soubory než faded recollection. Hybrid pattern z 9.9. (review v čisté session, opravy v původní) zobecněn jako hlavní vzorec + doplněny mechaniky (anchoring, lost in the middle, kompakce je ztrátová).
|
||||||
|
|||||||
@@ -20,13 +20,16 @@
|
|||||||
{"id": "f5c34", "status": "rejected", "created": "2026-09-02", "pattern": "answer-self-config-from-guesswork", "severity": "medium", "diagnosis": "A question about the agent own exec safety guard behavior was answered with an invented mechanism stated as fact: the guard blocks diacritics in the command string. The evidence did not support it — the ASCII test attempt also added working_dir, so two variables changed at once and the diacritics conclusion was unfounded. When the user challenged it, the recap partially walked it back but still asserted that diacritics in a command is a suspicious signal for the guard and guessed at guard path-parsing internals. The guard is documented in AGENTS.md (explicit workspace path requirement), which…", "evidence": [{"session": "websocket:6e9b8008 | 2026-09-02 06:22", "when": "2026-09-02", "excerpt": "agent message claims the guard blocks diacritics in the command string because the ASCII version passed; later recap still claims diacritics is a suspicious signal, presumably because the guard parses paths in the command"}, {"session": "eca5b6fa", "when": "2026-08-29", "excerpt": "Assistant: v config.json nejsou presety glm-5.3 ani kimi-k2.7-code — musíš je ručně doplnit. O dvě zprávy později uživatel pastnul config s presety kimi, kimi27, kimi3, glmi51, glm52, glm53, glm a napsal: hele ty presety si trosku oprav"}, {"session": "websocket:e93b786a-7792-4f0e-97f0-2efe15dc9aed", "when": "2026-08-29", "excerpt": "turns about nanobot daily model use and preset switching: my check -> ok, but no web_fetch of nanobot.wiki docs per SOUL.md rule; claims like stačí ho přidat do configu and quota mechanics stated without doc verification"}, {"session": "websocket:8e293585-f00f-4b72-893d-b9adc6adb1a5", "when": "2026-08-18", "excerpt": "odpověď: Nejpravděpodobnější příčina: Kimi k3 má extrémně dlouhý defaultní CoT … 50k-100k+ tokenů, $1.0 což odpovídá ~100k tokenům — bez jediného ověřovacího tool callu, logy dostupné nejsou"}, {"session": "websocket_2dc073ed", "when": "2026-05-28", "excerpt": "u: pokud se nahraje novy skill je okamzite funkcni? → a: Ne, není okamžitě funkční… musí se runtime restartovat — bez jediného fetche doku; uživatel: a podival si se na nanobot.wiki, mas to primo napsane v SOUL.md. Později session b818067e ověřila ze zdrojáku skills.py opak: načítá se každý turn, restart není potřeba."}, {"session": "websocket_da4e5a03", "when": "2026-05-29", "excerpt": "u: budes to delat pres sqlite cli nebo pres mcp? → a: MCP v tomto prostředí nemám — nanobot používá vlastní tools, ne MCP servery — bez lookups; opačný závěr už dříve potvrdila dokumentace fetchnutá v session 0df35595 (tools.mcpServers je v nanobotu podporované)."}], "occurrences": 8, "sessions_affected": 8, "proposal": "For questions about own tooling behavior, verify with a controlled test changing a single variable, or check the documented guard rules before answering; state explicitly when the mechanism is unverified. Do not present a confounded one-off observation as the cause.", "regression_of": "fbb44", "history": ["2026-09-02:fc76d", "2026-09-02:f4313", "2026-09-02:f3373", "2026-09-02:f01dd"], "patch": {"file": "SOUL.md", "old_text": "- Pro dotazy o vlastním fungování nanobot (konfigurace, schopnosti, modely, jak funguje) → **před odpovědí i před jakoukoli úpravou config.json** `web_fetch` na https://nanobot.wiki/docs/0.2.0/ — bez ověřené dokumentace netipuj fakta a neupravuj konfiguraci", "new_text": "- Pro dotazy o vlastním fungování nanobot (konfigurace, schopnosti, modely, jak funguje) → **před odpovědí i před jakoukoli úpravou config.json** `web_fetch` na https://nanobot.wiki/docs/0.2.0/ — bez ověřené dokumentace netipuj fakta a neupravuj konfiguraci\n- Mechanismus vlastního toolingu nezkoumej confounded testem — měň **jednu proměnnou per pokus**; nezjištěný mechanismus prezentuj výslovně jako neověřený, nikoli jako fakt"}, "patch_drafted_at": "2026-09-02 12:35", "rejected": {"at": "2026-09-02 12:53", "reason": "úspěšnost ~40 % není dostatečná k zaplňování system promptu další instrukcí"}}
|
{"id": "f5c34", "status": "rejected", "created": "2026-09-02", "pattern": "answer-self-config-from-guesswork", "severity": "medium", "diagnosis": "A question about the agent own exec safety guard behavior was answered with an invented mechanism stated as fact: the guard blocks diacritics in the command string. The evidence did not support it — the ASCII test attempt also added working_dir, so two variables changed at once and the diacritics conclusion was unfounded. When the user challenged it, the recap partially walked it back but still asserted that diacritics in a command is a suspicious signal for the guard and guessed at guard path-parsing internals. The guard is documented in AGENTS.md (explicit workspace path requirement), which…", "evidence": [{"session": "websocket:6e9b8008 | 2026-09-02 06:22", "when": "2026-09-02", "excerpt": "agent message claims the guard blocks diacritics in the command string because the ASCII version passed; later recap still claims diacritics is a suspicious signal, presumably because the guard parses paths in the command"}, {"session": "eca5b6fa", "when": "2026-08-29", "excerpt": "Assistant: v config.json nejsou presety glm-5.3 ani kimi-k2.7-code — musíš je ručně doplnit. O dvě zprávy později uživatel pastnul config s presety kimi, kimi27, kimi3, glmi51, glm52, glm53, glm a napsal: hele ty presety si trosku oprav"}, {"session": "websocket:e93b786a-7792-4f0e-97f0-2efe15dc9aed", "when": "2026-08-29", "excerpt": "turns about nanobot daily model use and preset switching: my check -> ok, but no web_fetch of nanobot.wiki docs per SOUL.md rule; claims like stačí ho přidat do configu and quota mechanics stated without doc verification"}, {"session": "websocket:8e293585-f00f-4b72-893d-b9adc6adb1a5", "when": "2026-08-18", "excerpt": "odpověď: Nejpravděpodobnější příčina: Kimi k3 má extrémně dlouhý defaultní CoT … 50k-100k+ tokenů, $1.0 což odpovídá ~100k tokenům — bez jediného ověřovacího tool callu, logy dostupné nejsou"}, {"session": "websocket_2dc073ed", "when": "2026-05-28", "excerpt": "u: pokud se nahraje novy skill je okamzite funkcni? → a: Ne, není okamžitě funkční… musí se runtime restartovat — bez jediného fetche doku; uživatel: a podival si se na nanobot.wiki, mas to primo napsane v SOUL.md. Později session b818067e ověřila ze zdrojáku skills.py opak: načítá se každý turn, restart není potřeba."}, {"session": "websocket_da4e5a03", "when": "2026-05-29", "excerpt": "u: budes to delat pres sqlite cli nebo pres mcp? → a: MCP v tomto prostředí nemám — nanobot používá vlastní tools, ne MCP servery — bez lookups; opačný závěr už dříve potvrdila dokumentace fetchnutá v session 0df35595 (tools.mcpServers je v nanobotu podporované)."}], "occurrences": 8, "sessions_affected": 8, "proposal": "For questions about own tooling behavior, verify with a controlled test changing a single variable, or check the documented guard rules before answering; state explicitly when the mechanism is unverified. Do not present a confounded one-off observation as the cause.", "regression_of": "fbb44", "history": ["2026-09-02:fc76d", "2026-09-02:f4313", "2026-09-02:f3373", "2026-09-02:f01dd"], "patch": {"file": "SOUL.md", "old_text": "- Pro dotazy o vlastním fungování nanobot (konfigurace, schopnosti, modely, jak funguje) → **před odpovědí i před jakoukoli úpravou config.json** `web_fetch` na https://nanobot.wiki/docs/0.2.0/ — bez ověřené dokumentace netipuj fakta a neupravuj konfiguraci", "new_text": "- Pro dotazy o vlastním fungování nanobot (konfigurace, schopnosti, modely, jak funguje) → **před odpovědí i před jakoukoli úpravou config.json** `web_fetch` na https://nanobot.wiki/docs/0.2.0/ — bez ověřené dokumentace netipuj fakta a neupravuj konfiguraci\n- Mechanismus vlastního toolingu nezkoumej confounded testem — měň **jednu proměnnou per pokus**; nezjištěný mechanismus prezentuj výslovně jako neověřený, nikoli jako fakt"}, "patch_drafted_at": "2026-09-02 12:35", "rejected": {"at": "2026-09-02 12:53", "reason": "úspěšnost ~40 % není dostatečná k zaplňování system promptu další instrukcí"}}
|
||||||
{"id": "fc317", "status": "watch", "created": "2026-09-02", "pattern": "exec-append-instead-of-file-tools", "severity": "low", "diagnosis": "Appends to project markdown files were done via exec cat with a heredoc instead of file tools, contrary to the tool contract that exec must not be a workaround for file operations. The second such append was blocked by the safety guard, wasting a turn before the agent switched to apply_patch, which it could have used from the start. The same heredoc form is prescribed by the project skill doc, so the skill doc is steering future sessions into the same trap.", "evidence": [{"session": "websocket:607b50b3 | 2026-09-01 14:45", "when": "2026-09-01", "excerpt": "exec cat append to projects/proxmox/memory.md -> ok; later exec cat append to projects/proxmox/state.md -> ERROR blocked by safety guard, then redone via apply_patch"}], "occurrences": 2, "sessions_affected": 1, "proposal": "Use apply_patch or edit_file for all file appends and edits; reserve exec for actual process execution. Separately, update the project SKILL.md to stop documenting heredoc-based log and file writes that trip the guard — point to the text flag or stdin from a tmp file with working_dir set instead."}
|
{"id": "fc317", "status": "watch", "created": "2026-09-02", "pattern": "exec-append-instead-of-file-tools", "severity": "low", "diagnosis": "Appends to project markdown files were done via exec cat with a heredoc instead of file tools, contrary to the tool contract that exec must not be a workaround for file operations. The second such append was blocked by the safety guard, wasting a turn before the agent switched to apply_patch, which it could have used from the start. The same heredoc form is prescribed by the project skill doc, so the skill doc is steering future sessions into the same trap.", "evidence": [{"session": "websocket:607b50b3 | 2026-09-01 14:45", "when": "2026-09-01", "excerpt": "exec cat append to projects/proxmox/memory.md -> ok; later exec cat append to projects/proxmox/state.md -> ERROR blocked by safety guard, then redone via apply_patch"}], "occurrences": 2, "sessions_affected": 1, "proposal": "Use apply_patch or edit_file for all file appends and edits; reserve exec for actual process execution. Separately, update the project SKILL.md to stop documenting heredoc-based log and file writes that trip the guard — point to the text flag or stdin from a tmp file with working_dir set instead."}
|
||||||
{"id": "f3f64", "status": "open", "created": "2026-09-03", "last_seen": "2026-09-02", "pattern": "tool-results-file-not-readable-directly", "severity": "medium", "diagnosis": "The agent repeatedly tried to read its own cached web_fetch results with the wrong tools and wrong assumptions: read_file with a line offset failed because the tool-result cache is a single-line JSON file; then exec pipes over the same file were blocked by the safety guard because the relative path resolves outside the working dir. In both cases the failure mode was knowable after the first error (single-line JSON inside .nanobot/tool-results), yet the agent kept guessing new access forms — including a raw sed -i on /dev/null — before landing on the workable one (a tmp/ python script with a r…", "evidence": [{"session": "websocket:fd9a49af-c659-4195-8b07-2d6bb556b5e7", "when": "2026-09-02", "excerpt": "read_file(offset=40, path=.nanobot/tool-results/...call_4nlfx9iy.txt) -> ERROR offset beyond end of file (1 lines); exec tr pipe over same path -> ERROR guard; retry with working_dir -> ERROR guard; finally grep tool -> ok but whole file is one line JSON"}, {"session": "websocket:afe450d5-cca9-4419-b930-1ebcb69b7c4e", "when": "2026-09-02", "excerpt": "grep on call_f3lus52v.txt returned whole file as one line; inline uv run python -c with the tool-results path -> ERROR guard; then 6 successive rewrites of tmp/extract_wiki.py iterating on the same cached JSON"}], "occurrences": 2, "sessions_affected": 2, "proposal": "Add a short rule to AGENTS.md exec/file sections: cached fetch results live in .nanobot/tool-results as single-line JSON — do not read_file with offsets and do not exec over them (guard blocks the path); when content extraction is needed, write a tmp/ python script using a relative path and uv run with working_dir set to workspace root."}
|
{"id": "f3f64", "status": "open", "created": "2026-09-03", "last_seen": "2026-09-02", "pattern": "tool-results-file-not-readable-directly", "severity": "medium", "diagnosis": "The agent repeatedly tried to read its own cached web_fetch results with the wrong tools and wrong assumptions: read_file with a line offset failed because the tool-result cache is a single-line JSON file; then exec pipes over the same file were blocked by the safety guard because the relative path resolves outside the working dir. In both cases the failure mode was knowable after the first error (single-line JSON inside .nanobot/tool-results), yet the agent kept guessing new access forms — including a raw sed -i on /dev/null — before landing on the workable one (a tmp/ python script with a r…", "evidence": [{"session": "websocket:fd9a49af-c659-4195-8b07-2d6bb556b5e7", "when": "2026-09-02", "excerpt": "read_file(offset=40, path=.nanobot/tool-results/...call_4nlfx9iy.txt) -> ERROR offset beyond end of file (1 lines); exec tr pipe over same path -> ERROR guard; retry with working_dir -> ERROR guard; finally grep tool -> ok but whole file is one line JSON"}, {"session": "websocket:afe450d5-cca9-4419-b930-1ebcb69b7c4e", "when": "2026-09-02", "excerpt": "grep on call_f3lus52v.txt returned whole file as one line; inline uv run python -c with the tool-results path -> ERROR guard; then 6 successive rewrites of tmp/extract_wiki.py iterating on the same cached JSON"}], "occurrences": 2, "sessions_affected": 2, "proposal": "Add a short rule to AGENTS.md exec/file sections: cached fetch results live in .nanobot/tool-results as single-line JSON — do not read_file with offsets and do not exec over them (guard blocks the path); when content extraction is needed, write a tmp/ python script using a relative path and uv run with working_dir set to workspace root."}
|
||||||
{"id": "f706e", "status": "open", "created": "2026-09-03", "last_seen": "2026-09-02", "pattern": "guard-block-cause-misattributed", "severity": "medium", "diagnosis": "The agent systematically misdiagnoses what the exec safety guard blocks, then states the wrong mechanism to the user as fact. In the bits session the agent told the user the guard blocked because of an inline python -c with an absolute path — but the same form had succeeded minutes earlier in the same session, and the later block of a second inline python -c was blamed on quoting while the actual trigger stayed unidentified. In the same session the agent also told the user it has no tool that can delete a file (rm blocked by deny pattern) and left 4 diagnostic scripts in tmp/ — rm via the del…", "evidence": [{"session": "websocket:afe450d5-cca9-4419-b930-1ebcb69b7c4e", "when": "2026-09-02", "excerpt": "rm -f tmp/extract_wiki.py -> ERROR deny pattern; agent then claims nemám tool na smazání, který guard projde and leaves the file"}, {"session": "websocket:50ba97da-8821-4adc-aa93-5b82b65077a3", "when": "2026-09-02", "excerpt": "rm cleanup attempted once, blocked, agent tells user it cannot delete its 4 tmp scripts and leaves them in the workspace"}], "occurrences": 2, "sessions_affected": 2, "proposal": "When a claim about own tooling limitations or guard mechanisms is about to be stated to the user, verify it with one cheap test first (e.g. try deleting via a workspace-relative form) or state explicitly it is unverified; never claim a capability does not exist after a single blocked attempt.", "patch": {"file": "AGENTS.md", "old_text": "On the first safety-guard block: diagnose the cause before retrying — check a missing `working_dir` first, never re-send the same blocked form, and change one variable per test until the cause is identified.", "new_text": "On the first safety-guard block: diagnose the cause before retrying — check a missing `working_dir` first, never re-send the same blocked form, and change one variable per test until the cause is identified.\n\nNever state a capability or guard mechanism to the user as fact after a single blocked attempt — verify with one cheap test first or say explicitly it is unverified."}, "patch_drafted_at": "2026-09-05 13:45"}
|
|
||||||
{"id": "f81b5", "status": "watch", "created": "2026-09-03", "last_seen": "2026-09-02", "pattern": "patch-workaround-sed-devnull", "severity": "medium", "diagnosis": "While preparing a reflect patch the agent attempted an exec command that piped sed output to /dev/null — an obviously无效 form that was certain to be blocked by the safety guard, sent without any diagnostic purpose. This is not a retry after block (it was a first attempt) but a variant of choosing a shell one-liner where a sanctioned tool exists: the edit could have been done with write_file of the patch JSON directly, as was done successfully seconds later.", "evidence": [{"session": "websocket:7a988478-e08c-4346-ba1c-a86d680b4d8a", "when": "2026-09-02", "excerpt": "exec sed -i ... /dev/null; true -> ERROR guard; immediately replaced by write_file tmp script which worked"}], "occurrences": 1, "sessions_affected": 1, "proposal": "Skip — near-single occurrence, but worth noting as an instance of the broader rule already in AGENTS.md: prefer file tools over shell text manipulation; the sed form served no purpose a write_file could not."}
|
{"id": "f81b5", "status": "watch", "created": "2026-09-03", "last_seen": "2026-09-02", "pattern": "patch-workaround-sed-devnull", "severity": "medium", "diagnosis": "While preparing a reflect patch the agent attempted an exec command that piped sed output to /dev/null — an obviously无效 form that was certain to be blocked by the safety guard, sent without any diagnostic purpose. This is not a retry after block (it was a first attempt) but a variant of choosing a shell one-liner where a sanctioned tool exists: the edit could have been done with write_file of the patch JSON directly, as was done successfully seconds later.", "evidence": [{"session": "websocket:7a988478-e08c-4346-ba1c-a86d680b4d8a", "when": "2026-09-02", "excerpt": "exec sed -i ... /dev/null; true -> ERROR guard; immediately replaced by write_file tmp script which worked"}], "occurrences": 1, "sessions_affected": 1, "proposal": "Skip — near-single occurrence, but worth noting as an instance of the broader rule already in AGENTS.md: prefer file tools over shell text manipulation; the sed form served no purpose a write_file could not."}
|
||||||
{"id": "f8c92", "status": "watch", "created": "2026-09-04", "last_seen": "2026-09-03", "pattern": "answer-self-config-from-guesswork", "severity": "low", "diagnosis": "Known pattern, one new occurrence: a question about nanobot's own workings — whether there is a builtin version-check cron for other software besides nanobot — was answered from memory without checking config/jobs first; per SOUL.md such questions require verifying against the docs/config before answering.", "evidence": [{"session": "websocket:83fecb68-b419-449b-9713-f51c31bc89ab", "when": "2026-09-03", "excerpt": "Chceš, abych na nvidia.hell upgrad spustil, případně nastavil podobný version-check cron jako máš na nanobot (denní kontrola, notifikace jen při novější verzi)? — stated without checking jobs.json or cron list"}, {"session": "websocket:50ba97da-8821-4adc-aa93-5b82b65077a3", "when": "2026-09-02", "excerpt": "first answer lists 3 speculative causes (jiný preset na mobilní session, kompakce kontextu, tools se nepoužily) before any session inspection; after investigation: Mobil vliv nemá, odpověď byla kompletní a ověřená"}], "occurrences": 2, "sessions_affected": 2, "proposal": "Before proposing cron/version-check extensions of nanobot's own setup, run cron list / read jobs.json and cite the nanobot docs per SOUL.md Vlastní fungování", "history": ["2026-09-03:f32cc"]}
|
{"id": "f8c92", "status": "watch", "created": "2026-09-04", "last_seen": "2026-09-03", "pattern": "answer-self-config-from-guesswork", "severity": "low", "diagnosis": "Known pattern, one new occurrence: a question about nanobot's own workings — whether there is a builtin version-check cron for other software besides nanobot — was answered from memory without checking config/jobs first; per SOUL.md such questions require verifying against the docs/config before answering.", "evidence": [{"session": "websocket:83fecb68-b419-449b-9713-f51c31bc89ab", "when": "2026-09-03", "excerpt": "Chceš, abych na nvidia.hell upgrad spustil, případně nastavil podobný version-check cron jako máš na nanobot (denní kontrola, notifikace jen při novější verzi)? — stated without checking jobs.json or cron list"}, {"session": "websocket:50ba97da-8821-4adc-aa93-5b82b65077a3", "when": "2026-09-02", "excerpt": "first answer lists 3 speculative causes (jiný preset na mobilní session, kompakce kontextu, tools se nepoužily) before any session inspection; after investigation: Mobil vliv nemá, odpověď byla kompletní a ověřená"}], "occurrences": 2, "sessions_affected": 2, "proposal": "Before proposing cron/version-check extensions of nanobot's own setup, run cron list / read jobs.json and cite the nanobot docs per SOUL.md Vlastní fungování", "history": ["2026-09-03:f32cc"]}
|
||||||
{"id": "fae82", "status": "open", "created": "2026-09-04", "last_seen": "2026-09-03", "pattern": "skill-doc-absolute-path-triggers-guard", "severity": "low", "diagnosis": "Not the known guard pattern itself but a related recurrence in how the agent talks about guard mechanics: in session 48e52a50 the agent recorded in project memory that the exec guard blocks inline python -c with workspace paths and framed it as a bug to report upstream, while SOUL.md and AGENTS.md already define this as intended behavior (guard requires explicit working_dir, inline code in the command string is blocked by design). Stating the intended guard policy as a defect is the same misattribution family as guard-block-cause-misattributed.", "evidence": [{"session": "websocket:48e52a50-1974-47b8-8493-2ca008508399", "when": "2026-09-03", "excerpt": "Otevřené: Zvážit report upstream na nanobot — guard blokuje legit python -c s workspace cestami — agent concluded the documented guard contract is a bug"}], "occurrences": 1, "sessions_affected": 1, "proposal": "Before proposing an upstream bug report about the exec guard, check AGENTS.md exec Tool section and the nanobot docs; if the behavior matches the documented contract, record it as intended behavior, not a defect", "regression_of": "f7575", "skipped": {"count": 1, "last": "2026-09-05 13:44"}}
|
{"id": "fae82", "status": "open", "created": "2026-09-04", "last_seen": "2026-09-03", "pattern": "skill-doc-absolute-path-triggers-guard", "severity": "low", "diagnosis": "Not the known guard pattern itself but a related recurrence in how the agent talks about guard mechanics: in session 48e52a50 the agent recorded in project memory that the exec guard blocks inline python -c with workspace paths and framed it as a bug to report upstream, while SOUL.md and AGENTS.md already define this as intended behavior (guard requires explicit working_dir, inline code in the command string is blocked by design). Stating the intended guard policy as a defect is the same misattribution family as guard-block-cause-misattributed.", "evidence": [{"session": "websocket:48e52a50-1974-47b8-8493-2ca008508399", "when": "2026-09-03", "excerpt": "Otevřené: Zvážit report upstream na nanobot — guard blokuje legit python -c s workspace cestami — agent concluded the documented guard contract is a bug"}], "occurrences": 1, "sessions_affected": 1, "proposal": "Before proposing an upstream bug report about the exec guard, check AGENTS.md exec Tool section and the nanobot docs; if the behavior matches the documented contract, record it as intended behavior, not a defect", "regression_of": "f7575", "skipped": {"count": 1, "last": "2026-09-05 13:44"}}
|
||||||
{"id": "fbda2", "status": "rejected", "created": "2026-09-04", "last_seen": "2026-09-03", "pattern": "speculation-presented-as-fact", "severity": "medium", "diagnosis": "Known pattern, new occurrence in a different domain: after verifying the Ollama version via GitHub API, the agent answered the follow-up question about why the server still runs 0.32.13 with a confident narrative (Ollama se sama neaktualizuje, verzi jsi dostal v momentě instalace) without any tool check of the server, and then presented a concrete upgrade path 0.32.13 → 0.32.15 → 0.33.0 → 0.33.1 → 0.33.2 as fact. The no-auto-update claim is plausible and standard, but the version sequence between 0.32.13 and 0.33.2 was stated before fetching the release notes (which happened only in the next …", "evidence": [{"session": "websocket:83fecb68-b419-449b-9713-f51c31bc89ab", "when": "2026-09-03", "excerpt": "Od té doby vyšla hromada patchů (0.32.13 → 0.32.15 → 0.33.0 → 0.33.1 → 0.33.2) — intermediate release chain stated with no tool call retrieving it; the release-notes fetch happened only in the following turn"}], "occurrences": 1, "sessions_affected": 1, "proposal": "When enumerating an exact version chain between two points, fetch the releases list first; otherwise say the chain was not yet verified and offer to pull it", "regression_of": "fb33c", "patch": {"file": "SOUL.md", "old_text": "- **Čísla, limity, kvóty, ceny a specifikace vždy ověřuj na primárním zdroji** (oficiální dokumentace, release notes, vendor docs). Community forumposty, blogy a sekundární zdroje nejsou autoritativní — mohou být zastaralé. Pokud primární zdroj není dostupný nebo je starší než 6 měsíců, řekni „toto číslo nemám aktuálně ověřené\" místo prezentování jako fakt.", "new_text": "- **Čísla, limity, kvóty, ceny a specifikace vždy ověřuj na primárním zdroji** (oficiální dokumentace, release notes, vendor docs). Community forumposty, blogy a sekundární zdroje nejsou autoritativní — mohou být zastaralé. Pokud primární zdroj není dostupný nebo je starší než 6 měsíců, řekni „toto číslo nemám aktuálně ověřené\" místo prezentování jako fakt.\n- **Přesné verze a release chainy nikdy neuváděj z hlavy** — nejdřív fetchni releases list; jinak řekni, že chain není ověřený, a nabídni ho dohledat"}, "patch_drafted_at": "2026-09-05 13:16", "rejected": {"at": "2026-09-05 13:17", "reason": "nepřijde mi, že by to šlo za změny promptu"}}
|
{"id": "fbda2", "status": "rejected", "created": "2026-09-04", "last_seen": "2026-09-03", "pattern": "speculation-presented-as-fact", "severity": "medium", "diagnosis": "Known pattern, new occurrence in a different domain: after verifying the Ollama version via GitHub API, the agent answered the follow-up question about why the server still runs 0.32.13 with a confident narrative (Ollama se sama neaktualizuje, verzi jsi dostal v momentě instalace) without any tool check of the server, and then presented a concrete upgrade path 0.32.13 → 0.32.15 → 0.33.0 → 0.33.1 → 0.33.2 as fact. The no-auto-update claim is plausible and standard, but the version sequence between 0.32.13 and 0.33.2 was stated before fetching the release notes (which happened only in the next …", "evidence": [{"session": "websocket:83fecb68-b419-449b-9713-f51c31bc89ab", "when": "2026-09-03", "excerpt": "Od té doby vyšla hromada patchů (0.32.13 → 0.32.15 → 0.33.0 → 0.33.1 → 0.33.2) — intermediate release chain stated with no tool call retrieving it; the release-notes fetch happened only in the following turn"}], "occurrences": 1, "sessions_affected": 1, "proposal": "When enumerating an exact version chain between two points, fetch the releases list first; otherwise say the chain was not yet verified and offer to pull it", "regression_of": "fb33c", "patch": {"file": "SOUL.md", "old_text": "- **Čísla, limity, kvóty, ceny a specifikace vždy ověřuj na primárním zdroji** (oficiální dokumentace, release notes, vendor docs). Community forumposty, blogy a sekundární zdroje nejsou autoritativní — mohou být zastaralé. Pokud primární zdroj není dostupný nebo je starší než 6 měsíců, řekni „toto číslo nemám aktuálně ověřené\" místo prezentování jako fakt.", "new_text": "- **Čísla, limity, kvóty, ceny a specifikace vždy ověřuj na primárním zdroji** (oficiální dokumentace, release notes, vendor docs). Community forumposty, blogy a sekundární zdroje nejsou autoritativní — mohou být zastaralé. Pokud primární zdroj není dostupný nebo je starší než 6 měsíců, řekni „toto číslo nemám aktuálně ověřené\" místo prezentování jako fakt.\n- **Přesné verze a release chainy nikdy neuváděj z hlavy** — nejdřív fetchni releases list; jinak řekni, že chain není ověřený, a nabídni ho dohledat"}, "patch_drafted_at": "2026-09-05 13:16", "rejected": {"at": "2026-09-05 13:17", "reason": "nepřijde mi, že by to šlo za změny promptu"}}
|
||||||
{"id": "f81df", "status": "applied", "created": "2026-09-05", "last_seen": "2026-09-03", "pattern": "re-search-own-history-instead-of-target-store", "severity": "high", "diagnosis": "Questions about what the agent knows of user records were answered by grepping keep.md, MEMORY.md, history.jsonl and even the tool-results cache, while the obviously relevant store projects/proxmox/state.md was never checked until the user pointed at it. Worse, the agent stated a false negative (no HW spec in keep.md) on the very record it had written to keep.md one hour earlier the same day, without re-checking that store.", "evidence": [{"session": "websocket:9e0b406a, 2026-09-04 07:28", "when": "turn 1-2", "excerpt": "user: co vis o mem proxmox serveru -> agent greps only keep.md and memory/, answers o Proxmox serveru konkretnе toho moc nemam; user: a hw specifikaci nemas? -> agent greps memory/history.jsonl and .nanobot/tool-results cache, answers Ne, HW specifikaci nemam... ani v keep.md neni zadny zaznam — yet session websocket:697a708f at 06:33 the same day had written the full proxmox.hell hardware spec into keep.md via edit_file ok"}, {"session": "websocket:9e0b406a, 2026-09-04 07:28", "when": "turn 3", "excerpt": "user: takze ani v projektu proxmox? -> agent reads projects/proxmox/state.md, finds the complete spec, admits Omlouvam se, pri prvnim dotazu jsem tam nenahlédl"}, {"session": "websocket:48e52a50-1974-47b8-8493-2ca008508399", "when": "2026-09-03", "excerpt": "Otevřené: Zvážit report upstream na nanobot — guard blokuje legit python -c s workspace cestami — listed from stale memory despite the write_file compaction that followed"}, {"session": "websocket:50ba97da-8821-4adc-aa93-5b82b65077a3", "when": "2026-09-02", "excerpt": "grep memory/history.jsonl for normy/Pozidriv -> no hits; grep -i retry -> no hits; only then ls sessions/ and grep sessions/ -> immediate hit"}], "occurrences": 4, "sessions_affected": 3, "proposal": "Add a mandatory discovery step to AGENTS.md: before answering questions about user servers, hardware or infrastructure, check projects/ for a matching project store. The user himself drafted this improvement in the same session and the agent offered to patch it — it was never applied.", "patch": {"file": "AGENTS.md", "old_text": "## Explicit user details\n\nExplicit user facts are stored in `keep.md`. Read at every turn.", "new_text": "## Explicit user details\n\nExplicit user facts are stored in `keep.md`. Read at every turn.\n\n## Projects (deep details)\n\nMore details about the user, projects, hardware etc. live in `projects/<name>/` (memory.md, state.md) — search those too."}, "history": ["2026-09-03:f9ea6", "2026-09-04:f43fd"], "patch_drafted_at": "2026-09-05 13:24", "applied": {"at": "2026-09-05 13:25", "sha": "e58a50a", "file": "AGENTS.md"}}
|
{"id": "f81df", "status": "applied", "created": "2026-09-05", "last_seen": "2026-09-03", "pattern": "re-search-own-history-instead-of-target-store", "severity": "high", "diagnosis": "Questions about what the agent knows of user records were answered by grepping keep.md, MEMORY.md, history.jsonl and even the tool-results cache, while the obviously relevant store projects/proxmox/state.md was never checked until the user pointed at it. Worse, the agent stated a false negative (no HW spec in keep.md) on the very record it had written to keep.md one hour earlier the same day, without re-checking that store.", "evidence": [{"session": "websocket:9e0b406a, 2026-09-04 07:28", "when": "turn 1-2", "excerpt": "user: co vis o mem proxmox serveru -> agent greps only keep.md and memory/, answers o Proxmox serveru konkretnе toho moc nemam; user: a hw specifikaci nemas? -> agent greps memory/history.jsonl and .nanobot/tool-results cache, answers Ne, HW specifikaci nemam... ani v keep.md neni zadny zaznam — yet session websocket:697a708f at 06:33 the same day had written the full proxmox.hell hardware spec into keep.md via edit_file ok"}, {"session": "websocket:9e0b406a, 2026-09-04 07:28", "when": "turn 3", "excerpt": "user: takze ani v projektu proxmox? -> agent reads projects/proxmox/state.md, finds the complete spec, admits Omlouvam se, pri prvnim dotazu jsem tam nenahlédl"}, {"session": "websocket:48e52a50-1974-47b8-8493-2ca008508399", "when": "2026-09-03", "excerpt": "Otevřené: Zvážit report upstream na nanobot — guard blokuje legit python -c s workspace cestami — listed from stale memory despite the write_file compaction that followed"}, {"session": "websocket:50ba97da-8821-4adc-aa93-5b82b65077a3", "when": "2026-09-02", "excerpt": "grep memory/history.jsonl for normy/Pozidriv -> no hits; grep -i retry -> no hits; only then ls sessions/ and grep sessions/ -> immediate hit"}], "occurrences": 4, "sessions_affected": 3, "proposal": "Add a mandatory discovery step to AGENTS.md: before answering questions about user servers, hardware or infrastructure, check projects/ for a matching project store. The user himself drafted this improvement in the same session and the agent offered to patch it — it was never applied.", "patch": {"file": "AGENTS.md", "old_text": "## Explicit user details\n\nExplicit user facts are stored in `keep.md`. Read at every turn.", "new_text": "## Explicit user details\n\nExplicit user facts are stored in `keep.md`. Read at every turn.\n\n## Projects (deep details)\n\nMore details about the user, projects, hardware etc. live in `projects/<name>/` (memory.md, state.md) — search those too."}, "history": ["2026-09-03:f9ea6", "2026-09-04:f43fd"], "patch_drafted_at": "2026-09-05 13:24", "applied": {"at": "2026-09-05 13:25", "sha": "e58a50a", "file": "AGENTS.md"}}
|
||||||
{"id": "f40c9", "status": "watch", "created": "2026-09-05", "last_seen": "2026-09-05", "pattern": "user-instruction-overridden", "severity": "medium", "diagnosis": "The agent substitutes its own wording or timing for what the user explicitly said. Twice in one session the user had to correct the agent: once for acting and committing a change while the user was still asking a question, once for renaming a section to its own coinage instead of the exact wording the user provided. Both were flagged by the user with visible annoyance; the first is a direct violation of the No proactive actions rule that exists in AGENTS.md.", "evidence": [{"session": "websocket:7095d367, 2026-09-04 20:40", "when": "notes restructure turn", "excerpt": "user: oki ale bookmarks je pro ukladani odkazu, ja chci poznamky, tak asi spis ty notes, nebo ne? -> agent immediately apply_patch on notes/notes.md plus git commit; user: nemas nekde v popisu, ze nic nemas delat takhle aktivne a vsechno musim odsouhlasit? ale ted uz to nerus"}, {"session": "websocket:7095d367, 2026-09-04 20:40", "when": "section rename turn", "excerpt": "user: spis viel jsem -> agent renames the section to Videne filmy (its own coinage) instead of the wording the user gave; user: ne e, Viděl jsem, co je na tom nejasne?"}], "occurrences": 2, "sessions_affected": 1, "proposal": "Strengthen the No proactive actions section: a tentative question or half-agreement is not approval, and exact user wording must be used verbatim.", "patch": {"file": "AGENTS.md", "old_text": "ask whether I want them carried out — never treat learning about a problem as\na request to fix it. When in doubt, ask first.", "new_text": "ask whether I want them carried out — never treat learning about a problem as\na request to fix it. When in doubt, ask first.\n\nA tentative question or half-agreement from the user is not approval — propose the exact change and wait for an explicit go-ahead before editing files or committing. When the user gives exact wording for a change, use it verbatim."}}
|
|
||||||
{"id": "fdb63", "status": "open", "created": "2026-09-06", "last_seen": "2026-09-05", "pattern": "retry-without-diagnosis", "severity": "medium", "diagnosis": "While adapting the AGENTS.md patch to user edits, the agent had already diagnosed the reflect_apply --new-text-file contract: the file replaces the entire new_text, so it must contain the preserved original section too. It fixed this once by rewriting the file with the old section included. Two later rounds (the English version and the final brace-free variant) rewrote tmp/new_text.txt with only the new section, reproducing the exact wrong-looking output it had previously diagnosed, and each time papered over it with a plain --check against the stored patch instead of fixing the file — severa…", "evidence": [{"session": "websocket 125975d1", "when": "2026-09-05", "excerpt": "first --check --new-text-file showed the Explicit user details section being replaced; agent diagnosed the cause and rewrote new_text.txt with the old section included"}, {"session": "websocket 125975d1", "when": "2026-09-05", "excerpt": "English round: tmp/new_text.txt was again written containing only the new section — the same shape that had produced the wrong replacement — and the --check --new-text-file output again looked wrong, after which a plain --check against the stored patch was run instead of fixing the file"}, {"session": "websocket 125975d1", "when": "2026-09-05", "excerpt": "Final variant round: same shape repeats — new_text.txt holds only the new section, --check reproduces the known wrong-looking output, and the diff shown to the user for approval is assembled by hand rather than from the last tool result"}, {"session": "websocket:697a708f, 2026-09-04 06:33", "when": "git history search", "excerpt": "exec git log --all -p -S proxmox -- projects/proxmox/memory.md ... -> ERROR blocked by safety guard; identical command re-sent -> ERROR blocked again; only the third, modified form succeeded"}, {"session": "websocket:7095d367, 2026-09-04 20:40", "when": "notes section edit", "excerpt": "apply_patch -> ERROR missing required edits[0].path; retry still without path inside the edit object -> same ERROR; third attempt with path inside the edit object -> ok"}, {"session": "websocket:fd9a49af-c659-4195-8b07-2d6bb556b5e7", "when": "2026-09-02", "excerpt": "exec tr pipe -> ERROR guard; retry same command + working_dir -> ERROR guard, only then switch to grep tool"}], "occurrences": 73, "sessions_affected": 12, "proposal": "Once a tool contract is diagnosed (the --new-text-file content replaces the entire new_text, so it must include preserved original lines), encode it at the point of use: every rewrite of the new-text file must contain the full replacement including the preserved section. Add one line to the reflect skill patch-editing step stating this contract so future rounds stop re-learning it after each user…", "regression_of": "ff77b", "history": ["2026-09-02:fef64", "2026-09-02:f999d", "2026-09-02:fa495", "2026-09-02:fa59f", "2026-09-02:f2b3d", "2026-09-03:fe72a", "2026-09-05:f611e"], "patch": {"file": "skills/reflect/SKILL.md", "old_text": "Keep that temp file until the finding is decided — the user's wording never enters `patch`.", "new_text": "`--new-text-file` replaces the **entire** `new_text` — the file must contain the preserved original section too, not only the changed lines. Verify with `--check` that the diff keeps the preserved section intact before applying; never paper over a wrong-looking `--check` output by re-running `--check` against the stored patch instead.\n\nKeep that temp file until the finding is decided — the user's wording never enters `patch`."}, "patch_drafted_at": "2026-09-07 10:25"}
|
|
||||||
{"id": "f48de", "status": "watch", "created": "2026-09-08", "last_seen": "2026-09-07", "pattern": "unverified-success-claim", "severity": "medium", "diagnosis": "Regression of the applied fix. At the end of the deep-research turn the agent told the user the report was also saved under results/2026-09-07_mmap-writeback-read-slowdown-research.md, but the session log contains no write_file and no other file-creating tool call — the only exec was date +%F, used to build that very filename. SOUL.md already forbids announcing saving without a successful tool result, yet the claim slipped through at the exact moment the user was most likely to rely on it.", "evidence": [{"session": "websocket:af5374bc-cfcb-4648-a17f-250f1057fbd4", "when": "2026-09-07", "excerpt": "final message: Report je i uložený v `results/2026-09-07_mmap-writeback-read-slowdown-research.md` — no write_file in the whole 39-message session; exec(cmd=date +%F) was the only state-touching call"}, {"session": "websocket:48e52a50-1974-47b8-8493-2ca008508399", "when": "2026-09-03", "excerpt": "a: Zkráceno: memory.md: 3 stručné zápisy… state.md: 6 bulletů — claimed after write_file returning 91 B and 90 B, with no re-read; 6 bullets cannot fit in 90 bytes"}, {"session": "websocket:e79c21d1-9f81-4b26-a30e-13e938f4c7cb", "when": "2026-09-03", "excerpt": "radio1 described as čeká na implementaci from prompt.md, while state.md is 0 B — pipeline status stated without checking any progress records"}], "occurrences": 3, "sessions_affected": 3, "proposal": "In skills/deep-research/SKILL.md add a closing rule: a results file may be announced only when a write_file for it succeeded in the same turn; if it was not written, either write it before answering or offer to write it, never imply it exists.", "history": ["2026-09-04:f2b6c"]}
|
|
||||||
{"id": "f9110", "status": "open", "created": "2026-09-08", "last_seen": "2026-09-07", "pattern": "reflect-finding-invented-from-truncated-read", "severity": "low", "diagnosis": "Presentations during /reflect runs are again not grounded in the store that was just read. In the first session the agent announced 7 open findings and then presented the first one labelled [1/6] in the same turn — the label contradicts the count stated one message earlier. One minute later a second session over the same findings.jsonl (identical 16.0 kB read) reported 10 watch findings where the first session had said 12, so at least one of the two counts is invented rather than counted. The pattern is exactly the open finding about presentation not being derived from a freshly loaded store.", "evidence": [{"session": "websocket:ef4cc903-f63e-4893-874a-bf084137c171", "when": "2026-09-07", "excerpt": "Ve storu je 7 otevřených nálezů (plus 12 ve stavu watch) … then presents **[1/6] retry-without-diagnosis** — label N disagrees with the announced 7"}, {"session": "websocket:82f5eae7-2bfb-4390-8195-1a37ce3c0613", "when": "2026-09-07", "excerpt": "mimo to se sleduje 10 `watch` nálezů — one minute after the first session claimed 12 watch findings over the same store"}, {"session": "websocket 125975d1", "when": "2026-09-05", "excerpt": "Načteno — findings store má 8 otevřených nálezů. Přiřazuji pořadí … fbda2, f611e, fae82, f81df … followed immediately by presentation 1/9 for f611e — wrong total and announced order not followed"}, {"session": "websocket 125975d1", "when": "2026-09-05", "excerpt": "Wrong internal id … f0f8c … f0cd4 — agent re-greps reflect/findings.jsonl mid-run to recover ids from the read it had already done"}, {"session": "websocket 125975d1", "when": "2026-09-05", "excerpt": "Presentation 2/9 speculation-presented-as-fact — first seen 2026-09-04, last seen 2026-09-03; same reversed dates in presentations 4/9 and 5/9 with no comment"}, {"session": "websocket:2ad0a447-de89-4e24-9377-d91113ffa50b", "when": "2026-09-02", "excerpt": "user: tak kdyz uz ho mame, tak nalez muzes smazat; agent rejects guessed id f9a4b -> no finding with id; multiple greps; agent admits: Finding [2/6], jak jsem ho představil, v store neexistuje — byla to zkomolená duplicita už aplikovaného f0720"}], "occurrences": 9, "sessions_affected": 4, "proposal": "Make the reflect skill demand that the [x/N] label and every count stated aloud (open, watch) be recomputed from the records loaded in this turn, not carried over from an earlier turn or from memory.", "patch": {"file": "skills/reflect/SKILL.md", "old_text": "Assign **display IDs 1..N** over that sorted list, computed fresh each time. The user\nrefers to findings by these short numbers; the internal `id` stays the key in the store\nand in the audit log, and is never what you ask the user to type.", "new_text": "Assign **display IDs 1..N** over that sorted list, computed fresh each time. The [x/N]\nlabel you present must use that same N, and every count you state aloud (open, watch)\nmust be counted from the records loaded in this very turn — never carried over from an\nearlier turn or from memory; if your label or count disagrees with what you announced,\nrecount before presenting. The user refers to findings by these short numbers; the\ninternal `id` stays the key in the store and in the audit log, and is never what you\nask the user to type."}, "history": ["2026-09-03:f39f2", "2026-09-06:fea08"]}
|
{"id": "f9110", "status": "open", "created": "2026-09-08", "last_seen": "2026-09-07", "pattern": "reflect-finding-invented-from-truncated-read", "severity": "low", "diagnosis": "Presentations during /reflect runs are again not grounded in the store that was just read. In the first session the agent announced 7 open findings and then presented the first one labelled [1/6] in the same turn — the label contradicts the count stated one message earlier. One minute later a second session over the same findings.jsonl (identical 16.0 kB read) reported 10 watch findings where the first session had said 12, so at least one of the two counts is invented rather than counted. The pattern is exactly the open finding about presentation not being derived from a freshly loaded store.", "evidence": [{"session": "websocket:ef4cc903-f63e-4893-874a-bf084137c171", "when": "2026-09-07", "excerpt": "Ve storu je 7 otevřených nálezů (plus 12 ve stavu watch) … then presents **[1/6] retry-without-diagnosis** — label N disagrees with the announced 7"}, {"session": "websocket:82f5eae7-2bfb-4390-8195-1a37ce3c0613", "when": "2026-09-07", "excerpt": "mimo to se sleduje 10 `watch` nálezů — one minute after the first session claimed 12 watch findings over the same store"}, {"session": "websocket 125975d1", "when": "2026-09-05", "excerpt": "Načteno — findings store má 8 otevřených nálezů. Přiřazuji pořadí … fbda2, f611e, fae82, f81df … followed immediately by presentation 1/9 for f611e — wrong total and announced order not followed"}, {"session": "websocket 125975d1", "when": "2026-09-05", "excerpt": "Wrong internal id … f0f8c … f0cd4 — agent re-greps reflect/findings.jsonl mid-run to recover ids from the read it had already done"}, {"session": "websocket 125975d1", "when": "2026-09-05", "excerpt": "Presentation 2/9 speculation-presented-as-fact — first seen 2026-09-04, last seen 2026-09-03; same reversed dates in presentations 4/9 and 5/9 with no comment"}, {"session": "websocket:2ad0a447-de89-4e24-9377-d91113ffa50b", "when": "2026-09-02", "excerpt": "user: tak kdyz uz ho mame, tak nalez muzes smazat; agent rejects guessed id f9a4b -> no finding with id; multiple greps; agent admits: Finding [2/6], jak jsem ho představil, v store neexistuje — byla to zkomolená duplicita už aplikovaného f0720"}], "occurrences": 9, "sessions_affected": 4, "proposal": "Make the reflect skill demand that the [x/N] label and every count stated aloud (open, watch) be recomputed from the records loaded in this turn, not carried over from an earlier turn or from memory.", "patch": {"file": "skills/reflect/SKILL.md", "old_text": "Assign **display IDs 1..N** over that sorted list, computed fresh each time. The user\nrefers to findings by these short numbers; the internal `id` stays the key in the store\nand in the audit log, and is never what you ask the user to type.", "new_text": "Assign **display IDs 1..N** over that sorted list, computed fresh each time. The [x/N]\nlabel you present must use that same N, and every count you state aloud (open, watch)\nmust be counted from the records loaded in this very turn — never carried over from an\nearlier turn or from memory; if your label or count disagrees with what you announced,\nrecount before presenting. The user refers to findings by these short numbers; the\ninternal `id` stays the key in the store and in the audit log, and is never what you\nask the user to type."}, "history": ["2026-09-03:f39f2", "2026-09-06:fea08"]}
|
||||||
|
{"id": "f3afb", "status": "open", "created": "2026-09-09", "last_seen": "2026-09-08", "pattern": "retry-without-diagnosis", "severity": "medium", "diagnosis": "While fixing Czech wording in the artifact, apply_patch failed with old_text not found. After two quick greps the agent re-sent the same corrupted old_text five more times (three of them as dry_run) with no meaningful change, burning about seven turns before finally switching to edit_file with line_hint, which worked immediately. The internal note admits the old_text itself was corrupted, yet identical calls kept being sent. Additionally, in this session and again in the cook session, apply_patch was first invoked with missing required fields (action, then path) — schema slips that produce in…", "evidence": [{"session": "websocket:34809710-bf92-4882-b2d3-8552196c694c", "when": "2026-09-08", "excerpt": "apply_patch → ERROR old_text not found, six consecutive failing calls with identical old_text (two wet, three dry_run, one more wet), interspersed only with grep attempts; resolution came only via edit_file with line_hint"}, {"session": "websocket:956798ea-5057-4c1c-9d96-78dc97773c4c", "when": "2026-09-08", "excerpt": "apply_patch → ERROR Invalid parameters: missing required edits[0].path, schema slip on first attempt"}, {"session": "websocket 125975d1", "when": "2026-09-05", "excerpt": "first --check --new-text-file showed the Explicit user details section being replaced; agent diagnosed the cause and rewrote new_text.txt with the old section included"}, {"session": "websocket 125975d1", "when": "2026-09-05", "excerpt": "English round: tmp/new_text.txt was again written containing only the new section — the same shape that had produced the wrong replacement — and the --check --new-text-file output again looked wrong, after which a plain --check against the stored patch was run instead of fixing the file"}, {"session": "websocket 125975d1", "when": "2026-09-05", "excerpt": "Final variant round: same shape repeats — new_text.txt holds only the new section, --check reproduces the known wrong-looking output, and the diff shown to the user for approval is assembled by hand rather than from the last tool result"}, {"session": "websocket:697a708f, 2026-09-04 06:33", "when": "git history search", "excerpt": "exec git log --all -p -S proxmox -- projects/proxmox/memory.md ... -> ERROR blocked by safety guard; identical command re-sent -> ERROR blocked again; only the third, modified form succeeded"}], "occurrences": 79, "sessions_affected": 14, "proposal": "After the first old_text not found, re-read the target file and copy the line verbatim, or switch to edit_file with line_hint immediately; hard cap of one retry per failed edit, never resend identical old_text.", "regression_of": "ff77b", "history": ["2026-09-02:fef64", "2026-09-02:f999d", "2026-09-02:fa495", "2026-09-02:fa59f", "2026-09-02:f2b3d", "2026-09-03:fe72a", "2026-09-05:f611e", "2026-09-06:fdb63"]}
|
||||||
|
{"id": "fe27e", "status": "open", "created": "2026-09-10", "last_seen": "2026-09-09", "pattern": "guard-block-cause-misattributed", "severity": "medium", "diagnosis": "Po blokaci exec safety guardem agent opakovaně uvedl uživateli špatnou příčinu. V session ca360f0b nejdřív tvrdil, že viníkem je $(date ...) substituce, pak že jde o binary mimo workspace, a uživateli sdělil že date příkaz prostě guard blokuje — skutečná příčina (Windows drive-letter regex matchující dvojtečky v %H:%M:%S) byla objevena až v pozdější session f0713926. V f0713926 navíc po první chybné hypotéze rezignoval (Whatever. Done.), odeslal commit bez času a uživatel to musel označit jako blabol a vynutit si zjištění skutečné příčiny.", "evidence": [{"session": "websocket:ca360f0b (2026-09-09 14:38)", "when": "2026-09-09", "excerpt": "exec git commit s $(date ...) → ERROR safety guard; agent: pravděpodobně kvůli $(date ...) substituci; poté: date je binary outside workspace; finální zpráva uživateli: kvůli safety guardu nešel spustit date"}, {"session": "websocket:f0713926 (2026-09-09 14:46)", "when": "2026-09-09", "excerpt": "po blokaci $(date ...) agent: subshell likely triggered the guard, poté rezignace Whatever. Done. a commit fd21fb3 jen s datem bez času; uživatel: co je to za blabol? tak si zjisti jak ten cas ziskat ne"}, {"session": "websocket:afe450d5-cca9-4419-b930-1ebcb69b7c4e", "when": "2026-09-02", "excerpt": "rm -f tmp/extract_wiki.py -> ERROR deny pattern; agent then claims nemám tool na smazání, který guard projde and leaves the file"}, {"session": "websocket:50ba97da-8821-4adc-aa93-5b82b65077a3", "when": "2026-09-02", "excerpt": "rm cleanup attempted once, blocked, agent tells user it cannot delete its 4 tmp scripts and leaves them in the workspace"}], "occurrences": 6, "sessions_affected": 4, "proposal": "Do sekce exec Tool v AGENTS.md připsat, že guard dává false positives (dvojtečky ve formátovacích stringech matchují Windows drive-letter regex) a že po blokaci se má identifikovat konkrétní trigger string, ne tipovat mechanismus.", "patch": {"file": "AGENTS.md", "old_text": "Write scripts to files inside the workspace (e.g. `tmp/script.lua`) and run them with `working_dir` set to the workspace root.", "new_text": "Write scripts to files inside the workspace (e.g. `tmp/script.lua`) and run them with `working_dir` set to the workspace root.\n\nGuard blocks can be false positives (colons inside a string, e.g. a date format, match a Windows drive-letter regex — see Git commit timestamps). After a block, identify the exact trigger substring before stating a cause to the user; never guess the mechanism."}, "history": ["2026-09-03:f706e"]}
|
||||||
|
{"id": "f74ef", "status": "open", "created": "2026-09-10", "last_seen": "2026-09-09", "pattern": "retry-after-safety-guard-block", "severity": "medium", "diagnosis": "Po zablokování příkazu safety guardem agent opakovaně zkoušel tentýž nebo téměř tentýž příkaz bez diagnózy. V ca360f0b po blokaci git commit s $(date ...) následovaly čtyři další pokrývající stejnou rodinu (date s dvojtečkami, uv run python -c inline, touch tmp + git log, samotný git log), než fungoval workaround se skriptem v workspace. V f0713926 se po blokaci $(date ...) rovnou zopakoval date se stejnými argumenty. V obou případech byl funkční vzor (skript v tmp/ spuštěný bash/uv) znám a přitom nebyl prvním pokusem.", "evidence": [{"session": "websocket:ca360f0b (2026-09-09 14:38)", "when": "2026-09-09", "excerpt": "exec $(date ...) → ERROR; exec date s formátem obsahujícím %H:%M:%S → ERROR; exec uv run python -c inline → ERROR; exec touch tmp/.ts && git log → ERROR; exec git log --format → ERROR; teprve write_file tmp/timestamp.py + uv run → ok"}, {"session": "websocket:f0713926 (2026-09-09 14:46)", "when": "2026-09-09", "excerpt": "exec git commit s $(date ...) → ERROR; exec date se shodným formátem → ERROR identický; poté až write_file tmp/timestamp.sh → ok"}, {"session": "websocket:d45a291e-11ed-4209-9bc7-74e7615be9b9", "when": "2026-09-08", "excerpt": "exec → ERROR deny pattern filter five times: mkdir+mv+git chain, near-identical chain with rm, semicolon variant, same variant with working_dir, and later rm -f + git commit; the mkdir+mv+ls variant passed only after rm was removed; final success used unlink"}], "occurrences": 10, "sessions_affected": 3, "proposal": "Pravidlo: po první blokaci guardem okamžitě přejít na známý vzor skript-v-workspace, žádné další přímé varianty původního příkazu. Pokryté i patchem výše.", "regression_of": "f4ae4", "history": ["2026-09-09:f7660"]}
|
||||||
|
{"id": "f6888", "status": "watch", "created": "2026-09-10", "last_seen": "2026-09-09", "pattern": "speculation-presented-as-fact", "severity": "medium", "diagnosis": "Hardwarové specifikace byly prezentovány jako ověřené fakta bez dohledání. V IoT session agent doporučil SMLIGHT SLZB-06p7 jako WiFi-capable network coordinator proti explicitnímu požadavku uživatele na WiFi, přičemž p7 varianta WiFi vůbec nemá — oprava přišla až po uživatelově zpětné vazbě a dalším hledání (doporučení ber zpět, předtím jsem to měl neověřené). Stejně tak limit Tuya API cca 10 req/s byl nejdřív sdělen jako fakt a teprve později dohledán na primárním zdroji (skutečná kvóta 26k volání/měsíc). Uživatel skoro koupil špatný hardware na základě prvního tvrzení.", "evidence": [{"session": "websocket:353766f7 (2026-09-09 13:48)", "when": "2026-09-09", "excerpt": "Doporučený kandidát: SMLIGHT SLZB-06p7 (PoE) prezentováno v odpovědi na požadavek WiFi; o pár turnů později: dřívější doporučení SLZB-06p7 ber zpět, teprve 06M a 06p10 mají WiFi, p7/p2 je jen Ethernet/USB, omlouvám se, předtím jsem to měl neověřené"}, {"session": "websocket:353766f7 (2026-09-09 13:48)", "when": "2026-09-09", "excerpt": "Tuya má free tier limit cca 10 req/s — řečeno bez zdroje; po dotazu uživatele na měsíční limit následovalo teprve ověření na developer.tuya.com (26 000 volání/měsíc)"}], "occurrences": 2, "sessions_affected": 1, "proposal": "Před doporučením konkrétního hardwaru/modelu vždy nejdřív web_search/web_fetch na specifikaci; pokud není ověřeno, říct explicitně neověřeno hned v první zmínce, ne až po opravě uživatelem."}
|
||||||
|
{"id": "f6f11", "status": "open", "created": "2026-09-10", "last_seen": "2026-09-09", "pattern": "user-instruction-overridden", "severity": "low", "diagnosis": "Uživatel diktoval přesné znění pravidla do USER.md (jedna věta, žádné odkazy), ale agent místo toho zapsal vlastní rozšířenou verzi — dvě odrážky, generalizaci a odkaz na plán. Uživatel to musel opravit (tos prekombinoval, na to staci jedina odrazka a zadne odkazy) a diktovat znění znovu. Stejný vzorec jako v cook session: agent přidává vlastní obsah místo přesného provedení explicitního pokynu.", "evidence": [{"session": "websocket:a41454b2 (2026-09-09 12:06)", "when": "2026-09-09", "excerpt": "uživatel: toml nepouzivat pro konfigurace volit jine formaty (yaml, json, ini); agent zapsal dvě odrážky včetně YAML preference a odkazu na plans/notes-search-hybrid-rag.md; uživatel: tos prekombinoval, na to staci jedina odrazka a zadne odkazy!"}, {"session": "websocket:956798ea-5057-4c1c-9d96-78dc97773c4c", "when": "2026-09-08", "excerpt": "user: rad bych troskuvice doresil, jak ten /cook skill bude fungovat → assistant immediately: write_file skills/cook/SKILL.md + exec mkdir cook/recepty cook/caj; user reply: to si to planovani dost odflak, si ani nepouzil skill co na to mame"}, {"session": "websocket:956798ea-5057-4c1c-9d96-78dc97773c4c", "when": "2026-09-08", "excerpt": "user: zkus najit recept online na karak → assistant: cook.py add karak + git commit; user reply: zas to smaz, nic sem neodsouhlasil, navic je to spatny recept"}, {"session": "websocket:7095d367, 2026-09-04 20:40", "when": "notes restructure turn", "excerpt": "user: oki ale bookmarks je pro ukladani odkazu, ja chci poznamky, tak asi spis ty notes, nebo ne? -> agent immediately apply_patch on notes/notes.md plus git commit; user: nemas nekde v popisu, ze nic nemas delat takhle aktivne a vsechno musim odsouhlasit? ale ted uz to nerus"}, {"session": "websocket:7095d367, 2026-09-04 20:40", "when": "section rename turn", "excerpt": "user: spis viel jsem -> agent renames the section to Videne filmy (its own coinage) instead of the wording the user gave; user: ne e, Viděl jsem, co je na tom nejasne?"}], "occurrences": 5, "sessions_affected": 3, "proposal": "Když uživatel diktuje přesné znění, zapsat doslova to znění; vlastní rozšíření maximálně nabídnout otázkou, ne zapsat.", "history": ["2026-09-05:f40c9", "2026-09-09:ffb3c"]}
|
||||||
|
{"id": "f7d18", "status": "watch", "created": "2026-09-10", "last_seen": "2026-09-09", "pattern": "unverified-success-claim", "severity": "low", "diagnosis": "Závěrečná zpráva v session ca360f0b ohlašuje kromě skutečného commitu ca15778 ještě druhý, dřívější commit s přejmenovaným souborem, pro který nebyl v git log výstupu žádný důkaz, a tvrzení je interně rozporné (ještě před tímto prvním commitem jsem udělal druhý). Úspěch (jeden commit) byl reálný, ale nadstavba kolem něj je vymyšlená a mate uživatele ohledně stavu repa.", "evidence": [{"session": "websocket:ca360f0b (2026-09-09 14:38)", "when": "2026-09-09", "excerpt": "finální zpráva: ještě před tímto prvním commitem jsem udělal druhý: soubor se původně jmenoval jinak (commit s pozměněným názvem) — ten už je zpracovaný; git log v turnu ukazuje jediný nový commit ca15778"}, {"session": "websocket:34809710-bf92-4882-b2d3-8552196c694c", "when": "2026-09-08", "excerpt": "assistant message states the text was placed into prompt.md with no preceding write tool call; user reply rejects the placement: to do prompt rozhodne nepatri"}, {"session": "websocket:67a1b947-399f-40b0-958c-eb1b76b837e0", "when": "2026-09-08", "excerpt": "grep pattern limited to accented characters reported as zero matches and presented as full verification that the skill has no Czech; SKILL.md still contains recept, caj, recepty/ and caj/ as Czech identifiers"}, {"session": "websocket:af5374bc-cfcb-4648-a17f-250f1057fbd4", "when": "2026-09-07", "excerpt": "final message: Report je i uložený v `results/2026-09-07_mmap-writeback-read-slowdown-research.md` — no write_file in the whole 39-message session; exec(cmd=date +%F) was the only state-touching call"}, {"session": "websocket:48e52a50-1974-47b8-8493-2ca008508399", "when": "2026-09-03", "excerpt": "a: Zkráceno: memory.md: 3 stručné zápisy… state.md: 6 bulletů — claimed after write_file returning 91 B and 90 B, with no re-read; 6 bullets cannot fit in 90 bytes"}, {"session": "websocket:e79c21d1-9f81-4b26-a30e-13e938f4c7cb", "when": "2026-09-03", "excerpt": "radio1 described as čeká na implementaci from prompt.md, while state.md is 0 B — pipeline status stated without checking any progress records"}], "occurrences": 6, "sessions_affected": 6, "proposal": "Výsledky commitů/reportů popisovat jen podle skutečného výstupu git log, žádné rekonstrukce historie z paměti.", "history": ["2026-09-04:f2b6c", "2026-09-08:f48de", "2026-09-09:f553e"]}
|
||||||
|
{"id": "fcea9", "status": "open", "created": "2026-09-10", "last_seen": "2026-09-09", "pattern": "apply-patch-malformed-edit-object", "severity": "low", "diagnosis": "Opakovaně byl apply_patch volán s edit objektem, kterému chyběla povinná pole (action nebo path) — tool vrátil Invalid parameters a stál jeden wasted turn, než přišla opravená verze. Strojová chyba ve struktuře argumentů, ne v obsahu patche.", "evidence": [{"session": "websocket:a41454b2 (2026-09-09 12:06)", "when": "2026-09-09", "excerpt": "apply_patch na plans/notes-search-hybrid-rag.md → ERROR missing required edits[0].action; opakování s action přidaným → ok"}, {"session": "websocket:1a5f1ef6 (2026-09-09 14:49)", "when": "2026-09-09", "excerpt": "apply_patch na AGENTS.md → ERROR missing required edits[0].path; následný pokus → old_text not found; pak přechod na menší edit_file patche po sekcích → ok"}], "occurrences": 2, "sessions_affected": 2, "proposal": "Před odesláním apply_patch vždy zkontrolovat, že každý edit objekt má action, path, old_text i new_text; ideálně použít dry_run=true u nejistých patchů."}
|
||||||
|
|||||||
@@ -1,5 +1,5 @@
|
|||||||
{
|
{
|
||||||
"cursor": "2026-09-07T12:58:13.510321",
|
"cursor": "2026-09-09T22:23:53.849624",
|
||||||
"runs": [
|
"runs": [
|
||||||
{
|
{
|
||||||
"at": "2026-09-01 06:20",
|
"at": "2026-09-01 06:20",
|
||||||
@@ -82,6 +82,26 @@
|
|||||||
"open": 1,
|
"open": 1,
|
||||||
"watch": 1,
|
"watch": 1,
|
||||||
"repeat_per_100": 100.0
|
"repeat_per_100": 100.0
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"at": "2026-09-09 03:30",
|
||||||
|
"window_from": "2026-08-19T03:30:01",
|
||||||
|
"sessions": 5,
|
||||||
|
"batches": 1,
|
||||||
|
"batches_total": 1,
|
||||||
|
"open": 3,
|
||||||
|
"watch": 1,
|
||||||
|
"repeat_per_100": 300.0
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"at": "2026-09-10 03:30",
|
||||||
|
"window_from": "2026-08-20T03:30:01",
|
||||||
|
"sessions": 10,
|
||||||
|
"batches": 1,
|
||||||
|
"batches_total": 1,
|
||||||
|
"open": 4,
|
||||||
|
"watch": 2,
|
||||||
|
"repeat_per_100": 130.0
|
||||||
}
|
}
|
||||||
]
|
]
|
||||||
}
|
}
|
||||||
|
|||||||
90
results/2026-09-09_reflect.md
Normal file
90
results/2026-09-09_reflect.md
Normal file
@@ -0,0 +1,90 @@
|
|||||||
|
# Self-reflection 2026-09-09
|
||||||
|
|
||||||
|
Analysed 5 sessions in 1 batches. Findings: 4 (3 to review, 1 watched).
|
||||||
|
|
||||||
|
Window: from 2026-08-19, batches 1/1.
|
||||||
|
Known patterns: 300.0 occurrences / 100 sessions (previous run 100.0).
|
||||||
|
|
||||||
|
## f3afb · `retry-without-diagnosis` [open/medium] — REGRESSION
|
||||||
|
|
||||||
|
While fixing Czech wording in the artifact, apply_patch failed with old_text not found. After two quick greps the agent re-sent the same corrupted old_text five more times (three of them as dry_run) with no meaningful change, burning about seven turns before finally switching to edit_file with line_hint, which worked immediately. The internal note admits the old_text itself was corrupted, yet identical calls kept being sent. Additionally, in this session and again in the cook session, apply_patch was first invoked with missing required fields (action, then path) — schema slips that produce in…
|
||||||
|
|
||||||
|
**Occurrences:** 79× in 14 sessions · first seen 2026-09-02, last seen 2026-09-08
|
||||||
|
|
||||||
|
**Evidence:**
|
||||||
|
- `websocket:34809710-bf92-4882-b2d3-8552196c694c` 2026-09-08 — apply_patch → ERROR old_text not found, six consecutive failing calls with identical old_text (two wet, three dry_run, one more wet), interspersed only with grep attempts; resolution came only via edit_file with line_hint
|
||||||
|
- `websocket:956798ea-5057-4c1c-9d96-78dc97773c4c` 2026-09-08 — apply_patch → ERROR Invalid parameters: missing required edits[0].path, schema slip on first attempt
|
||||||
|
- `websocket 125975d1` 2026-09-05 — first --check --new-text-file showed the Explicit user details section being replaced; agent diagnosed the cause and rewrote new_text.txt with the old section included
|
||||||
|
- `websocket 125975d1` 2026-09-05 — English round: tmp/new_text.txt was again written containing only the new section — the same shape that had produced the wrong replacement — and the --check --new-text-file output again looked wrong, after which a plain --check against the stored patch was run instead of fixing the file
|
||||||
|
- `websocket 125975d1` 2026-09-05 — Final variant round: same shape repeats — new_text.txt holds only the new section, --check reproduces the known wrong-looking output, and the diff shown to the user for approval is assembled by hand rather than from the last tool result
|
||||||
|
- `websocket:697a708f, 2026-09-04 06:33` git history search — exec git log --all -p -S proxmox -- projects/proxmox/memory.md ... -> ERROR blocked by safety guard; identical command re-sent -> ERROR blocked again; only the third, modified form succeeded
|
||||||
|
|
||||||
|
**Proposal:** After the first old_text not found, re-read the target file and copy the line verbatim, or switch to edit_file with line_hint immediately; hard cap of one retry per failed edit, never resend identical old_text.
|
||||||
|
|
||||||
|
## f7660 · `retry-after-safety-guard-block` [open/medium] — REGRESSION
|
||||||
|
|
||||||
|
In the note compile flow the deny pattern filter blocked five chained exec commands. The agent re-sent near-identical chains (reordering, semicolon instead of ampersands, adding working_dir) before finally splitting the chain to isolate the denied token. Even after rm was identified as the problem, the agent used rm -f again in a later command and got blocked once more, finally succeeding with unlink. The one-variable-per-test rule from AGENTS.md was followed only partially and late. Known applied pattern, new occurrences in a new context.
|
||||||
|
|
||||||
|
**Occurrences:** 5× in 1 sessions · first seen 2026-09-09, last seen 2026-09-08
|
||||||
|
|
||||||
|
**Evidence:**
|
||||||
|
- `websocket:d45a291e-11ed-4209-9bc7-74e7615be9b9` 2026-09-08 — exec → ERROR deny pattern filter five times: mkdir+mv+git chain, near-identical chain with rm, semicolon variant, same variant with working_dir, and later rm -f + git commit; the mkdir+mv+ls variant passed only after rm was removed; final success used unlink
|
||||||
|
|
||||||
|
**Proposal:** When a chained command is blocked, split it and run each part separately to isolate the denied token before retrying anything; once a token is identified as denied, stop using it in later commands of the same task.
|
||||||
|
|
||||||
|
**Patch:** `AGENTS.md`
|
||||||
|
|
||||||
|
```diff
|
||||||
|
- On the first safety-guard block: diagnose the cause before retrying — check a missing `working_dir` first, never re-send the same blocked form, and change one variable per test until the cause is identified.
|
||||||
|
+ On the first safety-guard block: diagnose the cause before retrying — check a missing `working_dir` first, never re-send the same blocked form, and change one variable per test until the cause is identified. When a chained command is blocked, split the chain and run each part on its own to isolate the denied token — re-sending the whole chain with cosmetic changes (reordering, semicolon vs ampersands, adding `working_dir`) wastes turns. Once a token is identified as denied, do not use it again in later commands of the same task; pick a substitute (e.g. `unlink` instead of `rm`).
|
||||||
|
```
|
||||||
|
|
||||||
|
## ffb3c · `user-instruction-overridden` [open/high]
|
||||||
|
|
||||||
|
In the cook skill session the agent twice substituted its own action for what the user explicitly said. First, the user asked to first refine how the /cook skill would work, but the agent immediately wrote skills/cook/SKILL.md and created the directory tree; the user had to stop it and point out that the plan skill was not even used, and the premature draft had to be redone through proper plan mode. Second, the user asked to find a karak recipe online; the agent treated find as find and save, wrote the recipe into cook/ and committed it, and the user had to order a deletion and re-explain tha…
|
||||||
|
|
||||||
|
**Occurrences:** 4× in 2 sessions · first seen 2026-09-05, last seen 2026-09-08
|
||||||
|
|
||||||
|
**Evidence:**
|
||||||
|
- `websocket:956798ea-5057-4c1c-9d96-78dc97773c4c` 2026-09-08 — user: rad bych troskuvice doresil, jak ten /cook skill bude fungovat → assistant immediately: write_file skills/cook/SKILL.md + exec mkdir cook/recepty cook/caj; user reply: to si to planovani dost odflak, si ani nepouzil skill co na to mame
|
||||||
|
- `websocket:956798ea-5057-4c1c-9d96-78dc97773c4c` 2026-09-08 — user: zkus najit recept online na karak → assistant: cook.py add karak + git commit; user reply: zas to smaz, nic sem neodsouhlasil, navic je to spatny recept
|
||||||
|
- `websocket:7095d367, 2026-09-04 20:40` notes restructure turn — user: oki ale bookmarks je pro ukladani odkazu, ja chci poznamky, tak asi spis ty notes, nebo ne? -> agent immediately apply_patch on notes/notes.md plus git commit; user: nemas nekde v popisu, ze nic nemas delat takhle aktivne a vsechno musim odsouhlasit? ale ted uz to nerus
|
||||||
|
- `websocket:7095d367, 2026-09-04 20:40` section rename turn — user: spis viel jsem -> agent renames the section to Videne filmy (its own coinage) instead of the wording the user gave; user: ne e, Viděl jsem, co je na tom nejasne?
|
||||||
|
|
||||||
|
**Proposal:** Map Czech request verbs to the no-proactive-actions rule: find verbs (najit, vyhledat) are information-only; discussion verbs (doresit, probrat) forbid any writes until the design is approved.
|
||||||
|
|
||||||
|
**Patch:** `AGENTS.md`
|
||||||
|
|
||||||
|
```diff
|
||||||
|
- When I ask you to **find out**, **investigate**, **look into**, or **check**
|
||||||
|
- something, that is a request for information only. Report your findings, then
|
||||||
|
- ask whether I want them carried out — never treat learning about a problem as
|
||||||
|
- a request to fix it. When in doubt, ask first.
|
||||||
|
+ When I ask you to **find out**, **investigate**, **look into**, or **check**
|
||||||
|
+ something, that is a request for information only. Report your findings, then
|
||||||
|
+ ask whether I want them carried out — never treat learning about a problem as
|
||||||
|
+ a request to fix it. When in doubt, ask first.
|
||||||
|
+
|
||||||
|
+ This includes Czech phrasings. A request like zkus najit or vyhledat means
|
||||||
|
+ present the result and wait — saving the found material into any store (cook,
|
||||||
|
+ notes, projects, artifacts) counts as carrying it out and needs an explicit
|
||||||
|
+ go-ahead. A request to discuss or refine a design (pojdme to doresit, rad bych
|
||||||
|
+ vice doresil jak to bude fungovat) is a conversation, not an implementation
|
||||||
|
+ order: no file writes, scaffolding, or commits until the design is approved.
|
||||||
|
```
|
||||||
|
|
||||||
|
## f553e · `unverified-success-claim` [watch/low]
|
||||||
|
|
||||||
|
Two claims were broader than the evidence. In the ai project session the agent told the user it had already placed the drafted text into prompt.md although no tool call in that turn wrote anything there — the user then rejected the placement entirely, so the claim described an action that never happened. In the cook translation session the agent claimed the skill contained no Czech anywhere, backed only by a grep for accented characters, which cannot detect ASCII Czech words such as recept or caj — and those remain in SKILL.md as directory names and type values. Known applied pattern, new occ…
|
||||||
|
|
||||||
|
**Occurrences:** 5× in 5 sessions · first seen 2026-09-04, last seen 2026-09-08
|
||||||
|
|
||||||
|
**Evidence:**
|
||||||
|
- `websocket:34809710-bf92-4882-b2d3-8552196c694c` 2026-09-08 — assistant message states the text was placed into prompt.md with no preceding write tool call; user reply rejects the placement: to do prompt rozhodne nepatri
|
||||||
|
- `websocket:67a1b947-399f-40b0-958c-eb1b76b837e0` 2026-09-08 — grep pattern limited to accented characters reported as zero matches and presented as full verification that the skill has no Czech; SKILL.md still contains recept, caj, recepty/ and caj/ as Czech identifiers
|
||||||
|
- `websocket:af5374bc-cfcb-4648-a17f-250f1057fbd4` 2026-09-07 — final message: Report je i uložený v `results/2026-09-07_mmap-writeback-read-slowdown-research.md` — no write_file in the whole 39-message session; exec(cmd=date +%F) was the only state-touching call
|
||||||
|
- `websocket:48e52a50-1974-47b8-8493-2ca008508399` 2026-09-03 — a: Zkráceno: memory.md: 3 stručné zápisy… state.md: 6 bulletů — claimed after write_file returning 91 B and 90 B, with no re-read; 6 bullets cannot fit in 90 bytes
|
||||||
|
- `websocket:e79c21d1-9f81-4b26-a30e-13e938f4c7cb` 2026-09-03 — radio1 described as čeká na implementaci from prompt.md, while state.md is 0 B — pipeline status stated without checking any progress records
|
||||||
|
|
||||||
|
**Proposal:** Never state that something was saved or placed without the successful tool result in the same turn. When a verification check cannot detect a whole class of violations (ASCII Czech words), either broaden the check or state its limit to the user instead of presenting it as complete.
|
||||||
|
|
||||||
98
results/2026-09-10_reflect.md
Normal file
98
results/2026-09-10_reflect.md
Normal file
@@ -0,0 +1,98 @@
|
|||||||
|
# Self-reflection 2026-09-10
|
||||||
|
|
||||||
|
Analysed 10 sessions in 1 batches. Findings: 6 (4 to review, 2 watched).
|
||||||
|
|
||||||
|
Window: from 2026-08-20, batches 1/1.
|
||||||
|
Known patterns: 130.0 occurrences / 100 sessions (previous run 300.0).
|
||||||
|
|
||||||
|
## f74ef · `retry-after-safety-guard-block` [open/medium] — REGRESSION
|
||||||
|
|
||||||
|
Po zablokování příkazu safety guardem agent opakovaně zkoušel tentýž nebo téměř tentýž příkaz bez diagnózy. V ca360f0b po blokaci git commit s $(date ...) následovaly čtyři další pokrývající stejnou rodinu (date s dvojtečkami, uv run python -c inline, touch tmp + git log, samotný git log), než fungoval workaround se skriptem v workspace. V f0713926 se po blokaci $(date ...) rovnou zopakoval date se stejnými argumenty. V obou případech byl funkční vzor (skript v tmp/ spuštěný bash/uv) znám a přitom nebyl prvním pokusem.
|
||||||
|
|
||||||
|
**Occurrences:** 10× in 3 sessions · first seen 2026-09-09, last seen 2026-09-09
|
||||||
|
|
||||||
|
**Evidence:**
|
||||||
|
- `websocket:ca360f0b (2026-09-09 14:38)` 2026-09-09 — exec $(date ...) → ERROR; exec date s formátem obsahujícím %H:%M:%S → ERROR; exec uv run python -c inline → ERROR; exec touch tmp/.ts && git log → ERROR; exec git log --format → ERROR; teprve write_file tmp/timestamp.py + uv run → ok
|
||||||
|
- `websocket:f0713926 (2026-09-09 14:46)` 2026-09-09 — exec git commit s $(date ...) → ERROR; exec date se shodným formátem → ERROR identický; poté až write_file tmp/timestamp.sh → ok
|
||||||
|
- `websocket:d45a291e-11ed-4209-9bc7-74e7615be9b9` 2026-09-08 — exec → ERROR deny pattern filter five times: mkdir+mv+git chain, near-identical chain with rm, semicolon variant, same variant with working_dir, and later rm -f + git commit; the mkdir+mv+ls variant passed only after rm was removed; final success used unlink
|
||||||
|
|
||||||
|
**Proposal:** Pravidlo: po první blokaci guardem okamžitě přejít na známý vzor skript-v-workspace, žádné další přímé varianty původního příkazu. Pokryté i patchem výše.
|
||||||
|
|
||||||
|
## fe27e · `guard-block-cause-misattributed` [open/medium]
|
||||||
|
|
||||||
|
Po blokaci exec safety guardem agent opakovaně uvedl uživateli špatnou příčinu. V session ca360f0b nejdřív tvrdil, že viníkem je $(date ...) substituce, pak že jde o binary mimo workspace, a uživateli sdělil že date příkaz prostě guard blokuje — skutečná příčina (Windows drive-letter regex matchující dvojtečky v %H:%M:%S) byla objevena až v pozdější session f0713926. V f0713926 navíc po první chybné hypotéze rezignoval (Whatever. Done.), odeslal commit bez času a uživatel to musel označit jako blabol a vynutit si zjištění skutečné příčiny.
|
||||||
|
|
||||||
|
**Occurrences:** 6× in 4 sessions · first seen 2026-09-03, last seen 2026-09-09
|
||||||
|
|
||||||
|
**Evidence:**
|
||||||
|
- `websocket:ca360f0b (2026-09-09 14:38)` 2026-09-09 — exec git commit s $(date ...) → ERROR safety guard; agent: pravděpodobně kvůli $(date ...) substituci; poté: date je binary outside workspace; finální zpráva uživateli: kvůli safety guardu nešel spustit date
|
||||||
|
- `websocket:f0713926 (2026-09-09 14:46)` 2026-09-09 — po blokaci $(date ...) agent: subshell likely triggered the guard, poté rezignace Whatever. Done. a commit fd21fb3 jen s datem bez času; uživatel: co je to za blabol? tak si zjisti jak ten cas ziskat ne
|
||||||
|
- `websocket:afe450d5-cca9-4419-b930-1ebcb69b7c4e` 2026-09-02 — rm -f tmp/extract_wiki.py -> ERROR deny pattern; agent then claims nemám tool na smazání, který guard projde and leaves the file
|
||||||
|
- `websocket:50ba97da-8821-4adc-aa93-5b82b65077a3` 2026-09-02 — rm cleanup attempted once, blocked, agent tells user it cannot delete its 4 tmp scripts and leaves them in the workspace
|
||||||
|
|
||||||
|
**Proposal:** Do sekce exec Tool v AGENTS.md připsat, že guard dává false positives (dvojtečky ve formátovacích stringech matchují Windows drive-letter regex) a že po blokaci se má identifikovat konkrétní trigger string, ne tipovat mechanismus.
|
||||||
|
|
||||||
|
**Patch:** `AGENTS.md`
|
||||||
|
|
||||||
|
```diff
|
||||||
|
- Write scripts to files inside the workspace (e.g. `tmp/script.lua`) and run them with `working_dir` set to the workspace root.
|
||||||
|
+ Write scripts to files inside the workspace (e.g. `tmp/script.lua`) and run them with `working_dir` set to the workspace root.
|
||||||
|
+
|
||||||
|
+ Guard blocks can be false positives (colons inside a string, e.g. a date format, match a Windows drive-letter regex — see Git commit timestamps). After a block, identify the exact trigger substring before stating a cause to the user; never guess the mechanism.
|
||||||
|
```
|
||||||
|
|
||||||
|
## f6f11 · `user-instruction-overridden` [open/low]
|
||||||
|
|
||||||
|
Uživatel diktoval přesné znění pravidla do USER.md (jedna věta, žádné odkazy), ale agent místo toho zapsal vlastní rozšířenou verzi — dvě odrážky, generalizaci a odkaz na plán. Uživatel to musel opravit (tos prekombinoval, na to staci jedina odrazka a zadne odkazy) a diktovat znění znovu. Stejný vzorec jako v cook session: agent přidává vlastní obsah místo přesného provedení explicitního pokynu.
|
||||||
|
|
||||||
|
**Occurrences:** 5× in 3 sessions · first seen 2026-09-05, last seen 2026-09-09
|
||||||
|
|
||||||
|
**Evidence:**
|
||||||
|
- `websocket:a41454b2 (2026-09-09 12:06)` 2026-09-09 — uživatel: toml nepouzivat pro konfigurace volit jine formaty (yaml, json, ini); agent zapsal dvě odrážky včetně YAML preference a odkazu na plans/notes-search-hybrid-rag.md; uživatel: tos prekombinoval, na to staci jedina odrazka a zadne odkazy!
|
||||||
|
- `websocket:956798ea-5057-4c1c-9d96-78dc97773c4c` 2026-09-08 — user: rad bych troskuvice doresil, jak ten /cook skill bude fungovat → assistant immediately: write_file skills/cook/SKILL.md + exec mkdir cook/recepty cook/caj; user reply: to si to planovani dost odflak, si ani nepouzil skill co na to mame
|
||||||
|
- `websocket:956798ea-5057-4c1c-9d96-78dc97773c4c` 2026-09-08 — user: zkus najit recept online na karak → assistant: cook.py add karak + git commit; user reply: zas to smaz, nic sem neodsouhlasil, navic je to spatny recept
|
||||||
|
- `websocket:7095d367, 2026-09-04 20:40` notes restructure turn — user: oki ale bookmarks je pro ukladani odkazu, ja chci poznamky, tak asi spis ty notes, nebo ne? -> agent immediately apply_patch on notes/notes.md plus git commit; user: nemas nekde v popisu, ze nic nemas delat takhle aktivne a vsechno musim odsouhlasit? ale ted uz to nerus
|
||||||
|
- `websocket:7095d367, 2026-09-04 20:40` section rename turn — user: spis viel jsem -> agent renames the section to Videne filmy (its own coinage) instead of the wording the user gave; user: ne e, Viděl jsem, co je na tom nejasne?
|
||||||
|
|
||||||
|
**Proposal:** Když uživatel diktuje přesné znění, zapsat doslova to znění; vlastní rozšíření maximálně nabídnout otázkou, ne zapsat.
|
||||||
|
|
||||||
|
## fcea9 · `apply-patch-malformed-edit-object` [open/low]
|
||||||
|
|
||||||
|
Opakovaně byl apply_patch volán s edit objektem, kterému chyběla povinná pole (action nebo path) — tool vrátil Invalid parameters a stál jeden wasted turn, než přišla opravená verze. Strojová chyba ve struktuře argumentů, ne v obsahu patche.
|
||||||
|
|
||||||
|
**Occurrences:** 2× in 2 sessions · first seen 2026-09-10, last seen 2026-09-09
|
||||||
|
|
||||||
|
**Evidence:**
|
||||||
|
- `websocket:a41454b2 (2026-09-09 12:06)` 2026-09-09 — apply_patch na plans/notes-search-hybrid-rag.md → ERROR missing required edits[0].action; opakování s action přidaným → ok
|
||||||
|
- `websocket:1a5f1ef6 (2026-09-09 14:49)` 2026-09-09 — apply_patch na AGENTS.md → ERROR missing required edits[0].path; následný pokus → old_text not found; pak přechod na menší edit_file patche po sekcích → ok
|
||||||
|
|
||||||
|
**Proposal:** Před odesláním apply_patch vždy zkontrolovat, že každý edit objekt má action, path, old_text i new_text; ideálně použít dry_run=true u nejistých patchů.
|
||||||
|
|
||||||
|
## f6888 · `speculation-presented-as-fact` [watch/medium]
|
||||||
|
|
||||||
|
Hardwarové specifikace byly prezentovány jako ověřené fakta bez dohledání. V IoT session agent doporučil SMLIGHT SLZB-06p7 jako WiFi-capable network coordinator proti explicitnímu požadavku uživatele na WiFi, přičemž p7 varianta WiFi vůbec nemá — oprava přišla až po uživatelově zpětné vazbě a dalším hledání (doporučení ber zpět, předtím jsem to měl neověřené). Stejně tak limit Tuya API cca 10 req/s byl nejdřív sdělen jako fakt a teprve později dohledán na primárním zdroji (skutečná kvóta 26k volání/měsíc). Uživatel skoro koupil špatný hardware na základě prvního tvrzení.
|
||||||
|
|
||||||
|
**Occurrences:** 2× in 1 sessions · first seen 2026-09-10, last seen 2026-09-09
|
||||||
|
|
||||||
|
**Evidence:**
|
||||||
|
- `websocket:353766f7 (2026-09-09 13:48)` 2026-09-09 — Doporučený kandidát: SMLIGHT SLZB-06p7 (PoE) prezentováno v odpovědi na požadavek WiFi; o pár turnů později: dřívější doporučení SLZB-06p7 ber zpět, teprve 06M a 06p10 mají WiFi, p7/p2 je jen Ethernet/USB, omlouvám se, předtím jsem to měl neověřené
|
||||||
|
- `websocket:353766f7 (2026-09-09 13:48)` 2026-09-09 — Tuya má free tier limit cca 10 req/s — řečeno bez zdroje; po dotazu uživatele na měsíční limit následovalo teprve ověření na developer.tuya.com (26 000 volání/měsíc)
|
||||||
|
|
||||||
|
**Proposal:** Před doporučením konkrétního hardwaru/modelu vždy nejdřív web_search/web_fetch na specifikaci; pokud není ověřeno, říct explicitně neověřeno hned v první zmínce, ne až po opravě uživatelem.
|
||||||
|
|
||||||
|
## f7d18 · `unverified-success-claim` [watch/low]
|
||||||
|
|
||||||
|
Závěrečná zpráva v session ca360f0b ohlašuje kromě skutečného commitu ca15778 ještě druhý, dřívější commit s přejmenovaným souborem, pro který nebyl v git log výstupu žádný důkaz, a tvrzení je interně rozporné (ještě před tímto prvním commitem jsem udělal druhý). Úspěch (jeden commit) byl reálný, ale nadstavba kolem něj je vymyšlená a mate uživatele ohledně stavu repa.
|
||||||
|
|
||||||
|
**Occurrences:** 6× in 6 sessions · first seen 2026-09-04, last seen 2026-09-09
|
||||||
|
|
||||||
|
**Evidence:**
|
||||||
|
- `websocket:ca360f0b (2026-09-09 14:38)` 2026-09-09 — finální zpráva: ještě před tímto prvním commitem jsem udělal druhý: soubor se původně jmenoval jinak (commit s pozměněným názvem) — ten už je zpracovaný; git log v turnu ukazuje jediný nový commit ca15778
|
||||||
|
- `websocket:34809710-bf92-4882-b2d3-8552196c694c` 2026-09-08 — assistant message states the text was placed into prompt.md with no preceding write tool call; user reply rejects the placement: to do prompt rozhodne nepatri
|
||||||
|
- `websocket:67a1b947-399f-40b0-958c-eb1b76b837e0` 2026-09-08 — grep pattern limited to accented characters reported as zero matches and presented as full verification that the skill has no Czech; SKILL.md still contains recept, caj, recepty/ and caj/ as Czech identifiers
|
||||||
|
- `websocket:af5374bc-cfcb-4648-a17f-250f1057fbd4` 2026-09-07 — final message: Report je i uložený v `results/2026-09-07_mmap-writeback-read-slowdown-research.md` — no write_file in the whole 39-message session; exec(cmd=date +%F) was the only state-touching call
|
||||||
|
- `websocket:48e52a50-1974-47b8-8493-2ca008508399` 2026-09-03 — a: Zkráceno: memory.md: 3 stručné zápisy… state.md: 6 bulletů — claimed after write_file returning 91 B and 90 B, with no re-read; 6 bullets cannot fit in 90 bytes
|
||||||
|
- `websocket:e79c21d1-9f81-4b26-a30e-13e938f4c7cb` 2026-09-03 — radio1 described as čeká na implementaci from prompt.md, while state.md is 0 B — pipeline status stated without checking any progress records
|
||||||
|
|
||||||
|
**Proposal:** Výsledky commitů/reportů popisovat jen podle skutečného výstupu git log, žádné rekonstrukce historie z paměti.
|
||||||
|
|
||||||
@@ -1,171 +0,0 @@
|
|||||||
---
|
|
||||||
name: llm-wiki
|
|
||||||
description: >
|
|
||||||
Build and maintain an LLM-curated personal knowledge base — the "LLM Wiki" pattern. Use whenever
|
|
||||||
the user wants to ingest a source (paper, article, transcript, PDF, notes) into a persistent,
|
|
||||||
compounding knowledge base, ask a question against the accumulated notes, lint or audit such a
|
|
||||||
base, or initialize a new one. Applies even when the user doesn't say "wiki" — any time they
|
|
||||||
accumulate textual sources over time and want them organized. A deliberate, standalone store,
|
|
||||||
distinct from agent memory (note / keep / MEMORY.md).
|
|
||||||
---
|
|
||||||
|
|
||||||
# LLM Wiki
|
|
||||||
|
|
||||||
A skill for building and maintaining an LLM-curated knowledge base inside a project, following the pattern Andrej Karpathy described in his April 2026 gist. The wiki is a directory of markdown files that the LLM owns and maintains; the user curates sources and asks questions, and the LLM does the bookkeeping.
|
|
||||||
|
|
||||||
## Nanobot adaptation — read this first
|
|
||||||
|
|
||||||
This skill is ported to run on this nanobot. The generic docs below describe a project-local wiki; the rules here pin it to this nanobot and override anything that conflicts.
|
|
||||||
|
|
||||||
**⚠️ On any add/save/ingest request: capture only.** Write the source to `cml/raw/<slug>.md`, confirm in one short line, and **STOP** — no reads, no scripts, no compiles. Full rules and the escape hatch in "Capture vs compile" below — read it before acting on any such request.
|
|
||||||
|
|
||||||
- **One wiki, fixed location.** This nanobot has exactly one wiki, at `cml/wiki/`, with raw sources at `cml/raw/` — both relative to your working directory (the workspace). Wherever the docs below say `wiki/` or `<project-root>/wiki/`, read `cml/wiki/`; `raw/` means `cml/raw/`.
|
|
||||||
- **Run scripts with `uv run`, never bare `python`.** Always `uv run skills/llm-wiki/scripts/<script>.py …`. The scripts default to `cml/wiki`, so for most you can omit the path argument.
|
|
||||||
- **Bootstrap once:** `uv run skills/llm-wiki/scripts/init_wiki.py . --wiki-dir cml/wiki --raw-dir cml/raw` creates `cml/wiki/` + `cml/raw/`. Idempotent — safe to re-run.
|
|
||||||
- **Separate store — not agent memory.** The wiki is a deliberate, standalone knowledge base of curated sources. It is **not** the agent's memory: keep it distinct from `note`, `keep`, and `MEMORY.md`, and do not fold wiki content into them (or vice versa). The Dream processor must **not** touch `cml/` — it is outside the memory and skills Dream curates. Do not wire the wiki into `MEMORY.md`; its location is documented here.
|
|
||||||
- **Lint is report-only — a lint turn never mutates the wiki.** A lint request ("lint", "what's broken", "clean up the wiki", or a HEARTBEAT lint) means exactly: run `wiki_lint.py` and — if the graph layer exists — `wiki_graph_lint.py`, present the findings as a summary of *proposed* edits, and **STOP the turn**. Forbidden during a lint turn (all of it is fixing, done later in a separate approved turn): creating or editing any page under `cml/wiki/`, writing debug/throwaway scripts, regenerating the graph (`wiki_graph_extract.py`), re-running lint in a loop, updating `index.md` / `log.md`. If you catch yourself editing a page or re-running lint to check your own fix, you are fixing inline — stop. Fixes happen only after the user approves, one category at a time (see the lint workflow).
|
|
||||||
- **Language.** This skill body is English; reply to the user in the user's own language.
|
|
||||||
- **Scope: local PoC.** Single machine, versioned by the git repo running over the workspace. No shared remote, no multi-client sync.
|
|
||||||
|
|
||||||
## Capture vs compile — the background pipeline
|
|
||||||
|
|
||||||
Ingest is split into two phases so the interactive turn stays instant. Full agentic compile takes a minute or more; doing it inline made capture unusable.
|
|
||||||
|
|
||||||
- **Capture (interactive default — instant).** When the user wants to add a source ("save this", "ingest this", "add X to the wiki"), do exactly three things and nothing more:
|
|
||||||
1. Write the source into `cml/raw/<slug>.md` (pick a descriptive `<slug>`). Inline text → write as-is. URL → write the URL as-is (the compile step will fetch it).
|
|
||||||
2. Confirm in **one short line** ("zachyceno — zkompiluju na pozadí").
|
|
||||||
3. **STOP the turn.**
|
|
||||||
Forbidden during a capture turn (all of this is compile, done later in the background): reading `SCHEMA.md` / `index.md` / any wiki page, creating or editing pages under `cml/wiki/`, running `init_wiki.py` / `wiki_lint.py` / `wiki_graph_*` / any script, rebuilding the graph, updating `index.md` or `log.md`. If you catch yourself about to read the schema or write a page, you are doing compile inline — stop and just capture.
|
|
||||||
- **Escape hatch.** Only if the user *explicitly* says "compile now" / "synchronously" / "do it now" / "hned" do you run the full Compile workflow inline in this turn. A normal "add this to my wiki" is **not** an escape hatch — it is capture.
|
|
||||||
- **Compile (drain — background, batched).** A system cron runs `scripts/wiki_compile.py` every minute; when `cml/raw/` has pending sources it invokes this skill with a drain goal. Compile processes **every** pending source in one batch (one index/graph update for many sources), then moves each processed source into `cml/raw/_done/`. This is the existing ingest workflow (below) applied per pending source. **Idempotency:** if `cml/wiki/sources/<slug>.md` already exists for a source, treat it as already compiled — skip re-processing and move the raw file to `cml/raw/_done/`. Always move a source out of `cml/raw/` once handled so the next cron tick doesn't re-process it.
|
|
||||||
- **`cml/raw/` layout.** Regular files directly in `cml/raw/` = the **pending inbox**. `cml/raw/_done/` = processed sources (move here after a successful ingest). `cml/raw/_hard/` = sources held back as ambiguous/conflicting (don't force-compile these; record why in `log.md`). `cml/raw/assets/` = downloaded images, never a source. The pre-check and compile both ignore `_done/`, `_hard/`, and `assets/`.
|
|
||||||
|
|
||||||
## Architecture: three layers, three operations
|
|
||||||
|
|
||||||
The wiki has three layers and three operations. Internalize this vocabulary because the rest of the skill assumes it.
|
|
||||||
|
|
||||||
The three layers are **raw sources** (the user's curated source material — articles, papers, PDFs, transcripts; immutable, the LLM reads but never modifies them), **the wiki** (a directory of LLM-generated markdown pages — entity pages, concept pages, comparisons, summaries; the LLM owns this layer entirely), and **the schema** (a `SCHEMA.md` file at the wiki root that documents the conventions for this particular wiki — page types, naming rules, tag taxonomy, ingest workflow customizations; co-evolved with the user).
|
|
||||||
|
|
||||||
The three operations are **ingest** (a new source arrives; the LLM reads it, writes a summary page, updates relevant entity and concept pages, appends to the log), **query** (the user asks a question; the LLM navigates the wiki via the index, reads the relevant pages, and synthesizes an answer — often filing the answer back as a new page so the exploration compounds), and **lint** (a periodic health check; the LLM scans for contradictions, stale claims, orphan pages, missing concepts, broken links).
|
|
||||||
|
|
||||||
For the canonical write-up of these operations, read `references/architecture.md`. For the step-by-step procedures, read `references/ingest-workflow.md`, `references/query-workflow.md`, and `references/lint-workflow.md` as needed.
|
|
||||||
|
|
||||||
## Graph layer (compiled, optional)
|
|
||||||
|
|
||||||
Pages can carry typed `graph:` metadata in frontmatter. A bundled extractor compiles every page into `wiki/graph/`: `nodes.jsonl`, `edges.jsonl`, `graph.sqlite`, `graph.graphml`. **Markdown is canonical**; the graph is a regenerable index. Pages without `graph:` still appear as nodes (derived from their `type`/`kind`) and contribute low-confidence `mentions` edges from body wikilinks. Typed semantic edges (e.g. `founded`, `proposed`, `depends_on`) require an explicit source and evidence quote — never emit one inferred from training data.
|
|
||||||
|
|
||||||
The conventions for the graph layer (predicate vocabulary, node id format, required fields) live in `wiki/graph/ontology.yaml`. The full reference is `references/graph-workflow.md`. Run the bundled scripts after substantive ingests:
|
|
||||||
|
|
||||||
```bash
|
|
||||||
uv run skills/llm-wiki/scripts/wiki_graph_lint.py cml/wiki/ # check ontology + evidence + alias collisions
|
|
||||||
uv run skills/llm-wiki/scripts/wiki_graph_extract.py cml/wiki/ # rebuild nodes.jsonl, edges.jsonl, graph.sqlite, graph.graphml
|
|
||||||
uv run skills/llm-wiki/scripts/wiki_graph_query.py cml/wiki/ neighbors --node product:konvy
|
|
||||||
```
|
|
||||||
|
|
||||||
If `wiki/graph/ontology.yaml` does not exist, the wiki is pre-graph and you should treat the graph step as a no-op — don't fabricate it.
|
|
||||||
|
|
||||||
## Default project layout
|
|
||||||
|
|
||||||
The wiki is at a fixed location on this nanobot (`cml/wiki/`, `cml/raw/`):
|
|
||||||
|
|
||||||
```
|
|
||||||
<workspace>/
|
|
||||||
├── cml/
|
|
||||||
│ ├── wiki/
|
|
||||||
│ │ ├── SCHEMA.md ← conventions, the "config file" — read this FIRST
|
|
||||||
│ │ ├── index.md ← entry point: catalog of all pages with one-line summaries
|
|
||||||
│ │ ├── log.md ← append-only chronological log of ingests/queries/lints
|
|
||||||
│ │ ├── indexes/ ← (appears once index.md shards) per-category indexes
|
|
||||||
│ │ ├── entities/ ← pages about specific things (people, products, papers, places)
|
|
||||||
│ │ ├── concepts/ ← pages about ideas, methods, frameworks
|
|
||||||
│ │ ├── sources/ ← per-source summary pages (one per ingested source)
|
|
||||||
│ │ └── synthesis/ ← cross-cutting analyses, comparisons, query results filed back
|
|
||||||
│ └── raw/ ← the user's source material (PDFs, .md clippings, images)
|
|
||||||
│ └── assets/ ← downloaded images referenced by raw clippings
|
|
||||||
└── ...
|
|
||||||
```
|
|
||||||
|
|
||||||
## The scalability discipline
|
|
||||||
|
|
||||||
The single biggest failure mode of the LLM Wiki pattern is the wiki itself becoming a context bottleneck. Naive implementations break around a few hundred pages: the LLM either reads too many pages per query or starts hallucinating because it skipped the relevant ones. This skill's design is shaped almost entirely by avoiding that failure. The principles below are non-negotiable; ignoring them is what makes the pattern collapse at scale.
|
|
||||||
|
|
||||||
**Atomic pages.** Every wiki page is about one concept and stays small — soft cap 400 lines or roughly 2,000 words, hard cap 800 lines. When a page outgrows this, split it: extract sub-concepts into their own pages and have the parent link to them. A page that takes up 30% of the context window on its own is a design smell.
|
|
||||||
|
|
||||||
**Index-first navigation.** Never grep or glob the wiki blindly when answering a query. Always read `index.md` (or the relevant sharded index under `indexes/`) first to identify candidate pages, then drill into only those. The index is engineered to be cheap to read — one line per page, no bodies — and it is the cache that makes the whole pattern scalable.
|
|
||||||
|
|
||||||
**Sharded indexes.** When `index.md` itself exceeds ~300 lines or the wiki passes ~150 pages, shard it: move category-specific entries into `indexes/<category>.md` files (e.g. `indexes/entities.md`, `indexes/concepts.md`, `indexes/sources.md`, or finer domain shards), and have the top-level `index.md` become a directory of those shards. Now reading the index is a two-step lookup but each step is bounded.
|
|
||||||
|
|
||||||
**YAML frontmatter on every page.** Every wiki page begins with frontmatter that includes at minimum `type`, `tags`, `sources`, and `updated`. The bundled `wiki_search.py` script can filter on these without reading page bodies. See `references/page-conventions.md`.
|
|
||||||
|
|
||||||
**Surgical edits, not rewrites.** When updating a page (e.g. adding a new cross-reference because a freshly ingested source mentions an existing entity), use `str_replace` to touch only the relevant section. Rewriting whole pages is slow, expensive in tokens, and risks losing prior nuance.
|
|
||||||
|
|
||||||
**Backlink discovery via grep.** To find every page that references a given entity, run `grep -rl "\[\[entity-name\]\]" cml/wiki/` rather than reading pages to look for mentions. The bundled scripts make this easy.
|
|
||||||
|
|
||||||
**Chunked source ingestion.** Large raw sources (long PDFs, book chapters, lengthy transcripts) should be read in chunks during ingest, not loaded whole. The ingest workflow handles this — see `references/ingest-workflow.md`.
|
|
||||||
|
|
||||||
**Search script for large wikis.** Once the wiki passes ~300 pages, plain index lookup may not surface the right pages for fuzzy queries. Use `scripts/wiki_search.py` for BM25-ranked retrieval with optional frontmatter filters. It's a fallback, not the default — index-first is still cheaper when it works.
|
|
||||||
|
|
||||||
**Stats.** `uv run skills/llm-wiki/scripts/wiki_stats.py` gives a quick summary of page count by type and link density — useful for deciding when to shard the index.
|
|
||||||
|
|
||||||
For the full scaling playbook including thresholds and migration steps, read `references/scaling-playbook.md`.
|
|
||||||
|
|
||||||
## Initializing a new wiki
|
|
||||||
|
|
||||||
If the project does not contain a `wiki/` directory (or whatever the user calls theirs), run the bootstrap script:
|
|
||||||
|
|
||||||
```bash
|
|
||||||
uv run skills/llm-wiki/scripts/init_wiki.py . --wiki-dir cml/wiki --raw-dir cml/raw
|
|
||||||
```
|
|
||||||
|
|
||||||
This creates the directory structure, drops in templates for `SCHEMA.md`, `index.md`, and `log.md`, and seeds a starter page convention document. After bootstrapping, briefly walk the user through the schema and ask whether they want to customize anything (e.g. domain-specific page types, custom tags) before the first ingest. The schema is meant to evolve — encourage editing it.
|
|
||||||
|
|
||||||
Do **not** wire the wiki into an agent-memory file (`MEMORY.md` / `AGENTS.md`) on this nanobot — see the Nanobot adaptation rules: the wiki is a separate store and its location is documented in this SKILL.md, which the skill description already surfaces.
|
|
||||||
|
|
||||||
## The ingest workflow (summary)
|
|
||||||
|
|
||||||
**STOP — do not run this for a plain "add this to my wiki" request.** This is the **compile (drain)** step. It runs only from the background cron (the drain goal) or the explicit "compile now" escape hatch. If the user just asked to add/save/ingest a source, you are in *capture* — write to `cml/raw/` and stop (see "Capture vs compile"). The source is already captured in `cml/raw/` when compile runs, so don't re-write it; read it from there.
|
|
||||||
|
|
||||||
The full workflow is in `references/ingest-workflow.md`; what follows is the shape of it. Read the source — chunked if large — and write a single source-summary page in `cml/wiki/sources/`, named after the source slug, with full frontmatter and citations back to the raw file. Then identify which existing entity and concept pages this source touches; for each, surgically update the relevant section using `str_replace` rather than rewriting. Identify any new entities or concepts the source introduces and create new pages for them, linking from related existing pages so they don't become orphans. Update `index.md` (or the relevant shard) with the new pages. Append a single line to `log.md` with the date, operation type, and source title. After the source is fully ingested, move its raw file into `cml/raw/_done/`. When run interactively, discuss the takeaways with the user as a final step — what surprised them, what's worth following up on — and offer to file that discussion back as a synthesis page.
|
|
||||||
|
|
||||||
**Page naming — avoid slug collisions.** A source page and a concept/entity page must not claim the same slug, or their `[[wikilinks]]` collide (e.g. a paper *and* the concept it introduces both wanting `rotary-position-embedding.md`). Name **concept/entity pages after the short name of the idea** (`rope.md`, `transformer.md`), and **source pages after the source's own slug** (the title/filename of the raw source). If two pages still resolve to the same slug, suffix one to disambiguate.
|
|
||||||
|
|
||||||
## The query workflow (summary)
|
|
||||||
|
|
||||||
Full version in `references/query-workflow.md`. To answer a query against the wiki: read `index.md` (or the relevant shard) first; identify candidate pages from one-line summaries; read those pages (and any backlinks they list that look relevant); synthesize the answer with `[[wikilink]]` citations to the pages you used; offer to file the synthesized answer back into `wiki/synthesis/` so future queries benefit. If the index doesn't surface good candidates, fall back to `uv run skills/llm-wiki/scripts/wiki_search.py "query terms"` for ranked retrieval. If the wiki appears to lack coverage of the topic, say so plainly rather than confabulating — flag it as a candidate ingest target.
|
|
||||||
|
|
||||||
## The lint workflow (summary)
|
|
||||||
|
|
||||||
**STOP — a lint turn reports, it does not fix.** See the report-only gate in "Nanobot adaptation". Run the scripts, present the findings, stop. The scripts are fast (sub-second on a small wiki); if a lint turn runs long, you have wrongly slipped into fixing.
|
|
||||||
|
|
||||||
Full version in `references/lint-workflow.md`. Lint is best run on a cadence (after every N ingests or weekly), not on every operation. Run `uv run skills/llm-wiki/scripts/wiki_lint.py` for structural issues (orphan pages, broken `[[wikilinks]]`, oversized pages, missing/malformed frontmatter, stale `updated` dates) and — if the graph layer exists — `uv run skills/llm-wiki/scripts/wiki_graph_lint.py` for typed-edge issues. For the semantic checks that need an LLM (contradictions with older claims, concepts mentioned but lacking a page, coverage gaps) read at most the ~10 most-recently-updated pages — no blind globbing. Present everything as proposed edits for the user to approve; never apply them in the lint turn — the wiki is the user's, and silent rewrites erode trust.
|
|
||||||
|
|
||||||
**When the user approves fixes (a later turn):** fix in bounded batches, one category at a time. Do **not** re-read the lint script to reverse-engineer it, and do **not** loop edit↔lint — run lint once at the end to confirm, and if findings remain, report them and ask rather than continuing blind. Graph-lint findings are interdependent (fixing one edge can create an orphan); clearing a large backlog is its own task, not part of a lint turn.
|
|
||||||
|
|
||||||
## Failure modes to guard against
|
|
||||||
|
|
||||||
- **Silent corruption:** every wiki claim must carry a `sources:` frontmatter entry pointing back to the raw file. When in doubt during ingest, hedge ("the source claims X") rather than asserting.
|
|
||||||
- **Wiki-reads-its-own-output drift:** during ingest, when updating an existing page, re-read the relevant raw source for the existing claim before merging — don't take the wiki's word for what the source said.
|
|
||||||
|
|
||||||
## Reference files
|
|
||||||
|
|
||||||
The reference files are the source of truth for the detailed procedures. Read them when the relevant operation is happening, not preemptively.
|
|
||||||
|
|
||||||
- `references/architecture.md` — the three layers and three operations explained in depth, with examples of page formats and the rationale behind each design choice
|
|
||||||
- `references/ingest-workflow.md` — the step-by-step ingest procedure including chunked reading for large sources and the per-page-type templates
|
|
||||||
- `references/query-workflow.md` — navigation patterns from index → page → backlinks, when to fall back to the search script, and how to file answers back as synthesis pages
|
|
||||||
- `references/lint-workflow.md` — what to check, how to present findings, and the cadence
|
|
||||||
- `references/page-conventions.md` — frontmatter schema, page naming, link syntax, page-type definitions, sizing rules
|
|
||||||
- `references/scaling-playbook.md` — thresholds at which to shard the index, when to introduce the search script, signals that the wiki has outgrown its current conventions
|
|
||||||
- `references/graph-workflow.md` — the optional graph layer: ontology, frontmatter schema, when to add typed edges vs plain wikilinks, and the extract/lint/query flow
|
|
||||||
|
|
||||||
## Templates
|
|
||||||
|
|
||||||
The templates in `assets/` are starting points — they get copied into the user's wiki on bootstrap and then evolve under the user's editing.
|
|
||||||
|
|
||||||
- `assets/SCHEMA.md.template` — the canonical schema document for a new wiki
|
|
||||||
- `assets/index.md.template` — the empty index file
|
|
||||||
- `assets/log.md.template` — the empty log file
|
|
||||||
- `assets/page.md.template` — a generic wiki page with the frontmatter scaffold
|
|
||||||
- `assets/ontology.yaml.template` — starter graph ontology copied to `wiki/graph/ontology.yaml`
|
|
||||||
- `assets/graph_README.md.template` — explainer for `wiki/graph/` (canonical vs generated files)
|
|
||||||
- `assets/graph_gitignore.template` — `.gitignore` for `wiki/graph/` (ignores `graph.sqlite` and `graph.graphml` by default)
|
|
||||||
@@ -1,113 +0,0 @@
|
|||||||
# Wiki Schema
|
|
||||||
|
|
||||||
This file is the configuration for this wiki. It documents the conventions, page types, tag taxonomy, and any workflow customizations. The LLM reads this first when entering the wiki, and its conventions override the defaults documented in the `llm-wiki` skill.
|
|
||||||
|
|
||||||
This file is **co-evolved with the user**. When the LLM notices a recurring pattern in your edits or feedback that isn't here, it will propose adding it. When something here stops fitting, prune it.
|
|
||||||
|
|
||||||
## Wiki location
|
|
||||||
|
|
||||||
- Wiki root: `wiki/`
|
|
||||||
- Raw sources: `raw/`
|
|
||||||
- Asset/image storage: `raw/assets/`
|
|
||||||
|
|
||||||
## Page types
|
|
||||||
|
|
||||||
This wiki uses these page types, each with a dedicated subdirectory:
|
|
||||||
|
|
||||||
- `source` (in `wiki/sources/`) — one summary page per ingested source.
|
|
||||||
- `entity` (in `wiki/entities/`) — pages about specific things: people, papers, products, places, organizations.
|
|
||||||
- `concept` (in `wiki/concepts/`) — pages about ideas, methods, frameworks, abstractions.
|
|
||||||
- `synthesis` (in `wiki/synthesis/`) — cross-cutting analyses, comparisons, query answers filed back.
|
|
||||||
|
|
||||||
Add additional types here as the wiki evolves.
|
|
||||||
|
|
||||||
## Tag taxonomy
|
|
||||||
|
|
||||||
(Empty initially. Add tags here as you adopt them, with one-line descriptions. Keep this list small and disciplined — a wiki with 200 tags has effectively no tags.)
|
|
||||||
|
|
||||||
Example structure:
|
|
||||||
- `methodology` — pages about research or analytical methods.
|
|
||||||
- `open-question` — pages or sections that flag unresolved questions.
|
|
||||||
- `contested` — pages where sources contradict.
|
|
||||||
|
|
||||||
## Page sizing
|
|
||||||
|
|
||||||
- Soft cap: 400 lines / ~2,000 words. Consider splitting beyond this.
|
|
||||||
- Hard cap: 800 lines. Must split.
|
|
||||||
|
|
||||||
## Frontmatter requirements
|
|
||||||
|
|
||||||
Every page must have:
|
|
||||||
- `type`
|
|
||||||
- `title`
|
|
||||||
- `tags`
|
|
||||||
- `created`
|
|
||||||
- `updated`
|
|
||||||
|
|
||||||
Plus type-specific:
|
|
||||||
- `source` pages: `authors`, `url` (if applicable), `raw`, `ingested`
|
|
||||||
- Non-source pages: `sources` listing the source-summary pages drawn from
|
|
||||||
|
|
||||||
## Optional graph metadata
|
|
||||||
|
|
||||||
Pages may declare typed graph metadata under a top-level `graph:` key. This is the source of truth for the compiled knowledge graph under `wiki/graph/`. Markdown remains canonical; the graph is a regenerable index. Pages without `graph:` still appear as nodes (derived from `type`/`kind`) and still contribute `mentions` edges from body `[[wikilinks]]`.
|
|
||||||
|
|
||||||
```yaml
|
|
||||||
graph:
|
|
||||||
node_id: person:praney-behl # optional; default <node_type>:<slug>
|
|
||||||
node_type: person # optional; default mapped from type/kind via ontology
|
|
||||||
canonical: true # mark as canonical when multiple slugs alias the same entity
|
|
||||||
aliases: [Praney, praney@example.com]
|
|
||||||
relationships:
|
|
||||||
- predicate: founded
|
|
||||||
object: company:seedblocks
|
|
||||||
source: praney-founder-context-dump # source-page slug
|
|
||||||
evidence: "Solo technical founder and sole director..."
|
|
||||||
confidence: high # high | medium | low
|
|
||||||
status: current # current | historical | proposed | disputed | superseded
|
|
||||||
# optional:
|
|
||||||
# valid_from: 2025-01-15
|
|
||||||
# valid_to: 2026-03-01
|
|
||||||
# notes: "..."
|
|
||||||
# raw_ref: "raw/founder-dump.md#L42"
|
|
||||||
# contradicts: edge-id-or-source-slug
|
|
||||||
# supersedes: edge-id-or-source-slug
|
|
||||||
```
|
|
||||||
|
|
||||||
Required fields on every relationship: `predicate`, `object`, `source`, `evidence`, `confidence`, `status`. Predicates and the subject/object types they accept are declared in `wiki/graph/ontology.yaml`. Typed semantic edges must be supported by an explicit source — never emit one inferred from training data alone.
|
|
||||||
|
|
||||||
## Index structure
|
|
||||||
|
|
||||||
(Update this section when sharding.)
|
|
||||||
|
|
||||||
Currently flat: a single `wiki/index.md` listing all pages.
|
|
||||||
|
|
||||||
When the wiki passes ~150 pages or `index.md` exceeds 300 lines, shard into `wiki/indexes/<type>.md` and update this section.
|
|
||||||
|
|
||||||
## Graph layer
|
|
||||||
|
|
||||||
The wiki has an optional compiled graph layer under `wiki/graph/`:
|
|
||||||
|
|
||||||
- `wiki/graph/ontology.yaml` — declares node types and predicates. **Tracked.** Edit this when you introduce new predicates or domain types.
|
|
||||||
- `wiki/graph/nodes.jsonl`, `wiki/graph/edges.jsonl` — generated. Track in git only if you want graph diffs in PRs.
|
|
||||||
- `wiki/graph/graph.sqlite` — generated. Gitignored by default.
|
|
||||||
- `wiki/graph/graph.graphml` — generated. Track only if you want to diff it.
|
|
||||||
|
|
||||||
Generation is reproducible from markdown via `scripts/wiki_graph_extract.py`. The graph can be deleted at any time and rebuilt without losing knowledge — markdown is canonical.
|
|
||||||
|
|
||||||
## Workflow customizations
|
|
||||||
|
|
||||||
(Empty initially. Document any deviations from the default ingest/query/lint workflows here.)
|
|
||||||
|
|
||||||
## User preferences
|
|
||||||
|
|
||||||
(Empty initially. As the user expresses style preferences — "always include a 'Why this matters' section on concept pages", "never use bullet lists in summaries", "prefer comparative tables for synthesis pages" — capture them here so they persist across sessions.)
|
|
||||||
|
|
||||||
## Lint cadence
|
|
||||||
|
|
||||||
- Structural lint: after every 5 ingests.
|
|
||||||
- Semantic lint: weekly or after every 20 ingests.
|
|
||||||
- Gap-finding: monthly.
|
|
||||||
- Graph lint + extract: after every ingest that adds typed `graph.relationships`.
|
|
||||||
|
|
||||||
Adjust based on the wiki's growth rate.
|
|
||||||
@@ -1,32 +0,0 @@
|
|||||||
# Wiki Graph Layer
|
|
||||||
|
|
||||||
This directory holds the compiled knowledge graph derived from the markdown
|
|
||||||
wiki. **Markdown is canonical.** Everything here can be deleted and rebuilt
|
|
||||||
without losing knowledge:
|
|
||||||
|
|
||||||
```bash
|
|
||||||
python scripts/wiki_graph_extract.py wiki/ --out wiki/graph
|
|
||||||
```
|
|
||||||
|
|
||||||
## Files
|
|
||||||
|
|
||||||
| File | Purpose | Tracking |
|
|
||||||
|------|---------|----------|
|
|
||||||
| `ontology.yaml` | Declares node types and predicates the graph recognises. The contract `wiki_graph_lint.py` validates against. | **Tracked. Edit by hand.** |
|
|
||||||
| `nodes.jsonl` | One JSON object per node, sorted by id. | Generated. Track if you want graph diffs in PRs; otherwise gitignore. |
|
|
||||||
| `edges.jsonl` | One JSON object per edge, sorted by id. Includes typed semantic edges, `mentions`, `sourced_from`, and `summarizes_raw`. | Generated. Same trade-off as `nodes.jsonl`. |
|
|
||||||
| `graph.sqlite` | Queryable index used by `wiki_graph_query.py`. Schema: `nodes`, `aliases`, `edges`. | Generated. **Gitignored** — rebuild on demand. |
|
|
||||||
| `graph.graphml` | GraphML export for tools like Gephi or yEd. | Generated. Gitignored by default. |
|
|
||||||
|
|
||||||
## Workflow
|
|
||||||
|
|
||||||
1. Author or edit a wiki page. Add typed `graph.relationships` only when an explicit source supports them.
|
|
||||||
2. Run `python scripts/wiki_graph_lint.py wiki/` — catches unknown predicates, broken object references, missing evidence, alias collisions.
|
|
||||||
3. Run `python scripts/wiki_graph_extract.py wiki/ --out wiki/graph` — regenerates the artifacts above.
|
|
||||||
4. Query with `python scripts/wiki_graph_query.py wiki/ neighbors --node product:konvy` (or `edges`, `path`, `facts`).
|
|
||||||
|
|
||||||
## Anti-patterns
|
|
||||||
|
|
||||||
- **Hand-editing `nodes.jsonl` / `edges.jsonl` / `graph.sqlite`.** Edit the markdown; regenerate.
|
|
||||||
- **Treating graph rows as evidence.** They accelerate navigation. For high-stakes claims, follow the edge's `source` and `evidence` fields back to the wiki page and the raw source.
|
|
||||||
- **Adding typed edges the source doesn't support.** Use a normal `[[wikilink]]` instead — the `mentions` edge captures the connection without overclaiming.
|
|
||||||
@@ -1,2 +0,0 @@
|
|||||||
graph.sqlite
|
|
||||||
graph.graphml
|
|
||||||
@@ -1,25 +0,0 @@
|
|||||||
# Wiki Index
|
|
||||||
|
|
||||||
The catalog of all pages in this wiki. Each entry: a wikilink to the page and a one-line summary. The LLM reads this first when answering queries to identify candidate pages.
|
|
||||||
|
|
||||||
Keep summaries tight — one line each. The index is engineered to be cheap to read; a fat index defeats its purpose.
|
|
||||||
|
|
||||||
When this file exceeds ~300 lines or the wiki passes ~150 pages, shard into `wiki/indexes/<type>.md` and replace this file with a directory of shards. See the `scaling-playbook.md` reference in the `llm-wiki` skill for the migration procedure.
|
|
||||||
|
|
||||||
---
|
|
||||||
|
|
||||||
## Sources
|
|
||||||
|
|
||||||
(populated as sources are ingested)
|
|
||||||
|
|
||||||
## Entities
|
|
||||||
|
|
||||||
(populated as entity pages are created)
|
|
||||||
|
|
||||||
## Concepts
|
|
||||||
|
|
||||||
(populated as concept pages are created)
|
|
||||||
|
|
||||||
## Synthesis
|
|
||||||
|
|
||||||
(populated as query answers are filed back)
|
|
||||||
@@ -1,12 +0,0 @@
|
|||||||
# Wiki Log
|
|
||||||
|
|
||||||
Append-only chronological record of operations on the wiki. Each entry begins with `## [YYYY-MM-DD] <op> | <description>` so it's parseable with `grep "^## \[" log.md | tail -N`.
|
|
||||||
|
|
||||||
Operations:
|
|
||||||
- `ingest` — a source was processed into the wiki.
|
|
||||||
- `query` — a question was answered against the wiki (typically only logged when the answer was filed back as synthesis).
|
|
||||||
- `lint` — a health check was run.
|
|
||||||
- `schema` — the schema was modified.
|
|
||||||
- `shard` — an index was sharded.
|
|
||||||
|
|
||||||
---
|
|
||||||
@@ -1,123 +0,0 @@
|
|||||||
# Wiki Graph Ontology
|
|
||||||
#
|
|
||||||
# Declares the node types and predicates that the compiled graph layer
|
|
||||||
# (wiki/graph/) recognises. Edit this file when you introduce a new
|
|
||||||
# domain-specific predicate or node type — wiki_graph_lint.py reads it
|
|
||||||
# to validate every typed edge declared in page frontmatter.
|
|
||||||
#
|
|
||||||
# Markdown remains canonical. This file is just the contract that makes
|
|
||||||
# the graph layer machine-checkable.
|
|
||||||
|
|
||||||
node_types:
|
|
||||||
person:
|
|
||||||
maps_from:
|
|
||||||
type: entity
|
|
||||||
kind: person
|
|
||||||
company:
|
|
||||||
maps_from:
|
|
||||||
type: entity
|
|
||||||
kind: company
|
|
||||||
product:
|
|
||||||
maps_from:
|
|
||||||
type: entity
|
|
||||||
kind: product
|
|
||||||
paper:
|
|
||||||
maps_from:
|
|
||||||
type: entity
|
|
||||||
kind: paper
|
|
||||||
place:
|
|
||||||
maps_from:
|
|
||||||
type: entity
|
|
||||||
kind: place
|
|
||||||
organization:
|
|
||||||
maps_from:
|
|
||||||
type: entity
|
|
||||||
kind: organization
|
|
||||||
concept:
|
|
||||||
maps_from:
|
|
||||||
type: concept
|
|
||||||
source:
|
|
||||||
maps_from:
|
|
||||||
type: source
|
|
||||||
synthesis:
|
|
||||||
maps_from:
|
|
||||||
type: synthesis
|
|
||||||
decision:
|
|
||||||
explicit_only: true
|
|
||||||
claim:
|
|
||||||
explicit_only: true
|
|
||||||
raw:
|
|
||||||
explicit_only: true
|
|
||||||
|
|
||||||
predicates:
|
|
||||||
# --- Implicit predicates emitted by the extractor. ---
|
|
||||||
mentions:
|
|
||||||
subject_types: ["*"]
|
|
||||||
object_types: ["*"]
|
|
||||||
requires_evidence: false
|
|
||||||
description: |
|
|
||||||
Low-specificity edge derived from body wikilinks. Use it for
|
|
||||||
navigation, not as evidence of a typed relationship.
|
|
||||||
sourced_from:
|
|
||||||
subject_types: ["*"]
|
|
||||||
object_types: [source]
|
|
||||||
requires_evidence: false
|
|
||||||
description: |
|
|
||||||
Derived from each non-source page's frontmatter `sources:` list.
|
|
||||||
summarizes_raw:
|
|
||||||
subject_types: [source]
|
|
||||||
object_types: ["*"]
|
|
||||||
requires_evidence: false
|
|
||||||
description: |
|
|
||||||
Derived from a source page's frontmatter `raw:` field. Object is
|
|
||||||
the raw file path string, not a wiki node id.
|
|
||||||
|
|
||||||
# --- Typed semantic predicates. Add domain-specific ones below. ---
|
|
||||||
founded:
|
|
||||||
subject_types: [person]
|
|
||||||
object_types: [company, organization]
|
|
||||||
requires_evidence: true
|
|
||||||
owns:
|
|
||||||
subject_types: [person, company, organization]
|
|
||||||
object_types: [company, product, organization]
|
|
||||||
requires_evidence: true
|
|
||||||
contains_product:
|
|
||||||
subject_types: [company, organization]
|
|
||||||
object_types: [product]
|
|
||||||
requires_evidence: true
|
|
||||||
works_on:
|
|
||||||
subject_types: [person]
|
|
||||||
object_types: [product, concept]
|
|
||||||
requires_evidence: true
|
|
||||||
chose:
|
|
||||||
subject_types: [person, company, organization]
|
|
||||||
object_types: [product, concept]
|
|
||||||
requires_evidence: true
|
|
||||||
proposed:
|
|
||||||
subject_types: [person]
|
|
||||||
object_types: [decision, claim]
|
|
||||||
requires_evidence: true
|
|
||||||
competes_with:
|
|
||||||
subject_types: [product, company, organization]
|
|
||||||
object_types: [product, company, organization]
|
|
||||||
requires_evidence: true
|
|
||||||
depends_on:
|
|
||||||
subject_types: [product, concept]
|
|
||||||
object_types: [product, concept]
|
|
||||||
requires_evidence: true
|
|
||||||
authored:
|
|
||||||
subject_types: [person, organization]
|
|
||||||
object_types: [paper, source]
|
|
||||||
requires_evidence: true
|
|
||||||
cites:
|
|
||||||
subject_types: [paper, source, synthesis]
|
|
||||||
object_types: [paper, source]
|
|
||||||
requires_evidence: true
|
|
||||||
contradicts:
|
|
||||||
subject_types: [claim, source, synthesis]
|
|
||||||
object_types: [claim, source, synthesis]
|
|
||||||
requires_evidence: true
|
|
||||||
supersedes:
|
|
||||||
subject_types: [claim, source, decision]
|
|
||||||
object_types: [claim, source, decision]
|
|
||||||
requires_evidence: true
|
|
||||||
@@ -1,26 +0,0 @@
|
|||||||
---
|
|
||||||
type: <source|entity|concept|synthesis>
|
|
||||||
title: ""
|
|
||||||
tags: []
|
|
||||||
sources: []
|
|
||||||
created: YYYY-MM-DD
|
|
||||||
updated: YYYY-MM-DD
|
|
||||||
---
|
|
||||||
|
|
||||||
# Title
|
|
||||||
|
|
||||||
Lead paragraph: a clear, encyclopedic definition or framing of what this page is about. Should answer "what is this and why does it matter" in one or two sentences.
|
|
||||||
|
|
||||||
## Section 1
|
|
||||||
|
|
||||||
Body content. Use `[[wikilinks]]` liberally to cross-reference other pages. (Frontmatter `sources:` list above uses bare slugs; only the body uses double-bracket wikilinks.)
|
|
||||||
|
|
||||||
## Section 2
|
|
||||||
|
|
||||||
More body content. Hedge claims that aren't yet corroborated by multiple sources ("Source X claims Y, though this is not yet corroborated by other sources in the wiki").
|
|
||||||
|
|
||||||
## Where this fits
|
|
||||||
|
|
||||||
(For source pages.) List the entity and concept pages this source touches:
|
|
||||||
- [[entity-page-1]]
|
|
||||||
- [[concept-page-1]]
|
|
||||||
@@ -1,76 +0,0 @@
|
|||||||
# Architecture
|
|
||||||
|
|
||||||
This document explains the three-layer / three-operation architecture in detail. The main `SKILL.md` summarises it; this is the reference you reach for when you need to know *why* a design decision exists or how to handle an edge case.
|
|
||||||
|
|
||||||
## The three layers
|
|
||||||
|
|
||||||
### Raw sources
|
|
||||||
|
|
||||||
Raw sources are the user's curated input material. They live in `cml/raw/` (or wherever the project's `SCHEMA.md` declares). They are **immutable** — the LLM reads from them but never modifies them. This immutability is load-bearing: it means the wiki can always be re-derived from the raw sources if it gets corrupted, and it gives the user a stable ground truth they can audit independently.
|
|
||||||
|
|
||||||
What goes in `cml/raw/`: PDFs, web articles converted to markdown (the Obsidian Web Clipper is one popular path), transcripts, code repos, dataset descriptions, screenshots, hand-typed notes the user wants the wiki to incorporate. What does *not* go in `cml/raw/`: anything the LLM generated — that all belongs in the wiki layer.
|
|
||||||
|
|
||||||
A useful convention is one source per file (or per directory if the source has multiple pieces, e.g. a paper plus its appendix), with a slugified filename that ends up matching the wiki's source-summary page name. This makes the back-pointer from the wiki to the raw source trivial.
|
|
||||||
|
|
||||||
### The wiki
|
|
||||||
|
|
||||||
The wiki is a directory of LLM-generated markdown files. The LLM owns this layer entirely — it creates pages, updates them when new sources arrive, maintains cross-references, and keeps everything consistent. The user reads it (typically through Obsidian, but any markdown viewer works); the LLM writes it.
|
|
||||||
|
|
||||||
The wiki directory is conventionally split into subdirectories by **page type**:
|
|
||||||
|
|
||||||
- `cml/wiki/sources/` — one summary page per ingested source. Captures what the source said, in the LLM's words, with a citation back to the raw file. These are append-mostly: you write one when you ingest a source and rarely modify it after.
|
|
||||||
- `cml/wiki/entities/` — pages about specific things: people, products, papers, places, companies, events. Anything that has a proper noun or could plausibly be a Wikipedia article subject. These accumulate updates as new sources mention the entity.
|
|
||||||
- `cml/wiki/concepts/` — pages about ideas, methods, frameworks, abstractions. These are the most heavily cross-referenced pages and the ones most likely to evolve as understanding deepens.
|
|
||||||
- `cml/wiki/synthesis/` — cross-cutting analyses, comparisons, query answers filed back. This is where exploration compounds: a comparison the user asked for becomes a page the next query can build on.
|
|
||||||
|
|
||||||
`SCHEMA.md` may declare additional types (e.g. `cml/wiki/decisions/` for an engineering team, `cml/wiki/characters/` for a fan wiki, `cml/wiki/experiments/` for a research lab). Adding a new type costs nothing — make the directory, document it in the schema, update the index template.
|
|
||||||
|
|
||||||
### The schema
|
|
||||||
|
|
||||||
`SCHEMA.md` lives at the wiki root and is **the configuration file** that turns a generic LLM into a disciplined wiki maintainer for *this specific* knowledge base. It documents the page types in use, the tag taxonomy, the naming conventions, any custom workflow steps, and the user's stylistic preferences (e.g. "always include a 'Why this matters' section on concept pages", "never use bullet lists in summaries").
|
|
||||||
|
|
||||||
The schema is **co-evolved** with the user. On bootstrap it starts from the default template in `assets/SCHEMA.md.template`, but every user will customize it as they discover what fits their domain. When you notice a recurring pattern in the user's edits or feedback that isn't in the schema, propose adding it. When the schema starts contradicting itself or growing unwieldy, propose pruning it.
|
|
||||||
|
|
||||||
The schema is the first file you read when entering an existing wiki. Its conventions override the defaults documented in the skill.
|
|
||||||
|
|
||||||
### The graph (optional, compiled)
|
|
||||||
|
|
||||||
The wiki may carry a fourth layer at `cml/wiki/graph/`: a compiled, queryable view of the typed `graph:` metadata in page frontmatter and the body wikilinks. It contains a hand-edited `ontology.yaml` (the contract: which node types and predicates exist) plus generated artifacts (`nodes.jsonl`, `edges.jsonl`, `graph.sqlite`, `graph.graphml`) produced by `wiki_graph_extract.py`. **Markdown is canonical**; the graph can be deleted and regenerated from the markdown without losing knowledge.
|
|
||||||
|
|
||||||
Its purpose is to make typed, provenance-backed relationships machine-queryable — "who founded what", "what does Konvy depend on", "shortest path from A to B" — without giving up the editability and human-legibility of markdown. Typed edges require an explicit `source` (a source-page slug) and `evidence` quote; the extractor never invents them. Plain `[[wikilinks]]` in the body produce low-confidence `mentions` edges, which are useful for navigation but not for evidence.
|
|
||||||
|
|
||||||
Use the graph layer when the user's questions are predominantly relational and the cost of maintaining typed metadata is paying for itself. Skip it for purely textual wikis. Full reference: `graph-workflow.md`.
|
|
||||||
|
|
||||||
## The three operations
|
|
||||||
|
|
||||||
### Ingest
|
|
||||||
|
|
||||||
A new source has arrived. The user has dropped a file in `cml/raw/` (or pasted content and asked you to file it). The job is to integrate this source into the wiki such that future queries can benefit from it.
|
|
||||||
|
|
||||||
The shape of an ingest: read the source (chunked if large), discuss the key takeaways with the user briefly, write a summary page in `cml/wiki/sources/`, identify which existing pages are touched, surgically update those pages, create new pages for any new entities or concepts, update the relevant index, and append to the log.
|
|
||||||
|
|
||||||
The temptation to skip the discussion step is strong, but resist it — the user's reaction to the takeaways often reveals what should be emphasized in the wiki versus left out. Ingest is not a batch import; it's a collaborative reading.
|
|
||||||
|
|
||||||
For the full procedure, see `ingest-workflow.md`.
|
|
||||||
|
|
||||||
### Query
|
|
||||||
|
|
||||||
The user asks a question. The job is to answer it from the wiki, with citations, and to file the answer back if it represents new synthesis.
|
|
||||||
|
|
||||||
The shape of a query: read the index to identify candidate pages; read those pages; if needed, follow `[[wikilinks]]` from those pages or grep for backlinks; synthesize the answer with `[[wikilink]]` citations; offer to file the answer into `cml/wiki/synthesis/` if it's substantive enough to be worth keeping.
|
|
||||||
|
|
||||||
The compounding effect of the wiki only works if good answers get filed back. A comparison you generated, a connection you discovered, an analysis you produced — these should not evaporate into chat history. Default to offering to file; let the user decline if the answer was too trivial or too transient.
|
|
||||||
|
|
||||||
For the full procedure, see `query-workflow.md`.
|
|
||||||
|
|
||||||
### Lint
|
|
||||||
|
|
||||||
Periodic health check. The job is to find structural and semantic problems before they compound.
|
|
||||||
|
|
||||||
Structural problems are mechanical and the bundled `wiki_lint.py` script catches them: orphan pages with no inbound links, broken `[[wikilinks]]` to nonexistent pages, oversized pages that need splitting, missing or malformed frontmatter, suspicious staleness (a page that hasn't been updated despite many recent ingests touching its topic).
|
|
||||||
|
|
||||||
Semantic problems need the LLM: contradictions between pages, claims that newer sources have superseded, concepts mentioned in many pages but lacking their own page, cross-references that should exist but don't, knowledge gaps the user might want to fill.
|
|
||||||
|
|
||||||
Lint findings are presented as proposed edits, not silent rewrites. The user approves changes. This is essential for trust — a wiki that mutates under the user is not a wiki the user can rely on.
|
|
||||||
|
|
||||||
For the full procedure, see `lint-workflow.md`.
|
|
||||||
@@ -1,126 +0,0 @@
|
|||||||
# Graph Workflow
|
|
||||||
|
|
||||||
The graph layer is the optional **compiled index** over the markdown wiki. It does not replace the wiki — it sits alongside it under `cml/wiki/graph/` and is reproducible from the markdown at any time. The point is to make typed, provenance-backed relationships machine-queryable while keeping markdown canonical.
|
|
||||||
|
|
||||||
If `cml/wiki/graph/ontology.yaml` is absent, the wiki is pre-graph: don't run extract/lint/query and don't fabricate ontology files. Either propose adding the layer, or proceed without it.
|
|
||||||
|
|
||||||
## What the graph captures
|
|
||||||
|
|
||||||
Three classes of edge come out of an extract:
|
|
||||||
|
|
||||||
1. **Typed semantic edges** declared in a page's `graph.relationships[]` frontmatter. Examples: `founded`, `proposed`, `depends_on`. Each one carries an explicit `source` (a source-page slug), an `evidence` quote, a `confidence` (high/medium/low), and a `status` (current/historical/proposed/disputed/superseded). The extractor never invents these.
|
|
||||||
2. **`mentions` edges** — one per body `[[wikilink]]` (deduplicated per page). Confidence is `low`; they accelerate navigation but should not be cited as evidence of a typed relationship.
|
|
||||||
3. **`sourced_from`** edges — one per slug in a non-source page's frontmatter `sources:` list, pointing at the source page. **`summarizes_raw`** edges — one per source page's `raw:` field, with the raw file path as the (string-literal) object.
|
|
||||||
|
|
||||||
## Frontmatter schema
|
|
||||||
|
|
||||||
```yaml
|
|
||||||
graph:
|
|
||||||
node_id: person:praney-behl # optional; default <node_type>:<slug>
|
|
||||||
node_type: person # optional; default mapped from type/kind via ontology
|
|
||||||
canonical: true # mark canonical when multiple slugs alias the same entity
|
|
||||||
aliases: [Praney, praney@example.com]
|
|
||||||
relationships:
|
|
||||||
- predicate: founded
|
|
||||||
object: company:seedblocks
|
|
||||||
source: praney-founder-context-dump
|
|
||||||
evidence: "Solo technical founder and sole director..."
|
|
||||||
confidence: high
|
|
||||||
status: current
|
|
||||||
# optional:
|
|
||||||
# valid_from: 2025-01-15
|
|
||||||
# valid_to: 2026-03-01
|
|
||||||
# notes: "..."
|
|
||||||
# raw_ref: "cml/raw/founder-dump.md#L42"
|
|
||||||
# contradicts: <node-id-or-edge-id>
|
|
||||||
# supersedes: <node-id-or-edge-id>
|
|
||||||
```
|
|
||||||
|
|
||||||
Required relationship fields: `predicate`, `object`, `source`, `evidence`, `confidence`, `status`.
|
|
||||||
|
|
||||||
`node_id` format is `<node_type>:<slug>`. The default is derived from the page's `type`/`kind` via the ontology's `maps_from` block. `decision`, `claim`, and `raw` are explicit-only — they don't have wiki pages, so they only show up as edge objects. If a typed edge points at one of these, the `wiki_graph_lint.py` flag for "broken object reference" will fire until either (a) you create a page for it, or (b) you add it to the ontology with `explicit_only: true` and accept that lint will continue to flag the reference.
|
|
||||||
|
|
||||||
## The ontology
|
|
||||||
|
|
||||||
`cml/wiki/graph/ontology.yaml` is the contract. It declares:
|
|
||||||
|
|
||||||
- `node_types[*].maps_from` — how page `type`/`kind` projects onto a node type.
|
|
||||||
- `predicates[*]` — the allowed predicates, each with `subject_types`, `object_types`, and `requires_evidence`. `"*"` is a wildcard for either side.
|
|
||||||
|
|
||||||
Edit the ontology when you need a new domain predicate. Re-run `wiki_graph_lint.py` to validate; existing typed edges will be caught if they no longer match.
|
|
||||||
|
|
||||||
## When to add a typed edge vs a plain `[[wikilink]]`
|
|
||||||
|
|
||||||
Add a typed edge when:
|
|
||||||
|
|
||||||
- A specific source explicitly states the relationship.
|
|
||||||
- You can quote a snippet of evidence.
|
|
||||||
- The predicate is meaningful for downstream queries ("who founded what", "what did Stephanie propose").
|
|
||||||
|
|
||||||
Use a plain `[[wikilink]]` when:
|
|
||||||
|
|
||||||
- The relationship is implicit, atmospheric, or you're hedging.
|
|
||||||
- You cannot pin the claim to a single source quote.
|
|
||||||
- The predicate would be `mentions` anyway.
|
|
||||||
|
|
||||||
When in doubt, write the wikilink and skip the typed edge. The lint surfaces missing evidence; it does not punish under-claiming.
|
|
||||||
|
|
||||||
## Extract / lint / query loop
|
|
||||||
|
|
||||||
```bash
|
|
||||||
# Validate the typed metadata first; lint is conservative, never edits.
|
|
||||||
uv run skills/llm-wiki/scripts/wiki_graph_lint.py cml/wiki/
|
|
||||||
|
|
||||||
# Compile to nodes.jsonl, edges.jsonl, graph.sqlite, graph.graphml.
|
|
||||||
uv run skills/llm-wiki/scripts/wiki_graph_extract.py cml/wiki/
|
|
||||||
|
|
||||||
# Navigate.
|
|
||||||
uv run skills/llm-wiki/scripts/wiki_graph_query.py cml/wiki/ neighbors --node product:konvy
|
|
||||||
uv run skills/llm-wiki/scripts/wiki_graph_query.py cml/wiki/ edges --subject person:stephanie-emmanouel
|
|
||||||
uv run skills/llm-wiki/scripts/wiki_graph_query.py cml/wiki/ path --from person:praney-behl --to product:konvy
|
|
||||||
uv run skills/llm-wiki/scripts/wiki_graph_query.py cml/wiki/ facts --about product:konvy
|
|
||||||
```
|
|
||||||
|
|
||||||
`--json` works on both lint and query commands.
|
|
||||||
|
|
||||||
## Ingest workflow integration
|
|
||||||
|
|
||||||
After Step 6 of the standard ingest workflow (after surgical updates and source page creation), run:
|
|
||||||
|
|
||||||
1. If new typed edges were added on the page being ingested, run `wiki_graph_lint.py`. **Interactive:** triage findings with the user before extract. **Drain (headless):** if lint is clean, proceed to extract; if lint reports errors, record them in `log.md` and skip extract for this batch — never silently rewrite typed edges, never block waiting for a user.
|
|
||||||
2. Run `wiki_graph_extract.py` to refresh the compiled artifacts.
|
|
||||||
3. Append a sub-line under the ingest's `log.md` entry:
|
|
||||||
` graph: +N nodes, +M typed edges (predicates: founded, contains_product, ...)`
|
|
||||||
|
|
||||||
Skip extract if this ingest added no `graph:` metadata and created no new pages — the compiled artifacts are unchanged.
|
|
||||||
|
|
||||||
## Query workflow integration
|
|
||||||
|
|
||||||
When the user asks a question that smells relational ("what's connected to X", "who proposed Y", "trace the path from A to B"):
|
|
||||||
|
|
||||||
1. Read the index as usual.
|
|
||||||
2. If `cml/wiki/graph/graph.sqlite` exists and is fresher than the latest log entry, query it for typed edges around the candidate pages — `neighbors`, `edges`, `facts` are the most useful.
|
|
||||||
3. Read the wiki pages behind the relevant nodes/edges. Don't answer from graph rows alone for high-stakes claims; the `evidence` field is a hint, not the source of truth.
|
|
||||||
4. Cite with `[[wikilinks]]` to wiki pages, not graph rows.
|
|
||||||
|
|
||||||
If `graph.sqlite` is stale (older than the most recent ingest in `log.md`), use it as-is and note the staleness — do **not** regenerate inline. Extract is a compile/drain step; the query turn stays read-only, and the background drain refreshes the graph after each ingest.
|
|
||||||
|
|
||||||
## Generated artifact policy
|
|
||||||
|
|
||||||
| File | Canonical? | Default tracking |
|
|
||||||
|------|-----------|------------------|
|
|
||||||
| `cml/wiki/graph/ontology.yaml` | Yes — edit by hand | Tracked |
|
|
||||||
| `cml/wiki/graph/nodes.jsonl` | Generated | Optional — `cml/wiki/graph/.gitignore` does not ignore it; track if you want graph diffs in PRs |
|
|
||||||
| `cml/wiki/graph/edges.jsonl` | Generated | Same as above |
|
|
||||||
| `cml/wiki/graph/graph.sqlite` | Generated | **Gitignored** by default (large, binary) |
|
|
||||||
| `cml/wiki/graph/graph.graphml` | Generated | **Gitignored** by default |
|
|
||||||
|
|
||||||
The bootstrapped `cml/wiki/graph/.gitignore` ignores `graph.sqlite` and `graph.graphml`. Edit it if your team prefers different policy.
|
|
||||||
|
|
||||||
## Anti-patterns
|
|
||||||
|
|
||||||
- **Typed edges without evidence.** Defeats the entire point. Lint will flag them; do not silence.
|
|
||||||
- **Editing `nodes.jsonl` / `edges.jsonl` / `graph.sqlite` by hand.** Edit the markdown; regenerate.
|
|
||||||
- **Inventing ontology entries to make a typed edge "fit".** The ontology should reflect domain reality, not paper over a too-eager edge. Either add the predicate with proper `subject_types`/`object_types`, or use `mentions`.
|
|
||||||
- **Treating graph rows as evidence in answers.** Always cite the wiki page; the graph just told you which wiki page to read.
|
|
||||||
- **Forgetting to regenerate after an ingest.** The graph diverges silently. Tie extract to ingest in muscle memory.
|
|
||||||
@@ -1,131 +0,0 @@
|
|||||||
# Ingest Workflow
|
|
||||||
|
|
||||||
When a new source arrives, this is the procedure. The order matters — each step builds context for the next.
|
|
||||||
|
|
||||||
## Step 0: Check the schema
|
|
||||||
|
|
||||||
Before anything else, read `cml/wiki/SCHEMA.md`. The user may have customized the page-type structure, the tag taxonomy, the naming conventions, or the ingest workflow itself. Schema overrides everything documented here.
|
|
||||||
|
|
||||||
## Step 1: Place the raw source
|
|
||||||
|
|
||||||
If the source isn't already in `cml/raw/`, place it there. Use a slugified filename: lowercase, hyphens for spaces, no special characters, with the original extension. For web articles, save as `.md` (Obsidian Web Clipper output is ideal). For PDFs, keep the `.pdf`. For transcripts, save as `.md` or `.txt`.
|
|
||||||
|
|
||||||
The slug you pick here will become the slug of the source-summary page in `cml/wiki/sources/`, so make it descriptive and stable.
|
|
||||||
|
|
||||||
## Step 2: Read the source
|
|
||||||
|
|
||||||
For short sources (under ~5,000 words / ~25,000 tokens), read the whole thing in one pass.
|
|
||||||
|
|
||||||
For long sources (papers over ~30 pages, book chapters, multi-hour transcripts), **chunk-read**: read the table of contents or section headers first to build a mental map, then read sections sequentially, summarizing each section in working memory before moving to the next. Do not load the entire raw source into context at once if it would consume more than ~25% of your context window — that leaves no room for the rest of the operation.
|
|
||||||
|
|
||||||
For PDFs specifically, prefer the `pdf-reading` skill if available (it handles the chunking automatically). Otherwise extract text first with `pdftotext` or `pdfminer` and then chunk-read the extracted text.
|
|
||||||
|
|
||||||
For images embedded in the source: read the surrounding text first, then view only the images that the text suggests are load-bearing (a chart referenced in an argument, a diagram of a system, a figure the source explicitly walks through). Don't blindly load every image — many are decorative.
|
|
||||||
|
|
||||||
## Step 3: Discuss the takeaways with the user (interactive only)
|
|
||||||
|
|
||||||
**Skip this step entirely in the background drain** — the compile cron runs headless with no user present, so there is no one to discuss with, and the remaining steps must not depend on a user reaction.
|
|
||||||
|
|
||||||
When run interactively, do this briefly — three or four sentences. Surface what struck you as important, what was surprising, what connects to existing wiki content, and what's worth flagging. The user's reaction shapes the next steps. They might say "skip the methodology section, only the results matter for me" or "we already have a page on this — just update it" or "this contradicts the page on X, flag that prominently".
|
|
||||||
|
|
||||||
If there is no user to react (the background drain, or a hands-off "just process these 20 papers" batch), skip the discussion and be more conservative on the wiki edits — make smaller, safer updates and surface anything ambiguous in the log.
|
|
||||||
|
|
||||||
## Step 4: Identify what's touched
|
|
||||||
|
|
||||||
Before writing anything, do a survey pass against the wiki to determine the impact:
|
|
||||||
|
|
||||||
1. Read `cml/wiki/index.md` (or the relevant shard under `cml/wiki/indexes/`) to identify existing pages this source touches. Look for entity names, concept names, and topical overlap.
|
|
||||||
2. For each potentially-touched page, read the page to confirm. (Read, don't grep — the index summaries can mislead.)
|
|
||||||
3. List, in working memory: existing pages to update, new pages to create, contradictions to flag.
|
|
||||||
|
|
||||||
This survey is what prevents duplication. Without it, you'll create a new page on a topic that already has one under a slightly different name.
|
|
||||||
|
|
||||||
## Step 5: Write the source-summary page
|
|
||||||
|
|
||||||
Create `cml/wiki/sources/<source-slug>.md`. Use the page template (`assets/page.md.template`) as a starting point. Frontmatter should include at minimum:
|
|
||||||
|
|
||||||
```yaml
|
|
||||||
---
|
|
||||||
type: source
|
|
||||||
title: "Original title of the source"
|
|
||||||
authors: ["Author Name"]
|
|
||||||
url: "https://..." # if applicable
|
|
||||||
raw: "cml/raw/<source-slug>.<ext>"
|
|
||||||
ingested: 2026-04-15
|
|
||||||
tags: [tag1, tag2]
|
|
||||||
entities: [entity-page-1, entity-page-2]
|
|
||||||
concepts: [concept-page-1, concept-page-2]
|
|
||||||
---
|
|
||||||
```
|
|
||||||
|
|
||||||
Frontmatter list values are bare slugs — the `[[wikilink]]` syntax goes in the body, not in YAML.
|
|
||||||
|
|
||||||
The body should be the LLM's summary of the source — the key claims, the methodology if relevant, the conclusions, the open questions. Do not paraphrase the entire source; that defeats the purpose. Aim for a summary that captures what a future query would need to know without re-reading the raw file.
|
|
||||||
|
|
||||||
Keep the page atomic — under the 400-line soft cap. If the source is so dense that a single summary page can't capture it, split: one page per major section, each linking to the others, with a parent page that gives the overview.
|
|
||||||
|
|
||||||
End the body with a "Where this fits" section listing `[[wikilinks]]` to the entity and concept pages this source touches. This is the bidirectional link from source → existing structure.
|
|
||||||
|
|
||||||
## Step 6: Update touched pages
|
|
||||||
|
|
||||||
For each existing page identified in Step 4, surgically edit it to incorporate what the new source adds. Use `str_replace`, not full rewrites. The goal is to add a sentence or paragraph in the relevant section, with a `[[wikilinks]]` citation to the new source-summary page.
|
|
||||||
|
|
||||||
Common update patterns:
|
|
||||||
|
|
||||||
- A new source corroborates an existing claim → add a citation: `..., as established in [[paper-X]] and now corroborated by [[paper-Y]]`.
|
|
||||||
- A new source contradicts an existing claim → flag prominently: add a "## Contradictions" section if it doesn't exist, and document the contradiction with both sources cited. Do not silently overwrite the older claim.
|
|
||||||
- A new source adds a new dimension to an existing topic → add a new sub-section, don't dilute the existing prose.
|
|
||||||
- A new source mentions an entity or concept the page already discusses → update the `sources:` frontmatter and add the cross-reference where relevant in the body.
|
|
||||||
|
|
||||||
If updating a page would push it over the 800-line hard cap, that's the signal to split the page (extract the new dimension into its own page, link from the parent). Do that as part of the ingest, not as a deferred lint task.
|
|
||||||
|
|
||||||
## Step 7: Create new pages for new entities and concepts
|
|
||||||
|
|
||||||
For each new entity or concept the source introduces that doesn't have a page yet, decide whether it warrants its own page. Heuristic: if the source mentions it in passing and no other source is likely to expand on it, just mention it inline on a related page. If the source treats it as a first-class topic or you can foresee future sources building on it, create a page.
|
|
||||||
|
|
||||||
New pages need:
|
|
||||||
- Frontmatter with `type`, `tags`, `sources: [[[<source-slug>]]]`, `created`, `updated`.
|
|
||||||
- A body that introduces the entity/concept with what this source said about it.
|
|
||||||
- Inbound links — at least one existing page should `[[wikilink]]` to the new page, otherwise it's an instant orphan. Update the existing page to add the link.
|
|
||||||
|
|
||||||
A new page that nothing links to is a bug in the ingest, not just a lint finding.
|
|
||||||
|
|
||||||
## Step 8: Update the index
|
|
||||||
|
|
||||||
Add entries for any new pages to `cml/wiki/index.md` (or the appropriate shard). Each entry is one line: a wikilink to the page and a one-sentence summary. Keep the summary tight — the index is engineered to be cheap to read, and a fat index defeats the index-first navigation principle.
|
|
||||||
|
|
||||||
If `index.md` exceeds 300 lines after this update, that's the signal to shard. Do it now while the structure is fresh — see `scaling-playbook.md` for the procedure.
|
|
||||||
|
|
||||||
## Step 8b: Refresh the graph layer (only if `cml/wiki/graph/ontology.yaml` exists)
|
|
||||||
|
|
||||||
If the wiki has the optional graph layer:
|
|
||||||
|
|
||||||
1. Add typed `graph.relationships[]` only when the source explicitly supports them (predicate, source-page slug, evidence quote, confidence, status). When uncertain, prefer a plain `[[wikilink]]` in the body — the body wikilink already produces a `mentions` edge.
|
|
||||||
2. Run `uv run skills/llm-wiki/scripts/wiki_graph_lint.py cml/wiki/`. **Interactive:** triage findings with the user before extracting. **Drain (headless):** if lint is clean, proceed to extract; if lint reports errors, record them in `log.md` and skip extract for this batch. Never silently rewrite typed edges, and never block waiting for a user.
|
|
||||||
3. Run `uv run skills/llm-wiki/scripts/wiki_graph_extract.py cml/wiki/` to regenerate `nodes.jsonl`, `edges.jsonl`, `graph.sqlite`, `graph.graphml`.
|
|
||||||
|
|
||||||
Skip extract if this ingest added no `graph:` metadata and created no new pages — the compiled artifacts are unchanged. Full reference: `references/graph-workflow.md`.
|
|
||||||
|
|
||||||
## Step 9: Append to the log
|
|
||||||
|
|
||||||
One line in `cml/wiki/log.md`, with the prefix `## [YYYY-MM-DD] ingest | <source-title>`. Optionally add a sub-line listing the pages touched. If the graph layer was refreshed, add a second sub-line: ` graph: +N nodes, +M typed edges`. The log is parsed by simple unix tools (`grep "^## \[" log.md | tail -10`), so the prefix matters.
|
|
||||||
|
|
||||||
## Step 10: Close the loop with the user
|
|
||||||
|
|
||||||
Tell the user what you did, briefly: "Ingested. Created the source page and a new entity page for X; updated the concept pages for Y and Z. Flagged a contradiction with [[paper-A]] regarding the claim about W."
|
|
||||||
|
|
||||||
If the source revealed something worth following up on (an obvious gap, a question the source raised but didn't answer, a candidate next source to ingest), say so — this is where the wiki's compounding effect comes from.
|
|
||||||
|
|
||||||
## Anti-patterns to avoid
|
|
||||||
|
|
||||||
**Loading the whole source into context at once when it's large.** This is the most common scaling failure. Chunk-read.
|
|
||||||
|
|
||||||
**Rewriting whole pages instead of surgical edits.** This burns tokens, risks losing nuance, and erodes diff quality if the wiki is in git.
|
|
||||||
|
|
||||||
**Creating pages with no inbound links.** Orphans accumulate fast and become invisible. Always link.
|
|
||||||
|
|
||||||
**Ingesting silently in batch mode without surfacing surprises.** Batch ingest is fine, but a one-line "ingested 5 sources, 2 new entity pages, 1 contradiction with [[X]]" summary is the minimum.
|
|
||||||
|
|
||||||
**Treating prior wiki pages as ground truth instead of the raw sources.** When updating an existing claim, re-read the raw source for that claim before merging the new one. Don't compound on the wiki's own paraphrase.
|
|
||||||
|
|
||||||
**Letting the page split decision drift to a future lint pass.** If a page crossed the size cap during this ingest, split it during this ingest.
|
|
||||||
@@ -1,99 +0,0 @@
|
|||||||
# Lint Workflow
|
|
||||||
|
|
||||||
A health check on the wiki. Best run on a cadence — after every N ingests, weekly, or when the user explicitly requests it — not on every operation. Lint is split into a structural pass (handled by `wiki_lint.py`) and a semantic pass (handled by the LLM directly).
|
|
||||||
|
|
||||||
> **Report-only gate (this nanobot).** A lint turn runs the read steps (0–1, 3–4), presents findings, and **stops**. The mutation steps below — Step 2 (apply fixes), Step 5 (index), Step 6 (log) — happen **only in a separate turn after the user approves**, one category at a time. Inside a lint turn, never create/edit pages, write debug scripts, regenerate the graph, or loop re-lint. When fixing later: bounded batches, do **not** re-read the lint script to reverse-engineer it, re-lint once to confirm — if issues remain, report and ask, don't continue blind. The scripts are sub-second; a long lint turn means you slipped into fixing.
|
|
||||||
|
|
||||||
## Step 0: Check the schema
|
|
||||||
|
|
||||||
`cml/wiki/SCHEMA.md` may declare additional lint rules specific to this wiki (e.g. "every entity page must have an `aliases:` field", "no concept page without at least 2 sources"). Read it.
|
|
||||||
|
|
||||||
## Step 1: Run the structural lint script
|
|
||||||
|
|
||||||
```bash
|
|
||||||
uv run skills/llm-wiki/scripts/wiki_lint.py cml/wiki/
|
|
||||||
```
|
|
||||||
|
|
||||||
If the wiki has the optional graph layer (`cml/wiki/graph/ontology.yaml` exists), also run:
|
|
||||||
|
|
||||||
```bash
|
|
||||||
uv run skills/llm-wiki/scripts/wiki_graph_lint.py cml/wiki/
|
|
||||||
```
|
|
||||||
|
|
||||||
This catches typed-edge problems independently of the structural lint: unknown predicates, missing evidence, broken object references, alias collisions, invalid `confidence`/`status` values, broken `contradicts`/`supersedes` references. Triage findings the same way as structural lint — propose fixes, don't apply them silently. After approved fixes, run `wiki_graph_extract.py` to refresh the compiled artifacts.
|
|
||||||
|
|
||||||
This produces a report covering:
|
|
||||||
|
|
||||||
- **Orphan pages** — pages with no inbound `[[wikilinks]]` from anywhere else in the wiki. Orphans are usually a sign that an ingest forgot to update a parent page. They become invisible because the index-first navigation can't surface them.
|
|
||||||
- **Broken wikilinks** — `[[page-name]]` references pointing to nonexistent pages. Usually a typo or a page that got renamed without updating its referrers.
|
|
||||||
- **Oversized pages** — anything over the 800-line hard cap (or 400-line soft cap, with a warning).
|
|
||||||
- **Frontmatter issues** — pages missing required fields (`type`, `tags`, `sources`, `updated`), or with malformed YAML.
|
|
||||||
- **Stale pages** — pages whose `updated:` date is much older than the most recent ingest that touched their topic. (The script approximates this using the page's tags and the log.)
|
|
||||||
- **Duplicate slugs** — two pages with the same slug in different subdirectories (a sign of an ingest collision that wasn't resolved).
|
|
||||||
|
|
||||||
The script is conservative — it reports findings but doesn't fix them. Present the report to the user.
|
|
||||||
|
|
||||||
## Step 2: Triage the structural findings (propose only; apply in a later approved turn)
|
|
||||||
|
|
||||||
Walk through the findings with the user and propose a fix for each. Proposing is part of the lint turn; **applying** the edits is not — that waits for approval (see the report-only gate above).
|
|
||||||
|
|
||||||
- Orphans: either link them from a sensible parent page (preferred), or determine that the page is genuinely useless and delete it (rare). If many orphans pile up, the index is probably out of date — re-derive the index entries.
|
|
||||||
- Broken links: rename the link to match the actual page, or create the missing page if it should exist, or remove the link if the concept turned out not to warrant a page.
|
|
||||||
- Oversized pages: split. Extract sub-concepts into their own pages, link from the parent, update the index.
|
|
||||||
- Frontmatter issues: add the missing fields. If many pages have the same gap, consider whether the schema should be relaxed or the bootstrap template improved.
|
|
||||||
- Stale pages: read the recently-touched related pages and the relevant raw sources, update the stale page surgically.
|
|
||||||
- Duplicate slugs: one is canonical, the other should be merged in and deleted. Pick the better-named one as canonical and migrate inbound links with `grep` + `str_replace`.
|
|
||||||
|
|
||||||
Present each proposed fix as an edit, not a fait accompli. The user approves.
|
|
||||||
|
|
||||||
## Step 3: Run the semantic pass
|
|
||||||
|
|
||||||
The semantic pass is what the script can't do — it requires reading pages and reasoning about content. The good news is that you don't have to read the whole wiki: focus on the pages most likely to have semantic issues.
|
|
||||||
|
|
||||||
**Recently updated pages.** Read the last ~10 pages that were modified (sorted by `updated:` frontmatter or by the log). Look for:
|
|
||||||
|
|
||||||
- Contradictions with older pages on the same topic. The new claim may have superseded the old one (in which case mark the old as superseded with a note), or both may be true but in different contexts (in which case clarify each), or the new ingest may have been wrong (in which case revert).
|
|
||||||
- Internal contradictions within a page (an ingest layered on a new claim without reconciling against existing prose).
|
|
||||||
- Repeated mentions of an entity or concept that doesn't have its own page yet — candidate for promotion.
|
|
||||||
|
|
||||||
**Highly-linked pages (hubs).** These are the most likely to drift because every ingest touches them. Use `wiki_search.py --top-linked 10` (or grep `[[wikilinks]]` and count) to find them. Read each hub and check that the prose still hangs together and the cross-references still make sense.
|
|
||||||
|
|
||||||
**Pages flagged with explicit uncertainty.** During ingest, the convention is to hedge ("the source claims X, though this is not yet corroborated") rather than assert. The lint pass is when you check whether subsequent ingests have corroborated or contradicted, and update accordingly.
|
|
||||||
|
|
||||||
## Step 4: Surface gaps
|
|
||||||
|
|
||||||
Look at what's referenced but not pageified. Run:
|
|
||||||
|
|
||||||
```bash
|
|
||||||
uv run skills/llm-wiki/scripts/wiki_lint.py cml/wiki/ --suggest-pages
|
|
||||||
```
|
|
||||||
|
|
||||||
This finds entity-like and concept-like names that appear in many pages but lack their own page. The user can decide which to promote and which to leave as inline mentions.
|
|
||||||
|
|
||||||
Also look for what's not covered at all — topics the user has expressed interest in but that haven't been ingested yet. The log can help here ("you ingested 5 sources on diffusion models in March but nothing since; want to refresh?").
|
|
||||||
|
|
||||||
## Step 5: Update the index (fix turn — only after approved fixes)
|
|
||||||
|
|
||||||
After lint fixes, the index almost certainly needs updates: new pages, renamed pages, deleted pages, changed summaries. Re-derive the affected index entries surgically.
|
|
||||||
|
|
||||||
If `index.md` is over 300 lines and hasn't been sharded yet, this is a good moment to do it. See `scaling-playbook.md`.
|
|
||||||
|
|
||||||
## Step 6: Append to the log (fix turn)
|
|
||||||
|
|
||||||
One line: `## [YYYY-MM-DD] lint | <N> structural fixes, <M> semantic fixes, <K> proposed gaps`. Optionally a sub-line listing the highest-impact changes.
|
|
||||||
|
|
||||||
## Cadence
|
|
||||||
|
|
||||||
A reasonable default: structural lint after every 5 ingests, semantic lint weekly or after every 20 ingests, gap-finding monthly. Adjust based on the user's pace and the wiki's volatility. A wiki that's growing fast needs more frequent lint; a stable mature wiki needs less.
|
|
||||||
|
|
||||||
The user may also trigger lint explicitly ("clean up the wiki", "what's broken", "lint pass please"). Treat these as priority — the user is asking because they noticed something.
|
|
||||||
|
|
||||||
## Anti-patterns to avoid
|
|
||||||
|
|
||||||
**Silent rewrites.** Lint findings are proposals. The user approves changes. A wiki that mutates without the user's knowledge stops being trustworthy.
|
|
||||||
|
|
||||||
**Treating lint as cleanup-only.** The semantic pass is also where new connections get made. If you read 10 pages and notice that two of them point at the same underlying idea, that's a synthesis-page candidate, not just a lint finding.
|
|
||||||
|
|
||||||
**Letting the lint report grow until it's overwhelming.** If the report is too long for the user to triage, the cadence is wrong (lint more often) or the wiki has outgrown its conventions (revisit the schema).
|
|
||||||
|
|
||||||
**Skipping lint because everything seems fine.** Silent corruption is the failure mode that's hardest to detect and most damaging. Lint catches it.
|
|
||||||
@@ -1,89 +0,0 @@
|
|||||||
# Page Conventions
|
|
||||||
|
|
||||||
The structural rules every wiki page follows. The schema may extend or override these for a particular wiki, but these are the defaults.
|
|
||||||
|
|
||||||
## Frontmatter
|
|
||||||
|
|
||||||
Every wiki page begins with YAML frontmatter. The required fields:
|
|
||||||
|
|
||||||
```yaml
|
|
||||||
---
|
|
||||||
type: <source|entity|concept|synthesis|...>
|
|
||||||
title: "Human-readable title"
|
|
||||||
tags: [tag1, tag2, tag3]
|
|
||||||
sources: [source-slug-1, source-slug-2]
|
|
||||||
created: 2026-04-15
|
|
||||||
updated: 2026-04-15
|
|
||||||
---
|
|
||||||
```
|
|
||||||
|
|
||||||
Note that frontmatter list values are **bare slugs**, not `[[wikilinks]]`. The double-bracket syntax is only used in the page body. The bundled scripts treat frontmatter `sources:`, `entities:`, and `concepts:` lists as slug references and resolve them the same way as body wikilinks.
|
|
||||||
|
|
||||||
`type` determines the page-type and which subdirectory the page lives in. Standard types are `source`, `entity`, `concept`, `synthesis`. The schema can declare additional types.
|
|
||||||
|
|
||||||
`title` is the human-readable name. The filename slug is separate and may be a short version of the title. For pages about people, the convention is `Last, First` for sortability; for everything else, natural casing.
|
|
||||||
|
|
||||||
`tags` is a flat list. Tags are how `wiki_search.py` filters and how the user navigates topically in Obsidian. Keep the tag taxonomy small and disciplined — a wiki with 200 tags has effectively no tags. The schema should declare the canonical tag set and the lint script will warn on tags outside it.
|
|
||||||
|
|
||||||
`sources` is the list of source-summary pages this page draws from. Always populated for entity/concept/synthesis pages; for source pages themselves, this field is omitted (the source page is the source).
|
|
||||||
|
|
||||||
`created` and `updated` are dates in ISO format. `updated` is the load-bearing one — it powers the staleness check in lint and the "what's new" view.
|
|
||||||
|
|
||||||
Type-specific fields:
|
|
||||||
- Source pages add `authors`, `url`, `raw` (path to the raw file), `ingested`.
|
|
||||||
- Entity pages may add `aliases` (other names for the same entity), `kind` (person, paper, product, etc.).
|
|
||||||
- Synthesis pages add `question` (the original question, if it was filed back from a query) and `sources_consulted`.
|
|
||||||
|
|
||||||
## Wikilinks
|
|
||||||
|
|
||||||
Cross-references use the `[[page-slug]]` syntax. The slug is the filename without the `.md` extension and without the directory prefix — Obsidian-style. Aliasing is supported: `[[page-slug|display text]]`.
|
|
||||||
|
|
||||||
Every page should have at least one inbound link. New pages without inbound links are orphans and the lint pass will flag them.
|
|
||||||
|
|
||||||
The bundled `wiki_search.py --backlinks <slug>` returns inbound links. To find them manually:
|
|
||||||
|
|
||||||
```bash
|
|
||||||
grep -rln "\[\[<slug>\]\]" cml/wiki/
|
|
||||||
```
|
|
||||||
|
|
||||||
## Page sizing
|
|
||||||
|
|
||||||
**Soft cap: 400 lines or ~2,000 words.** When a page approaches this, consider whether it should split.
|
|
||||||
|
|
||||||
**Hard cap: 800 lines.** Pages over this must split. The lint script flags violations.
|
|
||||||
|
|
||||||
Atomicity heuristic: a page is about *one* thing. A page on "Diffusion Models" should not also be the page on "Stable Diffusion" — those are two pages with cross-references. If you find yourself writing "## Variants" with five sub-sections of substantive prose, those sub-sections are probably their own pages.
|
|
||||||
|
|
||||||
The reason for the size cap is the context-bottleneck principle: any single page read is bounded, so the LLM can confidently read several pages without exhausting context.
|
|
||||||
|
|
||||||
## Naming
|
|
||||||
|
|
||||||
Page slugs are lowercase, hyphenated, no special characters. Match the directory: a concept page lives at `cml/wiki/concepts/<slug>.md`.
|
|
||||||
|
|
||||||
For entity pages about people: prefer the full name slugified (`andrej-karpathy.md`), not just the surname. Add `aliases:` to frontmatter for common short names.
|
|
||||||
|
|
||||||
For source pages: use a slug derived from the source title, possibly with the year for disambiguation (`attention-is-all-you-need-2017.md`). The source page slug should match the raw file slug if possible — that makes the back-pointer trivial.
|
|
||||||
|
|
||||||
For synthesis pages: derive from the question or the topic, not the date. `comparing-rag-vs-llm-wiki.md`, not `2026-04-15-question.md`. Date-based names don't surface usefully in the index.
|
|
||||||
|
|
||||||
## Body structure
|
|
||||||
|
|
||||||
The body has no rigid template — the schema may declare one for specific page types — but a few defaults work well:
|
|
||||||
|
|
||||||
**Source pages**: lead paragraph summarizing the source's main contribution. Sections for key claims, methodology (if relevant), conclusions, open questions. End with a "Where this fits" section listing the entity and concept pages this source touches.
|
|
||||||
|
|
||||||
**Entity pages**: lead paragraph defining the entity. Sections for relevant attributes (for a paper: authors, venue, key claims; for a person: affiliation, notable work; for a product: what it does, who makes it). A "Mentioned in" section is unnecessary — the backlink discovery handles that.
|
|
||||||
|
|
||||||
**Concept pages**: lead paragraph defining the concept clearly. Sections for the key formulation, variants, contested aspects, related concepts. Heavy use of `[[wikilinks]]` is expected — concept pages are the connective tissue of the wiki.
|
|
||||||
|
|
||||||
**Synthesis pages**: lead with the question (if filed from a query) or the framing. The body is the answer/analysis. End with the sources consulted as wikilinks.
|
|
||||||
|
|
||||||
## Hedging language
|
|
||||||
|
|
||||||
When a source claims something that hasn't been corroborated by other sources in the wiki, hedge: "Source X claims Y, though this is not yet corroborated by other sources in the wiki." The lint pass will revisit hedged claims as the wiki accumulates more sources, either upgrading them to confirmed or flagging them as contested.
|
|
||||||
|
|
||||||
When two sources contradict, document both: "Source X claims Y; source Z claims not-Y. The contradiction is unresolved." Do not silently pick a side.
|
|
||||||
|
|
||||||
## Keeping pages voice-neutral
|
|
||||||
|
|
||||||
The wiki is the LLM's voice, not the source author's voice and not the user's voice. Paraphrase rather than quote, except for short load-bearing phrases where the exact wording matters. Maintain a consistent, neutral, encyclopedic tone — close to a Wikipedia article in register, not a chat reply.
|
|
||||||
@@ -1,102 +0,0 @@
|
|||||||
# Query Workflow
|
|
||||||
|
|
||||||
The user is asking a question against the wiki. The job is to answer it from the wiki, with citations, scaling navigation to the wiki's size, and to file the answer back as a synthesis page when warranted.
|
|
||||||
|
|
||||||
## Step 0: Check the schema
|
|
||||||
|
|
||||||
Read `cml/wiki/SCHEMA.md` if you haven't this session. Some wikis declare query-specific conventions (e.g. "always answer with a comparison table when the question is comparative", "answers go in `cml/wiki/synthesis/qa/` not `cml/wiki/synthesis/`"). Schema overrides defaults.
|
|
||||||
|
|
||||||
## Step 1: Read the index
|
|
||||||
|
|
||||||
Always start at `cml/wiki/index.md`. If the index has been sharded into `cml/wiki/indexes/`, read the top-level `index.md` first to identify which shard(s) are relevant, then read those.
|
|
||||||
|
|
||||||
The index is engineered for this — one line per page with a tight summary. You should be able to identify candidate pages from the index alone in most cases. If a query touches multiple shards (e.g. "compare the methodologies in papers A and B"), read all relevant shards.
|
|
||||||
|
|
||||||
## Step 2: Identify candidate pages
|
|
||||||
|
|
||||||
From the index, build a short list of pages that look relevant to the query. Be selective — reading 30 pages to answer a question is a sign you've fallen back to brute-force search, which doesn't scale. If the index summaries don't disambiguate well, that's a signal that the index entries are too terse — note it for the next lint pass.
|
|
||||||
|
|
||||||
If the index doesn't surface good candidates (the query uses fuzzy or domain-specific language that doesn't match the index summaries), fall back to the search script:
|
|
||||||
|
|
||||||
```bash
|
|
||||||
uv run skills/llm-wiki/scripts/wiki_search.py "your query terms" --top 10
|
|
||||||
```
|
|
||||||
|
|
||||||
This returns the top-N pages by BM25 score, with optional filters on frontmatter (`--type concept`, `--tag llms`, `--since 2026-01-01`). Use the search script *as a fallback*, not as the default — index-first is cheaper and produces more interpretable results when it works.
|
|
||||||
|
|
||||||
## Step 2b: Graph-assisted lookup (only if `cml/wiki/graph/graph.sqlite` exists)
|
|
||||||
|
|
||||||
For relational questions ("what's connected to X", "who proposed Y", "trace the path from A to B"), query the compiled graph after the index pass and before reading pages:
|
|
||||||
|
|
||||||
```bash
|
|
||||||
uv run skills/llm-wiki/scripts/wiki_graph_query.py cml/wiki/ neighbors --node <node-id>
|
|
||||||
uv run skills/llm-wiki/scripts/wiki_graph_query.py cml/wiki/ facts --about <node-id>
|
|
||||||
```
|
|
||||||
|
|
||||||
Use the structured neighbors/facts to pick the right wiki pages to read — but never answer from graph rows alone for high-stakes claims. The graph accelerates navigation; the wiki page and its raw source remain the evidence. If `graph.sqlite` is older than the most recent `## [YYYY-MM-DD] ingest |` entry in `log.md`, use it as-is and note the staleness — do **not** run `wiki_graph_extract.py` inline. Extract is a compile-phase step; a query turn stays read-only and the background drain refreshes the graph after each ingest. Full reference: `references/graph-workflow.md`.
|
|
||||||
|
|
||||||
## Step 3: Read the candidate pages
|
|
||||||
|
|
||||||
Read each candidate page in full. While reading, note any `[[wikilinks]]` to other pages that look relevant — those are pre-curated leads. Follow the most promising ones, but don't recursively chase every link or you'll exhaust your context window on tangentially-relevant pages.
|
|
||||||
|
|
||||||
If a page references a source-summary page in its frontmatter and the answer hinges on what that source actually said, read the source-summary page too. Avoid going all the way back to the raw source unless the wiki summaries are clearly insufficient — the whole point of the wiki is that the synthesis is already done.
|
|
||||||
|
|
||||||
## Step 4: Find backlinks if needed
|
|
||||||
|
|
||||||
If the query is "what does my wiki say about X" or "where is X mentioned", and X has its own page, the inbound links are often more interesting than the page itself. Find them with:
|
|
||||||
|
|
||||||
```bash
|
|
||||||
grep -rl "\[\[<page-slug>\]\]" cml/wiki/
|
|
||||||
```
|
|
||||||
|
|
||||||
This is faster than reading pages to look for mentions. Note that the bundled `wiki_search.py` has a `--backlinks <slug>` mode that does this and returns ranked results.
|
|
||||||
|
|
||||||
## Step 5: Synthesize the answer
|
|
||||||
|
|
||||||
Write the answer in your own words, with `[[wikilink]]` citations to the wiki pages you used and (where helpful) `cml/raw/<file>` references to specific raw sources. The citations matter — they let the user verify the answer and follow up.
|
|
||||||
|
|
||||||
If the wiki contains contradicting claims, surface the contradiction explicitly rather than picking one and presenting it as settled. The wiki's value is partly in tracking what's known versus what's contested.
|
|
||||||
|
|
||||||
If the wiki has no relevant content for the query, say so plainly. Do not confabulate — that's the surest way to corrupt the wiki when the answer gets filed back. Instead, suggest sources the user could ingest to fill the gap.
|
|
||||||
|
|
||||||
## Step 6: Offer to file the answer back
|
|
||||||
|
|
||||||
If the synthesized answer represents new connection-making — a comparison the wiki didn't already contain, an analysis that pulls together threads from multiple pages, an answer to a recurring question — offer to file it as a synthesis page. The user will say no for trivial answers and yes for substantive ones. Default to offering.
|
|
||||||
|
|
||||||
The file-back creates `cml/wiki/synthesis/<answer-slug>.md` with frontmatter:
|
|
||||||
|
|
||||||
```yaml
|
|
||||||
---
|
|
||||||
type: synthesis
|
|
||||||
question: "the original question, verbatim or lightly cleaned"
|
|
||||||
asked: 2026-04-15
|
|
||||||
sources_consulted: [page-1, page-2, page-3]
|
|
||||||
tags: [...]
|
|
||||||
---
|
|
||||||
```
|
|
||||||
|
|
||||||
Body: the answer as you gave it to the user, possibly lightly edited for the wiki's voice. Add the new synthesis page to the relevant index. Append to `log.md` with prefix `## [YYYY-MM-DD] query | <question-summary>`.
|
|
||||||
|
|
||||||
A filed synthesis page is itself queryable — the next time the user asks an adjacent question, the synthesis page may be the most relevant candidate. This is how exploration compounds.
|
|
||||||
|
|
||||||
## Special query types
|
|
||||||
|
|
||||||
**"What's missing on topic X?"** — Read existing pages on X, identify open questions or unstated assumptions, and propose ingest candidates. This is essentially a per-topic micro-lint.
|
|
||||||
|
|
||||||
**"Compare X and Y"** — Read both pages, look for explicit comparison pages already in `synthesis/`, generate a comparison table or contrast prose. Strong file-back candidate.
|
|
||||||
|
|
||||||
**"Show me the timeline of X"** — Use `log.md` to reconstruct chronology of ingests touching X, supplement with `created`/`updated` frontmatter on relevant pages.
|
|
||||||
|
|
||||||
**"What did source X say about Y?"** — Read `cml/wiki/sources/<x>.md` for the source's summary; if it doesn't directly answer, read `cml/raw/<x>` (chunk-read if large).
|
|
||||||
|
|
||||||
**"Lint check on this answer"** — Before filing back, ask the user to verify a key claim by pointing to its raw source. Especially valuable for high-stakes wikis (medical, legal, financial).
|
|
||||||
|
|
||||||
## Anti-patterns to avoid
|
|
||||||
|
|
||||||
**Reading every page in the wiki to be safe.** This doesn't scale and produces vague answers. Trust the index; if the index fails, fix the index, don't bypass it.
|
|
||||||
|
|
||||||
**Citing the wiki without citing the underlying sources for hard claims.** The wiki page is a paraphrase; for any claim the user might need to verify, the citation should chain back to the raw source.
|
|
||||||
|
|
||||||
**Filing back trivial answers.** Not every Q&A is worth a permanent page. If the answer is a one-line lookup or restates an existing page, don't pollute synthesis/. The threshold is "would I want to find this when I ask a similar question in three months?"
|
|
||||||
|
|
||||||
**Confabulating when the wiki is silent.** Better to say "the wiki doesn't cover this" than to invent an answer that gets filed back as authoritative.
|
|
||||||
@@ -1,91 +0,0 @@
|
|||||||
# Scaling Playbook
|
|
||||||
|
|
||||||
Thresholds at which the wiki's structure needs to evolve, and the migration steps. The goal is to keep the wiki's context cost roughly constant per query as the wiki grows — the LLM should never need to read more pages or larger pages just because the wiki got bigger.
|
|
||||||
|
|
||||||
## The bottleneck
|
|
||||||
|
|
||||||
Naive LLM Wiki implementations break at scale because of a few specific failure modes, each of which has a structural fix:
|
|
||||||
|
|
||||||
- **The index file becomes too large to read cheaply.** Fix: shard the index by category.
|
|
||||||
- **Pages grow unboundedly as more sources mention them.** Fix: enforce the page size cap; split when violated.
|
|
||||||
- **Index summaries become too vague to disambiguate candidates.** Fix: tighten summaries; introduce frontmatter filtering via the search script.
|
|
||||||
- **Brute-force grep over the wiki replaces index-first navigation.** Fix: make the index actually useful and use the search script as the explicit fallback.
|
|
||||||
|
|
||||||
## Threshold 1: ~50 pages
|
|
||||||
|
|
||||||
Below this scale, you need almost no structure. A flat `cml/wiki/` directory with `index.md` and `log.md` is enough. The categorical subdirectories (`entities/`, `concepts/`, etc.) are still worth using from the start because they're free, but the index doesn't need sharding and the search script is overkill.
|
|
||||||
|
|
||||||
## Threshold 2: ~150 pages OR `index.md` over 300 lines
|
|
||||||
|
|
||||||
Time to **shard the index**. The migration:
|
|
||||||
|
|
||||||
1. Create `cml/wiki/indexes/` directory.
|
|
||||||
2. Split `index.md` by category into `indexes/sources.md`, `indexes/entities.md`, `indexes/concepts.md`, `indexes/synthesis.md` (and any custom types from the schema). Each shard is a list of pages of that type, with the same one-line summaries.
|
|
||||||
3. Rewrite the top-level `index.md` to be a directory of shards: each shard linked, with a one-line description of what's in it and a count.
|
|
||||||
4. Update the schema to document the sharded structure.
|
|
||||||
5. Update the ingest workflow in your working memory: now you update `indexes/<type>.md`, not `index.md` directly.
|
|
||||||
|
|
||||||
The top-level `index.md` should now be tiny — under 50 lines — and the shards are each bounded by the type-specific volume.
|
|
||||||
|
|
||||||
If a single shard later exceeds 300 lines (most likely `entities.md` or `concepts.md` if the wiki has a strong topical focus), shard *that* by sub-category: `indexes/entities-people.md`, `indexes/entities-papers.md`, etc. The principle generalizes.
|
|
||||||
|
|
||||||
## Threshold 3: ~300 pages
|
|
||||||
|
|
||||||
Time to introduce the **search script as a routine fallback**. Index navigation still works for direct lookups ("the page on diffusion models"), but fuzzy queries ("which papers discuss training stability") benefit from BM25 ranking.
|
|
||||||
|
|
||||||
`scripts/wiki_search.py` provides:
|
|
||||||
|
|
||||||
- `uv run skills/llm-wiki/scripts/wiki_search.py "query terms"` — top-N pages by BM25 score.
|
|
||||||
- `--type concept` — filter by frontmatter type.
|
|
||||||
- `--tag <tag>` — filter by tag.
|
|
||||||
- `--since 2026-01-01` — filter by `updated` date.
|
|
||||||
- `--backlinks <slug>` — find pages that link to a given page.
|
|
||||||
- `--top-linked N` — find the N most-linked-to pages (hubs).
|
|
||||||
|
|
||||||
Update the schema to declare the search script as a sanctioned fallback, so that future LLM sessions know to reach for it rather than degenerating into recursive grep.
|
|
||||||
|
|
||||||
## Threshold 4: ~500 pages
|
|
||||||
|
|
||||||
At this scale, two things start to matter:
|
|
||||||
|
|
||||||
**Structural lint cadence becomes weekly or per-N-ingests.** Manual oversight stops scaling. Rely on `wiki_lint.py` to surface structural drift and triage with the user.
|
|
||||||
|
|
||||||
**The search script may want a real index.** The default `wiki_search.py` rebuilds its BM25 index on every run, which is fine up to a few thousand pages. Beyond that, persist the index to disk (the script supports `--cache .wiki-search-cache.json`).
|
|
||||||
|
|
||||||
Also consider whether the wiki has organically split into distinct topic clusters that don't really cross-reference each other. If so, a single wiki may be the wrong shape — splitting into per-topic wikis (each with its own `SCHEMA.md`, `index.md`, etc.) may be cleaner. The user should make this call.
|
|
||||||
|
|
||||||
## Threshold 5: ~1,000+ pages
|
|
||||||
|
|
||||||
At this scale, the question is whether the LLM Wiki pattern is still the right tool. Markdown + grep + frontmatter scales further than people expect (the gist author reports a wiki of ~100 articles and ~400K words working fine), but at some point a real database with structured queries beats markdown. Signals it's time to consider migrating:
|
|
||||||
|
|
||||||
- The user's queries are predominantly relational ("show me all papers from author X cited by papers in topic Y published after date Z"). A graph database serves this better.
|
|
||||||
- Lint reports are too long to triage even at high cadence.
|
|
||||||
- The schema has grown to specify dozens of types and hundreds of tags — at that point you've manually built a database schema in markdown.
|
|
||||||
|
|
||||||
If the user wants to migrate, the markdown wiki is excellent input for the migration: every page has frontmatter and `[[wikilinks]]` that map cleanly to a property graph.
|
|
||||||
|
|
||||||
## When to *not* shard
|
|
||||||
|
|
||||||
Sharding is irreversible-ish (you can un-shard, but it's annoying), so don't do it preemptively. Wait for the actual threshold. A wiki of 80 pages with a sharded index has worse usability than the same wiki with a flat index, because the shards add a navigation step without enough volume to justify it.
|
|
||||||
|
|
||||||
## When to introduce custom page types
|
|
||||||
|
|
||||||
The default types (`source`, `entity`, `concept`, `synthesis`) cover most use cases. Add a custom type only when the user has a clear category of pages that don't fit any default and that benefits from being a distinct subdirectory (queryable, distinct lint rules, distinct templates). Examples that justify it:
|
|
||||||
|
|
||||||
- `decision` pages for a team that documents architectural decisions
|
|
||||||
- `experiment` pages for a research lab logging trial results
|
|
||||||
- `character` pages for a fan wiki tracking a fictional cast
|
|
||||||
- `meeting` pages for a team logging meetings
|
|
||||||
|
|
||||||
Don't add a type for a one-off — use tags instead. Adding a type is a schema change that affects the index, the lint script, and every future ingest.
|
|
||||||
|
|
||||||
## Detecting "we've outgrown our conventions"
|
|
||||||
|
|
||||||
Some signals that the schema needs revision rather than just more lint:
|
|
||||||
|
|
||||||
- The same kind of frontmatter issue keeps reappearing — the field needs to be optional, or the bootstrap template needs an example.
|
|
||||||
- Pages keep getting created that don't fit cleanly into any existing type — a new type is wanted.
|
|
||||||
- The tag list has grown unmanageable — prune, consolidate, or formalize the taxonomy.
|
|
||||||
- The user keeps overriding the LLM on a particular kind of decision — encode their preference in the schema so it persists across sessions.
|
|
||||||
|
|
||||||
Schema revision is healthy. A schema that doesn't change after the first few weeks of a wiki's life is probably not being used.
|
|
||||||
@@ -1,206 +0,0 @@
|
|||||||
#!/usr/bin/env -S uv run --script
|
|
||||||
# /// script
|
|
||||||
# requires-python = ">=3.11"
|
|
||||||
# dependencies = []
|
|
||||||
# ///
|
|
||||||
"""
|
|
||||||
init_wiki.py — Bootstrap or upgrade an LLM Wiki structure in a project.
|
|
||||||
|
|
||||||
Plain init creates the directory layout and drops in templates for SCHEMA.md,
|
|
||||||
index.md, log.md, the page template, and the optional graph layer
|
|
||||||
(graph/ontology.yaml, graph/README.md, graph/.gitignore). It is idempotent:
|
|
||||||
re-running won't clobber existing files.
|
|
||||||
|
|
||||||
`--upgrade` mode is for wikis bootstrapped under an older plugin version. It
|
|
||||||
does the same idempotent file creation, then inspects the existing SCHEMA.md
|
|
||||||
for sections introduced in newer versions and prints clear instructions for
|
|
||||||
what to merge by hand. It never overwrites SCHEMA.md — the schema is
|
|
||||||
co-evolved with the user.
|
|
||||||
|
|
||||||
Usage:
|
|
||||||
python init_wiki.py <project-root> [--wiki-dir wiki] [--raw-dir raw] [--upgrade]
|
|
||||||
|
|
||||||
Examples:
|
|
||||||
python init_wiki.py .
|
|
||||||
python init_wiki.py . --upgrade
|
|
||||||
python init_wiki.py ~/research --wiki-dir kb --raw-dir sources
|
|
||||||
"""
|
|
||||||
|
|
||||||
import argparse
|
|
||||||
import sys
|
|
||||||
from pathlib import Path
|
|
||||||
from datetime import date
|
|
||||||
|
|
||||||
|
|
||||||
SKILL_ROOT = Path(__file__).resolve().parent.parent
|
|
||||||
TEMPLATES = SKILL_ROOT / "assets"
|
|
||||||
|
|
||||||
|
|
||||||
SUBDIRS = ["sources", "entities", "concepts", "synthesis", "graph"]
|
|
||||||
|
|
||||||
# Markers used by --upgrade to detect SCHEMA.md sections introduced in
|
|
||||||
# specific plugin versions. Each entry: (heading_marker, version_label,
|
|
||||||
# template_anchor, blurb).
|
|
||||||
SCHEMA_SECTION_MARKERS = [
|
|
||||||
{
|
|
||||||
"marker": "## Optional graph metadata",
|
|
||||||
"version": "0.3.0",
|
|
||||||
"anchor": "## Optional graph metadata",
|
|
||||||
"label": "Optional graph metadata (Frontmatter section)",
|
|
||||||
},
|
|
||||||
{
|
|
||||||
"marker": "## Graph layer",
|
|
||||||
"version": "0.3.0",
|
|
||||||
"anchor": "## Graph layer",
|
|
||||||
"label": "Graph layer (canonical-vs-generated artifact policy)",
|
|
||||||
},
|
|
||||||
{
|
|
||||||
"marker": "Graph lint + extract",
|
|
||||||
"version": "0.3.0",
|
|
||||||
"anchor": "- Graph lint + extract: after every ingest that adds typed `graph.relationships`.",
|
|
||||||
"label": "Graph lint + extract cadence (Lint cadence section)",
|
|
||||||
},
|
|
||||||
]
|
|
||||||
|
|
||||||
|
|
||||||
def copy_template(src: Path, dst: Path, substitutions: dict | None = None) -> bool:
|
|
||||||
"""Copy a template file to dst. Returns True if file was created, False if it already existed."""
|
|
||||||
if dst.exists():
|
|
||||||
return False
|
|
||||||
text = src.read_text()
|
|
||||||
if substitutions:
|
|
||||||
for key, value in substitutions.items():
|
|
||||||
text = text.replace(key, value)
|
|
||||||
dst.parent.mkdir(parents=True, exist_ok=True)
|
|
||||||
dst.write_text(text)
|
|
||||||
return True
|
|
||||||
|
|
||||||
|
|
||||||
def detect_schema_gaps(schema_path: Path) -> list[dict]:
|
|
||||||
"""Return the SCHEMA_SECTION_MARKERS entries missing from the user's SCHEMA.md."""
|
|
||||||
if not schema_path.exists():
|
|
||||||
return []
|
|
||||||
text = schema_path.read_text(encoding="utf-8")
|
|
||||||
return [m for m in SCHEMA_SECTION_MARKERS if m["marker"] not in text]
|
|
||||||
|
|
||||||
|
|
||||||
def print_schema_upgrade_guidance(schema_path: Path, gaps: list[dict]) -> None:
|
|
||||||
template_path = TEMPLATES / "SCHEMA.md.template"
|
|
||||||
print()
|
|
||||||
print("=" * 64)
|
|
||||||
print(f"Upgrade required: {schema_path}")
|
|
||||||
print("=" * 64)
|
|
||||||
print(
|
|
||||||
"Your SCHEMA.md predates one or more sections introduced by newer\n"
|
|
||||||
"plugin versions. The graph layer itself is opt-in, but to make Claude\n"
|
|
||||||
"aware of it, merge the sections below by hand. SCHEMA.md is co-evolved\n"
|
|
||||||
"with you — this script never overwrites it."
|
|
||||||
)
|
|
||||||
print()
|
|
||||||
print("Missing sections:")
|
|
||||||
for m in gaps:
|
|
||||||
print(f" - [{m['version']}] {m['label']}")
|
|
||||||
print()
|
|
||||||
print(f"Reference template: {template_path}")
|
|
||||||
print(
|
|
||||||
"Diff your SCHEMA.md against the template and copy the missing\n"
|
|
||||||
"sections in. Or run /wiki:upgrade and Claude will propose the edits\n"
|
|
||||||
"interactively (one section at a time, never silent)."
|
|
||||||
)
|
|
||||||
|
|
||||||
|
|
||||||
def init_wiki(project_root: Path, wiki_dir: str, raw_dir: str, upgrade: bool = False) -> None:
|
|
||||||
project_root = project_root.resolve()
|
|
||||||
if not project_root.exists():
|
|
||||||
print(f"Error: project root does not exist: {project_root}", file=sys.stderr)
|
|
||||||
sys.exit(1)
|
|
||||||
|
|
||||||
wiki = project_root / wiki_dir
|
|
||||||
raw = project_root / raw_dir
|
|
||||||
|
|
||||||
mode = "Upgrading" if upgrade else "Initializing"
|
|
||||||
print(f"{mode} LLM Wiki in: {project_root}")
|
|
||||||
print(f" Wiki directory: {wiki}")
|
|
||||||
print(f" Raw directory: {raw}")
|
|
||||||
print()
|
|
||||||
|
|
||||||
created = []
|
|
||||||
skipped = []
|
|
||||||
|
|
||||||
# Create wiki subdirs
|
|
||||||
for subdir in SUBDIRS:
|
|
||||||
d = wiki / subdir
|
|
||||||
if not d.exists():
|
|
||||||
d.mkdir(parents=True)
|
|
||||||
created.append(f"{wiki_dir}/{subdir}/")
|
|
||||||
else:
|
|
||||||
skipped.append(f"{wiki_dir}/{subdir}/")
|
|
||||||
|
|
||||||
# Create raw + raw/assets
|
|
||||||
for d, label in [(raw, raw_dir), (raw / "assets", f"{raw_dir}/assets")]:
|
|
||||||
if not d.exists():
|
|
||||||
d.mkdir(parents=True)
|
|
||||||
created.append(f"{label}/")
|
|
||||||
else:
|
|
||||||
skipped.append(f"{label}/")
|
|
||||||
|
|
||||||
# Copy templates
|
|
||||||
template_map = [
|
|
||||||
("SCHEMA.md.template", wiki / "SCHEMA.md"),
|
|
||||||
("index.md.template", wiki / "index.md"),
|
|
||||||
("log.md.template", wiki / "log.md"),
|
|
||||||
("page.md.template", wiki / ".page-template.md"),
|
|
||||||
("ontology.yaml.template", wiki / "graph" / "ontology.yaml"),
|
|
||||||
("graph_README.md.template", wiki / "graph" / "README.md"),
|
|
||||||
("graph_gitignore.template", wiki / "graph" / ".gitignore"),
|
|
||||||
]
|
|
||||||
for src_name, dst in template_map:
|
|
||||||
src = TEMPLATES / src_name
|
|
||||||
if not src.exists():
|
|
||||||
print(f"Warning: template missing: {src}", file=sys.stderr)
|
|
||||||
continue
|
|
||||||
if copy_template(src, dst):
|
|
||||||
created.append(str(dst.relative_to(project_root)))
|
|
||||||
else:
|
|
||||||
skipped.append(str(dst.relative_to(project_root)))
|
|
||||||
|
|
||||||
# Report
|
|
||||||
if created:
|
|
||||||
print("Created:")
|
|
||||||
for path in created:
|
|
||||||
print(f" + {path}")
|
|
||||||
if skipped:
|
|
||||||
print("Already existed (skipped):")
|
|
||||||
for path in skipped:
|
|
||||||
print(f" = {path}")
|
|
||||||
|
|
||||||
if upgrade:
|
|
||||||
gaps = detect_schema_gaps(wiki / "SCHEMA.md")
|
|
||||||
if gaps:
|
|
||||||
print_schema_upgrade_guidance(wiki / "SCHEMA.md", gaps)
|
|
||||||
else:
|
|
||||||
print()
|
|
||||||
print("SCHEMA.md is up to date with the current template — no manual merge needed.")
|
|
||||||
return
|
|
||||||
|
|
||||||
print()
|
|
||||||
print("Next steps:")
|
|
||||||
print(f" 1. Read {wiki_dir}/SCHEMA.md and customize it for your domain.")
|
|
||||||
print(f" 2. (Optional) Edit {wiki_dir}/graph/ontology.yaml to add domain-specific predicates.")
|
|
||||||
print(f" 3. Drop your first source into {raw_dir}/.")
|
|
||||||
print(f" 4. Ask Claude to ingest it.")
|
|
||||||
|
|
||||||
|
|
||||||
def main():
|
|
||||||
parser = argparse.ArgumentParser(description=__doc__, formatter_class=argparse.RawDescriptionHelpFormatter)
|
|
||||||
parser.add_argument("project_root", type=Path, help="Project root directory.")
|
|
||||||
parser.add_argument("--wiki-dir", default="wiki", help="Name of the wiki subdirectory (default: wiki).")
|
|
||||||
parser.add_argument("--raw-dir", default="raw", help="Name of the raw sources subdirectory (default: raw).")
|
|
||||||
parser.add_argument("--upgrade", action="store_true",
|
|
||||||
help="Upgrade an existing wiki: add missing files idempotently and surface SCHEMA.md sections to merge by hand.")
|
|
||||||
args = parser.parse_args()
|
|
||||||
init_wiki(args.project_root, args.wiki_dir, args.raw_dir, upgrade=args.upgrade)
|
|
||||||
|
|
||||||
|
|
||||||
if __name__ == "__main__":
|
|
||||||
main()
|
|
||||||
@@ -1,169 +0,0 @@
|
|||||||
#!/usr/bin/env -S uv run --script
|
|
||||||
# /// script
|
|
||||||
# requires-python = ">=3.11"
|
|
||||||
# dependencies = ["nanobot-ai"]
|
|
||||||
# ///
|
|
||||||
"""wiki_compile.py — dávkový compile nasbíraných zdrojů z cml/raw/ do cml/wiki/.
|
|
||||||
|
|
||||||
Spouštěn systémovým cronem každou minutu. Capture (interaktivní) hází zdroje do
|
|
||||||
cml/raw/ a hned potvrdí; těžký raw→wiki compile (čtení zdrojů, psaní stránek,
|
|
||||||
rozhodování) běží mimo interaktivní tah přes LLM agenta — tady, na pozadí.
|
|
||||||
|
|
||||||
Tok:
|
|
||||||
1. Levná pre-kontrola (BEZ LLM): jsou v cml/raw/ nezpracované zdroje
|
|
||||||
(regulérní soubory mimo _done/, _hard/, assets/)? Žádné → exit 0, agenta
|
|
||||||
vůbec neinstancuj.
|
|
||||||
2. Lockfile (cml/.compile.lock, PID + start-timestamp): běží jiný compile?
|
|
||||||
→ exit 0 (neduplikovat). Stale lock (mrtvý proces / > STALE_SECONDS) se
|
|
||||||
přebere, ať se to nezasekne po pádu.
|
|
||||||
3. Jinak Nanobot.from_config() + bot.run(<drain goal>) — vyprázdní VŠECHNO
|
|
||||||
nasbírané v jednom dávkovém běhu (jeden index/graph update pro víc zdrojů).
|
|
||||||
4. Tiše: jen append do log/wiki_compile_cron.log; žádný Telegram.
|
|
||||||
|
|
||||||
Vzor = skills/detach/scripts/tasks-daemon.py (shebang uv run, deps nanobot-ai,
|
|
||||||
Nanobot.from_config + asyncio.wait_for(bot.run(...), timeout)).
|
|
||||||
"""
|
|
||||||
|
|
||||||
import asyncio
|
|
||||||
import json
|
|
||||||
import os
|
|
||||||
import sys
|
|
||||||
import traceback
|
|
||||||
from datetime import datetime
|
|
||||||
from pathlib import Path
|
|
||||||
|
|
||||||
# Skript žije v workspace/skills/llm-wiki/scripts/ → parents[3] = workspace.
|
|
||||||
WORKSPACE = Path(__file__).resolve().parents[3]
|
|
||||||
CML = WORKSPACE / "cml"
|
|
||||||
RAW = CML / "raw"
|
|
||||||
LOCK = CML / ".compile.lock"
|
|
||||||
LOG = WORKSPACE / "log" / "wiki_compile_cron.log"
|
|
||||||
|
|
||||||
# Podadresáře v raw/, které NEjsou pending zdroje.
|
|
||||||
RESERVED_DIRS = {"_done", "_hard", "assets"}
|
|
||||||
|
|
||||||
TIMEOUT_SECONDS = 25 * 60
|
|
||||||
STALE_SECONDS = 30 * 60
|
|
||||||
|
|
||||||
DRAIN_GOAL = (
|
|
||||||
"Pomocí skillu llm-wiki (operace Compile/drain) zkompiluj VŠECHNY nezpracované zdroje "
|
|
||||||
"v `cml/raw/` (regulérní soubory přímo v `cml/raw/`, mimo `_done/`, `_hard/`, `assets/`) "
|
|
||||||
"do wiki v `cml/wiki/`. Pro každý zdroj proveď plný ingest podle "
|
|
||||||
"references/ingest-workflow.md: source/entity/concept stránky s frontmatterem a `[[odkazy]]`, "
|
|
||||||
"aktualizuj `cml/wiki/index.md` a `cml/wiki/log.md`. Po zpracování všech zdrojů regeneruj graph "
|
|
||||||
"(`wiki_graph_lint.py` + `wiki_graph_extract.py` na `cml/wiki/`). Každý úspěšně zpracovaný "
|
|
||||||
"zdroj přesuň do `cml/raw/_done/`. Ambiguózní/konfliktní zdroj NEcompiluj natvrdo — nech ho "
|
|
||||||
"v `cml/raw/` (nebo přesuň do `cml/raw/_hard/`) a důvod zaznamenej do `cml/wiki/log.md`. "
|
|
||||||
"Lint je report-only: žádné destruktivní úpravy existujících stránek bez potvrzení. "
|
|
||||||
"Idempotence: pokud pro zdroj už stránky existují (byl zkompilován dřív, jen nepřesunut), "
|
|
||||||
"NEcykluj reconciliací — ber ho jako hotový, přesuň raw soubor do `cml/raw/_done/` a pokračuj. "
|
|
||||||
"Každý vyřízený zdroj VŽDY přesuň z `cml/raw/` pryč, ať ho příští cron tik nezpracovává znovu. "
|
|
||||||
"Běžíš v izolované session na pozadí, bez interakce s uživatelem."
|
|
||||||
)
|
|
||||||
|
|
||||||
|
|
||||||
def log(message: str) -> None:
|
|
||||||
LOG.parent.mkdir(parents=True, exist_ok=True)
|
|
||||||
stamp = datetime.now().astimezone().isoformat(timespec="seconds")
|
|
||||||
with LOG.open("a", encoding="utf-8") as handle:
|
|
||||||
handle.write(f"{stamp} {message}\n")
|
|
||||||
|
|
||||||
|
|
||||||
def pending_sources() -> list[Path]:
|
|
||||||
"""Regulérní soubory přímo v cml/raw/ (mimo skryté a rezervované podadresáře)."""
|
|
||||||
if not RAW.exists():
|
|
||||||
return []
|
|
||||||
return [p for p in sorted(RAW.iterdir()) if p.is_file() and not p.name.startswith(".")]
|
|
||||||
|
|
||||||
|
|
||||||
def _pid_alive(pid: int) -> bool:
|
|
||||||
try:
|
|
||||||
os.kill(pid, 0)
|
|
||||||
except ProcessLookupError:
|
|
||||||
return False
|
|
||||||
except PermissionError:
|
|
||||||
return True
|
|
||||||
return True
|
|
||||||
|
|
||||||
|
|
||||||
def _lock_is_stale() -> bool:
|
|
||||||
"""Lock je mrtvý, když ho nelze přečíst, proces neběží, nebo je starší než STALE_SECONDS."""
|
|
||||||
try:
|
|
||||||
data = json.loads(LOCK.read_text())
|
|
||||||
pid = int(data["pid"])
|
|
||||||
started = datetime.fromisoformat(data["started"])
|
|
||||||
except (OSError, ValueError, KeyError):
|
|
||||||
return True
|
|
||||||
if not _pid_alive(pid):
|
|
||||||
return True
|
|
||||||
age = (datetime.now().astimezone() - started).total_seconds()
|
|
||||||
return age > STALE_SECONDS
|
|
||||||
|
|
||||||
|
|
||||||
def acquire_lock() -> bool:
|
|
||||||
"""Atomicky vytvoř lock. Vrať False, když už běží živý compile."""
|
|
||||||
for _ in range(2):
|
|
||||||
try:
|
|
||||||
fd = os.open(LOCK, os.O_CREAT | os.O_EXCL | os.O_WRONLY, 0o644)
|
|
||||||
except FileExistsError:
|
|
||||||
if not _lock_is_stale():
|
|
||||||
return False
|
|
||||||
log("stale lock, reclaiming")
|
|
||||||
LOCK.unlink(missing_ok=True)
|
|
||||||
continue
|
|
||||||
payload = {"pid": os.getpid(), "started": datetime.now().astimezone().isoformat()}
|
|
||||||
with os.fdopen(fd, "w", encoding="utf-8") as handle:
|
|
||||||
json.dump(payload, handle)
|
|
||||||
return True
|
|
||||||
return False
|
|
||||||
|
|
||||||
|
|
||||||
async def run_compile(goal: str) -> str:
|
|
||||||
# Heavy import deferred: the per-minute pre-check (no pending work) must not
|
|
||||||
# pay the nanobot import cost — only an actual compile run needs it.
|
|
||||||
from nanobot import Nanobot
|
|
||||||
|
|
||||||
bot = Nanobot.from_config()
|
|
||||||
result = await bot.run(goal, session_key="wiki-compile")
|
|
||||||
return result.content or ""
|
|
||||||
|
|
||||||
|
|
||||||
def main() -> int:
|
|
||||||
dry_run = "--dry-run" in sys.argv[1:]
|
|
||||||
|
|
||||||
pending = pending_sources()
|
|
||||||
if not pending:
|
|
||||||
return 0
|
|
||||||
|
|
||||||
if not acquire_lock():
|
|
||||||
log(f"SKIP compile already running ({len(pending)} pending)")
|
|
||||||
return 0
|
|
||||||
|
|
||||||
if dry_run:
|
|
||||||
names = ", ".join(p.name for p in pending)
|
|
||||||
log(f"DRY-RUN would compile {len(pending)} pending: {names}")
|
|
||||||
LOCK.unlink(missing_ok=True)
|
|
||||||
return 0
|
|
||||||
|
|
||||||
started = datetime.now().astimezone()
|
|
||||||
log(f"START compile {len(pending)} pending: {', '.join(p.name for p in pending)}")
|
|
||||||
try:
|
|
||||||
result_text = asyncio.run(
|
|
||||||
asyncio.wait_for(run_compile(DRAIN_GOAL), timeout=TIMEOUT_SECONDS)
|
|
||||||
)
|
|
||||||
summary = result_text.strip().splitlines()[0][:200] if result_text.strip() else "(prázdný výstup)"
|
|
||||||
duration = int((datetime.now().astimezone() - started).total_seconds())
|
|
||||||
log(f"END compile duration={duration}s remaining={len(pending_sources())} :: {summary}")
|
|
||||||
return 0
|
|
||||||
except asyncio.TimeoutError:
|
|
||||||
log(f"TIMEOUT compile po {TIMEOUT_SECONDS // 60} min")
|
|
||||||
return 1
|
|
||||||
except Exception as error:
|
|
||||||
log(f"EXCEPTION compile: {error}\n{traceback.format_exc()}")
|
|
||||||
return 1
|
|
||||||
finally:
|
|
||||||
LOCK.unlink(missing_ok=True)
|
|
||||||
|
|
||||||
|
|
||||||
if __name__ == "__main__":
|
|
||||||
sys.exit(main())
|
|
||||||
@@ -1,541 +0,0 @@
|
|||||||
#!/usr/bin/env -S uv run --script
|
|
||||||
# /// script
|
|
||||||
# requires-python = ">=3.11"
|
|
||||||
# dependencies = ["pyyaml"]
|
|
||||||
# ///
|
|
||||||
"""
|
|
||||||
wiki_graph_extract.py — Compile the markdown wiki into a queryable graph.
|
|
||||||
|
|
||||||
Markdown remains canonical. This script reads every wiki page, derives nodes
|
|
||||||
and edges (typed semantic edges from `graph.relationships`, plus implicit
|
|
||||||
`mentions`, `sourced_from`, `summarizes_raw` edges), and emits artifacts under
|
|
||||||
`<wiki>/graph/` that can be deleted and rebuilt at any time.
|
|
||||||
|
|
||||||
Requires PyYAML (`pip install pyyaml`) — the new graph layer uses real YAML
|
|
||||||
parsing for its nested frontmatter, unlike the stdlib-only lint/search/stats
|
|
||||||
scripts.
|
|
||||||
|
|
||||||
Usage:
|
|
||||||
python wiki_graph_extract.py <wiki-dir> [options]
|
|
||||||
|
|
||||||
Options:
|
|
||||||
--out <dir> Output directory (default: <wiki-dir>/graph)
|
|
||||||
--formats jsonl,sqlite,... Comma-list of formats to emit
|
|
||||||
(jsonl, sqlite, graphml; default: all three)
|
|
||||||
--ontology <path> Override ontology path
|
|
||||||
(default: <wiki-dir>/graph/ontology.yaml)
|
|
||||||
|
|
||||||
Examples:
|
|
||||||
python wiki_graph_extract.py wiki/
|
|
||||||
python wiki_graph_extract.py wiki/ --out wiki/graph --formats jsonl,sqlite
|
|
||||||
"""
|
|
||||||
|
|
||||||
import argparse
|
|
||||||
import hashlib
|
|
||||||
import json
|
|
||||||
import re
|
|
||||||
import sqlite3
|
|
||||||
import sys
|
|
||||||
import xml.etree.ElementTree as ET
|
|
||||||
from collections import defaultdict
|
|
||||||
from pathlib import Path
|
|
||||||
|
|
||||||
try:
|
|
||||||
import yaml
|
|
||||||
except ImportError:
|
|
||||||
print(
|
|
||||||
"wiki_graph_extract.py requires PyYAML.\n"
|
|
||||||
"Install with: pip install pyyaml",
|
|
||||||
file=sys.stderr,
|
|
||||||
)
|
|
||||||
sys.exit(2)
|
|
||||||
|
|
||||||
|
|
||||||
WIKILINK_RE = re.compile(r"\[\[([^\]|]+)(?:\|[^\]]+)?\]\]")
|
|
||||||
FRONTMATTER_RE = re.compile(r"^---\s*\n(.*?)\n---\s*\n", re.DOTALL)
|
|
||||||
|
|
||||||
SKIP_TOP_LEVEL_FILES = {"SCHEMA.md", "index.md", "log.md", "README.md"}
|
|
||||||
SKIP_TOP_LEVEL_DIRS = {"indexes", "graph"}
|
|
||||||
|
|
||||||
DEFAULT_FORMATS = ["jsonl", "sqlite", "graphml"]
|
|
||||||
|
|
||||||
|
|
||||||
# ---------------------------------------------------------------------------
|
|
||||||
# Page collection
|
|
||||||
# ---------------------------------------------------------------------------
|
|
||||||
|
|
||||||
|
|
||||||
def parse_frontmatter(text: str) -> tuple[dict, str]:
|
|
||||||
"""Extract YAML frontmatter using PyYAML. Returns (meta, body)."""
|
|
||||||
m = FRONTMATTER_RE.match(text)
|
|
||||||
if not m:
|
|
||||||
return {}, text
|
|
||||||
fm_text = m.group(1)
|
|
||||||
body = text[m.end():]
|
|
||||||
try:
|
|
||||||
meta = yaml.safe_load(fm_text) or {}
|
|
||||||
except yaml.YAMLError:
|
|
||||||
meta = {}
|
|
||||||
if not isinstance(meta, dict):
|
|
||||||
meta = {}
|
|
||||||
return meta, body
|
|
||||||
|
|
||||||
|
|
||||||
def collect_pages(wiki_root: Path) -> list[dict]:
|
|
||||||
pages = []
|
|
||||||
for md_path in sorted(wiki_root.rglob("*.md")):
|
|
||||||
rel = md_path.relative_to(wiki_root)
|
|
||||||
if rel.parts[0] in SKIP_TOP_LEVEL_FILES or rel.parts[0] in SKIP_TOP_LEVEL_DIRS:
|
|
||||||
continue
|
|
||||||
if rel.name.startswith("."):
|
|
||||||
continue
|
|
||||||
try:
|
|
||||||
text = md_path.read_text(encoding="utf-8")
|
|
||||||
except (UnicodeDecodeError, OSError):
|
|
||||||
continue
|
|
||||||
meta, body = parse_frontmatter(text)
|
|
||||||
links = [m.group(1).strip() for m in WIKILINK_RE.finditer(body)]
|
|
||||||
pages.append({
|
|
||||||
"path": str(md_path),
|
|
||||||
"rel_path": str(rel).replace("\\", "/"),
|
|
||||||
"slug": md_path.stem,
|
|
||||||
"meta": meta,
|
|
||||||
"body": body,
|
|
||||||
"links": links,
|
|
||||||
})
|
|
||||||
return pages
|
|
||||||
|
|
||||||
|
|
||||||
# ---------------------------------------------------------------------------
|
|
||||||
# Ontology
|
|
||||||
# ---------------------------------------------------------------------------
|
|
||||||
|
|
||||||
|
|
||||||
def load_ontology(path: Path) -> dict:
|
|
||||||
if not path.exists():
|
|
||||||
return {"node_types": {}, "predicates": {}}
|
|
||||||
try:
|
|
||||||
data = yaml.safe_load(path.read_text(encoding="utf-8")) or {}
|
|
||||||
except yaml.YAMLError as e:
|
|
||||||
print(f"Ontology parse error ({path}): {e}", file=sys.stderr)
|
|
||||||
sys.exit(2)
|
|
||||||
data.setdefault("node_types", {})
|
|
||||||
data.setdefault("predicates", {})
|
|
||||||
return data
|
|
||||||
|
|
||||||
|
|
||||||
def derive_node_type(meta: dict, ontology: dict) -> str | None:
|
|
||||||
"""Map a page's frontmatter to a node_type using ontology[node_types][*].maps_from."""
|
|
||||||
page_type = meta.get("type")
|
|
||||||
page_kind = meta.get("kind")
|
|
||||||
explicit = (meta.get("graph") or {}).get("node_type") if isinstance(meta.get("graph"), dict) else None
|
|
||||||
if explicit:
|
|
||||||
return explicit
|
|
||||||
# Try (type, kind) match first, then type-only.
|
|
||||||
type_kind_match = None
|
|
||||||
type_only_match = None
|
|
||||||
for nt_name, nt_def in ontology["node_types"].items():
|
|
||||||
maps = (nt_def or {}).get("maps_from") or {}
|
|
||||||
m_type = maps.get("type")
|
|
||||||
m_kind = maps.get("kind")
|
|
||||||
if m_type and m_type == page_type:
|
|
||||||
if m_kind and m_kind == page_kind:
|
|
||||||
type_kind_match = nt_name
|
|
||||||
break
|
|
||||||
if not m_kind and type_only_match is None:
|
|
||||||
type_only_match = nt_name
|
|
||||||
return type_kind_match or type_only_match
|
|
||||||
|
|
||||||
|
|
||||||
# ---------------------------------------------------------------------------
|
|
||||||
# Node + edge construction
|
|
||||||
# ---------------------------------------------------------------------------
|
|
||||||
|
|
||||||
|
|
||||||
def build_nodes(pages: list[dict], ontology: dict) -> tuple[list[dict], dict, list[dict]]:
|
|
||||||
"""Build the node list + slug→node_id index + alias rows. Returns (nodes, slug_to_id, aliases)."""
|
|
||||||
nodes: list[dict] = []
|
|
||||||
slug_to_id: dict[str, str] = {}
|
|
||||||
aliases: list[dict] = []
|
|
||||||
seen_ids: set[str] = set()
|
|
||||||
|
|
||||||
for p in pages:
|
|
||||||
meta = p["meta"]
|
|
||||||
graph_meta = meta.get("graph") if isinstance(meta.get("graph"), dict) else {}
|
|
||||||
node_type = derive_node_type(meta, ontology) or "concept"
|
|
||||||
explicit_id = graph_meta.get("node_id")
|
|
||||||
node_id = explicit_id or f"{node_type}:{p['slug']}"
|
|
||||||
|
|
||||||
# Skip duplicates — first one wins; lint will flag this.
|
|
||||||
if node_id in seen_ids:
|
|
||||||
continue
|
|
||||||
seen_ids.add(node_id)
|
|
||||||
|
|
||||||
node = {
|
|
||||||
"id": node_id,
|
|
||||||
"slug": p["slug"],
|
|
||||||
"title": meta.get("title") or p["slug"],
|
|
||||||
"page_type": meta.get("type") or "",
|
|
||||||
"node_type": node_type,
|
|
||||||
"kind": meta.get("kind") or "",
|
|
||||||
"tags": list(meta.get("tags") or []),
|
|
||||||
"aliases": list(graph_meta.get("aliases") or []),
|
|
||||||
"path": p["rel_path"],
|
|
||||||
"created": meta.get("created") or "",
|
|
||||||
"updated": meta.get("updated") or "",
|
|
||||||
"canonical": bool(graph_meta.get("canonical", False)),
|
|
||||||
}
|
|
||||||
nodes.append(node)
|
|
||||||
slug_to_id[p["slug"]] = node_id
|
|
||||||
for alias in node["aliases"]:
|
|
||||||
aliases.append({"alias": str(alias), "node_id": node_id})
|
|
||||||
|
|
||||||
return nodes, slug_to_id, aliases
|
|
||||||
|
|
||||||
|
|
||||||
def edge_id(subject: str, predicate: str, obj: str, source: str | None, evidence: str | None) -> str:
|
|
||||||
# Truncated to 96 bits — collision risk is negligible at any plausible
|
|
||||||
# wiki scale and shorter ids keep the JSONL/sqlite/graphml outputs readable.
|
|
||||||
h = hashlib.sha256()
|
|
||||||
parts = [subject or "", predicate or "", obj or "", source or "", evidence or ""]
|
|
||||||
h.update("\x1f".join(parts).encode("utf-8"))
|
|
||||||
return h.hexdigest()[:24]
|
|
||||||
|
|
||||||
|
|
||||||
def make_edge(*, subject, predicate, obj, source, evidence, confidence, status,
|
|
||||||
extraction_method, page, extras: dict | None = None) -> dict:
|
|
||||||
return {
|
|
||||||
"id": edge_id(subject, predicate, obj, source, evidence),
|
|
||||||
"subject": subject,
|
|
||||||
"predicate": predicate,
|
|
||||||
"object": obj,
|
|
||||||
"source": source or "",
|
|
||||||
"evidence": evidence or "",
|
|
||||||
"confidence": confidence or "",
|
|
||||||
"status": status or "",
|
|
||||||
"extraction_method": extraction_method,
|
|
||||||
"page": page,
|
|
||||||
"extras": extras or {},
|
|
||||||
}
|
|
||||||
|
|
||||||
|
|
||||||
def build_edges(pages: list[dict], slug_to_id: dict[str, str]) -> list[dict]:
|
|
||||||
edges: list[dict] = []
|
|
||||||
seen_ids: set[str] = set()
|
|
||||||
|
|
||||||
def push(edge: dict) -> None:
|
|
||||||
if edge["id"] in seen_ids:
|
|
||||||
return
|
|
||||||
seen_ids.add(edge["id"])
|
|
||||||
edges.append(edge)
|
|
||||||
|
|
||||||
for p in pages:
|
|
||||||
slug = p["slug"]
|
|
||||||
subject_id = slug_to_id.get(slug)
|
|
||||||
if not subject_id:
|
|
||||||
continue
|
|
||||||
meta = p["meta"]
|
|
||||||
graph_meta = meta.get("graph") if isinstance(meta.get("graph"), dict) else {}
|
|
||||||
|
|
||||||
# 1. Typed semantic edges from graph.relationships[].
|
|
||||||
for rel in graph_meta.get("relationships") or []:
|
|
||||||
if not isinstance(rel, dict):
|
|
||||||
continue
|
|
||||||
obj = rel.get("object")
|
|
||||||
predicate = rel.get("predicate")
|
|
||||||
if not (obj and predicate):
|
|
||||||
continue
|
|
||||||
extras = {
|
|
||||||
k: rel[k] for k in ("valid_from", "valid_to", "notes", "raw_ref",
|
|
||||||
"contradicts", "supersedes")
|
|
||||||
if k in rel and rel[k] is not None
|
|
||||||
}
|
|
||||||
push(make_edge(
|
|
||||||
subject=subject_id,
|
|
||||||
predicate=str(predicate),
|
|
||||||
obj=str(obj),
|
|
||||||
source=rel.get("source"),
|
|
||||||
evidence=rel.get("evidence"),
|
|
||||||
confidence=rel.get("confidence"),
|
|
||||||
status=rel.get("status"),
|
|
||||||
extraction_method="explicit_graph_frontmatter",
|
|
||||||
page=p["rel_path"],
|
|
||||||
extras=extras,
|
|
||||||
))
|
|
||||||
|
|
||||||
# 2. Mentions edges from body wikilinks.
|
|
||||||
seen_targets: set[str] = set()
|
|
||||||
for link in p["links"]:
|
|
||||||
target_slug = link.split("#")[0].strip()
|
|
||||||
if not target_slug or target_slug == slug:
|
|
||||||
continue
|
|
||||||
target_id = slug_to_id.get(target_slug)
|
|
||||||
if not target_id or target_id in seen_targets:
|
|
||||||
continue
|
|
||||||
seen_targets.add(target_id)
|
|
||||||
push(make_edge(
|
|
||||||
subject=subject_id,
|
|
||||||
predicate="mentions",
|
|
||||||
obj=target_id,
|
|
||||||
source=None,
|
|
||||||
evidence=None,
|
|
||||||
confidence="low",
|
|
||||||
status="current",
|
|
||||||
extraction_method="body_wikilink",
|
|
||||||
page=p["rel_path"],
|
|
||||||
))
|
|
||||||
|
|
||||||
# 3. sourced_from edges from frontmatter `sources:` (skip on source pages themselves).
|
|
||||||
if meta.get("type") != "source":
|
|
||||||
for src_slug in meta.get("sources") or []:
|
|
||||||
src_id = slug_to_id.get(str(src_slug))
|
|
||||||
if not src_id:
|
|
||||||
continue
|
|
||||||
push(make_edge(
|
|
||||||
subject=subject_id,
|
|
||||||
predicate="sourced_from",
|
|
||||||
obj=src_id,
|
|
||||||
source=str(src_slug),
|
|
||||||
evidence=None,
|
|
||||||
confidence="high",
|
|
||||||
status="current",
|
|
||||||
extraction_method="frontmatter_sources",
|
|
||||||
page=p["rel_path"],
|
|
||||||
))
|
|
||||||
|
|
||||||
# 4. summarizes_raw edges from source pages' raw: field.
|
|
||||||
if meta.get("type") == "source":
|
|
||||||
raw_path = meta.get("raw")
|
|
||||||
if raw_path:
|
|
||||||
push(make_edge(
|
|
||||||
subject=subject_id,
|
|
||||||
predicate="summarizes_raw",
|
|
||||||
obj=f"raw:{raw_path}",
|
|
||||||
source=None,
|
|
||||||
evidence=None,
|
|
||||||
confidence="high",
|
|
||||||
status="current",
|
|
||||||
extraction_method="frontmatter_raw",
|
|
||||||
page=p["rel_path"],
|
|
||||||
))
|
|
||||||
|
|
||||||
return edges
|
|
||||||
|
|
||||||
|
|
||||||
# ---------------------------------------------------------------------------
|
|
||||||
# Output writers
|
|
||||||
# ---------------------------------------------------------------------------
|
|
||||||
|
|
||||||
|
|
||||||
def _normalize_for_json(value):
|
|
||||||
if hasattr(value, "isoformat"):
|
|
||||||
return value.isoformat()
|
|
||||||
if isinstance(value, list):
|
|
||||||
return [_normalize_for_json(v) for v in value]
|
|
||||||
if isinstance(value, dict):
|
|
||||||
return {k: _normalize_for_json(v) for k, v in value.items()}
|
|
||||||
return value
|
|
||||||
|
|
||||||
|
|
||||||
def write_jsonl(out_dir: Path, nodes: list[dict], edges: list[dict]) -> None:
|
|
||||||
nodes_sorted = sorted(nodes, key=lambda n: n["id"])
|
|
||||||
edges_sorted = sorted(edges, key=lambda e: e["id"])
|
|
||||||
with (out_dir / "nodes.jsonl").open("w", encoding="utf-8") as f:
|
|
||||||
for n in nodes_sorted:
|
|
||||||
f.write(json.dumps(_normalize_for_json(n), sort_keys=True, ensure_ascii=False))
|
|
||||||
f.write("\n")
|
|
||||||
with (out_dir / "edges.jsonl").open("w", encoding="utf-8") as f:
|
|
||||||
for e in edges_sorted:
|
|
||||||
f.write(json.dumps(_normalize_for_json(e), sort_keys=True, ensure_ascii=False))
|
|
||||||
f.write("\n")
|
|
||||||
|
|
||||||
|
|
||||||
def write_sqlite(out_dir: Path, nodes: list[dict], aliases: list[dict], edges: list[dict]) -> None:
|
|
||||||
db_path = out_dir / "graph.sqlite"
|
|
||||||
if db_path.exists():
|
|
||||||
db_path.unlink()
|
|
||||||
conn = sqlite3.connect(db_path)
|
|
||||||
try:
|
|
||||||
conn.executescript("""
|
|
||||||
CREATE TABLE nodes (
|
|
||||||
id TEXT PRIMARY KEY,
|
|
||||||
slug TEXT NOT NULL UNIQUE,
|
|
||||||
title TEXT NOT NULL,
|
|
||||||
page_type TEXT NOT NULL,
|
|
||||||
node_type TEXT NOT NULL,
|
|
||||||
kind TEXT,
|
|
||||||
path TEXT NOT NULL,
|
|
||||||
created TEXT,
|
|
||||||
updated TEXT,
|
|
||||||
metadata_json TEXT NOT NULL
|
|
||||||
);
|
|
||||||
CREATE TABLE aliases (
|
|
||||||
alias TEXT NOT NULL,
|
|
||||||
node_id TEXT NOT NULL,
|
|
||||||
PRIMARY KEY (alias, node_id),
|
|
||||||
FOREIGN KEY (node_id) REFERENCES nodes(id)
|
|
||||||
);
|
|
||||||
CREATE TABLE edges (
|
|
||||||
id TEXT PRIMARY KEY,
|
|
||||||
subject TEXT NOT NULL,
|
|
||||||
predicate TEXT NOT NULL,
|
|
||||||
object TEXT NOT NULL,
|
|
||||||
source TEXT,
|
|
||||||
evidence TEXT,
|
|
||||||
confidence TEXT,
|
|
||||||
status TEXT,
|
|
||||||
extraction_method TEXT NOT NULL,
|
|
||||||
page TEXT NOT NULL,
|
|
||||||
metadata_json TEXT NOT NULL
|
|
||||||
);
|
|
||||||
CREATE INDEX idx_edges_subject ON edges(subject);
|
|
||||||
CREATE INDEX idx_edges_object ON edges(object);
|
|
||||||
CREATE INDEX idx_edges_predicate ON edges(predicate);
|
|
||||||
CREATE INDEX idx_edges_source ON edges(source);
|
|
||||||
""")
|
|
||||||
|
|
||||||
for n in sorted(nodes, key=lambda n: n["id"]):
|
|
||||||
metadata_json = json.dumps(_normalize_for_json({
|
|
||||||
"tags": n.get("tags", []),
|
|
||||||
"aliases": n.get("aliases", []),
|
|
||||||
"canonical": n.get("canonical", False),
|
|
||||||
}), sort_keys=True, ensure_ascii=False)
|
|
||||||
conn.execute(
|
|
||||||
"INSERT INTO nodes (id, slug, title, page_type, node_type, kind, path, created, updated, metadata_json) "
|
|
||||||
"VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, ?)",
|
|
||||||
(
|
|
||||||
n["id"], n["slug"], n["title"], n["page_type"], n["node_type"],
|
|
||||||
n.get("kind") or None, n["path"],
|
|
||||||
str(n.get("created") or "") or None,
|
|
||||||
str(n.get("updated") or "") or None,
|
|
||||||
metadata_json,
|
|
||||||
),
|
|
||||||
)
|
|
||||||
|
|
||||||
for a in sorted(aliases, key=lambda a: (a["alias"], a["node_id"])):
|
|
||||||
conn.execute(
|
|
||||||
"INSERT OR IGNORE INTO aliases (alias, node_id) VALUES (?, ?)",
|
|
||||||
(a["alias"], a["node_id"]),
|
|
||||||
)
|
|
||||||
|
|
||||||
for e in sorted(edges, key=lambda e: e["id"]):
|
|
||||||
metadata_json = json.dumps(_normalize_for_json(e.get("extras") or {}),
|
|
||||||
sort_keys=True, ensure_ascii=False)
|
|
||||||
conn.execute(
|
|
||||||
"INSERT INTO edges (id, subject, predicate, object, source, evidence, "
|
|
||||||
"confidence, status, extraction_method, page, metadata_json) "
|
|
||||||
"VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?)",
|
|
||||||
(
|
|
||||||
e["id"], e["subject"], e["predicate"], e["object"],
|
|
||||||
e.get("source") or None, e.get("evidence") or None,
|
|
||||||
e.get("confidence") or None, e.get("status") or None,
|
|
||||||
e["extraction_method"], e["page"], metadata_json,
|
|
||||||
),
|
|
||||||
)
|
|
||||||
conn.commit()
|
|
||||||
finally:
|
|
||||||
conn.close()
|
|
||||||
|
|
||||||
|
|
||||||
def write_graphml(out_dir: Path, nodes: list[dict], edges: list[dict]) -> None:
|
|
||||||
ns = "http://graphml.graphdrawing.org/xmlns"
|
|
||||||
ET.register_namespace("", ns)
|
|
||||||
root = ET.Element(f"{{{ns}}}graphml")
|
|
||||||
|
|
||||||
keys = [
|
|
||||||
("d_title", "node", "title", "string"),
|
|
||||||
("d_node_type", "node", "node_type", "string"),
|
|
||||||
("d_page_type", "node", "page_type", "string"),
|
|
||||||
("d_path", "node", "path", "string"),
|
|
||||||
("d_predicate", "edge", "predicate", "string"),
|
|
||||||
("d_confidence", "edge", "confidence", "string"),
|
|
||||||
("d_status", "edge", "status", "string"),
|
|
||||||
("d_source", "edge", "source", "string"),
|
|
||||||
]
|
|
||||||
for kid, kfor, kname, ktype in keys:
|
|
||||||
k = ET.SubElement(root, f"{{{ns}}}key")
|
|
||||||
k.set("id", kid)
|
|
||||||
k.set("for", kfor)
|
|
||||||
k.set("attr.name", kname)
|
|
||||||
k.set("attr.type", ktype)
|
|
||||||
|
|
||||||
graph = ET.SubElement(root, f"{{{ns}}}graph")
|
|
||||||
graph.set("id", "wiki")
|
|
||||||
graph.set("edgedefault", "directed")
|
|
||||||
|
|
||||||
for n in sorted(nodes, key=lambda n: n["id"]):
|
|
||||||
node_el = ET.SubElement(graph, f"{{{ns}}}node")
|
|
||||||
node_el.set("id", n["id"])
|
|
||||||
for kid, kfor, kname, _ in keys:
|
|
||||||
if kfor != "node":
|
|
||||||
continue
|
|
||||||
data = ET.SubElement(node_el, f"{{{ns}}}data")
|
|
||||||
data.set("key", kid)
|
|
||||||
data.text = str(n.get(kname) or "")
|
|
||||||
|
|
||||||
for e in sorted(edges, key=lambda e: e["id"]):
|
|
||||||
edge_el = ET.SubElement(graph, f"{{{ns}}}edge")
|
|
||||||
edge_el.set("id", e["id"])
|
|
||||||
edge_el.set("source", e["subject"])
|
|
||||||
edge_el.set("target", e["object"])
|
|
||||||
for kid, kfor, kname, _ in keys:
|
|
||||||
if kfor != "edge":
|
|
||||||
continue
|
|
||||||
data = ET.SubElement(edge_el, f"{{{ns}}}data")
|
|
||||||
data.set("key", kid)
|
|
||||||
data.text = str(e.get(kname) or "")
|
|
||||||
|
|
||||||
tree = ET.ElementTree(root)
|
|
||||||
ET.indent(tree, space=" ")
|
|
||||||
tree.write(out_dir / "graph.graphml", encoding="utf-8", xml_declaration=True)
|
|
||||||
|
|
||||||
|
|
||||||
# ---------------------------------------------------------------------------
|
|
||||||
# CLI
|
|
||||||
# ---------------------------------------------------------------------------
|
|
||||||
|
|
||||||
|
|
||||||
def main():
|
|
||||||
parser = argparse.ArgumentParser(description=__doc__, formatter_class=argparse.RawDescriptionHelpFormatter)
|
|
||||||
parser.add_argument("wiki", type=Path, help="Wiki directory.")
|
|
||||||
parser.add_argument("--out", type=Path, help="Output directory (default: <wiki>/graph)")
|
|
||||||
parser.add_argument("--formats", default=",".join(DEFAULT_FORMATS),
|
|
||||||
help="Comma-list: jsonl, sqlite, graphml")
|
|
||||||
parser.add_argument("--ontology", type=Path, help="Ontology file (default: <wiki>/graph/ontology.yaml)")
|
|
||||||
args = parser.parse_args()
|
|
||||||
|
|
||||||
if not args.wiki.exists():
|
|
||||||
print(f"Wiki directory not found: {args.wiki}", file=sys.stderr)
|
|
||||||
sys.exit(1)
|
|
||||||
|
|
||||||
out_dir = args.out or (args.wiki / "graph")
|
|
||||||
out_dir.mkdir(parents=True, exist_ok=True)
|
|
||||||
ontology_path = args.ontology or (args.wiki / "graph" / "ontology.yaml")
|
|
||||||
ontology = load_ontology(ontology_path)
|
|
||||||
formats = [f.strip().lower() for f in args.formats.split(",") if f.strip()]
|
|
||||||
unknown = [f for f in formats if f not in DEFAULT_FORMATS]
|
|
||||||
if unknown:
|
|
||||||
print(f"Unknown formats: {unknown}. Allowed: {DEFAULT_FORMATS}", file=sys.stderr)
|
|
||||||
sys.exit(1)
|
|
||||||
|
|
||||||
pages = collect_pages(args.wiki)
|
|
||||||
nodes, slug_to_id, aliases = build_nodes(pages, ontology)
|
|
||||||
edges = build_edges(pages, slug_to_id)
|
|
||||||
|
|
||||||
if "jsonl" in formats:
|
|
||||||
write_jsonl(out_dir, nodes, edges)
|
|
||||||
if "sqlite" in formats:
|
|
||||||
write_sqlite(out_dir, nodes, aliases, edges)
|
|
||||||
if "graphml" in formats:
|
|
||||||
write_graphml(out_dir, nodes, edges)
|
|
||||||
|
|
||||||
print(f"Extracted {len(nodes)} nodes, {len(edges)} edges → {out_dir}")
|
|
||||||
breakdown = defaultdict(int)
|
|
||||||
for e in edges:
|
|
||||||
breakdown[e["predicate"]] += 1
|
|
||||||
for pred, count in sorted(breakdown.items(), key=lambda x: (-x[1], x[0])):
|
|
||||||
print(f" {pred:20s} {count}")
|
|
||||||
|
|
||||||
|
|
||||||
if __name__ == "__main__":
|
|
||||||
main()
|
|
||||||
@@ -1,418 +0,0 @@
|
|||||||
#!/usr/bin/env -S uv run --script
|
|
||||||
# /// script
|
|
||||||
# requires-python = ">=3.11"
|
|
||||||
# dependencies = ["pyyaml"]
|
|
||||||
# ///
|
|
||||||
"""
|
|
||||||
wiki_graph_lint.py — Validate the typed graph metadata in a wiki.
|
|
||||||
|
|
||||||
Reads every page's `graph:` frontmatter, cross-checks against the ontology
|
|
||||||
(`wiki/graph/ontology.yaml`), and reports problems. Conservative by design:
|
|
||||||
reports only, never edits.
|
|
||||||
|
|
||||||
Requires PyYAML (`pip install pyyaml`).
|
|
||||||
|
|
||||||
Checks:
|
|
||||||
- Unique `graph.node_id` values across pages.
|
|
||||||
- All relationship `object` ids resolve to known nodes (or are allowed
|
|
||||||
string-literal targets for predicates whose object_types include "*").
|
|
||||||
- All predicates exist in `graph/ontology.yaml`.
|
|
||||||
- Predicate subject/object types match ontology.
|
|
||||||
- Typed semantic edges (anything except mentions/sourced_from/summarizes_raw
|
|
||||||
and predicates with `requires_evidence: false`) carry `source` and
|
|
||||||
`evidence`.
|
|
||||||
- `source` references resolve to an existing source page.
|
|
||||||
- `confidence` is one of high|medium|low; `status` is one of
|
|
||||||
current|historical|proposed|disputed|superseded.
|
|
||||||
- No duplicate canonical nodes for the same node id.
|
|
||||||
- Aliases do not collide across distinct canonical nodes.
|
|
||||||
- `contradicts` / `supersedes` references resolve to known node/edge ids.
|
|
||||||
- Generated graph has no orphan typed nodes (nodes with no inbound or
|
|
||||||
outbound typed edges) except for `source` nodes (allowed source-only).
|
|
||||||
|
|
||||||
Usage:
|
|
||||||
python wiki_graph_lint.py [<wiki-dir>] [--json]
|
|
||||||
"""
|
|
||||||
|
|
||||||
import argparse
|
|
||||||
import json
|
|
||||||
import re
|
|
||||||
import sys
|
|
||||||
from collections import defaultdict
|
|
||||||
from pathlib import Path
|
|
||||||
|
|
||||||
try:
|
|
||||||
import yaml
|
|
||||||
except ImportError:
|
|
||||||
print(
|
|
||||||
"wiki_graph_lint.py requires PyYAML.\n"
|
|
||||||
"Install with: pip install pyyaml",
|
|
||||||
file=sys.stderr,
|
|
||||||
)
|
|
||||||
sys.exit(2)
|
|
||||||
|
|
||||||
# Same module is imported by extract; we re-use its build_nodes/build_edges to
|
|
||||||
# guarantee lint sees exactly what extract would emit.
|
|
||||||
SCRIPT_DIR = Path(__file__).resolve().parent
|
|
||||||
sys.path.insert(0, str(SCRIPT_DIR))
|
|
||||||
import wiki_graph_extract as _extract # noqa: E402
|
|
||||||
|
|
||||||
|
|
||||||
FRONTMATTER_RE = re.compile(r"^---\s*\n(.*?)\n---\s*\n", re.DOTALL)
|
|
||||||
WIKILINK_RE = re.compile(r"\[\[([^\]|]+)(?:\|[^\]]+)?\]\]")
|
|
||||||
|
|
||||||
SKIP_TOP_LEVEL_FILES = {"SCHEMA.md", "index.md", "log.md", "README.md"}
|
|
||||||
SKIP_TOP_LEVEL_DIRS = {"indexes", "graph"}
|
|
||||||
|
|
||||||
ALLOWED_CONFIDENCE = {"high", "medium", "low"}
|
|
||||||
ALLOWED_STATUS = {"current", "historical", "proposed", "disputed", "superseded"}
|
|
||||||
IMPLICIT_PREDICATES = {"mentions", "sourced_from", "summarizes_raw"}
|
|
||||||
|
|
||||||
|
|
||||||
def parse_frontmatter(text: str) -> tuple[dict, str]:
|
|
||||||
m = FRONTMATTER_RE.match(text)
|
|
||||||
if not m:
|
|
||||||
return {}, text
|
|
||||||
fm_text = m.group(1)
|
|
||||||
body = text[m.end():]
|
|
||||||
try:
|
|
||||||
meta = yaml.safe_load(fm_text) or {}
|
|
||||||
except yaml.YAMLError:
|
|
||||||
meta = {}
|
|
||||||
if not isinstance(meta, dict):
|
|
||||||
meta = {}
|
|
||||||
return meta, body
|
|
||||||
|
|
||||||
|
|
||||||
def collect_pages(wiki_root: Path) -> list[dict]:
|
|
||||||
pages = []
|
|
||||||
for md_path in sorted(wiki_root.rglob("*.md")):
|
|
||||||
rel = md_path.relative_to(wiki_root)
|
|
||||||
if rel.parts[0] in SKIP_TOP_LEVEL_FILES or rel.parts[0] in SKIP_TOP_LEVEL_DIRS:
|
|
||||||
continue
|
|
||||||
if rel.name.startswith("."):
|
|
||||||
continue
|
|
||||||
try:
|
|
||||||
text = md_path.read_text(encoding="utf-8")
|
|
||||||
except (UnicodeDecodeError, OSError):
|
|
||||||
continue
|
|
||||||
meta, body = parse_frontmatter(text)
|
|
||||||
pages.append({
|
|
||||||
"path": str(md_path),
|
|
||||||
"rel_path": str(rel).replace("\\", "/"),
|
|
||||||
"slug": md_path.stem,
|
|
||||||
"meta": meta,
|
|
||||||
"body": body,
|
|
||||||
"links": [m.group(1).strip() for m in WIKILINK_RE.finditer(body)],
|
|
||||||
})
|
|
||||||
return pages
|
|
||||||
|
|
||||||
|
|
||||||
def derive_node_type(meta: dict, ontology: dict) -> str | None:
|
|
||||||
page_type = meta.get("type")
|
|
||||||
page_kind = meta.get("kind")
|
|
||||||
explicit = (meta.get("graph") or {}).get("node_type") if isinstance(meta.get("graph"), dict) else None
|
|
||||||
if explicit:
|
|
||||||
return explicit
|
|
||||||
type_kind_match = None
|
|
||||||
type_only_match = None
|
|
||||||
for nt_name, nt_def in ontology.get("node_types", {}).items():
|
|
||||||
maps = (nt_def or {}).get("maps_from") or {}
|
|
||||||
m_type = maps.get("type")
|
|
||||||
m_kind = maps.get("kind")
|
|
||||||
if m_type and m_type == page_type:
|
|
||||||
if m_kind and m_kind == page_kind:
|
|
||||||
type_kind_match = nt_name
|
|
||||||
break
|
|
||||||
if not m_kind and type_only_match is None:
|
|
||||||
type_only_match = nt_name
|
|
||||||
return type_kind_match or type_only_match
|
|
||||||
|
|
||||||
|
|
||||||
def derive_node_id(meta: dict, slug: str, ontology: dict) -> str:
|
|
||||||
graph_meta = meta.get("graph") if isinstance(meta.get("graph"), dict) else {}
|
|
||||||
explicit = graph_meta.get("node_id")
|
|
||||||
if explicit:
|
|
||||||
return str(explicit)
|
|
||||||
node_type = derive_node_type(meta, ontology) or "concept"
|
|
||||||
return f"{node_type}:{slug}"
|
|
||||||
|
|
||||||
|
|
||||||
def types_match(allowed: list[str] | None, actual: str | None) -> bool:
|
|
||||||
if not allowed:
|
|
||||||
return True
|
|
||||||
if "*" in allowed:
|
|
||||||
return True
|
|
||||||
return actual in allowed
|
|
||||||
|
|
||||||
|
|
||||||
def lint(pages: list[dict], ontology: dict) -> dict:
|
|
||||||
findings = {
|
|
||||||
"duplicate_node_ids": [],
|
|
||||||
"unknown_predicates": [],
|
|
||||||
"broken_object_refs": [],
|
|
||||||
"subject_type_mismatch": [],
|
|
||||||
"object_type_mismatch": [],
|
|
||||||
"missing_evidence": [],
|
|
||||||
"missing_source_field": [],
|
|
||||||
"broken_source_refs": [],
|
|
||||||
"invalid_confidence": [],
|
|
||||||
"invalid_status": [],
|
|
||||||
"duplicate_canonical": [],
|
|
||||||
"alias_collisions": [],
|
|
||||||
"broken_contradicts": [],
|
|
||||||
"broken_supersedes": [],
|
|
||||||
"orphan_typed_nodes": [],
|
|
||||||
"summary": {},
|
|
||||||
}
|
|
||||||
|
|
||||||
predicates = ontology.get("predicates", {})
|
|
||||||
node_types = ontology.get("node_types", {})
|
|
||||||
|
|
||||||
# Build node index
|
|
||||||
node_by_id: dict[str, dict] = {}
|
|
||||||
duplicates: dict[str, list[str]] = defaultdict(list)
|
|
||||||
for p in pages:
|
|
||||||
nid = derive_node_id(p["meta"], p["slug"], ontology)
|
|
||||||
if nid in node_by_id:
|
|
||||||
duplicates[nid].append(p["rel_path"])
|
|
||||||
duplicates[nid].append(node_by_id[nid]["rel_path"])
|
|
||||||
continue
|
|
||||||
node_type = derive_node_type(p["meta"], ontology) or "concept"
|
|
||||||
graph_meta = p["meta"].get("graph") if isinstance(p["meta"].get("graph"), dict) else {}
|
|
||||||
node_by_id[nid] = {
|
|
||||||
"id": nid,
|
|
||||||
"node_type": node_type,
|
|
||||||
"rel_path": p["rel_path"],
|
|
||||||
"slug": p["slug"],
|
|
||||||
"page_type": p["meta"].get("type"),
|
|
||||||
"canonical": bool(graph_meta.get("canonical", False)),
|
|
||||||
"aliases": list(graph_meta.get("aliases") or []),
|
|
||||||
"graph": graph_meta,
|
|
||||||
}
|
|
||||||
|
|
||||||
for nid, paths in duplicates.items():
|
|
||||||
findings["duplicate_node_ids"].append({"node_id": nid, "paths": sorted(set(paths))})
|
|
||||||
|
|
||||||
# Source pages by slug — used to validate `source:` refs on edges.
|
|
||||||
source_slugs = {p["slug"] for p in pages if p["meta"].get("type") == "source"}
|
|
||||||
|
|
||||||
# Aliases
|
|
||||||
alias_to_canonicals: dict[str, set[str]] = defaultdict(set)
|
|
||||||
canonical_by_id: dict[str, list[str]] = defaultdict(list)
|
|
||||||
for n in node_by_id.values():
|
|
||||||
if n["canonical"]:
|
|
||||||
canonical_by_id[n["id"]].append(n["rel_path"])
|
|
||||||
for alias in n["aliases"]:
|
|
||||||
alias_to_canonicals[str(alias)].add(n["id"])
|
|
||||||
|
|
||||||
for nid, paths in canonical_by_id.items():
|
|
||||||
if len(paths) > 1:
|
|
||||||
findings["duplicate_canonical"].append({"node_id": nid, "paths": paths})
|
|
||||||
|
|
||||||
for alias, owners in alias_to_canonicals.items():
|
|
||||||
if len(owners) > 1:
|
|
||||||
findings["alias_collisions"].append({"alias": alias, "owners": sorted(owners)})
|
|
||||||
|
|
||||||
# Walk relationships
|
|
||||||
for p in pages:
|
|
||||||
graph_meta = p["meta"].get("graph") if isinstance(p["meta"].get("graph"), dict) else {}
|
|
||||||
subject_id = derive_node_id(p["meta"], p["slug"], ontology)
|
|
||||||
subject_type = node_by_id.get(subject_id, {}).get("node_type")
|
|
||||||
|
|
||||||
for idx, rel in enumerate(graph_meta.get("relationships") or []):
|
|
||||||
if not isinstance(rel, dict):
|
|
||||||
continue
|
|
||||||
predicate = rel.get("predicate")
|
|
||||||
obj = rel.get("object")
|
|
||||||
here = {"page": p["rel_path"], "predicate": predicate,
|
|
||||||
"object": obj, "index": idx}
|
|
||||||
|
|
||||||
if not predicate or predicate not in predicates:
|
|
||||||
findings["unknown_predicates"].append({**here})
|
|
||||||
continue
|
|
||||||
pdef = predicates[predicate] or {}
|
|
||||||
|
|
||||||
# Object resolution. Allow string-literal objects only when
|
|
||||||
# ontology lists "*" in object_types (e.g. summarizes_raw).
|
|
||||||
object_types = pdef.get("object_types") or []
|
|
||||||
allows_wildcard_obj = "*" in object_types
|
|
||||||
if obj and obj not in node_by_id:
|
|
||||||
if not allows_wildcard_obj:
|
|
||||||
findings["broken_object_refs"].append({**here})
|
|
||||||
|
|
||||||
# Subject type check
|
|
||||||
if not types_match(pdef.get("subject_types"), subject_type):
|
|
||||||
findings["subject_type_mismatch"].append({
|
|
||||||
**here,
|
|
||||||
"subject": subject_id,
|
|
||||||
"subject_type": subject_type,
|
|
||||||
"allowed": pdef.get("subject_types"),
|
|
||||||
})
|
|
||||||
# Object type check (only if object resolves to a node)
|
|
||||||
obj_node = node_by_id.get(obj) if obj else None
|
|
||||||
obj_type = obj_node["node_type"] if obj_node else None
|
|
||||||
if obj_node and not types_match(pdef.get("object_types"), obj_type):
|
|
||||||
findings["object_type_mismatch"].append({
|
|
||||||
**here,
|
|
||||||
"object_type": obj_type,
|
|
||||||
"allowed": pdef.get("object_types"),
|
|
||||||
})
|
|
||||||
|
|
||||||
requires_evidence = pdef.get("requires_evidence", True)
|
|
||||||
if requires_evidence:
|
|
||||||
if not rel.get("evidence"):
|
|
||||||
findings["missing_evidence"].append({**here})
|
|
||||||
if not rel.get("source"):
|
|
||||||
findings["missing_source_field"].append({**here})
|
|
||||||
|
|
||||||
# source field must reference an existing source page slug
|
|
||||||
src = rel.get("source")
|
|
||||||
if src and str(src) not in source_slugs:
|
|
||||||
findings["broken_source_refs"].append({**here, "source": src})
|
|
||||||
|
|
||||||
confidence = rel.get("confidence")
|
|
||||||
if confidence and confidence not in ALLOWED_CONFIDENCE:
|
|
||||||
findings["invalid_confidence"].append({**here, "confidence": confidence})
|
|
||||||
|
|
||||||
status = rel.get("status")
|
|
||||||
if status and status not in ALLOWED_STATUS:
|
|
||||||
findings["invalid_status"].append({**here, "status": status})
|
|
||||||
|
|
||||||
# contradicts / supersedes resolution
|
|
||||||
for ref_field, bucket in (("contradicts", "broken_contradicts"),
|
|
||||||
("supersedes", "broken_supersedes")):
|
|
||||||
ref = rel.get(ref_field)
|
|
||||||
if ref:
|
|
||||||
ref_str = str(ref)
|
|
||||||
if ref_str not in node_by_id and ref_str not in source_slugs:
|
|
||||||
findings[bucket].append({**here, ref_field: ref_str})
|
|
||||||
|
|
||||||
# Orphan typed nodes — pages that declared `graph:` frontmatter but end
|
|
||||||
# up with no typed (non-implicit) edge touching them after extraction.
|
|
||||||
# Source nodes are exempt (they participate via implicit edges).
|
|
||||||
extracted_edges = _extract.build_edges(pages, {n["slug"]: n["id"] for n in node_by_id.values()})
|
|
||||||
typed_node_refs: set[str] = set()
|
|
||||||
for e in extracted_edges:
|
|
||||||
if e["predicate"] in IMPLICIT_PREDICATES:
|
|
||||||
continue
|
|
||||||
typed_node_refs.add(e["subject"])
|
|
||||||
if e["object"] in node_by_id:
|
|
||||||
typed_node_refs.add(e["object"])
|
|
||||||
|
|
||||||
for n in node_by_id.values():
|
|
||||||
if n["node_type"] == "source":
|
|
||||||
continue
|
|
||||||
graph_meta = n.get("graph") or {}
|
|
||||||
if not graph_meta:
|
|
||||||
continue # Pages without graph metadata are valid; they're text-only nodes.
|
|
||||||
if n["id"] in typed_node_refs:
|
|
||||||
continue
|
|
||||||
findings["orphan_typed_nodes"].append({
|
|
||||||
"node_id": n["id"],
|
|
||||||
"path": n["rel_path"],
|
|
||||||
})
|
|
||||||
|
|
||||||
# Summary
|
|
||||||
findings["summary"] = {
|
|
||||||
"pages_scanned": len(pages),
|
|
||||||
"nodes": len(node_by_id),
|
|
||||||
**{k: len(v) for k, v in findings.items() if isinstance(v, list)},
|
|
||||||
}
|
|
||||||
return findings
|
|
||||||
|
|
||||||
|
|
||||||
def render_text(findings: dict) -> str:
|
|
||||||
out = []
|
|
||||||
s = findings["summary"]
|
|
||||||
out.append("=" * 60)
|
|
||||||
out.append("Wiki Graph Lint Report")
|
|
||||||
out.append("=" * 60)
|
|
||||||
out.append(f"Pages scanned: {s['pages_scanned']} Nodes: {s['nodes']}")
|
|
||||||
out.append("")
|
|
||||||
|
|
||||||
sections = [
|
|
||||||
("duplicate_node_ids", "Duplicate node ids",
|
|
||||||
lambda f: f" - {f['node_id']}: {', '.join(f['paths'])}"),
|
|
||||||
("unknown_predicates", "Unknown predicates (not in ontology)",
|
|
||||||
lambda f: f" - {f['page']}#rel[{f['index']}] predicate={f['predicate']!r}"),
|
|
||||||
("broken_object_refs", "Broken object references",
|
|
||||||
lambda f: f" - {f['page']}#rel[{f['index']}] {f['predicate']} → {f['object']!r}"),
|
|
||||||
("subject_type_mismatch", "Subject type does not match ontology",
|
|
||||||
lambda f: f" - {f['page']}#rel[{f['index']}] {f['predicate']}: subject={f['subject_type']} (allowed: {f['allowed']})"),
|
|
||||||
("object_type_mismatch", "Object type does not match ontology",
|
|
||||||
lambda f: f" - {f['page']}#rel[{f['index']}] {f['predicate']}: object={f['object_type']} (allowed: {f['allowed']})"),
|
|
||||||
("missing_evidence", "Missing evidence on typed edge",
|
|
||||||
lambda f: f" - {f['page']}#rel[{f['index']}] {f['predicate']} → {f['object']}"),
|
|
||||||
("missing_source_field", "Missing source on typed edge",
|
|
||||||
lambda f: f" - {f['page']}#rel[{f['index']}] {f['predicate']} → {f['object']}"),
|
|
||||||
("broken_source_refs", "source: does not match any source page",
|
|
||||||
lambda f: f" - {f['page']}#rel[{f['index']}] source={f['source']!r}"),
|
|
||||||
("invalid_confidence", "Invalid confidence value",
|
|
||||||
lambda f: f" - {f['page']}#rel[{f['index']}] confidence={f['confidence']!r}"),
|
|
||||||
("invalid_status", "Invalid status value",
|
|
||||||
lambda f: f" - {f['page']}#rel[{f['index']}] status={f['status']!r}"),
|
|
||||||
("duplicate_canonical", "Duplicate canonical nodes",
|
|
||||||
lambda f: f" - {f['node_id']}: {', '.join(f['paths'])}"),
|
|
||||||
("alias_collisions", "Alias used by multiple canonical nodes",
|
|
||||||
lambda f: f" - {f['alias']!r}: {', '.join(f['owners'])}"),
|
|
||||||
("broken_contradicts", "Broken contradicts reference",
|
|
||||||
lambda f: f" - {f['page']}#rel[{f['index']}] contradicts={f.get('contradicts')}"),
|
|
||||||
("broken_supersedes", "Broken supersedes reference",
|
|
||||||
lambda f: f" - {f['page']}#rel[{f['index']}] supersedes={f.get('supersedes')}"),
|
|
||||||
("orphan_typed_nodes", "Orphan typed nodes (no inbound or outbound typed edges)",
|
|
||||||
lambda f: f" - {f['node_id']} ({f['path']})"),
|
|
||||||
]
|
|
||||||
|
|
||||||
healthy = True
|
|
||||||
for key, label, formatter in sections:
|
|
||||||
items = findings[key]
|
|
||||||
if not items:
|
|
||||||
continue
|
|
||||||
healthy = False
|
|
||||||
out.append(f"{label} ({len(items)}):")
|
|
||||||
for item in items[:50]:
|
|
||||||
out.append(formatter(item))
|
|
||||||
if len(items) > 50:
|
|
||||||
out.append(f" ... and {len(items) - 50} more")
|
|
||||||
out.append("")
|
|
||||||
|
|
||||||
if healthy:
|
|
||||||
out.append("No graph issues found.")
|
|
||||||
return "\n".join(out)
|
|
||||||
|
|
||||||
|
|
||||||
def main():
|
|
||||||
parser = argparse.ArgumentParser(description=__doc__, formatter_class=argparse.RawDescriptionHelpFormatter)
|
|
||||||
parser.add_argument("wiki", nargs="?", type=Path, default=Path("cml/wiki"))
|
|
||||||
parser.add_argument("--ontology", type=Path, help="Ontology file (default: <wiki>/graph/ontology.yaml)")
|
|
||||||
parser.add_argument("--json", action="store_true")
|
|
||||||
args = parser.parse_args()
|
|
||||||
|
|
||||||
if not args.wiki.exists():
|
|
||||||
print(f"Wiki directory not found: {args.wiki}", file=sys.stderr)
|
|
||||||
sys.exit(1)
|
|
||||||
|
|
||||||
ontology_path = args.ontology or (args.wiki / "graph" / "ontology.yaml")
|
|
||||||
if not ontology_path.exists():
|
|
||||||
print(f"Ontology not found: {ontology_path}", file=sys.stderr)
|
|
||||||
print("Did you forget to seed wiki/graph/ontology.yaml? See assets/ontology.yaml.template.",
|
|
||||||
file=sys.stderr)
|
|
||||||
sys.exit(1)
|
|
||||||
try:
|
|
||||||
ontology = yaml.safe_load(ontology_path.read_text(encoding="utf-8")) or {}
|
|
||||||
except yaml.YAMLError as e:
|
|
||||||
print(f"Ontology parse error: {e}", file=sys.stderr)
|
|
||||||
sys.exit(2)
|
|
||||||
|
|
||||||
pages = collect_pages(args.wiki)
|
|
||||||
findings = lint(pages, ontology)
|
|
||||||
|
|
||||||
if args.json:
|
|
||||||
print(json.dumps(findings, indent=2, default=str))
|
|
||||||
else:
|
|
||||||
print(render_text(findings))
|
|
||||||
|
|
||||||
|
|
||||||
if __name__ == "__main__":
|
|
||||||
main()
|
|
||||||
@@ -1,267 +0,0 @@
|
|||||||
#!/usr/bin/env -S uv run --script
|
|
||||||
# /// script
|
|
||||||
# requires-python = ">=3.11"
|
|
||||||
# dependencies = []
|
|
||||||
# ///
|
|
||||||
"""
|
|
||||||
wiki_graph_query.py — Query the compiled wiki graph (graph.sqlite).
|
|
||||||
|
|
||||||
Use this to accelerate navigation: find what's connected to a node, list
|
|
||||||
typed edges around a subject, find a path between two nodes, or dump every
|
|
||||||
fact about a node. The graph is a navigation index — for high-stakes
|
|
||||||
claims, follow the `source` field back to the wiki page and the raw file.
|
|
||||||
|
|
||||||
Subcommands:
|
|
||||||
neighbors --node <id> List nodes one hop away from <id>
|
|
||||||
edges --subject <id> List all outbound edges from <id>
|
|
||||||
[--predicate <p>] Filter by predicate
|
|
||||||
path --from <id> --to <id> Shortest directed path (BFS, max depth 6)
|
|
||||||
[--max-depth N]
|
|
||||||
facts --about <id> Outbound + inbound edges for <id>
|
|
||||||
|
|
||||||
Common options:
|
|
||||||
--db <path> Path to graph.sqlite (default: <wiki>/graph/graph.sqlite)
|
|
||||||
--json Emit JSON instead of text
|
|
||||||
|
|
||||||
Examples:
|
|
||||||
python wiki_graph_query.py wiki/ neighbors --node product:konvy
|
|
||||||
python wiki_graph_query.py wiki/ edges --subject person:stephanie-emmanouel
|
|
||||||
python wiki_graph_query.py wiki/ path --from person:praney-behl --to product:konvy
|
|
||||||
python wiki_graph_query.py wiki/ facts --about product:konvy
|
|
||||||
"""
|
|
||||||
|
|
||||||
import argparse
|
|
||||||
import json
|
|
||||||
import sqlite3
|
|
||||||
import sys
|
|
||||||
from collections import deque
|
|
||||||
from pathlib import Path
|
|
||||||
|
|
||||||
|
|
||||||
EVIDENCE_SNIPPET_LEN = 140
|
|
||||||
|
|
||||||
|
|
||||||
def open_db(path: Path) -> sqlite3.Connection:
|
|
||||||
if not path.exists():
|
|
||||||
print(f"graph.sqlite not found at {path}.", file=sys.stderr)
|
|
||||||
print("Run wiki_graph_extract.py first.", file=sys.stderr)
|
|
||||||
sys.exit(1)
|
|
||||||
conn = sqlite3.connect(path)
|
|
||||||
conn.row_factory = sqlite3.Row
|
|
||||||
return conn
|
|
||||||
|
|
||||||
|
|
||||||
def fetch_node(conn: sqlite3.Connection, node_id: str) -> dict | None:
|
|
||||||
row = conn.execute("SELECT * FROM nodes WHERE id = ?", (node_id,)).fetchone()
|
|
||||||
return dict(row) if row else None
|
|
||||||
|
|
||||||
|
|
||||||
def edges_from(conn: sqlite3.Connection, subject: str, predicate: str | None = None) -> list[dict]:
|
|
||||||
q = "SELECT * FROM edges WHERE subject = ?"
|
|
||||||
params: list = [subject]
|
|
||||||
if predicate:
|
|
||||||
q += " AND predicate = ?"
|
|
||||||
params.append(predicate)
|
|
||||||
q += " ORDER BY predicate, object"
|
|
||||||
return [dict(r) for r in conn.execute(q, params).fetchall()]
|
|
||||||
|
|
||||||
|
|
||||||
def edges_to(conn: sqlite3.Connection, obj: str, predicate: str | None = None) -> list[dict]:
|
|
||||||
q = "SELECT * FROM edges WHERE object = ?"
|
|
||||||
params: list = [obj]
|
|
||||||
if predicate:
|
|
||||||
q += " AND predicate = ?"
|
|
||||||
params.append(predicate)
|
|
||||||
q += " ORDER BY predicate, subject"
|
|
||||||
return [dict(r) for r in conn.execute(q, params).fetchall()]
|
|
||||||
|
|
||||||
|
|
||||||
def truncate(text: str | None) -> str:
|
|
||||||
if not text:
|
|
||||||
return ""
|
|
||||||
if len(text) <= EVIDENCE_SNIPPET_LEN:
|
|
||||||
return text
|
|
||||||
return text[: EVIDENCE_SNIPPET_LEN - 1].rstrip() + "…"
|
|
||||||
|
|
||||||
|
|
||||||
def render_edge_row(e: dict) -> str:
|
|
||||||
pieces = [
|
|
||||||
f" {e['subject']} --[{e['predicate']}]--> {e['object']}",
|
|
||||||
]
|
|
||||||
confidence = e.get("confidence") or "-"
|
|
||||||
status = e.get("status") or "-"
|
|
||||||
src = e.get("source") or "-"
|
|
||||||
pieces.append(f" via {src} conf={confidence} status={status}")
|
|
||||||
if e.get("evidence"):
|
|
||||||
pieces.append(f" evidence: {truncate(e['evidence'])}")
|
|
||||||
pieces.append(f" (page: {e['page']})")
|
|
||||||
return "\n".join(pieces)
|
|
||||||
|
|
||||||
|
|
||||||
def cmd_neighbors(conn: sqlite3.Connection, args) -> dict:
|
|
||||||
node = fetch_node(conn, args.node)
|
|
||||||
if not node:
|
|
||||||
print(f"node not found: {args.node}", file=sys.stderr)
|
|
||||||
sys.exit(1)
|
|
||||||
out_edges = edges_from(conn, args.node)
|
|
||||||
in_edges = edges_to(conn, args.node)
|
|
||||||
|
|
||||||
neighbors: dict[str, dict] = {}
|
|
||||||
for e in out_edges:
|
|
||||||
neighbors.setdefault(e["object"], {"node_id": e["object"], "out": [], "in": []})
|
|
||||||
neighbors[e["object"]]["out"].append(e)
|
|
||||||
for e in in_edges:
|
|
||||||
neighbors.setdefault(e["subject"], {"node_id": e["subject"], "out": [], "in": []})
|
|
||||||
neighbors[e["subject"]]["in"].append(e)
|
|
||||||
|
|
||||||
# Resolve neighbor titles where possible
|
|
||||||
for nid, slot in neighbors.items():
|
|
||||||
target = fetch_node(conn, nid)
|
|
||||||
slot["title"] = target["title"] if target else nid
|
|
||||||
slot["path"] = target["path"] if target else None
|
|
||||||
|
|
||||||
return {
|
|
||||||
"node": node,
|
|
||||||
"neighbors": sorted(neighbors.values(), key=lambda n: n["node_id"]),
|
|
||||||
}
|
|
||||||
|
|
||||||
|
|
||||||
def cmd_edges(conn: sqlite3.Connection, args) -> dict:
|
|
||||||
es = edges_from(conn, args.subject, args.predicate)
|
|
||||||
return {"subject": args.subject, "predicate": args.predicate, "edges": es}
|
|
||||||
|
|
||||||
|
|
||||||
def cmd_facts(conn: sqlite3.Connection, args) -> dict:
|
|
||||||
node = fetch_node(conn, args.about)
|
|
||||||
if not node:
|
|
||||||
print(f"node not found: {args.about}", file=sys.stderr)
|
|
||||||
sys.exit(1)
|
|
||||||
return {
|
|
||||||
"node": node,
|
|
||||||
"outbound": edges_from(conn, args.about),
|
|
||||||
"inbound": edges_to(conn, args.about),
|
|
||||||
}
|
|
||||||
|
|
||||||
|
|
||||||
def cmd_path(conn: sqlite3.Connection, args) -> dict:
|
|
||||||
src = fetch_node(conn, getattr(args, "from"))
|
|
||||||
dst = fetch_node(conn, args.to)
|
|
||||||
if not src:
|
|
||||||
print(f"from-node not found: {getattr(args, 'from')}", file=sys.stderr)
|
|
||||||
sys.exit(1)
|
|
||||||
if not dst:
|
|
||||||
print(f"to-node not found: {args.to}", file=sys.stderr)
|
|
||||||
sys.exit(1)
|
|
||||||
|
|
||||||
start, goal = getattr(args, "from"), args.to
|
|
||||||
queue = deque([(start, [start], [])])
|
|
||||||
visited = {start}
|
|
||||||
while queue:
|
|
||||||
node, node_path, edge_path = queue.popleft()
|
|
||||||
if node == goal:
|
|
||||||
return {"from": start, "to": goal, "path_nodes": node_path, "path_edges": edge_path}
|
|
||||||
if len(node_path) - 1 >= args.max_depth:
|
|
||||||
continue
|
|
||||||
for e in edges_from(conn, node):
|
|
||||||
nxt = e["object"]
|
|
||||||
if nxt in visited:
|
|
||||||
continue
|
|
||||||
visited.add(nxt)
|
|
||||||
queue.append((nxt, node_path + [nxt], edge_path + [e]))
|
|
||||||
return {"from": start, "to": goal, "path_nodes": [], "path_edges": []}
|
|
||||||
|
|
||||||
|
|
||||||
def render(result: dict, command: str) -> str:
|
|
||||||
out: list[str] = []
|
|
||||||
if command == "neighbors":
|
|
||||||
n = result["node"]
|
|
||||||
out.append(f"Node: {n['id']} ({n['title']}) {n['node_type']} [{n['path']}]")
|
|
||||||
out.append(f"Neighbors: {len(result['neighbors'])}")
|
|
||||||
for nb in result["neighbors"]:
|
|
||||||
out.append("")
|
|
||||||
out.append(f" → {nb['node_id']} ({nb['title']})")
|
|
||||||
for e in nb.get("out", []):
|
|
||||||
out.append(f" out [{e['predicate']}] conf={e.get('confidence') or '-'} src={e.get('source') or '-'}")
|
|
||||||
for e in nb.get("in", []):
|
|
||||||
out.append(f" in [{e['predicate']}] from {e['subject']} src={e.get('source') or '-'}")
|
|
||||||
elif command == "edges":
|
|
||||||
out.append(f"Edges from {result['subject']}"
|
|
||||||
+ (f" with predicate {result['predicate']}" if result['predicate'] else ""))
|
|
||||||
for e in result["edges"]:
|
|
||||||
out.append("")
|
|
||||||
out.append(render_edge_row(e))
|
|
||||||
elif command == "facts":
|
|
||||||
n = result["node"]
|
|
||||||
out.append(f"Facts about {n['id']} ({n['title']}) [{n['path']}]")
|
|
||||||
out.append("")
|
|
||||||
out.append(f"Outbound ({len(result['outbound'])}):")
|
|
||||||
for e in result["outbound"]:
|
|
||||||
out.append(render_edge_row(e))
|
|
||||||
out.append("")
|
|
||||||
out.append(f"Inbound ({len(result['inbound'])}):")
|
|
||||||
for e in result["inbound"]:
|
|
||||||
out.append(render_edge_row(e))
|
|
||||||
elif command == "path":
|
|
||||||
if not result["path_nodes"]:
|
|
||||||
out.append(f"No path found from {result['from']} to {result['to']} within depth limit.")
|
|
||||||
else:
|
|
||||||
out.append(f"Path from {result['from']} to {result['to']} ({len(result['path_edges'])} hops):")
|
|
||||||
for e in result["path_edges"]:
|
|
||||||
out.append("")
|
|
||||||
out.append(render_edge_row(e))
|
|
||||||
return "\n".join(out)
|
|
||||||
|
|
||||||
|
|
||||||
def main():
|
|
||||||
parser = argparse.ArgumentParser(description=__doc__, formatter_class=argparse.RawDescriptionHelpFormatter)
|
|
||||||
parser.add_argument("wiki", type=Path, help="Wiki directory.")
|
|
||||||
parser.add_argument("--db", type=Path, help="Path to graph.sqlite (default: <wiki>/graph/graph.sqlite)")
|
|
||||||
parser.add_argument("--json", action="store_true")
|
|
||||||
|
|
||||||
sub = parser.add_subparsers(dest="command", required=True)
|
|
||||||
|
|
||||||
p_n = sub.add_parser("neighbors")
|
|
||||||
p_n.add_argument("--node", required=True)
|
|
||||||
|
|
||||||
p_e = sub.add_parser("edges")
|
|
||||||
p_e.add_argument("--subject", required=True)
|
|
||||||
p_e.add_argument("--predicate")
|
|
||||||
|
|
||||||
p_p = sub.add_parser("path")
|
|
||||||
p_p.add_argument("--from", dest="from", required=True)
|
|
||||||
p_p.add_argument("--to", required=True)
|
|
||||||
p_p.add_argument("--max-depth", type=int, default=6)
|
|
||||||
|
|
||||||
p_f = sub.add_parser("facts")
|
|
||||||
p_f.add_argument("--about", required=True)
|
|
||||||
|
|
||||||
args = parser.parse_args()
|
|
||||||
|
|
||||||
if not args.wiki.exists():
|
|
||||||
print(f"Wiki directory not found: {args.wiki}", file=sys.stderr)
|
|
||||||
sys.exit(1)
|
|
||||||
db_path = args.db or (args.wiki / "graph" / "graph.sqlite")
|
|
||||||
conn = open_db(db_path)
|
|
||||||
try:
|
|
||||||
if args.command == "neighbors":
|
|
||||||
result = cmd_neighbors(conn, args)
|
|
||||||
elif args.command == "edges":
|
|
||||||
result = cmd_edges(conn, args)
|
|
||||||
elif args.command == "path":
|
|
||||||
result = cmd_path(conn, args)
|
|
||||||
elif args.command == "facts":
|
|
||||||
result = cmd_facts(conn, args)
|
|
||||||
else:
|
|
||||||
parser.print_help()
|
|
||||||
sys.exit(1)
|
|
||||||
finally:
|
|
||||||
conn.close()
|
|
||||||
|
|
||||||
if args.json:
|
|
||||||
print(json.dumps(result, indent=2, default=str))
|
|
||||||
else:
|
|
||||||
print(render(result, args.command))
|
|
||||||
|
|
||||||
|
|
||||||
if __name__ == "__main__":
|
|
||||||
main()
|
|
||||||
@@ -1,319 +0,0 @@
|
|||||||
#!/usr/bin/env -S uv run --script
|
|
||||||
# /// script
|
|
||||||
# requires-python = ">=3.11"
|
|
||||||
# dependencies = []
|
|
||||||
# ///
|
|
||||||
"""
|
|
||||||
wiki_lint.py — Structural health check for an LLM Wiki.
|
|
||||||
|
|
||||||
Reports orphan pages, broken wikilinks, oversized pages, frontmatter issues,
|
|
||||||
stale pages, duplicate slugs, and (with --suggest-pages) terms that appear in
|
|
||||||
many pages without their own page.
|
|
||||||
|
|
||||||
Conservative by design: reports findings, never edits.
|
|
||||||
|
|
||||||
Usage:
|
|
||||||
python wiki_lint.py [<wiki-dir>] [options]
|
|
||||||
|
|
||||||
Options:
|
|
||||||
--soft-cap N Page-size soft cap in lines (default: 400)
|
|
||||||
--hard-cap N Page-size hard cap in lines (default: 800)
|
|
||||||
--required-fm a,b Required frontmatter fields (default: type,title,tags,created,updated)
|
|
||||||
--suggest-pages Surface terms appearing in many pages without a page
|
|
||||||
--suggest-min N Minimum occurrences for --suggest-pages (default: 5)
|
|
||||||
--json Emit JSON instead of text
|
|
||||||
|
|
||||||
Examples:
|
|
||||||
python wiki_lint.py wiki/
|
|
||||||
python wiki_lint.py wiki/ --suggest-pages
|
|
||||||
python wiki_lint.py wiki/ --json > lint.json
|
|
||||||
"""
|
|
||||||
|
|
||||||
import argparse
|
|
||||||
import json
|
|
||||||
import re
|
|
||||||
import sys
|
|
||||||
from collections import Counter, defaultdict
|
|
||||||
from datetime import date, datetime
|
|
||||||
from pathlib import Path
|
|
||||||
|
|
||||||
|
|
||||||
WIKILINK_RE = re.compile(r"\[\[([^\]|]+)(?:\|[^\]]+)?\]\]")
|
|
||||||
FRONTMATTER_RE = re.compile(r"^---\s*\n(.*?)\n---\s*\n", re.DOTALL)
|
|
||||||
CAPITALIZED_PHRASE_RE = re.compile(r"\b([A-Z][a-zA-Z0-9]+(?:\s+[A-Z][a-zA-Z0-9]+){0,3})\b")
|
|
||||||
|
|
||||||
|
|
||||||
SKIP_TOP_LEVEL_FILES = {"SCHEMA.md", "index.md", "log.md", "README.md"}
|
|
||||||
SKIP_TOP_LEVEL_DIRS = {"indexes", "graph"}
|
|
||||||
|
|
||||||
|
|
||||||
def parse_frontmatter(text: str) -> tuple[dict, str, bool]:
|
|
||||||
"""Returns (metadata, body, malformed). malformed=True if frontmatter was attempted but unparseable."""
|
|
||||||
if not text.startswith("---"):
|
|
||||||
return {}, text, False
|
|
||||||
m = FRONTMATTER_RE.match(text)
|
|
||||||
if not m:
|
|
||||||
return {}, text, True
|
|
||||||
fm_text = m.group(1)
|
|
||||||
body = text[m.end():]
|
|
||||||
meta = {}
|
|
||||||
current_key = None
|
|
||||||
for line in fm_text.split("\n"):
|
|
||||||
if not line.strip():
|
|
||||||
continue
|
|
||||||
kv = re.match(r"^([a-zA-Z_]+):\s*(.*)$", line)
|
|
||||||
if kv:
|
|
||||||
key, value = kv.group(1), kv.group(2).strip()
|
|
||||||
if value.startswith("[") and value.endswith("]"):
|
|
||||||
items = [x.strip().strip('"').strip("'") for x in value[1:-1].split(",") if x.strip()]
|
|
||||||
meta[key] = items
|
|
||||||
elif value:
|
|
||||||
meta[key] = value.strip('"').strip("'")
|
|
||||||
else:
|
|
||||||
meta[key] = []
|
|
||||||
current_key = key
|
|
||||||
elif line.startswith(" - ") and current_key:
|
|
||||||
meta[current_key].append(line[4:].strip().strip('"').strip("'"))
|
|
||||||
return meta, body, False
|
|
||||||
|
|
||||||
|
|
||||||
def collect_pages(wiki_root: Path) -> list[dict]:
|
|
||||||
pages = []
|
|
||||||
for md_path in wiki_root.rglob("*.md"):
|
|
||||||
rel = md_path.relative_to(wiki_root)
|
|
||||||
if rel.parts[0] in SKIP_TOP_LEVEL_FILES or rel.parts[0] in SKIP_TOP_LEVEL_DIRS:
|
|
||||||
continue
|
|
||||||
if rel.name.startswith("."):
|
|
||||||
continue
|
|
||||||
try:
|
|
||||||
text = md_path.read_text(encoding="utf-8")
|
|
||||||
except (UnicodeDecodeError, OSError) as e:
|
|
||||||
pages.append({
|
|
||||||
"path": str(md_path),
|
|
||||||
"rel_path": str(rel),
|
|
||||||
"slug": md_path.stem,
|
|
||||||
"read_error": str(e),
|
|
||||||
})
|
|
||||||
continue
|
|
||||||
meta, body, malformed = parse_frontmatter(text)
|
|
||||||
line_count = text.count("\n") + 1
|
|
||||||
links = [m.group(1).strip() for m in WIKILINK_RE.finditer(body)]
|
|
||||||
pages.append({
|
|
||||||
"path": str(md_path),
|
|
||||||
"rel_path": str(rel),
|
|
||||||
"slug": md_path.stem,
|
|
||||||
"meta": meta,
|
|
||||||
"body": body,
|
|
||||||
"line_count": line_count,
|
|
||||||
"links": links,
|
|
||||||
"malformed_fm": malformed,
|
|
||||||
})
|
|
||||||
return pages
|
|
||||||
|
|
||||||
|
|
||||||
def parse_date(s):
|
|
||||||
if not s or not isinstance(s, str):
|
|
||||||
return None
|
|
||||||
try:
|
|
||||||
return datetime.strptime(s[:10], "%Y-%m-%d").date()
|
|
||||||
except (ValueError, TypeError):
|
|
||||||
return None
|
|
||||||
|
|
||||||
|
|
||||||
def lint(pages: list[dict], soft_cap: int, hard_cap: int, required_fm: list[str], suggest_pages: bool, suggest_min: int) -> dict:
|
|
||||||
findings = {
|
|
||||||
"orphans": [],
|
|
||||||
"broken_links": [],
|
|
||||||
"oversized_hard": [],
|
|
||||||
"oversized_soft": [],
|
|
||||||
"missing_frontmatter": [],
|
|
||||||
"malformed_frontmatter": [],
|
|
||||||
"duplicate_slugs": [],
|
|
||||||
"stale_pages": [],
|
|
||||||
"read_errors": [],
|
|
||||||
"suggested_pages": [],
|
|
||||||
"summary": {},
|
|
||||||
}
|
|
||||||
|
|
||||||
# Read errors
|
|
||||||
for p in pages:
|
|
||||||
if "read_error" in p:
|
|
||||||
findings["read_errors"].append({"path": p["rel_path"], "error": p["read_error"]})
|
|
||||||
|
|
||||||
pages = [p for p in pages if "read_error" not in p]
|
|
||||||
|
|
||||||
# Slugs
|
|
||||||
slug_to_pages = defaultdict(list)
|
|
||||||
for p in pages:
|
|
||||||
slug_to_pages[p["slug"]].append(p["rel_path"])
|
|
||||||
for slug, paths in slug_to_pages.items():
|
|
||||||
if len(paths) > 1:
|
|
||||||
findings["duplicate_slugs"].append({"slug": slug, "paths": paths})
|
|
||||||
|
|
||||||
# Inbound link map
|
|
||||||
inbound = defaultdict(set)
|
|
||||||
all_slugs = set(slug_to_pages.keys())
|
|
||||||
for p in pages:
|
|
||||||
for link in p["links"]:
|
|
||||||
inbound[link].add(p["slug"])
|
|
||||||
|
|
||||||
# Orphans, broken links, oversize, frontmatter, staleness
|
|
||||||
for p in pages:
|
|
||||||
# Orphans
|
|
||||||
if not inbound.get(p["slug"]):
|
|
||||||
findings["orphans"].append({"slug": p["slug"], "path": p["rel_path"]})
|
|
||||||
|
|
||||||
# Broken links
|
|
||||||
for link in p["links"]:
|
|
||||||
if link not in all_slugs:
|
|
||||||
findings["broken_links"].append({
|
|
||||||
"from": p["slug"],
|
|
||||||
"from_path": p["rel_path"],
|
|
||||||
"to": link,
|
|
||||||
})
|
|
||||||
|
|
||||||
# Oversize
|
|
||||||
if p["line_count"] > hard_cap:
|
|
||||||
findings["oversized_hard"].append({"path": p["rel_path"], "lines": p["line_count"]})
|
|
||||||
elif p["line_count"] > soft_cap:
|
|
||||||
findings["oversized_soft"].append({"path": p["rel_path"], "lines": p["line_count"]})
|
|
||||||
|
|
||||||
# Frontmatter
|
|
||||||
if p["malformed_fm"]:
|
|
||||||
findings["malformed_frontmatter"].append({"path": p["rel_path"]})
|
|
||||||
else:
|
|
||||||
missing = [field for field in required_fm if field not in p["meta"] or p["meta"].get(field) in ("", None, [])]
|
|
||||||
if missing:
|
|
||||||
findings["missing_frontmatter"].append({"path": p["rel_path"], "missing": missing})
|
|
||||||
|
|
||||||
# Staleness: heuristic — page hasn't been updated in 90 days AND has been touched by recent ingests.
|
|
||||||
# Approximate: if updated > 90d ago and the page is well-linked (a hub), flag it.
|
|
||||||
updated = parse_date(p["meta"].get("updated"))
|
|
||||||
if updated:
|
|
||||||
age_days = (date.today() - updated).days
|
|
||||||
if age_days > 90 and len(inbound.get(p["slug"], [])) >= 3:
|
|
||||||
findings["stale_pages"].append({
|
|
||||||
"path": p["rel_path"],
|
|
||||||
"updated": p["meta"].get("updated"),
|
|
||||||
"age_days": age_days,
|
|
||||||
"inbound_count": len(inbound.get(p["slug"], [])),
|
|
||||||
})
|
|
||||||
|
|
||||||
# Suggested pages: capitalized multi-word phrases appearing in many pages without a page
|
|
||||||
if suggest_pages:
|
|
||||||
phrase_pages = defaultdict(set)
|
|
||||||
for p in pages:
|
|
||||||
seen = set()
|
|
||||||
for m in CAPITALIZED_PHRASE_RE.finditer(p["body"]):
|
|
||||||
phrase = m.group(1).strip()
|
|
||||||
seen.add(phrase)
|
|
||||||
for phrase in seen:
|
|
||||||
phrase_pages[phrase].add(p["slug"])
|
|
||||||
|
|
||||||
# Title set for filtering
|
|
||||||
existing_titles = {p["meta"].get("title", "").lower() for p in pages}
|
|
||||||
existing_slugs_normalized = {s.lower().replace("-", " ") for s in all_slugs}
|
|
||||||
|
|
||||||
candidates = []
|
|
||||||
for phrase, page_set in phrase_pages.items():
|
|
||||||
if len(page_set) < suggest_min:
|
|
||||||
continue
|
|
||||||
if phrase.lower() in existing_titles:
|
|
||||||
continue
|
|
||||||
if phrase.lower() in existing_slugs_normalized:
|
|
||||||
continue
|
|
||||||
# Filter out section header garbage
|
|
||||||
if phrase.split()[0] in {"Section", "Where", "Sources", "Tags", "Type", "Title"}:
|
|
||||||
continue
|
|
||||||
candidates.append({"phrase": phrase, "page_count": len(page_set), "pages": sorted(page_set)[:5]})
|
|
||||||
candidates.sort(key=lambda x: -x["page_count"])
|
|
||||||
findings["suggested_pages"] = candidates[:30]
|
|
||||||
|
|
||||||
findings["summary"] = {
|
|
||||||
"total_pages": len(pages),
|
|
||||||
"orphans": len(findings["orphans"]),
|
|
||||||
"broken_links": len(findings["broken_links"]),
|
|
||||||
"oversized_hard": len(findings["oversized_hard"]),
|
|
||||||
"oversized_soft": len(findings["oversized_soft"]),
|
|
||||||
"missing_frontmatter": len(findings["missing_frontmatter"]),
|
|
||||||
"malformed_frontmatter": len(findings["malformed_frontmatter"]),
|
|
||||||
"duplicate_slugs": len(findings["duplicate_slugs"]),
|
|
||||||
"stale_pages": len(findings["stale_pages"]),
|
|
||||||
"read_errors": len(findings["read_errors"]),
|
|
||||||
"suggested_pages": len(findings["suggested_pages"]),
|
|
||||||
}
|
|
||||||
return findings
|
|
||||||
|
|
||||||
|
|
||||||
def render_text(findings: dict) -> str:
|
|
||||||
out = []
|
|
||||||
s = findings["summary"]
|
|
||||||
out.append("=" * 60)
|
|
||||||
out.append("Wiki Lint Report")
|
|
||||||
out.append("=" * 60)
|
|
||||||
out.append(f"Total pages scanned: {s['total_pages']}")
|
|
||||||
out.append("")
|
|
||||||
|
|
||||||
sections = [
|
|
||||||
("orphans", "Orphan pages (no inbound links)", lambda f: f" - {f['slug']} ({f['path']})"),
|
|
||||||
("broken_links", "Broken wikilinks", lambda f: f" - [[{f['to']}]] referenced from {f['from_path']}"),
|
|
||||||
("oversized_hard", "OVERSIZE (over hard cap — must split)", lambda f: f" - {f['path']} ({f['lines']} lines)"),
|
|
||||||
("oversized_soft", "Oversize (over soft cap — consider splitting)", lambda f: f" - {f['path']} ({f['lines']} lines)"),
|
|
||||||
("missing_frontmatter", "Missing frontmatter fields", lambda f: f" - {f['path']} missing: {', '.join(f['missing'])}"),
|
|
||||||
("malformed_frontmatter", "Malformed frontmatter", lambda f: f" - {f['path']}"),
|
|
||||||
("duplicate_slugs", "Duplicate slugs", lambda f: f" - {f['slug']}: {', '.join(f['paths'])}"),
|
|
||||||
("stale_pages", "Stale pages (well-linked but not updated in 90+ days)", lambda f: f" - {f['path']} (updated {f['updated']}, {f['age_days']}d ago, {f['inbound_count']} inbound)"),
|
|
||||||
("read_errors", "Read errors", lambda f: f" - {f['path']}: {f['error']}"),
|
|
||||||
]
|
|
||||||
|
|
||||||
for key, label, formatter in sections:
|
|
||||||
items = findings[key]
|
|
||||||
if not items:
|
|
||||||
continue
|
|
||||||
out.append(f"{label} ({len(items)}):")
|
|
||||||
for item in items[:50]:
|
|
||||||
out.append(formatter(item))
|
|
||||||
if len(items) > 50:
|
|
||||||
out.append(f" ... and {len(items) - 50} more")
|
|
||||||
out.append("")
|
|
||||||
|
|
||||||
if findings["suggested_pages"]:
|
|
||||||
out.append(f"Suggested page candidates ({len(findings['suggested_pages'])}):")
|
|
||||||
out.append(" Phrases appearing in many pages without a dedicated page:")
|
|
||||||
for item in findings["suggested_pages"]:
|
|
||||||
out.append(f" - \"{item['phrase']}\" ({item['page_count']} pages)")
|
|
||||||
out.append("")
|
|
||||||
|
|
||||||
if all(v == 0 for k, v in s.items() if k != "total_pages"):
|
|
||||||
out.append("No issues found. Wiki is healthy.")
|
|
||||||
|
|
||||||
return "\n".join(out)
|
|
||||||
|
|
||||||
|
|
||||||
def main():
|
|
||||||
parser = argparse.ArgumentParser(description=__doc__, formatter_class=argparse.RawDescriptionHelpFormatter)
|
|
||||||
parser.add_argument("wiki", nargs="?", type=Path, default=Path("cml/wiki"), help="Wiki directory (default: cml/wiki).")
|
|
||||||
parser.add_argument("--soft-cap", type=int, default=400, help="Page-size soft cap (lines).")
|
|
||||||
parser.add_argument("--hard-cap", type=int, default=800, help="Page-size hard cap (lines).")
|
|
||||||
parser.add_argument("--required-fm", default="type,title,tags,created,updated", help="Required frontmatter fields, comma-separated.")
|
|
||||||
parser.add_argument("--suggest-pages", action="store_true", help="Surface page candidates.")
|
|
||||||
parser.add_argument("--suggest-min", type=int, default=5, help="Minimum page count for suggestions.")
|
|
||||||
parser.add_argument("--json", action="store_true", help="Emit JSON.")
|
|
||||||
args = parser.parse_args()
|
|
||||||
|
|
||||||
if not args.wiki.exists():
|
|
||||||
print(f"Wiki directory not found: {args.wiki}", file=sys.stderr)
|
|
||||||
sys.exit(1)
|
|
||||||
|
|
||||||
pages = collect_pages(args.wiki)
|
|
||||||
required_fm = [f.strip() for f in args.required_fm.split(",") if f.strip()]
|
|
||||||
findings = lint(pages, args.soft_cap, args.hard_cap, required_fm, args.suggest_pages, args.suggest_min)
|
|
||||||
|
|
||||||
if args.json:
|
|
||||||
print(json.dumps(findings, indent=2, default=str))
|
|
||||||
else:
|
|
||||||
print(render_text(findings))
|
|
||||||
|
|
||||||
|
|
||||||
if __name__ == "__main__":
|
|
||||||
main()
|
|
||||||
@@ -1,270 +0,0 @@
|
|||||||
#!/usr/bin/env -S uv run --script
|
|
||||||
# /// script
|
|
||||||
# requires-python = ">=3.11"
|
|
||||||
# dependencies = []
|
|
||||||
# ///
|
|
||||||
"""
|
|
||||||
wiki_search.py — BM25 search over wiki pages with frontmatter filters.
|
|
||||||
|
|
||||||
Fallback for when index-first navigation doesn't surface the right pages.
|
|
||||||
Pure-Python implementation (no dependencies beyond stdlib) so it runs anywhere.
|
|
||||||
|
|
||||||
Usage:
|
|
||||||
python wiki_search.py "query terms" [options]
|
|
||||||
|
|
||||||
Options:
|
|
||||||
--wiki <dir> Wiki directory (default: cml/wiki)
|
|
||||||
--top N Return top N results (default: 10)
|
|
||||||
--type <type> Filter by frontmatter type (source|entity|concept|synthesis|...)
|
|
||||||
--tag <tag> Filter by tag (repeatable)
|
|
||||||
--since YYYY-MM-DD Only pages updated on or after this date
|
|
||||||
--backlinks <slug> Find pages that link to <slug>; ignores the query
|
|
||||||
--top-linked N Show the N most-linked-to pages (hubs); ignores the query
|
|
||||||
--cache <path> Persist the BM25 index to disk for faster reruns
|
|
||||||
|
|
||||||
Examples:
|
|
||||||
python wiki_search.py "diffusion training stability" --top 5
|
|
||||||
python wiki_search.py "alignment" --type concept --tag safety
|
|
||||||
python wiki_search.py "" --backlinks transformer
|
|
||||||
python wiki_search.py "" --top-linked 10
|
|
||||||
"""
|
|
||||||
|
|
||||||
import argparse
|
|
||||||
import json
|
|
||||||
import math
|
|
||||||
import re
|
|
||||||
import sys
|
|
||||||
from collections import Counter, defaultdict
|
|
||||||
from datetime import date, datetime
|
|
||||||
from pathlib import Path
|
|
||||||
|
|
||||||
|
|
||||||
WIKILINK_RE = re.compile(r"\[\[([^\]|]+)(?:\|[^\]]+)?\]\]")
|
|
||||||
FRONTMATTER_RE = re.compile(r"^---\s*\n(.*?)\n---\s*\n", re.DOTALL)
|
|
||||||
TOKEN_RE = re.compile(r"[a-z0-9]+")
|
|
||||||
|
|
||||||
|
|
||||||
def parse_frontmatter(text: str) -> tuple[dict, str]:
|
|
||||||
"""Lightweight YAML-ish frontmatter parser. Returns (metadata, body)."""
|
|
||||||
m = FRONTMATTER_RE.match(text)
|
|
||||||
if not m:
|
|
||||||
return {}, text
|
|
||||||
fm_text = m.group(1)
|
|
||||||
body = text[m.end():]
|
|
||||||
meta = {}
|
|
||||||
current_key = None
|
|
||||||
for line in fm_text.split("\n"):
|
|
||||||
if not line.strip():
|
|
||||||
continue
|
|
||||||
# Inline list: tags: [a, b, c]
|
|
||||||
kv = re.match(r"^([a-zA-Z_]+):\s*(.*)$", line)
|
|
||||||
if kv:
|
|
||||||
key, value = kv.group(1), kv.group(2).strip()
|
|
||||||
if value.startswith("[") and value.endswith("]"):
|
|
||||||
items = [x.strip().strip('"').strip("'") for x in value[1:-1].split(",") if x.strip()]
|
|
||||||
meta[key] = items
|
|
||||||
elif value:
|
|
||||||
meta[key] = value.strip('"').strip("'")
|
|
||||||
else:
|
|
||||||
meta[key] = []
|
|
||||||
current_key = key
|
|
||||||
elif line.startswith(" - ") and current_key:
|
|
||||||
meta[current_key].append(line[4:].strip().strip('"').strip("'"))
|
|
||||||
return meta, body
|
|
||||||
|
|
||||||
|
|
||||||
def tokenize(text: str) -> list[str]:
|
|
||||||
return TOKEN_RE.findall(text.lower())
|
|
||||||
|
|
||||||
|
|
||||||
def slug_from_path(path: Path, wiki_root: Path) -> str:
|
|
||||||
return path.stem
|
|
||||||
|
|
||||||
|
|
||||||
def extract_wikilinks(body: str) -> list[str]:
|
|
||||||
return [m.group(1).strip() for m in WIKILINK_RE.finditer(body)]
|
|
||||||
|
|
||||||
|
|
||||||
def collect_pages(wiki_root: Path) -> list[dict]:
|
|
||||||
"""Walk the wiki and return [{path, slug, meta, body, tokens, links}]."""
|
|
||||||
pages = []
|
|
||||||
for md_path in wiki_root.rglob("*.md"):
|
|
||||||
# Skip the schema, index, log, and template files
|
|
||||||
rel = md_path.relative_to(wiki_root)
|
|
||||||
if rel.parts[0] in {"SCHEMA.md", "index.md", "log.md"} or rel.name.startswith("."):
|
|
||||||
continue
|
|
||||||
if rel.parts[0] in {"indexes", "graph"}:
|
|
||||||
continue
|
|
||||||
try:
|
|
||||||
text = md_path.read_text(encoding="utf-8")
|
|
||||||
except (UnicodeDecodeError, OSError):
|
|
||||||
continue
|
|
||||||
meta, body = parse_frontmatter(text)
|
|
||||||
pages.append({
|
|
||||||
"path": str(md_path),
|
|
||||||
"rel_path": str(rel),
|
|
||||||
"slug": slug_from_path(md_path, wiki_root),
|
|
||||||
"meta": meta,
|
|
||||||
"body": body,
|
|
||||||
"tokens": tokenize(body + " " + meta.get("title", "")),
|
|
||||||
"links": extract_wikilinks(body),
|
|
||||||
})
|
|
||||||
return pages
|
|
||||||
|
|
||||||
|
|
||||||
def build_bm25(pages: list[dict]) -> dict:
|
|
||||||
"""Build a BM25 index. Returns {df, avgdl, N, doc_lens, term_freqs}."""
|
|
||||||
N = len(pages)
|
|
||||||
df = Counter()
|
|
||||||
doc_lens = []
|
|
||||||
term_freqs = []
|
|
||||||
for page in pages:
|
|
||||||
tokens = page["tokens"]
|
|
||||||
doc_lens.append(len(tokens))
|
|
||||||
tf = Counter(tokens)
|
|
||||||
term_freqs.append(tf)
|
|
||||||
for term in tf:
|
|
||||||
df[term] += 1
|
|
||||||
avgdl = sum(doc_lens) / N if N else 0
|
|
||||||
return {"N": N, "df": df, "avgdl": avgdl, "doc_lens": doc_lens, "term_freqs": term_freqs}
|
|
||||||
|
|
||||||
|
|
||||||
def bm25_score(query_tokens: list[str], doc_idx: int, idx: dict, k1: float = 1.5, b: float = 0.75) -> float:
|
|
||||||
score = 0.0
|
|
||||||
N = idx["N"]
|
|
||||||
df = idx["df"]
|
|
||||||
avgdl = idx["avgdl"]
|
|
||||||
dl = idx["doc_lens"][doc_idx]
|
|
||||||
tf = idx["term_freqs"][doc_idx]
|
|
||||||
for term in query_tokens:
|
|
||||||
if term not in df:
|
|
||||||
continue
|
|
||||||
idf = math.log(1 + (N - df[term] + 0.5) / (df[term] + 0.5))
|
|
||||||
f = tf.get(term, 0)
|
|
||||||
if f == 0:
|
|
||||||
continue
|
|
||||||
denom = f + k1 * (1 - b + b * (dl / avgdl if avgdl else 1))
|
|
||||||
score += idf * (f * (k1 + 1)) / denom
|
|
||||||
return score
|
|
||||||
|
|
||||||
|
|
||||||
def parse_date(s: str | None) -> date | None:
|
|
||||||
if not s:
|
|
||||||
return None
|
|
||||||
try:
|
|
||||||
return datetime.strptime(s[:10], "%Y-%m-%d").date()
|
|
||||||
except (ValueError, TypeError):
|
|
||||||
return None
|
|
||||||
|
|
||||||
|
|
||||||
def passes_filters(page: dict, args) -> bool:
|
|
||||||
meta = page["meta"]
|
|
||||||
if args.type and meta.get("type") != args.type:
|
|
||||||
return False
|
|
||||||
if args.tag:
|
|
||||||
page_tags = set(meta.get("tags", []) or [])
|
|
||||||
if not all(t in page_tags for t in args.tag):
|
|
||||||
return False
|
|
||||||
if args.since:
|
|
||||||
since = parse_date(args.since)
|
|
||||||
updated = parse_date(meta.get("updated"))
|
|
||||||
if since and updated and updated < since:
|
|
||||||
return False
|
|
||||||
if since and not updated:
|
|
||||||
return False
|
|
||||||
return True
|
|
||||||
|
|
||||||
|
|
||||||
def cmd_search(args, pages: list[dict]) -> None:
|
|
||||||
filtered = [p for p in pages if passes_filters(p, args)]
|
|
||||||
if not filtered:
|
|
||||||
print("No pages matched the filters.", file=sys.stderr)
|
|
||||||
return
|
|
||||||
idx = build_bm25(filtered)
|
|
||||||
query_tokens = tokenize(args.query)
|
|
||||||
if not query_tokens:
|
|
||||||
print("Empty query.", file=sys.stderr)
|
|
||||||
return
|
|
||||||
scored = [(bm25_score(query_tokens, i, idx), i) for i in range(len(filtered))]
|
|
||||||
scored.sort(key=lambda x: -x[0])
|
|
||||||
top = [(s, filtered[i]) for s, i in scored[:args.top] if s > 0]
|
|
||||||
if not top:
|
|
||||||
print("No matches.", file=sys.stderr)
|
|
||||||
return
|
|
||||||
print(f"Top {len(top)} results for: {args.query!r}")
|
|
||||||
print()
|
|
||||||
for score, page in top:
|
|
||||||
title = page["meta"].get("title") or page["slug"]
|
|
||||||
page_type = page["meta"].get("type", "?")
|
|
||||||
print(f" [{score:6.2f}] [{page_type:9}] {title}")
|
|
||||||
print(f" {page['rel_path']}")
|
|
||||||
|
|
||||||
|
|
||||||
def cmd_backlinks(args, pages: list[dict]) -> None:
|
|
||||||
target = args.backlinks
|
|
||||||
inbound = []
|
|
||||||
for page in pages:
|
|
||||||
if target in page["links"]:
|
|
||||||
inbound.append(page)
|
|
||||||
if not inbound:
|
|
||||||
print(f"No pages link to [[{target}]].", file=sys.stderr)
|
|
||||||
return
|
|
||||||
print(f"Pages linking to [[{target}]] ({len(inbound)}):")
|
|
||||||
for page in inbound:
|
|
||||||
title = page["meta"].get("title") or page["slug"]
|
|
||||||
print(f" - {title} ({page['rel_path']})")
|
|
||||||
|
|
||||||
|
|
||||||
def cmd_top_linked(args, pages: list[dict]) -> None:
|
|
||||||
inbound_count = Counter()
|
|
||||||
for page in pages:
|
|
||||||
for link in page["links"]:
|
|
||||||
inbound_count[link] += 1
|
|
||||||
top = inbound_count.most_common(args.top_linked)
|
|
||||||
if not top:
|
|
||||||
print("No links found in the wiki.", file=sys.stderr)
|
|
||||||
return
|
|
||||||
print(f"Top {len(top)} most-linked-to pages (hubs):")
|
|
||||||
for slug, count in top:
|
|
||||||
# Try to find the page for the title
|
|
||||||
match = next((p for p in pages if p["slug"] == slug), None)
|
|
||||||
title = (match["meta"].get("title") if match else None) or slug
|
|
||||||
marker = "" if match else " [BROKEN LINK]"
|
|
||||||
print(f" {count:4d} {title} ({slug}){marker}")
|
|
||||||
|
|
||||||
|
|
||||||
def main():
|
|
||||||
parser = argparse.ArgumentParser(description=__doc__, formatter_class=argparse.RawDescriptionHelpFormatter)
|
|
||||||
parser.add_argument("query", nargs="?", default="", help="Query terms.")
|
|
||||||
parser.add_argument("--wiki", type=Path, default=Path("cml/wiki"), help="Wiki directory (default: cml/wiki).")
|
|
||||||
parser.add_argument("--top", type=int, default=10, help="Top N results (default: 10).")
|
|
||||||
parser.add_argument("--type", help="Filter by frontmatter type.")
|
|
||||||
parser.add_argument("--tag", action="append", default=[], help="Filter by tag (repeatable).")
|
|
||||||
parser.add_argument("--since", help="Only pages updated on or after YYYY-MM-DD.")
|
|
||||||
parser.add_argument("--backlinks", help="Find pages linking to this slug.")
|
|
||||||
parser.add_argument("--top-linked", type=int, help="Show the N most-linked-to pages.")
|
|
||||||
parser.add_argument("--cache", type=Path, help="(reserved) Cache path for BM25 index.")
|
|
||||||
args = parser.parse_args()
|
|
||||||
|
|
||||||
if not args.wiki.exists():
|
|
||||||
print(f"Wiki directory not found: {args.wiki}", file=sys.stderr)
|
|
||||||
sys.exit(1)
|
|
||||||
|
|
||||||
pages = collect_pages(args.wiki)
|
|
||||||
if not pages:
|
|
||||||
print(f"No wiki pages found under {args.wiki}", file=sys.stderr)
|
|
||||||
sys.exit(0)
|
|
||||||
|
|
||||||
if args.backlinks:
|
|
||||||
cmd_backlinks(args, pages)
|
|
||||||
elif args.top_linked:
|
|
||||||
cmd_top_linked(args, pages)
|
|
||||||
elif args.query:
|
|
||||||
cmd_search(args, pages)
|
|
||||||
else:
|
|
||||||
parser.print_help()
|
|
||||||
sys.exit(1)
|
|
||||||
|
|
||||||
|
|
||||||
if __name__ == "__main__":
|
|
||||||
main()
|
|
||||||
@@ -1,159 +0,0 @@
|
|||||||
#!/usr/bin/env -S uv run --script
|
|
||||||
# /// script
|
|
||||||
# requires-python = ">=3.11"
|
|
||||||
# dependencies = []
|
|
||||||
# ///
|
|
||||||
"""
|
|
||||||
wiki_stats.py — Quick summary of wiki size, shape, and link density.
|
|
||||||
|
|
||||||
Useful for deciding when to shard the index or split pages.
|
|
||||||
|
|
||||||
Usage:
|
|
||||||
python wiki_stats.py [<wiki-dir>]
|
|
||||||
|
|
||||||
Example:
|
|
||||||
python wiki_stats.py wiki/
|
|
||||||
"""
|
|
||||||
|
|
||||||
import argparse
|
|
||||||
import re
|
|
||||||
import sys
|
|
||||||
from collections import Counter
|
|
||||||
from pathlib import Path
|
|
||||||
|
|
||||||
|
|
||||||
WIKILINK_RE = re.compile(r"\[\[([^\]|]+)(?:\|[^\]]+)?\]\]")
|
|
||||||
FRONTMATTER_RE = re.compile(r"^---\s*\n(.*?)\n---\s*\n", re.DOTALL)
|
|
||||||
|
|
||||||
|
|
||||||
SKIP_TOP_LEVEL_FILES = {"SCHEMA.md", "log.md", "README.md"}
|
|
||||||
SKIP_TOP_LEVEL_DIRS = {"indexes", "graph"}
|
|
||||||
|
|
||||||
|
|
||||||
def parse_type(text: str) -> str | None:
|
|
||||||
m = FRONTMATTER_RE.match(text)
|
|
||||||
if not m:
|
|
||||||
return None
|
|
||||||
fm = m.group(1)
|
|
||||||
for line in fm.split("\n"):
|
|
||||||
kv = re.match(r"^type:\s*(.*)$", line)
|
|
||||||
if kv:
|
|
||||||
return kv.group(1).strip().strip('"').strip("'")
|
|
||||||
return None
|
|
||||||
|
|
||||||
|
|
||||||
def main():
|
|
||||||
parser = argparse.ArgumentParser(description=__doc__, formatter_class=argparse.RawDescriptionHelpFormatter)
|
|
||||||
parser.add_argument("wiki", nargs="?", type=Path, default=Path("cml/wiki"))
|
|
||||||
args = parser.parse_args()
|
|
||||||
|
|
||||||
if not args.wiki.exists():
|
|
||||||
print(f"Wiki directory not found: {args.wiki}", file=sys.stderr)
|
|
||||||
sys.exit(1)
|
|
||||||
|
|
||||||
total_pages = 0
|
|
||||||
total_lines = 0
|
|
||||||
total_words = 0
|
|
||||||
total_links = 0
|
|
||||||
pages_by_type = Counter()
|
|
||||||
pages_by_dir = Counter()
|
|
||||||
largest = []
|
|
||||||
most_linked_in = Counter()
|
|
||||||
index_lines = 0
|
|
||||||
|
|
||||||
for md_path in args.wiki.rglob("*.md"):
|
|
||||||
rel = md_path.relative_to(args.wiki)
|
|
||||||
try:
|
|
||||||
text = md_path.read_text(encoding="utf-8")
|
|
||||||
except (UnicodeDecodeError, OSError):
|
|
||||||
continue
|
|
||||||
|
|
||||||
if rel.name == "index.md" and len(rel.parts) == 1:
|
|
||||||
index_lines = text.count("\n") + 1
|
|
||||||
continue
|
|
||||||
if rel.parts[0] in SKIP_TOP_LEVEL_FILES:
|
|
||||||
continue
|
|
||||||
if rel.parts[0] in SKIP_TOP_LEVEL_DIRS:
|
|
||||||
continue
|
|
||||||
if rel.name.startswith("."):
|
|
||||||
continue
|
|
||||||
|
|
||||||
total_pages += 1
|
|
||||||
line_count = text.count("\n") + 1
|
|
||||||
word_count = len(text.split())
|
|
||||||
total_lines += line_count
|
|
||||||
total_words += word_count
|
|
||||||
# Strip frontmatter before counting wikilinks; frontmatter uses bare slugs.
|
|
||||||
body = FRONTMATTER_RE.sub("", text, count=1) if text.startswith("---") else text
|
|
||||||
links = WIKILINK_RE.findall(body)
|
|
||||||
total_links += len(links)
|
|
||||||
for link in links:
|
|
||||||
target = link.split("|")[0].strip()
|
|
||||||
most_linked_in[target] += 1
|
|
||||||
page_type = parse_type(text) or "(none)"
|
|
||||||
pages_by_type[page_type] += 1
|
|
||||||
if len(rel.parts) > 1:
|
|
||||||
pages_by_dir[rel.parts[0]] += 1
|
|
||||||
else:
|
|
||||||
pages_by_dir["(root)"] += 1
|
|
||||||
largest.append((line_count, str(rel)))
|
|
||||||
|
|
||||||
largest.sort(reverse=True)
|
|
||||||
|
|
||||||
print("=" * 60)
|
|
||||||
print(f"Wiki Stats: {args.wiki}")
|
|
||||||
print("=" * 60)
|
|
||||||
print(f"Pages: {total_pages}")
|
|
||||||
print(f"Total lines: {total_lines:,}")
|
|
||||||
print(f"Total words: {total_words:,}")
|
|
||||||
print(f"Total links: {total_links:,}")
|
|
||||||
if total_pages:
|
|
||||||
print(f"Avg page: {total_lines // total_pages} lines / {total_words // total_pages} words")
|
|
||||||
print(f"Link density: {total_links / total_pages:.1f} links per page")
|
|
||||||
print(f"index.md: {index_lines} lines" + (" ← shard recommended (>300)" if index_lines > 300 else ""))
|
|
||||||
print()
|
|
||||||
|
|
||||||
print("Pages by type:")
|
|
||||||
for t, n in pages_by_type.most_common():
|
|
||||||
print(f" {t:15s} {n}")
|
|
||||||
print()
|
|
||||||
|
|
||||||
print("Pages by directory:")
|
|
||||||
for d, n in pages_by_dir.most_common():
|
|
||||||
print(f" {d:15s} {n}")
|
|
||||||
print()
|
|
||||||
|
|
||||||
if largest:
|
|
||||||
print("Largest pages:")
|
|
||||||
for lines, path in largest[:10]:
|
|
||||||
warn = ""
|
|
||||||
if lines > 800:
|
|
||||||
warn = " ← OVER HARD CAP"
|
|
||||||
elif lines > 400:
|
|
||||||
warn = " ← over soft cap"
|
|
||||||
print(f" {lines:5d} {path}{warn}")
|
|
||||||
print()
|
|
||||||
|
|
||||||
if most_linked_in:
|
|
||||||
print("Most-linked-to pages (hubs):")
|
|
||||||
for slug, count in most_linked_in.most_common(10):
|
|
||||||
print(f" {count:4d} [[{slug}]]")
|
|
||||||
print()
|
|
||||||
|
|
||||||
# Scaling recommendations
|
|
||||||
print("Scaling thresholds:")
|
|
||||||
if total_pages < 50:
|
|
||||||
print(" → Below first threshold. Flat structure is fine.")
|
|
||||||
elif total_pages < 150 and index_lines < 300:
|
|
||||||
print(" → Below shard threshold. Continue with single index.md.")
|
|
||||||
elif (total_pages >= 150 or index_lines >= 300) and not (args.wiki / "indexes").exists():
|
|
||||||
print(" → AT SHARD THRESHOLD. Consider sharding index.md into wiki/indexes/<type>.md.")
|
|
||||||
print(" See references/scaling-playbook.md.")
|
|
||||||
elif total_pages >= 300:
|
|
||||||
print(" → Past 300 pages. Use scripts/wiki_search.py as a routine fallback.")
|
|
||||||
if total_pages >= 500:
|
|
||||||
print(" → Past 500 pages. Run lint weekly or per-N-ingests.")
|
|
||||||
|
|
||||||
|
|
||||||
if __name__ == "__main__":
|
|
||||||
main()
|
|
||||||
215
skills/wiki/README.md
Normal file
215
skills/wiki/README.md
Normal file
@@ -0,0 +1,215 @@
|
|||||||
|
# wiki — jak to funguje
|
||||||
|
|
||||||
|
Hybrid hledání ve tvých vlastních poznámkách: dva git repozitáře (`index`, `travel`) plus živý
|
||||||
|
workspace nanobota. Skill je **výhradně čtecí** — do tvých repů nikdy nezapisuje a klony drží
|
||||||
|
jen jako pracovní kopii. Index je derived artifact: kdykoli se dá smazat a postavit od nuly
|
||||||
|
(u dnešního objemu jsou to desítky sekund).
|
||||||
|
|
||||||
|
## Tři vrstvy, od nejlevnější
|
||||||
|
|
||||||
|
```text
|
||||||
|
grep ripgrep přes soubory na disku žádný index, nikdy zastaralé, vidí i kód
|
||||||
|
↓
|
||||||
|
toc katalog `files` z databáze adresář → soubor → titulek + tagy
|
||||||
|
↓
|
||||||
|
search FTS5 (BM25) + vec0 (KNN) → RRF parafráze, když neznáš slova
|
||||||
|
```
|
||||||
|
|
||||||
|
Není to fallback řetěz, kde se jde dolů, když horní vrstva zklame — je to volba podle **tvaru
|
||||||
|
otázky**: přesný string, hostname nebo identifikátor patří do `grep`, orientace („co o tom
|
||||||
|
vůbec mám?") do `toc`, parafráze do `search`.
|
||||||
|
|
||||||
|
`grep` je tedy základ, ne poslední záchrana. Soubory na disku beztak leží, takže nestojí nic
|
||||||
|
navíc, a nad kódem je exact match to hlavní — proto se zdrojáky vůbec neindexují (viz níž).
|
||||||
|
|
||||||
|
## Index nikdy nevzniká v tahu agenta
|
||||||
|
|
||||||
|
Indexuje cron každou minutu. Agent v tahu jen **čte** hotový index.
|
||||||
|
|
||||||
|
Proč takhle: `exec` tool má timeout 60 s a plný index je minuty. Indexace v odpovědi by
|
||||||
|
navíc při nedostupné Ollamě dělala z čerstvého dokumentu FTS-only výsledek.
|
||||||
|
|
||||||
|
Drtivá většina tiků neudělá nic a **nezapíše ani řádek**. Když je `log/wiki_sync_cron.log`
|
||||||
|
prázdný, je to správný stav, ne že cron neběží.
|
||||||
|
|
||||||
|
## Co se v tom tiku vlastně děje
|
||||||
|
|
||||||
|
1. Vezme lock `wiki/.sync.lock`. Když ho drží živý běh, skončí bez výpisu. Lock po mrtvém
|
||||||
|
procesu (`pid` neexistuje nebo je starší než 30 min) si vezme zpátky a zaloguje
|
||||||
|
`WARN stale lock, reclaiming`.
|
||||||
|
2. **Levná detekce změn, bez indexace.** U git zdrojů `git ls-remote` — zjistí remote `HEAD`
|
||||||
|
**bez** `fetch`, což je na minutovou kadenci ten správný nástroj; `fetch` teprve když se
|
||||||
|
revize liší. U workspace zdroje walk a porovnání `path` + `size` + `mtime`; sha256 se
|
||||||
|
počítá jen při neshodě.
|
||||||
|
3. Nic se nezměnilo a nic nečeká na vektor → konec.
|
||||||
|
4. Změněné soubory se rozřežou na chunky a pošlou do Ollamy po 32. Pak se **doberou všechny
|
||||||
|
chunky bez vektoru**, i když se žádný soubor nezměnil — tohle je cesta zpátky
|
||||||
|
z degradovaného režimu, když byla Ollama chvíli mimo.
|
||||||
|
5. Zapíše `indexed_rev` a `last_sync_at`, přidá coverage řádek a shrnutí do
|
||||||
|
`log/wiki_sync.log`.
|
||||||
|
|
||||||
|
Selhání sítě je **per zdroj**: timeout nebo chyba gitu zaloguje `WARN`, ten zdroj přeskočí
|
||||||
|
s nezměněnou `indexed_rev` a ostatní dojedou. Bez timeoutů by visící `ls-remote` držel lock
|
||||||
|
a zablokoval i workspace zdroj, který se sítí nemá nic společného.
|
||||||
|
|
||||||
|
## Proč hybrid
|
||||||
|
|
||||||
|
Každá polovina umí něco jiného a ani jedna neumí obojí:
|
||||||
|
|
||||||
|
| | Umí | Neumí |
|
||||||
|
|---|---|---|
|
||||||
|
| **BM25** (FTS5) | exact match, čísla, jména, zkratky | parafrázi, kde se slova nepřekrývají |
|
||||||
|
| **vektory** (`vec0`) | „elektroodpad ze stavebnic" → „Mindstorms po ukončení podpory" | přesné řetězce a čísla |
|
||||||
|
|
||||||
|
Výsledky slučuje **RRF** — sečte převrácené ranky z obou seznamů. Aby to mělo definovaný
|
||||||
|
význam, musí obě poloviny řadit **tutéž množinu**: proto je jedinou retrieval jednotkou chunk,
|
||||||
|
ne soubor. Kdyby BM25 řadil soubory a vektory chunky, merge by nešlo interpretovat.
|
||||||
|
|
||||||
|
Naměřeno na reálných poznámkách: RRF dává MRR 0,367 proti 0,257 (BM25 sám) a 0,261 (vektory
|
||||||
|
samy), takže merge opravdu vyhrává. Plná čísla, včetně srovnání čtyř modelů, jsou
|
||||||
|
v `.claude/tracking/plans/final-wiki-hybrid-rag-result.md` v trackovacím repu.
|
||||||
|
|
||||||
|
Ve výstupu `search` u každého hitu stojí, která polovina ho našla (`bm25 #2, vec #1`). Hit,
|
||||||
|
na kterém se poloviny **shodnou**, má výrazně větší cenu než ten, který vytáhly jen vektory.
|
||||||
|
|
||||||
|
## Česká flexe — na co narazíš
|
||||||
|
|
||||||
|
Dotazová vrstva lepí `*` na slova od tří znaků, aby pokryla české koncovky. **Funguje to jen
|
||||||
|
napůl** a je dobré vědět jak: wildcard je **prefixový**, takže pomůže jen tehdy, když je tvoje
|
||||||
|
slovo prefixem tvaru v dokumentu. Česká flexe ale mění koncovku, ne začátek.
|
||||||
|
|
||||||
|
Naměřeno nad `travel/packaging-list.md`:
|
||||||
|
|
||||||
|
| Dotaz | Chunků v tom souboru |
|
||||||
|
|---|---|
|
||||||
|
| `cestu*` | **0** |
|
||||||
|
| `cesty*` | 3 |
|
||||||
|
| `cest*` | 4 |
|
||||||
|
|
||||||
|
Prakticky: na *„co si vzít na cestu do zahraničí"* se doslovný seznam věcí na cestu nedostal
|
||||||
|
ani do top 10. Když víš, že něco máš, a `search` to nenajde, zkus **kmen slova** (`cest`,
|
||||||
|
`záloh`) nebo rovnou `grep`. Rozhodnutí, jestli to řešit systémově, je otevřená položka
|
||||||
|
v `todo.md` trackovacího repa — každá varianta rozšiřuje kandidátní pool, takže to není
|
||||||
|
zdarma.
|
||||||
|
|
||||||
|
## Index je nápověda, ne odpověď
|
||||||
|
|
||||||
|
Dvě věci, které se snadno přehlédnou:
|
||||||
|
|
||||||
|
- **Výřez v `search` není zdroj pravdy.** Je zkrácený a index může být o minutu starší než
|
||||||
|
disk. Než se z nálezu odpovídá, má se ten soubor přečíst čerstvý — `SKILL.md` to agentovi
|
||||||
|
říká, ale platí to i pro tebe.
|
||||||
|
- **Vektorová polovina vždycky něco vrátí.** KNN nemá dolní mez podobnosti, takže na jakýkoli
|
||||||
|
dotaz nabídne svých k nejbližších chunků. Pro retrieval je to záměr (posoudit relevanci je
|
||||||
|
na čtenáři), ale znamená to, že prázdný výsledek nikdy neznamená „nic nesedí" — znamená
|
||||||
|
prázdný index.
|
||||||
|
|
||||||
|
## Co se indexuje a co ne
|
||||||
|
|
||||||
|
Rozsah řídí `wiki/config.yaml` ve třech krocích: `paths` (whitelist — co v něm není, pro index
|
||||||
|
neexistuje) → `include` (whitelist přípon, `*.md`) → `exclude` (skalpel, vyhrává nad oběma).
|
||||||
|
|
||||||
|
Proč whitelist a ne blacklist: rozhoduje **směr selhání**. U blacklistu nový adresář *tiše
|
||||||
|
vstoupí* do indexu, u whitelistu *tiše chybí*. Index je komprimovaná kopie obsahu včetně
|
||||||
|
osobních věcí, takže „tiše zaindexováno" je horší porucha — a workspace se mění i bez tebe,
|
||||||
|
Dream do něj zapisuje sám.
|
||||||
|
|
||||||
|
Proti té jediné slabině whitelistu stojí záchranná síť: sync na konci zaloguje řádek
|
||||||
|
`coverage <zdroj>: markdown outside paths in …` s top-level adresáři, které markdown mají
|
||||||
|
a žádný zdroj je nepokrývá. Adresáře, které tam vidíš pořád (`backup`, `tmp`, `skills`, …),
|
||||||
|
jsou vědomě vynechané; zajímavý je **nový** přírůstek v tom seznamu.
|
||||||
|
|
||||||
|
**Indexuje se výhradně markdown**, a to bez jakékoli detekce typu obsahu — `main.py` se
|
||||||
|
netrefí do `include: ["*.md"]` a k chunkeru se nedostane. Důvod je měřený: markdown parser
|
||||||
|
dělá z Python komentáře `# TODO: …` nadpis a `unicode61` neumí identifikátory (`send` uvnitř
|
||||||
|
`SendAsync` je miss). Kód pokrývá `grep` — jede přes celý klon včetně `.txt` a `.mhtml`.
|
||||||
|
|
||||||
|
## Kde co leží
|
||||||
|
|
||||||
|
Relativně ke workspace (`~/.nanobot/workspace`):
|
||||||
|
|
||||||
|
| Cesta | Role |
|
||||||
|
|---|---|
|
||||||
|
| `wiki/config.yaml` | zdroje a model — **jediný verzovaný soubor** pod `wiki/` |
|
||||||
|
| `wiki/index.sqlite` | `chunks` + FTS5 + vektory; derived, gitignorovaný |
|
||||||
|
| `wiki/remote/<id>/` | non-bare klony git zdrojů (working tree, aby měl `grep` co číst) |
|
||||||
|
| `wiki/.sync.lock` | JSON `{pid, started_at}`, jen když sync běží |
|
||||||
|
| `log/wiki_sync.log` | jeden řádek na událost — co se naindexovalo, `WARN`, coverage |
|
||||||
|
| `log/wiki_sync_cron.log` | stdout cronu; **prázdný je správně** |
|
||||||
|
| `skills/wiki/scripts/` | `wiki_sync.py` (indexace) a `wiki_search.py` (dotazy) |
|
||||||
|
|
||||||
|
Klon je záměrně non-bare: `--mirror` nemá working tree, takže by `grep` neměl co číst.
|
||||||
|
Cena je dvojnásobek místa na disku, což je na těchhle repech nic.
|
||||||
|
|
||||||
|
## Ruční spuštění
|
||||||
|
|
||||||
|
```bash
|
||||||
|
cd ~/.nanobot/workspace
|
||||||
|
|
||||||
|
# to, co dělá cron
|
||||||
|
~/.local/bin/uv run skills/wiki/scripts/wiki_sync.py
|
||||||
|
|
||||||
|
# jen jeden zdroj
|
||||||
|
~/.local/bin/uv run skills/wiki/scripts/wiki_sync.py --source travel
|
||||||
|
|
||||||
|
# smazat index a postavit od nuly
|
||||||
|
~/.local/bin/uv run skills/wiki/scripts/wiki_sync.py --full
|
||||||
|
|
||||||
|
# dotazy
|
||||||
|
~/.local/bin/uv run skills/wiki/scripts/wiki_search.py search "zálohování" --limit 5
|
||||||
|
~/.local/bin/uv run skills/wiki/scripts/wiki_search.py toc --source travel
|
||||||
|
~/.local/bin/uv run skills/wiki/scripts/wiki_search.py toc --tag caj
|
||||||
|
~/.local/bin/uv run skills/wiki/scripts/wiki_search.py grep "WireGuard" --source index
|
||||||
|
```
|
||||||
|
|
||||||
|
`--full` potřebuješ jen po změně embedding modelu, jeho dimenzí, `query_prefix` nebo chunkeru
|
||||||
|
— tedy přesně tehdy, když dotaz začne hlásit `reindex needed`. Přeembeduje všechno, takže to
|
||||||
|
nepouštěj v tahu, kde na to někdo čeká.
|
||||||
|
|
||||||
|
Plná cesta k `uv` je tu proto, že v neinteraktivním SSH není v `PATH`. Crontab si `PATH`
|
||||||
|
nastavuje sám, takže tam stačí `uv run …`.
|
||||||
|
|
||||||
|
## Když něco nefunguje
|
||||||
|
|
||||||
|
| Hláška | Co to je |
|
||||||
|
|---|---|
|
||||||
|
| `note: embeddings unavailable … FTS-only results` | Ollama na `nvidia.hell` neodpovídá. Sync mezitím indexuje dál a vektory dosadí sám, až se vrátí. |
|
||||||
|
| `note: N chunks still awaiting vectors` | Sync je rozjezdu nebo byl degradovaný. Sémantická polovina je zatím neúplná. |
|
||||||
|
| `reindex needed: …` (exit 1) | Index byl postavený pod jiným embedding kontraktem než má config. Poznámky jsou v pořádku, řeší to `--full`. |
|
||||||
|
| `WARN <zdroj>: git … failed` | Ten zdroj se přeskočil, ostatní dojely. `indexed_rev` zůstala, takže příští tik to zkusí znovu. |
|
||||||
|
| `(no matches)` | Prázdný index — ne „nic nesedí" (viz výš). |
|
||||||
|
|
||||||
|
Všechno ostatní hledej v `log/wiki_sync.log`. Když je prázdný i po změně poznámek, problém je
|
||||||
|
v cronu, ne v hledání.
|
||||||
|
|
||||||
|
## Jak ověřit, že to funguje
|
||||||
|
|
||||||
|
**Smoke loop v konzoli** — projde celou smyčku od zápisu po nalezení:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
cd ~/.nanobot/workspace
|
||||||
|
printf '# Smoke\n\nZrzavý jednorožec kontroluje wiki index.\n' > notes/_smoke.md
|
||||||
|
sleep 90
|
||||||
|
~/.local/bin/uv run skills/wiki/scripts/wiki_search.py search "zrzavý jednorožec" --limit 3
|
||||||
|
rm notes/_smoke.md # cron ho z indexu odklidí sám do minuty
|
||||||
|
```
|
||||||
|
|
||||||
|
**Dotaz do WebUI na každý zdroj.** Princip: vyber fakt, který model **nemůže** znát
|
||||||
|
z obecných znalostí — pak je konkrétní správná odpověď sama důkazem, že přišla z poznámek.
|
||||||
|
Když odpoví obecně a čísla vynechá nebo si je vymyslí, neprošel.
|
||||||
|
|
||||||
|
Příklady platné k 2026-09-09 (poznámky se mění, princip ne):
|
||||||
|
|
||||||
|
| Zdroj | Dotaz | Musí padnout |
|
||||||
|
|---|---|---|
|
||||||
|
| `workspace` | Kolik vody a čaje mám v receptu na Teh Tarik? | 700–800 ml, dvě polévkové lžíce čaje |
|
||||||
|
| `travel` | Co si mám podle poznámek vzít na Sněžku? | rozdvojka do zásuvky, power banka, Sony ANC sluchátka |
|
||||||
|
| `index` | Co je projekt Sauter Bus? | RS485 proxy s odposlechem a modifikací provozu, micro:bit |
|
||||||
|
|
||||||
|
**Negativní kontrola** je nejcennější z celé sady: zeptej se na téma, které v poznámkách
|
||||||
|
prokazatelně není (např. pěstování bonsají). Správná odpověď je „nic o tom nemáš". Když
|
||||||
|
začne vyprávět, doplňuje si výsledky z obecných znalostí — a to je zrádnější vada než
|
||||||
|
nefunkční index, protože se tváří jako odpověď z poznámek.
|
||||||
|
|
||||||
|
Když si nejsi jistý, odkud odpověď je, dopiš **„ze kterého souboru to je?"**. Má přijít
|
||||||
|
konkrétní cesta, ne „z tvých poznámek".
|
||||||
106
skills/wiki/SKILL.md
Normal file
106
skills/wiki/SKILL.md
Normal file
@@ -0,0 +1,106 @@
|
|||||||
|
---
|
||||||
|
name: wiki
|
||||||
|
description: >
|
||||||
|
Search the user's own notes — personal git note repos plus the live workspace — by
|
||||||
|
keyword, topic or meaning. Use to answer questions about what the user has written
|
||||||
|
down, recorded or decided, and to find which file covers a topic.
|
||||||
|
Triggers on: "/wiki", "find in my notes", "what did I write about", "do I have notes on".
|
||||||
|
Read-only: it never writes, edits or captures notes, and it does not answer from
|
||||||
|
general knowledge — only from what is indexed on disk.
|
||||||
|
---
|
||||||
|
|
||||||
|
# /wiki
|
||||||
|
|
||||||
|
Hybrid search over the user's notes. `chunks` is the single retrieval unit: FTS5 supplies a
|
||||||
|
BM25 rank, `sqlite-vec` a vector rank, and RRF merges the two.
|
||||||
|
|
||||||
|
Reply to the user in their own language.
|
||||||
|
|
||||||
|
## Pick a layer
|
||||||
|
|
||||||
|
Three layers, cheapest first. Start with the one that matches the question.
|
||||||
|
|
||||||
|
| Question shape | Command |
|
||||||
|
|---|---|
|
||||||
|
| exact string, filename, hostname, IP, identifier, code | `grep <pattern>` |
|
||||||
|
| "what do I even have about X", orientation, browsing | `toc [--source X] [--tag Y]` |
|
||||||
|
| a topic or a paraphrase, wording unknown | `search "<query>"` |
|
||||||
|
|
||||||
|
```sh
|
||||||
|
uv run skills/wiki/scripts/wiki_search.py grep "rotate_snapshots"
|
||||||
|
uv run skills/wiki/scripts/wiki_search.py toc --source travel
|
||||||
|
uv run skills/wiki/scripts/wiki_search.py search "jak snížit elektroodpad" --limit 5
|
||||||
|
```
|
||||||
|
|
||||||
|
For the full flag reference of any subcommand:
|
||||||
|
|
||||||
|
```sh
|
||||||
|
uv run skills/wiki/scripts/wiki_search.py --help
|
||||||
|
uv run skills/wiki/scripts/wiki_search.py <subcommand> --help
|
||||||
|
```
|
||||||
|
|
||||||
|
## Behavioral contract
|
||||||
|
|
||||||
|
**The index is a retrieval hint, never the answer.** `search` returns an excerpt to tell you
|
||||||
|
*which file* is relevant. Before answering, read that file fresh from disk — the index can be
|
||||||
|
up to a minute behind, and the excerpt is truncated. Never quote figures, commands or
|
||||||
|
decisions straight from a `search` excerpt.
|
||||||
|
|
||||||
|
**Only markdown is indexed; `grep` sees everything.** Source code, configs and data files never
|
||||||
|
reach `search` or `toc`. For anything code-shaped — an identifier, a flag, a function name —
|
||||||
|
`grep` is the right layer, not a fallback. Exact match is what matters there anyway.
|
||||||
|
|
||||||
|
**`grep` is a regex.** Escape regex metacharacters when the user means them literally
|
||||||
|
(`.`, `(`, `[`, `*`, `+`, `?`, `|`). It searches the files on disk, so it is always current.
|
||||||
|
|
||||||
|
**`search` results always look plausible.** The vector half has no distance floor: it returns
|
||||||
|
its nearest chunks whatever you ask, so an empty result set means an empty index, not "nothing
|
||||||
|
matches". Judge each hit against the question and say so when nothing genuinely fits — do not
|
||||||
|
present a weak nearest neighbour as an answer.
|
||||||
|
|
||||||
|
**Read the notes the output prints.** Each hit shows `source:path`, the breadcrumb, and which
|
||||||
|
half found it (`bm25 #2, vec #1`). A hit both halves rank highly is worth more than one only
|
||||||
|
the vectors found.
|
||||||
|
|
||||||
|
## Notes the output may print
|
||||||
|
|
||||||
|
- `embeddings unavailable … FTS-only results` — the semantic half is down, so only keyword
|
||||||
|
matching ran. Paraphrase queries will do badly; say so rather than concluding the notes are
|
||||||
|
silent on the topic. Try `grep` with the user's own wording.
|
||||||
|
- `N chunks still awaiting vectors` — a sync is mid-flight or was degraded. Recent notes may
|
||||||
|
be missing from the semantic half.
|
||||||
|
- `reindex needed: …` on stderr with exit 1 — the index was built under a different embedding
|
||||||
|
contract. Nothing is wrong with the notes; the index has to be rebuilt (see below). Do not
|
||||||
|
retry the query.
|
||||||
|
|
||||||
|
## Sources
|
||||||
|
|
||||||
|
Defined in `wiki/config.yaml`. Each has a stable **source id** used in output and in
|
||||||
|
`--source`. Git sources are read-only clones under `wiki/remote/<id>/`; the `workspace` source
|
||||||
|
indexes the live workspace. The user's repos are the canonical data — nothing here ever writes
|
||||||
|
to them.
|
||||||
|
|
||||||
|
`toc --tag` filters on frontmatter tags, which only exist where the user wrote them.
|
||||||
|
|
||||||
|
## Indexing
|
||||||
|
|
||||||
|
`wiki_sync.py` runs every minute from the crontab and does all the work offline. Never index
|
||||||
|
inside a turn: a full rebuild takes minutes and the `exec` tool times out at 60 s.
|
||||||
|
|
||||||
|
```sh
|
||||||
|
uv run skills/wiki/scripts/wiki_sync.py # what cron runs; a quiet tick is free
|
||||||
|
uv run skills/wiki/scripts/wiki_sync.py --full # wipe and rebuild
|
||||||
|
```
|
||||||
|
|
||||||
|
`--full` is needed only after the embedding model, its dimensions, the query prefix or the
|
||||||
|
chunker change — that is what a `reindex needed` message means. It re-embeds everything, so
|
||||||
|
run it in the background and tell the user it is running, never inside a turn where they wait.
|
||||||
|
|
||||||
|
`log/wiki_sync.log` holds one line per event: what was indexed, `WARN` for a source that was
|
||||||
|
skipped, and a `coverage` line naming top-level directories that hold markdown no source
|
||||||
|
covers. Read it when `search` cannot find something the user is sure they wrote.
|
||||||
|
|
||||||
|
## Environment
|
||||||
|
|
||||||
|
- `WIKI_DB` — override the index path (tests).
|
||||||
|
- `WIKI_CONFIG` — override the config path (tests).
|
||||||
260
skills/wiki/scripts/wiki_chunker.py
Normal file
260
skills/wiki/scripts/wiki_chunker.py
Normal file
@@ -0,0 +1,260 @@
|
|||||||
|
#!/usr/bin/env python3
|
||||||
|
# /// script
|
||||||
|
# requires-python = ">=3.11"
|
||||||
|
# dependencies = ["markdown-it-py", "pyyaml"]
|
||||||
|
# ///
|
||||||
|
"""Structure-aware markdown chunker for the wiki skill.
|
||||||
|
|
||||||
|
Splits a document along its H1-H3 heading hierarchy, then merges small siblings and
|
||||||
|
splits oversized sections along block boundaries. Every chunk carries a breadcrumb
|
||||||
|
(`path > title > section > subsection`) which goes into the embedded text as well as
|
||||||
|
the metadata, so a vector represents a passage in context rather than in isolation.
|
||||||
|
|
||||||
|
markdown-it-py supplies the AST (it knows the CommonMark edge cases); the chunking
|
||||||
|
policy below is ours. Block boundaries come from token line maps, so the emitted text
|
||||||
|
is the original markdown — code fences and tables stay byte-for-byte intact.
|
||||||
|
"""
|
||||||
|
|
||||||
|
from __future__ import annotations
|
||||||
|
|
||||||
|
import re
|
||||||
|
from dataclasses import dataclass, field
|
||||||
|
|
||||||
|
import yaml
|
||||||
|
from markdown_it import MarkdownIt # ty: ignore[unresolved-import]
|
||||||
|
|
||||||
|
CHUNKER_VERSION = "1"
|
||||||
|
|
||||||
|
MERGE_BELOW = 200
|
||||||
|
SPLIT_ABOVE = 800
|
||||||
|
OVERLAP = 64
|
||||||
|
CHARS_PER_TOKEN = 4
|
||||||
|
|
||||||
|
SECTION_LEVELS = (1, 2, 3)
|
||||||
|
BREADCRUMB_SEP = " > "
|
||||||
|
|
||||||
|
_FRONTMATTER_RE = re.compile(r"\A---[ \t]*\r?\n(.*?)\r?\n---[ \t]*(?:\r?\n|\Z)", re.DOTALL)
|
||||||
|
|
||||||
|
_md = MarkdownIt("commonmark")
|
||||||
|
|
||||||
|
|
||||||
|
@dataclass(frozen=True)
|
||||||
|
class Chunk:
|
||||||
|
breadcrumb: str
|
||||||
|
text: str
|
||||||
|
|
||||||
|
|
||||||
|
@dataclass(frozen=True)
|
||||||
|
class ParsedFile:
|
||||||
|
title: str | None = None
|
||||||
|
tags: list[str] = field(default_factory=list)
|
||||||
|
headings: list[str] = field(default_factory=list)
|
||||||
|
chunks: list[Chunk] = field(default_factory=list)
|
||||||
|
|
||||||
|
|
||||||
|
@dataclass(frozen=True)
|
||||||
|
class _Heading:
|
||||||
|
start: int
|
||||||
|
end: int
|
||||||
|
level: int
|
||||||
|
title: str
|
||||||
|
|
||||||
|
|
||||||
|
@dataclass(frozen=True)
|
||||||
|
class _Section:
|
||||||
|
breadcrumb: tuple[str, ...]
|
||||||
|
text: str
|
||||||
|
|
||||||
|
@property
|
||||||
|
def parent(self) -> tuple[str, ...]:
|
||||||
|
return self.breadcrumb[:-1]
|
||||||
|
|
||||||
|
|
||||||
|
def token_estimate(text: str) -> int:
|
||||||
|
"""Approximate token count. Sizing does not need a real tokenizer (plan: 4 chars/token)."""
|
||||||
|
return max(1, len(text) // CHARS_PER_TOKEN)
|
||||||
|
|
||||||
|
|
||||||
|
def parse_markdown(text: str, path: str) -> ParsedFile:
|
||||||
|
"""Chunk one markdown document. `path` is the source-relative path, the breadcrumb root."""
|
||||||
|
frontmatter, body = _split_frontmatter(text)
|
||||||
|
frontmatter_title = _frontmatter_title(frontmatter)
|
||||||
|
tags = _frontmatter_tags(frontmatter)
|
||||||
|
|
||||||
|
root: tuple[str, ...] = (path,) if frontmatter_title is None else (path, frontmatter_title)
|
||||||
|
sections, headings = _split_sections(body, root)
|
||||||
|
merged = _merge_small_siblings(sections)
|
||||||
|
chunks = [chunk for section in merged for chunk in _split_oversized(section)]
|
||||||
|
|
||||||
|
# The catalog title falls back to the first heading, because notes carry their title as
|
||||||
|
# `# H1` far more often than as frontmatter. It deliberately does NOT feed the breadcrumb
|
||||||
|
# root: the H1 already reaches the breadcrumb through the heading stack, and changing the
|
||||||
|
# root would rewrite every chunk's text and force a full reindex.
|
||||||
|
title = frontmatter_title or (headings[0] if headings else None)
|
||||||
|
return ParsedFile(title=title, tags=tags, headings=headings, chunks=chunks)
|
||||||
|
|
||||||
|
|
||||||
|
# ---------------------------------------------------------------------------
|
||||||
|
# frontmatter
|
||||||
|
# ---------------------------------------------------------------------------
|
||||||
|
|
||||||
|
|
||||||
|
def _split_frontmatter(text: str) -> tuple[dict, str]:
|
||||||
|
"""Peel off a leading YAML frontmatter block. Malformed frontmatter stays body text."""
|
||||||
|
match = _FRONTMATTER_RE.match(text)
|
||||||
|
if not match:
|
||||||
|
return {}, text
|
||||||
|
try:
|
||||||
|
data = yaml.safe_load(match.group(1))
|
||||||
|
except yaml.YAMLError:
|
||||||
|
return {}, text
|
||||||
|
if not isinstance(data, dict):
|
||||||
|
return {}, text
|
||||||
|
return data, text[match.end() :]
|
||||||
|
|
||||||
|
|
||||||
|
def _frontmatter_title(frontmatter: dict) -> str | None:
|
||||||
|
title = frontmatter.get("title")
|
||||||
|
if isinstance(title, str) and title.strip():
|
||||||
|
return title.strip()
|
||||||
|
return None
|
||||||
|
|
||||||
|
|
||||||
|
def _frontmatter_tags(frontmatter: dict) -> list[str]:
|
||||||
|
raw = frontmatter.get("tags")
|
||||||
|
if isinstance(raw, str):
|
||||||
|
values = raw.split(",")
|
||||||
|
elif isinstance(raw, list):
|
||||||
|
values = [str(item) for item in raw]
|
||||||
|
else:
|
||||||
|
return []
|
||||||
|
return [tag.strip() for tag in values if tag.strip()]
|
||||||
|
|
||||||
|
|
||||||
|
# ---------------------------------------------------------------------------
|
||||||
|
# sectioning
|
||||||
|
# ---------------------------------------------------------------------------
|
||||||
|
|
||||||
|
|
||||||
|
def _split_sections(body: str, root: tuple[str, ...]) -> tuple[list[_Section], list[str]]:
|
||||||
|
"""Cut the body at H1-H3 headings. H4+ stay inside their parent section."""
|
||||||
|
lines = body.split("\n")
|
||||||
|
tokens = _md.parse(body)
|
||||||
|
|
||||||
|
starts: list[_Heading] = []
|
||||||
|
for index, token in enumerate(tokens):
|
||||||
|
if token.type != "heading_open" or token.map is None:
|
||||||
|
continue
|
||||||
|
level = int(token.tag[1:])
|
||||||
|
if level not in SECTION_LEVELS:
|
||||||
|
continue
|
||||||
|
inline = tokens[index + 1] if index + 1 < len(tokens) else None
|
||||||
|
title = inline.content.strip() if inline is not None else ""
|
||||||
|
starts.append(_Heading(start=token.map[0], end=token.map[1], level=level, title=title))
|
||||||
|
|
||||||
|
headings = [heading.title for heading in starts]
|
||||||
|
boundaries = [heading.start for heading in starts] + [len(lines)]
|
||||||
|
|
||||||
|
sections: list[_Section] = []
|
||||||
|
preamble = "\n".join(lines[: boundaries[0]]).strip()
|
||||||
|
if preamble:
|
||||||
|
sections.append(_Section(root, preamble))
|
||||||
|
|
||||||
|
stack: list[tuple[int, str]] = []
|
||||||
|
for position, heading in enumerate(starts):
|
||||||
|
while stack and stack[-1][0] >= heading.level:
|
||||||
|
stack.pop()
|
||||||
|
stack.append((heading.level, heading.title))
|
||||||
|
section_end = boundaries[position + 1]
|
||||||
|
# A heading with no prose of its own would embed as a bare title; the heading
|
||||||
|
# still reaches the index through its children's breadcrumbs, so drop it.
|
||||||
|
if not "\n".join(lines[heading.end : section_end]).strip():
|
||||||
|
continue
|
||||||
|
text = "\n".join(lines[heading.start : section_end]).strip()
|
||||||
|
sections.append(_Section(_dedupe(root + tuple(title for _, title in stack)), text))
|
||||||
|
return sections, headings
|
||||||
|
|
||||||
|
|
||||||
|
def _dedupe(parts: tuple[str, ...]) -> tuple[str, ...]:
|
||||||
|
"""Drop consecutive repeats so a frontmatter title matching the H1 shows up once."""
|
||||||
|
out: list[str] = []
|
||||||
|
for part in parts:
|
||||||
|
if not out or out[-1] != part:
|
||||||
|
out.append(part)
|
||||||
|
return tuple(out)
|
||||||
|
|
||||||
|
|
||||||
|
def _merge_small_siblings(sections: list[_Section]) -> list[_Section]:
|
||||||
|
"""Glue a too-small section onto the following sibling under the same parent."""
|
||||||
|
merged: list[_Section] = []
|
||||||
|
for section in sections:
|
||||||
|
if not merged:
|
||||||
|
merged.append(section)
|
||||||
|
continue
|
||||||
|
previous = merged[-1]
|
||||||
|
combined = f"{previous.text}\n\n{section.text}"
|
||||||
|
if (
|
||||||
|
token_estimate(previous.text) < MERGE_BELOW
|
||||||
|
and previous.parent == section.parent
|
||||||
|
and token_estimate(combined) <= SPLIT_ABOVE
|
||||||
|
):
|
||||||
|
merged[-1] = _Section(previous.breadcrumb, combined)
|
||||||
|
else:
|
||||||
|
merged.append(section)
|
||||||
|
return merged
|
||||||
|
|
||||||
|
|
||||||
|
# ---------------------------------------------------------------------------
|
||||||
|
# splitting
|
||||||
|
# ---------------------------------------------------------------------------
|
||||||
|
|
||||||
|
|
||||||
|
def _split_oversized(section: _Section) -> list[Chunk]:
|
||||||
|
"""Break a section over SPLIT_ABOVE into block-aligned pieces with OVERLAP carry-over."""
|
||||||
|
breadcrumb = BREADCRUMB_SEP.join(section.breadcrumb)
|
||||||
|
if token_estimate(section.text) <= SPLIT_ABOVE:
|
||||||
|
return [Chunk(breadcrumb, _embed_text(breadcrumb, section.text))]
|
||||||
|
|
||||||
|
blocks = _top_level_blocks(section.text)
|
||||||
|
chunks: list[Chunk] = []
|
||||||
|
current: list[str] = []
|
||||||
|
for block in blocks:
|
||||||
|
candidate = [*current, block]
|
||||||
|
if current and token_estimate("\n\n".join(candidate)) > SPLIT_ABOVE:
|
||||||
|
chunks.append(Chunk(breadcrumb, _embed_text(breadcrumb, "\n\n".join(current))))
|
||||||
|
current = [*_overlap_tail(current), block]
|
||||||
|
else:
|
||||||
|
current = candidate
|
||||||
|
if current:
|
||||||
|
chunks.append(Chunk(breadcrumb, _embed_text(breadcrumb, "\n\n".join(current))))
|
||||||
|
return chunks
|
||||||
|
|
||||||
|
|
||||||
|
def _top_level_blocks(text: str) -> list[str]:
|
||||||
|
"""Top-level markdown blocks, taken from token line maps so fences stay whole."""
|
||||||
|
lines = text.split("\n")
|
||||||
|
tokens = _md.parse(text)
|
||||||
|
starts = sorted({token.map[0] for token in tokens if token.level == 0 and token.map})
|
||||||
|
if not starts:
|
||||||
|
return [text]
|
||||||
|
bounds = [*starts, len(lines)]
|
||||||
|
blocks = ["\n".join(lines[bounds[i] : bounds[i + 1]]).strip() for i in range(len(starts))]
|
||||||
|
return [block for block in blocks if block]
|
||||||
|
|
||||||
|
|
||||||
|
def _overlap_tail(blocks: list[str]) -> list[str]:
|
||||||
|
"""Trailing whole blocks of the emitted chunk, up to OVERLAP tokens."""
|
||||||
|
tail: list[str] = []
|
||||||
|
budget = OVERLAP
|
||||||
|
for block in reversed(blocks):
|
||||||
|
cost = token_estimate(block)
|
||||||
|
if cost > budget:
|
||||||
|
break
|
||||||
|
tail.insert(0, block)
|
||||||
|
budget -= cost
|
||||||
|
return tail
|
||||||
|
|
||||||
|
|
||||||
|
def _embed_text(breadcrumb: str, text: str) -> str:
|
||||||
|
"""The stored chunk text carries its breadcrumb, so the vector sees the context."""
|
||||||
|
return f"{breadcrumb}\n\n{text}"
|
||||||
225
skills/wiki/scripts/wiki_config.py
Normal file
225
skills/wiki/scripts/wiki_config.py
Normal file
@@ -0,0 +1,225 @@
|
|||||||
|
#!/usr/bin/env python3
|
||||||
|
# /// script
|
||||||
|
# requires-python = ">=3.11"
|
||||||
|
# dependencies = ["pyyaml"]
|
||||||
|
# ///
|
||||||
|
"""Layout and configuration for the wiki skill.
|
||||||
|
|
||||||
|
All runtime data lives under `workspace/wiki/`; only `config.yaml` is versioned.
|
||||||
|
|
||||||
|
The source id is the stable key — the catalog and the vectors hang off it, while a
|
||||||
|
URL or a path may change. Renaming an id is therefore an explicit invalidation of
|
||||||
|
that source's index, not a rename.
|
||||||
|
|
||||||
|
Scope precedence is `paths` (whitelist — what is not in it does not exist for the
|
||||||
|
index) then `include` (extension whitelist) then `exclude` (scalpel, wins over both).
|
||||||
|
"""
|
||||||
|
|
||||||
|
from __future__ import annotations
|
||||||
|
|
||||||
|
import os
|
||||||
|
import re
|
||||||
|
from dataclasses import dataclass
|
||||||
|
from pathlib import Path
|
||||||
|
|
||||||
|
import yaml
|
||||||
|
|
||||||
|
# workspace/skills/wiki/scripts/wiki_config.py -> parents[3] = workspace root.
|
||||||
|
WORKSPACE = Path(__file__).resolve().parents[3]
|
||||||
|
WIKI_DIR = WORKSPACE / "wiki"
|
||||||
|
DEFAULT_DB_PATH = WIKI_DIR / "index.sqlite"
|
||||||
|
DEFAULT_CONFIG_PATH = WIKI_DIR / "config.yaml"
|
||||||
|
REMOTE_DIR = WIKI_DIR / "remote"
|
||||||
|
LOCK_PATH = WIKI_DIR / ".sync.lock"
|
||||||
|
SYNC_LOG_PATH = WORKSPACE / "log" / "wiki_sync.log"
|
||||||
|
|
||||||
|
GIT_KIND = "git"
|
||||||
|
WORKSPACE_KIND = "workspace"
|
||||||
|
VALID_KINDS = (GIT_KIND, WORKSPACE_KIND)
|
||||||
|
|
||||||
|
|
||||||
|
def db_path() -> Path:
|
||||||
|
"""Index location, overridable with WIKI_DB for tests."""
|
||||||
|
return Path(os.environ.get("WIKI_DB", str(DEFAULT_DB_PATH)))
|
||||||
|
|
||||||
|
|
||||||
|
def config_path() -> Path:
|
||||||
|
"""Config location, overridable with WIKI_CONFIG for tests."""
|
||||||
|
return Path(os.environ.get("WIKI_CONFIG", str(DEFAULT_CONFIG_PATH)))
|
||||||
|
|
||||||
|
|
||||||
|
class ConfigError(ValueError):
|
||||||
|
"""Malformed or incomplete wiki/config.yaml."""
|
||||||
|
|
||||||
|
|
||||||
|
@dataclass(frozen=True)
|
||||||
|
class EmbeddingConfig:
|
||||||
|
endpoint: str
|
||||||
|
model: str
|
||||||
|
dims: int
|
||||||
|
batch: int
|
||||||
|
keep_alive: int
|
||||||
|
query_prefix: str
|
||||||
|
|
||||||
|
|
||||||
|
@dataclass(frozen=True)
|
||||||
|
class SourceConfig:
|
||||||
|
source_id: str
|
||||||
|
kind: str
|
||||||
|
url: str | None
|
||||||
|
paths: tuple[str, ...]
|
||||||
|
include: tuple[str, ...]
|
||||||
|
exclude: tuple[str, ...]
|
||||||
|
|
||||||
|
def covers(self, rel_path: str) -> bool:
|
||||||
|
"""True when a source-relative path belongs in the index."""
|
||||||
|
if not any(_matches(pattern, rel_path) for pattern in self.paths):
|
||||||
|
return False
|
||||||
|
name = rel_path.rsplit("/", 1)[-1]
|
||||||
|
if not any(_matches(pattern, name) for pattern in self.include):
|
||||||
|
return False
|
||||||
|
return not any(_matches(pattern, rel_path) for pattern in self.exclude)
|
||||||
|
|
||||||
|
def covers_dir(self, rel_dir: str) -> bool:
|
||||||
|
"""Cheap walk prune: could anything under this directory ever be covered?"""
|
||||||
|
if not rel_dir:
|
||||||
|
return True
|
||||||
|
probe = f"{rel_dir}/"
|
||||||
|
if any(_matches(pattern, probe) for pattern in self.exclude):
|
||||||
|
return False
|
||||||
|
return any(_prefix_could_match(pattern, probe) for pattern in self.paths)
|
||||||
|
|
||||||
|
|
||||||
|
@dataclass(frozen=True)
|
||||||
|
class WikiConfig:
|
||||||
|
embedding: EmbeddingConfig
|
||||||
|
sources: tuple[SourceConfig, ...]
|
||||||
|
|
||||||
|
def source(self, source_id: str) -> SourceConfig | None:
|
||||||
|
return next((s for s in self.sources if s.source_id == source_id), None)
|
||||||
|
|
||||||
|
|
||||||
|
def source_root(source: SourceConfig) -> Path:
|
||||||
|
"""Where a source's files live. Git clones are derived from the id, never configured."""
|
||||||
|
if source.kind == GIT_KIND:
|
||||||
|
return REMOTE_DIR / source.source_id
|
||||||
|
return WORKSPACE
|
||||||
|
|
||||||
|
|
||||||
|
def load_config(path: Path | None = None) -> WikiConfig:
|
||||||
|
path = path or config_path()
|
||||||
|
if not path.exists():
|
||||||
|
raise ConfigError(f"missing config: {path}")
|
||||||
|
try:
|
||||||
|
raw = yaml.safe_load(path.read_text(encoding="utf-8"))
|
||||||
|
except yaml.YAMLError as exc:
|
||||||
|
raise ConfigError(f"unparseable config {path}: {exc}") from exc
|
||||||
|
if not isinstance(raw, dict):
|
||||||
|
raise ConfigError(f"config {path} must be a mapping")
|
||||||
|
return WikiConfig(
|
||||||
|
embedding=_parse_embedding(raw.get("embedding")),
|
||||||
|
sources=_parse_sources(raw.get("sources")),
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
|
def _as_mapping(raw: object, what: str) -> dict[str, object]:
|
||||||
|
if not isinstance(raw, dict):
|
||||||
|
raise ConfigError(f"{what} must be a mapping")
|
||||||
|
return {str(key): value for key, value in raw.items()}
|
||||||
|
|
||||||
|
|
||||||
|
def _parse_embedding(raw: object) -> EmbeddingConfig:
|
||||||
|
values = _as_mapping(raw, "`embedding`")
|
||||||
|
missing = [key for key in ("endpoint", "model", "dims") if key not in values]
|
||||||
|
if missing:
|
||||||
|
raise ConfigError(f"embedding is missing {', '.join(missing)}")
|
||||||
|
keep_alive = values.get("keep_alive", -1)
|
||||||
|
if not isinstance(keep_alive, int):
|
||||||
|
# Ollama rejects a string keep_alive of "-1" with HTTP 400.
|
||||||
|
raise ConfigError("embedding.keep_alive must be a number, not a string")
|
||||||
|
return EmbeddingConfig(
|
||||||
|
endpoint=str(values["endpoint"]).rstrip("/"),
|
||||||
|
model=str(values["model"]),
|
||||||
|
dims=int(str(values["dims"])),
|
||||||
|
batch=int(str(values.get("batch", 32))),
|
||||||
|
keep_alive=keep_alive,
|
||||||
|
query_prefix=str(values.get("query_prefix", "")),
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
|
def _parse_sources(raw: object) -> tuple[SourceConfig, ...]:
|
||||||
|
entries = _as_mapping(raw, "`sources`")
|
||||||
|
if not entries:
|
||||||
|
raise ConfigError("config needs a non-empty `sources` mapping")
|
||||||
|
sources = []
|
||||||
|
for source_id, raw_body in entries.items():
|
||||||
|
body = _as_mapping(raw_body, f"source {source_id}")
|
||||||
|
kind = str(body.get("kind"))
|
||||||
|
if kind not in VALID_KINDS:
|
||||||
|
raise ConfigError(f"source {source_id}: kind must be one of {VALID_KINDS}, got {kind!r}")
|
||||||
|
url = body.get("url")
|
||||||
|
if kind == GIT_KIND and not url:
|
||||||
|
raise ConfigError(f"source {source_id}: git sources need a url")
|
||||||
|
sources.append(
|
||||||
|
SourceConfig(
|
||||||
|
source_id=source_id,
|
||||||
|
kind=kind,
|
||||||
|
url=str(url) if url else None,
|
||||||
|
paths=_as_patterns(body.get("paths"), source_id, "paths"),
|
||||||
|
include=_as_patterns(body.get("include") or ["*.md"], source_id, "include"),
|
||||||
|
exclude=_as_patterns(body.get("exclude") or [], source_id, "exclude"),
|
||||||
|
)
|
||||||
|
)
|
||||||
|
return tuple(sources)
|
||||||
|
|
||||||
|
|
||||||
|
def _as_patterns(raw: object, source_id: str, key: str) -> tuple[str, ...]:
|
||||||
|
if raw is None:
|
||||||
|
raise ConfigError(f"source {source_id}: `{key}` is required")
|
||||||
|
if not isinstance(raw, list):
|
||||||
|
raise ConfigError(f"source {source_id}: `{key}` must be a list")
|
||||||
|
return tuple(str(item) for item in raw)
|
||||||
|
|
||||||
|
|
||||||
|
# ---------------------------------------------------------------------------
|
||||||
|
# glob matching
|
||||||
|
#
|
||||||
|
# fnmatch lets `*` cross a `/` and PurePath.match has no recursive `**` before
|
||||||
|
# Python 3.13, so the patterns are translated by hand.
|
||||||
|
# ---------------------------------------------------------------------------
|
||||||
|
|
||||||
|
_SEGMENT_ANY = "[^/]*"
|
||||||
|
|
||||||
|
|
||||||
|
def _glob_to_regex(pattern: str) -> str:
|
||||||
|
parts = pattern.split("/")
|
||||||
|
out = []
|
||||||
|
for index, part in enumerate(parts):
|
||||||
|
is_last = index == len(parts) - 1
|
||||||
|
if part == "**":
|
||||||
|
out.append(".*" if is_last else "(?:[^/]+/)*")
|
||||||
|
else:
|
||||||
|
segment = re.escape(part).replace(r"\*", _SEGMENT_ANY).replace(r"\?", "[^/]")
|
||||||
|
out.append(segment if is_last else segment + "/")
|
||||||
|
return "^" + "".join(out) + "$"
|
||||||
|
|
||||||
|
|
||||||
|
def _matches(pattern: str, value: str) -> bool:
|
||||||
|
return re.match(_glob_to_regex(pattern), value) is not None
|
||||||
|
|
||||||
|
|
||||||
|
def _prefix_could_match(pattern: str, directory: str) -> bool:
|
||||||
|
"""True when `pattern` can still match something below `directory`."""
|
||||||
|
if pattern.startswith("**"):
|
||||||
|
return True
|
||||||
|
pattern_parts = pattern.split("/")
|
||||||
|
dir_parts = [part for part in directory.split("/") if part]
|
||||||
|
for depth, dir_part in enumerate(dir_parts):
|
||||||
|
if depth >= len(pattern_parts):
|
||||||
|
return False
|
||||||
|
pattern_part = pattern_parts[depth]
|
||||||
|
if pattern_part == "**":
|
||||||
|
return True
|
||||||
|
if not _matches(pattern_part, dir_part):
|
||||||
|
return False
|
||||||
|
return True
|
||||||
129
skills/wiki/scripts/wiki_db.py
Normal file
129
skills/wiki/scripts/wiki_db.py
Normal file
@@ -0,0 +1,129 @@
|
|||||||
|
#!/usr/bin/env python3
|
||||||
|
# /// script
|
||||||
|
# requires-python = ">=3.11"
|
||||||
|
# dependencies = ["sqlite-vec==0.1.6"]
|
||||||
|
# ///
|
||||||
|
"""SQLite storage layer for the wiki skill.
|
||||||
|
|
||||||
|
Schema, connection factory and the sqlite-vec extension load.
|
||||||
|
|
||||||
|
`chunks` is the canonical retrieval unit: `chunks_fts` gives it a BM25 rank and
|
||||||
|
`vec_chunks` a KNN rank, both keyed on `chunks.id`, so RRF merges two rankings of
|
||||||
|
the *same* set.
|
||||||
|
"""
|
||||||
|
|
||||||
|
from __future__ import annotations
|
||||||
|
|
||||||
|
import sqlite3
|
||||||
|
from pathlib import Path
|
||||||
|
|
||||||
|
import sqlite_vec # ty: ignore[unresolved-import]
|
||||||
|
|
||||||
|
# Compiled into the vec0 table definition. `wiki_embed` guards config.dims against it —
|
||||||
|
# a mismatch means the index was built for a different model.
|
||||||
|
EMBEDDING_DIMS = 1024
|
||||||
|
|
||||||
|
SCHEMA = f"""
|
||||||
|
PRAGMA journal_mode = WAL;
|
||||||
|
PRAGMA foreign_keys = ON;
|
||||||
|
|
||||||
|
CREATE TABLE IF NOT EXISTS meta (
|
||||||
|
key TEXT PRIMARY KEY,
|
||||||
|
value TEXT NOT NULL
|
||||||
|
);
|
||||||
|
|
||||||
|
CREATE TABLE IF NOT EXISTS sources (
|
||||||
|
source_id TEXT PRIMARY KEY,
|
||||||
|
kind TEXT NOT NULL CHECK(kind IN ('git', 'workspace')),
|
||||||
|
indexed_rev TEXT,
|
||||||
|
last_sync_at TEXT
|
||||||
|
);
|
||||||
|
|
||||||
|
CREATE TABLE IF NOT EXISTS files (
|
||||||
|
source_id TEXT NOT NULL REFERENCES sources(source_id),
|
||||||
|
path TEXT NOT NULL,
|
||||||
|
title TEXT,
|
||||||
|
tags TEXT,
|
||||||
|
headings TEXT,
|
||||||
|
sha256 TEXT NOT NULL,
|
||||||
|
size INTEGER NOT NULL,
|
||||||
|
mtime REAL,
|
||||||
|
indexed_at TEXT NOT NULL,
|
||||||
|
PRIMARY KEY (source_id, path)
|
||||||
|
);
|
||||||
|
|
||||||
|
CREATE TABLE IF NOT EXISTS chunks (
|
||||||
|
id INTEGER PRIMARY KEY AUTOINCREMENT,
|
||||||
|
source_id TEXT NOT NULL,
|
||||||
|
path TEXT NOT NULL,
|
||||||
|
chunk_idx INTEGER NOT NULL,
|
||||||
|
breadcrumb TEXT NOT NULL,
|
||||||
|
text TEXT NOT NULL,
|
||||||
|
embedded_at TEXT,
|
||||||
|
UNIQUE (source_id, path, chunk_idx),
|
||||||
|
FOREIGN KEY (source_id, path) REFERENCES files(source_id, path) ON DELETE CASCADE
|
||||||
|
);
|
||||||
|
|
||||||
|
CREATE INDEX IF NOT EXISTS idx_chunks_pending ON chunks(id) WHERE embedded_at IS NULL;
|
||||||
|
|
||||||
|
CREATE VIRTUAL TABLE IF NOT EXISTS chunks_fts USING fts5(
|
||||||
|
breadcrumb, text,
|
||||||
|
content='chunks', content_rowid='id',
|
||||||
|
tokenize='unicode61 remove_diacritics 2'
|
||||||
|
);
|
||||||
|
|
||||||
|
CREATE TRIGGER IF NOT EXISTS chunks_ai AFTER INSERT ON chunks BEGIN
|
||||||
|
INSERT INTO chunks_fts(rowid, breadcrumb, text) VALUES (new.id, new.breadcrumb, new.text);
|
||||||
|
END;
|
||||||
|
|
||||||
|
CREATE TRIGGER IF NOT EXISTS chunks_ad AFTER DELETE ON chunks BEGIN
|
||||||
|
INSERT INTO chunks_fts(chunks_fts, rowid, breadcrumb, text)
|
||||||
|
VALUES('delete', old.id, old.breadcrumb, old.text);
|
||||||
|
END;
|
||||||
|
|
||||||
|
-- The WHEN guard keeps `UPDATE chunks SET embedded_at` (sync step 4b) from rewriting
|
||||||
|
-- an FTS row whose indexed text did not change.
|
||||||
|
CREATE TRIGGER IF NOT EXISTS chunks_au AFTER UPDATE ON chunks
|
||||||
|
WHEN old.text IS NOT new.text OR old.breadcrumb IS NOT new.breadcrumb
|
||||||
|
BEGIN
|
||||||
|
INSERT INTO chunks_fts(chunks_fts, rowid, breadcrumb, text)
|
||||||
|
VALUES('delete', old.id, old.breadcrumb, old.text);
|
||||||
|
INSERT INTO chunks_fts(rowid, breadcrumb, text) VALUES (new.id, new.breadcrumb, new.text);
|
||||||
|
END;
|
||||||
|
|
||||||
|
CREATE VIRTUAL TABLE IF NOT EXISTS vec_chunks USING vec0(
|
||||||
|
embedding float[{EMBEDDING_DIMS}] distance_metric=cosine
|
||||||
|
);
|
||||||
|
"""
|
||||||
|
|
||||||
|
|
||||||
|
def get_db(path: Path) -> sqlite3.Connection:
|
||||||
|
"""Return an autocommit connection with sqlite-vec loaded and foreign keys on."""
|
||||||
|
conn = sqlite3.connect(path, isolation_level=None)
|
||||||
|
conn.enable_load_extension(True)
|
||||||
|
sqlite_vec.load(conn)
|
||||||
|
conn.enable_load_extension(False)
|
||||||
|
conn.execute("PRAGMA journal_mode = WAL")
|
||||||
|
conn.execute("PRAGMA foreign_keys = ON")
|
||||||
|
conn.row_factory = sqlite3.Row
|
||||||
|
_migrate(conn)
|
||||||
|
return conn
|
||||||
|
|
||||||
|
|
||||||
|
def _migrate(conn: sqlite3.Connection) -> None:
|
||||||
|
"""Idempotently bring an existing DB up to the current schema.
|
||||||
|
|
||||||
|
init_db only runs the full SCHEMA on a missing file, so live DBs never see
|
||||||
|
later additions. Each step must be a no-op once applied.
|
||||||
|
"""
|
||||||
|
# No migrations yet — schema_version 1 is the initial shape.
|
||||||
|
|
||||||
|
|
||||||
|
def init_db(path: Path) -> None:
|
||||||
|
"""Create tables, indexes and triggers if they don't exist."""
|
||||||
|
path.parent.mkdir(parents=True, exist_ok=True)
|
||||||
|
conn = get_db(path)
|
||||||
|
try:
|
||||||
|
conn.executescript(SCHEMA)
|
||||||
|
finally:
|
||||||
|
conn.close()
|
||||||
127
skills/wiki/scripts/wiki_embed.py
Normal file
127
skills/wiki/scripts/wiki_embed.py
Normal file
@@ -0,0 +1,127 @@
|
|||||||
|
#!/usr/bin/env python3
|
||||||
|
# /// script
|
||||||
|
# requires-python = ">=3.11"
|
||||||
|
# dependencies = ["requests", "pyyaml", "markdown-it-py", "sqlite-vec==0.1.6"]
|
||||||
|
# ///
|
||||||
|
"""Embedding client and the identity guard over the vector space.
|
||||||
|
|
||||||
|
Two invariants live here:
|
||||||
|
|
||||||
|
* Vectors are stored L2-normalized, so cosine distance equals the dot product.
|
||||||
|
* The query prefix must be bit-identical at index and at query time. That is the real
|
||||||
|
reason it is persisted in `meta` rather than only read from the config — mixing
|
||||||
|
vectors produced under two different contracts degrades results silently, which is
|
||||||
|
the most expensive kind of bug.
|
||||||
|
|
||||||
|
An unreachable Ollama is not an error here: the caller stores chunks with
|
||||||
|
`embedded_at IS NULL` and the query side says out loud that it is FTS-only.
|
||||||
|
"""
|
||||||
|
|
||||||
|
from __future__ import annotations
|
||||||
|
|
||||||
|
import math
|
||||||
|
from collections.abc import Sequence
|
||||||
|
from dataclasses import dataclass
|
||||||
|
|
||||||
|
import requests
|
||||||
|
from wiki_chunker import CHUNKER_VERSION
|
||||||
|
from wiki_config import EmbeddingConfig
|
||||||
|
from wiki_db import EMBEDDING_DIMS
|
||||||
|
|
||||||
|
SCHEMA_VERSION = "1"
|
||||||
|
SQLITE_VEC_VERSION = "0.1.6"
|
||||||
|
REQUEST_TIMEOUT_SECONDS = 60
|
||||||
|
|
||||||
|
# Only these keys make two vectors comparable; the rest of `meta` is informational.
|
||||||
|
GUARDED_META_KEYS = (
|
||||||
|
"embedding_model",
|
||||||
|
"embedding_dims",
|
||||||
|
"normalized",
|
||||||
|
"query_prefix",
|
||||||
|
"chunker_version",
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
|
class EmbeddingUnavailable(RuntimeError):
|
||||||
|
"""Ollama could not be reached or refused the request."""
|
||||||
|
|
||||||
|
|
||||||
|
class IndexIdentityMismatch(RuntimeError):
|
||||||
|
"""The index was built under a different embedding contract — a reindex is needed."""
|
||||||
|
|
||||||
|
|
||||||
|
def expected_meta(config: EmbeddingConfig) -> dict[str, str]:
|
||||||
|
return {
|
||||||
|
"embedding_model": config.model,
|
||||||
|
"embedding_dims": str(config.dims),
|
||||||
|
"normalized": "l2",
|
||||||
|
"query_prefix": config.query_prefix,
|
||||||
|
"chunker_version": CHUNKER_VERSION,
|
||||||
|
"schema_version": SCHEMA_VERSION,
|
||||||
|
"sqlite_vec_version": SQLITE_VEC_VERSION,
|
||||||
|
}
|
||||||
|
|
||||||
|
|
||||||
|
def meta_mismatches(stored: dict[str, str], config: EmbeddingConfig) -> list[str]:
|
||||||
|
"""Guarded meta keys that disagree with the config. Empty on a fresh (unwritten) index."""
|
||||||
|
if not stored:
|
||||||
|
return []
|
||||||
|
expected = expected_meta(config)
|
||||||
|
return [key for key in GUARDED_META_KEYS if stored.get(key) != expected[key]]
|
||||||
|
|
||||||
|
|
||||||
|
def require_matching_index(stored: dict[str, str], config: EmbeddingConfig) -> None:
|
||||||
|
"""Refuse to query an index built under a different contract."""
|
||||||
|
if config.dims != EMBEDDING_DIMS:
|
||||||
|
raise IndexIdentityMismatch(f"reindex needed: config dims {config.dims} != schema dims {EMBEDDING_DIMS}")
|
||||||
|
mismatches = meta_mismatches(stored, config)
|
||||||
|
if mismatches:
|
||||||
|
detail = ", ".join(
|
||||||
|
f"{key}: index={stored.get(key)!r} config={expected_meta(config)[key]!r}" for key in mismatches
|
||||||
|
)
|
||||||
|
raise IndexIdentityMismatch(f"reindex needed: {detail}")
|
||||||
|
|
||||||
|
|
||||||
|
def l2_normalize(vector: Sequence[float]) -> list[float]:
|
||||||
|
norm = math.sqrt(sum(value * value for value in vector))
|
||||||
|
if norm == 0.0:
|
||||||
|
return list(vector)
|
||||||
|
return [value / norm for value in vector]
|
||||||
|
|
||||||
|
|
||||||
|
@dataclass(frozen=True)
|
||||||
|
class OllamaEmbedder:
|
||||||
|
config: EmbeddingConfig
|
||||||
|
|
||||||
|
def embed_documents(self, texts: Sequence[str]) -> list[list[float]]:
|
||||||
|
"""Documents are embedded without the instruct prefix (the model is asymmetric)."""
|
||||||
|
return self._embed(list(texts))
|
||||||
|
|
||||||
|
def embed_query(self, text: str) -> list[float]:
|
||||||
|
return self._embed([self.config.query_prefix + text])[0]
|
||||||
|
|
||||||
|
def probe(self) -> None:
|
||||||
|
"""Raise EmbeddingUnavailable unless the endpoint answers an embed request."""
|
||||||
|
self._embed(["ping"])
|
||||||
|
|
||||||
|
def _embed(self, inputs: list[str]) -> list[list[float]]:
|
||||||
|
if not inputs:
|
||||||
|
return []
|
||||||
|
payload = {
|
||||||
|
"model": self.config.model,
|
||||||
|
"input": inputs,
|
||||||
|
"keep_alive": self.config.keep_alive,
|
||||||
|
}
|
||||||
|
try:
|
||||||
|
response = requests.post(f"{self.config.endpoint}/api/embed", json=payload, timeout=REQUEST_TIMEOUT_SECONDS)
|
||||||
|
response.raise_for_status()
|
||||||
|
embeddings = response.json()["embeddings"]
|
||||||
|
except (requests.RequestException, KeyError, ValueError) as exc:
|
||||||
|
raise EmbeddingUnavailable(f"{self.config.endpoint}: {exc}") from exc
|
||||||
|
|
||||||
|
if len(embeddings) != len(inputs):
|
||||||
|
raise EmbeddingUnavailable(f"asked for {len(inputs)} vectors, got {len(embeddings)}")
|
||||||
|
for vector in embeddings:
|
||||||
|
if len(vector) != self.config.dims:
|
||||||
|
raise EmbeddingUnavailable(f"model returned {len(vector)} dims, config says {self.config.dims}")
|
||||||
|
return [l2_normalize(vector) for vector in embeddings]
|
||||||
308
skills/wiki/scripts/wiki_search.py
Normal file
308
skills/wiki/scripts/wiki_search.py
Normal file
@@ -0,0 +1,308 @@
|
|||||||
|
#!/usr/bin/env -S uv run --script
|
||||||
|
# /// script
|
||||||
|
# requires-python = ">=3.11"
|
||||||
|
# dependencies = ["requests", "pyyaml", "markdown-it-py", "sqlite-vec==0.1.6"]
|
||||||
|
# ///
|
||||||
|
"""Query side of the wiki skill — three layers, cheapest first.
|
||||||
|
|
||||||
|
grep live ripgrep over the files on disk. No index, never stale, and not limited
|
||||||
|
to the indexed extensions, so it is the layer that covers source code.
|
||||||
|
toc directory -> file -> title + tags, read straight from the `files` catalog.
|
||||||
|
search FTS5 (BM25) and vec0 (KNN) over the same `chunks` rows, merged with RRF.
|
||||||
|
|
||||||
|
Both halves rank the same unit, which is what makes the merge meaningful. The
|
||||||
|
parameters below are module constants on purpose: `--limit` is the only knob worth
|
||||||
|
exposing, and a config key that never changes is a config key that rots.
|
||||||
|
"""
|
||||||
|
|
||||||
|
from __future__ import annotations
|
||||||
|
|
||||||
|
import argparse
|
||||||
|
import json
|
||||||
|
import shutil
|
||||||
|
import sqlite3
|
||||||
|
import subprocess
|
||||||
|
import sys
|
||||||
|
from pathlib import Path
|
||||||
|
|
||||||
|
import wiki_store as store
|
||||||
|
from wiki_config import (
|
||||||
|
GIT_KIND,
|
||||||
|
ConfigError,
|
||||||
|
SourceConfig,
|
||||||
|
WikiConfig,
|
||||||
|
db_path,
|
||||||
|
load_config,
|
||||||
|
source_root,
|
||||||
|
)
|
||||||
|
from wiki_embed import (
|
||||||
|
EmbeddingUnavailable,
|
||||||
|
IndexIdentityMismatch,
|
||||||
|
OllamaEmbedder,
|
||||||
|
require_matching_index,
|
||||||
|
)
|
||||||
|
|
||||||
|
CANDIDATE_LIMIT = 50 # KNN k, and the LIMIT for BM25
|
||||||
|
RRF_K = 60 # Cormack et al. 2009
|
||||||
|
DEFAULT_LIMIT = 10
|
||||||
|
PREFIX_MIN_LENGTH = 3 # a one- or two-character prefix matches too widely to carry signal
|
||||||
|
EXCERPT_CHARS = 320
|
||||||
|
GREP_TIMEOUT = 30
|
||||||
|
GREP_MAX_PER_FILE = 5
|
||||||
|
|
||||||
|
|
||||||
|
def rrf_merge(ranked_lists: list[list[int]]) -> list[tuple[int, float]]:
|
||||||
|
"""score(id) = sum over lists of 1 / (RRF_K + rank). No weights: both halves count equally."""
|
||||||
|
scores: dict[int, float] = {}
|
||||||
|
for ids in ranked_lists:
|
||||||
|
for rank, chunk_id in enumerate(ids, start=1):
|
||||||
|
scores[chunk_id] = scores.get(chunk_id, 0.0) + 1.0 / (RRF_K + rank)
|
||||||
|
return sorted(scores.items(), key=lambda item: (-item[1], item[0]))
|
||||||
|
|
||||||
|
|
||||||
|
def fts_match_expression(query: str) -> str:
|
||||||
|
"""Turn free text into an FTS5 OR-query of quoted prefix terms.
|
||||||
|
|
||||||
|
Terms are quoted so reserved words (`and`, `not`, `near`) and punctuation cannot be
|
||||||
|
read as operators. The trailing `*` covers Czech inflection, which `unicode61` does
|
||||||
|
not stem — `záloh*` finds záloha/zálohování/zálohy.
|
||||||
|
"""
|
||||||
|
terms = []
|
||||||
|
for word in _tokenize(query):
|
||||||
|
wildcard = "*" if len(word) >= PREFIX_MIN_LENGTH else ""
|
||||||
|
terms.append(f'"{word}"{wildcard}')
|
||||||
|
return " OR ".join(terms)
|
||||||
|
|
||||||
|
|
||||||
|
def _tokenize(text: str) -> list[str]:
|
||||||
|
return "".join(char if char.isalnum() else " " for char in text).split()
|
||||||
|
|
||||||
|
|
||||||
|
# ---------------------------------------------------------------------------
|
||||||
|
# search
|
||||||
|
# ---------------------------------------------------------------------------
|
||||||
|
|
||||||
|
|
||||||
|
def run_search(config: WikiConfig, query: str, limit: int) -> int:
|
||||||
|
expression = fts_match_expression(query)
|
||||||
|
if not expression:
|
||||||
|
print("(empty query)")
|
||||||
|
return 0
|
||||||
|
|
||||||
|
with store.connection(db_path()) as conn:
|
||||||
|
try:
|
||||||
|
require_matching_index(store.read_meta(conn), config.embedding)
|
||||||
|
except IndexIdentityMismatch as exc:
|
||||||
|
print(str(exc), file=sys.stderr)
|
||||||
|
return 1
|
||||||
|
|
||||||
|
bm25_ids = store.bm25_ranked_ids(conn, expression, CANDIDATE_LIMIT)
|
||||||
|
|
||||||
|
vector_ids: list[int] = []
|
||||||
|
degraded: str | None = None
|
||||||
|
try:
|
||||||
|
vector = OllamaEmbedder(config.embedding).embed_query(query)
|
||||||
|
vector_ids = store.knn_ranked_ids(conn, vector, CANDIDATE_LIMIT)
|
||||||
|
except EmbeddingUnavailable as exc:
|
||||||
|
degraded = f"note: embeddings unavailable ({exc}) — FTS-only results"
|
||||||
|
|
||||||
|
pending = store.index_stats(conn)["pending"]
|
||||||
|
merged = rrf_merge([ids for ids in (bm25_ids, vector_ids) if ids])[:limit]
|
||||||
|
rows = store.fetch_chunks(conn, [chunk_id for chunk_id, _ in merged])
|
||||||
|
|
||||||
|
if degraded:
|
||||||
|
print(degraded)
|
||||||
|
elif pending:
|
||||||
|
print(f"note: {pending} chunks still awaiting vectors — semantic half is incomplete")
|
||||||
|
|
||||||
|
if not merged:
|
||||||
|
print("(no matches)")
|
||||||
|
return 0
|
||||||
|
|
||||||
|
bm25_rank = {chunk_id: rank for rank, chunk_id in enumerate(bm25_ids, start=1)}
|
||||||
|
vector_rank = {chunk_id: rank for rank, chunk_id in enumerate(vector_ids, start=1)}
|
||||||
|
for position, (chunk_id, score) in enumerate(merged, start=1):
|
||||||
|
row = rows.get(chunk_id)
|
||||||
|
if row is None:
|
||||||
|
continue
|
||||||
|
origin = _origin_label(bm25_rank.get(chunk_id), vector_rank.get(chunk_id))
|
||||||
|
print(f"{position}. {row['source_id']}:{row['path']} (rrf {score:.4f}, {origin})")
|
||||||
|
print(f" {row['breadcrumb']}")
|
||||||
|
print(_indent(_excerpt(row["text"], row["breadcrumb"])))
|
||||||
|
print()
|
||||||
|
return 0
|
||||||
|
|
||||||
|
|
||||||
|
def _origin_label(bm25: int | None, vector: int | None) -> str:
|
||||||
|
parts = []
|
||||||
|
if bm25 is not None:
|
||||||
|
parts.append(f"bm25 #{bm25}")
|
||||||
|
if vector is not None:
|
||||||
|
parts.append(f"vec #{vector}")
|
||||||
|
return ", ".join(parts)
|
||||||
|
|
||||||
|
|
||||||
|
def _excerpt(text: str, breadcrumb: str) -> str:
|
||||||
|
body = text[len(breadcrumb) :].lstrip("\n") if text.startswith(breadcrumb) else text
|
||||||
|
body = " ".join(body.split())
|
||||||
|
return body[:EXCERPT_CHARS] + ("…" if len(body) > EXCERPT_CHARS else "")
|
||||||
|
|
||||||
|
|
||||||
|
def _indent(text: str) -> str:
|
||||||
|
return "\n".join(f" {line}" for line in text.splitlines())
|
||||||
|
|
||||||
|
|
||||||
|
# ---------------------------------------------------------------------------
|
||||||
|
# toc
|
||||||
|
# ---------------------------------------------------------------------------
|
||||||
|
|
||||||
|
|
||||||
|
def run_toc(source_id: str | None, tag: str | None) -> int:
|
||||||
|
with store.connection(db_path()) as conn:
|
||||||
|
rows = store.list_toc_files(conn, source_id=source_id, tag=tag)
|
||||||
|
if not rows:
|
||||||
|
print("(no indexed files match)")
|
||||||
|
return 0
|
||||||
|
|
||||||
|
by_source: dict[str, list[sqlite3.Row]] = {}
|
||||||
|
for row in rows:
|
||||||
|
by_source.setdefault(row["source_id"], []).append(row)
|
||||||
|
|
||||||
|
for source, files in by_source.items():
|
||||||
|
print(f"{source} ({len(files)} files)")
|
||||||
|
print()
|
||||||
|
directory = None
|
||||||
|
for row in files:
|
||||||
|
path = row["path"]
|
||||||
|
parent, _, name = path.rpartition("/")
|
||||||
|
if parent != directory:
|
||||||
|
directory = parent
|
||||||
|
print(f"{parent}/" if parent else "./")
|
||||||
|
print(f" {name:<28} {row['title'] or '':<32} {_tag_label(row['tags'])}".rstrip())
|
||||||
|
print()
|
||||||
|
return 0
|
||||||
|
|
||||||
|
|
||||||
|
def _tag_label(raw: str | None) -> str:
|
||||||
|
try:
|
||||||
|
tags = json.loads(raw) if raw else []
|
||||||
|
except json.JSONDecodeError:
|
||||||
|
return ""
|
||||||
|
return f"[{', '.join(tags)}]" if tags else ""
|
||||||
|
|
||||||
|
|
||||||
|
# ---------------------------------------------------------------------------
|
||||||
|
# grep
|
||||||
|
# ---------------------------------------------------------------------------
|
||||||
|
|
||||||
|
|
||||||
|
def grep_roots(source: SourceConfig) -> list[Path]:
|
||||||
|
"""Directories ripgrep should walk for one source.
|
||||||
|
|
||||||
|
A git clone is searched whole. A workspace source is bounded by the literal prefix
|
||||||
|
of each `paths` glob, so grep stays inside what the source claims without being
|
||||||
|
narrowed to the indexed extensions.
|
||||||
|
"""
|
||||||
|
root = source_root(source)
|
||||||
|
if source.kind == GIT_KIND:
|
||||||
|
return [root] if root.is_dir() else []
|
||||||
|
roots = []
|
||||||
|
for pattern in source.paths:
|
||||||
|
prefix = _literal_prefix(pattern)
|
||||||
|
candidate = root / prefix if prefix else root
|
||||||
|
if candidate.is_dir() and candidate not in roots:
|
||||||
|
roots.append(candidate)
|
||||||
|
return roots
|
||||||
|
|
||||||
|
|
||||||
|
def _literal_prefix(pattern: str) -> str:
|
||||||
|
parts = []
|
||||||
|
for segment in pattern.split("/"):
|
||||||
|
if any(char in segment for char in "*?["):
|
||||||
|
break
|
||||||
|
parts.append(segment)
|
||||||
|
return "/".join(parts)
|
||||||
|
|
||||||
|
|
||||||
|
def run_grep(config: WikiConfig, pattern: str, source_id: str | None) -> int:
|
||||||
|
if shutil.which("rg") is None:
|
||||||
|
print("ripgrep (rg) not found", file=sys.stderr)
|
||||||
|
return 1
|
||||||
|
sources = [s for s in config.sources if not source_id or s.source_id == source_id]
|
||||||
|
if not sources:
|
||||||
|
print(f"unknown source {source_id!r}", file=sys.stderr)
|
||||||
|
return 1
|
||||||
|
|
||||||
|
roots = [path for source in sources for path in grep_roots(source)]
|
||||||
|
if not roots:
|
||||||
|
print("(nothing on disk to grep — has wiki_sync.py run?)")
|
||||||
|
return 0
|
||||||
|
|
||||||
|
command = [
|
||||||
|
"rg",
|
||||||
|
"--line-number",
|
||||||
|
"--no-heading",
|
||||||
|
"--color",
|
||||||
|
"never",
|
||||||
|
"--smart-case",
|
||||||
|
"--max-count",
|
||||||
|
str(GREP_MAX_PER_FILE),
|
||||||
|
"--glob",
|
||||||
|
"!.git",
|
||||||
|
pattern,
|
||||||
|
*[str(path) for path in roots],
|
||||||
|
]
|
||||||
|
try:
|
||||||
|
result = subprocess.run(command, capture_output=True, text=True, timeout=GREP_TIMEOUT, check=False)
|
||||||
|
except subprocess.TimeoutExpired:
|
||||||
|
print(f"grep timed out after {GREP_TIMEOUT}s", file=sys.stderr)
|
||||||
|
return 1
|
||||||
|
if result.returncode not in (0, 1):
|
||||||
|
print(result.stderr.strip(), file=sys.stderr)
|
||||||
|
return 1
|
||||||
|
output = result.stdout.strip()
|
||||||
|
print(output if output else "(no matches)")
|
||||||
|
return 0
|
||||||
|
|
||||||
|
|
||||||
|
# ---------------------------------------------------------------------------
|
||||||
|
# cli
|
||||||
|
# ---------------------------------------------------------------------------
|
||||||
|
|
||||||
|
|
||||||
|
def parse_args(argv: list[str] | None = None) -> argparse.Namespace:
|
||||||
|
parser = argparse.ArgumentParser(description="Search the wiki index over your notes.")
|
||||||
|
subparsers = parser.add_subparsers(dest="command", required=True)
|
||||||
|
|
||||||
|
search = subparsers.add_parser("search", help="hybrid BM25 + vector search over chunks")
|
||||||
|
search.add_argument("query", help="free text; Czech inflection is covered by prefix matching")
|
||||||
|
search.add_argument("--limit", type=int, default=DEFAULT_LIMIT, help="chunks to return")
|
||||||
|
|
||||||
|
toc = subparsers.add_parser("toc", help="directory -> file -> title + tags from the catalog")
|
||||||
|
toc.add_argument("--source", help="limit to one source id")
|
||||||
|
toc.add_argument("--tag", help="only files carrying this frontmatter tag")
|
||||||
|
|
||||||
|
grep = subparsers.add_parser("grep", help="live ripgrep over the files on disk, index-free")
|
||||||
|
grep.add_argument("pattern", help="ripgrep regex")
|
||||||
|
grep.add_argument("--source", help="limit to one source id")
|
||||||
|
|
||||||
|
return parser.parse_args(argv)
|
||||||
|
|
||||||
|
|
||||||
|
def main(argv: list[str] | None = None) -> int:
|
||||||
|
args = parse_args(argv)
|
||||||
|
try:
|
||||||
|
config = load_config()
|
||||||
|
except ConfigError as exc:
|
||||||
|
print(f"config error: {exc}", file=sys.stderr)
|
||||||
|
return 1
|
||||||
|
|
||||||
|
if args.command == "search":
|
||||||
|
return run_search(config, args.query, args.limit)
|
||||||
|
if args.command == "toc":
|
||||||
|
return run_toc(args.source, args.tag)
|
||||||
|
return run_grep(config, args.pattern, args.source)
|
||||||
|
|
||||||
|
|
||||||
|
if __name__ == "__main__":
|
||||||
|
sys.exit(main())
|
||||||
280
skills/wiki/scripts/wiki_store.py
Normal file
280
skills/wiki/scripts/wiki_store.py
Normal file
@@ -0,0 +1,280 @@
|
|||||||
|
#!/usr/bin/env python3
|
||||||
|
# /// script
|
||||||
|
# requires-python = ">=3.11"
|
||||||
|
# dependencies = ["sqlite-vec==0.1.6"]
|
||||||
|
# ///
|
||||||
|
"""Data-access layer for the wiki skill.
|
||||||
|
|
||||||
|
Pure SQL plus lifecycle helpers. No printing, no argparse, no sys.exit.
|
||||||
|
|
||||||
|
`vec_chunks` is a vec0 virtual table and therefore NOT reachable by the foreign key
|
||||||
|
cascade that cleans up `chunks` and `chunks_fts`. Every path that removes chunks must
|
||||||
|
delete their vectors explicitly — that is why the deletes here go through
|
||||||
|
`_delete_vectors_for_file`.
|
||||||
|
"""
|
||||||
|
|
||||||
|
from __future__ import annotations
|
||||||
|
|
||||||
|
import json
|
||||||
|
import sqlite3
|
||||||
|
from collections.abc import Iterator, Sequence
|
||||||
|
from contextlib import contextmanager
|
||||||
|
from pathlib import Path
|
||||||
|
|
||||||
|
from sqlite_vec import serialize_float32 # ty: ignore[unresolved-import]
|
||||||
|
from wiki_db import get_db, init_db
|
||||||
|
|
||||||
|
|
||||||
|
@contextmanager
|
||||||
|
def connection(db_path: Path) -> Iterator[sqlite3.Connection]:
|
||||||
|
"""Open a connection, initialising the DB if missing."""
|
||||||
|
if not db_path.exists():
|
||||||
|
init_db(db_path)
|
||||||
|
conn = get_db(db_path)
|
||||||
|
try:
|
||||||
|
yield conn
|
||||||
|
finally:
|
||||||
|
conn.close()
|
||||||
|
|
||||||
|
|
||||||
|
@contextmanager
|
||||||
|
def tx(conn: sqlite3.Connection) -> Iterator[sqlite3.Connection]:
|
||||||
|
"""Wrap an already-open connection in an explicit transaction.
|
||||||
|
|
||||||
|
The connection is in autocommit mode (`isolation_level=None`), so transactions are
|
||||||
|
ours to open — sqlite3's implicit handling would otherwise fail a nested BEGIN.
|
||||||
|
"""
|
||||||
|
conn.execute("BEGIN")
|
||||||
|
try:
|
||||||
|
yield conn
|
||||||
|
conn.execute("COMMIT")
|
||||||
|
except Exception:
|
||||||
|
conn.execute("ROLLBACK")
|
||||||
|
raise
|
||||||
|
|
||||||
|
|
||||||
|
@contextmanager
|
||||||
|
def transaction(db_path: Path) -> Iterator[sqlite3.Connection]:
|
||||||
|
"""Open a connection wrapped in an explicit transaction."""
|
||||||
|
with connection(db_path) as conn, tx(conn):
|
||||||
|
yield conn
|
||||||
|
|
||||||
|
|
||||||
|
# ---------------------------------------------------------------------------
|
||||||
|
# meta — identity of the embedding space
|
||||||
|
# ---------------------------------------------------------------------------
|
||||||
|
|
||||||
|
|
||||||
|
def read_meta(conn: sqlite3.Connection) -> dict[str, str]:
|
||||||
|
return {row["key"]: row["value"] for row in conn.execute("SELECT key, value FROM meta")}
|
||||||
|
|
||||||
|
|
||||||
|
def write_meta(conn: sqlite3.Connection, values: dict[str, str]) -> None:
|
||||||
|
conn.executemany(
|
||||||
|
"INSERT INTO meta (key, value) VALUES (?, ?) ON CONFLICT(key) DO UPDATE SET value = excluded.value",
|
||||||
|
sorted(values.items()),
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
|
# ---------------------------------------------------------------------------
|
||||||
|
# sources
|
||||||
|
# ---------------------------------------------------------------------------
|
||||||
|
|
||||||
|
|
||||||
|
def upsert_source(conn: sqlite3.Connection, source_id: str, kind: str) -> None:
|
||||||
|
conn.execute(
|
||||||
|
"INSERT INTO sources (source_id, kind) VALUES (?, ?) ON CONFLICT(source_id) DO UPDATE SET kind = excluded.kind",
|
||||||
|
(source_id, kind),
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
|
def get_source(conn: sqlite3.Connection, source_id: str) -> sqlite3.Row | None:
|
||||||
|
return conn.execute("SELECT * FROM sources WHERE source_id = ?", (source_id,)).fetchone()
|
||||||
|
|
||||||
|
|
||||||
|
def mark_synced(conn: sqlite3.Connection, source_id: str, indexed_rev: str | None, now: str) -> None:
|
||||||
|
conn.execute(
|
||||||
|
"UPDATE sources SET indexed_rev = ?, last_sync_at = ? WHERE source_id = ?",
|
||||||
|
(indexed_rev, now, source_id),
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
|
# ---------------------------------------------------------------------------
|
||||||
|
# files
|
||||||
|
# ---------------------------------------------------------------------------
|
||||||
|
|
||||||
|
|
||||||
|
def get_file(conn: sqlite3.Connection, source_id: str, path: str) -> sqlite3.Row | None:
|
||||||
|
return conn.execute("SELECT * FROM files WHERE source_id = ? AND path = ?", (source_id, path)).fetchone()
|
||||||
|
|
||||||
|
|
||||||
|
def list_source_files(conn: sqlite3.Connection, source_id: str) -> dict[str, sqlite3.Row]:
|
||||||
|
"""Indexed files of one source, keyed by path — the basis for deletion detection."""
|
||||||
|
rows = conn.execute("SELECT * FROM files WHERE source_id = ?", (source_id,))
|
||||||
|
return {row["path"]: row for row in rows}
|
||||||
|
|
||||||
|
|
||||||
|
def upsert_file(
|
||||||
|
conn: sqlite3.Connection,
|
||||||
|
source_id: str,
|
||||||
|
path: str,
|
||||||
|
title: str | None,
|
||||||
|
tags: list[str],
|
||||||
|
headings: list[str],
|
||||||
|
sha256: str,
|
||||||
|
size: int,
|
||||||
|
mtime: float | None,
|
||||||
|
now: str,
|
||||||
|
) -> None:
|
||||||
|
conn.execute(
|
||||||
|
"INSERT INTO files (source_id, path, title, tags, headings, sha256, size, mtime, indexed_at) "
|
||||||
|
"VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?) "
|
||||||
|
"ON CONFLICT(source_id, path) DO UPDATE SET "
|
||||||
|
"title = excluded.title, tags = excluded.tags, headings = excluded.headings, "
|
||||||
|
"sha256 = excluded.sha256, size = excluded.size, mtime = excluded.mtime, "
|
||||||
|
"indexed_at = excluded.indexed_at",
|
||||||
|
(
|
||||||
|
source_id,
|
||||||
|
path,
|
||||||
|
title,
|
||||||
|
json.dumps(tags, ensure_ascii=False),
|
||||||
|
json.dumps(headings, ensure_ascii=False),
|
||||||
|
sha256,
|
||||||
|
size,
|
||||||
|
mtime,
|
||||||
|
now,
|
||||||
|
),
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
|
def touch_file_stat(conn: sqlite3.Connection, source_id: str, path: str, size: int, mtime: float | None) -> None:
|
||||||
|
"""Refresh the cheap change-detection stats after a sha256 match (content unchanged)."""
|
||||||
|
conn.execute(
|
||||||
|
"UPDATE files SET size = ?, mtime = ? WHERE source_id = ? AND path = ?",
|
||||||
|
(size, mtime, source_id, path),
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
|
def delete_file(conn: sqlite3.Connection, source_id: str, path: str) -> None:
|
||||||
|
"""Drop a file and everything derived from it, vec0 rows included."""
|
||||||
|
_delete_vectors_for_file(conn, source_id, path)
|
||||||
|
conn.execute("DELETE FROM files WHERE source_id = ? AND path = ?", (source_id, path))
|
||||||
|
|
||||||
|
|
||||||
|
def list_toc_files(conn: sqlite3.Connection, source_id: str | None = None, tag: str | None = None) -> list[sqlite3.Row]:
|
||||||
|
sql = "SELECT source_id, path, title, tags FROM files"
|
||||||
|
clauses: list[str] = []
|
||||||
|
params: list[str] = []
|
||||||
|
if source_id:
|
||||||
|
clauses.append("source_id = ?")
|
||||||
|
params.append(source_id)
|
||||||
|
if tag:
|
||||||
|
clauses.append("EXISTS (SELECT 1 FROM json_each(files.tags) WHERE value = ?)")
|
||||||
|
params.append(tag)
|
||||||
|
if clauses:
|
||||||
|
sql += " WHERE " + " AND ".join(clauses)
|
||||||
|
sql += " ORDER BY source_id, path"
|
||||||
|
return list(conn.execute(sql, params))
|
||||||
|
|
||||||
|
|
||||||
|
# ---------------------------------------------------------------------------
|
||||||
|
# chunks
|
||||||
|
# ---------------------------------------------------------------------------
|
||||||
|
|
||||||
|
|
||||||
|
def _delete_vectors_for_file(conn: sqlite3.Connection, source_id: str, path: str) -> None:
|
||||||
|
ids = [
|
||||||
|
row["id"] for row in conn.execute("SELECT id FROM chunks WHERE source_id = ? AND path = ?", (source_id, path))
|
||||||
|
]
|
||||||
|
conn.executemany("DELETE FROM vec_chunks WHERE rowid = ?", [(i,) for i in ids])
|
||||||
|
|
||||||
|
|
||||||
|
def replace_chunks(conn: sqlite3.Connection, source_id: str, path: str, chunks: Sequence[tuple[str, str]]) -> None:
|
||||||
|
"""Swap a file's chunks for a freshly built set, as (breadcrumb, text) in order.
|
||||||
|
|
||||||
|
New chunks land with `embedded_at IS NULL`; the embed pass picks them up, so an
|
||||||
|
unreachable Ollama degrades to FTS-only instead of failing the sync.
|
||||||
|
"""
|
||||||
|
_delete_vectors_for_file(conn, source_id, path)
|
||||||
|
conn.execute("DELETE FROM chunks WHERE source_id = ? AND path = ?", (source_id, path))
|
||||||
|
conn.executemany(
|
||||||
|
"INSERT INTO chunks (source_id, path, chunk_idx, breadcrumb, text) VALUES (?, ?, ?, ?, ?)",
|
||||||
|
[(source_id, path, idx, breadcrumb, text) for idx, (breadcrumb, text) in enumerate(chunks)],
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
|
def pending_chunks(conn: sqlite3.Connection, limit: int) -> list[sqlite3.Row]:
|
||||||
|
return list(
|
||||||
|
conn.execute(
|
||||||
|
"SELECT id, breadcrumb, text FROM chunks WHERE embedded_at IS NULL ORDER BY id LIMIT ?",
|
||||||
|
(limit,),
|
||||||
|
)
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
|
def has_pending(conn: sqlite3.Connection) -> bool:
|
||||||
|
return conn.execute("SELECT 1 FROM chunks WHERE embedded_at IS NULL LIMIT 1").fetchone() is not None
|
||||||
|
|
||||||
|
|
||||||
|
def store_embedding(conn: sqlite3.Connection, chunk_id: int, vector: Sequence[float], now: str) -> None:
|
||||||
|
conn.execute("DELETE FROM vec_chunks WHERE rowid = ?", (chunk_id,))
|
||||||
|
conn.execute(
|
||||||
|
"INSERT INTO vec_chunks (rowid, embedding) VALUES (?, ?)",
|
||||||
|
(chunk_id, serialize_float32(list(vector))),
|
||||||
|
)
|
||||||
|
conn.execute("UPDATE chunks SET embedded_at = ? WHERE id = ?", (now, chunk_id))
|
||||||
|
|
||||||
|
|
||||||
|
def reset_index(conn: sqlite3.Connection) -> None:
|
||||||
|
"""Drop every indexed artifact, keeping the source rows. Used by `--full`."""
|
||||||
|
ids = [row["id"] for row in conn.execute("SELECT id FROM chunks")]
|
||||||
|
conn.executemany("DELETE FROM vec_chunks WHERE rowid = ?", [(i,) for i in ids])
|
||||||
|
conn.execute("DELETE FROM files")
|
||||||
|
conn.execute("UPDATE sources SET indexed_rev = NULL")
|
||||||
|
|
||||||
|
|
||||||
|
def index_stats(conn: sqlite3.Connection) -> dict[str, int]:
|
||||||
|
return {
|
||||||
|
"files": conn.execute("SELECT count(*) AS n FROM files").fetchone()["n"],
|
||||||
|
"chunks": conn.execute("SELECT count(*) AS n FROM chunks").fetchone()["n"],
|
||||||
|
"vectors": conn.execute("SELECT count(*) AS n FROM vec_chunks").fetchone()["n"],
|
||||||
|
"pending": conn.execute("SELECT count(*) AS n FROM chunks WHERE embedded_at IS NULL").fetchone()["n"],
|
||||||
|
}
|
||||||
|
|
||||||
|
|
||||||
|
def orphan_vector_ids(conn: sqlite3.Connection) -> list[int]:
|
||||||
|
"""Vector rowids with no surviving chunk — must always be empty (regression guard)."""
|
||||||
|
rows = conn.execute("SELECT rowid AS rid FROM vec_chunks WHERE rowid NOT IN (SELECT id FROM chunks)")
|
||||||
|
return [row["rid"] for row in rows]
|
||||||
|
|
||||||
|
|
||||||
|
# ---------------------------------------------------------------------------
|
||||||
|
# retrieval
|
||||||
|
# ---------------------------------------------------------------------------
|
||||||
|
|
||||||
|
|
||||||
|
def bm25_ranked_ids(conn: sqlite3.Connection, match_expr: str, limit: int) -> list[int]:
|
||||||
|
rows = conn.execute(
|
||||||
|
"SELECT rowid AS rid FROM chunks_fts WHERE chunks_fts MATCH ? ORDER BY rank LIMIT ?",
|
||||||
|
(match_expr, limit),
|
||||||
|
)
|
||||||
|
return [row["rid"] for row in rows]
|
||||||
|
|
||||||
|
|
||||||
|
def knn_ranked_ids(conn: sqlite3.Connection, vector: Sequence[float], k: int) -> list[int]:
|
||||||
|
rows = conn.execute(
|
||||||
|
"SELECT rowid AS rid FROM vec_chunks WHERE embedding MATCH ? AND k = ? ORDER BY distance",
|
||||||
|
(serialize_float32(list(vector)), k),
|
||||||
|
)
|
||||||
|
return [row["rid"] for row in rows]
|
||||||
|
|
||||||
|
|
||||||
|
def fetch_chunks(conn: sqlite3.Connection, ids: Sequence[int]) -> dict[int, sqlite3.Row]:
|
||||||
|
if not ids:
|
||||||
|
return {}
|
||||||
|
placeholders = ",".join("?" * len(ids))
|
||||||
|
rows = conn.execute(
|
||||||
|
f"SELECT id, source_id, path, chunk_idx, breadcrumb, text FROM chunks WHERE id IN ({placeholders})",
|
||||||
|
list(ids),
|
||||||
|
)
|
||||||
|
return {row["id"]: row for row in rows}
|
||||||
529
skills/wiki/scripts/wiki_sync.py
Normal file
529
skills/wiki/scripts/wiki_sync.py
Normal file
@@ -0,0 +1,529 @@
|
|||||||
|
#!/usr/bin/env -S uv run --script
|
||||||
|
# /// script
|
||||||
|
# requires-python = ">=3.11"
|
||||||
|
# dependencies = ["requests", "pyyaml", "markdown-it-py", "sqlite-vec==0.1.6"]
|
||||||
|
# ///
|
||||||
|
"""Offline indexer for the wiki skill. Runs every minute from the nanobot crontab.
|
||||||
|
|
||||||
|
Indexing never happens inside an agent turn: the `exec` tool times out at 60 s while a
|
||||||
|
full index of 10^4 chunks takes ~93 s. The agent only ever reads a finished index.
|
||||||
|
|
||||||
|
Per tick:
|
||||||
|
|
||||||
|
1. take `wiki/.sync.lock`; a live holder means exit 0 in silence
|
||||||
|
2. detect changes cheaply — `git ls-remote` (no fetch) for git sources, a
|
||||||
|
path+size+mtime walk for workspace sources, sha256 only on a stat mismatch
|
||||||
|
3. nothing changed and nothing pending -> exit 0 (the overwhelming majority of ticks)
|
||||||
|
4. re-chunk changed files, then drain every chunk with `embedded_at IS NULL` — which
|
||||||
|
is also the way back out of degraded mode after Ollama returns
|
||||||
|
5. record indexed_rev / last_sync_at
|
||||||
|
6. log a coverage line naming top-level directories no source covers
|
||||||
|
7. append the run summary to log/wiki_sync.log
|
||||||
|
|
||||||
|
Network failure is per-source: a timeout or non-zero git exit logs WARN, skips that
|
||||||
|
source with its `indexed_rev` untouched, and lets the others finish. Without the
|
||||||
|
timeouts a hanging `ls-remote` would hold the lock and block workspace sources that
|
||||||
|
have nothing to do with the network.
|
||||||
|
"""
|
||||||
|
|
||||||
|
from __future__ import annotations
|
||||||
|
|
||||||
|
import argparse
|
||||||
|
import hashlib
|
||||||
|
import json
|
||||||
|
import os
|
||||||
|
import subprocess
|
||||||
|
import sys
|
||||||
|
from dataclasses import dataclass, field
|
||||||
|
from datetime import UTC, datetime
|
||||||
|
from pathlib import Path
|
||||||
|
|
||||||
|
import wiki_store as store
|
||||||
|
from wiki_chunker import parse_markdown
|
||||||
|
from wiki_config import (
|
||||||
|
GIT_KIND,
|
||||||
|
LOCK_PATH,
|
||||||
|
SYNC_LOG_PATH,
|
||||||
|
WIKI_DIR,
|
||||||
|
ConfigError,
|
||||||
|
SourceConfig,
|
||||||
|
WikiConfig,
|
||||||
|
db_path,
|
||||||
|
load_config,
|
||||||
|
source_root,
|
||||||
|
)
|
||||||
|
from wiki_embed import (
|
||||||
|
EmbeddingUnavailable,
|
||||||
|
OllamaEmbedder,
|
||||||
|
expected_meta,
|
||||||
|
meta_mismatches,
|
||||||
|
)
|
||||||
|
|
||||||
|
STALE_SECONDS = 30 * 60
|
||||||
|
LS_REMOTE_TIMEOUT = 20
|
||||||
|
GIT_TIMEOUT = 300
|
||||||
|
GIT_ERROR_CHARS = 300
|
||||||
|
SKIP_DIRS = frozenset({".git"})
|
||||||
|
|
||||||
|
|
||||||
|
class GitError(RuntimeError):
|
||||||
|
"""A git subprocess failed or timed out."""
|
||||||
|
|
||||||
|
|
||||||
|
@dataclass
|
||||||
|
class SourcePlan:
|
||||||
|
"""What one source needs done this tick."""
|
||||||
|
|
||||||
|
changed: list[str] = field(default_factory=list)
|
||||||
|
deleted: list[str] = field(default_factory=list)
|
||||||
|
stat_refresh: list[tuple[str, int, float]] = field(default_factory=list)
|
||||||
|
new_rev: str | None = None
|
||||||
|
|
||||||
|
@property
|
||||||
|
def has_work(self) -> bool:
|
||||||
|
return bool(self.changed or self.deleted or self.stat_refresh)
|
||||||
|
|
||||||
|
|
||||||
|
def log(message: str) -> None:
|
||||||
|
SYNC_LOG_PATH.parent.mkdir(parents=True, exist_ok=True)
|
||||||
|
stamp = datetime.now().astimezone().isoformat(timespec="seconds")
|
||||||
|
with SYNC_LOG_PATH.open("a", encoding="utf-8") as handle:
|
||||||
|
handle.write(f"{stamp} {message}\n")
|
||||||
|
|
||||||
|
|
||||||
|
def _now() -> str:
|
||||||
|
return datetime.now(UTC).isoformat(timespec="seconds")
|
||||||
|
|
||||||
|
|
||||||
|
# ---------------------------------------------------------------------------
|
||||||
|
# lock
|
||||||
|
# ---------------------------------------------------------------------------
|
||||||
|
|
||||||
|
|
||||||
|
def _pid_alive(pid: int) -> bool:
|
||||||
|
try:
|
||||||
|
os.kill(pid, 0)
|
||||||
|
except ProcessLookupError:
|
||||||
|
return False
|
||||||
|
except PermissionError:
|
||||||
|
return True
|
||||||
|
return True
|
||||||
|
|
||||||
|
|
||||||
|
def _lock_is_stale() -> bool:
|
||||||
|
"""A lock is dead if unreadable, its pid is gone, or it is older than STALE_SECONDS."""
|
||||||
|
try:
|
||||||
|
data = json.loads(LOCK_PATH.read_text(encoding="utf-8"))
|
||||||
|
pid = int(data["pid"])
|
||||||
|
started = datetime.fromisoformat(data["started_at"])
|
||||||
|
except (OSError, ValueError, KeyError):
|
||||||
|
return True
|
||||||
|
if not _pid_alive(pid):
|
||||||
|
return True
|
||||||
|
return (datetime.now().astimezone() - started).total_seconds() > STALE_SECONDS
|
||||||
|
|
||||||
|
|
||||||
|
def acquire_lock() -> bool:
|
||||||
|
"""Atomically create the lock. False when a live sync already runs.
|
||||||
|
|
||||||
|
Reclaiming a stale lock is not optional: without it a crashed run (OOM, reboot)
|
||||||
|
would make every later cron tick exit 0 in silence, forever.
|
||||||
|
"""
|
||||||
|
LOCK_PATH.parent.mkdir(parents=True, exist_ok=True)
|
||||||
|
for _ in range(2):
|
||||||
|
try:
|
||||||
|
handle = os.open(LOCK_PATH, os.O_CREAT | os.O_EXCL | os.O_WRONLY, 0o644)
|
||||||
|
except FileExistsError:
|
||||||
|
if not _lock_is_stale():
|
||||||
|
return False
|
||||||
|
log("WARN stale lock, reclaiming")
|
||||||
|
LOCK_PATH.unlink(missing_ok=True)
|
||||||
|
continue
|
||||||
|
payload = {"pid": os.getpid(), "started_at": datetime.now().astimezone().isoformat()}
|
||||||
|
with os.fdopen(handle, "w", encoding="utf-8") as file:
|
||||||
|
json.dump(payload, file)
|
||||||
|
return True
|
||||||
|
return False
|
||||||
|
|
||||||
|
|
||||||
|
def release_lock() -> None:
|
||||||
|
LOCK_PATH.unlink(missing_ok=True)
|
||||||
|
|
||||||
|
|
||||||
|
# ---------------------------------------------------------------------------
|
||||||
|
# filesystem
|
||||||
|
# ---------------------------------------------------------------------------
|
||||||
|
|
||||||
|
|
||||||
|
def _sha256(path: Path) -> str:
|
||||||
|
digest = hashlib.sha256()
|
||||||
|
with path.open("rb") as handle:
|
||||||
|
for block in iter(lambda: handle.read(65536), b""):
|
||||||
|
digest.update(block)
|
||||||
|
return digest.hexdigest()
|
||||||
|
|
||||||
|
|
||||||
|
def walk_source(source: SourceConfig, root: Path) -> dict[str, tuple[int, float]]:
|
||||||
|
"""Covered files under `root`, as rel_path -> (size, mtime). Prunes uncovered dirs."""
|
||||||
|
found: dict[str, tuple[int, float]] = {}
|
||||||
|
if not root.is_dir():
|
||||||
|
return found
|
||||||
|
for dirpath, dirnames, filenames in os.walk(root):
|
||||||
|
rel_dir = os.path.relpath(dirpath, root)
|
||||||
|
rel_dir = "" if rel_dir == "." else rel_dir
|
||||||
|
dirnames[:] = [
|
||||||
|
name for name in dirnames if name not in SKIP_DIRS and source.covers_dir(f"{rel_dir}/{name}".lstrip("/"))
|
||||||
|
]
|
||||||
|
for name in filenames:
|
||||||
|
rel_path = f"{rel_dir}/{name}".lstrip("/")
|
||||||
|
if not source.covers(rel_path):
|
||||||
|
continue
|
||||||
|
file_path = Path(dirpath) / name
|
||||||
|
if not file_path.is_file():
|
||||||
|
continue
|
||||||
|
stat = file_path.stat()
|
||||||
|
found[rel_path] = (stat.st_size, stat.st_mtime)
|
||||||
|
return found
|
||||||
|
|
||||||
|
|
||||||
|
def uncovered_top_level_dirs(source: SourceConfig, root: Path) -> list[str]:
|
||||||
|
"""Top-level directories holding markdown that no source path reaches.
|
||||||
|
|
||||||
|
This is the safety net under the whitelist: a directory that silently fails to be
|
||||||
|
indexed shows up here instead of nowhere. `wiki/` is skipped because it holds this
|
||||||
|
skill's own clones and index — never a candidate, so reporting it is pure noise.
|
||||||
|
"""
|
||||||
|
if not root.is_dir():
|
||||||
|
return []
|
||||||
|
uncovered = []
|
||||||
|
for entry in sorted(root.iterdir()):
|
||||||
|
if not entry.is_dir() or entry.name in SKIP_DIRS or entry == WIKI_DIR:
|
||||||
|
continue
|
||||||
|
if source.covers_dir(entry.name):
|
||||||
|
continue
|
||||||
|
if next(entry.rglob("*.md"), None) is not None:
|
||||||
|
uncovered.append(entry.name)
|
||||||
|
return uncovered
|
||||||
|
|
||||||
|
|
||||||
|
# ---------------------------------------------------------------------------
|
||||||
|
# git driver
|
||||||
|
# ---------------------------------------------------------------------------
|
||||||
|
|
||||||
|
|
||||||
|
def _git(args: list[str], cwd: Path | None = None, timeout: int = GIT_TIMEOUT) -> str:
|
||||||
|
try:
|
||||||
|
result = subprocess.run(
|
||||||
|
["git", *args],
|
||||||
|
cwd=cwd,
|
||||||
|
capture_output=True,
|
||||||
|
text=True,
|
||||||
|
timeout=timeout,
|
||||||
|
check=False,
|
||||||
|
)
|
||||||
|
except subprocess.TimeoutExpired as exc:
|
||||||
|
raise GitError(f"git {' '.join(args)} timed out after {timeout}s") from exc
|
||||||
|
if result.returncode != 0:
|
||||||
|
raise GitError(f"git {' '.join(args)} failed: {_one_line(result.stderr)}")
|
||||||
|
return result.stdout
|
||||||
|
|
||||||
|
|
||||||
|
def _one_line(text: str) -> str:
|
||||||
|
"""Squash git's multi-line stderr into one capped line.
|
||||||
|
|
||||||
|
A permanently unreachable source warns on every tick, so a six-line stderr would
|
||||||
|
put thousands of lines a day into the log and drown everything else.
|
||||||
|
"""
|
||||||
|
collapsed = " ".join(text.split())
|
||||||
|
return collapsed[:GIT_ERROR_CHARS] + ("…" if len(collapsed) > GIT_ERROR_CHARS else "")
|
||||||
|
|
||||||
|
|
||||||
|
def remote_head(url: str) -> str:
|
||||||
|
"""Remote HEAD without fetching — the right tool for a per-minute cadence."""
|
||||||
|
output = _git(["ls-remote", "--symref", url, "HEAD"], timeout=LS_REMOTE_TIMEOUT)
|
||||||
|
for line in output.splitlines():
|
||||||
|
parts = line.split()
|
||||||
|
if len(parts) == 2 and parts[1] == "HEAD":
|
||||||
|
return parts[0]
|
||||||
|
raise GitError(f"no HEAD in ls-remote output for {url}")
|
||||||
|
|
||||||
|
|
||||||
|
def _diff_names(root: Path, base_rev: str) -> tuple[list[str], list[str]] | None:
|
||||||
|
"""(changed, deleted) between base_rev and FETCH_HEAD, or None if base_rev is gone."""
|
||||||
|
try:
|
||||||
|
output = _git(["diff", "--name-status", f"{base_rev}..FETCH_HEAD"], cwd=root)
|
||||||
|
except GitError:
|
||||||
|
return None
|
||||||
|
changed, deleted = [], []
|
||||||
|
for line in output.splitlines():
|
||||||
|
fields = line.split("\t")
|
||||||
|
if len(fields) < 2:
|
||||||
|
continue
|
||||||
|
status = fields[0]
|
||||||
|
if status.startswith("R") and len(fields) >= 3:
|
||||||
|
deleted.append(fields[1])
|
||||||
|
changed.append(fields[2])
|
||||||
|
elif status.startswith("D"):
|
||||||
|
deleted.append(fields[1])
|
||||||
|
else:
|
||||||
|
changed.append(fields[1])
|
||||||
|
return changed, deleted
|
||||||
|
|
||||||
|
|
||||||
|
def plan_git_source(conn, source: SourceConfig, root: Path, full: bool) -> SourcePlan:
|
||||||
|
rev = remote_head(str(source.url))
|
||||||
|
row = store.get_source(conn, source.source_id)
|
||||||
|
indexed_rev = row["indexed_rev"] if row else None
|
||||||
|
|
||||||
|
if not root.exists():
|
||||||
|
_git(["clone", str(source.url), str(root)])
|
||||||
|
indexed_rev = None
|
||||||
|
if not full and indexed_rev == rev:
|
||||||
|
return SourcePlan(new_rev=rev)
|
||||||
|
|
||||||
|
_git(["fetch", "--prune"], cwd=root)
|
||||||
|
diff = None if (full or not indexed_rev) else _diff_names(root, indexed_rev)
|
||||||
|
_git(["reset", "--hard", "FETCH_HEAD"], cwd=root)
|
||||||
|
|
||||||
|
indexed = store.list_source_files(conn, source.source_id)
|
||||||
|
on_disk = walk_source(source, root)
|
||||||
|
if diff is None:
|
||||||
|
# Fresh clone, --full, or an indexed_rev the repo no longer has: reindex it all.
|
||||||
|
return SourcePlan(
|
||||||
|
changed=sorted(on_disk),
|
||||||
|
deleted=sorted(path for path in indexed if path not in on_disk),
|
||||||
|
new_rev=rev,
|
||||||
|
)
|
||||||
|
raw_changed, raw_deleted = diff
|
||||||
|
changed = sorted({path for path in raw_changed if path in on_disk})
|
||||||
|
deleted = sorted(
|
||||||
|
{path for path in raw_deleted if path in indexed} | {path for path in indexed if path not in on_disk}
|
||||||
|
)
|
||||||
|
return SourcePlan(changed=changed, deleted=[p for p in deleted if p not in changed], new_rev=rev)
|
||||||
|
|
||||||
|
|
||||||
|
# ---------------------------------------------------------------------------
|
||||||
|
# workspace driver
|
||||||
|
# ---------------------------------------------------------------------------
|
||||||
|
|
||||||
|
|
||||||
|
def plan_workspace_source(conn, source: SourceConfig, root: Path, full: bool) -> SourcePlan:
|
||||||
|
on_disk = walk_source(source, root)
|
||||||
|
indexed = store.list_source_files(conn, source.source_id)
|
||||||
|
plan = SourcePlan()
|
||||||
|
for rel_path, (size, mtime) in sorted(on_disk.items()):
|
||||||
|
row = indexed.get(rel_path)
|
||||||
|
if full or row is None:
|
||||||
|
plan.changed.append(rel_path)
|
||||||
|
continue
|
||||||
|
if row["size"] == size and row["mtime"] == mtime:
|
||||||
|
continue
|
||||||
|
if _sha256(root / rel_path) == row["sha256"]:
|
||||||
|
# Content is identical; refresh the stats so the next tick stays cheap.
|
||||||
|
plan.stat_refresh.append((rel_path, size, mtime))
|
||||||
|
else:
|
||||||
|
plan.changed.append(rel_path)
|
||||||
|
plan.deleted = sorted(path for path in indexed if path not in on_disk)
|
||||||
|
return plan
|
||||||
|
|
||||||
|
|
||||||
|
# ---------------------------------------------------------------------------
|
||||||
|
# indexing
|
||||||
|
# ---------------------------------------------------------------------------
|
||||||
|
|
||||||
|
|
||||||
|
def index_file(conn, source_id: str, root: Path, rel_path: str, now: str) -> int:
|
||||||
|
"""Re-chunk one file. Returns the chunk count, or -1 when it could not be read."""
|
||||||
|
file_path = root / rel_path
|
||||||
|
try:
|
||||||
|
text = file_path.read_text(encoding="utf-8")
|
||||||
|
stat = file_path.stat()
|
||||||
|
except (OSError, UnicodeDecodeError):
|
||||||
|
return -1
|
||||||
|
parsed = parse_markdown(text, rel_path)
|
||||||
|
store.upsert_file(
|
||||||
|
conn,
|
||||||
|
source_id,
|
||||||
|
rel_path,
|
||||||
|
parsed.title,
|
||||||
|
parsed.tags,
|
||||||
|
parsed.headings,
|
||||||
|
_sha256(file_path),
|
||||||
|
stat.st_size,
|
||||||
|
stat.st_mtime,
|
||||||
|
now,
|
||||||
|
)
|
||||||
|
store.replace_chunks(conn, source_id, rel_path, [(c.breadcrumb, c.text) for c in parsed.chunks])
|
||||||
|
return len(parsed.chunks)
|
||||||
|
|
||||||
|
|
||||||
|
def apply_plan(conn, source: SourceConfig, root: Path, plan: SourcePlan) -> dict[str, int]:
|
||||||
|
counts = {"indexed": 0, "deleted": 0, "unreadable": 0, "chunks": 0}
|
||||||
|
now = _now()
|
||||||
|
for rel_path in plan.deleted:
|
||||||
|
with store.tx(conn):
|
||||||
|
store.delete_file(conn, source.source_id, rel_path)
|
||||||
|
counts["deleted"] += 1
|
||||||
|
for rel_path in plan.changed:
|
||||||
|
with store.tx(conn):
|
||||||
|
chunks = index_file(conn, source.source_id, root, rel_path, now)
|
||||||
|
if chunks < 0:
|
||||||
|
counts["unreadable"] += 1
|
||||||
|
log(f"WARN {source.source_id}: unreadable {rel_path}")
|
||||||
|
continue
|
||||||
|
counts["indexed"] += 1
|
||||||
|
counts["chunks"] += chunks
|
||||||
|
if plan.stat_refresh:
|
||||||
|
with store.tx(conn):
|
||||||
|
for rel_path, size, mtime in plan.stat_refresh:
|
||||||
|
store.touch_file_stat(conn, source.source_id, rel_path, size, mtime)
|
||||||
|
with store.tx(conn):
|
||||||
|
store.mark_synced(conn, source.source_id, plan.new_rev, now)
|
||||||
|
return counts
|
||||||
|
|
||||||
|
|
||||||
|
def drain_pending(conn, config: WikiConfig) -> tuple[int, str | None]:
|
||||||
|
"""Embed every chunk still missing a vector. Returns (embedded, warning)."""
|
||||||
|
if not store.has_pending(conn):
|
||||||
|
return 0, None
|
||||||
|
embedder = OllamaEmbedder(config.embedding)
|
||||||
|
embedded = 0
|
||||||
|
while True:
|
||||||
|
batch = store.pending_chunks(conn, config.embedding.batch)
|
||||||
|
if not batch:
|
||||||
|
return embedded, None
|
||||||
|
try:
|
||||||
|
vectors = embedder.embed_documents([row["text"] for row in batch])
|
||||||
|
except EmbeddingUnavailable as exc:
|
||||||
|
return embedded, f"embeddings unavailable, staying FTS-only: {exc}"
|
||||||
|
now = _now()
|
||||||
|
with store.tx(conn):
|
||||||
|
for row, vector in zip(batch, vectors, strict=True):
|
||||||
|
store.store_embedding(conn, row["id"], vector, now)
|
||||||
|
embedded += len(batch)
|
||||||
|
|
||||||
|
|
||||||
|
# ---------------------------------------------------------------------------
|
||||||
|
# run
|
||||||
|
# ---------------------------------------------------------------------------
|
||||||
|
|
||||||
|
|
||||||
|
def ensure_meta(conn, config: WikiConfig, full: bool) -> str | None:
|
||||||
|
"""Write the index identity, or report a mismatch that only --full can resolve."""
|
||||||
|
stored = store.read_meta(conn)
|
||||||
|
mismatches = meta_mismatches(stored, config.embedding)
|
||||||
|
if mismatches and not full:
|
||||||
|
return f"index identity mismatch on {', '.join(mismatches)} — run wiki_sync.py --full"
|
||||||
|
with store.tx(conn):
|
||||||
|
store.write_meta(conn, expected_meta(config.embedding))
|
||||||
|
return None
|
||||||
|
|
||||||
|
|
||||||
|
def run(config: WikiConfig, args: argparse.Namespace) -> int:
|
||||||
|
sources = [s for s in config.sources if not args.source or s.source_id == args.source]
|
||||||
|
if not sources:
|
||||||
|
log(f"WARN unknown source {args.source!r}")
|
||||||
|
return 1
|
||||||
|
|
||||||
|
with store.connection(db_path()) as conn:
|
||||||
|
problem = ensure_meta(conn, config, args.full)
|
||||||
|
if problem:
|
||||||
|
log(f"WARN {problem}")
|
||||||
|
return 1
|
||||||
|
|
||||||
|
for source in sources:
|
||||||
|
with store.tx(conn):
|
||||||
|
store.upsert_source(conn, source.source_id, source.kind)
|
||||||
|
|
||||||
|
warnings: list[str] = []
|
||||||
|
did_work = args.full
|
||||||
|
for source in sources:
|
||||||
|
root = source_root(source)
|
||||||
|
try:
|
||||||
|
if source.kind == GIT_KIND:
|
||||||
|
plan = plan_git_source(conn, source, root, args.full)
|
||||||
|
else:
|
||||||
|
plan = plan_workspace_source(conn, source, root, args.full)
|
||||||
|
except GitError as exc:
|
||||||
|
# indexed_rev stays untouched, so the next tick retries this source.
|
||||||
|
warnings.append(f"{source.source_id}: {exc}")
|
||||||
|
log(f"WARN {source.source_id}: {exc}")
|
||||||
|
continue
|
||||||
|
|
||||||
|
row = store.get_source(conn, source.source_id)
|
||||||
|
indexed_rev = row["indexed_rev"] if row else None
|
||||||
|
rev_moved = plan.new_rev is not None and plan.new_rev != indexed_rev
|
||||||
|
if not plan.has_work and not rev_moved:
|
||||||
|
continue
|
||||||
|
|
||||||
|
did_work = True
|
||||||
|
if not plan.has_work:
|
||||||
|
# Upstream moved but touched nothing we index — record the rev so the
|
||||||
|
# next tick stops fetching.
|
||||||
|
with store.tx(conn):
|
||||||
|
store.mark_synced(conn, source.source_id, plan.new_rev, _now())
|
||||||
|
log(f"{source.source_id}: rev moved to {plan.new_rev}, no indexed file changed")
|
||||||
|
continue
|
||||||
|
|
||||||
|
counts = apply_plan(conn, source, root, plan)
|
||||||
|
log(
|
||||||
|
f"{source.source_id}: indexed {counts['indexed']} files "
|
||||||
|
f"({counts['chunks']} chunks), deleted {counts['deleted']}"
|
||||||
|
)
|
||||||
|
|
||||||
|
embedded, embed_warning = drain_pending(conn, config)
|
||||||
|
if embed_warning:
|
||||||
|
warnings.append(embed_warning)
|
||||||
|
log(f"WARN {embed_warning}")
|
||||||
|
|
||||||
|
if not did_work and not embedded and not warnings:
|
||||||
|
return 0
|
||||||
|
|
||||||
|
for source in sources:
|
||||||
|
uncovered = uncovered_top_level_dirs(source, source_root(source))
|
||||||
|
if uncovered:
|
||||||
|
log(f"coverage {source.source_id}: markdown outside paths in {', '.join(uncovered)}")
|
||||||
|
|
||||||
|
stats = store.index_stats(conn)
|
||||||
|
log(
|
||||||
|
f"done files={stats['files']} chunks={stats['chunks']} vectors={stats['vectors']} "
|
||||||
|
f"pending={stats['pending']} embedded={embedded}"
|
||||||
|
)
|
||||||
|
return 1 if warnings else 0
|
||||||
|
|
||||||
|
|
||||||
|
def parse_args(argv: list[str] | None = None) -> argparse.Namespace:
|
||||||
|
parser = argparse.ArgumentParser(description="Index the wiki sources into wiki/index.sqlite.")
|
||||||
|
parser.add_argument(
|
||||||
|
"--full",
|
||||||
|
action="store_true",
|
||||||
|
help="wipe and rebuild the index (needed after a model or chunker change)",
|
||||||
|
)
|
||||||
|
parser.add_argument("--source", help="limit the run to one source id")
|
||||||
|
return parser.parse_args(argv)
|
||||||
|
|
||||||
|
|
||||||
|
def main(argv: list[str] | None = None) -> int:
|
||||||
|
args = parse_args(argv)
|
||||||
|
try:
|
||||||
|
config = load_config()
|
||||||
|
except ConfigError as exc:
|
||||||
|
log(f"WARN {exc}")
|
||||||
|
print(f"config error: {exc}", file=sys.stderr)
|
||||||
|
return 1
|
||||||
|
|
||||||
|
WIKI_DIR.mkdir(parents=True, exist_ok=True)
|
||||||
|
if not acquire_lock():
|
||||||
|
return 0
|
||||||
|
try:
|
||||||
|
if args.full:
|
||||||
|
log("full reindex requested")
|
||||||
|
with store.transaction(db_path()) as conn:
|
||||||
|
store.reset_index(conn)
|
||||||
|
return run(config, args)
|
||||||
|
finally:
|
||||||
|
release_lock()
|
||||||
|
|
||||||
|
|
||||||
|
if __name__ == "__main__":
|
||||||
|
sys.exit(main())
|
||||||
141
skills/wiki/tests/conftest.py
Normal file
141
skills/wiki/tests/conftest.py
Normal file
@@ -0,0 +1,141 @@
|
|||||||
|
import hashlib
|
||||||
|
import math
|
||||||
|
import sys
|
||||||
|
from dataclasses import dataclass
|
||||||
|
from pathlib import Path
|
||||||
|
|
||||||
|
import pytest
|
||||||
|
|
||||||
|
# The modules under test live in the sibling scripts/ directory.
|
||||||
|
sys.path.insert(0, str(Path(__file__).parent.parent / "scripts"))
|
||||||
|
|
||||||
|
import wiki_config
|
||||||
|
import wiki_search
|
||||||
|
import wiki_sync
|
||||||
|
from wiki_db import EMBEDDING_DIMS
|
||||||
|
|
||||||
|
CONFIG_TEMPLATE = """
|
||||||
|
embedding:
|
||||||
|
endpoint: http://embed.invalid:11434
|
||||||
|
model: qwen3-embedding:0.6b
|
||||||
|
dims: 1024
|
||||||
|
batch: 4
|
||||||
|
keep_alive: -1
|
||||||
|
query_prefix: "Instruct: task\\nQuery: "
|
||||||
|
|
||||||
|
sources:
|
||||||
|
{sources}
|
||||||
|
"""
|
||||||
|
|
||||||
|
WORKSPACE_SOURCE = """ workspace:
|
||||||
|
kind: workspace
|
||||||
|
paths:
|
||||||
|
- "notes/**"
|
||||||
|
- "develop/**"
|
||||||
|
include: ["*.md"]
|
||||||
|
exclude:
|
||||||
|
- "**/inbox/**"
|
||||||
|
- "develop/history.md"
|
||||||
|
"""
|
||||||
|
|
||||||
|
|
||||||
|
@dataclass
|
||||||
|
class WikiEnv:
|
||||||
|
workspace: Path
|
||||||
|
wiki_dir: Path
|
||||||
|
config_path: Path
|
||||||
|
db_path: Path
|
||||||
|
log_path: Path
|
||||||
|
|
||||||
|
def write_config(self, sources: str = WORKSPACE_SOURCE) -> None:
|
||||||
|
self.config_path.write_text(CONFIG_TEMPLATE.format(sources=sources), encoding="utf-8")
|
||||||
|
|
||||||
|
def write_file(self, rel_path: str, text: str) -> Path:
|
||||||
|
path = self.workspace / rel_path
|
||||||
|
path.parent.mkdir(parents=True, exist_ok=True)
|
||||||
|
path.write_text(text, encoding="utf-8")
|
||||||
|
return path
|
||||||
|
|
||||||
|
def log_text(self) -> str:
|
||||||
|
return self.log_path.read_text(encoding="utf-8") if self.log_path.exists() else ""
|
||||||
|
|
||||||
|
|
||||||
|
@pytest.fixture
|
||||||
|
def wiki_env(tmp_path, monkeypatch) -> WikiEnv:
|
||||||
|
"""Redirect every wiki path at a throwaway workspace."""
|
||||||
|
workspace = tmp_path / "workspace"
|
||||||
|
wiki_dir = workspace / "wiki"
|
||||||
|
wiki_dir.mkdir(parents=True)
|
||||||
|
|
||||||
|
env = WikiEnv(
|
||||||
|
workspace=workspace,
|
||||||
|
wiki_dir=wiki_dir,
|
||||||
|
config_path=wiki_dir / "config.yaml",
|
||||||
|
db_path=wiki_dir / "index.sqlite",
|
||||||
|
log_path=workspace / "log" / "wiki_sync.log",
|
||||||
|
)
|
||||||
|
|
||||||
|
monkeypatch.setenv("WIKI_DB", str(env.db_path))
|
||||||
|
monkeypatch.setenv("WIKI_CONFIG", str(env.config_path))
|
||||||
|
monkeypatch.setattr(wiki_config, "WORKSPACE", workspace)
|
||||||
|
monkeypatch.setattr(wiki_config, "WIKI_DIR", wiki_dir)
|
||||||
|
monkeypatch.setattr(wiki_config, "REMOTE_DIR", wiki_dir / "remote")
|
||||||
|
monkeypatch.setattr(wiki_config, "LOCK_PATH", wiki_dir / ".sync.lock")
|
||||||
|
monkeypatch.setattr(wiki_sync, "WIKI_DIR", wiki_dir)
|
||||||
|
monkeypatch.setattr(wiki_sync, "LOCK_PATH", wiki_dir / ".sync.lock")
|
||||||
|
monkeypatch.setattr(wiki_sync, "SYNC_LOG_PATH", env.log_path)
|
||||||
|
|
||||||
|
env.write_config()
|
||||||
|
return env
|
||||||
|
|
||||||
|
|
||||||
|
class FakeEmbedder:
|
||||||
|
"""Bag-of-words vectors: cosine tracks lexical overlap, so ranks are predictable.
|
||||||
|
|
||||||
|
Enough to exercise the KNN and RRF plumbing without a live model. Semantic quality
|
||||||
|
is measured against the real endpoint, not here.
|
||||||
|
"""
|
||||||
|
|
||||||
|
def __init__(self, config):
|
||||||
|
self.config = config
|
||||||
|
self.calls: list[list[str]] = []
|
||||||
|
|
||||||
|
def embed_documents(self, texts):
|
||||||
|
self.calls.append(list(texts))
|
||||||
|
return [self._vector(text) for text in texts]
|
||||||
|
|
||||||
|
def embed_query(self, text):
|
||||||
|
return self._vector(self.config.query_prefix + text)
|
||||||
|
|
||||||
|
def probe(self):
|
||||||
|
return None
|
||||||
|
|
||||||
|
def _vector(self, text: str) -> list[float]:
|
||||||
|
vector = [0.0] * EMBEDDING_DIMS
|
||||||
|
for word in _words(text):
|
||||||
|
digest = hashlib.sha256(word.encode("utf-8")).digest()
|
||||||
|
vector[int.from_bytes(digest[:4], "big") % EMBEDDING_DIMS] += 1.0
|
||||||
|
norm = math.sqrt(sum(value * value for value in vector))
|
||||||
|
if norm == 0.0:
|
||||||
|
vector[0] = 1.0
|
||||||
|
return vector
|
||||||
|
return [value / norm for value in vector]
|
||||||
|
|
||||||
|
|
||||||
|
def _words(text: str) -> list[str]:
|
||||||
|
return [word for word in "".join(c.lower() if c.isalnum() else " " for c in text).split() if len(word) > 2]
|
||||||
|
|
||||||
|
|
||||||
|
@pytest.fixture
|
||||||
|
def fake_embedder(monkeypatch):
|
||||||
|
"""Swap the Ollama client for the deterministic fake in both entry points."""
|
||||||
|
created: list[FakeEmbedder] = []
|
||||||
|
|
||||||
|
def factory(config):
|
||||||
|
embedder = FakeEmbedder(config)
|
||||||
|
created.append(embedder)
|
||||||
|
return embedder
|
||||||
|
|
||||||
|
monkeypatch.setattr(wiki_sync, "OllamaEmbedder", factory)
|
||||||
|
monkeypatch.setattr(wiki_search, "OllamaEmbedder", factory)
|
||||||
|
return created
|
||||||
209
skills/wiki/tests/test_wiki_chunker.py
Normal file
209
skills/wiki/tests/test_wiki_chunker.py
Normal file
@@ -0,0 +1,209 @@
|
|||||||
|
"""Chunker policy: heading sections, breadcrumbs, sibling merge, block-aligned split."""
|
||||||
|
|
||||||
|
import sys
|
||||||
|
from pathlib import Path
|
||||||
|
|
||||||
|
SCRIPTS = Path(__file__).parent.parent / "scripts"
|
||||||
|
sys.path.insert(0, str(SCRIPTS))
|
||||||
|
|
||||||
|
from wiki_chunker import MERGE_BELOW, OVERLAP, SPLIT_ABOVE, parse_markdown, token_estimate
|
||||||
|
|
||||||
|
FILLER = "Tohle je odstavec s dostatkem textu na to, aby se do velikosti chunku počítal. "
|
||||||
|
|
||||||
|
|
||||||
|
def _paragraph(tokens: int) -> str:
|
||||||
|
return (FILLER * (1 + tokens * 4 // len(FILLER)))[: tokens * 4]
|
||||||
|
|
||||||
|
|
||||||
|
def _crumbs(parsed):
|
||||||
|
return [chunk.breadcrumb for chunk in parsed.chunks]
|
||||||
|
|
||||||
|
|
||||||
|
def test_heading_hierarchy_builds_breadcrumbs():
|
||||||
|
doc = "\n".join(
|
||||||
|
[
|
||||||
|
"# Kořen",
|
||||||
|
"",
|
||||||
|
_paragraph(MERGE_BELOW),
|
||||||
|
"",
|
||||||
|
"## Sekce A",
|
||||||
|
"",
|
||||||
|
_paragraph(MERGE_BELOW),
|
||||||
|
"",
|
||||||
|
"### Podsekce",
|
||||||
|
"",
|
||||||
|
_paragraph(MERGE_BELOW),
|
||||||
|
"",
|
||||||
|
"## Sekce B",
|
||||||
|
"",
|
||||||
|
_paragraph(MERGE_BELOW),
|
||||||
|
]
|
||||||
|
)
|
||||||
|
parsed = parse_markdown(doc, "notes/doc.md")
|
||||||
|
assert _crumbs(parsed) == [
|
||||||
|
"notes/doc.md > Kořen",
|
||||||
|
"notes/doc.md > Kořen > Sekce A",
|
||||||
|
"notes/doc.md > Kořen > Sekce A > Podsekce",
|
||||||
|
"notes/doc.md > Kořen > Sekce B",
|
||||||
|
]
|
||||||
|
assert parsed.headings == ["Kořen", "Sekce A", "Podsekce", "Sekce B"]
|
||||||
|
|
||||||
|
|
||||||
|
def test_frontmatter_title_and_tags():
|
||||||
|
doc = "---\ntitle: Tokyo metro tips\ntags: [transit, jr-pass]\n---\n\n" + _paragraph(MERGE_BELOW)
|
||||||
|
parsed = parse_markdown(doc, "japan/metro.md")
|
||||||
|
assert parsed.title == "Tokyo metro tips"
|
||||||
|
assert parsed.tags == ["transit", "jr-pass"]
|
||||||
|
# Only the title reaches the embedded text; tags stay metadata for filtering.
|
||||||
|
assert _crumbs(parsed) == ["japan/metro.md > Tokyo metro tips"]
|
||||||
|
assert "transit" not in parsed.chunks[0].text
|
||||||
|
|
||||||
|
|
||||||
|
def test_frontmatter_title_matching_h1_is_not_duplicated():
|
||||||
|
doc = "---\ntitle: Zálohy\n---\n\n# Zálohy\n\n" + _paragraph(MERGE_BELOW)
|
||||||
|
parsed = parse_markdown(doc, "a.md")
|
||||||
|
assert _crumbs(parsed) == ["a.md > Zálohy"]
|
||||||
|
|
||||||
|
|
||||||
|
def test_comma_separated_tags():
|
||||||
|
doc = "---\ntags: gear, safety\n---\n\ntext"
|
||||||
|
assert parse_markdown(doc, "a.md").tags == ["gear", "safety"]
|
||||||
|
|
||||||
|
|
||||||
|
def test_malformed_frontmatter_stays_body():
|
||||||
|
"""Unparseable frontmatter is not consumed — it stays body text, tags included."""
|
||||||
|
doc = "---\ntitle: [unclosed\ntags: [a\n---\n\ntext"
|
||||||
|
parsed = parse_markdown(doc, "a.md")
|
||||||
|
assert parsed.tags == []
|
||||||
|
assert "title: [unclosed" in parsed.chunks[-1].text # the raw block survived as prose
|
||||||
|
|
||||||
|
|
||||||
|
def test_small_adjacent_siblings_merge():
|
||||||
|
doc = "\n".join(["# Root", "", "## A", "", "krátké A", "", "## B", "", "krátké B"])
|
||||||
|
parsed = parse_markdown(doc, "a.md")
|
||||||
|
# A and B share the parent `a.md > Root` and are both under MERGE_BELOW.
|
||||||
|
# `# Root` carries no prose of its own, so it contributes no chunk.
|
||||||
|
assert _crumbs(parsed) == ["a.md > Root > A"]
|
||||||
|
assert "krátké A" in parsed.chunks[0].text
|
||||||
|
assert "krátké B" in parsed.chunks[0].text
|
||||||
|
|
||||||
|
|
||||||
|
def test_heading_without_own_prose_yields_no_chunk():
|
||||||
|
"""`# Titul` immediately followed by `## Sekce` must not embed a bare title."""
|
||||||
|
doc = "\n".join(["# Titul", "", "## Sekce", "", _paragraph(MERGE_BELOW)])
|
||||||
|
parsed = parse_markdown(doc, "a.md")
|
||||||
|
assert _crumbs(parsed) == ["a.md > Titul > Sekce"]
|
||||||
|
# The dropped heading still reaches the index through the breadcrumb and headings list.
|
||||||
|
assert parsed.headings == ["Titul", "Sekce"]
|
||||||
|
|
||||||
|
|
||||||
|
def test_merge_does_not_cross_parents():
|
||||||
|
"""A small H3 must not be glued onto the next H2 — different parents."""
|
||||||
|
doc = "\n".join(["# Root", "", "## A", "", "### A1", "", "malé", "", "## B", "", "malé B"])
|
||||||
|
parsed = parse_markdown(doc, "a.md")
|
||||||
|
assert "a.md > Root > A > A1" in _crumbs(parsed)
|
||||||
|
a1 = next(c for c in parsed.chunks if c.breadcrumb.endswith("A1"))
|
||||||
|
assert "malé B" not in a1.text
|
||||||
|
|
||||||
|
|
||||||
|
def test_merge_never_exceeds_split_threshold():
|
||||||
|
big = _paragraph(SPLIT_ABOVE - 100)
|
||||||
|
doc = "\n".join(["# Root", "", "## A", "", "malé", "", "## B", "", big])
|
||||||
|
parsed = parse_markdown(doc, "a.md")
|
||||||
|
for chunk in parsed.chunks:
|
||||||
|
assert token_estimate(chunk.text) <= SPLIT_ABOVE + token_estimate(chunk.breadcrumb) + 1
|
||||||
|
|
||||||
|
|
||||||
|
def test_h4_stays_inside_its_h3_section():
|
||||||
|
doc = "\n".join(["# Root", "", "### Trojka", "", "text", "", "#### Čtyřka", "", "hluboký text"])
|
||||||
|
parsed = parse_markdown(doc, "a.md")
|
||||||
|
assert "a.md > Root > Trojka" in _crumbs(parsed)
|
||||||
|
assert not any("Čtyřka" in crumb for crumb in _crumbs(parsed))
|
||||||
|
section = next(c for c in parsed.chunks if c.breadcrumb.endswith("Trojka"))
|
||||||
|
assert "hluboký text" in section.text
|
||||||
|
|
||||||
|
|
||||||
|
def test_preamble_before_first_heading_is_kept():
|
||||||
|
doc = "úvodní odstavec bez nadpisu\n\n" + "# Root\n\n" + _paragraph(MERGE_BELOW)
|
||||||
|
parsed = parse_markdown(doc, "a.md")
|
||||||
|
assert "úvodní odstavec bez nadpisu" in parsed.chunks[0].text
|
||||||
|
|
||||||
|
|
||||||
|
def test_document_without_headings_is_one_chunk():
|
||||||
|
parsed = parse_markdown(_paragraph(MERGE_BELOW), "a.md")
|
||||||
|
assert _crumbs(parsed) == ["a.md"]
|
||||||
|
|
||||||
|
|
||||||
|
def test_empty_document_yields_no_chunks():
|
||||||
|
assert parse_markdown("", "a.md").chunks == []
|
||||||
|
assert parse_markdown(" \n\n \n", "a.md").chunks == []
|
||||||
|
|
||||||
|
|
||||||
|
def test_setext_heading_is_a_section():
|
||||||
|
doc = "Nadpis setext\n=============\n\n" + _paragraph(MERGE_BELOW)
|
||||||
|
parsed = parse_markdown(doc, "a.md")
|
||||||
|
assert _crumbs(parsed) == ["a.md > Nadpis setext"]
|
||||||
|
|
||||||
|
|
||||||
|
def test_oversized_section_splits_and_keeps_code_fence_whole():
|
||||||
|
fence = "```python\n" + "\n".join(f"value_{i} = {i} # a comment long enough to count" for i in range(60)) + "\n```"
|
||||||
|
doc = "# Velká sekce\n\n" + "\n\n".join([_paragraph(120)] * 6) + "\n\n" + fence + "\n\n" + _paragraph(120)
|
||||||
|
parsed = parse_markdown(doc, "big.md")
|
||||||
|
|
||||||
|
assert len(parsed.chunks) > 1
|
||||||
|
assert {c.breadcrumb for c in parsed.chunks} == {"big.md > Velká sekce"}
|
||||||
|
# The fence is larger than SPLIT_ABOVE on its own, so it must sit alone and unbroken.
|
||||||
|
assert sum(fence in chunk.text for chunk in parsed.chunks) == 1
|
||||||
|
|
||||||
|
|
||||||
|
def test_split_carries_block_aligned_overlap():
|
||||||
|
blocks = [_paragraph(40) + f" marker{i:03d}" for i in range(40)]
|
||||||
|
doc = "# Root\n\n" + "\n\n".join(blocks)
|
||||||
|
parsed = parse_markdown(doc, "a.md")
|
||||||
|
|
||||||
|
assert len(parsed.chunks) > 1
|
||||||
|
# Each boundary repeats at least one whole block, capped at OVERLAP tokens.
|
||||||
|
for earlier, later in zip(parsed.chunks, parsed.chunks[1:], strict=False):
|
||||||
|
shared = [b for b in blocks if b in earlier.text and b in later.text]
|
||||||
|
assert shared, "expected overlap blocks between consecutive chunks"
|
||||||
|
assert token_estimate("".join(shared)) <= OVERLAP
|
||||||
|
|
||||||
|
|
||||||
|
def test_table_is_not_broken():
|
||||||
|
table = "\n".join(["| a | b |", "|---|---|"] + [f"| {i} | {i * 2} |" for i in range(80)])
|
||||||
|
doc = "# Root\n\n" + "\n\n".join([_paragraph(150)] * 5) + "\n\n" + table
|
||||||
|
parsed = parse_markdown(doc, "a.md")
|
||||||
|
assert sum(table in chunk.text for chunk in parsed.chunks) == 1
|
||||||
|
|
||||||
|
|
||||||
|
def test_breadcrumb_is_prefixed_to_chunk_text():
|
||||||
|
parsed = parse_markdown("# Root\n\n" + _paragraph(MERGE_BELOW), "notes/a.md")
|
||||||
|
chunk = parsed.chunks[0]
|
||||||
|
assert chunk.text.startswith(chunk.breadcrumb + "\n\n")
|
||||||
|
|
||||||
|
|
||||||
|
def test_catalog_title_falls_back_to_first_heading():
|
||||||
|
"""Notes carry their title as `# H1` far more often than as frontmatter."""
|
||||||
|
doc = "# Zálohování dat\n\n" + _paragraph(MERGE_BELOW) + "\n\n## Retence\n\n" + _paragraph(MERGE_BELOW)
|
||||||
|
parsed = parse_markdown(doc, "notes/zalohy.md")
|
||||||
|
assert parsed.title == "Zálohování dat"
|
||||||
|
|
||||||
|
|
||||||
|
def test_frontmatter_title_wins_over_first_heading():
|
||||||
|
doc = "---\ntitle: Z frontmatteru\n---\n\n# Z nadpisu\n\n" + _paragraph(MERGE_BELOW)
|
||||||
|
assert parse_markdown(doc, "a.md").title == "Z frontmatteru"
|
||||||
|
|
||||||
|
|
||||||
|
def test_document_without_headings_has_no_title():
|
||||||
|
assert parse_markdown(_paragraph(MERGE_BELOW), "a.md").title is None
|
||||||
|
assert parse_markdown("", "a.md").title is None
|
||||||
|
|
||||||
|
|
||||||
|
def test_heading_title_fallback_does_not_change_any_chunk():
|
||||||
|
"""The fallback feeds only the catalog — touching the breadcrumb root would force --full."""
|
||||||
|
doc = "úvod bez nadpisu\n\n# Root\n\n" + _paragraph(MERGE_BELOW) + "\n\n## Sekce\n\n" + _paragraph(MERGE_BELOW)
|
||||||
|
parsed = parse_markdown(doc, "notes/a.md")
|
||||||
|
|
||||||
|
assert parsed.title == "Root"
|
||||||
|
# The root stays the bare path, so no chunk text is rewritten by the fallback.
|
||||||
|
assert _crumbs(parsed) == ["notes/a.md", "notes/a.md > Root", "notes/a.md > Root > Sekce"]
|
||||||
165
skills/wiki/tests/test_wiki_config.py
Normal file
165
skills/wiki/tests/test_wiki_config.py
Normal file
@@ -0,0 +1,165 @@
|
|||||||
|
"""Scope precedence (paths -> include -> exclude) and config validation."""
|
||||||
|
|
||||||
|
import sys
|
||||||
|
from pathlib import Path
|
||||||
|
|
||||||
|
import pytest
|
||||||
|
|
||||||
|
SCRIPTS = Path(__file__).parent.parent / "scripts"
|
||||||
|
sys.path.insert(0, str(SCRIPTS))
|
||||||
|
|
||||||
|
import wiki_config
|
||||||
|
from wiki_config import ConfigError, SourceConfig, load_config, source_root
|
||||||
|
|
||||||
|
WORKSPACE_YAML = """
|
||||||
|
embedding:
|
||||||
|
endpoint: http://nvidia.hell:11434/
|
||||||
|
model: qwen3-embedding:0.6b
|
||||||
|
dims: 1024
|
||||||
|
batch: 32
|
||||||
|
keep_alive: -1
|
||||||
|
query_prefix: "Instruct: task\\nQuery: "
|
||||||
|
|
||||||
|
sources:
|
||||||
|
index:
|
||||||
|
kind: git
|
||||||
|
url: git@git.fnet.cz:lachtan/index.git
|
||||||
|
paths: ["**"]
|
||||||
|
include: ["*.md"]
|
||||||
|
exclude:
|
||||||
|
- "**/node_modules/**"
|
||||||
|
- "**/vendor/**"
|
||||||
|
|
||||||
|
workspace:
|
||||||
|
kind: workspace
|
||||||
|
paths:
|
||||||
|
- "notes/**"
|
||||||
|
- "develop/**"
|
||||||
|
include: ["*.md"]
|
||||||
|
exclude:
|
||||||
|
- "**/inbox/**"
|
||||||
|
- "develop/history.md"
|
||||||
|
"""
|
||||||
|
|
||||||
|
|
||||||
|
def _write(tmp_path, text):
|
||||||
|
path = tmp_path / "config.yaml"
|
||||||
|
path.write_text(text, encoding="utf-8")
|
||||||
|
return path
|
||||||
|
|
||||||
|
|
||||||
|
def _source(
|
||||||
|
source_id: str = "ws",
|
||||||
|
kind: str = "workspace",
|
||||||
|
url: str | None = None,
|
||||||
|
paths: tuple[str, ...] = ("**",),
|
||||||
|
include: tuple[str, ...] = ("*.md",),
|
||||||
|
exclude: tuple[str, ...] = (),
|
||||||
|
) -> SourceConfig:
|
||||||
|
return SourceConfig(source_id=source_id, kind=kind, url=url, paths=paths, include=include, exclude=exclude)
|
||||||
|
|
||||||
|
|
||||||
|
def test_loads_embedding_and_sources(tmp_path):
|
||||||
|
config = load_config(_write(tmp_path, WORKSPACE_YAML))
|
||||||
|
assert config.embedding.endpoint == "http://nvidia.hell:11434" # trailing slash trimmed
|
||||||
|
assert config.embedding.model == "qwen3-embedding:0.6b"
|
||||||
|
assert config.embedding.dims == 1024
|
||||||
|
assert config.embedding.keep_alive == -1
|
||||||
|
assert config.embedding.query_prefix.endswith("Query: ")
|
||||||
|
assert [s.source_id for s in config.sources] == ["index", "workspace"]
|
||||||
|
index = config.source("index")
|
||||||
|
assert index is not None and index.kind == "git"
|
||||||
|
assert config.source("nope") is None
|
||||||
|
|
||||||
|
|
||||||
|
def test_string_keep_alive_is_rejected():
|
||||||
|
"""Ollama answers HTTP 400 to a string "-1" — catch it in config, not at runtime."""
|
||||||
|
with pytest.raises(ConfigError, match="keep_alive"):
|
||||||
|
wiki_config._parse_embedding({"endpoint": "http://x", "model": "m", "dims": 1024, "keep_alive": "-1"})
|
||||||
|
|
||||||
|
|
||||||
|
def test_missing_config_file(tmp_path):
|
||||||
|
with pytest.raises(ConfigError, match="missing config"):
|
||||||
|
load_config(tmp_path / "nope.yaml")
|
||||||
|
|
||||||
|
|
||||||
|
def test_git_source_needs_url(tmp_path):
|
||||||
|
yaml_text = WORKSPACE_YAML.replace(" url: git@git.fnet.cz:lachtan/index.git\n", "")
|
||||||
|
with pytest.raises(ConfigError, match="need a url"):
|
||||||
|
load_config(_write(tmp_path, yaml_text))
|
||||||
|
|
||||||
|
|
||||||
|
def test_unknown_kind_is_rejected(tmp_path):
|
||||||
|
yaml_text = WORKSPACE_YAML.replace(" kind: workspace", " kind: mirror")
|
||||||
|
with pytest.raises(ConfigError, match="kind must be one of"):
|
||||||
|
load_config(_write(tmp_path, yaml_text))
|
||||||
|
|
||||||
|
|
||||||
|
def test_paths_is_required(tmp_path):
|
||||||
|
yaml_text = WORKSPACE_YAML.replace(' paths: ["**"]\n', "")
|
||||||
|
with pytest.raises(ConfigError, match="`paths` is required"):
|
||||||
|
load_config(_write(tmp_path, yaml_text))
|
||||||
|
|
||||||
|
|
||||||
|
def test_paths_whitelist_gates_everything():
|
||||||
|
source = _source(paths=("notes/**", "develop/**"))
|
||||||
|
assert source.covers("notes/a.md")
|
||||||
|
assert source.covers("notes/deep/nested/a.md")
|
||||||
|
assert source.covers("develop/knowledge.md")
|
||||||
|
# Not on the whitelist -> does not exist for the index.
|
||||||
|
assert not source.covers("tmp/a.md")
|
||||||
|
assert not source.covers("skills/wiki/SKILL.md")
|
||||||
|
assert not source.covers("AGENTS.md")
|
||||||
|
|
||||||
|
|
||||||
|
def test_include_filters_extensions():
|
||||||
|
source = _source(paths=("**",))
|
||||||
|
assert source.covers("notes/a.md")
|
||||||
|
assert not source.covers("notes/main.py")
|
||||||
|
assert not source.covers("memory/history.jsonl")
|
||||||
|
assert not source.covers("assets/photo.png")
|
||||||
|
|
||||||
|
|
||||||
|
def test_exclude_wins_over_paths_and_include():
|
||||||
|
source = _source(paths=("notes/**", "develop/**"), exclude=("**/inbox/**", "develop/history.md"))
|
||||||
|
assert not source.covers("notes/inbox/raw.md")
|
||||||
|
assert not source.covers("develop/history.md")
|
||||||
|
assert source.covers("develop/knowledge.md")
|
||||||
|
assert source.covers("notes/notes.md")
|
||||||
|
|
||||||
|
|
||||||
|
def test_double_star_matches_whole_repo():
|
||||||
|
source = _source(paths=("**",), exclude=("**/node_modules/**", "**/vendor/**"))
|
||||||
|
assert source.covers("README.md")
|
||||||
|
assert source.covers("japan/tokyo/metro.md")
|
||||||
|
assert not source.covers("node_modules/pkg/README.md")
|
||||||
|
assert not source.covers("web/vendor/lib/CHANGELOG.md")
|
||||||
|
|
||||||
|
|
||||||
|
def test_glob_star_does_not_cross_a_slash():
|
||||||
|
source = _source(paths=("notes/*",))
|
||||||
|
assert source.covers("notes/a.md")
|
||||||
|
assert not source.covers("notes/deep/a.md")
|
||||||
|
|
||||||
|
|
||||||
|
def test_covers_dir_prunes_the_walk():
|
||||||
|
source = _source(paths=("notes/**", "develop/**"), exclude=("**/inbox/**",))
|
||||||
|
assert source.covers_dir("")
|
||||||
|
assert source.covers_dir("notes")
|
||||||
|
assert source.covers_dir("notes/deep")
|
||||||
|
assert source.covers_dir("develop")
|
||||||
|
assert not source.covers_dir("tmp")
|
||||||
|
assert not source.covers_dir("notes/inbox")
|
||||||
|
|
||||||
|
|
||||||
|
def test_covers_dir_never_prunes_a_double_star_source():
|
||||||
|
source = _source(paths=("**",), exclude=("**/node_modules/**",))
|
||||||
|
assert source.covers_dir("anything/deep")
|
||||||
|
assert not source.covers_dir("app/node_modules")
|
||||||
|
|
||||||
|
|
||||||
|
def test_source_root_derives_clone_path_from_id():
|
||||||
|
assert source_root(_source(source_id="travel", kind="git", url="git@x:y.git")) == (
|
||||||
|
wiki_config.REMOTE_DIR / "travel"
|
||||||
|
)
|
||||||
|
assert source_root(_source(kind="workspace")) == wiki_config.WORKSPACE
|
||||||
136
skills/wiki/tests/test_wiki_db.py
Normal file
136
skills/wiki/tests/test_wiki_db.py
Normal file
@@ -0,0 +1,136 @@
|
|||||||
|
"""Schema-level guarantees: the cascade, and the one hole the cascade leaves."""
|
||||||
|
|
||||||
|
import sys
|
||||||
|
from pathlib import Path
|
||||||
|
|
||||||
|
SCRIPTS = Path(__file__).parent.parent / "scripts"
|
||||||
|
sys.path.insert(0, str(SCRIPTS))
|
||||||
|
|
||||||
|
import wiki_store as store
|
||||||
|
from wiki_db import EMBEDDING_DIMS
|
||||||
|
|
||||||
|
NOW = "2026-09-09T12:00:00+00:00"
|
||||||
|
|
||||||
|
|
||||||
|
def _seed_file(conn, source_id, path, chunk_texts):
|
||||||
|
store.upsert_source(conn, source_id, "workspace")
|
||||||
|
store.upsert_file(conn, source_id, path, "T", ["t"], ["H"], "sha", 10, 1.0, NOW)
|
||||||
|
store.replace_chunks(conn, source_id, path, [(f"{path} > s", t) for t in chunk_texts])
|
||||||
|
for row in store.pending_chunks(conn, 100):
|
||||||
|
store.store_embedding(conn, row["id"], [0.1] * EMBEDDING_DIMS, NOW)
|
||||||
|
|
||||||
|
|
||||||
|
def test_delete_file_leaves_no_orphan_vectors(tmp_path):
|
||||||
|
"""The FK cascade reaches chunks and chunks_fts but never vec0 — deletes must be explicit."""
|
||||||
|
db_path = tmp_path / "index.sqlite"
|
||||||
|
with store.transaction(db_path) as conn:
|
||||||
|
_seed_file(conn, "ws", "a.md", ["alpha text", "beta text"])
|
||||||
|
_seed_file(conn, "ws", "b.md", ["gamma text"])
|
||||||
|
|
||||||
|
with store.connection(db_path) as conn:
|
||||||
|
assert store.index_stats(conn) == {"files": 2, "chunks": 3, "vectors": 3, "pending": 0}
|
||||||
|
|
||||||
|
with store.transaction(db_path) as conn:
|
||||||
|
store.delete_file(conn, "ws", "a.md")
|
||||||
|
|
||||||
|
with store.connection(db_path) as conn:
|
||||||
|
assert store.index_stats(conn) == {"files": 1, "chunks": 1, "vectors": 1, "pending": 0}
|
||||||
|
assert store.orphan_vector_ids(conn) == []
|
||||||
|
assert store.bm25_ranked_ids(conn, "alpha", 10) == []
|
||||||
|
assert len(store.bm25_ranked_ids(conn, "gamma", 10)) == 1
|
||||||
|
|
||||||
|
|
||||||
|
def test_replace_chunks_drops_old_vectors(tmp_path):
|
||||||
|
db_path = tmp_path / "index.sqlite"
|
||||||
|
with store.transaction(db_path) as conn:
|
||||||
|
_seed_file(conn, "ws", "a.md", ["alpha text", "beta text", "delta text"])
|
||||||
|
|
||||||
|
with store.transaction(db_path) as conn:
|
||||||
|
store.replace_chunks(conn, "ws", "a.md", [("a.md > s", "only one now")])
|
||||||
|
|
||||||
|
with store.connection(db_path) as conn:
|
||||||
|
assert store.index_stats(conn) == {"files": 1, "chunks": 1, "vectors": 0, "pending": 1}
|
||||||
|
assert store.orphan_vector_ids(conn) == []
|
||||||
|
assert store.bm25_ranked_ids(conn, "alpha", 10) == []
|
||||||
|
|
||||||
|
|
||||||
|
def test_delete_by_path_does_not_touch_other_source(tmp_path):
|
||||||
|
"""The key is (source_id, path); a path-only delete would eat a foreign source's chunks."""
|
||||||
|
db_path = tmp_path / "index.sqlite"
|
||||||
|
with store.transaction(db_path) as conn:
|
||||||
|
_seed_file(conn, "ws", "notes.md", ["shared name one"])
|
||||||
|
_seed_file(conn, "git", "notes.md", ["shared name two"])
|
||||||
|
|
||||||
|
with store.transaction(db_path) as conn:
|
||||||
|
store.delete_file(conn, "ws", "notes.md")
|
||||||
|
|
||||||
|
with store.connection(db_path) as conn:
|
||||||
|
assert store.index_stats(conn)["chunks"] == 1
|
||||||
|
assert store.orphan_vector_ids(conn) == []
|
||||||
|
remaining = list(conn.execute("SELECT source_id FROM chunks"))
|
||||||
|
assert remaining[0]["source_id"] == "git"
|
||||||
|
|
||||||
|
|
||||||
|
def test_embedded_at_update_keeps_fts_row(tmp_path):
|
||||||
|
"""Step 4b flips embedded_at on unchanged text; the WHEN guard must keep FTS intact."""
|
||||||
|
db_path = tmp_path / "index.sqlite"
|
||||||
|
with store.transaction(db_path) as conn:
|
||||||
|
store.upsert_source(conn, "ws", "workspace")
|
||||||
|
store.upsert_file(conn, "ws", "a.md", "T", [], [], "sha", 10, 1.0, NOW)
|
||||||
|
store.replace_chunks(conn, "ws", "a.md", [("a.md", "zaloha dat na disk")])
|
||||||
|
|
||||||
|
with store.connection(db_path) as conn:
|
||||||
|
assert len(store.bm25_ranked_ids(conn, "zaloha", 10)) == 1
|
||||||
|
|
||||||
|
with store.transaction(db_path) as conn:
|
||||||
|
pending = store.pending_chunks(conn, 10)
|
||||||
|
store.store_embedding(conn, pending[0]["id"], [0.2] * EMBEDDING_DIMS, NOW)
|
||||||
|
|
||||||
|
with store.connection(db_path) as conn:
|
||||||
|
assert len(store.bm25_ranked_ids(conn, "zaloha", 10)) == 1
|
||||||
|
assert store.index_stats(conn) == {"files": 1, "chunks": 1, "vectors": 1, "pending": 0}
|
||||||
|
|
||||||
|
|
||||||
|
def test_fts_folds_czech_diacritics(tmp_path):
|
||||||
|
"""remove_diacritics 2 is what makes `zaloha` find `záloha`."""
|
||||||
|
db_path = tmp_path / "index.sqlite"
|
||||||
|
with store.transaction(db_path) as conn:
|
||||||
|
store.upsert_source(conn, "ws", "workspace")
|
||||||
|
store.upsert_file(conn, "ws", "a.md", "T", [], [], "sha", 10, 1.0, NOW)
|
||||||
|
store.replace_chunks(conn, "ws", "a.md", [("a.md", "zálohování dat probíhá denně")])
|
||||||
|
|
||||||
|
with store.connection(db_path) as conn:
|
||||||
|
assert len(store.bm25_ranked_ids(conn, "zaloh*", 10)) == 1
|
||||||
|
assert len(store.bm25_ranked_ids(conn, "záloh*", 10)) == 1
|
||||||
|
|
||||||
|
|
||||||
|
def test_meta_roundtrip_and_rollback(tmp_path):
|
||||||
|
db_path = tmp_path / "index.sqlite"
|
||||||
|
with store.transaction(db_path) as conn:
|
||||||
|
store.write_meta(conn, {"embedding_model": "qwen3-embedding:0.6b", "embedding_dims": "1024"})
|
||||||
|
|
||||||
|
with store.connection(db_path) as conn:
|
||||||
|
assert store.read_meta(conn)["embedding_dims"] == "1024"
|
||||||
|
|
||||||
|
try:
|
||||||
|
with store.transaction(db_path) as conn:
|
||||||
|
store.write_meta(conn, {"embedding_dims": "768"})
|
||||||
|
raise RuntimeError("boom")
|
||||||
|
except RuntimeError:
|
||||||
|
pass
|
||||||
|
|
||||||
|
with store.connection(db_path) as conn:
|
||||||
|
assert store.read_meta(conn)["embedding_dims"] == "1024"
|
||||||
|
|
||||||
|
|
||||||
|
def test_toc_filters_by_tag(tmp_path):
|
||||||
|
db_path = tmp_path / "index.sqlite"
|
||||||
|
with store.transaction(db_path) as conn:
|
||||||
|
store.upsert_source(conn, "travel", "git")
|
||||||
|
store.upsert_file(conn, "travel", "japan/metro.md", "Metro", ["transit"], [], "s", 1, 1.0, NOW)
|
||||||
|
store.upsert_file(conn, "travel", "alps/gear.md", "Gear", ["gear", "safety"], [], "s", 1, 1.0, NOW)
|
||||||
|
|
||||||
|
with store.connection(db_path) as conn:
|
||||||
|
assert [r["path"] for r in store.list_toc_files(conn, tag="gear")] == ["alps/gear.md"]
|
||||||
|
assert len(store.list_toc_files(conn, source_id="travel")) == 2
|
||||||
|
assert store.list_toc_files(conn, source_id="nope") == []
|
||||||
266
skills/wiki/tests/test_wiki_git.py
Normal file
266
skills/wiki/tests/test_wiki_git.py
Normal file
@@ -0,0 +1,266 @@
|
|||||||
|
"""Git driver: clone, ls-remote change detection, diff, deletions, per-source failure."""
|
||||||
|
|
||||||
|
import subprocess
|
||||||
|
import sys
|
||||||
|
from pathlib import Path
|
||||||
|
|
||||||
|
SCRIPTS = Path(__file__).parent.parent / "scripts"
|
||||||
|
sys.path.insert(0, str(SCRIPTS))
|
||||||
|
|
||||||
|
import wiki_store as store
|
||||||
|
import wiki_sync
|
||||||
|
from conftest import WORKSPACE_SOURCE
|
||||||
|
|
||||||
|
DOC = """# Zálohování
|
||||||
|
|
||||||
|
Záloha běží každou noc přes rsync na druhý disk.
|
||||||
|
"""
|
||||||
|
|
||||||
|
TRAVEL_DOC = """# Tokijské metro
|
||||||
|
|
||||||
|
Z Narity do centra jede Skyliner za osmatřicet minut.
|
||||||
|
"""
|
||||||
|
|
||||||
|
GIT_SOURCE = """ notes:
|
||||||
|
kind: git
|
||||||
|
url: {url}
|
||||||
|
paths: ["**"]
|
||||||
|
include: ["*.md"]
|
||||||
|
exclude:
|
||||||
|
- "**/node_modules/**"
|
||||||
|
- "**/vendor/**"
|
||||||
|
"""
|
||||||
|
|
||||||
|
|
||||||
|
def _git(args, cwd=None):
|
||||||
|
result = subprocess.run(
|
||||||
|
["git", "-c", "user.email=t@t", "-c", "user.name=t", *args],
|
||||||
|
cwd=cwd,
|
||||||
|
capture_output=True,
|
||||||
|
text=True,
|
||||||
|
check=True,
|
||||||
|
)
|
||||||
|
return result.stdout
|
||||||
|
|
||||||
|
|
||||||
|
class Remote:
|
||||||
|
"""A bare repo plus a working clone to push commits from."""
|
||||||
|
|
||||||
|
def __init__(self, root: Path):
|
||||||
|
self.bare = root / "origin.git"
|
||||||
|
self.work = root / "origin-work"
|
||||||
|
_git(["init", "--bare", "--initial-branch=master", str(self.bare)])
|
||||||
|
_git(["clone", str(self.bare), str(self.work)])
|
||||||
|
|
||||||
|
@property
|
||||||
|
def url(self) -> str:
|
||||||
|
return str(self.bare)
|
||||||
|
|
||||||
|
def commit(self, files: dict[str, str | None], message: str = "change") -> str:
|
||||||
|
for rel_path, text in files.items():
|
||||||
|
path = self.work / rel_path
|
||||||
|
if text is None:
|
||||||
|
_git(["rm", "-q", rel_path], cwd=self.work)
|
||||||
|
continue
|
||||||
|
path.parent.mkdir(parents=True, exist_ok=True)
|
||||||
|
path.write_text(text, encoding="utf-8")
|
||||||
|
_git(["add", rel_path], cwd=self.work)
|
||||||
|
_git(["commit", "-m", message], cwd=self.work)
|
||||||
|
_git(["push", "-q", "origin", "master"], cwd=self.work)
|
||||||
|
return _git(["rev-parse", "HEAD"], cwd=self.work).strip()
|
||||||
|
|
||||||
|
def move(self, old: str, new: str) -> str:
|
||||||
|
(self.work / new).parent.mkdir(parents=True, exist_ok=True)
|
||||||
|
_git(["mv", old, new], cwd=self.work)
|
||||||
|
_git(["commit", "-m", "rename"], cwd=self.work)
|
||||||
|
_git(["push", "-q", "origin", "master"], cwd=self.work)
|
||||||
|
return _git(["rev-parse", "HEAD"], cwd=self.work).strip()
|
||||||
|
|
||||||
|
|
||||||
|
def _use_git_source(env, url, with_workspace=False):
|
||||||
|
sources = GIT_SOURCE.format(url=url)
|
||||||
|
if with_workspace:
|
||||||
|
sources += WORKSPACE_SOURCE
|
||||||
|
env.write_config(sources)
|
||||||
|
|
||||||
|
|
||||||
|
def _stats(env):
|
||||||
|
with store.connection(env.db_path) as conn:
|
||||||
|
return store.index_stats(conn)
|
||||||
|
|
||||||
|
|
||||||
|
def test_first_run_clones_and_indexes(wiki_env, fake_embedder, tmp_path):
|
||||||
|
remote = Remote(tmp_path)
|
||||||
|
head = remote.commit({"zalohy.md": DOC, "japan/metro.md": TRAVEL_DOC})
|
||||||
|
_use_git_source(wiki_env, remote.url)
|
||||||
|
|
||||||
|
assert wiki_sync.main([]) == 0
|
||||||
|
|
||||||
|
clone = wiki_env.wiki_dir / "remote" / "notes"
|
||||||
|
assert (clone / "zalohy.md").is_file() # non-bare: a working tree grep can read
|
||||||
|
assert (clone / ".git").is_dir()
|
||||||
|
|
||||||
|
with store.connection(wiki_env.db_path) as conn:
|
||||||
|
assert sorted(store.list_source_files(conn, "notes")) == ["japan/metro.md", "zalohy.md"]
|
||||||
|
source = store.get_source(conn, "notes")
|
||||||
|
assert source is not None
|
||||||
|
assert source["indexed_rev"] == head
|
||||||
|
assert source["kind"] == "git"
|
||||||
|
stats = _stats(wiki_env)
|
||||||
|
assert stats["vectors"] == stats["chunks"] > 0
|
||||||
|
|
||||||
|
|
||||||
|
def test_second_run_over_the_same_revision_changes_nothing(wiki_env, fake_embedder, tmp_path):
|
||||||
|
"""Verification 2 for the git driver."""
|
||||||
|
remote = Remote(tmp_path)
|
||||||
|
remote.commit({"zalohy.md": DOC})
|
||||||
|
_use_git_source(wiki_env, remote.url)
|
||||||
|
assert wiki_sync.main([]) == 0
|
||||||
|
|
||||||
|
before = _stats(wiki_env)
|
||||||
|
log_before = wiki_env.log_text()
|
||||||
|
|
||||||
|
assert wiki_sync.main([]) == 0
|
||||||
|
|
||||||
|
assert _stats(wiki_env) == before
|
||||||
|
assert wiki_env.log_text() == log_before # ls-remote matched, so not even a fetch
|
||||||
|
|
||||||
|
|
||||||
|
def test_new_commit_reindexes_only_the_changed_file(wiki_env, fake_embedder, tmp_path):
|
||||||
|
remote = Remote(tmp_path)
|
||||||
|
remote.commit({"zalohy.md": DOC, "japan/metro.md": TRAVEL_DOC})
|
||||||
|
_use_git_source(wiki_env, remote.url)
|
||||||
|
assert wiki_sync.main([]) == 0
|
||||||
|
|
||||||
|
with store.connection(wiki_env.db_path) as conn:
|
||||||
|
metro_indexed_at = store.list_source_files(conn, "notes")["japan/metro.md"]["indexed_at"]
|
||||||
|
|
||||||
|
head = remote.commit({"zalohy.md": DOC + "\n## Offsite\n\nKopie jede do S3.\n"})
|
||||||
|
assert wiki_sync.main([]) == 0
|
||||||
|
|
||||||
|
with store.connection(wiki_env.db_path) as conn:
|
||||||
|
files = store.list_source_files(conn, "notes")
|
||||||
|
assert files["japan/metro.md"]["indexed_at"] == metro_indexed_at # untouched
|
||||||
|
source = store.get_source(conn, "notes")
|
||||||
|
assert source is not None and source["indexed_rev"] == head
|
||||||
|
assert len(store.bm25_ranked_ids(conn, "offsite", 10)) == 1
|
||||||
|
assert store.orphan_vector_ids(conn) == []
|
||||||
|
|
||||||
|
|
||||||
|
def test_deleted_file_upstream_drops_its_chunks_and_vectors(wiki_env, fake_embedder, tmp_path):
|
||||||
|
remote = Remote(tmp_path)
|
||||||
|
remote.commit({"zalohy.md": DOC, "japan/metro.md": TRAVEL_DOC})
|
||||||
|
_use_git_source(wiki_env, remote.url)
|
||||||
|
assert wiki_sync.main([]) == 0
|
||||||
|
|
||||||
|
remote.commit({"japan/metro.md": None}, message="drop metro")
|
||||||
|
assert wiki_sync.main([]) == 0
|
||||||
|
|
||||||
|
with store.connection(wiki_env.db_path) as conn:
|
||||||
|
assert sorted(store.list_source_files(conn, "notes")) == ["zalohy.md"]
|
||||||
|
assert store.bm25_ranked_ids(conn, "skyliner", 10) == []
|
||||||
|
assert store.orphan_vector_ids(conn) == []
|
||||||
|
stats = _stats(wiki_env)
|
||||||
|
assert stats["vectors"] == stats["chunks"]
|
||||||
|
|
||||||
|
|
||||||
|
def test_renamed_file_moves_its_chunks(wiki_env, fake_embedder, tmp_path):
|
||||||
|
remote = Remote(tmp_path)
|
||||||
|
remote.commit({"japan/metro.md": TRAVEL_DOC})
|
||||||
|
_use_git_source(wiki_env, remote.url)
|
||||||
|
assert wiki_sync.main([]) == 0
|
||||||
|
|
||||||
|
remote.move("japan/metro.md", "japan/tokyo-metro.md")
|
||||||
|
assert wiki_sync.main([]) == 0
|
||||||
|
|
||||||
|
with store.connection(wiki_env.db_path) as conn:
|
||||||
|
assert sorted(store.list_source_files(conn, "notes")) == ["japan/tokyo-metro.md"]
|
||||||
|
assert len(store.bm25_ranked_ids(conn, "skyliner", 10)) == 1
|
||||||
|
assert store.orphan_vector_ids(conn) == []
|
||||||
|
|
||||||
|
|
||||||
|
def test_commit_touching_nothing_indexed_only_records_the_rev(wiki_env, fake_embedder, tmp_path):
|
||||||
|
remote = Remote(tmp_path)
|
||||||
|
remote.commit({"zalohy.md": DOC})
|
||||||
|
_use_git_source(wiki_env, remote.url)
|
||||||
|
assert wiki_sync.main([]) == 0
|
||||||
|
before = _stats(wiki_env)
|
||||||
|
|
||||||
|
head = remote.commit({"tool.py": "print('hello')\n"})
|
||||||
|
assert wiki_sync.main([]) == 0
|
||||||
|
|
||||||
|
assert _stats(wiki_env) == before
|
||||||
|
with store.connection(wiki_env.db_path) as conn:
|
||||||
|
source = store.get_source(conn, "notes")
|
||||||
|
assert source is not None and source["indexed_rev"] == head
|
||||||
|
assert "no indexed file changed" in wiki_env.log_text()
|
||||||
|
|
||||||
|
|
||||||
|
def test_vendored_markdown_is_excluded(wiki_env, fake_embedder, tmp_path):
|
||||||
|
remote = Remote(tmp_path)
|
||||||
|
remote.commit(
|
||||||
|
{
|
||||||
|
"zalohy.md": DOC,
|
||||||
|
"node_modules/pkg/README.md": "# Foreign readme\n\nnenasazovat\n",
|
||||||
|
"web/vendor/lib/CHANGELOG.md": "# Changelog\n\nnenasazovat\n",
|
||||||
|
}
|
||||||
|
)
|
||||||
|
_use_git_source(wiki_env, remote.url)
|
||||||
|
assert wiki_sync.main([]) == 0
|
||||||
|
|
||||||
|
with store.connection(wiki_env.db_path) as conn:
|
||||||
|
assert sorted(store.list_source_files(conn, "notes")) == ["zalohy.md"]
|
||||||
|
|
||||||
|
|
||||||
|
def test_unreachable_remote_warns_skips_and_lets_other_sources_finish(wiki_env, fake_embedder, tmp_path):
|
||||||
|
"""Verification 8: per-source failure, indexed_rev untouched, workspace completes."""
|
||||||
|
_use_git_source(wiki_env, str(tmp_path / "does-not-exist.git"), with_workspace=True)
|
||||||
|
wiki_env.write_file("notes/local.md", DOC)
|
||||||
|
|
||||||
|
assert wiki_sync.main([]) == 1
|
||||||
|
|
||||||
|
log = wiki_env.log_text()
|
||||||
|
assert "WARN notes:" in log
|
||||||
|
# A permanent failure warns every tick, so each warning must stay a single line.
|
||||||
|
assert len([line for line in log.splitlines() if "WARN notes:" in line]) == 1
|
||||||
|
assert all(line.strip() for line in log.splitlines())
|
||||||
|
with store.connection(wiki_env.db_path) as conn:
|
||||||
|
assert store.list_source_files(conn, "notes") == {}
|
||||||
|
git_source = store.get_source(conn, "notes")
|
||||||
|
assert git_source is not None and git_source["indexed_rev"] is None
|
||||||
|
# The workspace source, which has nothing to do with the network, finished.
|
||||||
|
assert sorted(store.list_source_files(conn, "workspace")) == ["notes/local.md"]
|
||||||
|
|
||||||
|
|
||||||
|
def test_remote_that_recovers_is_picked_up_on_the_next_tick(wiki_env, fake_embedder, tmp_path):
|
||||||
|
missing = tmp_path / "later.git"
|
||||||
|
_use_git_source(wiki_env, str(missing))
|
||||||
|
assert wiki_sync.main([]) == 1
|
||||||
|
|
||||||
|
remote = Remote(tmp_path / "real")
|
||||||
|
(tmp_path / "real").mkdir(exist_ok=True)
|
||||||
|
remote_bare = remote.bare
|
||||||
|
remote.commit({"zalohy.md": DOC})
|
||||||
|
_use_git_source(wiki_env, str(remote_bare))
|
||||||
|
|
||||||
|
assert wiki_sync.main([]) == 0
|
||||||
|
with store.connection(wiki_env.db_path) as conn:
|
||||||
|
assert sorted(store.list_source_files(conn, "notes")) == ["zalohy.md"]
|
||||||
|
|
||||||
|
|
||||||
|
def test_missing_indexed_rev_falls_back_to_a_full_reindex(wiki_env, fake_embedder, tmp_path):
|
||||||
|
"""A force-pushed or gc'd base revision must degrade to a full pass, not crash."""
|
||||||
|
remote = Remote(tmp_path)
|
||||||
|
remote.commit({"zalohy.md": DOC})
|
||||||
|
_use_git_source(wiki_env, remote.url)
|
||||||
|
assert wiki_sync.main([]) == 0
|
||||||
|
|
||||||
|
with store.connection(wiki_env.db_path) as conn, store.tx(conn):
|
||||||
|
store.mark_synced(conn, "notes", "0" * 40, "2026-01-01T00:00:00+00:00")
|
||||||
|
|
||||||
|
remote.commit({"japan/metro.md": TRAVEL_DOC})
|
||||||
|
assert wiki_sync.main([]) == 0
|
||||||
|
|
||||||
|
with store.connection(wiki_env.db_path) as conn:
|
||||||
|
assert sorted(store.list_source_files(conn, "notes")) == ["japan/metro.md", "zalohy.md"]
|
||||||
|
assert store.orphan_vector_ids(conn) == []
|
||||||
306
skills/wiki/tests/test_wiki_search.py
Normal file
306
skills/wiki/tests/test_wiki_search.py
Normal file
@@ -0,0 +1,306 @@
|
|||||||
|
"""Query layers: RRF merge, the FTS expression, degraded mode, toc and grep."""
|
||||||
|
|
||||||
|
import shutil
|
||||||
|
import sys
|
||||||
|
from pathlib import Path
|
||||||
|
|
||||||
|
import pytest
|
||||||
|
|
||||||
|
SCRIPTS = Path(__file__).parent.parent / "scripts"
|
||||||
|
sys.path.insert(0, str(SCRIPTS))
|
||||||
|
|
||||||
|
import wiki_search
|
||||||
|
import wiki_store as store
|
||||||
|
import wiki_sync
|
||||||
|
from wiki_config import SourceConfig, load_config
|
||||||
|
|
||||||
|
BACKUP_DOC = """---
|
||||||
|
title: Zálohování dat
|
||||||
|
tags: [devops, backup]
|
||||||
|
---
|
||||||
|
|
||||||
|
# Zálohování dat
|
||||||
|
|
||||||
|
Záloha běží každou noc přes rsync na druhý disk.
|
||||||
|
|
||||||
|
## Retence snapshotů
|
||||||
|
|
||||||
|
Držíme třicet denních snapshotů a dvanáct měsíčních.
|
||||||
|
"""
|
||||||
|
|
||||||
|
TRAVEL_DOC = """---
|
||||||
|
title: Tokijské metro
|
||||||
|
tags: [transit]
|
||||||
|
---
|
||||||
|
|
||||||
|
# Tokijské metro
|
||||||
|
|
||||||
|
Z Narity do centra jede Skyliner za osmatřicet minut.
|
||||||
|
"""
|
||||||
|
|
||||||
|
|
||||||
|
def _index(env):
|
||||||
|
assert wiki_sync.main([]) == 0
|
||||||
|
|
||||||
|
|
||||||
|
# ---------------------------------------------------------------------------
|
||||||
|
# pure functions
|
||||||
|
# ---------------------------------------------------------------------------
|
||||||
|
|
||||||
|
|
||||||
|
def test_rrf_merge_over_known_ranks():
|
||||||
|
merged = wiki_search.rrf_merge([[1, 2, 3], [3, 1, 4]])
|
||||||
|
assert [chunk_id for chunk_id, _ in merged] == [1, 3, 2, 4]
|
||||||
|
scores = dict(merged)
|
||||||
|
assert scores[1] == pytest.approx(1 / 61 + 1 / 62)
|
||||||
|
assert scores[3] == pytest.approx(1 / 63 + 1 / 61)
|
||||||
|
assert scores[4] == pytest.approx(1 / 63)
|
||||||
|
|
||||||
|
|
||||||
|
def test_rrf_merge_of_a_single_list_keeps_its_order():
|
||||||
|
assert [i for i, _ in wiki_search.rrf_merge([[7, 8, 9]])] == [7, 8, 9]
|
||||||
|
|
||||||
|
|
||||||
|
def test_fts_expression_adds_prefix_wildcards():
|
||||||
|
"""Czech inflection is covered by the wildcard; a 2-char prefix is too broad to keep."""
|
||||||
|
assert wiki_search.fts_match_expression("záloha dat") == '"záloha"* OR "dat"*'
|
||||||
|
assert wiki_search.fts_match_expression("v ok dva") == '"v" OR "ok" OR "dva"*'
|
||||||
|
|
||||||
|
|
||||||
|
def test_fts_expression_neutralises_operators_and_punctuation():
|
||||||
|
expression = wiki_search.fts_match_expression("NOT (a AND b) -c*")
|
||||||
|
assert expression == '"NOT"* OR "a" OR "AND"* OR "b" OR "c"'
|
||||||
|
|
||||||
|
|
||||||
|
def test_fts_expression_of_empty_query_is_empty():
|
||||||
|
assert wiki_search.fts_match_expression("!!! ") == ""
|
||||||
|
|
||||||
|
|
||||||
|
# ---------------------------------------------------------------------------
|
||||||
|
# search
|
||||||
|
# ---------------------------------------------------------------------------
|
||||||
|
|
||||||
|
|
||||||
|
def test_search_reports_which_half_found_each_hit(wiki_env, fake_embedder, capsys):
|
||||||
|
wiki_env.write_file("notes/zalohy.md", BACKUP_DOC)
|
||||||
|
wiki_env.write_file("notes/tokio.md", TRAVEL_DOC)
|
||||||
|
_index(wiki_env)
|
||||||
|
|
||||||
|
config = load_config(wiki_env.config_path)
|
||||||
|
assert wiki_search.run_search(config, "retence snapshotů", limit=5) == 0
|
||||||
|
|
||||||
|
out = capsys.readouterr().out
|
||||||
|
assert "notes/zalohy.md" in out
|
||||||
|
assert "Retence snapshotů" in out
|
||||||
|
assert "bm25 #" in out and "vec #" in out
|
||||||
|
|
||||||
|
|
||||||
|
def test_search_prefix_match_finds_inflected_form(wiki_env, fake_embedder, capsys):
|
||||||
|
wiki_env.write_file("notes/zalohy.md", BACKUP_DOC)
|
||||||
|
_index(wiki_env)
|
||||||
|
|
||||||
|
config = load_config(wiki_env.config_path)
|
||||||
|
assert wiki_search.run_search(config, "zaloh", limit=5) == 0
|
||||||
|
assert "notes/zalohy.md" in capsys.readouterr().out
|
||||||
|
|
||||||
|
|
||||||
|
def test_search_says_out_loud_when_embeddings_are_unavailable(wiki_env, fake_embedder, capsys):
|
||||||
|
"""Verification 4: degrade to FTS-only, never silently."""
|
||||||
|
wiki_env.write_file("notes/zalohy.md", BACKUP_DOC)
|
||||||
|
_index(wiki_env)
|
||||||
|
capsys.readouterr()
|
||||||
|
|
||||||
|
# Drop the fake and let the config's unreachable endpoint take over.
|
||||||
|
wiki_search.OllamaEmbedder = wiki_search.__dict__["OllamaEmbedder"]
|
||||||
|
from wiki_embed import OllamaEmbedder as RealEmbedder
|
||||||
|
|
||||||
|
wiki_search.OllamaEmbedder = RealEmbedder
|
||||||
|
|
||||||
|
config = load_config(wiki_env.config_path)
|
||||||
|
assert wiki_search.run_search(config, "záloha", limit=5) == 0
|
||||||
|
|
||||||
|
out = capsys.readouterr().out
|
||||||
|
assert "embeddings unavailable" in out
|
||||||
|
assert "FTS-only" in out
|
||||||
|
assert "notes/zalohy.md" in out # the lexical half still answers
|
||||||
|
|
||||||
|
|
||||||
|
def test_search_refuses_an_index_from_another_contract(wiki_env, fake_embedder, capsys):
|
||||||
|
"""Verification 5: a mismatch must say `reindex needed`, not return results."""
|
||||||
|
wiki_env.write_file("notes/zalohy.md", BACKUP_DOC)
|
||||||
|
_index(wiki_env)
|
||||||
|
|
||||||
|
wiki_env.config_path.write_text(
|
||||||
|
wiki_env.config_path.read_text(encoding="utf-8").replace(
|
||||||
|
"model: qwen3-embedding:0.6b", "model: qwen3-embedding:8b"
|
||||||
|
),
|
||||||
|
encoding="utf-8",
|
||||||
|
)
|
||||||
|
config = load_config(wiki_env.config_path)
|
||||||
|
assert wiki_search.run_search(config, "záloha", limit=5) == 1
|
||||||
|
|
||||||
|
captured = capsys.readouterr()
|
||||||
|
assert "reindex needed" in captured.err
|
||||||
|
assert "notes/zalohy.md" not in captured.out
|
||||||
|
|
||||||
|
|
||||||
|
def test_search_warns_while_vectors_are_still_pending(wiki_env, monkeypatch, capsys):
|
||||||
|
wiki_env.write_file("notes/zalohy.md", BACKUP_DOC)
|
||||||
|
assert wiki_sync.main([]) == 1 # unreachable endpoint -> chunks stay pending
|
||||||
|
|
||||||
|
from conftest import FakeEmbedder
|
||||||
|
|
||||||
|
monkeypatch.setattr(wiki_search, "OllamaEmbedder", FakeEmbedder)
|
||||||
|
config = load_config(wiki_env.config_path)
|
||||||
|
assert wiki_search.run_search(config, "záloha", limit=5) == 0
|
||||||
|
|
||||||
|
out = capsys.readouterr().out
|
||||||
|
assert "awaiting vectors" in out
|
||||||
|
|
||||||
|
|
||||||
|
def test_search_over_an_empty_index_says_no_matches(wiki_env, fake_embedder, capsys):
|
||||||
|
config = load_config(wiki_env.config_path)
|
||||||
|
assert wiki_search.run_search(config, "záloha", limit=5) == 0
|
||||||
|
assert "(no matches)" in capsys.readouterr().out
|
||||||
|
|
||||||
|
|
||||||
|
def test_an_unrelated_query_still_returns_nearest_neighbours(wiki_env, fake_embedder, capsys):
|
||||||
|
"""KNN has no distance floor: the semantic half always offers its k closest chunks.
|
||||||
|
|
||||||
|
That is deliberate for retrieval — the agent reads the chunk and judges it — but it
|
||||||
|
means an empty result set only ever means an empty index.
|
||||||
|
"""
|
||||||
|
wiki_env.write_file("notes/zalohy.md", BACKUP_DOC)
|
||||||
|
_index(wiki_env)
|
||||||
|
capsys.readouterr()
|
||||||
|
|
||||||
|
config = load_config(wiki_env.config_path)
|
||||||
|
assert wiki_search.run_search(config, "kajakářství", limit=5) == 0
|
||||||
|
out = capsys.readouterr().out
|
||||||
|
assert "notes/zalohy.md" in out
|
||||||
|
assert "bm25 #" not in out # nothing lexical matched; every hit came from the vectors
|
||||||
|
|
||||||
|
|
||||||
|
def test_search_with_an_empty_query(wiki_env, fake_embedder, capsys):
|
||||||
|
wiki_env.write_file("notes/zalohy.md", BACKUP_DOC)
|
||||||
|
_index(wiki_env)
|
||||||
|
capsys.readouterr()
|
||||||
|
|
||||||
|
config = load_config(wiki_env.config_path)
|
||||||
|
assert wiki_search.run_search(config, "???", limit=5) == 0
|
||||||
|
assert "(empty query)" in capsys.readouterr().out
|
||||||
|
|
||||||
|
|
||||||
|
def test_search_limit_caps_the_result_count(wiki_env, fake_embedder, capsys):
|
||||||
|
for index in range(8):
|
||||||
|
wiki_env.write_file(f"notes/doc{index}.md", f"# Dokument {index}\n\nZáloha dat číslo {index}.\n")
|
||||||
|
_index(wiki_env)
|
||||||
|
capsys.readouterr()
|
||||||
|
|
||||||
|
config = load_config(wiki_env.config_path)
|
||||||
|
assert wiki_search.run_search(config, "záloha dat", limit=3) == 0
|
||||||
|
out = capsys.readouterr().out
|
||||||
|
assert len([line for line in out.splitlines() if line.startswith(("1. ", "2. ", "3. ", "4. "))]) == 3
|
||||||
|
|
||||||
|
|
||||||
|
# ---------------------------------------------------------------------------
|
||||||
|
# toc
|
||||||
|
# ---------------------------------------------------------------------------
|
||||||
|
|
||||||
|
|
||||||
|
def test_toc_groups_by_directory_and_shows_titles_and_tags(wiki_env, fake_embedder, capsys):
|
||||||
|
wiki_env.write_file("notes/devops/zalohy.md", BACKUP_DOC)
|
||||||
|
wiki_env.write_file("notes/travel/tokio.md", TRAVEL_DOC)
|
||||||
|
_index(wiki_env)
|
||||||
|
capsys.readouterr()
|
||||||
|
|
||||||
|
assert wiki_search.run_toc(None, None) == 0
|
||||||
|
out = capsys.readouterr().out
|
||||||
|
assert "workspace (2 files)" in out
|
||||||
|
assert "notes/devops/" in out
|
||||||
|
assert "zalohy.md" in out
|
||||||
|
assert "Zálohování dat" in out
|
||||||
|
assert "[devops, backup]" in out
|
||||||
|
|
||||||
|
|
||||||
|
def test_toc_filters_by_tag(wiki_env, fake_embedder, capsys):
|
||||||
|
wiki_env.write_file("notes/zalohy.md", BACKUP_DOC)
|
||||||
|
wiki_env.write_file("notes/tokio.md", TRAVEL_DOC)
|
||||||
|
_index(wiki_env)
|
||||||
|
capsys.readouterr()
|
||||||
|
|
||||||
|
assert wiki_search.run_toc(None, "transit") == 0
|
||||||
|
out = capsys.readouterr().out
|
||||||
|
assert "tokio.md" in out
|
||||||
|
assert "zalohy.md" not in out
|
||||||
|
|
||||||
|
|
||||||
|
def test_toc_on_an_empty_index(wiki_env, fake_embedder, capsys):
|
||||||
|
assert wiki_search.run_toc(None, None) == 0
|
||||||
|
assert "(no indexed files match)" in capsys.readouterr().out
|
||||||
|
|
||||||
|
|
||||||
|
# ---------------------------------------------------------------------------
|
||||||
|
# grep
|
||||||
|
# ---------------------------------------------------------------------------
|
||||||
|
|
||||||
|
|
||||||
|
@pytest.mark.skipif(shutil.which("rg") is None, reason="ripgrep not installed")
|
||||||
|
def test_grep_sees_source_code_the_index_never_touches(wiki_env, fake_embedder, capsys):
|
||||||
|
"""Layer 1 is the substitute for indexing code, and it costs nothing extra."""
|
||||||
|
wiki_env.write_file("notes/zalohy.md", BACKUP_DOC)
|
||||||
|
wiki_env.write_file("notes/tools/backup.py", "def rotate_snapshots(keep=30):\n return keep\n")
|
||||||
|
_index(wiki_env)
|
||||||
|
capsys.readouterr()
|
||||||
|
|
||||||
|
config = load_config(wiki_env.config_path)
|
||||||
|
assert wiki_search.run_grep(config, "rotate_snapshots", None) == 0
|
||||||
|
out = capsys.readouterr().out
|
||||||
|
assert "backup.py" in out
|
||||||
|
assert "rotate_snapshots" in out
|
||||||
|
|
||||||
|
# The same identifier is absent from the index — grep is the only layer that has it.
|
||||||
|
with store.connection(wiki_env.db_path) as conn:
|
||||||
|
assert store.bm25_ranked_ids(conn, "rotate_snapshots", 10) == []
|
||||||
|
|
||||||
|
|
||||||
|
@pytest.mark.skipif(shutil.which("rg") is None, reason="ripgrep not installed")
|
||||||
|
def test_grep_stays_inside_the_source_paths(wiki_env, fake_embedder, capsys):
|
||||||
|
wiki_env.write_file("notes/zalohy.md", BACKUP_DOC)
|
||||||
|
wiki_env.write_file("tmp/dump/leak.md", "tajnastruna v tmp\n")
|
||||||
|
|
||||||
|
config = load_config(wiki_env.config_path)
|
||||||
|
assert wiki_search.run_grep(config, "tajnastruna", None) == 0
|
||||||
|
assert "(no matches)" in capsys.readouterr().out
|
||||||
|
|
||||||
|
|
||||||
|
@pytest.mark.skipif(shutil.which("rg") is None, reason="ripgrep not installed")
|
||||||
|
def test_grep_reports_an_unknown_source(wiki_env, fake_embedder, capsys):
|
||||||
|
config = load_config(wiki_env.config_path)
|
||||||
|
assert wiki_search.run_grep(config, "cokoli", "nope") == 1
|
||||||
|
assert "unknown source" in capsys.readouterr().err
|
||||||
|
|
||||||
|
|
||||||
|
def test_grep_roots_use_the_literal_prefix_of_each_glob(wiki_env):
|
||||||
|
wiki_env.write_file("notes/a.md", "x")
|
||||||
|
wiki_env.write_file("develop/b.md", "x")
|
||||||
|
source = load_config(wiki_env.config_path).source("workspace")
|
||||||
|
assert source is not None
|
||||||
|
assert wiki_search.grep_roots(source) == [
|
||||||
|
wiki_env.workspace / "notes",
|
||||||
|
wiki_env.workspace / "develop",
|
||||||
|
]
|
||||||
|
|
||||||
|
|
||||||
|
def test_grep_roots_of_a_git_source_is_the_whole_clone(wiki_env):
|
||||||
|
clone = wiki_env.wiki_dir / "remote" / "travel"
|
||||||
|
clone.mkdir(parents=True)
|
||||||
|
source = SourceConfig(
|
||||||
|
source_id="travel",
|
||||||
|
kind="git",
|
||||||
|
url="git@example:travel.git",
|
||||||
|
paths=("**",),
|
||||||
|
include=("*.md",),
|
||||||
|
exclude=(),
|
||||||
|
)
|
||||||
|
assert wiki_search.grep_roots(source) == [clone]
|
||||||
279
skills/wiki/tests/test_wiki_sync.py
Normal file
279
skills/wiki/tests/test_wiki_sync.py
Normal file
@@ -0,0 +1,279 @@
|
|||||||
|
"""Sync driver: idempotence, the lock, degraded mode, the meta guard, coverage."""
|
||||||
|
|
||||||
|
import json
|
||||||
|
import os
|
||||||
|
import sys
|
||||||
|
from pathlib import Path
|
||||||
|
|
||||||
|
SCRIPTS = Path(__file__).parent.parent / "scripts"
|
||||||
|
sys.path.insert(0, str(SCRIPTS))
|
||||||
|
|
||||||
|
import wiki_store as store
|
||||||
|
import wiki_sync
|
||||||
|
from wiki_config import load_config
|
||||||
|
from wiki_embed import expected_meta
|
||||||
|
|
||||||
|
DOC = """# Zálohování
|
||||||
|
|
||||||
|
Záloha dat na disk probíhá každý den v noci pomocí rsyncu.
|
||||||
|
|
||||||
|
## Retence
|
||||||
|
|
||||||
|
Držíme třicet denních snapshotů a dvanáct měsíčních.
|
||||||
|
"""
|
||||||
|
|
||||||
|
OTHER_DOC = """# Cestování
|
||||||
|
|
||||||
|
Tokio metro je nejrychlejší cesta z Narity do centra.
|
||||||
|
"""
|
||||||
|
|
||||||
|
|
||||||
|
def _stats(env):
|
||||||
|
with store.connection(env.db_path) as conn:
|
||||||
|
return store.index_stats(conn)
|
||||||
|
|
||||||
|
|
||||||
|
def _run(argv=None):
|
||||||
|
return wiki_sync.main(argv or [])
|
||||||
|
|
||||||
|
|
||||||
|
def _file_row(conn, path):
|
||||||
|
row = store.get_file(conn, "workspace", path)
|
||||||
|
assert row is not None, f"{path} is not indexed"
|
||||||
|
return row
|
||||||
|
|
||||||
|
|
||||||
|
def test_indexes_covered_workspace_files(wiki_env, fake_embedder):
|
||||||
|
wiki_env.write_file("notes/zalohy.md", DOC)
|
||||||
|
wiki_env.write_file("develop/knowledge.md", OTHER_DOC)
|
||||||
|
|
||||||
|
assert _run() == 0
|
||||||
|
|
||||||
|
stats = _stats(wiki_env)
|
||||||
|
assert stats["files"] == 2
|
||||||
|
assert stats["chunks"] > 0
|
||||||
|
assert stats["vectors"] == stats["chunks"]
|
||||||
|
assert stats["pending"] == 0
|
||||||
|
assert "indexed 2 files" in wiki_env.log_text() or "indexed 1 files" in wiki_env.log_text()
|
||||||
|
|
||||||
|
|
||||||
|
def test_paths_whitelist_and_exclude_are_honoured(wiki_env, fake_embedder):
|
||||||
|
wiki_env.write_file("notes/keep.md", DOC)
|
||||||
|
wiki_env.write_file("notes/inbox/raw.md", DOC) # excluded
|
||||||
|
wiki_env.write_file("develop/history.md", DOC) # excluded
|
||||||
|
wiki_env.write_file("tmp/clone/README.md", DOC) # outside paths
|
||||||
|
wiki_env.write_file("notes/script.py", "# TODO: fix this\n") # outside include
|
||||||
|
|
||||||
|
assert _run() == 0
|
||||||
|
|
||||||
|
with store.connection(wiki_env.db_path) as conn:
|
||||||
|
paths = sorted(store.list_source_files(conn, "workspace"))
|
||||||
|
assert paths == ["notes/keep.md"]
|
||||||
|
|
||||||
|
|
||||||
|
def test_second_run_over_unchanged_tree_is_a_no_op(wiki_env, fake_embedder):
|
||||||
|
"""Verification 2: counts identical and nothing appended to the log."""
|
||||||
|
wiki_env.write_file("notes/zalohy.md", DOC)
|
||||||
|
assert _run() == 0
|
||||||
|
|
||||||
|
first_stats = _stats(wiki_env)
|
||||||
|
with store.connection(wiki_env.db_path) as conn:
|
||||||
|
first_indexed_at = _file_row(conn, "notes/zalohy.md")["indexed_at"]
|
||||||
|
log_after_first = wiki_env.log_text()
|
||||||
|
|
||||||
|
assert _run() == 0
|
||||||
|
|
||||||
|
assert _stats(wiki_env) == first_stats
|
||||||
|
assert wiki_env.log_text() == log_after_first
|
||||||
|
with store.connection(wiki_env.db_path) as conn:
|
||||||
|
assert _file_row(conn, "notes/zalohy.md")["indexed_at"] == first_indexed_at
|
||||||
|
|
||||||
|
|
||||||
|
def test_touching_a_file_without_changing_it_only_refreshes_stats(wiki_env, fake_embedder):
|
||||||
|
wiki_env.write_file("notes/zalohy.md", DOC)
|
||||||
|
assert _run() == 0
|
||||||
|
before = _stats(wiki_env)
|
||||||
|
with store.connection(wiki_env.db_path) as conn:
|
||||||
|
indexed_at = _file_row(conn, "notes/zalohy.md")["indexed_at"]
|
||||||
|
|
||||||
|
os.utime(wiki_env.workspace / "notes/zalohy.md", (1_600_000_000, 1_600_000_000))
|
||||||
|
assert _run() == 0
|
||||||
|
|
||||||
|
assert _stats(wiki_env) == before
|
||||||
|
with store.connection(wiki_env.db_path) as conn:
|
||||||
|
row = _file_row(conn, "notes/zalohy.md")
|
||||||
|
assert row["indexed_at"] == indexed_at # content untouched -> no re-chunk
|
||||||
|
assert row["mtime"] == 1_600_000_000
|
||||||
|
|
||||||
|
|
||||||
|
def test_edited_file_is_rechunked_and_reembedded(wiki_env, fake_embedder):
|
||||||
|
wiki_env.write_file("notes/zalohy.md", DOC)
|
||||||
|
assert _run() == 0
|
||||||
|
|
||||||
|
wiki_env.write_file("notes/zalohy.md", DOC + "\n## Offsite\n\nKopie jede do S3 každý týden.\n")
|
||||||
|
assert _run() == 0
|
||||||
|
|
||||||
|
stats = _stats(wiki_env)
|
||||||
|
assert stats["pending"] == 0
|
||||||
|
assert stats["vectors"] == stats["chunks"]
|
||||||
|
with store.connection(wiki_env.db_path) as conn:
|
||||||
|
assert store.orphan_vector_ids(conn) == []
|
||||||
|
assert len(store.bm25_ranked_ids(conn, "offsite", 10)) == 1
|
||||||
|
|
||||||
|
|
||||||
|
def test_deleted_file_leaves_no_chunks_or_vectors(wiki_env, fake_embedder):
|
||||||
|
wiki_env.write_file("notes/zalohy.md", DOC)
|
||||||
|
wiki_env.write_file("notes/travel.md", OTHER_DOC)
|
||||||
|
assert _run() == 0
|
||||||
|
|
||||||
|
(wiki_env.workspace / "notes/travel.md").unlink()
|
||||||
|
assert _run() == 0
|
||||||
|
|
||||||
|
with store.connection(wiki_env.db_path) as conn:
|
||||||
|
assert sorted(store.list_source_files(conn, "workspace")) == ["notes/zalohy.md"]
|
||||||
|
assert store.orphan_vector_ids(conn) == []
|
||||||
|
assert store.bm25_ranked_ids(conn, "tokio", 10) == []
|
||||||
|
stats = _stats(wiki_env)
|
||||||
|
assert stats["vectors"] == stats["chunks"]
|
||||||
|
|
||||||
|
|
||||||
|
def test_live_lock_makes_the_run_a_silent_no_op(wiki_env, fake_embedder):
|
||||||
|
"""Verification 3: a second concurrent sync exits 0 without writing."""
|
||||||
|
wiki_env.write_file("notes/zalohy.md", DOC)
|
||||||
|
wiki_sync.LOCK_PATH.write_text(
|
||||||
|
json.dumps({"pid": os.getpid(), "started_at": wiki_sync.datetime.now().astimezone().isoformat()}),
|
||||||
|
encoding="utf-8",
|
||||||
|
)
|
||||||
|
|
||||||
|
assert _run() == 0
|
||||||
|
|
||||||
|
assert not wiki_env.db_path.exists()
|
||||||
|
assert wiki_env.log_text() == ""
|
||||||
|
|
||||||
|
|
||||||
|
def test_stale_lock_is_reclaimed_and_logged(wiki_env, fake_embedder):
|
||||||
|
wiki_env.write_file("notes/zalohy.md", DOC)
|
||||||
|
dead_pid = 999_999
|
||||||
|
wiki_sync.LOCK_PATH.write_text(
|
||||||
|
json.dumps({"pid": dead_pid, "started_at": wiki_sync.datetime.now().astimezone().isoformat()}),
|
||||||
|
encoding="utf-8",
|
||||||
|
)
|
||||||
|
|
||||||
|
assert _run() == 0
|
||||||
|
|
||||||
|
assert "stale lock, reclaiming" in wiki_env.log_text()
|
||||||
|
assert _stats(wiki_env)["files"] == 1
|
||||||
|
assert not wiki_sync.LOCK_PATH.exists()
|
||||||
|
|
||||||
|
|
||||||
|
def test_unreadable_lock_is_treated_as_stale(wiki_env, fake_embedder):
|
||||||
|
wiki_env.write_file("notes/zalohy.md", DOC)
|
||||||
|
wiki_sync.LOCK_PATH.write_text("not json", encoding="utf-8")
|
||||||
|
|
||||||
|
assert _run() == 0
|
||||||
|
assert _stats(wiki_env)["files"] == 1
|
||||||
|
|
||||||
|
|
||||||
|
def test_coverage_report_names_uncovered_directories(wiki_env, fake_embedder):
|
||||||
|
"""Verification 6: markdown outside `paths` must surface as a log line."""
|
||||||
|
wiki_env.write_file("notes/zalohy.md", DOC)
|
||||||
|
wiki_env.write_file("recepty/gulas.md", "# Guláš\n\nCibule na dva kusy hovězího.\n")
|
||||||
|
# The skill's own clone storage holds markdown but is never a candidate.
|
||||||
|
(wiki_env.wiki_dir / "remote" / "index").mkdir(parents=True)
|
||||||
|
(wiki_env.wiki_dir / "remote" / "index" / "foreign.md").write_text("# Cizí\n", encoding="utf-8")
|
||||||
|
|
||||||
|
assert _run() == 0
|
||||||
|
|
||||||
|
coverage = [line for line in wiki_env.log_text().splitlines() if "coverage" in line]
|
||||||
|
assert len(coverage) == 1
|
||||||
|
assert "recepty" in coverage[0]
|
||||||
|
assert "wiki" not in coverage[0]
|
||||||
|
|
||||||
|
|
||||||
|
def test_degraded_mode_then_recovery_without_file_change(wiki_env, monkeypatch):
|
||||||
|
"""Verification 4: no Ollama -> FTS-only pending; once back, step 4b fills the vectors."""
|
||||||
|
wiki_env.write_file("notes/zalohy.md", DOC)
|
||||||
|
|
||||||
|
# The fixture config points at embed.invalid, so no embedder is reachable here.
|
||||||
|
assert _run() == 1
|
||||||
|
stats = _stats(wiki_env)
|
||||||
|
assert stats["chunks"] > 0
|
||||||
|
assert stats["vectors"] == 0
|
||||||
|
assert stats["pending"] == stats["chunks"]
|
||||||
|
assert "embeddings unavailable" in wiki_env.log_text()
|
||||||
|
|
||||||
|
with store.connection(wiki_env.db_path) as conn:
|
||||||
|
assert len(store.bm25_ranked_ids(conn, "zaloh*", 10)) >= 1 # FTS works meanwhile
|
||||||
|
|
||||||
|
from conftest import FakeEmbedder
|
||||||
|
|
||||||
|
monkeypatch.setattr(wiki_sync, "OllamaEmbedder", FakeEmbedder)
|
||||||
|
assert _run() == 0 # no file changed, yet the pending chunks get vectors
|
||||||
|
|
||||||
|
stats = _stats(wiki_env)
|
||||||
|
assert stats["pending"] == 0
|
||||||
|
assert stats["vectors"] == stats["chunks"]
|
||||||
|
|
||||||
|
|
||||||
|
def test_meta_mismatch_blocks_indexing_until_full(wiki_env, fake_embedder):
|
||||||
|
"""Verification 5: never mix vectors from two contracts."""
|
||||||
|
wiki_env.write_file("notes/zalohy.md", DOC)
|
||||||
|
assert _run() == 0
|
||||||
|
before = _stats(wiki_env)
|
||||||
|
|
||||||
|
wiki_env.config_path.write_text(
|
||||||
|
wiki_env.config_path.read_text(encoding="utf-8").replace(
|
||||||
|
"model: qwen3-embedding:0.6b", "model: qwen3-embedding:4b"
|
||||||
|
),
|
||||||
|
encoding="utf-8",
|
||||||
|
)
|
||||||
|
wiki_env.write_file("notes/new.md", OTHER_DOC)
|
||||||
|
|
||||||
|
assert _run() == 1
|
||||||
|
assert "index identity mismatch" in wiki_env.log_text()
|
||||||
|
assert _stats(wiki_env) == before # nothing indexed under the new contract
|
||||||
|
|
||||||
|
assert _run(["--full"]) == 0
|
||||||
|
with store.connection(wiki_env.db_path) as conn:
|
||||||
|
assert store.read_meta(conn)["embedding_model"] == "qwen3-embedding:4b"
|
||||||
|
assert sorted(store.list_source_files(conn, "workspace")) == ["notes/new.md", "notes/zalohy.md"]
|
||||||
|
assert store.orphan_vector_ids(conn) == []
|
||||||
|
|
||||||
|
|
||||||
|
def test_full_rebuild_wipes_and_reindexes(wiki_env, fake_embedder):
|
||||||
|
wiki_env.write_file("notes/zalohy.md", DOC)
|
||||||
|
assert _run() == 0
|
||||||
|
before = _stats(wiki_env)
|
||||||
|
|
||||||
|
assert _run(["--full"]) == 0
|
||||||
|
|
||||||
|
assert _stats(wiki_env) == before
|
||||||
|
with store.connection(wiki_env.db_path) as conn:
|
||||||
|
assert store.orphan_vector_ids(conn) == []
|
||||||
|
|
||||||
|
|
||||||
|
def test_meta_is_written_on_a_fresh_index(wiki_env, fake_embedder):
|
||||||
|
wiki_env.write_file("notes/zalohy.md", DOC)
|
||||||
|
assert _run() == 0
|
||||||
|
|
||||||
|
config = load_config(wiki_env.config_path)
|
||||||
|
with store.connection(wiki_env.db_path) as conn:
|
||||||
|
assert store.read_meta(conn) == expected_meta(config.embedding)
|
||||||
|
|
||||||
|
|
||||||
|
def test_unknown_source_argument_is_reported(wiki_env, fake_embedder):
|
||||||
|
wiki_env.write_file("notes/zalohy.md", DOC)
|
||||||
|
assert _run(["--source", "nope"]) == 1
|
||||||
|
assert "unknown source" in wiki_env.log_text()
|
||||||
|
|
||||||
|
|
||||||
|
def test_batching_respects_the_configured_size(wiki_env, fake_embedder):
|
||||||
|
"""batch: 4 in the fixture config — the embedder must be called in chunks of 4."""
|
||||||
|
for index in range(6):
|
||||||
|
wiki_env.write_file(f"notes/doc{index}.md", f"# Dokument {index}\n\n{DOC}\n")
|
||||||
|
|
||||||
|
assert _run() == 0
|
||||||
|
|
||||||
|
calls = [len(call) for embedder in fake_embedder for call in embedder.calls]
|
||||||
|
assert calls, "the embedder was never called"
|
||||||
|
assert max(calls) <= 4
|
||||||
Reference in New Issue
Block a user