mirror of
https://github.com/tiennm99/goclaw.git
synced 2026-10-17 02:14:09 +00:00
fix(vault): search document bodies and match any query word (#1554)
vault_search only indexed title + path + the auto-summary, and the summary is written from the first 3000 runes of the file. Anything the summary left out (room names, prices, codes) could never be found. On top of that the FTS query used plainto_tsquery, which ANDs every word, so a normal question like "what is the price of the Deluxe Ocean Suite?" matched nothing even when the key words were indexed. - Add vault_document_chunks (migration 98): the file body split into chunks with their own tsvector and embedding. body_indexed_hash on vault_documents records which content_hash the chunks came from, so unchanged files are not re-chunked or re-embedded. - The enrich worker rebuilds chunks when a file changes. This needs no LLM, so it runs even when no provider is configured. - Rescan backfills chunks for docs indexed before this change, since they already have a summary and never go back through the worker. - FTS matches any query word; ts_rank still ranks docs with more matching words first. Both FTS and vector search look at the doc and its chunks and score each doc by its best hit. SQLite is unchanged: its vault search is LIKE on title/path only.
This commit is contained in:
1 parent
c48feda334
commit
65dbbcfd59
13 files changed
+683
-69
No files matched your search
@@ -86,7 +86,8 @@ Document registry: metadata pointers. Content lives on filesystem; registry hold
|
||||
| `doc_type` | TEXT | context, memory, note, skill, episodic |
|
||||
| `content_hash` | TEXT | SHA-256 of file content (detects changes) |
|
||||
| `embedding` | vector(1536) | pgvector: semantic similarity |
|
||||
| `tsv` | tsvector | Generated: FTS index on title+path |
|
||||
| `tsv` | tsvector | Generated: FTS index on title+path+summary |
|
||||
| `body_indexed_hash` | TEXT | `content_hash` the body chunks were built from (NULL = not chunked yet) |
|
||||
| `metadata` | JSONB | Optional custom fields |
|
||||
| `created_at`, `updated_at` | TIMESTAMPTZ | Timestamps |
|
||||
| **Unique constraint** | (agent_id, scope, path) | One doc per path per scope |
|
||||
@@ -99,6 +100,19 @@ Document registry: metadata pointers. Content lives on filesystem; registry hold
|
||||
- `idx_vault_docs_embedding` — HNSW vector (semantic search)
|
||||
- `idx_vault_docs_tsv` — GIN FTS index (keyword search)
|
||||
|
||||
### vault_document_chunks
|
||||
|
||||
The body of each text document, split into ~1000-char chunks (200-char overlap), so search reaches content that the auto-summary leaves out. The enrich worker rebuilds the chunks from the workspace file whenever `content_hash` moves past `body_indexed_hash`. `POST /v1/vault/rescan` also backfills docs that have no chunks yet. Media and `document` types are not chunked.
|
||||
|
||||
| Column | Type | Notes |
|
||||
|--------|------|-------|
|
||||
| `document_id` | UUID | Parent doc (cascade delete) |
|
||||
| `chunk_index` | INT | Position in the body |
|
||||
| `start_line`, `end_line` | INT | Line range in the file |
|
||||
| `text` | TEXT | Chunk content |
|
||||
| `embedding` | vector(1536) | Only the first 64 chunks of a doc are embedded |
|
||||
| `tsv` | tsvector | Generated: FTS on `text` ('simple' config) |
|
||||
|
||||
### vault_links
|
||||
|
||||
Bidirectional links between documents (wikilinks, explicit references).
|
||||
@@ -192,8 +206,8 @@ Hybrid search integrates vault FTS, vector embeddings, episodic memory, and know
|
||||
|
||||
`VaultStore.Search(ctx, opts VaultSearchOptions)` on single vault:
|
||||
|
||||
- **FTS**: PostgreSQL `plainto_tsquery()` on tsv (title+path keywords)
|
||||
- **Vector**: pgvector cosine similarity on embedding (semantic)
|
||||
- **FTS**: matches any query word (OR), so natural-language questions work; docs matching more words rank higher. Runs on the doc `tsv` and on body chunks, and a doc scores by its best hit
|
||||
- **Vector**: pgvector cosine similarity on the doc embedding and on chunk embeddings, best hit per doc
|
||||
- **Combined scoring**: Normalize each method's scores (0–1), then apply query-time weights
|
||||
- **Results:** Top N documents with score
|
||||
|
||||
|
||||
Reference in new issue
Block a user