Commit Graph
4893 Commits
Author SHA1 Message Date
Alex 8b099ef1aa fix(wiki): monochrome icon for the New wiki source option
The add-source grid inverts black icons to white (dark:invert), but the wiki used book.svg (fill #6A4DF4), which inverted to lime. Point the wiki ingestor at a new black book-mono.svg so it renders white like the others; book.svg stays purple for its other use (PromptsModal, no invert there).
2026-06-22 23:39:03 +01:00
Alex f809f660f4 fix(wiki): maintain sources.tokens for wikis (sum of page token_counts)
sources.tokens was only set at ingest, so wikis showed no/stale token count on the card (blank wikis showed nothing; converted ones showed the stale original count). rebuild_wiki_directory_structure (called after every wiki mutation) now also sets sources.tokens to the sum of wiki_pages.token_count; blank wiki create sets tokens=0. So the card reflects live wiki content.
2026-06-22 23:33:47 +01:00
Alex 2616ec5d1b fix(wiki): detect wiki sources by config.kind, not type
A converted source keeps its origin type (e.g. crawler) with the wiki-ness in config.kind=wiki (D5/D20), so the UI's type==='wiki' checks treated converted sources as normal (wrong view, convert action still offered). Sources.tsx now keys isWiki + the view branch on config.kind==='wiki' (with a type fallback for blank-created wikis). The paginated endpoint already returns config.
2026-06-22 23:26:12 +01:00
Alex 54773cdf0f feat(wiki): open source view on card click (wiki viewer or normal view)
Clicking a source card opens its view via the existing setDocumentToView handler (wiki -> WikiViewer, nested -> FileTree/ConnectorTree, else -> Chunks). The 3-dots Button and both DropdownMenuContent menus stopPropagation so clicks on the trigger or menu items don't bubble (through the React tree, even though Radix portals them) to the card's onClick. Adds cursor + keyboard/role a11y matching AgentCard.
2026-06-22 23:21:11 +01:00
Alex 8a0f7024e3 fix(wiki): convert reassembles pages from existing chunks (crawler/remote)
convert_source_to_wiki now builds pages from the source's already-ingested vector-store chunks instead of re-parsing storage files, so it works for crawler/remote/connector sources (which have no stored files; file_path empty). Groups by metadata file_path/file_name (URL- and connector-aware; URLs normalized to virtual paths) rather than the raw source. Passes embeddings_key to create_vectorstore in BOTH convert and reembed_wiki_page (fixes a TypeError crash on faiss/elasticsearch). Conservative overlap-trim (min length). Deletes the original chunks after reassembling so retrieval has no duplicate/stale content.
2026-06-22 23:03:52 +01:00
Alex 80533f9383 feat(wiki): convert-to-wiki UX, living-wiki naming, surface agent edits
Unit 7 of F-Wiki (D20/D23/D24). Source row action 'Convert to wiki' (owner/editor only) -> confirm naming the cost (lossy/one-way/skipped) -> poll task status -> after-summary; blank source enables inline. Agent wiki writes (create/str_replace/insert/delete/rename) render as a prominent top-level card in the conversation (outside the collapsed tool-calls accordion) with the page path; reads stay normal. 'Living wiki'/agent-editable naming + explainer; 'New wiki' blank-source fast path retained.
2026-06-22 21:05:35 +01:00
Alex 919fe10ab9 feat(wiki): human page editing + provenance stamps
Unit 6 of F-Wiki (D22 + D24). Migration 0024 adds wiki_pages.updated_via; set to 'agent' by WikiTool + convert, 'human' by the edit/seed endpoints (content-hash short-circuit preserves it). WikiViewer gains a markdown editor (Save via PUT with expected_version; 409 reloads latest while keeping the draft; 403 graceful) and a per-page provenance stamp (editor/when/version). Edit gated on write access client-side; backend remains the real authz.
2026-06-22 20:42:37 +01:00
Alex 0c9e0313bd feat(wiki): convert existing source to wiki + human page edit endpoint
Unit 5 of F-Wiki (D20-D23). convert_source_to_wiki task reuses reingest's storage file-load + parser to materialize files->wiki_pages (one page/file), skips/reports non-text, re-embeds per page, and flips kind=wiki + exposure=agentic_tool only when pages were created. POST /wiki/convert (explicit, write-authz; blank source enables inline, fileful enqueues the task; rejects mid-ingest). PUT /wiki/page for human edits (write-authz, optimistic version -> 409, re-embed). PATCH /config preserves kind (kind changes only via convert).
2026-06-22 20:16:18 +01:00
Alex 5110189143 feat(wiki): create-wiki route + read endpoints + minimal read-only UI
Unit 4 of F-Wiki. POST /api/sources/wiki creates a type=wiki/kind=wiki source with no ingest task (optional seed page enqueues reembed). GET /wiki/pages + /wiki/page serve the tree + fresh page content, read-access gated (owner or team grant), path-validated. Frontend: 'Create Wiki' ingestor entry, read-only WikiViewer (FileTree + react-markdown, no raw HTML), sync/reingest hidden for wiki sources.
2026-06-22 18:41:54 +01:00
Alex 98aa8db953 feat(wiki): WikiTool with injection + write authz
Unit 3 of F-Wiki. WikiTool (view/create/str_replace/insert/delete/rename) over wiki_pages: exact-case unique str_replace (no silent multi-replace), optimistic version on edits (WikiPageConflict), reads served fresh from Postgres, untrusted-content fencing on reads, 1MB page cap. Injected via add_wiki_tool only for writable wiki sources (effective_write_owner: owner/team-editor; viewers get nothing), scoped to one source_id. Each mutation enqueues reembed_wiki_page (owner as user, per-page idempotency key) and rebuilds directory_structure. Shared validate_tool_path extracted from MemoryTool.
2026-06-22 18:18:04 +01:00
Alex d68be86244 feat(wiki): reembed_wiki_page durable task
Per-page re-embed (Unit 2 of F-Wiki): targeted delete of the page's old chunks, re-chunk via the source's chunking config, add_chunk with reingest-matching metadata (source=path), set embed_status embedded/failed. Durable + idempotent (key=content_hash), mirroring reingest_source_task. Missing page => purge only.
2026-06-22 17:49:04 +01:00
Alex 8aa0facf4a feat(wiki): wiki_pages table + repository + targeted chunk delete
Migration 0023_wiki_pages (source-scoped; version/content_hash/embed_status; FK source_id->sources ON DELETE CASCADE; UNIQUE(source_id,path) + prefix index). WikiPagesRepository: path-keyed CRUD with content-hash short-circuit, version bump, and move-with-reject. BaseVectorStore.delete_chunks_by_source_path (loop default) + parameterized pgvector override (DELETE WHERE source_id=%s AND metadata->>'source'=%s). Unit 1 of F-Wiki.
2026-06-22 17:40:41 +01:00
Alex a709b7ccdc feat: semantic chunking + pgvector hybrid (BM25+vector) retriever (#2553) 2026-06-22 17:04:13 +01:00
Alex c6a33eca32 Merge pull request #2552 from arc53/rag-per-source-config
feat: per-source RAG configuration (retrieval strategies, chunking, exposure, prescreen)
2026-06-22 14:00:02 +01:00
Alex 057b510adf fix: better error logging 2026-06-22 13:47:09 +01:00
Alex a9068faefa feat: small mixes on mixes sources 2026-06-22 13:13:51 +01:00
Alex babc067aa4 feat: remove old chunk management 2026-06-22 10:40:33 +01:00
Alex f2e6967ec9 feat: redesign advanced settings panel as settings rows with dividers
Group eyebrow headers with short live/re-ingest tags, lighter field labels, hairline dividers between rows, compact right-aligned controls, and hide the single-option Retriever select.
2026-06-21 14:15:09 +01:00
Alex e18886b4f1 fix: rename create-flow section to "Advanced settings" with neutral toggle color 2026-06-21 14:02:44 +01:00
Alex e9f90a3f87 chore: rename source config menu/title to "Source settings" 2026-06-21 13:59:10 +01:00
Alex f4ca9ad858 fix: make retrieval options modal scroll on desktop
The upload and edit-config modals dropped their height cap on sm+ screens (sm:max-h-none), so an expanded Retrieval options panel overflowed the viewport with no scroll. Keep the max-h cap at all breakpoints (max-h is a cap, not a fixed height, so short modals still size naturally) and let the existing overflow-y-auto content area scroll.

Also fix a tsc error in the RetrievalOptions serialization test (optionsToConfig returns SourceConfig whose retrieval is optional).
2026-06-21 09:47:02 +01:00
Alex f6400cd736 feat: per-source RAG configuration (retrieval strategies, chunking, exposure, prescreen)
Introduces a per-source config contract that makes RAG behavior strategy-dispatched instead of a single hardcoded path. Every source gains a validated JSONB config; an empty/absent config reproduces current behavior byte-for-byte, and the whole path is gated by PER_SOURCE_RETRIEVAL_ENABLED.

Foundation: sources.config JSONB column + migration 0022_source_config; SourceConfig/ChunkingConfig/RetrievalConfig pydantic models (strict on write, lenient on read); ChunkerCreator and RetrieverCreator.register registries; config threaded through the upload routes, ingest/remote/connector workers, and reingest.

Retrieval: a Dispatcher groups sources by retriever key (all-classic collapses to today's single ClassicRAG under one shared token budget; non-classic retrievers get their own instance), removing the previous single-global-retriever collapse in stream_processor. Per-source chunks, score_threshold (honored for pgvector/mongodb, safely ignored elsewhere), and rephrase_query toggle. New PATCH /api/sources/<id>/config with team-aware (effective_write_owner) authz and a requires_reingest signal.

Chunking strategies: recursive, markdown, parent_child (selectable per source; re-ingest to apply). Search exposure: per-source prefetch vs agentic_tool for agentic/research agents. Map-reduce prescreen: optional LLM relevance pre-filter implemented as a composable post-retrieval stage that wraps any retriever.

Backend and frontend (shared Retrieval options panel + edit modal) with tests; backend suite and frontend vitest green. Excludes the wiki and GraphRAG flagships.
2026-06-20 21:54:23 +01:00
Alex cfda009be0 Merge pull request #2551 from arc53/collisons-in-messages
fix: classify FK violations in message journal to skip pointless seq-retry
2026-06-18 17:10:08 +01:00
Alex 1475ac2bf9 fix: fk and collision issue 2026-06-18 16:21:42 +01:00
Alex f27444fe48 feat: mini llm fixes 2026-06-18 13:35:37 +01:00
Alex 7b0f5d777b chore: update readme 2026-06-17 15:49:41 +01:00
Alex b7a8eaeb9f Merge pull request #2548 from arc53/team
feat: teams
2026-06-17 10:21:42 +01:00
Alex 2923f85ce3 fix: better s3 bucket var names 2026-06-16 18:08:04 +01:00
Alex 7e7b92aa5e fix: timezoned timestamps 2026-06-16 18:02:23 +01:00
Alex 1b7f84d981 Merge remote-tracking branch 'origin/main' into team 2026-06-16 17:34:27 +01:00
Alex 0ced2d734c Merge pull request #2545 from ManishMadan2882/main
(feat:dateTime) consistent d/m/y
2026-06-16 17:27:47 +01:00
Alex 26d741ab94 feat: notifications on share 2026-06-16 14:50:55 +01:00
Alex 6ad42043e7 fix: dont share assets with yourself 2026-06-16 14:09:36 +01:00
Alex d988d63b0f feat: ignore .bak files 2026-06-16 12:37:19 +01:00
Alex 11aee54a07 fix: remove picker .bak 2026-06-16 12:36:08 +01:00
Alex 8c8eb3c679 fix: ci fixes 2026-06-16 12:22:07 +01:00
Alex 884bea7286 fix: test 2026-06-16 12:14:39 +01:00
Alex d37e6bec51 feat: better tags and dropdowns for sharing 2026-06-16 12:14:28 +01:00
Alex b97a12870c fix: input box 2026-06-16 11:02:51 +01:00
Alex ed83fb783c fix: tool calls on free model 2026-06-16 10:50:33 +01:00
Alex d3dc976d3c feat: team design changes 2026-06-16 10:48:42 +01:00
Alex d21f4758be feat: teams 2026-06-16 09:21:21 +01:00
GH Action - Upstream Sync e5a2f3c079 Merge branch 'main' of https://github.com/arc53/DocsGPT 2026-06-16 04:18:01 +00:00
ManishMadan2882 d0dc52f925 (feat:dateTime) consistent d/m/y 2026-06-16 01:44:54 +02:00
Alex df41410772 Update roadmap to mark agent scheduling as complete 2026-06-15 14:48:06 +01:00
Alex f6a371be7d Merge pull request #2539 from arc53/admin-dashboard
feat: admin dashboard
2026-06-15 12:03:43 +01:00
Alex c772ebc60f fix: namespace 2026-06-15 11:44:02 +01:00
Alex 9a349401f4 feat: admin dashboard 2026-06-15 11:30:02 +01:00
Alex 49fb28acd3 Merge pull request #2538 from arc53/rbac-users
feat: admin role rbac
2026-06-14 23:08:37 +01:00
Alex 82cd7b7d49 fix: tests 2026-06-14 22:38:28 +01:00