feat(tts): provider capabilities schema + Gemini TTS + dynamic param forms

Baseline groundwork for TTS expansion plan. Introduces a capabilities
system that the follow-up plan (phase-01..04) builds on.

Backend:
- `audio.ParamSchema` / `ProviderCapabilities` types + per-provider
  capabilities.go for edge, elevenlabs, minimax, openai, gemini.
- Gemini TTS provider (client, models, voices, wav encoder,
  multi-speaker via SpeakerVoice, audio-tag aware prompts).
- TTSOptions gains `Params map[string]any` (read-only) + `Speakers`.
- `VoiceListProvider` interface decouples HTTP voice handler from
  provider-specific impls.
- `nested_keys.go` resolves dot-separated param paths for nested
  provider bodies (voice_settings.stability etc.).
- Characterization + defaults-invariant tests per provider lock
  nil-params byte-equivalence before new params land.
- `/v1/tts/capabilities` HTTP endpoint + integration coverage.
- Dual-read tests (PG + SQLite) for tts_config.

Frontend (web + desktop):
- `DynamicParamForm` with depends-on evaluation, split into
  fields/logic modules. Slider primitive added.
- `AudioTagPicker` + `MultiSpeakerEditor` for Gemini.
- `voice-picker` refactored toward portal UX; combobox tightened.
- tts-capabilities API client + typed hooks.
- i18n catalogs (en/vi/zh) expanded; parity tests guard key drift.
- TTS page reorganised (voice-model-section removed; playground +
  credentials + provider-setup split cleanly).

Docs: codebase-summary, project-changelog, tts-provider-capabilities.
This commit is contained in:
viettranx committed 2026-04-20 00:05:19 +07:00
1 parent 7639a8c013
commit 613b6e38d7
123 files changed
+12233 -994

No files matched your search

+129
View File
@@ -0,0 +1,129 @@
# Codebase Summary
High-level map of GoClaw modules and key cross-cutting concerns.
For system design see `docs/00-architecture-overview.md`; for API contract see `docs/18-http-api.md`.
---
## Module Map
| Path | Purpose |
|------|---------|
| `cmd/` | CLI (cobra): `serve`, `onboard`, `migrate` commands |
| `internal/agent/` | Agent loop (think→act→observe), router, input guard |
| `internal/audio/` | TTS provider layer (see §TTS below) |
| `internal/bootstrap/` | SOUL/IDENTITY system prompts + per-user seed |
| `internal/channels/` | Telegram, Feishu, Zalo, Discord, WhatsApp connectors |
| `internal/config/` | JSON5 config loading + env overlay |
| `internal/crypto/` | AES-256-GCM encryption for API keys |
| `internal/gateway/` | WS + HTTP server, client, method router |
| `internal/http/` | HTTP API handlers (`/v1/*`) |
| `internal/i18n/` | Backend message catalog (EN/VI/ZH) + `T(locale, key, args)` |
| `internal/memory/` | pgvector 3-tier memory system |
| `internal/mcp/` | Model Context Protocol bridge |
| `internal/permissions/` | RBAC: admin / operator / viewer |
| `internal/pipeline/` | 8-stage agent pipeline |
| `internal/providers/` | LLM providers (Anthropic, OpenAI-compat, Qwen, Claude CLI) |
| `internal/store/` | Store interfaces + PG + SQLite implementations |
| `internal/tools/` | Tool registry (filesystem, exec, web, MCP, delegate) |
| `internal/tts/` | Back-compat alias package for old import paths |
| `internal/vault/` | Knowledge Vault: wikilinks, hybrid search, FS sync |
| `migrations/` | PostgreSQL migration files |
| `ui/web/` | React SPA (Vite, Tailwind, Radix UI, Zustand) |
| `ui/desktop/` | Wails v2 desktop app (SQLite, embedded gateway) |
---
## TTS Subsystem
### Overview
TTS lives in `internal/audio/` (canonical) with a back-compat alias at `internal/tts/`.
Five providers ship out-of-the-box: OpenAI, ElevenLabs, Edge TTS, MiniMax, Gemini.
### Provider Capabilities Schema
Each provider implements `audio.TTSProvider` and optionally `audio.CapabilitiesProvider`:
```go
type CapabilitiesProvider interface {
Capabilities() ProviderCapabilities
}
type ProviderCapabilities struct {
Params []ParamSchema
CustomFeatures CustomFeatures
}
```
`ParamSchema` carries `Key`, `Type` (string/number/bool/select/textarea), `Label`, `Help`, `Default`,
`Min`/`Max`, `Options` (for select), `DependsOn` (all-of conditions), and `Hidden` flag.
The UI reads `GET /v1/tts/capabilities` and renders per-provider param editors dynamically.
See `docs/tts-provider-capabilities.md` for full schema documentation.
### Providers
| Provider | Package | Voice source | Key param |
|----------|---------|--------------|-----------|
| OpenAI | `internal/audio/openai/` | Static list | `speed`, `response_format`, `instructions` |
| ElevenLabs | `internal/audio/elevenlabs/` | Dynamic (API) | `voiceSettings.*`, `seed`, `language_code` |
| Edge TTS | `internal/audio/edge/` | Static list | `rate`, `pitch`, `volume` |
| MiniMax | `internal/audio/minimax/` | Dynamic (API) | `speed`, `vol`, `pitch`, `emotion`, `audio.*` |
| Gemini | `internal/audio/gemini/` | Static list (30) | multi-speaker, audio tags, 70+ languages |
### Storage: Dual-Read
Tenant TTS config is stored in `system_config` with two complementary strategies:
- **Legacy flat keys** (`tts.provider`, `tts.voice_id`, `tts.api_key`, …) — written by all versions.
- **Params blob** (`tts.<provider>.params` JSON) — written by v3.x+ for per-provider param overrides.
On read, both paths are merged; blob wins on key conflict. This allows gradual migration:
old clients see flat keys; new clients see the full params blob.
### Manager
`audio.Manager` is the central registry:
- `RegisterProvider(p TTSProvider)` — replaces any existing provider by the same name.
- `Primary()` — returns current primary provider name from config.
- `Synthesize(ctx, text, opts)` — delegates to active provider.
- `ListVoices(ctx, opts)` — delegates to `VoiceListProvider` if implemented.
### HTTP Endpoints
| Endpoint | Role | Notes |
|----------|------|-------|
| `POST /v1/tts/synthesize` | operator | Streams WAV response |
| `POST /v1/tts/test-connection` | operator | Ephemeral provider, returns latency |
| `GET /v1/tts/capabilities` | operator | ProviderCapabilities JSON per provider |
| `GET /v1/tts/config` | admin | Tenant TTS config |
| `POST /v1/tts/config` | admin | Save tenant TTS config |
### Gemini Specifics
- Models: `gemini-2.5-flash-preview-tts`, `gemini-2.5-pro-preview-tts` (preview).
- Multi-speaker: up to 2 simultaneous speakers, each with distinct voice + name annotation.
- Audio tags: inline `<say-as>` / style directives via bracketed prompts.
- Sentinel errors: `ErrInvalidVoice`, `ErrInvalidModel`, `ErrSpeakerLimit` → HTTP 422 with i18n message.
- SSRF guard: `api_base` override validated by `validateProviderURL()` — blocks 127.0.0.1/localhost.
### i18n
Backend validation errors use `i18n.T(locale, key, args...)` pattern.
Locale is extracted from `Accept-Language` HTTP header by `enrichContext` middleware.
UI param labels/help text live in:
- `ui/web/src/i18n/locales/{en,vi,zh}/tts.json`
- `ui/desktop/frontend/src/i18n/locales/{en,vi,zh}/tts.json`
Parity enforced by `ui/web/src/__tests__/i18n-tts-key-parity.test.ts` (vitest).
---
## Key Conventions
- **Store layer:** Interface-based; PG (`store/pg/`) + SQLite (`store/sqlitestore/`). Raw SQL, `$1/$2` params.
- **Context propagation:** `store.WithLocale`, `store.WithUserID`, `store.WithTenantID`, etc.
- **Security logs:** `slog.Warn("security.*")` for all security events.
- **SSRF prevention:** `validateProviderURL()` in `internal/http/tts_validate.go`.
- **i18n keys:** Add to `internal/i18n/keys.go` + 3 catalog Go files; UI strings in 3 locale JSON dirs.
- **Migrations:** PG (`migrations/`) + SQLite (`store/sqlitestore/schema.go`) — always update both.