* feat(audio): add openai_compat TTS/STT provider for self-hosted endpoints
Adds a provider pair speaking the OpenAI audio wire format against any
compatible endpoint (gpu-manager, Speaches, vLLM, LocalAI, llama.cpp,
Ollama), so a self-hosted engine no longer needs a bespoke HTTP shim
implementing goclaw's proprietary /transcribe_audio contract.
Kept distinct from the openai package on purpose. Both speak the same
wire format, but "openai" carries api.openai.com semantics, and
audio.IsVoiceCompatible applies OpenAI's voice allowlist to any provider
by that name — FilterVoiceForProvider then silently substitutes "alloy"
for anything outside it. A self-hosted voice ID such as Piper's
"fr_FR-gilles-low" would be replaced with no error and no log line, and
the caller would hear the wrong language. Providers without validation
rules pass through untouched, which is the correct behaviour for an
engine whose voice namespace is its own.
Other deliberate departures from the openai package:
- api_base is required. There is no public endpoint to fall back to, and
defaulting to api.openai.com would send self-hosted traffic to a vendor.
- An empty api_key omits the Authorization header rather than sending an
empty Bearer token; self-hosted engines commonly have no auth.
- Empty voice/model are omitted from the request instead of sent blank,
since some engines resolve the model from the requested voice.
- The multipart filename is never blank: several engines type the audio
by extension rather than Content-Type and reject what they cannot type.
Endpoint error bodies are preserved in the returned error so a format
mismatch (e.g. an engine that cannot encode mp3) is diagnosable.
* feat(audio): wire openai_compat TTS/STT provider into gateway config
Adds tts.openai_compat config block and registers both providers at
startup. api_base is the enable switch rather than an API key, since
self-hosted endpoints commonly have no auth and there is no vendor
default to fall back to.
The STT chain is now built from the providers that actually registered
instead of being hardcoded to {elevenlabs, proxy}. Transcribe skips
unregistered names with a warning on every call, so a static chain
naming absent providers is log noise on the hot path. openai_compat goes
first when present: it is an explicit operator choice and keeps audio on
the local network. "proxy" stays last, as BridgeLegacySTT registers it
later per channel.
Behaviour is unchanged when tts.openai_compat.api_base is unset: the
chain resolves to {elevenlabs, proxy} exactly as before.
---------
Co-authored-by: Bruno Clermont <bruno.clermont@gmail.com>
An agent's tts_voice_id (e.g. "en-US-GuyNeural" for Edge) was applied
regardless of the selected TTS provider, causing OpenAI and other
providers to reject unknown voice IDs.
- Add internal/audio/voice_compat.go with IsVoiceCompatible,
GetProviderDefaultVoice, and FilterVoiceForProvider helpers
- Update TtsTool.resolveVoiceAndModel to report whether the voice
came from an agent override (voiceFromAgent)
- Apply applyVoiceCompat per-provider branch in Execute so
incompatible agent voices fall back to the provider's default
with a slog.Warn for operator visibility
- Add voice_compat_test.go with table-driven tests for all cases
Co-authored-by: Bruno Clermont <bruno.clermont@gmail.com>
- config: align struct tag formatting in TelemetryConfig for readability
- audio tests: modernize test patterns
- cache, http, hooks, providers: small style and cleanup passes
- No functional changes
MiniMax /v1/get_voice does not return explicit gender/language fields,
but the system-voice naming convention exposes them:
- voice_id prefix `male-*` / `female-*` (legacy Chinese voices)
- voice_name suffix `*_Man` / `*_Boy` / `*_Lord` / `*_Speaker` → male
- voice_name suffix `*_Lady` / `*_Girl` / `*_Belle` / `*_Lass` → female
- voice_name prefix `English_*` / `Chinese_*` / `Japanese_*` /
`Korean_*` / `French_*` / etc. → language
`parseMinimaxLabels` is best-effort — returns nil for unparseable IDs
(e.g. `moss_audio_<uuid>` cloned voices), so the picker shows no badge
rather than a wrong one. The picker already renders `voice.labels` as
badges; nothing to wire on the FE side.
8-case unit test covers the matrix.
Google does not publish gender for the 30 prebuilt Gemini voices —
inclusive design — but does publish a one-word style descriptor
(Bright, Firm, Smooth, Mature, Lively, etc.). Surface it so users can
pick by character.
- `audio.VoiceOption` gains `Labels map[string]string` (provider-
specific descriptors). ElevenLabs already returns labels via API;
Gemini now hardcodes `{"style": "..."}` per voice from the official
catalog.
- Web + desktop `mapCapVoiceToPortal` passes labels through. Both
pickers' `LABEL_KEYS` now include `"style"`, so the existing badge
rendering picks it up — first 1–2 entries shown next to the name.
- TS `VoiceOption` interface (web + desktop) gains optional `labels`.
No gender field is invented for Gemini voices.
Post-review cleanup of Phase 4. Closes Finding #9 properly and corrects
the Finding #13 documentation lie surfaced in the code-review report.
Capability schema:
- Replace `AgentOverridable bool` with `AgentOverridableAs string` on
ParamSchema. Empty string = not overridable; non-empty = the generic
key alias (`"speed"`, `"emotion"`, `"style"`).
- Each provider declaration now carries the alias inline, so the
generic↔native mapping has a single TS-readable source.
Frontend:
- Web `tts-override-block.tsx` drops the inline `GENERIC_TO_NATIVE`
literal and derives the bidirectional adapter from the filtered
capability params (each param self-describes its alias). Adapter
tests rewritten around the new shape.
- Desktop `AgentDetailPanel.tsx` drops the 45-line inline IIFE in
favour of a new `<TtsOverrideFineTune>` component that uses the
same alias-based mapping.
Backend:
- Move `AgentTTSParamsAllowedKeys` + `ValidateAgentTTSParams` to
`internal/audio/agent_params_adapter.go`. HTTP `validate.go` and WS
`gateway/methods/agents_update.go` both delegate, eliminating the
duplicated `{speed, emotion, style}` literal.
Cleanup:
- Delete orphan i18n keys `MsgTtsParamInvalidJSON` and
`MsgTtsParamDependsOn` from `keys.go` + en/vi/zh catalogs (no
in-code references; DependsOn is FE-only, JSON parse failures
already surface via slog).
Documentation:
- `prompt-settings-section.tsx` Finding #13 comment rewritten to
honestly describe the best-effort merge into a fresh local copy of
the cached `otherConfig` prop. Concurrent-tab clobber remains
possible — server-side JSON-merge-patch endpoint planned for v2.
Tests: 9 backend suites (race), web 217/217, desktop build clean,
both Go build tags pass.
Phase 4 — final phase of the TTS params/layout/agent-override plan.
Adds a 3-key allow-list (`speed`, `emotion`, `style`) per agent stored
in `agents.other_config.tts_params`. Backend resolves and merges into
`opts.Params` PER ATTEMPT inside the fallback loop so each provider
sees its own native shape — never the primary's keys when fallback
runs (Finding #1 critical).
Backend:
- `AgentOverridable bool` on `audio.ParamSchema`. UI filter reads this
flag from /v1/tts/capabilities; no separate TS literal mirror —
capabilities API is the single source of truth (Finding #9).
- `audio.AdaptAgentParams(generic, provider)` maps the 3 generic keys
to provider-native paths (e.g. `speed` → `voice_settings.speed` for
ElevenLabs, flat `speed` for OpenAI/MiniMax, dropped for Edge/Gemini).
- `Manager.SynthesizeWithFallbackAdapted` adapts inside the loop so
fallback providers receive correctly-shaped params.
- `manager_auto.go` and `tools/tts.go` Execute do per-attempt adaptation
on the tenant + direct + fallback call sites.
- Drop log bumped to `slog.Info("tts.agent.params.dropped", ...)` for
audit trail when a generic key isn't supported by the active provider.
- Cross-check test asserts every adapter switch case has at least one
capability ParamSchema with `AgentOverridable: true`, and vice versa.
Security (red-team findings):
- Allow-list ENFORCED at write path: `validateAgentTTSParams` in HTTP
`handleUpdate` AND WS `agents_update` rejects any `tts_params` key
outside `{speed, emotion, style}` (Finding #5).
- 64KB body cap on agent PUT via `http.MaxBytesReader` (Finding #6).
- Explicit tenant-scope guard after `agents.GetByID` (Finding #12).
- Concurrent-tab clobber: handleSave merges `tts_params` into a fresh
copy of `otherConfig` rather than reusing stale state (Finding #13).
- Rate-limit verified — RoleAdmin gate sufficient for v1 (Finding #15).
Frontend (web + desktop):
- `TtsOverrideBlock` rewritten: filters capability params to
`agent_overridable === true`, renders via `DynamicParamForm`. Hides
entirely for providers with no overridable params (Edge, Gemini).
- Bidirectional adapter (generic ↔ capability-native form state) so
agent storage stays in generic keys while UI works in native paths.
25 round-trip tests cover all 5 providers.
- Desktop `AgentDetailPanel` gains an inline fine-tune section gated
on `globalProvider`, reusing the desktop `DynamicParamForm`.
i18n: `tts.override.params.title` ("Fine-tune") added to web + desktop
en/vi/zh.
Tests: all 9 backend suites green (race), web 214/214, desktop build
clean, both Go build tags pass.
- Add per-tenant TTS configuration endpoints (GET/POST /v1/tts/config)
with RoleAdmin auth instead of master scope
- Implement TenantTTSResolver for channels to use tenant-specific
TTS providers and auto mode settings
- Add per-agent voice override in channel TTS auto-apply:
- Extend OutboundMessage with AgentID and AgentOtherConfig
- Inject AgentAudioSnapshot in dispatch.go from outbound message
- MaybeApply reads tts_voice_id/tts_model_id from agent context
- Fix events.go to use RunContext.TenantID directly (H3)
- Fix dispatch.go error notification to use sendCtx (H1)
- Update UI to use new /v1/tts/config endpoints
Implement legacy STT bridge adapter to support existing per-channel STTProxyURL database configuration. Wraps arbitrary proxy endpoints and auto-registers them via Manager at boot. Includes integration tests for URL validation, authentication, and error handling.
Add Transcribe() method to Manager with STT provider dispatch and channel-specific overrides. Implements channelSTTOverrides map, WithChannel context propagation, RegisterChannelSTT registration, and resolveSTTChain fallback logic. Includes manager-level unit tests.
Implement STT proxy provider that wraps arbitrary HTTP-based STT services. Ported 12 test cases from legacy Telegram STT integration to validate multipart/form-data submission, language/model overrides, and API error handling.
Implement native ElevenLabs Scribe STT provider with POST /v1/speech-to-text endpoint. Supports 20MB audio cap, multipart form submission, configurable model and language. Includes comprehensive unit tests for happy path, audio size validation, API errors, and edge cases.
Add STTInput, STTOptions, and TranscriptResult types to support native STT providers (ElevenLabs Scribe). Includes audio_duration_secs JSON tag for transcription metadata.
Add StreamingTTSProvider interface for audio streaming support. Implement voice
caching with TTL+LRU. Add ElevenLabs model validation and voice list retrieval.
