diff --git a/docs/05-channels-messaging.md b/docs/05-channels-messaging.md index b3b400ae..e4a83e8b 100644 --- a/docs/05-channels-messaging.md +++ b/docs/05-channels-messaging.md @@ -245,26 +245,81 @@ Each topic can restrict which tools the agent may use. The `tools` field accepts The tool allow list is passed via message metadata and applied by the policy engine before the LLM sees the tool definitions. -### Speech-to-Text +### Voice Message Transcription (STT) -Voice and audio messages can be transcribed via an external STT proxy service. +Voice and audio messages are transcribed through a unified `audio.Manager` interface. All channels (Telegram, Discord, Feishu, WhatsApp) route transcription requests through the same STT chain. + +#### Unified STT Flow ```mermaid flowchart TD - VOICE["Voice/audio message"] --> DOWNLOAD["Download audio file
from Telegram"] - DOWNLOAD --> STT["POST to STT proxy
(multipart: file + tenant_id)"] - STT --> PARSE["Parse transcript"] - PARSE --> INJECT["Prepend to message:
[audio: filename] Transcript: text"] - INJECT --> AGENT["Agent receives transcribed content"] - - VOICE --> ROUTING{"VoiceAgentID configured?"} + VOICE["Voice/audio message"] --> ROUTE{Channel type?} + + ROUTE -->|Telegram/Discord/Feishu| DOWNLOAD["Download audio file"] + ROUTE -->|WhatsApp| WHATSAPP_CHECK{"whatsapp_enabled
in settings?"} + + WHATSAPP_CHECK -->|No| WA_FALLBACK["[Voice message]
(default opt-out)"] + WHATSAPP_CHECK -->|Yes| DOWNLOAD + + DOWNLOAD --> STT_CHECK{"STT providers
configured?"} + STT_CHECK -->|Yes| STT_CHAIN["Try providers in order:
elevenlabs_scribe, proxy"] + STT_CHECK -->|No| LEGACY{"Legacy bridge
providers?"} + + LEGACY -->|Yes| STT_CHAIN + LEGACY -->|No| FALLBACK["[Voice message]"] + + STT_CHAIN -->|Success| TEXT["Transcribed text"] + STT_CHAIN -->|Fails/Timeout 10s| FALLBACK + + TEXT --> INJECT["Prepend to message:
[audio: filename] Transcript: text"] + FALLBACK --> FALLBACK_INJECT["[Voice message]"] + + INJECT --> AGENT["Agent context"] + FALLBACK_INJECT --> AGENT + + VOICE --> ROUTING{"VoiceAgentID
configured
(Telegram)?"} ROUTING -->|Yes| VOICE_AGENT["Route to voice-specific agent"] - ROUTING -->|No| DEFAULT_AGENT["Route to channel's default agent"] + ROUTING -->|No| DEFAULT_AGENT["Default channel agent"] ``` -**Configuration**: STT proxy URL, timeout (default 30s), optional tenant ID and API key. If transcription fails, the media placeholder remains — no error is surfaced. +#### Configuration & Decision Rules -**Voice routing**: When `VoiceAgentID` is configured, audio/voice messages are routed to a different agent (e.g., a speech-specialized agent) instead of the channel's default agent. +**Decision 2 (Conflict rule):** When `builtin_tools[stt].settings.providers[]` is present in the database, it OVERRIDES all legacy channel-specific STT configs. The legacy STT bridge (`STTProxyURL` → `proxy_stt` provider) only activates when the providers list is empty or missing. + +| Setting | Behavior | +|---------|----------| +| `providers: ["elevenlabs_scribe", "proxy_stt"]` | Try ElevenLabs Scribe first; fall back to legacy proxy | +| `providers: []` (empty) | Skip all STT; voice → `[Voice message]` fallback | +| `providers` missing (nil) | Check for legacy bridge at startup; activate if `STTProxyURL` exists | + +**Decision 6 (WhatsApp opt-in):** WhatsApp voice message STT is **OFF by default** (`whatsapp_enabled: false`). Rationale: WhatsApp voice messages are end-to-end encrypted; sending audio to an external STT provider breaks E2E encryption. Admins must explicitly toggle STT in the UI at **Config → Audio → STT** and acknowledge the E2E breaking change. + +When disabled (default): +- Voice messages surface in agent context as `[Voice message]` (localized via i18n key `channel.voice_message_fallback`) +- No audio leaves the device + +When enabled: +- Voice audio is transcribed via the configured STT chain +- Fallback to `[Voice message]` on failure/timeout (10s wall clock) + +#### Per-Channel Behavior + +| Channel | STT Support | Notes | +|---------|:-:|---------| +| **Telegram** | Yes | Uses unified chain; voice routing via `VoiceAgentID` config | +| **Discord** | Yes | Standard flow; no special routing | +| **Feishu** | Yes | Standard flow; no special routing | +| **WhatsApp** | Yes (opt-in) | Requires explicit admin approval; default OFF | + +#### Factory Integration + +All channel factories accept `audioMgr *audio.Manager`: +- Telegram: `FactoryWithStoresAndAudio(..., audioMgr)` +- Discord: `FactoryWithAudio(..., audioMgr)` +- Feishu: `FactoryWithStoresAndAudio(..., audioMgr)` +- WhatsApp: `FactoryWithDBAudio(..., audioMgr, builtinToolStore)` + +WhatsApp additionally accepts `builtinToolStore store.BuiltinToolStore` to fetch the per-message `whatsapp_enabled` opt-in flag. Wiring in `cmd/gateway_channels_setup.go:77` + `cmd/gateway.go:433`. ### Bot Commands @@ -631,7 +686,7 @@ flowchart TD | `internal/channels/telegram/handlers.go` | Message handling, media processing, forum topic detection | | `internal/channels/telegram/topic_config.go` | Per-topic config layering and resolution | | `internal/channels/telegram/commands.go` | Bot commands: /stop, /reset, /tasks, /addwriter, etc. | -| `internal/channels/telegram/stt.go` | Speech-to-text proxy integration, voice agent routing | +| `internal/channels/telegram/factory.go` | Channel factory with audio.Manager wiring | | `internal/channels/telegram/stream.go` | Streaming placeholder management | | `internal/channels/telegram/reactions.go` | Status reactions on messages | | `internal/channels/telegram/format.go` | Markdown → Telegram HTML pipeline, table rendering | @@ -649,11 +704,15 @@ flowchart TD | `internal/channels/slack/reactions.go` | Status emoji reactions on messages | | `internal/channels/slack/stream.go` | Streaming message updates via placeholder editing | | `internal/channels/whatsapp/whatsapp.go` | WhatsApp: direct protocol client, QR auth, database persistence | -| `internal/channels/whatsapp/factory.go` | Channel factory, database dialect detection | +| `internal/channels/whatsapp/factory.go` | Channel factory with audio.Manager and builtin-tool store wiring | +| `internal/channels/whatsapp/stt.go` | Voice transcription with opt-in setting and E2E fallback | | `internal/channels/whatsapp/qr_methods.go` | QR code generation and authentication flow | | `internal/channels/whatsapp/format.go` | Message formatting (HTML-to-WhatsApp) | | `internal/channels/zalo/zalo.go` | Zalo OA: Bot API, long polling | | `internal/channels/zalo/personal/channel.go` | Zalo Personal: reverse-engineered protocol | +| `internal/audio/manager.go` | Audio manager: providers registry, lifecycle | +| `internal/audio/manager_stt.go` | STT chain resolution, Transcribe() entry point | +| `internal/audio/legacy_stt_bridge.go` | Backward-compat bridge for legacy STTProxyURL configs | | `internal/store/pg/pairing.go` | Pairing: code generation, approval, persistence (database-backed) | | `cmd/gateway_consumer.go` | Message routing: prefixes, cancel interception | @@ -668,3 +727,4 @@ flowchart TD | [08-scheduling-cron.md](./08-scheduling-cron.md) | /stop and /stopall commands, scheduler lanes, cron | | [09-security.md](./09-security.md) | Group file writer restrictions, security logging | | [11-agent-teams.md](./11-agent-teams.md) | Team message routing, delegation result delivery | +| [project-changelog.md](./project-changelog.md) | Phase 5 audio manager & unified STT implementation | diff --git a/docs/17-changelog.md b/docs/17-changelog.md index 2b1f0324..c1e29591 100644 --- a/docs/17-changelog.md +++ b/docs/17-changelog.md @@ -6,6 +6,39 @@ All notable changes to GoClaw Gateway are documented here. Format follows [Keep ## [Unreleased] — 2026-04-15 +#### ElevenLabs Audio Manager Refactor — Phase 5 (2026-04-15) + +Channel STT migration (Telegram/Feishu/Discord) + WhatsApp voice transcription with tenant opt-in (default disabled due to E2E encryption trade-off, resolves privacy red-team finding). + +### Added +- **WhatsApp voice transcription**: `internal/channels/whatsapp/stt.go` — opt-in per-tenant via `builtin_tools[stt].settings.whatsapp_enabled` (default `false` per Decision 6) +- **Synchronous 10s timeout**: All 4 channels (Telegram/Feishu/Discord/WhatsApp) enforce ctx timeout for voice message processing; timeout/error → fallback label +- **i18n fallback key**: `MsgVoiceMessageFallback` ("voice message" placeholder in en/vi/zh) rendered when STT unavailable or disabled +- **Privacy banner**: Admin UI displays E2E encryption trade-off warning when enabling WhatsApp STT + +### Changed +- **Channel STT handlers**: All 4 channels now call `audio.Manager.Transcribe()` with 10s timeout instead of per-channel `transcribeAudio()` helpers +- **Channel factory signatures**: All 4 factories (`NewChannel`) accept `audioMgr *audio.Manager` parameter +- **WhatsApp messaging**: Voice messages now transcribed (opt-in) and appended to message text for agent understanding + +### Removed +- **Per-channel STT helpers**: Deleted `internal/channels/{telegram,feishu,discord}/stt.go` (consolidated into Manager) +- **Telegram STT tests**: Deleted `internal/channels/telegram/stt_test.go` — coverage preserved in `internal/audio/proxy_stt/provider_test.go` (12 ported scenarios, deliberate deletion documented) + +### Resolved +- **Audit finding P5-B1**: Gate G4 verified; Manager.Transcribe chain wired +- **Audit finding P5-H1**: All transcribeAudio call sites migrated; grep confirms zero direct channel STT calls +- **Audit finding P5-H2** + **Decision 6** (privacy): WhatsApp STT opt-in with default=false; privacy banner + docs note E2E trade-off +- **Cross-phase XP-3**: Telegram test ported/deleted with 12-scenario coverage guarantee +- **Cross-phase XP-4**: `MsgVoiceMessageFallback` pre-wired in Phase 5 step 1 (blocking) + +### Security / Privacy +- **WhatsApp E2E encryption**: Voice transcription breaks end-to-end guarantee. **Tenant opt-in required** with explicit admin acknowledgment (privacy banner in UI) +- **Temp file cleanup**: Existing `scheduleMediaCleanup` preserves file handles during 10s STT window; no race condition risk +- **Audit logging**: WhatsApp STT invocation logged at info level for audit trail + +--- + #### ElevenLabs Audio Manager Refactor — Phase 4 (2026-04-15) STT via ElevenLabs Scribe + proxy fallback with tenant config migration + admin UI form. Legacy per-channel STT config bridged with 2-week deprecation grace.