From 4de94c1d7c9d7968e45d805f00c233f55088b59c Mon Sep 17 00:00:00 2001 From: Duy /zuey/ Date: Sun, 31 May 2026 12:23:40 +0700 Subject: [PATCH] fix(telegram): repair voice transcription routing Squash merge PR #102 after resolving dev changelog conflict. PR CI run 26704061639 passed release-versioning, go, and web. --- docs/03-tools-system.md | 2 +- docs/05-channels-messaging.md | 8 +- docs/project-changelog.md | 15 ++ internal/channels/telegram/handlers.go | 50 +++++-- internal/channels/telegram/media_test.go | 141 ++++++++++++++++++ internal/tools/read_audio_resolve.go | 36 +---- internal/tools/read_audio_resolve_test.go | 51 +++++++ ...-01-characterize-telegram-voice-failure.md | 43 ++++++ .../phase-02-fix-telegram-stt-routing.md | 40 +++++ .../phase-03-harden-read-audio-fallback.md | 41 +++++ .../phase-04-tests-and-docs.md | 37 +++++ .../plan.md | 38 +++++ ...60528-1837-telegram-voice-transcription.md | 59 ++++++++ 13 files changed, 508 insertions(+), 53 deletions(-) create mode 100644 plans/260528-1837-telegram-voice-transcription-fix/phase-01-characterize-telegram-voice-failure.md create mode 100644 plans/260528-1837-telegram-voice-transcription-fix/phase-02-fix-telegram-stt-routing.md create mode 100644 plans/260528-1837-telegram-voice-transcription-fix/phase-03-harden-read-audio-fallback.md create mode 100644 plans/260528-1837-telegram-voice-transcription-fix/phase-04-tests-and-docs.md create mode 100644 plans/260528-1837-telegram-voice-transcription-fix/plan.md create mode 100644 plans/260528-1837-telegram-voice-transcription-fix/reports/debugger-260528-1837-telegram-voice-transcription.md diff --git a/docs/03-tools-system.md b/docs/03-tools-system.md index fc054083..97dd65e8 100644 --- a/docs/03-tools-system.md +++ b/docs/03-tools-system.md @@ -164,7 +164,7 @@ Memory layers: L1 (`memory_search`) returns ranked abstracts; L2 (`memory_expand | Tool | Description | |---|---| | `read_image` | Analyze/describe images using vision AI (Gemini, Anthropic, OpenRouter, DashScope) | -| `read_audio` | Transcribe audio to text | +| `read_audio` | Transcribe audio to text using Gemini File API, native OpenAI audio input, or OpenAI-compatible transcription models; unsupported provider/model routes fail closed instead of sending audio as image data | | `read_document` | Extract and analyze documents (PDF, images) via Gemini or Resolve service | | `read_video` | Analyze/transcribe video content | diff --git a/docs/05-channels-messaging.md b/docs/05-channels-messaging.md index cc32f8a7..657a6d3c 100644 --- a/docs/05-channels-messaging.md +++ b/docs/05-channels-messaging.md @@ -287,7 +287,7 @@ flowchart TD WHATSAPP_CHECK -->|Yes| DOWNLOAD DOWNLOAD --> STT_CHECK{"STT providers
configured?"} - STT_CHECK -->|Yes| STT_CHAIN["Try providers in order:
elevenlabs_scribe, proxy"] + STT_CHECK -->|Yes| STT_CHAIN["Try providers in order:
elevenlabs, proxy"] STT_CHECK -->|No| LEGACY{"Legacy bridge
providers?"} LEGACY -->|Yes| STT_CHAIN @@ -309,11 +309,11 @@ flowchart TD #### Configuration & Decision Rules -**Decision 2 (Conflict rule):** When `builtin_tools[stt].settings.providers[]` is present in the database, it OVERRIDES all legacy channel-specific STT configs. The legacy STT bridge (`STTProxyURL` → `proxy_stt` provider) only activates when the providers list is empty or missing. +**Decision 2 (Conflict rule):** When `builtin_tools[stt].settings.providers[]` is present in the database, it OVERRIDES all legacy channel-specific STT configs. The legacy STT bridge (`STTProxyURL` → `proxy` provider) only activates when the providers list is empty or missing. | Setting | Behavior | |---------|----------| -| `providers: ["elevenlabs_scribe", "proxy_stt"]` | Try ElevenLabs Scribe first; fall back to legacy proxy | +| `providers: ["elevenlabs", "proxy"]` | Try ElevenLabs Scribe first; fall back to legacy proxy | | `providers: []` (empty) | Skip all STT; voice → `[Voice message]` fallback | | `providers` missing (nil) | Check for legacy bridge at startup; activate if `STTProxyURL` exists | @@ -331,7 +331,7 @@ When enabled: | Channel | STT Support | Notes | |---------|:-:|---------| -| **Telegram** | Yes | Uses unified chain; voice routing via `VoiceAgentID` config | +| **Telegram** | Yes | Uses unified chain; preserves Telegram voice MIME; legacy proxy override is keyed by platform type `telegram`; voice routing via `VoiceAgentID` config | | **Discord** | Yes | Standard flow; no special routing | | **Feishu** | Yes | Standard flow; no special routing | | **WhatsApp** | Yes (opt-in) | Requires explicit admin approval; default OFF | diff --git a/docs/project-changelog.md b/docs/project-changelog.md index 326e7cd2..3dd60edc 100644 --- a/docs/project-changelog.md +++ b/docs/project-changelog.md @@ -6,6 +6,21 @@ Significant changes, features, and fixes in reverse chronological order. ## 2026-05-29 +### Telegram voice transcription and read_audio fallback (issue #85) + +**Fixes** + +- Telegram voice/audio STT now preserves Telegram's detected MIME type, including `audio/ogg; codecs=opus`, instead of forcing every STT request to `audio/ogg`. +- Telegram STT now resolves channel-scoped legacy proxy overrides by platform type `telegram`, so DB channel instances with custom names still select the Telegram override. +- `read_audio` no longer sends unsupported audio routes as `providers.ImageContent`; unsupported provider/model combinations now fail closed with an audio-specific error that names supported routes. + +**Tests** + +- Added Telegram STT regression coverage for channel-type override selection and MIME preservation. +- Added `read_audio` regression coverage proving unsupported providers are not called through chat/image fallback. + +--- + ### RapidAPI cron SecureCLI diagnostics (issue #74) **Fixes** diff --git a/internal/channels/telegram/handlers.go b/internal/channels/telegram/handlers.go index bd92c93e..78a7f652 100644 --- a/internal/channels/telegram/handlers.go +++ b/internal/channels/telegram/handlers.go @@ -473,22 +473,13 @@ func (c *Channel) processResolvedMessage(ctx context.Context, rctx resolvedMessa m := &mediaList[i] switch m.Type { case "audio", "voice": - var transcript string - var sttErr error - if c.audioMgr != nil { - sttCtx, cancel := context.WithTimeout(ctx, 10*time.Second) - res, err := c.audioMgr.Transcribe(sttCtx, audio.STTInput{FilePath: m.FilePath, MimeType: "audio/ogg"}, audio.STTOptions{}) - cancel() - if err == nil && res != nil { - transcript = res.Text - } else { - sttErr = err - } - } + transcript, sttErr := c.transcribeMediaAudio(ctx, *m) if sttErr != nil { slog.Warn("telegram: STT transcription failed", - "type", m.Type, "error", sttErr) - } else { + "type", m.Type, + "mime", telegramAudioSTTMime(m.ContentType), + "error", sttErr) + } else if transcript != "" { m.Transcript = transcript } case "document": @@ -726,3 +717,34 @@ func (c *Channel) processResolvedMessage(ctx context.Context, rctx resolvedMessa c.GroupHistory().Clear(rctx.localKey) } } + +func (c *Channel) transcribeMediaAudio(ctx context.Context, m MediaInfo) (string, error) { + if c.audioMgr == nil { + return "", nil + } + + sttCtx, cancel := context.WithTimeout(ctx, 10*time.Second) + defer cancel() + sttCtx = audio.WithChannel(sttCtx, c.Type()) + + res, err := c.audioMgr.Transcribe(sttCtx, audio.STTInput{ + FilePath: m.FilePath, + MimeType: telegramAudioSTTMime(m.ContentType), + Filename: m.FileName, + }, audio.STTOptions{}) + if err != nil { + return "", err + } + if res == nil || strings.TrimSpace(res.Text) == "" { + return "", nil + } + return res.Text, nil +} + +func telegramAudioSTTMime(contentType string) string { + mime := strings.TrimSpace(contentType) + if mime == "" || strings.EqualFold(mime, "application/octet-stream") { + return "audio/ogg" + } + return mime +} diff --git a/internal/channels/telegram/media_test.go b/internal/channels/telegram/media_test.go index a9822c70..08446c30 100644 --- a/internal/channels/telegram/media_test.go +++ b/internal/channels/telegram/media_test.go @@ -1,15 +1,33 @@ package telegram import ( + "context" + "net/http" + "net/http/httptest" + "os" "strings" "testing" "github.com/mymmrac/telego" + "github.com/nextlevelbuilder/goclaw/internal/audio" "github.com/nextlevelbuilder/goclaw/internal/bus" "github.com/nextlevelbuilder/goclaw/internal/channels" + "github.com/nextlevelbuilder/goclaw/internal/config" ) +type telegramTestSTT struct { + name string + input audio.STTInput +} + +func (s *telegramTestSTT) Name() string { return s.name } + +func (s *telegramTestSTT) Transcribe(_ context.Context, in audio.STTInput, _ audio.STTOptions) (*audio.TranscriptResult, error) { + s.input = in + return &audio.TranscriptResult{Text: "xin chao", Provider: s.name}, nil +} + // --- buildMediaTags tests --- // TestBuildMediaTags_NoTranscript_Legacy verifies that the pre-patch behaviour is @@ -154,6 +172,129 @@ func TestBuildMediaTags_UnknownType(t *testing.T) { } } +func TestTranscribeMediaAudioUsesChannelTypeAndPreservesMIME(t *testing.T) { + mgr := audio.NewManager(audio.ManagerConfig{}) + tenantSTT := &telegramTestSTT{name: "tenant"} + telegramSTT := &telegramTestSTT{name: "proxy"} + mgr.RegisterSTT(tenantSTT) + mgr.SetSTTChain([]string{"tenant"}) + mgr.RegisterChannelSTT(channels.TypeTelegram, telegramSTT) + + ch := &Channel{ + BaseChannel: channels.NewBaseChannel("telegram-main", nil, nil), + audioMgr: mgr, + } + ch.SetType(channels.TypeTelegram) + + got, err := ch.transcribeMediaAudio(context.Background(), MediaInfo{ + Type: "voice", + FilePath: "/tmp/voice.ogg", + FileName: "voice.ogg", + ContentType: "audio/ogg; codecs=opus", + }) + if err != nil { + t.Fatalf("transcribeMediaAudio returned error: %v", err) + } + if got != "xin chao" { + t.Fatalf("transcript = %q, want channel override transcript", got) + } + if tenantSTT.input.FilePath != "" { + t.Fatalf("tenant STT received input; expected Telegram channel override to win") + } + if telegramSTT.input.MimeType != "audio/ogg; codecs=opus" { + t.Fatalf("mime = %q, want preserved Telegram voice MIME", telegramSTT.input.MimeType) + } + if telegramSTT.input.FilePath != "/tmp/voice.ogg" { + t.Fatalf("file path = %q, want /tmp/voice.ogg", telegramSTT.input.FilePath) + } +} + +func TestTelegramAudioSTTMimeFallback(t *testing.T) { + tests := []struct { + name string + contentType string + want string + }{ + {name: "preserves opus", contentType: "audio/ogg; codecs=opus", want: "audio/ogg; codecs=opus"}, + {name: "empty defaults ogg", contentType: "", want: "audio/ogg"}, + {name: "octet stream defaults ogg", contentType: "application/octet-stream", want: "audio/ogg"}, + {name: "trims whitespace", contentType: " audio/mpeg ", want: "audio/mpeg"}, + } + + for _, tt := range tests { + t.Run(tt.name, func(t *testing.T) { + if got := telegramAudioSTTMime(tt.contentType); got != tt.want { + t.Fatalf("telegramAudioSTTMime(%q) = %q, want %q", tt.contentType, got, tt.want) + } + }) + } +} + +func TestResolveMediaTelegramVoiceDownloadsOggOpusFixture(t *testing.T) { + const token = "123456789:aaaabbbbaaaabbbbaaaabbbbaaaabbbbccc" + const fileID = "telegram-voice-file" + oggOpus := []byte("OggS\x00\x02telegram-voice-opus-fixture") + + server := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { + switch r.URL.Path { + case "/bot" + token + "/getFile": + w.Header().Set("Content-Type", "application/json") + _, _ = w.Write([]byte(`{"ok":true,"result":{"file_id":"` + fileID + `","file_unique_id":"voice-unique","file_path":"voice/file.ogg"}}`)) + case "/file/bot" + token + "/voice/file.ogg": + w.Header().Set("Content-Type", "audio/ogg") + _, _ = w.Write(oggOpus) + default: + t.Fatalf("unexpected Telegram API path: %s", r.URL.Path) + } + })) + defer server.Close() + + ch, err := New(config.TelegramConfig{ + Token: token, + APIServer: server.URL, + }, nil, nil, nil) + if err != nil { + t.Fatalf("New channel: %v", err) + } + + mediaList, mediaErrors := ch.resolveMedia(context.Background(), &telego.Message{ + Voice: &telego.Voice{ + FileID: fileID, + MimeType: "audio/ogg; codecs=opus", + FileSize: int64(len(oggOpus)), + }, + }) + if len(mediaErrors) > 0 { + t.Fatalf("resolveMedia errors: %+v", mediaErrors) + } + if len(mediaList) != 1 { + t.Fatalf("got %d media items, want 1", len(mediaList)) + } + + voice := mediaList[0] + if voice.Type != "voice" { + t.Fatalf("media type = %q, want voice", voice.Type) + } + if voice.ContentType != "audio/ogg; codecs=opus" { + t.Fatalf("content type = %q, want Telegram voice MIME", voice.ContentType) + } + if voice.FilePath == "" { + t.Fatalf("expected downloaded file path") + } + t.Cleanup(func() { _ = os.Remove(voice.FilePath) }) + + data, err := os.ReadFile(voice.FilePath) + if err != nil { + t.Fatalf("read downloaded voice fixture: %v", err) + } + if !strings.HasPrefix(string(data), "OggS") { + t.Fatalf("downloaded fixture does not look like OGG data: %q", string(data)) + } + if got := telegramAudioSTTMime(voice.ContentType); got != "audio/ogg; codecs=opus" { + t.Fatalf("STT mime = %q, want preserved Telegram voice MIME", got) + } +} + func TestExtractMediaRefsPreservesDocumentMetadata(t *testing.T) { msg := &telego.Message{ Document: &telego.Document{ diff --git a/internal/tools/read_audio_resolve.go b/internal/tools/read_audio_resolve.go index f7f62a1e..bbfdcc0b 100644 --- a/internal/tools/read_audio_resolve.go +++ b/internal/tools/read_audio_resolve.go @@ -72,7 +72,7 @@ func (t *ReadAudioTool) resolveAudioFile(ctx context.Context, mediaID string) (p // callProvider dispatches audio analysis to the appropriate provider API. // Gemini: uses File API (upload → poll → file_data in generateContent). // OpenAI: uses input_audio content part in chat completions. -// Others: falls back to base64 in image_url (best effort). +// Transcription models: use the OpenAI-compatible audio transcription endpoint. func (t *ReadAudioTool) callProvider(ctx context.Context, cp credentialProvider, providerName, model string, params map[string]any) ([]byte, *providers.Usage, error) { prompt := GetParamString(params, "prompt", "Analyze this audio and describe its contents.") data, _ := params["data"].([]byte) @@ -157,39 +157,7 @@ func (t *ReadAudioTool) callProvider(ctx context.Context, cp credentialProvider, } } - // Other providers: try standard Chat API with base64 audio as image_url (best effort). - p, err := t.registry.Get(ctx, providerName) - if err != nil { - return nil, nil, fmt.Errorf("provider %q not available: %w", providerName, err) - } - - slog.Info("read_audio: using chat API fallback", "provider", providerName, "model", model, "size", len(data)) - chatReq := providers.ChatRequest{ - Messages: []providers.Message{ - { - Role: "user", - Content: prompt, - Images: []providers.ImageContent{{MimeType: mime, Data: base64.StdEncoding.EncodeToString(data)}}, - }, - }, - Model: model, - Options: map[string]any{ - "max_tokens": 16384, - "temperature": 0.2, - }, - } - reservation, reserveErr := reserveToolLLMUsage(ctx, t.usageCaps, t.Name(), providerName, model, chatReq) - if reserveErr != nil { - return nil, nil, reserveErr - } - resp, err := p.Chat(ctx, chatReq) - if reservation != nil { - reservation.Reconcile(ctx, resp, err) - } - if err != nil { - return nil, nil, fmt.Errorf("chat API: %w", err) - } - return []byte(resp.Content), resp.Usage, nil + return nil, nil, fmt.Errorf("read_audio: unsupported audio route for provider %q (type %q) model %q; supported routes are Gemini File API, native OpenAI input_audio, or OpenAI-compatible transcription models", providerName, ptype, model) } // openaiAudioCall sends audio to OpenAI using the input_audio content part. diff --git a/internal/tools/read_audio_resolve_test.go b/internal/tools/read_audio_resolve_test.go index 0323a9d5..df671dba 100644 --- a/internal/tools/read_audio_resolve_test.go +++ b/internal/tools/read_audio_resolve_test.go @@ -4,8 +4,31 @@ import ( "context" "strings" "testing" + + "github.com/nextlevelbuilder/goclaw/internal/providers" ) +type readAudioUnsupportedProvider struct { + name string + chatCalls int + images int +} + +func (p *readAudioUnsupportedProvider) Name() string { return p.name } +func (p *readAudioUnsupportedProvider) DefaultModel() string { return "claude-3-5-sonnet-latest" } + +func (p *readAudioUnsupportedProvider) Chat(_ context.Context, req providers.ChatRequest) (*providers.ChatResponse, error) { + p.chatCalls++ + for _, msg := range req.Messages { + p.images += len(msg.Images) + } + return &providers.ChatResponse{Content: "unexpected"}, nil +} + +func (p *readAudioUnsupportedProvider) ChatStream(_ context.Context, _ providers.ChatRequest, _ func(providers.StreamChunk)) (*providers.ChatResponse, error) { + return nil, nil +} + // TestReadAudioCallProvider_TranscriptionModelWithoutCreds_FailsFast asserts // that when no API credentials are present, a transcription-named model // returns a clear error rather than silently falling back to chat/completions @@ -71,3 +94,31 @@ func TestReadAudioCallProvider_GeminiWithoutCreds_FailsFast(t *testing.T) { t.Errorf("expected error to mention credentials, got: %v", err) } } + +func TestReadAudioCallProvider_UnsupportedProviderDoesNotSendAudioAsImage(t *testing.T) { + reg := providers.NewRegistry(nil) + fake := &readAudioUnsupportedProvider{name: "anthropic"} + reg.Register(fake) + tool := &ReadAudioTool{registry: reg} + + params := map[string]any{ + "_provider_type": "anthropic", + "data": []byte{0x4f, 0x67, 0x67, 0x53}, + "mime": "audio/ogg; codecs=opus", + } + + _, _, err := tool.callProvider(context.Background(), nil, "anthropic", "claude-3-5-sonnet-latest", params) + if err == nil { + t.Fatalf("expected unsupported audio route error, got nil") + } + msg := strings.ToLower(err.Error()) + if !strings.Contains(msg, "unsupported audio route") { + t.Fatalf("expected unsupported audio route error, got: %v", err) + } + if !strings.Contains(msg, "anthropic") || !strings.Contains(msg, "claude-3-5-sonnet-latest") { + t.Fatalf("expected provider and model in error, got: %v", err) + } + if fake.chatCalls != 0 || fake.images != 0 { + t.Fatalf("unsupported audio route called chat fallback: calls=%d images=%d", fake.chatCalls, fake.images) + } +} diff --git a/plans/260528-1837-telegram-voice-transcription-fix/phase-01-characterize-telegram-voice-failure.md b/plans/260528-1837-telegram-voice-transcription-fix/phase-01-characterize-telegram-voice-failure.md new file mode 100644 index 00000000..645b629b --- /dev/null +++ b/plans/260528-1837-telegram-voice-transcription-fix/phase-01-characterize-telegram-voice-failure.md @@ -0,0 +1,43 @@ +# Phase 01 - Characterize Telegram Voice Failure + +## Overview +- Priority: P0 +- Status: Complete +- Purpose: Lock current failing behavior before code changes. + +## Context Links +- Issue: https://github.com/digitopvn/goclaw/issues/85 +- Debug report: `reports/debugger-260528-1837-telegram-voice-transcription.md` + +## Requirements +- Build a small unit-level characterization for Telegram voice `MediaInfo`. +- Prove `msg.Voice.MimeType` is available and should not be discarded. +- Prove channel context is required for legacy Telegram STT proxy selection. +- Prove `read_audio` must not route audio bytes through image payload fallback. + +## Characterization Notes +- `resolveMedia` already classifies Telegram `msg.Voice` as `MediaInfo{Type: "voice"}` and copies `msg.Voice.MimeType` into `ContentType`; the loss happens later when `processResolvedMessage` hardcodes STT MIME as `audio/ogg`. +- `audio.Manager` already supports channel-scoped STT overrides through `audio.WithChannel(ctx, "telegram")`; Telegram does not set that context before calling `Transcribe`. +- `read_audio` still has a generic chat fallback that attaches audio bytes as `providers.ImageContent`, which matches the provider-side "image format illegal" failure. + +## Related Code Files +- Modify: `internal/channels/telegram/media_test.go` +- Modify: `internal/channels/telegram/handlers_test.go` or add focused helper tests near Telegram package. +- Modify: `internal/audio/manager_stt_test.go` only if existing coverage lacks the channel override case. + +## Implementation Steps +1. Add a Telegram voice media test using a `telego.Message{Voice: ...}` fixture. +2. Assert media type is `voice`, MIME is preserved, file path is populated by a stub download path. +3. Add/extend audio manager test proving `audio.WithChannel(ctx, "telegram")` selects the channel-scoped STT provider. +4. Avoid live Telegram or provider calls. + +## Success Criteria +- Deterministic regression tests can be added without live Telegram or provider calls. +- Phase 02 owns Telegram STT context/MIME test and fix. +- Phase 03 owns `read_audio` unsupported-provider regression and fix. + +## Risks +- `processResolvedMessage` is large; prefer extracting a narrow helper only if needed for testability. + +## Open Questions +- None. diff --git a/plans/260528-1837-telegram-voice-transcription-fix/phase-02-fix-telegram-stt-routing.md b/plans/260528-1837-telegram-voice-transcription-fix/phase-02-fix-telegram-stt-routing.md new file mode 100644 index 00000000..fcbaad92 --- /dev/null +++ b/plans/260528-1837-telegram-voice-transcription-fix/phase-02-fix-telegram-stt-routing.md @@ -0,0 +1,40 @@ +# Phase 02 - Fix Telegram STT Routing + +## Overview +- Priority: P0 +- Status: Complete +- Purpose: Make Telegram voice STT use the right MIME and provider chain. + +## Requirements +- Use `m.ContentType` for audio/voice STT, with safe fallback to `audio/ogg`. +- Wrap STT context with `audio.WithChannel(ctx, c.Type())`. +- Preserve current 10s timeout and fallback behavior. +- Do not change WhatsApp opt-in semantics. + +## Related Code Files +- Modify: `internal/channels/telegram/handlers.go` +- Optional: create helper in `internal/channels/telegram/stt.go` if `handlers.go` growth needs containment. + +## Implementation Steps +1. Introduce a small helper for audio MIME fallback: empty or `application/octet-stream` -> `audio/ogg`. +2. Replace hardcoded `MimeType: "audio/ogg"` with the helper result. +3. Call `audio.WithChannel(sttCtx, c.Type())` before `audioMgr.Transcribe`. +4. Preserve `m.Transcript = res.Text` only on non-empty success. +5. Log provider failure as Telegram audio STT failure with type, MIME, and provider error. + +## Implementation Notes +- Used `c.Type()` instead of `c.Name()` so DB channel instances like `telegram-main` still resolve the legacy override registered for platform type `telegram`. +- Added a focused regression where tenant STT is configured but Telegram channel-scoped STT wins and receives `audio/ogg; codecs=opus`. +- Empty transcripts remain non-fatal; no transcript tag is added. + +## Success Criteria +- Telegram legacy `stt_proxy_url` can be selected via channel override. +- Telegram OGG/Opus MIME survives into STT input. +- Existing Telegram media flow still publishes `bus.InboundMessage.Media`. + +## Security Considerations +- No new secret storage. +- Do not log raw audio path if it can expose tenant filesystem details beyond existing debug behavior; prefer basename when possible. + +## Open Questions +- None. diff --git a/plans/260528-1837-telegram-voice-transcription-fix/phase-03-harden-read-audio-fallback.md b/plans/260528-1837-telegram-voice-transcription-fix/phase-03-harden-read-audio-fallback.md new file mode 100644 index 00000000..0578a4f7 --- /dev/null +++ b/plans/260528-1837-telegram-voice-transcription-fix/phase-03-harden-read-audio-fallback.md @@ -0,0 +1,41 @@ +# Phase 03 - Harden Read Audio Fallback + +## Overview +- Priority: P0 +- Status: Complete +- Purpose: Stop audio being sent as image payload. + +## Requirements +- `read_audio` must never put audio bytes in `providers.ImageContent`. +- Unsupported provider/model routes must return an audio-specific error. +- Preserve working routes: + - Gemini File API. + - Native OpenAI input audio. + - OpenAI transcription endpoint for transcription model names. + +## Related Code Files +- Modify: `internal/tools/read_audio_resolve.go` +- Modify: `internal/tools/read_audio_resolve_test.go` +- Optional: `internal/tools/read_audio.go` description to mention ``. + +## Implementation Steps +1. Replace "Other providers" image fallback with explicit unsupported audio route error. +2. Include provider name, provider type, model, and accepted options in the error, without secrets. +3. Add regression test using an OpenAI-compatible non-transcription provider/model that previously went through `Images`. +4. Add test ensuring `` id still resolves through `read_audio`. + +## Implementation Notes +- Removed generic chat/image fallback from `read_audio`. +- Added regression proving unsupported audio routes do not call provider chat and do not create `providers.ImageContent`. +- Preserved Gemini File API, native OpenAI input audio, and OpenAI-compatible transcription model routes. + +## Success Criteria +- The observed "image format illegal" class cannot be produced by GoClaw for audio fallback. +- User-facing error says audio provider/model unsupported or STT provider unavailable. +- Existing Gemini/OpenAI/transcription tests still pass. + +## Risks +- Some provider may have relied on chat fallback for audio. Treat as unsafe legacy path because it is semantically wrong and produced this issue. + +## Open Questions +- DashScope/Qwen audio chat can be added later as a first-class verified route. diff --git a/plans/260528-1837-telegram-voice-transcription-fix/phase-04-tests-and-docs.md b/plans/260528-1837-telegram-voice-transcription-fix/phase-04-tests-and-docs.md new file mode 100644 index 00000000..256fa271 --- /dev/null +++ b/plans/260528-1837-telegram-voice-transcription-fix/phase-04-tests-and-docs.md @@ -0,0 +1,37 @@ +# Phase 04 - Tests and Docs + +## Overview +- Priority: P1 +- Status: Complete +- Purpose: Verify fix and sync public docs. + +## Requirements +- Add regression tests for Telegram voice, STT channel override, and read_audio no-image fallback. +- Update docs to reflect actual STT provider IDs and failure behavior. +- Do not add load/stress tests. + +## Related Code Files +- Modify: `docs/05-channels-messaging.md` +- Modify: `docs/03-tools-system.md` if read_audio contract changes. +- Modify: `docs/project-changelog.md` +- Run: package-level Go tests for touched packages. + +## Implementation Steps +1. Run `go test ./internal/channels/telegram ./internal/audio ./internal/tools`. +2. If shared code changes, run `go test ./internal/agent ./internal/channels/... ./internal/tools`. +3. Update docs after tests pass. +4. Record issue #85 in changelog with root cause and validation commands. + +## Implementation Notes +- Updated channel docs to use actual provider IDs: `elevenlabs` and `proxy`. +- Documented Telegram voice MIME preservation and platform-type STT override behavior. +- Documented `read_audio` fail-closed behavior for unsupported audio routes. +- Added a Telegram Bot API fixture test that downloads a voice OGG file and verifies `audio/ogg; codecs=opus` survives from `telego.Voice` through media resolution into STT MIME selection. + +## Success Criteria +- Tests pass locally. +- Docs no longer claim provider IDs that do not exist in code. +- Issue acceptance criteria are traceable to test names. + +## Open Questions +- None. diff --git a/plans/260528-1837-telegram-voice-transcription-fix/plan.md b/plans/260528-1837-telegram-voice-transcription-fix/plan.md new file mode 100644 index 00000000..b498a9a0 --- /dev/null +++ b/plans/260528-1837-telegram-voice-transcription-fix/plan.md @@ -0,0 +1,38 @@ +# Telegram Voice Transcription Fix Plan + +## Context +- Issue: https://github.com/digitopvn/goclaw/issues/85 +- Worktree: `/Users/duynguyen/.codex/worktrees/codex/issue-85-telegram-voice-transcription-plan` +- Branch: `codex/issue-85-telegram-voice-transcription-plan` +- Base: `origin/dev` at `5017f7ca` after merging latest dev into the issue branch +- Debug report: `reports/debugger-260528-1837-telegram-voice-transcription.md` + +## Goal +Fix Telegram app-recorded voice messages so `` is treated as audio end-to-end, transcribed through the STT path, and never routed through image payloads. + +## Verified Findings +- Telegram downloads `msg.Voice` as media type `voice` and carries `msg.Voice.MimeType`, but STT call currently forces MIME to `audio/ogg` instead of using the detected content type: `internal/channels/telegram/media.go:160`, `internal/channels/telegram/handlers.go:474`. +- Telegram calls `audioMgr.Transcribe` without `audio.WithChannel(ctx, c.Name())`, so channel-scoped legacy STT proxy overrides cannot be selected: `internal/audio/manager.go:17`, `internal/audio/manager_stt.go:69`, `internal/audio/legacy_stt_bridge.go:81`. +- When Telegram STT fails or is not configured, the message still reaches the agent with `` and media attachment. `read_audio` can then be invoked: `internal/channels/telegram/handlers.go:488`, `internal/channels/telegram/handlers.go:709`, `internal/agent/loop_input_media.go:120`. +- `read_audio` fallback for non-Gemini/non-OpenAI/non-transcription providers sends audio bytes as `providers.ImageContent`, matching the observed "image format illegal" provider error: `internal/tools/read_audio_resolve.go:160`. +- Docs say unified STT should cover Telegram and use the configured provider chain, but current names and behavior drift from implementation: `docs/05-channels-messaging.md:262`. + +## Phases +- [x] [Phase 01 - Characterize Failure](phase-01-characterize-telegram-voice-failure.md) +- [x] [Phase 02 - Fix Telegram STT Routing](phase-02-fix-telegram-stt-routing.md) +- [x] [Phase 03 - Harden Read Audio Fallback](phase-03-harden-read-audio-fallback.md) +- [x] [Phase 04 - Tests and Docs](phase-04-tests-and-docs.md) + +## Dependencies +- No schema migration expected. +- If adding audio transcoding, decide between built-in ffmpeg dependency vs provider-compatible no-conversion path first. +- Keep WhatsApp opt-in behavior unchanged. + +## Success Criteria +- Telegram `Voice` messages preserve `audio/ogg` / `audio/ogg; codecs=opus` metadata into STT and media refs. +- Channel-scoped STT proxy works for Telegram via `audio.WithChannel`. +- `read_audio` never sends audio as image payload for unsupported provider routes; errors are audio-specific and actionable. +- Regression tests cover Telegram voice media, STT channel context, and `read_audio` unsupported fallback. + +## Open Questions +- None. diff --git a/plans/260528-1837-telegram-voice-transcription-fix/reports/debugger-260528-1837-telegram-voice-transcription.md b/plans/260528-1837-telegram-voice-transcription-fix/reports/debugger-260528-1837-telegram-voice-transcription.md new file mode 100644 index 00000000..bcbf04bd --- /dev/null +++ b/plans/260528-1837-telegram-voice-transcription-fix/reports/debugger-260528-1837-telegram-voice-transcription.md @@ -0,0 +1,59 @@ +# Telegram Voice Transcription - Investigation Report + +## Executive Summary +- **Issue:** Telegram app-recorded voice messages fail transcription; provider returns image-format error. +- **Impact:** Telegram voice workflow blocked when auto-STT misses and agent/tool fallback is used. +- **Root cause:** Audio/voice can fall through to `read_audio`; unsupported provider fallback encodes audio as image content. +- **Status:** Root cause isolated; implementation plan ready. +- **Fix:** Preserve voice MIME + channel context in Telegram STT, then block or convert unsafe `read_audio` fallback. + +## Evidence +- Telegram voice is detected as voice: + - `internal/channels/telegram/media.go:160` handles `msg.Voice`. + - `internal/channels/telegram/media.go:167` appends `MediaInfo{Type: "voice"}`. + - `internal/channels/telegram/media.go:171` stores `msg.Voice.MimeType`. +- Telegram STT loses metadata: + - `internal/channels/telegram/handlers.go:474` enters audio/voice branch. + - `internal/channels/telegram/handlers.go:480` calls `audioMgr.Transcribe(... MimeType: "audio/ogg")`. + - This ignores `m.ContentType` and does not set `audio.WithChannel(ctx, c.Name())`. +- Channel-scoped STT exists but is skipped by Telegram call: + - `internal/audio/manager.go:17` defines `WithChannel`. + - `internal/audio/manager_stt.go:69` checks channel override before default chain. + - `internal/audio/legacy_stt_bridge.go:81` registers channel-scoped proxy providers for Telegram/Feishu/Discord. +- Fallback reaches agent/tool path: + - `internal/channels/telegram/handlers.go:488` only logs STT failure. + - `internal/channels/telegram/handlers.go:516` emits media tags, transcript optional. + - `internal/channels/telegram/handlers.go:709` publishes media files to the agent. + - `internal/agent/loop_input_media.go:120` collects audio refs for `read_audio`. +- Observed image-format error maps to tool fallback: + - `internal/tools/read_audio_resolve.go:72` documents provider dispatch. + - `internal/tools/read_audio_resolve.go:160` sends "Other providers" through chat API with `Images`. + - `internal/tools/read_audio_resolve.go:172` puts audio bytes in `providers.ImageContent`. + +## Root Cause +Primary root cause: `read_audio` has a best-effort chat fallback that represents audio as image data. For OpenAI-compatible providers without a transcription model name or dedicated audio route, this generates an image payload and surfaces provider errors like "image format illegal". + +Contributing causes: +- Telegram STT hardcodes `audio/ogg`, losing actual MIME detail such as `audio/ogg; codecs=opus`. +- Telegram STT does not attach channel context, so legacy channel-specific proxy STT cannot win. +- Telegram keeps processing after STT failure with only a warn log, so the LLM can call `read_audio` and hit the unsafe fallback. +- Docs describe a unified STT chain, but UI/code naming currently uses `elevenlabs` / `proxy`, while docs show `elevenlabs_scribe` / `proxy_stt`. + +## Recommended Plan +### Immediate (P0) +- Add focused tests proving Telegram voice path uses actual voice MIME and `audio.WithChannel`. +- Change Telegram STT call to pass `MimeType: fallbackAudioMime(m.ContentType)` and a channel-scoped context. +- Keep failure behavior user-friendly: log audio-specific cause and keep `` tag, but do not imply image analysis. + +### Short-Term (P1) +- Change `read_audio` fallback: if provider route is not Gemini, native OpenAI audio, or transcription endpoint, return clear audio-specific unsupported-provider error instead of `Images`. +- Consider explicit provider capability check for audio input/transcription before dispatch. +- Add regression around DashScope/openai-compatible non-transcription model to ensure no image payload is produced. + +### Longer-Term (P2) +- Decide conversion layer: ffmpeg OGG/Opus to WAV/MP3 before STT when selected provider cannot accept OGG/Opus. +- Sync docs and UI labels with actual provider ids. + +## Open Questions +- Should `builtin_tools[stt]` provider order drive Telegram STT now, or only after a separate STT settings migration? +- Should conversion be implemented now, or only after provider rejection proves it is needed?