fix(telegram): repair voice transcription routing

Squash merge PR #102 after resolving dev changelog conflict. PR CI run 26704061639 passed release-versioning, go, and web.
This commit is contained in:
Duy /zuey/ authored and GitHub committed 2026-05-31 12:23:40 +07:00
1 parent cf66bda618
commit 4de94c1d7c
13 files changed
+508 -53

No files matched your search

+1 -1
View File
@@ -164,7 +164,7 @@ Memory layers: L1 (`memory_search`) returns ranked abstracts; L2 (`memory_expand
| Tool | Description |
|---|---|
| `read_image` | Analyze/describe images using vision AI (Gemini, Anthropic, OpenRouter, DashScope) |
| `read_audio` | Transcribe audio to text |
| `read_audio` | Transcribe audio to text using Gemini File API, native OpenAI audio input, or OpenAI-compatible transcription models; unsupported provider/model routes fail closed instead of sending audio as image data |
| `read_document` | Extract and analyze documents (PDF, images) via Gemini or Resolve service |
| `read_video` | Analyze/transcribe video content |
+4 -4
View File
@@ -287,7 +287,7 @@ flowchart TD
WHATSAPP_CHECK -->|Yes| DOWNLOAD
DOWNLOAD --> STT_CHECK{"STT providers<br/>configured?"}
STT_CHECK -->|Yes| STT_CHAIN["Try providers in order:<br/>elevenlabs_scribe, proxy"]
STT_CHECK -->|Yes| STT_CHAIN["Try providers in order:<br/>elevenlabs, proxy"]
STT_CHECK -->|No| LEGACY{"Legacy bridge<br/>providers?"}
LEGACY -->|Yes| STT_CHAIN
@@ -309,11 +309,11 @@ flowchart TD
#### Configuration & Decision Rules
**Decision 2 (Conflict rule):** When `builtin_tools[stt].settings.providers[]` is present in the database, it OVERRIDES all legacy channel-specific STT configs. The legacy STT bridge (`STTProxyURL` → `proxy_stt` provider) only activates when the providers list is empty or missing.
**Decision 2 (Conflict rule):** When `builtin_tools[stt].settings.providers[]` is present in the database, it OVERRIDES all legacy channel-specific STT configs. The legacy STT bridge (`STTProxyURL` → `proxy` provider) only activates when the providers list is empty or missing.
| Setting | Behavior |
|---------|----------|
| `providers: ["elevenlabs_scribe", "proxy_stt"]` | Try ElevenLabs Scribe first; fall back to legacy proxy |
| `providers: ["elevenlabs", "proxy"]` | Try ElevenLabs Scribe first; fall back to legacy proxy |
| `providers: []` (empty) | Skip all STT; voice → `[Voice message]` fallback |
| `providers` missing (nil) | Check for legacy bridge at startup; activate if `STTProxyURL` exists |
@@ -331,7 +331,7 @@ When enabled:
| Channel | STT Support | Notes |
|---------|:-:|---------|
| **Telegram** | Yes | Uses unified chain; voice routing via `VoiceAgentID` config |
| **Telegram** | Yes | Uses unified chain; preserves Telegram voice MIME; legacy proxy override is keyed by platform type `telegram`; voice routing via `VoiceAgentID` config |
| **Discord** | Yes | Standard flow; no special routing |
| **Feishu** | Yes | Standard flow; no special routing |
| **WhatsApp** | Yes (opt-in) | Requires explicit admin approval; default OFF |
+15
View File
@@ -6,6 +6,21 @@ Significant changes, features, and fixes in reverse chronological order.
## 2026-05-29
### Telegram voice transcription and read_audio fallback (issue #85)
**Fixes**
- Telegram voice/audio STT now preserves Telegram's detected MIME type, including `audio/ogg; codecs=opus`, instead of forcing every STT request to `audio/ogg`.
- Telegram STT now resolves channel-scoped legacy proxy overrides by platform type `telegram`, so DB channel instances with custom names still select the Telegram override.
- `read_audio` no longer sends unsupported audio routes as `providers.ImageContent`; unsupported provider/model combinations now fail closed with an audio-specific error that names supported routes.
**Tests**
- Added Telegram STT regression coverage for channel-type override selection and MIME preservation.
- Added `read_audio` regression coverage proving unsupported providers are not called through chat/image fallback.
---
### RapidAPI cron SecureCLI diagnostics (issue #74)
**Fixes**
+36 -14
View File
@@ -473,22 +473,13 @@ func (c *Channel) processResolvedMessage(ctx context.Context, rctx resolvedMessa
m := &mediaList[i]
switch m.Type {
case "audio", "voice":
var transcript string
var sttErr error
if c.audioMgr != nil {
sttCtx, cancel := context.WithTimeout(ctx, 10*time.Second)
res, err := c.audioMgr.Transcribe(sttCtx, audio.STTInput{FilePath: m.FilePath, MimeType: "audio/ogg"}, audio.STTOptions{})
cancel()
if err == nil && res != nil {
transcript = res.Text
} else {
sttErr = err
}
}
transcript, sttErr := c.transcribeMediaAudio(ctx, *m)
if sttErr != nil {
slog.Warn("telegram: STT transcription failed",
"type", m.Type, "error", sttErr)
} else {
"type", m.Type,
"mime", telegramAudioSTTMime(m.ContentType),
"error", sttErr)
} else if transcript != "" {
m.Transcript = transcript
}
case "document":
@@ -726,3 +717,34 @@ func (c *Channel) processResolvedMessage(ctx context.Context, rctx resolvedMessa
c.GroupHistory().Clear(rctx.localKey)
}
}
func (c *Channel) transcribeMediaAudio(ctx context.Context, m MediaInfo) (string, error) {
if c.audioMgr == nil {
return "", nil
}
sttCtx, cancel := context.WithTimeout(ctx, 10*time.Second)
defer cancel()
sttCtx = audio.WithChannel(sttCtx, c.Type())
res, err := c.audioMgr.Transcribe(sttCtx, audio.STTInput{
FilePath: m.FilePath,
MimeType: telegramAudioSTTMime(m.ContentType),
Filename: m.FileName,
}, audio.STTOptions{})
if err != nil {
return "", err
}
if res == nil || strings.TrimSpace(res.Text) == "" {
return "", nil
}
return res.Text, nil
}
func telegramAudioSTTMime(contentType string) string {
mime := strings.TrimSpace(contentType)
if mime == "" || strings.EqualFold(mime, "application/octet-stream") {
return "audio/ogg"
}
return mime
}
+141
View File
@@ -1,15 +1,33 @@
package telegram
import (
"context"
"net/http"
"net/http/httptest"
"os"
"strings"
"testing"
"github.com/mymmrac/telego"
"github.com/nextlevelbuilder/goclaw/internal/audio"
"github.com/nextlevelbuilder/goclaw/internal/bus"
"github.com/nextlevelbuilder/goclaw/internal/channels"
"github.com/nextlevelbuilder/goclaw/internal/config"
)
type telegramTestSTT struct {
name string
input audio.STTInput
}
func (s *telegramTestSTT) Name() string { return s.name }
func (s *telegramTestSTT) Transcribe(_ context.Context, in audio.STTInput, _ audio.STTOptions) (*audio.TranscriptResult, error) {
s.input = in
return &audio.TranscriptResult{Text: "xin chao", Provider: s.name}, nil
}
// --- buildMediaTags tests ---
// TestBuildMediaTags_NoTranscript_Legacy verifies that the pre-patch behaviour is
@@ -154,6 +172,129 @@ func TestBuildMediaTags_UnknownType(t *testing.T) {
}
}
func TestTranscribeMediaAudioUsesChannelTypeAndPreservesMIME(t *testing.T) {
mgr := audio.NewManager(audio.ManagerConfig{})
tenantSTT := &telegramTestSTT{name: "tenant"}
telegramSTT := &telegramTestSTT{name: "proxy"}
mgr.RegisterSTT(tenantSTT)
mgr.SetSTTChain([]string{"tenant"})
mgr.RegisterChannelSTT(channels.TypeTelegram, telegramSTT)
ch := &Channel{
BaseChannel: channels.NewBaseChannel("telegram-main", nil, nil),
audioMgr: mgr,
}
ch.SetType(channels.TypeTelegram)
got, err := ch.transcribeMediaAudio(context.Background(), MediaInfo{
Type: "voice",
FilePath: "/tmp/voice.ogg",
FileName: "voice.ogg",
ContentType: "audio/ogg; codecs=opus",
})
if err != nil {
t.Fatalf("transcribeMediaAudio returned error: %v", err)
}
if got != "xin chao" {
t.Fatalf("transcript = %q, want channel override transcript", got)
}
if tenantSTT.input.FilePath != "" {
t.Fatalf("tenant STT received input; expected Telegram channel override to win")
}
if telegramSTT.input.MimeType != "audio/ogg; codecs=opus" {
t.Fatalf("mime = %q, want preserved Telegram voice MIME", telegramSTT.input.MimeType)
}
if telegramSTT.input.FilePath != "/tmp/voice.ogg" {
t.Fatalf("file path = %q, want /tmp/voice.ogg", telegramSTT.input.FilePath)
}
}
func TestTelegramAudioSTTMimeFallback(t *testing.T) {
tests := []struct {
name string
contentType string
want string
}{
{name: "preserves opus", contentType: "audio/ogg; codecs=opus", want: "audio/ogg; codecs=opus"},
{name: "empty defaults ogg", contentType: "", want: "audio/ogg"},
{name: "octet stream defaults ogg", contentType: "application/octet-stream", want: "audio/ogg"},
{name: "trims whitespace", contentType: " audio/mpeg ", want: "audio/mpeg"},
}
for _, tt := range tests {
t.Run(tt.name, func(t *testing.T) {
if got := telegramAudioSTTMime(tt.contentType); got != tt.want {
t.Fatalf("telegramAudioSTTMime(%q) = %q, want %q", tt.contentType, got, tt.want)
}
})
}
}
func TestResolveMediaTelegramVoiceDownloadsOggOpusFixture(t *testing.T) {
const token = "123456789:aaaabbbbaaaabbbbaaaabbbbaaaabbbbccc"
const fileID = "telegram-voice-file"
oggOpus := []byte("OggS\x00\x02telegram-voice-opus-fixture")
server := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
switch r.URL.Path {
case "/bot" + token + "/getFile":
w.Header().Set("Content-Type", "application/json")
_, _ = w.Write([]byte(`{"ok":true,"result":{"file_id":"` + fileID + `","file_unique_id":"voice-unique","file_path":"voice/file.ogg"}}`))
case "/file/bot" + token + "/voice/file.ogg":
w.Header().Set("Content-Type", "audio/ogg")
_, _ = w.Write(oggOpus)
default:
t.Fatalf("unexpected Telegram API path: %s", r.URL.Path)
}
}))
defer server.Close()
ch, err := New(config.TelegramConfig{
Token: token,
APIServer: server.URL,
}, nil, nil, nil)
if err != nil {
t.Fatalf("New channel: %v", err)
}
mediaList, mediaErrors := ch.resolveMedia(context.Background(), &telego.Message{
Voice: &telego.Voice{
FileID: fileID,
MimeType: "audio/ogg; codecs=opus",
FileSize: int64(len(oggOpus)),
},
})
if len(mediaErrors) > 0 {
t.Fatalf("resolveMedia errors: %+v", mediaErrors)
}
if len(mediaList) != 1 {
t.Fatalf("got %d media items, want 1", len(mediaList))
}
voice := mediaList[0]
if voice.Type != "voice" {
t.Fatalf("media type = %q, want voice", voice.Type)
}
if voice.ContentType != "audio/ogg; codecs=opus" {
t.Fatalf("content type = %q, want Telegram voice MIME", voice.ContentType)
}
if voice.FilePath == "" {
t.Fatalf("expected downloaded file path")
}
t.Cleanup(func() { _ = os.Remove(voice.FilePath) })
data, err := os.ReadFile(voice.FilePath)
if err != nil {
t.Fatalf("read downloaded voice fixture: %v", err)
}
if !strings.HasPrefix(string(data), "OggS") {
t.Fatalf("downloaded fixture does not look like OGG data: %q", string(data))
}
if got := telegramAudioSTTMime(voice.ContentType); got != "audio/ogg; codecs=opus" {
t.Fatalf("STT mime = %q, want preserved Telegram voice MIME", got)
}
}
func TestExtractMediaRefsPreservesDocumentMetadata(t *testing.T) {
msg := &telego.Message{
Document: &telego.Document{
+2 -34
View File
@@ -72,7 +72,7 @@ func (t *ReadAudioTool) resolveAudioFile(ctx context.Context, mediaID string) (p
// callProvider dispatches audio analysis to the appropriate provider API.
// Gemini: uses File API (upload → poll → file_data in generateContent).
// OpenAI: uses input_audio content part in chat completions.
// Others: falls back to base64 in image_url (best effort).
// Transcription models: use the OpenAI-compatible audio transcription endpoint.
func (t *ReadAudioTool) callProvider(ctx context.Context, cp credentialProvider, providerName, model string, params map[string]any) ([]byte, *providers.Usage, error) {
prompt := GetParamString(params, "prompt", "Analyze this audio and describe its contents.")
data, _ := params["data"].([]byte)
@@ -157,39 +157,7 @@ func (t *ReadAudioTool) callProvider(ctx context.Context, cp credentialProvider,
}
}
// Other providers: try standard Chat API with base64 audio as image_url (best effort).
p, err := t.registry.Get(ctx, providerName)
if err != nil {
return nil, nil, fmt.Errorf("provider %q not available: %w", providerName, err)
}
slog.Info("read_audio: using chat API fallback", "provider", providerName, "model", model, "size", len(data))
chatReq := providers.ChatRequest{
Messages: []providers.Message{
{
Role: "user",
Content: prompt,
Images: []providers.ImageContent{{MimeType: mime, Data: base64.StdEncoding.EncodeToString(data)}},
},
},
Model: model,
Options: map[string]any{
"max_tokens": 16384,
"temperature": 0.2,
},
}
reservation, reserveErr := reserveToolLLMUsage(ctx, t.usageCaps, t.Name(), providerName, model, chatReq)
if reserveErr != nil {
return nil, nil, reserveErr
}
resp, err := p.Chat(ctx, chatReq)
if reservation != nil {
reservation.Reconcile(ctx, resp, err)
}
if err != nil {
return nil, nil, fmt.Errorf("chat API: %w", err)
}
return []byte(resp.Content), resp.Usage, nil
return nil, nil, fmt.Errorf("read_audio: unsupported audio route for provider %q (type %q) model %q; supported routes are Gemini File API, native OpenAI input_audio, or OpenAI-compatible transcription models", providerName, ptype, model)
}
// openaiAudioCall sends audio to OpenAI using the input_audio content part.
+51
View File
@@ -4,8 +4,31 @@ import (
"context"
"strings"
"testing"
"github.com/nextlevelbuilder/goclaw/internal/providers"
)
type readAudioUnsupportedProvider struct {
name string
chatCalls int
images int
}
func (p *readAudioUnsupportedProvider) Name() string { return p.name }
func (p *readAudioUnsupportedProvider) DefaultModel() string { return "claude-3-5-sonnet-latest" }
func (p *readAudioUnsupportedProvider) Chat(_ context.Context, req providers.ChatRequest) (*providers.ChatResponse, error) {
p.chatCalls++
for _, msg := range req.Messages {
p.images += len(msg.Images)
}
return &providers.ChatResponse{Content: "unexpected"}, nil
}
func (p *readAudioUnsupportedProvider) ChatStream(_ context.Context, _ providers.ChatRequest, _ func(providers.StreamChunk)) (*providers.ChatResponse, error) {
return nil, nil
}
// TestReadAudioCallProvider_TranscriptionModelWithoutCreds_FailsFast asserts
// that when no API credentials are present, a transcription-named model
// returns a clear error rather than silently falling back to chat/completions
@@ -71,3 +94,31 @@ func TestReadAudioCallProvider_GeminiWithoutCreds_FailsFast(t *testing.T) {
t.Errorf("expected error to mention credentials, got: %v", err)
}
}
func TestReadAudioCallProvider_UnsupportedProviderDoesNotSendAudioAsImage(t *testing.T) {
reg := providers.NewRegistry(nil)
fake := &readAudioUnsupportedProvider{name: "anthropic"}
reg.Register(fake)
tool := &ReadAudioTool{registry: reg}
params := map[string]any{
"_provider_type": "anthropic",
"data": []byte{0x4f, 0x67, 0x67, 0x53},
"mime": "audio/ogg; codecs=opus",
}
_, _, err := tool.callProvider(context.Background(), nil, "anthropic", "claude-3-5-sonnet-latest", params)
if err == nil {
t.Fatalf("expected unsupported audio route error, got nil")
}
msg := strings.ToLower(err.Error())
if !strings.Contains(msg, "unsupported audio route") {
t.Fatalf("expected unsupported audio route error, got: %v", err)
}
if !strings.Contains(msg, "anthropic") || !strings.Contains(msg, "claude-3-5-sonnet-latest") {
t.Fatalf("expected provider and model in error, got: %v", err)
}
if fake.chatCalls != 0 || fake.images != 0 {
t.Fatalf("unsupported audio route called chat fallback: calls=%d images=%d", fake.chatCalls, fake.images)
}
}
@@ -0,0 +1,43 @@
# Phase 01 - Characterize Telegram Voice Failure
## Overview
- Priority: P0
- Status: Complete
- Purpose: Lock current failing behavior before code changes.
## Context Links
- Issue: https://github.com/digitopvn/goclaw/issues/85
- Debug report: `reports/debugger-260528-1837-telegram-voice-transcription.md`
## Requirements
- Build a small unit-level characterization for Telegram voice `MediaInfo`.
- Prove `msg.Voice.MimeType` is available and should not be discarded.
- Prove channel context is required for legacy Telegram STT proxy selection.
- Prove `read_audio` must not route audio bytes through image payload fallback.
## Characterization Notes
- `resolveMedia` already classifies Telegram `msg.Voice` as `MediaInfo{Type: "voice"}` and copies `msg.Voice.MimeType` into `ContentType`; the loss happens later when `processResolvedMessage` hardcodes STT MIME as `audio/ogg`.
- `audio.Manager` already supports channel-scoped STT overrides through `audio.WithChannel(ctx, "telegram")`; Telegram does not set that context before calling `Transcribe`.
- `read_audio` still has a generic chat fallback that attaches audio bytes as `providers.ImageContent`, which matches the provider-side "image format illegal" failure.
## Related Code Files
- Modify: `internal/channels/telegram/media_test.go`
- Modify: `internal/channels/telegram/handlers_test.go` or add focused helper tests near Telegram package.
- Modify: `internal/audio/manager_stt_test.go` only if existing coverage lacks the channel override case.
## Implementation Steps
1. Add a Telegram voice media test using a `telego.Message{Voice: ...}` fixture.
2. Assert media type is `voice`, MIME is preserved, file path is populated by a stub download path.
3. Add/extend audio manager test proving `audio.WithChannel(ctx, "telegram")` selects the channel-scoped STT provider.
4. Avoid live Telegram or provider calls.
## Success Criteria
- Deterministic regression tests can be added without live Telegram or provider calls.
- Phase 02 owns Telegram STT context/MIME test and fix.
- Phase 03 owns `read_audio` unsupported-provider regression and fix.
## Risks
- `processResolvedMessage` is large; prefer extracting a narrow helper only if needed for testability.
## Open Questions
- None.
@@ -0,0 +1,40 @@
# Phase 02 - Fix Telegram STT Routing
## Overview
- Priority: P0
- Status: Complete
- Purpose: Make Telegram voice STT use the right MIME and provider chain.
## Requirements
- Use `m.ContentType` for audio/voice STT, with safe fallback to `audio/ogg`.
- Wrap STT context with `audio.WithChannel(ctx, c.Type())`.
- Preserve current 10s timeout and fallback behavior.
- Do not change WhatsApp opt-in semantics.
## Related Code Files
- Modify: `internal/channels/telegram/handlers.go`
- Optional: create helper in `internal/channels/telegram/stt.go` if `handlers.go` growth needs containment.
## Implementation Steps
1. Introduce a small helper for audio MIME fallback: empty or `application/octet-stream` -> `audio/ogg`.
2. Replace hardcoded `MimeType: "audio/ogg"` with the helper result.
3. Call `audio.WithChannel(sttCtx, c.Type())` before `audioMgr.Transcribe`.
4. Preserve `m.Transcript = res.Text` only on non-empty success.
5. Log provider failure as Telegram audio STT failure with type, MIME, and provider error.
## Implementation Notes
- Used `c.Type()` instead of `c.Name()` so DB channel instances like `telegram-main` still resolve the legacy override registered for platform type `telegram`.
- Added a focused regression where tenant STT is configured but Telegram channel-scoped STT wins and receives `audio/ogg; codecs=opus`.
- Empty transcripts remain non-fatal; no transcript tag is added.
## Success Criteria
- Telegram legacy `stt_proxy_url` can be selected via channel override.
- Telegram OGG/Opus MIME survives into STT input.
- Existing Telegram media flow still publishes `bus.InboundMessage.Media`.
## Security Considerations
- No new secret storage.
- Do not log raw audio path if it can expose tenant filesystem details beyond existing debug behavior; prefer basename when possible.
## Open Questions
- None.
@@ -0,0 +1,41 @@
# Phase 03 - Harden Read Audio Fallback
## Overview
- Priority: P0
- Status: Complete
- Purpose: Stop audio being sent as image payload.
## Requirements
- `read_audio` must never put audio bytes in `providers.ImageContent`.
- Unsupported provider/model routes must return an audio-specific error.
- Preserve working routes:
- Gemini File API.
- Native OpenAI input audio.
- OpenAI transcription endpoint for transcription model names.
## Related Code Files
- Modify: `internal/tools/read_audio_resolve.go`
- Modify: `internal/tools/read_audio_resolve_test.go`
- Optional: `internal/tools/read_audio.go` description to mention `<media:voice>`.
## Implementation Steps
1. Replace "Other providers" image fallback with explicit unsupported audio route error.
2. Include provider name, provider type, model, and accepted options in the error, without secrets.
3. Add regression test using an OpenAI-compatible non-transcription provider/model that previously went through `Images`.
4. Add test ensuring `<media:voice>` id still resolves through `read_audio`.
## Implementation Notes
- Removed generic chat/image fallback from `read_audio`.
- Added regression proving unsupported audio routes do not call provider chat and do not create `providers.ImageContent`.
- Preserved Gemini File API, native OpenAI input audio, and OpenAI-compatible transcription model routes.
## Success Criteria
- The observed "image format illegal" class cannot be produced by GoClaw for audio fallback.
- User-facing error says audio provider/model unsupported or STT provider unavailable.
- Existing Gemini/OpenAI/transcription tests still pass.
## Risks
- Some provider may have relied on chat fallback for audio. Treat as unsafe legacy path because it is semantically wrong and produced this issue.
## Open Questions
- DashScope/Qwen audio chat can be added later as a first-class verified route.
@@ -0,0 +1,37 @@
# Phase 04 - Tests and Docs
## Overview
- Priority: P1
- Status: Complete
- Purpose: Verify fix and sync public docs.
## Requirements
- Add regression tests for Telegram voice, STT channel override, and read_audio no-image fallback.
- Update docs to reflect actual STT provider IDs and failure behavior.
- Do not add load/stress tests.
## Related Code Files
- Modify: `docs/05-channels-messaging.md`
- Modify: `docs/03-tools-system.md` if read_audio contract changes.
- Modify: `docs/project-changelog.md`
- Run: package-level Go tests for touched packages.
## Implementation Steps
1. Run `go test ./internal/channels/telegram ./internal/audio ./internal/tools`.
2. If shared code changes, run `go test ./internal/agent ./internal/channels/... ./internal/tools`.
3. Update docs after tests pass.
4. Record issue #85 in changelog with root cause and validation commands.
## Implementation Notes
- Updated channel docs to use actual provider IDs: `elevenlabs` and `proxy`.
- Documented Telegram voice MIME preservation and platform-type STT override behavior.
- Documented `read_audio` fail-closed behavior for unsupported audio routes.
- Added a Telegram Bot API fixture test that downloads a voice OGG file and verifies `audio/ogg; codecs=opus` survives from `telego.Voice` through media resolution into STT MIME selection.
## Success Criteria
- Tests pass locally.
- Docs no longer claim provider IDs that do not exist in code.
- Issue acceptance criteria are traceable to test names.
## Open Questions
- None.
@@ -0,0 +1,38 @@
# Telegram Voice Transcription Fix Plan
## Context
- Issue: https://github.com/digitopvn/goclaw/issues/85
- Worktree: `/Users/duynguyen/.codex/worktrees/codex/issue-85-telegram-voice-transcription-plan`
- Branch: `codex/issue-85-telegram-voice-transcription-plan`
- Base: `origin/dev` at `5017f7ca` after merging latest dev into the issue branch
- Debug report: `reports/debugger-260528-1837-telegram-voice-transcription.md`
## Goal
Fix Telegram app-recorded voice messages so `<media:voice>` is treated as audio end-to-end, transcribed through the STT path, and never routed through image payloads.
## Verified Findings
- Telegram downloads `msg.Voice` as media type `voice` and carries `msg.Voice.MimeType`, but STT call currently forces MIME to `audio/ogg` instead of using the detected content type: `internal/channels/telegram/media.go:160`, `internal/channels/telegram/handlers.go:474`.
- Telegram calls `audioMgr.Transcribe` without `audio.WithChannel(ctx, c.Name())`, so channel-scoped legacy STT proxy overrides cannot be selected: `internal/audio/manager.go:17`, `internal/audio/manager_stt.go:69`, `internal/audio/legacy_stt_bridge.go:81`.
- When Telegram STT fails or is not configured, the message still reaches the agent with `<media:voice>` and media attachment. `read_audio` can then be invoked: `internal/channels/telegram/handlers.go:488`, `internal/channels/telegram/handlers.go:709`, `internal/agent/loop_input_media.go:120`.
- `read_audio` fallback for non-Gemini/non-OpenAI/non-transcription providers sends audio bytes as `providers.ImageContent`, matching the observed "image format illegal" provider error: `internal/tools/read_audio_resolve.go:160`.
- Docs say unified STT should cover Telegram and use the configured provider chain, but current names and behavior drift from implementation: `docs/05-channels-messaging.md:262`.
## Phases
- [x] [Phase 01 - Characterize Failure](phase-01-characterize-telegram-voice-failure.md)
- [x] [Phase 02 - Fix Telegram STT Routing](phase-02-fix-telegram-stt-routing.md)
- [x] [Phase 03 - Harden Read Audio Fallback](phase-03-harden-read-audio-fallback.md)
- [x] [Phase 04 - Tests and Docs](phase-04-tests-and-docs.md)
## Dependencies
- No schema migration expected.
- If adding audio transcoding, decide between built-in ffmpeg dependency vs provider-compatible no-conversion path first.
- Keep WhatsApp opt-in behavior unchanged.
## Success Criteria
- Telegram `Voice` messages preserve `audio/ogg` / `audio/ogg; codecs=opus` metadata into STT and media refs.
- Channel-scoped STT proxy works for Telegram via `audio.WithChannel`.
- `read_audio` never sends audio as image payload for unsupported provider routes; errors are audio-specific and actionable.
- Regression tests cover Telegram voice media, STT channel context, and `read_audio` unsupported fallback.
## Open Questions
- None.
@@ -0,0 +1,59 @@
# Telegram Voice Transcription - Investigation Report
## Executive Summary
- **Issue:** Telegram app-recorded voice messages fail transcription; provider returns image-format error.
- **Impact:** Telegram voice workflow blocked when auto-STT misses and agent/tool fallback is used.
- **Root cause:** Audio/voice can fall through to `read_audio`; unsupported provider fallback encodes audio as image content.
- **Status:** Root cause isolated; implementation plan ready.
- **Fix:** Preserve voice MIME + channel context in Telegram STT, then block or convert unsafe `read_audio` fallback.
## Evidence
- Telegram voice is detected as voice:
- `internal/channels/telegram/media.go:160` handles `msg.Voice`.
- `internal/channels/telegram/media.go:167` appends `MediaInfo{Type: "voice"}`.
- `internal/channels/telegram/media.go:171` stores `msg.Voice.MimeType`.
- Telegram STT loses metadata:
- `internal/channels/telegram/handlers.go:474` enters audio/voice branch.
- `internal/channels/telegram/handlers.go:480` calls `audioMgr.Transcribe(... MimeType: "audio/ogg")`.
- This ignores `m.ContentType` and does not set `audio.WithChannel(ctx, c.Name())`.
- Channel-scoped STT exists but is skipped by Telegram call:
- `internal/audio/manager.go:17` defines `WithChannel`.
- `internal/audio/manager_stt.go:69` checks channel override before default chain.
- `internal/audio/legacy_stt_bridge.go:81` registers channel-scoped proxy providers for Telegram/Feishu/Discord.
- Fallback reaches agent/tool path:
- `internal/channels/telegram/handlers.go:488` only logs STT failure.
- `internal/channels/telegram/handlers.go:516` emits media tags, transcript optional.
- `internal/channels/telegram/handlers.go:709` publishes media files to the agent.
- `internal/agent/loop_input_media.go:120` collects audio refs for `read_audio`.
- Observed image-format error maps to tool fallback:
- `internal/tools/read_audio_resolve.go:72` documents provider dispatch.
- `internal/tools/read_audio_resolve.go:160` sends "Other providers" through chat API with `Images`.
- `internal/tools/read_audio_resolve.go:172` puts audio bytes in `providers.ImageContent`.
## Root Cause
Primary root cause: `read_audio` has a best-effort chat fallback that represents audio as image data. For OpenAI-compatible providers without a transcription model name or dedicated audio route, this generates an image payload and surfaces provider errors like "image format illegal".
Contributing causes:
- Telegram STT hardcodes `audio/ogg`, losing actual MIME detail such as `audio/ogg; codecs=opus`.
- Telegram STT does not attach channel context, so legacy channel-specific proxy STT cannot win.
- Telegram keeps processing after STT failure with only a warn log, so the LLM can call `read_audio` and hit the unsafe fallback.
- Docs describe a unified STT chain, but UI/code naming currently uses `elevenlabs` / `proxy`, while docs show `elevenlabs_scribe` / `proxy_stt`.
## Recommended Plan
### Immediate (P0)
- Add focused tests proving Telegram voice path uses actual voice MIME and `audio.WithChannel`.
- Change Telegram STT call to pass `MimeType: fallbackAudioMime(m.ContentType)` and a channel-scoped context.
- Keep failure behavior user-friendly: log audio-specific cause and keep `<media:voice>` tag, but do not imply image analysis.
### Short-Term (P1)
- Change `read_audio` fallback: if provider route is not Gemini, native OpenAI audio, or transcription endpoint, return clear audio-specific unsupported-provider error instead of `Images`.
- Consider explicit provider capability check for audio input/transcription before dispatch.
- Add regression around DashScope/openai-compatible non-transcription model to ensure no image payload is produced.
### Longer-Term (P2)
- Decide conversion layer: ffmpeg OGG/Opus to WAV/MP3 before STT when selected provider cannot accept OGG/Opus.
- Sync docs and UI labels with actual provider ids.
## Open Questions
- Should `builtin_tools[stt]` provider order drive Telegram STT now, or only after a separate STT settings migration?
- Should conversion be implemented now, or only after provider rejection proves it is needed?