mirror of
https://github.com/tiennm99/goclaw.git
synced 2026-10-19 08:15:12 +00:00
fix(telegram): repair voice transcription routing
Squash merge PR #102 after resolving dev changelog conflict. PR CI run 26704061639 passed release-versioning, go, and web.
This commit is contained in:
1 parent
cf66bda618
commit
4de94c1d7c
13 files changed
+508
-53
No files matched your search
@@ -164,7 +164,7 @@ Memory layers: L1 (`memory_search`) returns ranked abstracts; L2 (`memory_expand
|
||||
| Tool | Description |
|
||||
|---|---|
|
||||
| `read_image` | Analyze/describe images using vision AI (Gemini, Anthropic, OpenRouter, DashScope) |
|
||||
| `read_audio` | Transcribe audio to text |
|
||||
| `read_audio` | Transcribe audio to text using Gemini File API, native OpenAI audio input, or OpenAI-compatible transcription models; unsupported provider/model routes fail closed instead of sending audio as image data |
|
||||
| `read_document` | Extract and analyze documents (PDF, images) via Gemini or Resolve service |
|
||||
| `read_video` | Analyze/transcribe video content |
|
||||
|
||||
|
||||
@@ -287,7 +287,7 @@ flowchart TD
|
||||
WHATSAPP_CHECK -->|Yes| DOWNLOAD
|
||||
|
||||
DOWNLOAD --> STT_CHECK{"STT providers<br/>configured?"}
|
||||
STT_CHECK -->|Yes| STT_CHAIN["Try providers in order:<br/>elevenlabs_scribe, proxy"]
|
||||
STT_CHECK -->|Yes| STT_CHAIN["Try providers in order:<br/>elevenlabs, proxy"]
|
||||
STT_CHECK -->|No| LEGACY{"Legacy bridge<br/>providers?"}
|
||||
|
||||
LEGACY -->|Yes| STT_CHAIN
|
||||
@@ -309,11 +309,11 @@ flowchart TD
|
||||
|
||||
#### Configuration & Decision Rules
|
||||
|
||||
**Decision 2 (Conflict rule):** When `builtin_tools[stt].settings.providers[]` is present in the database, it OVERRIDES all legacy channel-specific STT configs. The legacy STT bridge (`STTProxyURL` → `proxy_stt` provider) only activates when the providers list is empty or missing.
|
||||
**Decision 2 (Conflict rule):** When `builtin_tools[stt].settings.providers[]` is present in the database, it OVERRIDES all legacy channel-specific STT configs. The legacy STT bridge (`STTProxyURL` → `proxy` provider) only activates when the providers list is empty or missing.
|
||||
|
||||
| Setting | Behavior |
|
||||
|---------|----------|
|
||||
| `providers: ["elevenlabs_scribe", "proxy_stt"]` | Try ElevenLabs Scribe first; fall back to legacy proxy |
|
||||
| `providers: ["elevenlabs", "proxy"]` | Try ElevenLabs Scribe first; fall back to legacy proxy |
|
||||
| `providers: []` (empty) | Skip all STT; voice → `[Voice message]` fallback |
|
||||
| `providers` missing (nil) | Check for legacy bridge at startup; activate if `STTProxyURL` exists |
|
||||
|
||||
@@ -331,7 +331,7 @@ When enabled:
|
||||
|
||||
| Channel | STT Support | Notes |
|
||||
|---------|:-:|---------|
|
||||
| **Telegram** | Yes | Uses unified chain; voice routing via `VoiceAgentID` config |
|
||||
| **Telegram** | Yes | Uses unified chain; preserves Telegram voice MIME; legacy proxy override is keyed by platform type `telegram`; voice routing via `VoiceAgentID` config |
|
||||
| **Discord** | Yes | Standard flow; no special routing |
|
||||
| **Feishu** | Yes | Standard flow; no special routing |
|
||||
| **WhatsApp** | Yes (opt-in) | Requires explicit admin approval; default OFF |
|
||||
|
||||
@@ -6,6 +6,21 @@ Significant changes, features, and fixes in reverse chronological order.
|
||||
|
||||
## 2026-05-29
|
||||
|
||||
### Telegram voice transcription and read_audio fallback (issue #85)
|
||||
|
||||
**Fixes**
|
||||
|
||||
- Telegram voice/audio STT now preserves Telegram's detected MIME type, including `audio/ogg; codecs=opus`, instead of forcing every STT request to `audio/ogg`.
|
||||
- Telegram STT now resolves channel-scoped legacy proxy overrides by platform type `telegram`, so DB channel instances with custom names still select the Telegram override.
|
||||
- `read_audio` no longer sends unsupported audio routes as `providers.ImageContent`; unsupported provider/model combinations now fail closed with an audio-specific error that names supported routes.
|
||||
|
||||
**Tests**
|
||||
|
||||
- Added Telegram STT regression coverage for channel-type override selection and MIME preservation.
|
||||
- Added `read_audio` regression coverage proving unsupported providers are not called through chat/image fallback.
|
||||
|
||||
---
|
||||
|
||||
### RapidAPI cron SecureCLI diagnostics (issue #74)
|
||||
|
||||
**Fixes**
|
||||
|
||||
@@ -473,22 +473,13 @@ func (c *Channel) processResolvedMessage(ctx context.Context, rctx resolvedMessa
|
||||
m := &mediaList[i]
|
||||
switch m.Type {
|
||||
case "audio", "voice":
|
||||
var transcript string
|
||||
var sttErr error
|
||||
if c.audioMgr != nil {
|
||||
sttCtx, cancel := context.WithTimeout(ctx, 10*time.Second)
|
||||
res, err := c.audioMgr.Transcribe(sttCtx, audio.STTInput{FilePath: m.FilePath, MimeType: "audio/ogg"}, audio.STTOptions{})
|
||||
cancel()
|
||||
if err == nil && res != nil {
|
||||
transcript = res.Text
|
||||
} else {
|
||||
sttErr = err
|
||||
}
|
||||
}
|
||||
transcript, sttErr := c.transcribeMediaAudio(ctx, *m)
|
||||
if sttErr != nil {
|
||||
slog.Warn("telegram: STT transcription failed",
|
||||
"type", m.Type, "error", sttErr)
|
||||
} else {
|
||||
"type", m.Type,
|
||||
"mime", telegramAudioSTTMime(m.ContentType),
|
||||
"error", sttErr)
|
||||
} else if transcript != "" {
|
||||
m.Transcript = transcript
|
||||
}
|
||||
case "document":
|
||||
@@ -726,3 +717,34 @@ func (c *Channel) processResolvedMessage(ctx context.Context, rctx resolvedMessa
|
||||
c.GroupHistory().Clear(rctx.localKey)
|
||||
}
|
||||
}
|
||||
|
||||
func (c *Channel) transcribeMediaAudio(ctx context.Context, m MediaInfo) (string, error) {
|
||||
if c.audioMgr == nil {
|
||||
return "", nil
|
||||
}
|
||||
|
||||
sttCtx, cancel := context.WithTimeout(ctx, 10*time.Second)
|
||||
defer cancel()
|
||||
sttCtx = audio.WithChannel(sttCtx, c.Type())
|
||||
|
||||
res, err := c.audioMgr.Transcribe(sttCtx, audio.STTInput{
|
||||
FilePath: m.FilePath,
|
||||
MimeType: telegramAudioSTTMime(m.ContentType),
|
||||
Filename: m.FileName,
|
||||
}, audio.STTOptions{})
|
||||
if err != nil {
|
||||
return "", err
|
||||
}
|
||||
if res == nil || strings.TrimSpace(res.Text) == "" {
|
||||
return "", nil
|
||||
}
|
||||
return res.Text, nil
|
||||
}
|
||||
|
||||
func telegramAudioSTTMime(contentType string) string {
|
||||
mime := strings.TrimSpace(contentType)
|
||||
if mime == "" || strings.EqualFold(mime, "application/octet-stream") {
|
||||
return "audio/ogg"
|
||||
}
|
||||
return mime
|
||||
}
|
||||
@@ -1,15 +1,33 @@
|
||||
package telegram
|
||||
|
||||
import (
|
||||
"context"
|
||||
"net/http"
|
||||
"net/http/httptest"
|
||||
"os"
|
||||
"strings"
|
||||
"testing"
|
||||
|
||||
"github.com/mymmrac/telego"
|
||||
|
||||
"github.com/nextlevelbuilder/goclaw/internal/audio"
|
||||
"github.com/nextlevelbuilder/goclaw/internal/bus"
|
||||
"github.com/nextlevelbuilder/goclaw/internal/channels"
|
||||
"github.com/nextlevelbuilder/goclaw/internal/config"
|
||||
)
|
||||
|
||||
type telegramTestSTT struct {
|
||||
name string
|
||||
input audio.STTInput
|
||||
}
|
||||
|
||||
func (s *telegramTestSTT) Name() string { return s.name }
|
||||
|
||||
func (s *telegramTestSTT) Transcribe(_ context.Context, in audio.STTInput, _ audio.STTOptions) (*audio.TranscriptResult, error) {
|
||||
s.input = in
|
||||
return &audio.TranscriptResult{Text: "xin chao", Provider: s.name}, nil
|
||||
}
|
||||
|
||||
// --- buildMediaTags tests ---
|
||||
|
||||
// TestBuildMediaTags_NoTranscript_Legacy verifies that the pre-patch behaviour is
|
||||
@@ -154,6 +172,129 @@ func TestBuildMediaTags_UnknownType(t *testing.T) {
|
||||
}
|
||||
}
|
||||
|
||||
func TestTranscribeMediaAudioUsesChannelTypeAndPreservesMIME(t *testing.T) {
|
||||
mgr := audio.NewManager(audio.ManagerConfig{})
|
||||
tenantSTT := &telegramTestSTT{name: "tenant"}
|
||||
telegramSTT := &telegramTestSTT{name: "proxy"}
|
||||
mgr.RegisterSTT(tenantSTT)
|
||||
mgr.SetSTTChain([]string{"tenant"})
|
||||
mgr.RegisterChannelSTT(channels.TypeTelegram, telegramSTT)
|
||||
|
||||
ch := &Channel{
|
||||
BaseChannel: channels.NewBaseChannel("telegram-main", nil, nil),
|
||||
audioMgr: mgr,
|
||||
}
|
||||
ch.SetType(channels.TypeTelegram)
|
||||
|
||||
got, err := ch.transcribeMediaAudio(context.Background(), MediaInfo{
|
||||
Type: "voice",
|
||||
FilePath: "/tmp/voice.ogg",
|
||||
FileName: "voice.ogg",
|
||||
ContentType: "audio/ogg; codecs=opus",
|
||||
})
|
||||
if err != nil {
|
||||
t.Fatalf("transcribeMediaAudio returned error: %v", err)
|
||||
}
|
||||
if got != "xin chao" {
|
||||
t.Fatalf("transcript = %q, want channel override transcript", got)
|
||||
}
|
||||
if tenantSTT.input.FilePath != "" {
|
||||
t.Fatalf("tenant STT received input; expected Telegram channel override to win")
|
||||
}
|
||||
if telegramSTT.input.MimeType != "audio/ogg; codecs=opus" {
|
||||
t.Fatalf("mime = %q, want preserved Telegram voice MIME", telegramSTT.input.MimeType)
|
||||
}
|
||||
if telegramSTT.input.FilePath != "/tmp/voice.ogg" {
|
||||
t.Fatalf("file path = %q, want /tmp/voice.ogg", telegramSTT.input.FilePath)
|
||||
}
|
||||
}
|
||||
|
||||
func TestTelegramAudioSTTMimeFallback(t *testing.T) {
|
||||
tests := []struct {
|
||||
name string
|
||||
contentType string
|
||||
want string
|
||||
}{
|
||||
{name: "preserves opus", contentType: "audio/ogg; codecs=opus", want: "audio/ogg; codecs=opus"},
|
||||
{name: "empty defaults ogg", contentType: "", want: "audio/ogg"},
|
||||
{name: "octet stream defaults ogg", contentType: "application/octet-stream", want: "audio/ogg"},
|
||||
{name: "trims whitespace", contentType: " audio/mpeg ", want: "audio/mpeg"},
|
||||
}
|
||||
|
||||
for _, tt := range tests {
|
||||
t.Run(tt.name, func(t *testing.T) {
|
||||
if got := telegramAudioSTTMime(tt.contentType); got != tt.want {
|
||||
t.Fatalf("telegramAudioSTTMime(%q) = %q, want %q", tt.contentType, got, tt.want)
|
||||
}
|
||||
})
|
||||
}
|
||||
}
|
||||
|
||||
func TestResolveMediaTelegramVoiceDownloadsOggOpusFixture(t *testing.T) {
|
||||
const token = "123456789:aaaabbbbaaaabbbbaaaabbbbaaaabbbbccc"
|
||||
const fileID = "telegram-voice-file"
|
||||
oggOpus := []byte("OggS\x00\x02telegram-voice-opus-fixture")
|
||||
|
||||
server := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
|
||||
switch r.URL.Path {
|
||||
case "/bot" + token + "/getFile":
|
||||
w.Header().Set("Content-Type", "application/json")
|
||||
_, _ = w.Write([]byte(`{"ok":true,"result":{"file_id":"` + fileID + `","file_unique_id":"voice-unique","file_path":"voice/file.ogg"}}`))
|
||||
case "/file/bot" + token + "/voice/file.ogg":
|
||||
w.Header().Set("Content-Type", "audio/ogg")
|
||||
_, _ = w.Write(oggOpus)
|
||||
default:
|
||||
t.Fatalf("unexpected Telegram API path: %s", r.URL.Path)
|
||||
}
|
||||
}))
|
||||
defer server.Close()
|
||||
|
||||
ch, err := New(config.TelegramConfig{
|
||||
Token: token,
|
||||
APIServer: server.URL,
|
||||
}, nil, nil, nil)
|
||||
if err != nil {
|
||||
t.Fatalf("New channel: %v", err)
|
||||
}
|
||||
|
||||
mediaList, mediaErrors := ch.resolveMedia(context.Background(), &telego.Message{
|
||||
Voice: &telego.Voice{
|
||||
FileID: fileID,
|
||||
MimeType: "audio/ogg; codecs=opus",
|
||||
FileSize: int64(len(oggOpus)),
|
||||
},
|
||||
})
|
||||
if len(mediaErrors) > 0 {
|
||||
t.Fatalf("resolveMedia errors: %+v", mediaErrors)
|
||||
}
|
||||
if len(mediaList) != 1 {
|
||||
t.Fatalf("got %d media items, want 1", len(mediaList))
|
||||
}
|
||||
|
||||
voice := mediaList[0]
|
||||
if voice.Type != "voice" {
|
||||
t.Fatalf("media type = %q, want voice", voice.Type)
|
||||
}
|
||||
if voice.ContentType != "audio/ogg; codecs=opus" {
|
||||
t.Fatalf("content type = %q, want Telegram voice MIME", voice.ContentType)
|
||||
}
|
||||
if voice.FilePath == "" {
|
||||
t.Fatalf("expected downloaded file path")
|
||||
}
|
||||
t.Cleanup(func() { _ = os.Remove(voice.FilePath) })
|
||||
|
||||
data, err := os.ReadFile(voice.FilePath)
|
||||
if err != nil {
|
||||
t.Fatalf("read downloaded voice fixture: %v", err)
|
||||
}
|
||||
if !strings.HasPrefix(string(data), "OggS") {
|
||||
t.Fatalf("downloaded fixture does not look like OGG data: %q", string(data))
|
||||
}
|
||||
if got := telegramAudioSTTMime(voice.ContentType); got != "audio/ogg; codecs=opus" {
|
||||
t.Fatalf("STT mime = %q, want preserved Telegram voice MIME", got)
|
||||
}
|
||||
}
|
||||
|
||||
func TestExtractMediaRefsPreservesDocumentMetadata(t *testing.T) {
|
||||
msg := &telego.Message{
|
||||
Document: &telego.Document{
|
||||
|
||||
@@ -72,7 +72,7 @@ func (t *ReadAudioTool) resolveAudioFile(ctx context.Context, mediaID string) (p
|
||||
// callProvider dispatches audio analysis to the appropriate provider API.
|
||||
// Gemini: uses File API (upload → poll → file_data in generateContent).
|
||||
// OpenAI: uses input_audio content part in chat completions.
|
||||
// Others: falls back to base64 in image_url (best effort).
|
||||
// Transcription models: use the OpenAI-compatible audio transcription endpoint.
|
||||
func (t *ReadAudioTool) callProvider(ctx context.Context, cp credentialProvider, providerName, model string, params map[string]any) ([]byte, *providers.Usage, error) {
|
||||
prompt := GetParamString(params, "prompt", "Analyze this audio and describe its contents.")
|
||||
data, _ := params["data"].([]byte)
|
||||
@@ -157,39 +157,7 @@ func (t *ReadAudioTool) callProvider(ctx context.Context, cp credentialProvider,
|
||||
}
|
||||
}
|
||||
|
||||
// Other providers: try standard Chat API with base64 audio as image_url (best effort).
|
||||
p, err := t.registry.Get(ctx, providerName)
|
||||
if err != nil {
|
||||
return nil, nil, fmt.Errorf("provider %q not available: %w", providerName, err)
|
||||
}
|
||||
|
||||
slog.Info("read_audio: using chat API fallback", "provider", providerName, "model", model, "size", len(data))
|
||||
chatReq := providers.ChatRequest{
|
||||
Messages: []providers.Message{
|
||||
{
|
||||
Role: "user",
|
||||
Content: prompt,
|
||||
Images: []providers.ImageContent{{MimeType: mime, Data: base64.StdEncoding.EncodeToString(data)}},
|
||||
},
|
||||
},
|
||||
Model: model,
|
||||
Options: map[string]any{
|
||||
"max_tokens": 16384,
|
||||
"temperature": 0.2,
|
||||
},
|
||||
}
|
||||
reservation, reserveErr := reserveToolLLMUsage(ctx, t.usageCaps, t.Name(), providerName, model, chatReq)
|
||||
if reserveErr != nil {
|
||||
return nil, nil, reserveErr
|
||||
}
|
||||
resp, err := p.Chat(ctx, chatReq)
|
||||
if reservation != nil {
|
||||
reservation.Reconcile(ctx, resp, err)
|
||||
}
|
||||
if err != nil {
|
||||
return nil, nil, fmt.Errorf("chat API: %w", err)
|
||||
}
|
||||
return []byte(resp.Content), resp.Usage, nil
|
||||
return nil, nil, fmt.Errorf("read_audio: unsupported audio route for provider %q (type %q) model %q; supported routes are Gemini File API, native OpenAI input_audio, or OpenAI-compatible transcription models", providerName, ptype, model)
|
||||
}
|
||||
|
||||
// openaiAudioCall sends audio to OpenAI using the input_audio content part.
|
||||
|
||||
@@ -4,8 +4,31 @@ import (
|
||||
"context"
|
||||
"strings"
|
||||
"testing"
|
||||
|
||||
"github.com/nextlevelbuilder/goclaw/internal/providers"
|
||||
)
|
||||
|
||||
type readAudioUnsupportedProvider struct {
|
||||
name string
|
||||
chatCalls int
|
||||
images int
|
||||
}
|
||||
|
||||
func (p *readAudioUnsupportedProvider) Name() string { return p.name }
|
||||
func (p *readAudioUnsupportedProvider) DefaultModel() string { return "claude-3-5-sonnet-latest" }
|
||||
|
||||
func (p *readAudioUnsupportedProvider) Chat(_ context.Context, req providers.ChatRequest) (*providers.ChatResponse, error) {
|
||||
p.chatCalls++
|
||||
for _, msg := range req.Messages {
|
||||
p.images += len(msg.Images)
|
||||
}
|
||||
return &providers.ChatResponse{Content: "unexpected"}, nil
|
||||
}
|
||||
|
||||
func (p *readAudioUnsupportedProvider) ChatStream(_ context.Context, _ providers.ChatRequest, _ func(providers.StreamChunk)) (*providers.ChatResponse, error) {
|
||||
return nil, nil
|
||||
}
|
||||
|
||||
// TestReadAudioCallProvider_TranscriptionModelWithoutCreds_FailsFast asserts
|
||||
// that when no API credentials are present, a transcription-named model
|
||||
// returns a clear error rather than silently falling back to chat/completions
|
||||
@@ -71,3 +94,31 @@ func TestReadAudioCallProvider_GeminiWithoutCreds_FailsFast(t *testing.T) {
|
||||
t.Errorf("expected error to mention credentials, got: %v", err)
|
||||
}
|
||||
}
|
||||
|
||||
func TestReadAudioCallProvider_UnsupportedProviderDoesNotSendAudioAsImage(t *testing.T) {
|
||||
reg := providers.NewRegistry(nil)
|
||||
fake := &readAudioUnsupportedProvider{name: "anthropic"}
|
||||
reg.Register(fake)
|
||||
tool := &ReadAudioTool{registry: reg}
|
||||
|
||||
params := map[string]any{
|
||||
"_provider_type": "anthropic",
|
||||
"data": []byte{0x4f, 0x67, 0x67, 0x53},
|
||||
"mime": "audio/ogg; codecs=opus",
|
||||
}
|
||||
|
||||
_, _, err := tool.callProvider(context.Background(), nil, "anthropic", "claude-3-5-sonnet-latest", params)
|
||||
if err == nil {
|
||||
t.Fatalf("expected unsupported audio route error, got nil")
|
||||
}
|
||||
msg := strings.ToLower(err.Error())
|
||||
if !strings.Contains(msg, "unsupported audio route") {
|
||||
t.Fatalf("expected unsupported audio route error, got: %v", err)
|
||||
}
|
||||
if !strings.Contains(msg, "anthropic") || !strings.Contains(msg, "claude-3-5-sonnet-latest") {
|
||||
t.Fatalf("expected provider and model in error, got: %v", err)
|
||||
}
|
||||
if fake.chatCalls != 0 || fake.images != 0 {
|
||||
t.Fatalf("unsupported audio route called chat fallback: calls=%d images=%d", fake.chatCalls, fake.images)
|
||||
}
|
||||
}
|
||||
+43
@@ -0,0 +1,43 @@
|
||||
# Phase 01 - Characterize Telegram Voice Failure
|
||||
|
||||
## Overview
|
||||
- Priority: P0
|
||||
- Status: Complete
|
||||
- Purpose: Lock current failing behavior before code changes.
|
||||
|
||||
## Context Links
|
||||
- Issue: https://github.com/digitopvn/goclaw/issues/85
|
||||
- Debug report: `reports/debugger-260528-1837-telegram-voice-transcription.md`
|
||||
|
||||
## Requirements
|
||||
- Build a small unit-level characterization for Telegram voice `MediaInfo`.
|
||||
- Prove `msg.Voice.MimeType` is available and should not be discarded.
|
||||
- Prove channel context is required for legacy Telegram STT proxy selection.
|
||||
- Prove `read_audio` must not route audio bytes through image payload fallback.
|
||||
|
||||
## Characterization Notes
|
||||
- `resolveMedia` already classifies Telegram `msg.Voice` as `MediaInfo{Type: "voice"}` and copies `msg.Voice.MimeType` into `ContentType`; the loss happens later when `processResolvedMessage` hardcodes STT MIME as `audio/ogg`.
|
||||
- `audio.Manager` already supports channel-scoped STT overrides through `audio.WithChannel(ctx, "telegram")`; Telegram does not set that context before calling `Transcribe`.
|
||||
- `read_audio` still has a generic chat fallback that attaches audio bytes as `providers.ImageContent`, which matches the provider-side "image format illegal" failure.
|
||||
|
||||
## Related Code Files
|
||||
- Modify: `internal/channels/telegram/media_test.go`
|
||||
- Modify: `internal/channels/telegram/handlers_test.go` or add focused helper tests near Telegram package.
|
||||
- Modify: `internal/audio/manager_stt_test.go` only if existing coverage lacks the channel override case.
|
||||
|
||||
## Implementation Steps
|
||||
1. Add a Telegram voice media test using a `telego.Message{Voice: ...}` fixture.
|
||||
2. Assert media type is `voice`, MIME is preserved, file path is populated by a stub download path.
|
||||
3. Add/extend audio manager test proving `audio.WithChannel(ctx, "telegram")` selects the channel-scoped STT provider.
|
||||
4. Avoid live Telegram or provider calls.
|
||||
|
||||
## Success Criteria
|
||||
- Deterministic regression tests can be added without live Telegram or provider calls.
|
||||
- Phase 02 owns Telegram STT context/MIME test and fix.
|
||||
- Phase 03 owns `read_audio` unsupported-provider regression and fix.
|
||||
|
||||
## Risks
|
||||
- `processResolvedMessage` is large; prefer extracting a narrow helper only if needed for testability.
|
||||
|
||||
## Open Questions
|
||||
- None.
|
||||
+40
@@ -0,0 +1,40 @@
|
||||
# Phase 02 - Fix Telegram STT Routing
|
||||
|
||||
## Overview
|
||||
- Priority: P0
|
||||
- Status: Complete
|
||||
- Purpose: Make Telegram voice STT use the right MIME and provider chain.
|
||||
|
||||
## Requirements
|
||||
- Use `m.ContentType` for audio/voice STT, with safe fallback to `audio/ogg`.
|
||||
- Wrap STT context with `audio.WithChannel(ctx, c.Type())`.
|
||||
- Preserve current 10s timeout and fallback behavior.
|
||||
- Do not change WhatsApp opt-in semantics.
|
||||
|
||||
## Related Code Files
|
||||
- Modify: `internal/channels/telegram/handlers.go`
|
||||
- Optional: create helper in `internal/channels/telegram/stt.go` if `handlers.go` growth needs containment.
|
||||
|
||||
## Implementation Steps
|
||||
1. Introduce a small helper for audio MIME fallback: empty or `application/octet-stream` -> `audio/ogg`.
|
||||
2. Replace hardcoded `MimeType: "audio/ogg"` with the helper result.
|
||||
3. Call `audio.WithChannel(sttCtx, c.Type())` before `audioMgr.Transcribe`.
|
||||
4. Preserve `m.Transcript = res.Text` only on non-empty success.
|
||||
5. Log provider failure as Telegram audio STT failure with type, MIME, and provider error.
|
||||
|
||||
## Implementation Notes
|
||||
- Used `c.Type()` instead of `c.Name()` so DB channel instances like `telegram-main` still resolve the legacy override registered for platform type `telegram`.
|
||||
- Added a focused regression where tenant STT is configured but Telegram channel-scoped STT wins and receives `audio/ogg; codecs=opus`.
|
||||
- Empty transcripts remain non-fatal; no transcript tag is added.
|
||||
|
||||
## Success Criteria
|
||||
- Telegram legacy `stt_proxy_url` can be selected via channel override.
|
||||
- Telegram OGG/Opus MIME survives into STT input.
|
||||
- Existing Telegram media flow still publishes `bus.InboundMessage.Media`.
|
||||
|
||||
## Security Considerations
|
||||
- No new secret storage.
|
||||
- Do not log raw audio path if it can expose tenant filesystem details beyond existing debug behavior; prefer basename when possible.
|
||||
|
||||
## Open Questions
|
||||
- None.
|
||||
+41
@@ -0,0 +1,41 @@
|
||||
# Phase 03 - Harden Read Audio Fallback
|
||||
|
||||
## Overview
|
||||
- Priority: P0
|
||||
- Status: Complete
|
||||
- Purpose: Stop audio being sent as image payload.
|
||||
|
||||
## Requirements
|
||||
- `read_audio` must never put audio bytes in `providers.ImageContent`.
|
||||
- Unsupported provider/model routes must return an audio-specific error.
|
||||
- Preserve working routes:
|
||||
- Gemini File API.
|
||||
- Native OpenAI input audio.
|
||||
- OpenAI transcription endpoint for transcription model names.
|
||||
|
||||
## Related Code Files
|
||||
- Modify: `internal/tools/read_audio_resolve.go`
|
||||
- Modify: `internal/tools/read_audio_resolve_test.go`
|
||||
- Optional: `internal/tools/read_audio.go` description to mention `<media:voice>`.
|
||||
|
||||
## Implementation Steps
|
||||
1. Replace "Other providers" image fallback with explicit unsupported audio route error.
|
||||
2. Include provider name, provider type, model, and accepted options in the error, without secrets.
|
||||
3. Add regression test using an OpenAI-compatible non-transcription provider/model that previously went through `Images`.
|
||||
4. Add test ensuring `<media:voice>` id still resolves through `read_audio`.
|
||||
|
||||
## Implementation Notes
|
||||
- Removed generic chat/image fallback from `read_audio`.
|
||||
- Added regression proving unsupported audio routes do not call provider chat and do not create `providers.ImageContent`.
|
||||
- Preserved Gemini File API, native OpenAI input audio, and OpenAI-compatible transcription model routes.
|
||||
|
||||
## Success Criteria
|
||||
- The observed "image format illegal" class cannot be produced by GoClaw for audio fallback.
|
||||
- User-facing error says audio provider/model unsupported or STT provider unavailable.
|
||||
- Existing Gemini/OpenAI/transcription tests still pass.
|
||||
|
||||
## Risks
|
||||
- Some provider may have relied on chat fallback for audio. Treat as unsafe legacy path because it is semantically wrong and produced this issue.
|
||||
|
||||
## Open Questions
|
||||
- DashScope/Qwen audio chat can be added later as a first-class verified route.
|
||||
@@ -0,0 +1,37 @@
|
||||
# Phase 04 - Tests and Docs
|
||||
|
||||
## Overview
|
||||
- Priority: P1
|
||||
- Status: Complete
|
||||
- Purpose: Verify fix and sync public docs.
|
||||
|
||||
## Requirements
|
||||
- Add regression tests for Telegram voice, STT channel override, and read_audio no-image fallback.
|
||||
- Update docs to reflect actual STT provider IDs and failure behavior.
|
||||
- Do not add load/stress tests.
|
||||
|
||||
## Related Code Files
|
||||
- Modify: `docs/05-channels-messaging.md`
|
||||
- Modify: `docs/03-tools-system.md` if read_audio contract changes.
|
||||
- Modify: `docs/project-changelog.md`
|
||||
- Run: package-level Go tests for touched packages.
|
||||
|
||||
## Implementation Steps
|
||||
1. Run `go test ./internal/channels/telegram ./internal/audio ./internal/tools`.
|
||||
2. If shared code changes, run `go test ./internal/agent ./internal/channels/... ./internal/tools`.
|
||||
3. Update docs after tests pass.
|
||||
4. Record issue #85 in changelog with root cause and validation commands.
|
||||
|
||||
## Implementation Notes
|
||||
- Updated channel docs to use actual provider IDs: `elevenlabs` and `proxy`.
|
||||
- Documented Telegram voice MIME preservation and platform-type STT override behavior.
|
||||
- Documented `read_audio` fail-closed behavior for unsupported audio routes.
|
||||
- Added a Telegram Bot API fixture test that downloads a voice OGG file and verifies `audio/ogg; codecs=opus` survives from `telego.Voice` through media resolution into STT MIME selection.
|
||||
|
||||
## Success Criteria
|
||||
- Tests pass locally.
|
||||
- Docs no longer claim provider IDs that do not exist in code.
|
||||
- Issue acceptance criteria are traceable to test names.
|
||||
|
||||
## Open Questions
|
||||
- None.
|
||||
@@ -0,0 +1,38 @@
|
||||
# Telegram Voice Transcription Fix Plan
|
||||
|
||||
## Context
|
||||
- Issue: https://github.com/digitopvn/goclaw/issues/85
|
||||
- Worktree: `/Users/duynguyen/.codex/worktrees/codex/issue-85-telegram-voice-transcription-plan`
|
||||
- Branch: `codex/issue-85-telegram-voice-transcription-plan`
|
||||
- Base: `origin/dev` at `5017f7ca` after merging latest dev into the issue branch
|
||||
- Debug report: `reports/debugger-260528-1837-telegram-voice-transcription.md`
|
||||
|
||||
## Goal
|
||||
Fix Telegram app-recorded voice messages so `<media:voice>` is treated as audio end-to-end, transcribed through the STT path, and never routed through image payloads.
|
||||
|
||||
## Verified Findings
|
||||
- Telegram downloads `msg.Voice` as media type `voice` and carries `msg.Voice.MimeType`, but STT call currently forces MIME to `audio/ogg` instead of using the detected content type: `internal/channels/telegram/media.go:160`, `internal/channels/telegram/handlers.go:474`.
|
||||
- Telegram calls `audioMgr.Transcribe` without `audio.WithChannel(ctx, c.Name())`, so channel-scoped legacy STT proxy overrides cannot be selected: `internal/audio/manager.go:17`, `internal/audio/manager_stt.go:69`, `internal/audio/legacy_stt_bridge.go:81`.
|
||||
- When Telegram STT fails or is not configured, the message still reaches the agent with `<media:voice>` and media attachment. `read_audio` can then be invoked: `internal/channels/telegram/handlers.go:488`, `internal/channels/telegram/handlers.go:709`, `internal/agent/loop_input_media.go:120`.
|
||||
- `read_audio` fallback for non-Gemini/non-OpenAI/non-transcription providers sends audio bytes as `providers.ImageContent`, matching the observed "image format illegal" provider error: `internal/tools/read_audio_resolve.go:160`.
|
||||
- Docs say unified STT should cover Telegram and use the configured provider chain, but current names and behavior drift from implementation: `docs/05-channels-messaging.md:262`.
|
||||
|
||||
## Phases
|
||||
- [x] [Phase 01 - Characterize Failure](phase-01-characterize-telegram-voice-failure.md)
|
||||
- [x] [Phase 02 - Fix Telegram STT Routing](phase-02-fix-telegram-stt-routing.md)
|
||||
- [x] [Phase 03 - Harden Read Audio Fallback](phase-03-harden-read-audio-fallback.md)
|
||||
- [x] [Phase 04 - Tests and Docs](phase-04-tests-and-docs.md)
|
||||
|
||||
## Dependencies
|
||||
- No schema migration expected.
|
||||
- If adding audio transcoding, decide between built-in ffmpeg dependency vs provider-compatible no-conversion path first.
|
||||
- Keep WhatsApp opt-in behavior unchanged.
|
||||
|
||||
## Success Criteria
|
||||
- Telegram `Voice` messages preserve `audio/ogg` / `audio/ogg; codecs=opus` metadata into STT and media refs.
|
||||
- Channel-scoped STT proxy works for Telegram via `audio.WithChannel`.
|
||||
- `read_audio` never sends audio as image payload for unsupported provider routes; errors are audio-specific and actionable.
|
||||
- Regression tests cover Telegram voice media, STT channel context, and `read_audio` unsupported fallback.
|
||||
|
||||
## Open Questions
|
||||
- None.
|
||||
+59
@@ -0,0 +1,59 @@
|
||||
# Telegram Voice Transcription - Investigation Report
|
||||
|
||||
## Executive Summary
|
||||
- **Issue:** Telegram app-recorded voice messages fail transcription; provider returns image-format error.
|
||||
- **Impact:** Telegram voice workflow blocked when auto-STT misses and agent/tool fallback is used.
|
||||
- **Root cause:** Audio/voice can fall through to `read_audio`; unsupported provider fallback encodes audio as image content.
|
||||
- **Status:** Root cause isolated; implementation plan ready.
|
||||
- **Fix:** Preserve voice MIME + channel context in Telegram STT, then block or convert unsafe `read_audio` fallback.
|
||||
|
||||
## Evidence
|
||||
- Telegram voice is detected as voice:
|
||||
- `internal/channels/telegram/media.go:160` handles `msg.Voice`.
|
||||
- `internal/channels/telegram/media.go:167` appends `MediaInfo{Type: "voice"}`.
|
||||
- `internal/channels/telegram/media.go:171` stores `msg.Voice.MimeType`.
|
||||
- Telegram STT loses metadata:
|
||||
- `internal/channels/telegram/handlers.go:474` enters audio/voice branch.
|
||||
- `internal/channels/telegram/handlers.go:480` calls `audioMgr.Transcribe(... MimeType: "audio/ogg")`.
|
||||
- This ignores `m.ContentType` and does not set `audio.WithChannel(ctx, c.Name())`.
|
||||
- Channel-scoped STT exists but is skipped by Telegram call:
|
||||
- `internal/audio/manager.go:17` defines `WithChannel`.
|
||||
- `internal/audio/manager_stt.go:69` checks channel override before default chain.
|
||||
- `internal/audio/legacy_stt_bridge.go:81` registers channel-scoped proxy providers for Telegram/Feishu/Discord.
|
||||
- Fallback reaches agent/tool path:
|
||||
- `internal/channels/telegram/handlers.go:488` only logs STT failure.
|
||||
- `internal/channels/telegram/handlers.go:516` emits media tags, transcript optional.
|
||||
- `internal/channels/telegram/handlers.go:709` publishes media files to the agent.
|
||||
- `internal/agent/loop_input_media.go:120` collects audio refs for `read_audio`.
|
||||
- Observed image-format error maps to tool fallback:
|
||||
- `internal/tools/read_audio_resolve.go:72` documents provider dispatch.
|
||||
- `internal/tools/read_audio_resolve.go:160` sends "Other providers" through chat API with `Images`.
|
||||
- `internal/tools/read_audio_resolve.go:172` puts audio bytes in `providers.ImageContent`.
|
||||
|
||||
## Root Cause
|
||||
Primary root cause: `read_audio` has a best-effort chat fallback that represents audio as image data. For OpenAI-compatible providers without a transcription model name or dedicated audio route, this generates an image payload and surfaces provider errors like "image format illegal".
|
||||
|
||||
Contributing causes:
|
||||
- Telegram STT hardcodes `audio/ogg`, losing actual MIME detail such as `audio/ogg; codecs=opus`.
|
||||
- Telegram STT does not attach channel context, so legacy channel-specific proxy STT cannot win.
|
||||
- Telegram keeps processing after STT failure with only a warn log, so the LLM can call `read_audio` and hit the unsafe fallback.
|
||||
- Docs describe a unified STT chain, but UI/code naming currently uses `elevenlabs` / `proxy`, while docs show `elevenlabs_scribe` / `proxy_stt`.
|
||||
|
||||
## Recommended Plan
|
||||
### Immediate (P0)
|
||||
- Add focused tests proving Telegram voice path uses actual voice MIME and `audio.WithChannel`.
|
||||
- Change Telegram STT call to pass `MimeType: fallbackAudioMime(m.ContentType)` and a channel-scoped context.
|
||||
- Keep failure behavior user-friendly: log audio-specific cause and keep `<media:voice>` tag, but do not imply image analysis.
|
||||
|
||||
### Short-Term (P1)
|
||||
- Change `read_audio` fallback: if provider route is not Gemini, native OpenAI audio, or transcription endpoint, return clear audio-specific unsupported-provider error instead of `Images`.
|
||||
- Consider explicit provider capability check for audio input/transcription before dispatch.
|
||||
- Add regression around DashScope/openai-compatible non-transcription model to ensure no image payload is produced.
|
||||
|
||||
### Longer-Term (P2)
|
||||
- Decide conversion layer: ffmpeg OGG/Opus to WAV/MP3 before STT when selected provider cannot accept OGG/Opus.
|
||||
- Sync docs and UI labels with actual provider ids.
|
||||
|
||||
## Open Questions
|
||||
- Should `builtin_tools[stt]` provider order drive Telegram STT now, or only after a separate STT settings migration?
|
||||
- Should conversion be implemented now, or only after provider rejection proves it is needed?
|
||||
Reference in new issue
Block a user