mirror of
https://github.com/tiennm99/ccs.git
synced 2026-10-14 12:16:38 +00:00
feat(proxy): complete openai routing scope
This commit is contained in:
1 parent
b346693c3a
commit
17187c4abd
24 files changed
+1819
-53
No files matched your search
@@ -67,6 +67,10 @@ ccs proxy start hf
|
|||||||
eval "$(ccs proxy activate)"
|
eval "$(ccs proxy activate)"
|
||||||
```
|
```
|
||||||
|
|
||||||
|
The proxy also supports request-time `profile:model` selectors, scenario-based
|
||||||
|
model routing through `proxy.routing`, and explicit activation helpers such as
|
||||||
|
`ccs proxy activate --fish`.
|
||||||
|
|
||||||
Guide: [OpenAI-Compatible Provider Routing](./docs/openai-compatible-providers.md)
|
Guide: [OpenAI-Compatible Provider Routing](./docs/openai-compatible-providers.md)
|
||||||
|
|
||||||
### Related Project: claude-code-router
|
### Related Project: claude-code-router
|
||||||
|
|||||||
@@ -68,8 +68,22 @@ ccs proxy status
|
|||||||
ccs proxy stop
|
ccs proxy stop
|
||||||
```
|
```
|
||||||
|
|
||||||
`ccs proxy activate` prints the `ANTHROPIC_*` exports needed for a local
|
Useful variants:
|
||||||
Anthropic-compatible session against the running proxy.
|
|
||||||
|
```bash
|
||||||
|
ccs proxy start hf --host 127.0.0.1
|
||||||
|
ccs proxy activate --fish
|
||||||
|
```
|
||||||
|
|
||||||
|
`ccs proxy activate` now prints the full local runtime contract:
|
||||||
|
|
||||||
|
- `ANTHROPIC_BASE_URL`
|
||||||
|
- `ANTHROPIC_AUTH_TOKEN`
|
||||||
|
- `ANTHROPIC_MODEL` plus tier defaults when present
|
||||||
|
- `DISABLE_TELEMETRY`
|
||||||
|
- `DISABLE_COST_WARNINGS`
|
||||||
|
- `API_TIMEOUT_MS`
|
||||||
|
- `NO_PROXY`
|
||||||
|
|
||||||
## One Active Proxy Profile
|
## One Active Proxy Profile
|
||||||
|
|
||||||
@@ -82,6 +96,56 @@ The current runtime is a single local proxy daemon.
|
|||||||
This is intentional to avoid breaking an in-flight Claude session by swapping
|
This is intentional to avoid breaking an in-flight Claude session by swapping
|
||||||
its upstream provider out from under it.
|
its upstream provider out from under it.
|
||||||
|
|
||||||
|
## Request-Time Routing
|
||||||
|
|
||||||
|
The proxy is no longer limited to the startup profile's default model.
|
||||||
|
|
||||||
|
Supported request-time selectors:
|
||||||
|
|
||||||
|
- `profile:model`
|
||||||
|
Example: `deepseek:deepseek-reasoner`
|
||||||
|
- `profile`
|
||||||
|
Example: `openrouter`
|
||||||
|
- plain model ids
|
||||||
|
Example: `deepseek-chat`
|
||||||
|
|
||||||
|
Routing behavior:
|
||||||
|
|
||||||
|
1. `profile:model` wins immediately.
|
||||||
|
2. Scenario routing may override the active profile when configured.
|
||||||
|
3. Plain model ids are matched against the configured OpenAI-compatible
|
||||||
|
profiles before falling back to the active profile.
|
||||||
|
|
||||||
|
This means a Claude session launched through one compatible profile can still
|
||||||
|
request another compatible profile/model when the proxy can resolve it safely.
|
||||||
|
|
||||||
|
## Scenario Routing
|
||||||
|
|
||||||
|
Scenario routing is now supported through `proxy.routing` in your CCS config.
|
||||||
|
|
||||||
|
Example `~/.ccs/config.yaml`:
|
||||||
|
|
||||||
|
```yaml
|
||||||
|
proxy:
|
||||||
|
routing:
|
||||||
|
default: "deepseek:deepseek-chat"
|
||||||
|
background: "ollama:qwen2.5-coder:0.5b"
|
||||||
|
think: "deepseek:deepseek-reasoner"
|
||||||
|
longContext: "openrouter:google/gemini-2.5-pro"
|
||||||
|
longContextThreshold: 60000
|
||||||
|
webSearch: "openrouter:perplexity/sonar-pro"
|
||||||
|
```
|
||||||
|
|
||||||
|
Current scenario detection:
|
||||||
|
|
||||||
|
- `background`: requested model contains `haiku`
|
||||||
|
- `think`: Anthropic `thinking` is enabled
|
||||||
|
- `longContext`: estimated request tokens exceed `longContextThreshold`
|
||||||
|
- `webSearch`: tool list includes `web_search`
|
||||||
|
- `default`: fallback selector when the above do not apply
|
||||||
|
|
||||||
|
Routing decisions are logged through CCS structured logs.
|
||||||
|
|
||||||
## How Profile Detection Works
|
## How Profile Detection Works
|
||||||
|
|
||||||
CCS keeps these profiles in the normal API/settings-profile flow.
|
CCS keeps these profiles in the normal API/settings-profile flow.
|
||||||
@@ -102,6 +166,79 @@ OpenAI-compatible endpoints such as:
|
|||||||
|
|
||||||
are routed through the local proxy for Claude-target launches.
|
are routed through the local proxy for Claude-target launches.
|
||||||
|
|
||||||
|
## Provider Setup
|
||||||
|
|
||||||
|
### DeepSeek
|
||||||
|
|
||||||
|
Use a settings profile whose env looks like:
|
||||||
|
|
||||||
|
```json
|
||||||
|
{
|
||||||
|
"env": {
|
||||||
|
"ANTHROPIC_BASE_URL": "https://api.deepseek.com/v1",
|
||||||
|
"ANTHROPIC_AUTH_TOKEN": "sk-...",
|
||||||
|
"ANTHROPIC_MODEL": "deepseek-chat",
|
||||||
|
"CCS_DROID_PROVIDER": "generic-chat-completion-api"
|
||||||
|
}
|
||||||
|
}
|
||||||
|
```
|
||||||
|
|
||||||
|
Typical override target:
|
||||||
|
|
||||||
|
- `deepseek:deepseek-reasoner`
|
||||||
|
|
||||||
|
### OpenRouter
|
||||||
|
|
||||||
|
```json
|
||||||
|
{
|
||||||
|
"env": {
|
||||||
|
"ANTHROPIC_BASE_URL": "https://openrouter.ai/api/v1",
|
||||||
|
"ANTHROPIC_AUTH_TOKEN": "sk-or-...",
|
||||||
|
"ANTHROPIC_MODEL": "openai/gpt-4.1-mini",
|
||||||
|
"CCS_DROID_PROVIDER": "generic-chat-completion-api"
|
||||||
|
}
|
||||||
|
}
|
||||||
|
```
|
||||||
|
|
||||||
|
Useful when you want:
|
||||||
|
|
||||||
|
- model fan-out behind one provider profile
|
||||||
|
- long-context or web-search scenario targets
|
||||||
|
|
||||||
|
### Ollama / Local Gateways
|
||||||
|
|
||||||
|
```json
|
||||||
|
{
|
||||||
|
"env": {
|
||||||
|
"ANTHROPIC_BASE_URL": "http://127.0.0.1:11434",
|
||||||
|
"ANTHROPIC_AUTH_TOKEN": "ollama",
|
||||||
|
"ANTHROPIC_MODEL": "qwen3-coder",
|
||||||
|
"CCS_DROID_PROVIDER": "generic-chat-completion-api"
|
||||||
|
}
|
||||||
|
}
|
||||||
|
```
|
||||||
|
|
||||||
|
For self-signed HTTPS gateways, add `CCS_OPENAI_PROXY_INSECURE=1`.
|
||||||
|
|
||||||
|
### DashScope / Qwen Compatible Mode
|
||||||
|
|
||||||
|
DashScope's compatible endpoint works even when older settings files still
|
||||||
|
carry a stale Anthropic-style provider hint:
|
||||||
|
|
||||||
|
```json
|
||||||
|
{
|
||||||
|
"env": {
|
||||||
|
"ANTHROPIC_BASE_URL": "https://dashscope-us.aliyuncs.com/compatible-mode/v1",
|
||||||
|
"ANTHROPIC_AUTH_TOKEN": "sk-...",
|
||||||
|
"ANTHROPIC_MODEL": "qwen3.6-plus",
|
||||||
|
"CCS_DROID_PROVIDER": "anthropic"
|
||||||
|
}
|
||||||
|
}
|
||||||
|
```
|
||||||
|
|
||||||
|
CCS now infers the OpenAI-compatible route from the base URL and does not let
|
||||||
|
that stale provider hint block proxy routing.
|
||||||
|
|
||||||
## Self-Signed TLS
|
## Self-Signed TLS
|
||||||
|
|
||||||
If your upstream gateway uses a self-signed or privately issued certificate,
|
If your upstream gateway uses a self-signed or privately issued certificate,
|
||||||
@@ -124,29 +261,50 @@ That flag is respected by both:
|
|||||||
|
|
||||||
- `ccs <profile>` with Claude target: auto-starts the local proxy when needed
|
- `ccs <profile>` with Claude target: auto-starts the local proxy when needed
|
||||||
- `ccs proxy start <profile>`: starts the proxy explicitly
|
- `ccs proxy start <profile>`: starts the proxy explicitly
|
||||||
|
- `GET /`: proxy info and bound profile details
|
||||||
- `GET /health`: proxy liveness check
|
- `GET /health`: proxy liveness check
|
||||||
- `GET /v1/models`: local view of the configured model mapping
|
- `GET /v1/models`: local view of the configured model mapping
|
||||||
- `POST /v1/messages`: Anthropic-compatible request entrypoint
|
- `POST /v1/messages`: Anthropic-compatible request entrypoint
|
||||||
|
|
||||||
|
## Troubleshooting
|
||||||
|
|
||||||
|
### Missing or invalid local proxy token
|
||||||
|
|
||||||
|
- Re-run `eval "$(ccs proxy activate)"`
|
||||||
|
- Check `ccs proxy status` and confirm the expected profile is running
|
||||||
|
|
||||||
|
### Self-signed or private CA upstream
|
||||||
|
|
||||||
|
- Add `CCS_OPENAI_PROXY_INSECURE=1` to the profile settings
|
||||||
|
- Restart the proxy after changing the setting
|
||||||
|
|
||||||
|
### Port conflict on `3456`
|
||||||
|
|
||||||
|
- Start with a fixed port: `ccs proxy start hf --port 3457`
|
||||||
|
- Re-run `ccs proxy activate` after changing the port
|
||||||
|
|
||||||
|
### Provider returns `429` or empty upstream output
|
||||||
|
|
||||||
|
- CCS now preserves upstream rate-limit errors and retry headers
|
||||||
|
- Empty or malformed provider JSON is returned as Anthropic-style `api_error`
|
||||||
|
|
||||||
|
### Requests route to the wrong model/profile
|
||||||
|
|
||||||
|
- Use an explicit selector such as `profile:model`
|
||||||
|
- Review `proxy.routing` if scenario routing is enabled
|
||||||
|
- Check CCS structured logs in `~/.ccs/logs/current.jsonl` for routing decisions
|
||||||
|
|
||||||
## Validation
|
## Validation
|
||||||
|
|
||||||
The shipped coverage includes:
|
The shipped coverage includes:
|
||||||
|
|
||||||
- unit tests for OpenAI-compatible profile detection
|
- unit tests for OpenAI-compatible profile detection
|
||||||
- unit tests for Anthropic -> OpenAI request translation
|
- unit tests for Anthropic -> OpenAI request translation
|
||||||
|
- unit tests for request-time profile/model routing and scenario routing
|
||||||
- unit tests for multi-line SSE parsing
|
- unit tests for multi-line SSE parsing
|
||||||
- integration tests for `/v1/messages` request/response translation
|
- integration tests for `/v1/messages` request/response translation
|
||||||
|
- integration tests for rate limits, empty upstream responses, timeout handling,
|
||||||
|
thinking/tool-call chunk streaming, and request-time routing
|
||||||
- integration tests for daemon lifecycle and `/health` / `/v1/models`
|
- integration tests for daemon lifecycle and `/health` / `/v1/models`
|
||||||
- e2e tests for `ccs proxy` lifecycle
|
- e2e tests for `ccs proxy` lifecycle
|
||||||
- e2e tests for `ccs <profile>` auto-routing through a mock upstream
|
- e2e tests for `ccs <profile>` auto-routing through a mock upstream
|
||||||
|
|
||||||
## Current Scope
|
|
||||||
|
|
||||||
The current implementation focuses on the core routing path:
|
|
||||||
|
|
||||||
- local proxy lifecycle
|
|
||||||
- Anthropic/OpenAI request-response translation
|
|
||||||
- Claude-target settings profile auto-routing
|
|
||||||
|
|
||||||
Scenario-based routing and token-count-driven model switching remain follow-up
|
|
||||||
work if they are needed beyond the base provider-routing flow.
|
|
||||||
@@ -41,7 +41,7 @@ All major modularization work is complete. The codebase evolved from monolithic
|
|||||||
|
|
||||||
### Recent Fixes
|
### Recent Fixes
|
||||||
|
|
||||||
- **2026-04-14**: **#991** CCS now auto-routes Claude-target settings profiles that use OpenAI-compatible endpoints through a local Anthropic-compatible proxy instead of sending raw Anthropic `/v1/messages` traffic directly to chat-completions backends. The new `ccs proxy` command supports `start`, `status`, `activate`, and `stop`, the runtime exposes `/health` and `/v1/models`, the shared SSE parser now handles multi-line `data:` payloads, and the feature ships with dedicated unit, integration, and e2e coverage.
|
- **2026-04-14**: **#991** CCS now auto-routes Claude-target settings profiles that use OpenAI-compatible endpoints through a local Anthropic-compatible proxy instead of sending raw Anthropic `/v1/messages` traffic directly to chat-completions backends. The `ccs proxy` command now supports `start`, `status`, `activate`, and `stop` with explicit host binding, shell-aware activation helpers, and a fuller local runtime env contract. The proxy surface now exposes `GET /`, `/health`, `/v1/models`, and `/v1/messages`, logs routing decisions into CCS structured logs, supports Anthropic image blocks plus request-time `profile:model` overrides, and adds config-driven scenario routing (`background`, `think`, `longContext`, `webSearch`) on top of the compatible-profile path. Coverage now includes request routing, rate-limit/timeout/empty-upstream failures, chunked tool-call streaming, and disconnect cleanup alongside the existing unit, integration, and e2e suites.
|
||||||
- **2026-04-10**: **#765** `/providers` now includes a first-class Hugging Face preset for API Profiles. CCS exposes Hugging Face Inference Providers through the existing OpenAI-compatible profile flow with the official router endpoint `https://router.huggingface.co/v1`, a short `hf` default profile name, and `hf` preset alias support for both the dashboard chooser and `ccs api create --preset hf`.
|
- **2026-04-10**: **#765** `/providers` now includes a first-class Hugging Face preset for API Profiles. CCS exposes Hugging Face Inference Providers through the existing OpenAI-compatible profile flow with the official router endpoint `https://router.huggingface.co/v1`, a short `hf` default profile name, and `hf` preset alias support for both the dashboard chooser and `ccs api create --preset hf`.
|
||||||
- **2026-04-10**: **#944** Image Analysis auth readiness no longer collapses to native Read when merged runtime-status dependency overrides include a missing initializer value. CCS now preserves default dependency functions when override entries are `undefined`, still reads token-backed auth status directly in the local readiness path, and includes regression coverage for the missing-initializer case that previously surfaced as `deps.initializeAccounts is not a function`.
|
- **2026-04-10**: **#944** Image Analysis auth readiness no longer collapses to native Read when merged runtime-status dependency overrides include a missing initializer value. CCS now preserves default dependency functions when override entries are `undefined`, still reads token-backed auth status directly in the local readiness path, and includes regression coverage for the missing-initializer case that previously surfaced as `deps.initializeAccounts is not a function`.
|
||||||
- **2026-04-10**: **#945** CCS now normalizes Gemini CLI and Antigravity tier signals around an explicit `free / pro / ultra / unknown` model, preserves raw tier ids such as `g1-pro-tier`, enriches Gemini quota responses with provider entitlement evidence, classifies `MODEL_CAPACITY_EXHAUSTED` separately from auth/entitlement failures, fixes the Antigravity CLI quota table so live quota-derived tiers no longer collapse back to stale `unknown`, adds Gemini tier ids to CLI quota output, extends Gemini Flash Lite grouping to cover `gemini-3.1-flash-lite-preview`, and allows Gemini account surfaces to render the same tier badge semantics as Antigravity.
|
- **2026-04-10**: **#945** CCS now normalizes Gemini CLI and Antigravity tier signals around an explicit `free / pro / ultra / unknown` model, preserves raw tier ids such as `g1-pro-tier`, enriches Gemini quota responses with provider entitlement evidence, classifies `MODEL_CAPACITY_EXHAUSTED` separately from auth/entitlement failures, fixes the Antigravity CLI quota table so live quota-derived tiers no longer collapse back to stale `unknown`, adds Gemini tier ids to CLI quota output, extends Gemini Flash Lite grouping to cover `gemini-3.1-flash-lite-preview`, and allows Gemini account surfaces to render the same tier badge semantics as Antigravity.
|
||||||
|
|||||||
@@ -36,12 +36,15 @@ function showHelp(): number {
|
|||||||
console.log('');
|
console.log('');
|
||||||
console.log('Options:');
|
console.log('Options:');
|
||||||
console.log(' --port <n> Override the local proxy port (default: 3456)');
|
console.log(' --port <n> Override the local proxy port (default: 3456)');
|
||||||
|
console.log(' --host <addr> Bind the proxy server to a specific host (default: 127.0.0.1)');
|
||||||
console.log(' --shell <name> activate only: auto|bash|zsh|fish|powershell');
|
console.log(' --shell <name> activate only: auto|bash|zsh|fish|powershell');
|
||||||
|
console.log(' --fish activate only: shorthand for --shell fish');
|
||||||
console.log(' --insecure Disable upstream TLS verification');
|
console.log(' --insecure Disable upstream TLS verification');
|
||||||
console.log('');
|
console.log('');
|
||||||
console.log('Examples:');
|
console.log('Examples:');
|
||||||
console.log(' ccs proxy start hf');
|
console.log(' ccs proxy start hf');
|
||||||
console.log(' eval "$(ccs proxy activate)"');
|
console.log(' eval "$(ccs proxy activate)"');
|
||||||
|
console.log(' ccs proxy activate --fish');
|
||||||
console.log(' ccs proxy status');
|
console.log(' ccs proxy status');
|
||||||
console.log(' ccs proxy stop');
|
console.log(' ccs proxy stop');
|
||||||
console.log('');
|
console.log('');
|
||||||
@@ -61,11 +64,14 @@ function resolveProfile(profileName: string) {
|
|||||||
async function handleStart(args: string[]): Promise<number> {
|
async function handleStart(args: string[]): Promise<number> {
|
||||||
const profileName = args.find((arg) => !arg.startsWith('-'));
|
const profileName = args.find((arg) => !arg.startsWith('-'));
|
||||||
if (!profileName) {
|
if (!profileName) {
|
||||||
console.error(fail('Usage: ccs proxy start <profile> [--port <n>] [--insecure]'));
|
console.error(
|
||||||
|
fail('Usage: ccs proxy start <profile> [--port <n>] [--host <addr>] [--insecure]')
|
||||||
|
);
|
||||||
return 1;
|
return 1;
|
||||||
}
|
}
|
||||||
|
|
||||||
const portValue = parseOptionValue(args, '--port');
|
const portValue = parseOptionValue(args, '--port');
|
||||||
|
const host = parseOptionValue(args, '--host');
|
||||||
const port = portValue ? Number.parseInt(portValue, 10) || 3456 : undefined;
|
const port = portValue ? Number.parseInt(portValue, 10) || 3456 : undefined;
|
||||||
let profile;
|
let profile;
|
||||||
try {
|
try {
|
||||||
@@ -77,6 +83,7 @@ async function handleStart(args: string[]): Promise<number> {
|
|||||||
|
|
||||||
const result = await startOpenAICompatProxy(profile, {
|
const result = await startOpenAICompatProxy(profile, {
|
||||||
...(port ? { port } : {}),
|
...(port ? { port } : {}),
|
||||||
|
...(host ? { host } : {}),
|
||||||
insecure: args.includes('--insecure'),
|
insecure: args.includes('--insecure'),
|
||||||
});
|
});
|
||||||
|
|
||||||
@@ -101,6 +108,10 @@ async function handleStatus(): Promise<number> {
|
|||||||
}
|
}
|
||||||
|
|
||||||
console.log(ok(`Proxy running on port ${status.port}`));
|
console.log(ok(`Proxy running on port ${status.port}`));
|
||||||
|
if (status.host) {
|
||||||
|
console.log(` Host: ${status.host}`);
|
||||||
|
console.log(` Local URL: http://${status.host}:${status.port}`);
|
||||||
|
}
|
||||||
console.log(` Profile: ${status.profileName}`);
|
console.log(` Profile: ${status.profileName}`);
|
||||||
console.log(` Base URL: ${status.baseUrl}`);
|
console.log(` Base URL: ${status.baseUrl}`);
|
||||||
if (status.model) {
|
if (status.model) {
|
||||||
@@ -119,7 +130,7 @@ async function handleActivate(args: string[]): Promise<number> {
|
|||||||
return 1;
|
return 1;
|
||||||
}
|
}
|
||||||
|
|
||||||
const shell = detectShell(parseOptionValue(args, '--shell'));
|
const shell = detectShell(args.includes('--fish') ? 'fish' : parseOptionValue(args, '--shell'));
|
||||||
let profile;
|
let profile;
|
||||||
try {
|
try {
|
||||||
profile = resolveProfile(status.profileName);
|
profile = resolveProfile(status.profileName);
|
||||||
@@ -128,7 +139,13 @@ async function handleActivate(args: string[]): Promise<number> {
|
|||||||
return 1;
|
return 1;
|
||||||
}
|
}
|
||||||
|
|
||||||
const env = buildOpenAICompatProxyEnv(profile, status.port, status.authToken);
|
const env = buildOpenAICompatProxyEnv(
|
||||||
|
profile,
|
||||||
|
status.port,
|
||||||
|
status.authToken,
|
||||||
|
undefined,
|
||||||
|
status.host || '127.0.0.1'
|
||||||
|
);
|
||||||
Object.entries(env).forEach(([key, value]) => {
|
Object.entries(env).forEach(([key, value]) => {
|
||||||
console.log(formatExportLine(shell, key, value));
|
console.log(formatExportLine(shell, key, value));
|
||||||
});
|
});
|
||||||
|
|||||||
@@ -384,6 +384,18 @@ function mergeWithDefaults(partial: Partial<UnifiedConfig>): UnifiedConfig {
|
|||||||
: defaults.cliproxy.routing?.strategy,
|
: defaults.cliproxy.routing?.strategy,
|
||||||
},
|
},
|
||||||
},
|
},
|
||||||
|
proxy: {
|
||||||
|
routing: {
|
||||||
|
default: partial.proxy?.routing?.default ?? defaults.proxy?.routing?.default,
|
||||||
|
background: partial.proxy?.routing?.background ?? defaults.proxy?.routing?.background,
|
||||||
|
think: partial.proxy?.routing?.think ?? defaults.proxy?.routing?.think,
|
||||||
|
longContext: partial.proxy?.routing?.longContext ?? defaults.proxy?.routing?.longContext,
|
||||||
|
webSearch: partial.proxy?.routing?.webSearch ?? defaults.proxy?.routing?.webSearch,
|
||||||
|
longContextThreshold:
|
||||||
|
partial.proxy?.routing?.longContextThreshold ??
|
||||||
|
defaults.proxy?.routing?.longContextThreshold,
|
||||||
|
},
|
||||||
|
},
|
||||||
logging: {
|
logging: {
|
||||||
enabled: partial.logging?.enabled ?? DEFAULT_LOGGING_CONFIG.enabled,
|
enabled: partial.logging?.enabled ?? DEFAULT_LOGGING_CONFIG.enabled,
|
||||||
level: partial.logging?.level ?? DEFAULT_LOGGING_CONFIG.level,
|
level: partial.logging?.level ?? DEFAULT_LOGGING_CONFIG.level,
|
||||||
@@ -674,6 +686,17 @@ function generateYamlWithComments(config: UnifiedConfig): string {
|
|||||||
);
|
);
|
||||||
lines.push('');
|
lines.push('');
|
||||||
|
|
||||||
|
if (config.proxy?.routing) {
|
||||||
|
lines.push('# ----------------------------------------------------------------------------');
|
||||||
|
lines.push('# Proxy Routing: OpenAI-compatible local proxy model selection rules');
|
||||||
|
lines.push('# Use profile:model selectors to force a target profile and upstream model.');
|
||||||
|
lines.push('# ----------------------------------------------------------------------------');
|
||||||
|
lines.push(
|
||||||
|
yaml.dump({ proxy: config.proxy }, { indent: 2, lineWidth: -1, quotingType: '"' }).trim()
|
||||||
|
);
|
||||||
|
lines.push('');
|
||||||
|
}
|
||||||
|
|
||||||
if (config.logging) {
|
if (config.logging) {
|
||||||
lines.push('# ----------------------------------------------------------------------------');
|
lines.push('# ----------------------------------------------------------------------------');
|
||||||
lines.push('# Logging: CCS-owned structured runtime logs');
|
lines.push('# Logging: CCS-owned structured runtime logs');
|
||||||
|
|||||||
@@ -498,6 +498,19 @@ export interface ProxyLocalConfig {
|
|||||||
auto_start: boolean;
|
auto_start: boolean;
|
||||||
}
|
}
|
||||||
|
|
||||||
|
export interface OpenAICompatProxyRoutingConfig {
|
||||||
|
default?: string;
|
||||||
|
background?: string;
|
||||||
|
think?: string;
|
||||||
|
longContext?: string;
|
||||||
|
webSearch?: string;
|
||||||
|
longContextThreshold?: number;
|
||||||
|
}
|
||||||
|
|
||||||
|
export interface OpenAICompatProxyConfig {
|
||||||
|
routing?: OpenAICompatProxyRoutingConfig;
|
||||||
|
}
|
||||||
|
|
||||||
/**
|
/**
|
||||||
* CLIProxy server configuration section.
|
* CLIProxy server configuration section.
|
||||||
* Controls whether CCS uses local or remote CLIProxyAPI instance.
|
* Controls whether CCS uses local or remote CLIProxyAPI instance.
|
||||||
@@ -856,6 +869,8 @@ export interface UnifiedConfig {
|
|||||||
profiles: Record<string, ProfileConfig>;
|
profiles: Record<string, ProfileConfig>;
|
||||||
/** CLIProxy configuration */
|
/** CLIProxy configuration */
|
||||||
cliproxy: CLIProxyConfig;
|
cliproxy: CLIProxyConfig;
|
||||||
|
/** OpenAI-compatible local proxy configuration */
|
||||||
|
proxy?: OpenAICompatProxyConfig;
|
||||||
/** CCS-owned structured logging configuration */
|
/** CCS-owned structured logging configuration */
|
||||||
logging?: LoggingConfig;
|
logging?: LoggingConfig;
|
||||||
/** User preferences */
|
/** User preferences */
|
||||||
@@ -934,6 +949,12 @@ export const DEFAULT_CLIPROXY_SERVER_CONFIG: CliproxyServerConfig = {
|
|||||||
},
|
},
|
||||||
};
|
};
|
||||||
|
|
||||||
|
export const DEFAULT_OPENAI_COMPAT_PROXY_CONFIG: OpenAICompatProxyConfig = {
|
||||||
|
routing: {
|
||||||
|
longContextThreshold: 60_000,
|
||||||
|
},
|
||||||
|
};
|
||||||
|
|
||||||
/**
|
/**
|
||||||
* Create an empty unified config with defaults.
|
* Create an empty unified config with defaults.
|
||||||
*/
|
*/
|
||||||
@@ -958,6 +979,11 @@ export function createEmptyUnifiedConfig(): UnifiedConfig {
|
|||||||
strategy: 'round-robin',
|
strategy: 'round-robin',
|
||||||
},
|
},
|
||||||
},
|
},
|
||||||
|
proxy: {
|
||||||
|
routing: {
|
||||||
|
...DEFAULT_OPENAI_COMPAT_PROXY_CONFIG.routing,
|
||||||
|
},
|
||||||
|
},
|
||||||
logging: { ...DEFAULT_LOGGING_CONFIG },
|
logging: { ...DEFAULT_LOGGING_CONFIG },
|
||||||
preferences: {
|
preferences: {
|
||||||
theme: 'system',
|
theme: 'system',
|
||||||
|
|||||||
@@ -4,6 +4,7 @@ import { startOpenAICompatProxyServer } from './server/proxy-server';
|
|||||||
|
|
||||||
interface RuntimeOptions {
|
interface RuntimeOptions {
|
||||||
port: number;
|
port: number;
|
||||||
|
host: string;
|
||||||
profileName: string;
|
profileName: string;
|
||||||
settingsPath: string;
|
settingsPath: string;
|
||||||
authToken: string;
|
authToken: string;
|
||||||
@@ -12,6 +13,7 @@ interface RuntimeOptions {
|
|||||||
|
|
||||||
function parseArgs(argv: string[]): RuntimeOptions {
|
function parseArgs(argv: string[]): RuntimeOptions {
|
||||||
let port = 3456;
|
let port = 3456;
|
||||||
|
let host = '127.0.0.1';
|
||||||
let profileName = '';
|
let profileName = '';
|
||||||
let settingsPath = '';
|
let settingsPath = '';
|
||||||
let authToken = '';
|
let authToken = '';
|
||||||
@@ -23,6 +25,10 @@ function parseArgs(argv: string[]): RuntimeOptions {
|
|||||||
port = Number.parseInt(argv[++i] || '', 10) || port;
|
port = Number.parseInt(argv[++i] || '', 10) || port;
|
||||||
continue;
|
continue;
|
||||||
}
|
}
|
||||||
|
if (arg === '--host' && argv[i + 1]) {
|
||||||
|
host = argv[++i] || host;
|
||||||
|
continue;
|
||||||
|
}
|
||||||
if (arg === '--profile' && argv[i + 1]) {
|
if (arg === '--profile' && argv[i + 1]) {
|
||||||
profileName = argv[++i] || '';
|
profileName = argv[++i] || '';
|
||||||
continue;
|
continue;
|
||||||
@@ -40,7 +46,7 @@ function parseArgs(argv: string[]): RuntimeOptions {
|
|||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
return { port, profileName, settingsPath, authToken, insecure };
|
return { port, host, profileName, settingsPath, authToken, insecure };
|
||||||
}
|
}
|
||||||
|
|
||||||
function startRuntime(options: RuntimeOptions): void {
|
function startRuntime(options: RuntimeOptions): void {
|
||||||
@@ -62,6 +68,7 @@ function startRuntime(options: RuntimeOptions): void {
|
|||||||
|
|
||||||
const server = startOpenAICompatProxyServer({
|
const server = startOpenAICompatProxyServer({
|
||||||
profile,
|
profile,
|
||||||
|
host: options.host,
|
||||||
port: options.port,
|
port: options.port,
|
||||||
authToken: options.authToken,
|
authToken: options.authToken,
|
||||||
insecure: options.insecure,
|
insecure: options.insecure,
|
||||||
|
|||||||
@@ -9,6 +9,7 @@ import {
|
|||||||
export interface OpenAICompatProxySession {
|
export interface OpenAICompatProxySession {
|
||||||
profileName: string;
|
profileName: string;
|
||||||
settingsPath: string;
|
settingsPath: string;
|
||||||
|
host: string;
|
||||||
port: number;
|
port: number;
|
||||||
baseUrl: string;
|
baseUrl: string;
|
||||||
authToken: string;
|
authToken: string;
|
||||||
|
|||||||
@@ -225,10 +225,11 @@ export async function stopOpenAICompatProxy(): Promise<{ success: boolean; error
|
|||||||
|
|
||||||
export async function startOpenAICompatProxy(
|
export async function startOpenAICompatProxy(
|
||||||
profile: OpenAICompatProfileConfig,
|
profile: OpenAICompatProfileConfig,
|
||||||
options: { port?: number; insecure?: boolean } = {}
|
options: { port?: number; host?: string; insecure?: boolean } = {}
|
||||||
): Promise<StartOpenAICompatProxyResult> {
|
): Promise<StartOpenAICompatProxyResult> {
|
||||||
return withOpenAICompatProxyLock(async () => {
|
return withOpenAICompatProxyLock(async () => {
|
||||||
const status = await getOpenAICompatProxyStatus();
|
const status = await getOpenAICompatProxyStatus();
|
||||||
|
const host = options.host?.trim() || status.host || '127.0.0.1';
|
||||||
const port =
|
const port =
|
||||||
typeof options.port === 'number'
|
typeof options.port === 'number'
|
||||||
? options.port
|
? options.port
|
||||||
@@ -245,7 +246,12 @@ export async function startOpenAICompatProxy(
|
|||||||
if (!Number.isInteger(port) || port < 1 || port > 65535) {
|
if (!Number.isInteger(port) || port < 1 || port > 65535) {
|
||||||
return { success: false, port, error: `Invalid port: ${port}` };
|
return { success: false, port, error: `Invalid port: ${port}` };
|
||||||
}
|
}
|
||||||
if (status.running && status.profileName === profile.profileName && status.port === port) {
|
if (
|
||||||
|
status.running &&
|
||||||
|
status.profileName === profile.profileName &&
|
||||||
|
status.port === port &&
|
||||||
|
(status.host || '127.0.0.1') === host
|
||||||
|
) {
|
||||||
return {
|
return {
|
||||||
success: true,
|
success: true,
|
||||||
alreadyRunning: true,
|
alreadyRunning: true,
|
||||||
@@ -304,6 +310,8 @@ export async function startOpenAICompatProxy(
|
|||||||
daemonEntry,
|
daemonEntry,
|
||||||
'--port',
|
'--port',
|
||||||
String(port),
|
String(port),
|
||||||
|
'--host',
|
||||||
|
host,
|
||||||
'--profile',
|
'--profile',
|
||||||
profile.profileName,
|
profile.profileName,
|
||||||
'--settings-path',
|
'--settings-path',
|
||||||
@@ -323,6 +331,7 @@ export async function startOpenAICompatProxy(
|
|||||||
writeOpenAICompatProxySession({
|
writeOpenAICompatProxySession({
|
||||||
profileName: profile.profileName,
|
profileName: profile.profileName,
|
||||||
settingsPath: profile.settingsPath,
|
settingsPath: profile.settingsPath,
|
||||||
|
host,
|
||||||
port,
|
port,
|
||||||
baseUrl: profile.baseUrl,
|
baseUrl: profile.baseUrl,
|
||||||
authToken,
|
authToken,
|
||||||
|
|||||||
@@ -4,11 +4,17 @@ export function buildOpenAICompatProxyEnv(
|
|||||||
profile: OpenAICompatProfileConfig,
|
profile: OpenAICompatProfileConfig,
|
||||||
port: number,
|
port: number,
|
||||||
authToken: string,
|
authToken: string,
|
||||||
claudeConfigDir?: string
|
claudeConfigDir?: string,
|
||||||
|
host = '127.0.0.1'
|
||||||
): Record<string, string> {
|
): Record<string, string> {
|
||||||
|
const localBaseUrl = `http://${host}:${port}`;
|
||||||
return {
|
return {
|
||||||
ANTHROPIC_BASE_URL: `http://127.0.0.1:${port}`,
|
ANTHROPIC_BASE_URL: localBaseUrl,
|
||||||
ANTHROPIC_AUTH_TOKEN: authToken,
|
ANTHROPIC_AUTH_TOKEN: authToken,
|
||||||
|
DISABLE_TELEMETRY: '1',
|
||||||
|
DISABLE_COST_WARNINGS: '1',
|
||||||
|
API_TIMEOUT_MS: '600000',
|
||||||
|
NO_PROXY: host === '127.0.0.1' ? '127.0.0.1,localhost' : `${host},127.0.0.1,localhost`,
|
||||||
...(profile.model ? { ANTHROPIC_MODEL: profile.model } : {}),
|
...(profile.model ? { ANTHROPIC_MODEL: profile.model } : {}),
|
||||||
...(profile.opusModel ? { ANTHROPIC_DEFAULT_OPUS_MODEL: profile.opusModel } : {}),
|
...(profile.opusModel ? { ANTHROPIC_DEFAULT_OPUS_MODEL: profile.opusModel } : {}),
|
||||||
...(profile.sonnetModel ? { ANTHROPIC_DEFAULT_SONNET_MODEL: profile.sonnetModel } : {}),
|
...(profile.sonnetModel ? { ANTHROPIC_DEFAULT_SONNET_MODEL: profile.sonnetModel } : {}),
|
||||||
|
|||||||
@@ -0,0 +1,222 @@
|
|||||||
|
import { loadConfigSafe, loadSettings } from '../utils/config-manager';
|
||||||
|
import { expandPath } from '../utils/helpers';
|
||||||
|
import type { ProxyOpenAIRequest } from './transformers/request-transformer';
|
||||||
|
import {
|
||||||
|
loadOpenAICompatProxyRoutingConfig,
|
||||||
|
type OpenAICompatProxyRoutingConfig,
|
||||||
|
} from './routing-config';
|
||||||
|
import { resolveOpenAICompatProfileConfig, type OpenAICompatProfileConfig } from './profile-router';
|
||||||
|
|
||||||
|
export type ProxyRoutingScenario = 'default' | 'background' | 'think' | 'longContext' | 'webSearch';
|
||||||
|
|
||||||
|
export interface ProxyRequestRoute {
|
||||||
|
profile: OpenAICompatProfileConfig;
|
||||||
|
model?: string;
|
||||||
|
scenario?: ProxyRoutingScenario;
|
||||||
|
estimatedTokens: number;
|
||||||
|
source:
|
||||||
|
| 'explicit-profile'
|
||||||
|
| 'scenario'
|
||||||
|
| 'profile-model-match'
|
||||||
|
| 'profile-name'
|
||||||
|
| 'request-model'
|
||||||
|
| 'active-default';
|
||||||
|
}
|
||||||
|
|
||||||
|
function loadOpenAICompatProfiles(
|
||||||
|
activeProfile: OpenAICompatProfileConfig
|
||||||
|
): OpenAICompatProfileConfig[] {
|
||||||
|
const config = loadConfigSafe();
|
||||||
|
const profiles = [activeProfile];
|
||||||
|
|
||||||
|
for (const [profileName, settingsPath] of Object.entries(config.profiles)) {
|
||||||
|
if (profileName === activeProfile.profileName) {
|
||||||
|
continue;
|
||||||
|
}
|
||||||
|
|
||||||
|
try {
|
||||||
|
const expandedPath = expandPath(settingsPath);
|
||||||
|
const settings = loadSettings(expandedPath);
|
||||||
|
const profile = resolveOpenAICompatProfileConfig(
|
||||||
|
profileName,
|
||||||
|
expandedPath,
|
||||||
|
settings.env || {}
|
||||||
|
);
|
||||||
|
if (profile) {
|
||||||
|
profiles.push(profile);
|
||||||
|
}
|
||||||
|
} catch {
|
||||||
|
// Ignore invalid profiles while routing a live request.
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
return profiles;
|
||||||
|
}
|
||||||
|
|
||||||
|
function resolveSelectorTarget(
|
||||||
|
selector: string,
|
||||||
|
activeProfile: OpenAICompatProfileConfig,
|
||||||
|
profiles: OpenAICompatProfileConfig[]
|
||||||
|
): { profile: OpenAICompatProfileConfig; model?: string; explicitProfile: boolean } | null {
|
||||||
|
const trimmed = selector.trim();
|
||||||
|
if (!trimmed) {
|
||||||
|
return null;
|
||||||
|
}
|
||||||
|
|
||||||
|
const colonIndex = trimmed.indexOf(':');
|
||||||
|
if (colonIndex > 0) {
|
||||||
|
const profileName = trimmed.slice(0, colonIndex).trim();
|
||||||
|
const profile = profiles.find((candidate) => candidate.profileName === profileName);
|
||||||
|
if (profile) {
|
||||||
|
const model = trimmed.slice(colonIndex + 1).trim() || profile.model;
|
||||||
|
return { profile, model, explicitProfile: true };
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
const namedProfile = profiles.find((candidate) => candidate.profileName === trimmed);
|
||||||
|
if (namedProfile) {
|
||||||
|
return { profile: namedProfile, model: namedProfile.model, explicitProfile: false };
|
||||||
|
}
|
||||||
|
|
||||||
|
return {
|
||||||
|
profile: activeProfile,
|
||||||
|
model: trimmed,
|
||||||
|
explicitProfile: false,
|
||||||
|
};
|
||||||
|
}
|
||||||
|
|
||||||
|
function profileSupportsModel(profile: OpenAICompatProfileConfig, model: string): boolean {
|
||||||
|
return [profile.model, profile.opusModel, profile.sonnetModel, profile.haikuModel].some(
|
||||||
|
(candidate) => typeof candidate === 'string' && candidate === model
|
||||||
|
);
|
||||||
|
}
|
||||||
|
|
||||||
|
function estimateTokens(request: ProxyOpenAIRequest): number {
|
||||||
|
let characters = 0;
|
||||||
|
for (const message of request.messages) {
|
||||||
|
if (typeof message.content === 'string') {
|
||||||
|
characters += message.content.length;
|
||||||
|
continue;
|
||||||
|
}
|
||||||
|
if (Array.isArray(message.content)) {
|
||||||
|
for (const part of message.content) {
|
||||||
|
characters += part.type === 'text' ? part.text.length : part.image_url.url.length;
|
||||||
|
}
|
||||||
|
}
|
||||||
|
if (Array.isArray(message.tool_calls)) {
|
||||||
|
for (const toolCall of message.tool_calls) {
|
||||||
|
characters += toolCall.function.name.length + toolCall.function.arguments.length;
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
if (Array.isArray(request.tools)) {
|
||||||
|
characters += JSON.stringify(request.tools).length;
|
||||||
|
}
|
||||||
|
|
||||||
|
return Math.max(1, Math.ceil(characters / 4));
|
||||||
|
}
|
||||||
|
|
||||||
|
function detectScenario(
|
||||||
|
request: ProxyOpenAIRequest,
|
||||||
|
requestedModel: string | undefined,
|
||||||
|
routing: OpenAICompatProxyRoutingConfig
|
||||||
|
): { scenario?: ProxyRoutingScenario; selector?: string; estimatedTokens: number } {
|
||||||
|
const estimatedTokens = estimateTokens(request);
|
||||||
|
const hasWebSearchTool =
|
||||||
|
request.tools?.some((tool) => tool.function.name === 'web_search') === true;
|
||||||
|
const thinkingEnabled =
|
||||||
|
request.reasoning?.enabled === true || typeof request.reasoning_effort === 'string';
|
||||||
|
const modelId = requestedModel || '';
|
||||||
|
const longContextThreshold = routing.longContextThreshold ?? 60_000;
|
||||||
|
|
||||||
|
if (hasWebSearchTool && routing.webSearch) {
|
||||||
|
return { scenario: 'webSearch', selector: routing.webSearch, estimatedTokens };
|
||||||
|
}
|
||||||
|
if (thinkingEnabled && routing.think) {
|
||||||
|
return { scenario: 'think', selector: routing.think, estimatedTokens };
|
||||||
|
}
|
||||||
|
if (estimatedTokens > longContextThreshold && routing.longContext) {
|
||||||
|
return { scenario: 'longContext', selector: routing.longContext, estimatedTokens };
|
||||||
|
}
|
||||||
|
if (modelId.toLowerCase().includes('haiku') && routing.background) {
|
||||||
|
return { scenario: 'background', selector: routing.background, estimatedTokens };
|
||||||
|
}
|
||||||
|
if (!requestedModel && routing.default) {
|
||||||
|
return { scenario: 'default', selector: routing.default, estimatedTokens };
|
||||||
|
}
|
||||||
|
|
||||||
|
return { estimatedTokens };
|
||||||
|
}
|
||||||
|
|
||||||
|
export function resolveProxyRequestRoute(
|
||||||
|
activeProfile: OpenAICompatProfileConfig,
|
||||||
|
request: ProxyOpenAIRequest
|
||||||
|
): ProxyRequestRoute {
|
||||||
|
const profiles = loadOpenAICompatProfiles(activeProfile);
|
||||||
|
const requestedModel = request.model?.trim() || undefined;
|
||||||
|
const explicitTarget = requestedModel
|
||||||
|
? resolveSelectorTarget(requestedModel, activeProfile, profiles)
|
||||||
|
: null;
|
||||||
|
const routing = loadOpenAICompatProxyRoutingConfig();
|
||||||
|
|
||||||
|
if (explicitTarget?.explicitProfile) {
|
||||||
|
return {
|
||||||
|
profile: explicitTarget.profile,
|
||||||
|
model: explicitTarget.model,
|
||||||
|
estimatedTokens: estimateTokens(request),
|
||||||
|
source: 'explicit-profile',
|
||||||
|
};
|
||||||
|
}
|
||||||
|
|
||||||
|
const scenario = detectScenario(request, requestedModel, routing);
|
||||||
|
if (scenario.selector) {
|
||||||
|
const scenarioTarget = resolveSelectorTarget(scenario.selector, activeProfile, profiles);
|
||||||
|
if (scenarioTarget) {
|
||||||
|
return {
|
||||||
|
profile: scenarioTarget.profile,
|
||||||
|
model: scenarioTarget.model,
|
||||||
|
scenario: scenario.scenario,
|
||||||
|
estimatedTokens: scenario.estimatedTokens,
|
||||||
|
source: 'scenario',
|
||||||
|
};
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
if (explicitTarget && explicitTarget.profile.profileName !== activeProfile.profileName) {
|
||||||
|
return {
|
||||||
|
profile: explicitTarget.profile,
|
||||||
|
model: explicitTarget.model,
|
||||||
|
estimatedTokens: scenario.estimatedTokens,
|
||||||
|
source: 'profile-name',
|
||||||
|
};
|
||||||
|
}
|
||||||
|
|
||||||
|
if (requestedModel) {
|
||||||
|
const matchedProfile = profiles.find((profile) =>
|
||||||
|
profileSupportsModel(profile, requestedModel)
|
||||||
|
);
|
||||||
|
if (matchedProfile) {
|
||||||
|
return {
|
||||||
|
profile: matchedProfile,
|
||||||
|
model: requestedModel,
|
||||||
|
estimatedTokens: scenario.estimatedTokens,
|
||||||
|
source: 'profile-model-match',
|
||||||
|
};
|
||||||
|
}
|
||||||
|
|
||||||
|
return {
|
||||||
|
profile: activeProfile,
|
||||||
|
model: requestedModel,
|
||||||
|
estimatedTokens: scenario.estimatedTokens,
|
||||||
|
source: 'request-model',
|
||||||
|
};
|
||||||
|
}
|
||||||
|
|
||||||
|
return {
|
||||||
|
profile: activeProfile,
|
||||||
|
model: activeProfile.model,
|
||||||
|
estimatedTokens: scenario.estimatedTokens,
|
||||||
|
source: 'active-default',
|
||||||
|
};
|
||||||
|
}
|
||||||
@@ -0,0 +1,74 @@
|
|||||||
|
import * as fs from 'fs';
|
||||||
|
import * as yaml from 'js-yaml';
|
||||||
|
import { getActiveConfigPath, getConfigPath } from '../utils/config-manager';
|
||||||
|
|
||||||
|
export interface OpenAICompatProxyRoutingConfig {
|
||||||
|
default?: string;
|
||||||
|
background?: string;
|
||||||
|
think?: string;
|
||||||
|
longContext?: string;
|
||||||
|
webSearch?: string;
|
||||||
|
longContextThreshold?: number;
|
||||||
|
}
|
||||||
|
|
||||||
|
function readRawConfigObject(configPath: string): Record<string, unknown> | null {
|
||||||
|
if (!fs.existsSync(configPath)) {
|
||||||
|
return null;
|
||||||
|
}
|
||||||
|
|
||||||
|
const raw = fs.readFileSync(configPath, 'utf8');
|
||||||
|
const parsed =
|
||||||
|
configPath.endsWith('.yaml') || configPath.endsWith('.yml') ? yaml.load(raw) : JSON.parse(raw);
|
||||||
|
return typeof parsed === 'object' && parsed !== null ? (parsed as Record<string, unknown>) : null;
|
||||||
|
}
|
||||||
|
|
||||||
|
function normalizeRoutingConfig(value: unknown): OpenAICompatProxyRoutingConfig {
|
||||||
|
if (typeof value !== 'object' || value === null) {
|
||||||
|
return {};
|
||||||
|
}
|
||||||
|
|
||||||
|
const config = value as Record<string, unknown>;
|
||||||
|
return {
|
||||||
|
default:
|
||||||
|
typeof config.default === 'string' && config.default.trim()
|
||||||
|
? config.default.trim()
|
||||||
|
: undefined,
|
||||||
|
background:
|
||||||
|
typeof config.background === 'string' && config.background.trim()
|
||||||
|
? config.background.trim()
|
||||||
|
: undefined,
|
||||||
|
think:
|
||||||
|
typeof config.think === 'string' && config.think.trim() ? config.think.trim() : undefined,
|
||||||
|
longContext:
|
||||||
|
typeof config.longContext === 'string' && config.longContext.trim()
|
||||||
|
? config.longContext.trim()
|
||||||
|
: undefined,
|
||||||
|
webSearch:
|
||||||
|
typeof config.webSearch === 'string' && config.webSearch.trim()
|
||||||
|
? config.webSearch.trim()
|
||||||
|
: undefined,
|
||||||
|
longContextThreshold:
|
||||||
|
typeof config.longContextThreshold === 'number' &&
|
||||||
|
Number.isFinite(config.longContextThreshold)
|
||||||
|
? config.longContextThreshold
|
||||||
|
: undefined,
|
||||||
|
};
|
||||||
|
}
|
||||||
|
|
||||||
|
export function loadOpenAICompatProxyRoutingConfig(): OpenAICompatProxyRoutingConfig {
|
||||||
|
const activePath = getActiveConfigPath();
|
||||||
|
const rawConfig = readRawConfigObject(activePath);
|
||||||
|
if (rawConfig?.proxy && typeof rawConfig.proxy === 'object') {
|
||||||
|
return normalizeRoutingConfig((rawConfig.proxy as Record<string, unknown>).routing);
|
||||||
|
}
|
||||||
|
|
||||||
|
const legacyPath = getConfigPath();
|
||||||
|
if (legacyPath !== activePath) {
|
||||||
|
const legacyConfig = readRawConfigObject(legacyPath);
|
||||||
|
if (legacyConfig?.proxy && typeof legacyConfig.proxy === 'object') {
|
||||||
|
return normalizeRoutingConfig((legacyConfig.proxy as Record<string, unknown>).routing);
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
return {};
|
||||||
|
}
|
||||||
@@ -1,12 +1,15 @@
|
|||||||
import * as http from 'http';
|
import * as http from 'http';
|
||||||
import type { Dispatcher } from 'undici';
|
import type { Dispatcher } from 'undici';
|
||||||
import type { OpenAICompatProfileConfig } from '../profile-router';
|
import type { OpenAICompatProfileConfig } from '../profile-router';
|
||||||
|
import { resolveProxyRequestRoute } from '../request-router';
|
||||||
import { ProxyRequestTransformer } from '../transformers/request-transformer';
|
import { ProxyRequestTransformer } from '../transformers/request-transformer';
|
||||||
import { ProxySseStreamTransformer } from '../transformers/sse-stream-transformer';
|
import { ProxySseStreamTransformer } from '../transformers/sse-stream-transformer';
|
||||||
import { resolveOpenAIChatCompletionsUrl } from '../upstream-url';
|
import { resolveOpenAIChatCompletionsUrl } from '../upstream-url';
|
||||||
|
import { createLogger } from '../../services/logging';
|
||||||
import { pipeWebResponseToNode, readJsonBody, writeJson } from './http-helpers';
|
import { pipeWebResponseToNode, readJsonBody, writeJson } from './http-helpers';
|
||||||
|
|
||||||
const REQUEST_TIMEOUT_MS = 30_000;
|
const REQUEST_TIMEOUT_MS = 600_000;
|
||||||
|
const logger = createLogger('proxy:openai-compat:messages');
|
||||||
|
|
||||||
class ProxyInputError extends Error {
|
class ProxyInputError extends Error {
|
||||||
constructor(message: string) {
|
constructor(message: string) {
|
||||||
@@ -23,7 +26,10 @@ function buildUpstreamHeaders(profile: OpenAICompatProfileConfig): Record<string
|
|||||||
};
|
};
|
||||||
}
|
}
|
||||||
|
|
||||||
function buildUpstreamBody(profile: OpenAICompatProfileConfig, rawBody: unknown): string {
|
function buildUpstreamRequest(
|
||||||
|
profile: OpenAICompatProfileConfig,
|
||||||
|
rawBody: unknown
|
||||||
|
): { body: string; route: ReturnType<typeof resolveProxyRequestRoute> } {
|
||||||
let transformed;
|
let transformed;
|
||||||
try {
|
try {
|
||||||
const transformer = new ProxyRequestTransformer();
|
const transformer = new ProxyRequestTransformer();
|
||||||
@@ -32,12 +38,13 @@ function buildUpstreamBody(profile: OpenAICompatProfileConfig, rawBody: unknown)
|
|||||||
const message = error instanceof Error ? error.message : 'Invalid Anthropic request';
|
const message = error instanceof Error ? error.message : 'Invalid Anthropic request';
|
||||||
throw new ProxyInputError(message);
|
throw new ProxyInputError(message);
|
||||||
}
|
}
|
||||||
|
const route = resolveProxyRequestRoute(profile, transformed);
|
||||||
const body = {
|
const body = {
|
||||||
...transformed,
|
...transformed,
|
||||||
model: transformed.model || profile.model,
|
model: route.model || route.profile.model,
|
||||||
stream: transformed.stream === true,
|
stream: transformed.stream === true,
|
||||||
};
|
};
|
||||||
return JSON.stringify(body);
|
return { body: JSON.stringify(body), route };
|
||||||
}
|
}
|
||||||
|
|
||||||
export function extractIncomingProxyToken(headers: http.IncomingHttpHeaders): string | null {
|
export function extractIncomingProxyToken(headers: http.IncomingHttpHeaders): string | null {
|
||||||
@@ -88,6 +95,20 @@ function buildFetchInit(
|
|||||||
return init;
|
return init;
|
||||||
}
|
}
|
||||||
|
|
||||||
|
function getRequestTimeoutMs(): number {
|
||||||
|
const rawValue = process.env.CCS_OPENAI_PROXY_REQUEST_TIMEOUT_MS;
|
||||||
|
if (!rawValue) {
|
||||||
|
return REQUEST_TIMEOUT_MS;
|
||||||
|
}
|
||||||
|
|
||||||
|
const parsed = Number.parseInt(rawValue, 10);
|
||||||
|
return Number.isFinite(parsed) && parsed > 0 ? parsed : REQUEST_TIMEOUT_MS;
|
||||||
|
}
|
||||||
|
|
||||||
|
function formatTimeoutDuration(timeoutMs: number): string {
|
||||||
|
return timeoutMs % 1000 === 0 ? `${timeoutMs / 1000} seconds` : `${timeoutMs}ms`;
|
||||||
|
}
|
||||||
|
|
||||||
export async function handleProxyMessagesRequest(
|
export async function handleProxyMessagesRequest(
|
||||||
req: http.IncomingMessage,
|
req: http.IncomingMessage,
|
||||||
res: http.ServerResponse,
|
res: http.ServerResponse,
|
||||||
@@ -98,6 +119,9 @@ export async function handleProxyMessagesRequest(
|
|||||||
const transformer = new ProxySseStreamTransformer();
|
const transformer = new ProxySseStreamTransformer();
|
||||||
|
|
||||||
if (!validateIncomingProxyAuth(req.headers, expectedAuthToken)) {
|
if (!validateIncomingProxyAuth(req.headers, expectedAuthToken)) {
|
||||||
|
logger.warn('auth.invalid', 'Rejected proxy message request with invalid auth token', {
|
||||||
|
remoteAddress: req.socket.remoteAddress || null,
|
||||||
|
});
|
||||||
await pipeWebResponseToNode(
|
await pipeWebResponseToNode(
|
||||||
transformer.error(401, 'authentication_error', 'Missing or invalid local proxy token'),
|
transformer.error(401, 'authentication_error', 'Missing or invalid local proxy token'),
|
||||||
res
|
res
|
||||||
@@ -105,34 +129,77 @@ export async function handleProxyMessagesRequest(
|
|||||||
return;
|
return;
|
||||||
}
|
}
|
||||||
|
|
||||||
|
let timeoutMs = REQUEST_TIMEOUT_MS;
|
||||||
try {
|
try {
|
||||||
const rawBody = await readJsonBody(req);
|
const rawBody = await readJsonBody(req);
|
||||||
const upstreamBody = buildUpstreamBody(profile, rawBody);
|
const upstream = buildUpstreamRequest(profile, rawBody);
|
||||||
|
logger.info('request.forward', 'Forwarding Anthropic request to OpenAI-compatible upstream', {
|
||||||
|
profileName: upstream.route.profile.profileName,
|
||||||
|
provider: upstream.route.profile.provider,
|
||||||
|
baseUrl: upstream.route.profile.baseUrl,
|
||||||
|
model: upstream.route.model || upstream.route.profile.model || null,
|
||||||
|
routeSource: upstream.route.source,
|
||||||
|
scenario: upstream.route.scenario || null,
|
||||||
|
estimatedTokens: upstream.route.estimatedTokens,
|
||||||
|
});
|
||||||
const controller = new AbortController();
|
const controller = new AbortController();
|
||||||
const timeout = setTimeout(() => controller.abort(), REQUEST_TIMEOUT_MS);
|
timeoutMs = getRequestTimeoutMs();
|
||||||
const abortOnDisconnect = () => {
|
const timeout = setTimeout(() => controller.abort(), timeoutMs);
|
||||||
|
const abortOnDisconnect = (source: string) => {
|
||||||
if (!controller.signal.aborted && !res.writableEnded) {
|
if (!controller.signal.aborted && !res.writableEnded) {
|
||||||
|
logger.info(
|
||||||
|
'request.disconnect',
|
||||||
|
'Aborting upstream request after local client disconnect',
|
||||||
|
{
|
||||||
|
profileName: profile.profileName,
|
||||||
|
source,
|
||||||
|
}
|
||||||
|
);
|
||||||
controller.abort();
|
controller.abort();
|
||||||
}
|
}
|
||||||
};
|
};
|
||||||
|
|
||||||
req.on('aborted', abortOnDisconnect);
|
req.on('aborted', () => abortOnDisconnect('req.aborted'));
|
||||||
req.on('close', abortOnDisconnect);
|
req.on('close', () => abortOnDisconnect('req.close'));
|
||||||
res.on('close', abortOnDisconnect);
|
req.socket?.on('close', () => abortOnDisconnect('req.socket.close'));
|
||||||
|
res.on('close', () => abortOnDisconnect('res.close'));
|
||||||
|
res.socket?.on('close', () => abortOnDisconnect('res.socket.close'));
|
||||||
|
const disconnectPoll = setInterval(() => {
|
||||||
|
if (
|
||||||
|
req.destroyed ||
|
||||||
|
res.destroyed ||
|
||||||
|
req.socket?.destroyed === true ||
|
||||||
|
res.socket?.destroyed === true
|
||||||
|
) {
|
||||||
|
abortOnDisconnect('poll.destroyed');
|
||||||
|
}
|
||||||
|
}, 50);
|
||||||
|
|
||||||
try {
|
try {
|
||||||
const upstreamResponse = await fetch(
|
const upstreamResponse = await fetch(
|
||||||
resolveOpenAIChatCompletionsUrl(profile.baseUrl),
|
resolveOpenAIChatCompletionsUrl(upstream.route.profile.baseUrl),
|
||||||
buildFetchInit(profile, upstreamBody, controller.signal, insecureDispatcher)
|
buildFetchInit(upstream.route.profile, upstream.body, controller.signal, insecureDispatcher)
|
||||||
);
|
);
|
||||||
clearTimeout(timeout);
|
clearTimeout(timeout);
|
||||||
|
clearInterval(disconnectPoll);
|
||||||
|
logger.info('response.received', 'Received upstream response', {
|
||||||
|
profileName: profile.profileName,
|
||||||
|
routedProfileName: upstream.route.profile.profileName,
|
||||||
|
status: upstreamResponse.status,
|
||||||
|
});
|
||||||
const response = await transformer.transform(upstreamResponse);
|
const response = await transformer.transform(upstreamResponse);
|
||||||
await pipeWebResponseToNode(response, res);
|
await pipeWebResponseToNode(response, res);
|
||||||
} finally {
|
} finally {
|
||||||
clearTimeout(timeout);
|
clearTimeout(timeout);
|
||||||
|
clearInterval(disconnectPoll);
|
||||||
}
|
}
|
||||||
} catch (error) {
|
} catch (error) {
|
||||||
const message = error instanceof Error ? error.message : 'Unknown proxy error';
|
const message = error instanceof Error ? error.message : 'Unknown proxy error';
|
||||||
|
logger.error('request.failed', 'Proxy message request failed', {
|
||||||
|
profileName: profile.profileName,
|
||||||
|
error: message,
|
||||||
|
abort: error instanceof Error && error.name === 'AbortError',
|
||||||
|
});
|
||||||
const status =
|
const status =
|
||||||
error instanceof Error && error.name === 'AbortError'
|
error instanceof Error && error.name === 'AbortError'
|
||||||
? 502
|
? 502
|
||||||
@@ -149,7 +216,7 @@ export async function handleProxyMessagesRequest(
|
|||||||
status,
|
status,
|
||||||
type,
|
type,
|
||||||
error instanceof Error && error.name === 'AbortError'
|
error instanceof Error && error.name === 'AbortError'
|
||||||
? 'The upstream provider did not respond within 30 seconds'
|
? `The upstream provider did not respond within ${formatTimeoutDuration(timeoutMs)}`
|
||||||
: message
|
: message
|
||||||
),
|
),
|
||||||
res
|
res
|
||||||
|
|||||||
@@ -2,6 +2,7 @@ import * as http from 'http';
|
|||||||
import { Agent } from 'undici';
|
import { Agent } from 'undici';
|
||||||
import type { OpenAICompatProfileConfig } from '../profile-router';
|
import type { OpenAICompatProfileConfig } from '../profile-router';
|
||||||
import { OPENAI_COMPAT_PROXY_SERVICE_NAME } from '../proxy-daemon-paths';
|
import { OPENAI_COMPAT_PROXY_SERVICE_NAME } from '../proxy-daemon-paths';
|
||||||
|
import { createLogger } from '../../services/logging';
|
||||||
import {
|
import {
|
||||||
handleProxyMessagesRequest,
|
handleProxyMessagesRequest,
|
||||||
handleProxyModelsRequest,
|
handleProxyModelsRequest,
|
||||||
@@ -11,12 +12,19 @@ import { writeJson } from './http-helpers';
|
|||||||
|
|
||||||
export interface OpenAICompatProxyServerOptions {
|
export interface OpenAICompatProxyServerOptions {
|
||||||
profile: OpenAICompatProfileConfig;
|
profile: OpenAICompatProfileConfig;
|
||||||
|
host?: string;
|
||||||
port: number;
|
port: number;
|
||||||
authToken: string;
|
authToken: string;
|
||||||
insecure?: boolean;
|
insecure?: boolean;
|
||||||
}
|
}
|
||||||
|
|
||||||
export function startOpenAICompatProxyServer(options: OpenAICompatProxyServerOptions): http.Server {
|
export function startOpenAICompatProxyServer(options: OpenAICompatProxyServerOptions): http.Server {
|
||||||
|
const host = options.host?.trim() || '127.0.0.1';
|
||||||
|
const logger = createLogger('proxy:openai-compat', {
|
||||||
|
profileName: options.profile.profileName,
|
||||||
|
host,
|
||||||
|
port: options.port,
|
||||||
|
});
|
||||||
const insecureDispatcher = options.insecure
|
const insecureDispatcher = options.insecure
|
||||||
? new Agent({ connect: { rejectUnauthorized: false } })
|
? new Agent({ connect: { rejectUnauthorized: false } })
|
||||||
: undefined;
|
: undefined;
|
||||||
@@ -31,12 +39,31 @@ export function startOpenAICompatProxyServer(options: OpenAICompatProxyServerOpt
|
|||||||
writeJson(res, 200, {
|
writeJson(res, 200, {
|
||||||
ok: true,
|
ok: true,
|
||||||
service: OPENAI_COMPAT_PROXY_SERVICE_NAME,
|
service: OPENAI_COMPAT_PROXY_SERVICE_NAME,
|
||||||
|
host,
|
||||||
profile: options.profile.profileName,
|
profile: options.profile.profileName,
|
||||||
port: options.port,
|
port: options.port,
|
||||||
});
|
});
|
||||||
return;
|
return;
|
||||||
}
|
}
|
||||||
|
|
||||||
|
if (method === 'GET' && pathname === '/') {
|
||||||
|
writeJson(res, 200, {
|
||||||
|
ok: true,
|
||||||
|
service: OPENAI_COMPAT_PROXY_SERVICE_NAME,
|
||||||
|
bind: {
|
||||||
|
host,
|
||||||
|
port: options.port,
|
||||||
|
},
|
||||||
|
profile: {
|
||||||
|
name: options.profile.profileName,
|
||||||
|
provider: options.profile.provider,
|
||||||
|
model: options.profile.model || null,
|
||||||
|
},
|
||||||
|
endpoints: ['/health', '/v1/messages', '/v1/models'],
|
||||||
|
});
|
||||||
|
return;
|
||||||
|
}
|
||||||
|
|
||||||
if (method === 'GET' && pathname === '/v1/models') {
|
if (method === 'GET' && pathname === '/v1/models') {
|
||||||
if (!validateIncomingProxyAuth(req.headers, options.authToken)) {
|
if (!validateIncomingProxyAuth(req.headers, options.authToken)) {
|
||||||
writeJson(res, 401, {
|
writeJson(res, 401, {
|
||||||
@@ -63,13 +90,21 @@ export function startOpenAICompatProxyServer(options: OpenAICompatProxyServerOpt
|
|||||||
return;
|
return;
|
||||||
}
|
}
|
||||||
|
|
||||||
|
logger.warn('http.not_found', 'Rejected unknown proxy route', {
|
||||||
|
method,
|
||||||
|
pathname,
|
||||||
|
});
|
||||||
writeJson(res, 404, { error: 'Not found' });
|
writeJson(res, 404, { error: 'Not found' });
|
||||||
});
|
});
|
||||||
|
|
||||||
|
logger.info('server.start', 'OpenAI-compatible proxy server listening', {
|
||||||
|
baseUrl: `http://${host}:${options.port}`,
|
||||||
|
});
|
||||||
server.on('close', () => {
|
server.on('close', () => {
|
||||||
|
logger.info('server.stop', 'OpenAI-compatible proxy server stopped');
|
||||||
void insecureDispatcher?.close();
|
void insecureDispatcher?.close();
|
||||||
});
|
});
|
||||||
|
|
||||||
server.listen(options.port, '127.0.0.1');
|
server.listen(options.port, host);
|
||||||
return server;
|
return server;
|
||||||
}
|
}
|
||||||
@@ -1,20 +1,97 @@
|
|||||||
import { translateAnthropicRequest } from '../../cursor/cursor-anthropic-translator';
|
interface AnthropicThinking {
|
||||||
|
type?: 'enabled' | 'disabled' | string;
|
||||||
|
budget_tokens?: number;
|
||||||
|
}
|
||||||
|
|
||||||
|
interface AnthropicTextBlock {
|
||||||
|
type: 'text';
|
||||||
|
text?: string;
|
||||||
|
}
|
||||||
|
|
||||||
|
interface AnthropicImageBlock {
|
||||||
|
type: 'image';
|
||||||
|
source?: {
|
||||||
|
type?: string;
|
||||||
|
media_type?: string;
|
||||||
|
data?: string;
|
||||||
|
};
|
||||||
|
}
|
||||||
|
|
||||||
|
interface AnthropicToolUseBlock {
|
||||||
|
type: 'tool_use';
|
||||||
|
id?: string;
|
||||||
|
name?: string;
|
||||||
|
input?: Record<string, unknown>;
|
||||||
|
}
|
||||||
|
|
||||||
|
interface AnthropicToolResultBlock {
|
||||||
|
type: 'tool_result';
|
||||||
|
tool_use_id?: string;
|
||||||
|
content?: unknown;
|
||||||
|
}
|
||||||
|
|
||||||
|
type AnthropicContentBlock =
|
||||||
|
| AnthropicTextBlock
|
||||||
|
| AnthropicImageBlock
|
||||||
|
| AnthropicToolUseBlock
|
||||||
|
| AnthropicToolResultBlock
|
||||||
|
| { type: string; [key: string]: unknown };
|
||||||
|
|
||||||
|
interface AnthropicMessage {
|
||||||
|
role?: 'user' | 'assistant' | string;
|
||||||
|
content?: string | AnthropicContentBlock[];
|
||||||
|
}
|
||||||
|
|
||||||
interface AnthropicProxyRequestShape {
|
interface AnthropicProxyRequestShape {
|
||||||
|
model?: unknown;
|
||||||
|
system?: unknown;
|
||||||
|
messages?: unknown;
|
||||||
max_tokens?: unknown;
|
max_tokens?: unknown;
|
||||||
temperature?: unknown;
|
temperature?: unknown;
|
||||||
top_p?: unknown;
|
top_p?: unknown;
|
||||||
stop_sequences?: unknown;
|
stop_sequences?: unknown;
|
||||||
metadata?: unknown;
|
metadata?: unknown;
|
||||||
tools?: unknown;
|
tools?: unknown;
|
||||||
|
stream?: unknown;
|
||||||
|
thinking?: AnthropicThinking;
|
||||||
}
|
}
|
||||||
|
|
||||||
export interface ProxyOpenAIRequest extends ReturnType<typeof translateAnthropicRequest> {
|
interface OpenAITextPart {
|
||||||
max_tokens?: number;
|
type: 'text';
|
||||||
temperature?: number;
|
text: string;
|
||||||
top_p?: number;
|
}
|
||||||
stop?: string[];
|
|
||||||
metadata?: Record<string, unknown>;
|
interface OpenAIImagePart {
|
||||||
|
type: 'image_url';
|
||||||
|
image_url: {
|
||||||
|
url: string;
|
||||||
|
};
|
||||||
|
}
|
||||||
|
|
||||||
|
type OpenAIContentPart = OpenAITextPart | OpenAIImagePart;
|
||||||
|
|
||||||
|
interface OpenAIMessage {
|
||||||
|
role: 'system' | 'user' | 'assistant' | 'tool';
|
||||||
|
content: string | OpenAIContentPart[] | null;
|
||||||
|
tool_call_id?: string;
|
||||||
|
tool_calls?: Array<{
|
||||||
|
id: string;
|
||||||
|
type: 'function';
|
||||||
|
function: {
|
||||||
|
name: string;
|
||||||
|
arguments: string;
|
||||||
|
};
|
||||||
|
}>;
|
||||||
|
}
|
||||||
|
|
||||||
|
export interface ProxyOpenAIRequest {
|
||||||
|
model?: string;
|
||||||
|
stream: boolean;
|
||||||
|
reasoning_effort?: string;
|
||||||
|
reasoning?: {
|
||||||
|
enabled: boolean;
|
||||||
|
effort: string;
|
||||||
|
};
|
||||||
tools?: Array<{
|
tools?: Array<{
|
||||||
type: 'function';
|
type: 'function';
|
||||||
function: {
|
function: {
|
||||||
@@ -23,6 +100,23 @@ export interface ProxyOpenAIRequest extends ReturnType<typeof translateAnthropic
|
|||||||
parameters: Record<string, unknown>;
|
parameters: Record<string, unknown>;
|
||||||
};
|
};
|
||||||
}>;
|
}>;
|
||||||
|
messages: OpenAIMessage[];
|
||||||
|
max_tokens?: number;
|
||||||
|
temperature?: number;
|
||||||
|
top_p?: number;
|
||||||
|
stop?: string[];
|
||||||
|
metadata?: Record<string, unknown>;
|
||||||
|
}
|
||||||
|
|
||||||
|
const TOOL_RESULT_SERIALIZATION_FALLBACK = '[unserializable content]';
|
||||||
|
const TOOL_USE_ARGUMENTS_FALLBACK = '{}';
|
||||||
|
|
||||||
|
function assertObject(value: unknown, label: string): Record<string, unknown> {
|
||||||
|
if (typeof value !== 'object' || value === null) {
|
||||||
|
throw new Error(`${label} must be an object`);
|
||||||
|
}
|
||||||
|
|
||||||
|
return value as Record<string, unknown>;
|
||||||
}
|
}
|
||||||
|
|
||||||
function asNumber(value: unknown): number | undefined {
|
function asNumber(value: unknown): number | undefined {
|
||||||
@@ -46,6 +140,90 @@ function asMetadata(value: unknown): Record<string, unknown> | undefined {
|
|||||||
: undefined;
|
: undefined;
|
||||||
}
|
}
|
||||||
|
|
||||||
|
function safeJsonStringify(value: unknown, fallback: string): string {
|
||||||
|
try {
|
||||||
|
const serialized = JSON.stringify(value);
|
||||||
|
return typeof serialized === 'string' ? serialized : fallback;
|
||||||
|
} catch {
|
||||||
|
return fallback;
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
function flattenTextContent(content: unknown, label: string): string {
|
||||||
|
if (typeof content === 'string') {
|
||||||
|
return content;
|
||||||
|
}
|
||||||
|
if (!Array.isArray(content)) {
|
||||||
|
throw new Error(`${label} must be a string or content block array`);
|
||||||
|
}
|
||||||
|
|
||||||
|
return content
|
||||||
|
.map((block, index) => {
|
||||||
|
const parsed = assertObject(block, `${label}[${index}]`);
|
||||||
|
if (parsed.type !== 'text') {
|
||||||
|
throw new Error(`${label}[${index}].type "${String(parsed.type)}" is not supported`);
|
||||||
|
}
|
||||||
|
return typeof parsed.text === 'string' ? parsed.text : '';
|
||||||
|
})
|
||||||
|
.join('\n');
|
||||||
|
}
|
||||||
|
|
||||||
|
function toToolResultContent(content: unknown, label: string): string {
|
||||||
|
if (content === undefined) {
|
||||||
|
return '';
|
||||||
|
}
|
||||||
|
if (typeof content === 'string') {
|
||||||
|
return content;
|
||||||
|
}
|
||||||
|
if (Array.isArray(content)) {
|
||||||
|
return flattenTextContent(content, label);
|
||||||
|
}
|
||||||
|
return safeJsonStringify(content, TOOL_RESULT_SERIALIZATION_FALLBACK);
|
||||||
|
}
|
||||||
|
|
||||||
|
function createFallbackToolId(messageIndex: number, blockIndex: number): string {
|
||||||
|
return `toolu_proxy_fallback_${messageIndex}_${blockIndex}`;
|
||||||
|
}
|
||||||
|
|
||||||
|
function toImagePart(block: AnthropicImageBlock, label: string): OpenAIImagePart {
|
||||||
|
const source = block.source;
|
||||||
|
if (!source || source.type !== 'base64' || !source.media_type || !source.data) {
|
||||||
|
throw new Error(`${label}.source must be a base64 image payload`);
|
||||||
|
}
|
||||||
|
|
||||||
|
return {
|
||||||
|
type: 'image_url',
|
||||||
|
image_url: {
|
||||||
|
url: `data:${source.media_type};base64,${source.data}`,
|
||||||
|
},
|
||||||
|
};
|
||||||
|
}
|
||||||
|
|
||||||
|
function isImageBlock(block: AnthropicContentBlock): block is AnthropicImageBlock {
|
||||||
|
return block.type === 'image';
|
||||||
|
}
|
||||||
|
|
||||||
|
function isToolUseBlock(block: AnthropicContentBlock): block is AnthropicToolUseBlock {
|
||||||
|
return block.type === 'tool_use';
|
||||||
|
}
|
||||||
|
|
||||||
|
function isToolResultBlock(block: AnthropicContentBlock): block is AnthropicToolResultBlock {
|
||||||
|
return block.type === 'tool_result';
|
||||||
|
}
|
||||||
|
|
||||||
|
function flushUserContent(messages: OpenAIMessage[], parts: OpenAIContentPart[]): void {
|
||||||
|
if (parts.length === 0) {
|
||||||
|
return;
|
||||||
|
}
|
||||||
|
|
||||||
|
const onlyText = parts.every((part) => part.type === 'text');
|
||||||
|
messages.push({
|
||||||
|
role: 'user',
|
||||||
|
content: onlyText ? parts.map((part) => (part as OpenAITextPart).text).join('\n') : [...parts],
|
||||||
|
});
|
||||||
|
parts.length = 0;
|
||||||
|
}
|
||||||
|
|
||||||
function transformTools(value: unknown): ProxyOpenAIRequest['tools'] {
|
function transformTools(value: unknown): ProxyOpenAIRequest['tools'] {
|
||||||
if (!Array.isArray(value)) {
|
if (!Array.isArray(value)) {
|
||||||
return undefined;
|
return undefined;
|
||||||
@@ -71,19 +249,179 @@ function transformTools(value: unknown): ProxyOpenAIRequest['tools'] {
|
|||||||
return tools.length > 0 ? tools : undefined;
|
return tools.length > 0 ? tools : undefined;
|
||||||
}
|
}
|
||||||
|
|
||||||
|
function mapThinkingToReasoning(
|
||||||
|
thinking: AnthropicThinking | undefined
|
||||||
|
): Pick<ProxyOpenAIRequest, 'reasoning' | 'reasoning_effort'> {
|
||||||
|
if (!thinking || thinking.type === 'disabled') {
|
||||||
|
return {};
|
||||||
|
}
|
||||||
|
|
||||||
|
if (thinking.type !== 'enabled') {
|
||||||
|
throw new Error('thinking.type must be "enabled" or "disabled"');
|
||||||
|
}
|
||||||
|
|
||||||
|
const effort =
|
||||||
|
typeof thinking.budget_tokens === 'number' && thinking.budget_tokens >= 8192
|
||||||
|
? 'high'
|
||||||
|
: 'medium';
|
||||||
|
|
||||||
|
return {
|
||||||
|
reasoning_effort: effort,
|
||||||
|
reasoning: {
|
||||||
|
enabled: true,
|
||||||
|
effort,
|
||||||
|
},
|
||||||
|
};
|
||||||
|
}
|
||||||
|
|
||||||
|
function transformMessages(messagesValue: unknown): OpenAIMessage[] {
|
||||||
|
if (!Array.isArray(messagesValue)) {
|
||||||
|
throw new Error('messages must be an array');
|
||||||
|
}
|
||||||
|
|
||||||
|
const translatedMessages: OpenAIMessage[] = [];
|
||||||
|
|
||||||
|
messagesValue.forEach((message, messageIndex) => {
|
||||||
|
const parsedMessage = assertObject(message, `messages[${messageIndex}]`) as AnthropicMessage;
|
||||||
|
const role = parsedMessage.role;
|
||||||
|
if (role !== 'user' && role !== 'assistant') {
|
||||||
|
throw new Error(`messages[${messageIndex}].role must be "user" or "assistant"`);
|
||||||
|
}
|
||||||
|
|
||||||
|
const content = parsedMessage.content;
|
||||||
|
if (typeof content === 'string') {
|
||||||
|
translatedMessages.push({ role, content });
|
||||||
|
return;
|
||||||
|
}
|
||||||
|
|
||||||
|
if (!Array.isArray(content)) {
|
||||||
|
throw new Error(`messages[${messageIndex}].content must be a string or array`);
|
||||||
|
}
|
||||||
|
|
||||||
|
const userParts: OpenAIContentPart[] = [];
|
||||||
|
const assistantTextParts: string[] = [];
|
||||||
|
const toolCalls: NonNullable<OpenAIMessage['tool_calls']> = [];
|
||||||
|
let sawToolResult = false;
|
||||||
|
|
||||||
|
content.forEach((block, blockIndex) => {
|
||||||
|
const parsed = assertObject(
|
||||||
|
block,
|
||||||
|
`messages[${messageIndex}].content[${blockIndex}]`
|
||||||
|
) as AnthropicContentBlock;
|
||||||
|
|
||||||
|
if (parsed.type === 'text') {
|
||||||
|
const text = typeof parsed.text === 'string' ? parsed.text : '';
|
||||||
|
if (role === 'user') {
|
||||||
|
userParts.push({ type: 'text', text });
|
||||||
|
} else {
|
||||||
|
assistantTextParts.push(text);
|
||||||
|
}
|
||||||
|
return;
|
||||||
|
}
|
||||||
|
|
||||||
|
if (isImageBlock(parsed)) {
|
||||||
|
if (role !== 'user') {
|
||||||
|
throw new Error(
|
||||||
|
`messages[${messageIndex}].content[${blockIndex}] image requires user role`
|
||||||
|
);
|
||||||
|
}
|
||||||
|
userParts.push(toImagePart(parsed, `messages[${messageIndex}].content[${blockIndex}]`));
|
||||||
|
return;
|
||||||
|
}
|
||||||
|
|
||||||
|
if (isToolUseBlock(parsed)) {
|
||||||
|
if (role !== 'assistant') {
|
||||||
|
throw new Error(
|
||||||
|
`messages[${messageIndex}].content[${blockIndex}] tool_use requires assistant role`
|
||||||
|
);
|
||||||
|
}
|
||||||
|
toolCalls.push({
|
||||||
|
id:
|
||||||
|
typeof parsed.id === 'string' && parsed.id.length > 0
|
||||||
|
? parsed.id
|
||||||
|
: createFallbackToolId(messageIndex, blockIndex),
|
||||||
|
type: 'function',
|
||||||
|
function: {
|
||||||
|
name: typeof parsed.name === 'string' ? parsed.name : 'tool',
|
||||||
|
arguments: safeJsonStringify(parsed.input ?? {}, TOOL_USE_ARGUMENTS_FALLBACK),
|
||||||
|
},
|
||||||
|
});
|
||||||
|
return;
|
||||||
|
}
|
||||||
|
|
||||||
|
if (isToolResultBlock(parsed)) {
|
||||||
|
if (role !== 'user') {
|
||||||
|
throw new Error(
|
||||||
|
`messages[${messageIndex}].content[${blockIndex}] tool_result requires user role`
|
||||||
|
);
|
||||||
|
}
|
||||||
|
if (typeof parsed.tool_use_id !== 'string' || parsed.tool_use_id.trim().length === 0) {
|
||||||
|
throw new Error(
|
||||||
|
`messages[${messageIndex}].content[${blockIndex}].tool_use_id must be a non-empty string`
|
||||||
|
);
|
||||||
|
}
|
||||||
|
sawToolResult = true;
|
||||||
|
flushUserContent(translatedMessages, userParts);
|
||||||
|
translatedMessages.push({
|
||||||
|
role: 'tool',
|
||||||
|
tool_call_id: parsed.tool_use_id,
|
||||||
|
content: toToolResultContent(
|
||||||
|
parsed.content,
|
||||||
|
`messages[${messageIndex}].content[${blockIndex}].content`
|
||||||
|
),
|
||||||
|
});
|
||||||
|
return;
|
||||||
|
}
|
||||||
|
|
||||||
|
throw new Error(
|
||||||
|
`messages[${messageIndex}].content[${blockIndex}].type "${String(parsed.type)}" is not supported`
|
||||||
|
);
|
||||||
|
});
|
||||||
|
|
||||||
|
if (role === 'assistant') {
|
||||||
|
translatedMessages.push({
|
||||||
|
role: 'assistant',
|
||||||
|
content: assistantTextParts.join('\n'),
|
||||||
|
tool_calls: toolCalls.length > 0 ? toolCalls : undefined,
|
||||||
|
});
|
||||||
|
return;
|
||||||
|
}
|
||||||
|
|
||||||
|
if (userParts.length > 0 || !sawToolResult) {
|
||||||
|
flushUserContent(translatedMessages, userParts);
|
||||||
|
}
|
||||||
|
});
|
||||||
|
|
||||||
|
return translatedMessages;
|
||||||
|
}
|
||||||
|
|
||||||
export class ProxyRequestTransformer {
|
export class ProxyRequestTransformer {
|
||||||
transform(raw: unknown): ProxyOpenAIRequest {
|
transform(raw: unknown): ProxyOpenAIRequest {
|
||||||
const translated = translateAnthropicRequest(raw);
|
const source = assertObject(raw || {}, 'request') as AnthropicProxyRequestShape;
|
||||||
const source = (raw || {}) as AnthropicProxyRequestShape;
|
const messages = transformMessages(source.messages);
|
||||||
|
const system = source.system;
|
||||||
|
const allMessages =
|
||||||
|
system !== undefined
|
||||||
|
? [
|
||||||
|
{ role: 'system', content: flattenTextContent(system, 'system') } as OpenAIMessage,
|
||||||
|
...messages,
|
||||||
|
]
|
||||||
|
: messages;
|
||||||
|
|
||||||
return {
|
return {
|
||||||
...translated,
|
model:
|
||||||
|
typeof source.model === 'string' && source.model.trim().length > 0
|
||||||
|
? source.model.trim()
|
||||||
|
: undefined,
|
||||||
|
stream: source.stream === true,
|
||||||
|
messages: allMessages,
|
||||||
max_tokens: asNumber(source.max_tokens),
|
max_tokens: asNumber(source.max_tokens),
|
||||||
temperature: asNumber(source.temperature),
|
temperature: asNumber(source.temperature),
|
||||||
top_p: asNumber(source.top_p),
|
top_p: asNumber(source.top_p),
|
||||||
stop: asStringArray(source.stop_sequences),
|
stop: asStringArray(source.stop_sequences),
|
||||||
metadata: asMetadata(source.metadata),
|
metadata: asMetadata(source.metadata),
|
||||||
tools: transformTools(source.tools),
|
tools: transformTools(source.tools),
|
||||||
|
...mapThinkingToReasoning(source.thinking),
|
||||||
};
|
};
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
@@ -40,6 +40,19 @@ export interface CLIProxyVariantsConfig {
|
|||||||
[profileName: string]: CLIProxyVariantConfig;
|
[profileName: string]: CLIProxyVariantConfig;
|
||||||
}
|
}
|
||||||
|
|
||||||
|
export interface OpenAICompatProxyRoutingConfig {
|
||||||
|
default?: string;
|
||||||
|
background?: string;
|
||||||
|
think?: string;
|
||||||
|
longContext?: string;
|
||||||
|
webSearch?: string;
|
||||||
|
longContextThreshold?: number;
|
||||||
|
}
|
||||||
|
|
||||||
|
export interface OpenAICompatProxyConfig {
|
||||||
|
routing?: OpenAICompatProxyRoutingConfig;
|
||||||
|
}
|
||||||
|
|
||||||
/**
|
/**
|
||||||
* Main CCS configuration
|
* Main CCS configuration
|
||||||
* Located at: ~/.ccs/config.json
|
* Located at: ~/.ccs/config.json
|
||||||
@@ -51,6 +64,8 @@ export interface Config {
|
|||||||
profile_targets?: Record<string, TargetType>;
|
profile_targets?: Record<string, TargetType>;
|
||||||
/** User-defined CLIProxy profile variants (optional) */
|
/** User-defined CLIProxy profile variants (optional) */
|
||||||
cliproxy?: CLIProxyVariantsConfig;
|
cliproxy?: CLIProxyVariantsConfig;
|
||||||
|
/** OpenAI-compatible local proxy configuration (optional) */
|
||||||
|
proxy?: OpenAICompatProxyConfig;
|
||||||
/** Legacy continuity inheritance mapping (profile -> source account) */
|
/** Legacy continuity inheritance mapping (profile -> source account) */
|
||||||
continuity_inherit_from_account?: Record<string, string>;
|
continuity_inherit_from_account?: Record<string, string>;
|
||||||
}
|
}
|
||||||
|
|||||||
@@ -14,6 +14,8 @@ export type {
|
|||||||
ProfilesRegistry,
|
ProfilesRegistry,
|
||||||
CLIProxyVariantConfig,
|
CLIProxyVariantConfig,
|
||||||
CLIProxyVariantsConfig,
|
CLIProxyVariantsConfig,
|
||||||
|
OpenAICompatProxyConfig,
|
||||||
|
OpenAICompatProxyRoutingConfig,
|
||||||
} from './config';
|
} from './config';
|
||||||
export { isConfig, isSettings } from './config';
|
export { isConfig, isSettings } from './config';
|
||||||
|
|
||||||
|
|||||||
@@ -673,6 +673,10 @@ router.put('/', (req: Request, res: Response): void => {
|
|||||||
currentConfig.cliproxy = mergeCliproxyConfig(currentConfig, config);
|
currentConfig.cliproxy = mergeCliproxyConfig(currentConfig, config);
|
||||||
}
|
}
|
||||||
|
|
||||||
|
if (config.proxy !== undefined) {
|
||||||
|
currentConfig.proxy = config.proxy;
|
||||||
|
}
|
||||||
|
|
||||||
if (config.preferences !== undefined) {
|
if (config.preferences !== undefined) {
|
||||||
currentConfig.preferences = config.preferences;
|
currentConfig.preferences = config.preferences;
|
||||||
}
|
}
|
||||||
|
|||||||
@@ -68,20 +68,42 @@ describe('proxy command e2e', () => {
|
|||||||
'utf8'
|
'utf8'
|
||||||
);
|
);
|
||||||
|
|
||||||
const started = runCli(['proxy', 'start', 'hf', '--port', String(port)]);
|
const started = runCli(['proxy', 'start', 'hf', '--port', String(port), '--host', '127.0.0.1']);
|
||||||
expect(started.status).toBe(0);
|
expect(started.status).toBe(0);
|
||||||
|
|
||||||
const status = runCli(['proxy', 'status']);
|
const status = runCli(['proxy', 'status']);
|
||||||
expect(status.stdout).toContain(`Proxy running on port ${port}`);
|
expect(status.stdout).toContain(`Proxy running on port ${port}`);
|
||||||
|
expect(status.stdout).toContain('Host: 127.0.0.1');
|
||||||
expect(status.stdout).toContain('Profile: hf');
|
expect(status.stdout).toContain('Profile: hf');
|
||||||
|
|
||||||
const activate = runCli(['proxy', 'activate', '--shell', 'bash']);
|
const activate = runCli(['proxy', 'activate', '--shell', 'bash']);
|
||||||
expect(activate.stdout).toContain(`export ANTHROPIC_BASE_URL='http://127.0.0.1:${port}'`);
|
expect(activate.stdout).toContain(`export ANTHROPIC_BASE_URL='http://127.0.0.1:${port}'`);
|
||||||
expect(activate.stdout).toMatch(/export ANTHROPIC_AUTH_TOKEN='[a-f0-9]{48}'/);
|
expect(activate.stdout).toMatch(/export ANTHROPIC_AUTH_TOKEN='[a-f0-9]{48}'/);
|
||||||
|
expect(activate.stdout).toContain("export DISABLE_TELEMETRY='1'");
|
||||||
|
expect(activate.stdout).toContain("export DISABLE_COST_WARNINGS='1'");
|
||||||
|
expect(activate.stdout).toContain("export API_TIMEOUT_MS='600000'");
|
||||||
|
expect(activate.stdout).toContain("export NO_PROXY='127.0.0.1,localhost'");
|
||||||
|
|
||||||
|
const activateFish = runCli(['proxy', 'activate', '--fish']);
|
||||||
|
expect(activateFish.stdout).toContain(`set -gx ANTHROPIC_BASE_URL 'http://127.0.0.1:${port}'`);
|
||||||
|
|
||||||
const health = await fetch(`http://127.0.0.1:${port}/health`);
|
const health = await fetch(`http://127.0.0.1:${port}/health`);
|
||||||
expect(health.status).toBe(200);
|
expect(health.status).toBe(200);
|
||||||
|
|
||||||
|
const info = await fetch(`http://127.0.0.1:${port}/`);
|
||||||
|
expect(info.status).toBe(200);
|
||||||
|
await expect(info.json()).resolves.toMatchObject({
|
||||||
|
ok: true,
|
||||||
|
service: 'ccs-openai-compat-proxy',
|
||||||
|
bind: {
|
||||||
|
host: '127.0.0.1',
|
||||||
|
port,
|
||||||
|
},
|
||||||
|
profile: {
|
||||||
|
name: 'hf',
|
||||||
|
},
|
||||||
|
});
|
||||||
|
|
||||||
const stopped = runCli(['proxy', 'stop']);
|
const stopped = runCli(['proxy', 'stop']);
|
||||||
expect(stopped.status).toBe(0);
|
expect(stopped.status).toBe(0);
|
||||||
}, 35000);
|
}, 35000);
|
||||||
|
|||||||
@@ -0,0 +1,259 @@
|
|||||||
|
import { afterEach, beforeEach, describe, expect, it } from 'bun:test';
|
||||||
|
import getPort from 'get-port';
|
||||||
|
import * as fs from 'fs';
|
||||||
|
import * as http from 'http';
|
||||||
|
import * as os from 'os';
|
||||||
|
import * as path from 'path';
|
||||||
|
import { startOpenAICompatProxyServer } from '../../../src/proxy/server/proxy-server';
|
||||||
|
import type { OpenAICompatProfileConfig } from '../../../src/proxy/profile-router';
|
||||||
|
|
||||||
|
let proxyServer: http.Server;
|
||||||
|
let upstreamServer: http.Server;
|
||||||
|
let upstreamSockets = new Set<import('net').Socket>();
|
||||||
|
let upstreamPort: number;
|
||||||
|
let proxyPort: number;
|
||||||
|
let tempDir: string;
|
||||||
|
let originalTimeoutEnv: string | undefined;
|
||||||
|
let originalCcsHome: string | undefined;
|
||||||
|
|
||||||
|
async function startUpstream(
|
||||||
|
handler: (req: http.IncomingMessage, res: http.ServerResponse) => Promise<void> | void
|
||||||
|
): Promise<void> {
|
||||||
|
upstreamServer = http.createServer((req, res) => {
|
||||||
|
void Promise.resolve(handler(req, res));
|
||||||
|
});
|
||||||
|
upstreamServer.on('connection', (socket) => {
|
||||||
|
upstreamSockets.add(socket);
|
||||||
|
socket.on('close', () => {
|
||||||
|
upstreamSockets.delete(socket);
|
||||||
|
});
|
||||||
|
});
|
||||||
|
await new Promise<void>((resolve) =>
|
||||||
|
upstreamServer.listen(upstreamPort, '127.0.0.1', () => resolve())
|
||||||
|
);
|
||||||
|
}
|
||||||
|
|
||||||
|
async function requestProxy(payload: unknown, signal?: AbortSignal): Promise<Response> {
|
||||||
|
return fetch(`http://127.0.0.1:${proxyPort}/v1/messages`, {
|
||||||
|
method: 'POST',
|
||||||
|
headers: {
|
||||||
|
'Content-Type': 'application/json',
|
||||||
|
'x-api-key': 'test-proxy-token',
|
||||||
|
},
|
||||||
|
body: JSON.stringify(payload),
|
||||||
|
signal,
|
||||||
|
});
|
||||||
|
}
|
||||||
|
|
||||||
|
beforeEach(async () => {
|
||||||
|
upstreamPort = await getPort();
|
||||||
|
proxyPort = await getPort();
|
||||||
|
tempDir = fs.mkdtempSync(path.join(os.tmpdir(), 'ccs-proxy-edge-'));
|
||||||
|
originalTimeoutEnv = process.env.CCS_OPENAI_PROXY_REQUEST_TIMEOUT_MS;
|
||||||
|
originalCcsHome = process.env.CCS_HOME;
|
||||||
|
process.env.CCS_HOME = tempDir;
|
||||||
|
});
|
||||||
|
|
||||||
|
afterEach(async () => {
|
||||||
|
if (originalTimeoutEnv !== undefined) {
|
||||||
|
process.env.CCS_OPENAI_PROXY_REQUEST_TIMEOUT_MS = originalTimeoutEnv;
|
||||||
|
} else {
|
||||||
|
delete process.env.CCS_OPENAI_PROXY_REQUEST_TIMEOUT_MS;
|
||||||
|
}
|
||||||
|
if (originalCcsHome !== undefined) {
|
||||||
|
process.env.CCS_HOME = originalCcsHome;
|
||||||
|
} else {
|
||||||
|
delete process.env.CCS_HOME;
|
||||||
|
}
|
||||||
|
|
||||||
|
if (proxyServer) {
|
||||||
|
await new Promise<void>((resolve) => proxyServer.close(() => resolve()));
|
||||||
|
}
|
||||||
|
if (upstreamServer) {
|
||||||
|
for (const socket of upstreamSockets) {
|
||||||
|
socket.destroy();
|
||||||
|
}
|
||||||
|
upstreamSockets = new Set();
|
||||||
|
await new Promise<void>((resolve) => upstreamServer.close(() => resolve()));
|
||||||
|
}
|
||||||
|
fs.rmSync(tempDir, { recursive: true, force: true });
|
||||||
|
});
|
||||||
|
|
||||||
|
describe('openai proxy message edge cases', () => {
|
||||||
|
async function startProxyWithHandler(
|
||||||
|
handler: (req: http.IncomingMessage, res: http.ServerResponse) => Promise<void> | void
|
||||||
|
) {
|
||||||
|
await startUpstream(handler);
|
||||||
|
const profile: OpenAICompatProfileConfig = {
|
||||||
|
profileName: 'hf',
|
||||||
|
settingsPath: '/tmp/hf.settings.json',
|
||||||
|
baseUrl: `http://127.0.0.1:${upstreamPort}`,
|
||||||
|
apiKey: 'hf_token',
|
||||||
|
provider: 'generic-chat-completion-api',
|
||||||
|
model: 'hf-model',
|
||||||
|
};
|
||||||
|
proxyServer = startOpenAICompatProxyServer({
|
||||||
|
profile,
|
||||||
|
port: proxyPort,
|
||||||
|
authToken: 'test-proxy-token',
|
||||||
|
});
|
||||||
|
}
|
||||||
|
|
||||||
|
it('preserves rate-limit errors from the upstream provider', async () => {
|
||||||
|
await startProxyWithHandler((_req, res) => {
|
||||||
|
res.writeHead(429, {
|
||||||
|
'Content-Type': 'application/json',
|
||||||
|
'Retry-After': '9',
|
||||||
|
});
|
||||||
|
res.end(JSON.stringify({ error: { message: 'rate limited' } }));
|
||||||
|
});
|
||||||
|
|
||||||
|
const response = await requestProxy({
|
||||||
|
model: 'hf-model',
|
||||||
|
messages: [{ role: 'user', content: 'hello' }],
|
||||||
|
});
|
||||||
|
|
||||||
|
expect(response.status).toBe(429);
|
||||||
|
expect(response.headers.get('retry-after')).toBe('9');
|
||||||
|
await expect(response.json()).resolves.toMatchObject({
|
||||||
|
type: 'error',
|
||||||
|
error: {
|
||||||
|
type: 'rate_limit_error',
|
||||||
|
message: 'rate limited',
|
||||||
|
},
|
||||||
|
});
|
||||||
|
});
|
||||||
|
|
||||||
|
it('returns api_error when the upstream JSON response has no usable choices', async () => {
|
||||||
|
await startProxyWithHandler((_req, res) => {
|
||||||
|
res.writeHead(200, { 'Content-Type': 'application/json' });
|
||||||
|
res.end(JSON.stringify({ id: 'chatcmpl_empty', choices: [] }));
|
||||||
|
});
|
||||||
|
|
||||||
|
const response = await requestProxy({
|
||||||
|
model: 'hf-model',
|
||||||
|
messages: [{ role: 'user', content: 'hello' }],
|
||||||
|
});
|
||||||
|
|
||||||
|
expect(response.status).toBe(502);
|
||||||
|
await expect(response.json()).resolves.toMatchObject({
|
||||||
|
type: 'error',
|
||||||
|
error: {
|
||||||
|
type: 'api_error',
|
||||||
|
message: 'Failed to translate Cursor JSON response',
|
||||||
|
},
|
||||||
|
});
|
||||||
|
});
|
||||||
|
|
||||||
|
it('streams thinking deltas and chunked tool-call arguments back as Anthropic SSE', async () => {
|
||||||
|
await startProxyWithHandler((_req, res) => {
|
||||||
|
res.writeHead(200, { 'Content-Type': 'text/event-stream' });
|
||||||
|
res.write(
|
||||||
|
'data: {"id":"chatcmpl_1","model":"hf-model","choices":[{"index":0,"delta":{"role":"assistant"}}]}\n\n'
|
||||||
|
);
|
||||||
|
res.write(
|
||||||
|
'data: {"id":"chatcmpl_1","model":"hf-model","choices":[{"index":0,"delta":{"reasoning_content":"Need to search first."}}]}\n\n'
|
||||||
|
);
|
||||||
|
res.write(
|
||||||
|
'data: {"id":"chatcmpl_1","model":"hf-model","choices":[{"index":0,"delta":{"tool_calls":[{"index":0,"id":"call_1","type":"function","function":{"name":"search","arguments":"{\\"q\\":\\""}}]}}]}\n\n'
|
||||||
|
);
|
||||||
|
res.write(
|
||||||
|
'data: {"id":"chatcmpl_1","model":"hf-model","choices":[{"index":0,"delta":{"tool_calls":[{"index":0,"function":{"arguments":"docs\\"}"}}]}}]}\n\n'
|
||||||
|
);
|
||||||
|
res.write(
|
||||||
|
'data: {"id":"chatcmpl_1","model":"hf-model","choices":[{"index":0,"delta":{},"finish_reason":"tool_calls"}],"usage":{"prompt_tokens":8,"completion_tokens":6}}\n\n'
|
||||||
|
);
|
||||||
|
res.end('data: [DONE]\n\n');
|
||||||
|
});
|
||||||
|
|
||||||
|
const response = await requestProxy({
|
||||||
|
model: 'hf-model',
|
||||||
|
stream: true,
|
||||||
|
messages: [{ role: 'user', content: 'search docs' }],
|
||||||
|
});
|
||||||
|
|
||||||
|
const body = await response.text();
|
||||||
|
expect(response.status).toBe(200);
|
||||||
|
expect(body).toContain('"type":"thinking_delta"');
|
||||||
|
expect(body).toContain('"type":"tool_use"');
|
||||||
|
expect(body).toContain('"partial_json":"{\\"q\\":\\""');
|
||||||
|
expect(body).toContain('"partial_json":"docs\\"}"');
|
||||||
|
expect(body).toContain('event: message_stop');
|
||||||
|
});
|
||||||
|
|
||||||
|
it('returns a timeout error when the upstream does not respond in time', async () => {
|
||||||
|
process.env.CCS_OPENAI_PROXY_REQUEST_TIMEOUT_MS = '50';
|
||||||
|
await startProxyWithHandler(async (req, _res) => {
|
||||||
|
await new Promise<void>((resolve) => req.on('close', () => resolve()));
|
||||||
|
});
|
||||||
|
|
||||||
|
const response = await requestProxy({
|
||||||
|
model: 'hf-model',
|
||||||
|
messages: [{ role: 'user', content: 'hello' }],
|
||||||
|
});
|
||||||
|
|
||||||
|
expect(response.status).toBe(502);
|
||||||
|
await expect(response.json()).resolves.toMatchObject({
|
||||||
|
type: 'error',
|
||||||
|
error: {
|
||||||
|
type: 'api_error',
|
||||||
|
message: 'The upstream provider did not respond within 50ms',
|
||||||
|
},
|
||||||
|
});
|
||||||
|
});
|
||||||
|
|
||||||
|
it('aborts the upstream request when the client disconnects mid-flight', async () => {
|
||||||
|
await startProxyWithHandler(() => {});
|
||||||
|
|
||||||
|
await new Promise<void>((resolve) => {
|
||||||
|
const request = http.request(
|
||||||
|
{
|
||||||
|
hostname: '127.0.0.1',
|
||||||
|
port: proxyPort,
|
||||||
|
path: '/v1/messages',
|
||||||
|
method: 'POST',
|
||||||
|
headers: {
|
||||||
|
'Content-Type': 'application/json',
|
||||||
|
'x-api-key': 'test-proxy-token',
|
||||||
|
},
|
||||||
|
},
|
||||||
|
() => resolve()
|
||||||
|
);
|
||||||
|
|
||||||
|
request.write(
|
||||||
|
JSON.stringify({
|
||||||
|
model: 'hf-model',
|
||||||
|
messages: [{ role: 'user', content: 'hello' }],
|
||||||
|
})
|
||||||
|
);
|
||||||
|
request.end();
|
||||||
|
|
||||||
|
setTimeout(() => {
|
||||||
|
request.destroy(new Error('client aborted'));
|
||||||
|
resolve();
|
||||||
|
}, 50);
|
||||||
|
});
|
||||||
|
|
||||||
|
const logPath = path.join(tempDir, '.ccs', 'logs', 'current.jsonl');
|
||||||
|
await Promise.race([
|
||||||
|
new Promise<void>((resolve, reject) => {
|
||||||
|
const startedAt = Date.now();
|
||||||
|
const timer = setInterval(() => {
|
||||||
|
if (fs.existsSync(logPath)) {
|
||||||
|
const content = fs.readFileSync(logPath, 'utf8');
|
||||||
|
if (content.includes('"event":"request.disconnect"')) {
|
||||||
|
clearInterval(timer);
|
||||||
|
resolve();
|
||||||
|
return;
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
if (Date.now() - startedAt > 1500) {
|
||||||
|
clearInterval(timer);
|
||||||
|
reject(new Error('proxy did not log disconnect cleanup'));
|
||||||
|
}
|
||||||
|
}, 50);
|
||||||
|
}),
|
||||||
|
]);
|
||||||
|
});
|
||||||
|
});
|
||||||
@@ -173,16 +173,44 @@ describe('openai proxy messages endpoint', () => {
|
|||||||
expect(body.error?.type).toBe('authentication_error');
|
expect(body.error?.type).toBe('authentication_error');
|
||||||
});
|
});
|
||||||
|
|
||||||
it('returns invalid_request_error for unsupported Anthropic content blocks', async () => {
|
it('translates supported Anthropic image blocks into OpenAI image_url content', async () => {
|
||||||
const response = await requestProxy({
|
const response = await requestProxy({
|
||||||
model: 'hf-model',
|
model: 'hf-model',
|
||||||
messages: [{ role: 'user', content: [{ type: 'image' }] }],
|
messages: [
|
||||||
|
{
|
||||||
|
role: 'user',
|
||||||
|
content: [
|
||||||
|
{ type: 'text', text: 'Describe this' },
|
||||||
|
{
|
||||||
|
type: 'image',
|
||||||
|
source: {
|
||||||
|
type: 'base64',
|
||||||
|
media_type: 'image/png',
|
||||||
|
data: 'ZmFrZQ==',
|
||||||
|
},
|
||||||
|
},
|
||||||
|
],
|
||||||
|
},
|
||||||
|
],
|
||||||
});
|
});
|
||||||
const body = (await response.json()) as { error?: { type?: string; message?: string } };
|
|
||||||
|
|
||||||
expect(response.status).toBe(400);
|
expect(response.status).toBe(200);
|
||||||
expect(body.error?.type).toBe('invalid_request_error');
|
const parsedUpstream = upstreamBody as {
|
||||||
expect(body.error?.message).toContain('is not supported');
|
messages?: Array<{
|
||||||
|
role: string;
|
||||||
|
content: Array<{ type: string; text?: string; image_url?: { url?: string } }>;
|
||||||
|
}>;
|
||||||
|
};
|
||||||
|
expect(parsedUpstream.messages?.[0]).toEqual({
|
||||||
|
role: 'user',
|
||||||
|
content: [
|
||||||
|
{ type: 'text', text: 'Describe this' },
|
||||||
|
{
|
||||||
|
type: 'image_url',
|
||||||
|
image_url: { url: 'data:image/png;base64,ZmFrZQ==' },
|
||||||
|
},
|
||||||
|
],
|
||||||
|
});
|
||||||
});
|
});
|
||||||
|
|
||||||
it('accepts query strings and trailing slashes on the messages route', async () => {
|
it('accepts query strings and trailing slashes on the messages route', async () => {
|
||||||
|
|||||||
@@ -0,0 +1,217 @@
|
|||||||
|
import { afterEach, beforeEach, describe, expect, it } from 'bun:test';
|
||||||
|
import getPort from 'get-port';
|
||||||
|
import * as fs from 'fs';
|
||||||
|
import * as http from 'http';
|
||||||
|
import * as os from 'os';
|
||||||
|
import * as path from 'path';
|
||||||
|
import { startOpenAICompatProxyServer } from '../../../src/proxy/server/proxy-server';
|
||||||
|
import type { OpenAICompatProfileConfig } from '../../../src/proxy/profile-router';
|
||||||
|
|
||||||
|
let originalCcsHome: string | undefined;
|
||||||
|
let tempDir: string;
|
||||||
|
let proxyServer: http.Server;
|
||||||
|
let upstreamServers: http.Server[] = [];
|
||||||
|
let proxyPort: number;
|
||||||
|
|
||||||
|
function startMockUpstream(
|
||||||
|
port: number,
|
||||||
|
hitLabel: string,
|
||||||
|
hits: string[],
|
||||||
|
bodies: Array<{ label: string; body: unknown }>
|
||||||
|
): Promise<void> {
|
||||||
|
return new Promise((resolve) => {
|
||||||
|
const server = http.createServer(async (req, res) => {
|
||||||
|
let body = '';
|
||||||
|
for await (const chunk of req) {
|
||||||
|
body += chunk.toString();
|
||||||
|
}
|
||||||
|
hits.push(hitLabel);
|
||||||
|
bodies.push({ label: hitLabel, body: JSON.parse(body) });
|
||||||
|
|
||||||
|
res.writeHead(200, { 'Content-Type': 'application/json' });
|
||||||
|
res.end(
|
||||||
|
JSON.stringify({
|
||||||
|
id: `chatcmpl_${hitLabel}`,
|
||||||
|
model: hitLabel,
|
||||||
|
choices: [
|
||||||
|
{
|
||||||
|
index: 0,
|
||||||
|
message: { role: 'assistant', content: `Reply from ${hitLabel}` },
|
||||||
|
finish_reason: 'stop',
|
||||||
|
},
|
||||||
|
],
|
||||||
|
usage: { prompt_tokens: 2, completion_tokens: 3 },
|
||||||
|
})
|
||||||
|
);
|
||||||
|
});
|
||||||
|
upstreamServers.push(server);
|
||||||
|
server.listen(port, '127.0.0.1', () => resolve());
|
||||||
|
});
|
||||||
|
}
|
||||||
|
|
||||||
|
function writeSettings(profileName: string, env: Record<string, string>): string {
|
||||||
|
const settingsPath = path.join(tempDir, '.ccs', `${profileName}.settings.json`);
|
||||||
|
fs.writeFileSync(settingsPath, JSON.stringify({ env }, null, 2), 'utf8');
|
||||||
|
return settingsPath;
|
||||||
|
}
|
||||||
|
|
||||||
|
async function requestProxy(payload: unknown): Promise<Response> {
|
||||||
|
return fetch(`http://127.0.0.1:${proxyPort}/v1/messages`, {
|
||||||
|
method: 'POST',
|
||||||
|
headers: {
|
||||||
|
'Content-Type': 'application/json',
|
||||||
|
'x-api-key': 'test-proxy-token',
|
||||||
|
},
|
||||||
|
body: JSON.stringify(payload),
|
||||||
|
});
|
||||||
|
}
|
||||||
|
|
||||||
|
beforeEach(async () => {
|
||||||
|
originalCcsHome = process.env.CCS_HOME;
|
||||||
|
tempDir = fs.mkdtempSync(path.join(os.tmpdir(), 'ccs-proxy-routing-'));
|
||||||
|
fs.mkdirSync(path.join(tempDir, '.ccs'), { recursive: true });
|
||||||
|
process.env.CCS_HOME = tempDir;
|
||||||
|
proxyPort = await getPort();
|
||||||
|
});
|
||||||
|
|
||||||
|
afterEach(async () => {
|
||||||
|
await Promise.all(
|
||||||
|
upstreamServers.map(
|
||||||
|
(server) =>
|
||||||
|
new Promise<void>((resolve) => {
|
||||||
|
server.close(() => resolve());
|
||||||
|
})
|
||||||
|
)
|
||||||
|
);
|
||||||
|
upstreamServers = [];
|
||||||
|
if (proxyServer) {
|
||||||
|
await new Promise<void>((resolve) => proxyServer.close(() => resolve()));
|
||||||
|
}
|
||||||
|
if (originalCcsHome !== undefined) {
|
||||||
|
process.env.CCS_HOME = originalCcsHome;
|
||||||
|
} else {
|
||||||
|
delete process.env.CCS_HOME;
|
||||||
|
}
|
||||||
|
fs.rmSync(tempDir, { recursive: true, force: true });
|
||||||
|
});
|
||||||
|
|
||||||
|
describe('openai proxy request routing', () => {
|
||||||
|
it('routes explicit profile:model selectors to the matching upstream profile', async () => {
|
||||||
|
const primaryPort = await getPort();
|
||||||
|
const secondaryPort = await getPort();
|
||||||
|
const hits: string[] = [];
|
||||||
|
const bodies: Array<{ label: string; body: unknown }> = [];
|
||||||
|
await startMockUpstream(primaryPort, 'primary', hits, bodies);
|
||||||
|
await startMockUpstream(secondaryPort, 'secondary', hits, bodies);
|
||||||
|
|
||||||
|
const primarySettings = writeSettings('hf', {
|
||||||
|
ANTHROPIC_BASE_URL: `http://127.0.0.1:${primaryPort}`,
|
||||||
|
ANTHROPIC_AUTH_TOKEN: 'hf_token',
|
||||||
|
ANTHROPIC_MODEL: 'hf-default',
|
||||||
|
CCS_DROID_PROVIDER: 'generic-chat-completion-api',
|
||||||
|
});
|
||||||
|
const secondarySettings = writeSettings('deepseek', {
|
||||||
|
ANTHROPIC_BASE_URL: `http://127.0.0.1:${secondaryPort}`,
|
||||||
|
ANTHROPIC_AUTH_TOKEN: 'deepseek_token',
|
||||||
|
ANTHROPIC_MODEL: 'deepseek-chat',
|
||||||
|
CCS_DROID_PROVIDER: 'generic-chat-completion-api',
|
||||||
|
});
|
||||||
|
|
||||||
|
fs.writeFileSync(
|
||||||
|
path.join(tempDir, '.ccs', 'config.json'),
|
||||||
|
JSON.stringify({ profiles: { hf: primarySettings, deepseek: secondarySettings } }, null, 2),
|
||||||
|
'utf8'
|
||||||
|
);
|
||||||
|
|
||||||
|
const profile: OpenAICompatProfileConfig = {
|
||||||
|
profileName: 'hf',
|
||||||
|
settingsPath: primarySettings,
|
||||||
|
baseUrl: `http://127.0.0.1:${primaryPort}`,
|
||||||
|
apiKey: 'hf_token',
|
||||||
|
provider: 'generic-chat-completion-api',
|
||||||
|
model: 'hf-default',
|
||||||
|
};
|
||||||
|
proxyServer = startOpenAICompatProxyServer({
|
||||||
|
profile,
|
||||||
|
port: proxyPort,
|
||||||
|
authToken: 'test-proxy-token',
|
||||||
|
});
|
||||||
|
|
||||||
|
const response = await requestProxy({
|
||||||
|
model: 'deepseek:deepseek-reasoner',
|
||||||
|
messages: [{ role: 'user', content: 'hello' }],
|
||||||
|
});
|
||||||
|
expect(response.status).toBe(200);
|
||||||
|
expect(await response.json()).toMatchObject({
|
||||||
|
content: [{ type: 'text', text: 'Reply from secondary' }],
|
||||||
|
});
|
||||||
|
expect(hits).toEqual(['secondary']);
|
||||||
|
expect(bodies[0]?.body).toMatchObject({ model: 'deepseek-reasoner' });
|
||||||
|
});
|
||||||
|
|
||||||
|
it('routes thinking requests through the configured think scenario', async () => {
|
||||||
|
const primaryPort = await getPort();
|
||||||
|
const thinkPort = await getPort();
|
||||||
|
const hits: string[] = [];
|
||||||
|
const bodies: Array<{ label: string; body: unknown }> = [];
|
||||||
|
await startMockUpstream(primaryPort, 'primary', hits, bodies);
|
||||||
|
await startMockUpstream(thinkPort, 'thinker', hits, bodies);
|
||||||
|
|
||||||
|
const primarySettings = writeSettings('hf', {
|
||||||
|
ANTHROPIC_BASE_URL: `http://127.0.0.1:${primaryPort}`,
|
||||||
|
ANTHROPIC_AUTH_TOKEN: 'hf_token',
|
||||||
|
ANTHROPIC_MODEL: 'hf-default',
|
||||||
|
CCS_DROID_PROVIDER: 'generic-chat-completion-api',
|
||||||
|
});
|
||||||
|
const thinkSettings = writeSettings('thinker', {
|
||||||
|
ANTHROPIC_BASE_URL: `http://127.0.0.1:${thinkPort}`,
|
||||||
|
ANTHROPIC_AUTH_TOKEN: 'think_token',
|
||||||
|
ANTHROPIC_MODEL: 'deepseek-reasoner',
|
||||||
|
CCS_DROID_PROVIDER: 'generic-chat-completion-api',
|
||||||
|
});
|
||||||
|
|
||||||
|
fs.writeFileSync(
|
||||||
|
path.join(tempDir, '.ccs', 'config.json'),
|
||||||
|
JSON.stringify(
|
||||||
|
{
|
||||||
|
profiles: { hf: primarySettings, thinker: thinkSettings },
|
||||||
|
proxy: {
|
||||||
|
routing: {
|
||||||
|
think: 'thinker:deepseek-reasoner',
|
||||||
|
},
|
||||||
|
},
|
||||||
|
},
|
||||||
|
null,
|
||||||
|
2
|
||||||
|
),
|
||||||
|
'utf8'
|
||||||
|
);
|
||||||
|
|
||||||
|
const profile: OpenAICompatProfileConfig = {
|
||||||
|
profileName: 'hf',
|
||||||
|
settingsPath: primarySettings,
|
||||||
|
baseUrl: `http://127.0.0.1:${primaryPort}`,
|
||||||
|
apiKey: 'hf_token',
|
||||||
|
provider: 'generic-chat-completion-api',
|
||||||
|
model: 'hf-default',
|
||||||
|
};
|
||||||
|
proxyServer = startOpenAICompatProxyServer({
|
||||||
|
profile,
|
||||||
|
port: proxyPort,
|
||||||
|
authToken: 'test-proxy-token',
|
||||||
|
});
|
||||||
|
|
||||||
|
const response = await requestProxy({
|
||||||
|
model: 'hf-default',
|
||||||
|
thinking: { type: 'enabled', budget_tokens: 9000 },
|
||||||
|
messages: [{ role: 'user', content: 'think hard' }],
|
||||||
|
});
|
||||||
|
|
||||||
|
expect(response.status).toBe(200);
|
||||||
|
expect(await response.json()).toMatchObject({
|
||||||
|
content: [{ type: 'text', text: 'Reply from thinker' }],
|
||||||
|
});
|
||||||
|
expect(hits).toEqual(['thinker']);
|
||||||
|
expect(bodies[0]?.body).toMatchObject({ model: 'deepseek-reasoner' });
|
||||||
|
});
|
||||||
|
});
|
||||||
@@ -0,0 +1,194 @@
|
|||||||
|
import { afterEach, beforeEach, describe, expect, it } from 'bun:test';
|
||||||
|
import * as fs from 'fs';
|
||||||
|
import * as os from 'os';
|
||||||
|
import * as path from 'path';
|
||||||
|
import { resolveProxyRequestRoute } from '../../../src/proxy/request-router';
|
||||||
|
import { resolveOpenAICompatProfileConfig } from '../../../src/proxy/profile-router';
|
||||||
|
import { loadSettings } from '../../../src/utils/config-manager';
|
||||||
|
|
||||||
|
let originalCcsHome: string | undefined;
|
||||||
|
let tempDir: string;
|
||||||
|
|
||||||
|
function writeSettings(profileName: string, env: Record<string, string>): string {
|
||||||
|
const settingsPath = path.join(tempDir, '.ccs', `${profileName}.settings.json`);
|
||||||
|
fs.writeFileSync(settingsPath, JSON.stringify({ env }, null, 2), 'utf8');
|
||||||
|
return settingsPath;
|
||||||
|
}
|
||||||
|
|
||||||
|
function buildProfile(profileName: string) {
|
||||||
|
const settingsPath = path.join(tempDir, '.ccs', `${profileName}.settings.json`);
|
||||||
|
const profile = resolveOpenAICompatProfileConfig(
|
||||||
|
profileName,
|
||||||
|
settingsPath,
|
||||||
|
loadSettings(settingsPath).env || {}
|
||||||
|
);
|
||||||
|
expect(profile).toBeTruthy();
|
||||||
|
return profile!;
|
||||||
|
}
|
||||||
|
|
||||||
|
beforeEach(() => {
|
||||||
|
originalCcsHome = process.env.CCS_HOME;
|
||||||
|
tempDir = fs.mkdtempSync(path.join(os.tmpdir(), 'ccs-proxy-router-'));
|
||||||
|
fs.mkdirSync(path.join(tempDir, '.ccs'), { recursive: true });
|
||||||
|
});
|
||||||
|
|
||||||
|
afterEach(() => {
|
||||||
|
if (originalCcsHome !== undefined) {
|
||||||
|
process.env.CCS_HOME = originalCcsHome;
|
||||||
|
} else {
|
||||||
|
delete process.env.CCS_HOME;
|
||||||
|
}
|
||||||
|
fs.rmSync(tempDir, { recursive: true, force: true });
|
||||||
|
});
|
||||||
|
|
||||||
|
describe('resolveProxyRequestRoute', () => {
|
||||||
|
beforeEach(() => {
|
||||||
|
process.env.CCS_HOME = tempDir;
|
||||||
|
|
||||||
|
const profiles = {
|
||||||
|
hf: writeSettings('hf', {
|
||||||
|
ANTHROPIC_BASE_URL: 'https://router.huggingface.co/v1',
|
||||||
|
ANTHROPIC_AUTH_TOKEN: 'hf_token',
|
||||||
|
ANTHROPIC_MODEL: 'hf-default',
|
||||||
|
CCS_DROID_PROVIDER: 'generic-chat-completion-api',
|
||||||
|
}),
|
||||||
|
deepseek: writeSettings('deepseek', {
|
||||||
|
ANTHROPIC_BASE_URL: 'https://api.deepseek.com/v1',
|
||||||
|
ANTHROPIC_AUTH_TOKEN: 'deepseek_token',
|
||||||
|
ANTHROPIC_MODEL: 'deepseek-chat',
|
||||||
|
CCS_DROID_PROVIDER: 'generic-chat-completion-api',
|
||||||
|
}),
|
||||||
|
thinker: writeSettings('thinker', {
|
||||||
|
ANTHROPIC_BASE_URL: 'https://thinking.example.com/v1',
|
||||||
|
ANTHROPIC_AUTH_TOKEN: 'think_token',
|
||||||
|
ANTHROPIC_MODEL: 'deepseek-reasoner',
|
||||||
|
CCS_DROID_PROVIDER: 'generic-chat-completion-api',
|
||||||
|
}),
|
||||||
|
search: writeSettings('search', {
|
||||||
|
ANTHROPIC_BASE_URL: 'https://search.example.com/v1',
|
||||||
|
ANTHROPIC_AUTH_TOKEN: 'search_token',
|
||||||
|
ANTHROPIC_MODEL: 'sonar-pro',
|
||||||
|
CCS_DROID_PROVIDER: 'generic-chat-completion-api',
|
||||||
|
}),
|
||||||
|
background: writeSettings('background', {
|
||||||
|
ANTHROPIC_BASE_URL: 'https://background.example.com/v1',
|
||||||
|
ANTHROPIC_AUTH_TOKEN: 'background_token',
|
||||||
|
ANTHROPIC_MODEL: 'qwen-small',
|
||||||
|
CCS_DROID_PROVIDER: 'generic-chat-completion-api',
|
||||||
|
}),
|
||||||
|
long: writeSettings('long', {
|
||||||
|
ANTHROPIC_BASE_URL: 'https://long.example.com/v1',
|
||||||
|
ANTHROPIC_AUTH_TOKEN: 'long_token',
|
||||||
|
ANTHROPIC_MODEL: 'gemini-2.5-pro',
|
||||||
|
CCS_DROID_PROVIDER: 'generic-chat-completion-api',
|
||||||
|
}),
|
||||||
|
};
|
||||||
|
|
||||||
|
fs.writeFileSync(
|
||||||
|
path.join(tempDir, '.ccs', 'config.json'),
|
||||||
|
JSON.stringify(
|
||||||
|
{
|
||||||
|
profiles,
|
||||||
|
proxy: {
|
||||||
|
routing: {
|
||||||
|
default: 'hf:hf-default',
|
||||||
|
background: 'background:qwen-small',
|
||||||
|
think: 'thinker:deepseek-reasoner',
|
||||||
|
longContext: 'long:gemini-2.5-pro',
|
||||||
|
longContextThreshold: 10,
|
||||||
|
webSearch: 'search:sonar-pro',
|
||||||
|
},
|
||||||
|
},
|
||||||
|
},
|
||||||
|
null,
|
||||||
|
2
|
||||||
|
),
|
||||||
|
'utf8'
|
||||||
|
);
|
||||||
|
});
|
||||||
|
|
||||||
|
it('uses explicit profile:model selectors', () => {
|
||||||
|
const route = resolveProxyRequestRoute(buildProfile('hf'), {
|
||||||
|
model: 'deepseek:deepseek-reasoner',
|
||||||
|
stream: true,
|
||||||
|
messages: [{ role: 'user', content: 'plan this' }],
|
||||||
|
});
|
||||||
|
|
||||||
|
expect(route.profile.profileName).toBe('deepseek');
|
||||||
|
expect(route.model).toBe('deepseek-reasoner');
|
||||||
|
expect(route.source).toBe('explicit-profile');
|
||||||
|
});
|
||||||
|
|
||||||
|
it('matches plain model ids to the profile that owns them', () => {
|
||||||
|
const route = resolveProxyRequestRoute(buildProfile('hf'), {
|
||||||
|
model: 'deepseek-chat',
|
||||||
|
stream: true,
|
||||||
|
messages: [{ role: 'user', content: 'hello' }],
|
||||||
|
});
|
||||||
|
|
||||||
|
expect(route.profile.profileName).toBe('deepseek');
|
||||||
|
expect(route.model).toBe('deepseek-chat');
|
||||||
|
expect(route.source).toBe('profile-model-match');
|
||||||
|
});
|
||||||
|
|
||||||
|
it('routes thinking requests through the configured think scenario', () => {
|
||||||
|
const route = resolveProxyRequestRoute(buildProfile('hf'), {
|
||||||
|
model: 'hf-default',
|
||||||
|
stream: true,
|
||||||
|
reasoning_effort: 'high',
|
||||||
|
reasoning: { enabled: true, effort: 'high' },
|
||||||
|
messages: [{ role: 'user', content: 'work this out carefully' }],
|
||||||
|
});
|
||||||
|
|
||||||
|
expect(route.profile.profileName).toBe('thinker');
|
||||||
|
expect(route.model).toBe('deepseek-reasoner');
|
||||||
|
expect(route.scenario).toBe('think');
|
||||||
|
expect(route.source).toBe('scenario');
|
||||||
|
});
|
||||||
|
|
||||||
|
it('routes long-context requests through the configured longContext scenario', () => {
|
||||||
|
const route = resolveProxyRequestRoute(buildProfile('hf'), {
|
||||||
|
model: 'hf-default',
|
||||||
|
stream: true,
|
||||||
|
messages: [{ role: 'user', content: 'x'.repeat(120) }],
|
||||||
|
});
|
||||||
|
|
||||||
|
expect(route.profile.profileName).toBe('long');
|
||||||
|
expect(route.model).toBe('gemini-2.5-pro');
|
||||||
|
expect(route.scenario).toBe('longContext');
|
||||||
|
expect(route.estimatedTokens).toBeGreaterThan(10);
|
||||||
|
});
|
||||||
|
|
||||||
|
it('routes web_search tool requests through the configured webSearch scenario', () => {
|
||||||
|
const route = resolveProxyRequestRoute(buildProfile('hf'), {
|
||||||
|
model: 'hf-default',
|
||||||
|
stream: true,
|
||||||
|
messages: [{ role: 'user', content: 'search the docs' }],
|
||||||
|
tools: [
|
||||||
|
{
|
||||||
|
type: 'function',
|
||||||
|
function: {
|
||||||
|
name: 'web_search',
|
||||||
|
parameters: { type: 'object' },
|
||||||
|
},
|
||||||
|
},
|
||||||
|
],
|
||||||
|
});
|
||||||
|
|
||||||
|
expect(route.profile.profileName).toBe('search');
|
||||||
|
expect(route.model).toBe('sonar-pro');
|
||||||
|
expect(route.scenario).toBe('webSearch');
|
||||||
|
});
|
||||||
|
|
||||||
|
it('routes haiku requests through the configured background scenario', () => {
|
||||||
|
const route = resolveProxyRequestRoute(buildProfile('hf'), {
|
||||||
|
model: 'claude-3-haiku',
|
||||||
|
stream: true,
|
||||||
|
messages: [{ role: 'user', content: 'run this in the background' }],
|
||||||
|
});
|
||||||
|
|
||||||
|
expect(route.profile.profileName).toBe('background');
|
||||||
|
expect(route.model).toBe('qwen-small');
|
||||||
|
expect(route.scenario).toBe('background');
|
||||||
|
});
|
||||||
|
});
|
||||||
@@ -28,6 +28,7 @@ describe('ProxyRequestTransformer', () => {
|
|||||||
|
|
||||||
expect(result.stream).toBe(true);
|
expect(result.stream).toBe(true);
|
||||||
expect(result.reasoning_effort).toBe('high');
|
expect(result.reasoning_effort).toBe('high');
|
||||||
|
expect(result.reasoning).toEqual({ enabled: true, effort: 'high' });
|
||||||
expect(result.max_tokens).toBe(1024);
|
expect(result.max_tokens).toBe(1024);
|
||||||
expect(result.temperature).toBe(0.2);
|
expect(result.temperature).toBe(0.2);
|
||||||
expect(result.top_p).toBe(0.9);
|
expect(result.top_p).toBe(0.9);
|
||||||
@@ -42,6 +43,43 @@ describe('ProxyRequestTransformer', () => {
|
|||||||
});
|
});
|
||||||
});
|
});
|
||||||
|
|
||||||
|
it('translates base64 image blocks into OpenAI image_url parts', () => {
|
||||||
|
const transformer = new ProxyRequestTransformer();
|
||||||
|
const result = transformer.transform({
|
||||||
|
messages: [
|
||||||
|
{
|
||||||
|
role: 'user',
|
||||||
|
content: [
|
||||||
|
{ type: 'text', text: 'Describe this image' },
|
||||||
|
{
|
||||||
|
type: 'image',
|
||||||
|
source: {
|
||||||
|
type: 'base64',
|
||||||
|
media_type: 'image/png',
|
||||||
|
data: 'ZmFrZS1pbWFnZS1ieXRlcw==',
|
||||||
|
},
|
||||||
|
},
|
||||||
|
],
|
||||||
|
},
|
||||||
|
],
|
||||||
|
});
|
||||||
|
|
||||||
|
expect(result.messages).toEqual([
|
||||||
|
{
|
||||||
|
role: 'user',
|
||||||
|
content: [
|
||||||
|
{ type: 'text', text: 'Describe this image' },
|
||||||
|
{
|
||||||
|
type: 'image_url',
|
||||||
|
image_url: {
|
||||||
|
url: 'data:image/png;base64,ZmFrZS1pbWFnZS1ieXRlcw==',
|
||||||
|
},
|
||||||
|
},
|
||||||
|
],
|
||||||
|
},
|
||||||
|
]);
|
||||||
|
});
|
||||||
|
|
||||||
it('drops malformed optional fields but preserves the translated core request', () => {
|
it('drops malformed optional fields but preserves the translated core request', () => {
|
||||||
const transformer = new ProxyRequestTransformer();
|
const transformer = new ProxyRequestTransformer();
|
||||||
const result = transformer.transform({
|
const result = transformer.transform({
|
||||||
|
|||||||
Reference in new issue
Block a user