Provider Compatibility Shim

Localhost proxy that scrubs response-only fields the Copilot SDK echoes back on followup requests, for OpenAI-compatible providers whose validators don't accept them. Works for any provider that strict-validates the OpenAI request schema - cerebras (original) and openai-strict (Ollama, vLLM, HuggingFace TGI, etc.) are both supported through a single shared shim.

Compatibility is configured per provider: set compat: on each provider block that needs it. A single always-on localhost proxy routes every such provider through its own /p/<name> route to that provider's own upstream, so any number of providers can run with compat set at the same time.

For the bigger picture - when to reach for a strict OpenAI-compatible endpoint at all (self-hosted vs. cloud, and which model to run) - see Inference. For the full provider field schema, see Inference Providers.

When to enable

Set compat: in a provider block when all of:

• auth_mode: byok
• provider: openai
• base_url: points at a strict OpenAI-compatible endpoint

Value	Target provider(s)	Default upstream when base_url unset
cerebras	Cerebras Cloud (api.cerebras.ai)	https://api.cerebras.ai/v1
openai-strict	Ollama, vLLM, HuggingFace TGI, any strict-spec endpoint	none - base_url is required (the endpoint varies per backend; startup fails if unset)

Leave off for OpenAI proper, Anthropic, and Azure OpenAI - they accept the extra SDK fields silently.

Symptom of running a strict provider with the shim off: tool-using turns fail with 400 status code (no body) from the SDK. The actual provider error (messages.N.assistant.refusal: property … is unsupported, or invalid message content type: <nil> from Ollama) is swallowed by the openai-node APIError handler, so the chat appears to hang on the second LLM call after any tool runs.

How it works

Copilot CLI ──► 127.0.0.1:8888/p/<provider> (shim) ──► that provider's strict upstream
                       │
                       │ request: scrub assistant.refusal & assistant.parsed;
                       │          coerce content:null → "" on assistant/tool msgs
                       │ response: pipe through unbuffered (SSE preserved)

A single localhost proxy starts unconditionally on 127.0.0.1:8888 early in buildServer() (server.ts) and is held by the compat registry (src/adapters/strict-compat/compat-registry.ts). If the port is already in use it logs a warning and boot continues - compat simply won't apply.

The proxy serves /p/<provider>/... routes. Each time the inference config loads or reloads, updateCompatRoutes() rebuilds the route map { providerName → base_url } for every provider whose compat is set and whose base_url is present. A request to /p/<provider>/chat/completions forwards to that provider's base_url, with the path remainder joined onto the upstream's path prefix. An unknown provider name returns 502.

When an agent resolves a BYOK provider that has compat set, the provider resolver points the SDK provider block's baseUrl at the proxy route (http://127.0.0.1:8888/p/<provider>) instead of the raw base_url. Providers without compat keep their raw base_url. For cerebras, base_url defaults to https://api.cerebras.ai/v1; for openai-strict, base_url is required.

Responses stream back via res.pipe() - no buffering, no extra latency, SSE token deltas pass through unchanged.

Configuration

Set the compat field in the provider definition:

# config/packages/pimcore_agent.yaml
pimcore_agent:
    inference:
        default_provider: local-cerebras
        providers:
            local-cerebras:
                driver: copilot
                auth_mode: byok
                provider: openai
                base_url: 'https://api.cerebras.ai/v1'
                token: '${CEREBRAS_API_KEY}'
                compat: cerebras
                default_model: gpt-oss-120b
                available_models:
                    gpt-oss-120b:
                        max_context_window_tokens: 131072
                        max_prompt_tokens: 120000
                        max_output_tokens: 16384
                    gpt-oss-20b:
                        max_context_window_tokens: 131072
                        max_prompt_tokens: 120000
                        max_output_tokens: 16384

            local-ollama:
                driver: copilot
                auth_mode: byok
                provider: openai
                base_url: 'http://host.docker.internal:11434/v1'
                token: 'ollama'
                compat: openai-strict
                default_model: 'gemma3:27b'
                available_models:
                    'gemma3:27b':
                        max_context_window_tokens: 128000
                        max_prompt_tokens: 120000
                        max_output_tokens: 8000
                    'llama3.1:8b':
                        max_context_window_tokens: 131072
                        max_prompt_tokens: 120000
                        max_output_tokens: 8000

What gets scrubbed / coerced

The sanitizer modifies every messages[] object where applicable. Implementation: agent-server/src/adapters/strict-compat/sanitize.ts.

Operation	Field(s)	Origin	Why
Strip	refusal	OpenAI structured-outputs response field (2024)	Always null in practice; openai-node attaches it to every response; strict endpoints reject with wrong_api_format
Strip	parsed	OpenAI SDK .parse() helper for structured outputs	SDK-internal - never belongs on the wire against any provider; OpenAI accepts silently, strict endpoints reject
Coerce	content: null → ""	Copilot CLI initialises assistant messages with content: null and only replaces it when the choice carries text	Strict validators (Ollama Go: invalid message content type: <nil>; vLLM/HF: Pydantic error) reject a null content on tool-call-only assistant turns. Applies defensively to tool role messages too. Coercing to "" is safe - the OpenAI spec allows both "" and null; strict servers require a string.

Not stripped, deliberately: reasoning (gpt-oss chain-of-thought working memory, billed via completion_tokens_details.reasoning_tokens; stripping it caused observed runaway loops where the model lost track of its own plan), tool_calls, tool_call_id, content (when non-null).

Adding a new field to the scrub list

When a provider rejects a new property (look for Model call failed lines with wrong_api_format in the agent-server log):

1. Add the destructure target and the touched predicate in src/adapters/strict-compat/sanitize.ts.
2. Add a unit test in tests/adapters/strict-compat-sanitize.test.ts - both stripped-when-present and other-fields-untouched.
3. Add a row to the scrub table above.

Bar for stripping: the field must be unambiguously not in OpenAI's documented request schema for that role AND the upstream must hard-reject it. Documented OpenAI fields the upstream just doesn't implement (like refusal) are workarounds, not fixes.

Adding support for a new provider

To run another strict OpenAI-compat backend through the shim, just add a provider with compat: set and a base_url: - no code change is needed; the always-on proxy picks it up on the next config load via updateCompatRoutes().

Any non-empty compat value (e.g. cerebras, openai-strict) enables the shim for that provider - presence is what matters, not the specific value. The single shared sanitizeForStrictCompat in src/adapters/strict-compat/sanitize.ts handles the scrubbing - no new sanitizer module is needed unless the provider requires genuinely different field treatment.

Only add a sibling sanitizer under src/adapters/strict-compat/ if a provider needs field handling that would break other providers if applied universally (e.g. stripping reasoning, which Cerebras expects and uses).

Limitations

• Localhost-only binding (127.0.0.1). Never expose externally. Logs are intentionally minimal - method + path + scrub count, never headers or bodies.
• No retry logic. Upstream 5xx propagates as 5xx; the Copilot SDK handles retries.
• Streaming preservation depends on node:https.request + res.pipe(). Switching to fetch().arrayBuffer() to add response-side sanitization would re-buffer the entire response and lose token-by-token SSE - don't.
• Request-side only. The shim never touches response bodies.

Context

Same class of bug landed client-side fixes in vercel/ai#15042 (Cerebras + reasoning_content), langchain-ai/langchainjs#8643, zed-industries/zed#36215. The parsed echo is a Copilot CLI bug regardless of provider - github/copilot-cli#2553 tracks the related class.