Model client

How Felix talks to LLM providers.

The model layer lives in its own workspace package, packages/ai (felix_ai), which may not import felix — an invariant test walks every import node, so a lazy in-function import is not an escape hatch. That boundary is what makes model-agnosticism structural rather than a claim: anything the harness injects arrives as a Protocol (ToolSchema, ModelConfig) or an explicit sink (felix_ai.observability for counters, felix_ai.context for the prompt-cache key).

felix.patterns.model keeps only what genuinely needs the harness: resolving FELIX_MODEL_ROUTES against Settings, metering a turn through record_usage, and the tracing wrappers that give each provider call a span. Fallback and escalation live in felix.patterns.model_composites — deciding what to do when an answer is unavailable or not good enough is a policy, not routing.

Escalation bills two turns

confidence_escalation calls the primary, judges the answer, and calls the stronger model — two billed turns for one reply. The composite folds the discarded turn’s usage into the result it returns, because a caller can only meter what it is handed; without that, limits.max_cost_usd under-counts an escalating run by whatever the first turn cost.

Provider registry

A provider is a descriptor — a wire format, an endpoint, and where its credential lives — so adding one is a row rather than a module. Providers register at import time:

Provider	Wire format	Configured with
anthropic	Anthropic Messages	FELIX_ANTHROPIC_API_KEY
openai	OpenAI Chat Completions	FELIX_OPENAI_API_KEY, optional FELIX_LITELLM_BASE_URL
workers_ai	OpenAI Chat Completions	api_key, account_id, optional gateway_id
groq together deepseek cerebras fireworks openrouter xai mistral google	OpenAI Chat Completions	api_key

Everything past the first three is configured through FELIX_MODEL_PROVIDER_OPTIONS rather than a settings field per vendor:

{"workers_ai": {"api_key": "...", "account_id": "...", "gateway_id": "default"}}

That exists because Settings ignores unknown env vars, so FELIX_MYPROVIDER_API_KEY never lands anywhere — without it a registered third-party provider had no way to be given a key at all. An entry also overrides a built-in’s named field, which is how you point one at a gateway.

Every option value joins the redaction list except the ones the provider consumes as addressing — base_url, any {placeholder} its endpoint templates, and its header options — so a credential cannot reach tool output whatever the option is called. Matching secret-looking names was the previous rule, and it failed open for exactly the third-party providers this map exists to serve. The converse is worth knowing: an unrecognised option is redacted too, so a long non-secret value there will be masked out of tool results. Keep credentials out of base_url — it is exempt by definition, and also reaches server logs through connection errors.

The descriptor carries the two things that are not uniform. base_url_default may hold {option} placeholders — Cloudflare puts the account id in the URL path — and header_options sends a header only when its option is set, which is how one provider covers both direct and AI-Gateway-routed calls (cf-aig-gateway-id) instead of being two providers.

Add a provider with registry.register_model_provider(name, factory) from a plugin’s register(), or register_model_provider directly, before the first agent build — no edits to the react loop. The wire formats and transport are public (felix_ai.wire: OpenAICompletionsClient, AnthropicMessagesClient, post_with_retry, map_stop, parse_tool_arguments), because re-deriving retry-on-429, SSE parsing and usage accounting is most of the work of writing a provider.

Report usage, or the budgets fail open

record_usage is the only feed for limits.max_input_tokens, max_output_tokens and max_cost_usd. A provider that reports nothing leaves the run uncapped — it warns and increments felix_model_unmetered, but nothing stops it. On the OpenAI wire format a streamed response carries usage only when the request sets stream_options.include_usage; forget it and every streamed run is free as far as the budgets are concerned. tests/conformance/test_model_provider.py asserts this against every wire format.

Pricing, and refusing to guess

felix_ai.catalog records one entry per model family — context window, max output, accepted request parameters, and rates. An entry that does not state rates does not have them: pricing is None, the model contributes zero to spend, and is_priced() reports it as unmeterable. The hosted tier ships without per-token rates, deliberately, with one exception: Workers AI. Cloudflare meters it in neurons but publishes a per-token rate for each model, so the catalog carries those rates under @cf/ keys, which the same weights served by Groq or Together never match. The 10,000 free neurons a day are not subtracted, so spend reads high rather than low.

A manifest that declares limits.max_cost_usd on a model with no known rates is refused at compile, pointing at spec.model.price. Only a declared cap is refused; an unset one is filled from DEFAULT_LIMITS, and refusing on that would break every deployment on an unpriced model over a ceiling the author never asked for.

Costs are keyed on the wire model, while reporting keeps the logical route name.

Logical routes

Manifests declare spec.model.id as a logical id. Ops maps it via FELIX_MODEL_ROUTES JSON (falls back to baked-in defaults):

Logical id	Provider	Wire model
claude-opus	anthropic	claude-opus-5
claude-sonnet	anthropic	claude-sonnet-5
claude-haiku	anthropic	claude-haiku-4-5
claude-fable	anthropic	claude-fable-5
gpt-4.1	openai	gpt-4.1
gpt-4.1-mini	openai	gpt-4.1-mini
glm-5.3-cf	workers_ai	@cf/zai-org/glm-5.3 (text only)
glm-5.3-flash-cf	workers_ai	@cf/zai-org/glm-5.3-flash
deepseek-v4-pro-cf	workers_ai	@cf/deepseek-ai/deepseek-v4-pro-0813 (text only)
deepseek-v4-flash-cf	workers_ai	@cf/deepseek-ai/deepseek-v4-flash-0731 (text only)
qwen3.8-27b-cf	workers_ai	@cf/qwen/qwen3.8-27b
kimi-k2.7-code-cf	workers_ai	@cf/moonshotai/kimi-k2.7-code
kimi-k2-cf	workers_ai	@cf/moonshotai/kimi-k2.6
gpt-oss-120b-cf	workers_ai	@cf/openai/gpt-oss-120b (text only)
gpt-oss-20b-cf	workers_ai	@cf/openai/gpt-oss-20b (text only)
glm-flash-cf	workers_ai	@cf/zai-org/glm-4.7-flash (text only)
llama-3-pro	workers_ai	@cf/zai-org/glm-5.3 (legacy id)
llama-3-fast	workers_ai	@cf/zai-org/glm-5.3-flash (legacy id)

Wire ids are complete as written — the Anthropic routes carry no date suffix. Two legacy logical ids, claude-sonnet-4 and claude-haiku-4, still resolve so older manifests keep building; both now point at the current model in their tier (claude-sonnet-5 and claude-haiku-4-5) rather than the snapshot they were named for. Prefer the unsuffixed ids in new manifests.

Empty id → FELIX_DEFAULT_MODEL_ID (default claude-sonnet). An id that is in neither the baked-in table nor FELIX_MODEL_ROUTES fails the build.

Images and text-only routes

The catalog says a model is text-only only where it can vouch for it — the text_only flag, set on the gpt-oss, GLM Flash, Llama 3.3 and Qwen3 Workers AI entries and on Jev and Clef. The ("text",) modality every other entry inherits is a default, not a claim: the llama family key also matches llama3.2-vision. A route can state what it accepts itself, which wins over the catalog:

FELIX_MODEL_ROUTES='{"seer":{"provider":"workers_ai","model":"@cf/meta/llama-3.2-11b-vision-instruct","modalities":["text","image"]}}'

When the primary route is vouched text-only:

Situation	What happens
spec.model.vision_model or FELIX_DEFAULT_VISION_MODEL_ID names a route that can see	Each call carrying an image goes to that route; every other call stays on the primary. Metering follows the route that answered, and the switch counts as felix_model_switch{reason="vision"}.
Neither is set, and a user turn carries an image	/chat and /v1/chat/completions answer 422 naming the route, before the agent is built or any model called.
An image reaches the route mid-run anyway (a fallback, a tool result)	It is replaced by [image omitted: model route '…' does not accept images] and counted in felix_model_images_omitted.

The 422 and the build read one plan, so they agree about a route. A vision route that is named but cannot serve — missing from FELIX_MODEL_ROUTES, or text-only itself — fails the build, as an unroutable fallback does, and felix doctor fails on it.

The image stays in the thread’s history, so once a thread has been shown a picture every later turn of it goes to the vision route, until compaction or a window drops the image. felix_model_switch counts each of those calls. felix_model_images_omitted counts each image once, when it arrives. Usage is metered under the route that answered: the composite stamps it on the result rather than on itself. The vision route fails over to whichever of spec.model.fallbacks can see (a text-only fallback is skipped), and has no escalation of its own. The thread is compacted against the smaller of the two routes’ windows, since either may answer its next turn; a declared spec.session.context_window_tokens still wins.

Images a tool returns

A tool returns an image for the model to see with ToolOutputDict.attachments. The runner puts it on the tool message, and each wire renders it the one way its API accepts:

Wire	Rendering
Anthropic	Inside the tool_result, as [text, image…] content beside the call that produced it. A result without an image stays a plain string, for the prompt cache.
OpenAI chat completions (and every OpenAI-compatible provider)	The tool message stays text, and one user turn follows the run of tool messages. Each image appears under a line naming its tool and marking it as data, not instructions. Never inside the run: tool messages must follow their assistant turn contiguously.

Only inline bytes are rendered on a tool message, so a remote URL there is never fetched. The session log keeps a felix-file:// reference, resolved to bytes just before the wire like an upload. A tool image reaching a text-only route is replaced by the “image omitted” line.

A route the catalog cannot vouch for either way is sent the image unchanged, as before this check existed. felix doctor notes a text-only default with no vision route.

Client surface

chat(messages, tools, opts?) → ModelChatResult
stream_chat(...)             → AsyncGenerator[str, ModelChatResult]
count_tokens?(...)           → int   # Anthropic implements; others may omit

opts carries temperature, max tokens, and an abort signal from LimitState so wall-clock breaches cancel in-flight HTTP.

Anthropic / OpenAI specifics

• Tools — local Pydantic models compile to JSON Schema; remote MCP tools pass raw_input_schema through verbatim.
• Prompt caching — spec.model.cache tags ephemeral breakpoints (system / tools / tail) where the provider supports it.
• Thinking — spec.model.thinking_budget enables extended thinking; thinking blocks round-trip on continuations.
• Usage — normalized to {input, output, cache_creation?, cache_read?} so limit accounting is provider-agnostic.

Fallbacks and escalation

Configured on spec.model:

Mechanism	Behavior
fallbacks: [...]	On provider error (5xx / 429 / network), retry the same call against the next logical id
confidence_escalation	After chat, if the reply looks low-confidence, re-call escalate_to. A streamed turn settles the reply first and then chunks it, so it is judged too — escalation trades time-to-first-token for that. With decider: true the judgement is the decider’s, not the length-and-markers heuristic

Both are eager-built so unknown ids fail at build time, not mid-request. A switch increments the felix_model_switch counter (labels from, to, reason ∈ provider_error / low_confidence); it does not write an audit row.

Decision models

Some calls decide rather than write — which tool fits, which sub-agent takes a request, whether a reply answers, whether a text is an injection. Those used to ask a chat model for “a number only” or {"score": ...} and parse the reply, with no idea how sure it was. felix_ai.decide is a separate seam for models that answer typed questions:

Question	Answer
Choice(instructions, criteria) — one of ≤255 options	the option, a probability per option, a confidence
Score(instructions, levels) — 2–10 ordered levels	a probability-weighted level, a confidence
Noul(instructions) — true or false	the probability it is true

It is a registry beside the model registry, not a model provider: a decision provider has no chat, so routing one from spec.model.id would build a client that fails on its first turn. Routes live in FELIX_DECISION_ROUTES and credentials in the same FELIX_MODEL_PROVIDER_OPTIONS entry as the provider of the same name:

Route	Provider	Wire model	Configured with
clef	workers_ai — …/accounts/{account_id}/ai/run/@cf/cloudflare/clef	@cf/cloudflare/clef	api_key, account_id, optional gateway_id
clef-flash	workers_ai — …/ai/run/@cf/cloudflare/clef-flash	@cf/cloudflare/clef-flash	as clef
jev	typesafe — api.typesafe.ai/v1/systemone	jev-latest	api_key (refused at compile without one)
jev-cf	workers_ai — …/accounts/{account_id}/ai/run	typesafe/jev	api_key, account_id, optional gateway_id
(yours)	llm	any FELIX_MODEL_ROUTES id	the chat provider’s own

llm answers the same questions with a chat model through output_schema, so nothing depends on a second vendor; it reports its pick with no confidence, because a chat model’s self-assessed certainty is not calibrated, and consumers treat that as “no signal” rather than as sure.

Clef is Cloudflare’s open-weight decision model and speaks Jev’s API, so both run through one client. They differ only in how Workers AI is called: a Cloudflare-hosted @cf/ model is run by path, with state and questions at the top of the body and its short name (clef, clef-flash) in model; a partner model like typesafe/jev is posted to /ai/run with its fields nested under input. Clef costs $0.24 per million input tokens and clef-flash $0.09, Jev $0.042; output is unbilled for all three, and HTTP 529 is retried like 429.

The harness half, felix.decisions, builds a MeteredDecider per compile. Every decision goes through record_usage — the same path as a turn, so it counts against limits.max_cost_usd — and every answer is checked at that boundary whatever backend gave it: a choice that was not offered, a probability that is NaN or outside [0, 1], an unanswered question, or a wrong answer type is an error, never a value. A plugin adds a backend with register_decision_provider(name, factory).

Each consumer opts in with its own manifest flag and keeps its old path as the fallback — see spec.decider for the list.

Transient messages

A per-request note in the prompt costs the conversation its cache. It is present in one request and gone from the next, so it splits the cached prefix where it stood: as a system message the Anthropic wire folds it into the cached system block, and as a user message it takes the newest-message breakpoint. ChatMessage.transient marks a message for this request only. The Anthropic wire sends it after the cache breakpoint, the OpenAI wire sends it last, and the react loop attaches it to the run’s first model call and never writes it to the session log. The skill hint and recalled procedures travel this way.

Tip

For an open-weight stack, give workers_ai an entry in FELIX_MODEL_PROVIDER_OPTIONS and point routes at the -cf ids (see the bundled oss-only / hybrid-router manifests); Anthropic and OpenAI keys can stay unset.