Deploy

Felix runs as containers you operate — CPython behind your own ingress, with no serverless or edge runtime anywhere in the harness. A CDN may sit in front for DNS, TLS and WAF only.

Piece	Local	Production
API	felix-api on :8080	Same image behind your ingress
Worker	Taskiq consumer	Same image, felix-worker
Scheduler	Taskiq scheduler	Same image, felix-scheduler
Postgres + pgvector	Compose service	Managed Postgres (Neon, RDS, Cloud SQL, …)
Valkey / Redis	Compose	Memorystore / ElastiCache / self-hosted
Object store	fs (default)	s3 / gcs (+ image extras)

Compose

make bootstrap   # .env with generated passwords, lean install (once per checkout)
make up          # a one-shot migrate service applies the schema before api/worker/scheduler start

Lean hosts: make up-lite. MinIO profile: make up-full with FELIX_DOCKER_EXTRAS=aws.

Overlays

Each is a Compose overlay on the base stack, composable with the others. Run from the repo root — the Makefile passes --project-directory ..

Target	Adds
make up-lite	Tighter mem_limit; no host ports for DB/cache
make up-pooled	PgBouncer in transaction mode
make up-replicas	Two API replicas behind one nginx origin
make up-observability	OTel Collector, Prometheus, Grafana, Jaeger, Loki, Postgres/Valkey exporters
make up-self	The self-build stack: a shell runner container that holds no secrets and runs every shell tool, and no host ports for DB, cache or object store. Writes FELIX_SHELL_RUNNER_TOKEN into .env first

Observability stack

make up-observability   # Grafana :3000 · Prometheus :9090 · Jaeger :16686

Budget roughly 1.7 GiB of additional mem_limit, so this does not belong on the same host as make up-lite. It also rebuilds the image with the otel extra — the lean default omits it, and without it OTLP export logs one warning at startup and ships nothing while every other signal keeps working.

/metrics is authenticated (its labels carry tenant-supplied manifest ids and remote MCP tool names), and Prometheus has no env expansion in scrape configs — so scripts/metrics-token.sh, which make up-observability runs for you, mints a second API key with an empty scopes array beside your operator key and writes the bare token to a gitignored file. /metrics has no scope gate, so that key reads metrics and is refused everywhere else.

See observability for the metric catalog and span schema.

Sending traces to a backend you already run

Felix exports OTLP, so any backend that reads it works — a collector, a hosted vendor, an LLM-observability console you self-host elsewhere. None of that runs inside the Felix compose project: every FELIX_OTEL_* setting is passed through compose.yml, so make up exports with no overlay at all.

FELIX_DOCKER_EXTRAS=otel            # the exporter lives behind an extra
FELIX_OTEL_ENABLED=true
FELIX_OTEL_PROTOCOL=http            # grpc for a collector on 4317; http for a hosted ingest
FELIX_OTEL_ENDPOINT=https://ingest.example.com/v1/otel
FELIX_OTEL_INSECURE=false
FELIX_OTEL_HEADERS=authorization=Basic <credential>
FELIX_OTEL_SERVICE_NAME=felix-prod  # when several deployments share one backend

Felix needs no vendor SDK and no extra dependency. Spans carry the gen_ai.* semantic-convention attributes, so a backend that understands them renders a chat as a generation with model, token usage and cost rather than as a bare span — the integration is an endpoint and a header, and nothing in the harness knows which product is on the other end.

Run uv run felix doctor after: it reports whether the exporter is actually installed, in every environment. Enabling export on a lean image logs one warning and sends nothing, and an empty dashboard is otherwise the only sign.

Identity reaches the backend by default

user.id and the tenant id are on spans unless you set FELIX_OTEL_CAPTURE_IDENTITY=false. Prompts and completions are not — they sit outside governance content screening, so FELIX_OTEL_CAPTURE_CONTENT=true is an explicit opt-in.

A process has one OTLP destination. To send to Jaeger and a second backend, run the observability overlay too, leave FELIX_OTEL_ENDPOINT pointing at the collector, and add an otlphttp exporter there — fanning out is the collector’s job. The second backend’s credential goes in the collector’s config, not in .env: the overlay blanks FELIX_OTEL_HEADERS so a header meant for a hosted ingest is not sent to a local collector that never asked for one.

Sizing

Each worker process carries its own connection pool, so raise the two together or the second one becomes the ceiling.

Setting	Default	Notes
FELIX_WORKERS	1	Granian/uvicorn worker processes
FELIX_DB_POOL_SIZE	10	Per worker
FELIX_DB_MAX_OVERFLOW	20	Per worker, on top of the pool
FELIX_DB_POOL_TIMEOUT_SECONDS	30	How long a request queues before failing
FELIX_DB_POOL_PRE_PING	true	Set false against a direct Postgres
FELIX_DB_PREPARED_STATEMENTS	true	Set false behind a pooler that does not track them — see below

The effective ceiling is WORKERS × (POOL_SIZE + MAX_OVERFLOW) connections. Past it, requests queue for FELIX_DB_POOL_TIMEOUT_SECONDS and then fail — so check the ceiling against your database’s own max_connections before raising either number.

FELIX_DB_POOL_PRE_PING costs a round trip on every checkout to discover a connection a pooler closed server-side. That earns its cost behind PgBouncer, RDS Proxy, or Cloud SQL and wastes it against a Postgres you connect to directly.

Behind a connection pooler

Four workers on the defaults is 120 connections against a stock Postgres max_connections of 100 — and raising max_connections trades one wall for another, since every backend costs memory. A transaction-mode pooler multiplexes many cheap client connections onto few server ones instead.

make up-pooled   # adds PgBouncer in transaction mode in front of Postgres

Felix’s own pool then becomes a pool of client connections to PgBouncer, which are cheap. PGBOUNCER_POOL_SIZE is the real server-side limit — raise the former freely, size the latter against your database.

Run migrations against Postgres directly rather than through the pooler. Nothing breaks under transaction pooling; a migration is a one-off admin action with no reason to go through a multiplexer sized for request traffic.

Prepared statements

This one fails on the sixth query, not the first

psycopg3 automatically prepares a statement after its fifth execution. Under transaction pooling the sixth lands on a different server connection, where that statement was never created, and the query fails. Five identical queries succeed first, so the symptom looks unrelated to the change that caused it.

Either the pooler tracks prepared statements or Felix stops making them — never neither:

Pooler	Setting
PgBouncer ≥ 1.21 with max_prepared_statements > 0	leave FELIX_DB_PREPARED_STATEMENTS=true
PgBouncer with max_prepared_statements = 0	set FELIX_DB_PREPARED_STATEMENTS=false
RDS Proxy	set FELIX_DB_PREPARED_STATEMENTS=false
Direct Postgres, or session-mode pooling	leave it true

make up-pooled sets max_prepared_statements non-zero, so the default is correct there.

RDS Proxy is the case that forces the decision rather than merely allowing it: it pins the session when it sees a prepared statement, holding a server connection for the life of the client one — which defeats the multiplexing you deployed it for. There the setting costs you nothing and buys back the pooling.

Helm

Chart: deploy/helm/felix in felix-run/felix.

helm upgrade --install felix ./deploy/helm/felix \
  --set secrets.databaseUrl="$FELIX_DATABASE_URL" \
  --set secrets.redisUrl="$FELIX_REDIS_URL" \
  --set secrets.consumerSharedSecret="$FELIX_CONSUMER_SHARED_SECRET" \
  --set secrets.jwksPublic="$FELIX_JWKS_PUBLIC" \
  --set secrets.jwksPrivate="$FELIX_JWKS_PRIVATE"

A pre-install/pre-upgrade Job runs felix migrate head (migrate.enabled=true). Rolling the schema back is a separate, deliberate step: felix migrate <revision> --down runs each revision’s downgrade(), which drops what it added. Without --down, asking for a revision the database has already passed is refused (exit 2) rather than reported as done. The chart runs api, worker, and scheduler. For FELIX_OBJECT_STORE=fs, enable persistence.enabled=true.

Set serviceMonitor.enabled=true for a Prometheus Operator scrape of /metrics. It reads a bearer token from a Secret you create — use a Felix API key with an empty scopes array, the same least-privilege shape scripts/metrics-token.sh mints for Compose:

kubectl create secret generic felix-metrics-scrape --from-literal=token="$UNSCOPED_API_KEY"
helm upgrade --install felix ./deploy/helm/felix \
  --set serviceMonitor.enabled=true \
  --set serviceMonitor.bearerTokenSecret.name=felix-metrics-scrape

Cloud notes: deploy/aws/README.md, deploy/gcp/README.md.

Auth (production)

Mode	Notes
jwt	Preferred — set FELIX_JWKS_PUBLIC / verifiers. Add GitHub login to let people mint their own tokens from org membership
api_key	FELIX_AUTH_API_KEYS map. Tenants are the orgs sharing an install, and keys and tenants are the install operator’s configuration, with no key table or key API: adding an org or a key is an edit and a restart. An org’s members log in through GitHub login or your IdP instead
none	Loopback only — see below

none cannot be exposed

FELIX_AUTH_MODE=none is refused whenever FELIX_HOST is reachable off-host — in every environment. FELIX_ALLOW_INSECURE only relaxes the non-development environment check; it does not permit a public unauthenticated bind. FELIX_HOST defaults to 127.0.0.1, containers set 0.0.0.0 explicitly and pair it with a real auth mode, and Compose defaults to api_key with a key generated by make up.

/internal/* needs FELIX_CONSUMER_SHARED_SECRET when auth is not open.

Session leases and retried sends

Two settings decide how the harness treats clients that share a thread and clients that retry.

Setting	Default
FELIX_LEASE_ENFORCE	advisory	advisory: the routes that drive a thread check X-Felix-Lease-Token only when a client sends it, so a script or a /v1 call is never refused. strict: a driving request without the header is 409 lease_held while another client holds the thread exclusively — an unheld thread, or one only observers watch, still passes — and /v1/chat/completions with a user, which names the same thread, is checked too.
FELIX_IDEMPOTENCY_TTL_SECONDS	86400	How long an Idempotency-Key on POST /chat or POST /chat/stream is remembered, so a resend inside it is answered from the first request rather than run again.

strict is for a deployment where people run more than one client against the same threads — a browser tab and the terminal, say — and want a write that forgot its token refused rather than taken as a second driver. It is a consistency control between cooperating clients, not an authorization boundary: anyone in the tenant may take a lease. Both clients here present a token on every write, watching with an observer hold when another client drives, so they behave the same under either setting. Leases and keys live in Redis when FELIX_REDIS_URL is set, and per process without it — where each replica grants and remembers its own. See Leases and Idempotency-Key for the wire.

Cron schedules

Scheduled jobs (PUT /jobs/{name}) use standard five-field cron, evaluated in UTC:

┌─ minute (0-59)
│ ┌─ hour (0-23)
│ │ ┌─ day of month (1-31)
│ │ │ ┌─ month (1-12)
│ │ │ │ ┌─ day of week (0-6, Sunday = 0)
│ │ │ │ │
* * * * *

Expression	Meaning
*/10 * * * *	Every ten minutes
0 3 * * *	Daily at 03:00
0 9 * * 1	Mondays at 09:00

Note

The worker’s run_scheduled_jobs cron sweeps due rows every minute, but the sweep is enqueued by felix-scheduler. Run all three containers — API, worker and scheduler — or no job, retention sweep, memory consolidation, skill evaluation or skill improvement ever fires. Durable runs are the exception: each worker polls for them itself (below), and the scheduler’s once-a-minute fiber_scheduler sweep is only the backstop.

Durable runs

A durable run is a pending fiber until a worker claims it, and nothing tells the worker when one is submitted — so each worker polls. The claim is the same lease plus FOR UPDATE SKIP LOCKED the cron sweep takes, so any number of workers, polling or sweeping, never run one fiber twice.

Setting	Default
FELIX_FIBER_POLL_SECONDS	1.0	How often each worker checks for due fibers. 0 turns the poll off and leaves only the once-a-minute sweep, which is how durable runs were started before 0.5.0 — each waited up to a minute before its first model call.
FELIX_FIBER_CONCURRENCY	8	Fibers one worker advances at once. A run parked on an approval holds its slot until someone answers, so this bounds how many such waits can pile up before a new run waits for a slot. A full worker leaves further runs unclaimed for another worker.
FELIX_FIBER_MAX_ATTEMPTS	5	Consecutive failed steps before a fiber is dead; retries back off 1m doubling to 1h.

Credentials the model never holds

An agent’s http_tools can call an authenticated API without the model holding the key. Register the key once, per tenant, against one exact host; a manifest names the id in spec.network.credentials:

FELIX_EGRESS_CREDENTIALS='{
  "github": {"host": "api.github.com", "path_prefix": "/repos/acme/",
             "secret": "secret:GITHUB_TOKEN", "scheme": "Bearer", "tenants": ["acme"]}
}'

Key	Required	Notes
host	yes	One exact host; no wildcard, scheme, port or path.
port	no	The port the API listens on; absent is the scheme’s default. Another port on the host gets no key.
path_prefix	no	Default /. Compared with the path as sent, after .. is resolved; encoded traversal is refused.
header	no	Default Authorization. Host, Content-Length, Cookie and the other transport headers are refused.
scheme	no	A word put before the value, such as Bearer; empty sends the value alone.
secret	yes	secret:NAME through FELIX_SECRETS_BACKEND, or a literal. Resolved when the agent compiles and masked everywhere from then. A value under 8 characters, with surrounding whitespace or a control character, or one the backend cannot find fails the compile (422), naming the id and never the value.
tenants	yes	Tenant ids that may name it, or "*" for all. Any manifest in such a tenant may bind it; "*" and default include anonymous callers.

The egress transport attaches the header after the host check, over https only (plain http only with FELIX_ENVIRONMENT=development and FELIX_ALLOW_INSECURE=true), and judges each redirect hop again, and strips the header from any request the credential does not cover. A malformed registry fails the boot. The manifest must also set network.allow_hosts, naming the host, so what the agent reads with the key can only go where its list allows.

Setting	Default
FELIX_EGRESS_CREDENTIALS	empty	The registry above, as JSON.

Completion webhooks

A durable run can announce its end to endpoints its manifest names in spec.execution.webhooks. The manifest carries only ids; the operator registers what each id means:

FELIX_WEBHOOK_ENDPOINTS='{
  "ops":   {"url": "https://hooks.example.com/felix", "secret": "secret:OPS_WEBHOOK", "tenants": ["acme"]},
  "audit": {"url": "http://audit.internal:8080/runs", "secret": "whsec_...", "tenants": "*", "private": true}
}'

Key	Required	Notes
url	yes	Absolute https. http only with FELIX_ENVIRONMENT=development and FELIX_ALLOW_INSECURE=true. No user:pass@.
secret	yes	Signs every delivery. A whsec_… base64 key (what Standard Webhooks libraries generate), any other string, or secret:NAME resolved through FELIX_SECRETS_BACKEND. Masked in audit, session and tool output in every process.
tenants	yes	Tenant ids that may name the endpoint, or "*" for all. There is no default: an endpoint open to every tenant lets any tenant’s run arrive signed with its secret, so it has to be written out — and that receiver must check tenant_id.
private	no	true exempts this URL from the egress guard so it may resolve to a private address. Operator-only; redirects are still refused.

A malformed registry fails the boot. A run naming an id that is unregistered or not allowed for its tenant is refused at enqueue with the same 422 unknown webhook endpoint: <id> either way, so the answer does not reveal what other tenants have registered. Removing or narrowing an endpoint later marks pending deliveries to it dead rather than sending them.

Setting	Default
FELIX_WEBHOOK_ENDPOINTS	empty	The registry above, as JSON.
FELIX_WEBHOOK_MAX_ATTEMPTS	8	Tries per endpoint before dead; backoff 1m doubling to 1h.
FELIX_WEBHOOK_TIMEOUT_SECONDS	10	Above 0, at most 60; a larger value fails the boot. Whole-attempt bound, including a receiver that answers slowly. A sweep starts no delivery after the first 60 seconds of its 120-second claim, and the cap keeps one attempt inside that.

Delivery runs in the worker’s webhook_delivery cron, so it needs felix-scheduler like every other periodic task. Each attempt increments felix_webhook_delivery{kind, endpoint, outcome} with kind="run" and outcome one of delivered, retry, dead. The payload and how to verify it: REST API → Completion webhooks.

The same registry carries imported skills’ update events: FELIX_SKILL_UPDATE_WEBHOOKS binds a tenant to endpoint ids here, and those deliveries count under kind="skill_update". The kind label is new with them; a query that ignores it now counts both, so filter {kind="run"} for runs alone. A receiver shared by both dispatches on the body’s type. See Skill import → Update notifications.

Inbound triggers

An event from elsewhere (a pushed commit, a failed payment, an alert) can start a durable run. The operator registers each trigger; senders POST to /triggers/{id} (the endpoint):

FELIX_TRIGGERS='{
  "gh-push": {"tenant": "acme", "manifest": "ci-triage", "scheme": "github",
              "secret": "secret:GH_HOOK", "events": ["push", "pull_request"],
              "prompt": "A {event} event arrived from GitHub ({delivery}). Triage it."},
  "alerts":  {"tenant": "acme", "manifest": "oncall", "scheme": "standard",
              "secret": ["whsec_old...", "whsec_new..."], "scopes": ["oncall:run"], "rate_limit": 20,
              "prompt": "An alert fired. Investigate and summarise."}
}'

Key	Required	Notes
tenant	yes	The tenant the run belongs to.
manifest	yes	The agent that runs. It must list the trigger_event tool, or every delivery is refused with 503.
scheme	yes	standard (Standard Webhooks: webhook-id, webhook-timestamp, webhook-signature, a 5-minute window) or github (X-Hub-Signature-256).
secret	yes	secret:NAME, or a list of them: any one verifies, so rotate by adding the new secret, switching the sender, then dropping the old. A literal is accepted only with FELIX_ENVIRONMENT=development and FELIX_AUTH_MODE=none. A key under 16 bytes, as written or as resolved, is refused: an unset variable behind secret:NAME answers every delivery 503 rather than verifying against an empty key. whsec_… values are base64-decoded (whitespace ignored), as Standard Webhooks libraries generate them.
prompt	yes	The run’s instruction. {trigger} is the id; {event} is the event type only when events lists it (else unknown), because GitHub does not sign that header; {delivery} is the delivery id when it is a short token. Nothing from the body is filled in.
events	no	github only. Event types that start a run (GitHub’s X-GitHub-Event). Others are answered 200 {"status":"ignored"}. GitHub’s ping is ignored unless listed, so creating a hook starts no run.
scopes	no	Granted to the run’s principal, trigger:<id>, for the manifest’s auth.inbound.required_scopes and spec.policies. None by default.
rate_limit	no	Runs started per minute, 1–10000, default 60. Counted only for new deliveries that would start a run: not for a bad signature, a replay or an ignored event.

A malformed registry fails the boot. The path takes no bearer, and only while FELIX_TRIGGERS is set; the signature is the credential. The payload never becomes the run’s instruction: the model reads it by calling trigger_event, whose output content screening treats like an MCP server’s, and a payload carrying a prompt-injection marker is withheld from it whatever the manifest’s screening says. A manifest that limits auth.inbound.schemes must name trigger. Add spec.execution.webhooks to the manifest to send the run’s result on: event in, result out.

Setting	Default
FELIX_TRIGGERS	empty	The registry above, as JSON.
FELIX_IDEMPOTENCY_TTL_SECONDS	86400	How long a delivery is remembered, so a replay starts nothing. Per process without FELIX_REDIS_URL. GitHub signs no timestamp, so a captured GitHub delivery replayed after this starts a new run.

A trigger that runs a workflow

A trigger may name a workflow instead of a manifest and prompt, and map payload fields to the workflow’s inputs by JSON Pointer:

FELIX_TRIGGERS='{
  "gh-issue": {"tenant": "acme", "workflow": "triage", "scheme": "github",
               "secret": "secret:GH_HOOK", "events": ["issues"],
               "inputs": {"number": "/issue/number", "title": "/issue/title", "kind": "$event"}}
}'

Each value must be what the workflow declares for that input, or the delivery is 422 invalid_payload and starts nothing (checked before the rate limit). $event is a type events lists; $delivery is the delivery id as a short token. The sender’s strings, objects and arrays are recorded on the run as untrusted_inputs, and every agent step of a trigger-started run reads what it is given inside a fence made new for each prompt — a convention the model is told to honour, so keep inbound screening on the manifests the workflow reaches. Anyone with workflows:write can change what the trigger’s scopes reach; pin the workflow with "revision" when it grants scopes.

Slack

With the optional felix-channels plugin installed, people can talk to an agent from Slack: a mention in a channel, or a DM to the app. Each message starts a durable run as its sender, a Slack thread is one Felix conversation, and the worker posts the answer back in the thread when the run ends.

Install the plugin in both the API and the worker image: FELIX_EXTRAS=channels when building (docker build --build-arg FELIX_EXTRAS="channels" .), or uv sync --extra channels from a checkout. Without it, FELIX_CHANNELS is inert. The worker must run with felix-scheduler, as for every durable run.

Create the Slack app (api.slack.com/apps):

1. Bot token scopes: app_mentions:read, chat:write, im:history. Install it to the workspace and keep the bot token (xoxb-…) and, from Basic Information, the signing secret.
2. Event Subscriptions: request URL https://<your-felix>/channels/slack/<id>/events, where <id> is the key below. Slack sends a signed challenge, which Felix answers once the entry exists. Subscribe to the bot events app_mention and message.im.
3. App Home: allow users to send messages from the messages tab, for DMs.

FELIX_CHANNELS='{
  "acme-slack": {"platform": "slack", "tenant": "acme", "manifest": "assistant",
                 "team_id": "T0123ABCD",
                 "signing_secret": "secret:SLACK_SIGNING_SECRET", "bot_token": "secret:SLACK_BOT_TOKEN"}
}'

Key	Required	Notes
platform	yes	slack.
tenant, manifest	yes	Where the conversation runs. A manifest that limits auth.inbound.schemes must name slack.
team_id	yes	The workspace (T…). A request from any other workspace is ignored.
signing_secret, bot_token	yes	secret:NAME; a literal only with FELIX_ENVIRONMENT=development and FELIX_AUTH_MODE=none. A signing secret that does not resolve, or resolves under 16 bytes, answers 503.
scopes	no	Granted to every sender, slack:<team>:<user>, for auth.inbound.required_scopes and spec.policies. None by default; there is no per-user account linking yet, so everyone in the workspace has the same scopes.
channels, users	no	Slack ids that may talk to the agent; anyone in the workspace when unset.
rate_limit	no	Attempts per minute for the whole channel, 1–10000, default 60.
user_rate_limit	no	Attempts per minute for one sender, at most rate_limit, default 10, so one person cannot spend everyone’s budget.

A malformed registry fails the boot. Requests are verified with the signing secret (a 5-minute window); Slack’s retries are recognised by their event id and start nothing (across replicas only with FELIX_REDIS_URL). Edits, deletions and bots’ messages, the app’s own replies among them, are ignored, and so are people from other organisations in a Slack Connect channel (or another workspace of the same Enterprise Grid org). While a conversation’s run is still going, a message there is told to wait, at most once a minute. If the manifest’s inbound auth refuses the channel (its scopes, or auth.inbound.schemes without slack), every sender is refused alike: the conversation is told the agent isn’t available, at most once a minute, and the log says channel_misconfigured. Replies are escaped (no @channel pings or disguised links from the model) and posted with link previews off; a failed or expired run is answered with a short note, never its error. Replies are sent once: if Slack is down when the run ends, that reply is lost. Not yet: approvals and ask_user as Slack buttons (a run waiting on an approval waits for the web client or the API), Telegram, and per-user linking.

Setting	Default
FELIX_CHANNELS	empty	The registry above, as JSON. Read by the felix-channels plugin.
FELIX_SECRET_NAMES	empty	Add the channel’s secret names (SLACK_SIGNING_SECRET,SLACK_BOT_TOKEN), so the worker, where channel runs execute, masks them from boot rather than from its first reply.

Skill library

Agent Skills come from the image’s bundled skills/, from FELIX_SKILLS_DIR, from SKILL.md keys an operator uploads to the object store, and from each tenant’s skill library, which agents with spec.skill_authoring and operators with skills:write write through review. See Concepts → Skill.

Setting	Default
FELIX_SKILLS_DIR	empty	An extra directory of SKILL.md packages, searched after the bundled skills/. The library refuses to save a name either one holds.
FELIX_SKILL_PUBLISH_MIN_QUALITY	0	0–100. Refuse a publish whose quality review scores below this; 0 turns the bar off.
FELIX_SKILL_PUBLISH_BLOCK_ON_ADVISORY	false	Also refuse a version whose security scan is advisory (medium findings), not only fail.
FELIX_SKILL_PUBLISH_REQUIRE_EVAL	false	Refuse a publish unless the version has a succeeded evaluation that counts.
FELIX_SKILL_PUBLISH_MIN_EVAL_UPLIFT	unset	−100–100. Refuse a publish unless the version’s latest counting evaluation has at least this uplift; unset is no floor.

A version whose security scan fails never publishes, whatever these say. They are the deployment’s floor: a tenant’s own policy (PATCH /skill-library/-/policy) can tighten every one and loosen none. A rollback is judged without the two evaluation settings.

Skill imports

Browsing and importing skills from GitHub (GET /skill-library/-/browse, POST /skill-library/-/import) runs on the API; every call goes to api.github.com through the egress guard. The settings are read at boot. See Skill import.

Setting	Default
FELIX_SKILL_IMPORT_GITHUB_TOKEN	empty	Reaches private repositories and lifts GitHub’s limit from 60 calls an hour to 5,000. Read-only Contents access is enough. Hydrated from the secrets backend and masked like any credential; sent only as a header to api.github.com.
FELIX_SKILL_IMPORT_SOURCES	empty	Comma-separated [<tenant>=]github:<owner>[/<repo>[/<path>]] globs a browse or import may name; an entry with a tenant serves that tenant only, and one without a glob covers everything under it. Empty allows any GitHub source.
FELIX_SKILL_IMPORT_MIN_AGE_DAYS	0	0–365. Refuse (403 too_recent) files this tenant first saw fewer than this many days ago, by Felix’s own clock. Tenants can raise it, never lower it.
FELIX_SKILL_IMPORT_CALLS_PER_HOUR	500	1–100000. GitHub calls one tenant’s browses and imports may make per hour (429 rate_limited).
FELIX_SKILL_IMPORT_CALLS_PER_HOUR_TOTAL	4000	1–1000000. The same for the whole deployment, protecting the token’s own limit.
FELIX_SKILL_IMPORT_CHECK_HOURS	0	0–168; 0 is off. Check every imported skill against its origin this often, from the worker’s skill_upstream_checks task, spending at most half of each budget above. Needs felix-scheduler, and the worker needs the same FELIX_SKILL_IMPORT_SOURCES and FELIX_SKILL_IMPORT_GITHUB_TOKEN as the API; felix doctor notes this when it is set.
FELIX_SKILL_UPDATE_WEBHOOKS	empty	Comma-separated tenant=endpoint_id (acme=ops,acme=ci,beta=beta-hook): where each tenant’s skill.update_available events go. Each id must be a FELIX_WEBHOOK_ENDPOINTS endpoint open to that tenant, or boot fails; no wildcard, and a tenant not listed gets none. Set it on the API and the worker alike. See Update notifications.

A malformed FELIX_SKILL_IMPORT_SOURCES entry stops the API at boot. With the token set, boot also refuses an empty list, an entry naming no tenant, and an entry whose owner is a glob — each would let one tenant read whatever the token reads — unless FELIX_ENVIRONMENT=development and FELIX_AUTH_MODE=none together. The call budgets are shared across replicas through FELIX_REDIS_URL; without Redis each replica keeps its own.

Skill jobs

Evaluations and AI improvements from accepted feedback run in the worker’s skill_jobs task, enqueued every minute by felix-scheduler — run it alongside felix-worker, or a queued evaluation stays queued and accepted feedback is never rewritten. The API never enqueues to the worker directly. Each sweep runs at most five jobs of each kind, and one sweep runs at a time across every worker, held by a lease row that works through a transaction pooler, so the PgBouncer overlay needs nothing special. A worker that dies mid-sweep frees it within the job deadline plus five minutes.

Setting	Default
FELIX_SKILL_IMPROVE_MODEL	empty	The FELIX_MODEL_ROUTES id that rewrites a skill from accepted feedback; empty is FELIX_DEFAULT_MODEL_ID. The rewrite is saved as a draft, never published.
FELIX_SKILL_EVAL_MODEL	empty	The route that answers each evaluation scenario, with and without the skill; empty is the default route.
FELIX_SKILL_EVAL_JUDGE_MODEL	empty	The route that scores the answers; empty is the default route.
FELIX_SKILL_EVAL_MAX_SCENARIOS	10	1–10. Scenarios one evaluation runs; each is four model calls.
FELIX_SKILL_EVAL_MAX_TOKENS	2048	64–32000. Output-token cap on each evaluation call (answers, generated scenarios, judge scores).
FELIX_SKILL_IMPROVE_MAX_TOKENS	8192	256–64000. Output-token cap on the rewritten SKILL.md.
FELIX_SKILL_JOB_DEADLINE_SECONDS	1200	1–86400. Wall clock for one job; past it the job ends failed.
FELIX_SKILL_JOBS_MAX_QUEUED	20	1–10000. Per tenant: evaluations and accepted improvements queued or running at once.
FELIX_SKILL_JOBS_DAILY_LIMIT	200	1–100000. Per tenant: skill jobs created per UTC day.

Past either per-tenant cap, queueing another job is refused 429 skill_jobs_cap_reached, which waiting fixes. Every job’s model calls are metered to the skill’s tenant like a turn. A job is claimed at most three times before it fails attempts_exhausted.

Model routes

FELIX_DEFAULT_MODEL_ID (default claude-sonnet) picks the model a manifest gets when it does not name one. FELIX_MODEL_ROUTES is a JSON object that overrides or extends the logical-id table, so manifests stay portable across deployments:

FELIX_DEFAULT_MODEL_ID=claude-sonnet
FELIX_MODEL_ROUTES='{
  "claude-sonnet": {"provider": "anthropic", "model": "claude-sonnet-5"},
  "house-fast":    {"provider": "workers_ai", "model": "@cf/zai-org/glm-5.3-flash"}
}'

Providers are anthropic, openai, workers_ai (Cloudflare Workers AI), groq, together, deepseek, cerebras, fireworks, openrouter, xai, mistral and google, plus any registered by a plugin. Point FELIX_LITELLM_BASE_URL at a local gateway to keep traffic in your own network. Everything past the first two is configured through FELIX_MODEL_PROVIDER_OPTIONS:

FELIX_MODEL_PROVIDER_OPTIONS='{"workers_ai":{"api_key":"...","account_id":"...","gateway_id":"default"}}'

That one entry is all the built-in Workers AI routes need: glm-5.3-cf (GLM-5.3, 1M window), glm-5.3-flash-cf (GLM-5.3 Flash, 1M window, vision), deepseek-v4-pro-cf and deepseek-v4-flash-cf (DeepSeek V4, 1M window), qwen3.8-27b-cf (Qwen 3.8 27B, 262K window, vision), kimi-k2.7-code-cf (Kimi K2.7 Code, 262K window, vision), kimi-k2-cf (Kimi K2.6, 262K window), gpt-oss-120b-cf, gpt-oss-20b-cf and glm-flash-cf. The legacy llama-3-pro and llama-3-fast ids, which used to route to a local Ollama, now point at GLM-5.3 and GLM-5.3 Flash. All of them call tools and are priced, so limits.max_cost_usd is enforced on them; GLM-5.3, DeepSeek V4 and Kimi K2.7 Code need the Workers Paid plan or AI Gateway credits. To run on open weights wherever a manifest names no model, and move the memory embedder there as well:

FELIX_DEFAULT_MODEL_ID=glm-5.3-cf
FELIX_MEMORY_EMBEDDER=workers_ai

gpt-oss and GLM Flash are text-only. A default that points at one should name a route that can see for turns that carry an image — Kimi is the Workers AI one — or every such turn is refused with a 422 naming the route:

FELIX_DEFAULT_MODEL_ID=gpt-oss-120b-cf
FELIX_DEFAULT_VISION_MODEL_ID=kimi-k2-cf

A route of your own declares "modalities": ["text", "image"] when the model behind it can see. See Images and text-only routes.

Provider names in FELIX_MODEL_ROUTES are resolved against the registry at startup, so a typo fails immediately rather than on the first request that happens to take that route. A manifest referring to house-fast works everywhere the route is defined. See Model client.

Embeddings

One embedder per deployment serves the vector channel of memory recall and of the documents corpus. Without one, both still retrieve by full text (and memory by topic key); hits report lexical only.

Setting	Default	Notes
FELIX_MEMORY_EMBEDDER	auto	auto embeds locally with sentence_transformers when felix-harness[embeddings] is installed, and otherwise runs no vector channel. none turns it off. sentence_transformers requires the extra. Any registered provider that serves /embeddings (openai, workers_ai, google) is also accepted, resolved through the same descriptor as its model client. An unregistered name fails startup validation.
FELIX_MEMORY_EMBEDDING_MODEL	bge-base-en-v1.5	The local model. For a provider embedder, only an explicitly set value is sent; unset, the provider’s own default embedding model is used.
FELIX_MEMORY_EMBEDDING_DIM	768	Declares the width of the memory_vectors.embedding column, which is vector(768) from the baseline migration. Changing it means re-embedding every row, not flipping a setting.

auto is local only. It never picks a hosted embedder: that would send every stored memory to whichever provider has a key configured, and the column is sized for the local model. Choosing a provider is always an explicit setting. The local weights are fetched on first use, so the first embed after a start is slow and needs outbound access to the model hub.

The default image is built lean, without the embeddings extra, so auto there means no vector channel. Build with --build-arg FELIX_EXTRAS=embeddings to turn it on. On an image that has the extra, set FELIX_MEMORY_EMBEDDER=none to keep full-text-only recall.

Hosted workspaces

By default a workspace scope is a directory under FELIX_WORKSPACE_ROOT on the host. With FELIX_WORKSPACE_BACKEND=hosted each scope instead lives in its own sandbox: a Cloudflare Container (a Firecracker microVM) started with the internet off and an empty environment, reached through a gateway Worker that ships with the harness (deploy/cloudflare/workspace-gateway). Nothing a model writes or runs touches the host.

• The tools answer as they do locally. The file operations run in the sandbox with the local backend’s own code (no symlink followed, ranged reads, edit by rename); a conformance test holds the two backends to identical answers.
• shell_tools run in the sandbox, after the same allowlist and command screening as on the host, and never fall back to the host. The image is slim Debian with Python: a command that is not installed there fails as a missing binary.
• Files persist in R2. A sandbox stops after 10 idle minutes and its disk goes with it, so /workspace is backed up to an R2 bucket when a run that wrote files ends (and before the idle stop), and restored when the sandbox next starts. A restore that fails refuses the call rather than serving the scope empty.
• A thread’s repository is cloned into its sandbox. The person’s token goes to the gateway for the one clone and is added to the sandbox’s requests to github.com outside the container, which may fetch only that repository and only while the clone runs: the sandbox never holds the token and cannot push. The file listing and publish_commits read the repository there; publishing still goes through GitHub’s API. A repository opens only into an empty thread workspace, and removing it destroys the sandbox.
• Not served under hosted, and refused rather than run on the host: an image tool’s path, and publish_commits in a thread with no repository of its own, for any scope but deployment, which stays on the host.
• A scope starts empty on hosted. To carry what local scopes already hold, run felix workspace upload once after turning it on (--dry-run first); the local files stay where they are, and symlinks and files over the write cap are skipped and reported.

To turn it on, deploy the gateway, then point the harness at it:

cd deploy/cloudflare/workspace-gateway          # in the felix-run/felix checkout
cp wrangler.example.jsonc wrangler.jsonc && npm ci
npx wrangler r2 bucket create felix-workspaces
openssl rand -hex 32 > gateway-token            # keep it; it goes to both sides
npx wrangler secret put WORKSPACE_GATEWAY_TOKEN < gateway-token
npx wrangler deploy                             # builds the image: needs Docker

Setting (API and worker)	Value
FELIX_WORKSPACE_BACKEND	hosted
FELIX_WORKSPACE_GATEWAY_URL	the Worker’s https URL
FELIX_WORKSPACE_GATEWAY_TOKEN	the same token, 32+ characters

Boot refuses hosted without an https URL (plain http only for localhost) and a long enough token. Measured on the reference deployment: a sandbox’s cold start took 2.6–11s, a warm operation about 0.5s, and a backup about 2s.

Chat UI / docs Workers

Frontends live in felix-run/web. Deploy chat-ui with vars.FELIX_ORIGIN pointing at your public Felix API; docs is a static Workers site.

Browsers need a same-origin proxy

Felix has no CORS layer, on purpose, so a page on another origin cannot call it directly. A preflight OPTIONS gets a 401 (or a 405 under FELIX_AUTH_MODE=none) with no Access-Control-Allow-* headers, and the browser stops there. A plain GET still reaches Felix, but the page cannot read the response. There is no browser login (no cookies), so a page calling Felix directly would have to hold a bearer token or API key, where any script on the page can read it.

Put a server-side proxy on the page’s own origin instead. It holds the credential, adds Authorization, and forwards to Felix. The chat-ui Worker does exactly this: it forwards /api/* to FELIX_ORIGIN, and when CHAT_UI_KEY is set it first checks the browser’s x-chat-key. An nginx or Caddy location /api/ block that sets the header works the same way.

With GitHub login on, the chat-ui gate leads with Continue with GitHub and keeps the access key behind it (when CHAT_UI_KEY is set). A signed-in browser sends its own bearer, and the Worker forwards it in place of both x-chat-key and FELIX_API_KEY — so each person runs as their org’s tenant rather than as the deployment’s key — but only while GET /auth/methods reports bearer_required. Under FELIX_AUTH_MODE=none any bearer would be accepted, and the Worker refuses to let one past its key there. The session is the harness’s token, so it lasts FELIX_GITHUB_LOGIN_TTL_SECONDS and signing out of the page does not revoke it. The CLI, felix-client, curl and other services are not browsers, so none of this applies to them.

With Google sign-in on, the gate offers Continue with Google: as the card when it is the only way in, or under GitHub’s button when both are. Only the accounts on FELIX_GOOGLE_ACCOUNTS get through. The Worker passes /auth/google/* through as it does GitHub’s routes, and the redirect URI to register in Google Cloud is https://<chat-ui>/api/auth/google/callback. A Google session lasts FELIX_GOOGLE_LOGIN_TTL_SECONDS. The account chip renews it by sending the browser to Google and back to the same page, since Google has no code to enter in place. A durable run carries on at the harness and is rejoined on the way back. A Google session has no GitHub connection, so the chip neither shows one nor tries to revoke one on sign-out.

Where to next

• Getting started — local smoke
• REST API — surfaces
• Root README — lean extras / warehouse