Concepts

Felix is a managed agents harness you run yourself. You describe an agent declaratively; Felix compiles that description into a governed, observable, resumable runtime and serves it over several protocols at once.

The through-line is: the manifest is the program, and everything else is machinery that enforces it.

A run paused on a gated write_file, as chat-ui draws it: the file as it is and as it would be, the deadline after which the harness denies it, and what approving grants. Cropped from a capture against a local harness (v0.11.2); the call was denied. Full window.

Manifest

A YAML or JSON document with apiVersion: felix/v1 and kind: Agent. The schema is Pydantic v2 with extra="forbid", so an unknown key is a hard error rather than a silently ignored typo.

A manifest declares everything an agent needs: pattern, model, system prompt, tools, skills, MCP servers, A2A peers, client tools, memory backend, session strategy, auth requirements, policies, limits, guardrails, screening, approvals, and framework governance.

apiVersion: felix/v1
kind: Agent
metadata:
  name: quick
spec:
  pattern: react
  model:
    id: claude-sonnet
  tools: [calculator]

Every field is catalogued in the Manifest reference.

Where manifests come from

The request-path resolver walks four layers and returns the first hit:

1. Tenant Postgres — the tenant’s active pointer (source: tenant_postgres)
2. Tenant object store — manifests/{tenant_id}/{name}.json (source: tenant_object)
3. Global object store — manifests/{name}.json, affecting every tenant (source: global_object)
4. Bundled — the YAML shipped in the image (source: bundled)

So a tenant manifest named quick shadows the bundled quick for that tenant alone. Active pointers are cached for 30 seconds. Ten manifests ship in the box: quick, assistant, deep, router, oss-only, hybrid-router, support, decider-support, cowork and governed. assistant is quick with long-term memory — see Memory.

contributor and triage, the agents that build Felix itself, are in the image but not bundled. They need the builder stack’s workspace and shell runner, so they live in manifests/self/, and only a deployment that sets FELIX_MANIFESTS_DIR to that directory serves them; deploy/docker/compose.self.yml does. Any directory of manifest YAML works the same way, searched after the bundled set. A directory that is missing, or that holds a name the bundled set also has, is refused at boot rather than letting one shadow the other.

FELIX_MANIFEST_SOURCE=bundled collapses that to one layer: only the manifests baked into the image are served, no manifest store is constructed, and the write routes are not registered — so the verbs are absent from the app and from /openapi.json rather than present and refusing. It is the right posture for a deployment that never authors a manifest at runtime. Two things change when you flip it: every tenant falls back to the image’s file, dropping any per-tenant auth.inbound tightening, and threads pinned with governance.pin_compile see one 409 as the resolved version becomes null.

Writes go through the Management API, which is append-only and version-pinned: publishing inserts a new version row and flips a pointer, and rolling back flips the pointer again rather than rewriting content.

Canary rollouts

An active pointer can carry a canary_version and a canary_weight. Selection is deterministic, not random — the harness hashes tenant, thread, manifest name and both version numbers, then takes that modulo 100. A thread therefore stays on one side for the whole rollout instead of flapping between versions mid-conversation.

Compile pinning

With spec.governance.pin_compile, the manifest version a thread started on is frozen for that thread. If the active pointer moves underneath a live conversation, the next request fails with 409 rather than silently switching behaviour mid-run.

Tenant

Every row Felix writes is scoped to a tenant id, taken from the authenticated principal — never from a request field, so a caller cannot address another tenant’s data. Under FELIX_AUTH_MODE=none everything is the anonymous principal in tenant default.

Postgres row-level security is available as opt-in defence in depth. The policies themselves are not optional to install — 0006_tenant_rls is part of the migration chain, so any database at that revision or later has them. FELIX_DATABASE_RLS is the runtime half: set it true and the tenant id is applied per transaction and the policy enforces it; leave it false and Felix declares a bypass, so the policies are inert and query-layer scoping remains the isolation.

One caveat is worth checking rather than assuming, because it fails silently in the direction that matters: the policies are FORCEd, and a superuser or BYPASSRLS connection skips them entirely — so RLS can be switched on and enforcing nothing while everything appears to work. felix doctor reports whether it is actually in effect.

Thread

A thread is one conversation. Its id is always {tenant_id}:{suffix}, where clients supply only the suffix — a suffix containing : or # is rejected with 400 invalid_thread_id.

Session

The session is the append-only event log behind a thread. It lives outside the model’s context window: the log is the durable record, and a strategy decides what the model actually sees each turn.

Event kinds: message, tool_call, tool_result, thinking, audit, compaction, model_change, thinking_level_change, branch_summary, custom, label, session_info.

Strategies (spec.session.strategy):

Strategy	Behaviour
full_replay	Send the whole transcript. The default.
windowed:N	Send the last N events.
summarizing:N	Summarize everything older than the last N.
semantic:N	Retrieve the N most relevant events (needs the embeddings extra).
compacting	Compact on a token threshold, keeping keep_recent_tokens intact.

Because the log is authoritative and server-side, a client is a view, not the source of truth. It hydrates from GET /chat/sessions/{id}, whose snapshot carries the transcript, phase, thinking level, leaf, lease state and any queued steering.

A session moves through phases idle, turn, compaction, branch_summary, retry and aborted.

Branching

A thread is a tree, not a line. Fork copies a session into a new thread up to a chosen event. Rewind moves the active leaf back to an earlier event in the same thread; unless you opt out, the abandoned branch is summarized into a branch_summary event so the model does not lose what happened there.

Editing a sent message is the same move followed by a send. Chat UI rewinds the leaf to the event the message continued from, then sends the new text, so the edit becomes a sibling of the original. The model sees the conversation as if the original had never been sent, while the original and its replies stay on the session on the other branch, which is why the edit can be undone.

Leases

Two browser tabs on one session would race. Each client mints a holder id and takes a lease. One client holds it exclusive and drives the thread (reported as locked); any number hold it shared and watch read-only (reported as attached), each with a token of its own. Contention returns 409 for an exclusive acquire only: a shared one always succeeds, with held_by_other: true when someone else is driving. Observers never block an exclusive acquire and are never promoted, so a watcher that wants to drive takes the exclusive hold itself once it is free. A client that presents its token on a write (X-Felix-Lease-Token) is refused unless it is the driver’s — see Leases. Leases are Redis-backed, so they hold across replicas.

Pattern

The pattern is the control loop. Felix ships seven:

Pattern	Kind	Behaviour
react	single	Think → call tools → observe → repeat. The workhorse.
deep	single	Plans first, records steps as a plan document, then executes.
reflect	single	Generates, critiques against criteria, and retries below threshold.
plan_execute	single	Separate planner and executor models.
router	multi	Classifies the request and delegates to one sub-agent.
parallel	multi	Fans out to sub-agents and aggregates.
groupchat	multi	Sub-agents converse to a joint answer.

The registry is open — register_pattern(name, builder) adds your own, and nothing in core enumerates the built-in list. Multi-agent patterns require sub_agents and reject the direct outbound blocks (peers, containers, queues, sandboxes, browser_tools); a sub-agent’s own manifest declares those instead. See Patterns.

Skill

An Agent Skill: a SKILL.md with YAML frontmatter (name, description, and optionally license, compatibility, metadata, allowed-tools) and a Markdown body, plus optional bundle files under references/, scripts/, assets/ and evals/.

Skills are disclosed progressively. The compiled system prompt carries only the catalog, as an <available_skills> block of names and descriptions. activate_skill returns a skill’s body to the model as instructions, and lists its bundle files; read_skill_file(name, path) reads one (text only, at most 64 KiB, with a truncated flag). deactivate_skill switches one off. Activation is per (tenant, manifest) and only reaches skills in the manifest’s catalog.

Where skills come from

Source	Where	source	Written by
Bundled	the harness’s skills/ directory	bundled	the release
Host directory	FELIX_SKILLS_DIR	bundled	the operator
Operator upload	object-store keys skills/{tenant}/{name}/SKILL.md or skills/{name}/SKILL.md, and pinned …/{name}/{version}/SKILL.md	store	the operator, outside Felix
Tenant library	the live version of each skill in the tenant’s library, under skill-library/	library	an agent, an operator, an import from GitHub, or a promotion from a personal library, through review
Personal library	the live version of each skill in the caller’s own library, under skill-library/{tenant}/~{digest}/	library	the caller, or an agent under spec.personal_skills: write

A catalog is built in this order, and the first source to hold a name answers it:

1. The host wins. Bundled skills, then FELIX_SKILLS_DIR (a skill there replaces a bundled one of the same name). The library refuses to save a host name, or a name an operator uploaded at an unversioned key (409 name_shadows_host_skill).
2. The caller’s own library, live versions only, where the manifest sets spec.personal_skills: read. A caller with a verified identity has a personal library; an anonymous caller, or an API key with no subject of its own, has none. A personal skill shadows a tenant skill of the same name for that caller alone, and only an ambient one: a name the manifest declares in spec.skills always resolves from the tenant’s library. In the web UI it is My skills on the Skills page (?lib=me): the same list, editor and version actions as the team library, published by its owner through the same gate with no review queue — writing needs skills:personal — and a version once published can be promoted into the tenant library as a draft for its review queue. An administrator finds everyone’s under Personal libraries on the team library, listed only when they ask because each listing is audited, and opens one read-only (?lib=<digest>): readable and archivable, with no editor, decision, rollback or promotion.
3. The tenant library, live versions only. Without spec.skills_declared_only, every live library skill joins the catalog of every manifest in the tenant that builds one (any manifest with spec.skills, a skill tool, or spec.skill_authoring); with it, only declared names do.
4. Operator uploads, for a declared name nothing above holds: pinned key, then unversioned, the tenant’s keys before the shared ones.

One exception to library-before-uploads: an explicit version pin to an operator upload beats the library. A spec.skills ref with version: 0.1.0 is served skills/{tenant}/{name}/0.1.0/SKILL.md (or the shared skills/{name}/0.1.0/SKILL.md) when it exists, whatever the library’s live version is. A pin no upload holds still gets the library’s live version. The library reports shadows_operator_upload on a skill whose name is split this way.

A personal library belongs to one principal within one tenant, and only its owner’s turns load it. The owner saves and publishes in it through /skill-library/~me with the skills:personal scope, through the same publish gate as the tenant’s library but without its review queue; an agent writes to it under spec.personal_skills: write. Promotion proposes a version the owner has published to the whole tenant: its files are copied, byte for byte, into a draft of the tenant’s skill of the same name (source: promoted) that waits in the tenant’s ordinary review queue. Promotion never publishes, and the draft is judged as strictly as an agent’s draft: the copy rule runs on it, only its bundle’s own evals/ count toward an evaluation requirement, and it may not add, change or drop an evals/ file. The personal skill stays as it was.

Skills an agent writes

A manifest with spec.skill_authoring.enabled gets three more tools:

Tool	Arguments	Does
create_skill	name, description, body, reason	Saves a new skill to the tenant library at 0.1.0 (to the caller’s own under spec.personal_skills: write)
update_skill	name, body, reason, parent_version, description?	Saves a new version with a new body; every other file is kept from the parent
submit_skill_feedback	name, body, suggested_patch?	Files feedback on a library skill in this agent’s catalog; changes nothing

parent_version is required and must be the skill’s newest version that was not a rejected draft (newest_version in list_skills and activate_skill). A stale one is refused parent_changed; naming a rejected draft is refused parent_rejected. Because it is an argument, an approval of the call binds it. Every refusal comes back to the model as {"error": …}.

Drafts are the default. activate_skill hands a body to the model as instructions — a higher trust tier than recalled memory, which is fenced as reference. A skill an agent wrote from a turn that carried injected tool output would otherwise steer every later session in the tenant. So a save is an immutable draft that no catalog loads, and a person publishes it.

1. Draft — the save is validated, quality-reviewed (a 0–100 score) and security-scanned (pass, advisory or fail), and written as a new semver version. max_pending caps one manifest’s undecided drafts.

2. Review — an operator reads it in the review queue (GET /skill-library/-/review) and publishes or rejects it with a note.

3. Publish — the publish gate re-reads the bytes against their saved digests, re-validates and re-scans them, and applies the publish policy. The version becomes live_version; the one it replaces is archived. A rollback makes a once-published version live again.

spec.skill_authoring.mode: publish skips step 2: the draft is published at once if the gate passes, and stays a draft (with the reasons) if not. The manifest is refused unless an approvals rule gates every create_skill and update_skill call, so a person still reads each skill before it is saved. An agent’s edit of a version an operator wrote, imported or promoted waits for review in either mode, as does an edit built on such a version through drafts nobody has decided yet.

Skills from GitHub

An operator can import a skill published on GitHub into the library (felix skills add github:owner/repo/path). The import is a draft pinned to one commit of the repository’s own history, and it is published like any other, after review. Nothing resolves a GitHub ref when a manifest compiles.

Imported text is a third party’s, so it stays marked as such: the version and everything built on it carry lineage_import until an operator adopts a version, the publish gate blocks it on an advisory scan whatever the policy says, the catalog and list_skills mark it untrusted, and what the skill tools relay of it is content-screened — at least by the injection markers even where the manifest turns screening off. A deployment allowlists the repositories each tenant may name, and can hold back files Felix first saw too recently. Skill import has the whole surface.

Evaluations and feedback

An evaluation asks whether answers get better with the skill. The worker answers each scenario twice with FELIX_SKILL_EVAL_MODEL, without the skill and with its body fenced as untrusted reference; FELIX_SKILL_EVAL_JUDGE_MODEL scores both 0–100, and uplift is with-skill minus baseline. Scenarios come from the bundle’s evals/, else are generated from the SKILL.md, else are three defaults built around the description. The publish policy can require a succeeded evaluation (require_eval) and a minimum uplift (min_eval_uplift). Each evaluation says whether it counts_for_gate: for a version an agent wrote, only one on the bundle’s own evals/ scenarios counts, because generated and default scenarios are written from the agent’s own text. skill_authoring.auto_eval queues one for every draft the agent saves.

Feedback is filed by an agent (submit_skill_feedback) or an operator, on one version, and waits as pending. Only a person decides it. Accepting with improve (the default) queues an AI rewrite: the worker asks FELIX_SKILL_IMPROVE_MODEL for a revised SKILL.md and saves it as a draft for review. Nothing in the loop publishes. Evaluations and improvements run in the worker’s skill_jobs task, so they need felix-worker and felix-scheduler running.

chat-ui’s Skills page is the operator surface over all of this: the library list, a review queue and feedback inbox, the tenant’s publish policy, and for each skill its versions with diffs, the gate’s verdict, evaluations and feedback. Every publish confirm states the gate’s verdict before it asks. A skill is edited in the side pane, beside the skill’s own page or a conversation: Edit on its page, Edit to fix on a draft the gate refused, and New skill all open it there, and an unsaved edit survives switching tabs or moving between pages. A create_skill or update_skill result renders in the chat as a proposal card with Publish, Reject and Edit. The routes underneath are in the Management API.

chat-ui’s View setting (in the header menu, or /view) has three levels. Everyday is the conversation and what waits on you. Harness view adds the harness pages — Skills, Activity, Memory and the rest — to the sidebar, along with the run instrument, the context meter and each reply’s token counts. Harness + debug (or /debug) adds the instrument’s Debug tab, the event ids under each turn and tool card, and tool input and output left open. In Everyday, the sidebar shows a Skill drafts row only while a draft is waiting for review, which links straight to this page.

Tool

Every tool, wherever it runs, is compiled to the same interface and tagged with a transport that shows up in audit rows:

Transport	Runs where	Declared by
local	In the API process	spec.tools
mcp	A remote MCP server	spec.mcp_servers → {server}__{tool}
a2a	Another Felix peer	spec.peers → peer__{name}
container	An HTTPS container gateway	spec.containers
sandbox	A Docker sandbox (sandbox extra)	spec.sandboxes
browser	Playwright (browser extra)	spec.browser_tools
queue	A separate consumer, via a Redis list	spec.queues
client	The caller’s own machine	spec.client_tools

Tool failures come back as strings the model can read and react to, prefixed [error/invalid_arguments], [error/timeout], [error/blocked] or [error/internal]. A ref with fatal: true aborts the batch instead.

Client-executed tools

spec.client_tools inverts the usual direction: the harness emits a tool_request frame and blocks, and the browser or CLI runs the tool and answers with POST /chat/tool_result. This is a real round trip inside the model loop — a client that never answers stalls the run until the tool’s timeout. It is how the web clients run local shell commands and open local files against a workspace the server cannot see.

Queue tools

A queue tool pushes onto a Redis list and returns immediately; a separate consumer does the work and writes the result back with POST /internal/sessions/{session_id}/events, authenticated with X-Felix-Consumer-Secret. Use it for work that outlives a request.

Governance

Every tool is cloned through a fixed wrapper stack before the model can call it. The order is load-bearing and enforced by tests:

secret masking → policies → command screening → content screening → limits → guardrails → judges → approvals → permission mode → artifact spill

Layer	Manifest block	Question it answers
Secret masking	—	Is a resolved credential about to leak into output?
Policies	spec.policies	Does the caller hold the scopes this tool requires?
Command screening	spec.command_screening	Is this command allowed, denied, or does it need a human?
Content screening	spec.content_screening	Is this tool output trying to hijack the model?
Limits	spec.limits	Has this run exceeded its budget?
Guardrails	spec.guardrails	Does the text match a blocked pattern (e.g. PII)?
Judges	spec.guardrails.judges	Does a model score this as acceptable?
Approvals	spec.approvals	Has a human said yes?
Permission mode	spec.permissions	Does this conversation’s mode (plan, accept edits, bypass) let it run, or waive its approval?
Artifacts	spec.artifacts	Is this output too large to put in context?

A denial is stamped with a module-private marker, so a model cannot forge one by emitting a convincing-looking string. Governance fails closed — a missing store or an internal error denies rather than allows. See Governance.

spec.governance.frameworks additionally maps a manifest onto soc2 or eu_ai_act requirements and refuses to compile without the corresponding controls. That is a compile-time check, not a certification.

Memory

Two independent things share the word.

Session memory is the transcript, above — the checkpointer (spec.memory.checkpointer, Postgres) plus a strategy.

Durable memory is facts that outlive a thread. With spec.memory.capture.enabled, a small model extracts facts from each turn into memory_vectors (pgvector, HNSW) under kinds fact, event, instruction and task. Stored facts reach a ReAct agent as a prelude at the start of each run, and spec.memory.recall.tools adds remember, recall, forget and list_memories. spec.procedural_memory recalls stored how-tos and adds a remember_procedure tool.

Durable memory is keyed by tenant and manifest, not by caller: everyone who talks to one manifest in one tenant shares its facts. That is why quick stays stateless and the bundled assistant manifest — quick plus capture, recall tools, content screening and an approval on forget — sets auth.inbound.allow_anonymous: false. Under FELIX_AUTH_MODE=none every caller is anonymous, so assistant answers 401; call it with an API key or JWT.

Recall fuses a full-text channel, topic keys and a vector channel. The vector channel runs when FELIX_MEMORY_EMBEDDER resolves to an embedder: the default auto uses the local bge-base-en-v1.5 model when felix-harness[embeddings] is installed and runs without the vector channel otherwise. See Embeddings. Semantic session strategies and tools_retrieval also need the embeddings extra.

Durable execution

By default a run lives and dies with the request. With spec.execution.mode: durable, POST /chat enqueues a fiber and returns 202 with a resume_token; the fiber checkpoints into Postgres and a worker picks it up within about a second and resumes it after any restart, so a process restart does not lose the run. Poll GET /chat/runs/{resume_token} for the result, or name endpoints in spec.execution.webhooks and the worker POSTs the finished run to them, signed.

POST /chat/stream on a durable manifest streams the run — run_accepted, run_status, final — and a dropped client reattaches with the resume token rather than losing the run.

Steering

A run is not a black box once it starts. POST /chat/steer with kind: "steer" interrupts the remaining tool batch mid-run — or, sent while the thread is idle, is held and delivered at the start of its next run; kind: "follow_up" queues a message delivered once the run goes idle. POST /chat/abort stops it, and POST /chat/continue picks it back up with no new user message. The queues are Redis lists, so any replica can accept the steer.

Neither queue can be read back or withdrawn once a message is in it, so the web client does not put one there by default. A message written while a run is going waits in the tab, above the composer, and is sent as the next turn when the run finishes. Until then it can be edited, reordered or removed, or sent into the run as a steer from its own row. Stopping a run, or a run that fails, pauses that queue, and nothing more is sent until you resume it.

Interrupts

Three things pause a run until something answers out of band: an approval_required frame waiting on a human decision, a tool_request frame waiting on the client, and a ui_request frame asking the user to choose, confirm or type something. Each has a timeout and fails closed. See sticky interrupts. The card at the top of this page is an approval_required waiting on a decision, as chat-ui draws it.

Surfaces

The same compiled agent is reachable four ways, which is the point of compiling from a manifest rather than wiring a framework by hand:

Surface	Entry point
Native REST + SSE	POST /chat, POST /chat/stream
OpenAI-compatible	POST /v1/chat/completions (model = manifest name)
A2A JSON-RPC	POST /a2a
MCP	POST /mcp

Auth

Three modes, set by FELIX_AUTH_MODE:

• none — everything is anonymous in tenant default, and management scope checks are skipped. Legal only on a loopback bind; the harness refuses to start otherwise, in every environment.
• api_key — Authorization: Bearer, Authorization: ApiKey or X-Api-Key, compared in constant time against FELIX_AUTH_API_KEYS, which supplies the subject, tenant and scopes.
• jwt — verified against FELIX_JWT_VERIFIERS and FELIX_JWKS_PUBLIC. Mint tokens locally with felix mint-jwt.

A manifest adds its own inbound requirements through spec.auth.inbound — allow_anonymous and required_scopes are enforced per request. Management routes layer scopes on top; see Scopes. A second credential, X-Felix-Consumer-Secret, gates POST /internal/*.

Details in Auth.

Observability

Prometheus metrics at GET /metrics, counters prefixed felix_*. Audit rows are buffered by the process that emits them and flushed to Postgres on a timer plus a drain on shutdown — the agent loop runs in the API process, so the API flushes too, not only the worker. Optional OTLP tracing with FELIX_OTEL_ENABLED, and an optional append-only analytics spill (FELIX_WAREHOUSE: DuckDB, ClickHouse or Doris).

See Observability.

Three seams

• Session — append-only events with pluggable strategies (full_replay / windowed / summarizing / semantic / compacting).
• Pattern / model registries — open registration for loops (react, deep, …) and providers (anthropic, openai, workers_ai, groq, …), with the model layer in its own package that cannot import the harness.
• ToolExecutor — local / mcp / a2a / container / queue / sandbox / browser / client, wrapped by governance.

Request lifecycle

Putting it together, one POST /chat:

1. Middleware — request id, then body limit (1 MiB; 12 MiB on the two skill-library routes that carry a whole bundle), then per-tenant rate limit, then authentication. All four are pure ASGI, so a streamed response is not copied between layers.

2. Resolve — walk the four manifest layers, pick the canary variant, check the compile pin.

3. Authorize — apply spec.auth.inbound; screen the inbound user turn.

4. Compile — build_agent resolves the prompt and sub-agents, binds outbound tools, wires skills and memory, and wraps every tool in the governance stack.

5. Run — the pattern loop calls the model and dispatches tools, emitting audit rows and stream frames, and pausing on any interrupt.

6. Persist — append events to the session log; buffer audit and usage for flush.

For the mechanism at each step, start at Architecture.