Testing

How the Python harness is tested. Repo: felix-run/felix.

Layout

Suite	Path	Role
Unit	tests/unit/	Schema, patterns, governance, stores (often memory:// DB + object_store=memory), auth modes
Integration	tests/integration/	FastAPI HTTP surfaces (/health, /chat, /v1, …) against a test app
Conformance	tests/conformance/	One contract run against every backend of a store — session, memory, manifests, approvals, eval, documents, usage, fibers, retention, RLS, migrations
Smoke	tests/test_smoke.py	Coarse import / build / optional live-model checks
Eval	CLI + felix.eval	Offline datasets; --mock scores rubric without a live model

The conformance suite is the one to know about, because it can pass without proving anything. A store’s contract is written once and run against each backend — the in-memory arm always runs, but the Postgres arm runs only when FELIX_CONFORMANCE_DATABASE_URL points at a reachable database and skips otherwise. That skip is a real hole in coverage rather than a pass, which matters because the logic under test exists twice: once for memory:// and once in SQL. CI sets FELIX_CONFORMANCE_REQUIRE_POSTGRES=1 so a missing database fails the job instead of silently skipping half of it.

FELIX_CONFORMANCE_DATABASE_URL=postgresql+psycopg://user:pass@localhost:5432/db \
  ./scripts/test.sh tests/conformance -q

Every store that has gained a Postgres arm so far has produced a real divergence between the two copies, which is the argument for adding the rest. The eval store’s was the worst: put_dataset overwrote an existing item on the in-memory twin and raised UniqueViolation on Postgres, so the second run of any eval failed on every real deployment while the suite stayed green. Adding a backend to BACKENDS in a contract file makes it inherit every assertion in that file.

Caution

Run tests with ./scripts/test.sh, never a bare uv run pytest. The repo’s .env points FELIX_DATABASE_URL at a real Postgres and pydantic-settings reads it, so a bare run fails every database-touching test with a connection error that looks like a code bug. The script exports the in-memory environment — FELIX_DATABASE_URL=memory://ci, FELIX_OBJECT_STORE=memory — and blanks every vendor credential so a mis-routed model call cannot bill anyone. It is what make test and CI both run, and it forwards its arguments to pytest.

No Vitest / miniflare. Run via:

make check                                          # ruff + ty + pytest with the coverage floor + format check
make check-ci                                       # + every other gate CI runs without infrastructure
./scripts/test.sh                                   # the full suite
./scripts/test.sh tests/unit/test_react_loop.py -q  # one file
./scripts/test.sh -k compact                        # one theme

ty needs the optional extras (make install-full). On a lean install make check skips it, says so as its last line, and still exits 0; make check-ci and STRICT=1 make check fail instead.

Patterns when adding tests

1. Settings — tests/support/factories.py:make_settings(**overrides) builds the hermetic memory:// baseline (no inbound auth, in-memory object store); name only the fields the test cares about. Shared helpers live in tests/support/, and a test never imports another test module or a conftest.
2. Stub tools — pass an explicit ToolProvider / in-memory tools; don’t depend on full Compose wiring.
3. Manifests — build with apiVersion: felix/v1 dicts or parse_manifest; don’t require the on-disk bundle.
4. Tenant isolation — assert another tenant cannot read rows you wrote.
5. Async — mark with @pytest.mark.asyncio where the harness APIs are async.

Mock eval

felix eval -d my-dataset -m quick --fixture ./dataset.json --mock

--mock compares against rubric mock_answer / expect fields — CI-friendly without provider keys. Omit --mock to drive the candidate manifest through a real model.

CLI helpers used in CI / ops

Command	Purpose
felix migrate	Alembic upgrade
felix doctor	Read-only config / connectivity checks
felix eval …	Offline eval runs
felix mint-jwt	Self-issued JWT for jwt mode
felix bundle-manifests	Validate bundled YAML
felix validate-manifest <file>	Validate one manifest as PUT /manifests would, before it is stored
felix chat	Chat with a manifest on a running server, one line at a time (--url, --manifest, --api-key/FELIX_API_KEY, --stream); make cli runs it

Local stack for integration

make bootstrap && make up            # the whole stack in Docker; it migrates before the API starts
make db migrate dev                  # or: Postgres + Valkey in Docker, the API from the checkout

Lean hosts: make up-lite. Details: getting started, deploy.

Tip

Prefer unit tests with in-memory Postgres/object store for governance and pattern loops; reserve integration tests for HTTP contract and auth middleware behavior.