Testing

How the Python harness is tested. Repo: felix-run/felix.

Layout

Suite	Path	Role
Unit	tests/unit/	Schema, patterns, governance, stores (often memory:// DB + object_store=memory), auth modes
Integration	tests/integration/	FastAPI HTTP surfaces (/health, /chat, /v1, …) against a test app
Conformance	tests/conformance/	One contract run against every backend of a store — session, memory, migrations
Smoke	tests/test_smoke.py	Coarse import / build / optional live-model checks
Eval	CLI + felix.eval	Offline datasets; --mock scores rubric without a live model

The conformance suite is the one to know about, because it can pass without proving anything. A store’s contract is written once and run against each backend — the in-memory arm always runs, but the Postgres arm runs only when FELIX_CONFORMANCE_DATABASE_URL points at a reachable database (CI sets it from a service container) and skips otherwise. That skip is a real hole in coverage rather than a pass, which matters because the logic under test exists twice: once for memory:// and once in SQL. Run it against Postgres before trusting it.

No Vitest / miniflare. Run via:

make check          # ruff + ty + pytest
# or
uv run pytest
uv run pytest tests/unit -q
uv run pytest tests/integration -q

Patterns when adding tests

1. Settings — construct Settings(auth_mode="none", allow_insecure=True, database_url="memory://…", object_store="memory") so tests stay hermetic.
2. Stub tools — pass an explicit ToolProvider / in-memory tools; don’t depend on full Compose wiring.
3. Manifests — build with apiVersion: felix/v1 dicts or parse_manifest; don’t require the on-disk bundle.
4. Tenant isolation — assert another tenant cannot read rows you wrote.
5. Async — mark with @pytest.mark.asyncio where the harness APIs are async.

Mock eval

felix eval -d my-dataset -m quick --fixture ./dataset.json --mock

--mock compares against rubric mock_answer / expect fields — CI-friendly without provider keys. Omit --mock to drive the candidate manifest through a real model.

CLI helpers used in CI / ops

Command	Purpose
felix migrate	Alembic upgrade
felix doctor	Read-only config / connectivity checks
felix eval …	Offline eval runs
felix mint-jwt	Self-issued JWT for jwt mode
felix bundle-manifests	Validate bundled YAML

Local stack for integration

make install && make up && make migrate
# API :8080 — see getting started

Lean hosts: make up-lite. Details: getting started, deploy.

Prefer unit tests with in-memory Postgres/object store for governance and pattern loops; reserve integration tests for HTTP contract and auth middleware behavior.