Observability — counters, audit events, alerts

What Felix emits and where to look. Complements governance. Code: felix.observability + felix.audit in felix-run/felix.

Signal paths

Path	Sink
Counters / histograms	Prometheus (record_counter / record_histogram); scrape /metrics on the API, and FELIX_METRICS_PORT on the worker
Spans	In-process + optional OTLP when FELIX_OTEL_ENABLED=true (uv sync --extra otel)
Logs	Optional OTLP with FELIX_OTEL_LOGS=true, stamped with the trace that produced them
Audit events	Buffered → Postgres audit_events (worker flush); query via GET /audit

/metrics is authenticated, and not by accident

Label values include tenant-supplied manifest ids and remote MCP tool names, so an anonymous scrape would disclose every tenant’s manifests and tools. It is not in the public-path set and it is not rate-limit exempt. A scraper needs a credential — give it an API key with an empty scopes array: /metrics has no scope gate, so an unscoped key reads metrics and is refused everywhere else.

The worker’s FELIX_METRICS_PORT endpoint carries the same label values with no authentication, because the worker has no auth middleware. Keep it on an internal network; never publish it.

Counters are the alert signal. Audit rows are the trail after a threshold trips.

Counters

Every counter is created on first use and named felix_*. A manifest’s spec.observability.metrics allowlist is applied before the series is created, so a counter left out of a non-empty allowlist is never registered at all.

Counter	Labels	Incremented
felix_tool_calls	transport, status (ok / error / denied), manifest_id, error_code on the exception path	Once per tool dispatch
felix_tokens	manifest_id, model, kind (input / output)	By the token count of each model call
felix_policy_deny	manifest_id, tool, policy	A spec.policies rule denied the call
felix_secret_masking	manifest_id, tool	A resolved credential was stripped from tool output
felix_approval_required	manifest_id, tool, rule	A call hit an approval gate (rule: command for command screening)
felix_content_screening	manifest_id, tool, action	Post-call screening acted (action is the applied on_flag, or unavailable)
felix_inbound_screening	manifest_id, surface (turn / tool_arguments), action	Screening acted on the way in — the counterpart to the row above, and the one a tool-shaped panel misses
felix_control_unavailable	control	A named control could not run and was not treated as a pass
felix_control_degraded	control, reason	A control fell back to a weaker implementation (e.g. PII regex fallback)
felix_model_switch	from, to, reason (provider_error / low_confidence)	A fallback or confidence escalation fired
felix_decisions	decider, purpose, outcome (ok / error / invalid)	Once per call to a decision model; invalid means it answered, was billed, and the answer did not fit the question. Its tokens are in felix_tokens under model=<decider id>
felix_tool_selection	method (decider / unscored / unsure / error)	tools_retrieval.decider chose a shortlist, or fell back to the default ranking. A rising unsure share means top_k is too small for the catalogue
felix_router_choice	manifest_id, method (decider / decider_unsure / decider_error / llm / unmatched)	A router picked its sub-agent. Watch unmatched: the classifier named no child and the request went to the first one
felix_escalation_check	method (decider / error)	confidence_escalation.decider judged a reply, or fell back to the heuristic
felix_skill_suggestion	outcome (suggested / none / error)	spec.skill_suggestion ran for a turn
felix_model_retry / felix_model_retry_skipped	provider, plus status / reason	Provider call retried, or deliberately not retried (reason: quota)
felix_context_overflow_recovered	manifest_id, reason	The loop recovered from a context-window overflow
felix_run_stop_reason	manifest_id, reason	A run ended on anything other than a clean completion
felix_interrupted_tool_calls	manifest_id	A tool batch was cut short by a steer or abort
felix_browser_egress_blocked	reason	A browser tool was refused an outbound host
felix_egress_blocked	reason	An outbound request was refused by SSRF / egress policy
felix_model_timeout	provider	FELIX_MODEL_TIMEOUT_SECONDS elapsed on a provider call
felix_model_unmetered	manifest_id, model	A turn reported no usage — see the warning below
felix_model_unpriced	manifest_id, model	The turn was metered, but the model it was priced by has no entry in the catalog — see the warning below
felix_policy_unsatisfiable	manifest_id	A policy can never pass
felix_rule_targets_nothing	manifest_id, rule, kind	A rule matches no tool
felix_untrusted_tools_unscreened	manifest_id	Untrusted tool output reached the model without content screening
felix_imported_skills_unscreened	manifest_id	A compile offered an imported skill with content screening off, so only the injection markers screen it
felix_worker_task	task, status	Once per periodic worker sweep
felix_webhook_delivery	kind (run / skill_update), endpoint, outcome (delivered / retry / dead)	One signed webhook attempt: a finished durable run’s completion webhook, or a queued skill.update_available. dead is out of tries, or an endpoint no longer registered for — or, for skill_update, no longer bound to — the tenant. kind arrived with skill update notifications: a query that ignores it counts both, {kind="run"} counts runs alone. Watch dead: nobody waiting on that webhook heard about it
felix_skill_update_notification	outcome (enqueued / superseded)	A skill.update_available event before it is sent: enqueued when a check queues one, superseded when one never will be — a newer digest replaced it, the origin moved on, or the skill’s newest version already holds it
felix_buffer_dropped	buffer	Audit or usage rows dropped because a buffer hit its cap

Histograms

Histogram	Labels	Observes
felix_model_call_seconds	model, status	One provider call. A fallback chain contributes one observation per attempt
felix_tool_call_seconds	transport, manifest_id	One tool dispatch
felix_worker_task_seconds	task	One periodic sweep

felix_tool_call_seconds is labelled like felix_tool_calls and deliberately not by tool name: MCP tool names arrive from a remote server and are unbounded label values.

Two counters mean a control is not working

felix_model_unmetered — the turn counted against no budget, so limits.max_cost_usd and the token limits failed open for it. Usually a streamed response missing stream_options.include_usage.

felix_model_unpriced — the tokens were counted but the spend was not. The model has no entry in the pricing catalog and no spec.model.price override, so the row’s cost_usd is 0 while real money was spent: the token caps still bite, limits.max_cost_usd fails open, and anything summing /usage is reporting a floor rather than a total. A custom or self-hosted route is the usual cause. The fix is an override, not a dashboard note.

felix_buffer_dropped — audit or usage rows are gone from Postgres and the warehouse both. Silent data loss; nothing else reports it.

Watch felix_control_unavailable and felix_untrusted_tools_unscreened for the same reason.

There is no counter for limit breaches, judge outcomes, anomaly scans or auto-rollbacks. A limit or guardrail denial is visible as felix_tool_calls{status="denied"} and as a policy_deny audit row.

A zero on a governance panel is ambiguous

spec.observability.metrics is applied before the series is created, so a tenant that sets a non-empty allowlist suppresses every counter it does not name — including the governance-gap counters an operator is told to watch. Zero can mean “nothing happened” or “this tenant opted out of telling you”.

Spans

Span attributes follow the OpenTelemetry GenAI semantic conventions where one exists, so any OTLP backend renders a model call as a generation with token usage attached rather than as an anonymous timing bar.

Span	When	Key attributes
HTTP request	Per request, via opentelemetry-instrumentation-fastapi — the trace root	Standard HTTP semconv
manifest	One per compile	manifest_name, manifest_version
chat {model}	One per provider call	gen_ai.operation.name, gen_ai.system, gen_ai.request.model, gen_ai.response.model, gen_ai.usage.input_tokens, gen_ai.usage.output_tokens, gen_ai.usage.cache_*_input_tokens, felix.cost_usd, felix.model.route, felix.manifest.id
tool {name}	One per tool call	gen_ai.tool.name, gen_ai.operation.name=execute_tool, felix.tool.transport, status
worker {task}	One per periodic sweep	felix.worker.task

A fallback chain that tries two providers emits two chat spans: the span wraps the leaf client, not the composite, so a retry is visible rather than collapsed into one bar.

felix.usage.unmetered=true on a chat span is the span-level twin of felix_model_unmetered.

Sessions and identity

session.id and gen_ai.conversation.id carry {tenant}:{thread_id} whenever the request has a thread — namespaced so threads cannot collide across tenants. This is what makes a multi-turn conversation one session in a backend instead of N unrelated traces.

FELIX_OTEL_CAPTURE_IDENTITY (default true) additionally emits user.id / gen_ai.user.id from the auth principal and felix.tenant.id. Set it false when principal subjects are personal identifiers you would rather not export. The anonymous principal is never recorded — it is a default, not a person.

Prompts and completions are not captured

FELIX_OTEL_CAPTURE_CONTENT=true opts in, writing gen_ai.input.messages and gen_ai.output.messages (truncated at 32k characters). It defaults to off because a tracing backend is an egress destination, and span attributes do not pass through the governance content screening that guards tool output.

Know what the redaction covers

Captured content passes through redact_json — the same masking the audit path uses. That replaces values Felix knows are secrets (those hydrated from the secrets backend) by substring match. It is not pattern detection, so a credential a user types into a chat is exported verbatim.

Probe endpoints are not traced

/health, /live, /ready and /metrics are excluded, as are the ASGI http send / http receive child spans. Docker healthchecks and a Prometheus scrape run every 10–15s forever: left in, they accounted for 73% of traces and 59% of observations in a real export, against under 1% for agent activity. The request span still measures the whole request.

Audit events

Every row carries tenant_id, ts, event_type, manifest_id, principal_subj, status and payload_json. Seven types come from a turn — four by the agent loop, three by the governance wrappers on either side of it — and the skill tools and the skill library write the skill events below:

event_type	Emitted	status	Payload
user_input	Once per caller turn	ok	user_input (first 2000 chars), thread_id
tool_call	Per tool invocation that was not denied	ok, error	tool, tool_call_id, thread_id; error_code on an error row
policy_deny	Per tool invocation a governance wrapper or a plugin’s before_tool hook denied	denied	tool, tool_call_id, thread_id, control; hook when control is hook
final_response	Once per completed turn	ok, error (a fatal tool error, or the last round of tool calls had a refusal)	thread_id, chars, denied_calls
inbound_screening	Per screening decision on the way in	denied, quarantined, redacted, oversize, unavailable	surface (turn | tool_arguments)
guardrails_reply	PII matched in the answer	blocked, redacted	engine
judge_deny	A final_response judge scored under threshold	denied	judge, score, threshold

A run that recovered from a refusal still ends ok

final_response.status says whether the run ended on a refusal: one refused in an early round that went on to finish reads ok. payload.denied_calls counts every refused call across all rounds, so ok with a non-zero count is a run that met a refusal and recovered, and ok with 0 is one that never did. Each refusal also has its own policy_deny row.

policy_deny is not only policies — and is not every denial

The name is narrower than the behaviour in one direction and wider in the other. The loop emits policy_deny whenever the tool wrapper stack returns a deny marker — policies, command or content screening, limits, guardrails, tool-targeted judges, or a pending approval all land here. payload.control names which one — one of policy, limits, guardrails, approvals, command, screening — so GET /audit?event_type=policy_deny filtered on it answers “which layer denied” without correlating counters; a row from a harness older than this field carries no control key at all. Read payload.tool alongside it to know which tool.

What does not land there is anything decided outside a tool call: a turn refused on the way in is inbound_screening, and an answer blocked on the way out is guardrails_reply or judge_deny. An incident query filtered to policy_deny alone sees none of those three.

Skill events

skill_activation is written by activate_skill and deactivate_skill inside a turn. The rest are written by the skill library for work done outside one — a management route acting for an operator, or a worker job — so principal_subj is the operator, the authoring manifest’s id for an agent’s save, or skill-improver / skill-eval for the worker. The version events (skill_draft_saved through skill_archived, skill_adopted and skill_promoted included) carry the version’s origin_manifest_id as manifest_id — empty for an operator’s save or a promotion — and their payloads name skill, version, source, author, quality_score, security_status and library: org, or a personal library’s digest (~ and 32 hex characters), never the owner’s subject. Feedback events name skill, target_version, feedback_id, source and author, with the authoring manifest as manifest_id for an agent’s feedback.

event_type	Emitted	status	Payload adds
skill_activation	activate_skill / deactivate_skill	ok, unknown_skill	action, skill, thread_id, tool_call_id
skill_draft_saved	Every save, agent, operator, import or promotion (reason imported from <source>@<commit>, or updated from … for an update)	ok	parent, reason (first 200 chars, redacted), shadows_operator_upload, principal
skill_published	A publish	ok, blocked	previous, or reasons when blocked
skill_rolled_back	A rollback	ok, blocked	previous, or reasons when blocked
skill_rejected	A draft rejected	ok	note (first 200 chars)
skill_adopted	POST …/versions/{version}/adopt, after the new draft’s skill_draft_saved	ok	adopted_from (the version vouched for; version is the one saved), reason (first 200 chars, redacted), principal. Under FELIX_AUTH_MODE=none the principal is whatever the request claimed
skill_archived	DELETE /skill-library/{name}	ok	previous
skill_promoted	POST /skill-library/~me/{name}/versions/{version}/promote, after the tenant draft’s skill_draft_saved	ok	promoted_from (the personal version; version is the tenant draft), reason (first 200 chars, redacted), principal. library is the personal library it came from, so an audit:read holder can link the promoter’s subject to that digest
personal_library_accessed	An administrator’s GET /skill-library/-/personal, or any request under /skill-library/~{digest}	ok	library (the digest, or * for the listing), method, path
skill_feedback_submitted	Feedback filed	ok	feedback_status, body_chars, has_patch — never the body
skill_feedback_accepted / skill_feedback_rejected	A person’s decision	ok	improve
skill_feedback_applied / skill_feedback_failed	The worker’s improvement	ok	result_version, model, error
skill_eval_queued / skill_eval_finished	An evaluation queued / run	ok, failed	skill, version, eval_id, eval_status, scenario_source, baseline_score, with_skill_score, uplift, error
skill_policy_updated / skill_policy_deleted	PATCH / DELETE /skill-library/-/policy	ok	before, after (each with source), tenant
skill_update_notification_queued	A recorded check queued a skill.update_available (principal skill-upstream)	ok	skill, source, version, upstream_commit, upstream_tree_hash, endpoints, superseded (it replaced an undelivered one)

Payloads are secret-redacted before persist. Per-request audit caps drop excess events (with a truncation marker) so a runaway loop cannot flood storage.

Optional warehouse spill (FELIX_WAREHOUSE) copies audit/eval analytics after Postgres write — Postgres stays authoritative.

Incident walk

1. GET /audit?event_type=policy_deny or ?status=error — the tenant comes from your own principal, so there is no tenant query parameter to set.
2. Find the failing row → payload.tool names the tool; payload.thread_id locates the run.
3. Confirm with Prometheus: felix_tool_calls carries the transport and error_code the audit payload does not, filtered by manifest_id.
4. GET /audit/metrics for per-tool call counts and error rates over a window. Its avg_latency_ms currently always reads 0 — it looks for payload.latency_ms, which the tool_call payload does not write. Use felix_tool_call_seconds for latency.
5. For durable chats, poll GET /chat/runs/{resume_token} and inspect fiber/run status.

Known limitations

• No multiprocess metrics. prometheus_client runs in single-process mode, so a multi-worker Granian reports per-worker fragments of every counter. Run one worker per replica, or aggregate across a replica label.
• A reused metric name degrades silently. Registering a name under a second label set raises, and the sample is written as a logger.info line instead — the series simply never appears in /metrics.
• Head sampling only. FELIX_OTEL_SAMPLE_RATIO is parent-respecting, so a sampled request keeps all of its child spans.

Local

make up-observability brings up an OTel Collector, Prometheus, Grafana, Jaeger, Loki and Postgres/Valkey exporters against the stack, with datasources and one dashboard provisioned — including a governance row for the counters above. It also builds the image with the otel extra; without it, OTLP export logs one warning at startup and exports nothing while everything else looks healthy.

To export to a backend you already run instead of one this overlay starts, set the FELIX_OTEL_* variables in .env — compose.yml passes every one of them through to api and worker, so no overlay is involved and nothing in the harness names the destination. uv run felix doctor then reports whether the exporter is installed, in every environment including development; it checks the module the configured FELIX_OTEL_PROTOCOL actually needs, so an install carrying only the other one reads as broken rather than fine.

make up / make dev → hit /metrics (with a credential) and /audit. Deploy guidance: deploy.