Skip to content

Operational Log Events

This page covers how to monitor a running MindRoom: the health and readiness endpoints, the structured log events that are stable enough for metrics and alerts, the container metrics worth alerting on, and the debug logging you can turn on while troubleshooting. Most other log event names are internal diagnostics and can change in any release.

See also Routing & Responder Selection, Threads, Replies & Participation, and Access Control.

Health & Readiness

Method Endpoint Description
GET /api/health Liveness. Returns 200 with {"status": "healthy", ...}, or 503 with "status": "unhealthy" once startup is ready and an agent, team, or router has stopped syncing with Matrix
GET /api/ready Readiness. Returns {"status": "ready"} when startup has finished, otherwise 503 with {"status": "<phase>", "detail": "..."}

Use /api/health for startup and liveness probes and /api/ready for readiness probes in container orchestrators. Never point a startup probe at /api/ready: the one-time conversion after upgrading to conversation storage v2 keeps the process not ready for as long as it takes, a probe restart rolls back the database being converted, and a database larger than the probe allows never finishes converting. A startup probe on /api/health passes as soon as the process answers, as the bundled Helm charts configure it.

The /api/health body also contains:

  • last_sync_time: the oldest last successful Matrix sync among running entities.
  • stale_sync_entities: the names of entities whose Matrix sync is stale, when there are any.
  • e2ee: decryption-failure counters (see Troubleshooting Undecryptable Messages).
  • embedder: {"status": "failing", "detail": ...} with the cause while the semantic-search embedder is failing; this never makes the endpoint unhealthy.

The /api/ready phases are:

Phase Meaning
idle Process not started
starting Startup in progress; detail says what it is waiting for
ready Startup finished and requests are being served
failed Startup or runtime failure; detail gives the cause

While mindroom run waits for hosted pairing approval, or converts conversation storage on the first start after an upgrade, only these two endpoints answer: /api/health returns 200 and /api/ready returns 503 with phase starting and detail Waiting for local pairing approval or Converting conversation storage.

When an agent, team, or router receives no Matrix sync for 120 seconds, MindRoom restarts that entity's sync loop on its own. After 180 seconds without a sync, /api/health returns 503 and lists the entity in stale_sync_entities. While an entity is still making progress on a large backlog of received events, both wait up to MINDROOM_MATRIX_INGESTION_GRACE_SECONDS. Set the liveness probe's failureThreshold high enough that the sync-loop restart can recover before the container is restarted.

Collecting the Events

Set MINDROOM_LOG_FORMAT=json so each record is written as one JSON object per line to stderr and to the runtime log file under <storage>/logs/. Every record carries event, level, logger, and timestamp next to its own fields, so log collectors that parse JSON lines can filter on event and extract numeric fields.

{"sample_count": 1200, "p50_ms": 0.412, "p95_ms": 1.87, "p99_ms": 4.02, "max_ms": 11.3, "gc_collections_total": [5120, 466, 31], "max_lag_scheduled_at": "2026-01-01T12:00:41.250+00:00", "max_lag_observed_at": "2026-01-01T12:00:41.261+00:00", "event": "event_loop_scheduler_lag_summary", "logger": "mindroom.event_loop_stall", "level": "info", "timestamp": "2026-01-01T12:01:00.012345Z"}

event_loop_scheduler_lag_summary and worker_cold_start are info records, so keep LOG_LEVEL at INFO or lower, or raise only their loggers with MINDROOM_LOGGER_LEVELS=mindroom.event_loop_stall:INFO,mindroom.workers.progress:INFO. agent_error, event_loop_stall_detected, and event_loop_stall_ongoing are error records, and the other stable events below are warning records. See Environment Variables for the logging controls.

Events

The event names and fields below are stable operator signals; a record can carry further diagnostic fields that are not listed and can change. Each section lists the fields, a suggested alert, and where to look when it fires. The thresholds are starting points to tune against your own baseline, and the queries are generic pseudo-queries to translate into your log platform's syntax.

event_loop_scheduler_lag_summary

All agents, teams, and the router share one event loop. Every 60 seconds MindRoom logs how late a heartbeat scheduled every 50 ms actually ran.

Field Meaning
sample_count Heartbeat samples in the window
p50_ms Median heartbeat lateness in milliseconds
p95_ms 95th percentile heartbeat lateness in milliseconds
p99_ms 99th percentile heartbeat lateness in milliseconds
max_ms Worst heartbeat lateness in milliseconds
max_lag_scheduled_at UTC time at which the worst heartbeat was due
max_lag_observed_at UTC time at which the worst heartbeat ran
gc_collections_total Cumulative garbage collections per generation since process start

An idle or lightly loaded loop reports a p50_ms of a few milliseconds or less. A sustained high median means the loop is saturated and replies slow down, even while /api/health reports healthy, because liveness tracks Matrix sync rather than loop latency.

metric  = distribution of p50_ms
          from logs where event = "event_loop_scheduler_lag_summary"
alert if  median(metric) over 10 minutes > 20
          for 30 minutes

When it fires, look for event_loop_stall_detected records, which carry the blocking stack of long stalls. Every stall above MINDROOM_EVENT_LOOP_STALL_STACK_THRESHOLD_SECONDS (default 0.25, 0 or a negative value disables) also logs one event_loop_stall_stack record with the blocking stack and the stall's lag_ms once it ends, at most once every 10 seconds. Compare the frequency of event_loop_gc_collection records, which report garbage collections slower than 50 ms. Repeated large_streaming_edit_preview_prepared records with a growing original_size_bytes in one room point at one long streaming response. Setting MINDROOM_EVENT_LOOP_STALL_THRESHOLD_SECONDS to 0 or a negative value disables stall detection and this summary.

Once agents are ready, and then every MINDROOM_HEAP_REFREEZE_INTERVAL_SECONDS (default 3600, 0 disables, any other value below 60 fails startup), MindRoom collects garbage once and excludes the surviving objects from later collections, logging runtime_heap_refrozen. This keeps routine collections short on large processes at the cost of one full collection per interval. After the first refreeze, MindRoom also raises CPython's full-collection threshold and logs gc_full_collection_threshold_raised, so routine full collections run less often, by an amount that depends on the workload; unreachable reference cycles then wait longer to be freed. Memory held by objects that become garbage while frozen, such as agents a config reload replaced, is reclaimed only at the next refreeze, so resident memory can stay higher for up to one interval.

event_loop_stall_detected

Logged when the shared event loop has run no heartbeat for MINDROOM_EVENT_LOOP_STALL_THRESHOLD_SECONDS (default 5), so every agent, team, and router is frozen until it ends. While the stall lasts, an event_loop_stall_ongoing record with the same fields follows every 30 seconds.

Field Meaning
stalled_for_seconds Seconds since the last heartbeat
threshold_seconds The configured stall threshold
stack_capture_suppressed true when a stack was already captured in the last 30 seconds; the record then lacks the four fields below
stack Stack of the code blocking the event loop
process_cpu_seconds_since_heartbeat CPU seconds the process used since the last heartbeat
other_thread_stacks Stacks of up to eight other threads
omitted_thread_stack_count Number of other threads whose stacks were left out

Count the records and alert when stalls repeat, since each one delays every reply in the process. The stack names the blocking code; a process_cpu_seconds_since_heartbeat close to stalled_for_seconds points at CPU-bound work, and one near 0 at a blocking call such as file or network I/O.

event_loop_stall_ended

Logged when a stall that event_loop_stall_detected reported ends.

Field Meaning
stall_duration_seconds Seconds the event loop went without a heartbeat

Sum stall_duration_seconds over a window to see how long agents were frozen in it.

matrix_sync_stall_diagnostics

Logged when one agent, team, or router goes 90 seconds without Matrix sync progress, and repeated at most every 90 seconds while the stall lasts. It shows where that entity's sync and event-processing tasks are waiting, without message content, and does not affect health or readiness.

Field Meaning
agent Name of the entity that made no progress
no_progress_seconds Seconds since the last sync or event-processing progress
sync_age Seconds since the last sync activity, or null before the first sync result
generation Event-processing progress counter, or null when none is reported
snapshots Up to four task snapshots, each with task_name, await_chain, await_boundary, and truncated

Each report covers its own 90 seconds without progress, so five reports for one entity within 15 minutes mean at least seven and a half minutes, not necessarily continuous, without progress.

metric  = count of logs where event = "matrix_sync_stall_diagnostics"
          grouped by agent
alert if  sum(metric) over 15 minutes >= 5 for any agent

Read the await_chain snapshots to see where each task is waiting. Reports for every entity at once usually point at the homeserver or at a saturated event loop, so check event_loop_scheduler_lag_summary as well.

matrix_event_decryption_failed

Logged when an agent cannot decrypt a new message, in a room it has joined, from a sender it may reply to. Every agent in the room that receives the message logs it, and a burst of failures in one encryption session can log fewer records than messages, so count distinct event_id values to approximate affected messages. Troubleshooting Undecryptable Messages explains the usual cause and fix.

Field Meaning
room_id Room of the message
event_id Event ID of the message
sender Matrix user ID of the sender
sender_device_id Device the sender encrypted the message on
session_id Encryption session of the message
agent Agent that could not decrypt it
key_request_already_sent Whether a key request for that session was already pending
hint Fixed explanation of the usual cause

Count it grouped by agent and alert on a sustained rate. Failures from one sender_device_id point at that client, while failures across many senders for one agent point at that agent's device.

conversation_converted

Logged once for each conversation database the first start after an upgrade that changes the database layout converts, such as upgrading to conversation storage v2 or to one owner per conversation database, or that converts at its first use later, such as a restored backup. An upgrade the upgrade notes do not list converts without a backup, and the release before it reads the converted databases as they are, so going back needs no restore.

Field Meaning
path The converted database file
runs Runs in the converted database
messages Stored messages in the converted database
compacted Whether the file was compacted to return the space its old tables used
duration_seconds Time the conversion and compaction took

compacted is false when compacting would return little or the directory lacked the space the upgrade notes give for it; a conversation_compaction_failed warning names the database when compacting failed, and the conversion still stands. A database that could not convert at startup logs conversation_conversion_skipped_store instead.

tool_call_limit_reached

Each agent or team run has a tool-call budget, normally the entity's max_tool_calls_per_turn. Calls past the budget fail with a tool error, and this event is logged once when a run keeps requesting tools and is stopped, ending the turn with the text produced so far.

Field Meaning
entity Agent or team name; dynamic workflow subagent participants appear under the calling agent's name and room_agent participants under their own
budget The run's tool-call budget
model_requests Model requests the run made before it was stopped
alert on  each log where event = "tool_call_limit_reached"
notify    at most once per hour

Check whether the run was a runaway loop, such as the same tool call repeated or calls to unknown tools, or legitimate long work. Each call to a tool the run does not offer logs a tool_not_offered warning, and each call whose arguments cannot be decoded logs an invalid_tool_arguments warning with reason decode or not_object; both carry entity and tool_name, never the arguments. For legitimate long work, raise that entity's max_tool_calls_per_turn. Skill reviews also log this record under the reviewed agent's name, with a fixed budget of 16 that max_tool_calls_per_turn does not change.

failing_tool_call_refused

Logged for each call refused because its tool keeps failing, as max_tool_calls_per_turn describes.

Field Meaning
entity Agent or team name
tool_name The refused tool

The refused call's result quotes the tool's last error, and a tool that raises also logs each failure as tool_call_failed; one repeated error usually means an integration that is down, an expired grant, or arguments the model keeps resending.

failing_tool_ended_run

Logged once when the model calls a refused tool again and the turn ends.

Field Meaning
entity Agent or team name
tool_name The tool the model kept calling

tool_call_failed

Logged each time a tool call raises an error. A call stopped together with its turn logs error_type CancelledError.

Field Meaning
tool_name Name of the tool function
agent_name Agent that made the call, or null
error_type Class name of the error the tool raised
error_message The error's message, with credentials redacted
duration_ms Milliseconds from the call's start until it failed
correlation_id ID shared by the tool calls that answer one message
channel matrix, openai_compat, headless, mcp, or null

Count it grouped by tool_name and alert when one tool fails far more often than usual. Failures that share a correlation_id came from a single turn, and error_type and error_message give the cause. For a tool that uses an OAuth connection, also check for oauth_credentials_refresh_failed records.

worker_cold_start

Logged when a call had to wait for a dedicated worker of the Docker or Kubernetes backend that was not ready, such as a new, stopped, scaled-down, failed, or recreated worker, once it became ready or its start failed. A call to a ready worker logs nothing, and calls that wait for the same start share its one record.

Field Meaning
worker The worker's log ID, the hash other worker logs name it by and the end of its state directory name
backend kubernetes or docker
kind script for the worker of one background script run, subagent for the dedicated worker of one subagent, otherwise tool
scope The worker's scope: shared, user, user_agent, or unscoped for an agent without worker_scope
agent_name Agent the worker serves, as its worker key spells it, or null for a user worker, which serves every agent on that scope
outcome ready, timeout for a worker that did not become ready within the backend's ready timeout, or error
wait_ms Milliseconds from finding the worker not ready until it answered its health check, or until its start failed
pod_ready_ms Kubernetes only: milliseconds from the same point until Kubernetes reported the worker's pod ready, which covers applying or scaling up its Deployment, scheduling, image pull, and container start; the rest of wait_ms is the first health check through the worker's Service
{"worker": "3f9c2a7b1d4e8f60", "backend": "kubernetes", "kind": "tool", "scope": "user_agent", "agent_name": "code", "outcome": "ready", "wait_ms": 14211.6, "pod_ready_ms": 13190.2, "event": "worker_cold_start", "logger": "mindroom.workers.progress", "level": "info", "timestamp": "2026-01-01T12:00:14.211Z"}
chart     p50 and p95 of wait_ms where event = "worker_cold_start" and outcome = "ready", grouped by kind
alert on  count of logs where event = "worker_cold_start" and outcome != "ready" > 3 in 15 minutes

A high pod_ready_ms points at scheduling or image pulls: check for pending worker pods, node scale-up, and the worker image pull policy. A large gap between pod_ready_ms and wait_ms points at the worker's Service or a network policy between the primary and its workers. Many tool records for the same worker mean it is used less often than its idle timeout, such as workers.kubernetes.idleTimeoutSeconds, so it scales down between uses. A timeout means the worker did not become ready within workers.kubernetes.readyTimeoutSeconds or MINDROOM_DOCKER_WORKER_READY_TIMEOUT_SECONDS; the call that waited fails, and for a tool call a tool_call_failed record carries the error.

oauth_credentials_refresh_failed

Logged when MindRoom cannot refresh the access token of a stored OAuth connection, for a tool or an MCP server.

Field Meaning
provider_id OAuth provider ID
credential_service Credential service the connection is stored under
reason refresh_rejected, token_endpoint_changed, or provider_refresh_failed, described below
error_type OAuthRefreshRejectedError for the first two reasons, OAuthProviderError for provider_refresh_failed
oauth_error The provider's OAuth error code, such as invalid_grant, unrecognized for a nonstandard code, or null
has_refresh_token Whether the stored connection had a refresh token
expires_at Expiry of the stored access token as a Unix timestamp, or null
  • refresh_rejected: the provider rejected the refresh grant, for example because it was revoked or expired; MindRoom removes the connection, and the account must be connected again.
  • token_endpoint_changed: the provider's token endpoint changed since the account was connected, as Troubleshooting OAuth Connections describes.
  • provider_refresh_failed: any other failure, such as a network error or a provider outage; the connection is kept, so a later refresh can succeed.

A failure caused by an HTTP or network error also carries transport_error_category, transport_error_type, and, for an HTTP error, http_status_code. Count it grouped by provider_id and reason. Each refresh_rejected record is one connection that must be connected again, while repeated provider_refresh_failed records for one provider point at that provider, the network, or the OAuth client settings, which oauth_error invalid_client names.

provider_request_retried

Logged each time MindRoom sends a model request again: after a transient provider failure, such as a rate limit, an overload, a server error, or a dropped connection, or after a stalled stream. Each retry delays the reply, and a reply whose request fails every retry logs agent_error.

Field Meaning
model_id Model ID the request was sent to
reason transient_error for a failure the provider or the network reported, stream_stalled for a stream that went silent past its idle limit
error_type RateLimited for a throttled request, otherwise ProviderUnavailable
status_code HTTP status of the failure, 504 for a stalled stream, or null when there is none, as for a dropped connection
error The failure's message, redacted and truncated
attempt Number of this retry for its reason, starting at 1
max_retries Retries allowed for its reason
delay_seconds Seconds waited before this retry: the provider's Retry-After or Azure's retry-after-ms when it asked for at most a minute, otherwise a backoff that doubles with each retry

Count it grouped by model_id and alert when one model is retried far more often than usual. error_type, status_code, and error tell throttling apart from an outage or an unresponsive endpoint.

agent_error

Logged each time an agent's or team's reply fails and the user gets an error message instead.

Field Meaning
agent Agent or team name
error_type Class name of the error that ended the reply
error The error's repr, with credentials redacted
error_code MindRoom's code for a provider error that its type alone does not tell apart, such as quota_exhausted or missing_api_key, otherwise null
status_code HTTP status of a provider error, 401 for a credential failure MindRoom found before sending a request, or null when there is none or the error is not a provider error
provider Provider name of the model route for a provider error, otherwise null

These error_type values are stable; any other value is the class name of an unexpected error and can change:

  • RateLimited: the provider throttled the request, after any retries that were safe.
  • ProviderUnavailable: the provider or the network failed, or the stream stalled, after any retries that were safe.
  • AuthenticationFailed: the provider rejected the credentials, or none are set up.
  • ContextWindowExceeded: the conversation no longer fits the model's context window, or one request is larger than the account's tokens-per-minute limit.
  • InvalidRequest: the provider rejected the request, such as a malformed or oversized body or an unknown model.
  • ModelSafeguardRefusal: the model's safeguards refused the request.
  • ModelOutputLimitError: the model's responses kept stopping at the output limit before their tool calls were complete.
  • ProviderError: any other provider rejection, such as an HTTP 403, an HTTP 402, an OpenAI, Anthropic, Gemini, or Moonshot account with no credit left or past its monthly spend or usage limit, or a Z.ai account with no balance or an expired plan or package.

These error_code values are stable:

  • quota_exhausted: the provider refused the request over the account's billing, such as no credit left, a spend or usage limit reached, an expired plan, or any HTTP 402 (error_type ProviderError). MindRoom does not retry it. OpenRouter also sends a 402 while it settles earlier requests against a low balance, which a wait can let through.
  • missing_api_key: the model route has no credential set up (error_type AuthenticationFailed).

An AuthenticationFailed with error_code null is any other credential failure: a key the provider rejected, such as a revoked key, a key whose environment variable is unset, or a missing or expired CLI login. Alert on quota_exhausted and on AuthenticationFailed: they usually last until someone changes the account, the key, or the login.

Count it grouped by error_type and alert when one value is far more frequent than usual. RateLimited and ProviderUnavailable from many agents mean a provider is failing, while InvalidRequest or ContextWindowExceeded from one agent is more likely one conversation the provider cannot accept.

Container Metrics

With the Kubernetes worker backend, idle worker Deployments scale to zero replicas by design, so exclude Deployments labelled mindroom.ai/component=worker from alerts on Deployments without an available replica. Judge container memory by resident memory (RSS) or working set, not by total usage: total usage also counts the page cache that session databases, knowledge files, and logs fill, which the kernel reclaims under memory pressure. The heap probe reports the primary process's own rss_bytes.

Debug Logging

When a request to a model under models ends, completed or not, MindRoom logs an LLM usage event with provider and model identifiers, the stop_reason of a reply that finished (output_limit or content_filter for one the provider cut off), and, when the provider reports usage, input tokens, uncached input tokens, and a cache-read ratio, without prompt content.

To record full provider requests, enable debug.log_llm_requests:

debug:
  log_llm_requests: true
  llm_request_log_dir: null
  • log_llm_requests (bool, default false): write provider requests as JSONL, and record successful tool calls with timing in <storage>/tracking/tool_calls.jsonl, where <storage> is MINDROOM_STORAGE_PATH or the default mindroom_data (tool failures are always recorded there).
  • llm_request_log_dir (string, default null, meaning <storage>/logs/llm_requests): directory for the request logs.

Request logs hold each request as MindRoom hands it to the provider's client library, with its prompts, messages, tools, and attachments (on Gemini, in google-genai's field names and without the settings under http_options), beside the model parameters and requester and source Matrix event metadata. Credentials such as tokens, cookies, passwords, API keys, authorization headers, and PEM private keys are redacted, but these files can still contain sensitive prompt, argument, and result data, so leave the flag disabled unless you are actively debugging.

Export MINDROOM_TIMING=1 before startup to log timing for tool calls and one INFO-level Dispatch pipeline timing summary per turn, including time_to_model_request_ms and context, queue, payload, agent-build, and model spans. A reply whose tool call waited more than 1.5 seconds for a worker to start, or whose worker failed to start, also carries worker_wait_ms, the longest such wait, which matches wait_ms of that start's worker_cold_start record; a streamed reply also shows a warmup notice during that wait.

LOG_LEVEL=DEBUG MINDROOM_LOG_FORMAT=json MINDROOM_TIMING=1 mindroom run

Memory Growth

To find what grows the primary process's memory, set MINDROOM_HEAP_PROBE_INTERVAL_SECONDS in the process environment or the .env file next to config.yaml. MindRoom then logs one heap_type_probe event per interval with the 25 most common object types and their counts, resident memory (rss_bytes), allocator totals (arena_bytes, arena_in_use_bytes, arena_free_bytes, mmap_bytes), and python_allocated_blocks. Resident memory that grows while arena_in_use_bytes plus mmap_bytes and python_allocated_blocks stay flat suggests freed memory the allocator keeps. Growth in those suggests live objects, which the type histogram may not show because it does not count strings, bytes, or numbers. The histogram leaves out frozen objects and reports only their count as frozen_objects, so it shows what the runtime created since the last heap refreeze; set MINDROOM_HEAP_REFREEZE_INTERVAL_SECONDS=0 to see growth over the whole process lifetime. Each probe briefly pauses the event loop, so prefer intervals of several minutes on large processes. Unset or 0 disables the probe (the default), and any other value below 60 fails startup.