Skip to content

Voice Messages

MindRoom can surface Matrix voice messages as attachment-aware prompts for agents and teams. If STT is configured, MindRoom also transcribes the audio and routes it through the normal text pipeline. If STT is unavailable, disabled, or fails after media registration succeeds, the audio remains available as an attachment and falls back to 🎤 [Attached voice message]. If media download itself fails, dispatch continues with that text-only fallback and no attachment metadata.

Overview

When a voice message is received:

  1. The audio event is handled through the shared media pipeline.
  2. If voice STT and voice.visible_router_echo are enabled and the router is present and allowed to reply, the router immediately posts Router agent is transcribing….
  3. When audio download succeeds, it is decrypted, if needed, and registered as a context-scoped attachment while the placeholder is visible.
  4. If STT is configured and succeeds, the audio is transcribed and lightly normalized for mentions and ASR cleanup without creating chat commands.
  5. If STT is unavailable, disabled, or fails, MindRoom falls back to 🎤 [Attached voice message].
  6. The router replaces its placeholder with the normalized transcript or fallback text, or posts the fallback directly when STT is disabled.
  7. The normalized transcript or fallback prompt is dispatched using the normal routing and thread logic, with attachment metadata when registration succeeded.
  8. If routing is ambiguous in a multi-responder room, the router posts a visible handoff message.
  9. Otherwise, no extra router message is posted and the chosen agent or team replies directly.
  10. When download and registration succeeded, the responding entity receives the original audio attachment alongside the normalized transcript or fallback prompt; otherwise it receives the text fallback without an attachment.

Configuration

Enable STT and transcript normalization in config.yaml:

voice:
  enabled: true
  visible_router_echo: true
  stt:
    provider: openai
    model: gpt-4o-transcribe
    # Optional: custom service root or /v1 base URL
    # host: http://localhost:8080
  intelligence:
    model: default  # Model used for mention normalization and light ASR cleanup

Or use the dashboard's Voice tab.

With voice.enabled: false, audio messages are still surfaced as attachments with the fallback prompt. Enabling voice adds STT and transcript normalization on top of that attachment flow. With voice.visible_router_echo: true and voice.enabled: true, the router immediately posts a transcription placeholder and replaces that message with the normalized transcript or fallback text when it is present in the room and allowed to reply. When voice.enabled: false, the router posts the fallback text directly without claiming that transcription is running.

STT Providers

MindRoom uses the OpenAI-compatible transcription API. Any service that implements the /v1/audio/transcriptions endpoint will work.

OpenAI Transcription (Cloud)

voice:
  enabled: true
  stt:
    provider: openai
    model: gpt-4o-transcribe

Requires voice.stt.api_key or a named voice.stt.credentials_service; the default OpenAI credential service can resolve stored credentials or OPENAI_API_KEY.

Self-Hosted Whisper

voice:
  enabled: true
  stt:
    provider: openai_compatible
    model: whisper-1
    host: http://localhost:8080

The host may be either the service root or its /v1 base URL.

Use with faster-whisper-server or similar OpenAI-compatible STT servers.

Custom API Key

For self-hosted solutions that require authentication:

voice:
  enabled: true
  stt:
    provider: openai_compatible
    model: whisper-1
    host: http://localhost:8080
    api_key: your-custom-api-key

If a custom endpoint has no api_key, MindRoom sends a non-secret placeholder rather than requiring a cloud key. Cloud OpenAI transcription falls back to the OPENAI_API_KEY environment variable.

Transcript Normalization

The intelligence component uses an AI model to normalize transcriptions without turning speech into Matrix commands:

  1. Agent and team mentions - Converts spoken agent or team names to listed @agent or @team mentions
  2. Mention sanitization - Mentions of agents or teams not available in the current room have their @ stripped so the responder is not falsely targeted
  3. Command preservation - Never invents or rewrites spoken text into !command syntax
  4. Smart formatting - Handles light speech-recognition errors and natural-language variations

Intelligence Model

The intelligence model processes raw transcriptions to normalize agent and team mentions plus light ASR errors:

voice:
  intelligence:
    model: default  # Uses the default model from your models config

You can specify a different model for faster or more accurate transcript normalization.

How It Works

┌─────────────┐     ┌─────────────┐     ┌─────────────┐     ┌─────────────┐
│ Voice Msg   │────▶│ Download &  │────▶│ Transcribe  │────▶│ Format with │
│ (Audio)     │     │ Decrypt     │     │ (STT)       │     │ AI (LLM)    │
└─────────────┘     └─────────────┘     └─────────────┘     └─────────────┘
                                                                  │
                                                                  ▼
                                                         ┌──────────────────┐
                                                         │ Normal Dispatch  │
                                                         │ Decision         │
                                                         └──────────────────┘
                                                           │            │
                                                           │            │
                                                           ▼            ▼
                                                 ┌──────────────┐  ┌──────────────┐
                                                 │ Visible      │  │ No Visible   │
                                                 │ Router       │  │ Router       │
                                                 │ Handoff      │  │ Handoff      │
                                                 └──────────────┘  └──────────────┘
                                                           │            │
                                                           └──────┬─────┘
                                                                  ▼
                                                           ┌─────────────┐
                                                           │ Responder   │
                                                           │ Answers     │
                                                           └─────────────┘

Dispatch Behavior

Single-responder rooms or explicitly targeted audio

If only one eligible agent or team is visible, that responder answers the normalized audio event directly. If the audio caption or transcript explicitly mentions an agent or team, that targeted responder answers directly as well. In these cases, the router does not post an extra visible routing handoff. The transcript or fallback text is used internally for dispatch, not echoed to the room as a separate message. If voice STT and voice.visible_router_echo are enabled, the router still posts a display-only placeholder and replaces it with the normalized transcript or fallback text, but responders ignore that echo and continue responding to the original audio event. With STT disabled, the router posts the display-only fallback directly.

Multi-responder rooms where the router must choose

If multiple agents or teams are available and the audio does not already target one of them, the router uses the normalized text to do the usual routing step. The router then posts a normal handoff message such as @home could you help with this?. The selected agent or team responds to that router handoff, and the handoff carries the original audio attachment metadata forward. This is the case where a visible router message appears. If voice STT and voice.visible_router_echo are enabled, the router immediately posts a display-only transcription placeholder, replaces it with the normalized transcript or fallback text, and then posts the normal handoff. With STT disabled, the router posts the display-only fallback directly before the normal handoff.

No router, or router cannot reply

Audio still works when the router is absent. In that case, agents and teams handle the normalized audio directly using the same mention, thread, and permission rules as normal text messages. The same direct handling also applies when the router is present but is not allowed to reply to the original sender. In these cases, there is no visible router echo because the router does not handle the event. If multiple eligible responders remain and the audio does not already target one of them, there is no automatic handoff until the user mentions an agent or team.

Visibility rule

By default, MindRoom immediately posts a display-only router transcription placeholder when voice STT is enabled and the router is allowed to process the event, then replaces it with the normalized transcript or fallback text. With STT disabled, MindRoom posts the display-only fallback directly. The router handoff message appears only when the router must disambiguate between multiple eligible responders. If the responder is already clear from room shape, thread context, or explicit targeting, the chosen agent or team replies directly to the original audio event. Set voice.visible_router_echo: false to suppress the display-only echo without changing which event responders actually answer.

Attachment access

When media download and registration succeed, the original audio is registered as a context-scoped attachment before dispatch continues. That means the responding agent or team can inspect the file directly, use audio-capable models, or fetch it later with the attachments tool. This is true whether the prompt came from a transcript, a fallback message, or a router handoff. For successful STT turns, MindRoom adds hidden model-facing guidance that says the 🎤 text is already the transcript and the raw audio attachment is optional. For raw fallback turns, MindRoom does not add that guidance because the audio attachment remains the primary content.

Matrix Integration

Voice messages in Matrix are:

  • Detected as RoomMessageAudio or RoomEncryptedAudio events
  • Downloaded from the Matrix media server
  • Decrypted if end-to-end encrypted (using the encryption key from the event)
  • Registered as an audio attachment before dispatch when media download succeeds
  • Sent to the STT service via the OpenAI-compatible API when transcription is enabled
  • Normalized once per room and thread context, even though multiple bots may observe the event

Audio callbacks are registered on all bots because audio now follows the shared media pipeline. Shared normalization prevents repeated download and STT work for the same event. Reply-permission checks still use the original human sender, not a later router relay.

Environment Variables

Variable Description
OPENAI_API_KEY For OpenAI transcription (used as fallback if no api_key is configured)

Text-to-Speech Tools

MindRoom also supports text-to-speech (TTS) through agent tools. These are separate from voice message transcription and allow agents to generate audio responses:

  • Matrix Voice Message - One-call OpenAI TTS delivery to the current Matrix room or thread via matrix_voice_message
  • OpenAI - Speech synthesis via openai tool
  • ElevenLabs - High-quality AI voices and sound effects via eleven_labs tool
  • Cartesia - Voice AI with optional voice localization via cartesia tool
  • Groq - Fast speech generation via groq tool

Use matrix_voice_message when an agent should send a playable Opus Matrix voice note directly. It defaults to the current room and active thread, accepts thread_id="room" for room-level delivery, and can add readable text through companion_message.

See the Tools documentation for configuration details.

Voice Fallback (No STT Available)

When STT is unavailable, disabled, or transcription fails, MindRoom falls back to raw audio passthrough:

  1. When media download succeeds, the voice message audio is saved locally and registered as an attachment
  2. The normalized text becomes 🎤 [Attached voice message]
  3. Registered raw audio receives an attachment ID available to agents and teams in the room or thread context
  4. When registration succeeded, an agent or team automatically receives the raw audio as an Agno Audio object

If media download fails, the same fallback text is dispatched without an attachment ID or audio object.

This means voice messages still reach responders even without STT. Agents or teams with audio-capable models can process the raw audio directly, and tool-using responders can retrieve the file by attachment ID. Attachment IDs in this fallback path use the same context-scoping rules described in File & Video Attachments.

Limitations

  • Only OpenAI-compatible STT APIs are supported
  • Audio quality and background noise affect transcription accuracy
  • Without STT, routing has less textual context, so explicit @mentions or existing thread context are more reliable in multi-responder rooms
  • Without STT, responders receive raw audio instead of transcription, so the model or tools must support audio inputs to process it

Tips

  • Say the agent or team name first - "Hey @assistant, what's the weather?"
  • Use display names - The AI converts spoken names like "HomeAssistant" to the correct @home mention