Skip to content

Image Messages

MindRoom can process images sent to Matrix rooms, passing them to vision-capable agents and teams for analysis.

Overview

When a user sends an image in a Matrix room:

  1. The responder determines whether it should answer (via mention, thread participation, or DM)
  2. The image is downloaded and decrypted (if E2E encrypted)
  3. The image is wrapped as an agno.media.Image and passed to the AI model
  4. The responder replies with its analysis

Image support works automatically for agents and teams -- no configuration is needed. The selected model must support vision (e.g., Claude, GPT-5.6).

Supported Formats

MindRoom detects image format from file byte signatures:

  • PNG
  • JPEG
  • GIF
  • WebP
  • BMP
  • TIFF

If the declared MIME type in the Matrix event does not match the detected byte signature, MindRoom logs a warning and uses the detected type.

How It Works

┌─────────────┐     ┌─────────────┐     ┌─────────────┐
│ Image Msg   │────>│ Download &  │────>│ Pass to AI  │
│ (Matrix)    │     │ Decrypt     │     │ Model       │
└─────────────┘     └─────────────┘     └─────────────┘
                                              │
                                              v
                                        ┌─────────────┐
                                        │ Responder   │
                                        │ Replies     │
                                        └─────────────┘

Usage

Send an image in a Matrix room and mention the agent or team in the caption:

  • With caption: @assistant What does this diagram show? -- the caption is used as the prompt
  • Without caption: The agent receives [Attached image] as the prompt and describes what it sees
  • Bare filename: If the body is just a filename (e.g., IMG_1234.jpg), it is treated the same as no caption

Images work in both direct messages and threads, and with both individual agents and teams.

Captions (MSC2530)

If the Matrix event's filename field differs from body, the body is used as a user caption. This follows MSC2530 semantics and works with clients that set the caption in the body.

Image Persistence

Image bytes are saved under mindroom_data/incoming_media/, while their attachment metadata is saved under mindroom_data/attachments/; both are subject to the attachment retention policy. In addition to being passed to the AI model as vision input, each image is also registered as an att_* attachment ID so agents can reference it via tool calls. See Attachments for details on retention and context scoping.

Encryption

Both unencrypted and E2E encrypted images are supported. Encrypted images are decrypted transparently using the key material from the Matrix event.

Media Fallback

If a model rejects inline media (images, audio, video, or documents), MindRoom automatically retries the request without the inline media. The retried prompt includes [Inline media unavailable for this model] to inform the agent that attachments were dropped. Agents can still reference the files via attachment IDs and tools.

This fallback is transparent — no user action is required. Any ordinary failure of a media-bearing request triggers one retry without media — no error wording decides whether to retry, so unknown provider prose degrades gracefully instead of surfacing a raw provider error. Provider safeguard refusals are returned as refusals and are not retried without media. When the retry succeeds, the model route learns that the dropped media kinds are unsupported, and later requests omit them up front instead of paying a failed API call. This learned capability state is process-local and resets on restart. Payload-size and context-overflow rejections never teach the capability state, since dropping media can shrink an oversized request for reasons unrelated to media support. Transient failures (HTTP 5xx and 429 status codes on the provider exception) also never teach, since their retry can succeed simply because the outage or rate limit passed.

Limitations

  • Routing with multiple eligible responders -- without an @mention, the router uses the image caption to select among candidates only when room configuration and reply permissions leave multiple eligible agents or teams.
  • Bridge mention detection uses m.mentions in the event, falling back to parsing HTML pills from formatted_body when m.mentions is absent (e.g., mautrix-telegram). Bridges that set neither may not trigger agent responses.
  • Model support -- vision input requires a model that supports it. Text-only models reject inline images, and the media fallback retries without them so the agent still answers with a note that it cannot view the attachment.