Skip to content

Interactive Canvases

An agent can show the user a web page it wrote, such as a dashboard, report, slide deck, form, or picker, in the Canvas panel beside the conversation in MindRoom Chat. The user can work in the page and send an answer, which reaches the agent as the user's next message. Use a canvas when seeing or clicking beats reading or typing; for a quick choice between a few options, a plain reply or an interactive question is simpler.

A narrated film of all four: a scheduling grid, a data fit, a slide deck, and a trip planner.

A week grid of four people's availability; the pick goes back with Send, and the agent books the meeting after approval.

An interactive fit drawn in inline SVG with the theme variables; leaving out an outlier refits the curve live.

A deck the agent writes to a workspace file and shows by path; after a change request it rewrites the file and the open deck updates in place.

One canvas, two steps: pick a stay, then plan the days against a live budget, updated in place with canvas_event_id.

Turn canvases on

Canvases need two switches, and both are off by default.

First, give the agent the chat_ui toolkit with enable_show_canvas: true:

agents:
  planner:
    display_name: Planner
    role: Plan trips and compare the options visually.
    model: default
    tools:
      - chat_ui:
          enable_show_canvas: true

enable_show_canvas (boolean, default false) adds the show_canvas function; the rest of chat_ui works without it. It can also be set in the chat_ui tool's saved settings for every agent that uses chat_ui; an agent's own entry wins, as described in How Settings Combine. mindroom config init and newly provisioned hosted instances already turn it on for the starter Mind agent, so Mind needs only the Chat switch.

Second, enable canvases in the MindRoom Chat deployment's runtime config.json:

{
  "mindroom": {
    "canvas": {
      "enabled": true
    }
  }
}

The shipped MindRoom Chat configuration sets enabled to false, so only whoever deploys Chat can turn canvases on.

Let pages load libraries

Chart.js, D3, Mermaid, and KaTeX need no switch: MindRoom Chat serves its own copies to pages whose agent names them in show_canvas, and no viewer's address reaches a CDN. Pages can also load any other npm library from jsDelivr's npm CDN, https://cdn.jsdelivr.net/npm/. This takes two more switches, both off by default; turn on both, because a page that needs a library breaks where Chat blocks it.

agents:
  planner:
    display_name: Planner
    role: Plan trips and compare the options visually.
    model: default
    tools:
      - chat_ui:
          enable_show_canvas: true
          enable_canvas_libraries: true
{
  "mindroom": {
    "canvas": {
      "enabled": true,
      "libraries": true
    }
  }
}

enable_canvas_libraries (boolean, default false) tells the agent it may use the source, and Chat's libraries lets pages load from it. Pages may then load scripts, styles, and fonts from /npm/ paths only; fetching data, external images, workers, and every other site stay blocked, and libraries that evaluate strings, such as Alpine or Vue in-page templates, do not run. Opening a page that uses a CDN library tells jsDelivr the viewer's IP address and which files they load, which is why Chat keeps libraries off unless its deployment allows them.

Let pages load remote assets

To preview a website as a browser shows it, with its real images, fonts, and stylesheets, pages can load assets from any https: address. This also takes two switches, both off by default:

agents:
  designer:
    display_name: Designer
    role: Preview website elements as a browser shows them.
    model: default
    tools:
      - chat_ui:
          enable_show_canvas: true
          enable_canvas_remote_assets: true
{
  "mindroom": {
    "canvas": {
      "enabled": true,
      "remoteAssets": true
    }
  }
}

enable_canvas_remote_assets (boolean, default false) tells the agent it may load assets from any https: address, and Chat's remoteAssets lets pages load scripts, styles, images, fonts, and media from them. Fetching data, frames, workers, and form submissions stay blocked. Every host a page names then sees the viewer's IP address and when they look, and a page can put data in the addresses it loads, which is why Chat keeps remote assets off unless its deployment allows them.

What the user sees

Each canvas appears in the conversation as its title with an Open canvas button. Chat opens the panel by itself under the same conditions as other Chat UI actions; otherwise the user chooses Open canvas. The panel names the agent that made the page, and the user can resize it or expand it to the full width of the conversation. The room header's Canvas button lists the conversation's canvases, newest update first, and reopens one after the user closed it.

Pages in the thread

With inline=True, show_canvas shows the page in the conversation itself, at the width of a reply, instead of in the panel; its height follows the content. Agents prefer it for anything the user should see or answer in the flow: charts, diagrams, mockups, pickers, short forms, and multi-step questions. They use the panel for what stays open beside the conversation: dashboards, slides, long documents, and pages the user returns to. An inline page sends answers and keeps inputs and saved state like a panel canvas. Chat never opens the panel for it; Open canvas shows it there, and the thread then shows "Shown in the panel" in its place. Updates keep a canvas inline.

Chat rebuilds an inline page each time it scrolls back into view, so agents give form fields an id or name, or call saveState, to keep what the user entered. Forms use the themed mode, not raw or width, because a scaled-down preview shrinks fields and buttons. For the next step of a flow, the agent shows a new inline canvas with its reply or updates the same one.

For an accurate preview of a website or app, the agent adds raw=True: Chat adds no styles or theme variables of its own, so browser defaults apply as in a normal tab. width (320 to 2560 pixels) renders the page at the viewport the design targets, such as 1440 for desktop or 390 for a phone, and scales it down to fit the conversation; it never scales up. A preview with raw or width is laid out in a browser window 16:10 of its width (568 to 1080 pixels tall), so vh units work as in a browser. The thread shows as much of that window as the page fills, and a taller page, such as one with a 100vh hero section, scrolls inside it. Other inline pages grow with their content, so they must not use vh units. The panel shows raw pages too, at the panel's own width. raw and width belong to each update, so an update without them shows a themed page at reply width.

When the panel cannot open, the Open canvas button is disabled and a message below it says why:

Message Cause
Interactive panels are turned off in this app. The Chat deployment has not enabled canvases.
Interactive panels are not available in the app yet. Open this conversation in a browser. The native mobile app does not show canvases; Chat in a browser does.
End the call to open interactive panels. Chat does not run canvases during a call.

Other people in the room who use MindRoom Chat on the agent's homeserver see the title and Open canvas, not the inline page, and can answer in the panel; everyone else sees only the notice text, Interactive panel: <title>. Open it in MindRoom Chat to respond.

Find a canvas again

A canvas the user comes back to, such as a landing page an agent keeps current or a checklist used every day, does not have to be found in its thread again. Where canvases are on, Chat's sidebar gets a Canvases button once the user has a canvas; it lists the canvases agents made for the user: pinned ones first, then the most recently updated, with the thread each was shown in (by its summary, as the sidebar names it), the room, the agent, and when each was updated and created. Choosing a title opens the canvas at its latest version, filling the conversation's column; choosing the thread or the room opens that conversation.

Pins are kept in the user's Matrix account data (io.mindroom.pinned_canvases, room and event IDs only), so they show on every device. The rest of the list holds the canvases this browser has received, so a new device starts with recent canvases and older ones appear once their thread is opened. Users who don't want the button turn off Show Canvases in the sidebar under Settings โ†’ General โ†’ Interface; the choice follows them to every device. For a page the user returns to, update the same canvas with canvas_event_id rather than showing a new one, so its row and pin keep pointing at it.

Answering a canvas

Typing, toggling, and moving between steps inside the page send nothing. When the page submits an answer, Chat shows the user exactly what will be sent, and nothing is sent until the user chooses Send. To change a pending answer, the user chooses Discard and answers again.

The answer is an ordinary message from the user in the canvas's thread; it mentions the agent and starts the agent's next turn:

@mindroom_planner:example.org Canvas response ($canvas-event, revision $canvas-event): Pro plan
{"plan":"pro"}

The first ID is the canvas and the second is the version the user answered; when that is not the latest version, the user answered an earlier page. MindRoom Chat shows the answer as a one-line receipt that expands to the exact data. An answer can carry up to 512 KB of JSON data, enough for a long document the user edited in the canvas; larger or non-JSON data is ignored, so no Send prompt appears.

When the page throws an error, fails to load a file, or tries to load something Chat blocks, the panel lists up to five of those errors with a Tell agent button. Nothing is sent unless the user chooses it; the report reaches the agent like an answer:

@mindroom_planner:example.org Canvas error ($canvas-event, revision $canvas-event):
Uncaught ReferenceError: drawChart is not defined (line 12)
Blocked https://unpkg.com/chart.js (script-src-elem)

Let the agent read the page without a Send button

For a page whose choices the agent needs later, such as a checklist ticked over a few days, the agent shows the canvas with share_state=True. Chat then keeps a copy of what the page saved (its saved state and kept inputs) in the room, once the user has worked in the page and pauses for a moment, and when the panel closes or the tab is hidden. The copy never starts the agent's turn and never enters its conversation; the agent reads the newest copy when it wants, for example from a scheduled task:

chat_ui.read_canvas_state(canvas_event_id="$canvas-event")

It returns the state, the inputs, and when the user's Chat shared them, or says nothing was shared yet; when the newest copy in the room cannot be read, it returns an error rather than an older copy. Sharing is decided when a canvas is first shown, and its updates keep it; the panel tells the user "Saved in this room: agent and others here can read what you enter." Each copy stays in the room's history like a message, and logging out does not remove it.

Update a canvas

show_canvas returns the canvas event_id. Calling show_canvas again with canvas_event_id set to that ID replaces the page in place, for the next step of a flow or a new version of a file the agent edited, and the conversation keeps one card. An update may leave out title to keep the title the canvas was first shown with. An open panel switches to the new page at once, unless the user has been working in it; then Chat shows "agent updated this panel." with a Load update button, so unsent work is not lost. Every update stays available: once a canvas has more than one version, the panel header shows Version n of m with buttons to go back and forward, and an earlier version offers Show latest. The user can answer an earlier version; switching is unavailable while an answer waits for Send or is sending. An agent can update only canvases it showed to the same user in the same room and thread, or at room level; otherwise the call fails and the agent shows a new canvas instead.

Write a page

Pass the page as html, or as a path to an HTML file, which suits pages the agent builds and refines, such as slides. Paths follow the same rules as registering local files; prefer workspace-relative paths such as slides/deck.html.

chat_ui.show_canvas(
    title="Choose a plan",
    html="""
<style>
  .plans { display: grid; gap: 12px; grid-template-columns: repeat(auto-fit, minmax(160px, 1fr)); }
  .plan { background: var(--mr-surface); border: 1px solid var(--mr-border);
          border-radius: var(--mr-radius); padding: 16px; }
  button { background: var(--mr-accent); color: var(--mr-accent-text); border: 0;
           border-radius: 8px; padding: 8px 14px; cursor: pointer; }
</style>
<div class="plans">
  <div class="plan"><h3>Basic</h3><p>For trying things out.</p>
    <button onclick="mindroom.submit({plan: 'basic'}, {label: 'Basic plan'})">Choose Basic</button></div>
  <div class="plan"><h3>Pro</h3><p>For daily work.</p>
    <button onclick="mindroom.submit({plan: 'pro'}, {label: 'Pro plan'})">Choose Pro</button></div>
</div>
""",
)
  • Self-contained: write inline CSS and JavaScript. The page cannot fetch data, and it loads libraries or external assets only where libraries or remote assets are allowed; otherwise draw charts with inline SVG or a <canvas> element, and always embed images as data: URLs.
  • Submitting: call window.mindroom.submit(data, {label}) with JSON-serializable data and a short label, or use a <form>, whose fields become the data and whose data-mindroom-label attribute sets the label.
  • Responsive: the panel ranges from narrow to the full conversation width.
  • Kept inputs: inputs, selects, and textareas with an id or name keep their values on their own, across a reload, an update, and a switch to another version, matched by that id or name (a radio group by its name). When the page loads, Chat sets the kept values and fires input and change on each, so a page that redraws on those events shows them. Give a field a new id when its meaning changes between versions. Never kept: password, file, and hidden inputs, fields with autocomplete="off", and texts over 32K characters, which a page can keep with saveState.
  • Saved state: to keep anything else the user did in the page, call window.mindroom.saveState(value) on every change; Chat batches the writes, and a value over 256K characters of JSON throws. Later loads of the canvas, after a reload, an update, or a switch to another version, start with it in window.mindroom.state, which is undefined when nothing is saved. Read it defensively, since an earlier version of the page may have saved it in another shape. It stays in that browser and reaches the agent only through an answer, or when the canvas shares it. The browser keeps it for the 100 most recently saved canvases and deletes it when the user logs out.
  • Checking a page: an agent with the browser tool can open its page file with browser_control(action="open", paths=[...]) and take a screenshot before showing it. Chat's theme variables and window.mindroom are not there, so a page that gives its var(--mr-...) colors fallbacks looks closest to the panel. The browser also does not block everything the panel blocks.
  • Limits: the title is one line of at most 120 characters, and the page is UTF-8 text of at most 4 MB. Each update of a page larger than about 24 KB stores a new copy on the homeserver that users cannot delete, so iterate on large pages sparingly.

Chat exposes its current theme as CSS variables, so a page can match light and dark mode:

Variable Use
--mr-bg Page background (the default body background)
--mr-surface, --mr-surface-raised Cards and raised elements
--mr-border Borders and dividers
--mr-text, --mr-text-muted Text (the default body color) and secondary text
--mr-accent, --mr-accent-text Primary actions and the text on them
--mr-success, --mr-warning, --mr-danger Status colors
--mr-radius, --mr-font Corner radius and font family

For choices the variables cannot make, such as a chart palette, window.mindroom.colorScheme is light or dark.

Privacy and safety

  • The page cannot read the user's Matrix account, messages, cookies, or storage, and it cannot open pop-ups or navigate away; a page that tries is stopped, and Chat shows "This panel tried to leave its sandbox and was stopped." with a Reload panel button.
  • In an encrypted room, the page, the answer, and a shared copy of the page's state are end-to-end encrypted like other messages.
  • A canvas shared with share_state=True lets its agent, and anyone else in the room, read what the page saved without asking each time; Chat says so in the panel. Passwords and fields with autocomplete="off" are never kept automatically, but whatever the page passes to saveState is shared, so pages must keep secrets out of it.
  • A canvas is not a safe place for secrets: browsers do not let a page block WebRTC, and a page that loads libraries or remote assets can put data in the addresses it requests, so a malicious page could leak what the user types into it. Chat therefore names the agent that made each canvas and reminds the user that what they enter may leave the panel.
  • Agents should show only pages they wrote, never downloaded or untrusted pages, because every canvas appears as the agent's own.