AI & Generation

Use these tools to transcribe audio, generate images and videos, synthesize speech, and call provider-hosted media generation APIs.

What This Page Covers

This page documents the built-in tools in the ai-and-generation group. Use these tools when you need OpenAI- or Google-style multimodal generation, provider-specific media APIs, or text-to-speech and audio workflows.

Tools On This Page

• [openai] - OpenAI-backed transcription, image generation, and text-to-speech.
• [gemini] - Google-backed image generation and Vertex-only video generation.
• [groq] - Groq-backed audio transcription, translation, and speech generation.
• [replicate] - Replicate-hosted image or video generation from prompt-driven models.
• [fal] - Fal-hosted media generation and a fixed image-to-image workflow.
• [cartesia] - Voice listing, voice localization, and text-to-speech.
• [eleven_labs] - Voice listing, sound effect generation, and text-to-speech.
• [lumalabs] - Luma AI video generation and image-to-video workflows.
• [modelslabs] - ModelsLab media generation for PNG, JPG, MP4, GIF, MP3, and WAV outputs.

Common Setup Notes

Every tool on this page is status=requires_config in the live registry and is meant to be configured with provider credentials. For openai, gemini, groq, cartesia, eleven_labs, fal, and replicate, MindRoom applies the documented model defaults when no model is configured. Stored model settings and authored agent overrides take precedence over those defaults. These tools do not use an auth_provider, and src/mindroom/api/integrations.py currently only exposes Spotify OAuth routes, so setup is done through stored tool credentials or provider SDK environment variables rather than a dedicated dashboard OAuth flow. Password fields such as api_key should be stored through the dashboard or credential store instead of inline YAML. Missing optional dependencies can auto-install at first use unless MINDROOM_NO_AUTO_INSTALL_TOOLS=1 is set. Most generation calls on this page return ToolResult media attachments rather than only raw text, so they are best suited to agents that can pass generated images, videos, or audio back to the user. openai uses the OpenAI Python SDK and OPENAI_API_KEY. gemini uses GOOGLE_API_KEY in Gemini API mode, and MindRoom also maps provider name gemini to shared Google credentials in its provider credential helpers. The current upstream SDK implementations also honor provider env vars such as OPENAI_API_KEY, GOOGLE_API_KEY, GROQ_API_KEY, REPLICATE_API_KEY, FAL_API_KEY, CARTESIA_API_KEY, ELEVEN_LABS_API_KEY, LUMAAI_API_KEY, and MODELS_LAB_API_KEY.

The former desi_vocal toolkit was removed after DesiVocal announced its June 30, 2026 sunset. Use a supported speech toolkit such as eleven_labs or openai.

[openai]

openai is the general OpenAI media toolkit for audio transcription, image generation, and text-to-speech. The former dalle toolkit was removed because DALL-E 3 is no longer available through the OpenAI API. Replace the entire old dalle configuration block with this image-only configuration to use GPT Image 2.5 Sunburst:

agents:
  illustrator:
    tools:
      - openai:
          enable_transcription: false
          enable_speech_generation: false

Old dalle options such as model, n, size, quality, style, and enable_create_image do not transfer directly. Use the image_* options in the configuration table below for supported overrides, and update instructions that call create_image() to use generate_image().

What It Does

openai exposes transcribe_audio(audio_path), generate_image(prompt), and generate_speech(text_input). transcribe_audio() expects a local file path and sends it to the configured transcription model, which defaults to gpt-transcribe. generate_image() uses the configured image_model, defaults to gpt-image-2.5-sunburst, and returns attached image bytes rather than only a remote URL. generate_speech() uses the configured OpenAI TTS model, voice, and output format and returns an attached audio artifact.

Configuration

Example

agents:
  creator:
    tools:
      - openai:
          transcription_model: gpt-transcribe
          image_model: gpt-image-2.5-sunburst
          text_to_speech_voice: alloy

transcribe_audio("recordings/intro.wav")
generate_image("A retro-futurist Matrix control room with warm lighting.")
generate_speech("Status update complete.")

Notes

• transcribe_audio() expects a readable local path, not a URL.
• For image generation only, set enable_transcription: false and enable_speech_generation: false.

[gemini]

gemini is the Google media toolkit for image generation through Nano Banana and video generation through Veo.

What It Does

gemini exposes generate_image(prompt) and generate_video(prompt). generate_image() uses the Gemini content-generation API with the configured image_generation_model, which defaults to Nano Banana 2 (gemini-3.1-flash-image), and returns attached image bytes. generate_video() uses the configured video_generation_model, which defaults to Veo 3.1 (veo-3.1-generate-001), polls until the long-running operation completes, and returns attached video artifacts. The current implementation requires Vertex AI mode for video generation and returns an error if vertexai is not enabled. In non-Vertex mode, the tool uses the Gemini API through GOOGLE_API_KEY.

Configuration

Example

agents:
  studio:
    tools:
      - gemini:
          vertexai: true
          project_id: my-gcp-project
          location: us-central1
          image_generation_model: gemini-3.1-flash-image
          video_generation_model: veo-3.1-generate-001

generate_image("A minimal poster for a Matrix developer conference.")
generate_video("A slow cinematic flythrough of a neon data center.")

Notes

• generate_video() only works in Vertex AI mode on this branch.
• In MindRoom's provider credential helpers, gemini maps to shared Google credentials rather than its own independent provider bucket.
• The current tool polls every 5 seconds until the video operation finishes, and that polling interval is not exposed as a tool config field.

[groq]

groq is the audio-focused toolkit for fast transcription, translation, and speech generation.

What It Does

groq exposes transcribe_audio(audio_source), translate_audio(audio_source), and generate_speech(text_input). transcribe_audio() and translate_audio() accept either a local file path or a public URL. translate_audio() translates the source audio to English using the configured translation model. generate_speech() uses the configured Groq TTS model and voice and returns an attached WAV artifact. All three functions use the Groq SDK directly and require a Groq API key.

Configuration

Example

agents:
  audio:
    tools:
      - groq:
          transcription_model: whisper-large-v3
          tts_model: canopylabs/orpheus-v1-english
          tts_voice: troy

transcribe_audio("samples/interview.mp3")
translate_audio("https://example.com/spanish-briefing.mp3")
generate_speech("Your transcript is ready.")

Notes

• transcribe_audio() and translate_audio() are more flexible than [openai] because they accept either local files or public URLs.
• The current Groq TTS path always asks the API for wav output and returns an audio/wav artifact.
• Use [openai] instead if you want OpenAI transcription or OpenAI TTS specifically.

[replicate]

replicate is the generic Replicate wrapper for prompt-driven image or video generation.

What It Does

replicate exposes one call, generate_media(prompt). It runs the configured Replicate model with input={"prompt": prompt} and expects one FileOutput or an iterable of FileOutput objects. The current implementation infers whether each output is an image or a video from the returned file URL extension. Generated artifacts are attached by remote URL rather than downloaded into MindRoom-managed bytes.

Configuration

Example

agents:
  video:
    tools:
      - replicate:
          model: minimax/h3

generate_media("A short looping animation of code flowing across a terminal.")

Notes

• The current wrapper only supports models that accept a single prompt input field.
• Output parsing depends on file extensions in returned URLs, so nonstandard model outputs can fail even if the Replicate run itself succeeds.
• Use [fal], [lumalabs], or [modelslabs] instead when you want a narrower wrapper with a more opinionated provider-specific flow.

[fal]

fal is the Fal wrapper for prompt-driven media generation plus a dedicated image-to-image path.

What It Does

fal exposes generate_media(prompt) and, when enabled, image_to_image(prompt, image_url=None). generate_media() calls Fal with the configured model and a single prompt argument and returns the provider's image or video URLs as attachments. image_to_image() requires a source image URL and uses the fixed fal-ai/flux-2/edit workflow independently of the configured model. Both Fal methods use the configured API key and stream queue log messages to the MindRoom process logs while the job is running.

Configuration

Example

agents:
  visuals:
    tools:
      - fal:
          model: fal-ai/hunyuan-video-v1.5/text-to-video
          enable_image_to_image: true

generate_media("A cinematic drone shot over a rainy cyberpunk street.")
image_to_image(
    "Turn this product photo into a watercolor illustration.",
    image_url="https://example.com/source.png",
)

Notes

• model only affects generate_media().
• image_to_image() uses Fal's FLUX.2 edit route and sends the source image in image_urls.
• Returned media are attached by remote URL rather than stored bytes.

[cartesia]

cartesia is the voice toolkit for listing voices, localizing voices into new languages, and generating speech.

What It Does

cartesia exposes list_voices(), localize_voice(name, description, language, original_speaker_gender, voice_id=None), and text_to_speech(transcript, voice_id=None). list_voices() returns a filtered JSON list of voice IDs, names, descriptions, and languages. localize_voice() creates a localized derivative of an existing voice, using default_voice_id unless you pass a different voice_id. text_to_speech() uses the configured model_id and voice ID and returns attached MP3 audio bytes. The current implementation hardcodes MP3 output at 44.1 kHz and 128 kbps.

Configuration

Example

agents:
  voice:
    tools:
      - cartesia:
          model_id: sonic-3.6
          enable_localize_voice: true

list_voices()
localize_voice(
    name="French Support Voice",
    description="Warm and clear support voice.",
    language="fr",
    original_speaker_gender="female",
)
text_to_speech("Deployment complete.")

Notes

• localize_voice() is disabled by default, so voice cloning or localization is opt-in.
• voice_id can be overridden per call for both localize_voice() and text_to_speech().
• The current TTS path always returns MP3 bytes even though the tool config does not expose an output-format option.

[eleven_labs]

eleven_labs is the ElevenLabs toolkit for voices, sound effects, and text-to-speech.

What It Does

eleven_labs exposes get_voices(), generate_sound_effect(prompt, duration_seconds=None), and text_to_speech(prompt). get_voices() returns voice IDs, names, and descriptions from the ElevenLabs account. generate_sound_effect() turns a text description into an attached audio artifact. text_to_speech() uses the configured voice_id, model_id, and output_format and returns attached audio bytes. If target_directory is set, the current implementation also saves generated audio files to disk in that directory.

Configuration

Example

agents:
  audio_fx:
    tools:
      - eleven_labs:
          model_id: eleven_v3
          output_format: mp3_44100_64
          target_directory: generated-audio

get_voices()
generate_sound_effect("Mechanical keyboard typing in a quiet office.", duration_seconds=4)
text_to_speech("The build succeeded.")

Notes

• The default Eleven v3 model accepts up to 5,000 characters per request.
• target_directory is optional and only affects local file saving, not the returned attachment.
• The current implementation always emits audio/mpeg artifacts, even when you choose a PCM- or u-law-style output format.
• generate_sound_effect() is useful when you want non-speech audio from the same provider toolkit.

[lumalabs]

lumalabs is the Luma AI toolkit for text-to-video and image-to-video generation.

What It Does

lumalabs exposes generate_video(prompt, loop=False, aspect_ratio="16:9", keyframes=None) and image_to_video(prompt, start_image_url, end_image_url=None, loop=False, aspect_ratio="16:9"). Both calls create a Luma generation job and poll until it completes or times out. generate_video() optionally accepts provider-style keyframes, while image_to_video() builds the required keyframe structure from one or two image URLs. Completed jobs return remote video URL attachments. If wait_for_completion is false, the current implementation returns Async generation unsupported.

Configuration

Example

agents:
  motion:
    tools:
      - lumalabs:
          poll_interval: 5
          max_wait_time: 600

generate_video("A calm flythrough of a futuristic coworking space.", aspect_ratio="16:9")
image_to_video(
    "Animate this concept art into a short reveal shot.",
    start_image_url="https://example.com/frame0.png",
    end_image_url="https://example.com/frame1.png",
)

Notes

• image_to_video() requires remote image URLs, not local file paths.
• wait_for_completion: false does not currently provide a job handle or async response.
• Use [gemini] instead when you specifically want Google's Veo-backed video path.

[modelslabs]

modelslabs is the ModelsLab wrapper for provider-hosted PNG, JPG, MP4, GIF, MP3, or WAV generation.

What It Does

modelslabs exposes one call, generate_media(prompt). The current wrapper chooses one of several provider endpoints based on file_type and sends a fixed payload template for that media class. For PNG and JPG it uses the image endpoint; MP4 and GIF use the text-to-video endpoint and return future-link URLs with an ETA. For MP3 and WAV generation, it uses provider voice endpoints and returns audio URLs. If wait_for_completion is enabled, the tool polls the provider fetch endpoint until the media is ready or the timeout is reached.

Configuration

Example

agents:
  generator:
    tools:
      - modelslabs:
          file_type: gif
          wait_for_completion: true
          max_wait_time: 90

generate_media("A looping animation of messages flowing through a Matrix bridge.")

Notes

• Despite the broad provider branding, the current wrapper exposes one opinionated generate_media() path rather than a generic arbitrary-model interface.
• MP4 and GIF generation use a provider-side video template whose model and dimensions can be overridden with model_id, width, and height.
• Returned media are provider URLs, and the success message usually includes the provider ETA rather than immediate ready-to-view bytes.

Related Docs

• Tools Overview
• OpenAI-Compatible API