AI & Generation

These tools let an agent transcribe and translate audio, generate images, video, speech, music, and sound effects, and work with provider voices. Pick the toolkit for the provider you have an API key for:

Setup

Every tool on this page needs a provider API key, except gemini in Vertex AI mode, which uses Google Cloud authentication. Set api_key through the dashboard or credential store, not inline in YAML (see Security Restrictions). When api_key is unset, each toolkit reads its provider environment variable, listed in its configuration table. Missing Python dependencies install automatically on first use (see Automatic Dependency Installation). Every toolkit except modelslabs has enable_* fields to turn individual functions on or off, and all: true enables every function. Generated media is attached to the agent's reply. openai, gemini, groq, cartesia, and eleven_labs return the media files, while replicate, fal, lumalabs, and modelslabs return provider-hosted URLs.

openai

openai provides transcribe_audio(audio_path), generate_image(prompt), and generate_speech(text_input). transcribe_audio() takes a local file path, not a URL. The path follows the agent's file_access: with the default workspace it must be a regular file inside the agent workspace, and relative paths resolve from there.

agents:
  illustrator:
    tools:
      - openai:
          enable_transcription: false
          enable_speech_generation: false

The dalle toolkit no longer exists, and a config that lists it fails with Unknown tool 'dalle'. Replace the whole dalle block with the image-only openai block above, move supported overrides to the image_* options, and change instructions that call create_image() to generate_image(). The desi_vocal toolkit also no longer exists; use openai or eleven_labs for speech.

gemini

gemini provides generate_image(prompt) and generate_video(prompt). Generated images are square (1:1). generate_video() works only in Vertex AI mode and otherwise returns Video generation requires Vertex AI mode.

One gemini toolkit uses a single Vertex location for both functions. Google lists global, us, and eu for Gemini 3.1 Flash Image and us-central1 for Veo 3.1, with no shared location. To use both on Vertex AI, give each to a separate agent and disable the other function:

agents:
  illustrator:
    tools:
      - gemini:
          vertexai: true
          project_id: my-gcp-project
          location: global
          enable_generate_video: false
  filmmaker:
    tools:
      - gemini:
          vertexai: true
          project_id: my-gcp-project
          location: us-central1
          enable_generate_image: false

groq

groq provides transcribe_audio(audio_source), translate_audio(audio_source), and generate_speech(text_input). transcribe_audio() and translate_audio() accept a public http or https URL or a local file path. A local path follows the agent's file_access: with the default workspace it must be a regular file inside the agent workspace, and relative paths resolve from there. translate_audio() translates the audio to English. generate_speech() always returns WAV audio.

replicate

replicate provides generate_media(prompt), which runs the configured Replicate model with only a prompt input. It works only with models that accept a single prompt field and return files whose URLs end in an image or video extension.

fal

fal provides generate_media(prompt) and, when enabled, image_to_image(prompt, image_url). generate_media() runs the configured model with only a prompt input. image_to_image() edits the image at image_url with Fal's FLUX.2 edit model, regardless of model.

cartesia

cartesia provides list_voices(), localize_voice(name, description, language, original_speaker_gender, voice_id=None), and text_to_speech(transcript, voice_id=None). localize_voice() creates a new voice in another language from an existing one. Both localize_voice() and text_to_speech() use default_voice_id unless the call passes voice_id. text_to_speech() always returns MP3 audio at 44.1 kHz and 128 kbps.

eleven_labs

eleven_labs provides get_voices(), generate_sound_effect(prompt, duration_seconds=None), and text_to_speech(prompt). The default Eleven v4 model accepts up to 10,000 characters per request. Audio attachments are always labeled MP3 (audio/mpeg), even when output_format selects PCM or u-law.

lumalabs

lumalabs provides generate_video(prompt, loop=False, aspect_ratio="16:9", keyframes=None) and image_to_video(prompt, start_image_url, end_image_url=None, loop=False, aspect_ratio="16:9"). image_to_video() animates between one or two remote image URLs; local file paths are not accepted. Both functions wait for the video until it finishes or max_wait_time passes. Keep wait_for_completion: true, because false makes both functions return Async generation unsupported without a job handle.

modelslabs

modelslabs provides generate_media(prompt), and file_type decides what it generates:

model_id, width, and height apply to images and video. Without wait_for_completion, a queued job returns its URLs and the provider ETA, and those URLs may not be ready yet. With wait_for_completion: true, the tool checks the job about once per second for the provider ETA plus add_to_eta attempts, capped at max_wait_time attempts, and returns the finished URLs. If the job is still unfinished at the cap, the tool returns the queued URLs, and the job may still complete on the provider side. A job the provider rejects returns the provider's message instead of media.

agents:
  generator:
    tools:
      - modelslabs:
          file_type: gif
          wait_for_completion: true
          max_wait_time: 90

Related Docs

• Tools Overview
• OpenAI-Compatible API