Media understanding

OpenClaw can summarize inbound media (image/audio/video) before the reply pipeline runs, so command parsing and routing work off short text instead of raw bytes. Understanding auto-detects local tools or provider keys, or you can configure explicit models. Original media is always delivered to the model as usual; when understanding fails or is disabled, the reply flow continues unchanged.

Vendor plugins register capability metadata (which provider supports which media type, default model, priority). OpenClaw core owns the shared tools.media config, fallback order, and reply-pipeline integration.

How it works

Config

tools.media holds a shared model list plus per-capability overrides:

{  tools: {    media: {      concurrency: 2, // max concurrent capability runs (default)      models: [/* shared list, gate with capabilities */],      image: {/* optional overrides */},      audio: {        /* optional overrides */        echoTranscript: true,        echoFormat: '📝 "{transcript}"',      },      video: {/* optional overrides */},    },  },}

Per-capability (image/audio/video) keys:

Deepgram-specific options go under providerOptions.deepgram (the top-level deepgram: { detectLanguage, punctuate, smartFormat } field is deprecated but still read).

Model entries

Each models[] entry is a provider entry (default) or a CLI entry:

{  type: "provider", // default if omitted  provider: "openai",  model: "gpt-5.6-sol",  prompt: "Describe the image in <= 500 chars.",  maxChars: 500,  maxBytes: 10485760,  timeoutSeconds: 60,  capabilities: ["image"], // optional, for multi-modal shared entries  profile: "vision-profile",  preferredProfile: "vision-fallback",}

{  type: "cli",  command: "gemini",  args: [    "-m",    "gemini-3-flash",    "--allowed-tools",    "read_file",    "Read the media at {{MediaPath}} and describe it in <= {{MaxChars}} characters.",  ],  maxChars: 500,  maxBytes: 52428800,  timeoutSeconds: 120,  capabilities: ["video", "image"],}

Provider credentials

Provider media understanding uses the same auth resolution as normal model calls: auth profiles, environment variables, then models.providers.<providerId>.apiKey. tools.media.*.models[] entries do not accept an inline apiKey field.

{  models: {    providers: {      openai: { apiKey: "<OPENAI_API_KEY>" },      moonshot: { apiKey: "<MOONSHOT_API_KEY>" },    },  },}

See Tools and custom providers for profiles, env vars, and custom base URLs.

Rules and behavior

• Media exceeding maxBytes skips that model and tries the next one.
• Audio files under 1024 bytes are treated as empty/corrupt and skipped before transcription; the agent gets a deterministic placeholder transcript instead.
• If the active primary image model already supports vision natively, OpenClaw skips the [Image] summary block and passes the original image into the model directly. MiniMax is an exception: minimax, minimax-cn, minimax-portal, and minimax-portal-cn always route image understanding through the plugin-owned MiniMax-VL-01 media provider, even if legacy MiniMax M2.x chat metadata claims image input (only MiniMax-M3 and later are treated as natively vision-capable).
• If a Gateway/WebChat primary model is text-only, image attachments are preserved as offloaded media://inbound/* refs so image/PDF tools or a configured image model can still inspect them instead of losing the attachment.
• Explicit openclaw infer image describe --file <path> --model <provider/model> (alias: openclaw capability image describe) runs that image-capable provider/model directly, including Ollama refs such as ollama/qwen2.5vl:7b when a matching image-capable model is configured under models.providers.ollama.models[].
• If <capability>.enabled is not false but no models are configured, OpenClaw tries the active reply model when its provider supports the capability.

Auto-detect (default)

When tools.media.<capability>.enabled is not false and no models are configured, OpenClaw tries these in order and stops at the first working option:

To disable auto-detection for a capability:

{  tools: {    media: {      audio: {        enabled: false,      },    },  },}

Proxy support (audio/video provider calls)

Provider-based audio and video understanding honors standard outbound proxy environment variables, including NO_PROXY/no_proxy bypass rules: HTTPS_PROXY, HTTP_PROXY, ALL_PROXY, https_proxy, http_proxy, all_proxy. Lowercase vars take precedence over uppercase. If none are set, media understanding uses direct egress; if the proxy value is malformed, OpenClaw logs a warning and falls back to direct fetch. Image understanding does not go through this proxy path.

Capabilities

Set capabilities on a models[] entry to restrict it to specific media types. For shared lists, OpenClaw infers defaults per bundled provider:

For CLI entries, set capabilities explicitly to avoid surprising matches; if omitted, the entry is eligible for every capability list it appears in.

Provider support matrix

Model selection guidance

• Prefer the strongest current-generation model for each media capability when quality and safety matter.
• For tool-enabled agents handling untrusted inputs, avoid older/weaker media models.
• Keep at least one fallback per capability for availability (quality model + faster/cheaper model).
• CLI fallbacks (whisper-cli, whisper, gemini) help when provider APIs are unavailable.
• Known file-output modes are authoritative: an empty or missing inferred transcript file produces no transcript instead of falling back to CLI progress output.
• parakeet-mlx: use --output-format txt (or all) with --output-dir and the default {filename} output template. The upstream PARAKEET_OUTPUT_FORMAT and PARAKEET_OUTPUT_TEMPLATE environment variables are also honored. OpenClaw reads <output-dir>/<media-basename>.txt; the default srt format, other formats, and custom output templates continue to use stdout.

Attachment policy

Per-capability attachments controls which attachments are processed:

When mode: "all", outputs are labeled [Image 1/2], [Audio 2/2], etc.

File-attachment extraction

• Extracted file text is wrapped as untrusted external content before it's appended to the media prompt, using boundary markers like <<&lt;EXTERNAL_UNTRUSTED_CONTENT id=&quot;...&quot;&gt;>> / <<&lt;END_EXTERNAL_UNTRUSTED_CONTENT id=&quot;...&quot;&gt;>> plus a Source: External metadata line.
• This path intentionally omits the long SECURITY NOTICE: banner to keep the media prompt short; the boundary markers and metadata still apply.
• A file with no extractable text gets [No extractable text].
• If a PDF falls back to rendered page images, OpenClaw forwards those images to vision-capable reply models and keeps the placeholder [PDF content rendered to images] in the file block.

Config examples

{  tools: {    media: {      models: [        { provider: "openai", model: "gpt-5.6-sol", capabilities: ["image"] },        {          provider: "google",          model: "gemini-3-flash-preview",          capabilities: ["image", "audio", "video"],        },        {          type: "cli",          command: "gemini",          args: [            "-m",            "gemini-3-flash",            "--allowed-tools",            "read_file",            "Read the media at {{MediaPath}} and describe it in <= {{MaxChars}} characters.",          ],          capabilities: ["image", "video"],        },      ],      audio: {        attachments: { mode: "all", maxAttachments: 2 },      },      video: {        maxChars: 500,      },    },  },}

{  tools: {    media: {      audio: {        enabled: true,        models: [          { provider: "openai", model: "gpt-4o-mini-transcribe" },          {            type: "cli",            command: "whisper",            args: ["--model", "base", "{{MediaPath}}"],          },        ],      },      video: {        enabled: true,        maxChars: 500,        models: [          { provider: "google", model: "gemini-3-flash-preview" },          {            type: "cli",            command: "gemini",            args: [              "-m",              "gemini-3-flash",              "--allowed-tools",              "read_file",              "Read the media at {{MediaPath}} and describe it in <= {{MaxChars}} characters.",            ],          },        ],      },    },  },}

{  tools: {    media: {      image: {        enabled: true,        maxBytes: 10485760,        maxChars: 500,        models: [          { provider: "openai", model: "gpt-5.6-sol" },          { provider: "anthropic", model: "claude-opus-4-8" },          {            type: "cli",            command: "gemini",            args: [              "-m",              "gemini-3-flash",              "--allowed-tools",              "read_file",              "Read the media at {{MediaPath}} and describe it in <= {{MaxChars}} characters.",            ],          },        ],      },    },  },}

{  tools: {    media: {      image: {        models: [          {            provider: "google",            model: "gemini-3.1-pro-preview",            capabilities: ["image", "video", "audio"],          },        ],      },      audio: {        models: [          {            provider: "google",            model: "gemini-3.1-pro-preview",            capabilities: ["image", "video", "audio"],          },        ],      },      video: {        models: [          {            provider: "google",            model: "gemini-3.1-pro-preview",            capabilities: ["image", "video", "audio"],          },        ],      },    },  },}

Status output

When media understanding runs, /status includes a per-capability summary line:

📎 Media: image ok (openai/gpt-5.6-sol) · audio ok (whisper-cli observed=metal)

For preflight inventory, run openclaw capability audio providers. Local rows show the local fallback winner separately from global provider selection, readiness, and separate capable/requested/observed backend fields. The same local selection is available as an informational doctor finding:

openclaw doctor --lint --only core/doctor/local-audio-acceleration --severity-min info

Notes

• Understanding is best-effort. Errors do not block replies.
• Attachments are still passed to models even when understanding is disabled.
• Use scope to limit where understanding runs (for example, only DMs).

Related

• Configuration
• Image & media support