BoundaryML/baml · error

Cannot convert media types to JSON

Error message

Cannot convert media types to JSON

What it means

The GEPA optimizer serializes BamlValue test-case data to JSON to build prompts and merge variants. BamlValue::Media (images/audio/PDFs attached to test cases) has no plain-JSON representation the optimizer supports, so baml_value_to_json bails instead of silently dropping or mis-encoding the media. Use of media inputs/outputs is therefore unsupported in optimizer-managed test cases.

Source

Thrown at engine/baml-runtime/src/optimize/gepa_runtime.rs:446

            Ok(serde_json::Value::Array(items?))
        }
        BamlValue::Map(map) => {
            let obj: Result<serde_json::Map<_, _>> = map
                .iter()
                .map(|(k, v)| baml_value_to_json(v).map(|jv| (k.clone(), jv)))
                .collect();
            Ok(serde_json::Value::Object(obj?))
        }
        BamlValue::Class(_, fields) => {
            let mut obj = serde_json::Map::new();
            for (k, v) in fields {
                obj.insert(k.clone(), baml_value_to_json(v)?);
            }
            Ok(serde_json::Value::Object(obj))
        }
        BamlValue::Enum(_, val) => Ok(serde_json::Value::String(val.clone())),
        BamlValue::Media(_) => {
            anyhow::bail!("Cannot convert media types to JSON")
        }
    }
}

View on GitHub (pinned to bd85ce9dee)

Solutions

1. Remove media (image/audio/pdf) values from the test cases used by the optimizer; use text-only fixtures for optimization runs
2. Point the optimizer at a different, text-only test subset (via test filters/tags)
3. If the media is small, replace it with a textual description or a base64 string the prompt can consume, adjusting the BAML types accordingly
4. Restrict multimodal testing to the regular test runner (`baml-cli test`) rather than the GEPA optimizer

Example fix

// before (test case)
test ParseInvoice {
  args { doc { image "./invoice.png" } }
}

// after (text-only for optimization)
test ParseInvoice {
  args { doc_text "Invoice #123, total $42.00 ..." }
}

import { BamlValue } from '@baml-lang/core';
function containsMedia(v: any): boolean {
  if (!v || typeof v !== 'object') return false;
  if (v.kind === 'media' || v.type === 'media') return true;
  return Object.values(v).some(containsMedia);
}
if (testCases.some((t) => containsMedia(t.args))) throw new Error('Optimizer test set contains media values; use text-only fixtures');

try {
  const result = await optimizer.proposeImprovements(testCases);
} catch (e) {
  if (/Cannot convert media types/i.test(e.message)) {
    console.warn('Falling back to text-only test subset for optimization');
    return optimizer.proposeImprovements(testCases.filter((t) => !containsMedia(t.args)));
  }
  throw e;
}

When it happens

Trigger: Running propose_improvements or merge_variants where a test case's args, labels, or output contains a BamlValue::Media (image/audio/pdf), reached via nested Maps/Lists converted by baml_value_to_json.

Common situations: Optimizing a function whose tests pass image/pdf inputs (e.g. OCR or vision tasks); GEPA optimizer runs configured over multimodal test suites; accidentally including a media field in a test-case parameter map.

Understand the failure class

Background: UnsupportedOperationException and "is not supported" errors: when a library deliberately refuses a call — this error's family across 30 libraries.

Related errors

• Invalid number: {n}
• unknown feedback field(s) {}; only "title" and "description"
• feedback field "{key}" must be a string
• feedback payload must be a JSON object like {"title": "..."}
• unsupported pack target `{target_triple}`

AI-assisted analysis of BoundaryML/baml@bd85ce9dee (2026-09-12). Data as JSON: /api/errors/731d8cde6cf13708. Report an issue: GitHub.