mudler/LocalAI · error · ValueError

engine_args must be a JSON object, got {type(extra).__name__

Error message

engine_args must be a JSON object, got {type(extra).__name__}

What it means

Raised by _apply_engine_args in the LocalAI vLLM backend when the YAML model config's engine_args setting parses as valid JSON but is not a JSON object (dict). The value is meant to be overlaid onto a vllm AsyncEngineArgs dataclass via dataclasses.replace, which only works with key/value pairs. Any JSON array, string, number, or boolean triggers this ValueError.

Source

Thrown at backend/python/vllm/backend.py:138

        """Apply user-supplied engine_args (JSON object) onto an AsyncEngineArgs.

        Returns a new AsyncEngineArgs with the typed fields preserved and the
        user's overrides layered on top. Uses ``dataclasses.replace`` so vLLM's
        ``__post_init__`` re-runs and auto-converts dict-valued fields like
        ``compilation_config`` / ``attention_config`` into their dataclass form.
        ``speculative_config`` and ``kv_transfer_config`` are accepted as dicts
        directly (vLLM converts them at engine init).

        Unknown keys raise ValueError with the closest valid field as a hint.
        """
        if not engine_args_json:
            return engine_args
        try:
            extra = json.loads(engine_args_json)
        except json.JSONDecodeError as e:
            raise ValueError(f"engine_args is not valid JSON: {e}") from e
        if not isinstance(extra, dict):
            raise ValueError(
                f"engine_args must be a JSON object, got {type(extra).__name__}"
            )
        valid = {f.name for f in dataclasses.fields(type(engine_args))}
        for key in extra:
            if key not in valid:
                suggestion = difflib.get_close_matches(key, valid, n=1)
                hint = f" did you mean {suggestion[0]!r}?" if suggestion else ""
                raise ValueError(f"unknown engine_args key {key!r}.{hint}")
        return dataclasses.replace(engine_args, **extra)

    def _messages_to_dicts(self, messages):
        """Convert proto Messages to list of dicts suitable for apply_chat_template()."""
        result = []
        for msg in messages:
            d = {"role": msg.role, "content": msg.content or ""}
            if msg.name:
                d["name"] = msg.name
            if msg.tool_call_id:

View on GitHub (pinned to 44413a9d06)

Solutions

1. Change engine_args in the model YAML to a JSON object of AsyncEngineArgs field names, e.g. engine_args: '{"dtype": "auto", "max_model_len": 8192}'
2. Quote the whole JSON string in YAML if it contains special characters (single quotes are safest)
3. Remove the engine_args key entirely if you only need one of the typed fields (Quantization, MaxModelLen, GPUMemoryUtilization, ...) which have dedicated YAML keys
4. Check backend stderr — the backend prints 'engine_args error: ...' and exits the Load flow

Example fix

# before (model.yaml)
engine_args: 0.9
# after
engine_args: '{"gpu_memory_utilization": 0.9}'

import json
cfg = model_config.get('engine_args')
if cfg is not None:
    v = json.loads(cfg)
    assert isinstance(v, dict), f'engine_args must be a JSON object, got {type(v).__name__}'

def is_engine_args_object(s: str) -> bool:
    try:
        return isinstance(json.loads(s), dict)
    except json.JSONDecodeError:
        return False

try:
    load_model(cfg)
except ValueError as e:
    if 'engine_args must be a JSON object' in str(e):
        fix_engine_args_in_yaml(cfg)  # make it an object, reload
    else:
        raise

When it happens

Trigger: Setting engine_args: '[1,2,3]' or engine_args: '"auto"' or engine_args: '0.9' in a model YAML. The string is json.loads-ed successfully but isinstance(extra, dict) is False, so the backend raises before loading the model.

Common situations: Operators pasting a bare scalar (e.g. a dtype or tensor_parallel_size value) into engine_args instead of an object; wrapping keys in quotes so the whole thing parses as one string; copying a JSON array from docs; YAML unquoted scalars that look like numbers.

Related errors

• unknown engine_args key {key!r}.{hint}
• engine_args is not valid JSON: {e}
• HTTP ${response.status}
• Invalid YAML structure
• Model name is required

AI-assisted analysis of mudler/LocalAI@44413a9d06 (2026-08-15). Data as JSON: /api/errors/0380cfa2b071c673. Report an issue: GitHub.