Unity-Technologies/ml-agents · error · TrainerError

Metadata not found, resuming from an incompatible version of

Error message

Metadata not found, resuming from an incompatible version of ML-Agents.

What it means

GlobalTrainingStatus.load_state() restores trainer metadata from a saved 'training_status.json' produced by a previous run. The loaded dict must contain a 'metadata' key; if it doesn't, ML-Agents concludes the checkpoint was written by an incompatible version and raises TrainerError instead of silently resuming with partial state.

Source

Thrown at ml-agents/mlagents/trainers/training_status.py:83

    def load_state(path: str) -> None:
        """
        Load a JSON file that contains saved state.
        :param path: Path to the JSON file containing the state.
        """
        try:
            with open(path) as f:
                loaded_dict = json.load(f)
            # Compare the metadata
            _metadata = loaded_dict[StatusType.STATS_METADATA.value]
            StatusMetaData.from_dict(_metadata).check_compatibility(StatusMetaData())
            # Update saved state.
            GlobalTrainingStatus.saved_state.update(loaded_dict)
        except FileNotFoundError:
            logger.warning(
                "Training status file not found. Not all functions will resume properly."
            )
        except KeyError:
            raise TrainerError(
                "Metadata not found, resuming from an incompatible version of ML-Agents."
            )

    @staticmethod
    def save_state(path: str) -> None:
        """
        Save a JSON file that contains saved state.
        :param path: Path to the JSON file containing the state.
        """
        GlobalTrainingStatus.saved_state[
            StatusType.STATS_METADATA.value
        ] = StatusMetaData().to_dict()
        with open(path, "w") as f:
            json.dump(GlobalTrainingStatus.saved_state, f, indent=4)

    @staticmethod
    def set_parameter_state(category: str, key: StatusType, value: Any) -> None:
        """

View on GitHub (pinned to 3ecb446f75)

Solutions

Regenerate the run with the ML-Agents version you are using now — start training fresh rather than resuming the old checkpoint.
Check out/pip-install the ML-Agents version that originally created the checkpoint, resume it there, then upgrade.
Inspect training_status.json and confirm/add a top-level "metadata" key if you know the file is valid and merely edited.
Delete or archive the stale training_status.json so load_state fails cleanly at start rather than mid-run.

Example fix

// before (resume old checkpoint)
mlagents-learn config.yaml --run-id=old-run --resume
// raises: Metadata not found...
// after (fresh run with current version)
mlagents-learn config.yaml --run-id=new-run

Defensive patterns

Strategy: validation

Validate before calling

import json
from mlagents.trainers.exception import TrainerError

def validate_status_file(path):
    with open(path) as f:
        state = json.load(f)
    if "metadata" not in state:
        raise ValueError(f"{path} lacks 'metadata'; incompatible with this ML-Agents version")
    return state

Type guard

def is_compatible_status(state: dict) -> bool:
    return isinstance(state, dict) and "metadata" in state

Try / catch

from mlagents.trainers.exception import TrainerError
try:
    GlobalTrainingStatus.load_state(path)
except TrainerError as e:
    if "Metadata not found" in str(e):
        logger.warning("Checkpoint from incompatible ML-Agents version; starting fresh run")
    else:
        raise

Prevention

Pin the same mlagents version for checkpointing and resuming.
Keep training_status.json out of manual edit workflows.
Store the mlagents version alongside run artifacts and check before resuming.
Archive old checkpoints instead of resuming across major version jumps.

When it happens

Trigger: Calling GlobalTrainingStatus.load_state(path) (via mlagents-learn --resume) on a training_status.json whose top-level dict lacks the 'metadata' key — i.e. the file was saved by an older ML-Agents release with a different state schema, or the file was hand-edited/corrupted and 'metadata' was removed.

Common situations: Resuming a run that was checkpointed with a much older ML-Agents version after upgrading the package; copying a training_status.json from a different project/branch; manual edits to the status file.

Related errors

AI-assisted analysis of Unity-Technologies/ml-agents@3ecb446f75 (2026-09-02). Data as JSON: /api/errors/4fa3dcbf53508828. Report an issue: GitHub.