mlflow/mlflow · error · MlflowException

{exception_header} with string representation '{raw_artifact

Error message

{exception_header} with string representation '{raw_artifact}' that is neither a valid path to a file nor a JSON string.

What it means

_infer_artifact_type_and_ext determines how a custom metric value passed to mlflow.evaluate should be logged as an artifact. If a string is neither an existing file path nor parseable JSON, MLflow raises this MlflowException because it cannot infer a serialization type. The {exception_header} prefix names the offending argument (e.g. 'Value ... for custom metric "x"').

Source

Thrown at mlflow/models/evaluation/artifacts.py:165

    exception_header = (
        f"Custom metric function '{custom_metric_tuple.name}' at index "
        f"{custom_metric_tuple.index} in the `custom_metrics` parameter produced an "
        f"artifact '{artifact_name}'"
    )

    # Given a string, first see if it is a path. Otherwise, check if it is a JsonEvaluationArtifact
    if isinstance(raw_artifact, str):
        potential_path = pathlib.Path(raw_artifact)
        if potential_path.exists():
            raw_artifact = potential_path
        else:
            try:
                json.loads(raw_artifact)
                return _InferredArtifactProperties(
                    from_path=False, type=JsonEvaluationArtifact, ext=".json"
                )
            except JSONDecodeError:
                raise MlflowException(
                    f"{exception_header} with string representation '{raw_artifact}' that is "
                    f"neither a valid path to a file nor a JSON string."
                )

    # Type inference based on the file extension
    if isinstance(raw_artifact, pathlib.Path):
        if not raw_artifact.exists():
            raise MlflowException(f"{exception_header} with path '{raw_artifact}' does not exist.")
        if not raw_artifact.is_file():
            raise MlflowException(f"{exception_header} with path '{raw_artifact}' is not a file.")
        if raw_artifact.suffix not in _EXT_TO_ARTIFACT_MAP:
            raise MlflowException(
                f"{exception_header} with path '{raw_artifact}' does not match any of the supported"
                f" file extensions: {', '.join(_EXT_TO_ARTIFACT_MAP.keys())}."
            )
        return _InferredArtifactProperties(
            from_path=True, type=_EXT_TO_ARTIFACT_MAP[raw_artifact.suffix], ext=raw_artifact.suffix
        )

View on GitHub (pinned to 6a27f2decc)

Solutions

Serialize the string as JSON before returning it (json.dumps(value)).
Pass a real pathlib.Path to an existing supported file instead of a string path.
Return a typed EvaluationArtifact instance (e.g. JsonEvaluationArtifact) to skip inference.
If the value is genuinely text, wrap it as {"text": value} and log via a JSON artifact.

Example fix

// before
// def my_metric(...):
//     return "model said: hello"
// after
// import json
// def my_metric(...):
//     return json.dumps({"text": "model said: hello"})

Defensive patterns

Strategy: validation

Validate before calling

import json, pathlib
def inferable(v):
    if isinstance(v, pathlib.Path):
        return v.exists()
    if isinstance(v, str):
        return pathlib.Path(v).exists()
    try:
        json.loads(v)
        return True
    except Exception:
        return False
assert inferable(value), f"custom metric return {value!r} is neither an existing path nor JSON"

Type guard

import json, pathlib
def is_json_str(v: str) -> bool:
    try:
        json.loads(v)
        return True
    except json.JSONDecodeError:
        return False

Try / catch

from mlflow.exceptions import MlflowException
try:
    result = mlflow.evaluate(...)
except MlflowException as e:
    if "neither a valid path to a file nor a JSON string" in str(e):
        raise SystemExit("Fix custom metric return: wrap strings with json.dumps or return a real Path")
    raise

Prevention

Always json.dumps plain strings returned from custom metrics
Return pathlib.Path objects (not str) when referencing files
Prefer returning typed EvaluationArtifact instances to skip inference entirely

When it happens

Trigger: Returning a plain non-JSON string from a custom metric/float-castable value passed as an artifact, e.g. `pred = "result file abc"` or a string that is a path-like name but doesn't exist, and not a pathlib.Path — the JSON decode fallback then fails and this is raised.

Common situations: Returning free-form text (e.g. an LLM completion or error message) from a custom metric without specifying artifact_type; typos in file paths (string path pointing to a missing file); forgetting to wrap content in json.dumps.

Related errors

AI-assisted analysis of mlflow/mlflow@6a27f2decc (2026-08-29). Data as JSON: /api/errors/ee6fa0867cc81236. Report an issue: GitHub.