mlflow/mlflow · error · MlflowTraceArchivalMalformedTrace

{str(e)}

Error message

{str(e)}

What it means

Raised as MlflowTraceArchivalMalformedTrace by ArtifactRepo.upload_archived_trace_data when spans_to_traces_data_pb(trace_data.spans) fails while converting the TraceData spans to OTLP protobuf. The original error message is preserved in the new message; it indicates the archived trace payload is structurally invalid and cannot be serialized.

Source

Thrown at mlflow/store/artifact/artifact_repo.py:603

    def upload_archived_trace_data(self, trace_data: TraceData) -> None:
        """
        Upload archived trace data as OTLP protobuf to ``traces.pb``.

        Args:
            trace_data: The archived trace data as a ``TraceData`` object.
        """
        from mlflow.exceptions import MlflowTraceArchivalMalformedTrace
        from mlflow.tracing.otel.otel_archival import spans_to_traces_data_pb

        if not isinstance(trace_data, TraceData):
            raise MlflowException.invalid_parameter_value(
                "Archived trace data must be a TraceData object."
            )
        try:
            data = spans_to_traces_data_pb(trace_data.spans)
        except (MlflowException, TypeError, ValueError) as e:
            raise MlflowTraceArchivalMalformedTrace(str(e)) from e
        self.upload_archived_trace_data_bytes(data)

    def upload_archived_trace_data_bytes(self, data: bytes) -> None:
        """
        Upload serialized archived trace data bytes to ``traces.pb``.

        Backends can override this hook to avoid the default temp-file staging path when their
        storage SDK supports in-memory uploads or more efficient multipart transfer primitives.
        Overriding is useful for remote object stores where direct byte uploads can reduce local
        disk I/O and let the backend apply transport-specific optimizations.
        """
        with _write_local_temp_trace_data_pb_file(data) as temp_file:
            self.log_artifact(temp_file)

    def upload_attachment(self, attachment_id: str, content_bytes: bytes) -> None:
        _validate_attachment_path(attachment_id)
        with tempfile.TemporaryDirectory() as temp_dir:
            temp_file = Path(temp_dir, attachment_id)

View on GitHub (pinned to 6a27f2decc)

Solutions

1. Read the wrapped message to find the offending span/field and fix the Span construction (valid trace_id, span_id, timestamps, attribute values).
2. Sanitize span attributes before conversion — drop or coerce values that are not OTLP-compatible primitives (str, bool, int, float, bytes).
3. Validate spans by round-tripping through spans_to_traces_data_pb locally before upload and skip/log malformed spans.
4. Catch MlflowTraceArchivalMalformedTrace at the call site and decide whether to skip archival for that trace.

Example fix

// before
repo.upload_archived_trace_data(TraceData(spans=raw_spans))

// after
from mlflow.exceptions import MlflowTraceArchivalMalformedTrace
try:
    repo.upload_archived_trace_data(TraceData(spans=raw_spans))
except MlflowTraceArchivalMalformedTrace as e:
    logger.warning("Skipping malformed trace: %s", e)

def spans_look_valid(trace_data):
    for s in trace_data.spans:
        if not s.trace_id or not s.span_id or s.start_time is None or s.end_time is None:
            return False
        for v in (s.attributes or {}).values():
            if not isinstance(v, (str, bool, int, float, bytes)):
                return False
    return True

from mlflow.exceptions import MlflowTraceArchivalMalformedTrace
try:
    repo.upload_archived_trace_data(trace_data)
except MlflowTraceArchivalMalformedTrace as e:
    logger.warning("Skipping malformed archived trace: %s", e)

When it happens

Trigger: Calling upload_archived_trace_data with a TraceData whose .spans contain malformed/None span fields (missing trace_id/span_id, invalid timestamps or attribute types) such that spans_to_traces_data_pb raises MlflowException, TypeError, or ValueError.

Common situations: Traces ingested from external systems with non-OTLP-compatible attribute values; partially constructed Span entities missing required identifiers; version mismatches between the SDK that produced the spans and the current MLflow protobuf schema.

Related errors

• Unsupported trace location type: {trace_location.type}
• INVALID_PARAMETER_VALUE
• INVALID_PARAMETER_VALUE
• INVALID_PARAMETER_VALUE
• trace_id is required but was empty

AI-assisted analysis of mlflow/mlflow@6a27f2decc (2026-08-29). Data as JSON: /api/errors/f71584616dabb2b5. Report an issue: GitHub.