What causes "Failed to search traces" in mlflow/mlflow?

Retry the search — the failure is often transient on the long-running operation path

mlflow/mlflow · error · MlflowException

INTERNAL_ERROR

INTERNAL_ERROR

Error message

Failed to search traces

What it means

`search_traces` on Databricks V4 runs as a long-running operation; `_poll_search_traces_operation` polls the operation and, if the returned operation carries an `error` field, raises it as an MlflowException. When the server sent no message or no error code, MLflow defaults to the generic message "Failed to search traces" with INTERNAL_ERROR.

Source

Thrown at mlflow/store/tracking/databricks_rest_store.py:550

        trace_infos = [TraceInfo.from_proto(t) for t in response_proto.trace_infos]
        return trace_infos, response_proto.next_page_token or None

    def _poll_search_traces_operation(
        self,
        operation: SearchTracesOperation,
        *,
        poll_interval_seconds: float = _SEARCH_TRACES_POLL_INTERVAL_SECONDS,
    ) -> SearchTracesOperation:
        while not operation.done:
            time.sleep(poll_interval_seconds)
            operation = self._call_endpoint(
                GetOperationRequest,
                None,
                endpoint=f"{_V4_TRACE_REST_API_PATH_PREFIX}/search/operations/{operation.name}",
                response_proto=SearchTracesOperation(),
            )
        if operation.HasField("error"):
            raise MlflowException(
                operation.error.message or "Failed to search traces",
                error_code=operation.error.error_code or ErrorCode.Name(INTERNAL_ERROR),
            )
        return operation

    def _search_unified_traces(
        self,
        model_id: str,
        locations: list[str],
        filter_string: str | None = None,
        max_results: int = SEARCH_TRACES_DEFAULT_MAX_RESULTS,
        order_by: list[str] | None = None,
        page_token: str | None = None,
    ) -> tuple[list[TraceInfo], str | None]:
        sql_warehouse_id = MLFLOW_TRACING_SQL_WAREHOUSE_ID.get()
        if sql_warehouse_id is None:
            raise MlflowException.invalid_parameter_value(
                "SQL warehouse ID is required for searching traces by model ID in UC tables, "

View on GitHub (pinned to 6a27f2decc)

Solutions

Retry the search — the failure is often transient on the long-running operation path
Inspect server-side logs / operation status for the underlying cause; re-run with fewer results or narrower locations
Verify permissions and existence of the UC table / experiment being searched
Upgrade MLflow and Databricks backend; if the server error is recurring, file a report with the operation name

Example fix

// before
traces = client.search_traces(locations=[exp_id])  # sporadic INTERNAL_ERROR
// after
from tenacity import retry, stop_after_attempt
@retry(stop=stop_after_attempt(3))
def safe_search():
    return client.search_traces(locations=[exp_id])

Defensive patterns

Strategy: retry

Validate before calling

# preflight: verify the target is searchable
client.get_experiment(experiment_id)  # raises early on permission/existence issues

Try / catch

from mlflow.exceptions import MlflowException
import time
for attempt in range(3):
    try:
        traces = store.search_traces(locations=locations, max_results=50)
        break
    except MlflowException as e:
        if e.error_code == "INTERNAL_ERROR" and attempt < 2:
            time.sleep(2 ** attempt)
            continue
        raise

Prevention

Retry transient INTERNAL_ERROR searches with backoff
Reduce max_results / narrow locations for very large searches
Verify UC table permissions and existence before searching
Keep MLflow and the Databricks backend up to date

When it happens

Trigger: The server-side search operation failed asynchronously (e.g., backend query failure on the trace store, timeout, permission/permission-adjacent server error) and the polled operation response contains an error with an empty message.

Common situations: Large searches timing out server-side; transient Databricks backend failures; UC table issues (missing permissions, table dropped mid-query); flaky workspace availability.

Related errors

AI-assisted analysis of mlflow/mlflow@6a27f2decc (2026-08-29). Data as JSON: /api/errors/d51fe1b6461b4c05. Report an issue: GitHub.