{"record":{"id":"d51fe1b6461b4c05","repo":"mlflow/mlflow","slug":"internal-error-d51fe1","errorCode":"INTERNAL_ERROR","errorMessage":"Failed to search traces","messagePattern":"Failed to search traces","errorType":"exception","errorClass":"MlflowException","httpStatus":500,"severity":"error","filePath":"mlflow/store/tracking/databricks_rest_store.py","lineNumber":550,"sourceCode":"        trace_infos = [TraceInfo.from_proto(t) for t in response_proto.trace_infos]\n        return trace_infos, response_proto.next_page_token or None\n\n    def _poll_search_traces_operation(\n        self,\n        operation: SearchTracesOperation,\n        *,\n        poll_interval_seconds: float = _SEARCH_TRACES_POLL_INTERVAL_SECONDS,\n    ) -> SearchTracesOperation:\n        while not operation.done:\n            time.sleep(poll_interval_seconds)\n            operation = self._call_endpoint(\n                GetOperationRequest,\n                None,\n                endpoint=f\"{_V4_TRACE_REST_API_PATH_PREFIX}/search/operations/{operation.name}\",\n                response_proto=SearchTracesOperation(),\n            )\n        if operation.HasField(\"error\"):\n            raise MlflowException(\n                operation.error.message or \"Failed to search traces\",\n                error_code=operation.error.error_code or ErrorCode.Name(INTERNAL_ERROR),\n            )\n        return operation\n\n    def _search_unified_traces(\n        self,\n        model_id: str,\n        locations: list[str],\n        filter_string: str | None = None,\n        max_results: int = SEARCH_TRACES_DEFAULT_MAX_RESULTS,\n        order_by: list[str] | None = None,\n        page_token: str | None = None,\n    ) -> tuple[list[TraceInfo], str | None]:\n        sql_warehouse_id = MLFLOW_TRACING_SQL_WAREHOUSE_ID.get()\n        if sql_warehouse_id is None:\n            raise MlflowException.invalid_parameter_value(\n                \"SQL warehouse ID is required for searching traces by model ID in UC tables, \"","sourceCodeStart":532,"sourceCodeEnd":568,"githubUrl":"https://github.com/mlflow/mlflow/blob/6a27f2decc0b76eb1b54af31849784addb357dbc/mlflow/store/tracking/databricks_rest_store.py#L532-L568","documentation":"`search_traces` on Databricks V4 runs as a long-running operation; `_poll_search_traces_operation` polls the operation and, if the returned operation carries an `error` field, raises it as an MlflowException. When the server sent no message or no error code, MLflow defaults to the generic message \"Failed to search traces\" with INTERNAL_ERROR.","triggerScenarios":"The server-side search operation failed asynchronously (e.g., backend query failure on the trace store, timeout, permission/permission-adjacent server error) and the polled operation response contains an error with an empty message.","commonSituations":"Large searches timing out server-side; transient Databricks backend failures; UC table issues (missing permissions, table dropped mid-query); flaky workspace availability.","solutions":["Retry the search — the failure is often transient on the long-running operation path","Inspect server-side logs / operation status for the underlying cause; re-run with fewer results or narrower locations","Verify permissions and existence of the UC table / experiment being searched","Upgrade MLflow and Databricks backend; if the server error is recurring, file a report with the operation name"],"exampleFix":"// before\ntraces = client.search_traces(locations=[exp_id])  # sporadic INTERNAL_ERROR\n// after\nfrom tenacity import retry, stop_after_attempt\n@retry(stop=stop_after_attempt(3))\ndef safe_search():\n    return client.search_traces(locations=[exp_id])","handlingStrategy":"retry","validationCode":"# preflight: verify the target is searchable\nclient.get_experiment(experiment_id)  # raises early on permission/existence issues","typeGuard":null,"tryCatchPattern":"from mlflow.exceptions import MlflowException\nimport time\nfor attempt in range(3):\n    try:\n        traces = store.search_traces(locations=locations, max_results=50)\n        break\n    except MlflowException as e:\n        if e.error_code == \"INTERNAL_ERROR\" and attempt < 2:\n            time.sleep(2 ** attempt)\n            continue\n        raise","preventionTips":["Retry transient INTERNAL_ERROR searches with backoff","Reduce max_results / narrow locations for very large searches","Verify UC table permissions and existence before searching","Keep MLflow and the Databricks backend up to date"],"tags":["mlflow","databricks","tracing","internal-error","polling","transient"],"backgroundTag":"remote-operation-failed","analyzedSha":"6a27f2decc0b76eb1b54af31849784addb357dbc","analyzedAt":"2026-08-29T20:54:51.419Z","schemaVersion":2},"datasetVersion":"2026-08-29T22:17:34.462Z"}