mlflow/mlflow · warning

mlflow: flush failed (${reason}) attempt ${attempt}/${attemp

Error message

mlflow: flush failed (${reason}) attempt ${attempt}/${attempts}: ${String(err)}

What it means

The OpenClaw integration's `flushWithRetry` attempts to flush pending traces to MLflow with bounded retries and exponential backoff; each failed attempt emits this structured warning including the flush reason, attempt count, and error. After exhausting attempts it gives up, so traces from that flush may be lost.

Source

Thrown at libs/typescript/integrations/openclaw/src/service.ts:421

    };

    activeTraces.set(sessionKey, trace);
    evictOldest(activeTraces, MAX_ACTIVE_TRACES);

    return trace;
  }

  // Flush with exponential backoff retry
  async function flushWithRetry(reason: string): Promise<void> {
    const attempts = DEFAULT_FLUSH_RETRY_COUNT + 1;
    for (let attempt = 1; attempt <= attempts; attempt++) {
      try {
        await flushTraces();
        metrics.flushSuccesses += 1;
        return;
      } catch (err) {
        metrics.flushFailures += 1;
        log.warn(`mlflow: flush failed (${reason}) attempt ${attempt}/${attempts}: ${String(err)}`);
        if (attempt >= attempts) {
          return;
        }
        metrics.flushRetries += 1;
        const delayMs = Math.min(
          DEFAULT_FLUSH_RETRY_BASE_DELAY_MS * 2 ** (attempt - 1),
          MAX_FLUSH_RETRY_DELAY_MS,
        );
        await sleep(delayMs);
      }
    }
  }

  // =====================================================================
  // registerHooks — MUST be called during register(), not start().
  // OpenClaw only accepts api.on() subscriptions during the register phase.
  // Hooks guard on `initialized` so events before SDK init are silently skipped.
  // =====================================================================

View on GitHub (pinned to 6a27f2decc)

Solutions

1. Check the embedded `String(err)` for the root cause (connection refused vs HTTP status)
2. Verify the tracking server is reachable and the tracking URI is correct
3. Increase retry attempts/backoff for shutdown-time flushes
4. Ensure `stop()` is awaited and the server is available before process exit

Example fix

// before
service.stop() // fire-and-forget flush can fail unobserved
// after
await service.stop() // ensures flushWithRetry completes; inspect warnings if any

// health-check the tracking server before flush-heavy operations
const res = await fetch(`${trackingUri}/health`)
if (!res.ok) throw new Error('tracking server unavailable; flush will retry and may fail')

function isRetryable(err: unknown): boolean {
  const s = String(err)
  return /ECONNREFUSED|ETIMEDOUT|5\d\d/.test(s)
}

try {
  await flushWithRetry('stop')
} catch (err) {
  console.warn('flush ultimately failed after retries:', String(err))
}

When it happens

Trigger: Calling `stop()`, sweep intervals, or hook-triggered flushes when the MLflow tracking server is unreachable, returns errors (4xx/5xx), or artifact upload fails; retries continue until `attempts` is reached.

Common situations: MLflow server down or restarted during shutdown; wrong tracking URI; network timeouts in CI; auth token expired mid-session; flushing large trace payloads hitting request size limits.

Related errors

• Failed to download trace data for trace ${traceInfo.traceId}
• Azure ADLS Gen2 file creation failed: ${createResponse.statu
• Azure ADLS Gen2 data append failed: ${appendResponse.status}
• Azure ADLS Gen2 flush failed: ${flushResponse.status} ${flus
• Failed to upload to Azure ADLS Gen2: ${(error as Error).mess

AI-assisted analysis of mlflow/mlflow@6a27f2decc (2026-08-29). Data as JSON: /api/errors/6e1bf20ffb53b62a. Report an issue: GitHub.