ruvnet/ruflo · error · ResumeFailedError

--resume checkpoint not found: ${resumeFrom}

Error message

--resume checkpoint not found: ${resumeFrom}

What it means

Thrown as a ResumeFailedError by runNativeTraining() when the --resume option points at a path that does not exist on disk (existsSync returns false). This is the loud-failure design for explicit resume: a missing checkpoint must NOT silently fall through to fresh training, because that would hide data loss and destroy the epoch-position/optimizer-state the user expected to restore. ResumeFailedError is special-cased in the outer catch to re-throw rather than degrade to the null/WASM fallback used for other native-training failures.

Source

Thrown at v3/@claude-flow/cli/src/services/native-training.ts:116

      epochs,
      inputDim: dim,
      outputDim: dim,
    };
    // 0 (or omitted) disables validation; only pass a real holdout through.
    if (typeof validationSplit === 'number' && validationSplit > 0) {
      pipelineConfig.validationSplit = validationSplit;
    }
    const pipeline = new ruvllm.TrainingPipeline(pipelineConfig);

    // Resume BEFORE training. Prefer resumeFrom() (2.6.0 — epoch position +
    // optimizer state); fall back to loadCheckpoint() (2.5.7 — weights only).
    // Any failure with an explicit --resume is loud (ResumeFailedError),
    // never silent fresh training.
    let resumed = false;
    let resumeMode: 'resumeFrom' | 'loadCheckpoint' | undefined;
    if (resumeFrom) {
      if (!existsSync(resumeFrom)) {
        throw new ResumeFailedError(`--resume checkpoint not found: ${resumeFrom}`);
      }
      try {
        if (typeof pipeline.resumeFrom === 'function') {
          pipeline.resumeFrom(resumeFrom);
          resumeMode = 'resumeFrom';
        } else {
          const ok = pipeline.loadCheckpoint(resumeFrom);
          if (ok === false) throw new Error('loadCheckpoint returned false');
          resumeMode = 'loadCheckpoint';
        }
        resumed = true;
      } catch (e) {
        if (e instanceof ResumeFailedError) throw e;
        throw new ResumeFailedError(
          `--resume failed to load checkpoint ${resumeFrom}: ${(e as Error).message}`,
        );
      }
    }

View on GitHub (pinned to 6b01dc5a68)

Solutions

  1. Verify the file exists at the absolute path before passing it: use fs.realpathSync or fs.existsSync and log the resolved path.
  2. If the checkpoint was lost, drop --resume and start fresh training (the error is intentional — do not paper over it).
  3. Ensure ruvllm >=2.5.7 so that prior runs actually persist checkpoints, and confirm the checkpointPath you resume from is the one a successful run reported.

Example fix

// before
await runNativeTraining({ embeddings, epochs, batchSize, learningRate, dim, resumeFrom: opts.resume })
// after — validate existence with an absolute path first
import { existsSync, realpathSync } from 'fs';
import { resolve } from 'path';
const ckpt = opts.resume ? realpathSync(resolve(opts.resume)) : undefined;
if (opts.resume && !existsSync(ckpt)) {
  throw new Error(`refusing to resume: checkpoint not found at ${ckpt}`);
}
await runNativeTraining({ embeddings, epochs, batchSize, learningRate, dim, resumeFrom: ckpt })
Defensive patterns

Strategy: validation

Validate before calling

import { existsSync, realpathSync } from 'fs';
import { resolve } from 'path';
function resolveResumePath(resumeFrom?: string): string | undefined {
  if (!resumeFrom) return undefined;
  const abs = resolve(resumeFrom);
  if (!existsSync(abs)) {
    throw new Error(`--resume checkpoint not found at ${abs}; refusing to fresh-train silently`);
  }
  return realpathSync(abs);
}

Type guard

import { existsSync } from 'fs';
function isExistingCheckpoint(path: string): boolean {
  return typeof path === 'string' && path.length > 0 && existsSync(path);
}

Try / catch

import { ResumeFailedError } from './native-training.js';
try {
  await runNativeTraining({ ...opts, resumeFrom });
} catch (e) {
  if (e instanceof ResumeFailedError) {
    // explicit resume failure — do NOT retry as fresh training;
    // either restore the checkpoint file or drop --resume intentionally
    console.error(e.message);
    process.exit(1);
  }
  // other failures degrade to null/WASM fallback by design
}

Prevention

When it happens

Trigger: Invoking runNativeTraining({ resumeFrom: './ckpt.bin', ... }) where ./ckpt.bin was never written, was deleted, lives on a different machine, or the path is relative to the wrong working directory. Also fires when a previous training run failed before saveCheckpoint() wrote the file (checkpoint saving is best-effort and only present in ruvllm >=2.5.7).

Common situations: Pointing --resume at a checkpoint from a prior run that crashed before checkpointing; a path that was correct in one environment (CI) but absent in another (local); a relative path interpreted against an unexpected cwd; running on ruvllm <2.5.7 where saveCheckpoint writes nothing, so a 'checkpointPath' reported by a prior run never actually landed on disk.

Related errors


AI-assisted analysis of ruvnet/ruflo@6b01dc5a68 (2026-08-12). Data as JSON: /api/errors/7b440bcb2b5b6757. Report an issue: GitHub.