micro/go-micro · error

; additionally failed to checkpoint failed run

Error message

%w; additionally failed to checkpoint failed run: %v

What it means

When a step fails after all retries, runFrom marks the run failed and attempts to persist that terminal state via f.save. If the checkpoint save itself also errors, the original step error is wrapped with this message appending the save failure, so both causes are visible. It signals that the run failed AND its failure state could not be recorded, leaving the checkpoint stale (e.g. still showing running).

Solutions

1. Fix the underlying store error shown after 'additionally failed to checkpoint failed run:' — that is the save failure cause.
2. Check step failure cause (the first %w) — fix the step error first; the save failure is secondary.
3. Add retries/timeout headroom in the checkpoint store backend and verify credentials/connectivity.
4. Inspect the checkpoint store for orphaned 'running' runs after such incidents and reconcile them manually or via ResumePending.
5. Avoid passing an already-expired/canceled context into runFrom so the terminal save can complete.

Example fix

// before
run, err := f.Start(ctx, input) // ctx has 1s deadline, step takes 2s

// after
ctx, cancel := context.WithTimeout(context.Background(), 30*time.Second)
defer cancel()
run, err := f.Start(ctx, input)

// ensure the checkpoint backend is reachable before starting flows
if err := ckpt.Ping(ctx); err != nil {
    return fmt.Errorf("checkpoint store unavailable: %w", err)
}

run, err := f.Start(ctx, in)
if err != nil && strings.Contains(err.Error(), "additionally failed to checkpoint failed run") {
    log.Error("run failed AND state not persisted; check store health", "err", err)
    // fix cause, then reconcile orphaned runs via ResumePending
    return reconcile(ctx, run)
}

When it happens

Trigger: A step inside a flow run returns an error (after retries), and the subsequent f.checkpoint.Save(ctx, run) of the failed-run record also fails — e.g. the store is down, timed out, or the context was canceled (note the deferred cancel and ctx use).

Common situations: Checkpoint store outage or network partition coinciding with a genuine step failure; context deadline exceeded during a long failing step so the context is already canceled when save runs; misconfigured store credentials surfacing only on the write path.

Related errors

• flow checkpoint save
• agent checkpoint save
• flow has no checkpoint configured
• run not found
• agent: checkpointed run is input-required; resume with…

AI-assisted analysis of micro/go-micro@24529f1404 (2026-09-01). Data as JSON: /api/errors/a862ede52550b87d. Report an issue: GitHub.

Appendix: source

Thrown at flow/steps.go:604

			run.Steps[i].Status = "waiting"
			run.Status = "waiting"
			run.Await = &AwaitState{Step: step.Name, Key: await.Key, Prompt: await.Prompt}
			if saveErr := f.save(ctx, run); saveErr != nil {
				spanErr = saveErr
				return run, saveErr
			}
			f.log.Logf(logger.InfoLevel, "Flow %s run %s waiting for input %q at step %q", f.name, run.ID, await.Key, step.Name)
			return run, nil
		}
		if err != nil {
			spanErr = err
			run.Steps[i].Status = "failed"
			run.Steps[i].Error = err.Error()
			run.Steps[i].ErrorKind = string(ai.ClassifyError(err))
			run.Status = "failed"
			if saveErr := f.save(ctx, run); saveErr != nil {
				spanErr = saveErr
				return run, fmt.Errorf("%w; additionally failed to checkpoint failed run: %v", err, saveErr)
			}
			f.record(resultFromRun(f.opts.TriggerTopic, run))
			f.log.Logf(logger.ErrorLevel, "Flow %s run %s failed at step %q: %v", f.name, run.ID, step.Name, err)
			return run, err
		}

		run.State = out
		run.Steps[i].Status = "done"
		run.Steps[i].Result = truncate(out.String(), 200)
		if i+1 < len(steps) {
			run.State.Stage = steps[i+1].Name
		} else {
			run.State.Stage = ""
		}
		if err := f.save(ctx, run); err != nil {
			spanErr = err
			return run, err
		}

View on GitHub (pinned to 24529f1404)