{"record":{"id":"a862ede52550b87d","repo":"micro/go-micro","slug":"w-additionally-failed-to-checkpoint-failed-run","errorCode":null,"errorMessage":"%w; additionally failed to checkpoint failed run: %v","messagePattern":"%w; additionally failed to checkpoint failed run: (.+?)","errorType":"exception","errorClass":null,"httpStatus":null,"severity":"error","filePath":"flow/steps.go","lineNumber":604,"sourceCode":"\t\t\trun.Steps[i].Status = \"waiting\"\n\t\t\trun.Status = \"waiting\"\n\t\t\trun.Await = &AwaitState{Step: step.Name, Key: await.Key, Prompt: await.Prompt}\n\t\t\tif saveErr := f.save(ctx, run); saveErr != nil {\n\t\t\t\tspanErr = saveErr\n\t\t\t\treturn run, saveErr\n\t\t\t}\n\t\t\tf.log.Logf(logger.InfoLevel, \"Flow %s run %s waiting for input %q at step %q\", f.name, run.ID, await.Key, step.Name)\n\t\t\treturn run, nil\n\t\t}\n\t\tif err != nil {\n\t\t\tspanErr = err\n\t\t\trun.Steps[i].Status = \"failed\"\n\t\t\trun.Steps[i].Error = err.Error()\n\t\t\trun.Steps[i].ErrorKind = string(ai.ClassifyError(err))\n\t\t\trun.Status = \"failed\"\n\t\t\tif saveErr := f.save(ctx, run); saveErr != nil {\n\t\t\t\tspanErr = saveErr\n\t\t\t\treturn run, fmt.Errorf(\"%w; additionally failed to checkpoint failed run: %v\", err, saveErr)\n\t\t\t}\n\t\t\tf.record(resultFromRun(f.opts.TriggerTopic, run))\n\t\t\tf.log.Logf(logger.ErrorLevel, \"Flow %s run %s failed at step %q: %v\", f.name, run.ID, step.Name, err)\n\t\t\treturn run, err\n\t\t}\n\n\t\trun.State = out\n\t\trun.Steps[i].Status = \"done\"\n\t\trun.Steps[i].Result = truncate(out.String(), 200)\n\t\tif i+1 < len(steps) {\n\t\t\trun.State.Stage = steps[i+1].Name\n\t\t} else {\n\t\t\trun.State.Stage = \"\"\n\t\t}\n\t\tif err := f.save(ctx, run); err != nil {\n\t\t\tspanErr = err\n\t\t\treturn run, err\n\t\t}","sourceCodeStart":586,"sourceCodeEnd":622,"githubUrl":"https://github.com/micro/go-micro/blob/24529f140421a11a33b6999ab7944f2021cfd69c/flow/steps.go#L586-L622","documentation":"When a step fails after all retries, runFrom marks the run failed and attempts to persist that terminal state via f.save. If the checkpoint save itself also errors, the original step error is wrapped with this message appending the save failure, so both causes are visible. It signals that the run failed AND its failure state could not be recorded, leaving the checkpoint stale (e.g. still showing running).","triggerScenarios":"A step inside a flow run returns an error (after retries), and the subsequent f.checkpoint.Save(ctx, run) of the failed-run record also fails — e.g. the store is down, timed out, or the context was canceled (note the deferred cancel and ctx use).","commonSituations":"Checkpoint store outage or network partition coinciding with a genuine step failure; context deadline exceeded during a long failing step so the context is already canceled when save runs; misconfigured store credentials surfacing only on the write path.","solutions":["Fix the underlying store error shown after 'additionally failed to checkpoint failed run:' — that is the save failure cause.","Check step failure cause (the first %w) — fix the step error first; the save failure is secondary.","Add retries/timeout headroom in the checkpoint store backend and verify credentials/connectivity.","Inspect the checkpoint store for orphaned 'running' runs after such incidents and reconcile them manually or via ResumePending.","Avoid passing an already-expired/canceled context into runFrom so the terminal save can complete."],"exampleFix":"// before\nrun, err := f.Start(ctx, input) // ctx has 1s deadline, step takes 2s\n\n// after\nctx, cancel := context.WithTimeout(context.Background(), 30*time.Second)\ndefer cancel()\nrun, err := f.Start(ctx, input)","handlingStrategy":"try-catch","validationCode":"// ensure the checkpoint backend is reachable before starting flows\nif err := ckpt.Ping(ctx); err != nil {\n    return fmt.Errorf(\"checkpoint store unavailable: %w\", err)\n}","typeGuard":null,"tryCatchPattern":"run, err := f.Start(ctx, in)\nif err != nil && strings.Contains(err.Error(), \"additionally failed to checkpoint failed run\") {\n    log.Error(\"run failed AND state not persisted; check store health\", \"err\", err)\n    // fix cause, then reconcile orphaned runs via ResumePending\n    return reconcile(ctx, run)\n}","preventionTips":["Monitor checkpoint store health and alert before step failures cascade into save failures","Give flow operations generous context deadlines so terminal saves aren't canceled","Run periodic reconciliation for runs stuck in 'running' status","Log both wrapped causes — the step error and the save error — separately"],"tags":["go","flow","checkpoint","persistence"],"backgroundTag":"checkpoint-save-failed","analyzedSha":"24529f140421a11a33b6999ab7944f2021cfd69c","analyzedAt":"2026-09-01T02:52:24.923Z","contentChangedAt":null,"schemaVersion":2},"datasetVersion":"2026-09-08T05:18:18.240Z"}