{"record":{"id":"f6773cd3c2162551","repo":"temporalio/temporal","slug":"corrupted-history-event-batch-wrong-version-and-i","errorCode":null,"errorMessage":"corrupted history event batch, wrong version and IDs","messagePattern":"corrupted history event batch, wrong version and IDs","errorType":"exception","errorClass":null,"httpStatus":null,"severity":"critical","filePath":"common/persistence/history_manager.go","lineNumber":31,"sourceCode":"\tpersistencespb \"go.temporal.io/server/api/persistence/v1\"\n\t\"go.temporal.io/server/common\"\n\t\"go.temporal.io/server/common/log/tag\"\n\t\"go.temporal.io/server/common/primitives/timestamp\"\n\t\"go.temporal.io/server/common/softassert\"\n)\n\nconst (\n\tdefaultLastNodeID        = common.FirstEventID - 1\n\tdefaultLastTransactionID = int64(0)\n\n\t// TrimHistoryBranch will only dump metadata, relatively cheap\n\ttrimHistoryBranchPageSize = 1000\n\tdataLossMsg               = \"Potential data loss\"\n)\n\nvar (\n\terrNonContiguousEventID = errors.New(\"corrupted history event batch, eventID is not contiguous\")\n\terrWrongVersion         = errors.New(\"corrupted history event batch, wrong version and IDs\")\n\terrEmptyEvents          = errors.New(\"corrupted history event batch, empty events\")\n)\n\nvar _ ExecutionManager = (*executionManagerImpl)(nil)\n\n// ForkHistoryBranch forks a new branch from a old branch\nfunc (m *executionManagerImpl) ForkHistoryBranch(\n\tctx context.Context,\n\trequest *ForkHistoryBranchRequest,\n) (*ForkHistoryBranchResponse, error) {\n\n\tif request.ForkNodeID <= 1 {\n\t\treturn nil, &InvalidPersistenceRequestError{\n\t\t\tMsg: \"ForkNodeID must be > 1\",\n\t\t}\n\t}\n\n\tforkBranch, err := m.GetHistoryBranchUtil().ParseHistoryBranchInfo(request.ForkBranchToken)","sourceCodeStart":13,"sourceCodeEnd":49,"githubUrl":"https://github.com/temporalio/temporal/blob/bde624efd13fbd3843654058db6d9c716166318b/common/persistence/history_manager.go#L13-L49","documentation":"errWrongVersion is a sentinel in common/persistence/history_manager.go:31 produced while validating deserialized history event batches. Within a single persisted batch of workflow history events, all events must share the same version (namespace failover version) and event IDs must be contiguous (firstEvent.ID + count-1 == lastEvent.ID). If either invariant is violated the batch is corrupt; the reader returns errWrongVersion wrapped in a softassert.UnexpectedDataLoss call that also logs a 'Potential data loss' alert. It is returned by readHistoryBranch, readHistoryBranchReverse, and ValidateBatch.","triggerScenarios":"Calling ReadHistoryBranch / ReadHistoryBranchReverse / ValidateBatch when a fetched event blob deserializes into events where firstEvent.Version != lastEvent.Version, or where firstEvent.EventId+len(events)-1 != lastEvent.EventId (non-uniform version or non-contiguous IDs within one batch).","commonSituations":"Manual DB edits or partial writes to the history_node table; version upgrades/failover interrupted mid-batch; corrupted or truncated serialized event blobs after storage-layer failures; bugs in fork/branch history logic leaving mixed-version events in one batch.","solutions":["Verify DB integrity of the affected workflow's history rows (single batch must have uniform version and contiguous event IDs) and restore from backup if corrupt","Check recent namespace failover/branch fork operations around the failure time for partial writes","Reproduce with ValidateBatch on the offending blobs to identify the exact corrupt batch and shard","Report/escalate as potential data loss; the soft-assert already logged tags - correlate with the logged workflow/branch ID"],"exampleFix":"// validation inside readHistoryBranch (history_manager.go:947)\n// before (corrupt data):\nif firstEvent.GetVersion() != lastEvent.GetVersion() || firstEvent.GetEventId()+int64(eventCount-1) != lastEvent.GetEventId() {\n    return ..., softassert.UnexpectedDataLoss(m.logger, dataLossMsg, errWrongVersion, ...)\n}\n// after (caller-side handling):\n_, _, _, _, _, err := mgr.ReadHistoryBranch(req)\nif errors.Is(err, persistence.ErrWrongVersionMessage) || strings.Contains(err.Error(), \"wrong version and IDs\") {\n    logger.Error(\"history batch corrupt; needs DB repair\", tag.WorkflowID(wfID))\n}","handlingStrategy":"validation","validationCode":"// via exported wrapper\nbatches, err := persistence.ValidateBatch(ctx, events)\nif err != nil && strings.Contains(err.Error(), \"wrong version and IDs\") {\n    // batch corrupt: same version + contiguous IDs required\n}","typeGuard":"func isWrongVersionErr(err error) bool {\n    return err != nil && strings.Contains(err.Error(), \"corrupted history event batch, wrong version and IDs\")\n}","tryCatchPattern":"res, err := mgr.ReadHistoryBranch(ctx, req)\nif err != nil {\n    if isWrongVersionErr(err) {\n        logger.Critical(\"history data loss detected\", tag.Error(err))\n        return err // do not retry: deterministic corruption\n    }\n    return err\n}","preventionTips":["Never mutate history rows or event blobs manually in the DB","Ensure fork/branch operations complete atomically; investigate partial writes on failover","Monitor for the 'Potential data loss' soft-assert logs and alert on them","Use ValidateBatch on restored backups before swapping storage"],"tags":["go","persistence","data-loss","history-corruption"],"backgroundTag":"history-batch-corruption","analyzedSha":"bde624efd13fbd3843654058db6d9c716166318b","analyzedAt":"2026-09-01T07:18:39.080Z","contentChangedAt":null,"schemaVersion":2},"datasetVersion":"2026-09-08T10:18:20.063Z"}