temporalio/temporal · critical

corrupted history event batch, wrong version and IDs

Error message

corrupted history event batch, wrong version and IDs

What it means

errWrongVersion is a sentinel in common/persistence/history_manager.go:31 produced while validating deserialized history event batches. Within a single persisted batch of workflow history events, all events must share the same version (namespace failover version) and event IDs must be contiguous (firstEvent.ID + count-1 == lastEvent.ID). If either invariant is violated the batch is corrupt; the reader returns errWrongVersion wrapped in a softassert.UnexpectedDataLoss call that also logs a 'Potential data loss' alert. It is returned by readHistoryBranch, readHistoryBranchReverse, and ValidateBatch.

Source

Thrown at common/persistence/history_manager.go:31

	persistencespb "go.temporal.io/server/api/persistence/v1"
	"go.temporal.io/server/common"
	"go.temporal.io/server/common/log/tag"
	"go.temporal.io/server/common/primitives/timestamp"
	"go.temporal.io/server/common/softassert"
)

const (
	defaultLastNodeID        = common.FirstEventID - 1
	defaultLastTransactionID = int64(0)

	// TrimHistoryBranch will only dump metadata, relatively cheap
	trimHistoryBranchPageSize = 1000
	dataLossMsg               = "Potential data loss"
)

var (
	errNonContiguousEventID = errors.New("corrupted history event batch, eventID is not contiguous")
	errWrongVersion         = errors.New("corrupted history event batch, wrong version and IDs")
	errEmptyEvents          = errors.New("corrupted history event batch, empty events")
)

var _ ExecutionManager = (*executionManagerImpl)(nil)

// ForkHistoryBranch forks a new branch from a old branch
func (m *executionManagerImpl) ForkHistoryBranch(
	ctx context.Context,
	request *ForkHistoryBranchRequest,
) (*ForkHistoryBranchResponse, error) {

	if request.ForkNodeID <= 1 {
		return nil, &InvalidPersistenceRequestError{
			Msg: "ForkNodeID must be > 1",
		}
	}

	forkBranch, err := m.GetHistoryBranchUtil().ParseHistoryBranchInfo(request.ForkBranchToken)

View on GitHub (pinned to bde624efd1)

Solutions

1. Verify DB integrity of the affected workflow's history rows (single batch must have uniform version and contiguous event IDs) and restore from backup if corrupt
2. Check recent namespace failover/branch fork operations around the failure time for partial writes
3. Reproduce with ValidateBatch on the offending blobs to identify the exact corrupt batch and shard
4. Report/escalate as potential data loss; the soft-assert already logged tags - correlate with the logged workflow/branch ID

Example fix

// validation inside readHistoryBranch (history_manager.go:947)
// before (corrupt data):
if firstEvent.GetVersion() != lastEvent.GetVersion() || firstEvent.GetEventId()+int64(eventCount-1) != lastEvent.GetEventId() {
    return ..., softassert.UnexpectedDataLoss(m.logger, dataLossMsg, errWrongVersion, ...)
}
// after (caller-side handling):
_, _, _, _, _, err := mgr.ReadHistoryBranch(req)
if errors.Is(err, persistence.ErrWrongVersionMessage) || strings.Contains(err.Error(), "wrong version and IDs") {
    logger.Error("history batch corrupt; needs DB repair", tag.WorkflowID(wfID))
}

// via exported wrapper
batches, err := persistence.ValidateBatch(ctx, events)
if err != nil && strings.Contains(err.Error(), "wrong version and IDs") {
    // batch corrupt: same version + contiguous IDs required
}

func isWrongVersionErr(err error) bool {
    return err != nil && strings.Contains(err.Error(), "corrupted history event batch, wrong version and IDs")
}

res, err := mgr.ReadHistoryBranch(ctx, req)
if err != nil {
    if isWrongVersionErr(err) {
        logger.Critical("history data loss detected", tag.Error(err))
        return err // do not retry: deterministic corruption
    }
    return err
}

When it happens

Trigger: Calling ReadHistoryBranch / ReadHistoryBranchReverse / ValidateBatch when a fetched event blob deserializes into events where firstEvent.Version != lastEvent.Version, or where firstEvent.EventId+len(events)-1 != lastEvent.EventId (non-uniform version or non-contiguous IDs within one batch).

Common situations: Manual DB edits or partial writes to the history_node table; version upgrades/failover interrupted mid-batch; corrupted or truncated serialized event blobs after storage-layer failures; bugs in fork/branch history logic leaving mixed-version events in one batch.

Related errors

• corrupted history event batch, empty events
• page size to read history tasks must be positive
• history task from queue has nil blob
• enqueue task request task is nil
• queue already exists

AI-assisted analysis of temporalio/temporal@bde624efd1 (2026-09-01). Data as JSON: /api/errors/f6773cd3c2162551. Report an issue: GitHub.