weaviate/weaviate · error

transform value: %w

Error message

transform value: %w

What it means

writeIndividualNode optionally rewrites live (non-tombstone, non-empty) values through c.valueTransformer — the active edit transformer used e.g. when compression encoding changes across segment versions. If the transformer returns an error for the given value bytes, compaction aborts with this wrapped error. It means the on-disk value could not be converted to the new encoding/representation.

Source

Thrown at adapters/repos/db/lsmkv/compactor_replace.go:273

			// a secondary key is not guaranteed to be present for every index
			// position (e.g. segment written with a different secondary index
			// count). Skip positions beyond what the compactor expects.
			break
		}
		c.secIndexSizes[pos] += int64(segmentindex.TREE_KEY_STORE_OVERHEAD + len(sk))
	}
}

func (c *compactorReplace) writeIndividualNode(f *segmentindex.SegmentFile,
	offset int, key, value []byte, secondaryKeys [][]byte, tombstone bool,
) (segmentindex.Key, error) {
	// Rewrite live values through the active edit transformer before they hit
	// the merged segment. Tombstones and empty payloads carry nothing to
	// transform, so they are skipped.
	if c.valueTransformer != nil && !tombstone && len(value) > 0 {
		transformed, err := c.valueTransformer(value)
		if err != nil {
			return segmentindex.Key{}, fmt.Errorf("transform value: %w", err)
		}
		value = transformed
	}

	// Copy key bytes into stable arena memory. The reusable cursor reuses its
	// internal buffers on every next() call, so ki.Key / ki.SecondaryKeys stored
	// in the kis slice would otherwise be corrupted on the next iteration.
	keyCopy := c.arena.CopyKey(key)

	var secKeysCopy [][]byte
	if len(secondaryKeys) > 0 {
		secKeysCopy = make([][]byte, len(secondaryKeys))
		for i, sk := range secondaryKeys {
			secKeysCopy[i] = c.arena.CopyKey(sk)
		}
	}

	// value is NOT arena-copied because it is written synchronously to the

View on GitHub (pinned to 75aa4b6d11)

Solutions

1. Read the wrapped cause to identify whether it is a decompression failure (corrupt data) or an encode failure (incompatible value).
2. If compression settings were changed, verify the transformer supports values from all existing segment versions (upgrade path).
3. Restore the affected bucket from backup if the source segment is corrupt, then let compaction rebuild it.
4. Report the failing value pattern upstream if it reproduces on valid data — the transformer has a version-compatibility bug.

Example fix

// before: transformer assumes new encoding and fails on old values
transformed, err := c.valueTransformer(value)
// after (caller-side mitigation): re-vectorize/rewrite the collection
// or revert the compression setting change that forced the transform
//   client.schema.updateCollection({ compression: previousSetting })

// before enabling a new compression on a collection, round-trip test:
// transformed, err := transformer(existingValue); require.NoError(t, err)

func isTransformErr(err error) bool {
	return strings.Contains(err.Error(), "transform value:")
}

if err := bucket.Compact(ctx); err != nil {
	if isTransformErr(err) {
		log.Printf("value transform failed: %v — restore bucket from backup or revert compression setting", err)
	}
}

When it happens

Trigger: A replace-bucket compaction runs with a valueTransformer configured and the transformer function errors on a specific live value — e.g. decompression/compression failure (BQ/PQ/SQ compression change), value produced by an older incompatible segment version, or bit-corrupted value bytes.

Common situations: Upgrading Weaviate with a changed default compression so old segments must be rewritten; manually downgrading/switching compression options on a bucket; corrupted segment data after a crash.

Related errors

• could not copy all data into buffer
• pause compaction for rangeable in-memory rebuild: %w
• pause compaction: %w
• flush buffered: %w
• write compactorMap segment checksum: %w

AI-assisted analysis of weaviate/weaviate@75aa4b6d11 (2026-09-04). Data as JSON: /api/errors/6448d2a9449fcfec. Report an issue: GitHub.