{"record":{"id":"edbd86bfee00a652","repo":"microsoft/FASTER","slug":"metadata-of-size-commitmetadata-length-does-not-fit-in-delta","errorCode":null,"errorMessage":"Metadata of size {commitMetadata.Length} does not fit in delta log space of size {length}","messagePattern":"Metadata of size (.+?) does not fit in delta log space of size (.+?)","errorType":"exception","errorClass":"Exception","httpStatus":null,"severity":"error","filePath":"cs/src/core/Index/CheckpointManagement/DeviceLogCommitCheckpointManager.cs","lineNumber":266,"sourceCode":"                logTokenHistory[logTokenHistoryOffset] = logToken;\n                logTokenHistoryOffset = (byte)((logTokenHistoryOffset + 1) % logTokenCount);\n                if (prior != default)\n                    deviceFactory.Delete(checkpointNamingScheme.LogCheckpointBase(prior));\n            }\n        }\n\n        /// <inheritdoc />\n        public virtual unsafe void CommitLogIncrementalCheckpoint(Guid logToken, long version, byte[] commitMetadata, DeltaLog deltaLog)\n        {\n            deltaLog.Allocate(out int length, out long physicalAddress);\n            if (length < commitMetadata.Length)\n            {\n                deltaLog.Seal(0, DeltaLogEntryType.CHECKPOINT_METADATA);\n                deltaLog.Allocate(out length, out physicalAddress);\n                if (length < commitMetadata.Length)\n                {\n                    deltaLog.Seal(0);\n                    throw new Exception($\"Metadata of size {commitMetadata.Length} does not fit in delta log space of size {length}\");\n                }\n            }\n            fixed (byte* ptr = commitMetadata)\n            {\n                Buffer.MemoryCopy(ptr, (void*)physicalAddress, commitMetadata.Length, commitMetadata.Length);\n            }\n            deltaLog.Seal(commitMetadata.Length, DeltaLogEntryType.CHECKPOINT_METADATA);\n            deltaLog.FlushAsync().Wait();\n        }\n\n        /// <inheritdoc />\n        public IEnumerable<Guid> GetLogCheckpointTokens()\n        {\n            return deviceFactory.ListContents(checkpointNamingScheme.LogCheckpointBasePath()).Select(e => checkpointNamingScheme.Token(e));\n        }\n\n        /// <inheritdoc />\n        public virtual byte[] GetLogCheckpointMetadata(Guid logToken, DeltaLog deltaLog, bool scanDelta, long recoverTo)","sourceCodeStart":248,"sourceCodeEnd":284,"githubUrl":"https://github.com/microsoft/FASTER/blob/321d872eabda6a0345c8bd76419f89723ed864ae/cs/src/core/Index/CheckpointManagement/DeviceLogCommitCheckpointManager.cs#L248-L284","documentation":"During CommitLogIncrementalCheckpoint, the checkpoint metadata is serialized and an attempt is made to write it into a single delta-log record. If the serialized metadata length exceeds the space returned by deltaLog.Allocate, the manager seals the entry and throws. This means the incremental checkpoint metadata cannot fit in the allocated delta log entry, so the incremental checkpoint cannot proceed.","triggerScenarios":"Calling CommitLogIncrementalCheckpoint when commitMetadata.Length > the length returned by deltaLog.Allocate(out length, out physicalAddress); typically metadata grew beyond the expected delta record size.","commonSituations":"Very large numbers of partitions/segments or huge cookie blobs inflating metadata; misconfigured delta log entry size; running incremental checkpoints with metadata that only fits a full checkpoint.","solutions":["Take a full (non-incremental) checkpoint instead, which writes metadata through a path not bounded by a single delta entry.","Increase the delta log entry/segment size so metadata of the current size fits.","Reduce metadata size (e.g. fewer iterators, smaller cookie, compact recovery info) before checkpointing.","Update/upgrade the library if a fixed larger default delta space is available."],"exampleFix":"// before\nmanager.CommitLogIncrementalCheckpoint(token); // may throw if metadata too large\n\n// after\n// fall back to a full checkpoint when incremental fails\ntry {\n    manager.CommitLogIncrementalCheckpoint(token);\n} catch (Exception) {\n    manager.CommitLogCheckpoint(token); // full checkpoint path\n}","handlingStrategy":"fallback","validationCode":null,"typeGuard":null,"tryCatchPattern":"try { manager.CommitLogIncrementalCheckpoint(token); } catch (Exception ex) when (ex.Message.Contains(\"does not fit in delta log space\")) { manager.CommitLogCheckpoint(token); }","preventionTips":["Keep metadata small: limit iterator count and cookie size.","Provision delta log space for worst-case metadata size.","Prefer full checkpoints when metadata growth is expected."],"tags":["checkpoint","delta-log","capacity","fasterlog"],"backgroundTag":"payload-too-large","analyzedSha":"321d872eabda6a0345c8bd76419f89723ed864ae","analyzedAt":"2026-09-15T22:18:00.693Z","contentChangedAt":"2026-09-15T22:18:00.693Z","schemaVersion":2},"datasetVersion":"2026-09-16T04:17:20.429Z"}