{"record":{"id":"3df42e7ca62ffa65","repo":"VictoriaMetrics/VictoriaMetrics","slug":"invalid-mintimestamp-at-block-header-at-offset-d","errorCode":null,"errorMessage":"invalid MinTimestamp at block header at offset %d; got %d; cannot be smaller than %d","messagePattern":"invalid MinTimestamp at block header at offset (.+?); got (.+?); cannot be smaller than (.+?)","errorType":"exception","errorClass":null,"httpStatus":null,"severity":"error","filePath":"lib/storage/block_stream_reader.go","lineNumber":254,"sourceCode":"\t\treturn fmt.Errorf(\"cannot parse block header read from index data at offset %d: %w\", bsr.prevIndexBlockOffset(), err)\n\t}\n\tif len(tail) > 0 {\n\t\treturn fmt.Errorf(\"non-empty tail left after parsing block header at offset %d: %x\", bsr.prevIndexBlockOffset(), tail)\n\t}\n\tbsr.indexCursor = bsr.indexCursor[marshaledBlockHeaderSize:]\n\n\tbsr.blocksCount++\n\tif bsr.blocksCount > bsr.ph.BlocksCount {\n\t\treturn fmt.Errorf(\"too many blocks found in the block stream; got %d; cannot be bigger than %d\", bsr.blocksCount, bsr.ph.BlocksCount)\n\t}\n\n\t// Validate block header.\n\tbsr.rowsCount += uint64(bsr.Block.bh.RowsCount)\n\tif bsr.rowsCount > bsr.ph.RowsCount {\n\t\treturn fmt.Errorf(\"too many rows found in the block stream; got %d; cannot be bigger than %d\", bsr.rowsCount, bsr.ph.RowsCount)\n\t}\n\tif bsr.Block.bh.MinTimestamp < bsr.ph.MinTimestamp {\n\t\treturn fmt.Errorf(\"invalid MinTimestamp at block header at offset %d; got %d; cannot be smaller than %d\",\n\t\t\tbsr.prevIndexBlockOffset(), bsr.Block.bh.MinTimestamp, bsr.ph.MinTimestamp)\n\t}\n\tif bsr.Block.bh.MaxTimestamp > bsr.ph.MaxTimestamp {\n\t\treturn fmt.Errorf(\"invalid MaxTimestamp at block header at offset %d; got %d; cannot be bigger than %d\",\n\t\t\tbsr.prevIndexBlockOffset(), bsr.Block.bh.MaxTimestamp, bsr.ph.MaxTimestamp)\n\t}\n\tusePrevTimestamps := len(bsr.prevTimestampsData) > 0 && bsr.Block.bh.TimestampsBlockOffset == bsr.prevTimestampsBlockOffset\n\tif usePrevTimestamps {\n\t\tif int(bsr.Block.bh.TimestampsBlockSize) != len(bsr.prevTimestampsData) {\n\t\t\treturn fmt.Errorf(\"invalid TimestampsBlockSize at block header at offset %d; got %d; want %d\",\n\t\t\t\tbsr.prevIndexBlockOffset(), bsr.Block.bh.TimestampsBlockSize, len(bsr.prevTimestampsData))\n\t\t}\n\t} else if bsr.Block.bh.TimestampsBlockOffset != bsr.timestampsBlockOffset {\n\t\treturn fmt.Errorf(\"invalid TimestampsBlockOffset at block header at offset %d; got %d; want %d\",\n\t\t\tbsr.prevIndexBlockOffset(), bsr.Block.bh.TimestampsBlockOffset, bsr.timestampsBlockOffset)\n\t}\n\tif bsr.Block.bh.ValuesBlockOffset != bsr.valuesBlockOffset {\n\t\treturn fmt.Errorf(\"invalid ValuesBlockOffset at block header at offset %d; got %d; want %d\",","sourceCodeStart":236,"sourceCodeEnd":272,"githubUrl":"https://github.com/VictoriaMetrics/VictoriaMetrics/blob/5079fb58f1e8e62113f90c945ad71586c797d770/lib/storage/block_stream_reader.go#L236-L272","documentation":"Each part header declares the minimum timestamp (ph.MinTimestamp) across all rows in the part. Every block header must have MinTimestamp >= the part's minimum. A smaller value means the index content contradicts the part metadata — corruption or mismatched files. This guard keeps query-time timestamp filtering correct.","triggerScenarios":"Calling NextBlock when bsr.Block.bh.MinTimestamp < bsr.ph.MinTimestamp — typically an index file paired with the wrong part header, corrupted timestamp fields, or files mixed from different merges/versions.","commonSituations":"Manual part copies during migrations; interrupted compaction leaving stale metadata; disk corruption; externally rewritten VM files.","solutions":["Restore the entire part directory from a backup or healthy replica so header and index agree.","Check for leftover tmp/partial parts from interrupted merges and delete them.","Remove the inconsistent part and re-ingest the affected time range if no backup exists.","If many parts are affected, investigate the storage device for silent corruption."],"exampleFix":"// before: part header and index from different merges\n// invalid MinTimestamp at block header at offset 8192; got 100; cannot be smaller than 200\n// after: restore a self-consistent part\nrsync -a --checksum backup:/vmdata/parts/2026_01/ /vmdata/parts/2026_01/","handlingStrategy":"validation","validationCode":"// Ensure the part directory is complete and self-consistent before streaming:\nfunc partSelfConsistent(partPath string) error {\n\tfor _, f := range []string{\"metadata.json\", \"index.bin\", \"metaindex.bin\", \"timestamps.bin\", \"values.bin\"} {\n\t\tif _, err := os.Stat(filepath.Join(partPath, f)); err != nil {\n\t\t\treturn fmt.Errorf(\"missing %s in %s: %w\", f, partPath, err)\n\t\t}\n\t}\n\treturn nil\n}","typeGuard":null,"tryCatchPattern":"if err := bsr.Error(); err != nil {\n\tif strings.Contains(err.Error(), \"invalid MinTimestamp\") {\n\t\t// index contradicts part header: restore whole part or re-ingest range\n\t}\n\treturn err\n}","preventionTips":["Keep part header, index, and data files from the same merge together at all times.","Delete tmp/partial parts left behind by interrupted compactions.","Restore parts only from consistent, checksum-verified backups.","Check storage health if timestamp inconsistencies appear across many parts."],"tags":["go","victoriametrics","storage","corruption"],"backgroundTag":"index-corruption","analyzedSha":"5079fb58f1e8e62113f90c945ad71586c797d770","analyzedAt":"2026-09-03T18:10:26.153Z","contentChangedAt":"2026-09-03T18:10:26.153Z","schemaVersion":2},"datasetVersion":"2026-09-08T15:18:49.778Z"}