{"record":{"id":"93cfbe1427a81ce1","repo":"VictoriaMetrics/VictoriaMetrics","slug":"cannot-open-next-chunk-file-w","errorCode":null,"errorMessage":"cannot open next chunk file: %w","messagePattern":"cannot open next chunk file: %w","errorType":"exception","errorClass":null,"httpStatus":null,"severity":"critical","filePath":"lib/persistentqueue/persistentqueue.go","lineNumber":474,"sourceCode":"\tvar err error\n\tdst, err = q.readBlock(dst)\n\tif err != nil {\n\t\tif err == errEmptyQueue {\n\t\t\treturn dst, false\n\t\t}\n\t\tlogger.Panicf(\"FATAL: %s\", err)\n\t}\n\treturn dst, true\n}\n\nfunc (q *queue) readBlock(dst []byte) ([]byte, error) {\n\tstartTime := time.Now()\n\tdefer func() {\n\t\treadDurationSeconds.Add(time.Since(startTime).Seconds())\n\t}()\n\tif q.readerLocalOffset+q.maxBlockSize+8 > q.chunkFileSize {\n\t\tif err := q.nextChunkFileForRead(); err != nil {\n\t\t\treturn dst, fmt.Errorf(\"cannot open next chunk file: %w\", err)\n\t\t}\n\t}\n\nagain:\n\t// Read block len.\n\theader := headerBufPool.Get()\n\theader.B = bytesutil.ResizeNoCopyMayOverallocate(header.B, 8)\n\terr := q.readFull(header.B)\n\tblockLen := encoding.UnmarshalUint64(header.B)\n\theaderBufPool.Put(header)\n\tif err != nil {\n\t\tlogger.Errorf(\"skipping corrupted %q, since header with size 8 bytes cannot be read from it: %s\", q.readerPath, err)\n\t\tif err := q.skipBrokenChunkFile(); err != nil {\n\t\t\treturn dst, err\n\t\t}\n\t\tgoto again\n\t}\n\t// see https://github.com/VictoriaMetrics/VictoriaMetrics/pull/6241","sourceCodeStart":456,"sourceCodeEnd":492,"githubUrl":"https://github.com/VictoriaMetrics/VictoriaMetrics/blob/5079fb58f1e8e62113f90c945ad71586c797d770/lib/persistentqueue/persistentqueue.go#L456-L492","documentation":"Raised when readBlock cannot advance to the next chunk file for reading. The wrapped error comes from nextChunkFileForRead, which either fails checkReaderWriterOffsets (reader offset beyond writer offset — corrupted metainfo from an unclean shutdown) or fails to flush metainfo/open the chunk. This aborts the read and, via MustReadBlockNonblocking, panics with FATAL since reads cannot proceed.","triggerScenarios":"readBlock detects readerLocalOffset+maxBlockSize+8 > chunkFileSize and calls nextChunkFileForRead, which fails: either readerOffset (rounded up to the next chunk) exceeds writerOffset (corrupted/stale metainfo), or flushMetainfo cannot write to q.dir, or the next chunk file cannot be opened via filestream.MustOpen.","commonSituations":"Unclean shutdown (power loss, kill -9) leaving metainfo out of sync with chunk files; queue data directory partially deleted or restored from an inconsistent backup; read-only or full disk preventing metainfo flush during rollover.","solutions":["Compare readerOffset/writerOffset in the error: if reader > writer, the queue's metainfo is inconsistent — back up the queue directory, then either restore consistent metainfo or delete the queue directory so it is recreated (data loss).","Fix filesystem problems (full disk, permissions) if the wrapped error is 'cannot flush metainfo'.","Ensure clean shutdowns: use SIGTERM/SIGINT so MustClose flushes metainfo, and avoid kill -9 on the storage path.","Restore from a consistent snapshot of the data path rather than mixing chunk files and metainfo from different points in time."],"exampleFix":"// before: inconsistent dir after crash -> FATAL panic on read\n// readerOffset=1048576 cannot exceed writerOffset=999424\n// after: reset the corrupted queue dir (accept data loss) and restart\n# mv /data/<queue-name> /data/<queue-name>.corrupt\n# systemctl restart vmagent","handlingStrategy":"try-catch","validationCode":"// on startup, sanity-check that metainfo offsets are consistent with chunk files\n// (readerOffset <= writerOffset and chunk files exist for both offsets)\nfunc checkQueueConsistency(dir string, chunkFileSize int64) error {\n    ro, wo, err := readMetainfoOffsets(dir)\n    if err != nil { return err }\n    if ro > wo { return fmt.Errorf(\"corrupt metainfo: reader %d > writer %d\", ro, wo) }\n    return nil\n}","typeGuard":null,"tryCatchPattern":"// reads panic with FATAL on this error; intercept and reset the queue dir\ndefer func() {\n    if r := recover(); r != nil {\n        if strings.Contains(fmt.Sprint(r), \"cannot open next chunk file\") {\n            logger.Errorf(\"queue read rollover failed: %v; reset queue dir\", r)\n            os.RemoveAll(queueDir) // accept data loss, queue recreated on restart\n        }\n    }\n}()","preventionTips":["Always stop services with SIGTERM so metainfo is flushed on close.","Never restore or copy queue directories while the process is running.","Restore full-directory snapshots, never a mix of chunk and metainfo files.","Keep queue data on crash-safe storage (fsync'd local disk)."],"tags":["go","persistent-queue","data-corruption","unclean-shutdown"],"backgroundTag":"queue-offset-corruption","analyzedSha":"5079fb58f1e8e62113f90c945ad71586c797d770","analyzedAt":"2026-09-03T18:10:26.153Z","contentChangedAt":"2026-09-03T18:10:26.153Z","schemaVersion":2},"datasetVersion":"2026-09-08T15:18:49.778Z"}