VictoriaMetrics/VictoriaMetrics · critical

cannot open next chunk file: %w

Error message

cannot open next chunk file: %w

What it means

Raised when readBlock cannot advance to the next chunk file for reading. The wrapped error comes from nextChunkFileForRead, which either fails checkReaderWriterOffsets (reader offset beyond writer offset — corrupted metainfo from an unclean shutdown) or fails to flush metainfo/open the chunk. This aborts the read and, via MustReadBlockNonblocking, panics with FATAL since reads cannot proceed.

Source

Thrown at lib/persistentqueue/persistentqueue.go:474

	var err error
	dst, err = q.readBlock(dst)
	if err != nil {
		if err == errEmptyQueue {
			return dst, false
		}
		logger.Panicf("FATAL: %s", err)
	}
	return dst, true
}

func (q *queue) readBlock(dst []byte) ([]byte, error) {
	startTime := time.Now()
	defer func() {
		readDurationSeconds.Add(time.Since(startTime).Seconds())
	}()
	if q.readerLocalOffset+q.maxBlockSize+8 > q.chunkFileSize {
		if err := q.nextChunkFileForRead(); err != nil {
			return dst, fmt.Errorf("cannot open next chunk file: %w", err)
		}
	}

again:
	// Read block len.
	header := headerBufPool.Get()
	header.B = bytesutil.ResizeNoCopyMayOverallocate(header.B, 8)
	err := q.readFull(header.B)
	blockLen := encoding.UnmarshalUint64(header.B)
	headerBufPool.Put(header)
	if err != nil {
		logger.Errorf("skipping corrupted %q, since header with size 8 bytes cannot be read from it: %s", q.readerPath, err)
		if err := q.skipBrokenChunkFile(); err != nil {
			return dst, err
		}
		goto again
	}
	// see https://github.com/VictoriaMetrics/VictoriaMetrics/pull/6241

View on GitHub (pinned to 5079fb58f1)

Solutions

1. Compare readerOffset/writerOffset in the error: if reader > writer, the queue's metainfo is inconsistent — back up the queue directory, then either restore consistent metainfo or delete the queue directory so it is recreated (data loss).
2. Fix filesystem problems (full disk, permissions) if the wrapped error is 'cannot flush metainfo'.
3. Ensure clean shutdowns: use SIGTERM/SIGINT so MustClose flushes metainfo, and avoid kill -9 on the storage path.
4. Restore from a consistent snapshot of the data path rather than mixing chunk files and metainfo from different points in time.

Example fix

// before: inconsistent dir after crash -> FATAL panic on read
// readerOffset=1048576 cannot exceed writerOffset=999424
// after: reset the corrupted queue dir (accept data loss) and restart
# mv /data/<queue-name> /data/<queue-name>.corrupt
# systemctl restart vmagent

// on startup, sanity-check that metainfo offsets are consistent with chunk files
// (readerOffset <= writerOffset and chunk files exist for both offsets)
func checkQueueConsistency(dir string, chunkFileSize int64) error {
    ro, wo, err := readMetainfoOffsets(dir)
    if err != nil { return err }
    if ro > wo { return fmt.Errorf("corrupt metainfo: reader %d > writer %d", ro, wo) }
    return nil
}

// reads panic with FATAL on this error; intercept and reset the queue dir
defer func() {
    if r := recover(); r != nil {
        if strings.Contains(fmt.Sprint(r), "cannot open next chunk file") {
            logger.Errorf("queue read rollover failed: %v; reset queue dir", r)
            os.RemoveAll(queueDir) // accept data loss, queue recreated on restart
        }
    }
}()

When it happens

Trigger: readBlock detects readerLocalOffset+maxBlockSize+8 > chunkFileSize and calls nextChunkFileForRead, which fails: either readerOffset (rounded up to the next chunk) exceeds writerOffset (corrupted/stale metainfo), or flushMetainfo cannot write to q.dir, or the next chunk file cannot be opened via filestream.MustOpen.

Common situations: Unclean shutdown (power loss, kill -9) leaving metainfo out of sync with chunk files; queue data directory partially deleted or restored from an inconsistent backup; read-only or full disk preventing metainfo flush during rollover.

Related errors

• readerOffset=%d cannot exceed writerOffset=%d; it is likely
• unexpected number of bytes read; got %d bytes; want %d bytes
• cannot write header with size 8 bytes to %q: %w
• cannot write block contents with size %d bytes to %q: %w
• cannot flush metainfo: %w

AI-assisted analysis of VictoriaMetrics/VictoriaMetrics@5079fb58f1 (2026-09-03). Data as JSON: /api/errors/93cfbe1427a81ce1. Report an issue: GitHub.