nats-io/nats-server · critical

error opening msg block file [%q]: %v

Error message

error opening msg block file [%q]: %v

What it means

Returned when opening a message block's backing file (os.OpenFile with O_CREATE|O_RDWR) fails while recovering a stream. The block cannot be brought online, so the stream recovery for it fails. The wrapped %v carries the OS-level cause (permission denied, no such file, too many open files, etc.).

Source

Thrown at server/filestore.go:7570

		}
	}

	return fs.ld, firstErr
}

// Lock should be held.
func (mb *msgBlock) enableForWriting(fip bool) error {
	if mb == nil {
		return errNoMsgBlk
	}
	if mb.mfd != nil {
		return nil
	}
	mb.fs.dios.acquire()
	mfd, err := os.OpenFile(mb.mfn, os.O_CREATE|os.O_RDWR, defaultFilePerms)
	mb.fs.dios.release()
	if err != nil {
		return fmt.Errorf("error opening msg block file [%q]: %v", mb.mfn, err)
	}
	mb.mfd = mfd

	// Spin up our flusher loop if needed.
	if !fip {
		mb.spinUpFlushLoopLocked()
	}

	return nil
}

// Helper function to place a delete tombstone.
func (mb *msgBlock) writeTombstone(seq uint64, ts int64) error {
	return mb.writeMsgRecord(emptyRecordLen, seq|tbit, _EMPTY_, nil, nil, ts, true)
}

// Helper function to place a delete tombstone without flush.
// Lock should not be held.

View on GitHub (pinned to 3a66a489d2)

Solutions

1. Read the wrapped OS error: fix permissions (chown/chmod) or remount the storage if needed
2. Raise the file descriptor limit (ulimit -n / LimitNOFILE= high value in systemd) for large streams
3. Verify the datadir path is correct and the server user owns the jetstream directory
4. If the file is missing entirely, restore it from backup or remove the block record

Example fix

# before: EPERM/EMFILE on block files
chown -R nats:nats /var/lib/nats/jetstream
# systemd unit
[Service]
LimitNOFILE=1000000

// Go: pre-flight checks before starting/recovering a server
// 1. user owns datadir, 2. fd limit high enough for block count
if err := checkDatadirWritable(storeDir, runAsUser); err != nil {
    return fmt.Errorf("datadir not writable by %s: %w", runAsUser, err)
}
blocks, _ := filepath.Glob(filepath.Join(storeDir, "streams/*/msgblk/*.blk"))
if len(blocks) > fdSoftLimit/4 {
    log.Printf("raise LimitNOFILE: %d blocks may exhaust fds", len(blocks))
}

if err != nil {
    if strings.Contains(err.Error(), "error opening msg block file") && strings.Contains(err.Error(), "permission denied") {
        return fixOwnership(storeDir) // chown -R nats:nats, then re-open
    }
    if strings.Contains(err.Error(), "too many open files") {
        return raiseFdLimitAndRetry()
    }
    return err
}

When it happens

Trigger: Recover/openStream where mb.mfn cannot be opened: wrong permissions on the .blk file, datadir on a dead mount, or fd exhaustion (EMFILE) when a stream has thousands of blocks.

Common situations: Running nats-server as a different user than the datadir owner; NFS mount hung; ulimit -n too low for large streams with many blocks; SELinux/AppArmor blocking file access.

Related errors

• loading encryption for block %d failed: %w
• rebuildState for block %d failed: %w
• populating per-subject info for block %d failed: %w
• failed to create temporary file: %w
• could not create consumer directory - %v

AI-assisted analysis of nats-io/nats-server@3a66a489d2 (2026-09-02). Data as JSON: /api/errors/6dca2a8f4284284a. Report an issue: GitHub.