missing <REV> in --filename-format when gzip, rotation, or w

Error message

missing <REV> in --filename-format when gzip, rotation, or work dir enabled

What it means

nsq_to_file builds output filenames from --filename-format by substituting <TOPIC>, <HOST>, <PID> and <REV>. When gzip (-gzip), size/interval rotation (-rotate-size/-rotate-interval) or a separate work dir (-work-dir != -output-dir) is enabled, the <REV> placeholder is mandatory because each rotation/gzip cycle must produce a distinct filename via a monotonically increasing revision number. Without it, rotated files would overwrite each other, so generateFileName refuses to run.

Source

Thrown at apps/nsq_to_file/file_logger.go:407

}

func computeFilenameFormat(opts *Options, topic string) (string, error) {
	hostname, err := os.Hostname()
	if err != nil {
		return "", err
	}
	shortHostname := strings.Split(hostname, ".")[0]

	identifier := shortHostname
	if len(opts.HostIdentifier) != 0 {
		identifier = strings.ReplaceAll(opts.HostIdentifier, "<SHORT_HOST>", shortHostname)
		identifier = strings.ReplaceAll(identifier, "<HOSTNAME>", hostname)
	}

	cff := opts.FilenameFormat
	if opts.GZIP || opts.RotateSize > 0 || opts.RotateInterval > 0 || opts.WorkDir != opts.OutputDir {
		if !strings.Contains(cff, "<REV>") {
			return "", errors.New("missing <REV> in --filename-format when gzip, rotation, or work dir enabled")
		}
	} else {
		// remove <REV> as we don't need it
		cff = strings.ReplaceAll(cff, "<REV>", "")
	}
	cff = strings.ReplaceAll(cff, "<TOPIC>", topic)
	cff = strings.ReplaceAll(cff, "<HOST>", identifier)
	cff = strings.ReplaceAll(cff, "<PID>", fmt.Sprintf("%d", os.Getpid()))
	if opts.GZIP && !strings.HasSuffix(cff, ".gz") {
		cff = cff + ".gz"
	}

	return cff, nil
}

View on GitHub (pinned to 85cf10c09c)

Solutions

Add <REV> to --filename-format, e.g. --filename-format=<TOPIC>.<HOST>.<PID>.<REV>.log
Keep <REV> adjacent to the extension so rotated files sort naturally (<REV> expands to .000000001-style zero-padded numbers).
If you truly never rotate, disable --gzip, set --rotate-size=0 and --rotate-interval=0, and leave --work-dir equal to --output-dir so <REV> becomes optional.

Example fix

# before
nsq_to_file --topic=test --channel=ch --filename-format=<TOPIC>.<HOST>.log --gzip --output-dir=/data
# after
nsq_to_file --topic=test --channel=ch --filename-format=<TOPIC>.<HOST>.<REV>.log.gz --gzip --output-dir=/data

Defensive patterns

Strategy: validation

Validate before calling

func validateFilenameFormat(format string, gzip bool, rotateSize, rotateInterval int64, workDir, outputDir string) error {
	needsRev := gzip || rotateSize > 0 || rotateInterval > 0 || workDir != outputDir
	if needsRev && !strings.Contains(format, "<REV>") {
		return fmt.Errorf("filename-format %q needs <REV> when gzip/rotation/work-dir enabled", format)
	}
	return nil
}

// run before exec'ing nsq_to_file
if err := validateFilenameFormat(*filenameFormat, *gzip, *rotateSize, *rotateInterval, *workDir, *outputDir); err != nil {
	log.Fatal(err)
}

Prevention

Standardize on --filename-format=<TOPIC>.<HOST>.<PID>.<REV>.log in shared templates so <REV> is never omitted.
When enabling gzip or rotation later on an existing deployment, diff the full flag set against the filename format.

When it happens

Trigger: Starting nsq_to_file with a custom --filename-format that omits <REV> (e.g. '<TOPIC>.log') while any of --gzip, --rotate-size > 0, --rotate-interval > 0, or --work-dir different from --output-dir is set. Without rotation features <REV> is simply stripped from the format, which is why the check only fires in this branch.

Common situations: Customizing filename-format for log-shipping pipelines (e.g. to feed fluentd/logrotate) and then later enabling gzip or rotation; setting -work-dir for atomic rename-based writes while keeping a pretty filename format.

Related errors

AI-assisted analysis of nsqio/nsq@85cf10c09c (2026-08-16). Data as JSON: /api/errors/0dd649f0ad3a6848. Report an issue: GitHub.