What causes "path to directory not allowed: %q" in apache/beam?

Filter out directories before reading (skip entries ending in '/' or check the metadata).

path to directory not allowed

Error message

path to directory not allowed: %q

What it means

readFn rejects directory entries when DirectoryTreatment is set to directoryDisallow. MatchAll can expand to directories (e.g. an 'empty glob' match or a directory whose path matched); this DoFn refuses to read them and returns this error instead of silently skipping.

Solutions

Filter out directories before reading (skip entries ending in '/' or check the metadata).
Set the DirectoryTreatment option to allow/passthrough directories if that's intended.
Tighten the match pattern so it doesn't expand to directories.
Add an allowlist of file extensions to the match pattern.

Example fix

// before
p := beam.ParDo(s, &fileio.ReadFn{DirectoryTreatment: fileio.Disallow}, matched)
// after
p := beam.ParDo(s, &fileio.ReadFn{DirectoryTreatment: fileio.Passthrough}, matched)
// or filter directories before reading

Defensive patterns

Strategy: validation

Validate before calling

if strings.HasSuffix(metadata.Path, "/") || isDir(metadata.Path) {
	return nil // skip directories before readFn sees them
}

Type guard

func isFile(md FileMetadata) bool { return !strings.HasSuffix(md.Path, "/") }

Try / catch

if err := beam.Run(...); err != nil && strings.Contains(err.Error(), "path to directory not allowed") {
	log.Warn("match pattern expanded to a directory; filter or set DirectoryTreatment")
}

Prevention

End glob patterns with an extension filter (e.g. /*.json) to avoid directory matches.
Choose DirectoryTreatment explicitly instead of relying on defaults.
Filter directory entries in the match DoFn before readFn.
Test match patterns against real bucket layouts.

When it happens

Trigger: A FileMetadata whose Path is a directory flows into readFn.ProcessElement while the transform was configured with ReadDirectoriesDisallowed (the default disallow behavior).

Common situations: Glob patterns like 'dir/*' matched a subdirectory; the user assumed only files would be surfaced; forgot to set ReadDirectoriesPassthrough or filter out directories upstream.

Related errors

AI-assisted analysis of apache/beam@12126d8942 (2026-09-13). Data as JSON: /api/errors/b66440e78f657014. Report an issue: GitHub.

Appendix: source

Thrown at sdks/go/pkg/beam/io/fileio/read.go:121

	return beam.ParDo(s, newReadFn(option), col)
}

type readFn struct {
	Compression        compressionType
	DirectoryTreatment directoryTreatment
}

func newReadFn(option *readOption) *readFn {
	return &readFn{
		Compression:        option.Compression,
		DirectoryTreatment: option.DirectoryTreatment,
	}
}

func (fn *readFn) ProcessElement(metadata FileMetadata, emit func(ReadableFile)) error {
	if isDirectory(metadata.Path) {
		if fn.DirectoryTreatment == directoryDisallow {
			return fmt.Errorf("path to directory not allowed: %q", metadata.Path)
		}
		return nil
	}

	file := ReadableFile{
		Metadata:    metadata,
		Compression: fn.Compression,
	}

	emit(file)
	return nil
}

func isDirectory(path string) bool {
	if strings.HasSuffix(path, "/") || strings.HasSuffix(path, "\\") {
		return true
	}
	return false

View on GitHub (pinned to 12126d8942)