compression must be resolved into a concrete type before…

Error message

compression must be resolved into a concrete type before obtaining a reader

What it means

fileio's newDecompressionReader converts a raw file reader into a decompressing one based on a compressionType. compressionAuto is only a hint to be resolved during match/manifest processing; if it is still unresolved when a reader is requested, the library refuses with this error because it cannot know which decompressor to apply.

Solutions

Use fileio.MatchAll (or MatchAllFilepattern) with compression.Auto and then read via the returned match results so the library resolves Auto to a concrete type (Gzip/Uncompressed) per file.
If specifying compression manually, use a concrete type: compression.Gzip or compression.Uncompressed instead of compression.Auto.
If you manage readers yourself, resolve the compression from the file metadata (e.g. .gz extension / magic bytes) before requesting the reader.

Example fix

// before
files, _ := fileio.MatchAll(s, "gs://bucket/*.gz", fileio.MatchConfiguration{Compression: compression.Auto})
// reader built directly with compression still Auto -> error

// after
files := fileio.MatchAll(s, "gs://bucket/*.gz", fileio.MatchConfiguration{Compression: compression.Auto})
fileio.Read(s, files, func(ctx context.Context, src *fileio.ReadableFile) error {
	// src.Compression is resolved (Gzip/None) before the reader is opened
})

Defensive patterns

Strategy: validation

Validate before calling

if compression == compression.Auto {
	// resolve first: use fileio.MatchAll results, whose metadata carries the
	// concrete per-file compression, or specify Gzip/Uncompressed explicitly
}

Prevention

Never request readers while compression is still compression.Auto — let fileio.MatchAll resolve it.
Pass a concrete compression type (Gzip, Uncompressed) when you already know the file format.
Use the fileio.Read transform rather than hand-building decompression readers.

When it happens

Trigger: Using fileio.Match* with a fileio.MatchConfiguration{Compression: compression.Auto} but obtaining readers directly from the match results (or calling newDecompressionReader/Open paths) without first resolving the auto-detected compression per file, e.g. reading results.Metadata[i].Compression instead of letting the match result populate it.

Common situations: Manually constructing fileio.Read(s, col, compression.Auto) / building readers from fileio.MatchAllAll output while bypassing the compression resolution step; copying match configuration code that sets Auto but never calls the resolution that replaces it with Gzip or None.

Understand the failure class

Background: "Invalid state transition" errors: "status must be X, actually Y", "already rejected/charging/uninstalled", "cannot ... while running" — what they mean when a library rejects your call — this error's family across 31 libraries.

Related errors

AI-assisted analysis of apache/beam@12126d8942 (2026-09-13). Data as JSON: /api/errors/5bf0363c7c492736. Report an issue: GitHub.

Appendix: source

Thrown at sdks/go/pkg/beam/io/fileio/file.go:106

func compressionFromExt(path string) compressionType {
	switch filepath.Ext(path) {
	case ".gz":
		return compressionGzip
	default:
		return compressionUncompressed
	}
}

// newDecompressionReader returns an io.ReadCloser that can be used to read uncompressed data from
// reader, based on the specified compression. If the compression is compressionAuto, a non-nil
// error is returned. It is the caller's responsibility to close the returned reader.
func newDecompressionReader(
	reader io.ReadCloser,
	compression compressionType,
) (io.ReadCloser, error) {
	switch compression {
	case compressionAuto:
		return nil, errors.New(
			"compression must be resolved into a concrete type before obtaining a reader",
		)
	case compressionGzip:
		return newGzipReader(reader)
	default:
		return reader, nil
	}
}

// Read reads the entire file into memory and returns the contents.
func (f ReadableFile) Read(ctx context.Context) (data []byte, err error) {
	rc, err := f.Open(ctx)
	if err != nil {
		return nil, err
	}

	defer func() {
		closeErr := rc.Close()

View on GitHub (pinned to 12126d8942)