What causes "failed to scan %v for artifacts to stage" in apache/beam?

Verify the directory path exists and is a directory: os.Stat + Mode().IsDir() before calling StageDir.

failed to scan for artifacts to stage

Error message

failed to scan %v for artifacts to stage

What it means

scan() walks the staging source directory recursively collecting KeyedFile entries; it wraps any error from walk() (which delegates to os.ReadDir) as 'failed to scan %v for artifacts to stage'. This means the source directory could not be read at all — StageDir aborts before staging anything. Note that StageDir silently returns no artifacts (and nil error) if the directory is readable but empty.

Solutions

Verify the directory path exists and is a directory: os.Stat + Mode().IsDir() before calling StageDir.
Check read/execute permissions on the directory (and traverse permissions on parents) for the user running the pipeline.
Correct the source path passed to StageDir — it must be the directory containing artifacts, not a file.
Ensure the directory isn't removed by earlier pipeline steps or cleanup jobs before staging runs.

Example fix

// before
mds, err := artifact.StageDir(ctx, client, srcDir, token)
// after (validate first)
if info, statErr := os.Stat(srcDir); statErr != nil || !info.IsDir() {
	return fmt.Errorf("staging source %q is not a readable directory", srcDir)
}
mds, err := artifact.StageDir(ctx, client, srcDir, token)

Defensive patterns

Strategy: validation

Validate before calling

func validateStagingDir(dir string) error {
	info, err := os.Stat(dir)
	if err != nil {
		return fmt.Errorf("staging dir %q: %w", dir, err)
	}
	if !info.IsDir() {
		return fmt.Errorf("%q is not a directory", dir)
	}
	f, err := os.Open(dir)
	if err != nil {
		return fmt.Errorf("no read permission on %q: %w", dir, err)
	}
	f.Close()
	return nil
}

Try / catch

list, err := artifact.StageDir(ctx, client, srcDir, token)
if err != nil {
	if strings.Contains(err.Error(), "failed to scan") {
		return fmt.Errorf("check that %s exists, is a directory, and is readable: %w", srcDir, err)
	}
	return err
}
if len(list) == 0 {
	return fmt.Errorf("staging dir %s contained no artifacts", srcDir)
}

Prevention

os.Stat and confirm IsDir before calling StageDir.
Check directory read/execute permissions for the pipeline user.
Beware: an empty but readable directory yields no error and no artifacts — assert non-empty results.
Confirm no cleanup job deletes the directory between pipeline steps and staging.

When it happens

Trigger: os.ReadDir(dir) inside walk fails because the directory does not exist, is not a directory, or lacks read permission; os.ReadDir on a subdirectory errors mid-walk (note: subdirectory walk errors are ignored, only the top-level error propagates).

Common situations: Passing a wrong --artifact staging dir / working-dir path to the pipeline runner; the directory was deleted or renamed before staging; running the pipeline as a user without read permission on the directory; passing a file path instead of a directory.

Related errors

AI-assisted analysis of apache/beam@12126d8942 (2026-09-13). Data as JSON: /api/errors/c99c976c0fef0ea5. Report an issue: GitHub.

Appendix: source

Thrown at sdks/go/pkg/beam/artifact/stage.go:217

		if err == io.EOF {
			break
		}
		if err != nil {
			return "", err
		}
	}
	return hex.EncodeToString(sha256W.Sum(nil)), nil
}

// KeyedFile is a key and filename pair.
type KeyedFile struct {
	Key, Filename string
}

func scan(dir string) ([]KeyedFile, error) {
	var ret []KeyedFile
	if err := walk(dir, "", &ret); err != nil {
		return nil, errors.Wrapf(err, "failed to scan %v for artifacts to stage", dir)
	}
	return ret, nil
}

func walk(dir, key string, accum *[]KeyedFile) error {
	list, err := os.ReadDir(dir)
	if err != nil {
		return err
	}

	for _, elm := range list {
		k := makeKey(key, elm.Name())
		f := filepath.Join(dir, elm.Name())

		if elm.IsDir() {
			walk(f, k, accum)
			continue
		}

View on GitHub (pinned to 12126d8942)