apache/beam · error

failed to open file

Error message

failed to open file %s

What it means

stageFile in the Dataflow runner's staging library opens the local file to be uploaded (a worker binary or artifact) before pushing it to GCS. If os.Open fails, the upload aborts and the os error is wrapped with the filename for context. This means the pipeline could not even read the artifact it needs to stage to GCS before the job is submitted.

Solutions

1. Verify the filename exists and is a regular readable file before submitting: os.Stat(filename) and check err == nil and !info.IsDir().
2. If passing a worker binary, build it first (go build) and pass the absolute path.
3. Check file permissions (chmod/chown) and that the process user can read the path.
4. Check the wrapped inner error in the message: 'no such file or directory' means wrong path, 'permission denied' means fix ACLs.

Example fix

// before
hash, err := stageFile(ctx, project, url, "worker/beamapp")
// after
abs, _ := filepath.Abs("worker/beamapp")
if _, err := os.Stat(abs); err != nil {
    log.Fatalf("worker binary not found at %s: build it first", abs)
}
hash, err := stageFile(ctx, project, url, abs)

info, err := os.Stat(filename)
if err != nil {
    return fmt.Errorf("staging artifact %s not accessible: %w", filename, err)
}
if info.IsDir() {
    return fmt.Errorf("%s is a directory, expected a file", filename)
}

When it happens

Trigger: Calling stageFile (via StageModel or Execute/ResolveXLangArtifacts) with a filename that does not exist, is a directory, or is not readable by the current process. os.Open returns the underlying error which is wrapped here.

Common situations: Running a Dataflow pipeline with --worker_binary or an XLang artifact path that is wrong relative to the current working directory; the binary was never built; a path is passed without escaping spaces; running under a service account or container lacking read permission on the file.

Understand the failure class

Background: "open() failed", "failed to open file", "cannot create file" — what a file open error means and how to fix it — this error's family across 42 libraries.

Related errors

• invalid staging location
• no Google Cloud project specified. Use --project=
• no Google Cloud region specified. Use --region=
• bad I
• bad KV

AI-assisted analysis of apache/beam@12126d8942 (2026-09-13). Data as JSON: /api/errors/66dbe171651d4d11. Report an issue: GitHub.

Appendix: source

Thrown at sdks/go/pkg/beam/runners/dataflow/dataflowlib/stage.go:42

	"os"

	"cloud.google.com/go/storage"
	"github.com/apache/beam/sdks/v2/go/pkg/beam/core/graph"
	"github.com/apache/beam/sdks/v2/go/pkg/beam/core/runtime/xlangx"
	"github.com/apache/beam/sdks/v2/go/pkg/beam/internal/errors"
	"github.com/apache/beam/sdks/v2/go/pkg/beam/util/gcsx"
)

// StageModel uploads the pipeline model to GCS as a unique object.
func StageModel(ctx context.Context, project, modelURL string, model []byte) error {
	return upload(ctx, project, modelURL, bytes.NewReader(model))
}

// stageFile uploads a file to GCS, and returns the sha256 hash.
func stageFile(ctx context.Context, project, url, filename string) (string, error) {
	fd, err := os.Open(filename)
	if err != nil {
		return "", errors.Wrapf(err, "failed to open file %s", filename)
	}
	defer fd.Close()

	sha256W := sha256.New()
	tee := io.TeeReader(fd, sha256W)
	if err := upload(ctx, project, url, tee); err != nil {
		return "", err
	}
	hash := hex.EncodeToString(sha256W.Sum(nil))
	return hash, nil
}

func upload(ctx context.Context, project, object string, r io.Reader) error {
	bucket, obj, err := gcsx.ParseObject(object)
	if err != nil {
		return errors.Wrapf(err, "invalid staging location %v", object)
	}
	client, err := gcsx.NewClient(ctx, storage.ScopeReadWrite)

View on GitHub (pinned to 12126d8942)