What causes "error getting last modified time for %q: %v" in apache/beam?

Fix the underlying error in %v: 403 → grant read/objects.list IAM; 404 → file disappeared, adjust the glob or timing

error getting last modified time for

Error message

error getting last modified time for %q: %v

What it means

metadataFromFiles builds FileMetadata for each matched file; for filesystems implementing LastModifiedGetter, lastModified calls LastModified(ctx, path). If that call fails, the path and cause are wrapped with this message. This blocks constructing file metadata (e.g. for file size/time-based processing).

Solutions

Fix the underlying error in %v: 403 → grant read/objects.list IAM; 404 → file disappeared, adjust the glob or timing
Retry transient storage errors (5xx/timeouts) with backoff before failing the bundle
Ensure upstream retention policies don't delete matched files while the pipeline is running
If your custom filesystem can't support LastModified properly, don't implement LastModifiedGetter (the code degrades gracefully then)

Defensive patterns

Strategy: retry

Validate before calling

// before the pipeline: confirm objects are readable
attrs, err := client.Bucket(bkt).Object(name).Attrs(ctx)
if err != nil {
	log.Fatalf("object %s/%s: %v", bkt, name, err)
}

Try / catch

mTime, err := lmGetter.LastModified(ctx, path)
if err != nil {
	if isTransient(err) { return retryWithBackoff(ctx, path) }
	return time.Time{}, fmt.Errorf("last modified %q: %w", path, err)
}

Prevention

Ensure storage IAM (objects.get / storage.objects.list) for worker identities
Avoid GC/compaction deleting files while the pipeline runs; use retention windows
Retry transient 5xx/timeouts with exponential backoff

When it happens

Trigger: lmGetter.LastModified(ctx, path) returns an error (match.go:221) — invoked via metadataFromFiles after a successful match: permission denied on stat, file vanished between match and metadata read, or FS backend error (GCS 404/403).

Common situations: Files deleted by a compaction/GC job between matching and reading; GCS object permission changes; flaky storage backend returning 5xx; custom filesystem implementation whose LastModified misreports errors.

Understand the failure class

Background: "failed to read file", EACCES, ENOENT and "could not read <path>" errors: when a program can't read a file from disk — this error's family across 49 libraries.

Related errors

AI-assisted analysis of apache/beam@12126d8942 (2026-09-13). Data as JSON: /api/errors/20334059ecb356c6. Report an issue: GitHub.

Appendix: source

Thrown at sdks/go/pkg/beam/io/fileio/match.go:221

			Path:         path,
			Size:         size,
			LastModified: mTime,
		}
	}

	return metadata, nil
}

func lastModified(ctx context.Context, fs filesystem.Interface, path string) (time.Time, error) {
	lmGetter, ok := fs.(filesystem.LastModifiedGetter)
	if !ok {
		log.Warnf(ctx, "Filesystem %T does not implement filesystem.LastModifiedGetter", fs)
		return time.Time{}, nil
	}

	mTime, err := lmGetter.LastModified(ctx, path)
	if err != nil {
		return time.Time{}, fmt.Errorf("error getting last modified time for %q: %v", path, err)
	}

	return mTime, nil
}

// duplicateTreatment controls how duplicate matches are treated.
type duplicateTreatment int

const (
	// duplicateAllow allows duplicate matches.
	duplicateAllow duplicateTreatment = iota
	// duplicateAllowIfModified allows duplicate matches only if the file has been modified since it
	// was last observed.
	duplicateAllowIfModified
	// duplicateSkip skips duplicate matches.
	duplicateSkip
)

View on GitHub (pinned to 12126d8942)