{"record":{"id":"20334059ecb356c6","repo":"apache/beam","slug":"error-getting-last-modified-time-for-q-v","errorCode":null,"errorMessage":"error getting last modified time for %q: %v","messagePattern":"error getting last modified time for %q: (.+?)","errorType":"exception","errorClass":null,"httpStatus":null,"severity":"error","filePath":"sdks/go/pkg/beam/io/fileio/match.go","lineNumber":221,"sourceCode":"\t\t\tPath:         path,\n\t\t\tSize:         size,\n\t\t\tLastModified: mTime,\n\t\t}\n\t}\n\n\treturn metadata, nil\n}\n\nfunc lastModified(ctx context.Context, fs filesystem.Interface, path string) (time.Time, error) {\n\tlmGetter, ok := fs.(filesystem.LastModifiedGetter)\n\tif !ok {\n\t\tlog.Warnf(ctx, \"Filesystem %T does not implement filesystem.LastModifiedGetter\", fs)\n\t\treturn time.Time{}, nil\n\t}\n\n\tmTime, err := lmGetter.LastModified(ctx, path)\n\tif err != nil {\n\t\treturn time.Time{}, fmt.Errorf(\"error getting last modified time for %q: %v\", path, err)\n\t}\n\n\treturn mTime, nil\n}\n\n// duplicateTreatment controls how duplicate matches are treated.\ntype duplicateTreatment int\n\nconst (\n\t// duplicateAllow allows duplicate matches.\n\tduplicateAllow duplicateTreatment = iota\n\t// duplicateAllowIfModified allows duplicate matches only if the file has been modified since it\n\t// was last observed.\n\tduplicateAllowIfModified\n\t// duplicateSkip skips duplicate matches.\n\tduplicateSkip\n)\n","sourceCodeStart":203,"sourceCodeEnd":239,"githubUrl":"https://github.com/apache/beam/blob/12126d8942aaf848030c478b4c6a28c6af861c66/sdks/go/pkg/beam/io/fileio/match.go#L203-L239","documentation":"metadataFromFiles builds FileMetadata for each matched file; for filesystems implementing LastModifiedGetter, lastModified calls LastModified(ctx, path). If that call fails, the path and cause are wrapped with this message. This blocks constructing file metadata (e.g. for file size/time-based processing).","triggerScenarios":"lmGetter.LastModified(ctx, path) returns an error (match.go:221) — invoked via metadataFromFiles after a successful match: permission denied on stat, file vanished between match and metadata read, or FS backend error (GCS 404/403).","commonSituations":"Files deleted by a compaction/GC job between matching and reading; GCS object permission changes; flaky storage backend returning 5xx; custom filesystem implementation whose LastModified misreports errors.","solutions":["Fix the underlying error in %v: 403 → grant read/objects.list IAM; 404 → file disappeared, adjust the glob or timing","Retry transient storage errors (5xx/timeouts) with backoff before failing the bundle","Ensure upstream retention policies don't delete matched files while the pipeline is running","If your custom filesystem can't support LastModified properly, don't implement LastModifiedGetter (the code degrades gracefully then)"],"exampleFix":null,"handlingStrategy":"retry","validationCode":"// before the pipeline: confirm objects are readable\nattrs, err := client.Bucket(bkt).Object(name).Attrs(ctx)\nif err != nil {\n\tlog.Fatalf(\"object %s/%s: %v\", bkt, name, err)\n}","typeGuard":null,"tryCatchPattern":"mTime, err := lmGetter.LastModified(ctx, path)\nif err != nil {\n\tif isTransient(err) { return retryWithBackoff(ctx, path) }\n\treturn time.Time{}, fmt.Errorf(\"last modified %q: %w\", path, err)\n}","preventionTips":["Ensure storage IAM (objects.get / storage.objects.list) for worker identities","Avoid GC/compaction deleting files while the pipeline runs; use retention windows","Retry transient 5xx/timeouts with exponential backoff"],"tags":["filesystem","metadata","gcs","last-modified","beam-io"],"backgroundTag":"file-read-failed","analyzedSha":"12126d8942aaf848030c478b4c6a28c6af861c66","analyzedAt":"2026-09-13T01:50:10.254Z","contentChangedAt":"2026-09-13T01:50:10.254Z","schemaVersion":2},"datasetVersion":"2026-09-20T03:17:13.778Z"}