What causes "Target metadata lookup failed for path=%s" in apache/seatunnel?

Retry the job; these are usually transient storage errors — check the wrapped cause for specifics

apache/seatunnel · error · IOException

Target metadata lookup failed for path=%s

Error message

Target metadata lookup failed for path=%s

What it means

In UpdateFileMetadataLoader.loadPoints, per-path target metadata lookups run concurrently; an IOException from target.getFileStatus (other than FileNotFoundException, which is treated as 'not exists') is rethrown as 'Target metadata lookup failed for path=...'. It indicates a single-file stat call against the target filesystem failed.

Source

Thrown at seatunnel-connectors-v2/connector-file/connector-file-base/src/main/java/org/apache/seatunnel/connectors/seatunnel/file/source/reader/UpdateFileMetadataLoader.java:173

        long started = System.nanoTime();
        IOException failure = null;
        try {
            while (completed < requests.size()) {
                while (submitted < requests.size() && submitted - completed < maxInFlight) {
                    Request request = requests.get(submitted++);
                    inFlight.add(
                            completion.submit(
                                    () -> {
                                        int current = active.incrementAndGet();
                                        peakConcurrency.accumulateAndGet(current, Math::max);
                                        try {
                                            return new Lookup(
                                                    request,
                                                    target.getFileStatus(request.targetPath));
                                        } catch (FileNotFoundException ignored) {
                                            return new Lookup(request, null);
                                        } catch (IOException e) {
                                            throw new IOException(
                                                    "Target metadata lookup failed for path="
                                                            + mask(new Path(request.targetPath)),
                                                    e);
                                        } finally {
                                            active.decrementAndGet();
                                        }
                                    }));
                    peakInFlight = Math.max(peakInFlight, submitted - completed);
                }
                try {
                    Future<Lookup> completedFuture = completion.take();
                    inFlight.remove(completedFuture);
                    Lookup lookup = completedFuture.get();
                    ordered[lookup.request.order] = lookup.status;
                    completed++;
                } catch (InterruptedException e) {
                    Thread.currentThread().interrupt();
                    failure = new IOException("Interrupted during target_point_lookup", e);

View on GitHub (pinned to cf67b549a7)

Solutions

Retry the job; these are usually transient storage errors — check the wrapped cause for specifics
Confirm read permissions on the target path for the SeaTunnel process user
Reduce concurrency of point lookups or add backoff if the storage is rate-limiting
Validate target storage credentials/tokens haven't expired mid-run

Example fix

// before
// stat calls failing with 503 Slow Down
// after
// add retry policy with exponential backoff on the target filesystem client, rerun job

Defensive patterns

Strategy: retry

Validate before calling

// pre-flight stat on a sample target path
try { fs.getFileStatus(sampleTargetPath); }
catch (IOException e) { throw new IllegalStateException("Target stat probe failed", e); }

Try / catch

try {
    loader.load(requests);
} catch (IOException e) {
    if (e.getMessage().startsWith("Target metadata lookup failed")) {
        retryWithBackoff(e, 3);
    } else throw e;
}

Prevention

Add client-level retries with exponential backoff for stat/getFileStatus calls
Limit concurrent point lookups to avoid throttling
Check target path read permissions for the job's principal

When it happens

Trigger: getFileStatus on request.targetPath throwing IOException due to transient storage errors, permission problems, throttling, or connectivity failures to the target filesystem.

Common situations: Same as bulk listing but for point lookups: S3 throttling under many concurrent stats, HDFS DataNode/NameNode hiccups, expired session tokens during long jobs.

Understand the failure class

Background: "failed to read file", EACCES, ENOENT and "could not read <path>" errors: when a program can't read a file from disk — this error's family across 49 libraries.

Related errors

AI-assisted analysis of apache/seatunnel@cf67b549a7 (2026-09-10). Data as JSON: /api/errors/3308222e2e2d6c50. Report an issue: GitHub.