apache/beam · error · IllegalArgumentException

Unable to fetch file %s to be used locally to create a Kafka

Error message

Unable to fetch file %s to be used locally to create a Kafka Consumer.

What it means

identityOrGcsToLocalFile downloads a remote config file (e.g. a GCS-hosted Kafka config) to a local temp file so a Kafka Consumer can use it. If reading/writing the file fails with IOException, it is wrapped in an IllegalArgumentException saying the file could not be fetched locally.

Source

Thrown at sdks/java/io/kafka/src/main/java/org/apache/beam/sdk/io/kafka/KafkaReadSchemaTransformProvider.java:418

          try {
            Path localFile = Files.createTempFile("", "");
            LOG.info(
                "Downloading {} into local filesystem ({})", configStr, localFile.toAbsolutePath());
            // TODO(pabloem): Only copy if file does not exist.
            try (ReadableByteChannel channel =
                    FileSystems.open(FileSystems.match(configStr).metadata().get(0).resourceId());
                FileOutputStream outputStream = new FileOutputStream(localFile.toFile());
                WritableByteChannel outputChannel = Channels.newChannel(outputStream)) {
              ByteBuffer buffer = ByteBuffer.allocate(1024);
              while (channel.read(buffer) != -1) {
                buffer.flip();
                outputChannel.write(buffer);
                buffer.compact();
              }
            }
            return localFile.toAbsolutePath().toString();
          } catch (IOException e) {
            throw new IllegalArgumentException(
                String.format(
                    "Unable to fetch file %s to be used locally to create a Kafka Consumer.",
                    configStr),
                e);
          }
        } else {
          return configValue;
        }
      } else {
        return configValue;
      }
    }
  }
}

View on GitHub (pinned to 12126d8942)

Solutions

Verify the file exists and the gs:// path is correct (gsutil ls the object)
Check that the runtime has GCS credentials/permissions (service account with storage.objects.get)
Test local filesystem writability (temp dir space/permissions)
If the config is static, ship it as a local path on workers instead of a remote URI

Example fix

// before
throw new IllegalArgumentException(String.format("Unable to fetch file %s ...", configStr), e);
// after (caller-side): validate the file is fetchable first
if (!GcsUtil.isReadable(configStr)) {
  throw new IllegalArgumentException("Config file not accessible before building consumer: " + configStr);
}

Defensive patterns

Strategy: validation

Validate before calling

// verify the remote file is fetchable before building the consumer
java.nio.file.Path local = java.nio.file.Files.createTempFile("kafka", ".cfg");
// attempt copy first; only pass configStr if it succeeds

Try / catch

try {
  String localPath = identityOrGcsToLocalFile(configStr);
} catch (IllegalArgumentException e) {
  LOG.error("Kafka config file fetch failed: {}", e.getCause());
}

Prevention

Verify gs:// paths exist before submitting the pipeline
Grant worker service accounts read access to the config bucket
Prefer distributing config via local worker files when network access is restricted

When it happens

Trigger: apply() is given a configStr pointing to a remote file (GCS) that cannot be read or written locally — GCS object missing, credentials missing, network failure, or local temp-dir write failure.

Common situations: Passing a gs:// path with a typo or missing object, running without GCS access credentials, or an environment with a read-only temp directory.

Understand the failure class

Background: "File not found" and ENOENT errors: why libraries can't find a file that should exist — this error's family across 50 libraries.

Related errors

AI-assisted analysis of apache/beam@12126d8942 (2026-09-13). Data as JSON: /api/errors/11a8fa814e2a4dfe. Report an issue: GitHub.