apache/beam · error · RuntimeException

Failed to translate BigtableOptions to BigtableConfig

Error message

Failed to translate BigtableOptions to BigtableConfig

What it means

translateToBigtableConfig reads BigtableOptions (instance id, project id, table id, and key/config files) via filesystem ValueProviders. If any IOException occurs while reading those files, the method throws RuntimeException('Failed to translate BigtableOptions to BigtableConfig') wrapping it. This means pipeline construction failed because BigtableOptions could not be converted into a BigtableConfig.

Source

Thrown at sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/bigtable/BigtableConfigTranslator.java:454

            Credentials credentials =
                ((CredentialOptions.UserSuppliedCredentialOptions) credOptions).getCredential();
            builder.setCredentialFactory(FixedCredentialFactory.create(credentials));
            break;
          case SuppliedJson:
            CredentialOptions.JsonCredentialsOptions jsonCredentialsOptions =
                (CredentialOptions.JsonCredentialsOptions) credOptions;
            builder.setCredentialFactory(
                FixedCredentialFactory.create(
                    GoogleCredentials.fromStream(jsonCredentialsOptions.getInputStream())));
            break;
          case None:
            // pipelineOptions is ignored
            PipelineOptions pipelineOptions = PipelineOptionsFactory.create();
            builder.setCredentialFactory(NoopCredentialFactory.fromOptions(pipelineOptions));
            break;
        }
      } catch (IOException e) {
        throw new RuntimeException("Failed to translate BigtableOptions to BigtableConfig", e);
      }
    }

    return builder.build();
  }

  /** Translate BigtableOptions to BigtableReadOptions. */
  static BigtableReadOptions translateToBigtableReadOptions(
      BigtableReadOptions readOptions, BigtableOptions options) {
    BigtableReadOptions.Builder builder = readOptions.toBuilder();
    builder.setWaitTimeout(
        org.joda.time.Duration.millis(options.getRetryOptions().getReadPartialRowTimeoutMillis()));
    if (options.getCallOptionsConfig().getReadStreamRpcAttemptTimeoutMs().isPresent()) {
      builder.setAttemptTimeout(
          org.joda.time.Duration.millis(
              options.getCallOptionsConfig().getReadStreamRpcAttemptTimeoutMs().get()));
    }
    builder.setOperationTimeout(

View on GitHub (pinned to 12126d8942)

Solutions

Inspect the wrapped IOException cause to see which file path failed
Verify all file paths passed to BigtableOptions/BigtableReadOptions exist and are readable at translation time
Stage key/config files with --files or use GCS locations accessible to the pipeline
Pass credentials via Application Default Credentials instead of file paths to avoid file reads entirely

Example fix

// before
BigtableOptions options = new BigtableOptions().withKeyFile("/local/missing.p12");
// after
BigtableOptions options = new BigtableOptions(); // rely on ADC, or stage the file and use a correct, existing path

Defensive patterns

Strategy: validation

Validate before calling

for (String path : ImmutableList.of(keyFile, tempFile, changeStreamNamesFile)) {
  if (path != null && !new File(path).canRead()) {
    throw new IllegalStateException("Bigtable options file not readable: " + path);
  }
}

Try / catch

try {
  BigtableIO.write().withConfig(bigtableOptions).expand(rows);
} catch (RuntimeException e) {
  if (e.getCause() instanceof IOException) {
    // inspect e.getCause() for the failing file path and fix staging/permissions
  }
  throw e;
}

Prevention

Stage key/config files with the job (--files) or use GCS URIs accessible from workers
Prefer runtime-provided credentials (ADC) to eliminate file reads
Check file existence at pipeline-argument parsing time, before expansion

When it happens

Trigger: BigtableOptions or BigtableReadOptions holds file-backed ValueProviders (e.g. key file, temp pipeline file, change stream names file) whose paths cannot be opened/read at translation time.

Common situations: Missing key file path in a distributed/runtime environment, wrong path or filename typo, file unreadable due to permissions, running on a worker where the file was not staged.

Understand the failure class

Background: "failed to read file", EACCES, ENOENT and "could not read <path>" errors: when a program can't read a file from disk — this error's family across 49 libraries.

Related errors

AI-assisted analysis of apache/beam@12126d8942 (2026-09-13). Data as JSON: /api/errors/ac8dddc450bd789b. Report an issue: GitHub.