apache/iceberg · warning

Failed to load catalog: {}

Error message

Failed to load catalog: {}

What it means

Spark3Util resolves a possibly catalog-qualified identifier. When the CatalogManager's named catalog cannot be loaded (its plugin class fails to instantiate/configure), this warning is logged and null is returned so identifier resolution can try other candidates or fall back to the default catalog.

Source

Thrown at spark/v4.0/spark/src/main/java/org/apache/iceberg/spark/Spark3Util.java:817

  public static CatalogAndIdentifier catalogAndIdentifier(
      SparkSession spark, List<String> nameParts, CatalogPlugin defaultCatalog) {
    CatalogManager catalogManager = spark.sessionState().catalogManager();

    String[] currentNamespace;
    if (defaultCatalog.equals(catalogManager.currentCatalog())) {
      currentNamespace = catalogManager.currentNamespace();
    } else {
      currentNamespace = defaultCatalog.defaultNamespace();
    }

    Pair<CatalogPlugin, Identifier> catalogIdentifier =
        SparkUtil.catalogAndIdentifier(
            nameParts,
            catalogName -> {
              try {
                return catalogManager.catalog(catalogName);
              } catch (Exception e) {
                LOG.warn("Failed to load catalog: {}", catalogName, e);
                return null;
              }
            },
            Identifier::of,
            defaultCatalog,
            currentNamespace);
    return new CatalogAndIdentifier(catalogIdentifier);
  }

  private static TableCatalog asTableCatalog(CatalogPlugin catalog) {
    if (catalog instanceof TableCatalog) {
      return (TableCatalog) catalog;
    }

    throw new IllegalArgumentException(
        String.format(
            "Cannot use catalog %s(%s): not a TableCatalog",
            catalog.name(), catalog.getClass().getName()));

View on GitHub (pinned to 86d9c8fc54)

Solutions

  1. Read the logged exception to see why the catalog failed to load (ClassNotFound vs config error).
  2. Fix spark.sql.catalog.<name> properties: correct implementation class, warehouse, and credentials.
  3. Ensure the catalog's JAR (e.g. hive-metastore, aws-bundle, nessie) is on the Spark classpath.
  4. Use the fully qualified identifier with a valid catalog or fix the default catalog config.

Example fix

// before
spark.conf.set("spark.sql.catalog.myCat", "org.apache.iceberg.shaded.Whatever")

// after
spark.conf.set("spark.sql.catalog.myCat", "org.apache.iceberg.spark.SparkCatalog")
spark.conf.set("spark.sql.catalog.myCat.type", "hadoop")
spark.conf.set("spark.sql.catalog.myCat.warehouse", "s3://bucket/warehouse")
Defensive patterns

Strategy: validation

Validate before calling

// Pre-validate that a named catalog loads before resolving identifiers
try {
  CatalogPlugin c = catalogManager.catalog("myCat");
} catch (Exception e) {
  throw new IllegalArgumentException("Catalog myCat misconfigured: " + e.getMessage(), e);
}
String impl = spark.conf().get("spark.sql.catalog.myCat", null);
Preconditions.checkArgument(impl != null, "spark.sql.catalog.myCat must be set");

Try / catch

try {
  CatalogPlugin c = catalogManager.catalog(name);
} catch (Exception e) {
  LOG.warn("Failed to load catalog: {}", name, e);
  throw new IllegalArgumentException("Check spark.sql.catalog." + name + ".* settings", e);
}

Prevention

When it happens

Trigger: Resolving table identifiers like myCatalog.db.table (via Spark3Util / catalogAndIdentifier) where catalog 'myCatalog' is registered in Spark conf but its implementation class is missing or misconfigured — the catalog load throws and is swallowed with this warning.

Common situations: Typo in catalog implementation class name; missing connector JAR on the classpath; bad catalog options (wrong warehouse URI, missing credentials); using a catalog name that was removed from the config.

Understand the failure class

Background: "missing required config value" errors: why libraries refuse to start when a configuration key is empty, unset, or blank — this error's family across 48 libraries.

Related errors

AI-assisted analysis of apache/iceberg@86d9c8fc54 (2026-09-12). Data as JSON: /api/errors/b4a0d07b8631b20a. Report an issue: GitHub.