apache/shardingsphere · error · PipelineInvalidParameterException

Source storage units have different database types

Error message

Source storage units have different database types

What it means

When building a migration job configuration, MigrationJobAPI requires every source storage unit to share one database type: the first unit fixes result.sourceDatabaseType and any later unit with a different type aborts with PipelineInvalidParameterException. Heterogeneous sources are rejected because the pipeline assumes a single source dialect.

Source

Thrown at kernel/data-pipeline/scenario/migration/core/src/main/java/org/apache/shardingsphere/data/pipeline/scenario/migration/api/MigrationJobAPI.java:146

        for (MigrationSourceTargetEntry each : new HashSet<>(sourceTargetEntries).stream()
                .sorted(Comparator.comparing(MigrationSourceTargetEntry::getTargetTableName).thenComparing(each -> each.getSource().format())).collect(Collectors.toList())) {
            sourceDataNodes.computeIfAbsent(each.getTargetTableName(), key -> new LinkedList<>()).add(each.getSource());
            ShardingSpherePreconditions.checkState(1 == sourceDataNodes.get(each.getTargetTableName()).size(),
                    () -> new PipelineInvalidParameterException("More than one source table for " + each.getTargetTableName()));
            String dataSourceName = each.getSource().getDataSourceName();
            if (configSources.containsKey(dataSourceName)) {
                continue;
            }
            ShardingSpherePreconditions.checkContainsKey(metaDataDataSource, dataSourceName,
                    () -> new PipelineInvalidParameterException(dataSourceName + " doesn't exist. Run `SHOW MIGRATION SOURCE STORAGE UNITS;` to verify it."));
            Map<String, Object> sourceDataSourcePoolProps = dataSourceConfigSwapper.swapToMap(metaDataDataSource.get(dataSourceName));
            StandardPipelineDataSourceConfiguration sourceDataSourceConfig = new StandardPipelineDataSourceConfiguration(sourceDataSourcePoolProps);
            configSources.put(dataSourceName, buildYamlPipelineDataSourceConfiguration(sourceDataSourceConfig.getType(), sourceDataSourceConfig.getParameter()));
            DatabaseType sourceDatabaseType = sourceDataSourceConfig.getDatabaseType();
            if (null == result.getSourceDatabaseType()) {
                result.setSourceDatabaseType(sourceDatabaseType.getType());
            } else if (!result.getSourceDatabaseType().equals(sourceDatabaseType.getType())) {
                throw new PipelineInvalidParameterException("Source storage units have different database types");
            }
        }
        result.setSources(configSources);
        ShardingSphereDatabase targetDatabase = PipelineContextManager.getProxyContext().getMetaDataContexts().getMetaData().getDatabase(targetDatabaseName);
        PipelineDataSourceConfiguration targetPipelineDataSourceConfig = buildTargetPipelineDataSourceConfiguration(targetDatabase);
        result.setTarget(buildYamlPipelineDataSourceConfiguration(targetPipelineDataSourceConfig.getType(), targetPipelineDataSourceConfig.getParameter()));
        result.setTargetDatabaseType(targetPipelineDataSourceConfig.getDatabaseType().getType());
        List<JobDataNodeEntry> tablesFirstDataNodes = sourceDataNodes.entrySet().stream()
                .map(entry -> new JobDataNodeEntry(entry.getKey(), entry.getValue().subList(0, 1))).collect(Collectors.toList());
        result.setTargetTableNames(new ArrayList<>(sourceDataNodes.keySet()).stream().sorted().collect(Collectors.toList()));
        result.setTargetTableSchemaMap(buildTargetTableSchemaMap(sourceDataNodes));
        result.setTablesFirstDataNodes(new JobDataNodeLine(tablesFirstDataNodes).marshal());
        result.setJobShardingDataNodes(JobDataNodeLineConvertUtils.convertDataNodesToLines(sourceDataNodes).stream().map(JobDataNodeLine::marshal).collect(Collectors.toList()));
        result.setJobId(PipelineJobIdUtils.marshal(new MigrationJobId(contextKey, result.getJobShardingDataNodes())));
        return result;
    }
    
    private YamlPipelineDataSourceConfiguration buildYamlPipelineDataSourceConfiguration(final String type, final String param) {

View on GitHub (pinned to e952770a21)

Solutions

1. Run SHOW MIGRATION SOURCE STORAGE UNITS; and confirm the database type of every unit that owns the tables being migrated.
2. Split the migration into separate jobs per database type, each referencing only units of one engine.
3. If all sources are really the same engine, check each unit's JDBC URL and stored database type metadata for a misconfiguration (e.g. wrong URL pointing at a different server product).
4. Remove unused/misconfigured storage units from the schema so they cannot be picked up.

Example fix

-- before (t_order on MySQL unit, t_user on PostgreSQL unit)
MIGRATE TABLE db_0.t_order INTO db_1.t_order, db_0.t_user INTO db_1.t_user;
-- after (one job per engine)
MIGRATE TABLE db_0.t_order INTO db_1.t_order;
MIGRATE TABLE db_pg.t_user INTO db_1.t_user;

Set<String> sourceTypes = new HashSet<>();
for (String each : sourceDataNodes.keySet()) {
    sourceTypes.add(getStorageUnitDatabaseType(each)); // from SHOW STORAGE UNITS metadata
}
ShardingSpherePreconditions.checkState(sourceTypes.size() <= 1,
    () -> new IllegalArgumentException("migration sources must share one database type"));

try {
    api.migrate(...);
} catch (final PipelineInvalidParameterException ex) {
    if (ex.getMessage().contains("different database types")) { splitJobPerEngine(); } else { throw ex; }
}

When it happens

Trigger: Running MIGRATE TABLE or a migration job build where the sourceDataNodes span multiple storage units whose JDBC URLs resolve to different database types (e.g. one PostgreSQL and one MySQL unit, or one OpenGauss vs PostgreSQL).

Common situations: Mixing storage units of different engines in one schema and referencing tables from both in a single migration; a unit whose URL/driver metadata resolves to an unexpected database type; renaming or misconfiguring units so the wrong one matches table names.

Related errors

• 1
• Either key generator name or algorithm segment must be provi
• Unknown stream channel type `%s`.
• 0
• 12010

AI-assisted analysis of apache/shardingsphere@e952770a21 (2026-08-14). Data as JSON: /api/errors/65d1ead7d47d6e7d. Report an issue: GitHub.