alibaba/DataX · error · RuntimeException

Failed to join rows data, unsupported `format` from stream l

Error message

Failed to join rows data, unsupported `format` from stream load properties:

What it means

DorisStreamLoadObserver.addRows() only knows how to join buffered rows for two formats: CSV (joins with line_delimiter) and JSON (wraps rows into a JSON array with ',' separators). If options.getStreamLoadFormat() is anything else, this RuntimeException is thrown. Note the message does not interpolate the offending format — the '%s'-less string ends with a colon, so the bad value is not printed.

Source

Thrown at doriswriter/src/main/java/com/alibaba/datax/plugin/writer/doriswriter/DorisStreamLoadObserver.java:153

            return bos.array();
        }

        if (Keys.StreamLoadFormat.JSON.equals(options.getStreamLoadFormat())) {
            ByteBuffer bos = ByteBuffer.allocate(totalBytes + (rows.isEmpty() ? 2 : rows.size() + 1));
            bos.put("[".getBytes(StandardCharsets.UTF_8));
            byte[] jsonDelimiter = ",".getBytes(StandardCharsets.UTF_8);
            boolean isFirstElement = true;
            for (byte[] row : rows) {
                if (!isFirstElement) {
                    bos.put(jsonDelimiter);
                }
                bos.put(row);
                isFirstElement = false;
            }
            bos.put("]".getBytes(StandardCharsets.UTF_8));
            return bos.array();
        }
        throw new RuntimeException("Failed to join rows data, unsupported `format` from stream load properties:");
    }
    private Map<String, Object> put(String loadUrl, String label, byte[] data) throws IOException {
        LOG.info(String.format("Executing stream load to: '%s', size: '%s'", loadUrl, data.length));
        final HttpClientBuilder httpClientBuilder = HttpClients.custom()
                .setRedirectStrategy(new DefaultRedirectStrategy () {
                    @Override
                    protected boolean isRedirectable(String method) {
                        return true;
                    }
                });
        try ( CloseableHttpClient httpclient = httpClientBuilder.build()) {
            HttpPut httpPut = new HttpPut(loadUrl);
            httpPut.removeHeaders(HttpHeaders.CONTENT_LENGTH);
            httpPut.removeHeaders(HttpHeaders.TRANSFER_ENCODING);
            List<String> cols = options.getColumns();
            if (null != cols && !cols.isEmpty() && Keys.StreamLoadFormat.CSV.equals(options.getStreamLoadFormat())) {
                httpPut.setHeader("columns", String.join(",", cols.stream().map(f -> String.format("`%s`", f)).collect(Collectors.toList())));
            }

View on GitHub (pinned to 80ec23d5c5)

Solutions

1. Set the format to "csv" or "json" exactly (lowercase) in the doriswriter job configuration
2. For line-delimited JSON needs, use "json" format — the plugin wraps rows into a JSON array which Doris accepts
3. If you truly need another format, extend DorisStreamLoadObserver.addRows() with a branch for it and rebuild the plugin

Example fix

// before
"loadProps": { "format": "json_lines" }
// after
"loadProps": { "format": "json" }

// validate format before the job starts
String fmt = String.valueOf(loadProps.get("format")).trim().toLowerCase(Locale.ROOT);
if (!(fmt.equals("csv") || fmt.equals("json"))) {
    throw new IllegalArgumentException("doriswriter supports only 'csv' or 'json', got: " + loadProps.get("format"));
}

boolean isSupportedFormat(String f) {
    String s = f == null ? "csv" : f.trim().toLowerCase(Locale.ROOT);
    return s.equals("csv") || s.equals("json");
}

When it happens

Trigger: Setting "format" under loadProps (or the streamLoadFormat option) to a value other than 'csv' or 'json' — e.g. 'json_lines', 'arrow', or a typo like 'CSV ' with trailing whitespace. addRows() falls through both if-branches and hits the unconditional throw.

Common situations: Copy-pasting a Doris stream-load curl example that uses a format DataX's doriswriter does not implement; assuming the plugin supports every format the Doris stream load itself supports; case/whitespace mismatch in config.

Related errors

• CONFIG_ERROR
• 通道容量[%d]必须大于0.
• dx_digest paras length must be 3
• dx_digest paras index 1 must be md5 or sha1
• dx_digest paras index 2 must be toUpperCase or toLowerCase

AI-assisted analysis of alibaba/DataX@80ec23d5c5 (2026-08-14). Data as JSON: /api/errors/8bc65436143fc053. Report an issue: GitHub.