{"record":{"id":"987b7d5ad9829028","repo":"apache/beam","slug":"preprocess-validation-failure-of-stage-v-v","errorCode":null,"errorMessage":"preprocess validation failure of stage %v: %v","messagePattern":"preprocess validation failure of stage (.+?): (.+?)","errorType":"exception","errorClass":null,"httpStatus":null,"severity":"error","filePath":"sdks/go/pkg/beam/runners/prism/internal/preprocess.go","lineNumber":179,"sourceCode":"\tfacts, err := computeFacts(topological, comps)\n\tif err != nil {\n\t\terr = fmt.Errorf(\"error computing pipeline facts: %w\", err)\n\t\tj.SendMsg(err.Error())\n\t\tj.Failed(err)\n\t\treturn nil\n\t}\n\tfacts.ForcedRoots = forcedRoots\n\n\t// avoid \"unused\" warnings while keeping the older default approach available.\n\t_ = greedyFusion\n\t_ = defaultFusion\n\n\tstages := greedyFusion(topological, comps, facts)\n\n\tfor i, stg := range stages {\n\t\terr := finalizeStage(stg, comps, facts)\n\t\tif err != nil {\n\t\t\terr = fmt.Errorf(\"preprocess validation failure of stage %v: %v\", i, err)\n\t\t\tj.SendMsg(err.Error())\n\t\t\tj.Failed(err)\n\t\t\treturn nil\n\t\t}\n\t}\n\tvar stageDetails []any\n\tfor i, stg := range stages {\n\t\tvar transformNames []string\n\t\tfor _, tid := range stg.transforms {\n\t\t\ttransformNames = append(transformNames, comps.GetTransforms()[tid].GetUniqueName())\n\t\t}\n\t\tstageDetails = append(stageDetails,\n\t\t\tslog.Group(fmt.Sprintf(\"stage-%03d\", i),\n\t\t\t\tslog.String(\"environment\", stg.envID),\n\t\t\t\tslog.Any(\"transforms\", transformNames),\n\t\t\t),\n\t\t)\n\t}","sourceCodeStart":161,"sourceCodeEnd":197,"githubUrl":"https://github.com/apache/beam/blob/12126d8942aaf848030c478b4c6a28c6af861c66/sdks/go/pkg/beam/runners/prism/internal/preprocess.go#L161-L197","documentation":"After greedy fusion, Prism validates each fused stage via finalizeStage; a failure means the fused stage is internally inconsistent (bad wiring, unsupported combine, etc.). The error is wrapped with the stage index, reported to the job's message stream, and the job is Failed.","triggerScenarios":"executePipeline → preProcessGraph loops over stages produced by greedyFusion and finalizeStage returns an error for stage i — the pipeline graph yields a stage Prism cannot finalize.","commonSituations":"Pipelines with transform combinations that fuse into invalid stages (e.g. unsupported GBK/flatten placements after fusion); SDK pipeline shapes not exercised by Prism tests.","solutions":["Read the %v inner error for the specific stage and the stage index to locate the offending fused stage.","Restructure the pipeline (e.g. split complex composite DoFns, avoid problematic transform adjacency) so stages fuse differently.","Try forcing stage boundaries (e.g. reshuffles) to prevent the bad fusion.","Verify with DirectRunner and file a Prism bug with the pipeline if it's a valid graph."],"exampleFix":null,"handlingStrategy":"try-catch","validationCode":"// validate the graph on a permissive runner before prism stage fusion\nif err := directRun(pipeline); err != nil {\n    return fmt.Errorf(\"pipeline won't validate: %w\", err)\n}","typeGuard":null,"tryCatchPattern":"err := submitToPrism(pipeline)\nif err != nil && strings.Contains(err.Error(), \"preprocess validation failure of stage\") {\n    // restructure pipeline to change fusion, then retry once\n}","preventionTips":["Avoid exotic transform adjacencies that fuse into unsupported stages.","Insert explicit reshape/reshuffle boundaries to control fusion.","Keep prism and SDK versions aligned; validate on DirectRunner first."],"tags":["go","beam-prism","stage-fusion","preprocessing"],"backgroundTag":"internal-invariant-violation","analyzedSha":"12126d8942aaf848030c478b4c6a28c6af861c66","analyzedAt":"2026-09-13T01:50:10.254Z","contentChangedAt":"2026-09-13T01:50:10.254Z","schemaVersion":2},"datasetVersion":"2026-09-20T03:17:13.778Z"}