[MINOR] Clean up Spark 3.2 build/config/docs leftovers#12550
[MINOR] Clean up Spark 3.2 build/config/docs leftovers#12550LuciferYang wants to merge 2 commits into
Conversation
Follow-up to the Spark 3.2 code cleanup PRs (apache#12522, apache#12525, apache#12532, apache#12524, apache#12541, apache#12543, apache#12548). This picks up the remaining build-system, dev-script, and documentation leftovers that referred to the no-longer-supported Spark 3.2 shim. * `package/pom.xml`: drop the `<exclude>*spark3.2*</exclude>` entry from the maven-clean fileset. The pattern protected artifacts that are no longer produced. * `LICENSE`: drop 9 stale `./shims/spark32/...` paths from the Apache Spark attribution section (the directory was removed when Spark 3.2 support was dropped) and 6 stale `./backends-clickhouse/src-delta-20/...` paths from the Delta Lake attribution section (the directory was removed by apache#12524). * `dev/run-scala-test.sh`, `dev/bloop-test.sh`: remove the two `spark32`/`gluten-ut-spark32` module-map entries. Both mapped to filesystem paths that no longer exist. * `ep/build-clickhouse/src/resources/bin/gluten.sh`: drop the dead `if [ "$SPARK_MAJOR_MINOR_VERSION" == "3.2" ]; then GLUTEN_JARS=.../jars/spark32/*` branch. The remaining 3.3 branch and the else-fail path are unchanged. * `docs/developers/NewToGluten.md`: rewrite the TPC-H troubleshooting path example from the fictional `v2-bhj/spark322/5.txt` to a real supported path (`v1-bhj/spark35/5.txt`). No functional change; broader LICENSE hygiene (hyphenated `src-delta-23`/`src-delta-33` paths mismatching on-disk `src-delta23`/ `src-delta33`, and the moved `SparkWriteFilesCommitProtocol.scala`) is intentionally deferred to a separate PR since it is unrelated to Spark 3.2 removal.
There was a problem hiding this comment.
Pull request overview
Cleans up remaining build-script, packaging, and documentation references to the removed Spark 3.2 shim paths/artifacts to keep the repo consistent with the supported Spark version matrix.
Changes:
- Remove Spark 3.2-related excludes/paths from packaging cleanup config and
LICENSEattributions. - Prune dead
spark32module-map entries from dev test helper scripts. - Update ClickHouse
gluten.shversion branching and refresh the TPC-H golden-plan example path in developer docs.
Reviewed changes
Copilot reviewed 5 out of 6 changed files in this pull request and generated 1 comment.
Show a summary per file
| File | Description |
|---|---|
| package/pom.xml | Drops *spark3.2* clean-plugin exclude that no longer protects any produced artifacts. |
| LICENSE | Removes stale attributions for deleted shims/spark32 and src-delta-20 paths. |
| dev/run-scala-test.sh | Removes spark32/gluten-ut-spark32 module-map entries pointing to deleted directories. |
| dev/bloop-test.sh | Removes spark32/gluten-ut-spark32 module-map entries pointing to deleted directories. |
| ep/build-clickhouse/src/resources/bin/gluten.sh | Removes dead Spark 3.2 jar-branch; needs follow-up to align supported versions with check-env.sh. |
| docs/developers/NewToGluten.md | Updates TPC-H troubleshooting example paths to a currently supported Spark version. |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
| Actual Plan path: /tmp/tpch-approved-plan/v2-bhj/spark322/5.txt | ||
| Golden Plan path: /opt/gluten/backends-velox/target/scala-2.12/test-classes/tpch-approved-plan/v2-bhj/spark322/5.txt (VeloxTPCHSuite.scala:101) | ||
| Actual Plan path: /tmp/tpch-approved-plan/v1-bhj/spark35/5.txt | ||
| Golden Plan path: /opt/gluten/backends-velox/target/scala-2.12/test-classes/tpch-approved-plan/v1-bhj/spark35/5.txt (VeloxTPCHSuite.scala:101) |
There was a problem hiding this comment.
Good catch. Dropped the :101 line number in 87c3513 — the source has already drifted to line 107, and keeping a specific line number in a troubleshooting example just guarantees future drift. The filename alone is enough for navigation.
…ample Address Copilot review feedback on apache#12550: the example log referenced `VeloxTPCHSuite.scala:101` but the actual line has drifted (currently line 107). Drop the specific line number to avoid future drift; the filename is enough for the reader to navigate.
philo-he
left a comment
There was a problem hiding this comment.
Looks good. Could you confirm whether the CI failures are related? Thanks.
What changes were proposed in this pull request?
Follow-up to the Spark 3.2 code cleanup PRs (#12522, #12525, #12532, #12524, #12541, #12543, #12548). This picks up the remaining build-system, dev-script, and documentation leftovers that referred to the no-longer-supported Spark 3.2 shim.
package/pom.xmlDrop the
<exclude>*spark3.2*</exclude>entry from themaven-clean-pluginfileset. The pattern protected artifacts that are no longer produced. Sibling entries for the currently-supported versions (spark3.3throughspark4.1) are kept.LICENSE./shims/spark32/...paths from the Apache Spark attribution section — theshims/spark32/directory was removed when Spark 3.2 support was dropped. Sibling./shims/spark33/...paths kept../backends-clickhouse/src-delta-20/...paths from the Delta Lake attribution section — Delta 2.0 is the Spark-3.2-era Delta, and the directory was removed by [MINOR][CH] Remove orphan Spark 3.2 / Delta 2.0 source directories #12524. Siblingsrc-delta-23/src-delta-33lines are left untouched.dev/run-scala-test.sh,dev/bloop-test.shRemove the
spark32/gluten-ut-spark32module-map entries in both scripts. Both mapped toshims/spark32andgluten-ut/spark32filesystem paths that no longer exist.ep/build-clickhouse/src/resources/bin/gluten.shDrop the dead
if [ "$SPARK_MAJOR_MINOR_VERSION" == "3.2" ]; then GLUTEN_JARS=…/jars/spark32/*branch.SPARK_MAJOR_MINOR_VERSIONcannot be3.2on any supported build. The remaining3.3branch and the else-fail path are unchanged.docs/developers/NewToGluten.mdRewrite the TPC-H troubleshooting failure-message example from
/tmp/tpch-approved-plan/v2-bhj/spark322/5.txt(fictional subType + Spark 3.2.2) to a real path using a currently-supported version:/tmp/tpch-approved-plan/v1-bhj/spark35/5.txt. ThesubType()values arev1/v1-bhj/v1-ras/v1-bhj-ras.No functional change on any supported Spark version.
Out of scope (deferred)
Broader
LICENSEhygiene unrelated to Spark 3.2 (paths usingsrc-delta-23/src-delta-33with hyphens while on-disk directories aresrc-delta23/src-delta33, plus a staleSparkWriteFilesCommitProtocol.scalapath) is intentionally left to a separate PR.How was this patch tested?
grep -rn 'spark3\.2|shims/spark32|src-delta-20'against the touched files: 0 remaining occurrences.grep -rn 'spark32|spark3\.2'repo-wide (excluding target/generated/verbatim Delta-fork files and CH Spark-error-test golden data): confirms the only remainingspark32mentions are inside upstream Delta test files and Spark's own SQL-error golden output (both DO NOT TOUCH)..scala,.java,.cpp) files touched → no compile step needed.Was this patch authored or co-authored using generative AI tooling?
Generated-by: Claude claude-opus-4-7