Skip to content

[MINOR] Clean up Spark 3.2 build/config/docs leftovers#12550

Open
LuciferYang wants to merge 2 commits into
apache:mainfrom
LuciferYang:gluten-minor-build-spark32-leftovers
Open

[MINOR] Clean up Spark 3.2 build/config/docs leftovers#12550
LuciferYang wants to merge 2 commits into
apache:mainfrom
LuciferYang:gluten-minor-build-spark32-leftovers

Conversation

@LuciferYang

Copy link
Copy Markdown
Contributor

What changes were proposed in this pull request?

Follow-up to the Spark 3.2 code cleanup PRs (#12522, #12525, #12532, #12524, #12541, #12543, #12548). This picks up the remaining build-system, dev-script, and documentation leftovers that referred to the no-longer-supported Spark 3.2 shim.

package/pom.xml

Drop the <exclude>*spark3.2*</exclude> entry from the maven-clean-plugin fileset. The pattern protected artifacts that are no longer produced. Sibling entries for the currently-supported versions (spark3.3 through spark4.1) are kept.

LICENSE

  • Drop 9 stale ./shims/spark32/... paths from the Apache Spark attribution section — the shims/spark32/ directory was removed when Spark 3.2 support was dropped. Sibling ./shims/spark33/... paths kept.
  • Drop 6 stale ./backends-clickhouse/src-delta-20/... paths from the Delta Lake attribution section — Delta 2.0 is the Spark-3.2-era Delta, and the directory was removed by [MINOR][CH] Remove orphan Spark 3.2 / Delta 2.0 source directories #12524. Sibling src-delta-23/src-delta-33 lines are left untouched.

dev/run-scala-test.sh, dev/bloop-test.sh

Remove the spark32 / gluten-ut-spark32 module-map entries in both scripts. Both mapped to shims/spark32 and gluten-ut/spark32 filesystem paths that no longer exist.

ep/build-clickhouse/src/resources/bin/gluten.sh

Drop the dead if [ "$SPARK_MAJOR_MINOR_VERSION" == "3.2" ]; then GLUTEN_JARS=…/jars/spark32/* branch. SPARK_MAJOR_MINOR_VERSION cannot be 3.2 on any supported build. The remaining 3.3 branch and the else-fail path are unchanged.

docs/developers/NewToGluten.md

Rewrite the TPC-H troubleshooting failure-message example from /tmp/tpch-approved-plan/v2-bhj/spark322/5.txt (fictional subType + Spark 3.2.2) to a real path using a currently-supported version: /tmp/tpch-approved-plan/v1-bhj/spark35/5.txt. The subType() values are v1 / v1-bhj / v1-ras / v1-bhj-ras.

No functional change on any supported Spark version.

Out of scope (deferred)

Broader LICENSE hygiene unrelated to Spark 3.2 (paths using src-delta-23/src-delta-33 with hyphens while on-disk directories are src-delta23/src-delta33, plus a stale SparkWriteFilesCommitProtocol.scala path) is intentionally left to a separate PR.

How was this patch tested?

  • grep -rn 'spark3\.2|shims/spark32|src-delta-20' against the touched files: 0 remaining occurrences.
  • grep -rn 'spark32|spark3\.2' repo-wide (excluding target/generated/verbatim Delta-fork files and CH Spark-error-test golden data): confirms the only remaining spark32 mentions are inside upstream Delta test files and Spark's own SQL-error golden output (both DO NOT TOUCH).
  • No source (.scala, .java, .cpp) files touched → no compile step needed.

Was this patch authored or co-authored using generative AI tooling?

Generated-by: Claude claude-opus-4-7

Follow-up to the Spark 3.2 code cleanup PRs (apache#12522, apache#12525, apache#12532,
apache#12524, apache#12541, apache#12543, apache#12548). This picks up the remaining
build-system, dev-script, and documentation leftovers that referred to
the no-longer-supported Spark 3.2 shim.

* `package/pom.xml`: drop the `<exclude>*spark3.2*</exclude>` entry from
  the maven-clean fileset. The pattern protected artifacts that are no
  longer produced.
* `LICENSE`: drop 9 stale `./shims/spark32/...` paths from the Apache
  Spark attribution section (the directory was removed when Spark 3.2
  support was dropped) and 6 stale `./backends-clickhouse/src-delta-20/...`
  paths from the Delta Lake attribution section (the directory was
  removed by apache#12524).
* `dev/run-scala-test.sh`, `dev/bloop-test.sh`: remove the two
  `spark32`/`gluten-ut-spark32` module-map entries. Both mapped to
  filesystem paths that no longer exist.
* `ep/build-clickhouse/src/resources/bin/gluten.sh`: drop the dead
  `if [ "$SPARK_MAJOR_MINOR_VERSION" == "3.2" ]; then GLUTEN_JARS=.../jars/spark32/*`
  branch. The remaining 3.3 branch and the else-fail path are unchanged.
* `docs/developers/NewToGluten.md`: rewrite the TPC-H troubleshooting
  path example from the fictional `v2-bhj/spark322/5.txt` to a real
  supported path (`v1-bhj/spark35/5.txt`).

No functional change; broader LICENSE hygiene (hyphenated
`src-delta-23`/`src-delta-33` paths mismatching on-disk `src-delta23`/
`src-delta33`, and the moved `SparkWriteFilesCommitProtocol.scala`) is
intentionally deferred to a separate PR since it is unrelated to Spark
3.2 removal.
Copilot AI review requested due to automatic review settings July 17, 2026 11:32

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Cleans up remaining build-script, packaging, and documentation references to the removed Spark 3.2 shim paths/artifacts to keep the repo consistent with the supported Spark version matrix.

Changes:

  • Remove Spark 3.2-related excludes/paths from packaging cleanup config and LICENSE attributions.
  • Prune dead spark32 module-map entries from dev test helper scripts.
  • Update ClickHouse gluten.sh version branching and refresh the TPC-H golden-plan example path in developer docs.

Reviewed changes

Copilot reviewed 5 out of 6 changed files in this pull request and generated 1 comment.

Show a summary per file
File Description
package/pom.xml Drops *spark3.2* clean-plugin exclude that no longer protects any produced artifacts.
LICENSE Removes stale attributions for deleted shims/spark32 and src-delta-20 paths.
dev/run-scala-test.sh Removes spark32/gluten-ut-spark32 module-map entries pointing to deleted directories.
dev/bloop-test.sh Removes spark32/gluten-ut-spark32 module-map entries pointing to deleted directories.
ep/build-clickhouse/src/resources/bin/gluten.sh Removes dead Spark 3.2 jar-branch; needs follow-up to align supported versions with check-env.sh.
docs/developers/NewToGluten.md Updates TPC-H troubleshooting example paths to a currently supported Spark version.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Comment thread docs/developers/NewToGluten.md Outdated
Actual Plan path: /tmp/tpch-approved-plan/v2-bhj/spark322/5.txt
Golden Plan path: /opt/gluten/backends-velox/target/scala-2.12/test-classes/tpch-approved-plan/v2-bhj/spark322/5.txt (VeloxTPCHSuite.scala:101)
Actual Plan path: /tmp/tpch-approved-plan/v1-bhj/spark35/5.txt
Golden Plan path: /opt/gluten/backends-velox/target/scala-2.12/test-classes/tpch-approved-plan/v1-bhj/spark35/5.txt (VeloxTPCHSuite.scala:101)

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Good catch. Dropped the :101 line number in 87c3513 — the source has already drifted to line 107, and keeping a specific line number in a troubleshooting example just guarantees future drift. The filename alone is enough for navigation.

…ample

Address Copilot review feedback on apache#12550: the example log referenced
`VeloxTPCHSuite.scala:101` but the actual line has drifted (currently
line 107). Drop the specific line number to avoid future drift; the
filename is enough for the reader to navigate.
Copilot AI review requested due to automatic review settings July 17, 2026 11:45

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Copilot reviewed 5 out of 6 changed files in this pull request and generated no new comments.

@philo-he philo-he left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks good. Could you confirm whether the CI failures are related? Thanks.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants