[GLUTEN-12474][CORE] Preserve V1 write ordering for dynamic partition writes by wForget · Pull Request #12514 · apache/gluten

wForget · 2026-07-15T07:13:59Z

What changes are proposed in this pull request?

How was this patch tested?

Was this patch authored or co-authored using generative AI tooling?

Related issue: #12474

github-actions · 2026-07-15T07:18:10Z

Run Gluten Clickhouse CI on x86

wForget · 2026-07-15T08:53:52Z

Successfully reproduced this issue, https://github.com/apache/gluten/actions/runs/29396675906/job/87292215408?pr=12514

2026-07-15T08:05:01.9825974Z - Gluten - GLUTEN-12474: preserve ordering for dynamic partition writes *** FAILED ***
2026-07-15T08:05:01.9827943Z   org.apache.spark.SparkException: Job aborted due to stage failure: Task 0 in stage 64.0 failed 1 times, most recent failure: Lost task 0.0 in stage 64.0 (TID 82) (705c9412ef57 executor driver): org.apache.spark.SparkException: [TASK_WRITE_FAILED] Task failed while writing rows to file:/__w/gluten/gluten/gluten-ut/spark35/target/scala-2.13/test-classes/unit-tests-working-home/spark-warehouse/gluten_12474_tgt/.spark-staging-2f3d085e-dac8-4ee5-92b7-b06bc0b10194.
2026-07-15T08:05:01.9830247Z 	at org.apache.spark.sql.errors.QueryExecutionErrors$.taskFailedWhileWritingRowsError(QueryExecutionErrors.scala:775)
2026-07-15T08:05:01.9831086Z 	at org.apache.spark.sql.execution.datasources.FileFormatWriter$.executeTask(FileFormatWriter.scala:420)
2026-07-15T08:05:01.9831899Z 	at org.apache.spark.sql.execution.datasources.WriteFilesExec.$anonfun$doExecuteWrite$1(WriteFiles.scala:100)
2026-07-15T08:05:01.9832558Z 	at org.apache.spark.rdd.RDD.$anonfun$mapPartitionsInternal$2(RDD.scala:893)
2026-07-15T08:05:01.9833090Z 	at org.apache.spark.rdd.RDD.$anonfun$mapPartitionsInternal$2$adapted(RDD.scala:893)
2026-07-15T08:05:01.9833638Z 	at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:52)
2026-07-15T08:05:01.9834137Z 	at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:367)
2026-07-15T08:05:01.9834561Z 	at org.apache.spark.rdd.RDD.iterator(RDD.scala:331)
2026-07-15T08:05:01.9834982Z 	at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:93)
2026-07-15T08:05:01.9835494Z 	at org.apache.spark.TaskContext.runTaskWithListeners(TaskContext.scala:166)
2026-07-15T08:05:01.9835965Z 	at org.apache.spark.scheduler.Task.run(Task.scala:141)
2026-07-15T08:05:01.9836424Z 	at org.apache.spark.executor.Executor$TaskRunner.$anonfun$run$4(Executor.scala:620)
2026-07-15T08:05:01.9837231Z 	at org.apache.spark.util.SparkErrorUtils.tryWithSafeFinally(SparkErrorUtils.scala:64)
2026-07-15T08:05:01.9837858Z 	at org.apache.spark.util.SparkErrorUtils.tryWithSafeFinally$(SparkErrorUtils.scala:61)
2026-07-15T08:05:01.9838472Z 	at org.apache.spark.util.Utils$.tryWithSafeFinally(Utils.scala:94)
2026-07-15T08:05:01.9838952Z 	at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:623)
2026-07-15T08:05:01.9839993Z 	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)
2026-07-15T08:05:01.9840686Z 	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
2026-07-15T08:05:01.9841178Z 	at java.base/java.lang.Thread.run(Thread.java:833)
2026-07-15T08:05:01.9842856Z Caused by: org.apache.hadoop.fs.FileAlreadyExistsException: File already exists: file:/__w/gluten/gluten/gluten-ut/spark35/target/scala-2.13/test-classes/unit-tests-working-home/spark-warehouse/gluten_12474_tgt/.spark-staging-2f3d085e-dac8-4ee5-92b7-b06bc0b10194/_temporary/0/_temporary/attempt_202607150105011536507608415885554_0064_m_000000_82/day=2026-06-01/part-00000-2f3d085e-dac8-4ee5-92b7-b06bc0b10194.c000.snappy.orc
2026-07-15T08:05:01.9844817Z 	at org.apache.hadoop.fs.RawLocalFileSystem.create(RawLocalFileSystem.java:421)
2026-07-15T08:05:01.9845372Z 	at org.apache.hadoop.fs.RawLocalFileSystem.create(RawLocalFileSystem.java:459)
2026-07-15T08:05:01.9846018Z 	at org.apache.hadoop.fs.ChecksumFileSystem$ChecksumFSOutputSummer.<init>(ChecksumFileSystem.java:433)
2026-07-15T08:05:01.9846650Z 	at org.apache.hadoop.fs.ChecksumFileSystem.create(ChecksumFileSystem.java:521)
2026-07-15T08:05:01.9847179Z 	at org.apache.hadoop.fs.ChecksumFileSystem.create(ChecksumFileSystem.java:500)
2026-07-15T08:05:01.9847670Z 	at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:1195)
2026-07-15T08:05:01.9848100Z 	at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:1175)
2026-07-15T08:05:01.9848567Z 	at org.apache.orc.impl.PhysicalFsWriter.<init>(PhysicalFsWriter.java:94)
2026-07-15T08:05:01.9849202Z 	at org.apache.orc.impl.WriterImpl.<init>(WriterImpl.java:230)
2026-07-15T08:05:01.9849796Z 	at org.apache.orc.OrcFile.createWriter(OrcFile.java:1040)
2026-07-15T08:05:01.9850358Z 	at org.apache.spark.sql.execution.datasources.orc.OrcOutputWriter.<init>(OrcOutputWriter.scala:49)
2026-07-15T08:05:01.9851115Z 	at org.apache.spark.sql.execution.datasources.orc.OrcFileFormat$$anon$1.newInstance(OrcFileFormat.scala:89)
2026-07-15T08:05:01.9852012Z 	at org.apache.spark.sql.execution.datasources.BaseDynamicPartitionDataWriter.renewCurrentWriter(FileFormatDataWriter.scala:298)
2026-07-15T08:05:01.9852980Z 	at org.apache.spark.sql.execution.datasources.DynamicPartitionDataSingleWriter.write(FileFormatDataWriter.scala:365)
2026-07-15T08:05:01.9853873Z 	at org.apache.spark.sql.execution.datasources.FileFormatDataWriter.writeWithMetrics(FileFormatDataWriter.scala:85)
2026-07-15T08:05:01.9855256Z 	at org.apache.spark.sql.execution.datasources.FileFormatDataWriter.writeWithIterator(FileFormatDataWriter.scala:92)
2026-07-15T08:05:01.9856136Z 	at org.apache.spark.sql.execution.datasources.FileFormatWriter$.$anonfun$executeTask$1(FileFormatWriter.scala:403)
2026-07-15T08:05:01.9856886Z 	at org.apache.spark.util.Utils$.tryWithSafeFinallyAndFailureCallbacks(Utils.scala:1397)
2026-07-15T08:05:01.9857571Z 	at org.apache.spark.sql.execution.datasources.FileFormatWriter$.executeTask(FileFormatWriter.scala:410)
2026-07-15T08:05:01.9858070Z 	... 17 more

github-actions · 2026-07-15T09:18:21Z

Run Gluten Clickhouse CI on x86

github-actions · 2026-07-15T09:18:56Z

Run Gluten Clickhouse CI on x86

github-actions · 2026-07-15T10:12:03Z

Run Gluten Clickhouse CI on x86

github-actions · 2026-07-15T10:19:25Z

Run Gluten Clickhouse CI on x86

wForget · 2026-07-16T03:45:36Z

ClickHouse CI failures:

20:24:15  12:24:15.295 ERROR org.apache.spark.sql.execution.datasources.FileFormatWriter: Aborting job 1a7f0bae-125a-451d-a16d-04978ef8c357.
20:24:15  org.apache.spark.SparkException: Job aborted due to stage failure: Task 0 in stage 245.0 failed 1 times, most recent failure: Lost task 0.0 in stage 245.0 (TID 245) (gluten-gluten-ci-20575-cmcll-fz22k-wcm3r executor driver): org.apache.spark.SparkRuntimeException: [EXCEED_LIMIT_LENGTH] Exceeds char/varchar type length limitation: 5.
20:24:15  	at org.apache.spark.sql.errors.QueryExecutionErrors$.exceedMaxLimit(QueryExecutionErrors.scala:2577)
20:24:15  	at org.apache.spark.sql.errors.QueryExecutionErrors.exceedMaxLimit(QueryExecutionErrors.scala)
20:24:15  	at org.apache.spark.sql.catalyst.util.CharVarcharCodegenUtils.trimTrailingSpaces(CharVarcharCodegenUtils.java:31)
20:24:15  	at org.apache.spark.sql.catalyst.util.CharVarcharCodegenUtils.charTypeWriteSideCheck(CharVarcharCodegenUtils.java:44)
20:24:15  	at org.apache.spark.sql.catalyst.expressions.GeneratedClass$GeneratedIteratorForCodegenStage1.processNext(Unknown Source)
20:24:15  	at org.apache.spark.sql.execution.BufferedRowIterator.hasNext(BufferedRowIterator.java:43)
20:24:15  	at org.apache.spark.sql.execution.WholeStageCodegenEvaluatorFactory$WholeStageCodegenPartitionEvaluator$$anon$1.hasNext(WholeStageCodegenEvaluatorFactory.scala:43)
20:24:15  - length check for input string values: partitioned columns *** FAILED ***
20:24:15    java.lang.ClassCastException: class org.apache.spark.SparkRuntimeException cannot be cast to class org.apache.spark.SparkException (org.apache.spark.SparkRuntimeException and org.apache.spark.SparkException are in unnamed module of loader 'app')
20:24:15    at org.apache.spark.sql.CharVarcharTestSuite.$anonfun$$init$$84(CharVarcharTestSuite.scala:342)
20:24:15    at org.apache.spark.sql.CharVarcharTestSuite.$anonfun$$init$$84$adapted(CharVarcharTestSuite.scala:336)
20:24:15    at org.apache.spark.sql.CharVarcharTestSuite.$anonfun$testTableWrite$1(CharVarcharTestSuite.scala:309)
20:24:15    at scala.runtime.java8.JFunction0$mcV$sp.apply(JFunction0$mcV$sp.scala:18)
20:24:15    at org.apache.spark.util.SparkErrorUtils.tryWithSafeFinally(SparkErrorUtils.scala:64)
20:24:15    at org.apache.spark.util.SparkErrorUtils.tryWithSafeFinally$(SparkErrorUtils.scala:61)
20:24:15    at org.apache.spark.util.Utils$.tryWithSafeFinally(Utils.scala:94)
20:24:15    at org.apache.spark.sql.test.SQLTestUtilsBase.withTable(SQLTestUtils.scala:306)
20:24:15    at org.apache.spark.sql.test.SQLTestUtilsBase.withTable$(SQLTestUtils.scala:304)
20:24:15    at org.apache.spark.sql.FileSourceCharVarcharTestSuite.withTable(CharVarcharTestSuite.scala:953)
20:24:15    ...
......
20:24:24  12:24:24.247 ERROR org.apache.spark.sql.execution.datasources.FileFormatWriter: Aborting job 12d3d584-5c04-43b9-9661-c250ea02e220.
20:24:24  org.apache.spark.SparkException: Job aborted due to stage failure: Task 0 in stage 579.0 failed 1 times, most recent failure: Lost task 0.0 in stage 579.0 (TID 551) (gluten-gluten-ci-20575-cmcll-fz22k-wcm3r executor driver): org.apache.spark.SparkRuntimeException: [EXCEED_LIMIT_LENGTH] Exceeds char/varchar type length limitation: 5.
20:24:24  	at org.apache.spark.sql.errors.QueryExecutionErrors$.exceedMaxLimit(QueryExecutionErrors.scala:2577)
20:24:24  	at org.apache.spark.sql.errors.QueryExecutionErrors.exceedMaxLimit(QueryExecutionErrors.scala)
20:24:24  	at org.apache.spark.sql.catalyst.util.CharVarcharCodegenUtils.trimTrailingSpaces(CharVarcharCodegenUtils.java:31)
20:24:24  	at org.apache.spark.sql.catalyst.util.CharVarcharCodegenUtils.charTypeWriteSideCheck(CharVarcharCodegenUtils.java:44)
20:24:24  	at org.apache.spark.sql.catalyst.expressions.GeneratedClass$GeneratedIteratorForCodegenStage1.processNext(Unknown Source)
20:24:24  	at org.apache.spark.sql.execution.BufferedRowIterator.hasNext(BufferedRowIterator.java:43)
20:24:24  	at org.apache.spark.sql.execution.WholeStageCodegenEvaluatorFactory$WholeStageCodegenPartitionEvaluator$$anon$1.hasNext(WholeStageCodegenEvaluatorFactory.scala:43)
20:24:24  - char/varchar type values length check: partitioned columns of other types *** FAILED ***
20:24:24    java.lang.ClassCastException: class org.apache.spark.SparkRuntimeException cannot be cast to class org.apache.spark.SparkException (org.apache.spark.SparkRuntimeException and org.apache.spark.SparkException are in unnamed module of loader 'app')
20:24:24    at org.apache.spark.sql.FileSourceCharVarcharTestSuite.$anonfun$new$76(CharVarcharTestSuite.scala:1051)
20:24:24    at scala.runtime.java8.JFunction0$mcV$sp.apply(JFunction0$mcV$sp.scala:18)
20:24:24    at org.apache.spark.util.SparkErrorUtils.tryWithSafeFinally(SparkErrorUtils.scala:64)
20:24:24    at org.apache.spark.util.SparkErrorUtils.tryWithSafeFinally$(SparkErrorUtils.scala:61)
20:24:24    at org.apache.spark.util.Utils$.tryWithSafeFinally(Utils.scala:94)
20:24:24    at org.apache.spark.sql.test.SQLTestUtilsBase.withTable(SQLTestUtils.scala:306)
20:24:24    at org.apache.spark.sql.test.SQLTestUtilsBase.withTable$(SQLTestUtils.scala:304)
20:24:24    at org.apache.spark.sql.FileSourceCharVarcharTestSuite.withTable(CharVarcharTestSuite.scala:953)
20:24:24    at org.apache.spark.sql.FileSourceCharVarcharTestSuite.$anonfun$new$75(CharVarcharTestSuite.scala:1040)
20:24:24    at org.apache.spark.sql.FileSourceCharVarcharTestSuite.$anonfun$new$75$adapted(CharVarcharTestSuite.scala:1039)
20:24:24    ...

github-actions · 2026-07-17T06:37:29Z

Run Gluten Clickhouse CI on x86

github-actions · 2026-07-17T10:06:52Z

Run Gluten Clickhouse CI on x86

test GLUTEN-12474

5f5ecb9

github-actions Bot added the CORE works for Gluten Core label Jul 15, 2026

wForget changed the title ~~Test~~ Test GLUTEN-12474 Jul 15, 2026

Preserve V1 write ordering when inserting local sorts

91bf3e2

add unit tests for other spark version

1817d26

wForget mentioned this pull request Jul 15, 2026

[VL] Gluten fails to satisfy the ordering required by dynamic partition column #12474

Open

wForget added 2 commits July 15, 2026 17:46

fix

50ccf39

fix spotless check

9d9c4bc

wForget changed the title ~~Test GLUTEN-12474~~ [GLUTEN-12474][CORE] Preserve V1 write ordering for dynamic partition writes Jul 16, 2026

Enhance local sort handling for GlutenPlan support

71747fb

wForget force-pushed the GLUTEN-12474 branch from 2f0194c to 71747fb Compare July 17, 2026 10:06

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

[GLUTEN-12474][CORE] Preserve V1 write ordering for dynamic partition writes#12514

[GLUTEN-12474][CORE] Preserve V1 write ordering for dynamic partition writes#12514
wForget wants to merge 6 commits into
apache:mainfrom
wForget:GLUTEN-12474

wForget commented Jul 15, 2026 •

edited by github-actions Bot

Loading

Uh oh!

github-actions Bot commented Jul 15, 2026

Uh oh!

wForget commented Jul 15, 2026

Uh oh!

github-actions Bot commented Jul 15, 2026

Uh oh!

github-actions Bot commented Jul 15, 2026

Uh oh!

github-actions Bot commented Jul 15, 2026

Uh oh!

github-actions Bot commented Jul 15, 2026

Uh oh!

wForget commented Jul 16, 2026

Uh oh!

github-actions Bot commented Jul 17, 2026

Uh oh!

github-actions Bot commented Jul 17, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

Uh oh!

Conversation

wForget commented Jul 15, 2026 • edited by github-actions Bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What changes are proposed in this pull request?

How was this patch tested?

Was this patch authored or co-authored using generative AI tooling?

Uh oh!

github-actions Bot commented Jul 15, 2026

Uh oh!

wForget commented Jul 15, 2026

Uh oh!

github-actions Bot commented Jul 15, 2026

Uh oh!

github-actions Bot commented Jul 15, 2026

Uh oh!

github-actions Bot commented Jul 15, 2026

Uh oh!

github-actions Bot commented Jul 15, 2026

Uh oh!

wForget commented Jul 16, 2026

Uh oh!

github-actions Bot commented Jul 17, 2026

Uh oh!

github-actions Bot commented Jul 17, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

wForget commented Jul 15, 2026 •

edited by github-actions Bot

Loading