feat(sparkeks): add JAR/--class entrypoint support (chipmunk EMR->EKS migration, UoW-F3)#126
Open
prasadlohakpure wants to merge 6 commits into
Open
feat(sparkeks): add JAR/--class entrypoint support (chipmunk EMR->EKS migration, UoW-F3)#126prasadlohakpure wants to merge 6 commits into
prasadlohakpure wants to merge 6 commits into
Conversation
… migration UoW-F3) Enables the sparkeks plugin to run a custom-class (Scala/Java) Spark application via new optional jobParameters.EntryPoint + jobContext.Arguments. JAR mode sets Type:Scala/MainClass and builds args [appName, user, query, ...arguments, (resultURI)] server-side, matching com.pattern.chipmunk.Writer. SQL-wrapper path unchanged. Defensive default Spec.Type=Python guards templateless jobs. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Contributor
There was a problem hiding this comment.
Pull request overview
Adds opt-in JAR/--class entrypoint support to the sparkeks plugin so it can launch JVM Spark applications (e.g., Chipmunk Writer) in addition to the existing SQL-wrapper path, with a defensive fallback to ensure SparkApplication.Spec.Type is never submitted empty.
Changes:
- Introduces
entry_pointandargumentsjob-context fields to enable JAR-mode SparkApplication submission. - Updates SparkApplication spec generation to support a JAR-mode argument vector and to default
Spec.Typewhen not set by templates. - Adds unit tests covering JAR mode, JAR+return_result, SQL-wrapper non-regression, and the default-Type fallback.
Reviewed changes
Copilot reviewed 2 out of 2 changed files in this pull request and generated 2 comments.
| File | Description |
|---|---|
| internal/pkg/object/command/sparkeks/sparkeks.go | Adds JAR-mode (--class) SparkApplication configuration and a defensive default for Spec.Type. |
| internal/pkg/object/command/sparkeks/sparkeks_test.go | Adds tests validating the new JAR-mode behavior and guarding existing SQL-wrapper behavior. |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
| // - SQL-wrapper (Parameters.EntryPoint unset — existing behavior, byte-for-byte | ||
| // unchanged below): wrapper_uri points at a .py wrapper that reads the query from | ||
| // queryURI. | ||
| if execCtx.commandContext.WrapperURI != "" { |
| sparkApp.Spec.Arguments = []string{execCtx.appName, s3aQueryURI, execCtx.job.User, s3aResultURI} | ||
| sparkApp.Spec.MainApplicationFile = &mainAppFile | ||
|
|
||
| if jobContext.Parameters != nil && jobContext.Parameters.EntryPoint != "" { |
Extract entrypoint-specific spec (Type/MainClass/Arguments) into an entrypointStrategy interface with jarEntrypointStrategy and sqlWrapperEntrypointStrategy, selected by newEntrypointStrategy. Replaces the inline if/else in applySparkOperatorConfig. Behavior unchanged — all 4 unit tests still pass. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
…dded tests Move entrypointStrategy + jar/sqlWrapper strategies + newEntrypointStrategy and jarApplicationType into internal/pkg/object/command/sparkeks/entrypoint.go. sparkeks.go keeps applySparkOperatorConfig (delegates to newEntrypointStrategy) and the Gap-4 default. Revert the added sparkeks_test.go unit tests per request. Build + vet clean. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
…; multi JAR type - Select entrypoint strategy from wrapper_uri extension via a map (entrypointStrategiesByExt: .jar -> JAR, .py -> SQL wrapper), default SQL wrapper. - Generalize JAR application type: jarApplicationTypes map + resolveJarApplicationType, configurable via new jobParameters.ApplicationType (Scala/Java), default Scala. - Behavior-equivalent to the previous EntryPoint-based selection for all real commands; more extensible. Build + vet clean. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
monitorJobAndCollectLogs wrote the real error to stderr for ApplicationStateFailed, but the FailedSubmission/Unknown branch returned only a generic constant and never wrote anything to e.runtime.Stderr, silently dropping sparkApp.Status.AppState.ErrorMessage.
applySparkOperatorConfig already normalized WrapperURI, EventLogURI, queryURI, and resultURI via updateS3ToS3aURI, but merged arbitrary job/cluster properties into SparkConf verbatim. Any property value referencing a raw s3:// path (e.g. spark.kubernetes.driver.podTemplateFile) hit Hadoop's UnsupportedFileSystemException, since only s3a:// is registered without extra config.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
Adds JAR/
--classentrypoint support to the Heimdallsparkeksplugin so it can run a custom-class(Java/Scala) Spark application, not just the built-in SQL wrapper. Additive and opt-in — existing
sparkekscommands are byte-for-byte unchanged.Changes in
internal/pkg/object/command/sparkeks/sparkeks.go:jobParameters.EntryPoint(entry_point) andjobContext.Arguments(
arguments).applySparkOperatorConfig, whenParameters.EntryPointis set → JAR mode: setsSpec.Type = Scala,Spec.MainClass = EntryPoint,MainApplicationFile = wrapper_uri(as a jar),and builds
Spec.Arguments = [appName, user, query] + arguments + (resultURI if return_result).When unset → the existing SQL-wrapper path, unchanged.
Spec.TypetoPythonif still empty after config (the CRD field isrequired/enum-validated; a templateless job would otherwise submit an invalid empty
Type).Why
Foundation step (UoW-F3) of migrating Chipmunk's collection-build Spark jobs off EMR-on-EKS
onto Spark-on-EKS (
sparkeks). The chipmunk collection writer is a Scala JAR(
com.pattern.chipmunk.Writer); todaysparkekscan only launch the SQL wrapper, so it cannot runthat JAR. This change unlocks the
spark-chipmunk-ekscommand + operator wiring (a later PR).Arg order matters: the JAR-mode vector is
[appName, user, query, s3Target, (resultURI)], prependedserver-side — matching
Writer.scala's contract — deliberately NOT the SQL-wrapper order[appName, queryURI, user, resultURI].Migration roadmap (where this PR fits)
Foundation phase; this is step 3 (steps 1–3 are independent, none consumed until step 4 wires them):
snowflake-keysecret on sparkeks driver--classsupport in sparkeks plugin (THIS PR)spark-chipmunk-ekscommand +ChipmunkCollection(use_eks=)operator (depends on 1–3)Then per-pipeline cutovers (each keeps the EMR path runnable + validated before removal), then EMR
decommission.
Test plan
go build ./internal/...— clean.go vet ./internal/pkg/object/command/sparkeks/...— clean.go test ./internal/pkg/object/command/sparkeks/...— all pass, including 4 new tests:JAR mode (Type/MainClass/MainApplicationFile/4-elem args), JAR+return_result (5-elem, result URI
last), SQL-wrapper non-regression (both return_result states), and the Gap-4 default-Type fallback.
Notes / rollback
EntryPoint).go build ./...across the whole repo reports "function main is undeclared"for every
plugins/*package — these are-buildmode=pluginpackages with nofunc main()and failthe same way on an untouched
origin/maincheckout. Not introduced by this PR.🤖 Generated with Claude Code