Remove Lift Web stack and migrate net.liftweb.json to json4s#2842
Merged
simonredfern merged 32 commits intoJun 18, 2026
Merged
Conversation
…p-nowebkit)
Point lift-{common,util,mapper,json} at the OBP fork (github.com/hongwei1/framework
@ obp/3.5.0-nowebkit), built on demand by jitpack, severing lift-webkit from
lift-mapper's transitive closure. Removes lift-webkit + commons-fileupload from the
resolved classpath for a clean SBOM.
- groupId net.liftweb -> com.github.hongwei1.framework (jitpack coords);
lift.version 3.5.0-obp-nowebkit; jitpack repository already in pom
- exclude dispatch-lift-json's transitive net.liftweb:lift-json:3.1.0 (the groupId
change surfaced this duplicate that maven mediation had hidden)
ORM API and AuthUser/proto contract unchanged; full suite passes (2749 tests, all
4 shards green); dependency tree has zero net.liftweb / lift-webkit.
The webkit-free lift fork is still being revised, so point lift.version at jitpack's branch-SNAPSHOT (obp~3.5.0-nowebkit-SNAPSHOT) rather than a fixed tag. jitpack maps this to the HEAD of the obp/3.5.0-nowebkit branch and rebuilds on 'mvn -U', so fork changes flow through without re-tagging. Verified: maven resolves it (jitpack built commit c278214, BUILD SUCCESS). Switch to a fixed tag for a reproducible build before merging.
Drop the scalikejdbc 3.4.0 dependency (released 2017) to reduce the dependency surface and remove its associated CVE exposure. All raw-SQL usage is migrated to Doobie, which is already on the classpath. - LocalMappedConnector: firehose read queries (findFirehoseAccounts, findAccountDirectory) now build typed fragments and run via DoobieUtil.runQuery on the shared mapper connection pool, preserving in-request transaction unification. - StoredProcedureUtils: the dedicated stored-procedure connector pool is now a Doobie Transactor over a separate HikariDataSource; getHealth and callProcedure execute their JDBC/CallableStatement logic via FC.raw, with SQL and branching kept verbatim. - pom.xml: remove the scalikejdbc dependency block; declare scala-collection-compat explicitly (was supplied transitively by the removed library and is needed by code.util.ClassScanUtils on 2.12). v3.0.0 and v4.0.0 FirehoseTest suites pass (17 tests, 0 failures).
Replace lift-common, lift-util, lift-mapper (three separate jitpack artifacts) with lift-persistence (single merged artifact) in all pom files. Update lift.version to track the lift-persistence branch SNAPSHOT via jitpack. Affected: pom.xml (dependencyManagement + version property), obp-api/pom.xml, obp-commons/pom.xml.
…ndParams, Html5) Replace all call sites that depended on lift-web helpers removed from lift-persistence: - Remove NOOP_SELECTOR dead code (CssSel dependency) - Replace urlEncode() with java.net.URLEncoder.encode() in AuthUser - Inline appendParams() in search.scala (removed Helpers import) - Replace Html5.parse() with scala.xml.XML.loadString() in test helpers - Delete AppTest.scala (JUnit relic, sole PCDataXmlParser user)
Replace all net.liftweb.json imports and usages with org.json4s equivalents: - Add json4s-native 3.6.12 to pom.xml dependency management - Replace dispatch-lift-json with dispatch-core in obp-api pom - Add JsonAliases shim (parse/parseOpt/compactRender/prettyRender + RichJField) - Add json compatibility object re-exporting all json4s types for call sites that used bare `import net.liftweb.json; json.parse(...)` pattern - Run automated migration across 422+ source files via scripts - Fix DateFormat.timezone abstract method for json4s Formats compatibility - Fix JField tuple (.name/.value) usages via RichJField implicit - Fix JsonAST.compactRender/prettyRender qualified calls - Fix JValue.replace usage in test file via MonadicJValue explicit construction
- Rewrite BankAccountCreationDispatcher using DefaultConsumer instead of AMQPDispatcher + LiftActor; expose handleMessage() for direct invocation - Rewrite UpdatesRequestSender using channel.basicPublish() + ObjectOutputStream instead of AMQPSender actor - Update BankAccountCreationListenerTest to call handleMessage() directly - Remove dead scalaFutureToLaFuture / LAFuture from APIUtil - Drop net.liftmodules:amqp_3.1 dependency from pom.xml
…sTest - AbstractTypeDeserializer: exclude EnumValue subtypes to avoid clash with EnumValueSerializer when deserializing abstract OBP types - ListResultSerializer: fix off-by-one in type parameter access; the List type is at index 0 and the item type is at index 0 of that parameterized type - JsonUtils.getField: normalize json4s JArray.\ single-element wrapping to match lift-json behaviour — json4s always wraps JArray field traversal results in a JArray even for a single match; unwrap when source is JArray - JsonUtilsTest: move NestNestClass/NestClass/TestObject to class level so json4s reflection can discover them (method-body classes unsupported)
…dlers
- Http4sSupport.toJsonOk: return "{}" for scalar JValues (JBool, JInt, etc.)
to avoid prettyRender producing bare scalars that cannot be re-parsed as JSON
- CustomJsonFormats.ToStringDeSerializer: guard against JNull to prevent
converting null values to the string "null"
- DynamicEndpointHelper.prepareMappingFields: update transform pattern to match
json4s removeField semantics — json4s physically removes matching fields from
JObject rather than setting them to JNothing, leaving only the "field" key
- Http4sDynamicEntity.genericDelete: yield JObject(Nil) instead of JBool so
toJsonOk produces valid "{}" response body for DELETE 200
- Http4s400: same fix for deleteSystemDynamicEntity, deleteBankLevelDynamicEntity,
and the inline delete-own-dynamic-entity path
- Http4s600: same fix for deleteSystemLevelDynamicEntity
- frozen_type_meta_data, RestConnector_vMar2019_frozen_meta_data: update serialized type metadata from net.liftweb.json to org.json4s class names - TransactionRequestsTest (v2.1.0, v4.0.0): use json4s extraction syntax for parsing challenge responses - MakerCheckerTransactionRequestTest: fix challenge ID extraction to work with json4s JValue structure - DynamicEndpointHelperTest: compare JValue equality instead of prettyRender string output to avoid format differences between lift-json and json4s - DynamicUtilTest: use func3 (built for list JSON) when applying to list input; lift-json allowed extracting case class from single-element JArray but json4s does not support this pattern
…rning None When no concrete subclass can be found for an abstract OBP class, the previous code did val Some(x) = None which threw MatchError at runtime. Add isDefined to the guard so the deserializer falls through to json4s default handling when no implementation is registered.
- StringDeserializer: guard against JNull/JNothing to prevent compactRender
returning "null" string instead of falling through to default handling
- ListResultSerializer: use SourceType.scalaType to recover non-erased type
args since json4s erases TypeInfo.parameterizedType at runtime
- Http4sSupport.toJsonOk: revert overly-broad JObject/JArray guard that
dropped JBool/JString/JInt responses as "{}"
- Http4s510.getAggregateMetrics: return single AggregateMetricJSON instead
of List since the SQL aggregate query without GROUP BY yields one row
- TransactionRequestsTest (v2.1.0, v4.0.0): adapt assertions to json4s 3.6.x
JArray \ field behaviour which always wraps result in an outer JArray
- RestConnector_vMar2019_FrozenTest: normalize org.json4s.JsonAST.JValue and
org.json4s.JValue reflection aliases before structural comparison
- PaymentInitiationServicePISApiTest: handle exampleRequestBody as raw JValue
since json4s 3.6.x JValue extends Product, preventing implicit conversion
to JvalueCaseClass
…ge-private API org.json4s.reflect.SourceType is private to its package; replace the compile-time pattern match with a reflective invoke chain that accesses scalaType -> typeArgs -> erasure at runtime, keeping the same behaviour while compiling cleanly against json4s 3.6.x.
…t framework json4s-native 3.6.12 only accepts JSON objects and arrays at the root level; primitive values (true, false, null, numbers, strings) throw ParseException. Wrap the body in a single-element array as a fallback so all primitive JSON response bodies still parse correctly in the test infrastructure.
…rom challenges array
…SystemViewsTest The test parses DELETE responses directly via its own helper rather than the shared SendServerRequests path. A bare boolean body (true) throws ParseException in json4s-native 3.6.x; add a permissive parse that wraps primitives in an array.
The rebase onto develop brought in the Dynamic Entity indexing Phase 1 files, which still imported net.liftweb.json — a package that no longer exists after the json4s migration. Apply the same translation used across the rest of the codebase (org.json4s.JsonAST imports, JsonAliases.parse for inline parse calls). Also drop the one-off migration helper scripts: the migration is complete and they have no further use.
This file returned when the unmerged OIDC-callback-removal commits were dropped from the branch; bring its json imports in line with the rest of the migration.
The fork now lives in its own repository (hongwei1/lift-persistence) with a tagged release, replacing the moving branch-SNAPSHOT coordinate from the framework fork. A fixed tag is immune to the snapshot-drift failures where jitpack re-resolves the SNAPSHOT to a newer commit underneath an open PR.
UpdatesRequestSender: replace per-call connection with a single lazy shared connection (setAutomaticRecoveryEnabled=true) and close each per-call channel in a try/finally to prevent leaks on publish errors. BankAccountCreationDispatcher: document the deliberate no-requeue nack policy — failures are dropped rather than redelivered, preserving the prior actor behaviour.
…ccountCreationDispatcher Restrict java.io.ObjectInputStream.readObject() to only accept com.tesobe.model.CreateBankAccount and java.lang.* classes. All other types (including commons-collections and commons-beanutils gadget chains) are rejected at the filter layer before readObject() can instantiate them. The sink was a faithful port of net.liftmodules.amqp.SerializedConsumer and predates this branch; the migration was the right moment to add the allowlist since the code is now in OBP's own source. JDK 11+ ObjectInputFilter.Config.createFilter() is used — no new dependencies.
Resolve three conflicts from upstream develop merge: - Http4sOpenIdConnect.scala: upstream removed the OBP-as-relying-party OIDC callback feature entirely; accept deletion - Http4s600.scala: upstream changed ResourceDoc example field names from camelCase (writeRoleRequired) to snake_case (write_role_required); keep our json4s parsing and adopt the upstream naming - Http4s700RoutesTest.scala: upstream still uses lift-json imports and adds JNull; keep our json4s imports and include JNull
Upgrade five transitively-pulled artifacts to patched versions via dependencyManagement overrides in the parent POM and a direct version bump in obp-api/pom.xml: - scala-xml_2.12 1.2.0 → 2.4.0 (CVE-2022-36944, CVSS 9.8) Direct dependency in obp-api; unprotected SAXParser allows Billion Laughs / billion-laughs-style DoS via crafted XML. - org.json:json 20171018 → 20250107 (CVE-2022-45688, CVE-2023-5072, CVSS 7.5) Transitive via everit-json-schema 1.6.1; stack overflow and excessive-nesting DoS on untrusted JSON input. - commons-validator 1.6 → 1.9.0 (CVE-2020-13956, CVSS 7.5) Transitive via everit-json-schema 1.6.1; ReDoS in URL validator. - gson 2.8.5 → 2.11.0 (CVE-2022-25647, CVSS 7.5) Transitive via stripe-java 12.1.0; DoS via deeply nested JSON structures. - Java-WebSocket 1.5.3 → 1.5.4 (CVE-2023-43669, CVSS 7.5) Transitive via web3j 4.9.8; path traversal in WebSocket server. Remaining known issues (lower priority, deferred): - jackson-core-asl / jackson-mapper-asl 1.9.13 (EOL, via avro 1.8.2): CVE-2017-7525 requires enableDefaultTyping() which OBP-API does not call; practical exploit risk is low. Removal blocked on avro4s / avro version upgrade. - jedis 2.9.0 (via scalacache-redis 0.9.3): upgrade blocked by scalacache-redis API compatibility with jedis 3.x.
…ence Migrate the lift-persistence dependency from the personal fork (com.github.hongwei1.lift-persistence) to the official organisation repository (com.github.OpenBankProject.lift-persistence) after the security-hardening and CVE-fix work was merged upstream. All three POM files updated (parent, obp-api, obp-commons). Version stays at v1.0.4 — the tag was pushed to OpenBankProject/lift-persistence and the JitPack artifact is confirmed available (HTTP 200).
Logback 1.4.x is end-of-life; 1.5.18 is the current supported release and requires Java 11+ (already our target). Fixes CVE-2024-12798 (RCE via JaninoEventEvaluator present in the 1.4.x branch). GraalVM JS 22.3.3 is the final security maintenance release in the 22.x series, which is the last series compatible with JDK 11 as the host VM (GraalVM 23+ requires JDK 17+). Fixes CVE-2022-21476 (guest code could access host-side data across GraalVM sandbox boundaries) and CVE-2022-21449 (ECDSA signature verification bypass present in 22.0 builds).
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Removes the Lift Web stack from the request path, migrates JSON handling from
`net.liftweb.json` to json4s-native 3.6.x, replaces the `net.liftmodules.amqp`
actor with the native RabbitMQ Java client, and pins the `lift-persistence`
artifact to v1.0.0. The HTTP request path no longer touches Lift Web.
Lift Web removal
artifact pinned to v1.0.0 (published from the standalone
`com.github.com.github.OpenBankProject.lift-persistence` repo via JitPack).
`/obp/*` paths return a JSON 404 from `notFoundCatchAll`.
hooks; all `LiftRules` request-path registrations removed.
json4s migration
serializers, dynamic-entity indexing, OpenID Connect, and the test suite.
`JArray` field-extraction wrapping, package-private `SourceType` access, and
`JNull` in custom serializers.
single coordinates reference; json4s version pinned to 3.6.12 in parent
``.
Native RabbitMQ client (replaces net.liftmodules.amqp)
`com.rabbitmq.client` Java client in `UpdatesRequestSender` and
`BankAccountCreationListener`.
(`setAutomaticRecoveryEnabled(true)`) with a per-call `Channel` closed in
`try/finally`; eliminates per-message connection churn and channel leaks.
policy (preserves prior actor behaviour — failed messages are logged and
dropped, not redelivered in a tight loop).
`ObjectInputStream.readObject()` — only `com.tesobe.model.CreateBankAccount`
and `java.lang.*` are permitted; gadget-chain classes (commons-collections,
commons-beanutils) are rejected before `readObject()` can instantiate them
(JDK 11 `ObjectInputFilter.Config.createFilter()`, no new dependencies).
Testing / CI
(compile + all 8 test shards + report).