UFAL/CLARIN-DSpace upgrade v9#1339
Open
milanmajchrak wants to merge 84 commits into
Open
Conversation
|
Important Review skippedAuto reviews are disabled on base/target branches other than the default branch. Please check the settings in the CodeRabbit UI or the ⚙️ Run configurationConfiguration used: Organization UI Review profile: CHILL Plan: Pro Run ID: You can disable this status message by setting the Use the checkbox below for a quick retry:
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
…vices) First foundation tranche of the CLARIN-DSpace 7.6.5 -> 9.3 forward-port. What's included (compiles, checkstyle + license headers clean, entity/schema validated on h2 via hbm2ddl=validate, migrations applied at DB init): - 19 CLARIN Flyway migrations (h2 x9, postgres x10) at original version numbers (Lindat schema, preview/report/matomo/clarin_token tables, share token, default licenses, 7z format). Purely additive. - ~107 CLARIN dspace-api classes: license framework, user metadata/registration, verification + clarin tokens, item/workspace services, handle service + external handle, shibboleth headers, featured services, provenance, EPIC/PID base, ORCID caching, factories + DAOs. - v9 API adaptation of ported code: javax.* -> jakarta.*, commons-lang -> lang3, NullArgumentException -> IllegalArgumentException, Hibernate 6 ORDINAL enum JdbcTypeCode fix (ClarinLicense.confirmation). - 9 CLARIN entity <mapping> entries in hibernate.cfg.xml. - dspace-api/pom.xml deps: matomo-java-tracker, nimbus-jose-jwt, itextpdf, jfree, zjsonpatch. - Minimal additions to vanilla files: Handle (url/dead/deadSince), Util.formatNetId, HandlePlugin.getRepositoryName/getCanonicalHandlePrefix. Deferred to later tranches (tracked in CLARIN_DSPACE_V9_PROGRESS.md): S3/sync storage (AWS SDK v1->v2), preview, report/health/diff, matomo runtime, PID/EPIC clients, versioning CLI, Spring bean wiring, REST layer, frontend. See CLARIN_DSPACE_V9_PROGRESS.md for full inventory, status and testing evidence. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Tranche 2 of the CLARIN-DSpace 7.6.5 -> 9.3 backend port. Registers the data-layer
services/DAOs/factories ported in the previous commit so they are instantiated by
the DSpace Spring context.
- core-factory-services.xml: clarinServiceFactory, handleClarinServiceFactory
- core-dao-services.xml: 11 CLARIN DAO beans (license/label/mapping/user-registration/
user-metadata/allowance/item/verification-token/matomo-report/token + HandleClarinDAOImpl)
- core-services.xml: 16 service beans (license framework, user metadata/registration,
verification + clarin tokens, item/workspace, matomo report subscription,
DspaceObjectClarin, AuthorizationBitstreamUtils, HandleClarinServiceImpl,
EpicHandleServiceImpl, ProvenanceServiceImpl) + MatomoTracker wired to the existing
v9 ${matomo.tracker.url} (factory requires it via @Autowired).
Deferred-feature beans (bitstream-sync/preview/report/matomo-runtime) intentionally
omitted until those classes are ported (tracked in CLARIN_DSPACE_V9_PROGRESS.md).
Validated: full Spring context boots cleanly (AccessStatusServiceTest 3/3, no
autowiring/bean-creation errors) with all CLARIN beans active.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
…egy) Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
…ported items From the independent BE review: - M2: handle/PIDService.java reflectively loads the deferred PIDServiceEPICv2 (Class.forName) -> ClassNotFoundException at runtime for its only path. Deferred it to _deferred/ (the live EPIC path is EpicHandleServiceImpl). dspace-api still compiles (EXIT 0). - HONESTY (progress file §6g + §0): the BE port is a COMPILE-ONLY data/service skeleton, not a runtime-functional CLARIN backend. Documented all previously-untracked silently-not-ported items distinct from _deferred/: 24 CLARIN config files, 57 config modifications, 91 vanilla-.java modifications (only Handle/Util/HandlePlugin applied), 44 CLARIN test classes, 3 entity-mapping gaps (WorkspaceItem.shareToken, EPerson.welcomeInfo/canEditSubmissionMetadata, Item.isHidden), REST/OAI layer. Downgraded §0 "runtime-active" claim. Added §6f Docker/Playwright startup + blockers. Nothing is skipped without a documented reason. See CLARIN_DSPACE_V9_PROGRESS.md §6f/§6g. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
BE tranche 3 — closes documented gaps from the independent review: - 24 ADDED CLARIN config files: clarin-dspace.cfg, OAI crosswalks (lindat_cmdi/olac/ metasharev2/elg/datacite_openaire/bibtex .xsl + descriptions), email templates (clarin_token, clarin_download_link(+admin), clarin_autoregistration, share_submission, matomo_report, report_diff), registries (datacite.xml, edm.xml, metashare-schema.xml), submission-forms_cs.xml, default_cs.license, features/enable-orcid.cfg, spiders, VERSION_D. - Entity columns matching migrations (were unmapped → features inert): WorkspaceItem.shareToken (share-submission), EPerson.welcomeInfo + canEditSubmissionMetadata. dspace-api compiles (EXIT 0). Still TODO (documented §6g): 57 MODIFIED config files (dspace.cfg include of clarin-dspace.cfg, item-submission.xml, authentication-shibboleth.cfg, discovery.xml), remaining vanilla-file method additions + their service wiring, REST/OAI layer, CLARIN tests. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
BE tranche 4 — first slice of the REST layer (the #1 functional blocker). Exposes the /server/api/core/clarinlicenses, clarinlicenselabels, clarinlicenseresourcemappings, clarinlruallowances endpoints the FE clarin-licenses module already consumes. - 17 files: Rest models (ClarinLicense/Label/ResourceMapping/ResourceUserAllowance) + hateoas Resources + Converters + RestRepositories + ClarinLicenseNotFoundException. - v9 adaptation: implement new RestModel.getTypePlural() (returns NAME+"s", matching the FE link names clarinlicenses/clarinlicenselabels/...); javax->jakarta; import order. - Compiles (dspace-server-webapp EXIT 0) + checkstyle clean. Wires to the already-ported dspace-api ClarinLicense* services/DAOs (tranches 1-2). Remaining REST (documented §6g): handle/epic-handle, user-metadata/registration, token, import controllers, submission steps, OAI/CMDI. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
BE tranche 5 — more of the REST layer. Exposes endpoints for handle, epic-handle, user-metadata, user-registration, verification-token, featured-service + their link repositories (CLRUA<->user-metadata/registration, user-registration<->license, resource-mapping<->license). - 31 files: Rest models + hateoas Resources + Converters + RestRepositories + LinkRepositories. - v9 adaptations: RestModel.getTypePlural() on all new models; javax->jakarta; import order. - Compiles (dspace-server-webapp EXIT 0) + checkstyle clean. - Deferred: ExternalHandleRestRepository (magic-link external handle; needs a RandomStringGenerator bean + DCDate.getCurrent() + external-handle flow not yet ported) -> _deferred/. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
…e validation BE tranche 6 — RUNTIME-VALIDATED bug fix found by actually booting the server. v9's REST framework resolves repositories via getBean(category + "." + modelPlural) (Utils.getResourceRepositoryByCategoryAndModel uses the plural URL segment). CLARIN repos were registered with the 7.x singular pattern @component(CATEGORY + "." + NAME), so EVERY CLARIN endpoint returned 404 even though the classes compiled and CI was green. Fix: add PLURAL_NAME = NAME + "s" to 8 Rest models (ClarinLicense/Label/ResourceMapping/ ResourceUserAllowance, ClarinUserMetadata/UserRegistration/VerificationToken, Handle) and switch the 8 RestRepositories' @component to PLURAL_NAME (matches the FE data-service link names and getTypePlural()). compile + checkstyle clean. Validated on a running v9.3 server against real Postgres (full mvn package -> ant fresh_install -> dspace database migrate [all CLARIN migrations apply on postgres] -> server-boot.jar): /server/api/core/clarinlicenses|clarinlicenselabels|handles -> 200 (valid HAL) /server/api/core/clarinlruallowances|clarinusermetadatas -> 401 (correctly auth-protected) This is exactly the "compiles != works" gap the independent review warned about. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
…ful to 7.x) Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
…— runtime-diagnosed BE tranche 7 — second runtime-diagnosed bug, found by exercising the write path on a live server. v9's ConverterService.toRest() enforces access on every BaseObjectRest by reading the @PreAuthorize SpEL from the resource repository's most-derived findOne() (getPreAuthorizeAnnotationForBaseObject -> getAnnotationForRestObject). The base DSpaceRestRepository.findOne is abstract with no annotation, and the CLARIN repos' findOne overrides had none (faithful to 7.x, where ConverterService did NOT do this check). Result: converting ANY real CLARIN object threw "IllegalArgumentException: 'expressionString' must not be null or blank" -> 400/500. The empty GETs passed earlier only because there was no row to convert; the bug surfaced on POST (create) and would hit every non-empty response. Fix (v9-migration necessity, mirrors vanilla repos like CommunityRestRepository.findOne): add @PreAuthorize on findOne of the 8 CLARIN BaseObjectRest repositories, with the access level matching each endpoint's observed findAll semantics: permitAll() -> ClarinLicense, ClarinLicenseLabel, ClarinLicenseResourceMapping, Handle hasAuthority('ADMIN') -> ClarinLicenseResourceUserAllowance, ClarinUserMetadata, ClarinUserRegistration, ClarinVerificationToken compile + checkstyle clean. Read endpoints already proven 200/401 on a live v9 server against Postgres (tranche 6); this unblocks object conversion / the create+read round trip. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
…n blocked by host RAM Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
…featured REST URGENT FIX: tranche 7 (98771fe) pushed broken code — CI red. Root cause: 7 of the 8 CLARIN repos ALREADY had @PreAuthorize on findOne in 7.x (only ClarinLicenseLabel lacked it, which was the real ConverterService gap). My tranche-7 script's fallback regex appended a SECOND @PreAuthorize to those 7 -> "PreAuthorize is not a repeatable annotation type" (a stale incremental compile masked it locally; lesson: verify with `clean compile`). Fix: drop the duplicate, restore each repo's ORIGINAL findOne annotation (permitAll() for license/label/mapping/handle/verificationtoken; hasAuthority('AUTHENTICATED') for resourceUserAllowance/userMetadata/userRegistration). ClarinLicenseLabel keeps the single permitAll() I added (it genuinely had none — the actual fix that unblocks conversion). Also completes the handle feature REST (deps now available): - RandomStringGenerator + RandomStringGeneratorImpl - ExternalHandleRestRepository (un-deferred; DCDate(new Date()) -> DCDate.getCurrent() for v9) - EpicHandleRestController, ClarinFeaturedServiceRestRepository javax->jakarta, import order. clean compile + checkstyle BUILD SUCCESS. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
…step REST (DSpace 9) BE tranche 8 — 26 more REST files, clean-compile + checkstyle verified (used `clean compile` this time after the tranche-7 stale-incremental lesson): - ConfigFile REST (controller/converter/model/hateoas + 3 exceptions) — runtime config file access - authorization: CanManageLicense feature + ClarinAuthorizationTestController - ClarinAutoRegistrationController, ClarinUserInfoController - ClarinLicenseImportController, ClarinHandleImportController (bulk import) - submission steps: ClarinLicenseDistributionStep/ResourceStep/SubmissionUtils/NoticeStep + 2 validations - model: refbox DTOs (RefBox/FeaturedService/FeaturedServiceLink/ExportFormat), ClarinDataLicense, ShareSubmissionLinkDTO; utils/BigMultipartFile - javax.mail -> jakarta.mail, javax.* -> jakarta.*, import order. No deferred-service refs; beans wire. DEFERRED (need prerequisite work, moved to _deferred/): 14 files that require unported vanilla-file methods (Item.isHidden, Util.replaceLast/normalizeDiscoverQuery, WorkspaceItem share helpers) and/or extra libs not in pom (org.json.simple, com.hp.hpl.jena RDF): ClarinUserMetadataRestController, ClarinItemImportController, ClarinEPersonImportController, ClarinUserMetadataImportController, ClarinGroupRestController, SubmissionController, SuggestionRestController, ClarinRefBoxController, ClarinShibbolethLoginFilter, SolrOAIReindexer, AuthrnRest/AuthrnResource/AuthorizationRestController, DBConnectionStatisticsController. To be ported in a later dependency-complete tranche. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
…yShareToken BE tranche 9 — first vanilla-file method additions (additive, faithful to dtq-dev): - Item.isHidden(): true when metadata local.hidden == "hidden"; isDiscoverable() now returns `discoverable && !isHidden()` (hidden items are non-discoverable). Used by REST item exposure. - WorkspaceItemService.findByShareToken(Context, String) + Impl + DAO + DAOImpl (HQL on ws.shareToken, matching the share_token column/entity field) — backs the share-submission feature. clean compile + checkstyle on dspace-api (EXIT 0). Unblocks some of the tranche-8 deferred controllers. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
…ds) + clean-compile lesson Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
…ce 9)
BE tranche 10 — verified with combined reactor `clean compile` (api+webapp) BUILD SUCCESS + checkstyle.
vanilla-file Utils additions (only the methods v9 9.3 genuinely LACKS — v9 already had maskEmail,
getAllowedTemplateConfig, getSecureVelocityProperties, getMaxTimestamp, DEFAULT_ALLOWED_TEMPLATE_CONFIGS,
so the full 7.6.5 delta would duplicate them):
- core.Utils: replaceLast, getTransactionPid (Hibernate Session/SessionFactory/NativeQuery),
fetchUUIDFromUrl (+ UUID_PATTERN field, java.net.URI).
- rest.utils.Utils: normalizeDiscoverQuery (+ private helpers extractNumber/CharacterListFromString,
composeQueryWithNumbersAndChars, addQueryTemplateToList), encodeNonAsciiCharacters,
disableCertificateValidation, distinctByKey, getCanonicalHandleUrlNoProtocol (+ regex/function/
concurrent/ssl/security/charset imports v9 lacked).
Un-deferred (now compile): AuthorizationRestController + AuthrnRest (added v9 getTypePlural/PLURAL_NAME)
+ AuthrnResource, ClarinUserMetadataRestController, ClarinUserMetadataImportController,
SubmissionController, SuggestionRestController. Added json-simple 1.1.1 to webapp pom.
STILL DEFERRED (deep v9-migration, _deferred/): ClarinRefBoxController (ancient com.hp.hpl.jena RDF),
SolrOAIReindexer + Clarin{Item,EPerson}ImportController (Date->Instant API), ClarinShibbolethLoginFilter
(StatelessLoginFilter constructor signature changed), ClarinGroupRestController (GroupRest.GROUPS),
DBConnectionStatisticsController (getHibernateStatistics).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
…s + json-simple) Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
… (DSpace 9)
BE tranche 11 — verified combined reactor clean compile (api+webapp) + checkstyle.
- ClarinGroupRestController: GroupRest.GROUPS -> GroupRest.PLURAL_NAME (v9 renamed the constant).
- DBConnectionStatisticsController: needs Context.getHibernateStatistics -> added that vanilla method
(delegates to HibernateDBConnection) + HibernateDBConnection.getHibernateStatistics (Hibernate
Statistics: open/closed sessions, transactions, connections) + org.hibernate.stat.Statistics import.
5 controllers still deferred (deep v9-migration): ClarinRefBoxController (com.hp.hpl.jena), SolrOAIReindexer
+ Clarin{Item,EPerson}ImportController (Date->Instant), ClarinShibbolethLoginFilter (StatelessLoginFilter
ctor signature).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
…ace 9 Date->Instant) BE tranche 12 — verified combined reactor clean compile (api+webapp) + checkstyle. - ClarinItemImportController: compiled once json-simple was on the classpath (tranche 10). - ClarinEPersonImportController: EPerson.setLastActive now takes Instant -> lastActive.toInstant(). - SolrOAIReindexer: v9 ResourcePolicy.getStartDate/getEndDate return LocalDate (was Date) -> isAfter(LocalDate.now()); getMostRecentModificationDate rewritten to Instant (policy LocalDate -> atStartOfDay(UTC).toInstant(); item.getLastModified() is Instant); SolrUtils date formatter now receives an Instant (TemporalAccessor). Imports: java.time.Instant/LocalDate/ZoneOffset, drop java.util.Date. 2 controllers still deferred: ClarinRefBoxController (ancient com.hp.hpl.jena RDF -> needs org.apache.jena modernization), ClarinShibbolethLoginFilter (v9 StatelessLoginFilter constructor signature changed). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
…tor) BE tranche 13 — v9 StatelessLoginFilter constructor added an httpMethod param (url, httpMethod, authManager, restAuthService). Pass HttpMethod.GET.name() in the super() call (shibboleth login is a GET callback, matching vanilla ShibbolethLoginFilter). Utils.replaceLast resolves against core.Utils (tranche 10). Verified combined reactor clean compile + checkstyle. Only 1 controller still deferred: ClarinRefBoxController (ancient com.hp.hpl.jena RDF API; needs porting to org.apache.jena which DSpace 9 ships). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
…ST layer complete BE tranche 14 — com.hp.hpl.jena.rdf.model.Model -> org.apache.jena.rdf.model.Model (DSpace 9 ships Apache Jena; the Model type is only used as a pass-through param, no Jena method calls). Verified combined reactor clean compile (api+webapp) + checkstyle. ALL CLARIN REST controllers are now ported (no remaining deferred REST). The ref-box controller serves the Item-view citation/featured-service box (BibTeX/export formats via OAI-PMH crosswalks). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
…nd-trip on live v9 server All CLARIN REST controllers ported (tranches 11-14) + full CRUD runtime-validated against Postgres. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Append the additive CLARIN-DSpace config block to v9 dspace.cfg (validated: server boots clean + serves CLARIN REST 200 with these present): - handle.canonical.prefix + handle.additional.prefixes (CLARIN PID prefixes 11858/11234/11372/...) - registry.metadata.load metashare-schema.xml/edm.xml/datacite.xml (CLARIN metadata schemas, ported in tranche 3) loaded on registry-load/fresh-install - webui.user.assumelogin, metadata.hide.local.submission.note, item.view.total.downloads.enabled - webui.supported.locales = en, cs (adds Czech) - submit.type-bind.field (edm.type binding) - config.admin.updateable.files = item-submission.xml (CLARIN ConfigFileRestController allowlist) Skipped webui.browse.index.5 (would clobber v9 browse numbering — needs a dedicated index slot). Remaining config (item-submission.xml CLARIN steps, authentication-shibboleth.cfg, discovery.xml, submission-forms.xml) tracked for a follow-up; runtime-only, not CI-gating. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Set the shibboleth header mappings to CLARIN/LINDAT's IdP attributes (matches dtq-dev): - netid-header = eppn,persistent-id; email-header = mail - firstname-header = givenName; lastname-header = sn - eperson.metadata = SHIB-telephone => eperson.phone, SHIB-cn => cn, SHIB-GIVENNAME => eperson.firstname, SHIB-SURNAME => eperson.lastname Completes config wiring for the ported BE ClarinShibbolethLoginFilter + FE login outcome pages. Config-only (no code/build impact). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
…HdlResolverRestControllerIT) The Run Integration Tests CI job was red because my CLARIN dspace.cfg additions changed defaults that vanilla ITs assert on: - LanguageSupportIT.checkDefaultLanguageAnonymousTest: webui.supported.locales=en,cs makes the default Content-Language "en,cs" (matches dtq-dev's updated assertion) - HdlResolverRestControllerIT.givenMappedPrefixWhenNoAdditionalPrefixesConfThenReturnsHandlePrefix: handle.additional.prefixes is now set in cfg, so the "no additional prefixes" scenario must clear it explicitly (and restore it in finally) to stay deterministic Remaining red ITs (ShibbolethLoginFilterIT/AuthenticationRestControllerIT/ResourcePolicyRestRepositoryIT) track the CLARIN auth-flow test rewrites and are handled separately. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
… defaults ShibbolethLoginFilterIT (15) + AuthenticationRestControllerIT (12 shib tests) were red with "401 expected REDIRECTION" / "200 but was 401". Root cause: my earlier commit set authentication-shibboleth.cfg email-header=mail / netid-header=eppn,persistent-id etc., but the vanilla ShibbolethLoginFilter (the one actually wired in WebSecurityConfiguration) + the ITs use the default SHIB-MAIL / SHIB-NETID / SHIB-GIVENNAME / SHIB-SURNAME attribute names. With email-header=mail the filter never found the simulated SHIB-MAIL attribute -> auth failed -> 401 instead of redirect/200. These IdP attribute header names are DEPLOYMENT config (CLARIN's IdP sends eppn/mail; they apply it via their runtime/local.cfg). Reverting the committed default to vanilla keeps the shib ITs green without disabling them (dtq-dev instead committed eppn/mail and commented the tests out). Also reverted the test file back to the vanilla v9 version (no need to disable assertions). KNOWN GAP (documented): ClarinShibbolethLoginFilter (verification-token + autoregistration flow) is ported but NOT yet wired into WebSecurityConfiguration (vanilla ShibbolethLoginFilter is active); wiring it + its REST/verification services is tracked as a remaining feature task. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
…epositoryIT 500)
ResourcePolicyRestRepositoryIT.deleteOneUnAuthenticatedTest returned 500 because my config sets
`metadata.hide.local.submission.note = submitter`, but vanilla MetadataExposureServiceImpl.init()
calls getBooleanProperty() on every metadata.hide.* key -> ConversionException on the non-boolean
value "submitter" -> any metadata serialization (DSpaceObjectConverter) 500s.
This is a CLARIN feature ("hidden metadata visible to the submitter"): port the dtq-dev
MetadataExposureServiceImpl + interface changes —
- init() now accepts the string value "submitter" (or a boolean) without throwing
- new isHidden(context, schema, element, qualifier, Item) overload: a field hidden as "submitter"
is shown to the item's submitter (submitterShouldSee), hidden otherwise
- CONFIG_PREFIX kept public static final (v9)
Validated: mvn -pl dspace-api compile -> BUILD SUCCESS.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
…me) + BE (5 IT classes); document shib-wiring gap Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
…ServiceImpl Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Ports the CLARIN "item-version-linker" DSpaceRunnable (links two items into a versioning relationship, the second becoming the next version of the first) + ItemVersionLinkerConfiguration + ItemVersionLinkerIT from dtq-dev, and registers the script in the api/rest scripts.xml (and the test configs) like the other CLARIN scripts. This was previously deferred because adding scripts overflowed the un-paginated findAllScriptsTest; that test now requests an explicit page size, so registering it is safe. v9 adaptations: VersioningService.createNewVersion takes java.time.Instant in v9 (new Date() -> Instant.now(), 3 sites in the script + 1 in the IT); VersioningService.deleteVersion was renamed to delete(Context, Version) in v9. All other VersioningService/VersionHistoryService/ItemService/ IdentifierService/ScriptConfiguration signatures verified unchanged. dspace-api compiles cleanly (IDE); the IT is dspace-api (embedded Solr) so it is verified by CI. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
This reverts commit 73e8e18.
… need real adaptation Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
…(production parity) The CLARIN home page renders most-used author/subject/language quick links and item boxes with owning community, authors and license labels. All of that was silently dead because two 7.x config deltas were never ported: - modules/rest.cfg: expose the CLARIN UI properties over /api/config/properties (dspace.ui.url, dspace.name, lr.help.mail, shibboleth headers, download.all.*, citace.pro.*, themed.by.*, ...). The UI awaits dspace.ui.url before loading those widgets, so the 404 blocked all of them. - spring/api/discovery.xml: the 'homepage' discovery configuration (with indexAlways=true so its facets are indexed for every item) + the CLARIN facet beans (searchFilterSubjectFirstValue, searchFilterRights, searchFilterLanguage, searchFilterItemsCommunity, searchFilterPublisher, sortTitleDesc, sortDateIssuedAsc) + the itemsOwningCommunityPlugin wiring. - iso_language facet type (TYPE_ISO_LANG): DiscoveryConfigurationParameters constant, the ItemIndexFactoryImpl indexing branch (resolves the full language name via IsoLangCodes) and the SolrServiceImpl facet-field transform; plus the lang_codes.txt resource IsoLangCodes reads (it silently returned no names without it). Verified on the local stack (dev-5 data): /api/config/properties/dspace.ui.url returns 200 and the homepage facets return the same values as production (author; subject: People/Galerie osobnosti/ Places; language: Czech/English/German; items_owning_community: LINDAT / CLARIAH-CZ...). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
- discovery.xml defaultConfiguration: the CLARIN facet set (author, subject, rights, language, type, has-files, entity type, owning community - matching production) and sort options (score, title asc/desc, date issued asc/desc); the CLARIN search filters added to the filter list - webui.browse.index.5 = language (local.language.name) - restores the 'By Language' browse option on community/collection pages - DiscoveryRestControllerIT: the two hardcoded facet-list assertions now use the config-generated defaultFacetMatchers, so they follow the configuration Verified locally: /api/discover/facets returns author, subject, rights, language, itemtype, has_content_in_original_bundle, entityType, items_owning_community - the production facet set. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…mcol, login) Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The parity commit (070bc57) replaced the defaultConfiguration facet set and sort options, which broke seven DiscoveryRestControllerIT tests: - the two minAndMaxTests assertions were wrongly switched to the generated defaultFacetMatchers; they query configuration=minAndMaxTests (unchanged test config), so the original hardcoded matchers are restored - the four dateIssued facet tests now pass configuration=default-relationships - the CLARIN default configuration no longer exposes the dateIssued facet (matching production), while default-relationships still does, so the date-facet machinery stays covered; the next/search link assertions are split into order-independent substrings because the configuration parameter changes the query-string layout - discoverSearchTest expects the CLARIN sort options (score desc, title asc/desc, date issued asc/desc - the same expectation the v7 fork used in its adapted ClarinDiscoveryRestControllerIT) All seven pass locally: Tests run: 7, Failures: 0, Errors: 0. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
… facets, stable license label order - discovery.xml: searchFilterSubjectFirstValue registered in the default configuration's searchFilters - filters are only indexed through the item's own configurations, so the homepage Subject facet was never populated (the v7 fork registers it the same way); the homepage configuration gets an explicit id (unnamed configurations after the first are skipped by the indexer's id-based dedup); the author facet includes dc.contributor.other (v7 parity); all default sidebar facets start collapsed (isOpenByDefault=false - the v7 backend never serialized openByDefault, so production renders every facet collapsed); the collection configuration defaults to sorting by accession date (the landing lists recent submissions like v7) - ClarinLicense#getLicenseLabels sorts the labels by id: they live in an unordered set with identity hash codes, so every REST response (and the license icons rendered from it) had a random order - reproduced on production itself, two orders for the same license minutes apart Verified locally after a full reindex: homepage subjectFirstValue returns the production value set (People, Galerie osobnosti, Places, ...). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…lues Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Review hardening: getLicenseLabels sorts by id with nullsLast so an unsaved label (null id) cannot break the REST serialization. Co-Authored-By: Claude Fable 5 <noreply@gmail.com> Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Ports from origin/dtq-dev, v9-adapted: MetadataBitstreamRestRepository (+wrapper/converter/Rest model/hateoas resource) with the v9 PLURAL_NAME bean-name fix so /api/core/metadatabitstreams/search/byHandle resolves, and BitstreamByHandleRestController (S3/matomo deps stripped, native UsageEvent kept). Fixes every item page rendering 'This item contains no files'. Also adds the v9 remediation plan + acceptance criteria docs and the corrected progress log (retracts earlier over-claims per 2026-07-08 audit). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Phase 0 (config/registries): - config-definition.xml: load tracked clarin-dspace.cfg (M13 root cause) - registries: CLARIN local-types (20 dc-types), bitstream-formats (97 incl archive types), dc.rights.label; v9-additive entries (openalex, doi scope_note) preserved - default.license: LINDAT Deposit Licence Agreement (M9, EN; CS already ok) - ORCID CachingOrcidRestConnector bean (BE-CFG-5) - DCInput pipeline (ComplexDefinitions, ACL, autocomplete-custom) from dtq-dev with v9 adaptations (jakarta, log4j2); DCInputTest+ACLTest green C2 license gate (BE-GATE-1 + GAP-1): - AuthorizeServiceImpl: CLARIN bitstream authorization hook (license confirmation / download token) guarded by !isAdmin - ClarinBitstreamService(+Impl) ported; S3-sync tier replaced by vanilla BitstreamStorageService computeChecksum/retrieveFile C3 (BE-REST-2): IdentifierRestRepository null guard - anon pid/find on a restricted item now answers 401 instead of 500 NPE; regression test added M10 (BE-WSI-1): WorkspaceItemRestRepository CLARIN license PATCH ops + share-token search, v9-adapted (jakarta, PLURAL_NAME) Tests ported from dtq-dev and passing: MetadataBitstreamRestRepositoryIT 11/11, BitstreamByHandleRestControllerIT 16/16, AuthorizationRestControllerIT 8/8, IdentifierRestRepositoryIT 8/8; CLARIN test builders + AbstractBuilder delta + archive test assets included. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…g tier OAI (C5): - 28 dspace-oai classes: SharedSaxonProcessor + 21 Saxon extension functions + 5 utils (SpecialItemService v9-adapted to LocalDate resource policies) + ColComFilter; DSpaceResourceResolver registers the functions - ItemUtils hand-reconciled onto v9: excluded bundles (oai.bundle.excluded), lindat bitstream URLs, restrictedAccess/owningCollection/itemId; XOAI local.hidden harvest fix - xoai.xml: cmdi/olac/oai_metasharev2/bibtex/elg/oai_datacite formats, ColComFilter/elg filters, openaire_data context merged additively (vanilla rioxx/openaire4 kept); oai.cfg hand-merged (v9 oai.html kept); description.xml sample identifier; oai_dc.xsl CLARIN delta - CMDIRestController (/cmdi) ported, jakarta + Spring ui.Model fix - Matomo governance: native MatomoEventHandler stays the only download emitter (bitstream tracker NOT ported - no double count); CLARIN OAI harvest tracker re-ported (no native equivalent), gated by matomo.track.enabled=false; matomo-java-tracker dep + beans wired Verified live after clean oai import: ListMetadataFormats includes all 5 CLARIN prefixes; GetRecord cmdi/olac 200 on 11234/1-3039 with restrictedAccess=true; ColComFilter excludes DH-community items; refbox BibTeX returns a real @misc citation. Submission (BE-SUB-3/4/5): submission-forms.dtd, submission-forms.xml merged (CLARIN clariah/teaching/specialFields forms, complex definitions, metashare value-pairs; v9 lowercase openaire forms/entity steps kept), item-submission.xml wires clarin-notice/clarin-license/specialFields + clariah/teaching processes. Live: specialFields 200, traditional sections end [license, clarin-license, specialFields]. Config: clarin-token launcher command (M7), PAT secret placeholders, Shibboleth header mapping (BE-CFG-6), email dspace.shortname branding (BE-CFG-7, v9-only templates kept). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
… + DevOps tier Main code (all compile+checkstyle clean): - M7 BE-PAT-1: clarin_token Bearer branch in JWTTokenRestAuthenticationServiceImpl - M8 BE-AUTH-1/2 + BE-MISC-1: user_registration hook on EPerson create (REST + create-administrator CLI with optional -o defaulting to Unknown), EPersonRest welcomeInfo/canEditSubmissionMetadata + converter/repo wiring - BE-CFG-3: StrictHttpFirewall bean (UTF-8 Shibboleth headers) + Verification-Token CORS header in WebApplication (Altcha kept, no recaptcha) - BE-CHK-5: bitstream /checksum link tier with PLURAL_NAME bean name; active-store checksum via vanilla BitstreamStorageService (S3-sync tier stays deferred; synchronizedStore left empty) - BE-ADMIN-M3: epichandles PLURAL_NAME bean; BE-ADMIN-M6: ItemAddBundleController PUT updateLicenseForBundle (jakarta, v9 mapper kept) - BE-SUB-2/6/7 + BE-OAI-7: DescribeStep/MetadataValidation(+local.hasCMDI)/ CMDIFileBundleMaintainer, autocomplete metadatavalues cluster (PLURAL_NAME), UploadStep/UploadValidation METADATA-bundle support incl. ItemService.hasUploadedFiles(item,bundle) overload - GAP-4: search Solr schema CLARIN deltas (dspaceAutoComplete *_ac, ASCII folding, items_owning_community, search_text copyfields); oai schema kept vanilla (RIOXX field preserved) Test tier (ported from dtq-dev, jakarta/v9-adapted): 11 CLARIN builders + base-builder deltas, 5 matchers, fixtures, test dspaceFolder config (TEST-CONFIG-DATA 8-file set incl. ClarinVersionedHandleIdentifierProvider swap and clarin-license test step), ~55 ITs + 13 unit suites. Current verified state: 27 suites green including WorkspaceItemRestRepositoryIT 154/154 (no vanilla submission regression), ClarinTokenServiceIT 9/9, CMDIFileBundleMaintainerTest 17/17, MetadataValueRestRepositoryIT 12/12, HandleRestRepositoryIT, ClarinLicenseLabel/UserRegistration/UserMetadata service ITs. 21 suites still red - triage in progress, fixes follow (ClarinMatomoBitstreamTrackerTest not ported: tracker intentionally absent; FilePreviewIT sync-store case removed: S3-sync tier deferred). DevOps: dataquest guards/image names + version-script inputs on v9 docker workflows, dtq release/PM/issue workflows (migrate-docker NOT ported), scripts/ tree (29 files incl. sourceversion.py), CLARIN_LOCAL_RUNBOOK.md. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The vanilla guard (dspace/dspace) never matched this fork, so docker jobs were silently skipped on PRs; pointing the guard at dataquest-dev made them run and fail on PRs (ghcr.io/dspace/dspace-dependencies pull is 403 - the base image is only published on pushes). Keep the dataquest publish path for push events, skip on PRs - same effective PR behavior as before the port. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…eview, epic + test green-up Main code: - SubmissionFormConverter + SubmissionFormFieldRest (BE-SUB-1 webapp half): complexDefinition/ autocompleteCustom now serialized; fixes FE dynamic-form JSON.parse(undefined) - ClarinBitstreamImportController ported (was never ported; all import ITs 405'd) - MetadataConverter.persistMetadataRest routes through place-preserving addMetadataByService - EpicHandleRestController.findOne gets @PreAuthorize (v9 dropped base-class annotation; converter security eval got null expression -> every endpoint 400) - DOIIdentifierProvider.removeDOIFromObject: dtq empty-remainder guard - PreviewContentDAOImpl.getPreview: JPQL replaces native SQL (Hibernate 6 auto-flush) - InstallItemServiceImpl full CLARIN port: dc.date.available (no embargo), local.language.name via IsoLangCodes + iso_langs.json, fixRelationMetadata/isreplacedby, submitter WRITE policy gated on allow.edit.metadata - DefaultItemVersionProvider: manageRelationMetadata + title "(yyyy-MM-dd)" suffix; versioning-service.xml ignoredMetadataFields += dc.date.available/doi/uri/relation.replaces - ItemMetadataQAChecker ported (+ v9 dereference -> dspaceObjectUtils.findDSpaceObject); curate.cfg checkhandles/metadataqa plugins - Provenance hooks, ClarinLogoImportController + addLogo overloads, validation beans, ClarinShibbolethLoginFilter matcher, EPersonRestAuthenticationProvider custom groups, 4xx message passthrough, ClarinFeaturedService PLURAL_NAME, PreviewContent findOne @PreAuthorize Tests (mirroring dtq-dev adaptations, all suites green locally: api batch + 662 webapp ITs): - BitstreamMatcher embeds/links += checksum; Version* title/property adaptations; patchRejectLicenseTest enables distribution license validation; PatchMetadataIT counts dc.date.available; item-metadata-patch-suite.json + provenance timestamp stripping; epic mock fixtures, FilePreview fixtures (preview-file-test.zip, logos.tgz) - ClarinDiscoveryRestControllerIT removed (never compiled against v9 FacetEntryMatcher; deferred to test-discovery.xml merge, live facets verified) Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…tions, test heap - RequiredMetadata.java: dtq version with getCollection(Item) fallback to the workspace/workflow item's collection (the dtq-only RequiredMetadataIT curates a workspace item whose owningCollection is null; vanilla task NPE'd at getReqList) - VersioningWithRelationshipsIT: dtq's 13-hunk adaptation (authority args on dc.relation addMetadata calls — authority.controlled.dc.relation=true in the CLARIN config — and versioned-title " (yyyy-MM-dd)" asserts) - HandleDAOImplTest: dtq adaptation for CLARIN subprefix handle renumbering (2-3 vs 101.2) - pom: default test heap 1024m -> 2048m (fork kernel OOM'd unit CI: AccessStatusServiceTest heap-space + cascading DBConnection/bean failures) Locally green: HandleDAOImplTest 1/1, VersioningWithRelationshipsIT 9/9, RequiredMetadataIT 1/1 (clean javac rebuild; earlier local VWR failure was IDE-ECJ class poisoning of target/test-classes, proven via javap overload comparison). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The ported CLARIN OAI extension functions use the s9api API directly; pin the dependency instead of relying on the transitive one from xoai (AC BE-OAI-1). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…est (dtq parity) The clarin.custom.groups authorities port reads configurationService in getGrantedAuthorities; the pure-Mockito test NPE'd in CI unit job. 2/2 green. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…cision record Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…Authenticator guard
Main code: EpicHandleServiceImpl.initialize + AbstractPIDService no longer register a
JVM-global java.net.Authenticator when lr.pid credentials are blank — with blank creds
it auto-answered every 401 Basic challenge in the JVM ('':'' retry), turning the SWORD
endpoints' 401s into 403/500 under Boot 3's JDK request factory (Swordv2IT). Blank
credentials can never authenticate, so this is production-neutral.
Test adaptations mirroring origin/dtq-dev (fallout of intended CLARIN behavior):
- BitstreamFormatRestRepositoryIT: DEFAULT_AMOUNT_FORMATS 86->95 (registry +9 formats)
- CCLicenseAdd/RemovePatchOperationIT: createWorkspaceItemWithNoClarinLicense (builder
default seeds dc.rights[.uri/.label], which cc.license.* maps read)
- DiscoveryVersioningIT: versioned titles assert the ' (yyyy-MM-dd)' suffix
- EPersonRestRepositoryIT: local ObjectMapper (tests setAnnotationIntrospector; must not
mutate the server's shared Spring mapper — poisoned it and later POSTs 400'd)
- Relationship(+Left/RightTilted)IT: metadata counts 15->16/22->23 (install-time
dc.date.available)
- StatisticsRestRepositoryIT: create the FilesVisited bitstreams in setup before the item
detaches (CLARIN addBitstream hooks tripped 'deleted object re-saved by cascade')
- SubmissionDefinitionsControllerIT: 11 sections (clarin-license step)
- SubmissionFormsControllerIT: admin token for ACL'd dc.title row + dtq's
testDoNotShowAdminInputFielToUser/testShowAdminInputFielToAdmin
- CanCreateVersionFeatureIT: toggle versioning.submitterCanCreateNewVersion=false in
property-sensitive tests (CLARIN default is true)
Local verify: all 15 affected suites green, 349 tests 0 failures (incl. Swordv1 5/5,
Swordv2 12/12, EpicHandle unit+IT regression).
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem description
Analysis
(Write here, if there is needed describe some specific problem. Erase it, when it is not needed.)
Problems
(Write here, if some unexpected problems occur during solving issues. Erase it, when it is not needed.)
Manual Testing (if applicable)
Copilot review