Feature | CMG-703 | stack-to-stack migration with audit report#1078
Feature | CMG-703 | stack-to-stack migration with audit report#1078chetan-contentstack wants to merge 23 commits into
Conversation
🔒 Security Scan Results
⏱️ SLA Breach Summary
ℹ️ Vulnerabilities Without Available Fixes (Informational Only)The following vulnerabilities were detected but do not have fixes available (no upgrade or patch). These are excluded from failure thresholds:
Consider reviewing these vulnerabilities when fixes become available. |
7 similar comments
🔒 Security Scan Results
⏱️ SLA Breach Summary
ℹ️ Vulnerabilities Without Available Fixes (Informational Only)The following vulnerabilities were detected but do not have fixes available (no upgrade or patch). These are excluded from failure thresholds:
Consider reviewing these vulnerabilities when fixes become available. |
🔒 Security Scan Results
⏱️ SLA Breach Summary
ℹ️ Vulnerabilities Without Available Fixes (Informational Only)The following vulnerabilities were detected but do not have fixes available (no upgrade or patch). These are excluded from failure thresholds:
Consider reviewing these vulnerabilities when fixes become available. |
🔒 Security Scan Results
⏱️ SLA Breach Summary
ℹ️ Vulnerabilities Without Available Fixes (Informational Only)The following vulnerabilities were detected but do not have fixes available (no upgrade or patch). These are excluded from failure thresholds:
Consider reviewing these vulnerabilities when fixes become available. |
🔒 Security Scan Results
⏱️ SLA Breach Summary
ℹ️ Vulnerabilities Without Available Fixes (Informational Only)The following vulnerabilities were detected but do not have fixes available (no upgrade or patch). These are excluded from failure thresholds:
Consider reviewing these vulnerabilities when fixes become available. |
🔒 Security Scan Results
⏱️ SLA Breach Summary
ℹ️ Vulnerabilities Without Available Fixes (Informational Only)The following vulnerabilities were detected but do not have fixes available (no upgrade or patch). These are excluded from failure thresholds:
Consider reviewing these vulnerabilities when fixes become available. |
🔒 Security Scan Results
⏱️ SLA Breach Summary
ℹ️ Vulnerabilities Without Available Fixes (Informational Only)The following vulnerabilities were detected but do not have fixes available (no upgrade or patch). These are excluded from failure thresholds:
Consider reviewing these vulnerabilities when fixes become available. |
🔒 Security Scan Results
⏱️ SLA Breach Summary
ℹ️ Vulnerabilities Without Available Fixes (Informational Only)The following vulnerabilities were detected but do not have fixes available (no upgrade or patch). These are excluded from failure thresholds:
Consider reviewing these vulnerabilities when fixes become available. |
chetan-contentstack
requested review from
shradha-nahar,
umesh-more-cstk,
umeshmore45 and
vikrantraut-cstk
and removed request for
umeshmore45
🔒 Security Scan Results
⏱️ SLA Breach Summary
ℹ️ Vulnerabilities Without Available Fixes (Informational Only)The following vulnerabilities were detected but do not have fixes available (no upgrade or patch). These are excluded from failure thresholds:
Consider reviewing these vulnerabilities when fixes become available. |
There was a problem hiding this comment.
Pull request overview
Implements Contentstack stack-to-stack migration support end-to-end (export → validate → audit → map → test → final), adding new backend services/routes, upload-api validators/mappers, and UI flow updates (including cross-region source login/session handling and improved long-running migration UX).
Changes:
- Adds Contentstack export validation + locale extraction + upload-api validator/mapper support.
- Adds API support for source export/validation/audit (with sharded lowdb audit storage) and CLI import safeguards (link normalization + per-module fallback).
- Updates UI migration flow (audit step, long timeouts, better log parsing, and regional source-login token caching).
Reviewed changes
Copilot reviewed 67 out of 69 changed files in this pull request and generated 7 comments.
Show a summary per file
| File | Description |
|---|---|
| upload-api/tests/unit/validators/contentstack.validator.test.ts | Unit tests for Contentstack zip/json validator. |
| upload-api/tests/unit/services/contentstack.service.test.ts | Unit tests for Contentstack mapper service POSTing locales. |
| upload-api/tests/unit/services/contentstack.locales.test.ts | Unit tests for resolving export root + extracting locales. |
| upload-api/tests/unit/config/index.config.test.ts | Updates config env var precedence + mysql default tests. |
| upload-api/src/validators/index.ts | Registers Contentstack validator for zip/json formats. |
| upload-api/src/validators/contentstack.ts | Adds Contentstack zip/json validation logic. |
| upload-api/src/services/createMapper.ts | Adds Contentstack mapper selection. |
| upload-api/src/services/contentstack/locales.ts | Implements export-root resolution + locale extraction for upload-api. |
| upload-api/src/services/contentstack/index.ts | Creates Contentstack mapper (posts extracted locales). |
| upload-api/src/routes/index.ts | Tightens file path handling + response typing; fixes comment typo. |
| upload-api/src/config/index.ts | Aligns local path env precedence + formatting. |
| upload-api/migration-aem/package-lock.json | Removes @types/uuid entry from lockfile. |
| ui/src/utilities/migrationSourceSession.ts | Session storage helpers for source-region token caching. |
| ui/src/utilities/functions.ts | Ensures failure notifications always have non-empty text. |
| ui/src/utilities/constants.ts | Adds constant for popup postMessage source. |
| ui/src/services/api/user.service.ts | Adds profile fetch for an override token (regional login). |
| ui/src/services/api/upload.service.ts | Hardens getCall error fallback when err.response is missing. |
| ui/src/services/api/stacks.service.ts | Adds optional source_region + app token override when listing stacks. |
| ui/src/services/api/migration.service.ts | Adds source-config update, audit/export endpoints, and long timeouts. |
| ui/src/pages/Migration/index.tsx | Adds audit step + source_details persistence + multiple step flow changes. |
| ui/src/pages/Login/index.tsx | Adds popup regional login flow + return-to handling + safer errors. |
| ui/src/context/app/app.interface.ts | Extends migration state shape with source_details + audit + stepValue. |
| ui/src/components/TestMigration/index.tsx | Improves disable logic + state persistence + completion handling. |
| ui/src/components/TestMigration/index.scss | Tweaks layout spacing for test migration UI. |
| ui/src/components/MigrationFlowHeader/index.tsx | Makes CTA label dependent on step/CMS; fixes step gating logic. |
| ui/src/components/LogScreen/MigrationLogViewer.tsx | Buffers socket log chunks + improves parsing + completion handling. |
| ui/src/components/LogScreen/index.tsx | Same log chunk buffering + parsing improvements for test migration logs. |
| ui/src/components/LegacyCms/legacyCms.scss | Styles new Contentstack source panel UI. |
| ui/src/components/LegacyCms/Actions/LoadSelectCms.tsx | Initializes source_details and sets source_mode for Contentstack. |
| ui/src/components/DestinationStack/Actions/LoadLanguageMapper.tsx | Supports locale arrays as strings or objects (Contentstack format). |
| ui/src/components/Common/DeleteProjectModal/index.tsx | Fixes React prop casing: tabIndex. |
| ui/src/components/AuditReport/index.tsx | Adds UI for running/reviewing audit and persisting exclusions/summary. |
| ui/src/components/AuditReport/index.scss | Styling for audit report UI/table. |
| ui/src/cmsData/migrationSteps.json | Adds Audit step and renumbers steps to 6-step flow. |
| api/tests/unit/utils/normalize-entry-links.utils.test.ts | Tests link normalization across schema shapes (group/blocks/multiple). |
| api/tests/unit/utils/contentstack-user-orgs.utils.test.ts | Tests org mapping across region-specific role payload shapes. |
| api/tests/unit/services/validation.service.test.ts | Tests export root resolution + structure validation. |
| api/tests/unit/services/user.service.test.ts | Ensures profile response includes region and mapped orgs. |
| api/tests/unit/services/projects.service.test.ts | Updates step-progression tests + adds source/audit update tests. |
| api/tests/unit/services/migration.service.test.ts | Adds tests for export/validate/audit endpoints and step/status updates. |
| api/tests/unit/services/exportCli.service.test.ts | Tests CLI export execution + region handling + error cases. |
| api/tests/unit/services/contentMapper.service.test.ts | Updates expected step gating for mapper operations. |
| api/tests/unit/services/auth.service.test.ts | Updates org membership requirements + aligns error message constant. |
| api/tests/unit/services/audit.service.test.ts | Tests audit generation paths and edge cases. |
| api/tests/unit/routes/projects.routes.test.ts | Adds route wiring tests for new project endpoints. |
| api/tests/unit/routes/migration.routes.test.ts | Adds route wiring tests for new migration endpoints. |
| api/tests/unit/models/audit-lowdb.test.ts | Tests sharded audit persistence + retrieval behavior. |
| api/tests/unit/controllers/migration.controller.test.ts | Updates controller behavior to await long-running service calls. |
| api/src/utils/sanitize-path.utils.ts | Adds allowlisted export-root path assertion helper. |
| api/src/utils/normalize-entry-links.utils.ts | Adds export-wide legacy link normalization utility. |
| api/src/utils/contentstack-user-orgs.utils.ts | Adds robust org mapping util across role shapes/regions. |
| api/src/services/validation.service.ts | Adds Contentstack export structure validation + root resolution. |
| api/src/services/user.service.ts | Uses org mapping util + includes region in profile payload. |
| api/src/services/runCli.service.ts | Adds link normalization + module fallback + export-root resolution. |
| api/src/services/projects.service.ts | Adds source config + audit selection updates; adds audit step gating. |
| api/src/services/exportCli.service.ts | Implements Contentstack stack export via CLI with structured logging. |
| api/src/services/auth.service.ts | Uses org mapping util for admin/owner/member determination. |
| api/src/services/audit.service.ts | Implements audit generation + persistence to sharded lowdb store. |
| api/src/server.ts | Fixes log tailing to avoid awaitWriteFinish delays; adds offset reset. |
| api/src/routes/projects.routes.ts | Adds project routes for source-config and audit-selections. |
| api/src/routes/migration.routes.ts | Adds migration routes for export/validate/audit. |
| api/src/models/project-lowdb.ts | Extends project schema to store source_details/audit/validation/source_locales. |
| api/src/models/audit-lowdb.ts | Adds sharded audit storage model. |
| api/src/controllers/projects.controller.ts | Adds controllers for source-config + audit-selections. |
| api/src/controllers/migration.controller.ts | Adds controllers for export/validate/audit; awaits long operations. |
| api/src/constants/index.ts | Adds CMS.CONTENTSTACK, audit DB config, and renumbers step constants. |
| .gitignore | Ignores export-stack/ output directory. |
Files not reviewed (1)
- upload-api/migration-aem/package-lock.json: Language not supported
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
🔒 Security Scan Results
⏱️ SLA Breach Summary
ℹ️ Vulnerabilities Without Available Fixes (Informational Only)The following vulnerabilities were detected but do not have fixes available (no upgrade or patch). These are excluded from failure thresholds:
Consider reviewing these vulnerabilities when fixes become available. |
| // Default 30-minute cap on any single CLI invocation. Large stacks legitimately | ||
| // take many minutes to export, but a stuck child process should not hold an | ||
| // API request open indefinitely. Overridable via EXPORT_CLI_TIMEOUT_MS. | ||
| const DEFAULT_CLI_TIMEOUT_MS = 30 * 60 * 1000; |
There was a problem hiding this comment.
can you explain this why we are cli timeout?
There was a problem hiding this comment.
This was asked on Above comment By @vikrantraut-cstk.
#1078 (comment)
| break; | ||
| } | ||
| if (cms !== CMS.AEM) { | ||
| if (cms !== CMS.AEM && cms !== CMS.CONTENTSTACK) { |
There was a problem hiding this comment.
why we are not putting this in switch case
There was a problem hiding this comment.
@umesh-more-cstk These blocks are intentionally outside the switch (cms) — they run after the CMS-specific pre-processing. The first one applies to every CMS except AEM and Contentstack, and the second is Contentstack-only post-migration cleanup. Moving them into the switch would duplicate the logic across multiple cases (or several cases would fall through into Contentstack-style cleanup). Happy to refactor if you'd prefer, but the current shape avoids the duplication.
| ); | ||
|
|
||
| // Handle CMS-specific post-migration tasks | ||
| if (cms === CMS.CONTENTSTACK) { |
There was a problem hiding this comment.
why we are not putting this in switch case
🔒 Security Scan Results
⏱️ SLA Breach Summary
ℹ️ Vulnerabilities Without Available Fixes (Informational Only)The following vulnerabilities were detected but do not have fixes available (no upgrade or patch). These are excluded from failure thresholds:
Consider reviewing these vulnerabilities when fixes become available. |
1 similar comment
🔒 Security Scan Results
⏱️ SLA Breach Summary
ℹ️ Vulnerabilities Without Available Fixes (Informational Only)The following vulnerabilities were detected but do not have fixes available (no upgrade or patch). These are excluded from failure thresholds:
Consider reviewing these vulnerabilities when fixes become available. |
Adds Contentstack-to-Contentstack migration across regions/orgs: CLI-driven source export, export-structure validation, audit report (unused assets, unpublished entries, empty content types, unused global fields), and import via the official CLI with link-field normalization. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
… and environment variable usage
- audit.service: support all regions in buildContentstackUrl, quoted-token asset reference detection, optional chaining, drop unused assetUidSet. - runCli.service: replace literal current_step=6 with STEPPER_STEPS.MIGRATION. - projects.service: gate audit-summary requirement to Contentstack source so non-CS migrations aren't blocked at step 2. - Migration UI: derive next step from isContentstackSource in content mapper and test migration handlers.
…tack - migration.service: skip non-CS pre-create steps (fieldAttacher, marketplace apps, extensions, taxonomies, global fields, delta asset index) when cms === CONTENTSTACK; the CLI imports these directly from the export. - migration.controller: default to 200 when service returns void so Express doesn't crash with "Invalid status code: undefined". - contentMapper + projects.service: align iteration default to 1 so reads match writers and step 4 finds the audit-generated mappers. - post-rebase cleanup: drop dead AutoMappedMerge modal reference, remove duplicate updateNewMigrationData import, fix dangling setHasShownCompletionNotification reset effect.
chetan-contentstack
force-pushed
the
feature/cmg-703-stack-to-stack
branch
from
4bf9336 to
f32cf05
Compare
🔒 Security Scan Results
⏱️ SLA Breach Summary
ℹ️ Vulnerabilities Without Available Fixes (Informational Only)The following vulnerabilities were detected but do not have fixes available (no upgrade or patch). These are excluded from failure thresholds:
Consider reviewing these vulnerabilities when fixes become available. |
🔒 Security Scan Results
⏱️ SLA Breach Summary
ℹ️ Vulnerabilities Without Available Fixes (Informational Only)The following vulnerabilities were detected but do not have fixes available (no upgrade or patch). These are excluded from failure thresholds:
Consider reviewing these vulnerabilities when fixes become available. |
- sanitize-path: include space/parens in allowlist (upload-api preserves them in extracted filenames); throw on disallowed chars instead of silently stripping so tampered paths can't resolve elsewhere - runCli: validate user-controlled source paths in resolveSourcePathForImport through assertExportPathInAllowedRoot before handing to the CLI - upload-api routes: add extractedPath to FileProcessingResponse and drop the three (data as any) casts - audit.service: tolerate missing/corrupt assets/assets.json instead of aborting the whole audit run Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
🔒 Security Scan Results
⏱️ SLA Breach Summary
ℹ️ Vulnerabilities Without Available Fixes (Informational Only)The following vulnerabilities were detected but do not have fixes available (no upgrade or patch). These are excluded from failure thresholds:
Consider reviewing these vulnerabilities when fixes become available. |
…ecord Moves the stack-to-stack source app token out of browser sessionStorage and onto the authenticated user record (authentication.json). The token now survives across browser sessions and is never exposed to client-side storage. - api: extend AuthenticationDocument with source_session; add GET/PUT/DELETE /v2/user/source-session backed by user.service - ui: migrationSourceSession utility now async, backed by the new endpoints; Login + LoadUploadFile call sites updated to await Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Adds an explicit FileProcessingResult return type so every branch contributes the same shape (extractedPath optional). Resolves TS2339 at the route call sites that read data?.extractedPath after dropping the prior (data as any) casts. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
…to-stack # Conflicts: # api/src/services/migration.service.ts # api/src/services/runCli.service.ts # ui/src/pages/Migration/index.tsx
🔒 Security Scan Results
⏱️ SLA Breach Summary
ℹ️ Vulnerabilities Without Available Fixes (Informational Only)The following vulnerabilities were detected but do not have fixes available (no upgrade or patch). These are excluded from failure thresholds:
Consider reviewing these vulnerabilities when fixes become available. |
Adds unit tests for the new source-session service/controller/UI utility introduced by the backend-persisted source token work. Lifts api function coverage to 81.13% and ui to 83.89% so both packages clear their 80% threshold (previously 79.76% / 79.19%). Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
🔒 Security Scan Results
⏱️ SLA Breach Summary
ℹ️ Vulnerabilities Without Available Fixes (Informational Only)The following vulnerabilities were detected but do not have fixes available (no upgrade or patch). These are excluded from failure thresholds:
Consider reviewing these vulnerabilities when fixes become available. |
Summary
Implements CMG-703 — content migration from a source Contentstack stack to a destination stack across regions, organizations, or within the same org/region.
The flow: source-stack auth → CLI-driven export → export validation → audit report → destination config → content mapper → test migration → final migration.
Scope delivered
Source stack selection
LoadUploadFile.tsx).npx @contentstack/cli cm:stacks:exportvia the newexportCli.service.ts; exported data lands underapi/export-stack/<stackId>/.Export validation
validation.service.tsresolves the new branch-nested export layout (main/,master/, …) and reports missing modules (content_types,entries,assets,global_fields,locales) with a clean message.resolvedRooton the project so downstream steps use one canonical path.Audit report (new Step 3)
audit.service.ts+audit-lowdb.tsgenerate a per-project sharded audit covering:AuditReport/index.tsxlets users review and exclude items from the upcoming migration; exclusions are persisted on the project.GET /audit-data/:projectId.Destination configuration
getUserProfileWithToken) so users authenticate once per region and the resulting token is cached for reuse.contentstack-user-orgs.utils.ts.Test & final migration
runCli.service→cm:stacks:importpath with two new safeguards:normalize-entry-links.utils.tsrewrites legacy{ url, title }link values to{ href, title }(recursive throughgroup/blocks).MODULE_NOT_FOUND) trigger a per-module fallback so the rest of the import still succeeds.ERROR:lines from the CLI are treated as non-fatal warnings; only known fatal strings abort the run.Acceptance criteria mapping
audit-lowdb,/audit-data/:projectId)Known limitations (worth follow-up tickets)
source_branchonly; stacks with multiple branches need one project per branch. UI doesn't expose multi-branch selection yet.resolveContentstackExportRootis duplicated betweenapi/andupload-api/; consolidating requires a shared workspace package.