[codex] Document memory-efficient semantic dedup fitting

[lbliii]

Addresses the semantic-deduplication portion of #2144. The bundled 26.06 release-note disposition is tracked in existing PR #2143.

Summary

• documents KMeansStage.fit_data_fraction, SemanticDeduplicationWorkflow.fit_data_fraction, and TextSemanticDeduplicationWorkflow.kmeans_fit_data_fraction
• explains per-actor file sampling, validation, the two-pass execution path, peak-memory reduction, and added I/O
• documents direct KMeansStage.cache_path centroid persistence and its distinction from workflow cache paths
• corrects the output artifact tree: workflow wrappers do not save kmeans_centroids.npy

Validation

• fern check: 0 errors, 103 pre-existing warnings
• all 11 Python examples parse with ast
• source contract audit confirms all documented parameter names exist on current main
• link check baseline remains the 22 pre-existing errors outside this page

[copy-pr-bot]

This pull request requires additional validation before any workflows can run on NVIDIA's runners.

Pull request vetters can view their responsibilities here.

Contributors can view more details about this message here.

[greptile-apps]

Greptile Summary

This PR documents the memory-efficient semantic deduplication fitting feature across KMeansStage, SemanticDeduplicationWorkflow, and TextSemanticDeduplicationWorkflow. All documented parameter names, defaults, and behavioral descriptions were verified against the current main source.

• Adds a "Reduce K-means fitting memory" section explaining the two-pass execution path (file-based sampling for fitting, then group-by-group prediction), peak memory trade-off, and added I/O cost; includes working code examples for all three API layers.
• Corrects the output artifact tree by removing kmeans_centroids.npy from the workflow cache path and documenting that only a direct KMeansStage(cache_path=...) call writes that file.
• Adds kmeans_fit_data_fraction to the Quick Start and Advanced Configuration examples, and documents the fit_data_fraction parameter for the lower-level workflows.

Confidence Score: 5/5

Documentation-only change; no executable code is altered, and all documented parameter names and behaviors match the current source.

Every documented parameter (kmeans_fit_data_fraction on TextSemanticDeduplicationWorkflow, fit_data_fraction on SemanticDeduplicationWorkflow and KMeansStage, cache_path on KMeansStage) was confirmed present in the corresponding Python files. The two-pass algorithm description matches the _fit_pass/_predict_write_pass implementation, the centroid-save path (actor_index == 0, kmeans_centroids.npy) matches the source, and the correction removing kmeans_centroids.npy from the workflow output tree is consistent with cache_path=None being hardcoded in SemanticDeduplicationWorkflow._run_kmeans_stage. No source-contract violations were found.

No files require special attention.

Important Files Changed

Filename	Overview
fern/versions/main/pages/curate-text/process-data/deduplication/semdedup.mdx	Documentation-only change adding kmeans_fit_data_fraction/fit_data_fraction parameter docs, two-pass execution explanation, and centroid persistence clarification; all claims verified against source in kmeans.py and workflow.py.

_{Reviews (2): Last reviewed commit: "Merge branch 'main' into codex/docs-late..." | Re-trigger Greptile}

[github-actions]

🌿 Preview your docs: https://nvidia-preview-codex-docs-late-freeze-updates.docs.buildwithfern.com/nemo/curator

Here are the markdown pages you've updated:

• nemo/curator/curate-text/process-data/deduplication/semdedup