feat: extend meta classes by ceberam · Pull Request #617 · docling-project/docling-core

ceberam · 2026-05-19T14:32:34Z

No description provided.

github-actions · 2026-05-19T14:32:48Z

✅ DCO Check Passed

Thanks @ceberam, all your commits are properly signed off. 🎉

mergify · 2026-05-19T14:33:19Z

Merge Protections

Your pull request matches the following merge protections and will not be merged until they are valid.

🔴 Require two reviewer for test updates

Waiting for

#approved-reviews-by >= 2

This rule is failing.

When test data is updated, we require two reviewers

#approved-reviews-by >= 2

🟢 Enforce conventional commit

Wonderful, this rule succeeded.

Make sure that we follow https://www.conventionalcommits.org/en/v1.0.0/

title ~= ^(fix|feat|docs|style|refactor|perf|test|build|ci|chore|revert)(?:\(.+\))?(!)?:

ana-daniele · 2026-05-22T08:58:41Z

Changes

Backed up the rich version → feat/extend-meta-classes-backup (branch only, no PR)
Hard-reset feat/extend-meta-classes to main
Re-implemented the minimal scope:
- KeywordsMetaField and TopicsMetaField — values: list[str], deduplicated (order-preserving)
- Wired into BaseMeta (node-level) and new DocumentMeta (document-level, keywords + topics only)
- DoclingDocument.meta field added
- HTML / Markdown / Doclang serializers updated to render new fields as comma-separated values
- Dropped: Statement*, PropertyValue*, all confidence levels on new fields, title / authors / publication_date
- No schema version bump (additive, backward-compatible)
Added 8 tests covering round-trip, dedup, has_content, HTML escaping, and Markdown rendering
Squashed to 2 conventional commits (feat: + test:)
Force-pushed

Diff: +292 / -0 across 7 files (was +726 / -1019 in the rich version).

Tests: 22/22 in test_metadata.py, 389 in wider suite green.

Introduce two simple, additive meta fields: - KeywordsMetaField — unique list of keyword strings - TopicsMetaField — unique list of topic / subject strings Both attach to BaseMeta (node-level) and to the new DocumentMeta (document-level), which is wired into DoclingDocument.meta. HTML, Markdown, and Doclang serializers render the new fields as comma- separated values. Schema bump is intentionally avoided since the fields are optional and backward-compatible. Signed-off-by: Ana Daniele <ana.daniele@ibm.com>

Add round-trip, dedup, has_content, HTML escape, and Markdown rendering tests for KeywordsMetaField, TopicsMetaField, and DocumentMeta. Document-level meta absent-by-default is also covered. Signed-off-by: Ana Daniele <ana.daniele@ibm.com>

Convert tuple-form isinstance checks to PEP 604 unions and reorder a misplaced import flagged by ruff. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> Signed-off-by: Ana Daniele <ana.daniele@ibm.com>

Post-rebase onto main, the HTML metadata serializer now emits `data-meta-name="<field>"` instead of valueless `data-meta-<field>` and applies a single trailing html.escape pass. Drop the per-type escape from the Keywords/Topics branch (was producing double escaping after rebase) and update the two test assertions to the new attribute shape. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> Signed-off-by: Ana Daniele <ana.daniele@ibm.com>

Per discussion, DocLang is a standard whose extension is out of scope for this PR. Drop the KeywordsMetaField/TopicsMetaField branches from DoclangMetaSerializer and their imports; HTML and Markdown serialization remain. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> Signed-off-by: Ana Daniele <ana.daniele@ibm.com>

codecov · 2026-06-01T09:21:12Z

Codecov Report

✅ All modified and coverable lines are covered by tests.

📢 Thoughts on this report? Let us know!

ceberam · 2026-06-01T09:40:34Z

+    KEYWORDS = "keywords"  # unique keywords / keyphrases for the node content
+    TOPICS = "topics"  # unique topics / subjects for the node content


Could you please leverage docstrings in MetaFieldName class instead of inline comments? It would be create if you could fix the other attributes. Check TableFormerMode as an example of how we document enumerations.

ceberam · 2026-06-01T09:44:22Z

+class KeywordsMetaField(_ExtraAllowingModel):
+    """Container for a list of unique keywords / keyphrases."""
+
+    values: Annotated[list[str], Field(min_length=1)]


You could reuse a type that we created for unique lists: UniqueLists instead of list[str].

ceberam · 2026-06-01T09:46:44Z

+class TopicsMetaField(_ExtraAllowingModel):
+    """Container for a list of unique topics / subjects."""
+
+    values: Annotated[list[str], Field(min_length=1)]


Same as above

ceberam · 2026-06-01T09:48:51Z

+    keywords: Optional[KeywordsMetaField] = None
+    topics: Optional[TopicsMetaField] = None


Can we take advantage of this PR and add a good description of each metadata type? For instance, it would be helpful to make a clear distinction between entities, keywords, and topics. You can use the Annotated pattern and pydantic's Field function with the description and examples arguments.

ceberam · 2026-06-01T09:52:01Z

+class DocumentMeta(_ExtraAllowingModel):
+    """Document-level metadata for DoclingDocument."""
+
+    keywords: Optional[KeywordsMetaField] = None
+    topics: Optional[TopicsMetaField] = None
+
+


Please, remove this class. In the past we discussed about metadata at document level and we decided to use the body field of DoclingDocument to attach the document-level metadata.

ceberam · 2026-06-01T09:52:34Z

        # This is optional, e.g. a DoclingDocument could also be entirely
        # generated from synthetic data.
    )
+    meta: Optional[DocumentMeta] = None


Please, remove since not necessary. Users should use the body field for document-level metadata.

ceberam · 2026-06-01T09:54:51Z

+    assert field.values == ["nlp", "vision"]
+
+
+def test_keywords_and_topics_in_base_meta_roundtrip() -> None:


I would just extend the existing test_semantic_base_meta_fields_roundtrip_and_html_rendering, since it was built for testing any type of meta fields.

ceberam · 2026-06-01T09:57:13Z

+    doc = DoclingDocument(
+        name="doc-meta",
+        meta=DocumentMeta(
+            keywords=KeywordsMetaField(values=["llm", "rag"]),
+            topics=TopicsMetaField(values=["ai"]),
+        ),
+    )
+    roundtrip = DoclingDocument.model_validate(doc.model_dump(mode="json"))
+    assert roundtrip.meta is not None
+    assert roundtrip.meta.keywords is not None and roundtrip.meta.keywords.values == ["llm", "rag"]
+    assert roundtrip.meta.topics is not None and roundtrip.meta.topics.values == ["ai"]


Not sure if these lines test anything different with respect to the previous test.

ceberam · 2026-06-01T10:01:27Z

+    assert "[Keywords] ibm, zurich" in md
+    assert "[Topics] business" in md


Just FYI for a next PR: brackets should be escaped in markdown, since they are used to create links. This is the way we have serialized other metadata so far and therefore I would leave the implementation as it is in this PR. We can later think a nice way to add metadata in markdown and ensure it follows the right syntax.

ana-daniele force-pushed the feat/extend-meta-classes branch from 58dfa93 to eeb96d5 Compare May 22, 2026 08:54

ceberam marked this pull request as ready for review June 1, 2026 07:31

PeterStaar-IBM self-requested a review June 1, 2026 08:23

PeterStaar-IBM previously approved these changes Jun 1, 2026

View reviewed changes

ceberam assigned ana-daniele Jun 1, 2026

ana-daniele and others added 4 commits June 1, 2026 10:54

test: cover keywords and topics meta fields

3d1c482

Add round-trip, dedup, has_content, HTML escape, and Markdown rendering tests for KeywordsMetaField, TopicsMetaField, and DocumentMeta. Document-level meta absent-by-default is also covered. Signed-off-by: Ana Daniele <ana.daniele@ibm.com>

style: satisfy ruff UP038 and import order

4d5f289

Convert tuple-form isinstance checks to PEP 604 unions and reorder a misplaced import flagged by ruff. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> Signed-off-by: Ana Daniele <ana.daniele@ibm.com>

ana-daniele dismissed PeterStaar-IBM’s stale review via 22839e2 June 1, 2026 08:56

ana-daniele force-pushed the feat/extend-meta-classes branch from 2b193e6 to 22839e2 Compare June 1, 2026 08:56

ceberam commented Jun 1, 2026

View reviewed changes

		KEYWORDS = "keywords" # unique keywords / keyphrases for the node content
		TOPICS = "topics" # unique topics / subjects for the node content

		keywords: Optional[KeywordsMetaField] = None
		topics: Optional[TopicsMetaField] = None

		assert field.values == ["nlp", "vision"]


		def test_keywords_and_topics_in_base_meta_roundtrip() -> None:

		assert "[Keywords] ibm, zurich" in md
		assert "[Topics] business" in md

Conversation

ceberam commented May 19, 2026

Uh oh!

github-actions Bot commented May 19, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

mergify Bot commented May 19, 2026

Merge Protections

🔴 Require two reviewer for test updates

🟢 Enforce conventional commit

Uh oh!

ana-daniele commented May 22, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Changes

Uh oh!

codecov Bot commented Jun 1, 2026

Codecov Report

Uh oh!

ceberam Jun 1, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

ceberam Jun 1, 2026

Choose a reason for hiding this comment

Uh oh!

ceberam Jun 1, 2026

Choose a reason for hiding this comment

Uh oh!

ceberam Jun 1, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

ceberam Jun 1, 2026

Choose a reason for hiding this comment

Uh oh!

ceberam Jun 1, 2026

Choose a reason for hiding this comment

Uh oh!

ceberam Jun 1, 2026

Choose a reason for hiding this comment

Uh oh!

ceberam Jun 1, 2026

Choose a reason for hiding this comment

Uh oh!

ceberam Jun 1, 2026

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

github-actions Bot commented May 19, 2026 •

edited

Loading

ana-daniele commented May 22, 2026 •

edited

Loading

ceberam Jun 1, 2026 •

edited

Loading

ceberam Jun 1, 2026 •

edited

Loading