Skip to content

[Python][C++] Fix tree-model TsFileDataFrame returning empty data on reused readers and uppercase names#862

Open
Young-Leo wants to merge 2 commits into
apache:developfrom
Young-Leo:fix/tree-model-uppercase-read
Open

[Python][C++] Fix tree-model TsFileDataFrame returning empty data on reused readers and uppercase names#862
Young-Leo wants to merge 2 commits into
apache:developfrom
Young-Leo:fix/tree-model-uppercase-read

Conversation

@Young-Leo

Copy link
Copy Markdown
Contributor

Summary

When reading tree-model TsFiles through TsFileDataFrame, series reads could
return empty arrays in two situations. This PR fixes both, which together were
responsible for "read one series OK, then read another series returns empty" on
a reused reader.

Both fixes go on top of the existing tree-model support (#816).


Fix 1 — C++: make StringArrayDeviceID::split_table_name() idempotent

Problem
Device IDs are cached and reused across queries. split_table_name() /
init_prefix_segments() appended to prefix_segments_ every time it was
called, so on a reused reader the prefix segments accumulated across successive
queries. The stale/duplicated prefix segments corrupted the per-device path
columns, so a second series read on the same reader could match nothing and
return empty (and it also leaked the previously allocated segment pointers).

Change
init_prefix_segments() now clears and frees the existing prefix_segments_
before rebuilding, making split_table_name() idempotent. Repeated calls
produce a stable, correct segment vector.

Files: cpp/src/common/device_id.cc, test cpp/test/common/device_id_test.cc
(DeviceIdTest.SplitTableNameIsIdempotent).


Fix 2 — Python: read series with uppercase measurement / path names

Problem
Tree TsFiles have no table schema, so the Python dataset layer reads them by
synthesizing a virtual table via query_table_on_tree. Because the table model
is case-insensitive, the native layer normalizes the synthesized column names
and path segments to lower case (e.g. Temperature -> temperature), while
the dataset catalog keeps the original casing from timeseries metadata. The two
tree read paths then looked the field column and device path up
case-sensitively, so an uppercase measurement or path segment failed the lookup
and returned an empty array:

with TsFileDataFrame(path) as tsdf:
    tsdf["root.ln.wf01.wt01.Temperature"][:]
    # -> array([], dtype=float64)   # expected [0.5, 1.5, 2.5, 3.5, 4.5]

Change Match field column names and device path segments case-insensitively in _read_series_by_row_tree and _read_arrow_tree, consistent with the case-insensitive table model. The original casing is still surfaced to users; only the internal lookup against the lower-cased result set is normalized.

Files: python/tsfile/dataset/reader.py, regression test in python/tests/test_tsfile_dataset.py.


Tests

  • New C++ regression test: DeviceIdTest.SplitTableNameIsIdempotent.
  • New Python regression tests covering reused-reader reads and uppercase measurement/path names.
  • No public API change; logical series names keep their original casing.

…ppercase names

Two related fixes for reading tree-model TsFiles through TsFileDataFrame, which together caused 'read one series OK, then reading another series returns empty' on a reused reader.

C++: make StringArrayDeviceID::split_table_name()/init_prefix_segments() idempotent. Device IDs are cached and reused across queries; the previous code appended to prefix_segments_ on every call, so prefixes accumulated (and leaked) across successive queries and corrupted per-device path columns, making a second series read match nothing. Now it clears/frees existing prefixes before rebuilding. Adds DeviceIdTest.SplitTableNameIsIdempotent.

Python: read series with uppercase measurement/path names. query_table_on_tree synthesizes a case-insensitive table schema, so the native layer lower-cases column names and path segments, while the dataset catalog keeps the original casing. The tree read paths matched field columns and device paths case-sensitively and returned empty for uppercase names. _read_series_by_row_tree and _read_arrow_tree now match case-insensitively. Adds a regression test.
@Young-Leo Young-Leo force-pushed the fix/tree-model-uppercase-read branch from 6586660 to d13cd23 Compare July 9, 2026 13:03
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant