[Python][C++] Fix tree-model TsFileDataFrame returning empty data on reused readers and uppercase names#862
Open
Young-Leo wants to merge 2 commits into
Open
Conversation
…ppercase names Two related fixes for reading tree-model TsFiles through TsFileDataFrame, which together caused 'read one series OK, then reading another series returns empty' on a reused reader. C++: make StringArrayDeviceID::split_table_name()/init_prefix_segments() idempotent. Device IDs are cached and reused across queries; the previous code appended to prefix_segments_ on every call, so prefixes accumulated (and leaked) across successive queries and corrupted per-device path columns, making a second series read match nothing. Now it clears/frees existing prefixes before rebuilding. Adds DeviceIdTest.SplitTableNameIsIdempotent. Python: read series with uppercase measurement/path names. query_table_on_tree synthesizes a case-insensitive table schema, so the native layer lower-cases column names and path segments, while the dataset catalog keeps the original casing. The tree read paths matched field columns and device paths case-sensitively and returned empty for uppercase names. _read_series_by_row_tree and _read_arrow_tree now match case-insensitively. Adds a regression test.
6586660 to
d13cd23
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
When reading tree-model TsFiles through
TsFileDataFrame, series reads couldreturn empty arrays in two situations. This PR fixes both, which together were
responsible for "read one series OK, then read another series returns empty" on
a reused reader.
Both fixes go on top of the existing tree-model support (#816).
Fix 1 — C++: make
StringArrayDeviceID::split_table_name()idempotentProblem
Device IDs are cached and reused across queries.
split_table_name()/init_prefix_segments()appended toprefix_segments_every time it wascalled, so on a reused reader the prefix segments accumulated across successive
queries. The stale/duplicated prefix segments corrupted the per-device path
columns, so a second series read on the same reader could match nothing and
return empty (and it also leaked the previously allocated segment pointers).
Change
init_prefix_segments()now clears and frees the existingprefix_segments_before rebuilding, making
split_table_name()idempotent. Repeated callsproduce a stable, correct segment vector.
Files:
cpp/src/common/device_id.cc, testcpp/test/common/device_id_test.cc(
DeviceIdTest.SplitTableNameIsIdempotent).Fix 2 — Python: read series with uppercase measurement / path names
Problem
Tree TsFiles have no table schema, so the Python dataset layer reads them by
synthesizing a virtual table via
query_table_on_tree. Because the table modelis case-insensitive, the native layer normalizes the synthesized column names
and path segments to lower case (e.g.
Temperature->temperature), whilethe dataset catalog keeps the original casing from timeseries metadata. The two
tree read paths then looked the field column and device path up
case-sensitively, so an uppercase measurement or path segment failed the lookup
and returned an empty array:
Change Match field column names and device path segments case-insensitively in
_read_series_by_row_treeand_read_arrow_tree, consistent with the case-insensitive table model. The original casing is still surfaced to users; only the internal lookup against the lower-cased result set is normalized.Files:
python/tsfile/dataset/reader.py, regression test inpython/tests/test_tsfile_dataset.py.Tests
DeviceIdTest.SplitTableNameIsIdempotent.