Add ThingsBoard latest-telemetry and attributes DAOs on IoTDB Table Mode#113
Add ThingsBoard latest-telemetry and attributes DAOs on IoTDB Table Mode#113PDGGK wants to merge 8 commits into
Conversation
… Mode (GSOC-304 Wk 5) Implements the ThingsBoard AttributesDao SPI (v4.3.1.2, 14 methods) on IoTDB Table Mode against the entity_attributes table, as a Path-3 stretch artifact that stays inert by default. - 13/14 methods implemented; findNextBatch is a relational keyset-pagination migration helper with no IoTDB equivalent, so it throws UnsupportedOperationException. The three v4.3.1.2 additions (findAllKeysByEntityIdsAndScopeAsync, findLatestByEntityIdsAndScope and its async form) are implemented. - save is a tag-only delete-then-insert under an entity read lock plus a per-identity lock; find takes the same identity lock so the gap is invisible. - Inert by default: no @repository on the DAO; an independent database.attributes.type selector decoupled from the timeseries selectors; an AttributesDaoConflictGuard fail-fast; and a constructor cluster-mode acknowledgement gate (iotdb.attributes.cluster_mode). - Strategy F: eight compile-only ThingsBoard/commons stubs, excluded from the built jar. - Documented Phase-1 limitations: null version (IoTDB has no sequence), empty non-null-profile key lookup, best-effort bulk reads. - 114 unit tests and 11 Testcontainers integration tests against apache/iotdb:2.0.8-standalone. Signed-off-by: Zihan Dai <99155080+PDGGK@users.noreply.github.com>
…04 Wk 4) Adapts the Wk4 IoTDBTableLatestDao (single-table derived-latest, no shadow table) onto the merged PR-1 base so it coexists with the timeseries and attributes DAOs. No latest overlay yet — that is a follow-up. - LatestDao: findLatest (ORDER BY time DESC LIMIT 1), findAllLatest (LAST_BY(col,time) + MAX(time) GROUP BY key); saveLatest no-op and removeLatest half-open-window are documented derived-latest gaps. - Coherent opt-in activation via IoTDBTableLatestEnabledCondition (database.ts.type=iotdb-table AND database.ts_latest.type=iotdb-table + experimental flag), reusing the shared named session pool; a TimeseriesLatestDaoConflictGuard fails startup on a conflicting host DAO. - v4.3.1.2 TimeseriesLatestDao SPI stub (10 methods incl. findLatestByEntityIds /+Async) verified byte-for-byte against the real ThingsBoard source; pinned by StrategyFContractTest with a getDeclaredMethods length guard. - All three DAOs share one session pool via @ConditionalOnMissingBean(name). - 131 unit tests + 24 Testcontainers ITs (Latest 4 + Timeseries 9 + Attributes 11) against apache/iotdb:2.0.8-standalone. Signed-off-by: Zihan Dai <99155080+PDGGK@users.noreply.github.com>
Closes the saveLatest/removeLatest gaps the single-table derived-latest cannot satisfy (mentor-confirmed: add a minimal latest overlay rather than document the latest-only paths as unsupported). - New telemetry_latest overlay table (telemetry shape, TTL='INF'), created only under the latest selector via a second schema-bootstrap bean. - saveLatest does a tag-only delete-then-insert into the overlay under a per-identity lock; the async batch writer prevents distinguishing latest-only from full-save, so the overlay is written on every saveLatest (it is therefore a per-key latest store, written once per latest update). version stays null. - findLatest/findAllLatest merge the derived latest (telemetry) with the overlay by max timestamp per key (overlay wins ties); the merge is max-by-ts, never additive, so latest-only keys surface and stale overlay rows are shadowed. - removeLatest snapshots the merged latest under the per-identity lock, deletes the in-window overlay row, and on rewriteLatestIfDeleted resurrects the next-older telemetry value into the overlay. - Documented Phase-1 residuals (flagged for review): version always null (Cassandra parity), a telemetry-derived removeLatest race for full-save values, overlay TTL='INF' growth, and the same-ts overlay-wins tie-break. - 141 unit tests + 31 Testcontainers ITs (Latest 11) against apache/iotdb:2.0.8-standalone. Signed-off-by: Zihan Dai <99155080+PDGGK@users.noreply.github.com>
…304 PR-2) Make the non-null deviceProfileId degradation rationale precise in the DAO javadoc + the module README: it mirrors the official non-relational backend (CassandraBaseTimeseriesLatestDao.findAllKeysByDeviceProfileId also returns Collections.emptyList()), entity_attributes has no device_profile_id tag, and the sole upstream caller is DeviceProfileController GET /api/deviceProfile/devices/keys/attributes (TENANT_ADMIN, a config-time UI key enumeration) which tolerates an empty result. Failing loud would 500 under IoTDB while Cassandra returns empty -- an avoidable backend inconsistency. A real device->profile lookup remains a Phase-2 optional enhancement. Signed-off-by: Zihan Dai <99155080+PDGGK@users.noreply.github.com>
…raceful key discovery - saveLatest skips a backdated write so an out-of-order saveLatest never regresses an overlay-only latest below a newer stored value (max-ts-wins; an exact-ts tie still overwrites, preserving the documented overlay-wins-on-tie behaviour) - findLatest/findLatestOpt take the per-identity lock so a concurrent overlay delete-then-insert is never observed mid-window (findAllLatest stays unlocked) - findAllKeysByDeviceProfileId returns an empty list instead of throwing, so the TENANT_ADMIN config-UI key endpoint degrades gracefully instead of returning 500 (matching the non-relational backends; full tenant-wide discovery is deferred) - document the single-JVM overlay convergence limitation - add unit + Testcontainers regression tests for the out-of-order saveLatest case Signed-off-by: Zihan Dai <99155080+PDGGK@users.noreply.github.com>
- Key-discovery / batch SPI methods (findAllKeysByEntityIds / findLatestByEntityIds and their async forms) return an empty result instead of throwing UnsupportedOperationException, matching CassandraBaseTimeseriesLatestDao: they back reachable endpoints (the dashboard telemetry-key lookup and entity-delete housekeeping) where a throw would surface as an HTTP 500 / a failed cleanup task. - IoTDBTableAttributesDao.save returns a non-null version (the attribute lastUpdateTs) so BaseAttributesService#doSave never unboxes null; the write is a failure-safe insert-then-delete-older and point reads use ORDER BY time DESC LIMIT 1. - removeLatest deletes the overlay row only when the overlay row's own timestamp is inside the delete window, so an out-of-window overlay value survives as the next latest; the rewrite-resurrect path reads both telemetry and the overlay (max-ts-wins). - The saveLatest max-ts guard and the removeLatest snapshot read the derived and overlay latest B1-tolerantly (a same-timestamp two-type row is treated as absent so it never wedges a write or delete and self-heals on the next overwrite); the strict read paths still surface it. - Regression tests added for each; 153 unit + 37 Testcontainers IT green (cache off). Signed-off-by: Zihan Dai <99155080+PDGGK@users.noreply.github.com>
…ter gate Addresses the five review comments on this PR: - Lift the bounded read/task executor (init/submit/shutdown-drain, bounded ArrayBlockingQueue + AbortPolicy, task tracking) out of the three DAOs into IoTDBTableBaseDao so a future shutdown-race fix lives in one place; the DAOs now delegate. No behavior change. - Lift the small per-DAO utilities into the base too: sqlString (the single SQL single-quote escape point), kvEntry type mapping, requireTelemetryKey / requireKey blank checks, and entityPredicate. - Give the attributes DAO its own executor config (iotdb.attributes.executor .threads / .queue-capacity, defaults equal to iotdb.ts.read) instead of borrowing the timeseries read-executor settings, so the attributes path can be tuned independently of the telemetry selector. - Document that the latest overlay write path (saveLatest / removeLatest) shares the read-configured executor/queue and enrich the queue-full message so a write-burst rejection of a concurrent findLatest is not mistaken for a read misconfiguration. - Add a symmetric cluster-mode acknowledgement (iotdb.ts_latest.cluster_mode) to the latest overlay, mirroring the existing attributes gate, so the in-JVM Striped-lock cluster caveat is opt-in the same way on both paths. 155 unit + 37 Testcontainers IT green (cache off); checkstyle, spotless, apache-rat, and enforcer clean. Signed-off-by: Zihan Dai <99155080+PDGGK@users.noreply.github.com>
|
Thanks for the thorough review, @CritasWang — all five addressed in 511f9bf. 1. Duplicated bounded executor across the three DAOs ( 2. Duplicated small utilities ( 3. Attributes executor borrowed 4. Write path shares the read executor + "read"-named queue-full exception ( 5. Asymmetric cluster gate ( Re-verified with the build cache off: 155 unit + 37 Testcontainers IT (against |
CritasWang
left a comment
CritasWang
left a comment
There was a problem hiding this comment.
README is missing documentation for the new required property iotdb.ts_latest.cluster_mode
This commit adds a fail-fast constructor gate on iotdb.ts_latest.cluster_mode (must be sticky-routing or disabled), but the README was not updated:
- The configuration table only documents iotdb.attributes.cluster_mode; there is no entry for iotdb.ts_latest.cluster_mode.
- The latest-telemetry section still describes activation as just setting database.ts_latest.type=iotdb-table, with no mention that the new property is now mandatory.
Impact: any deployment configured per the current README (with database.ts_latest.type=iotdb-table set) will fail to start after this change, and the operator has no documentation to consult — the only explanation lives in the exception message and javadoc.
Suggestion: add a row to the README configuration table for iotdb.ts_latest.cluster_mode (mirroring the existing iotdb.attributes.cluster_mode entry: required when the latest DAO is active, allowed values, fail-fast behavior, and why the acknowledgement exists), and note the requirement in the latest-telemetry activation section. While at it, the new tuning knobs iotdb.attributes.executor.threads / iotdb.attributes.executor.queue-capacity should also be added to the table.
…in README Add the README configuration-table rows the review flagged as missing after the executor / cluster-gate changes: iotdb.ts_latest.cluster_mode (required and fail-fast when the latest overlay DAO is active, mirroring iotdb.attributes.cluster_mode) and the iotdb.attributes.executor.threads / .queue-capacity tuning knobs. Also note the ts_latest.cluster_mode requirement in the latest-telemetry activation section, so an operator activating the overlay has documentation to consult rather than only the exception message and javadoc. Signed-off-by: Zihan Dai <99155080+PDGGK@users.noreply.github.com>
|
Done in 9534959 — added the README configuration-table rows for |
2 participants
This is the second PR of the GSoC series (follows #110, the timeseries
TimeseriesDao, now merged). It adds the latest-telemetry and attributes storage paths for running ThingsBoard on Apache IoTDB 2.x Table Mode.What this adds
1.
IoTDBTableLatestDao— the ThingsBoardTimeseriesLatestDaoSPI on Table Mode.telemetrytable (ORDER BY time DESC LIMIT 1for a single key;LAST_BY(col, time) + MAX(time) GROUP BY keyfor all keys), so the normal full-save path needs no separate latest table.telemetry_latest) that captures the latest-only write/delete cases the single-table derived read cannot satisfy (e.g. the EntityViewLATEST_AND_WScopy path callssaveLatestwith no pairedsave()).findLatest/findAllLatestmerge the derived latest with the overlay by max timestamp per key (overlay wins an exact-ts tie). The overlay table is created only under the latest selector.removeLatestsnapshots the derived and overlay latest separately under a per-identity lock and deletes the overlay row only when the overlay row's own timestamp is inside the half-open delete window — so an out-of-window overlay value is never wiped by a tag-only delete and survives as the next latest; the rewrite-resurrect path looks before the window across both telemetry and the overlay (max-ts-wins).saveLatestorremoveLatest, and a subsequent overlay overwrite self-heals it); the strict read paths (findLatest/findAllLatest) still surface such a row as a fault.findAllKeysByEntityIds(+Async),findLatestByEntityIds(+Async)) return an empty result rather than throwing — matching the officialCassandraBaseTimeseriesLatestDao— because they back reachable endpoints (the dashboard "available telemetry keys" lookup, entity-delete housekeeping) where a throw would surface as an HTTP 500 / a failed cleanup task. Full derived implementations are Wk9 / Wk10.2.
IoTDBTableAttributesDao— the ThingsBoardAttributesDaoSPI on Table Mode, against a dedicatedentity_attributestable.database.attributes.type=iotdb-tableis set — a selector that no current ThingsBoard release exposes — so a normal deployment never instantiates it and attributes keep flowing to the host entity DB. No@Repository; anAttributesDaoConflictGuardfails startup if a conflicting hostAttributesDaois present; a constructor gate requires an explicitiotdb.attributes.cluster_modeacknowledgement.saveis a failure-safe insert-then-delete-older (insert the current row atlastUpdateTs, then delete strictly-older rows for the identity), so anINSERTfailure leaves the prior value intact; point reads useORDER BY time DESC LIMIT 1. It returns a non-null version (the attribute'slastUpdateTs), whichBaseAttributesService#doSaveunboxes intoAttributeKv(..., long).Both follow the same activation isolation + compile-only ThingsBoard stub surface (excluded from the built jar) as #110, and reuse the module's single named session pool (deduped via
@ConditionalOnMissingBean(name=...)); each DAO can be activated independently.Testing
Verified with the build cache off: 153 unit tests + 37 Testcontainers integration tests (Latest 16 / Timeseries 9 / Attributes 12) against
apache/iotdb:2.0.8-standalone; apache-rat, checkstyle, spotless, dependency-analyze, and enforcer all clean. The compile-only SPI stubs are pinned to the real ThingsBoard v4.3.1.2 signatures byStrategyFContractTest.Points I'd appreciate review on
saveLatestcannot tell a latest-only write from a full-save's pairedsaveLatest, sotelemetry_latestis a per-key latest store (one row per key), written on everysaveLatestexcept when a strictly newer latest already exists (max-ts-wins; an exact-ts tie still overwrites, preserving the documented same-timestamp behaviour). The overlay write is a delete-then-insert so a same-timestamp two-type change resolves to a clean single-typed row. It is effectively a latest table (at most one extra overlay write per in-order latest update); I'd welcome confirmation the shape is acceptable, or a preference for a narrower variant.IoTDBTableLatestDaojavadoc):versionis alwaysnull(IoTDB has no per-series sequence — same as the Cassandra backend); for a purely telemetry-derived (full-save) latest,removeLatestand a concurrent historical delete are separate futures, so a re-read can transiently disagree until the historical delete lands;telemetry_latestisTTL='INF'with no entity-level cleanup; a telemetry-only write (saveWithoutLatest/ "skip latest persistence") surfaces via the derived read; and the per-identity overlay lock is in-JVM (Striped), so in a clustered deployment two nodes writing the same identity concurrently can transiently leave two overlay rows until the next write (the max-ts read merge still returns the newer value, and the B1-tolerant guard/remove reads prevent a two-type overlay row from wedging writes) — a symmetric cluster-mode acknowledgement for the latest overlay is a mentor-review option.AttributesDao.findAllKeysByDeviceProfileIdwith a non-nullDeviceProfileIdreturns an empty list — matching the official non-relational backend (CassandraBaseTimeseriesLatestDao.findAllKeysByDeviceProfileIdalso returnsCollections.emptyList()); the sole caller is a TENANT_ADMIN config-time UI key-enumeration endpoint (GET /api/deviceProfile/devices/keys/attributes) that tolerates empty. The rationale is documented in the DAO javadoc + README; a real device→profile lookup is a future optional enhancement.Happy to iterate.