diff --git a/iotdb-thingsboard-table/README.md b/iotdb-thingsboard-table/README.md index 5217277e..fe995591 100644 --- a/iotdb-thingsboard-table/README.md +++ b/iotdb-thingsboard-table/README.md @@ -70,6 +70,121 @@ attribute/label DAOs are outside the current scope. > + write + delete only**. **Time-bucketed aggregation is NOT implemented yet**; > aggregation, latest telemetry, and attributes are outside the current scope. +## Entity attributes (Path-3 stretch, inert by default) + +`IoTDBTableAttributesDao` is an `AttributesDao` implementation for the IoTDB Table +Mode backend, storing entity attributes in the `entity_attributes` table. It is a +**Path-3 stretch artifact and is inert by default**: in Phase-1, entity attributes +stay in the host entity database (PostgreSQL), and this DAO only activates when an +operator opts in explicitly with `database.attributes.type=iotdb-table`. + +This attribute selector is **independent** of `database.ts.type` / +`database.ts_latest.type` — the attribute DAO routes separately from the +time-series DAOs (a piggy-back on the timeseries selector was deliberately +rejected). No shipped ThingsBoard release exposes a `database.attributes.type` +selector yet (open Q6 / ThingsBoard Discussion #15296), so a real Phase-1 +deployment never sets it; the activation condition stays false, no attribute bean +or session pool is created, and attributes keep flowing to the host entity-DB +`AttributesDao`. + +When activated, each identity tuple +`(tenant_id, entity_type, entity_id, attribute_scope, key)` holds exactly one +current row: `save` is a tag-only `DELETE` (no time predicate) followed by an +`INSERT` at `time = lastUpdateTs` with exactly one typed FIELD set, both under a +per-identity in-JVM lock so concurrent same-identity writes converge to a single +row. + +### Phase-1 attribute limitations (documented, not silently degraded) + +- **`version` is always `null`.** IoTDB has no SQL sequence, so `save` and + `removeAllWithVersions` return a `null` version (type-correct, matching the + Cassandra backend). Because the ThingsBoard service only emits an EDQS + attribute-delete notification when the version is non-null, that notification is + not driven in Phase-1. +- **`findNextBatch` is unsupported.** It is a relational keyset-pagination + migration helper with no IoTDB equivalent; it throws + `UnsupportedOperationException`. +- **`findAllKeysByDeviceProfileId` with a non-null profile returns an empty + list**, matching the official non-relational backend: + `CassandraBaseTimeseriesLatestDao.findAllKeysByDeviceProfileId` also returns + `Collections.emptyList()`, since a NoSQL / time-series store cannot do the + cross-device-table profile-dimension join. `entity_attributes` has no + `device_profile_id` tag and the module has no device→profile lookup. The sole + upstream caller is `DeviceProfileController` + `GET /api/deviceProfile/devices/keys/attributes` (TENANT_ADMIN, a config-time UI + key enumeration) which tolerates an empty result; failing loud would 500 under + IoTDB while Cassandra returns empty. The null-profile path returns the + tenant-wide distinct keys. A real device→profile lookup is a Phase-2 / Wk9 + optional enhancement. +- **`save` is non-atomic.** The tag-only `DELETE`-then-`INSERT` is two separate + statements with no rollback: if the `INSERT` fails after the `DELETE`, the value + is lost. A concurrent same-identity point `find` takes the same per-identity + lock and so never observes the transient empty window, but full-scope reads + (`find`-by-keys / `findAll` / `findLatestByEntityIdsAndScope`) are best-effort + (unlocked). +- **Single-JVM convergence only.** The per-identity lock converges writes only + within one JVM; cross-node single-writer safety is the operator's + responsibility, acknowledged via `iotdb.attributes.cluster_mode` (must be + `sticky-routing` or `disabled` when the DAO is active, else construction fails + fast). + +## Latest telemetry (derived + overlay) + +`IoTDBTableLatestDao` is a `TimeseriesLatestDao` implementation that serves the +latest value per `(tenant, entity, key)`. It activates only when all of +`database.ts.type=iotdb-table`, `database.ts_latest.type=iotdb-table`, and +`iotdb.ts.experimental-raw-only=true` are set (the timeseries selector is +required because the derived latest reads the `telemetry` table that only the +IoTDB writer populates). + +The latest value is read from **both** the historical `telemetry` table +(derived, `ORDER BY time DESC LIMIT 1` / `LAST_BY(col, time)`, engine-accelerated +by IoTDB's native last cache) **and** a minimal per-key `telemetry_latest` +**overlay** table, merged by the maximum timestamp per key (the overlay wins an +exact tie). The merge is max-by-ts, not additive, so a key present in both stores +is never double-counted. + +**The overlay is written on every `saveLatest`** (mentor-confirmed; a PR-2 review +flag). The module's historical `save()` write path is asynchronous and batched, +so at the moment `saveLatest` runs it cannot see whether a paired `telemetry` row +will be written. It therefore cannot distinguish a latest-only write (e.g. the +EntityView telemetry-copy `LATEST_AND_WS` / `saveTs=false` path, which calls +`saveLatest` with **no** paired `save()`) from a normal full-save, and writes the +overlay unconditionally as a delete-then-insert (one row per identity, under a +per-identity in-JVM lock). This **closes the latest-only data-loss gap** that a +no-shadow-table no-op would otherwise drop, at the cost of one extra overlay write +per latest update (equivalent to the standard ThingsBoard latest-table behavior). +`removeLatest` snapshots the merged latest under the per-identity lock and, when +that latest is inside the half-open `[startTs, endTs)` delete window, deletes the +overlay row; when `rewriteLatestIfDeleted` is set it resurrects the next-older +historical value (`telemetry` where `time < startTs`) back into the overlay and +returns it as the removing result's data (so TB emits a WS update rather than a +delete). + +### Residual latest limitations (documented, flagged for PR-2 review) + +- **`version` is always `null`.** IoTDB has no SQL sequence (same as Cassandra); + type-correct and contract-legal, but TB notifications that key off a non-null + version are not driven in Phase-1. +- **Telemetry-derived race residual.** Overlay-backed values are race-free under + the per-identity lock, but the historical `remove` runs as a separate future + from `removeLatest`, so a purely telemetry-derived (full-save) latest can + transiently still be read from `telemetry` until that historical delete commits + (eventually consistent). +- **Overlay growth.** `telemetry_latest` is `TTL='INF'` with no entity-level + cleanup, so under unbounded key cardinality it grows without bound (one row per + identity; bounded for normal key sets). +- **Same-timestamp cross-store type change.** The overlay wins an exact-ts tie, + continuing the documented same-timestamp (B1) limitation below. +- **Batch latest read / key discovery deferred.** `findLatestByEntityIds(Async)` + (new in v4.3.1.2) throws `UnsupportedOperationException` (Wk 10); the + key-discovery methods throw it too (Wk 9). + +The `telemetry_latest` table is created on startup by a **second** idempotent +schema bootstrap (`schema-iotdb-table-latest.sql`), registered only when the +latest selector is active and `iotdb.schema.bootstrap` is not disabled; when the +latest selector is off the overlay table is never created. + ## Known limitations **Same-timestamp type change across separate flushes.** The writer collapses @@ -98,11 +213,13 @@ Key activation and operational flags: | --- | --- | --- | | `database.ts.type` | _(unset)_ | Set to `iotdb-table` as the ThingsBoard historical-timeseries backend selector. | | `iotdb.ts.experimental-raw-only` | `false` | Explicit opt-in for this initial raw-only backend. Must be `true` together with `database.ts.type=iotdb-table`; write, raw read, and delete are implemented, while time-bucketed aggregation is outside the current scope. | +| `database.attributes.type` | _(unset)_ | Set to `iotdb-table` to opt in to the entity-attribute DAO (Path-3 stretch). Independent of the timeseries selectors. Unset in a real Phase-1 deployment, so the attribute DAO is inert by default. | +| `iotdb.attributes.cluster_mode` | _(empty)_ | Required when `database.attributes.type=iotdb-table`. Must be `sticky-routing` (per-identity writes pinned to one node) or `disabled` (single-node / acknowledged best-effort); any other value (including the empty default) fails construction fast, because the attribute write path converges only within a single JVM. | | `iotdb.host` / `iotdb.port` | `127.0.0.1` / `6667` | IoTDB node address. | | `iotdb.username` / `iotdb.password` | `root` / `root` | IoTDB credentials. | | `iotdb.database` | `thingsboard` | Target IoTDB database. | | `iotdb.session-pool-size` | `8` | Table session pool size. | -| `iotdb.schema.bootstrap` | `true` | When `true`, the module runs an idempotent startup bootstrap that reads `schema-iotdb-table.sql` from the classpath and creates the `telemetry` / `entity_attributes` tables (and database) on a fresh IoTDB before the first write. Set to `false` if you manage the schema out-of-band. | +| `iotdb.schema.bootstrap` | `true` | When `true`, the module runs an idempotent startup bootstrap that reads `schema-iotdb-table.sql` from the classpath and creates the `telemetry` / `entity_attributes` tables (and database) on a fresh IoTDB before the first write. When the latest selector is active it also runs a second bootstrap from `schema-iotdb-table-latest.sql` to create the `telemetry_latest` overlay table. Set to `false` if you manage the schema out-of-band. | The module is a Spring Boot **auto-configuration** (`IoTDBTableConfiguration`). Its deployment host is ThingsBoard 4.3.x, which runs on Spring Boot 3.5.x, so the @@ -165,5 +282,12 @@ docker compose -f docker-compose.test.yml down -v Initial module status: `IoTDBTableBaseDao` plus the `IoTDBTableTimeseriesDao` write, raw-read, and delete paths are implemented behind `database.ts.type=iotdb-table` and `iotdb.ts.experimental-raw-only=true`. -Without both properties, the module is inert. Aggregation, latest telemetry, and -attributes are outside the current scope. +Without both properties, the module is inert. Aggregation and latest telemetry +are outside the current scope. + +`IoTDBTableAttributesDao` is implemented as a **Path-3 stretch artifact**, inert +by default and activated only by the independent +`database.attributes.type=iotdb-table` opt-in (see the Entity attributes section +above and its Phase-1 limitations). In a real Phase-1 deployment the selector is +unset, so the attribute DAO never activates and attributes stay in the host +entity database. diff --git a/iotdb-thingsboard-table/pom.xml b/iotdb-thingsboard-table/pom.xml index 4f455aa6..43b65a29 100644 --- a/iotdb-thingsboard-table/pom.xml +++ b/iotdb-thingsboard-table/pom.xml @@ -299,6 +299,14 @@ org.apache.tsfile:common org.apache.iotdb:service-rpc org.springframework.boot:spring-boot + + org.apache.commons:commons-lang3 org.junit.jupiter:junit-jupiter-api diff --git a/iotdb-thingsboard-table/src/main/java/org/apache/iotdb/extras/thingsboard/table/IoTDBTableAttributesDao.java b/iotdb-thingsboard-table/src/main/java/org/apache/iotdb/extras/thingsboard/table/IoTDBTableAttributesDao.java index 188419b9..28533d39 100644 --- a/iotdb-thingsboard-table/src/main/java/org/apache/iotdb/extras/thingsboard/table/IoTDBTableAttributesDao.java +++ b/iotdb-thingsboard-table/src/main/java/org/apache/iotdb/extras/thingsboard/table/IoTDBTableAttributesDao.java @@ -18,25 +18,982 @@ package org.apache.iotdb.extras.thingsboard.table; +import org.apache.iotdb.isession.ITableSession; +import org.apache.iotdb.isession.SessionDataSet; import org.apache.iotdb.isession.pool.ITableSessionPool; +import com.google.common.util.concurrent.Futures; +import com.google.common.util.concurrent.ListenableFuture; +import com.google.common.util.concurrent.SettableFuture; +import com.google.common.util.concurrent.Striped; import lombok.extern.slf4j.Slf4j; +import org.apache.commons.lang3.tuple.Pair; +import org.apache.tsfile.enums.ColumnCategory; +import org.apache.tsfile.enums.TSDataType; +import org.apache.tsfile.write.record.Tablet; +import org.springframework.beans.factory.DisposableBean; +import org.thingsboard.server.common.data.AttributeScope; +import org.thingsboard.server.common.data.id.DeviceProfileId; +import org.thingsboard.server.common.data.id.EntityId; +import org.thingsboard.server.common.data.id.TenantId; +import org.thingsboard.server.common.data.kv.AttributeKvEntry; +import org.thingsboard.server.common.data.kv.BaseAttributeKvEntry; +import org.thingsboard.server.common.data.kv.BooleanDataEntry; +import org.thingsboard.server.common.data.kv.DataType; +import org.thingsboard.server.common.data.kv.DoubleDataEntry; +import org.thingsboard.server.common.data.kv.JsonDataEntry; +import org.thingsboard.server.common.data.kv.KvEntry; +import org.thingsboard.server.common.data.kv.LongDataEntry; +import org.thingsboard.server.common.data.kv.StringDataEntry; +import org.thingsboard.server.common.data.util.TbPair; +import org.thingsboard.server.dao.attributes.AttributesDao; +import org.thingsboard.server.dao.model.sql.AttributeKvEntity; + +import java.util.ArrayList; +import java.util.Collection; +import java.util.Collections; +import java.util.List; +import java.util.Objects; +import java.util.Optional; +import java.util.Set; +import java.util.UUID; +import java.util.concurrent.ArrayBlockingQueue; +import java.util.concurrent.Callable; +import java.util.concurrent.ConcurrentHashMap; +import java.util.concurrent.RejectedExecutionException; +import java.util.concurrent.ThreadFactory; +import java.util.concurrent.ThreadPoolExecutor; +import java.util.concurrent.TimeUnit; +import java.util.concurrent.atomic.AtomicBoolean; +import java.util.concurrent.atomic.AtomicInteger; +import java.util.concurrent.locks.Lock; +import java.util.concurrent.locks.ReadWriteLock; /** - * Attribute DAO skeleton for the IoTDB Table Mode backend. + * Entity-attribute DAO for the IoTDB Table Mode backend. + * + *

Spring activation: {@code database.attributes.type=iotdb-table}. NOTE: this activation + * property is the Phase-1 selector pending upstream ThingsBoard confirmation (open Q6 / ThingsBoard + * Discussion #15296); upstream ThingsBoard does not yet expose an {@code AttributesDao} selector, + * so the DAO is inert by default (no real Phase-1 deployment sets {@code + * database.attributes.type}, so the {@link IoTDBTableAttributesEnabledCondition} stays false and + * the bean is never instantiated). The selector is independent of {@code database.ts.type} / {@code + * database.ts_latest.type} (the attribute DAO routes separately); if Q6 resolves to a different + * property, the condition is updated. See the GSOC-304 design doc section 6.0 and the Wk5 decision + * note section 7. This is a Path-3 stretch artifact: Phase-1 attributes stay in the host entity DB. + * + *

This DAO is wired as an explicit {@code @Bean} in {@link IoTDBTableConfiguration} (guarded by + * the activation property) rather than via component scanning, so the {@code ITableSessionPool} + * constructor parameter is guaranteed to be created first; that avoids the + * {@code @ConditionalOnBean} bean-definition ordering trap, where a component-scanned bean can be + * condition-evaluated before the imported configuration's pool {@code @Bean} is registered. + * + *

Current-state (latest-only) contract on the historical-shaped {@code entity_attributes} table + * (design doc section 3.5): each identity tuple {@code (tenant_id, entity_type, entity_id, + * attribute_scope, key)} holds exactly one current row, so: + * + *

+ * + *

The {@code COLUMN_NAMES}/{@code DATA_TYPES}/{@code COLUMN_CATEGORIES} arrays below follow the + * {@code entity_attributes} DDL tag order in {@code schema-iotdb-table.sql} (attribute_scope, + * entity_type, tenant_id, key, entity_id, then the five typed FIELDs). The {@code time TIMESTAMP + * TIME} column is written via {@link Tablet#addTimestamp}, never as a {@code ColumnCategory.TIME} + * tablet column, so the three parallel arrays cover exactly the 10 non-time columns and must stay + * positionally aligned. + * + *

Every {@code SELECT}/{@code DELETE} predicate is keyed on the full identity (tenant_id + + * entity_type + entity_id, plus {@code attribute_scope} where scope-scoped, plus {@code key}) — a + * deliberate SUPERSET of ThingsBoard's relational {@code JpaAttributeDao}, which keys an attribute + * row by the entity UUID alone. The extra tag predicates are required because {@code + * entity_attributes} is a single multi-tenant / multi-entity IoTDB table; they prevent any + * cross-tenant, cross-entity, or cross-scope read or delete. + * + *

Phase-1 honest limitations (documented, not silently degraded): * - *

Spring activation is intentionally absent because upstream ThingsBoard does not expose an - * AttributesDao selector yet. The activation property and SPI binding should be introduced together - * when the attributes path is implemented. + *

* - *

Strategy F keeps this class free of ThingsBoard imports and interface clauses until the - * attribute path is implemented. + * @see "GSOC-304 design doc section 3.5 / 6.0" + * @see "GSOC-304 Wk5 attributes decision note" + * @since GSOC-304 Wk 5 attributes DAO */ @Slf4j -public class IoTDBTableAttributesDao extends IoTDBTableBaseDao { - // Not annotated @Repository yet: this must not auto-register until ThingsBoard exposes an - // AttributesDao selector and this module provides the matching implementation. - public IoTDBTableAttributesDao(ITableSessionPool tableSessionPool) { +public class IoTDBTableAttributesDao extends IoTDBTableBaseDao + implements AttributesDao, DisposableBean { + + static final String TABLE_NAME = "entity_attributes"; + + // NUL is the identity-lock key separator: it cannot appear in any tenant/entity UUID, + // entity-type/scope enum name, or telemetry key, so distinct identities can never collide + // into the same Striped lock stripe by string concatenation. + private static final char LOCK_KEY_SEPARATOR = '\u0000'; + private static final String SELECT_TYPED_COLUMNS = + "time, bool_v, long_v, double_v, str_v, json_v"; + + // The three parallel arrays below follow the entity_attributes DDL tag order + // (schema-iotdb-table.sql): attribute_scope, entity_type, tenant_id, key, entity_id (TAGs), + // then bool_v, long_v, double_v, str_v, json_v (FIELDs). They must stay positionally aligned and + // cover exactly the 10 non-time columns; the `time TIMESTAMP TIME` column is written through + // Tablet#addTimestamp (NOT a ColumnCategory.TIME entry), so COLUMN_CATEGORIES holds only TAG and + // FIELD. Rebuilding with a different tag order is a correctness bug (Wk5 risk: TAG-order rot). + private static final List COLUMN_NAMES = + List.of( + "attribute_scope", + "entity_type", + "tenant_id", + "key", + "entity_id", + "bool_v", + "long_v", + "double_v", + "str_v", + "json_v"); + private static final List DATA_TYPES = + List.of( + TSDataType.STRING, + TSDataType.STRING, + TSDataType.STRING, + TSDataType.STRING, + TSDataType.STRING, + TSDataType.BOOLEAN, + TSDataType.INT64, + TSDataType.DOUBLE, + TSDataType.STRING, + TSDataType.TEXT); + private static final List COLUMN_CATEGORIES = + List.of( + ColumnCategory.TAG, + ColumnCategory.TAG, + ColumnCategory.TAG, + ColumnCategory.TAG, + ColumnCategory.TAG, + ColumnCategory.FIELD, + ColumnCategory.FIELD, + ColumnCategory.FIELD, + ColumnCategory.FIELD, + ColumnCategory.FIELD); + + private final ThreadPoolExecutor ioExecutor; + private final Set> ioTasks = ConcurrentHashMap.newKeySet(); + private final AtomicBoolean accepting = new AtomicBoolean(true); + private final AtomicBoolean destroyed = new AtomicBoolean(false); + private final long shutdownDrainTimeoutMs; + // Per-identity write serialization (single-JVM convergence, design doc section 3.5). + private final Striped identityLocks = Striped.lock(256); + // Per-ENTITY read/write serialization guarding entity-wide removeAllByEntityId against concurrent + // single-identity save/delete. A per-key mutate (save/deleteIdentity) takes the entity READ lock + // (shared: many keys of the same entity may proceed concurrently); removeAllByEntityId takes the + // entity WRITE lock (exclusive) around its select+delete so no save can re-INSERT a key between + // the entity-wide select and delete. Lock ordering is ALWAYS entity-lock (outer) then + // per-identity lock (inner); removeAllByEntityId never takes an identity lock, so no cycle is + // possible. + private final Striped entityLocks = Striped.readWriteLock(256); + + public IoTDBTableAttributesDao(ITableSessionPool tableSessionPool, IoTDBTableConfig config) { super(tableSessionPool); + // Section 3.5 cluster opt-in validator: when the attribute DAO is active the operator must + // acknowledge cluster routing explicitly, because the delete-then-insert write path converges + // only within a single JVM. This is intentionally independent of ts.type / ts_latest.type + // (the attribute DAO routes separately). Fail fast at construction on an absent/invalid value. + requireClusterModeAcknowledged(config.getAttributes().getClusterMode()); + this.shutdownDrainTimeoutMs = config.getTs().getSave().getShutdownDrainTimeoutMs(); + int ioThreads = config.getTs().getRead().getThreads(); + int ioQueueCapacity = config.getTs().getRead().getQueueCapacity(); + this.ioExecutor = + new ThreadPoolExecutor( + ioThreads, + ioThreads, + 0L, + TimeUnit.MILLISECONDS, + new ArrayBlockingQueue<>(ioQueueCapacity), + ioThreadFactory(), + new ThreadPoolExecutor.AbortPolicy()); + } + + // ---- save: delete-then-insert under a per-identity lock ---- + + @Override + public ListenableFuture save( + TenantId tenantId, + EntityId entityId, + AttributeScope attributeScope, + AttributeKvEntry attribute) { + Objects.requireNonNull(tenantId, "tenantId"); + Objects.requireNonNull(entityId, "entityId"); + Objects.requireNonNull(attributeScope, "attributeScope"); + Objects.requireNonNull(attribute, "attribute"); + String key = requireKey(attribute.getKey()); + Lock entityReadLock = entityLock(tenantId, entityId).readLock(); + Lock lock = identityLock(tenantId, entityId, attributeScope, key); + return submitIoTask( + () -> { + // Lock ordering: entity read lock (outer) then per-identity lock (inner). The shared read + // lock lets concurrent saves of different keys on the same entity proceed, while + // excluding a concurrent entity-wide removeAllByEntityId (which holds the entity write + // lock). + entityReadLock.lock(); + try { + lock.lock(); + try { + doSave(tenantId, entityId, attributeScope, attribute, key); + } finally { + lock.unlock(); + } + } finally { + entityReadLock.unlock(); + } + // IoTDB has no sequence; Phase-1 returns a null version (see class javadoc). + return null; + }); + } + + private void doSave( + TenantId tenantId, + EntityId entityId, + AttributeScope attributeScope, + AttributeKvEntry attribute, + String key) + throws Exception { + String deleteSql = buildDeleteSql(tenantId, entityId, attributeScope, key); + Tablet tablet = buildTablet(tenantId, entityId, attributeScope, attribute, key); + try (ITableSession session = tableSessionPool.getSession()) { + // Step 1: tag-only DELETE removes the identity across all time (no time predicate); proven + // legal and full-range by the IoTDB analyzer (see Wk5 decision note evidence baseline). + session.executeNonQueryStatement(deleteSql); + // Step 2: insert the single current row at time = lastUpdateTs with one typed FIELD set. + session.insert(tablet); + } + } + + private Tablet buildTablet( + TenantId tenantId, + EntityId entityId, + AttributeScope attributeScope, + AttributeKvEntry attribute, + String key) { + Tablet tablet = new Tablet(TABLE_NAME, COLUMN_NAMES, DATA_TYPES, COLUMN_CATEGORIES, 1); + // entity_attributes declares an explicit `time TIMESTAMP TIME` column in DDL, but it is still + // the table's time column: it is written through the normal tablet timestamp mechanism (NOT a + // ColumnCategory.TIME entry). Use the attribute's last-update timestamp as the row time. + tablet.addTimestamp(0, attribute.getLastUpdateTs()); + // TAG values, in the DDL tag order (attribute_scope, entity_type, tenant_id, key, entity_id). + tablet.addValue("attribute_scope", 0, attributeScope.name()); + tablet.addValue("entity_type", 0, entityId.getEntityType().name()); + tablet.addValue("tenant_id", 0, tenantId.getId().toString()); + tablet.addValue("key", 0, key); + tablet.addValue("entity_id", 0, entityId.getId().toString()); + // FIELD values: exactly one typed column is non-null, chosen by the attribute's DataType. + DataType dataType = attribute.getDataType(); + tablet.addValue("bool_v", 0, dataType == DataType.BOOLEAN ? attribute.getValue() : null); + tablet.addValue("long_v", 0, dataType == DataType.LONG ? attribute.getValue() : null); + tablet.addValue("double_v", 0, dataType == DataType.DOUBLE ? attribute.getValue() : null); + tablet.addValue("str_v", 0, dataType == DataType.STRING ? attribute.getValue() : null); + tablet.addValue("json_v", 0, dataType == DataType.JSON ? attribute.getValue() : null); + tablet.setRowSize(1); + return tablet; + } + + // ---- find / findAll: synchronous reads on the calling thread ---- + + @Override + public Optional find( + TenantId tenantId, EntityId entityId, AttributeScope attributeScope, String attributeKey) { + Objects.requireNonNull(tenantId, "tenantId"); + Objects.requireNonNull(entityId, "entityId"); + Objects.requireNonNull(attributeScope, "attributeScope"); + String key = requireKey(attributeKey); + // Take the same per-identity lock save() holds across its (non-atomic) delete+insert, so a + // concurrent point-read observes either the old or the new value, never the transient empty gap + // between the DELETE and the INSERT. Deadlock-safe: find takes ONLY the identity lock (never + // the + // entity lock), so it cannot form a cycle with the entity-then-identity ordering used by save() + // and removeAllByEntityId(). Full-scope reads (find-by-keys / findAll) stay best-effort; see + // the + // README limitation. + Lock lock = identityLock(tenantId, entityId, attributeScope, key); + lock.lock(); + try { + return doFind(tenantId, entityId, attributeScope, key); + } catch (RuntimeException e) { + throw e; + } catch (Exception e) { + throw new IllegalStateException("Failed to read entity attribute", e); + } finally { + lock.unlock(); + } + } + + private Optional doFind( + TenantId tenantId, EntityId entityId, AttributeScope attributeScope, String key) + throws Exception { + String sql = buildFindSql(tenantId, entityId, attributeScope, key); + try (ITableSession session = tableSessionPool.getSession(); + SessionDataSet dataSet = session.executeQueryStatement(sql)) { + SessionDataSet.DataIterator row = dataSet.iterator(); + if (!row.next()) { + return Optional.empty(); + } + // B1 fail-fast: getEntry throws IllegalStateException if more than one typed column is set. + // delete-then-insert guarantees a single typed FIELD per identity, so this is defensive. + TypedKvValue value = getEntry(row); + if (!value.hasValue()) { + return Optional.empty(); + } + long ts = row.getTimestamp("time").getTime(); + return Optional.of(new BaseAttributeKvEntry(kvEntry(key, value), ts, null)); + } + } + + @Override + public List find( + TenantId tenantId, + EntityId entityId, + AttributeScope attributeScope, + Collection attributeKeys) { + Objects.requireNonNull(tenantId, "tenantId"); + Objects.requireNonNull(entityId, "entityId"); + Objects.requireNonNull(attributeScope, "attributeScope"); + Objects.requireNonNull(attributeKeys, "attributeKeys"); + if (attributeKeys.isEmpty()) { + return List.of(); + } + try { + return doFindKeyed(buildFindKeysSql(tenantId, entityId, attributeScope, attributeKeys)); + } catch (RuntimeException e) { + throw e; + } catch (Exception e) { + throw new IllegalStateException("Failed to read entity attributes by keys", e); + } + } + + @Override + public List findAll( + TenantId tenantId, EntityId entityId, AttributeScope attributeScope) { + Objects.requireNonNull(tenantId, "tenantId"); + Objects.requireNonNull(entityId, "entityId"); + Objects.requireNonNull(attributeScope, "attributeScope"); + try { + return doFindKeyed(buildFindAllSql(tenantId, entityId, attributeScope)); + } catch (RuntimeException e) { + throw e; + } catch (Exception e) { + throw new IllegalStateException("Failed to read all entity attributes", e); + } + } + + private List doFindKeyed(String sql) throws Exception { + List entries = new ArrayList<>(); + try (ITableSession session = tableSessionPool.getSession(); + SessionDataSet dataSet = session.executeQueryStatement(sql)) { + SessionDataSet.DataIterator row = dataSet.iterator(); + while (row.next()) { + String key = row.getString("key"); + TypedKvValue value = getEntry(row); + if (!value.hasValue()) { + continue; + } + long ts = row.getTimestamp("time").getTime(); + entries.add(new BaseAttributeKvEntry(kvEntry(key, value), ts, null)); + } + } + return entries; + } + + // ---- removeAll / removeAllWithVersions: per-key future + tag-only DELETE ---- + + @Override + public List> removeAll( + TenantId tenantId, EntityId entityId, AttributeScope attributeScope, List keys) { + Objects.requireNonNull(tenantId, "tenantId"); + Objects.requireNonNull(entityId, "entityId"); + Objects.requireNonNull(attributeScope, "attributeScope"); + Objects.requireNonNull(keys, "keys"); + // Validate every key before launching any delete task, so a blank key never leaves a partial + // delete behind (fail-fast before any side effect). + List validatedKeys = new ArrayList<>(keys.size()); + for (String rawKey : keys) { + validatedKeys.add(requireKey(rawKey)); + } + List> futures = new ArrayList<>(validatedKeys.size()); + for (String key : validatedKeys) { + futures.add( + submitIoTask( + () -> { + deleteIdentity(tenantId, entityId, attributeScope, key); + return key; + })); + } + return futures; + } + + @Override + public List>> removeAllWithVersions( + TenantId tenantId, EntityId entityId, AttributeScope attributeScope, List keys) { + Objects.requireNonNull(tenantId, "tenantId"); + Objects.requireNonNull(entityId, "entityId"); + Objects.requireNonNull(attributeScope, "attributeScope"); + Objects.requireNonNull(keys, "keys"); + // Validate every key before launching any delete task, so a blank key never leaves a partial + // delete behind (fail-fast before any side effect). + List validatedKeys = new ArrayList<>(keys.size()); + for (String rawKey : keys) { + validatedKeys.add(requireKey(rawKey)); + } + List>> futures = new ArrayList<>(validatedKeys.size()); + for (String key : validatedKeys) { + futures.add( + submitIoTask( + () -> { + deleteIdentity(tenantId, entityId, attributeScope, key); + // null version: IoTDB has no sequence (see class javadoc). The ThingsBoard service + // skips the EDQS attribute-delete notification when the version is null. + return TbPair.of(key, (Long) null); + })); + } + return futures; + } + + private void deleteIdentity( + TenantId tenantId, EntityId entityId, AttributeScope attributeScope, String key) + throws Exception { + // Lock ordering: entity read lock (outer) then per-identity lock (inner), matching save(), so a + // single-key delete is excluded from a concurrent entity-wide removeAllByEntityId. + Lock entityReadLock = entityLock(tenantId, entityId).readLock(); + Lock lock = identityLock(tenantId, entityId, attributeScope, key); + entityReadLock.lock(); + try { + lock.lock(); + try { + String sql = buildDeleteSql(tenantId, entityId, attributeScope, key); + try (ITableSession session = tableSessionPool.getSession()) { + session.executeNonQueryStatement(sql); + } + } finally { + lock.unlock(); + } + } finally { + entityReadLock.unlock(); + } + } + + // ---- findNextBatch: deferred relational migration helper ---- + + @Override + public List findNextBatch( + UUID entityId, int attributeType, int attributeKey, int batchSize) { + throw new UnsupportedOperationException( + "findNextBatch is a relational keyset-pagination migration helper not supported by the " + + "IoTDB Table Mode backend; see Wk5 decision note section 6"); + } + + // ---- key discovery + bulk latest read + removeAllByEntityId ---- + + @Override + public List findAllKeysByEntityIds(TenantId tenantId, List entityIds) { + Objects.requireNonNull(tenantId, "tenantId"); + Objects.requireNonNull(entityIds, "entityIds"); + if (entityIds.isEmpty()) { + return List.of(); + } + try { + return doFindDistinctKeys(buildKeysByEntityIdsSql(tenantId, entityIds, null)); + } catch (RuntimeException e) { + throw e; + } catch (Exception e) { + throw new IllegalStateException("Failed to read attribute keys by entity ids", e); + } + } + + @Override + public List findAllKeysByEntityIdsAndScope( + TenantId tenantId, List entityIds, AttributeScope scope) { + Objects.requireNonNull(tenantId, "tenantId"); + Objects.requireNonNull(entityIds, "entityIds"); + Objects.requireNonNull(scope, "scope"); + if (entityIds.isEmpty()) { + return List.of(); + } + try { + return doFindDistinctKeys(buildKeysByEntityIdsSql(tenantId, entityIds, scope)); + } catch (RuntimeException e) { + throw e; + } catch (Exception e) { + throw new IllegalStateException("Failed to read attribute keys by entity ids and scope", e); + } + } + + @Override + public ListenableFuture> findAllKeysByEntityIdsAndScopeAsync( + TenantId tenantId, List entityIds, AttributeScope scope) { + Objects.requireNonNull(tenantId, "tenantId"); + Objects.requireNonNull(entityIds, "entityIds"); + Objects.requireNonNull(scope, "scope"); + if (entityIds.isEmpty()) { + return Futures.immediateFuture(List.of()); + } + // [v4.3.1.2] async wrapper: run the same distinct-key read on the bounded IO executor. + return submitIoTask( + () -> doFindDistinctKeys(buildKeysByEntityIdsSql(tenantId, entityIds, scope))); + } + + @Override + public List findLatestByEntityIdsAndScope( + TenantId tenantId, List entityIds, AttributeScope scope) { + Objects.requireNonNull(tenantId, "tenantId"); + Objects.requireNonNull(entityIds, "entityIds"); + Objects.requireNonNull(scope, "scope"); + if (entityIds.isEmpty()) { + return List.of(); + } + // [v4.3.1.2] bulk latest read: each identity holds exactly one current row (delete-then-insert + // convergence), so a single SELECT over the entity OR-set in this scope already returns the + // latest value per (entity, key). Best-effort (unlocked) like find(keys)/findAll; see README. + try { + return doFindKeyed(buildLatestByEntityIdsSql(tenantId, entityIds, scope)); + } catch (RuntimeException e) { + throw e; + } catch (Exception e) { + throw new IllegalStateException( + "Failed to read latest attributes by entity ids and scope", e); + } + } + + @Override + public ListenableFuture> findLatestByEntityIdsAndScopeAsync( + TenantId tenantId, List entityIds, AttributeScope scope) { + Objects.requireNonNull(tenantId, "tenantId"); + Objects.requireNonNull(entityIds, "entityIds"); + Objects.requireNonNull(scope, "scope"); + if (entityIds.isEmpty()) { + return Futures.immediateFuture(List.of()); + } + // [v4.3.1.2] async wrapper: run the same bulk latest read on the bounded IO executor. + return submitIoTask(() -> doFindKeyed(buildLatestByEntityIdsSql(tenantId, entityIds, scope))); + } + + @Override + public List findAllKeysByDeviceProfileId( + TenantId tenantId, DeviceProfileId deviceProfileId) { + Objects.requireNonNull(tenantId, "tenantId"); + // A null deviceProfileId is the "all profiles" path: return the tenant-wide distinct keys, + // which entity_attributes CAN derive (mirrors the Wk4 latest DAO). + if (deviceProfileId == null) { + try { + return doFindDistinctKeys(buildKeysByTenantSql(tenantId)); + } catch (RuntimeException e) { + throw e; + } catch (Exception e) { + throw new IllegalStateException("Failed to read attribute keys by tenant", e); + } + } + // Non-null profile lookup returns an empty list, matching the official non-relational backend: + // CassandraBaseTimeseriesLatestDao.findAllKeysByDeviceProfileId also returns + // Collections.emptyList(), because a NoSQL / time-series store cannot do the cross-device-table + // profile-dimension join. The sole upstream caller is DeviceProfileController + // GET /api/deviceProfile/devices/keys/attributes (getAttributesKeys, @PreAuthorize + // TENANT_ADMIN) + // -- a config-time UI key-enumeration endpoint that tolerates an empty result; failing loud + // here + // would 500 under IoTDB while Cassandra returns empty, an avoidable backend inconsistency. + // entity_attributes has no device_profile_id tag and the module has no device -> profile + // lookup; + // a real cross-DB implementation is a Phase-2 / Wk9 optional enhancement, not Phase-1. + return Collections.emptyList(); + } + + @Override + public List> removeAllByEntityId( + TenantId tenantId, EntityId entityId) { + Objects.requireNonNull(tenantId, "tenantId"); + Objects.requireNonNull(entityId, "entityId"); + // Best-effort select-then-delete: IoTDB has no DELETE ... RETURNING, so the deleted (scope, + // key) + // pairs are gathered by a SELECT DISTINCT first, then a tag-only DELETE removes every attribute + // row for the entity across all scopes. The entity WRITE lock serializes this whole select+ + // delete against any concurrent same-entity save/deleteIdentity (which hold the entity READ + // lock), so a concurrent save cannot re-INSERT a key between the select and the delete and + // leave + // a row behind after a "remove all". A key inserted by another node (cluster mode) may still be + // deleted but not reported (documented Phase-1 window; see class javadoc). + Lock entityWriteLock = entityLock(tenantId, entityId).writeLock(); + entityWriteLock.lock(); + try { + List> removed = new ArrayList<>(); + String selectSql = buildScopeKeyByEntitySql(tenantId, entityId); + try (ITableSession session = tableSessionPool.getSession(); + SessionDataSet dataSet = session.executeQueryStatement(selectSql)) { + SessionDataSet.DataIterator row = dataSet.iterator(); + while (row.next()) { + AttributeScope scope = AttributeScope.valueOf(row.getString("attribute_scope")); + removed.add(Pair.of(scope, row.getString("key"))); + } + } + String deleteSql = buildDeleteByEntitySql(tenantId, entityId); + try (ITableSession session = tableSessionPool.getSession()) { + session.executeNonQueryStatement(deleteSql); + } + return removed; + } catch (RuntimeException e) { + throw e; + } catch (Exception e) { + throw new IllegalStateException("Failed to remove all attributes by entity id", e); + } finally { + entityWriteLock.unlock(); + } + } + + private List doFindDistinctKeys(String sql) throws Exception { + List keys = new ArrayList<>(); + try (ITableSession session = tableSessionPool.getSession(); + SessionDataSet dataSet = session.executeQueryStatement(sql)) { + SessionDataSet.DataIterator row = dataSet.iterator(); + while (row.next()) { + keys.add(row.getString("key")); + } + } + return keys; + } + + // ---- SQL builders ---- + + private String buildDeleteSql( + TenantId tenantId, EntityId entityId, AttributeScope attributeScope, String key) { + return "DELETE FROM " + + TABLE_NAME + + " WHERE " + + identityPredicate(tenantId, entityId, attributeScope, key); + } + + private String buildFindSql( + TenantId tenantId, EntityId entityId, AttributeScope attributeScope, String key) { + return "SELECT " + + SELECT_TYPED_COLUMNS + + " FROM " + + TABLE_NAME + + " WHERE " + + identityPredicate(tenantId, entityId, attributeScope, key); + } + + private String buildFindKeysSql( + TenantId tenantId, + EntityId entityId, + AttributeScope attributeScope, + Collection keys) { + StringBuilder in = new StringBuilder(); + boolean first = true; + for (String key : keys) { + if (!first) { + in.append(","); + } + in.append(sqlString(requireKey(key))); + first = false; + } + return "SELECT key, " + + SELECT_TYPED_COLUMNS + + " FROM " + + TABLE_NAME + + " WHERE " + + scopePredicate(tenantId, entityId, attributeScope) + + " AND key IN (" + + in + + ")"; + } + + private String buildFindAllSql( + TenantId tenantId, EntityId entityId, AttributeScope attributeScope) { + return "SELECT key, " + + SELECT_TYPED_COLUMNS + + " FROM " + + TABLE_NAME + + " WHERE " + + scopePredicate(tenantId, entityId, attributeScope); + } + + private String buildLatestByEntityIdsSql( + TenantId tenantId, List entityIds, AttributeScope scope) { + StringBuilder sql = + new StringBuilder("SELECT key, ") + .append(SELECT_TYPED_COLUMNS) + .append(" FROM ") + .append(TABLE_NAME) + .append(" WHERE tenant_id=") + .append(sqlString(tenantId.getId().toString())) + .append(" AND attribute_scope=") + .append(sqlString(scope.name())) + .append(" AND ("); + appendEntityOrSet(sql, entityIds); + sql.append(")"); + return sql.toString(); + } + + private String buildKeysByEntityIdsSql( + TenantId tenantId, List entityIds, AttributeScope scope) { + StringBuilder sql = + new StringBuilder("SELECT DISTINCT key FROM ") + .append(TABLE_NAME) + .append(" WHERE tenant_id=") + .append(sqlString(tenantId.getId().toString())); + if (scope != null) { + sql.append(" AND attribute_scope=").append(sqlString(scope.name())); + } + sql.append(" AND ("); + appendEntityOrSet(sql, entityIds); + sql.append(") ORDER BY key"); + return sql.toString(); + } + + private void appendEntityOrSet(StringBuilder sql, List entityIds) { + for (int i = 0; i < entityIds.size(); i++) { + EntityId entityId = Objects.requireNonNull(entityIds.get(i), "entityId"); + if (i > 0) { + sql.append(" OR "); + } + sql.append("(entity_type=") + .append(sqlString(entityId.getEntityType().name())) + .append(" AND entity_id=") + .append(sqlString(entityId.getId().toString())) + .append(")"); + } + } + + private String buildKeysByTenantSql(TenantId tenantId) { + return "SELECT DISTINCT key FROM " + + TABLE_NAME + + " WHERE tenant_id=" + + sqlString(tenantId.getId().toString()) + + " ORDER BY key"; + } + + private String buildScopeKeyByEntitySql(TenantId tenantId, EntityId entityId) { + return "SELECT DISTINCT attribute_scope, key FROM " + + TABLE_NAME + + " WHERE " + + entityPredicate(tenantId, entityId) + + " ORDER BY attribute_scope, key"; + } + + private String buildDeleteByEntitySql(TenantId tenantId, EntityId entityId) { + return "DELETE FROM " + TABLE_NAME + " WHERE " + entityPredicate(tenantId, entityId); + } + + private String identityPredicate( + TenantId tenantId, EntityId entityId, AttributeScope attributeScope, String key) { + return scopePredicate(tenantId, entityId, attributeScope) + " AND key=" + sqlString(key); + } + + private String scopePredicate( + TenantId tenantId, EntityId entityId, AttributeScope attributeScope) { + return entityPredicate(tenantId, entityId) + + " AND attribute_scope=" + + sqlString(attributeScope.name()); + } + + private String entityPredicate(TenantId tenantId, EntityId entityId) { + return "tenant_id=" + + sqlString(tenantId.getId().toString()) + + " AND entity_type=" + + sqlString(entityId.getEntityType().name()) + + " AND entity_id=" + + sqlString(entityId.getId().toString()); + } + + // ---- mapping + helpers ---- + + private KvEntry kvEntry(String key, TypedKvValue value) { + if (value.booleanValue() != null) { + return new BooleanDataEntry(key, value.booleanValue()); + } + if (value.longValue() != null) { + return new LongDataEntry(key, value.longValue()); + } + if (value.doubleValue() != null) { + return new DoubleDataEntry(key, value.doubleValue()); + } + if (value.stringValue() != null) { + return new StringDataEntry(key, value.stringValue()); + } + if (value.jsonValue() != null) { + return new JsonDataEntry(key, value.jsonValue()); + } + throw new IllegalArgumentException("Attribute row does not contain a typed value"); + } + + private Lock identityLock( + TenantId tenantId, EntityId entityId, AttributeScope attributeScope, String key) { + return identityLocks.get( + tenantId.getId().toString() + + LOCK_KEY_SEPARATOR + + entityId.getEntityType().name() + + LOCK_KEY_SEPARATOR + + entityId.getId().toString() + + LOCK_KEY_SEPARATOR + + attributeScope.name() + + LOCK_KEY_SEPARATOR + + key); + } + + private ReadWriteLock entityLock(TenantId tenantId, EntityId entityId) { + return entityLocks.get( + tenantId.getId().toString() + + LOCK_KEY_SEPARATOR + + entityId.getEntityType().name() + + LOCK_KEY_SEPARATOR + + entityId.getId().toString()); + } + + private static String sqlString(String value) { + // Single-quote SQL string literal. This relies on the IoTDB Table-Mode STRING-literal grammar + // (RelationalSql.g4): the single quote is the only special character and is escaped by doubling + // it (''), with NO backslash escaping. If a future IoTDB lexer adds backslash/other escapes, + // this doubling alone would no longer be sufficient and must be revisited. + return "'" + Objects.requireNonNull(value, "value").replace("'", "''") + "'"; + } + + private static String requireKey(String key) { + if (key == null || key.trim().isEmpty()) { + throw new IllegalArgumentException("Attribute key must not be blank"); + } + return key; + } + + private static void requireClusterModeAcknowledged(String clusterMode) { + String mode = clusterMode == null ? null : clusterMode.trim(); + if (mode == null || (!mode.equals("sticky-routing") && !mode.equals("disabled"))) { + throw new IllegalStateException( + "iotdb.attributes.cluster_mode must be explicitly set to 'sticky-routing' or 'disabled' " + + "when the IoTDB Table Mode attribute DAO is active; got '" + + clusterMode + + "'. The attribute write path (delete-then-insert under a per-identity in-JVM lock) " + + "converges only within a single JVM, so cross-node single-writer safety must be " + + "acknowledged. See the GSOC-304 Wk5 decision note section 3.5."); + } + } + + // ---- bounded IO executor (mirrors the Wk4 latest read executor) ---- + + @Override + public void destroy() { + if (!destroyed.compareAndSet(false, true)) { + return; + } + accepting.set(false); + IoTDBTableDaoShuttingDownException failure = shuttingDownException(); + for (Runnable dropped : ioExecutor.shutdownNow()) { + failDroppedIoTask(dropped, failure); + } + try { + ioExecutor.awaitTermination(shutdownDrainTimeoutMs, TimeUnit.MILLISECONDS); + } catch (InterruptedException e) { + Thread.currentThread().interrupt(); + } + for (IoTask task : ioTasks) { + task.fail(failure); + } + } + + private ListenableFuture submitIoTask(Callable callable) { + if (!accepting.get()) { + return Futures.immediateFailedFuture(shuttingDownException()); + } + IoTask task = new IoTask<>(callable); + ioTasks.add(task); + try { + ioExecutor.execute(task); + } catch (RejectedExecutionException e) { + if (!accepting.get() || ioExecutor.isShutdown()) { + task.fail(shuttingDownException()); + } else { + task.fail( + new IoTDBTableReadQueueFullException("IoTDB Table Mode attribute IO queue is full", e)); + } + ioTasks.remove(task); + return task.future(); + } + if (!accepting.get() && ioExecutor.remove(task)) { + task.fail(shuttingDownException()); + ioTasks.remove(task); + } + return task.future(); + } + + private void failDroppedIoTask(Runnable dropped, IoTDBTableDaoShuttingDownException failure) { + if (dropped instanceof IoTask task) { + task.fail(failure); + ioTasks.remove(task); + } + } + + private IoTDBTableDaoShuttingDownException shuttingDownException() { + return new IoTDBTableDaoShuttingDownException( + "IoTDB Table Mode attribute DAO is shutting down"); + } + + private static ThreadFactory ioThreadFactory() { + AtomicInteger sequence = new AtomicInteger(); + return runnable -> { + Thread thread = + new Thread(runnable, "iotdb-table-attributes-io-worker-" + sequence.incrementAndGet()); + thread.setDaemon(true); + return thread; + }; + } + + private final class IoTask implements Runnable { + private final Callable callable; + private final SettableFuture future = SettableFuture.create(); + + private IoTask(Callable callable) { + this.callable = Objects.requireNonNull(callable, "callable"); + } + + @Override + public void run() { + try { + future.set(callable.call()); + } catch (Throwable t) { + future.setException(t); + } finally { + ioTasks.remove(this); + } + } + + private ListenableFuture future() { + return future; + } + + private void fail(Throwable t) { + future.setException(t); + } } } diff --git a/iotdb-thingsboard-table/src/main/java/org/apache/iotdb/extras/thingsboard/table/IoTDBTableAttributesEnabledCondition.java b/iotdb-thingsboard-table/src/main/java/org/apache/iotdb/extras/thingsboard/table/IoTDBTableAttributesEnabledCondition.java new file mode 100644 index 00000000..aa76ab47 --- /dev/null +++ b/iotdb-thingsboard-table/src/main/java/org/apache/iotdb/extras/thingsboard/table/IoTDBTableAttributesEnabledCondition.java @@ -0,0 +1,47 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package org.apache.iotdb.extras.thingsboard.table; + +import org.springframework.context.annotation.Condition; +import org.springframework.context.annotation.ConditionContext; +import org.springframework.core.type.AnnotatedTypeMetadata; + +/** + * Enables the IoTDB Table Mode entity-attribute backend only when the ThingsBoard attribute + * selector {@code database.attributes.type=iotdb-table} is set (case-insensitively, trimmed). + * + *

This selector is INDEPENDENT of {@code database.ts.type} / {@code database.ts_latest.type}: + * the attribute DAO routes separately from the time-series DAOs (a piggy-back on the timeseries + * selector was deliberately rejected; see the Wk5 decision note section 2). Because upstream + * ThingsBoard does not expose an {@code AttributesDao} selector yet, no real Phase-1 deployment + * sets {@code database.attributes.type}; the property is therefore absent in practice, this + * condition returns false, the attribute bean is never instantiated, and attributes keep flowing to + * the host entity-DB {@code AttributesDao}. The DAO is thus inert by default and only activates + * when an operator opts in explicitly (Path-3 stretch artifact). + */ +final class IoTDBTableAttributesEnabledCondition implements Condition { + private static final String SELECTOR_PROPERTY = "database.attributes.type"; + private static final String SELECTOR_VALUE = "iotdb-table"; + + @Override + public boolean matches(ConditionContext context, AnnotatedTypeMetadata metadata) { + String selector = context.getEnvironment().getProperty(SELECTOR_PROPERTY); + return selector != null && SELECTOR_VALUE.equalsIgnoreCase(selector.trim()); + } +} diff --git a/iotdb-thingsboard-table/src/main/java/org/apache/iotdb/extras/thingsboard/table/IoTDBTableConfig.java b/iotdb-thingsboard-table/src/main/java/org/apache/iotdb/extras/thingsboard/table/IoTDBTableConfig.java index 19fe1a5f..b8d8b0c8 100644 --- a/iotdb-thingsboard-table/src/main/java/org/apache/iotdb/extras/thingsboard/table/IoTDBTableConfig.java +++ b/iotdb-thingsboard-table/src/main/java/org/apache/iotdb/extras/thingsboard/table/IoTDBTableConfig.java @@ -77,6 +77,8 @@ public class IoTDBTableConfig { @Valid private Ts ts = new Ts(); + @Valid private Attributes attributes = new Attributes(); + @Data public static class Ts { /** @@ -90,6 +92,24 @@ public static class Ts { @Valid private Read read = new Read(); } + /** + * Entity-attribute DAO configuration, bound from {@code iotdb.attributes.*}. Independent of + * {@code iotdb.ts.*} because the attribute DAO routes separately from the time-series DAOs. + */ + @Data + public static class Attributes { + /** + * Cluster routing acknowledgement for the entity-attribute DAO. The IoTDB Table Mode attribute + * write path is delete-then-insert under a per-identity in-JVM lock, which converges only + * inside a single JVM; cross-node single-writer safety is the operator's responsibility. When + * the DAO is activated this must be set explicitly to one of {@code sticky-routing} (all writes + * for a given identity are pinned to one node) or {@code disabled} (single-node / acknowledged + * best-effort); any other value (including the empty default) fails fast at construction. See + * the GSOC-304 Wk5 decision note section 3.5. + */ + private String clusterMode = ""; + } + @Data public static class Read { @Min(1) diff --git a/iotdb-thingsboard-table/src/main/java/org/apache/iotdb/extras/thingsboard/table/IoTDBTableConfiguration.java b/iotdb-thingsboard-table/src/main/java/org/apache/iotdb/extras/thingsboard/table/IoTDBTableConfiguration.java index c3977045..d898001a 100644 --- a/iotdb-thingsboard-table/src/main/java/org/apache/iotdb/extras/thingsboard/table/IoTDBTableConfiguration.java +++ b/iotdb-thingsboard-table/src/main/java/org/apache/iotdb/extras/thingsboard/table/IoTDBTableConfiguration.java @@ -68,8 +68,14 @@ public class IoTDBTableConfiguration { static final String IOTDB_TABLE_SESSION_POOL_BEAN_NAME = "iotdbThingsboardTableSessionPool"; static final String IOTDB_TABLE_TIMESERIES_DAO_BEAN_NAME = "ioTDBTableTimeseriesDao"; + static final String IOTDB_TABLE_LATEST_DAO_BEAN_NAME = "ioTDBTableLatestDao"; + static final String IOTDB_TABLE_ATTRIBUTES_DAO_BEAN_NAME = "ioTDBTableAttributesDao"; static final String TIMESERIES_DAO_CLASS_NAME = "org.thingsboard.server.dao.timeseries.TimeseriesDao"; + static final String TIMESERIES_LATEST_DAO_CLASS_NAME = + "org.thingsboard.server.dao.timeseries.TimeseriesLatestDao"; + static final String ATTRIBUTES_DAO_CLASS_NAME = + "org.thingsboard.server.dao.attributes.AttributesDao"; @Configuration(proxyBeanMethods = false) @ConditionalOnClass(name = TIMESERIES_DAO_CLASS_NAME) @@ -77,33 +83,16 @@ public class IoTDBTableConfiguration { @EnableConfigurationProperties(IoTDBTableConfig.class) static class EnabledRawOnlyConfiguration { - // This module implements only the timeseries backend. The latest-telemetry - // (database.ts_latest.type) and label (iotdb.labels.enabled) selectors are intentionally NOT - // included here: those DAOs do not exist yet, so they must not spin up a session pool or schema - // bootstrap for a backend that has not shipped. Those conditions return when the corresponding - // DAOs are implemented. + // The session pool, writer and schema bootstrap are owned by the timeseries backend + // (database.ts.type=iotdb-table + iotdb.ts.experimental-raw-only). The derived-latest DAO + // (database.ts_latest.type) is registered below and REUSES this pool; it adds the latest + // selector on top of the timeseries selector (see IoTDBTableLatestEnabledCondition), so it can + // never activate without the writer that populates the telemetry table it reads. The label + // (iotdb.labels.enabled) selector returns when that DAO is implemented. @Bean(name = IOTDB_TABLE_SESSION_POOL_BEAN_NAME, destroyMethod = "close") @ConditionalOnMissingBean(name = IOTDB_TABLE_SESSION_POOL_BEAN_NAME) ITableSessionPool tableSessionPool(IoTDBTableConfig config) { - String nodeUrl = config.getHost() + ":" + config.getPort(); - ITableSessionPool pool = - new TableSessionPoolBuilder() - .nodeUrls(List.of(nodeUrl)) - .user(config.getUsername()) - .password(config.getPassword()) - .database(config.getDatabase()) - .maxSize(config.getSessionPoolSize()) - .connectionTimeoutInMs(config.getConnectionTimeoutMs()) - .enableCompression(config.isEnableCompression()) - .build(); - log.info( - "IoTDB Table Mode session pool initialized: nodeUrl={}, database={}, poolSize={}, compression={}, defaultTtlMs(storageAccountingOnly)={}", - nodeUrl, - config.getDatabase(), - config.getSessionPoolSize(), - config.isEnableCompression(), - config.getDefaultTtlMs()); - return pool; + return buildSessionPool(config); } @Bean @@ -137,6 +126,39 @@ IoTDBTableTimeseriesDao ioTDBTableTimeseriesDao( return new IoTDBTableTimeseriesDao(tableSessionPool, timeseriesWriter, config); } + /** + * Fails startup (only when the latest selector is on) if the IoTDB latest backend is enabled + * but a conflicting non-IoTDB {@code TimeseriesLatestDao} is present, mirroring the fail-fast + * behavior of {@link #timeseriesDaoConflictGuard()} for the historical DAO so the latest path + * does not silently back off to a different backend while the timeseries path runs on IoTDB. + */ + @Bean + @ConditionalOnClass(name = TIMESERIES_LATEST_DAO_CLASS_NAME) + @Conditional(IoTDBTableLatestEnabledCondition.class) + static BeanFactoryPostProcessor timeseriesLatestDaoConflictGuard() { + return new TimeseriesLatestDaoConflictGuard(); + } + + /** + * Registers the derived-latest DAO. It is gated by {@link IoTDBTableLatestEnabledCondition} + * (the timeseries selector plus {@code database.ts_latest.type=iotdb-table}) so it only + * activates when the IoTDB writer that populates the telemetry table is also active, and reuses + * the module-owned named session pool. A conflicting host {@code TimeseriesLatestDao} fails + * startup fast via {@link #timeseriesLatestDaoConflictGuard()} rather than silently shadowing + * this DAO; the string-based missing-bean guard keeps auto-config metadata evaluation from + * loading ThingsBoard classes. + */ + @Bean + @ConditionalOnClass(name = TIMESERIES_LATEST_DAO_CLASS_NAME) + @ConditionalOnBean(name = IOTDB_TABLE_SESSION_POOL_BEAN_NAME) + @Conditional(IoTDBTableLatestEnabledCondition.class) + @ConditionalOnMissingBean(type = TIMESERIES_LATEST_DAO_CLASS_NAME) + IoTDBTableLatestDao ioTDBTableLatestDao( + @Qualifier(IOTDB_TABLE_SESSION_POOL_BEAN_NAME) ITableSessionPool tableSessionPool, + IoTDBTableConfig config) { + return new IoTDBTableLatestDao(tableSessionPool, config); + } + /** * Idempotent startup schema bootstrap. Only registered when the IoTDB Table Mode backend is * selected and explicitly enabled (same activation guard as the pool/DAO), the session pool @@ -154,6 +176,128 @@ IoTDBTableSchemaBootstrap schemaBootstrap( IoTDBTableConfig config) { return new IoTDBTableSchemaBootstrap(tableSessionPool, config); } + + /** + * Second idempotent startup schema bootstrap that creates the {@code telemetry_latest} overlay + * table from {@code schema-iotdb-table-latest.sql}. It is registered ONLY when the + * derived-latest DAO is active (same {@link IoTDBTableLatestEnabledCondition} guard, the {@code + * TimeseriesLatestDao} class and the module pool are present) and {@code + * iotdb.schema.bootstrap} is not disabled. It deliberately carries NO + * {@code @ConditionalOnMissingBean} so it always runs alongside {@link #schemaBootstrap()} (a + * distinct bean name); both resources are self-contained ({@code CREATE DATABASE IF NOT EXISTS} + * + {@code USE} + {@code CREATE TABLE IF NOT EXISTS}), so the two bootstrap beans are + * order-independent and idempotent. When the latest selector is off, the overlay table is never + * created (latest path stays inert). + */ + @Bean + @ConditionalOnClass(name = TIMESERIES_LATEST_DAO_CLASS_NAME) + @ConditionalOnBean(name = IOTDB_TABLE_SESSION_POOL_BEAN_NAME) + @Conditional(IoTDBTableLatestEnabledCondition.class) + @ConditionalOnProperty( + name = "iotdb.schema.bootstrap", + havingValue = "true", + matchIfMissing = true) + IoTDBTableSchemaBootstrap latestSchemaBootstrap( + @Qualifier(IOTDB_TABLE_SESSION_POOL_BEAN_NAME) ITableSessionPool tableSessionPool, + IoTDBTableConfig config) { + return new IoTDBTableSchemaBootstrap( + tableSessionPool, config, IoTDBTableSchemaBootstrap.LATEST_SCHEMA_RESOURCE); + } + } + + /** + * Entity-attribute backend, activated INDEPENDENTLY by {@code + * database.attributes.type=iotdb-table} (see {@link IoTDBTableAttributesEnabledCondition}). It is + * a separate inner configuration from {@link EnabledRawOnlyConfiguration} because the attribute + * DAO routes separately from the time-series DAOs: it must be able to activate on its own + * (attributes selector set, ts selectors unset) and must stay inert when no attributes selector + * is present. Because no shipped ThingsBoard release exposes {@code database.attributes.type}, + * the default Phase-1 deployment leaves it unset, this configuration is skipped, no session pool + * or attribute bean is created, and attributes keep flowing to the host entity-DB {@code + * AttributesDao} (Path-3 stretch, inert by default). + * + *

The session pool / schema bootstrap beans here reuse the same bean name as {@link + * EnabledRawOnlyConfiguration} and carry {@code @ConditionalOnMissingBean(name=...)}, so when + * both the timeseries and attributes selectors are on exactly one shared pool/bootstrap is + * created; when only the attributes selector is on this configuration brings them up on its own. + */ + @Configuration(proxyBeanMethods = false) + @ConditionalOnClass(name = ATTRIBUTES_DAO_CLASS_NAME) + @Conditional(IoTDBTableAttributesEnabledCondition.class) + @EnableConfigurationProperties(IoTDBTableConfig.class) + static class EnabledAttributesConfiguration { + + @Bean(name = IOTDB_TABLE_SESSION_POOL_BEAN_NAME, destroyMethod = "close") + @ConditionalOnMissingBean(name = IOTDB_TABLE_SESSION_POOL_BEAN_NAME) + ITableSessionPool tableSessionPool(IoTDBTableConfig config) { + return buildSessionPool(config); + } + + /** + * Fails startup before any IoTDB pool/bootstrap singleton is created if the explicit IoTDB + * attribute backend selection conflicts with a host-provided {@code AttributesDao}, mirroring + * {@code timeseriesDaoConflictGuard()} so the attribute path does not silently shadow a + * different backend. + */ + @Bean + static BeanFactoryPostProcessor attributesDaoConflictGuard() { + return new AttributesDaoConflictGuard(); + } + + /** + * Registers the entity-attribute DAO. The bean name {@code ioTDBTableAttributesDao} matches the + * default component-scan name, the string-based missing-bean guard avoids loading ThingsBoard + * classes while evaluating auto-configuration metadata, and the {@code @Bean} destroy method + * drains the DAO's IO executor on shutdown. + */ + @Bean(name = IOTDB_TABLE_ATTRIBUTES_DAO_BEAN_NAME, destroyMethod = "destroy") + @ConditionalOnBean(name = IOTDB_TABLE_SESSION_POOL_BEAN_NAME) + @ConditionalOnMissingBean(type = ATTRIBUTES_DAO_CLASS_NAME) + IoTDBTableAttributesDao ioTDBTableAttributesDao( + @Qualifier(IOTDB_TABLE_SESSION_POOL_BEAN_NAME) ITableSessionPool tableSessionPool, + IoTDBTableConfig config) { + return new IoTDBTableAttributesDao(tableSessionPool, config); + } + + /** + * Idempotent startup schema bootstrap for the attribute path. Shares the bean name with the + * timeseries configuration via {@code @ConditionalOnMissingBean}, so it only registers when the + * timeseries path has not already registered it. + */ + @Bean + @ConditionalOnBean(name = IOTDB_TABLE_SESSION_POOL_BEAN_NAME) + @ConditionalOnMissingBean(IoTDBTableSchemaBootstrap.class) + @ConditionalOnProperty( + name = "iotdb.schema.bootstrap", + havingValue = "true", + matchIfMissing = true) + IoTDBTableSchemaBootstrap attributesSchemaBootstrap( + @Qualifier(IOTDB_TABLE_SESSION_POOL_BEAN_NAME) ITableSessionPool tableSessionPool, + IoTDBTableConfig config) { + return new IoTDBTableSchemaBootstrap(tableSessionPool, config); + } + } + + private static ITableSessionPool buildSessionPool(IoTDBTableConfig config) { + String nodeUrl = config.getHost() + ":" + config.getPort(); + ITableSessionPool pool = + new TableSessionPoolBuilder() + .nodeUrls(List.of(nodeUrl)) + .user(config.getUsername()) + .password(config.getPassword()) + .database(config.getDatabase()) + .maxSize(config.getSessionPoolSize()) + .connectionTimeoutInMs(config.getConnectionTimeoutMs()) + .enableCompression(config.isEnableCompression()) + .build(); + log.info( + "IoTDB Table Mode session pool initialized: nodeUrl={}, database={}, poolSize={}, compression={}, defaultTtlMs(storageAccountingOnly)={}", + nodeUrl, + config.getDatabase(), + config.getSessionPoolSize(), + config.isEnableCompression(), + config.getDefaultTtlMs()); + return pool; } private static final class TimeseriesDaoConflictGuard implements BeanFactoryPostProcessor { @@ -207,4 +351,109 @@ private static Class resolveTimeseriesDaoClass(ConfigurableListableBeanFactor } } } + + private static final class TimeseriesLatestDaoConflictGuard implements BeanFactoryPostProcessor { + @Override + public void postProcessBeanFactory(ConfigurableListableBeanFactory beanFactory) + throws BeansException { + Class latestDaoType = resolveTimeseriesLatestDaoClass(beanFactory); + for (String beanName : beanFactory.getBeanNamesForType(latestDaoType, true, false)) { + if (!isIoTDBLatestDaoBean(beanFactory, beanName)) { + throw new IllegalStateException( + "database.ts_latest.type=iotdb-table with the IoTDB timeseries backend enabled, but a " + + "non-IoTDB TimeseriesLatestDao bean '" + + beanName + + "' is present; remove it or unset the IoTDB latest selector"); + } + } + } + + private static boolean isIoTDBLatestDaoBean( + ConfigurableListableBeanFactory beanFactory, String beanName) { + Class beanType = resolveBeanType(beanFactory, beanName); + if (beanType == null) { + throw new IllegalStateException( + "database.ts_latest.type=iotdb-table with the IoTDB timeseries backend enabled, but " + + "TimeseriesLatestDao bean '" + + beanName + + "' has no resolvable type; expose a concrete IoTDBTableLatestDao type or " + + "remove the bean"); + } + // beanType is guaranteed non-null here (the null case throws above). + return IoTDBTableLatestDao.class.isAssignableFrom(beanType); + } + + private static Class resolveBeanType( + ConfigurableListableBeanFactory beanFactory, String beanName) { + Class beanType = beanFactory.getType(beanName, false); + if (beanType != null || !beanFactory.containsBeanDefinition(beanName)) { + return beanType; + } + BeanDefinition beanDefinition = beanFactory.getBeanDefinition(beanName); + ResolvableType resolvableType = beanDefinition.getResolvableType(); + return resolvableType == ResolvableType.NONE ? null : resolvableType.resolve(); + } + + private static Class resolveTimeseriesLatestDaoClass( + ConfigurableListableBeanFactory beanFactory) { + try { + return ClassUtils.forName( + TIMESERIES_LATEST_DAO_CLASS_NAME, beanFactory.getBeanClassLoader()); + } catch (ClassNotFoundException e) { + throw new IllegalStateException( + "IoTDB Table Mode backend was enabled but TimeseriesLatestDao is not on the classpath", + e); + } + } + } + + private static final class AttributesDaoConflictGuard implements BeanFactoryPostProcessor { + @Override + public void postProcessBeanFactory(ConfigurableListableBeanFactory beanFactory) + throws BeansException { + Class attributesDaoType = resolveAttributesDaoClass(beanFactory); + for (String beanName : beanFactory.getBeanNamesForType(attributesDaoType, true, false)) { + if (!isIoTDBAttributesDaoBean(beanFactory, beanName)) { + throw new IllegalStateException( + "database.attributes.type=iotdb-table, but a non-IoTDB AttributesDao bean '" + + beanName + + "' is present; remove it or unset the IoTDB attributes selector"); + } + } + } + + private static boolean isIoTDBAttributesDaoBean( + ConfigurableListableBeanFactory beanFactory, String beanName) { + Class beanType = resolveBeanType(beanFactory, beanName); + if (beanType == null) { + throw new IllegalStateException( + "database.attributes.type=iotdb-table, but AttributesDao bean '" + + beanName + + "' has no resolvable type; expose a concrete IoTDBTableAttributesDao type or " + + "remove the bean"); + } + // beanType is guaranteed non-null here (the null case throws above). + return IoTDBTableAttributesDao.class.isAssignableFrom(beanType); + } + + private static Class resolveBeanType( + ConfigurableListableBeanFactory beanFactory, String beanName) { + Class beanType = beanFactory.getType(beanName, false); + if (beanType != null || !beanFactory.containsBeanDefinition(beanName)) { + return beanType; + } + BeanDefinition beanDefinition = beanFactory.getBeanDefinition(beanName); + ResolvableType resolvableType = beanDefinition.getResolvableType(); + return resolvableType == ResolvableType.NONE ? null : resolvableType.resolve(); + } + + private static Class resolveAttributesDaoClass(ConfigurableListableBeanFactory beanFactory) { + try { + return ClassUtils.forName(ATTRIBUTES_DAO_CLASS_NAME, beanFactory.getBeanClassLoader()); + } catch (ClassNotFoundException e) { + throw new IllegalStateException( + "IoTDB Table Mode backend was enabled but AttributesDao is not on the classpath", e); + } + } + } } diff --git a/iotdb-thingsboard-table/src/main/java/org/apache/iotdb/extras/thingsboard/table/IoTDBTableLatestDao.java b/iotdb-thingsboard-table/src/main/java/org/apache/iotdb/extras/thingsboard/table/IoTDBTableLatestDao.java index 60b0c2b2..c66e2f7e 100644 --- a/iotdb-thingsboard-table/src/main/java/org/apache/iotdb/extras/thingsboard/table/IoTDBTableLatestDao.java +++ b/iotdb-thingsboard-table/src/main/java/org/apache/iotdb/extras/thingsboard/table/IoTDBTableLatestDao.java @@ -18,25 +18,716 @@ package org.apache.iotdb.extras.thingsboard.table; +import org.apache.iotdb.isession.ITableSession; +import org.apache.iotdb.isession.SessionDataSet; import org.apache.iotdb.isession.pool.ITableSessionPool; +import com.google.common.util.concurrent.Futures; +import com.google.common.util.concurrent.ListenableFuture; +import com.google.common.util.concurrent.SettableFuture; +import com.google.common.util.concurrent.Striped; import lombok.extern.slf4j.Slf4j; +import org.apache.tsfile.enums.ColumnCategory; +import org.apache.tsfile.enums.TSDataType; +import org.apache.tsfile.write.record.Tablet; +import org.springframework.beans.factory.DisposableBean; +import org.springframework.boot.autoconfigure.condition.ConditionalOnBean; +import org.springframework.context.annotation.Conditional; +import org.springframework.stereotype.Repository; +import org.thingsboard.server.common.data.id.DeviceProfileId; +import org.thingsboard.server.common.data.id.EntityId; +import org.thingsboard.server.common.data.id.TenantId; +import org.thingsboard.server.common.data.kv.BasicTsKvEntry; +import org.thingsboard.server.common.data.kv.BooleanDataEntry; +import org.thingsboard.server.common.data.kv.DataType; +import org.thingsboard.server.common.data.kv.DeleteTsKvQuery; +import org.thingsboard.server.common.data.kv.DoubleDataEntry; +import org.thingsboard.server.common.data.kv.JsonDataEntry; +import org.thingsboard.server.common.data.kv.KvEntry; +import org.thingsboard.server.common.data.kv.LongDataEntry; +import org.thingsboard.server.common.data.kv.StringDataEntry; +import org.thingsboard.server.common.data.kv.TsKvEntry; +import org.thingsboard.server.common.data.kv.TsKvLatestRemovingResult; +import org.thingsboard.server.dao.timeseries.TimeseriesLatestDao; + +import java.util.ArrayList; +import java.util.LinkedHashMap; +import java.util.List; +import java.util.Map; +import java.util.Objects; +import java.util.Optional; +import java.util.concurrent.ArrayBlockingQueue; +import java.util.concurrent.Callable; +import java.util.concurrent.ConcurrentHashMap; +import java.util.concurrent.RejectedExecutionException; +import java.util.concurrent.ThreadFactory; +import java.util.concurrent.ThreadPoolExecutor; +import java.util.concurrent.TimeUnit; +import java.util.concurrent.atomic.AtomicBoolean; +import java.util.concurrent.atomic.AtomicInteger; +import java.util.concurrent.locks.Lock; /** - * Latest telemetry DAO skeleton for the IoTDB Table Mode backend. + * Latest telemetry DAO for the IoTDB Table Mode backend. + * + *

Spring activation ({@link IoTDBTableLatestEnabledCondition}): inert unless ALL of {@code + * database.ts.type=iotdb-table}, {@code database.ts_latest.type=iotdb-table} and {@code + * iotdb.ts.experimental-raw-only=true} are set. Requiring {@code database.ts.type=iotdb-table} (not + * just the latest selector) closes the split-config gap: the derived latest reads the {@code + * telemetry} table that only the IoTDB writer populates, so it must never activate without that + * writer. + * + *

Design (derived latest + a minimal per-key latest overlay): the latest value for an entity/key + * is read from BOTH the historical {@code telemetry} table (derived) AND a small {@code + * telemetry_latest} overlay table, merged by the maximum timestamp per key (the overlay wins an + * exact tie). The derived read stays primary and is engine-accelerated by IoTDB Table Mode's native + * last cache ({@code ORDER BY time DESC LIMIT 1} and {@code LAST_BY(col, time)}); the overlay + * supplements it. + * + *

Why the overlay is written on EVERY {@code saveLatest} (mentor-confirmed, PR-2 review + * flag): the module's historical {@code save()} write path is ASYNCHRONOUS and batched, so at + * the moment {@code saveLatest} runs it cannot see whether a paired {@code telemetry} row will be + * written (a normal full-save) or not (a latest-only write, e.g. the EntityView telemetry-copy + * {@code LATEST_AND_WS} / {@code saveTs=false} path). It therefore cannot distinguish latest-only + * from full-save and writes the overlay UNCONDITIONALLY (delete-then-insert, one row per identity). + * This closes the data-loss gap for latest-only writes that the earlier no-shadow-table design left + * open, at the cost of one extra overlay write per latest update (equivalent to the standard + * ThingsBoard latest-table behavior). The overlay is effectively a latest table, so this partially + * reverses the Phase-1 "no shadow table" choice; it is flagged here for the mentor's PR-2 review. + * + *

+ * + *

Residual Phase-1 limitations (documented, flagged for the mentor's PR-2 review): + * + *

* - *

This class is intentionally not annotated as a Spring bean. The latest-telemetry SPI binding - * is not implemented here, so registering it now would expose a {@code - * database.ts_latest.type=iotdb-table} selector that ThingsBoard cannot bind to a working DAO. + *

The key-discovery SPI methods ({@link #findAllKeysByDeviceProfileId}, {@link + * #findAllKeysByEntityIds}, {@link #findAllKeysByEntityIdsAsync}) are deferred to GSOC-304 Wk 9, + * and the batch {@link #findLatestByEntityIds}/{@link #findLatestByEntityIdsAsync} pair (new in + * ThingsBoard v4.3.1.2) to GSOC-304 Wk 10; all currently throw {@link + * UnsupportedOperationException}. * - *

Strategy F keeps this class free of ThingsBoard imports and interface clauses until the - * latest-telemetry path is implemented. + * @see "GSOC-304 design doc section 6.0" + * @see "GSOC-304 latest-overlay design note" + * @since GSOC-304 Wk 4 latest DAO */ @Slf4j -public class IoTDBTableLatestDao extends IoTDBTableBaseDao { - // Not annotated @Repository yet: auto-registering it now would advertise - // database.ts_latest.type=iotdb-table with no working DAO behind it. - public IoTDBTableLatestDao(ITableSessionPool tableSessionPool) { +@Repository +@ConditionalOnBean(name = IoTDBTableConfiguration.IOTDB_TABLE_SESSION_POOL_BEAN_NAME) +@Conditional(IoTDBTableLatestEnabledCondition.class) +public class IoTDBTableLatestDao extends IoTDBTableBaseDao + implements TimeseriesLatestDao, DisposableBean { + private static final String TABLE_NAME = IoTDBTableTimeseriesWriter.TABLE_NAME; + static final String TABLE_LATEST = "telemetry_latest"; + private static final String SELECT_TYPED_COLUMNS = + "time, bool_v, long_v, double_v, str_v, json_v"; + + // NUL is the identity-lock key separator: it cannot appear in any tenant/entity UUID, + // entity-type enum name, or telemetry key, so distinct identities can never collide into the same + // Striped lock stripe by string concatenation. + private static final char LOCK_KEY_SEPARATOR = '\u0000'; + + // The three parallel arrays below follow the telemetry_latest DDL tag order + // (schema-iotdb-table-latest.sql): entity_type, tenant_id, key, entity_id (TAGs), then bool_v, + // long_v, double_v, str_v, json_v (FIELDs) — the SAME shape as the historical telemetry table, so + // getEntry()/kvEntry() row mapping is reused. They must stay positionally aligned and cover + // exactly the 9 non-time columns; the `time TIMESTAMP TIME` column is written through + // Tablet#addTimestamp (NOT a ColumnCategory.TIME entry). Rebuilding with a different tag order is + // a correctness bug (TAG-order rot). + private static final List COLUMN_NAMES = + List.of( + "entity_type", + "tenant_id", + "key", + "entity_id", + "bool_v", + "long_v", + "double_v", + "str_v", + "json_v"); + private static final List DATA_TYPES = + List.of( + TSDataType.STRING, + TSDataType.STRING, + TSDataType.STRING, + TSDataType.STRING, + TSDataType.BOOLEAN, + TSDataType.INT64, + TSDataType.DOUBLE, + TSDataType.STRING, + TSDataType.TEXT); + private static final List COLUMN_CATEGORIES = + List.of( + ColumnCategory.TAG, + ColumnCategory.TAG, + ColumnCategory.TAG, + ColumnCategory.TAG, + ColumnCategory.FIELD, + ColumnCategory.FIELD, + ColumnCategory.FIELD, + ColumnCategory.FIELD, + ColumnCategory.FIELD); + + private final ThreadPoolExecutor readExecutor; + private final java.util.Set> readTasks = ConcurrentHashMap.newKeySet(); + private final AtomicBoolean accepting = new AtomicBoolean(true); + private final AtomicBoolean destroyed = new AtomicBoolean(false); + private final long shutdownDrainTimeoutMs; + // Per-identity write/snapshot serialization for the overlay (single-JVM convergence). Only + // saveLatest and removeLatest take it; the merged reads (findLatest/findAllLatest) stay + // best-effort/unlocked like the derived reads. + private final Striped identityLocks = Striped.lock(256); + + public IoTDBTableLatestDao(ITableSessionPool tableSessionPool, IoTDBTableConfig config) { super(tableSessionPool); + this.shutdownDrainTimeoutMs = config.getTs().getSave().getShutdownDrainTimeoutMs(); + int readThreads = config.getTs().getRead().getThreads(); + int readQueueCapacity = config.getTs().getRead().getQueueCapacity(); + this.readExecutor = + new ThreadPoolExecutor( + readThreads, + readThreads, + 0L, + TimeUnit.MILLISECONDS, + new ArrayBlockingQueue<>(readQueueCapacity), + readThreadFactory(), + new ThreadPoolExecutor.AbortPolicy()); + } + + @Override + public ListenableFuture> findLatestOpt( + TenantId tenantId, EntityId entityId, String key) { + Objects.requireNonNull(tenantId, "tenantId"); + Objects.requireNonNull(entityId, "entityId"); + String telemetryKey = requireTelemetryKey(key); + return submitReadTask(() -> doFindLatest(tenantId, entityId, telemetryKey)); + } + + @Override + public ListenableFuture findLatest(TenantId tenantId, EntityId entityId, String key) { + Objects.requireNonNull(tenantId, "tenantId"); + Objects.requireNonNull(entityId, "entityId"); + String telemetryKey = requireTelemetryKey(key); + return submitReadTask( + () -> + doFindLatest(tenantId, entityId, telemetryKey) + .orElseGet(() -> nullEntry(telemetryKey))); + } + + @Override + public ListenableFuture> findAllLatest(TenantId tenantId, EntityId entityId) { + Objects.requireNonNull(tenantId, "tenantId"); + Objects.requireNonNull(entityId, "entityId"); + return submitReadTask(() -> doFindAllLatest(tenantId, entityId)); + } + + @Override + public ListenableFuture saveLatest( + TenantId tenantId, EntityId entityId, TsKvEntry tsKvEntry) { + Objects.requireNonNull(tenantId, "tenantId"); + Objects.requireNonNull(entityId, "entityId"); + Objects.requireNonNull(tsKvEntry, "tsKvEntry"); + String key = requireTelemetryKey(tsKvEntry.getKey()); + // The async/batched historical save() cannot be observed here, so saveLatest cannot tell a + // latest-only write from a full-save and writes the per-key overlay UNCONDITIONALLY + // (delete-then-insert under a per-identity lock). This closes the latest-only data-loss gap + // (e.g. EntityView LATEST_AND_WS / saveTs=false) that a no-shadow-table no-op would drop. See + // the class javadoc (overlay rationale + PR-2 review flags). + Lock lock = identityLock(tenantId, entityId, key); + return submitReadTask( + () -> { + lock.lock(); + try { + upsertOverlay(tenantId, entityId, tsKvEntry, key); + } finally { + lock.unlock(); + } + // IoTDB has no sequence; Phase-1 returns a null version (see class javadoc). + return null; + }); + } + + @Override + public ListenableFuture removeLatest( + TenantId tenantId, EntityId entityId, DeleteTsKvQuery query) { + Objects.requireNonNull(tenantId, "tenantId"); + Objects.requireNonNull(entityId, "entityId"); + Objects.requireNonNull(query, "query"); + String telemetryKey = requireTelemetryKey(query.getKey()); + // Overlay-aware remove under the per-identity lock: snapshot the merged latest, and when it is + // inside the half-open [startTs, endTs) delete window delete the overlay row (and optionally + // resurrect the next-older historical value into the overlay). The result is HONEST about + // whether a latest value was actually affected, because TB consumes isRemoved() / getData() as + // real delete/update signals (DefaultTelemetrySubscriptionService.onTimeSeriesDelete). + Lock lock = identityLock(tenantId, entityId, telemetryKey); + return submitReadTask( + () -> { + lock.lock(); + try { + return doRemoveLatest(tenantId, entityId, query, telemetryKey); + } finally { + lock.unlock(); + } + }); + } + + @Override + public List findAllKeysByDeviceProfileId( + TenantId tenantId, DeviceProfileId deviceProfileId) { + // Key discovery is deferred to GSOC-304 Wk 9 (design doc 6.2 "key discovery methods"). + throw new UnsupportedOperationException( + "IoTDB Table Mode latest key discovery not implemented yet (GSOC-304 Wk 9)"); + } + + @Override + public List findAllKeysByEntityIds(TenantId tenantId, List entityIds) { + // Key discovery is deferred to GSOC-304 Wk 9 (design doc 6.2 "key discovery methods"). + throw new UnsupportedOperationException( + "IoTDB Table Mode latest key discovery not implemented yet (GSOC-304 Wk 9)"); + } + + @Override + public ListenableFuture> findAllKeysByEntityIdsAsync( + TenantId tenantId, List entityIds) { + // Key discovery is deferred to GSOC-304 Wk 9 (design doc 6.2 "key discovery methods"). + throw new UnsupportedOperationException( + "IoTDB Table Mode latest key discovery not implemented yet (GSOC-304 Wk 9)"); + } + + @Override + public List findLatestByEntityIds(TenantId tenantId, List entityIds) { + // Batch latest read (new in ThingsBoard v4.3.1.2) is deferred to GSOC-304 Wk 10 + // (design doc 6.2 "findLatestByEntityIds batch optimization"). + throw new UnsupportedOperationException( + "IoTDB Table Mode batch latest read not implemented yet (GSOC-304 Wk 10)"); + } + + @Override + public ListenableFuture> findLatestByEntityIdsAsync( + TenantId tenantId, List entityIds) { + // Batch latest read (new in ThingsBoard v4.3.1.2) is deferred to GSOC-304 Wk 10 + // (design doc 6.2 "findLatestByEntityIds batch optimization"). + throw new UnsupportedOperationException( + "IoTDB Table Mode batch latest read not implemented yet (GSOC-304 Wk 10)"); + } + + @Override + public void destroy() { + if (!destroyed.compareAndSet(false, true)) { + return; + } + accepting.set(false); + IoTDBTableDaoShuttingDownException failure = shuttingDownException(); + for (Runnable dropped : readExecutor.shutdownNow()) { + failDroppedReadTask(dropped, failure); + } + try { + readExecutor.awaitTermination(shutdownDrainTimeoutMs, TimeUnit.MILLISECONDS); + } catch (InterruptedException e) { + Thread.currentThread().interrupt(); + } + for (ReadTask task : readTasks) { + task.fail(failure); + } + } + + // ---- read merge (derived primary + overlay, max-ts-per-key, overlay wins on tie) ---- + + private Optional doFindLatest(TenantId tenantId, EntityId entityId, String key) + throws Exception { + // Derived read FIRST: getEntry fail-fast (IllegalStateException on >1 typed column, the + // documented B1 same-timestamp limitation) propagates so the future fails rather than returning + // bad data; the overlay read (delete-then-insert => single typed column) cannot trip it. + Optional derived = + readLatestRow(buildFindLatestSql(tenantId, entityId, key), key, "time"); + Optional overlay = + readLatestRow(buildFindLatestOverlaySql(tenantId, entityId, key), key, "time"); + return mergeLatest(derived, overlay); + } + + private List doFindAllLatest(TenantId tenantId, EntityId entityId) throws Exception { + Map byKey = new LinkedHashMap<>(); + // Derived: LAST_BY(col, time) + MAX(time) GROUP BY key gives one synthetic sparse row per key. + readEntriesInto(byKey, buildFindAllLatestSql(tenantId, entityId), "last_ts", false); + // Overlay: exactly one row per identity (delete-then-insert). Merge by max ts per key: overlay + // wins an exact tie and supplies any latest-only key the derived store never saw. + readEntriesInto(byKey, buildFindAllLatestOverlaySql(tenantId, entityId), "time", true); + return new ArrayList<>(byKey.values()); + } + + private Optional readLatestRow(String sql, String key, String tsColumn) + throws Exception { + try (ITableSession session = tableSessionPool.getSession(); + SessionDataSet dataSet = session.executeQueryStatement(sql)) { + SessionDataSet.DataIterator row = dataSet.iterator(); + if (!row.next()) { + return Optional.empty(); + } + TypedKvValue value = getEntry(row); + if (!value.hasValue()) { + return Optional.empty(); + } + long ts = row.getTimestamp(tsColumn).getTime(); + return Optional.of(new BasicTsKvEntry(ts, kvEntry(key, value))); + } + } + + private void readEntriesInto( + Map byKey, String sql, String tsColumn, boolean overlayWins) + throws Exception { + try (ITableSession session = tableSessionPool.getSession(); + SessionDataSet dataSet = session.executeQueryStatement(sql)) { + SessionDataSet.DataIterator row = dataSet.iterator(); + while (row.next()) { + String key = row.getString("key"); + // B1 fail-fast: getEntry throws IllegalStateException if a key's row has more than one + // typed column; the whole findAllLatest future then fails rather than silently skipping it. + TypedKvValue value = getEntry(row); + if (!value.hasValue()) { + continue; + } + long ts = row.getTimestamp(tsColumn).getTime(); + TsKvEntry entry = new BasicTsKvEntry(ts, kvEntry(key, value)); + if (overlayWins) { + // remap (existing=derived, incoming=overlay): overlay wins on tie (>=). + byKey.merge( + key, + entry, + (existing, incoming) -> incoming.getTs() >= existing.getTs() ? incoming : existing); + } else { + byKey.put(key, entry); + } + } + } + } + + private static Optional mergeLatest( + Optional derived, Optional overlay) { + if (derived.isEmpty()) { + return overlay; + } + if (overlay.isEmpty()) { + return derived; + } + // Max ts per key; the overlay wins an exact tie (continues the B1 same-timestamp limitation). + return overlay.get().getTs() >= derived.get().getTs() ? overlay : derived; + } + + // ---- overlay write (delete-then-insert) ---- + + private void upsertOverlay(TenantId tenantId, EntityId entityId, TsKvEntry tsKvEntry, String key) + throws Exception { + String deleteSql = buildDeleteLatestSql(tenantId, entityId, key); + Tablet tablet = buildLatestTablet(tenantId, entityId, tsKvEntry, key); + try (ITableSession session = tableSessionPool.getSession()) { + // Step 1: tag-only DELETE removes the identity's overlay row across all time (no time + // predicate). Step 2: insert the single current row at time = tsKvEntry.getTs(). + session.executeNonQueryStatement(deleteSql); + session.insert(tablet); + } + } + + private void deleteOverlay(TenantId tenantId, EntityId entityId, String key) throws Exception { + String sql = buildDeleteLatestSql(tenantId, entityId, key); + try (ITableSession session = tableSessionPool.getSession()) { + session.executeNonQueryStatement(sql); + } + } + + private Tablet buildLatestTablet( + TenantId tenantId, EntityId entityId, TsKvEntry tsKvEntry, String key) { + Tablet tablet = new Tablet(TABLE_LATEST, COLUMN_NAMES, DATA_TYPES, COLUMN_CATEGORIES, 1); + // telemetry_latest declares an explicit `time TIMESTAMP TIME` column in DDL, but it is still + // the table's time column: written through the normal tablet timestamp mechanism (NOT a + // ColumnCategory.TIME entry). Use the entry's timestamp as the row time. + tablet.addTimestamp(0, tsKvEntry.getTs()); + // TAG values, in the DDL tag order (entity_type, tenant_id, key, entity_id) — telemetry shape. + tablet.addValue("entity_type", 0, entityId.getEntityType().name()); + tablet.addValue("tenant_id", 0, tenantId.getId().toString()); + tablet.addValue("key", 0, key); + tablet.addValue("entity_id", 0, entityId.getId().toString()); + // FIELD values: exactly one typed column is non-null, chosen by the entry's DataType. + DataType dataType = tsKvEntry.getDataType(); + tablet.addValue("bool_v", 0, dataType == DataType.BOOLEAN ? tsKvEntry.getValue() : null); + tablet.addValue("long_v", 0, dataType == DataType.LONG ? tsKvEntry.getValue() : null); + tablet.addValue("double_v", 0, dataType == DataType.DOUBLE ? tsKvEntry.getValue() : null); + tablet.addValue("str_v", 0, dataType == DataType.STRING ? tsKvEntry.getValue() : null); + tablet.addValue("json_v", 0, dataType == DataType.JSON ? tsKvEntry.getValue() : null); + tablet.setRowSize(1); + return tablet; + } + + // ---- overlay-aware remove ---- + + private TsKvLatestRemovingResult doRemoveLatest( + TenantId tenantId, EntityId entityId, DeleteTsKvQuery query, String key) throws Exception { + // (1) Snapshot the merged latest (derived + overlay) under the per-identity lock; the overlay + // part of the snapshot is race-free wrt concurrent saveLatest/removeLatest on this identity. + Optional latest = doFindLatest(tenantId, entityId, key); + if (latest.isEmpty()) { + return new TsKvLatestRemovingResult(key, false); + } + long ts = latest.get().getTs(); + boolean inWindow = ts >= query.getStartTs() && ts < query.getEndTs(); + if (!inWindow) { + // (4) the current latest is outside the half-open [startTs, endTs) window: nothing removed, + // so TB is not told to delete a latest value that is still valid. + return new TsKvLatestRemovingResult(key, false); + } + // TB only invokes removeLatest with deleteLatest=true (BaseTimeseriesService gates it); the + // check here is defensive — when false, do not mutate the overlay and report nothing removed. + boolean deleteLatest = !Boolean.FALSE.equals(query.getDeleteLatest()); + if (!deleteLatest) { + return new TsKvLatestRemovingResult(key, false); + } + boolean rewrite = Boolean.TRUE.equals(query.getRewriteLatestIfDeleted()); + if (rewrite) { + // (5) resurrect the next-older historical value (telemetry, time < startTs) as the new latest + // by writing it into the overlay, and return it as data so onTimeSeriesDelete emits a WS + // UPDATE (removed=true, getData()=prior). + Optional prior = doFindHistoryBefore(tenantId, entityId, key, query.getStartTs()); + if (prior.isPresent()) { + upsertOverlay(tenantId, entityId, prior.get(), key); + return new TsKvLatestRemovingResult(prior.get(), null); + } + // No older history to resurrect: fall through to a plain latest delete. + } + // (3 + 6) delete the overlay row for this identity and report a real latest delete (WS DELETE). + deleteOverlay(tenantId, entityId, key); + return new TsKvLatestRemovingResult(key, true, null); + } + + private Optional doFindHistoryBefore( + TenantId tenantId, EntityId entityId, String key, long startTs) throws Exception { + return readLatestRow(buildRewriteHistorySql(tenantId, entityId, key, startTs), key, "time"); + } + + // ---- SQL builders ---- + + private String buildFindLatestSql(TenantId tenantId, EntityId entityId, String key) { + return "SELECT " + + SELECT_TYPED_COLUMNS + + " FROM " + + TABLE_NAME + + " WHERE " + + identityPredicate(tenantId, entityId, key) + + " ORDER BY time DESC LIMIT 1"; + } + + private String buildFindLatestOverlaySql(TenantId tenantId, EntityId entityId, String key) { + return "SELECT " + + SELECT_TYPED_COLUMNS + + " FROM " + + TABLE_LATEST + + " WHERE " + + identityPredicate(tenantId, entityId, key) + + " ORDER BY time DESC LIMIT 1"; + } + + private String buildFindAllLatestSql(TenantId tenantId, EntityId entityId) { + // GROUP BY key projects the key tag plus the per-column LAST_BY aggregate (value at max time) + // and MAX(time) for the entry timestamp. The other tags are fixed by the WHERE clause, so they + // do not need to be (and cannot be) projected as bare columns alongside GROUP BY key. + return "SELECT key," + + " LAST_BY(bool_v, time) AS bool_v," + + " LAST_BY(long_v, time) AS long_v," + + " LAST_BY(double_v, time) AS double_v," + + " LAST_BY(str_v, time) AS str_v," + + " LAST_BY(json_v, time) AS json_v," + + " MAX(time) AS last_ts" + + " FROM " + + TABLE_NAME + + " WHERE " + + entityPredicate(tenantId, entityId) + + " GROUP BY key"; + } + + private String buildFindAllLatestOverlaySql(TenantId tenantId, EntityId entityId) { + // Each identity holds exactly one overlay row (delete-then-insert), so no aggregation is + // needed. + return "SELECT key, " + + SELECT_TYPED_COLUMNS + + " FROM " + + TABLE_LATEST + + " WHERE " + + entityPredicate(tenantId, entityId); + } + + private String buildDeleteLatestSql(TenantId tenantId, EntityId entityId, String key) { + return "DELETE FROM " + TABLE_LATEST + " WHERE " + identityPredicate(tenantId, entityId, key); + } + + private String buildRewriteHistorySql( + TenantId tenantId, EntityId entityId, String key, long startTs) { + return "SELECT " + + SELECT_TYPED_COLUMNS + + " FROM " + + TABLE_NAME + + " WHERE " + + identityPredicate(tenantId, entityId, key) + + " AND time < " + + startTs + + " ORDER BY time DESC LIMIT 1"; + } + + private String identityPredicate(TenantId tenantId, EntityId entityId, String key) { + return entityPredicate(tenantId, entityId) + " AND key=" + sqlString(key); + } + + private String entityPredicate(TenantId tenantId, EntityId entityId) { + return "tenant_id=" + + sqlString(tenantId.getId().toString()) + + " AND entity_type=" + + sqlString(entityId.getEntityType().name()) + + " AND entity_id=" + + sqlString(entityId.getId().toString()); + } + + // ---- mapping + helpers ---- + + private static TsKvEntry nullEntry(String key) { + // SPI contract: findLatest returns this sentinel when the value is not present in the DB. + return new BasicTsKvEntry(System.currentTimeMillis(), new StringDataEntry(key, null)); + } + + private KvEntry kvEntry(String key, TypedKvValue value) { + if (value.booleanValue() != null) { + return new BooleanDataEntry(key, value.booleanValue()); + } + if (value.longValue() != null) { + return new LongDataEntry(key, value.longValue()); + } + if (value.doubleValue() != null) { + return new DoubleDataEntry(key, value.doubleValue()); + } + if (value.stringValue() != null) { + return new StringDataEntry(key, value.stringValue()); + } + if (value.jsonValue() != null) { + return new JsonDataEntry(key, value.jsonValue()); + } + throw new IllegalArgumentException("Telemetry row does not contain a typed value"); + } + + private Lock identityLock(TenantId tenantId, EntityId entityId, String key) { + return identityLocks.get( + tenantId.getId().toString() + + LOCK_KEY_SEPARATOR + + entityId.getEntityType().name() + + LOCK_KEY_SEPARATOR + + entityId.getId().toString() + + LOCK_KEY_SEPARATOR + + key); + } + + private static String sqlString(String value) { + return "'" + Objects.requireNonNull(value, "value").replace("'", "''") + "'"; + } + + private static String requireTelemetryKey(String key) { + if (key == null || key.trim().isEmpty()) { + throw new IllegalArgumentException("Telemetry key must not be blank"); + } + return key; + } + + private ListenableFuture submitReadTask(Callable callable) { + if (!accepting.get()) { + return Futures.immediateFailedFuture(shuttingDownException()); + } + ReadTask task = new ReadTask<>(callable); + readTasks.add(task); + try { + readExecutor.execute(task); + } catch (RejectedExecutionException e) { + if (!accepting.get() || readExecutor.isShutdown()) { + task.fail(shuttingDownException()); + } else { + task.fail(new IoTDBTableReadQueueFullException("IoTDB Table Mode latest queue is full", e)); + } + readTasks.remove(task); + return task.future(); + } + if (!accepting.get() && readExecutor.remove(task)) { + task.fail(shuttingDownException()); + readTasks.remove(task); + } + return task.future(); + } + + private void failDroppedReadTask(Runnable dropped, IoTDBTableDaoShuttingDownException failure) { + if (dropped instanceof ReadTask task) { + task.fail(failure); + readTasks.remove(task); + } + } + + private IoTDBTableDaoShuttingDownException shuttingDownException() { + return new IoTDBTableDaoShuttingDownException("IoTDB Table Mode latest DAO is shutting down"); + } + + private static ThreadFactory readThreadFactory() { + AtomicInteger sequence = new AtomicInteger(); + return runnable -> { + Thread thread = + new Thread(runnable, "iotdb-table-latest-read-worker-" + sequence.incrementAndGet()); + thread.setDaemon(true); + return thread; + }; + } + + private final class ReadTask implements Runnable { + private final Callable callable; + private final SettableFuture future = SettableFuture.create(); + + private ReadTask(Callable callable) { + this.callable = Objects.requireNonNull(callable, "callable"); + } + + @Override + public void run() { + try { + future.set(callable.call()); + } catch (Throwable t) { + future.setException(t); + } finally { + readTasks.remove(this); + } + } + + private ListenableFuture future() { + return future; + } + + private void fail(Throwable t) { + future.setException(t); + } } } diff --git a/iotdb-thingsboard-table/src/main/java/org/apache/iotdb/extras/thingsboard/table/IoTDBTableLatestEnabledCondition.java b/iotdb-thingsboard-table/src/main/java/org/apache/iotdb/extras/thingsboard/table/IoTDBTableLatestEnabledCondition.java new file mode 100644 index 00000000..d36e7090 --- /dev/null +++ b/iotdb-thingsboard-table/src/main/java/org/apache/iotdb/extras/thingsboard/table/IoTDBTableLatestEnabledCondition.java @@ -0,0 +1,56 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package org.apache.iotdb.extras.thingsboard.table; + +import org.springframework.context.annotation.Condition; +import org.springframework.context.annotation.ConditionContext; +import org.springframework.core.type.AnnotatedTypeMetadata; + +/** + * Activation guard for the derived-latest DAO. It mirrors {@link IoTDBTableRawOnlyEnabledCondition} + * (selector {@code database.ts.type=iotdb-table} plus the explicit {@code + * iotdb.ts.experimental-raw-only=true} opt-in) AND additionally requires {@code + * database.ts_latest.type=iotdb-table}. + * + *

Requiring the timeseries selector as well as the latest selector is deliberate: the latest + * value is derived from the {@code telemetry} table that only the IoTDB writer populates. Were the + * DAO to activate on {@code database.ts_latest.type=iotdb-table} alone, a deployment that left + * {@code database.ts.type} on another backend would read latest values from a table this module + * never writes. Coupling both selectors keeps the latest path coherent with the write path and + * leaves the module fully inert on any deployment that has not opted into the IoTDB Table Mode + * timeseries backend. + */ +final class IoTDBTableLatestEnabledCondition implements Condition { + + private static final String TS_SELECTOR_PROPERTY = "database.ts.type"; + private static final String TS_LATEST_SELECTOR_PROPERTY = "database.ts_latest.type"; + private static final String SELECTOR_VALUE = "iotdb-table"; + private static final String EXPERIMENTAL_PROPERTY = "iotdb.ts.experimental-raw-only"; + + @Override + public boolean matches(ConditionContext context, AnnotatedTypeMetadata metadata) { + String ts = context.getEnvironment().getProperty(TS_SELECTOR_PROPERTY); + String tsLatest = context.getEnvironment().getProperty(TS_LATEST_SELECTOR_PROPERTY); + boolean tsSelected = ts != null && SELECTOR_VALUE.equalsIgnoreCase(ts.trim()); + boolean tsLatestSelected = tsLatest != null && SELECTOR_VALUE.equalsIgnoreCase(tsLatest.trim()); + boolean experimental = + context.getEnvironment().getProperty(EXPERIMENTAL_PROPERTY, Boolean.class, false); + return tsSelected && tsLatestSelected && experimental; + } +} diff --git a/iotdb-thingsboard-table/src/main/java/org/apache/iotdb/extras/thingsboard/table/IoTDBTableSchemaBootstrap.java b/iotdb-thingsboard-table/src/main/java/org/apache/iotdb/extras/thingsboard/table/IoTDBTableSchemaBootstrap.java index dbb4dcd2..b620cbda 100644 --- a/iotdb-thingsboard-table/src/main/java/org/apache/iotdb/extras/thingsboard/table/IoTDBTableSchemaBootstrap.java +++ b/iotdb-thingsboard-table/src/main/java/org/apache/iotdb/extras/thingsboard/table/IoTDBTableSchemaBootstrap.java @@ -38,12 +38,18 @@ * *

On a fresh IoTDB the {@code telemetry} / {@code entity_attributes} tables (and their database) * do not exist, so the very first write would fail. This initializer runs once the session pool - * bean is up, reads {@code schema-iotdb-table.sql} from the classpath, and executes its statements. - * Every DDL statement uses {@code CREATE ... IF NOT EXISTS}, so a re-run returns SUCCESS without + * bean is up, reads a schema SQL resource from the classpath, and executes its statements. Every + * DDL statement uses {@code CREATE ... IF NOT EXISTS}, so a re-run returns SUCCESS without * throwing. As defense-in-depth for racy or partial schema states, an "already exists" failure is * also recognized by its structured IoTDB status code (with a message-substring fallback) and * tolerated rather than propagated, keeping the bootstrap idempotent. * + *

The schema resource is parameterized ({@link #SCHEMA_RESOURCE} default, overridable via the + * 3-arg constructor) so a SECOND bootstrap bean can load {@code schema-iotdb-table-latest.sql} (the + * {@code telemetry_latest} overlay) without re-running the base schema. Each resource is + * self-contained ({@code CREATE DATABASE IF NOT EXISTS} + {@code USE} + {@code CREATE TABLE IF NOT + * EXISTS}), so multiple bootstrap beans are order-independent and idempotent. + * *

Gated behind {@code iotdb.schema.bootstrap} (default {@code true}) so operators who manage the * schema out-of-band can disable it; see the module README. */ @@ -51,6 +57,7 @@ public class IoTDBTableSchemaBootstrap implements InitializingBean { static final String SCHEMA_RESOURCE = "schema-iotdb-table.sql"; + static final String LATEST_SCHEMA_RESOURCE = "schema-iotdb-table-latest.sql"; private static final String DEFAULT_SCHEMA_DATABASE = "thingsboard"; /** @@ -64,10 +71,17 @@ public class IoTDBTableSchemaBootstrap implements InitializingBean { private final ITableSessionPool tableSessionPool; private final IoTDBTableConfig config; + private final String schemaResource; public IoTDBTableSchemaBootstrap(ITableSessionPool tableSessionPool, IoTDBTableConfig config) { + this(tableSessionPool, config, SCHEMA_RESOURCE); + } + + public IoTDBTableSchemaBootstrap( + ITableSessionPool tableSessionPool, IoTDBTableConfig config, String schemaResource) { this.tableSessionPool = Objects.requireNonNull(tableSessionPool, "tableSessionPool"); this.config = Objects.requireNonNull(config, "config"); + this.schemaResource = Objects.requireNonNull(schemaResource, "schemaResource"); } @Override @@ -116,10 +130,10 @@ private static String requireValidDatabaseName(String database) { private String loadSchema(String database) throws IOException { try (InputStream stream = - IoTDBTableSchemaBootstrap.class.getClassLoader().getResourceAsStream(SCHEMA_RESOURCE)) { + IoTDBTableSchemaBootstrap.class.getClassLoader().getResourceAsStream(schemaResource)) { if (stream == null) { throw new IllegalStateException( - "IoTDB Table Mode schema resource not found on classpath: " + SCHEMA_RESOURCE); + "IoTDB Table Mode schema resource not found on classpath: " + schemaResource); } String schema = new String(stream.readAllBytes(), StandardCharsets.UTF_8); // Strip block and line comments so split-on-';' never executes a comment fragment. diff --git a/iotdb-thingsboard-table/src/main/resources/schema-iotdb-table-latest.sql b/iotdb-thingsboard-table/src/main/resources/schema-iotdb-table-latest.sql new file mode 100644 index 00000000..6df9f6ba --- /dev/null +++ b/iotdb-thingsboard-table/src/main/resources/schema-iotdb-table-latest.sql @@ -0,0 +1,38 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, + * software distributed under the License is distributed on an + * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY + * KIND, either express or implied. See the License for the + * specific language governing permissions and limitations + * under the License. + */ + +CREATE DATABASE IF NOT EXISTS thingsboard; +USE thingsboard; + +-- Per-key latest overlay written on every saveLatest (delete-then-insert, one row per identity). +-- TAG order is aligned with the historical `telemetry` table (entity_type, tenant_id, key, +-- entity_id) so the same getEntry()/kvEntry() row mapping is reused. TTL='INF' because a latest +-- value must never physically expire. +CREATE TABLE IF NOT EXISTS telemetry_latest ( + time TIMESTAMP TIME, + entity_type STRING TAG, -- DEVICE, ASSET, etc. + tenant_id STRING TAG, -- multi-tenant isolation + key STRING TAG, -- telemetry key name + entity_id STRING TAG, -- ThingsBoard entity UUID + bool_v BOOLEAN FIELD, + long_v INT64 FIELD, -- exactly one non-null per row, + double_v DOUBLE FIELD, -- mirroring AbstractTsKvEntity + str_v STRING FIELD, + json_v TEXT FIELD +) WITH (TTL='INF'); diff --git a/iotdb-thingsboard-table/src/provided/java/org/apache/commons/lang3/tuple/Pair.java b/iotdb-thingsboard-table/src/provided/java/org/apache/commons/lang3/tuple/Pair.java new file mode 100644 index 00000000..3d163830 --- /dev/null +++ b/iotdb-thingsboard-table/src/provided/java/org/apache/commons/lang3/tuple/Pair.java @@ -0,0 +1,84 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package org.apache.commons.lang3.tuple; + +import java.util.Objects; + +/** + * Compile-only stub of Apache Commons Lang3 {@code org.apache.commons.lang3.tuple.Pair}, provided + * so the IoTDB Table Mode DAO can bind the real ThingsBoard {@code AttributesDao.removeAllByEntityId} + * signature (which returns {@code List>}). commons-lang3 is not on this + * module's compile classpath; the real artifact is supplied by the host ThingsBoard runtime. This + * stub is excluded from the built jar (Strategy F) and only exposes the {@code of}/{@code getLeft}/ + * {@code getRight} surface the DAO uses. + */ +public final class Pair { + + private final L left; + private final R right; + + private Pair(L left, R right) { + this.left = left; + this.right = right; + } + + public static Pair of(L left, R right) { + return new Pair<>(left, right); + } + + public L getLeft() { + return left; + } + + public R getRight() { + return right; + } + + // Real commons-lang3 Pair implements Map.Entry, so TB service code reads pairs via + // getKey()/getValue(); mirror that surface (getKey == left, getValue == right). + public L getKey() { + return left; + } + + public R getValue() { + return right; + } + + @Override + public boolean equals(Object o) { + if (this == o) { + return true; + } + if (!(o instanceof Pair)) { + return false; + } + Pair pair = (Pair) o; + return Objects.equals(left, pair.left) && Objects.equals(right, pair.right); + } + + @Override + public int hashCode() { + return Objects.hash(left, right); + } + + @Override + public String toString() { + return "(" + left + "," + right + ")"; + } +} diff --git a/iotdb-thingsboard-table/src/provided/java/org/thingsboard/server/common/data/AttributeScope.java b/iotdb-thingsboard-table/src/provided/java/org/thingsboard/server/common/data/AttributeScope.java new file mode 100644 index 00000000..54c6ab1d --- /dev/null +++ b/iotdb-thingsboard-table/src/provided/java/org/thingsboard/server/common/data/AttributeScope.java @@ -0,0 +1,58 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +// Compile-only ThingsBoard stub (Strategy F). Verified against ThingsBoard v4.3.1.2 +// (commit c37fb509). +package org.thingsboard.server.common.data; + +import java.util.Arrays; +import java.util.Map; +import java.util.stream.Collectors; + +/** + * Compile-only ThingsBoard surface stub (Strategy F): mirrors {@code + * org.thingsboard.server.common.data.AttributeScope} so the IoTDB Table Mode DAO can bind the real + * {@code AttributesDao} SPI without depending on the upstream ThingsBoard artifact. Excluded from + * the built jar; replace with the provided-scope upstream dependency once it is resolvable. + * + *

The IoTDB {@code entity_attributes} table stores the scope by {@link #name()} (e.g. {@code + * SERVER_SCOPE}), not by {@link #getId()}; the integer id is retained only to match the real enum's + * shape (the custom {@code valueOf(int)} is part of the verified v4.3.1.2 surface). + */ +public enum AttributeScope { + CLIENT_SCOPE(1), + SERVER_SCOPE(2), + SHARED_SCOPE(3); + + private final int id; + + private static final Map VALUES = + Arrays.stream(values()).collect(Collectors.toMap(AttributeScope::getId, scope -> scope)); + + AttributeScope(int id) { + this.id = id; + } + + public int getId() { + return id; + } + + public static AttributeScope valueOf(int id) { + return VALUES.get(id); + } +} diff --git a/iotdb-thingsboard-table/src/provided/java/org/thingsboard/server/common/data/id/DeviceProfileId.java b/iotdb-thingsboard-table/src/provided/java/org/thingsboard/server/common/data/id/DeviceProfileId.java new file mode 100644 index 00000000..b8b20f36 --- /dev/null +++ b/iotdb-thingsboard-table/src/provided/java/org/thingsboard/server/common/data/id/DeviceProfileId.java @@ -0,0 +1,43 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +// Compile-only ThingsBoard stub (Strategy F). Verified against ThingsBoard v4.3.1.2 +// (commit c37fb509). +package org.thingsboard.server.common.data.id; + +import org.thingsboard.server.common.data.EntityType; + +import java.util.UUID; + +/** + * Compile-only ThingsBoard surface stub (Strategy F): mirrors {@code + * org.thingsboard.server.common.data.id.DeviceProfileId} so the IoTDB Table Mode DAO can bind the + * {@code AttributesDao.findAllKeysByDeviceProfileId} signature without depending on the upstream + * ThingsBoard artifact. Excluded from the built jar. + */ +public class DeviceProfileId extends UUIDBased implements EntityId { + + public DeviceProfileId(UUID id) { + super(id); + } + + @Override + public EntityType getEntityType() { + return EntityType.DEVICE_PROFILE; + } +} diff --git a/iotdb-thingsboard-table/src/provided/java/org/thingsboard/server/common/data/kv/AttributeKvEntry.java b/iotdb-thingsboard-table/src/provided/java/org/thingsboard/server/common/data/kv/AttributeKvEntry.java new file mode 100644 index 00000000..488ede65 --- /dev/null +++ b/iotdb-thingsboard-table/src/provided/java/org/thingsboard/server/common/data/kv/AttributeKvEntry.java @@ -0,0 +1,34 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +// Compile-only ThingsBoard stub (Strategy F). Verified against ThingsBoard v4.3.1.2 +// (commit c37fb509). +package org.thingsboard.server.common.data.kv; + +import org.thingsboard.server.common.data.HasVersion; + +/** + * Compile-only ThingsBoard surface stub (Strategy F): mirrors {@code + * org.thingsboard.server.common.data.kv.AttributeKvEntry} so the IoTDB Table Mode DAO can bind the + * real {@code AttributesDao} SPI without depending on the upstream ThingsBoard artifact. Excluded + * from the built jar. + */ +public interface AttributeKvEntry extends KvEntry, HasVersion { + + long getLastUpdateTs(); +} diff --git a/iotdb-thingsboard-table/src/provided/java/org/thingsboard/server/common/data/kv/BaseAttributeKvEntry.java b/iotdb-thingsboard-table/src/provided/java/org/thingsboard/server/common/data/kv/BaseAttributeKvEntry.java new file mode 100644 index 00000000..4e047128 --- /dev/null +++ b/iotdb-thingsboard-table/src/provided/java/org/thingsboard/server/common/data/kv/BaseAttributeKvEntry.java @@ -0,0 +1,139 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +// Compile-only ThingsBoard stub (Strategy F). Verified against ThingsBoard v4.3.1.2 +// (commit c37fb509). +package org.thingsboard.server.common.data.kv; + +import java.util.Objects; +import java.util.Optional; + +/** + * Compile-only ThingsBoard surface stub (Strategy F): mirrors {@code + * org.thingsboard.server.common.data.kv.BaseAttributeKvEntry}. Wraps a {@link KvEntry} together + * with its {@code lastUpdateTs} and optional {@code version}, delegating all {@link KvEntry} getters + * to the wrapped entry. Excluded from the built jar. + */ +public class BaseAttributeKvEntry implements AttributeKvEntry { + + private final long lastUpdateTs; + private final KvEntry kv; + private final Long version; + + public BaseAttributeKvEntry(KvEntry kv, long lastUpdateTs) { + this.kv = kv; + this.lastUpdateTs = lastUpdateTs; + this.version = null; + } + + // Real TB also exposes the (long, KvEntry) argument order; mirror it for stub fidelity. + public BaseAttributeKvEntry(long lastUpdateTs, KvEntry kv) { + this(kv, lastUpdateTs); + } + + public BaseAttributeKvEntry(KvEntry kv, long lastUpdateTs, Long version) { + this.kv = kv; + this.lastUpdateTs = lastUpdateTs; + this.version = version; + } + + @Override + public long getLastUpdateTs() { + return lastUpdateTs; + } + + @Override + public Long getVersion() { + return version; + } + + @Override + public String getKey() { + return kv.getKey(); + } + + @Override + public DataType getDataType() { + return kv.getDataType(); + } + + @Override + public Optional getBooleanValue() { + return kv.getBooleanValue(); + } + + @Override + public Optional getLongValue() { + return kv.getLongValue(); + } + + @Override + public Optional getDoubleValue() { + return kv.getDoubleValue(); + } + + @Override + public Optional getStrValue() { + return kv.getStrValue(); + } + + @Override + public Optional getJsonValue() { + return kv.getJsonValue(); + } + + @Override + public String getValueAsString() { + return kv.getValueAsString(); + } + + @Override + public Object getValue() { + return kv.getValue(); + } + + @Override + public boolean equals(Object o) { + if (this == o) { + return true; + } + if (!(o instanceof BaseAttributeKvEntry)) { + return false; + } + BaseAttributeKvEntry that = (BaseAttributeKvEntry) o; + return lastUpdateTs == that.lastUpdateTs + && Objects.equals(kv, that.kv) + && Objects.equals(version, that.version); + } + + @Override + public int hashCode() { + return Objects.hash(lastUpdateTs, kv, version); + } + + @Override + public String toString() { + return "BaseAttributeKvEntry(lastUpdateTs=" + + lastUpdateTs + + ", kv=" + + kv + + ", version=" + + version + + ")"; + } +} diff --git a/iotdb-thingsboard-table/src/provided/java/org/thingsboard/server/common/data/kv/TsKvLatestRemovingResult.java b/iotdb-thingsboard-table/src/provided/java/org/thingsboard/server/common/data/kv/TsKvLatestRemovingResult.java new file mode 100644 index 00000000..2d5defd0 --- /dev/null +++ b/iotdb-thingsboard-table/src/provided/java/org/thingsboard/server/common/data/kv/TsKvLatestRemovingResult.java @@ -0,0 +1,60 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package org.thingsboard.server.common.data.kv; + +public class TsKvLatestRemovingResult { + private final String key; + private final TsKvEntry data; + private final boolean removed; + private final Long version; + + public TsKvLatestRemovingResult(String key, boolean removed) { + this(key, removed, null); + } + + public TsKvLatestRemovingResult(TsKvEntry data, Long version) { + this.key = data.getKey(); + this.data = data; + this.removed = true; + this.version = version; + } + + public TsKvLatestRemovingResult(String key, boolean removed, Long version) { + this.key = key; + this.data = null; + this.removed = removed; + this.version = version; + } + + public String getKey() { + return key; + } + + public TsKvEntry getData() { + return data; + } + + public boolean isRemoved() { + return removed; + } + + public Long getVersion() { + return version; + } +} diff --git a/iotdb-thingsboard-table/src/provided/java/org/thingsboard/server/common/data/util/TbPair.java b/iotdb-thingsboard-table/src/provided/java/org/thingsboard/server/common/data/util/TbPair.java new file mode 100644 index 00000000..b1156b9b --- /dev/null +++ b/iotdb-thingsboard-table/src/provided/java/org/thingsboard/server/common/data/util/TbPair.java @@ -0,0 +1,81 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +// Compile-only ThingsBoard stub (Strategy F). Verified against ThingsBoard v4.3.1.2 +// (commit c37fb509). +package org.thingsboard.server.common.data.util; + +import java.util.Objects; + +/** + * Compile-only ThingsBoard surface stub (Strategy F): mirrors {@code + * org.thingsboard.server.common.data.util.TbPair} so the IoTDB Table Mode DAO can return the value + * shape expected by the real {@code AttributesDao.removeAllWithVersions} contract without depending + * on the upstream ThingsBoard artifact. Excluded from the built jar. + */ +public class TbPair { + private S first; + private T second; + + public TbPair(S first, T second) { + this.first = first; + this.second = second; + } + + public static TbPair of(S first, T second) { + return new TbPair<>(first, second); + } + + public S getFirst() { + return first; + } + + public void setFirst(S first) { + this.first = first; + } + + public T getSecond() { + return second; + } + + public void setSecond(T second) { + this.second = second; + } + + @Override + public boolean equals(Object o) { + if (this == o) { + return true; + } + if (!(o instanceof TbPair)) { + return false; + } + TbPair tbPair = (TbPair) o; + return Objects.equals(first, tbPair.first) && Objects.equals(second, tbPair.second); + } + + @Override + public int hashCode() { + return Objects.hash(first, second); + } + + @Override + public String toString() { + return "TbPair(first=" + first + ", second=" + second + ")"; + } +} diff --git a/iotdb-thingsboard-table/src/provided/java/org/thingsboard/server/dao/attributes/AttributesDao.java b/iotdb-thingsboard-table/src/provided/java/org/thingsboard/server/dao/attributes/AttributesDao.java new file mode 100644 index 00000000..c7cd680f --- /dev/null +++ b/iotdb-thingsboard-table/src/provided/java/org/thingsboard/server/dao/attributes/AttributesDao.java @@ -0,0 +1,94 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +// Compile-only ThingsBoard stub (Strategy F). Verified against ThingsBoard v4.3.1.2 +// (commit c37fb509). +package org.thingsboard.server.dao.attributes; + +import com.google.common.util.concurrent.ListenableFuture; +import org.apache.commons.lang3.tuple.Pair; +import org.thingsboard.server.common.data.AttributeScope; +import org.thingsboard.server.common.data.id.DeviceProfileId; +import org.thingsboard.server.common.data.id.EntityId; +import org.thingsboard.server.common.data.id.TenantId; +import org.thingsboard.server.common.data.kv.AttributeKvEntry; +import org.thingsboard.server.common.data.util.TbPair; +import org.thingsboard.server.dao.model.sql.AttributeKvEntity; + +import java.util.Collection; +import java.util.List; +import java.util.Optional; +import java.util.UUID; + +/** + * Compile-only ThingsBoard surface stub (Strategy F): mirrors {@code + * org.thingsboard.server.dao.attributes.AttributesDao} so the IoTDB Table Mode DAO can bind the real + * entity-attribute SPI without depending on the upstream ThingsBoard artifact. Excluded from the + * built jar; replace with the provided-scope upstream dependency once it is resolvable. + * + *

The 14 method signatures here pin the v4.3.1.2 SPI surface exactly (verified against the real + * {@code AttributesDao} at tag v4.3.1.2). {@code StrategyFContractTest} reflects over them so any + * accidental drift in the local surface breaks the build. + */ +public interface AttributesDao { + + Optional find( + TenantId tenantId, EntityId entityId, AttributeScope attributeScope, String attributeKey); + + List find( + TenantId tenantId, + EntityId entityId, + AttributeScope attributeScope, + Collection attributeKey); + + List findAll( + TenantId tenantId, EntityId entityId, AttributeScope attributeScope); + + ListenableFuture save( + TenantId tenantId, + EntityId entityId, + AttributeScope attributeScope, + AttributeKvEntry attribute); + + List> removeAll( + TenantId tenantId, EntityId entityId, AttributeScope attributeScope, List keys); + + List>> removeAllWithVersions( + TenantId tenantId, EntityId entityId, AttributeScope attributeScope, List keys); + + List findNextBatch( + UUID entityId, int attributeType, int attributeKey, int batchSize); + + List findAllKeysByDeviceProfileId(TenantId tenantId, DeviceProfileId deviceProfileId); + + List findAllKeysByEntityIds(TenantId tenantId, List entityIds); + + List findAllKeysByEntityIdsAndScope( + TenantId tenantId, List entityIds, AttributeScope scope); + + ListenableFuture> findAllKeysByEntityIdsAndScopeAsync( + TenantId tenantId, List entityIds, AttributeScope scope); + + List findLatestByEntityIdsAndScope( + TenantId tenantId, List entityIds, AttributeScope scope); + + ListenableFuture> findLatestByEntityIdsAndScopeAsync( + TenantId tenantId, List entityIds, AttributeScope scope); + + List> removeAllByEntityId(TenantId tenantId, EntityId entityId); +} diff --git a/iotdb-thingsboard-table/src/provided/java/org/thingsboard/server/dao/model/sql/AttributeKvEntity.java b/iotdb-thingsboard-table/src/provided/java/org/thingsboard/server/dao/model/sql/AttributeKvEntity.java new file mode 100644 index 00000000..3a745fd7 --- /dev/null +++ b/iotdb-thingsboard-table/src/provided/java/org/thingsboard/server/dao/model/sql/AttributeKvEntity.java @@ -0,0 +1,37 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +// Compile-only ThingsBoard stub (Strategy F). Verified against ThingsBoard v4.3.1.2 +// (commit c37fb509). +package org.thingsboard.server.dao.model.sql; + +/** + * Compile-only ThingsBoard surface stub (Strategy F): a deliberately empty placeholder for {@code + * org.thingsboard.server.dao.model.sql.AttributeKvEntity}. + * + *

The upstream type is a relational JPA entity ({@code @Entity}, {@code @EmbeddedId}, + * {@code jakarta.persistence}) used only by the keyset-pagination migration helper {@code + * AttributesDao.findNextBatch}. The IoTDB Table Mode backend documents that method as deferred (it + * throws {@code UnsupportedOperationException}; see the Wk5 decision note section 6), so this stub + * carries none of the ORM fields or annotations — it exists purely so the {@code AttributesDao} + * interface signature resolves at compile time. Excluded from the built jar; the IoTDB DAO never + * constructs or returns an instance. + */ +public final class AttributeKvEntity { + private AttributeKvEntity() {} +} diff --git a/iotdb-thingsboard-table/src/provided/java/org/thingsboard/server/dao/timeseries/TimeseriesLatestDao.java b/iotdb-thingsboard-table/src/provided/java/org/thingsboard/server/dao/timeseries/TimeseriesLatestDao.java new file mode 100644 index 00000000..d25edcf0 --- /dev/null +++ b/iotdb-thingsboard-table/src/provided/java/org/thingsboard/server/dao/timeseries/TimeseriesLatestDao.java @@ -0,0 +1,65 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package org.thingsboard.server.dao.timeseries; + +import com.google.common.util.concurrent.ListenableFuture; +import org.thingsboard.server.common.data.id.DeviceProfileId; +import org.thingsboard.server.common.data.id.EntityId; +import org.thingsboard.server.common.data.id.TenantId; +import org.thingsboard.server.common.data.kv.DeleteTsKvQuery; +import org.thingsboard.server.common.data.kv.TsKvEntry; +import org.thingsboard.server.common.data.kv.TsKvLatestRemovingResult; + +import java.util.List; +import java.util.Optional; + +/** + * Compile-only Strategy-F surface for ThingsBoard's {@code TimeseriesLatestDao} SPI. Pinned to + * ThingsBoard {@code v4.3.1.2} (tag commit {@code c37fb509}); re-verified against upstream on + * 2026-06-20. The {@code findLatestByEntityIds} / {@code findLatestByEntityIdsAsync} pair is new in + * v4.3.1.2 relative to v4.3.1.1. The method shapes the DAO depends on are pinned by {@code + * StrategyFContractTest}. + */ +public interface TimeseriesLatestDao { + + ListenableFuture> findLatestOpt( + TenantId tenantId, EntityId entityId, String key); + + ListenableFuture findLatest(TenantId tenantId, EntityId entityId, String key); + + ListenableFuture> findAllLatest(TenantId tenantId, EntityId entityId); + + ListenableFuture saveLatest(TenantId tenantId, EntityId entityId, TsKvEntry tsKvEntry); + + ListenableFuture removeLatest( + TenantId tenantId, EntityId entityId, DeleteTsKvQuery query); + + List findAllKeysByDeviceProfileId(TenantId tenantId, DeviceProfileId deviceProfileId); + + List findAllKeysByEntityIds(TenantId tenantId, List entityIds); + + ListenableFuture> findAllKeysByEntityIdsAsync( + TenantId tenantId, List entityIds); + + // New in ThingsBoard v4.3.1.2 (batch latest read). + List findLatestByEntityIds(TenantId tenantId, List entityIds); + + ListenableFuture> findLatestByEntityIdsAsync( + TenantId tenantId, List entityIds); +} diff --git a/iotdb-thingsboard-table/src/test/java/org/apache/iotdb/extras/thingsboard/table/IoTDBTableAttributesDaoIT.java b/iotdb-thingsboard-table/src/test/java/org/apache/iotdb/extras/thingsboard/table/IoTDBTableAttributesDaoIT.java new file mode 100644 index 00000000..86c6bd33 --- /dev/null +++ b/iotdb-thingsboard-table/src/test/java/org/apache/iotdb/extras/thingsboard/table/IoTDBTableAttributesDaoIT.java @@ -0,0 +1,644 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package org.apache.iotdb.extras.thingsboard.table; + +import org.apache.iotdb.isession.ITableSession; +import org.apache.iotdb.isession.SessionDataSet; +import org.apache.iotdb.isession.pool.ITableSessionPool; +import org.apache.iotdb.session.pool.TableSessionPoolBuilder; + +import com.google.common.util.concurrent.ListenableFuture; +import org.apache.commons.lang3.tuple.Pair; +import org.junit.jupiter.api.Tag; +import org.junit.jupiter.api.Test; +import org.testcontainers.containers.GenericContainer; +import org.testcontainers.containers.wait.strategy.Wait; +import org.testcontainers.junit.jupiter.Container; +import org.testcontainers.junit.jupiter.Testcontainers; +import org.testcontainers.utility.DockerImageName; +import org.thingsboard.server.common.data.AttributeScope; +import org.thingsboard.server.common.data.EntityType; +import org.thingsboard.server.common.data.id.EntityId; +import org.thingsboard.server.common.data.id.TenantId; +import org.thingsboard.server.common.data.kv.AttributeKvEntry; +import org.thingsboard.server.common.data.kv.BaseAttributeKvEntry; +import org.thingsboard.server.common.data.kv.BooleanDataEntry; +import org.thingsboard.server.common.data.kv.DataType; +import org.thingsboard.server.common.data.kv.DoubleDataEntry; +import org.thingsboard.server.common.data.kv.JsonDataEntry; +import org.thingsboard.server.common.data.kv.KvEntry; +import org.thingsboard.server.common.data.kv.LongDataEntry; +import org.thingsboard.server.common.data.kv.StringDataEntry; +import org.thingsboard.server.common.data.util.TbPair; + +import java.io.InputStream; +import java.nio.charset.StandardCharsets; +import java.time.Duration; +import java.util.ArrayList; +import java.util.HashMap; +import java.util.HashSet; +import java.util.List; +import java.util.Map; +import java.util.Set; +import java.util.UUID; +import java.util.concurrent.CountDownLatch; +import java.util.concurrent.ExecutorService; +import java.util.concurrent.Executors; +import java.util.concurrent.TimeUnit; + +import static org.junit.jupiter.api.Assertions.assertEquals; +import static org.junit.jupiter.api.Assertions.assertNull; +import static org.junit.jupiter.api.Assertions.assertTrue; + +@Tag("integration") +@Testcontainers(disabledWithoutDocker = true) +class IoTDBTableAttributesDaoIT { + // Cold testcontainer first writes/reads are slower than a warm production node, so the + // per-future assertion timeout is generous. + private static final int FUTURE_TIMEOUT_SECONDS = 30; + private static final Duration IOTDB_STARTUP_TIMEOUT = Duration.ofMinutes(3); + private static final Duration IOTDB_READY_TIMEOUT = Duration.ofSeconds(60); + private static final Duration IOTDB_READY_POLL_INTERVAL = Duration.ofMillis(500); + + @Container + static final GenericContainer IOTDB = + new GenericContainer<>(DockerImageName.parse("apache/iotdb:2.0.8-standalone")) + .withExposedPorts(6667) + // IoTDB binds its client RPC service to dn_rpc_address (default 127.0.0.1); bind to all + // interfaces so the Testcontainers port-mapped session handshake succeeds. + .withEnv("dn_rpc_address", "0.0.0.0") + .waitingFor(Wait.forListeningPort().withStartupTimeout(IOTDB_STARTUP_TIMEOUT)); + + @Test + void saveThenFind_roundTripsAllFiveTypesWithLastUpdateTs() throws Exception { + TestScope scope = scope("attr_types", "55555555-5555-5555-5555-555555555501"); + bootstrapSchema(scope.database()); + try (ITableSessionPool pool = newPool(scope.database())) { + IoTDBTableAttributesDao dao = new IoTDBTableAttributesDao(pool, config()); + try { + save(dao, scope, AttributeScope.SERVER_SCOPE, attr(1000L, "bool", bool("bool", true))); + save(dao, scope, AttributeScope.SERVER_SCOPE, attr(1001L, "long", lng("long", 42L))); + save(dao, scope, AttributeScope.SERVER_SCOPE, attr(1002L, "double", dbl("double", 4.2D))); + save(dao, scope, AttributeScope.SERVER_SCOPE, attr(1003L, "string", str("string", "v"))); + save( + dao, + scope, + AttributeScope.SERVER_SCOPE, + attr(1004L, "json", json("json", "{\"v\":1}"))); + + assertFind(dao, scope, AttributeScope.SERVER_SCOPE, "bool", 1000L, DataType.BOOLEAN, true); + assertFind(dao, scope, AttributeScope.SERVER_SCOPE, "long", 1001L, DataType.LONG, 42L); + assertFind(dao, scope, AttributeScope.SERVER_SCOPE, "double", 1002L, DataType.DOUBLE, 4.2D); + assertFind(dao, scope, AttributeScope.SERVER_SCOPE, "string", 1003L, DataType.STRING, "v"); + assertFind( + dao, scope, AttributeScope.SERVER_SCOPE, "json", 1004L, DataType.JSON, "{\"v\":1}"); + } finally { + dao.destroy(); + } + } + } + + @Test + void save_isDeleteThenInsert_keepsExactlyOneRowOnUpdate() throws Exception { + TestScope scope = scope("attr_update", "55555555-5555-5555-5555-555555555502"); + bootstrapSchema(scope.database()); + try (ITableSessionPool pool = newPool(scope.database())) { + IoTDBTableAttributesDao dao = new IoTDBTableAttributesDao(pool, config()); + try { + // First write a LONG at ts=1000, then overwrite the same identity with a STRING at ts=2000. + save(dao, scope, AttributeScope.SERVER_SCOPE, attr(1000L, "temp", lng("temp", 1L))); + save(dao, scope, AttributeScope.SERVER_SCOPE, attr(2000L, "temp", str("temp", "two"))); + + // delete-then-insert convergence: exactly one row, one typed FIELD, the newer value. + assertEquals(1, attributeRowCount(pool, scope, AttributeScope.SERVER_SCOPE, "temp")); + AttributeKvEntry found = + dao.find(scope.tenantId(), scope.entityId(), AttributeScope.SERVER_SCOPE, "temp") + .orElseThrow(); + assertEquals(DataType.STRING, found.getDataType()); + assertEquals("two", found.getValue()); + assertEquals(2000L, found.getLastUpdateTs()); + } finally { + dao.destroy(); + } + } + } + + @Test + void sameTimestampTypeChange_getEntryDoesNotThrow() throws Exception { + // B1 regression: rewriting the same identity at the SAME timestamp with a different type must + // still converge to one typed FIELD (delete-then-insert), so find()'s getEntry never sees two + // typed columns and never throws the B1 fail-fast. + TestScope scope = scope("attr_b1", "55555555-5555-5555-5555-555555555510"); + bootstrapSchema(scope.database()); + try (ITableSessionPool pool = newPool(scope.database())) { + IoTDBTableAttributesDao dao = new IoTDBTableAttributesDao(pool, config()); + try { + save(dao, scope, AttributeScope.SERVER_SCOPE, attr(3000L, "temp", lng("temp", 1L))); + save(dao, scope, AttributeScope.SERVER_SCOPE, attr(3000L, "temp", str("temp", "two"))); + + assertEquals(1, attributeRowCount(pool, scope, AttributeScope.SERVER_SCOPE, "temp")); + AttributeKvEntry found = + dao.find(scope.tenantId(), scope.entityId(), AttributeScope.SERVER_SCOPE, "temp") + .orElseThrow(); + assertEquals(DataType.STRING, found.getDataType()); + assertEquals("two", found.getValue()); + assertEquals(3000L, found.getLastUpdateTs()); + } finally { + dao.destroy(); + } + } + } + + @Test + void scopesAreIsolated() throws Exception { + TestScope scope = scope("attr_scope", "55555555-5555-5555-5555-555555555503"); + bootstrapSchema(scope.database()); + try (ITableSessionPool pool = newPool(scope.database())) { + IoTDBTableAttributesDao dao = new IoTDBTableAttributesDao(pool, config()); + try { + save(dao, scope, AttributeScope.CLIENT_SCOPE, attr(10L, "shared", lng("shared", 1L))); + save(dao, scope, AttributeScope.SERVER_SCOPE, attr(20L, "shared", lng("shared", 2L))); + + assertEquals( + 1L, + dao.find(scope.tenantId(), scope.entityId(), AttributeScope.CLIENT_SCOPE, "shared") + .orElseThrow() + .getValue()); + assertEquals( + 2L, + dao.find(scope.tenantId(), scope.entityId(), AttributeScope.SERVER_SCOPE, "shared") + .orElseThrow() + .getValue()); + + // Deleting the CLIENT scope key does not touch the SERVER scope key. + removeAll(dao, scope, AttributeScope.CLIENT_SCOPE, List.of("shared")); + assertTrue( + dao.find(scope.tenantId(), scope.entityId(), AttributeScope.CLIENT_SCOPE, "shared") + .isEmpty()); + assertTrue( + dao.find(scope.tenantId(), scope.entityId(), AttributeScope.SERVER_SCOPE, "shared") + .isPresent()); + } finally { + dao.destroy(); + } + } + } + + @Test + void findAllAndFindKeys_returnAllAttributesInScope() throws Exception { + TestScope scope = scope("attr_all", "55555555-5555-5555-5555-555555555504"); + bootstrapSchema(scope.database()); + try (ITableSessionPool pool = newPool(scope.database())) { + IoTDBTableAttributesDao dao = new IoTDBTableAttributesDao(pool, config()); + try { + save(dao, scope, AttributeScope.SERVER_SCOPE, attr(1L, "a", lng("a", 1L))); + save(dao, scope, AttributeScope.SERVER_SCOPE, attr(2L, "b", str("b", "x"))); + + List all = + dao.findAll(scope.tenantId(), scope.entityId(), AttributeScope.SERVER_SCOPE); + assertEquals(2, all.size()); + assertEquals(Set.of("a", "b"), new HashSet<>(all.stream().map(KvEntry::getKey).toList())); + + List some = + dao.find(scope.tenantId(), scope.entityId(), AttributeScope.SERVER_SCOPE, List.of("a")); + assertEquals(1, some.size()); + assertEquals("a", some.get(0).getKey()); + } finally { + dao.destroy(); + } + } + } + + @Test + void removeAll_thenFindIsEmpty_andVersionIsNull() throws Exception { + TestScope scope = scope("attr_remove", "55555555-5555-5555-5555-555555555505"); + bootstrapSchema(scope.database()); + try (ITableSessionPool pool = newPool(scope.database())) { + IoTDBTableAttributesDao dao = new IoTDBTableAttributesDao(pool, config()); + try { + save(dao, scope, AttributeScope.SERVER_SCOPE, attr(1L, "temp", lng("temp", 1L))); + + Long version = + dao.save( + scope.tenantId(), + scope.entityId(), + AttributeScope.SERVER_SCOPE, + attr(2L, "temp", lng("temp", 2L))) + .get(FUTURE_TIMEOUT_SECONDS, TimeUnit.SECONDS); + assertNull(version); + + List>> futures = + dao.removeAllWithVersions( + scope.tenantId(), scope.entityId(), AttributeScope.SERVER_SCOPE, List.of("temp")); + TbPair pair = futures.get(0).get(FUTURE_TIMEOUT_SECONDS, TimeUnit.SECONDS); + assertEquals("temp", pair.getFirst()); + assertNull(pair.getSecond()); + + assertTrue( + dao.find(scope.tenantId(), scope.entityId(), AttributeScope.SERVER_SCOPE, "temp") + .isEmpty()); + } finally { + dao.destroy(); + } + } + } + + @Test + void removeAllByEntityId_returnsScopeKeyPairsAndClearsEntity() throws Exception { + TestScope scope = scope("attr_byent", "55555555-5555-5555-5555-555555555506"); + bootstrapSchema(scope.database()); + try (ITableSessionPool pool = newPool(scope.database())) { + IoTDBTableAttributesDao dao = new IoTDBTableAttributesDao(pool, config()); + try { + save(dao, scope, AttributeScope.SERVER_SCOPE, attr(1L, "a", lng("a", 1L))); + save(dao, scope, AttributeScope.CLIENT_SCOPE, attr(2L, "b", str("b", "x"))); + + List> removed = + dao.removeAllByEntityId(scope.tenantId(), scope.entityId()); + Set pairKeys = new HashSet<>(); + for (Pair pair : removed) { + pairKeys.add(pair.getLeft().name() + ":" + pair.getRight()); + } + assertEquals(Set.of("SERVER_SCOPE:a", "CLIENT_SCOPE:b"), pairKeys); + + assertTrue( + dao.findAll(scope.tenantId(), scope.entityId(), AttributeScope.SERVER_SCOPE).isEmpty()); + assertTrue( + dao.findAll(scope.tenantId(), scope.entityId(), AttributeScope.CLIENT_SCOPE).isEmpty()); + } finally { + dao.destroy(); + } + } + } + + @Test + void keyDiscovery_distinctKeysAndDeviceProfileNullTenantWide() throws Exception { + TestScope scope = scope("attr_keys", "55555555-5555-5555-5555-555555555507"); + bootstrapSchema(scope.database()); + try (ITableSessionPool pool = newPool(scope.database())) { + IoTDBTableAttributesDao dao = new IoTDBTableAttributesDao(pool, config()); + try { + save(dao, scope, AttributeScope.SERVER_SCOPE, attr(1L, "a", lng("a", 1L))); + save(dao, scope, AttributeScope.CLIENT_SCOPE, attr(2L, "b", str("b", "x"))); + + assertEquals( + Set.of("a", "b"), + new HashSet<>(dao.findAllKeysByEntityIds(scope.tenantId(), List.of(scope.entityId())))); + assertEquals( + Set.of("a"), + new HashSet<>( + dao.findAllKeysByEntityIdsAndScope( + scope.tenantId(), List.of(scope.entityId()), AttributeScope.SERVER_SCOPE))); + // [v4.3.1.2] async key discovery returns the same set as the synchronous overload. + assertEquals( + Set.of("a"), + new HashSet<>( + dao.findAllKeysByEntityIdsAndScopeAsync( + scope.tenantId(), List.of(scope.entityId()), AttributeScope.SERVER_SCOPE) + .get(FUTURE_TIMEOUT_SECONDS, TimeUnit.SECONDS))); + assertEquals( + Set.of("a", "b"), + new HashSet<>(dao.findAllKeysByDeviceProfileId(scope.tenantId(), null))); + } finally { + dao.destroy(); + } + } + } + + @Test + void findLatestByEntityIdsAndScope_returnsLatestValuePerEntityAndKey() throws Exception { + // [v4.3.1.2] bulk latest read across an OR-set of entities in one scope. + TestScope scope = scope("attr_lat", "55555555-5555-5555-5555-555555555511"); + bootstrapSchema(scope.database()); + EntityId secondEntity = + new TestEntityId(UUID.fromString("55555555-5555-5555-5555-555555555512"), EntityType.ASSET); + try (ITableSessionPool pool = newPool(scope.database())) { + IoTDBTableAttributesDao dao = new IoTDBTableAttributesDao(pool, config()); + try { + // Two writes on the first entity (overwrite => latest), one on the second entity. + save(dao, scope, AttributeScope.SERVER_SCOPE, attr(1L, "temp", lng("temp", 1L))); + save(dao, scope, AttributeScope.SERVER_SCOPE, attr(2L, "temp", lng("temp", 9L))); + dao.save( + scope.tenantId(), + secondEntity, + AttributeScope.SERVER_SCOPE, + attr(3L, "hum", lng("hum", 5L))) + .get(FUTURE_TIMEOUT_SECONDS, TimeUnit.SECONDS); + // A different scope on the first entity must be excluded by the scope predicate. + save(dao, scope, AttributeScope.CLIENT_SCOPE, attr(4L, "ignored", lng("ignored", 7L))); + + List latestSync = + dao.findLatestByEntityIdsAndScope( + scope.tenantId(), + List.of(scope.entityId(), secondEntity), + AttributeScope.SERVER_SCOPE); + assertEquals(toValueMap(latestSync), Map.of("temp", 9L, "hum", 5L)); + + List latestAsync = + dao.findLatestByEntityIdsAndScopeAsync( + scope.tenantId(), + List.of(scope.entityId(), secondEntity), + AttributeScope.SERVER_SCOPE) + .get(FUTURE_TIMEOUT_SECONDS, TimeUnit.SECONDS); + assertEquals(toValueMap(latestAsync), Map.of("temp", 9L, "hum", 5L)); + } finally { + dao.destroy(); + } + } + } + + @Test + void keyEscaping_roundTripsKeyWithSingleQuote() throws Exception { + TestScope scope = scope("attr_esc", "55555555-5555-5555-5555-555555555508"); + bootstrapSchema(scope.database()); + try (ITableSessionPool pool = newPool(scope.database())) { + IoTDBTableAttributesDao dao = new IoTDBTableAttributesDao(pool, config()); + try { + String key = "a'b"; + save(dao, scope, AttributeScope.SERVER_SCOPE, attr(5L, key, lng(key, 8L))); + + AttributeKvEntry found = + dao.find(scope.tenantId(), scope.entityId(), AttributeScope.SERVER_SCOPE, key) + .orElseThrow(); + assertEquals(8L, found.getValue()); + + removeAll(dao, scope, AttributeScope.SERVER_SCOPE, List.of(key)); + assertTrue( + dao.find(scope.tenantId(), scope.entityId(), AttributeScope.SERVER_SCOPE, key) + .isEmpty()); + } finally { + dao.destroy(); + } + } + } + + @Test + void concurrentSaveSameIdentity_convergesToOneRow() throws Exception { + TestScope scope = scope("attr_conc", "55555555-5555-5555-5555-555555555509"); + bootstrapSchema(scope.database()); + try (ITableSessionPool pool = newPool(scope.database())) { + IoTDBTableConfig config = config(); + config.getTs().getRead().setThreads(8); + IoTDBTableAttributesDao dao = new IoTDBTableAttributesDao(pool, config); + ExecutorService callers = Executors.newFixedThreadPool(8); + try { + int writes = 50; + CountDownLatch start = new CountDownLatch(1); + List> futures = new ArrayList<>(); + List> submissions = new ArrayList<>(); + for (int i = 0; i < writes; i++) { + long value = i; + submissions.add( + callers.submit( + () -> { + try { + start.await(); + } catch (InterruptedException e) { + Thread.currentThread().interrupt(); + throw new RuntimeException(e); + } + synchronized (futures) { + futures.add( + dao.save( + scope.tenantId(), + scope.entityId(), + AttributeScope.SERVER_SCOPE, + attr(1000L + value, "temp", lng("temp", value)))); + } + })); + } + start.countDown(); + for (java.util.concurrent.Future submission : submissions) { + submission.get(FUTURE_TIMEOUT_SECONDS, TimeUnit.SECONDS); + } + for (ListenableFuture future : futures) { + future.get(FUTURE_TIMEOUT_SECONDS, TimeUnit.SECONDS); + } + + // §3.5 regression: single-JVM per-identity lock must converge to exactly one row. + assertEquals(1, attributeRowCount(pool, scope, AttributeScope.SERVER_SCOPE, "temp")); + } finally { + callers.shutdownNow(); + dao.destroy(); + } + } + } + + // ---- helpers ---- + + private void save( + IoTDBTableAttributesDao dao, + TestScope scope, + AttributeScope attributeScope, + AttributeKvEntry attribute) + throws Exception { + dao.save(scope.tenantId(), scope.entityId(), attributeScope, attribute) + .get(FUTURE_TIMEOUT_SECONDS, TimeUnit.SECONDS); + } + + private void removeAll( + IoTDBTableAttributesDao dao, + TestScope scope, + AttributeScope attributeScope, + List keys) + throws Exception { + for (ListenableFuture future : + dao.removeAll(scope.tenantId(), scope.entityId(), attributeScope, keys)) { + future.get(FUTURE_TIMEOUT_SECONDS, TimeUnit.SECONDS); + } + } + + private void assertFind( + IoTDBTableAttributesDao dao, + TestScope scope, + AttributeScope attributeScope, + String key, + long lastUpdateTs, + DataType dataType, + Object value) { + AttributeKvEntry entry = + dao.find(scope.tenantId(), scope.entityId(), attributeScope, key).orElseThrow(); + assertEquals(key, entry.getKey()); + assertEquals(dataType, entry.getDataType()); + assertEquals(value, entry.getValue()); + assertEquals(lastUpdateTs, entry.getLastUpdateTs()); + assertNull(entry.getVersion()); + } + + private Map toValueMap(List entries) { + Map values = new HashMap<>(); + for (AttributeKvEntry entry : entries) { + values.put(entry.getKey(), entry.getValue()); + } + return values; + } + + private int attributeRowCount( + ITableSessionPool pool, TestScope scope, AttributeScope attributeScope, String key) + throws Exception { + try (ITableSession session = pool.getSession(); + SessionDataSet dataSet = + session.executeQueryStatement( + "SELECT bool_v,long_v,double_v,str_v,json_v FROM entity_attributes WHERE tenant_id='" + + scope.tenantId().getId() + + "' AND entity_type='DEVICE' AND entity_id='" + + scope.entityId().getId() + + "' AND attribute_scope='" + + attributeScope.name() + + "' AND key='" + + key.replace("'", "''") + + "'")) { + SessionDataSet.DataIterator rows = dataSet.iterator(); + int rowCount = 0; + while (rows.next()) { + rowCount++; + } + return rowCount; + } + } + + private ITableSessionPool newPool(String database) { + TableSessionPoolBuilder builder = + new TableSessionPoolBuilder() + .nodeUrls(List.of("127.0.0.1:" + IOTDB.getMappedPort(6667))) + .user("root") + .password("root") + .maxSize(4); + if (database != null) { + builder.database(database); + } + return builder.build(); + } + + private void bootstrapSchema(String database) throws Exception { + awaitIoTDBReady(database); + + String schema; + try (InputStream stream = + IoTDBTableAttributesDaoIT.class + .getClassLoader() + .getResourceAsStream("schema-iotdb-table.sql")) { + schema = new String(stream.readAllBytes(), StandardCharsets.UTF_8); + } + schema = + schema + .replace( + "CREATE DATABASE IF NOT EXISTS thingsboard;", + "CREATE DATABASE IF NOT EXISTS " + database + ";") + .replace("USE thingsboard;", "USE " + database + ";"); + schema = schema.replaceAll("(?s)/\\*.*?\\*/", "").replaceAll("(?m)--.*$", ""); + try (ITableSessionPool bootstrapPool = newPool(null)) { + try (ITableSession session = bootstrapPool.getSession()) { + for (String statement : schema.split(";")) { + String trimmed = statement.trim(); + if (!trimmed.isEmpty()) { + session.executeNonQueryStatement(trimmed); + } + } + } + } + } + + private void awaitIoTDBReady(String database) throws Exception { + long deadlineNanos = System.nanoTime() + IOTDB_READY_TIMEOUT.toNanos(); + Exception lastFailure = null; + while (System.nanoTime() < deadlineNanos) { + try (ITableSessionPool bootstrapPool = newPool(null); + ITableSession session = bootstrapPool.getSession()) { + session.executeNonQueryStatement("CREATE DATABASE IF NOT EXISTS " + database); + return; + } catch (Exception e) { + lastFailure = e; + long remainingMillis = TimeUnit.NANOSECONDS.toMillis(deadlineNanos - System.nanoTime()); + if (remainingMillis <= 0) { + break; + } + Thread.sleep(Math.min(IOTDB_READY_POLL_INTERVAL.toMillis(), remainingMillis)); + } + } + throw new IllegalStateException( + "IoTDB did not accept table-session statements within " + IOTDB_READY_TIMEOUT, lastFailure); + } + + private IoTDBTableConfig config() { + IoTDBTableConfig config = new IoTDBTableConfig(); + config.getTs().getRead().setThreads(2); + config.getAttributes().setClusterMode("disabled"); + return config; + } + + private TestScope scope(String databasePrefix, String entityId) { + return new TestScope( + uniqueDatabase(databasePrefix), + new TenantId(UUID.fromString("44444444-4444-4444-4444-444444444444")), + new TestEntityId(UUID.fromString(entityId), EntityType.DEVICE)); + } + + private String uniqueDatabase(String prefix) { + // IoTDB caps database names at 64 chars; keep the per-test prefix short and append a trimmed + // UUID so the total length stays well within the limit. + String shortPrefix = prefix.length() > 12 ? prefix.substring(0, 12) : prefix; + String shortUuid = UUID.randomUUID().toString().replace("-", "").substring(0, 16); + return "tb_at_" + shortPrefix + "_" + shortUuid; + } + + private TestAttributeKvEntry attr(long lastUpdateTs, String key, KvEntry value) { + return new TestAttributeKvEntry(lastUpdateTs, value); + } + + private KvEntry bool(String key, boolean value) { + return new BooleanDataEntry(key, value); + } + + private KvEntry lng(String key, long value) { + return new LongDataEntry(key, value); + } + + private KvEntry dbl(String key, double value) { + return new DoubleDataEntry(key, value); + } + + private KvEntry str(String key, String value) { + return new StringDataEntry(key, value); + } + + private KvEntry json(String key, String value) { + return new JsonDataEntry(key, value); + } + + private record TestScope(String database, TenantId tenantId, EntityId entityId) {} + + private record TestEntityId(UUID id, EntityType entityType) implements EntityId { + @Override + public UUID getId() { + return id; + } + + @Override + public EntityType getEntityType() { + return entityType; + } + } + + private static final class TestAttributeKvEntry extends BaseAttributeKvEntry { + private TestAttributeKvEntry(long lastUpdateTs, KvEntry kv) { + super(kv, lastUpdateTs); + } + } +} diff --git a/iotdb-thingsboard-table/src/test/java/org/apache/iotdb/extras/thingsboard/table/IoTDBTableAttributesDaoTest.java b/iotdb-thingsboard-table/src/test/java/org/apache/iotdb/extras/thingsboard/table/IoTDBTableAttributesDaoTest.java new file mode 100644 index 00000000..9bb6f790 --- /dev/null +++ b/iotdb-thingsboard-table/src/test/java/org/apache/iotdb/extras/thingsboard/table/IoTDBTableAttributesDaoTest.java @@ -0,0 +1,929 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package org.apache.iotdb.extras.thingsboard.table; + +import org.apache.iotdb.isession.ITableSession; +import org.apache.iotdb.isession.SessionDataSet; +import org.apache.iotdb.isession.pool.ITableSessionPool; +import org.apache.iotdb.rpc.IoTDBConnectionException; +import org.apache.iotdb.rpc.StatementExecutionException; + +import com.google.common.util.concurrent.ListenableFuture; +import org.apache.commons.lang3.tuple.Pair; +import org.apache.tsfile.enums.ColumnCategory; +import org.apache.tsfile.write.record.Tablet; +import org.junit.jupiter.api.AfterEach; +import org.junit.jupiter.api.Test; +import org.mockito.ArgumentCaptor; +import org.thingsboard.server.common.data.AttributeScope; +import org.thingsboard.server.common.data.EntityType; +import org.thingsboard.server.common.data.id.DeviceProfileId; +import org.thingsboard.server.common.data.id.EntityId; +import org.thingsboard.server.common.data.id.TenantId; +import org.thingsboard.server.common.data.kv.AttributeKvEntry; +import org.thingsboard.server.common.data.kv.BaseAttributeKvEntry; +import org.thingsboard.server.common.data.kv.BooleanDataEntry; +import org.thingsboard.server.common.data.kv.DataType; +import org.thingsboard.server.common.data.kv.KvEntry; +import org.thingsboard.server.common.data.kv.LongDataEntry; +import org.thingsboard.server.common.data.kv.StringDataEntry; +import org.thingsboard.server.common.data.util.TbPair; + +import java.sql.Timestamp; +import java.util.ArrayList; +import java.util.HashMap; +import java.util.List; +import java.util.Map; +import java.util.Optional; +import java.util.UUID; +import java.util.concurrent.CountDownLatch; +import java.util.concurrent.TimeUnit; +import java.util.concurrent.atomic.AtomicInteger; + +import static org.junit.jupiter.api.Assertions.assertEquals; +import static org.junit.jupiter.api.Assertions.assertFalse; +import static org.junit.jupiter.api.Assertions.assertNull; +import static org.junit.jupiter.api.Assertions.assertThrows; +import static org.junit.jupiter.api.Assertions.assertTrue; +import static org.mockito.ArgumentMatchers.anyString; +import static org.mockito.Mockito.doAnswer; +import static org.mockito.Mockito.mock; +import static org.mockito.Mockito.never; +import static org.mockito.Mockito.timeout; +import static org.mockito.Mockito.verify; +import static org.mockito.Mockito.when; + +class IoTDBTableAttributesDaoTest { + private static final TenantId TENANT_ID = + new TenantId(UUID.fromString("11111111-1111-1111-1111-111111111111")); + private static final EntityId ENTITY_ID = + new TestEntityId(UUID.fromString("22222222-2222-2222-2222-222222222222"), EntityType.DEVICE); + private static final EntityId SECOND_ENTITY_ID = + new TestEntityId(UUID.fromString("33333333-3333-3333-3333-333333333333"), EntityType.ASSET); + + private final List daos = new ArrayList<>(); + + @AfterEach + void tearDown() { + for (IoTDBTableAttributesDao dao : daos) { + dao.destroy(); + } + daos.clear(); + } + + @Test + void save_buildsDeleteThenInsertWithLastUpdateTsTimeAndOneTypedField() throws Exception { + TestContext context = newContext(); + + Long version = + context + .dao() + .save( + TENANT_ID, + ENTITY_ID, + AttributeScope.SERVER_SCOPE, + attribute(7000L, "temperature", new LongDataEntry("temperature", 42L))) + .get(3, TimeUnit.SECONDS); + + // IoTDB has no sequence; the save returns a null version (Phase-1 contract). + assertNull(version); + + // delete-then-insert: the DELETE is a tag-only equality match (no time predicate) so it removes + // the identity across all time, then a single attribute row is inserted at time=lastUpdateTs. + ArgumentCaptor deleteSql = ArgumentCaptor.forClass(String.class); + verify(context.session(), timeout(3000)).executeNonQueryStatement(deleteSql.capture()); + assertEquals( + "DELETE FROM entity_attributes " + + "WHERE tenant_id='11111111-1111-1111-1111-111111111111' " + + "AND entity_type='DEVICE' " + + "AND entity_id='22222222-2222-2222-2222-222222222222' " + + "AND attribute_scope='SERVER_SCOPE' " + + "AND key='temperature'", + deleteSql.getValue()); + + ArgumentCaptor tablet = ArgumentCaptor.forClass(Tablet.class); + verify(context.session(), timeout(3000)).insert(tablet.capture()); + Tablet inserted = tablet.getValue(); + assertEquals("entity_attributes", inserted.getTableName()); + assertEquals(1, inserted.getRowSize()); + assertEquals(7000L, inserted.getTimestamp(0)); + } + + @Test + void save_tabletColumnsFollowNewDdlTagOrder() throws Exception { + // TAG-order rot guard (Wk5 risk): the Tablet column schema must follow the entity_attributes + // DDL tag order (attribute_scope, entity_type, tenant_id, key, entity_id), then the five typed + // FIELDs, with NO ColumnCategory.TIME entry (the time column is written via addTimestamp). + TestContext context = newContext(); + + context + .dao() + .save( + TENANT_ID, + ENTITY_ID, + AttributeScope.SERVER_SCOPE, + attribute(7000L, "temperature", new LongDataEntry("temperature", 42L))) + .get(3, TimeUnit.SECONDS); + + ArgumentCaptor tablet = ArgumentCaptor.forClass(Tablet.class); + verify(context.session(), timeout(3000)).insert(tablet.capture()); + Tablet inserted = tablet.getValue(); + + List columnNames = + inserted.getSchemas().stream() + .map(org.apache.tsfile.write.schema.IMeasurementSchema::getMeasurementName) + .toList(); + assertEquals( + List.of( + "attribute_scope", + "entity_type", + "tenant_id", + "key", + "entity_id", + "bool_v", + "long_v", + "double_v", + "str_v", + "json_v"), + columnNames); + // The tablet schema carries exactly the 10 non-time columns: 5 TAG then 5 FIELD, with no other + // category. The `time TIMESTAMP TIME` column is written via addTimestamp, not as a tablet + // column, so it never appears here (tsfile's ColumnCategory has only TAG/FIELD/ATTRIBUTE). + List categories = inserted.getColumnTypes(); + assertEquals(10, categories.size()); + for (int i = 0; i < 5; i++) { + assertEquals(ColumnCategory.TAG, categories.get(i), "tag column " + i); + } + for (int i = 5; i < 10; i++) { + assertEquals(ColumnCategory.FIELD, categories.get(i), "field column " + i); + } + } + + @Test + void save_runsDeleteBeforeInsert() throws Exception { + TestContext context = newContext(); + + context + .dao() + .save( + TENANT_ID, + ENTITY_ID, + AttributeScope.CLIENT_SCOPE, + attribute(1L, "k", new BooleanDataEntry("k", true))) + .get(3, TimeUnit.SECONDS); + + org.mockito.InOrder inOrder = org.mockito.Mockito.inOrder(context.session()); + inOrder.verify(context.session()).executeNonQueryStatement(anyString()); + inOrder.verify(context.session()).insert(org.mockito.ArgumentMatchers.any()); + } + + @Test + void find_buildsSingleRowSqlAndMapsToBaseAttributeKvEntry() throws Exception { + TestContext context = newContext(); + SessionDataSet dataSet = dataSet(row(150L, "long_v", 42L)); + when(context.session().executeQueryStatement(anyString())).thenReturn(dataSet); + + Optional result = + context.dao().find(TENANT_ID, ENTITY_ID, AttributeScope.SERVER_SCOPE, "temperature"); + + assertTrue(result.isPresent()); + AttributeKvEntry entry = result.get(); + assertEquals("temperature", entry.getKey()); + assertEquals(DataType.LONG, entry.getDataType()); + assertEquals(42L, entry.getValue()); + // The time column is the attribute's last-update timestamp. + assertEquals(150L, entry.getLastUpdateTs()); + assertNull(entry.getVersion()); + + ArgumentCaptor sql = ArgumentCaptor.forClass(String.class); + verify(context.session(), timeout(3000)).executeQueryStatement(sql.capture()); + assertEquals( + "SELECT time, bool_v, long_v, double_v, str_v, json_v FROM entity_attributes " + + "WHERE tenant_id='11111111-1111-1111-1111-111111111111' " + + "AND entity_type='DEVICE' " + + "AND entity_id='22222222-2222-2222-2222-222222222222' " + + "AND attribute_scope='SERVER_SCOPE' " + + "AND key='temperature'", + sql.getValue()); + } + + @Test + void find_returnsEmptyWhenNoRow() throws Exception { + TestContext context = newContext(); + SessionDataSet emptyDataSet = dataSet(); + when(context.session().executeQueryStatement(anyString())).thenReturn(emptyDataSet); + + Optional result = + context.dao().find(TENANT_ID, ENTITY_ID, AttributeScope.SERVER_SCOPE, "absent"); + + assertTrue(result.isEmpty()); + } + + @Test + void find_mapsAllFiveTypes() throws Exception { + assertSingleType("bool_v", true, DataType.BOOLEAN, true); + assertSingleType("long_v", 7L, DataType.LONG, 7L); + assertSingleType("double_v", 3.5D, DataType.DOUBLE, 3.5D); + assertSingleType("str_v", "hi", DataType.STRING, "hi"); + assertSingleType("json_v", "{\"a\":1}", DataType.JSON, "{\"a\":1}"); + } + + private void assertSingleType(String column, Object stored, DataType type, Object expected) + throws Exception { + TestContext context = newContext(); + SessionDataSet typedDataSet = dataSet(row(99L, column, stored)); + when(context.session().executeQueryStatement(anyString())).thenReturn(typedDataSet); + + AttributeKvEntry entry = + context.dao().find(TENANT_ID, ENTITY_ID, AttributeScope.SHARED_SCOPE, "k").orElseThrow(); + + assertEquals(type, entry.getDataType()); + assertEquals(expected, entry.getValue()); + assertEquals(99L, entry.getLastUpdateTs()); + } + + @Test + void findKeys_buildsInClauseAndMapsRowsPerKey() throws Exception { + TestContext context = newContext(); + SessionDataSet dataSet = + dataSet( + keyedRow("temperature", 100L, "long_v", 1L), + keyedRow("humidity", 200L, "double_v", 2.0D)); + when(context.session().executeQueryStatement(anyString())).thenReturn(dataSet); + + List result = + context + .dao() + .find( + TENANT_ID, + ENTITY_ID, + AttributeScope.SERVER_SCOPE, + List.of("temperature", "humidity")); + + assertEquals(2, result.size()); + assertEquals("temperature", result.get(0).getKey()); + assertEquals(1L, result.get(0).getValue()); + assertEquals(100L, result.get(0).getLastUpdateTs()); + assertEquals("humidity", result.get(1).getKey()); + + ArgumentCaptor sql = ArgumentCaptor.forClass(String.class); + verify(context.session(), timeout(3000)).executeQueryStatement(sql.capture()); + assertEquals( + "SELECT key, time, bool_v, long_v, double_v, str_v, json_v FROM entity_attributes " + + "WHERE tenant_id='11111111-1111-1111-1111-111111111111' " + + "AND entity_type='DEVICE' " + + "AND entity_id='22222222-2222-2222-2222-222222222222' " + + "AND attribute_scope='SERVER_SCOPE' " + + "AND key IN ('temperature','humidity')", + sql.getValue()); + } + + @Test + void findKeys_emptyCollectionReturnsEmptyAndSkipsQuery() throws Exception { + TestContext context = newContext(); + + assertEquals( + List.of(), + context.dao().find(TENANT_ID, ENTITY_ID, AttributeScope.SERVER_SCOPE, List.of())); + verify(context.pool(), never()).getSession(); + } + + @Test + void findAll_buildsScopeSqlAndMapsRowsPerKey() throws Exception { + TestContext context = newContext(); + SessionDataSet dataSet = + dataSet(keyedRow("a", 10L, "long_v", 1L), keyedRow("b", 20L, "str_v", "x")); + when(context.session().executeQueryStatement(anyString())).thenReturn(dataSet); + + List result = + context.dao().findAll(TENANT_ID, ENTITY_ID, AttributeScope.CLIENT_SCOPE); + + assertEquals(2, result.size()); + assertEquals("a", result.get(0).getKey()); + assertEquals("b", result.get(1).getKey()); + + ArgumentCaptor sql = ArgumentCaptor.forClass(String.class); + verify(context.session(), timeout(3000)).executeQueryStatement(sql.capture()); + assertEquals( + "SELECT key, time, bool_v, long_v, double_v, str_v, json_v FROM entity_attributes " + + "WHERE tenant_id='11111111-1111-1111-1111-111111111111' " + + "AND entity_type='DEVICE' " + + "AND entity_id='22222222-2222-2222-2222-222222222222' " + + "AND attribute_scope='CLIENT_SCOPE'", + sql.getValue()); + } + + @Test + void removeAll_returnsOneFuturePerKeyAndEachDeletes() throws Exception { + TestContext context = newContext(); + + List> futures = + context + .dao() + .removeAll( + TENANT_ID, ENTITY_ID, AttributeScope.SERVER_SCOPE, List.of("temperature", "speed")); + + assertEquals(2, futures.size()); + assertEquals("temperature", futures.get(0).get(3, TimeUnit.SECONDS)); + assertEquals("speed", futures.get(1).get(3, TimeUnit.SECONDS)); + + ArgumentCaptor sql = ArgumentCaptor.forClass(String.class); + verify(context.session(), timeout(3000).times(2)).executeNonQueryStatement(sql.capture()); + List statements = sql.getAllValues(); + assertTrue( + statements.stream() + .anyMatch( + s -> + s.equals( + "DELETE FROM entity_attributes " + + "WHERE tenant_id='11111111-1111-1111-1111-111111111111' " + + "AND entity_type='DEVICE' " + + "AND entity_id='22222222-2222-2222-2222-222222222222' " + + "AND attribute_scope='SERVER_SCOPE' " + + "AND key='temperature'"))); + assertTrue(statements.stream().anyMatch(s -> s.endsWith("AND key='speed'"))); + } + + @Test + void removeAllWithVersions_returnsTbPairKeyNullVersionPerKey() throws Exception { + TestContext context = newContext(); + + List>> futures = + context + .dao() + .removeAllWithVersions( + TENANT_ID, ENTITY_ID, AttributeScope.SERVER_SCOPE, List.of("temperature")); + + assertEquals(1, futures.size()); + TbPair pair = futures.get(0).get(3, TimeUnit.SECONDS); + assertEquals("temperature", pair.getFirst()); + assertNull(pair.getSecond()); + + verify(context.session(), timeout(3000)).executeNonQueryStatement(anyString()); + } + + @Test + void findNextBatch_throwsUnsupportedOperation() throws Exception { + TestContext context = newContext(); + + UnsupportedOperationException ex = + assertThrows( + UnsupportedOperationException.class, + () -> context.dao().findNextBatch(UUID.randomUUID(), 1, 1, 100)); + assertTrue(ex.getMessage().contains("migration helper")); + verify(context.pool(), never()).getSession(); + } + + @Test + void findAllKeysByEntityIds_buildsDistinctKeyByEntitySql() throws Exception { + TestContext context = newContext(); + SessionDataSet dataSet = dataSet(keyRow("temperature"), keyRow("humidity")); + when(context.session().executeQueryStatement(anyString())).thenReturn(dataSet); + + List keys = + context.dao().findAllKeysByEntityIds(TENANT_ID, List.of(ENTITY_ID, SECOND_ENTITY_ID)); + + assertEquals(List.of("temperature", "humidity"), keys); + + ArgumentCaptor sql = ArgumentCaptor.forClass(String.class); + verify(context.session(), timeout(3000)).executeQueryStatement(sql.capture()); + assertEquals( + "SELECT DISTINCT key FROM entity_attributes " + + "WHERE tenant_id='11111111-1111-1111-1111-111111111111' " + + "AND ((entity_type='DEVICE' AND entity_id='22222222-2222-2222-2222-222222222222') " + + "OR (entity_type='ASSET' AND entity_id='33333333-3333-3333-3333-333333333333')) " + + "ORDER BY key", + sql.getValue()); + } + + @Test + void findAllKeysByEntityIdsAndScope_addsScopePredicate() throws Exception { + TestContext context = newContext(); + SessionDataSet dataSet = dataSet(keyRow("temperature")); + when(context.session().executeQueryStatement(anyString())).thenReturn(dataSet); + + List keys = + context + .dao() + .findAllKeysByEntityIdsAndScope( + TENANT_ID, List.of(ENTITY_ID), AttributeScope.SHARED_SCOPE); + + assertEquals(List.of("temperature"), keys); + + ArgumentCaptor sql = ArgumentCaptor.forClass(String.class); + verify(context.session(), timeout(3000)).executeQueryStatement(sql.capture()); + assertEquals( + "SELECT DISTINCT key FROM entity_attributes " + + "WHERE tenant_id='11111111-1111-1111-1111-111111111111' " + + "AND attribute_scope='SHARED_SCOPE' " + + "AND ((entity_type='DEVICE' AND entity_id='22222222-2222-2222-2222-222222222222')) " + + "ORDER BY key", + sql.getValue()); + } + + @Test + void findAllKeysByEntityIdsAndScopeAsync_runsTheSameDistinctKeySql() throws Exception { + // [v4.3.1.2] async wrapper: same SQL as the synchronous overload, on the IO executor. + TestContext context = newContext(); + SessionDataSet dataSet = dataSet(keyRow("temperature")); + when(context.session().executeQueryStatement(anyString())).thenReturn(dataSet); + + List keys = + context + .dao() + .findAllKeysByEntityIdsAndScopeAsync( + TENANT_ID, List.of(ENTITY_ID), AttributeScope.SHARED_SCOPE) + .get(3, TimeUnit.SECONDS); + + assertEquals(List.of("temperature"), keys); + + ArgumentCaptor sql = ArgumentCaptor.forClass(String.class); + verify(context.session(), timeout(3000)).executeQueryStatement(sql.capture()); + assertEquals( + "SELECT DISTINCT key FROM entity_attributes " + + "WHERE tenant_id='11111111-1111-1111-1111-111111111111' " + + "AND attribute_scope='SHARED_SCOPE' " + + "AND ((entity_type='DEVICE' AND entity_id='22222222-2222-2222-2222-222222222222')) " + + "ORDER BY key", + sql.getValue()); + } + + @Test + void findAllKeysByEntityIdsAndScopeAsync_emptyEntityIdsReturnsImmediateEmpty() throws Exception { + TestContext context = newContext(); + + List keys = + context + .dao() + .findAllKeysByEntityIdsAndScopeAsync(TENANT_ID, List.of(), AttributeScope.SHARED_SCOPE) + .get(3, TimeUnit.SECONDS); + + assertEquals(List.of(), keys); + verify(context.pool(), never()).getSession(); + } + + @Test + void findLatestByEntityIdsAndScope_buildsBulkOrSetSqlAndMapsRows() throws Exception { + // [v4.3.1.2] bulk latest read: one SELECT over the entity OR-set in a scope, one row per + // (entity, key) because delete-then-insert keeps a single current row per identity. + TestContext context = newContext(); + SessionDataSet dataSet = + dataSet(keyedRow("a", 10L, "long_v", 1L), keyedRow("b", 20L, "str_v", "x")); + when(context.session().executeQueryStatement(anyString())).thenReturn(dataSet); + + List result = + context + .dao() + .findLatestByEntityIdsAndScope( + TENANT_ID, List.of(ENTITY_ID, SECOND_ENTITY_ID), AttributeScope.SERVER_SCOPE); + + assertEquals(2, result.size()); + assertEquals("a", result.get(0).getKey()); + assertEquals(1L, result.get(0).getValue()); + assertEquals(10L, result.get(0).getLastUpdateTs()); + assertNull(result.get(0).getVersion()); + assertEquals("b", result.get(1).getKey()); + + ArgumentCaptor sql = ArgumentCaptor.forClass(String.class); + verify(context.session(), timeout(3000)).executeQueryStatement(sql.capture()); + assertEquals( + "SELECT key, time, bool_v, long_v, double_v, str_v, json_v FROM entity_attributes " + + "WHERE tenant_id='11111111-1111-1111-1111-111111111111' " + + "AND attribute_scope='SERVER_SCOPE' " + + "AND ((entity_type='DEVICE' AND entity_id='22222222-2222-2222-2222-222222222222') " + + "OR (entity_type='ASSET' AND entity_id='33333333-3333-3333-3333-333333333333'))", + sql.getValue()); + } + + @Test + void findLatestByEntityIdsAndScopeAsync_runsTheSameBulkRead() throws Exception { + TestContext context = newContext(); + SessionDataSet dataSet = dataSet(keyedRow("a", 10L, "long_v", 1L)); + when(context.session().executeQueryStatement(anyString())).thenReturn(dataSet); + + List result = + context + .dao() + .findLatestByEntityIdsAndScopeAsync( + TENANT_ID, List.of(ENTITY_ID), AttributeScope.SERVER_SCOPE) + .get(3, TimeUnit.SECONDS); + + assertEquals(1, result.size()); + assertEquals("a", result.get(0).getKey()); + assertEquals(1L, result.get(0).getValue()); + + ArgumentCaptor sql = ArgumentCaptor.forClass(String.class); + verify(context.session(), timeout(3000)).executeQueryStatement(sql.capture()); + assertEquals( + "SELECT key, time, bool_v, long_v, double_v, str_v, json_v FROM entity_attributes " + + "WHERE tenant_id='11111111-1111-1111-1111-111111111111' " + + "AND attribute_scope='SERVER_SCOPE' " + + "AND ((entity_type='DEVICE' AND entity_id='22222222-2222-2222-2222-222222222222'))", + sql.getValue()); + } + + @Test + void findLatestByEntityIdsAndScopeAsync_emptyEntityIdsReturnsImmediateEmpty() throws Exception { + TestContext context = newContext(); + + List result = + context + .dao() + .findLatestByEntityIdsAndScopeAsync(TENANT_ID, List.of(), AttributeScope.SERVER_SCOPE) + .get(3, TimeUnit.SECONDS); + + assertEquals(List.of(), result); + verify(context.pool(), never()).getSession(); + } + + @Test + void findAllKeysByEntityIds_emptyListReturnsEmptyAndSkipsQuery() throws Exception { + TestContext context = newContext(); + + assertEquals(List.of(), context.dao().findAllKeysByEntityIds(TENANT_ID, List.of())); + verify(context.pool(), never()).getSession(); + } + + @Test + void findAllKeysByDeviceProfileId_nullProfileReturnsTenantWideDistinctKeys() throws Exception { + TestContext context = newContext(); + SessionDataSet dataSet = dataSet(keyRow("temperature"), keyRow("humidity")); + when(context.session().executeQueryStatement(anyString())).thenReturn(dataSet); + + List keys = context.dao().findAllKeysByDeviceProfileId(TENANT_ID, null); + + assertEquals(List.of("temperature", "humidity"), keys); + ArgumentCaptor sql = ArgumentCaptor.forClass(String.class); + verify(context.session(), timeout(3000)).executeQueryStatement(sql.capture()); + assertEquals( + "SELECT DISTINCT key FROM entity_attributes " + + "WHERE tenant_id='11111111-1111-1111-1111-111111111111' " + + "ORDER BY key", + sql.getValue()); + } + + @Test + void findAllKeysByDeviceProfileId_nonNullProfileReturnsEmptyDeferred() throws Exception { + TestContext context = newContext(); + + List keys = + context + .dao() + .findAllKeysByDeviceProfileId( + TENANT_ID, + new DeviceProfileId(UUID.fromString("44444444-4444-4444-4444-444444444444"))); + + assertEquals(List.of(), keys); + verify(context.pool(), never()).getSession(); + } + + @Test + void removeAllByEntityId_selectsThenDeletesAndReturnsScopeKeyPairs() throws Exception { + TestContext context = newContext(); + SessionDataSet selectResult = + dataSet(scopeKeyRow("SERVER_SCOPE", "temperature"), scopeKeyRow("CLIENT_SCOPE", "active")); + when(context.session().executeQueryStatement(anyString())).thenReturn(selectResult); + + List> removed = + context.dao().removeAllByEntityId(TENANT_ID, ENTITY_ID); + + assertEquals(2, removed.size()); + assertEquals(AttributeScope.SERVER_SCOPE, removed.get(0).getLeft()); + assertEquals("temperature", removed.get(0).getRight()); + assertEquals(AttributeScope.CLIENT_SCOPE, removed.get(1).getLeft()); + assertEquals("active", removed.get(1).getRight()); + + ArgumentCaptor selectSql = ArgumentCaptor.forClass(String.class); + verify(context.session(), timeout(3000)).executeQueryStatement(selectSql.capture()); + assertEquals( + "SELECT DISTINCT attribute_scope, key FROM entity_attributes " + + "WHERE tenant_id='11111111-1111-1111-1111-111111111111' " + + "AND entity_type='DEVICE' " + + "AND entity_id='22222222-2222-2222-2222-222222222222' " + + "ORDER BY attribute_scope, key", + selectSql.getValue()); + + ArgumentCaptor deleteSql = ArgumentCaptor.forClass(String.class); + verify(context.session(), timeout(3000)).executeNonQueryStatement(deleteSql.capture()); + assertEquals( + "DELETE FROM entity_attributes " + + "WHERE tenant_id='11111111-1111-1111-1111-111111111111' " + + "AND entity_type='DEVICE' " + + "AND entity_id='22222222-2222-2222-2222-222222222222'", + deleteSql.getValue()); + } + + @Test + void removeAllByEntityId_writeLockExcludesConcurrentSaveUntilItCompletes() throws Exception { + // removeAllByEntityId holds the per-entity WRITE lock around its select+delete; a concurrent + // same-entity save holds the per-entity READ lock, so the save cannot run its + // delete-then-insert + // until removeAllByEntityId releases the write lock. This deterministically proves the added + // serialization (no row can be re-inserted between the entity-wide select and delete). It is + // made deterministic by blocking the removeAll SELECT on a latch while the write lock is held, + // then asserting the save's writes have not yet executed. + TestContext context = newContext(); + CountDownLatch selectEntered = new CountDownLatch(1); + CountDownLatch releaseSelect = new CountDownLatch(1); + SessionDataSet selectResult = dataSet(scopeKeyRow("SERVER_SCOPE", "temperature")); + // The first query is removeAllByEntityId's SELECT DISTINCT; block it (while the entity write + // lock is held) until the test releases it. + when(context.session().executeQueryStatement(anyString())) + .thenAnswer( + invocation -> { + selectEntered.countDown(); + if (!releaseSelect.await(5, TimeUnit.SECONDS)) { + throw new AssertionError("removeAll SELECT was not released in time"); + } + return selectResult; + }); + + Thread remover = + new Thread(() -> context.dao().removeAllByEntityId(TENANT_ID, ENTITY_ID), "remover"); + remover.start(); + // Wait until removeAllByEntityId is inside its SELECT, i.e. holding the entity write lock. + assertTrue(selectEntered.await(5, TimeUnit.SECONDS), "removeAll did not reach its SELECT"); + + // Start a concurrent same-entity save; it must block on the entity read lock and NOT issue any + // DELETE/INSERT while the write lock is held. + ListenableFuture saveFuture = + context + .dao() + .save( + TENANT_ID, + ENTITY_ID, + AttributeScope.SERVER_SCOPE, + attribute(7L, "humidity", new LongDataEntry("humidity", 5L))); + + // Give the save's io-task a chance to run; it must remain blocked on the entity read lock. + Thread.sleep(200L); + verify(context.session(), never()).executeNonQueryStatement(anyString()); + assertFalse(saveFuture.isDone(), "save must not complete while removeAll holds the write lock"); + + // Release removeAll; it finishes (its DELETE runs), then the save acquires the read lock and + // runs its delete-then-insert. + releaseSelect.countDown(); + saveFuture.get(5, TimeUnit.SECONDS); + remover.join(5_000L); + + // Both the removeAll entity-wide DELETE and the save's identity DELETE must have executed; the + // save's INSERT (tablet) too. Two executeNonQueryStatement calls = removeAll DELETE + save + // DELETE (the save INSERT goes through session.insert, not executeNonQueryStatement). + verify(context.session(), timeout(3000).times(2)).executeNonQueryStatement(anyString()); + } + + @Test + void findTakesPerIdentityLockSoItCannotReadDuringTheDeleteInsertGap() throws Exception { + // find(single) takes the same per-identity lock save() holds across its (non-atomic) + // delete-then-insert, so a concurrent same-identity point-read cannot run its SELECT during the + // save's critical section -- it therefore never observes the transient empty window between the + // DELETE and the INSERT. Deterministic: block the save's DELETE on a latch while it holds the + // identity lock, then assert a concurrent same-identity find() stays blocked (never issues its + // SELECT) until the save releases the lock. + TestContext context = newContext(); + CountDownLatch deleteEntered = new CountDownLatch(1); + CountDownLatch releaseDelete = new CountDownLatch(1); + doAnswer( + invocation -> { + deleteEntered.countDown(); + if (!releaseDelete.await(5, TimeUnit.SECONDS)) { + throw new AssertionError("save DELETE was not released in time"); + } + return null; + }) + .when(context.session()) + .executeNonQueryStatement(anyString()); + SessionDataSet valueRow = dataSet(row(7L, "long_v", 5L)); + when(context.session().executeQueryStatement(anyString())).thenReturn(valueRow); + + ListenableFuture saveFuture = + context + .dao() + .save( + TENANT_ID, + ENTITY_ID, + AttributeScope.SERVER_SCOPE, + attribute(7L, "humidity", new LongDataEntry("humidity", 5L))); + assertTrue(deleteEntered.await(5, TimeUnit.SECONDS), "save did not reach its DELETE"); + + CountDownLatch findDone = new CountDownLatch(1); + boolean[] present = {false}; + Thread finder = + new Thread( + () -> { + present[0] = + context + .dao() + .find(TENANT_ID, ENTITY_ID, AttributeScope.SERVER_SCOPE, "humidity") + .isPresent(); + findDone.countDown(); + }, + "finder"); + finder.start(); + + // While the save holds the identity lock (its DELETE is blocked), find() must block on the same + // identity lock and NOT run its SELECT. + Thread.sleep(200L); + assertEquals( + 1L, findDone.getCount(), "find() must block on the per-identity lock during the save"); + verify(context.session(), never()).executeQueryStatement(anyString()); + + // Release the save; it completes and releases the identity lock; find() then runs and returns. + releaseDelete.countDown(); + saveFuture.get(5, TimeUnit.SECONDS); + assertTrue( + findDone.await(5, TimeUnit.SECONDS), + "find() should complete after the save releases the lock"); + finder.join(5_000L); + assertTrue(present[0], "find() must return the value, never the empty delete->insert gap"); + } + + @Test + void save_escapesSingleQuotesInScopeAndKey() throws Exception { + TestContext context = newContext(); + + context + .dao() + .save( + TENANT_ID, + ENTITY_ID, + AttributeScope.SERVER_SCOPE, + attribute(5L, "a'b", new StringDataEntry("a'b", "v"))) + .get(3, TimeUnit.SECONDS); + + ArgumentCaptor deleteSql = ArgumentCaptor.forClass(String.class); + verify(context.session(), timeout(3000)).executeNonQueryStatement(deleteSql.capture()); + assertTrue(deleteSql.getValue().contains("key='a''b'")); + } + + @Test + void save_blankKeyFailsFast() { + TestContext context = newContext(); + + IllegalArgumentException ex = + assertThrows( + IllegalArgumentException.class, + () -> + context + .dao() + .save( + TENANT_ID, + ENTITY_ID, + AttributeScope.SERVER_SCOPE, + attribute(1L, " ", new LongDataEntry(" ", 1L)))); + assertTrue(ex.getMessage().contains("must not be blank")); + } + + @Test + void constructor_failsWhenClusterModeUnset() { + ITableSessionPool pool = mock(ITableSessionPool.class); + IoTDBTableConfig config = new IoTDBTableConfig(); + config.getTs().getRead().setThreads(1); + // iotdb.attributes.cluster_mode defaults to blank -> must fail fast. + + IllegalStateException ex = + assertThrows(IllegalStateException.class, () -> new IoTDBTableAttributesDao(pool, config)); + assertTrue(ex.getMessage().contains("cluster_mode")); + } + + @Test + void constructor_acceptsDisabledClusterMode() { + ITableSessionPool pool = mock(ITableSessionPool.class); + IoTDBTableConfig config = new IoTDBTableConfig(); + config.getTs().getRead().setThreads(1); + config.getAttributes().setClusterMode("disabled"); + + IoTDBTableAttributesDao dao = new IoTDBTableAttributesDao(pool, config); + daos.add(dao); + } + + // ---- helpers (mirrors IoTDBTableLatestDaoTest) ---- + + private TestContext newContext() { + ITableSessionPool pool = mock(ITableSessionPool.class); + ITableSession session = mock(ITableSession.class); + try { + when(pool.getSession()).thenReturn(session); + } catch (IoTDBConnectionException e) { + throw new AssertionError(e); + } + IoTDBTableConfig config = new IoTDBTableConfig(); + config.getTs().getRead().setThreads(1); + config.getAttributes().setClusterMode("sticky-routing"); + IoTDBTableAttributesDao dao = new IoTDBTableAttributesDao(pool, config); + daos.add(dao); + return new TestContext(dao, pool, session); + } + + private TestAttributeKvEntry attribute(long lastUpdateTs, String key, KvEntry value) { + return new TestAttributeKvEntry(lastUpdateTs, value); + } + + private SessionDataSet dataSet(MockRow... rows) + throws IoTDBConnectionException, StatementExecutionException { + SessionDataSet dataSet = mock(SessionDataSet.class); + SessionDataSet.DataIterator iterator = mock(SessionDataSet.DataIterator.class); + AtomicInteger index = new AtomicInteger(-1); + when(dataSet.iterator()).thenReturn(iterator); + when(iterator.next()).thenAnswer(invocation -> index.incrementAndGet() < rows.length); + when(iterator.isNull(anyString())) + .thenAnswer(invocation -> rows[index.get()].isNull(invocation.getArgument(0))); + when(iterator.getBoolean(anyString())) + .thenAnswer(invocation -> rows[index.get()].get(invocation.getArgument(0))); + when(iterator.getLong(anyString())) + .thenAnswer(invocation -> rows[index.get()].get(invocation.getArgument(0))); + when(iterator.getDouble(anyString())) + .thenAnswer(invocation -> rows[index.get()].get(invocation.getArgument(0))); + when(iterator.getString(anyString())) + .thenAnswer(invocation -> String.valueOf(rows[index.get()].get(invocation.getArgument(0)))); + when(iterator.getTimestamp(anyString())) + .thenAnswer( + invocation -> new Timestamp((Long) rows[index.get()].get(invocation.getArgument(0)))); + return dataSet; + } + + /** A find()-single-key row exposing {@code time} plus one typed column. */ + private MockRow row(long ts, String column, Object value) { + Map columns = new HashMap<>(); + columns.put("time", ts); + columns.put(column, value); + return new MockRow(columns); + } + + /** A find(keys)/findAll row exposing {@code key}, {@code time} and one typed column. */ + private MockRow keyedRow(String key, long ts, String column, Object value) { + Map columns = new HashMap<>(); + columns.put("key", key); + columns.put("time", ts); + columns.put(column, value); + return new MockRow(columns); + } + + /** A DISTINCT key row exposing only the {@code key} column. */ + private MockRow keyRow(String key) { + Map columns = new HashMap<>(); + columns.put("key", key); + return new MockRow(columns); + } + + /** A DISTINCT (scope, key) row for removeAllByEntityId. */ + private MockRow scopeKeyRow(String scope, String key) { + Map columns = new HashMap<>(); + columns.put("attribute_scope", scope); + columns.put("key", key); + return new MockRow(columns); + } + + private static final class MockRow { + private final Map columns; + + private MockRow(Map columns) { + this.columns = columns; + } + + private boolean isNull(String column) { + return columns.get(column) == null; + } + + private Object get(String column) { + return columns.get(column); + } + } + + private record TestContext( + IoTDBTableAttributesDao dao, ITableSessionPool pool, ITableSession session) {} + + private record TestEntityId(UUID id, EntityType entityType) implements EntityId { + @Override + public UUID getId() { + return id; + } + + @Override + public EntityType getEntityType() { + return entityType; + } + } + + private static final class TestAttributeKvEntry extends BaseAttributeKvEntry { + private TestAttributeKvEntry(long lastUpdateTs, KvEntry kv) { + super(kv, lastUpdateTs); + } + } +} diff --git a/iotdb-thingsboard-table/src/test/java/org/apache/iotdb/extras/thingsboard/table/IoTDBTableAutoConfigurationTest.java b/iotdb-thingsboard-table/src/test/java/org/apache/iotdb/extras/thingsboard/table/IoTDBTableAutoConfigurationTest.java index d2179a05..7781f9f6 100644 --- a/iotdb-thingsboard-table/src/test/java/org/apache/iotdb/extras/thingsboard/table/IoTDBTableAutoConfigurationTest.java +++ b/iotdb-thingsboard-table/src/test/java/org/apache/iotdb/extras/thingsboard/table/IoTDBTableAutoConfigurationTest.java @@ -27,15 +27,19 @@ import org.springframework.boot.test.context.runner.ApplicationContextRunner; import org.springframework.context.annotation.Bean; import org.springframework.context.annotation.Configuration; +import org.thingsboard.server.common.data.id.DeviceProfileId; import org.thingsboard.server.common.data.id.EntityId; import org.thingsboard.server.common.data.id.TenantId; import org.thingsboard.server.common.data.kv.DeleteTsKvQuery; import org.thingsboard.server.common.data.kv.ReadTsKvQuery; import org.thingsboard.server.common.data.kv.ReadTsKvQueryResult; import org.thingsboard.server.common.data.kv.TsKvEntry; +import org.thingsboard.server.common.data.kv.TsKvLatestRemovingResult; import org.thingsboard.server.dao.timeseries.TimeseriesDao; +import org.thingsboard.server.dao.timeseries.TimeseriesLatestDao; import java.util.List; +import java.util.Optional; import static org.assertj.core.api.Assertions.assertThat; import static org.junit.jupiter.api.Assertions.assertFalse; @@ -82,6 +86,34 @@ void autoConfigDiscovery_withSelector_createsPoolAndDao() { }); } + @Test + void autoConfigDiscovery_withLatestSelector_createsLatestDaoAndDoesNotFailFast() { + contextRunner + .withPropertyValues( + "database.ts.type=iotdb-table", + "database.ts_latest.type=iotdb-table", + "iotdb.ts.experimental-raw-only=true", + "iotdb.host=localhost", + "iotdb.port=6667", + "iotdb.username=root", + "iotdb.password=root", + "iotdb.session-pool-size=8", + "iotdb.connection-timeout-ms=5000", + "iotdb.schema.bootstrap=false") + .run( + context -> { + // All three selectors on: our own latest DAO activates and the conflict guard, seeing + // only our IoTDB DAO, must NOT fail startup. + assertTrue( + context.getStartupFailure() == null, + "context should start when the only latest DAO is the IoTDB one"); + assertTrue( + context.containsBeanDefinition( + IoTDBTableConfiguration.IOTDB_TABLE_LATEST_DAO_BEAN_NAME)); + assertTrue(context.getBean(IoTDBTableLatestDao.class) != null); + }); + } + @Test void autoConfigDiscovery_withSelectorButNoExperimentalFlag_createsNoBeans() { contextRunner @@ -243,6 +275,65 @@ void hostProvidedTimeseriesDaoUsingModuleBeanName_failsFastWhenIoTDBBackendEnabl }); } + @Test + void hostProvidedTimeseriesLatestDao_failsFastWhenLatestBackendEnabled() { + contextRunner + .withUserConfiguration(HostTimeseriesLatestDaoConfiguration.class) + .withPropertyValues( + "database.ts.type=iotdb-table", + "database.ts_latest.type=iotdb-table", + "iotdb.ts.experimental-raw-only=true", + "iotdb.host=localhost", + "iotdb.port=6667", + "iotdb.username=root", + "iotdb.password=root", + "iotdb.session-pool-size=8", + "iotdb.connection-timeout-ms=5000", + "iotdb.schema.bootstrap=false") + .run( + context -> { + Throwable failure = context.getStartupFailure(); + assertTrue(failure != null, "context should fail on conflicting TimeseriesLatestDao"); + assertTrue( + failureContains( + failure, + "database.ts_latest.type=iotdb-table with the IoTDB timeseries backend " + + "enabled, but a non-IoTDB TimeseriesLatestDao bean " + + "'hostTimeseriesLatestDao' is present; remove it or unset the IoTDB " + + "latest selector"), + "startup failure should explain the conflicting TimeseriesLatestDao bean: " + + failure); + }); + } + + @Test + void hostProvidedTimeseriesLatestDaoUsingModuleBeanName_failsFastWhenLatestBackendEnabled() { + contextRunner + .withUserConfiguration(HostNamedTimeseriesLatestDaoConfiguration.class) + .withPropertyValues( + "database.ts.type=iotdb-table", + "database.ts_latest.type=iotdb-table", + "iotdb.ts.experimental-raw-only=true", + "iotdb.host=localhost", + "iotdb.port=6667", + "iotdb.username=root", + "iotdb.password=root", + "iotdb.session-pool-size=8", + "iotdb.connection-timeout-ms=5000", + "iotdb.schema.bootstrap=false") + .run( + context -> { + Throwable failure = context.getStartupFailure(); + assertTrue( + failure != null, + "context should fail on a host TimeseriesLatestDao using the module bean name"); + assertTrue( + failureContains(failure, "non-IoTDB TimeseriesLatestDao bean"), + "startup failure should explain the conflicting TimeseriesLatestDao bean: " + + failure); + }); + } + private static boolean failureContains(Throwable failure, String expected) { for (Throwable cause = failure; cause != null; cause = cause.getCause()) { String message = cause.getMessage(); @@ -302,4 +393,80 @@ public void cleanup(long systemTtl) { throw new UnsupportedOperationException(); } } + + @Configuration + static class HostTimeseriesLatestDaoConfiguration { + @Bean + TimeseriesLatestDao hostTimeseriesLatestDao() { + return new NoopTimeseriesLatestDao(); + } + } + + @Configuration + static class HostNamedTimeseriesLatestDaoConfiguration { + @Bean(name = IoTDBTableConfiguration.IOTDB_TABLE_LATEST_DAO_BEAN_NAME) + TimeseriesLatestDao ioTDBTableLatestDao() { + return new NoopTimeseriesLatestDao(); + } + } + + /** Minimal host-supplied {@link TimeseriesLatestDao} used only to trigger the conflict guard. */ + private static final class NoopTimeseriesLatestDao implements TimeseriesLatestDao { + @Override + public ListenableFuture> findLatestOpt( + TenantId tenantId, EntityId entityId, String key) { + throw new UnsupportedOperationException(); + } + + @Override + public ListenableFuture findLatest( + TenantId tenantId, EntityId entityId, String key) { + throw new UnsupportedOperationException(); + } + + @Override + public ListenableFuture> findAllLatest(TenantId tenantId, EntityId entityId) { + throw new UnsupportedOperationException(); + } + + @Override + public ListenableFuture saveLatest( + TenantId tenantId, EntityId entityId, TsKvEntry tsKvEntry) { + throw new UnsupportedOperationException(); + } + + @Override + public ListenableFuture removeLatest( + TenantId tenantId, EntityId entityId, DeleteTsKvQuery query) { + throw new UnsupportedOperationException(); + } + + @Override + public List findAllKeysByDeviceProfileId( + TenantId tenantId, DeviceProfileId deviceProfileId) { + throw new UnsupportedOperationException(); + } + + @Override + public List findAllKeysByEntityIds(TenantId tenantId, List entityIds) { + throw new UnsupportedOperationException(); + } + + @Override + public ListenableFuture> findAllKeysByEntityIdsAsync( + TenantId tenantId, List entityIds) { + throw new UnsupportedOperationException(); + } + + @Override + public List findLatestByEntityIds(TenantId tenantId, List entityIds) { + throw new UnsupportedOperationException(); + } + + @Override + public ListenableFuture> findLatestByEntityIdsAsync( + TenantId tenantId, List entityIds) { + throw new UnsupportedOperationException(); + } + } } diff --git a/iotdb-thingsboard-table/src/test/java/org/apache/iotdb/extras/thingsboard/table/IoTDBTableContextStartupTest.java b/iotdb-thingsboard-table/src/test/java/org/apache/iotdb/extras/thingsboard/table/IoTDBTableContextStartupTest.java index 0acca26b..a5bb7abd 100644 --- a/iotdb-thingsboard-table/src/test/java/org/apache/iotdb/extras/thingsboard/table/IoTDBTableContextStartupTest.java +++ b/iotdb-thingsboard-table/src/test/java/org/apache/iotdb/extras/thingsboard/table/IoTDBTableContextStartupTest.java @@ -51,9 +51,108 @@ void noActivation_contextStartsWithoutIoTDBBeans() { assertFalse(context.containsBeanDefinition("ioTDBTableTimeseriesDao")); assertFalse(context.containsBeanDefinition("ioTDBTableLatestDao")); assertFalse(context.containsBeanDefinition("ioTDBTableLabelDao")); + // Path-3 stretch: the attribute DAO is inert by default (no attributes selector set). + assertFalse(context.containsBeanDefinition("ioTDBTableAttributesDao")); }); } + // The attribute selector is INDEPENDENT of the timeseries selectors: when ONLY + // database.attributes.type=iotdb-table is set (and cluster_mode is acknowledged) the attribute + // DAO + its own pool/bootstrap activate, while the timeseries DAO stays absent. + @Test + void attributesSelectorAlone_activatesAttributesDaoAndPoolIndependentOfTs() { + contextRunner + .withPropertyValues( + "database.attributes.type=iotdb-table", + "iotdb.attributes.cluster_mode=disabled", + "iotdb.host=localhost", + "iotdb.port=6667", + "iotdb.username=root", + "iotdb.password=root", + "iotdb.session-pool-size=8", + "iotdb.connection-timeout-ms=5000", + "iotdb.schema.bootstrap=false") + .run( + context -> { + assertTrue(context.containsBean(SESSION_POOL_BEAN_NAME)); + assertTrue(context.containsBeanDefinition("ioTDBTableAttributesDao")); + assertTrue(context.getBean(IoTDBTableAttributesDao.class) != null); + // No timeseries DAO: the ts selector is unset. + assertFalse(context.containsBeanDefinition("ioTDBTableTimeseriesDao")); + assertTrue(context.getBeansOfType(TimeseriesDao.class).isEmpty()); + }); + } + + // Uppercase, trimmed attribute selector still activates (case-insensitive condition). + @Test + void attributesSelectorUppercase_stillActivatesAttributesDao() { + contextRunner + .withPropertyValues( + "database.attributes.type=IOTDB-TABLE", + "iotdb.attributes.cluster_mode=sticky-routing", + "iotdb.host=localhost", + "iotdb.port=6667", + "iotdb.username=root", + "iotdb.password=root", + "iotdb.session-pool-size=8", + "iotdb.connection-timeout-ms=5000", + "iotdb.schema.bootstrap=false") + .run( + context -> { + assertTrue(context.containsBean(SESSION_POOL_BEAN_NAME)); + assertTrue(context.containsBeanDefinition("ioTDBTableAttributesDao")); + }); + } + + // The timeseries selectors alone must NOT activate the attribute DAO (independent routing). + @Test + void tsSelectorAlone_doesNotActivateAttributesDao() { + contextRunner + .withPropertyValues( + "database.ts.type=iotdb-table", + "iotdb.ts.experimental-raw-only=true", + "iotdb.host=localhost", + "iotdb.port=6667", + "iotdb.username=root", + "iotdb.password=root", + "iotdb.session-pool-size=8", + "iotdb.connection-timeout-ms=5000", + "iotdb.schema.bootstrap=false") + .run(context -> assertFalse(context.containsBeanDefinition("ioTDBTableAttributesDao"))); + } + + // Both selectors ON together: the timeseries and attribute DAOs activate side by side and SHARE a + // SINGLE named session pool (deduped via @ConditionalOnMissingBean(name=...)), rather than each + // inner @Configuration spinning up its own competing pool. This is the cross-selector path the + // per-config pool javadoc asserts in prose; pin it here. + @Test + void bothSelectors_activateBothDaosSharingOneSessionPool() { + contextRunner + .withPropertyValues( + "database.ts.type=iotdb-table", + "iotdb.ts.experimental-raw-only=true", + "database.attributes.type=iotdb-table", + "iotdb.attributes.cluster_mode=disabled", + "iotdb.host=localhost", + "iotdb.port=6667", + "iotdb.username=root", + "iotdb.password=root", + "iotdb.session-pool-size=8", + "iotdb.connection-timeout-ms=5000", + "iotdb.schema.bootstrap=false") + .run( + context -> { + // exactly ONE shared pool, not one per inner @Configuration + assertTrue( + context.getBeanNamesForType(ITableSessionPool.class).length == 1, + "both selectors must share a single session pool"); + assertTrue(context.containsBean(SESSION_POOL_BEAN_NAME)); + // both DAOs are present, wired against that one pool + assertTrue(context.getBean(IoTDBTableTimeseriesDao.class) != null); + assertTrue(context.getBean(IoTDBTableAttributesDao.class) != null); + }); + } + @Test void tsTypeActivationWithExperimentalFlag_createsPoolAndTimeseries() { contextRunner diff --git a/iotdb-thingsboard-table/src/test/java/org/apache/iotdb/extras/thingsboard/table/IoTDBTableLatestDaoIT.java b/iotdb-thingsboard-table/src/test/java/org/apache/iotdb/extras/thingsboard/table/IoTDBTableLatestDaoIT.java new file mode 100644 index 00000000..7d0f497b --- /dev/null +++ b/iotdb-thingsboard-table/src/test/java/org/apache/iotdb/extras/thingsboard/table/IoTDBTableLatestDaoIT.java @@ -0,0 +1,770 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package org.apache.iotdb.extras.thingsboard.table; + +import org.apache.iotdb.isession.ITableSession; +import org.apache.iotdb.isession.SessionDataSet; +import org.apache.iotdb.isession.pool.ITableSessionPool; +import org.apache.iotdb.session.pool.TableSessionPoolBuilder; + +import com.google.common.util.concurrent.ListenableFuture; +import org.junit.jupiter.api.Tag; +import org.junit.jupiter.api.Test; +import org.testcontainers.containers.GenericContainer; +import org.testcontainers.containers.wait.strategy.Wait; +import org.testcontainers.junit.jupiter.Container; +import org.testcontainers.junit.jupiter.Testcontainers; +import org.testcontainers.utility.DockerImageName; +import org.thingsboard.server.common.data.EntityType; +import org.thingsboard.server.common.data.id.EntityId; +import org.thingsboard.server.common.data.id.TenantId; +import org.thingsboard.server.common.data.kv.BaseDeleteTsKvQuery; +import org.thingsboard.server.common.data.kv.DataType; +import org.thingsboard.server.common.data.kv.TsKvEntry; +import org.thingsboard.server.common.data.kv.TsKvLatestRemovingResult; + +import java.io.InputStream; +import java.nio.charset.StandardCharsets; +import java.time.Duration; +import java.util.ArrayList; +import java.util.Comparator; +import java.util.List; +import java.util.Optional; +import java.util.UUID; +import java.util.concurrent.ExecutionException; +import java.util.concurrent.TimeUnit; + +import static org.junit.jupiter.api.Assertions.assertEquals; +import static org.junit.jupiter.api.Assertions.assertFalse; +import static org.junit.jupiter.api.Assertions.assertInstanceOf; +import static org.junit.jupiter.api.Assertions.assertNotNull; +import static org.junit.jupiter.api.Assertions.assertNull; +import static org.junit.jupiter.api.Assertions.assertThrows; +import static org.junit.jupiter.api.Assertions.assertTrue; + +@Tag("integration") +@Testcontainers(disabledWithoutDocker = true) +class IoTDBTableLatestDaoIT { + // Cold testcontainer first writes/reads are slower than a warm production node, so the + // per-future assertion timeout is generous; production throughput is covered elsewhere. + private static final int FUTURE_TIMEOUT_SECONDS = 30; + private static final Duration IOTDB_STARTUP_TIMEOUT = Duration.ofMinutes(3); + private static final Duration IOTDB_READY_TIMEOUT = Duration.ofSeconds(60); + private static final Duration IOTDB_READY_POLL_INTERVAL = Duration.ofMillis(500); + + @Container + static final GenericContainer IOTDB = + new GenericContainer<>(DockerImageName.parse("apache/iotdb:2.0.8-standalone")) + .withExposedPorts(6667) + // IoTDB binds its client RPC service to dn_rpc_address (default 127.0.0.1); bind to all + // interfaces so the Testcontainers port-mapped session handshake succeeds. + .withEnv("dn_rpc_address", "0.0.0.0") + .waitingFor(Wait.forListeningPort().withStartupTimeout(IOTDB_STARTUP_TIMEOUT)); + + @Test + void saveThenFindLatest_roundTripsAllFiveTypes() throws Exception { + TestScope scope = + scope( + "latest_types", + "55555555-5555-5555-5555-555555555501", + "66666666-6666-6666-6666-666666666601"); + bootstrapSchema(scope.database()); + bootstrapLatestSchema(scope.database()); + try (ITableSessionPool pool = newPool(scope.database())) { + IoTDBTableConfig config = config(5); + IoTDBTableTimeseriesWriter writer = new IoTDBTableTimeseriesWriter(pool, config); + IoTDBTableTimeseriesDao tsDao = new IoTDBTableTimeseriesDao(pool, writer, config); + IoTDBTableLatestDao latestDao = new IoTDBTableLatestDao(pool, config); + try { + // Two rows per key at ascending timestamps; the latest read must return the newer value. + saveAll( + tsDao, + scope, + List.of( + entry(1000L, "bool", DataType.BOOLEAN, false), + entry(1010L, "bool", DataType.BOOLEAN, true), + entry(1000L, "long", DataType.LONG, 1L), + entry(1010L, "long", DataType.LONG, 42L), + entry(1000L, "double", DataType.DOUBLE, 1.0D), + entry(1010L, "double", DataType.DOUBLE, 4.2D), + entry(1000L, "string", DataType.STRING, "old"), + entry(1010L, "string", DataType.STRING, "value"), + entry(1000L, "json", DataType.JSON, "{\"v\":0}"), + entry(1010L, "json", DataType.JSON, "{\"v\":1}"))); + + assertLatest(latestDao, scope, "bool", 1010L, DataType.BOOLEAN, true); + assertLatest(latestDao, scope, "long", 1010L, DataType.LONG, 42L); + assertLatest(latestDao, scope, "double", 1010L, DataType.DOUBLE, 4.2D); + assertLatest(latestDao, scope, "string", 1010L, DataType.STRING, "value"); + assertLatest(latestDao, scope, "json", 1010L, DataType.JSON, "{\"v\":1}"); + } finally { + latestDao.destroy(); + tsDao.destroy(); + writer.destroy(); + } + } + } + + @Test + void findAllLatest_returnsLatestPerKeyWithoutTypeBackfill() throws Exception { + TestScope scope = + scope( + "latest_all", + "55555555-5555-5555-5555-555555555502", + "66666666-6666-6666-6666-666666666602"); + bootstrapSchema(scope.database()); + bootstrapLatestSchema(scope.database()); + try (ITableSessionPool pool = newPool(scope.database())) { + IoTDBTableConfig config = config(8); + IoTDBTableTimeseriesWriter writer = new IoTDBTableTimeseriesWriter(pool, config); + IoTDBTableTimeseriesDao tsDao = new IoTDBTableTimeseriesDao(pool, writer, config); + IoTDBTableLatestDao latestDao = new IoTDBTableLatestDao(pool, config); + try { + // "mixed" changes type over time: long_v@2000 then str_v@2010. LAST_BY must return only the + // newer string value and must NOT backfill the older long column into the aggregated row. + saveAll( + tsDao, + scope, + List.of( + entry(2000L, "mixed", DataType.LONG, 7L), + entry(2010L, "mixed", DataType.STRING, "latest"), + entry(2000L, "temperature", DataType.DOUBLE, 20.0D), + entry(2005L, "temperature", DataType.DOUBLE, 21.5D), + entry(2000L, "online", DataType.BOOLEAN, true))); + + List latest = + latestDao + .findAllLatest(scope.tenantId(), scope.entityId()) + .get(FUTURE_TIMEOUT_SECONDS, TimeUnit.SECONDS); + latest.sort(Comparator.comparing(TsKvEntry::getKey)); + + // Sorted by key, so the order is: mixed, online, temperature. + assertEquals(3, latest.size()); + assertEntry(latest.get(0), 2010L, "mixed", DataType.STRING, "latest"); + assertEntry(latest.get(1), 2000L, "online", DataType.BOOLEAN, true); + assertEntry(latest.get(2), 2005L, "temperature", DataType.DOUBLE, 21.5D); + } finally { + latestDao.destroy(); + tsDao.destroy(); + writer.destroy(); + } + } + } + + @Test + void findLatest_emptyReturnsNullStringEntryFallback() throws Exception { + TestScope scope = + scope( + "latest_empty", + "55555555-5555-5555-5555-555555555503", + "66666666-6666-6666-6666-666666666603"); + bootstrapSchema(scope.database()); + bootstrapLatestSchema(scope.database()); + try (ITableSessionPool pool = newPool(scope.database())) { + IoTDBTableConfig config = config(1); + IoTDBTableLatestDao latestDao = new IoTDBTableLatestDao(pool, config); + try { + Optional opt = + latestDao + .findLatestOpt(scope.tenantId(), scope.entityId(), "never-written") + .get(FUTURE_TIMEOUT_SECONDS, TimeUnit.SECONDS); + assertTrue(opt.isEmpty()); + + TsKvEntry fallback = + latestDao + .findLatest(scope.tenantId(), scope.entityId(), "never-written") + .get(FUTURE_TIMEOUT_SECONDS, TimeUnit.SECONDS); + assertEquals("never-written", fallback.getKey()); + assertEquals(DataType.STRING, fallback.getDataType()); + assertNull(fallback.getValue()); + } finally { + latestDao.destroy(); + } + } + } + + @Test + void findLatest_failsOnCrossBatchSameTimestampTypeChange() throws Exception { + TestScope scope = + scope( + "latest_stale", + "55555555-5555-5555-5555-555555555505", + "66666666-6666-6666-6666-666666666605"); + bootstrapSchema(scope.database()); + bootstrapLatestSchema(scope.database()); + try (ITableSessionPool pool = newPool(scope.database())) { + IoTDBTableConfig config = config(1); + IoTDBTableTimeseriesWriter writer = new IoTDBTableTimeseriesWriter(pool, config); + IoTDBTableTimeseriesDao tsDao = new IoTDBTableTimeseriesDao(pool, writer, config); + IoTDBTableLatestDao latestDao = new IoTDBTableLatestDao(pool, config); + try { + // Cross-batch same-timestamp type change leaves two typed columns set at time=4000 (the + // documented Phase-1 limitation). The derived DESC-LIMIT-1 latest read must fail-fast. + assertEquals( + 1, + tsDao + .save( + scope.tenantId(), scope.entityId(), entry(4000L, "flip", DataType.LONG, 7L), 0) + .get(FUTURE_TIMEOUT_SECONDS, TimeUnit.SECONDS)); + assertEquals( + 1, + tsDao + .save( + scope.tenantId(), + scope.entityId(), + entry(4000L, "flip", DataType.STRING, "seven"), + 0) + .get(FUTURE_TIMEOUT_SECONDS, TimeUnit.SECONDS)); + + ExecutionException failure = + assertThrows( + ExecutionException.class, + () -> + latestDao + .findLatest(scope.tenantId(), scope.entityId(), "flip") + .get(FUTURE_TIMEOUT_SECONDS, TimeUnit.SECONDS)); + assertInstanceOf(IllegalStateException.class, failure.getCause()); + assertTrue(failure.getCause().getMessage().contains("2 typed value columns set")); + } finally { + latestDao.destroy(); + tsDao.destroy(); + writer.destroy(); + } + } + } + + // ---- latest overlay (telemetry_latest): second bootstrap, saveLatest, merge, removeLatest ---- + + @Test + void latestOverlayBootstrap_createsTelemetryLatestTable() throws Exception { + TestScope scope = + scope( + "lt_boot", + "55555555-5555-5555-5555-555555555510", + "66666666-6666-6666-6666-666666666610"); + bootstrapSchema(scope.database()); + // The SECOND bootstrap (schema-iotdb-table-latest.sql via the 3-arg ctor) creates the overlay. + bootstrapLatestSchema(scope.database()); + try (ITableSessionPool pool = newPool(scope.database())) { + // Querying telemetry_latest must succeed (table exists); it is empty until a saveLatest. + try (ITableSession session = pool.getSession(); + SessionDataSet dataSet = + session.executeQueryStatement("SELECT key FROM telemetry_latest")) { + assertFalse(dataSet.iterator().next(), "telemetry_latest should exist and be empty"); + } + // Idempotency: a second bootstrap run must not throw. + bootstrapLatestSchema(scope.database()); + } + } + + @Test + void withoutLatestBootstrap_telemetryLatestTableIsNotCreated() throws Exception { + // The latest overlay DDL lives in a SEPARATE resource gated by the latest selector; the base + // bootstrap alone must NOT create telemetry_latest (latest-off => overlay table absent). + TestScope scope = + scope( + "lt_off", + "55555555-5555-5555-5555-555555555511", + "66666666-6666-6666-6666-666666666611"); + bootstrapSchema(scope.database()); + try (ITableSessionPool pool = newPool(scope.database()); + ITableSession session = pool.getSession()) { + assertThrows( + Exception.class, () -> session.executeQueryStatement("SELECT key FROM telemetry_latest")); + } + } + + @Test + void latestOnlySaveLatest_thenFindLatest_roundTripsWithoutPairedSave() throws Exception { + // Load-bearing EntityView LATEST_AND_WS case: saveLatest is called with NO paired tsDao.save(), + // so nothing writes the telemetry table. The overlay must capture the value and findLatest must + // return it (the no-shadow-table no-op would have dropped it). + TestScope scope = + scope( + "lt_only", + "55555555-5555-5555-5555-555555555512", + "66666666-6666-6666-6666-666666666612"); + bootstrapSchema(scope.database()); + bootstrapLatestSchema(scope.database()); + try (ITableSessionPool pool = newPool(scope.database())) { + IoTDBTableConfig config = config(1); + IoTDBTableLatestDao latestDao = new IoTDBTableLatestDao(pool, config); + try { + saveLatest(latestDao, scope, entry(1000L, "bool", DataType.BOOLEAN, true)); + saveLatest(latestDao, scope, entry(1000L, "long", DataType.LONG, 42L)); + saveLatest(latestDao, scope, entry(1000L, "double", DataType.DOUBLE, 4.2D)); + saveLatest(latestDao, scope, entry(1000L, "string", DataType.STRING, "value")); + saveLatest(latestDao, scope, entry(1000L, "json", DataType.JSON, "{\"v\":1}")); + + assertLatest(latestDao, scope, "bool", 1000L, DataType.BOOLEAN, true); + assertLatest(latestDao, scope, "long", 1000L, DataType.LONG, 42L); + assertLatest(latestDao, scope, "double", 1000L, DataType.DOUBLE, 4.2D); + assertLatest(latestDao, scope, "string", 1000L, DataType.STRING, "value"); + assertLatest(latestDao, scope, "json", 1000L, DataType.JSON, "{\"v\":1}"); + + // Overwrite is delete-then-insert: a single overlay row survives, holding the newer value. + saveLatest(latestDao, scope, entry(2000L, "long", DataType.LONG, 99L)); + assertLatest(latestDao, scope, "long", 2000L, DataType.LONG, 99L); + assertEquals(1, overlayRowCount(pool, scope, "long")); + } finally { + latestDao.destroy(); + } + } + } + + @Test + void findLatest_mergesTelemetryAndOverlayByMaxTs() throws Exception { + TestScope scope = + scope( + "lt_merge", + "55555555-5555-5555-5555-555555555513", + "66666666-6666-6666-6666-666666666613"); + bootstrapSchema(scope.database()); + bootstrapLatestSchema(scope.database()); + try (ITableSessionPool pool = newPool(scope.database())) { + IoTDBTableConfig config = config(2); + IoTDBTableTimeseriesWriter writer = new IoTDBTableTimeseriesWriter(pool, config); + IoTDBTableTimeseriesDao tsDao = new IoTDBTableTimeseriesDao(pool, writer, config); + IoTDBTableLatestDao latestDao = new IoTDBTableLatestDao(pool, config); + try { + // k_over: telemetry@1000 then a NEWER overlay@2000 -> overlay wins. + saveAll(tsDao, scope, List.of(entry(1000L, "k_over", DataType.LONG, 1L))); + saveLatest(latestDao, scope, entry(2000L, "k_over", DataType.LONG, 99L)); + assertLatest(latestDao, scope, "k_over", 2000L, DataType.LONG, 99L); + + // k_deriv: overlay@2000 then a NEWER telemetry@3000 -> derived wins. + saveLatest(latestDao, scope, entry(2000L, "k_deriv", DataType.LONG, 5L)); + saveAll(tsDao, scope, List.of(entry(3000L, "k_deriv", DataType.LONG, 7L))); + assertLatest(latestDao, scope, "k_deriv", 3000L, DataType.LONG, 7L); + } finally { + latestDao.destroy(); + tsDao.destroy(); + writer.destroy(); + } + } + } + + @Test + void findAllLatest_mixesDerivedAndOverlayOnlyKeys() throws Exception { + TestScope scope = + scope( + "lt_all_mix", + "55555555-5555-5555-5555-555555555514", + "66666666-6666-6666-6666-666666666614"); + bootstrapSchema(scope.database()); + bootstrapLatestSchema(scope.database()); + try (ITableSessionPool pool = newPool(scope.database())) { + IoTDBTableConfig config = config(4); + IoTDBTableTimeseriesWriter writer = new IoTDBTableTimeseriesWriter(pool, config); + IoTDBTableTimeseriesDao tsDao = new IoTDBTableTimeseriesDao(pool, writer, config); + IoTDBTableLatestDao latestDao = new IoTDBTableLatestDao(pool, config); + try { + // derived-only key, overlay-only key, and a key present in both (overlay newer -> wins). + saveAll( + tsDao, + scope, + List.of( + entry(1000L, "derived", DataType.DOUBLE, 1.5D), + entry(1000L, "both", DataType.LONG, 1L))); + saveLatest(latestDao, scope, entry(2000L, "overlay", DataType.STRING, "ov")); + saveLatest(latestDao, scope, entry(2000L, "both", DataType.LONG, 9L)); + + List latest = + latestDao + .findAllLatest(scope.tenantId(), scope.entityId()) + .get(FUTURE_TIMEOUT_SECONDS, TimeUnit.SECONDS); + latest.sort(Comparator.comparing(TsKvEntry::getKey)); + + assertEquals(3, latest.size()); + assertEntry(latest.get(0), 2000L, "both", DataType.LONG, 9L); + assertEntry(latest.get(1), 1000L, "derived", DataType.DOUBLE, 1.5D); + assertEntry(latest.get(2), 2000L, "overlay", DataType.STRING, "ov"); + } finally { + latestDao.destroy(); + tsDao.destroy(); + writer.destroy(); + } + } + } + + @Test + void removeLatest_ofOverlayValue_marksRemovedAndClearsOverlay() throws Exception { + TestScope scope = + scope( + "lt_rm", + "55555555-5555-5555-5555-555555555515", + "66666666-6666-6666-6666-666666666615"); + bootstrapSchema(scope.database()); + bootstrapLatestSchema(scope.database()); + try (ITableSessionPool pool = newPool(scope.database())) { + IoTDBTableConfig config = config(1); + IoTDBTableLatestDao latestDao = new IoTDBTableLatestDao(pool, config); + try { + saveLatest(latestDao, scope, entry(1500L, "k", DataType.LONG, 7L)); + assertLatest(latestDao, scope, "k", 1500L, DataType.LONG, 7L); + + TsKvLatestRemovingResult result = + latestDao + .removeLatest( + scope.tenantId(), scope.entityId(), new BaseDeleteTsKvQuery("k", 1000L, 2000L)) + .get(FUTURE_TIMEOUT_SECONDS, TimeUnit.SECONDS); + assertTrue(result.isRemoved()); + assertNull(result.getVersion()); + + // Overlay row deleted and (no telemetry) findLatest is now empty. + assertEquals(0, overlayRowCount(pool, scope, "k")); + assertTrue( + latestDao + .findLatestOpt(scope.tenantId(), scope.entityId(), "k") + .get(FUTURE_TIMEOUT_SECONDS, TimeUnit.SECONDS) + .isEmpty()); + } finally { + latestDao.destroy(); + } + } + } + + @Test + void removeLatest_rewriteResurrectsPriorHistoryIntoOverlay() throws Exception { + TestScope scope = + scope( + "lt_rw", + "55555555-5555-5555-5555-555555555516", + "66666666-6666-6666-6666-666666666616"); + bootstrapSchema(scope.database()); + bootstrapLatestSchema(scope.database()); + try (ITableSessionPool pool = newPool(scope.database())) { + IoTDBTableConfig config = config(2); + IoTDBTableTimeseriesWriter writer = new IoTDBTableTimeseriesWriter(pool, config); + IoTDBTableTimeseriesDao tsDao = new IoTDBTableTimeseriesDao(pool, writer, config); + IoTDBTableLatestDao latestDao = new IoTDBTableLatestDao(pool, config); + try { + // History: a prior value@1000 (outside the delete window) and the in-window latest@1500. + saveAll( + tsDao, + scope, + List.of(entry(1000L, "k", DataType.LONG, 10L), entry(1500L, "k", DataType.LONG, 20L))); + assertLatest(latestDao, scope, "k", 1500L, DataType.LONG, 20L); + + // Delete window [1200, 2000) covers the latest@1500 but not the prior@1000; rewrite=true. + TsKvLatestRemovingResult result = + latestDao + .removeLatest( + scope.tenantId(), + scope.entityId(), + new BaseDeleteTsKvQuery("k", 1200L, 2000L, true)) + .get(FUTURE_TIMEOUT_SECONDS, TimeUnit.SECONDS); + + // The removing result carries the resurrected prior value (WS UPDATE, not DELETE). + assertTrue(result.isRemoved()); + assertNull(result.getVersion()); + assertNotNull(result.getData()); + assertEquals(1000L, result.getData().getTs()); + assertEquals(DataType.LONG, result.getData().getDataType()); + assertEquals(10L, result.getData().getValue()); + + // The prior was written back into the overlay (one row, holding 1000/10). The in-window + // telemetry row is NOT deleted by this DAO (TB's historical remove is a separate path), so + // the derived read still sees 1500 -- assert the overlay write-back directly. + assertEquals(1, overlayRowCount(pool, scope, "k")); + OverlayRow overlay = overlayRow(pool, scope, "k"); + assertEquals(1000L, overlay.ts()); + assertEquals(10L, overlay.longValue()); + } finally { + latestDao.destroy(); + tsDao.destroy(); + writer.destroy(); + } + } + } + + private void assertLatest( + IoTDBTableLatestDao latestDao, + TestScope scope, + String key, + long expectedTs, + DataType dataType, + Object value) + throws Exception { + Optional opt = + latestDao + .findLatestOpt(scope.tenantId(), scope.entityId(), key) + .get(FUTURE_TIMEOUT_SECONDS, TimeUnit.SECONDS); + assertTrue(opt.isPresent(), "expected a latest value for key " + key); + assertEntry(opt.get(), expectedTs, key, dataType, value); + + TsKvEntry present = + latestDao + .findLatest(scope.tenantId(), scope.entityId(), key) + .get(FUTURE_TIMEOUT_SECONDS, TimeUnit.SECONDS); + assertEntry(present, expectedTs, key, dataType, value); + } + + private void assertEntry( + TsKvEntry entry, long expectedTs, String key, DataType dataType, Object value) { + assertEquals(expectedTs, entry.getTs()); + assertEquals(key, entry.getKey()); + assertEquals(dataType, entry.getDataType()); + assertEquals(value, entry.getValue()); + assertEquals(String.valueOf(value), entry.getValueAsString()); + } + + private ITableSessionPool newPool(String database) { + TableSessionPoolBuilder builder = + new TableSessionPoolBuilder() + .nodeUrls(List.of("127.0.0.1:" + IOTDB.getMappedPort(6667))) + .user("root") + .password("root") + .maxSize(4); + if (database != null) { + builder.database(database); + } + return builder.build(); + } + + private void bootstrapSchema(String database) throws Exception { + awaitIoTDBReady(database); + + String schema; + try (InputStream stream = + IoTDBTableLatestDaoIT.class + .getClassLoader() + .getResourceAsStream("schema-iotdb-table.sql")) { + schema = new String(stream.readAllBytes(), StandardCharsets.UTF_8); + } + schema = + schema + .replace( + "CREATE DATABASE IF NOT EXISTS thingsboard;", + "CREATE DATABASE IF NOT EXISTS " + database + ";") + .replace("USE thingsboard;", "USE " + database + ";"); + schema = schema.replaceAll("(?s)/\\*.*?\\*/", "").replaceAll("(?m)--.*$", ""); + try (ITableSessionPool bootstrapPool = newPool(null)) { + try (ITableSession session = bootstrapPool.getSession()) { + for (String statement : schema.split(";")) { + String trimmed = statement.trim(); + if (!trimmed.isEmpty()) { + session.executeNonQueryStatement(trimmed); + } + } + } + } + } + + private void bootstrapLatestSchema(String database) throws Exception { + // Exercise the REAL production bootstrap path: the 3-arg IoTDBTableSchemaBootstrap loading the + // telemetry_latest overlay resource. The bootstrap pool has no fixed database (CREATE DATABASE + // runs first), and the config carries the target database so the DDL is rewritten onto it. + awaitIoTDBReady(database); + IoTDBTableConfig config = new IoTDBTableConfig(); + config.setDatabase(database); + try (ITableSessionPool bootstrapPool = newPool(null)) { + new IoTDBTableSchemaBootstrap( + bootstrapPool, config, IoTDBTableSchemaBootstrap.LATEST_SCHEMA_RESOURCE) + .afterPropertiesSet(); + } + } + + private void awaitIoTDBReady(String database) throws Exception { + long deadlineNanos = System.nanoTime() + IOTDB_READY_TIMEOUT.toNanos(); + Exception lastFailure = null; + while (System.nanoTime() < deadlineNanos) { + try (ITableSessionPool bootstrapPool = newPool(null); + ITableSession session = bootstrapPool.getSession()) { + session.executeNonQueryStatement("CREATE DATABASE IF NOT EXISTS " + database); + return; + } catch (Exception e) { + lastFailure = e; + long remainingMillis = TimeUnit.NANOSECONDS.toMillis(deadlineNanos - System.nanoTime()); + if (remainingMillis <= 0) { + break; + } + Thread.sleep(Math.min(IOTDB_READY_POLL_INTERVAL.toMillis(), remainingMillis)); + } + } + throw new IllegalStateException( + "IoTDB did not accept table-session statements within " + IOTDB_READY_TIMEOUT, lastFailure); + } + + private IoTDBTableConfig config(int batchSize) { + IoTDBTableConfig config = new IoTDBTableConfig(); + config.getTs().getSave().setBatchSize(batchSize); + config.getTs().getSave().setMaxLingerMs(20L); + config.getTs().getSave().setRetryInitialBackoffMs(1L); + config.getTs().getSave().setRetryMaxBackoffMs(1L); + config.getTs().getRead().setThreads(1); + return config; + } + + private void saveAll(IoTDBTableTimeseriesDao dao, TestScope scope, List entries) + throws Exception { + List> futures = new ArrayList<>(); + for (TestTsKvEntry entry : entries) { + futures.add(dao.save(scope.tenantId(), scope.entityId(), entry, 0)); + } + for (ListenableFuture future : futures) { + assertEquals(1, future.get(FUTURE_TIMEOUT_SECONDS, TimeUnit.SECONDS)); + } + } + + private void saveLatest(IoTDBTableLatestDao dao, TestScope scope, TestTsKvEntry entry) + throws Exception { + // saveLatest writes the overlay only (no paired tsDao.save()) and returns a null version. + assertNull( + dao.saveLatest(scope.tenantId(), scope.entityId(), entry) + .get(FUTURE_TIMEOUT_SECONDS, TimeUnit.SECONDS)); + } + + private int overlayRowCount(ITableSessionPool pool, TestScope scope, String key) + throws Exception { + try (ITableSession session = pool.getSession(); + SessionDataSet dataSet = + session.executeQueryStatement( + "SELECT bool_v,long_v,double_v,str_v,json_v FROM telemetry_latest WHERE tenant_id='" + + scope.tenantId().getId() + + "' AND entity_type='DEVICE' AND entity_id='" + + scope.entityId().getId() + + "' AND key='" + + key.replace("'", "''") + + "'")) { + SessionDataSet.DataIterator rows = dataSet.iterator(); + int rowCount = 0; + while (rows.next()) { + rowCount++; + } + return rowCount; + } + } + + private OverlayRow overlayRow(ITableSessionPool pool, TestScope scope, String key) + throws Exception { + try (ITableSession session = pool.getSession(); + SessionDataSet dataSet = + session.executeQueryStatement( + "SELECT time, long_v FROM telemetry_latest WHERE tenant_id='" + + scope.tenantId().getId() + + "' AND entity_type='DEVICE' AND entity_id='" + + scope.entityId().getId() + + "' AND key='" + + key.replace("'", "''") + + "'")) { + SessionDataSet.DataIterator rows = dataSet.iterator(); + assertTrue(rows.next(), "expected one overlay row for key " + key); + return new OverlayRow(rows.getTimestamp("time").getTime(), rows.getLong("long_v")); + } + } + + private record OverlayRow(long ts, long longValue) {} + + private TestScope scope(String databasePrefix, String tenantId, String entityId) { + return new TestScope( + uniqueDatabase(databasePrefix), + new TenantId(UUID.fromString(tenantId)), + new TestEntityId(UUID.fromString(entityId), EntityType.DEVICE)); + } + + private String uniqueDatabase(String prefix) { + // IoTDB caps database names at 64 chars; keep the per-test prefix short and append a trimmed + // UUID so the total length stays well within the limit. + String shortPrefix = prefix.length() > 12 ? prefix.substring(0, 12) : prefix; + String shortUuid = UUID.randomUUID().toString().replace("-", "").substring(0, 16); + return "tb_lt_" + shortPrefix + "_" + shortUuid; + } + + private TestTsKvEntry entry(long ts, String key, DataType dataType, Object value) { + return new TestTsKvEntry(ts, key, dataType, value); + } + + private record TestScope(String database, TenantId tenantId, EntityId entityId) {} + + private record TestEntityId(UUID id, EntityType entityType) implements EntityId { + @Override + public UUID getId() { + return id; + } + + @Override + public EntityType getEntityType() { + return entityType; + } + } + + private record TestTsKvEntry(long ts, String key, DataType dataType, Object value) + implements TsKvEntry { + @Override + public long getTs() { + return ts; + } + + @Override + public String getKey() { + return key; + } + + @Override + public DataType getDataType() { + return dataType; + } + + @Override + public Optional getBooleanValue() { + return dataType == DataType.BOOLEAN ? Optional.of((Boolean) value) : Optional.empty(); + } + + @Override + public Optional getLongValue() { + return dataType == DataType.LONG ? Optional.of((Long) value) : Optional.empty(); + } + + @Override + public Optional getDoubleValue() { + return dataType == DataType.DOUBLE ? Optional.of((Double) value) : Optional.empty(); + } + + @Override + public Optional getStrValue() { + return dataType == DataType.STRING ? Optional.of((String) value) : Optional.empty(); + } + + @Override + public Optional getJsonValue() { + return dataType == DataType.JSON ? Optional.of((String) value) : Optional.empty(); + } + + @Override + public String getValueAsString() { + return String.valueOf(value); + } + + @Override + public Object getValue() { + return value; + } + + @Override + public Long getVersion() { + return null; + } + + @Override + public int getDataPoints() { + return 1; + } + } +} diff --git a/iotdb-thingsboard-table/src/test/java/org/apache/iotdb/extras/thingsboard/table/IoTDBTableLatestDaoTest.java b/iotdb-thingsboard-table/src/test/java/org/apache/iotdb/extras/thingsboard/table/IoTDBTableLatestDaoTest.java new file mode 100644 index 00000000..721c30a6 --- /dev/null +++ b/iotdb-thingsboard-table/src/test/java/org/apache/iotdb/extras/thingsboard/table/IoTDBTableLatestDaoTest.java @@ -0,0 +1,812 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package org.apache.iotdb.extras.thingsboard.table; + +import org.apache.iotdb.isession.ITableSession; +import org.apache.iotdb.isession.SessionDataSet; +import org.apache.iotdb.isession.pool.ITableSessionPool; +import org.apache.iotdb.rpc.IoTDBConnectionException; +import org.apache.iotdb.rpc.StatementExecutionException; + +import com.google.common.util.concurrent.ListenableFuture; +import org.apache.tsfile.enums.ColumnCategory; +import org.apache.tsfile.write.record.Tablet; +import org.apache.tsfile.write.schema.IMeasurementSchema; +import org.junit.jupiter.api.AfterEach; +import org.junit.jupiter.api.Test; +import org.mockito.ArgumentCaptor; +import org.mockito.InOrder; +import org.thingsboard.server.common.data.EntityType; +import org.thingsboard.server.common.data.id.EntityId; +import org.thingsboard.server.common.data.id.TenantId; +import org.thingsboard.server.common.data.kv.BaseDeleteTsKvQuery; +import org.thingsboard.server.common.data.kv.BasicTsKvEntry; +import org.thingsboard.server.common.data.kv.DataType; +import org.thingsboard.server.common.data.kv.TsKvEntry; +import org.thingsboard.server.common.data.kv.TsKvLatestRemovingResult; + +import java.sql.Timestamp; +import java.util.ArrayList; +import java.util.HashMap; +import java.util.List; +import java.util.Map; +import java.util.Optional; +import java.util.UUID; +import java.util.concurrent.ExecutionException; +import java.util.concurrent.TimeUnit; +import java.util.concurrent.atomic.AtomicInteger; + +import static org.junit.jupiter.api.Assertions.assertEquals; +import static org.junit.jupiter.api.Assertions.assertFalse; +import static org.junit.jupiter.api.Assertions.assertInstanceOf; +import static org.junit.jupiter.api.Assertions.assertNull; +import static org.junit.jupiter.api.Assertions.assertThrows; +import static org.junit.jupiter.api.Assertions.assertTrue; +import static org.mockito.ArgumentMatchers.any; +import static org.mockito.ArgumentMatchers.anyString; +import static org.mockito.Mockito.inOrder; +import static org.mockito.Mockito.mock; +import static org.mockito.Mockito.never; +import static org.mockito.Mockito.timeout; +import static org.mockito.Mockito.verify; +import static org.mockito.Mockito.when; + +class IoTDBTableLatestDaoTest { + private static final TenantId TENANT_ID = + new TenantId(UUID.fromString("11111111-1111-1111-1111-111111111111")); + private static final EntityId ENTITY_ID = + new TestEntityId(UUID.fromString("22222222-2222-2222-2222-222222222222"), EntityType.DEVICE); + + private static final String DERIVED_SQL_PREFIX = + "SELECT time, bool_v, long_v, double_v, str_v, json_v FROM telemetry " + + "WHERE tenant_id='11111111-1111-1111-1111-111111111111' " + + "AND entity_type='DEVICE' " + + "AND entity_id='22222222-2222-2222-2222-222222222222' "; + private static final String OVERLAY_SQL_PREFIX = + "SELECT time, bool_v, long_v, double_v, str_v, json_v FROM telemetry_latest " + + "WHERE tenant_id='11111111-1111-1111-1111-111111111111' " + + "AND entity_type='DEVICE' " + + "AND entity_id='22222222-2222-2222-2222-222222222222' "; + + private final List daos = new ArrayList<>(); + + @AfterEach + void tearDown() { + for (IoTDBTableLatestDao dao : daos) { + dao.destroy(); + } + daos.clear(); + } + + // ---- read merge: derived primary + overlay, max-ts-per-key, overlay wins on tie ---- + + @Test + void findLatestOpt_mergesDerivedAndOverlay_overlayNewerWins() throws Exception { + TestContext context = newContext(); + // derived@100 (long 7), overlay@150 (long 99) -> overlay is newer and wins. + stubReads(context.session(), rows(row(100L, "long_v", 7L)), rows(row(150L, "long_v", 99L))); + + Optional result = + context.dao().findLatestOpt(TENANT_ID, ENTITY_ID, "temperature").get(3, TimeUnit.SECONDS); + + assertTrue(result.isPresent()); + assertMappedEntry(result.get(), 150L, "temperature", DataType.LONG, 99L); + + List queries = captureQueries(context.session(), 2); + assertTrue( + queries.contains(DERIVED_SQL_PREFIX + "AND key='temperature' ORDER BY time DESC LIMIT 1"), + "derived point read SQL: " + queries); + assertTrue( + queries.contains(OVERLAY_SQL_PREFIX + "AND key='temperature' ORDER BY time DESC LIMIT 1"), + "overlay point read SQL: " + queries); + } + + @Test + void findLatestOpt_derivedNewerThanOverlay_derivedWins() throws Exception { + TestContext context = newContext(); + // derived@200 (double 3.5) beats overlay@150 (long 7). + stubReads(context.session(), rows(row(200L, "double_v", 3.5D)), rows(row(150L, "long_v", 7L))); + + Optional result = + context.dao().findLatestOpt(TENANT_ID, ENTITY_ID, "pressure").get(3, TimeUnit.SECONDS); + + assertTrue(result.isPresent()); + assertMappedEntry(result.get(), 200L, "pressure", DataType.DOUBLE, 3.5D); + } + + @Test + void findLatestOpt_equalTimestamp_overlayWinsTie() throws Exception { + TestContext context = newContext(); + // Same ts: the overlay value wins the tie (continues the B1 same-timestamp limitation). + stubReads(context.session(), rows(row(150L, "long_v", 1L)), rows(row(150L, "str_v", "ov"))); + + Optional result = + context.dao().findLatestOpt(TENANT_ID, ENTITY_ID, "k").get(3, TimeUnit.SECONDS); + + assertTrue(result.isPresent()); + assertMappedEntry(result.get(), 150L, "k", DataType.STRING, "ov"); + } + + @Test + void findLatestOpt_overlayOnlyKey_returnsOverlay() throws Exception { + TestContext context = newContext(); + // No derived row (the load-bearing latest-only / EntityView case): the overlay supplies it. + stubReads(context.session(), rows(), rows(row(150L, "long_v", 5L))); + + Optional result = + context.dao().findLatestOpt(TENANT_ID, ENTITY_ID, "latest-only").get(3, TimeUnit.SECONDS); + + assertTrue(result.isPresent()); + assertMappedEntry(result.get(), 150L, "latest-only", DataType.LONG, 5L); + } + + @Test + void findLatestOpt_returnsEmptyWhenNeitherStoreHasRow() throws Exception { + TestContext context = newContext(); + stubReads(context.session(), rows(), rows()); + + Optional result = + context.dao().findLatestOpt(TENANT_ID, ENTITY_ID, "absent").get(3, TimeUnit.SECONDS); + + assertTrue(result.isEmpty()); + } + + @Test + void findLatestOpt_failsFutureWhenDerivedRowHasTwoTypedColumns() throws Exception { + TestContext context = newContext(); + stubReads( + context.session(), rows(rowOf(3000L, Map.of("long_v", 7L, "str_v", "seven"))), rows()); + + Throwable cause = + assertFutureFailsWith( + context.dao().findLatestOpt(TENANT_ID, ENTITY_ID, "same-ts"), + IllegalStateException.class); + assertTrue(cause.getMessage().contains("2 typed value columns set")); + } + + @Test + void findLatest_returnsNullStringEntryFallbackWhenEmpty() throws Exception { + TestContext context = newContext(); + stubReads(context.session(), rows(), rows()); + + TsKvEntry fallback = + context.dao().findLatest(TENANT_ID, ENTITY_ID, "missing").get(3, TimeUnit.SECONDS); + + assertInstanceOf(BasicTsKvEntry.class, fallback); + assertEquals("missing", fallback.getKey()); + assertEquals(DataType.STRING, fallback.getDataType()); + assertNull(fallback.getValue()); + assertTrue(fallback.getStrValue().isEmpty()); + } + + @Test + void findLatest_returnsPresentEntryWithoutFallback() throws Exception { + TestContext context = newContext(); + stubReads(context.session(), rows(row(200L, "double_v", 3.5D)), rows()); + + TsKvEntry entry = + context.dao().findLatest(TENANT_ID, ENTITY_ID, "pressure").get(3, TimeUnit.SECONDS); + + assertMappedEntry(entry, 200L, "pressure", DataType.DOUBLE, 3.5D); + } + + @Test + void findAllLatest_mergesDerivedAndOverlayByMaxTsPerKey() throws Exception { + TestContext context = newContext(); + // derived: bool@110, count@120, label@130. overlay: count@125 (newer -> wins), extra@140 + // (only). + stubReads( + context.session(), + rows( + aggRow("bool", 110L, "bool_v", true), + aggRow("count", 120L, "long_v", 9L), + aggRow("label", 130L, "str_v", "ok")), + rows( + overlayAllRow("count", 125L, "long_v", 99L), + overlayAllRow("extra", 140L, "str_v", "x"))); + + List entries = + context.dao().findAllLatest(TENANT_ID, ENTITY_ID).get(3, TimeUnit.SECONDS); + entries.sort(java.util.Comparator.comparing(TsKvEntry::getKey)); + + assertEquals(4, entries.size()); + assertMappedEntry(entries.get(0), 110L, "bool", DataType.BOOLEAN, true); + assertMappedEntry(entries.get(1), 125L, "count", DataType.LONG, 99L); + assertMappedEntry(entries.get(2), 140L, "extra", DataType.STRING, "x"); + assertMappedEntry(entries.get(3), 130L, "label", DataType.STRING, "ok"); + + List queries = captureQueries(context.session(), 2); + assertTrue( + queries.contains( + "SELECT key, LAST_BY(bool_v, time) AS bool_v, LAST_BY(long_v, time) AS long_v, " + + "LAST_BY(double_v, time) AS double_v, LAST_BY(str_v, time) AS str_v, " + + "LAST_BY(json_v, time) AS json_v, MAX(time) AS last_ts FROM telemetry " + + "WHERE tenant_id='11111111-1111-1111-1111-111111111111' " + + "AND entity_type='DEVICE' " + + "AND entity_id='22222222-2222-2222-2222-222222222222' " + + "GROUP BY key"), + "derived findAll SQL: " + queries); + assertTrue( + queries.contains( + "SELECT key, time, bool_v, long_v, double_v, str_v, json_v FROM telemetry_latest " + + "WHERE tenant_id='11111111-1111-1111-1111-111111111111' " + + "AND entity_type='DEVICE' " + + "AND entity_id='22222222-2222-2222-2222-222222222222'"), + "overlay findAll SQL: " + queries); + } + + @Test + void findAllLatest_failsFutureWhenAggregatedKeyHasTwoTypedColumns() throws Exception { + TestContext context = newContext(); + stubReads( + context.session(), + rows( + aggRowOf("clean", 100L, Map.of("long_v", 1L)), + aggRowOf("dirty", 200L, Map.of("long_v", 2L, "str_v", "x"))), + rows()); + + Throwable cause = + assertFutureFailsWith( + context.dao().findAllLatest(TENANT_ID, ENTITY_ID), IllegalStateException.class); + assertTrue(cause.getMessage().contains("2 typed value columns set")); + } + + // ---- saveLatest: overlay delete-then-insert upsert ---- + + @Test + void saveLatest_upsertsOverlayWithDeleteThenInsertAndNullVersion() throws Exception { + TestContext context = newContext(); + + Long version = + context + .dao() + .saveLatest(TENANT_ID, ENTITY_ID, entry(500L, "temperature", DataType.LONG, 5L)) + .get(3, TimeUnit.SECONDS); + + // IoTDB has no sequence; saveLatest returns a null version (Phase-1 contract). + assertNull(version); + + ArgumentCaptor deleteSql = ArgumentCaptor.forClass(String.class); + verify(context.session(), timeout(3000)).executeNonQueryStatement(deleteSql.capture()); + assertEquals( + "DELETE FROM telemetry_latest " + + "WHERE tenant_id='11111111-1111-1111-1111-111111111111' " + + "AND entity_type='DEVICE' " + + "AND entity_id='22222222-2222-2222-2222-222222222222' " + + "AND key='temperature'", + deleteSql.getValue()); + + ArgumentCaptor tablet = ArgumentCaptor.forClass(Tablet.class); + verify(context.session(), timeout(3000)).insert(tablet.capture()); + Tablet inserted = tablet.getValue(); + assertEquals("telemetry_latest", inserted.getTableName()); + assertEquals(1, inserted.getRowSize()); + assertEquals(500L, inserted.getTimestamp(0)); + + // delete-then-insert ordering. + InOrder order = inOrder(context.session()); + order.verify(context.session()).executeNonQueryStatement(anyString()); + order.verify(context.session()).insert(any()); + } + + @Test + void saveLatest_tabletColumnsFollowLatestDdlTagOrder() throws Exception { + // TAG-order rot guard: the overlay Tablet must follow the telemetry_latest DDL tag order + // (entity_type, tenant_id, key, entity_id), then the five typed FIELDs, with NO + // ColumnCategory.TIME entry (the time column is written via addTimestamp). + TestContext context = newContext(); + + context + .dao() + .saveLatest(TENANT_ID, ENTITY_ID, entry(500L, "temperature", DataType.LONG, 5L)) + .get(3, TimeUnit.SECONDS); + + ArgumentCaptor tablet = ArgumentCaptor.forClass(Tablet.class); + verify(context.session(), timeout(3000)).insert(tablet.capture()); + Tablet inserted = tablet.getValue(); + + List columnNames = + inserted.getSchemas().stream().map(IMeasurementSchema::getMeasurementName).toList(); + assertEquals( + List.of( + "entity_type", + "tenant_id", + "key", + "entity_id", + "bool_v", + "long_v", + "double_v", + "str_v", + "json_v"), + columnNames); + List categories = inserted.getColumnTypes(); + assertEquals(9, categories.size()); + for (int i = 0; i < 4; i++) { + assertEquals(ColumnCategory.TAG, categories.get(i), "tag column " + i); + } + for (int i = 4; i < 9; i++) { + assertEquals(ColumnCategory.FIELD, categories.get(i), "field column " + i); + } + } + + @Test + void saveLatest_escapesSingleQuotesInKey() throws Exception { + TestContext context = newContext(); + + context + .dao() + .saveLatest(TENANT_ID, ENTITY_ID, entry(5L, "a'b", DataType.STRING, "v")) + .get(3, TimeUnit.SECONDS); + + ArgumentCaptor deleteSql = ArgumentCaptor.forClass(String.class); + verify(context.session(), timeout(3000)).executeNonQueryStatement(deleteSql.capture()); + assertTrue(deleteSql.getValue().contains("key='a''b'")); + } + + @Test + void saveLatest_rejectsBlankKeyBeforeAnyWork() { + TestContext context = newContext(); + + assertThrows( + IllegalArgumentException.class, + () -> context.dao().saveLatest(TENANT_ID, ENTITY_ID, entry(1L, " ", DataType.LONG, 1L))); + } + + // ---- removeLatest: overlay-aware ---- + + @Test + void removeLatest_inWindow_deletesOverlayRowAndReportsRemoved() throws Exception { + TestContext context = newContext(); + // Merged latest at ts=150, window [100, 200) covers it; default deleteLatest=true, no rewrite. + stubReads(context.session(), rows(row(150L, "long_v", 42L)), rows(row(150L, "long_v", 42L))); + + TsKvLatestRemovingResult result = + context + .dao() + .removeLatest(TENANT_ID, ENTITY_ID, new BaseDeleteTsKvQuery("temperature", 100L, 200L)) + .get(3, TimeUnit.SECONDS); + + assertEquals("temperature", result.getKey()); + assertTrue(result.isRemoved()); + assertNull(result.getVersion()); + assertNull(result.getData()); + + ArgumentCaptor deleteSql = ArgumentCaptor.forClass(String.class); + verify(context.session(), timeout(3000)).executeNonQueryStatement(deleteSql.capture()); + assertEquals( + "DELETE FROM telemetry_latest " + + "WHERE tenant_id='11111111-1111-1111-1111-111111111111' " + + "AND entity_type='DEVICE' " + + "AND entity_id='22222222-2222-2222-2222-222222222222' " + + "AND key='temperature'", + deleteSql.getValue()); + // Plain delete (no rewrite): no overlay INSERT. + verify(context.session(), never()).insert(any()); + } + + @Test + void removeLatest_overlayOnlyValue_inWindow_deletesOverlay() throws Exception { + TestContext context = newContext(); + // Latest-only value lives only in the overlay; an in-window remove must delete it. + stubReads(context.session(), rows(), rows(row(150L, "long_v", 42L))); + + TsKvLatestRemovingResult result = + context + .dao() + .removeLatest(TENANT_ID, ENTITY_ID, new BaseDeleteTsKvQuery("temperature", 100L, 200L)) + .get(3, TimeUnit.SECONDS); + + assertTrue(result.isRemoved()); + verify(context.session(), timeout(3000)).executeNonQueryStatement(anyString()); + verify(context.session(), never()).insert(any()); + } + + @Test + void removeLatest_outsideWindow_reportsNotRemovedAndNoMutation() throws Exception { + TestContext context = newContext(); + // Merged latest at ts=250, window [100, 200) does NOT cover it -> removed=false, no mutation. + stubReads(context.session(), rows(row(250L, "long_v", 42L)), rows(row(250L, "long_v", 42L))); + + TsKvLatestRemovingResult result = + context + .dao() + .removeLatest(TENANT_ID, ENTITY_ID, new BaseDeleteTsKvQuery("temperature", 100L, 200L)) + .get(3, TimeUnit.SECONDS); + + assertEquals("temperature", result.getKey()); + assertFalse(result.isRemoved()); + verify(context.session(), never()).executeNonQueryStatement(anyString()); + verify(context.session(), never()).insert(any()); + } + + @Test + void removeLatest_noLatest_reportsNotRemoved() throws Exception { + TestContext context = newContext(); + stubReads(context.session(), rows(), rows()); + + TsKvLatestRemovingResult result = + context + .dao() + .removeLatest(TENANT_ID, ENTITY_ID, new BaseDeleteTsKvQuery("temperature", 100L, 200L)) + .get(3, TimeUnit.SECONDS); + + assertEquals("temperature", result.getKey()); + assertFalse(result.isRemoved()); + verify(context.session(), never()).executeNonQueryStatement(anyString()); + verify(context.session(), never()).insert(any()); + } + + @Test + void removeLatest_rewrite_resurrectsPriorHistoryIntoOverlay() throws Exception { + TestContext context = newContext(); + // Merged latest at ts=150 in window; rewriteLatestIfDeleted=true; the next-older history value + // (telemetry, time < 100) is long 10 at ts=50 -> it is written back into the overlay and + // returned as data so onTimeSeriesDelete emits a WS UPDATE. + stubReads( + context.session(), + rows(row(150L, "long_v", 42L)), + rows(row(150L, "long_v", 42L)), + rows(row(50L, "long_v", 10L))); + + TsKvLatestRemovingResult result = + context + .dao() + .removeLatest( + TENANT_ID, ENTITY_ID, new BaseDeleteTsKvQuery("temperature", 100L, 200L, true)) + .get(3, TimeUnit.SECONDS); + + assertTrue(result.isRemoved()); + assertNull(result.getVersion()); + assertEquals("temperature", result.getKey()); + TsKvEntry data = result.getData(); + assertEquals(50L, data.getTs()); + assertEquals(DataType.LONG, data.getDataType()); + assertEquals(10L, data.getValue()); + + // The history-before query must have been issued against telemetry with a time-before + // predicate. + List queries = captureAllQueries(context.session()); + assertTrue( + queries.contains( + DERIVED_SQL_PREFIX + "AND key='temperature' AND time < 100 ORDER BY time DESC LIMIT 1"), + "rewrite history-before SQL: " + queries); + // The resurrected prior is written back into the overlay (delete-then-insert). + ArgumentCaptor tablet = ArgumentCaptor.forClass(Tablet.class); + verify(context.session(), timeout(3000)).insert(tablet.capture()); + assertEquals("telemetry_latest", tablet.getValue().getTableName()); + assertEquals(50L, tablet.getValue().getTimestamp(0)); + } + + @Test + void removeLatest_rewriteButNoOlderHistory_deletesOverlayAndReportsRemoved() throws Exception { + TestContext context = newContext(); + // In-window latest, rewrite requested, but there is no older history to resurrect -> plain + // overlay delete, removed=true with no data. + stubReads( + context.session(), rows(row(150L, "long_v", 42L)), rows(row(150L, "long_v", 42L)), rows()); + + TsKvLatestRemovingResult result = + context + .dao() + .removeLatest( + TENANT_ID, ENTITY_ID, new BaseDeleteTsKvQuery("temperature", 100L, 200L, true)) + .get(3, TimeUnit.SECONDS); + + assertTrue(result.isRemoved()); + assertNull(result.getData()); + verify(context.session(), timeout(3000)).executeNonQueryStatement(anyString()); + verify(context.session(), never()).insert(any()); + } + + @Test + void removeLatest_deleteLatestFalse_reportsNotRemovedAndNoMutation() throws Exception { + TestContext context = newContext(); + // deleteLatest=false (defensive branch; TB only calls removeLatest with deleteLatest=true): + // even an in-window latest must not be removed and the overlay is untouched. + stubReads(context.session(), rows(row(150L, "long_v", 42L)), rows(row(150L, "long_v", 42L))); + + TsKvLatestRemovingResult result = + context + .dao() + .removeLatest( + TENANT_ID, + ENTITY_ID, + new BaseDeleteTsKvQuery("temperature", 100L, 200L, false, false)) + .get(3, TimeUnit.SECONDS); + + assertFalse(result.isRemoved()); + verify(context.session(), never()).executeNonQueryStatement(anyString()); + verify(context.session(), never()).insert(any()); + } + + // ---- input validation / escaping ---- + + @Test + void findLatestOpt_escapesQuotesInKey() throws Exception { + TestContext context = newContext(); + stubReads(context.session(), rows(), rows()); + + context.dao().findLatestOpt(TENANT_ID, ENTITY_ID, "a'b").get(3, TimeUnit.SECONDS); + + List queries = captureQueries(context.session(), 2); + assertTrue(queries.stream().anyMatch(s -> s.contains("key='a''b'")), "queries: " + queries); + } + + @Test + void findLatestOpt_rejectsBlankKeyBeforeQuery() { + TestContext context = newContext(); + + assertThrows( + IllegalArgumentException.class, + () -> context.dao().findLatestOpt(TENANT_ID, ENTITY_ID, " ")); + verifyNoSession(context); + } + + // ---- helpers ---- + + private TestContext newContext() { + ITableSessionPool pool = mock(ITableSessionPool.class); + ITableSession session = mock(ITableSession.class); + try { + when(pool.getSession()).thenReturn(session); + } catch (IoTDBConnectionException e) { + throw new AssertionError(e); + } + IoTDBTableConfig config = new IoTDBTableConfig(); + config.getTs().getRead().setThreads(1); + IoTDBTableLatestDao dao = new IoTDBTableLatestDao(pool, config); + daos.add(dao); + return new TestContext(dao, pool, session); + } + + private void verifyNoSession(TestContext context) { + try { + verify(context.pool(), never()).getSession(); + } catch (IoTDBConnectionException e) { + throw new AssertionError(e); + } + } + + /** + * Dispatches each {@code executeQueryStatement} to a FRESH dataset built from the matching row + * array, keyed on the SQL: {@code telemetry_latest} -> overlay, {@code telemetry ... AND time <} + * -> history-before, otherwise -> derived telemetry. A fresh dataset per call gives each read an + * independent iterator (the derived + overlay merge issues two reads). + */ + private void stubReads(ITableSession session, MockRow[] derived, MockRow[] overlay) { + stubReads(session, derived, overlay, rows()); + } + + private void stubReads( + ITableSession session, MockRow[] derived, MockRow[] overlay, MockRow[] history) { + try { + when(session.executeQueryStatement(anyString())) + .thenAnswer( + invocation -> { + String sql = invocation.getArgument(0); + if (sql.contains("telemetry_latest")) { + return dataSet(overlay); + } + if (sql.contains(" AND time <")) { + return dataSet(history); + } + return dataSet(derived); + }); + } catch (IoTDBConnectionException | StatementExecutionException e) { + throw new AssertionError(e); + } + } + + private List captureQueries(ITableSession session, int expected) throws Exception { + ArgumentCaptor sql = ArgumentCaptor.forClass(String.class); + verify(session, timeout(3000).times(expected)).executeQueryStatement(sql.capture()); + return sql.getAllValues(); + } + + private List captureAllQueries(ITableSession session) throws Exception { + ArgumentCaptor sql = ArgumentCaptor.forClass(String.class); + verify(session, timeout(3000).atLeastOnce()).executeQueryStatement(sql.capture()); + return sql.getAllValues(); + } + + private void assertMappedEntry( + TsKvEntry entry, long ts, String key, DataType dataType, Object value) { + assertInstanceOf(BasicTsKvEntry.class, entry); + assertEquals(ts, entry.getTs()); + assertEquals(key, entry.getKey()); + assertEquals(dataType, entry.getDataType()); + assertEquals(value, entry.getValue()); + } + + private Throwable assertFutureFailsWith( + ListenableFuture future, Class expectedCause) throws Exception { + ExecutionException exception = + assertThrows(ExecutionException.class, () -> future.get(3, TimeUnit.SECONDS)); + assertInstanceOf(expectedCause, exception.getCause()); + return exception.getCause(); + } + + private SessionDataSet dataSet(MockRow... rows) + throws IoTDBConnectionException, StatementExecutionException { + SessionDataSet dataSet = mock(SessionDataSet.class); + SessionDataSet.DataIterator iterator = mock(SessionDataSet.DataIterator.class); + AtomicInteger index = new AtomicInteger(-1); + when(dataSet.iterator()).thenReturn(iterator); + when(iterator.next()).thenAnswer(invocation -> index.incrementAndGet() < rows.length); + when(iterator.isNull(anyString())) + .thenAnswer(invocation -> rows[index.get()].isNull(invocation.getArgument(0))); + when(iterator.getBoolean(anyString())) + .thenAnswer(invocation -> rows[index.get()].get(invocation.getArgument(0))); + when(iterator.getLong(anyString())) + .thenAnswer(invocation -> rows[index.get()].get(invocation.getArgument(0))); + when(iterator.getDouble(anyString())) + .thenAnswer(invocation -> rows[index.get()].get(invocation.getArgument(0))); + when(iterator.getString(anyString())) + .thenAnswer(invocation -> String.valueOf(rows[index.get()].get(invocation.getArgument(0)))); + when(iterator.getTimestamp(anyString())) + .thenAnswer( + invocation -> new Timestamp((Long) rows[index.get()].get(invocation.getArgument(0)))); + return dataSet; + } + + private MockRow[] rows(MockRow... rows) { + return rows; + } + + /** A single typed-value latest row exposing {@code time} plus the one set typed column. */ + private MockRow row(long ts, String column, Object value) { + Map columns = new HashMap<>(); + columns.put("time", ts); + columns.put(column, value); + return new MockRow(columns); + } + + /** A single typed-value row with explicit typed columns (for B1 multi-column cases). */ + private MockRow rowOf(long ts, Map typed) { + Map columns = new HashMap<>(typed); + columns.put("time", ts); + return new MockRow(columns); + } + + /** + * A derived findAllLatest aggregated row exposing {@code key}, {@code last_ts}, one typed col. + */ + private MockRow aggRow(String key, long lastTs, String column, Object value) { + Map columns = new HashMap<>(); + columns.put("key", key); + columns.put("last_ts", lastTs); + columns.put(column, value); + return new MockRow(columns); + } + + /** A derived findAllLatest aggregated row with explicit typed columns (B1 multi-column cases). */ + private MockRow aggRowOf(String key, long lastTs, Map typed) { + Map columns = new HashMap<>(typed); + columns.put("key", key); + columns.put("last_ts", lastTs); + return new MockRow(columns); + } + + /** An overlay findAllLatest row exposing {@code key}, {@code time} and one typed column. */ + private MockRow overlayAllRow(String key, long ts, String column, Object value) { + Map columns = new HashMap<>(); + columns.put("key", key); + columns.put("time", ts); + columns.put(column, value); + return new MockRow(columns); + } + + private TestTsKvEntry entry(long ts, String key, DataType dataType, Object value) { + return new TestTsKvEntry(ts, key, dataType, value); + } + + private static final class MockRow { + private final Map columns; + + private MockRow(Map columns) { + this.columns = columns; + } + + private boolean isNull(String column) { + return columns.get(column) == null; + } + + private Object get(String column) { + return columns.get(column); + } + } + + private record TestContext( + IoTDBTableLatestDao dao, ITableSessionPool pool, ITableSession session) {} + + private record TestEntityId(UUID id, EntityType entityType) implements EntityId { + @Override + public UUID getId() { + return id; + } + + @Override + public EntityType getEntityType() { + return entityType; + } + } + + private record TestTsKvEntry(long ts, String key, DataType dataType, Object value) + implements TsKvEntry { + @Override + public long getTs() { + return ts; + } + + @Override + public String getKey() { + return key; + } + + @Override + public DataType getDataType() { + return dataType; + } + + @Override + public Optional getBooleanValue() { + return dataType == DataType.BOOLEAN ? Optional.of((Boolean) value) : Optional.empty(); + } + + @Override + public Optional getLongValue() { + return dataType == DataType.LONG ? Optional.of((Long) value) : Optional.empty(); + } + + @Override + public Optional getDoubleValue() { + return dataType == DataType.DOUBLE ? Optional.of((Double) value) : Optional.empty(); + } + + @Override + public Optional getStrValue() { + return dataType == DataType.STRING ? Optional.of((String) value) : Optional.empty(); + } + + @Override + public Optional getJsonValue() { + return dataType == DataType.JSON ? Optional.of((String) value) : Optional.empty(); + } + + @Override + public String getValueAsString() { + return String.valueOf(value); + } + + @Override + public Object getValue() { + return value; + } + + @Override + public Long getVersion() { + return null; + } + + @Override + public int getDataPoints() { + return 1; + } + } +} diff --git a/iotdb-thingsboard-table/src/test/java/org/apache/iotdb/extras/thingsboard/table/StrategyFContractTest.java b/iotdb-thingsboard-table/src/test/java/org/apache/iotdb/extras/thingsboard/table/StrategyFContractTest.java index 8a1882d1..692fabfe 100644 --- a/iotdb-thingsboard-table/src/test/java/org/apache/iotdb/extras/thingsboard/table/StrategyFContractTest.java +++ b/iotdb-thingsboard-table/src/test/java/org/apache/iotdb/extras/thingsboard/table/StrategyFContractTest.java @@ -20,13 +20,18 @@ import com.google.common.util.concurrent.ListenableFuture; import org.junit.jupiter.api.Test; +import org.thingsboard.server.common.data.AttributeScope; +import org.thingsboard.server.common.data.id.DeviceProfileId; import org.thingsboard.server.common.data.id.EntityId; import org.thingsboard.server.common.data.id.TenantId; +import org.thingsboard.server.common.data.kv.AttributeKvEntry; import org.thingsboard.server.common.data.kv.DeleteTsKvQuery; import org.thingsboard.server.common.data.kv.ReadTsKvQuery; import org.thingsboard.server.common.data.kv.ReadTsKvQueryResult; import org.thingsboard.server.common.data.kv.TsKvEntry; +import org.thingsboard.server.dao.attributes.AttributesDao; import org.thingsboard.server.dao.timeseries.TimeseriesDao; +import org.thingsboard.server.dao.timeseries.TimeseriesLatestDao; import java.io.IOException; import java.lang.reflect.Method; @@ -34,8 +39,11 @@ import java.nio.file.Files; import java.nio.file.Path; import java.util.ArrayList; +import java.util.Collection; import java.util.Enumeration; import java.util.List; +import java.util.Optional; +import java.util.UUID; import java.util.jar.JarEntry; import java.util.jar.JarFile; @@ -158,6 +166,171 @@ private static void assertSpiMethod( "TimeseriesDao." + name + " parameter types drifted from the pinned SPI expectation"); } + @Test + void timeseriesLatestDaoSpiMethodsMatchExpectedSignatures() throws NoSuchMethodException { + // IoTDBTableLatestDao implements TimeseriesLatestDao; pin the exact SPI shapes it depends on so + // a silent drift in the compile-only surface fails the build. Verified against ThingsBoard + // v4.3.1.2 (commit c37fb509); findLatestByEntityIds/findLatestByEntityIdsAsync are new in + // v4.3.1.2 relative to v4.3.1.1. + assertLatestSpiMethod( + "findLatestOpt", + ListenableFuture.class, + new Class[] {TenantId.class, EntityId.class, String.class}); + assertLatestSpiMethod( + "findLatest", + ListenableFuture.class, + new Class[] {TenantId.class, EntityId.class, String.class}); + assertLatestSpiMethod( + "findAllLatest", ListenableFuture.class, new Class[] {TenantId.class, EntityId.class}); + assertLatestSpiMethod( + "saveLatest", + ListenableFuture.class, + new Class[] {TenantId.class, EntityId.class, TsKvEntry.class}); + assertLatestSpiMethod( + "removeLatest", + ListenableFuture.class, + new Class[] {TenantId.class, EntityId.class, DeleteTsKvQuery.class}); + assertLatestSpiMethod( + "findAllKeysByDeviceProfileId", + List.class, + new Class[] {TenantId.class, DeviceProfileId.class}); + assertLatestSpiMethod( + "findAllKeysByEntityIds", List.class, new Class[] {TenantId.class, List.class}); + assertLatestSpiMethod( + "findAllKeysByEntityIdsAsync", + ListenableFuture.class, + new Class[] {TenantId.class, List.class}); + assertLatestSpiMethod( + "findLatestByEntityIds", List.class, new Class[] {TenantId.class, List.class}); + assertLatestSpiMethod( + "findLatestByEntityIdsAsync", + ListenableFuture.class, + new Class[] {TenantId.class, List.class}); + + // The DAO is a genuine TimeseriesLatestDao implementation. + assertTrue(TimeseriesLatestDao.class.isAssignableFrom(IoTDBTableLatestDao.class)); + // The interface declares exactly the 10 pinned methods (catches additive drift too). + assertEquals(10, TimeseriesLatestDao.class.getDeclaredMethods().length); + } + + /** + * Asserts that {@link TimeseriesLatestDao} declares a method with exactly the given name, return + * type and ordered parameter types, pinning the v4.3.1.2 SPI surface this module consumes. + */ + private static void assertLatestSpiMethod( + String name, Class expectedReturn, Class[] expectedParams) + throws NoSuchMethodException { + Method method = TimeseriesLatestDao.class.getMethod(name, expectedParams); + assertEquals( + expectedReturn, + method.getReturnType(), + "TimeseriesLatestDao." + name + " return type drifted from the pinned SPI expectation"); + assertArrayEquals( + expectedParams, + method.getParameterTypes(), + "TimeseriesLatestDao." + name + " parameter types drifted from the pinned SPI expectation"); + } + + @Test + void attributesDaoSpiMethodsMatchExpectedSignatures() throws NoSuchMethodException { + // IoTDBTableAttributesDao implements AttributesDao; pin all 14 v4.3.1.2 SPI signatures so a + // silent change to the local compile-only surface (src/provided) breaks the build. Verified + // against ThingsBoard v4.3.1.2 (commit c37fb509). + assertAttributesSpiMethod( + "find", + Optional.class, + new Class[] {TenantId.class, EntityId.class, AttributeScope.class, String.class}); + assertAttributesSpiMethod( + "find", + List.class, + new Class[] {TenantId.class, EntityId.class, AttributeScope.class, Collection.class}); + assertAttributesSpiMethod( + "findAll", + List.class, + new Class[] {TenantId.class, EntityId.class, AttributeScope.class}); + assertAttributesSpiMethod( + "save", + ListenableFuture.class, + new Class[] { + TenantId.class, EntityId.class, AttributeScope.class, AttributeKvEntry.class + }); + assertAttributesSpiMethod( + "removeAll", + List.class, + new Class[] {TenantId.class, EntityId.class, AttributeScope.class, List.class}); + assertAttributesSpiMethod( + "removeAllWithVersions", + List.class, + new Class[] {TenantId.class, EntityId.class, AttributeScope.class, List.class}); + assertAttributesSpiMethod( + "findNextBatch", List.class, new Class[] {UUID.class, int.class, int.class, int.class}); + assertAttributesSpiMethod( + "findAllKeysByDeviceProfileId", + List.class, + new Class[] {TenantId.class, DeviceProfileId.class}); + assertAttributesSpiMethod( + "findAllKeysByEntityIds", List.class, new Class[] {TenantId.class, List.class}); + assertAttributesSpiMethod( + "findAllKeysByEntityIdsAndScope", + List.class, + new Class[] {TenantId.class, List.class, AttributeScope.class}); + assertAttributesSpiMethod( + "findAllKeysByEntityIdsAndScopeAsync", + ListenableFuture.class, + new Class[] {TenantId.class, List.class, AttributeScope.class}); + assertAttributesSpiMethod( + "findLatestByEntityIdsAndScope", + List.class, + new Class[] {TenantId.class, List.class, AttributeScope.class}); + assertAttributesSpiMethod( + "findLatestByEntityIdsAndScopeAsync", + ListenableFuture.class, + new Class[] {TenantId.class, List.class, AttributeScope.class}); + assertAttributesSpiMethod( + "removeAllByEntityId", List.class, new Class[] {TenantId.class, EntityId.class}); + + // The DAO is a genuine AttributesDao implementation. + assertTrue(AttributesDao.class.isAssignableFrom(IoTDBTableAttributesDao.class)); + // The interface declares exactly the 14 pinned methods (catches additive drift too). + assertEquals(14, AttributesDao.class.getDeclaredMethods().length); + } + + @Test + void attributeValueObjectGettersUsedByDaoExist() throws NoSuchMethodException { + // The DAO maps each row into a BaseAttributeKvEntry(KvEntry, ts, version) and reads back these + // value-object getters; lock them down so a stub edit that drops one fails the build. + AttributeKvEntry.class.getMethod("getLastUpdateTs"); + AttributeKvEntry.class.getMethod("getVersion"); + AttributeKvEntry.class.getMethod("getKey"); + AttributeKvEntry.class.getMethod("getDataType"); + AttributeKvEntry.class.getMethod("getValue"); + + // AttributeScope round-trips by name()/valueOf(String); the custom valueOf(int) is part of the + // verified surface too. + assertEquals(AttributeScope.SERVER_SCOPE, AttributeScope.valueOf("SERVER_SCOPE")); + assertEquals(AttributeScope.SERVER_SCOPE, AttributeScope.valueOf(2)); + AttributeScope.class.getMethod("name"); + AttributeScope.class.getMethod("valueOf", int.class); + } + + /** + * Asserts that {@link AttributesDao} declares a method with exactly the given name, return type + * and ordered parameter types, pinning the v4.3.1.2 SPI surface this module consumes. + */ + private static void assertAttributesSpiMethod( + String name, Class expectedReturn, Class[] expectedParams) + throws NoSuchMethodException { + Method method = AttributesDao.class.getMethod(name, expectedParams); + assertEquals( + expectedReturn, + method.getReturnType(), + "AttributesDao." + name + " return type drifted from the pinned SPI expectation"); + assertArrayEquals( + expectedParams, + method.getParameterTypes(), + "AttributesDao." + name + " parameter types drifted from the pinned SPI expectation"); + } + @Test void readTsKvQueryResultSurfaceUsedByDaoExists() throws NoSuchMethodException { // The raw read path constructs ReadTsKvQueryResult(queryId, entries, lastTs) and reads back the