[sentinel_one] Add Support for Optional Threat Timeline Collection by mohitjha-elastic · Pull Request #19990 · elastic/integrations

mohitjha-elastic · 2026-07-06T10:18:32Z

Proposed commit message

sentinel_one: Adds optional threat timeline collection to the threat data stream

This change adds optional threat timeline collection to the SentinelOne threat data stream.
When enabled, the integration retrieves paginated timeline entries for each threat and emits
each entry as a separate event under the `sentinel_one.threat.timeline.*` fields.
Additionally, STAR rule timeline entries are mapped to the ECS `rule.id`, `rule.name`, and
`rule.description` fields to provide enriched detection metadata.

Checklist

I have reviewed tips for building integrations and this pull request is aligned with them.
I have verified that all data streams collect metrics or logs.
I have added an entry to my package's changelog.yml file.
I have verified that Kibana version constraints are current according to guidelines.
I have verified that any added dashboard complies with Kibana's Dashboard good practices

How to test this PR locally

Clone integrations repo.
Install the elastic package locally.
Start the elastic stack using the elastic package.
Move to integrations/packages/sentinel_one directory.
Run the following command to run tests.

elastic-package test -v

infra-vault-gh-plugin-prod · 2026-07-06T10:18:37Z

Pinging @elastic/security-service-integrations (Team:Security-Service Integrations)

github-actions · 2026-07-06T10:20:52Z

Elastic Docs Style Checker (Vale)

Summary: 2 suggestions found

💡 Suggestions (2): Optional style improvements. Apply when helpful.

File	Line	Rule	Message
packages/sentinel_one/data_stream/threat/manifest.yml	29	Elastic.WordChoice	Consider using 'deactivate, deselect, hide, turn off' instead of 'Disable', unless the term is in the UI.
packages/sentinel_one/data_stream/threat/manifest.yml	38	Elastic.WordChoice	Consider using 'can, might' instead of 'may', unless the term is in the UI.

The Vale linter checks documentation changes against the Elastic Docs style guide. To use Vale locally or report issues, refer to Elastic style guide for Vale.

vera-review-bot · 2026-07-06T10:42:56Z

-            "cursor": (size(body.?data.orValue([])) > 0) ?
-              body.data.map(e, e.?threatInfo.updatedAt.orValue(null)).filter(v, v != null).as(updates,
-                updates.size() > 0 ?
+      !has(state.worklist) ? // Exit early due to GET failure.


Severity: 🟡 Medium confidence: medium path: packages/sentinel_one/data_stream/threat/agent/stream/httpjson_as_cel.yml.hbs:96

A list-fetch error after the first successful page is silently swallowed: the phase-2 empty-worklist branch overwrites the phase-1 error event because the !has(state.worklist) guard never fires once worklist has been set.

Details

When the threat-list request fails (non-200), phase 1 returns a single-object error {"events": {"error": {...}}, "want_more": false} and does not set worklist. Phase 2 is meant to bail out via !has(state.worklist) ? state. However, draining a page sets worklist to {"data": tail(...)}, which becomes {"data": []} for the last item and is never removed. So after the first successful fetch in a session, has(state.worklist) is permanently true. On the next interval, if the list fetch errors, state.worklist is still the stale {"data": []}, so phase 2 skips the guard, falls into the empty-worklist branch, and state.with({"events": [], ...}) overwrites the error object produced by phase 1. The result: the ERROR log / degraded-status signal is lost and the failure is silently skipped (the cursor is untouched so data is retried next interval, but the error is never surfaced). The // Exit early due to GET failure guard is effectively dead code after the first evaluation.

Recommendation:

Preserve a phase-1 error before entering the timeline stage. Because events is cleared between evaluations and phase-1 success never sets events, has(state.events) reliably means "the list fetch errored" — short-circuit on it:

).as(state, state.with( has(state.events) ? // list fetch failed this evaluation; preserve the error and skip the timeline stage. state : !has(state.worklist) ? state : (size(state.?worklist.?data.orValue([])) > 0) ? state.worklist.data[0].as(threat, // ... unchanged timeline logic ... ) : (state.?next_page.token.orValue(null) != null) ? {"events": [{"retry": true}], "want_more": true} : {"events": [], "want_more": false, "list_updated_at_gte": ""} ) )

🤖 AI-Generated Review | Vera Review Bot | 📚 Knowledge base: integration-skills

⚠️ Automated review — verify suggestions before applying.

vera-review-bot · 2026-07-06T10:42:56Z

+        (state.?next_page.token.orValue(null) != null) ?
+          {
+            "events": [],
+            "want_more": true,


Severity: 🔵 Low confidence: medium path: packages/sentinel_one/data_stream/threat/agent/stream/httpjson_as_cel.yml.hbs:206

The empty-worklist branch returns want_more: true with an empty events array; the input only re-evaluates on a non-empty events, so an empty threat-list page carrying a non-null nextCursor stalls pagination until the next interval instead of following the cursor immediately.

Details

In the phase-2 empty-worklist branch, when state.next_page.token is present the program returns {"events": [], "want_more": true}. The CEL input triggers an immediate re-evaluation on want_more: true only when at least one event was published; with an empty events array the pending next_page.token is not followed within the current collection cycle. Collection does resume on the next scheduled interval (state persists next_page.token and phase 1 re-fetches with the cursor), so this is not a permanent stall, but pages after an empty-but-more list response are delayed by one interval. This path is reached only if the list API returns an empty data array together with a non-null pagination.nextCursor.

Recommendation:

Emit a placeholder event on the continuation path so the re-evaluation actually fires, and drop it before indexing:

(state.?next_page.token.orValue(null) != null) ? {"events": [{"retry": true}], "want_more": true} : {"events": [], "want_more": false, "list_updated_at_gte": ""}

and discard the placeholder in the agent processors (or the ingest pipeline):

processors: - drop_event.when.equals.retry: true

🤖 AI-Generated Review | Vera Review Bot | 📚 Knowledge base: integration-skills

⚠️ Automated review — verify suggestions before applying.

elastic-vault-github-plugin-prod · 2026-07-06T11:10:02Z

🚀 Benchmarks report

To see the full report comment with /test benchmark fullreport

efd6 · 2026-07-07T00:43:02Z

  (
-    state.?want_more.orValue(false) ?
-      state.list_updated_at_gte
+    (size(state.?worklist.?data.orValue([])) > 0) ?


Suggested change

(size(state.?worklist.?data.orValue([])) > 0) ?

state.?worklist.data[0].hasValue() ?

efd6 · 2026-07-07T00:44:18Z

+              "list_updated_at_gte": (state.?cursor.?last_update_at.orValue("") != "") ?
+                string(state.cursor.last_update_at)
+              :
+                (now - duration(state.initial_interval)).format(time_layout.RFC3339),


Suggested change

"list_updated_at_gte": (state.?cursor.?last_update_at.orValue("") != "") ?

string(state.cursor.last_update_at)

:

(now - duration(state.initial_interval)).format(time_layout.RFC3339),

"list_updated_at_gte": state.?cursor.last_update_at.orValue((now - duration(state.initial_interval)).format(time_layout.RFC3339)),

This will require rejigging the code below. In general, avoid using "", null, 0 etc as a sentinel for absence; we have absence to signal absence.

~~Also, it looks like its supposed to be last_updated_at_gte instead of list_updated_at_gte ?~~ nvm

efd6 · 2026-07-07T00:45:47Z

+                    ?"token": has_more_list ? optional.of(body.pagination.nextCursor) : optional.none(),
+                  },
+                  "has_more_timeline": false,
+                  "list_updated_at_gte": has_more_list ? state.list_updated_at_gte : "",


Suggested change

"list_updated_at_gte": has_more_list ? state.list_updated_at_gte : "",

?"list_updated_at_gte": has_more_list ? optional.of(state.list_updated_at_gte) : optional.none(),

efd6 · 2026-07-07T00:46:18Z

-                    "last_update_at": (
-                      [?state.?cursor.last_update_at] + updates
-                    ).map(t, timestamp(t)).max().format(time_layout.RFC3339),
+                    "events": (size(body.?data.orValue([])) > 0) ?


Suggested change

"events": (size(body.?data.orValue([])) > 0) ?

"events": body.?data[0].hasValue() ?

efd6 · 2026-07-07T00:46:42Z

-                updates.size() > 0 ?
+      !has(state.worklist) ? // Exit early due to GET failure.
+        state
+      : (size(state.?worklist.?data.orValue([])) > 0) ?


Suggested change

: (size(state.?worklist.?data.orValue([])) > 0) ?

: state.?worklist.data[0].hasValue() ?

efd6 · 2026-07-07T00:47:09Z

+                  {
+                    "error": {
+                      "code": string(resp.StatusCode),
+                      "id": string(resp.Status),


Suggested change

"id": string(resp.Status),

"id": resp.Status,

efd6 · 2026-07-07T00:47:16Z

+                        (size(resp.Body) != 0) ?
+                          string(resp.Body)
+                        :
+                          string(resp.Status) + " (" + string(resp.StatusCode) + ")"


Suggested change

string(resp.Status) + " (" + string(resp.StatusCode) + ")"

resp.Status + " (" + string(resp.StatusCode) + ")"

efd6 · 2026-07-07T00:48:13Z

+                "next_chain": {},
+                "has_more_timeline": false,
+                "cursor": {
+                  "last_update_at": threat.?threatInfo.updatedAt.orValue(state.?cursor.last_update_at.orValue("")),


Avoid "" for absence (also below).

1. Refactor the CEL code as per comments. 2. Updated system test as per the updated CEL code.

elastic-vault-github-plugin-prod · 2026-07-07T09:47:09Z

✅ All changelog entries have the correct PR link.

vera-review-bot · 2026-07-07T09:53:04Z

No issues across the latest commits acd947c.

Review summary

Issues found across earlier commits b0c46c5 — 1 medium, 1 low

🟡 A list-fetch error after the first successful page is silently swallowed: the phase-2 empty-worklist branch overwrites the phase-1 error event because the !has(state.worklist) guard never fires once worklist has been set. (link) (Unresolved)
🔵 The empty-worklist branch returns want_more: true with an empty events array (link) (Unresolved)

A new commit triggers another review — at most once every 15 minutes. I skip the PR while it's approved or has merge conflicts.

🤖 AI-Generated Review | Vera Review Bot | 📚 Knowledge base: integration-skills

⚠️ Automated review — verify suggestions before applying.

kcreddy · 2026-07-07T09:50:13Z

+        type: bool
+        title: Enable Threat Timeline Collection
+        description: >-
+          When enabled, each threat is followed by paginated requests to the threat timeline endpoint (`/web/api/v2.1/threats/{threat_id}/timeline`). For each timeline history entry, one document is emitted containing the full threat response with that entry nested under `timeline`. Threats whose timeline returns HTTP 200 with an empty `data` array emit a single threat document. If the timeline endpoint returns HTTP 404 (for example, missing permissions or an unavailable feature), the threat document is still emitted and collection advances to the next threat. Disable to collect threats from the list endpoint only.


Does missing permissions also cause 404? In CEL code, 404 emits a threat document, but other non-200 returns error. So just want to clarify

kcreddy · 2026-07-07T09:52:51Z

-    "message": "Threat Detected: default.exe (malicious)",
+    "message": "STAR rule matched",


Looks like we are overriding previous value after adding timeline. This could break?

infra-vault-gh-plugin-prod · 2026-07-07T10:33:02Z

💚 Build Succeeded

Buildkite Build
Commit: acd947c

History

💚 Build #45525 succeeded b0c46c5

cc @mohitjha-elastic

Add support for optional threat timeline collection

719c810

mohitjha-elastic self-assigned this Jul 6, 2026

mohitjha-elastic requested review from a team as code owners July 6, 2026 10:18

Update changelog entry

b0c46c5

mohitjha-elastic changed the title ~~Add support for optional threat timeline collection~~ [sentinel_one] Add support for Optional Threat Timeline Collection Jul 6, 2026

mohitjha-elastic changed the title ~~[sentinel_one] Add support for Optional Threat Timeline Collection~~ [sentinel_one] Add Support for Optional Threat Timeline Collection Jul 6, 2026

mohitjha-elastic mentioned this pull request Jul 6, 2026

[sentinel_one] Migrate Threat Data stream to CEL #19881

Merged

5 tasks

vera-review-bot Bot reviewed Jul 6, 2026

View reviewed changes

efd6 reviewed Jul 7, 2026

View reviewed changes

Resolve review comments.

acd947c

1. Refactor the CEL code as per comments. 2. Updated system test as per the updated CEL code.

mohitjha-elastic requested a review from efd6 July 7, 2026 09:44

kcreddy reviewed Jul 7, 2026

View reviewed changes

	(size(state.?worklist.?data.orValue([])) > 0) ?
	state.?worklist.data[0].hasValue() ?

	"list_updated_at_gte": has_more_list ? state.list_updated_at_gte : "",
	?"list_updated_at_gte": has_more_list ? optional.of(state.list_updated_at_gte) : optional.none(),

	"events": (size(body.?data.orValue([])) > 0) ?
	"events": body.?data[0].hasValue() ?

	: (size(state.?worklist.?data.orValue([])) > 0) ?
	: state.?worklist.data[0].hasValue() ?

	string(resp.Status) + " (" + string(resp.StatusCode) + ")"
	resp.Status + " (" + string(resp.StatusCode) + ")"

		"message": "Threat Detected: default.exe (malicious)",
		"message": "STAR rule matched",

Uh oh!

Conversation

mohitjha-elastic commented Jul 6, 2026

Proposed commit message

Checklist

How to test this PR locally

Uh oh!

infra-vault-gh-plugin-prod Bot commented Jul 6, 2026

Uh oh!

github-actions Bot commented Jul 6, 2026

Elastic Docs Style Checker (Vale)

Uh oh!

vera-review-bot Bot Jul 6, 2026

Choose a reason for hiding this comment

Uh oh!

vera-review-bot Bot Jul 6, 2026

Choose a reason for hiding this comment

Uh oh!

elastic-vault-github-plugin-prod Bot commented Jul 6, 2026

🚀 Benchmarks report

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

kcreddy Jul 7, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

elastic-vault-github-plugin-prod Bot commented Jul 7, 2026

Uh oh!

vera-review-bot Bot commented Jul 7, 2026

Review summary

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

infra-vault-gh-plugin-prod Bot commented Jul 7, 2026

💚 Build Succeeded

History

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

kcreddy Jul 7, 2026 •

edited

Loading