aws: derive OTel-compatible trace.id from X-Ray trace ID in elb_logs

[grahamprimm]

Proposed commit message

aws: derive OTel-compatible trace.id from X-Ray trace ID in elb_logs

WHAT: The elb_logs ingest pipeline copied the raw ALB access log trace field (Root=1-{8 hex}-{24 hex}[;Self=...]) into ECS trace.id verbatim. This change replaces the verbatim set processor with a Painless script that strips the Root=1- prefix, drops any ;Self=/;Parent= segments, and concatenates the two hex segments into the OpenTelemetry/ECS 32-character hex form. Values that don't match the X-Ray v1 format fall back to the previous raw-copy behavior. The raw value remains available unchanged in aws.elb.trace_id.

WHY: APM agents using the X-Ray propagator (OTel xray, AWS X-Ray SDKs) derive a 32-character hex trace_id from the same X-Amzn-Trace-Id header the ALB writes to its access log. Because the pipeline stored the raw Root=1-... string, the two values never matched, so ELB access logs could not be correlated with application traces on trace.id in Kibana. With this change, a request that traverses an ALB lands on the same trace.id in both the access log and the application spans.

Checklist

• I have reviewed tips for building integrations and this pull request is aligned with them.
• I have verified that all data streams collect metrics or logs.
• I have added an entry to my package's changelog.yml file.
• I have verified that Kibana version constraints are current according to guidelines.
• I have verified that any added dashboard complies with Kibana's Dashboard good practices

Author's Checklist

• Pipeline test expectations (test-alb.log-expected.json), sample_event.json, and generated docs were updated by hand with the deterministic transform (Root=1-58337262-36d2... → 5833726236d2...); please confirm CI pipeline tests regenerate identically.
• Confirm behavior change to trace.id is acceptable as a minor version bump (raw value still preserved in aws.elb.trace_id).

How to test this PR locally

cd packages/aws
elastic-package stack up -d
elastic-package test pipeline --data-streams elb_logs -v

An ALB log line containing "Root=1-58337262-36d228ad5d99923122bbe354" should produce trace.id: 5833726236d228ad5d99923122bbe354 while aws.elb.trace_id retains the raw value. This matches the trace ID derivation performed by the OTel xray propagator, e.g. following https://opentelemetry.io/docs/languages/java/configuration/ with OTEL_PROPAGATORS=xray.

Related issues

🤖 Generated with Claude Code

[github-actions]

✅ Elastic Docs Style Checker (Vale)

No issues found on modified lines!

---

The Vale linter checks documentation changes against the Elastic Docs style guide. To use Vale locally or report issues, refer to Elastic style guide for Vale.

[github-actions]

TL;DR

Buildkite failed while uploading the pipeline because the post-checkout hook could not merge PR #20000 with current main. The conflicts are in packages/aws/changelog.yml and packages/aws/data_stream/elb_logs/elasticsearch/ingest_pipeline/default.yml, so no package tests ran.

Remediation

• Rebase or merge latest main into the PR branch and resolve the two conflicts.
• In packages/aws/changelog.yml, account for the existing top 6.21.0 entry on main; for this enhancement, either add the PR entry under the appropriate unreleased AWS version or bump to the next package version according to maintainer convention.
• In packages/aws/data_stream/elb_logs/elasticsearch/ingest_pipeline/default.yml, keep the new X-Ray-to-OTel trace.id script logic while preserving the processor tag added on main around the old trace.id processor.
• Re-run the Buildkite pipeline after the branch merges cleanly.

Investigation details

Root Cause

The failing step is :pipeline::arrow_up: Upload Pipeline: .buildkite/pipeline.yml, but the actual error happens earlier in .buildkite/hooks/post-checkout when Buildkite tries to create the PR merge branch. The merge against main conflicts in two files:

• packages/aws/changelog.yml: the PR adds a new 6.21.0 entry, while main already has 6.21.0 at the top for PR aws.inspector: fix CLOSED/SUPPRESSED findings staying ACTIVE #19918.
• packages/aws/data_stream/elb_logs/elasticsearch/ingest_pipeline/default.yml: the PR replaces the old trace.id set processor with a Painless script, while main has changed the same area to add required processor tags.

Evidence

• Build: https://buildkite.com/elastic/integrations/builds/45559
• Job/step: :pipeline::arrow_up: Upload Pipeline: .buildkite/pipeline.yml
• Key log excerpt:

Auto-merging packages/aws/changelog.yml
CONFLICT (content): Merge conflict in packages/aws/changelog.yml
Auto-merging packages/aws/data_stream/elb_logs/elasticsearch/ingest_pipeline/default.yml
CONFLICT (content): Merge conflict in packages/aws/data_stream/elb_logs/elasticsearch/ingest_pipeline/default.yml
Automatic merge failed; fix conflicts and then commit the result.
Merge failed: 1
🚨 Error: running "repository post-checkout" shell hook: The repository post-checkout hook exited with status 1

On current main, packages/aws/changelog.yml:2 already starts with version: "6.21.0", and packages/aws/data_stream/elb_logs/elasticsearch/ingest_pipeline/default.yml:176-180 still has the trace.id processor with a tag. The PR diff changes both same regions, which explains the post-checkout merge failure.

Verification

Not run locally; the failure occurs before test execution during Buildkite checkout/merge, and the log shows deterministic git merge conflicts that must be resolved in the PR branch.

---

What is this? | From workflow: PR Buildkite Detective

Give us feedback! React with 🚀 if perfect, 👍 if helpful, 👎 if not.

[elastic-vault-github-plugin-prod]

✅ All changelog entries have the correct PR link.

[elastic-vault-github-plugin-prod]

🚀 Benchmarks report

To see the full report comment with /test benchmark fullreport

[infra-vault-gh-plugin-prod]

💚 Build Succeeded

• Buildkite Build
• Commit: 1bb98f5

History

• 💔 Build #45559 failed 18012ad