Add Android Broker OCE Weekly Telemetry Report skill, Fixes AB#3601076 by shahzaibj · Pull Request #438 · AzureAD/android-complete

shahzaibj · 2026-06-10T05:43:59Z

Summary

Adds the oncall-weekly-telemetry-report Copilot skill to the android-complete repo. The skill automates the weekly Android Broker on-call (OCE) WoW + 60-day trend telemetry report, producing a polished self-contained HTML deliverable at ~/android-oce-reports/oncall-wow-report-<sunday>.html. It pulls from android_spans materialized views in Kusto, runs the mandatory originator pre-check on every regression, attributes spikes to PRs in broker/ and common/, and gates the publish on a multi-step validator + optional Playwright layout smoke test.

This is the first check-in of the skill. It has been used end-to-end to produce three weekly reports (2026-05-03, 2026-05-31); the hard-won lessons from those runs are baked into the assets so the next on-call who picks it up gets a much more boring experience than I did.

What's included

Path	Purpose
`.github/skills/oncall-weekly-telemetry-report/SKILL.md`	Full skill instructions inputs to confirm, 7-step workflow, hard rules, output checklist
`assets/report-template.html`	Canonical layout a real prior-week report kept verbatim. Authors edit in place. CSS + footer JS are canonical and ship with all the layout guards from past regressions
`assets/template-readme.md`	Author guide what changes per week, the two v8 layout traps (cards-touching + dim-row bleed), sparkline color palette, full CSS class reference (incl. status pills)
`assets/kusto-cheatsheet.md`	Schemas, helper funcs (`MergeAccountType`, `MergeIsSharedDevice`, `MergeUiRequiredExceptions`), HLL device-count gotcha, AADSTS reference table, ready-to-paste query templates
`assets/code-attribution-template.md`	Per-card 8-field checklist (Originator / Top throw site / Wrapper / Caller hot-spots / Underlying cause / Top error_messages / Likely PRs / Next step)
`assets/queries/*.kql`	13 canonical KQL templates: 60d trends (codes + types), WoW movers, reliability, latency, broker-version share, app share, attribution-union (all 7 dims, one round-trip), error-message-and-location (the mandatory originator pre-check), WoW-table sparkline source-of-truth, type sub-code decomposition
`assets/templates/{spike-card,traffic-attr-card,sparkline-footer}.html`	Copy-paste HTML snippets for the deep-dive attribution cards
`assets/bucket-trends.js`	Buckets every error code/type into 60d regression / spike / improvement / flat across the 8-week window. Run 4x (cross-product `{code,type} x {devices,requests}`). `--summary` and `--json` flags for clean output
`assets/summarize-attribution.js`	Rolls up 7-dim attribution slices for spike-attribution cards. Supports both per-dim files (legacy) and the preferred 1-query union form. Auto-detects schema produced by `scripts/run-kql.ps1`
`assets/agg.js`	Per-error per-dim top-N rollup with WoW deltas
`assets/find-suspect-prs.ps1`	Parallel pickaxe + grep across `broker/`+`common/` for a class/method symbol, returns PR numbers + GitHub URLs
`assets/scripts/bootstrap-report.ps1`	Bootstrap new report from template auto-computes reporting Sunday, prunes `_data/<old-sunday>/` older than 60 days, multi-marker collision detection (title + meta date + first KPI + size ratio) so populated reports are never silently overwritten
`assets/scripts/run-kql.ps1`	Direct-REST Kusto fallback when the Azure MCP server times out (frequent on per-error-code queries). Acquires token via `az`, POSTs to `/v2/rest/query`, writes JSON the JS helpers consume directly
`assets/scripts/visual-smoke.ps1`	Optional Playwright-based layout smoke test renders the report at 1400px in headless Chromium, screenshots to `~/android-oce-reports/_visual/`, runs DOM checks for element overflow inside `.dim`/`.attr-card` and adjacent-card gap < 8px. Catches rendered-layout bugs that HTML/CSS validation can't see
`assets/validate-report.ps1`	Pre-publish hard-fail validator 10 checks covering stale tokens, `devs`/`reqs` leaks in user-facing text, U+FFFD mojibake, Section 2 div balance, nested-callout detection, KPI sparkline coverage, code-attribution depth, attribution-card layout-guard CSS, fabricated-sparkline heuristic

Workflow overview

Bootstrap the report from the canonical template (scripts/bootstrap-report.ps1)
Pull WoW data via Kusto MCP, or scripts/run-kql.ps1 if the MCP times out reliability, latency, broker-version share, app share
Pull 60-day trend for both error_code and error_type; bucket 4x (cross-product {code,type} x {devices,requests}), union the regressions
Originator pre-check (mandatory) on every broker-tagged card via error-message-and-location.kql hard rule: if throw site is in common/ExceptionAdapter.* AND message starts with AADSTS, the originator is eSTS, not broker
7-dim spike attribution via one union query (span / calling app / active broker / broker version / account type / shared device / client SKU) + on-demand OS-version slice
PR-grep workflow read the full 4-week PR window first (<30 PRs, scannable end-to-end); use find-suspect-prs.ps1 only after the originator pre-check identifies a specific symbol
Traffic attribution distinguish "per-request rate moved" from "calling-app volume moved"
Validate validate-report.ps1 (hard checks) + optional scripts/visual-smoke.ps1 (DOM-based layout checks) before publish

Hard rules baked into the assets

Never sum(countDevices) always dcount_hll(hll_merge(countDevicesHll)) (HLL merge); summing per-row distinct counts double-counts cross-slice
Never sum percentiles percentile_tdigest(tdigest_merge(responseTimeTDigest), N, typeof(long)) only
Always use the auth-only denominator (SilentAuthStatsAllRequestsMetrics U InteractiveAuthStatsAllRequestsMetrics) for reliability %, never total broker spans
Always filter the partial in-progress week at the source with | where week < datetime(<END>)
Validator enforces: no {{TOKEN}} placeholders, no EXAMPLE CONTENT BELOW sentinels, no devs/reqs in user-facing text, no U+FFFD mojibake, no nested callouts in Section 2, no chartless KPI grid, full 8-field code-attribution on every card, layout-guard CSS present, no obviously-fabricated data-trend arrays

Triggers

/oncall-weekly-telemetry-report, "produce the OCE report", "weekly on-call report", "WoW telemetry report", "weekly broker health report", "generate this week's on-call summary"

Verification

Used end-to-end to produce the 2026-05-31 report (231 KB, 78 chart elements, 20 attribution cards). Validator clean (1 advisory warning about 4 fabricated low-volume sparklines correctly flagged). Visual smoke clean (0 overflow, 0 card-adjacency issues).
validate-report.ps1 runs in ~1 s and exits non-zero on any hard-check failure
scripts/visual-smoke.ps1 auto-installs Playwright + Chromium into %LOCALAPPDATA%\oce-skill-playwright on first run

Notes

Reports are written to ~/android-oce-reports/, not to the workspace, so they can never be accidentally committed
The skill ships with the lessons from three production runs already encoded: the v8 layout traps (cards-touching, dim-row text bleed), the UTF-8 PowerShell heredoc trap, the fabricated-sparkline guard, the smarter bootstrap collision detection, the MCP-fallback REST helper, and the bucket-trends summary/JSON-sidecar flags

Fixes AB#3601076

github-actions · 2026-06-10T05:44:30Z

❌ Work item link check failed. Description does not contain AB#{ID}.

Click here to Learn more.

Move 5 scripts (agg.js, bucket-trends.js, summarize-attribution.js, find-suspect-prs.ps1, validate-report.ps1) from assets/ root into assets/scripts/ where bootstrap-report.ps1, run-kql.ps1, and visual-smoke.ps1 already live. Move report-template.html and template-readme.md from assets/ root into assets/templates/ alongside the spike-card / traffic-attr-card / sparkline-footer snippets they're conceptually grouped with. Move kusto-cheatsheet.md and code-attribution-template.md from assets/ root into a new assets/docs/ folder. Update all cross-references in SKILL.md, the query templates, the script self-documentation, and the bootstrap script's template-lookup path. Validator + bootstrap still pass end-to-end after the move. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

shahzaibj · 2026-06-10T05:59:04Z

Pushed a follow-up commit (7c9c378) to reorganize the asset layout. Before the
move the assets/ root had a mix of scripts, docs, templates, and one
subfolder (scripts/) holding a different set of scripts files that shouldn't
have been at the root were inconsistent with their conceptually-matched
subfolders.

New layout:

assets/
 docs/                       (new)
    code-attribution-template.md
    kusto-cheatsheet.md
 queries/                    (unchanged)
    *.kql
 scripts/                    (consolidated)
    agg.js
    bootstrap-report.ps1
    bucket-trends.js
    find-suspect-prs.ps1
    run-kql.ps1
    summarize-attribution.js
    validate-report.ps1
    visual-smoke.ps1
 templates/                  (consolidated)
     README.md
     report-template.html
     sparkline-footer.html
     spike-card.html
     template-readme.md
     traffic-attr-card.html

All cross-references updated in SKILL.md, the query templates, the script
self-documentation, and bootstrap-report.ps1's template-lookup path. Git
recognized every move as a rename (95-100% similarity) so blame history is
preserved.

Validator + bootstrap still pass end-to-end after the move.

cacosta33 · 2026-06-15T17:53:47Z

+# Error types — by devices, then by requests (note --key)
+node .github\skills\oncall-weekly-telemetry-report\assets\bucket-trends.js <types.json> --start=2026-03-08 --end=2026-05-10 --key=unified_error_type
+node .github\skills\oncall-weekly-telemetry-report\assets\bucket-trends.js <types.json> --start=2026-03-08 --end=2026-05-10 --key=unified_error_type --metric=reqs
+```


These examples reference paths that don’t exist after the asset reorg. The files live under assets\scripts\..., but the commands use assets\bucket-trends.js, assets\find-suspect-prs.ps1, and assets\summarize-attribution.js. Anyone following the documented flow will hit file-not-found immediately.

Can we update these examples to:

assets\scripts\bucket-trends.js
assets\scripts\find-suspect-prs.ps1
assets\scripts\summarize-attribution.js

Fixed in dc574d3 - updated to assets\scripts\bucket-trends.js (4 occurrences across Steps 3 + 5).

cacosta33 · 2026-06-15T17:54:24Z

+# Searches both repos in parallel via `git log -S` (pickaxe on diff) AND `--grep` (subject).
+# Returns a unified table: repo | date | author | sha | PR# | URL | subject.
+.\.github\skills\oncall-weekly-telemetry-report\assets\find-suspect-prs.ps1 `
+  -Symbol 'ExceptionAdapter' -Since 2026-04-01 -Until 2026-05-09


Same command path issue as above

Fixed in dc574d3 - example now uses assets\scripts\find-suspect-prs.ps1.

cacosta33 · 2026-06-15T17:54:38Z

+
+```pwsh
+# Union mode (preferred). attr-union.json comes from attr-union-by-dim.kql.
+node .github\skills\oncall-weekly-telemetry-report\assets\summarize-attribution.js `


Same command path issue as above

Fixed in dc574d3 - example now uses assets\scripts\summarize-attribution.js (union mode).

cacosta33 · 2026-06-15T17:54:51Z

+Legacy per-dim mode (one JSON per dimension) is still supported for the rare wider-time-window case:
+
+```pwsh
+node .github\skills\oncall-weekly-telemetry-report\assets\summarize-attribution.js `


Same command path issue as above

Fixed in dc574d3 - example now uses assets\scripts\summarize-attribution.js (legacy per-dim mode).

cacosta33 · 2026-06-15T17:59:31Z

@@ -0,0 +1,531 @@
+---
+name: oncall-weekly-telemetry-report
+description: Generate the weekly Android Broker on-call (OCE) WoW + 60-day trend telemetry report as a polished self-contained HTML file. Use this skill for the weekly OCE rotation when asked to "produce the OCE report", "weekly on-call report", "WoW telemetry report", "weekly broker health report", or "generate this week's on-call summary". Pulls from `android_spans` materialized views, attributes regressions/improvements to PRs in `broker/` and `common/`, and writes to `oncall-wow-report-vN.html` at repo root.


For consistency purposes, you probably want to change this to $env:USERPROFILE\android-oce-reports\ so that it ensures that we don't write to repo root.

I know it's mentioned elsewhere to not write at repo root but this contradiction could potentially confuse to model in rare cases so probably best to just modify this

Good catch - fixed in dc574d3. The YAML description now says:

writes to $env:USERPROFILE\android-oce-reports\oncall-wow-report-YYYY-MM-DD.html (outside the workspace so reports are never committed)

so the model can't conflate it with repo-root anywhere.

cacosta33 · 2026-06-15T18:00:13Z

+
+Produce the weekly Android Broker on-call (OCE) telemetry report as a self-contained HTML file at `$env:USERPROFILE\android-oce-reports\oncall-wow-report-v{N+1}.html` (i.e. `~/android-oce-reports/`, outside the workspace so reports never accidentally get committed).
+
+The output mirrors the structure of the canonical template at [`assets/templates/report-template.html`](assets/templates/report-template.html) — copy it to `oncall-wow-report-v{N+1}.html` at repo root and edit in place. Do **not** redesign the layout each week.


Same as above in relation to ensuring we don't write to repo-root

Fixed in dc574d3 together with the L3 description. The intro line no longer mentions a v{N+1} filename or "at repo root" - it points directly at ~/android-oce-reports/oncall-wow-report-YYYY-MM-DD.html and notes that Step 1's bootstrap script does the copy into the user-home folder for you.

cacosta33 · 2026-06-15T18:06:45Z

+    } finally { Pop-Location }
+}
+
+if ($results.Count -eq 0) {


Nitpick:

This helper defaults RepoRoot to $env:USERPROFILE\Repos\android-complete. If that layout doesn’t exist, it only warns and then exits with “No PRs match...”, which is indistinguishable from a real no-match result. That can hide attribution candidates instead of failing loudly.

It would be safer to either:

fail when neither repo path exists, or
resolve paths relative to the current workspace rather than assuming a fixed ~/Repos/android-complete layout.

Both options applied in dc574d3:

-RepoRoot now defaults to git rev-parse --show-toplevel instead of ~/Repos/android-complete - running from inside any clone of android-complete (where broker/ and common/ are submodules) just works. Explicit -RepoRoot still wins.

The script now FAILS LOUDLY (exit 2 + actionable message naming the expected layout) when neither broker/ nor common/ exists under the resolved root, instead of warning and printing "No PRs match..." which was indistinguishable from a real no-match result.

If only one of the two is missing, it warns + continues with a "results will be incomplete" note.

somalaya · 2026-06-15T19:20:48Z

@@ -0,0 +1,531 @@
+---
+name: oncall-weekly-telemetry-report
+description: Generate the weekly Android Broker on-call (OCE) WoW + 60-day trend telemetry report as a polished self-contained HTML file. Use this skill for the weekly OCE rotation when asked to "produce the OCE report", "weekly on-call report", "WoW telemetry report", "weekly broker health report", or "generate this week's on-call summary". Pulls from `android_spans` materialized views, attributes regressions/improvements to PRs in `broker/` and `common/`, and writes to `oncall-wow-report-vN.html` at repo root.


The YAML description says oncall-wow-report-vN.html and line ~10 says oncall-wow-report-v{N+1}.html, but the bootstrap script, the output checklist, and the hard rules all use oncall-wow-report-YYYY-MM-DD.html (date-based). The date-based convention is what's actually implemented. The vN references look like leftovers from an earlier design - can we update them to match?

Good catch - those were leftovers from an earlier design that never got swept. Fixed in dc574d3: both the YAML description (L3) and the intro line (L10) now use the actual implemented convention (oncall-wow-report-YYYY-MM-DD.html in ~/android-oce-reports/).

somalaya · 2026-06-15T19:29:53Z

@@ -0,0 +1,531 @@
+---
+name: oncall-weekly-telemetry-report


Can we also add this skill to copilot-instructions.md skill table? I see all other skills are listed

Added in dc574d3 at the bottom of the Section 12 skills table:

| oncall-weekly-telemetry-report | .github/skills/oncall-weekly-telemetry-report/SKILL.md | "produce the OCE report", "weekly on-call report", "WoW telemetry report", "weekly broker health report", "generate this week''s on-call summary" |

SKILL.md fixes (cacosta33 + somalaya): - Update 4 example commands in Step 3 (bucket-trends.js x2) and Step 5 (summarize-attribution.js x2) to use the new assets/scripts/ paths so the documented flow doesn't hit file-not-found. - Update Step 4 find-suspect-prs.ps1 example to use assets/scripts/. - Replace 'writes to oncall-wow-report-vN.html at repo root' (YAML description line and intro line) with the actual implemented convention: \C:\Users\shjameel\android-oce-reports\oncall-wow-report-YYYY-MM-DD.html outside the workspace. The vN-and-repo-root phrasing was a leftover from an earlier design and contradicted the bootstrap script, output checklist, and hard rules. find-suspect-prs.ps1 (cacosta33): - Resolve RepoRoot from \git rev-parse --show-toplevel\ by default (works from any clone of android-complete) instead of assuming a fixed ~/Repos/android-complete layout. Explicit -RepoRoot still wins. - Fail loudly (exit 2 + actionable error message naming the expected layout) when neither broker/ nor common/ exists under the resolved root. Previous behaviour silently warned and printed 'No PRs match...' which was indistinguishable from a real no-match result and could hide attribution candidates. - Warn (but continue) when only one of the two subrepos is missing. copilot-instructions.md (somalaya): - Add the oncall-weekly-telemetry-report skill to the Section 12 skills table for consistency with every other skill in the repo. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

shahzaibj added 6 commits May 9, 2026 20:09

Add oncall telemetry weekly report skill

a36da31

Updates

c00e474

Updates to telemetry OCE report skill

047e3cb

Merge branch 'master' into shahzaibj/on-call-telemetry-report-skill

e8b5a42

Template fixes

427d8e4

More updates

a216b2c

shahzaibj requested a review from a team as a code owner June 10, 2026 05:44

shahzaibj self-assigned this Jun 10, 2026

github-actions Bot changed the title ~~Add Android Broker OCE Weekly Telemetry Report skill~~ Add Android Broker OCE Weekly Telemetry Report skill, Fixes AB#3601076 Jun 11, 2026

cacosta33 reviewed Jun 15, 2026

View reviewed changes

somalaya reviewed Jun 15, 2026

View reviewed changes

somalaya approved these changes Jun 16, 2026

View reviewed changes


		Produce the weekly Android Broker on-call (OCE) telemetry report as a self-contained HTML file at `$env:USERPROFILE\android-oce-reports\oncall-wow-report-v{N+1}.html` (i.e. `~/android-oce-reports/`, outside the workspace so reports never accidentally get committed).

		The output mirrors the structure of the canonical template at [`assets/templates/report-template.html`](assets/templates/report-template.html) — copy it to `oncall-wow-report-v{N+1}.html` at repo root and edit in place. Do not redesign the layout each week.

Conversation

shahzaibj commented Jun 10, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

What's included

Workflow overview

Hard rules baked into the assets

Triggers

Verification

Notes

Uh oh!

github-actions Bot commented Jun 10, 2026

Uh oh!

shahzaibj commented Jun 10, 2026

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

shahzaibj commented Jun 10, 2026 •

edited

Loading