InFlight

Benchmark air-traffic-control (ATC) speech-to-text across three models:

• wav2vec2 ATC — Jzuluaga/wav2vec2-large-960h-lv60-self-en-atc-atcosim, a wav2vec2-large fine-tuned on ATC speech, run fully on-device.
• Deepgram — nova-3 (cloud API).
• AssemblyAI — universal-3-pro (cloud API).

...on two ATC test sets:

• ATCO2 test-set-1h — Jzuluaga/atco2_corpus_1h — 871 segments of real, noisy tower/approach radio.
• ATCOSIM test — Jzuluaga/atcosim_corpus — 1,900 segments of clean, simulated ATC speech from non-native controllers.

📊 Results & analysis live in docs/DATASETS.md (regenerated by inflight report). 🧭 Research direction — where this is going (multimodal: sensors ↔ transcription) is in docs/RESEARCH_PLAN.md.

Setup

cd InFlight
uv sync                      # creates .venv and installs deps (incl. torch/transformers)
cp .env.example .env         # then paste your Deepgram + AssemblyAI keys

Keys (get them here):

• Deepgram — https://console.deepgram.com/
• AssemblyAI — https://www.assemblyai.com/app/account

The wav2vec2 model (~1.2 GB) downloads to the HuggingFace cache on first use and runs locally (CUDA → Apple MPS → CPU, auto-detected). No key needed.

Usage

uv run inflight datasets                      # list known datasets

# --- per dataset (default dataset is atco2) ---
uv run inflight download --dataset atcosim    # fetch from HuggingFace
uv run inflight extract  --dataset atcosim    # -> per-segment 16 kHz WAVs + manifest.jsonl
uv run inflight evaluate --dataset atcosim    # transcribe with all 3 models + score WER/CER

uv run inflight report                        # regenerate docs/DATASETS.md from results/

# quick smoke test on 10 segments:
uv run inflight evaluate --dataset atco2 --limit 10

# --- multimodal (atco2 only) ---
uv run inflight context --dataset atco2       # + surveillance callsigns, role, gold NER
uv run inflight oracle  --dataset atco2       # is the spoken callsign even in the list?

# RQ1: boost ASR with the surveillance callsign list, vs the un-boosted baseline
uv run inflight boost --airports LKPR,LSZH,LZIB              # commercial hubs
uv run inflight boost --airports LKPR,LSZH,LZIB --placebo    # control: WRONG callsigns

uv run inflight intent   # RQ3: extract (callsign, command, value); does boosting survive?
uv run inflight gate     # should we boost THIS utterance? gated vs always vs never
uv run inflight readback # read-back error detection; what ASR error costs a safety monitor

Multimodal context

inflight context fetches the official ATCO2 release (free, anonymous) and joins it onto the manifest by segment id, writing manifest_context.jsonl. The HuggingFace mirror we benchmark on carries only audio + transcript; the official release adds, per utterance:

• callsigns_nearby — the aircraft surveillance (OpenSky ADS-B) actually saw on frequency, each with a ready-made verbalization (RYR73AH : Ryan Air Seven Three Alfa Hotel). This is the contextual signal that Direction A of the research plan boosts on.
• entities — gold callsign / command / value spans.
• role — ATCO vs pilot — plus airport, controller position, VHF frequency, UTC timestamp, and nearby waypoints.

inflight oracle then answers the question that bounds the whole idea: is the callsign the pilot actually said even in that list? Corpus-wide, only 34.8% of the time — ranging from 79% at Prague to 12% at Bern, because ADS-B sees airliners, not light aircraft. See docs/RESEARCH_PLAN.md §6.

Does the context help? (RQ1)

inflight boost feeds that callsign list to the ASR as boosted terms and scores it against the un-boosted baseline. Headline — full write-up in docs/analysis/rq1_callsign_boosting.md:

Condition	Callsign accuracy	Δ	Δ WER
Commercial hubs (AssemblyAI)	36.1% → 69.7%	+33.6	−1.67
Placebo — same audio, wrong recording's callsigns	36.1% → 39.9%	+3.8	+0.82
GA fields — where ADS-B is blind	36.8% → 36.3%	−0.5	+4.97

The placebo is what makes it a result: +29.8 points are specific to knowing which aircraft are actually on frequency, not to being handed plausible aviation phrases. And boosting where surveillance is blind is worse than not boosting — it drags WER up by 5 points.

So naive boosting is not shippable. inflight gate fixes that: boost unless the ASR's own first pass heard a bare registration (a light aircraft — which ADS-B cannot see, and which the ASR already transcribes well). That keeps 97% of the callsign gain and turns the WER cost into a gain — +11.7 CSA at −0.42 WER, corpus-wide, beating never-boosting on both metrics at once. See docs/analysis/gating_policy.md.

Does it survive downstream? (Direction B)

Yes — the chain holds end to end:

surveillance context  ->  +33.6 callsign accuracy         (inflight boost)
                      ->  +12.0 callsign F1 in intent     (inflight intent)
                      ->  +73% more read-backs recovered  (inflight readback)

Callsign recognition is the bottleneck of the whole safety stack, and surveillance context is what relieves it.

But it is not yet enough. Even gated, a read-back monitor recovers only 28% of the read-backs a perfect transcript would find. A safety checker that silently misses seven clearances in ten is worse than useless. The method works; the transcripts do not — real-radio ASR at ~32% WER is the binding constraint on Direction B. See readback_detection.md.

Options (evaluate)

• --dataset atco2|atcosim — which test set (default atco2)
• --providers wav2vec2,deepgram,assemblyai — pick which to run (default: all three)
• --limit N — only the first N segments
• --workers K — concurrent API requests for the cloud providers (default 4). wav2vec2 always runs serially (single in-process model on GPU/CPU).
• --refresh — ignore cached transcripts and re-transcribe

Transcripts are cached to results/<dataset>/<provider>_transcripts.jsonl, so reruns and interruptions are cheap — only un-transcribed segments hit the API / model.

Output

Per dataset, under results/<dataset>/:

• summary.json — corpus WER/CER per provider
• per_segment.csv — reference + each provider's hypothesis, per-segment WER/CER
• Printed comparison table

inflight report rolls every dataset's summary.json up into docs/DATASETS.md (dataset statistics + performance comparison). Narrative analysis is hand-maintained in docs/analysis/.

Scoring

WER and CER are computed with jiwer after ATC-aware normalization of reference and hypothesis, identically: lowercase, strip punctuation, canonicalize numbers to their digit-by-digit reading (100 and one hundred both → one zero zero), and canonicalize ICAO-alphabet spellings (Alpha→alfa, Juliet→juliett, niner→nine). Corpus WER is word-weighted (total edits / total reference words). It lives in src/inflight/normalize.py.

Why the ATC-aware step matters. The obvious normalizer — expand digits, strip punctuation — charges a model for writing what the reference says: 27 vs two seven, Alpha vs alfa, 5,000 vs five thousand. Those are spelling differences, not recognition errors, and the penalty is not model-neutral: it is ~0 for a model fine-tuned on these references (wav2vec2) but 4–5 WER points for the cloud models, silently biasing every comparison. That legacy normalizer is kept as normalize() and reported as the WER (v1) column, so previously published numbers stay reproducible and the size of the artifact stays visible.

Adding a dataset

Add an entry to DATASETS in src/inflight/config.py:

• loader: "parquet" — a single parquet whose audio.bytes column holds WAV data (like ATCO2), or
• loader: "hf" — any HuggingFace dataset with audio + text columns; audio is decoded and resampled to 16 kHz on extract (like ATCOSIM).

Then download / extract / evaluate --dataset <key> just work.

Adding a model

Implement providers/base.Transcriber (one transcribe(wav_path) -> str method), register it in providers/__init__.get_provider. Set prefers_serial = True for local single-instance models so the evaluator runs them one segment at a time.

Layout

InFlight/
  src/inflight/
    config.py       # datasets registry + paths + provider settings
    download.py     # fetch dataset from HuggingFace (parquet or hf-datasets)
    extract.py      # -> per-segment 16 kHz wav files + manifest.jsonl
    normalize.py    # ATC text normalization (v2 = ATC-aware, v1 = legacy)
    metrics.py      # WER / CER via jiwer
    evaluate.py     # orchestration, caching, per-dataset CSV/JSON output
    context.py      # multimodal manifest: surveillance callsigns, role, gold NER
    callsign.py     # callsign matching + oracle recall of the surveillance list
    report.py       # roll results up into docs/DATASETS.md
    cli.py          # `inflight` command
    providers/
      wav2vec2_provider.py   # local HF wav2vec2 CTC
      deepgram_provider.py
      assemblyai_provider.py
  docs/
    DATASETS.md            # auto-generated: datasets + stats + performance
    RESEARCH_PLAN.md       # where this is going (multimodal: sensors <-> transcription)
    analysis/<dataset>.md  # hand-maintained narrative analysis
  results/<dataset>/       # eval outputs (gitignored)
../data/<dataset>/         # extracted audio + manifest (gitignored)
  <dataset>/manifest.jsonl          # audio + transcript
  <dataset>/manifest_context.jsonl  # + surveillance/role/NER  (inflight context)
  <dataset>/atco2_official/         # the official ATCO2 release (downloaded)