FlashVSR: From 17.2 FPS to 57.3 FPS at 768x1408

[hamuzhan]

Alright, I know this repo looks abandoned. I got obsessed with this model anyway, and honestly, finding performance wins has been a ridiculous dopamine loop. It is a little addictive. So I spent the last few days back in the code with one goal: squeeze Hopper until it has nothing left to give. Same model. Same quality target. Much higher speed. This time FlashVSR should finally earn its name. Here are the updates.

Hopper Tiny Inference Acceleration

Before, Then, Now

Point in time	GH200 throughput at 768x1408	What changed
Before	17.2 FPS	Stock FlashVSR path with cuDNN Conv3D and the bundled HMMA sparse-attention kernel
Then	38.585 FPS	First complete Hopper stack with tensor-core LQ Conv3D, channels-last decoder, WGMMA sparse attention, cache, and layout work
Now	57.256 FPS	Current recommended stack with streamed decode, decoder kernel work, DiT row fusion, threshold reuse, split-K MemBlocks, and strict route validation

That is 17.2 -> 38.585 -> 57.256 FPS on the same GH200 and 768x1408 target. The current result is a 3.33x increase over the original stock number and a further +48.4% over the first complete Hopper stack.

The 17.2 FPS number is the original GH200 denoise benchmark recorded when this work started. The 38.585 and 57.256 values use the current F=81 output-FPS harness. They describe the same model, GPU, and resolution, but the current harness is more complete and reproducible. The strict same-harness comparison for the latest work is 55.910 -> 57.256 FPS.

What Was Actually Slow

A big Hopper GPU does not automatically make FlashVSR fast. I profiled the real streaming workload instead of trusting isolated FLOP counts and found several very specific problems.

Problem	What I observed	Why it mattered
LQ Conv3D	cuDNN selected a path around 28 TFLOP/s for the projector shapes	The projector spent a large part of denoise time while barely using Hopper tensor cores
Sparse attention	The bundled path used Ampere-style HMMA behavior on Hopper	Attention remained the largest single compute block
Attention glue	Layout conversions, KV concatenation, pooling, mask selection, and RoPE moved data without advancing model work	Small costs repeated across 30 DiT blocks and every chunk
Decoder	Decoding was serialized, paid layout churn, copied recurrent state, and repeatedly materialized concat buffers	Decoder work blocked end-to-end throughput after denoise improved
Measurement	Fast paths could silently fall back and overlap could distort naive per-chunk timing	Optimizing an inactive route would create false wins

I added NVTX coverage, strict route counters, Nsight Systems and Nsight Compute harnesses, input provenance, per-chunk timing, tail latency, memory reporting, and fallback reporting before treating any optimization as real.

How I Fixed It

Put the LQ projector on Hopper tensor cores

The LQ projector Conv3D shapes were a poor cuDNN match. I reformulated the core operation as im2col plus BF16 GEMM, added H-axis tiling to bound transient memory, used persistent causal-pad buffers, and added an exact Triton im2col packer. This was the first major unlock.

Rebuild sparse attention for Hopper

The model depends on locality-constrained sparse attention, so dense attention was not an acceptable default replacement. I added Hopper WGMMA sparse attention, then a warp-specialized producer and consumer path with TMA loads, fused CSR construction, FFMA-form softmax math, strided IO, incremental pooled-K caching, a coalesced RoPE kernel, and zero-copy raster V handling.

Each attention route keeps the sparse-mask contract or is explicitly quality-gated. FP8 attention was investigated and rejected because it fell far below the 49 dB quality bar.

Stop paying decoder overhead every frame

The decoder now runs channels-last, overlaps chunk decode with denoise on a side stream, rotates recurrent state pointers instead of copying tensors, writes decoded chunks directly to final output, uses Triton pointwise and layout kernels, moves TGrow work to low resolution, and uses cuDNN fused Conv, Bias, Add, and ReLU paths when quality permits it.

The split-K MemBlock path removes the recurrent cat materialization. It splits the first convolution into current and past weight halves, then recombines both contributions through cuDNN.

Remove the remaining control-plane tax

The final GH200 set adds a Triton row-wise DiT LayerNorm to AdaLN fusion, a gated-residual kernel, and a steady mask-threshold cache. These are smaller wins than the original attention and decoder work, but they are measured, routed, and quality-gated like the rest.

Measured Results

768x1408, F=81, 77 output frames

xychart-beta
    title "GH200 throughput at 768x1408, F=81"
    x-axis ["Stock GH200", "First Hopper stack", "Current Hopper stack", "Goal"]
    y-axis "FPS" 0 --> 60
    bar [17.2, 38.585, 57.256, 60]

Loading

Configuration	FPS	Delta	Evidence
Stock GH200 path	17.200	baseline	Original GH200 denoise benchmark
First complete Hopper stack	38.585	+124.3%	Three fresh-process runs, median
Previous recommended preset	55.910	+225.1%	Three fresh-process runs, median
Updated recommended preset	57.256	+232.9% from stock, +48.4% from first Hopper stack, +2.41% from previous	57.256 / 57.213 / 57.358 FPS
Final strict smoke run	57.435	n/a	All requested fast paths executed, no fallback

1536x2560

xychart-beta
    title "GH200 throughput at 1536x2560"
    x-axis ["Initial stack", "Previous recommended", "Updated recommended"]
    y-axis "FPS" 0 --> 16
    bar [11.01, 15.07, 15.415]

Loading

Workload	FPS	Peak memory	Quality
1536x2560, F=41	15.415	46.46 GiB	Strict route pass
1536x2560, F=29	n/a	n/a	49.95 dB versus preceding preset

Quality And Correctness

Area	Result
Decoder cumulative stack	Bit-exact at F=29 and F=89
DiT row fusion, threshold cache, split-K combined	49.81 dB at F=29, 49.59 dB at F=89
1536x2560 quality spot	49.95 dB at F=29
Split-K isolated decoder tests	68.86 to 68.99 dB across production shapes
Strict route enforcement	Expected routes executed with no fallback errors

The new GH200 preset is intentionally explicit about this. The flags are opt-in, unsupported Hopper paths retain their existing fallback behavior, and the quality-gated preset is not described as lossless.

Recommended GH200 Controls

Control	What it does	Constraint
FLASHVSR_CONV3D_BACKEND=gemm	Uses the im2col plus tensor-core projector path	Hopper only
FLASHVSR_ATTN_BACKEND=triton2	Uses warp-specialized block-sparse attention	Hopper only, quality-gated against the reference attention implementation
FLASHVSR_DECODER_OVERLAP=1	Decodes chunks on a side stream while denoise continues	Bit-exact
FLASHVSR_TCDECODER_CUDNN_FUSED=1	Fuses decoder Conv, Bias, Add, and ReLU operations	Quality-gated
FLASHVSR_DIT_ROW_FUSION=1	Fuses affine-free LayerNorm, AdaLN, and gated residual work	Quality-gated
FLASHVSR_MASKGEN_THRESHOLD_CACHE=1	Reuses the previous steady-chunk threshold per DiT block	Quality-gated
FLASHVSR_TCDECODER_SPLITK_CONV=1	Removes recurrent decoder concat through split-weight convolution	Requires TCDECODER_CUDNN_FUSED=1, quality-gated

The full preset lives in examples/WanVSR/run_flashvsr_v1.1_tiny_gh200.sh.

Things I Tried And Rejected

Experiment	Result	Decision
1980 MHz graphics-clock lock	56.028 FPS versus 56.085 FPS fresh baseline	Rejected
Direct implicit Conv3D contraction	Improved isolated conv2, but dropped end-to-end throughput to 56.741 FPS	Rejected
Larger KV arena spare	Removed compactions, but did not repeat its probe win and added 8.7 GiB	Rejected
FP8 GEMM	Faster in isolation, but below the locked 49 dB quality gate	Experimental only
FP8 attention	34.96 dB optimistic probe	Rejected before kernel work
Aggressive attention overlap variants	Limited by softmax and WGMMA dependency behavior	Not promoted

Reproducibility And Profiling

• NVTX ranges cover chunk scheduling, DiT blocks, sparse attention, mask generation, LQ projection, decoder work, and color correction.
• run_pipe_target.py records input provenance, per-chunk timing, tail latency, peak memory, active backend routes, and fallback errors.
• FLASHVSR_REQUIRE_FASTPATHS=1 makes a benchmark fail if a requested route silently falls back.
• bench_phase7.sh runs the final GH200 stack and automatically selects a valid steady window for short clips.
• PHASE_BENCH_LOG.md contains the complete measurement ledger, quality probes, rejected experiments, and resolution spot checks.

Verification

venv/bin/python examples/WanVSR/test_phase5_lossless.py phase6

FLASHVSR_TEST_MASKGEN_THRESHOLD_CACHE=1 \
FLASHVSR_TEST_SPLITK_CONV=1 \
venv/bin/python examples/WanVSR/test_dit_row_fusion.py 89

FLASHVSR_REQUIRE_FASTPATHS=1 \
examples/WanVSR/profiling/bench_phase7.sh hopper-pr 3

FLASHVSR_PROF_W=1536 FLASHVSR_PROF_H=2560 \
FLASHVSR_PROF_FRAMES=41 FLASHVSR_BENCH_STEADY=2:3 \
FLASHVSR_REQUIRE_FASTPATHS=1 \
examples/WanVSR/profiling/bench_phase7.sh hopper-pr-1536 1

Commit Organization

Commit subjects are grouped by responsibility using perf, feat, docs, and test prefixes with scopes such as attention, decoder, rope, kv, lq, mask, and profiling.

[napervays]

Dude you’ve really done an amazing job

[FurkanGozukara]

amazing would any of these work on other gpus too? like rtx 3000 4000 series? 5000 series?