Add inference/generate pipeline to dynamic training

[dailykim149656-source]

Summary

Adds a standalone text-generation (inference) tool to the dynamic training pipeline, making it possible to run forward-only generation from a trained checkpoint on the ANE.

Motivation

The dynamic pipeline can train (train.m) and benchmark, but there was no way to actually generate text from a trained checkpoint. For a research project exploring ANE training, the ability to inspect model output — does the network actually learn anything? what does the text look like? — is the most direct feedback loop. This fills that gap with minimal new code: one new file, one Makefile target, one README section.

What it does

generate.m is a forward-only inference pipeline:

• Loads checkpoint weights — reads the checkpoint, fseek-skips past Adam optimizer state to load only the model weights
• Compiles only 3 forward kernels (sdpaFwd, woFwd, ffnFused) instead of the 10 kernels the trainer needs — no backward, no optimizer, no gradient machinery
• Sliding-window forward pass — advances the context window token-by-token, sampling the next token from logits
• Sampling modes — greedy decoding (--temp 0) and temperature + top-k sampling
• Detokenization — loads a llama2.c format tokenizer.bin from the working directory; if absent, prints raw token IDs as a fallback
• Prompt context — seeds generation from the first SEQ tokens of the data file

Usage

cd training_dynamic
make MODEL=stories110m generate    # or MODEL=qwen3_06b

# Greedy decoding (deterministic)
./generate --ckpt ane_stories110M_dyn_ckpt.bin --tokens 100

# Temperature sampling with top-k
./generate --ckpt ane_stories110M_dyn_ckpt.bin --tokens 200 --temp 0.8 --topk 40

CLI flags: --ckpt, --tokens, --temp, --topk, --data, --seed, --help.

Verification

• Builds clean (no warnings) for both stories110m and qwen3_06b model configs
• Runtime verified end-to-end: trained Stories110M for 200 steps (loss 6.39), then ran generate successfully — checkpoint load, kernel compilation (~800ms warm), forward pass, and token generation all working
• ~44-56ms/token on M2
• --help and missing-argument error handling confirmed

Files changed

File	Change
training/training_dynamic/generate.m	New — forward-only inference pipeline (~410 lines)
training/training_dynamic/Makefile	Add generate target, update clean
training/README.md	Document generate usage, add to Files table, renumber sections

Notes

• No changes to existing training code (train.m, cpu_ops.h, etc. are untouched)
• Follows the existing dynamic pipeline conventions: same config.h/io.h/cpu_ops.h/mil_dynamic.h headers, same MODEL-driven Makefile pattern, same checkpoint format
• The generate binary is added to .gitignore-style clean targets; the source file generate.m is committed

[chatgpt-codex-connector]

You have reached your Codex usage limits for code reviews. You can see your limits in the Codex usage dashboard.
To continue using code reviews, you can upgrade your account or add credits to your account and enable them for code reviews in your settings.

[chatgpt-codex-connector]

You have reached your Codex usage limits for code reviews. You can see your limits in the Codex usage dashboard.
To continue using code reviews, you can upgrade your account or add credits to your account and enable them for code reviews in your settings.