Skip to content

Pull requests: huggingface/tokenizers

New pull request New
113 Open 895 Closed
113 Open 895 Closed
Author
Filter by author
Loading
Label
Filter by label
Loading
Use alt + click/return to exclude labels
or ⇧ + click/return for logical OR
Projects
Filter by project
Loading
Milestones
Filter by milestone
Loading
Reviews
Filter by reviews
No reviews Review required Approved review Changes requested
Assignee
Filter by who’s assigned
Assigned to nobody Loading
Sort
Sort by
Newest Oldest Most commented Least commented Recently updated Least recently updated Best match
Most reactions
👍 👎 😄 🎉 😕 ❤️ 🚀 👀

Pull requests list

WordPiece via MPHF + bounded longest-match (PTRHash stream experiment)
#2179 opened Jul 9, 2026 by ArthurZucker Collaborator • Draft
3 of 5 tasks
1
decode: skip added-vocab check via id-range short-circuit
#2178 opened Jul 9, 2026 by sebpop Contributor Loading…
wip: more complete bench
#2177 opened Jul 9, 2026 by SBrandeis Contributor • Draft
1
experiment: icu_normalizer as a dependency
#2176 opened Jul 9, 2026 by SBrandeis Contributor • Draft
1
no alloc model, follow-ups
#2175 opened Jul 9, 2026 by SBrandeis Contributor • Draft
feat: T5 support (ai-generated)
#2174 opened Jul 9, 2026 by SBrandeis Contributor • Draft
1
chore(deps-dev): bump ws from 8.18.2 to 8.21.0 in /tokenizers/examples/unstable_wasm/www dependencies Pull requests that update a dependency file javascript Pull requests that update Javascript code
#2172 opened Jul 9, 2026 by dependabot Bot Loading…
1
dirty: thread-cache
#2170 opened Jul 8, 2026 by SBrandeis Contributor • Draft
1
Improve single thread performance of de-tokenization
#2166 opened Jul 8, 2026 by sebpop Contributor Loading…
8
Clarifications for contributing guide
#2162 opened Jul 7, 2026 by connor-makowski Loading…
2
Remove unreachable convert_offsets empty 0..0 branch
#2159 opened Jul 6, 2026 by yaohwang Loading…
3 tasks done
Make doc builds faster
#2158 opened Jul 6, 2026 by mishig25 Contributor Loading…
1
Do not panic when deserializing an invalid Precompiled normalizer
#2156 opened Jul 5, 2026 by v-code01 Loading…
2
Reject a zero length in the FixedLength pre-tokenizer instead of panicking
#2155 opened Jul 5, 2026 by v-code01 Loading…
1
Do not panic in the Strip decoder on crafted config or short tokens
#2154 opened Jul 5, 2026 by v-code01 Loading…
1
Do not panic in BPE builder when a merge exceeds the longest vocab token
#2153 opened Jul 5, 2026 by v-code01 Loading…
1
Validate TemplateProcessing on deserialization (fix panic on crafted tokenizer.json)
#2152 opened Jul 5, 2026 by v-code01 Loading…
1
Do not panic in Encoding::truncate when stride >= max_len
#2151 opened Jul 5, 2026 by v-code01 Loading…
2
Add a fast_split module
#2150 opened Jul 5, 2026 by ArthurZucker Collaborator Loading…
1
Map prepended text to a zero-width span to avoid overlapping offsets (#1775)
#2149 opened Jul 5, 2026 by v-code01 Loading…
2
experiment: reduce normalizer alloc further
#2148 opened Jul 3, 2026 by SBrandeis Contributor • Draft
1
chore(deps): bump js-yaml from 3.14.2 to 3.15.0 in /bindings/node dependencies Pull requests that update a dependency file javascript Pull requests that update Javascript code
#2131 opened Jun 30, 2026 by dependabot Bot Loading…
1
docs: add docstrings to Tokenizer::from_file, from_bytes, and from_petrained
#2124 opened Jun 23, 2026 by BBloggsbott Loading…
feat: train-encode split
#2119 opened Jun 18, 2026 by McPatate Member Loading…
4
chore(deps-dev): bump webpack-dev-server from 5.2.1 to 5.2.5 in /tokenizers/examples/unstable_wasm/www dependencies Pull requests that update a dependency file javascript Pull requests that update Javascript code
#2118 opened Jun 18, 2026 by dependabot Bot Loading…
1
ProTip! Find all pull requests that aren't related to any open issues with -linked:issue.