Download

Skill UI

Browse and discover 15558+ curated skills

All Development Artificial Intelligence Design & Creative Product & Business Data Science Marketing Soft Skills Productivity Engineering Languages

Search Unigram , found 2 results

Default Newest Most Downloaded

HuggingFace Tokenizer Suite

huggingface-tokenizers

Orchestra-Research/AI-Research-SKILLs

High-performance Rust-backed tokenization toolkit with Python bindings, offering BPE/WordPiece/Unigram, custom vocabulary training, alignment tracking, and production-ready integrations for NLP pipelines.

SentencePiece Tokenizer Library

Orchestra-Research/AI-Research-SKILLs

SentencePiece provides a language-independent tokenizer that trains BPE or Unigram vocabularies directly from raw Unicode text, ideal for multilingual, CJK, or reproducible NLP pipelines while staying lightweight and fast.

1

Language