# tiktoken - OpenAI의 Tokenizer

> Clean Markdown view of GeekNews topic #8091. Use the original source for factual precision when an external source URL is present.

## Metadata

- GeekNews HTML: [https://news.hada.io/topic?id=8091](https://news.hada.io/topic?id=8091)
- GeekNews Markdown: [https://news.hada.io/topic/8091.md](https://news.hada.io/topic/8091.md)
- Type: news
- Author: [xguru](https://news.hada.io/@xguru)
- Published: 2022-12-23T10:06:02+09:00
- Updated: 2022-12-23T10:06:02+09:00
- Original source: [github.com/openai](https://github.com/openai/tiktoken)
- Points: 4
- Comments: 0

## Topic Body

- OpenAI의 모델들과 사용하기 위한 빠른 BPE (Byte pair encoding) Tokenizer  
- 기존 Tokenizer(Huggingface)에 비해 3~6배 빠름   
- PyPI 로 설치 가능   
- Python + Rust 오픈소스

## Comments



_No public comments on this page._