bekan/testkaa-tokenizer

How to use

You can load this tokenizer directly using the transformers library in Python:

```python from transformers import AutoTokenizer

tokenizer = AutoTokenizer.from_pretrained("bekan/testkaa-tokenizer")

text = "C++ programmalastırıw tili." tokens = tokenizer.tokenize(text) print(tokens) ```