smoldocling pre tokenizer

#48
by anakin004 - opened

I am making support for this model on llama cpp and noticed that it has a pre tokenizer: name: tokenizer.ggml.pre, value: smoldocling, is there more information about this tokenizer I could get? Thanks!

Sign up or log in to comment