Multimodal ToolMessage
I'm trying to run a multimodal toolcalling agent, but when trying to add a base64 encoded image to a ToolMessage vLLM throws an error:
(partial stack trace)
ERROR 05-21 13:22:25 [chat_utils.py:1256] An error occurred in mistral_common while applying chat template
ERROR 05-21 13:22:25 [chat_utils.py:1256] Traceback (most recent call last):
ERROR 05-21 13:22:25 [chat_utils.py:1256] File "/home/admin/app/venv/lib/python3.12/site-packages/vllm/entrypoints/chat_utils.py", line 1238, in apply_mistral_chat_template
ERROR 05-21 13:22:25 [chat_utils.py:1256] return tokenizer.apply_chat_template(
ERROR 05-21 13:22:25 [chat_utils.py:1256] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
ERROR 05-21 13:22:25 [chat_utils.py:1256] File "/home/admin/app/venv/lib/python3.12/site-packages/vllm/transformers_utils/tokenizers/mistral.py", line 378, in apply_chat_template
ERROR 05-21 13:22:25 [chat_utils.py:1256] request = make_mistral_chat_completion_request(messages, tools)
It works fine within a UserMessage. Anyone else having similar problems or maybe a solution?