[RFC]: Multi-modality Support Refactoring

Multi-modality support was brought to vLLM recently, much thanks to https://github.com/vllm-project/vllm/pull/3042 from @xwjiang2010. Since then we have seen an increasing amount of interest in such models (from the number of pull requests and issues related). However, there are a few issues we should address with the current design before we bring in more features around multi-modality.

VisionLanguageConfig and MultiModalData

Currently the multimodal input can be either pixel_values or image_feaures for simplicity. While this works well with llava 1.5 where pixel_values are the only output from its ClipImageProcessor, this does not work well when it comes to supporting models with more complicated preprocessing to return multiple outputs.(e.g, llava 1.6, fuyu, etc). Developers could add additional preprocessing inside model implementation as a workaround, but this will be unmaintainable over time.

The overhead of requiring image_feature_size, image_token_id and image_input_shape is pushed to the user when these can/should be inferred from the model & processor config and not required at the inference time.

The current design assumes multi-modal inputs are already processed to be consumed by the model executable, but vLLM does not have a processor util. This blocks the vision model support on the OpenAI API server for end-to-end inference.

The current prompt format "<Image>" * 576 + prompt makes the underlying implementation easier (especially when it comes to profiling), but complicates the user experience compared to huggingface format "<Image>\n" + prompt and that has caused some confusion on what's needed to make multi-model work on vLLM.

Proposal
Most items in the above issues have been discussed and addressed in the original Llava1.5 PR as well as https://github.com/vllm-project/vllm/pull/3978. We propose a few high-level design decisions for the refactoring and welcome any feedback!

Adding a processor util - We can leverage out-of-box AutoProcessor from transformers the same way we have been doing with tokenizer as an attribute of LLMEngine (e.g., self.multi_modal_processor = AutoProcessor(model)). This allows us to support end-to-end inference with the API server as well as the LLM object.

Frontend input format:  Because of 1, we can keep the same format as HuggingFace since that's how users usually discover new models and it makes end-to-end integration test easier. Preprocessing should be hidden away from the interface and user. For example, this preprocessing step can be done inside LLMEngine.add_request() around the same place as https://github.com/vllm-project/vllm/blob/a134ef6f5e6c24d3cd459c63557e5db276db25b2/vllm/engine/llm_engine.py#L385-L391
Here's a pesudocode

if multi_modal_input is None:
   prompt_token_ids = self.encode_request(
       request_id=request_id,
       prompt=prompt,
       prompt_token_ids=prompt_token_ids,
       lora_request=lora_request)
else:
   # preprocessed_inputs is a dictionary of key(str)-value(tensor)
   # as output of self.multi_modal_processor
   preprocessed_inputs = self.preprocess_request(
       request_id=request_id,
       prompt=prompt,
       prompt_token_ids=prompt_token_ids,
       lora_request=lora_request,
       multi_modal_input=images)
   prompt_token_ids = preprocessed_inputs.pop("input_ids")
   multi_modal_data = MultiModalData(data=preprocessed_inputs)
...

and thus at LLM level, only image tensors will be required.

Refactor MultiModalData: Now this object simply holds the multi-modal data dictionary that we need for the model_executable. At inference time, data is unpacked in the forward pass - this approach is similar to transformer implementation of multi-modal models.
Refactor VisionLanguageConfig: This config is a lot simpler now. One caveat is that sometimes when the image features can be dynamic, users may specify an optional max_feature_size to help engine run the profiling for the worst-case scenario as well as to potentially abort certain requests.
Regarding the original image_feature as input type design: IMO LlaVA is a special case among multi-modal models since its vision encoder is detached from the language model and can be initialized separately, but in this case, one could argue that for the MultiModalProjector as well, and perhaps passing image_feature (outputs of CLIP) is a design decision not generalizable to all other models. Instead, passing multi-modal embeddings (outputs of CLIP -> Projector) at inference time is more flexible and should work nicely with other models. (One followup question is, does it make sense to actually define a separate Llava-no-clip module, since this is so specific to llava, to make our life easier?)

With the above changes, as an end-user, ideally you then should be able to do something like the following
from PIL import Image
from vllm import LLM
from vllm.config import VisionLanguageConfig

model_id = "llava-hf/llava-v1.6-mistral-7b-hf"
llm = LLM(model=model_id, multi_modal_input_type=VisionLanguageConfig.IMAGE_INPUT_TYPE.IMAGE) # This can also be EMBEDDINGS

prompt = "<image>\nUSER: What's the content of the image?\nASSISTANT:"

url = "https://www.ilankelman.org/stopsigns/australia.jpg"
image = Image.open(requests.get(url, stream=True).raw)

llm.generate(prompt, ..., multi_modal_input=image)

Under the hood, the pipeline is
prompt, image
-> prompt_token_ids, MultiModalData(data=preprocessed_inputs) # through preprocess within engine.add_request()
-> prompt_token_ids, pixel_values, image_sizes  # though unpacking in implementation of model's `forward`.

I will follow up with a series of PR for refactoring but please leave any feedback since this is a pretty significant interface change.
Report of performance regression
No response
Misc discussion on performance
No response
Your current environment (if you think it is necessary)
The output of `python collect_env.py`

Re: Issues
I fully agree with the issues which you have pointed out. I would like to add that the current prompt format is hardly extensible for multi-image input if we plan to pursue that further down the line. In #3978, I have proposed some ways of tackling the issue. I have thought about them more and have decided that they alone cannot provide the required flexibility, as explained below:

If there are only a small number of standard methods, we can provide a config option to choose which method to apply. I have added the image_openai attribute to VisionLanguageConfig to facilitate this.

I am not confident that this assumption would hold for very long, given the fast-changing pace of the field.

A more flexible option would be to pass the image(s) to the chat template (e.g. by setting the images attribute alongside role and content). This transfers the burden of implementation to the maintainers of the model on HuggingFace, making it more likely that vLLM users have to implement their own template. I have created ConversationMessage class to represent the dictionary for each message.

This is only suitable for cases where we only have to pass a single <image> token. The requirement of duplicating image tokens according to feature size should not be a concern of the chat template.
Re: Proposals
Here are my thoughts on each proposal:
1. Adding a processor util
I think that we should move this responsibility outside of the Engine class. This is because multi-modal input isn't necessarily limited to image data, so we should expect more data types to be added in the future. To avoid having to modify the core Engine logic each time, we can wrap the data with processor objects (with a common interface to process the data) before passing them into the Engine. This way, we can easily add new data types by simply defining a new processor class. For your reference, I have implemented this pattern in #4197.
2. Frontend input format
My comments on this are similar to 1. However, #4197 only refactors MultiModalData to define data processing logic. To avoid excessive duplication of the logic of encode_request, we should find a way to let MultiModalData control only parts of the process. Also, in my idea of MultiModalData, the processing logic should remain independent of the model architecture. I guess this is where Proposal 3 comes in.
3. Refactor MultiModalData.
I have refactored this class in #4197 according to this description, and it works well enough to support the image_size parameter of LLaVA-NeXT as shown in #4199.
4. Refactor VisionLanguageConfig
Currently in #4197, MultiModalData has to accept ModelConfig and VisionLanguageConfig separately. Perhaps we can make VisionLanguageConfig an attribute of ModelConfig so we do not have to pass in multiple parameters. Using this approach, we only have to add more attributes to ModelConfig instead of having to pass more config objects around in order to support additional multi-modal data types.
5. Regarding the original image_feature as input type design
TODO

@ywang96   Thanks for driving the integration of more MM models into VLLM. :heart_eyes:
It seems that there is no plan to refactor vision encoder  (todo in llava).
In my view, we should prioritize this, with performance being my main consideration.
By refactoring the vision encoder, we can establish an integration standard for MM models, similar to the our LLM models integration . This will not only ensure inference performance but also provide integration guidelines for the community
if I misunderstand, please correct me, thanks for your work again

Generally, I agreed with @DarkLight1337's opinion about integrating processing logics in MultiModalData to prevent modifying core code frequently. However, I think it's difficult to keep the processing logics fully independent from the model architecture.
For example, FuyuProcessor and Idefics2Processor will pad input_ids with image_feature_size during preprocess, while LlavaProcessor won't (I guess this is also why "<image>" * image_feature_size + prompt is used for llava). This means that we need to pad input_ids for llava manually.

cc @robertgshaw2-neuralmagic @mgoin (since NM's planned to work on whisper)
Thank you all for the feedback so far! I plan to address feedback altogether after meeting up with the core devs as well as getting perspectives from other community members who are working/plan to work on multi-modal models.
Some quick ones that I can answer now:

It seems that there is no plan to refactor vision encoder (todo in llava).

@jeejeelee This will need to be done regardless since it's inside the model implementation, this RFC is more around how we want to support multi-modal models in general, and thus focuses on the interface and component pattern.

However, I think it's difficult to keep the processing logics fully independent from the model architecture.

@DarkLight1337 @Isotr0py If this is just about where the processor should live, I'm indifferent between having it live inside LLMEngine or not. The tricky part IMO is that then we need to rework on the interface of LLMEngine to consume outputs of AutoProcessor as is.

I was under the impression that this is used to simplify the generation of the attention masks.
@DarkLight1337 That's correct too, but I'm worried that as the model gets more and more complicated, this approach might not be generalizable.

Since LLMEngine has support for an output processor interface, e.g. SequenceGroupOutputProcessor. Would it be reasonable within engine, to also add an InputProcessor interface?
This way engine can check for existing of an input processor, but the implementation in this case for llava's single image processing can live outside of engine. It's implementation could be as suggested, based on AutoProcessor.
As for supporting processing of something apart of an Image tag or varying formats - engine could only have a generic input processor executor, within the model executor's code, it would be up to the model implementation to define an input processor and pass it on to engine.

Generally, I agreed with @DarkLight1337's opinion about moving processing logics out from Engine to prevent modifying core code frequently. However, I think it's difficult to keep the processing logics fully independent from the model architecture.
For example, FuyuProcessor and Idefics2Processor will pad input_ids with image_feature_size during preprocess, while LlavaProcessor won't (I guess this is also why "<image>" * image_feature_size + prompt is used for llava). This means that we need to pad input_ids for llava manually. (maybe there is a better way to handle this? 🤔)

Perhaps we could follow a registry pattern and have each model  separately register how to preprocess the inputs? If the model does not do so, then the default implementation would be to pass the data to HuggingFace processors.

@Isotr0py Perhaps we could follow a registry pattern and have each model separately register how to preprocess the inputs? If the model does not do so, then the default implementation would be to pass the data to HuggingFace processors.

Yes, I agree that we can use processor registry to solve this. And it seems that transformers_utils/configs could be a good reference for this.

@Isotr0py Perhaps we could follow a registry pattern and have each model separately register how to preprocess the inputs? If the model does not do so, then the default implementation would be to pass the data to HuggingFace processors.

Yes, I agree that we can use processor registry to solve this. And it seems that transformers_utils/configs could be a good reference for this.

I have added an implementation of the processor registry to #4197.

2. Frontend input format
My comments on this are similar for Proposal 1. However, #4197 only refactors MultiModalData to define data processing logic. To avoid excessive duplication of the logic of encode_request, we should find a way to let MultiModalData control only parts of the process. Also, in my idea of MultiModalData, the processing logic should remain independent of the model architecture. I guess this is where Proposal 3 comes in: HuggingFace processors should output dictionaries with keys that match the parameter names of model.forward().

To solve the prompt format problem for LLaVA, I think we have to also deal with generating the attention masks in the processing framework. That would mean abstracting some of the logic of ModelRunner._prepare_prompt.

Just a heads up that #4228 will introduce another vision language model to vLLM, so our discussion should take that into account as well.

I discussed with @zhuohan123 offline about this - in particular regarding this comment

To avoid having to modify the core Engine logic each time, we can wrap the data with processor objects (with a common interface to process the data) before passing them into the Engine.

If vLLM's going to use out-of-box AutoProcessor anyways, then it's logical to make it an attribute of the engine (similar to what we did with tokenizer). As of now for the sake of simplicity, we could add something like self.processor = AutoProcessor(model_id) to this section if the model is an MM model.
https://github.com/vllm-project/vllm/blob/15436806912d7ad9371c8bcf6a46857590c107d2/vllm/engine/llm_engine.py#L136-L139
then at inference time, depending on if the request has multi-modal data or not, we process with it with either self.tokenizer or self.processor.
(IMO eventually, there shouldn't really be a separation between how we preprocess text data and multi-modal data as they should all go through one InputProcessor class, but that is probably a bigger engineering refactoring that we can leave for later.)
We can also add an additional parameter on the engine level to indicate that we're feeding the engine a already processed dictionary of tensors, so the preprocessing step with self.processor will be skipped. (Very similar to prompt vs prompt_token_ids)
@DarkLight1337 @Isotr0py WDYT?

I discussed with @zhuohan123 offline about this - in particular regarding this comment

To avoid having to modify the core Engine logic each time, we can wrap the data with processor objects (with a common interface to process the data) before passing them into the Engine.

If vLLM's going to use out-of-box AutoProcessor (which includes tokenizer) anyways, then it's logical to make it an attribute of the engine (similar to what we did with tokenizer). As of now for the sake of simplicity, we could add something like self.processor = AutoProcessor(model_id) to this section if the model is an MM model.
https://github.com/vllm-project/vllm/blob/15436806912d7ad9371c8bcf6a46857590c107d2/vllm/engine/llm_engine.py#L136-L139
then at inference time, depending on if the request has multi-modal data or not, we process with it with either self.tokenizer or self.processor.
(IMO eventually, there really shouldn't be a separation between how we preprocess text data and multi-modal data as they should all go through one InputProcessor class, but that is probably a bigger engineering refactoring that we can leave for later.)
We can also add an additional parameter on the engine level to indicate that we're feeding the engine an already processed dictionary of tensors, so the preprocessing step with self.processor will be skipped. (Very similar to prompt vs prompt_token_ids)
@DarkLight1337 @Isotr0py WDYT? Do you see any issue with this design?

This is somewhat similar to #4166 where I load the processing logic using AutoProcessor instead of AutoTokenizer for testing the HuggingFace implementation.
I think one potential issue of this design is that the direct dependency on HuggingFace (which we have no control over) would complicate efforts to apply additional preprocessing specific to certain HuggingFace processors (e.g. to adapt to our interface).
Since @Isotr0py 's comment, I have refactored the code in #4197 into using a registry pattern to apply the preprocessor, so that MultiModalData class itself no longer has any preprocessing logic.

@DarkLight1337 Thanks for sharing the thoughts! @zhuohan123 and I actually discussed about the use of AutoProcessor.
I think the point is that today vLLM already relies on AutoTokenizer, and most of model implementations we have in vLLM today are based on the implementation of such models in transformers, so I don't really think having this dependency is an big issue. Using AutoProcessor also allows us to abstract away from image in particular so that the same interface will work for other modalities (e.g, whisper) as well.
The original design of the prompt interface isn't very clean, and is very specific to LlaVa-1.5. I would like to emphasize that not every MM model has a "vision tower + projector + LM" architecture, so IMO the input format should really be one of raw inputs (images), processed inputs (outputs of autoprocessor) or embeddings (prompt embeddings + MM embeddings).
I will also be working on a PR so we can cross review each other's work.

One thing to add is that we would like to keep vLLM's end-user API easy to use. Having AutoProcessor outside of vLLM requires the user to create and pick the correct Processor for the specific model they are using, which can be error-prone. So I lean towards having AutoProcessor in vLLM and an end user can directly feed in the raw image (e.g. like a jpg image) to vLLM.

@DarkLight1337 Thanks for sharing the thoughts! @zhuohan123 and I actually discussed about the use of AutoProcessor.
I think the point is that today vLLM already relies on AutoTokenizer, and most of model implementations we have in vLLM today are based on the implementation of such models in transformers, so I don't really think having this dependency is an big issue. Using AutoProcessor also allows us to abstract away from image in particular so that the same interface will work for other modalities (e.g, whisper) as well.
The original design of the prompt interface isn't very clean, and is very specific to LlaVa-1.5. I would like to emphasize that not every MM model has a "vision tower + projector + LM" architecture, so IMO the input format should really be one of raw inputs (images), processed inputs (outputs of autoprocessor) or embeddings (prompt embeddings + MM embeddings).
I will also be working on a PR so we can cross review each other's work.

In this case, we would have to refactor the computation of attention masks so that it can accept single <image> token for LLaVA, since that is what its HuggingFace processor expects. How can we integrate this into vLLM's computation of the attention masks?

Regarding #4228, I think there may be a situation that some MM models don't have a Processor implemented.

In this case, we would have to refactor the computation of attention masks so that it can accept single <image> token for LLaVA, since that is what its HuggingFace processor expects.

@DarkLight1337 IMO, there may be a solution that we can inherit and modify the LLaVA processor to handle inputs_ids padding etc, so that it can create the right attention masks from current attention masks computation codes.

Regarding #4228, I think there may be a situation that some MM models don't have a Processor implemented.

In this case, we would have to refactor the computation of attention masks so that it can accept single <image> token for LLaVA, since that is what its HuggingFace processor expects.

@DarkLight1337 IMO, there may be a solution that we can inherit and modify the LLaVA processor to handle num_features calculation and inputs_ids padding etc, so that it can create the right attention masks from current attention masks computation codes.

I like the idea of simply inheriting from the existing HuggingFace processor. How should we ensure that our implementation is loaded instead of the HuggingFace one?

Also, I think that we should still wrap the input prompt to better distinguish the kwargs to pass to the HF processor from the other arguments to LLM.generate. It is rather awkward right now that we have to pass a list of multi-modal data with length equal to the input prompts.

I have noticed when using distributed inference on LLaVA-NeXT (#4199), there is a bug where the image tokens are not sent to the workers, resulting in an error when trying to merge the vision embeddings. This doesn't happen with LLaVA-1.5 because the model can be loaded inside a single GPU.
(Side note: I spent quite a bit of time figuring out the issue because the bug does not occur when I execute the code in VSCode Debugger. Not sure why that is the case.)

How should we ensure that our implementation is loaded instead of the HuggingFace one?

I think we can refer to get_config() in transformers_utils/config.py, but searching registried processor firstly then AutoProcessor, so that the get_processor() could be:
def get_processor(model: str,
               trust_remote_code: bool,
               revision: Optional[str] = None,
               code_revision: Optional[str] = None) -> ProcessorMixin:
    if processor.model_type in _PROCESSOR_REGISTRY:
        processor_class = _PROCESSOR_REGISTRY[processor.model_type]
        processor = processor_class.from_pretrained(model,
                                              revision=revision,
                                              code_revision=code_revision)
    try:
        processor = AutoProcessor.from_pretrained(
            model,
            trust_remote_code=trust_remote_code,
            revision=revision,
            code_revision=code_revision)
    except ValueError as e:
        # do something else

Hi guys,
It seems the current prefix caching does not work on multi modal model like llava because the hash of block only takes previous token ids and the image patch token is always the same but the corresponding images might differ.
Do you have any ideas about:

How to make the kv block of multi modal request have correct hash id?
How to inherit the kv cache blocks with images? Image might take several blocks but it is impossible to only inherit part of it because the image encoder need to take a whole image.

Thanks!

For reference, here are a few other GH issues that are related to this topic:

#416
#2757
#3519

Hi folks, just stumbled across this issue. It's great to see the proposed steps here as these were all things I ran into when implementing support for our multimodal audio model - in particular, not requiring pre-padding of the input with the special token is a huge simplification when the input (audio in our case) can be of arbitrary size.
The one other thing that might make sense here would be a rename of VisionLanguageConfig to MultimodalLanguageConfig or something similar, rather than having to add separate AudioLanguageConfig and other types in the future throughout all the layers of vLLM. This should be fairly straightforward with your refactoring as I think the only thing remaining in VisionLanguageConfig that we will care about will be the special token id.

The current prompt format "<Image>" * 576 + prompt makes the underlying implementation easier (especially when it comes to profiling), but complicates the user experience compared to huggingface format "<Image>\n" + prompt and that has caused some confusion on what's needed to make multi-model work on vLLM.

While messing with the OpenAI GPT-4V API, I found that this is quite vulnerable to token injection where the user includes <image> tokens in the text prompt. This causes the model to crash due to mismatched shapes when filling in the image token positions with the embeddings.

For folks who came across this RFC, I have been working closely with @DarkLight1337 on several PRs:

[x] #4910
[ ] #4328
[ ] #4197
[ ] #4200 (will work on this once all 3 above get merged)

The goal is to support end-to-end GPT4V-compatible inference by the next major release/meetup.

On parallel, I have been also helping on #4937 , and hopefully can get it merged before the next major release too.

Hi forks,
I think when we want to refactor the code, we should not only consider the multi modal input, but also the multi modal output.

Hi forks, I think when we want to refactor the code, we should not only consider the multi modal input, but also the multi modal output.

I personally totally agree with this, but for now multi-modal output isn't yet in the scope of the project until a reasonable model support and interest in it.

Can vllm support direct input of inputs_emb now? If so, we can leverage vllm's inference capabilities with minimal changes to the model inference code. Moreover, since model architectures are diverse, having vllm support direct input of inputs_emb would greatly enhance its applicability. Otherwise, each new model would require redevelopment, which is time-consuming and labor-intensive.
In the generate method, can it support input of embeddings? How is the progress on this?

Can vllm support direct input of inputs_emb now? If so, we can leverage vllm's inference capabilities with minimal changes to the model inference code. Moreover, since model architectures are diverse, having vllm support direct input of inputs_emb would greatly enhance its applicability. Otherwise, each new model would require redevelopment, which is time-consuming and labor-intensive.
In the generate method, can it support input of embeddings? How is the progress on this?

I don't think this is even supported for regular LLMs yet.

@nukes, once the PRs mentioned by @ywang96 land, I'm going to send a PR to integrate our audio model (https://ultravox.ai) using the same OpenAI interface (ie using image_url). It doesn't handle output yet, but this will get us halfway there.

@nukes, once the PRs mentioned by @ywang96 land, I'm going to send a PR to integrate our audio model (https://ultravox.ai) using the same OpenAI interface (ie using image_url). It doesn't handle output yet, but this will get us halfway there.

Nice work, just starred your repo. and joined the discord channel!

#4197 has finally been merged!

The current prompt format "<Image>" * 576 + prompt makes the underlying implementation easier (especially when it comes to profiling), but complicates the user experience compared to huggingface format "<Image>\n" + prompt and that has caused some confusion on what's needed to make multi-model work on vLLM.

I'm working on a new series of PRs that will resolve this issue by giving developers a way to indicate for each model how to insert image tokens into the prompt:

[ ] #5214
[ ] Implement insertion of image tokens and add documentation
[ ] Remove unnecessary attributes from VisionLanguageConfig, such as input_token_id, image_input_shape, and image_feature_size, since these values can be inferred from the HuggingFace config. It is not very friendly to have the user copy the required values from HuggingFace repo.
[ ] Maybe remove image_input_type? It may not be applicable to vision-language models outside LLaVA.

@DarkLight1337 @ywang96 Thanks for the ongoing work on improving multimodality support, looking forward to using it! Are these changes going to be part of the next release?

@DarkLight1337 @ywang96 Thanks for the ongoing work on improving multimodality support, looking forward to using it! Are these changes going to be part of the next release?

@akshay-loci we're on track to merge GPT4V compatible inference (i.e, the user facing changes) mentioned here before the next release. The developer facing changes are still WIP, and I will summarize another roadmap after the release.
For model support, it's best effort for now and I'm trying to see if we can add one or two more before the release.

For people who are following this thread and multi-modal support in general, we have made an update in the issue at the top so please take a look!

Remove unnecessary attributes from VisionLanguageConfig, such as input_token_id, image_input_shape, and image_feature_size, since these values can be inferred from the HuggingFace config. It is not very friendly to have the user copy the required values from HuggingFace repo.

@DarkLight1337 QQ: how can image_feature_size be found from the HuggingFace repo? I can't seem to find a fixed image_feature_size from model config. Does this issue have a WIP PR?

how can image_feature_size be found from the HuggingFace repo?

@CatherineSue To clarify, for now we need image_feature_size because the scheduler need to know in advance how much space the image embeddings take in the final embeddings passed to the language model. This will be no longer needed once we add the prompt preprocessing logic into vLLM (e.g, "<image> what's in this image?" -> "<image>" * 576 + "what's in this image"?). This preprocessing logic can get very complicated especially when the final image embedding shape depends on the image size/resolution itself, and currently this is implemented in the model forward pass in transformers (thus won't work in vLLM).
We're currently still discussing the best way to design this preprocessing logic into vLLM to make it easier for both users and developers, but ideas are welcomed!

I'm a bit confused about running llava-hf/llava-v1.6-mistral-7b-hf in the latest version of the Docker image. The first issue I had was the config checker looking for several key/value pairs in the config.json under text_config during startup that appear to have been moved to vision_config, such as num_attention_heads. When I copied in the needed key/values from mistralai/Mistral-7B-Instruct-v0.2 config.json, I got further in the startup, but now it's looking for image_size, which I don't see as a parameter defined in the VLM docs page. I also don't see this parameter being passed anywhere in the examples, am I missing something?

@FennFlyer I was able to run the example below on the main branch
from vllm import LLM
from vllm.multimodal.image import ImagePixelData
from vllm import SamplingParams

llm = LLM(
    model="llava-hf/llava-v1.6-mistral-7b-hf",
    image_input_type="pixel_values",
    image_token_id=32000,
    image_input_shape="1,3,336,336",
    image_feature_size=1176,
)

prompt = "[INST] " + "<image>" * 1176 + "\nWhat is shown in this image? [/INST]"
url = "https://h2o-release.s3.amazonaws.com/h2ogpt/bigben.jpg"
image = Image.open(BytesIO(requests.get(url).content))
sampling_params = SamplingParams(temperature=0.8, top_p=0.95, max_tokens=100)

outputs = llm.generate({
    "prompt": prompt,
    "multi_modal_data": ImagePixelData(image),
}, sampling_params=sampling_params)

generated_text = ""
for o in outputs:
    generated_text += o.outputs[0].text

print(f"LLM output:{generated_text}")

Output:
LLM output: The image shows a nighttime scene of the iconic Big Ben clock tower, also known as the Elizabeth Tower, which is located at the north end of the Palace of Westminster in London, UK. The tower is illuminated and stands out against the dark sky. In the foreground, there is a blurred view of traffic on a busy city street, which suggests the photo was taken from a moving vehicle or in a crowded area where motion blur is present.

@ywang96 @DarkLight1337 Hi, not sure if this is the correct place to post about this but thank you for your hard work on multimodal support for vLLM! I'm pretty interested in contributing to the multimodal effort specifically, so I was wondering if you guys had any suggestions on good starter issues to contribute to. If this is not the best place to ask, would it be possible to direct me to the correct place?

Hey @FennFlyer - dynamic image input shape is currently not supported yet, so you might need to take a look at the example configuration mentioned in #4199. This does mean your input image will be reshaped to the configuration you specified, so the output could be different from huggingface implementation. This is a limitation we're aware of and we're currently working on this!

Just want to make sure I'm understanding correctly, hosting this model via Docker is not working right now? I would need to use a standalone script and instantiate an LLM object for a one-shot inference? Here's the relevant bits of my docker-compose if I'm incorrect about that:
command: ["--model", "${VLLM_IMAGE_MODEL_ID}", "--gpu-memory-utilization", "0.75", "--host", "<IP_ADDRESS>", "--root-path", "/vllm-server",       "--image-input-type", "pixel_values", "--image-token-id", "32000", "--image-input-shape", "1,3,336,336", "--image-feature-size", "576",       "--chat-template", "template_llava.jinja"]
Thanks for the assist, I'm currently maintaining an Ollama image just for LlaVa alongside our multiple vLLM instances, so being able to consolidate on one hosting image would be great!

Hey @FennFlyer - dynamic image input shape is currently not supported yet, so you might need to take a look at the example configuration mentioned in #4199. This does mean your input image will be reshaped to the configuration you specified, so the output could be different from huggingface implementation. This is a limitation we're aware of and we're currently working on this!

Just want to make sure I'm understanding correctly, hosting this model via Docker is not working right now? I would need to use a standalone script and instantiate an LLM object for a one-shot inference? Here's the relevant bits of my docker-compose if I'm incorrect about that:
command: ["--model", "${VLLM_IMAGE_MODEL_ID}", "--gpu-memory-utilization", "0.75", "--host", "<IP_ADDRESS>", "--root-path", "/vllm-server", "--image-input-type", "pixel_values", "--image-token-id", "32000", "--image-input-shape", "1,3,336,336", "--image-feature-size", "576", "--chat-template", "template_llava.jinja"]
Thanks for the assist, I'm currently maintaining an Ollama image just for LlaVa alongside our multiple vLLM instances, so being able to consolidate on one (way more performant) hosting image would be great!

Can you provide more details about the error you're facing? By the way, check that the path to the chat template is correct.

Sure! Here's the error I'm getting. I get all the usual startup output, then hit this error. It definitely looks like I'm passing a starting parameter incorrectly, but I'm having trouble tracing back where that image size that it's checking in llava_next.py is coming from:
vllm-llava-server  | [rank0]: Traceback (most recent call last):
vllm-llava-server  | [rank0]:   File "/usr/lib/python3.10/runpy.py", line 196, in _run_module_as_main
vllm-llava-server  | [rank0]:     return _run_code(code, main_globals, None,
vllm-llava-server  | [rank0]:   File "/usr/lib/python3.10/runpy.py", line 86, in _run_code
vllm-llava-server  | [rank0]:     exec(code, run_globals)
vllm-llava-server  | [rank0]:   File "/usr/local/lib/python3.10/dist-packages/vllm/entrypoints/openai/api_server.py", line 196, in <module>
vllm-llava-server  | [rank0]:     engine = AsyncLLMEngine.from_engine_args(
vllm-llava-server  | [rank0]:   File "/usr/local/lib/python3.10/dist-packages/vllm/engine/async_llm_engine.py", line 398, in from_engine_args
vllm-llava-server  | [rank0]:     engine = cls(
vllm-llava-server  | [rank0]:   File "/usr/local/lib/python3.10/dist-packages/vllm/engine/async_llm_engine.py", line 349, in __init__
vllm-llava-server  | [rank0]:     self.engine = self._init_engine(*args, **kwargs)
vllm-llava-server  | [rank0]:   File "/usr/local/lib/python3.10/dist-packages/vllm/engine/async_llm_engine.py", line 473, in _init_engine
vllm-llava-server  | [rank0]:     return engine_class(*args, **kwargs)
vllm-llava-server  | [rank0]:   File "/usr/local/lib/python3.10/dist-packages/vllm/engine/llm_engine.py", line 236, in __init__
vllm-llava-server  | [rank0]:     self._initialize_kv_caches()
vllm-llava-server  | [rank0]:   File "/usr/local/lib/python3.10/dist-packages/vllm/engine/llm_engine.py", line 313, in _initialize_kv_caches
vllm-llava-server  | [rank0]:     self.model_executor.determine_num_available_blocks())
vllm-llava-server  | [rank0]:   File "/usr/local/lib/python3.10/dist-packages/vllm/executor/gpu_executor.py", line 75, in determine_num_available_blocks
vllm-llava-server  | [rank0]:     return self.driver_worker.determine_num_available_blocks()
vllm-llava-server  | [rank0]:   File "/usr/local/lib/python3.10/dist-packages/torch/utils/_contextlib.py", line 115, in decorate_context
vllm-llava-server  | [rank0]:     return func(*args, **kwargs)
vllm-llava-server  | [rank0]:   File "/usr/local/lib/python3.10/dist-packages/vllm/worker/worker.py", line 162, in determine_num_available_blocks
vllm-llava-server  | [rank0]:     self.model_runner.profile_run()
vllm-llava-server  | [rank0]:   File "/usr/local/lib/python3.10/dist-packages/torch/utils/_contextlib.py", line 115, in decorate_context
vllm-llava-server  | [rank0]:     return func(*args, **kwargs)
vllm-llava-server  | [rank0]:   File "/usr/local/lib/python3.10/dist-packages/vllm/worker/model_runner.py", line 844, in profile_run
vllm-llava-server  | [rank0]:     self.execute_model(seqs, kv_caches)
vllm-llava-server  | [rank0]:   File "/usr/local/lib/python3.10/dist-packages/torch/utils/_contextlib.py", line 115, in decorate_context
vllm-llava-server  | [rank0]:     return func(*args, **kwargs)
vllm-llava-server  | [rank0]:   File "/usr/local/lib/python3.10/dist-packages/vllm/worker/model_runner.py", line 749, in execute_model
vllm-llava-server  | [rank0]:     hidden_states = model_executable(
vllm-llava-server  | [rank0]:   File "/usr/local/lib/python3.10/dist-packages/torch/nn/modules/module.py", line 1532, in _wrapped_call_impl
vllm-llava-server  | [rank0]:     return self._call_impl(*args, **kwargs)
vllm-llava-server  | [rank0]:   File "/usr/local/lib/python3.10/dist-packages/torch/nn/modules/module.py", line 1541, in _call_impl
vllm-llava-server  | [rank0]:     return forward_call(*args, **kwargs)
vllm-llava-server  | [rank0]:   File "/usr/local/lib/python3.10/dist-packages/vllm/model_executor/models/llava_next.py", line 383, in forward
vllm-llava-server  | [rank0]:     image_input = self._parse_and_validate_image_input(**kwargs)
vllm-llava-server  | [rank0]:   File "/usr/local/lib/python3.10/dist-packages/vllm/model_executor/models/llava_next.py", line 196, in _parse_and_validate_image_input
vllm-llava-server  | [rank0]:     raise ValueError("Incorrect type of image sizes. "
vllm-llava-server  | [rank0]: ValueError: Incorrect type of image sizes. Got type: <class 'NoneType'>
vllm-llava-server exited with code 0

Sure! Here's the error I'm getting.

Are you loading the model from a remote HuggingFace repo or from a local one? Which model do you intend to use?

Update [6/25]
Synced with @ywang96 and @DarkLight1337 offline. These are a series of tasks for the next week. The goal is to lay the foundation with a better user and developer API. This will greatly benefit the work of supporting other VLM, passing embeddings and adding other modalities down the road.
See #5818 for upcoming API changes.

#5214
#5276
Remove VisionLanguageConfig.
Remove the support for passing in pixel_values and image_features.
Fix seq_len calculation (remove the dependency on image_feature_size) for profiling.

@DarkLight1337 I am loading from a local repo using the same setup as our other local chat models that I use vLLM for. I clone HF repo and point to the directory in the docker-compose with this config:
volumes:
      - ${MODEL_VOL}/${VLLM_IMAGE_MODEL_ID}:/vllm-workspace/${VLLM_IMAGE_MODEL_ID}
command: ["--model", "${VLLM_IMAGE_MODEL_ID}", "--gpu-memory-utilization", "0.75", "--host", "<IP_ADDRESS>", "--root-path", "/vllm-server",
      "--image-input-type", "pixel_values", "--image-token-id", "32000", "--image-input-shape", "1,3,336,336", "--image-feature-size", "576",
      "--chat-template", "template_llava.jinja"]

The model ID is loaded from my .env and points to the cloned HF model directory. I am using llava-hf/llava-v1.6-mistral-7b-hf and the directory structure is the following:
config.json
generation_config.json
.git
.gitattributes
model-00001-of-00004.safetensors
model-00002-of-00004.safetensors
model-00003-of-00004.safetensors
model-00004-of-00004.safetensors
model.safetensors.index.json
preprocessor_config.json
README.md
special_tokens_map.json
template_llava.jinja
tokenizer_config.json
tokenizer.json
tokenizer.model

Here is the current config.json I'm using. As I talked about in my initial post, vLLM seems to be looking in text_config for several key/value pairs, but in the most recent pull from HF, those are missing in that section and causes startup to fail at an earlier step, so I added them back in from the mistralai/Mistral-7B-Instruct-v0.2 config.json.

llava-hf/llava-v1.6-mistral-7b-hf

@FennFlyer could you open a separate issue so we can keep this issue clean just for discussion around multi-modality refactoring?
BTW - I just checked the latest huggingface repo and it does have the text_config in the config.json file.
https://huggingface.co/llava-hf/llava-v1.6-mistral-7b-hf/blob/main/config.json

@ywang96 @DarkLight1337 Hi, not sure if this is the correct place to post about this but thank you for your hard work on multimodal support for vLLM! I'm pretty interested in contributing to the multimodal effort specifically, so I was wondering if you guys had any suggestions on good starter issues to contribute to. I spent the last week familiarizing myself with parts of the codebase, but it seems most of the labeled starter issues don't have to do with multimodal support specifically. If this is not the best place to ask, would it be possible to direct me to the correct place?

Hi @chuyishang! Thank you for your interest and offering your help and I really appreciate it! Yes, I would say this is the place where you can check out work around multi-modality, and I often update the issue at the top with the latest updates!
In terms of suggestions on good starter issues to contribute to, I would say it'll be great if you can try out our VLM stack yourself and see if there's anything that we can improve. Especially when it comes to performance, a few things I'd like to work on but have got a chance to are:

Port CLIPAttention: I made a PR #5591 to port over the CLIPVisionModel from transformers to vLLM to be used inside Llava and Llava-NeXT, but currently it's still using CLIPAttention from transformers, so one contribution you can make is to rewrite that with vLLM primitive layers.
Rewrite MultiModalProjector: Layers in MultiModalProjector in Llava and Llava-NeXT, so it would be great to see if rewriting them with vLLM primitive layers will bring any performance benefit especially when the model is tensor parallelized. (See TODO here)

For those who are currently working on implementing new VLMs, please note the following:

Please update your PR according to https://github.com/vllm-project/vllm/pull/5896, otherwise your models will not function once merged.
If your model has a dynamic number of image features, please implement a way to calculate the number of image features based on image size. This makes the model easier to integrate with #5276.

Thanks in advance!

A friendly ping to this thread - @xwjiang2010, @DarkLight1337 and I have finished our second refactoring milestone and glad that we made it before the v0.5.1 release. To summarize:

We completely removed the need to specify image-related arguments when using LLM or launching the OpenAI API server. Simply specify model id and it'll be good to go!
The interface for LLM is now more intuitive, flexible and user-friendly. Prompts no longer need to be padded and are now fully compatible with huggingface formatting style.
Dynamic image shape/embedding is now supported, and to make this happen, we added InputRegistry for developers to apply the model's preprocessing logic ahead of forward pass time.

Next, we will be focusing on adding support for more vision language models and better GPU profiling strategy. A more detailed roadmap will be updated in the OP of this issue. As always, feedbacks and contributions are very much welcomed!

Another friendly ping for folks who are following this issue, we have updated the roadmap in the OP. Please feel free to leave feedbacks!

@ywang96 , is there any bounty program for vLLM?

@ywang96 , is there any bounty program for vLLM?

@huseinzol05 I'd really wish there were one but not for now - I personally believe in open source and the passion from all contributors!

Support multiple images as input whenever the model supports it

About this feature, I can give it a shot. Is there still any refactoring tasks needed to be done first (mentioned in https://github.com/vllm-project/vllm/issues/5820#issuecomment-2188393417)?
Thanks!

Support multiple images as input whenever the model supports it

About this feature, I can give it a shot. Is there still any refactoring tasks needed to be done first (mentioned in #5820 (comment))?
Thanks!

I already have a plan drawn up for supporting multiple images but haven't had the bandwidth to implement it:

We should update try_concat to stack the tensors instead of concatenating them.

The function currently works because existing models require input shape [B, (P), C, H, W] (P is an optional dimension for models that generate patches from the original input image), so that a batch containing B items each of shape [1, (P), C, H, W] is concatenated into [B, (P), C, H, W] as expected. However, this will not work for multiple image inputs; instead, we should stack B times [N, (P), C, H, W] into [B, N, (P), C, H, W], where N is the number of images.

In order to use the above stacking logic for both single- and multi-image inputs, we need to update all existing models to support [B, N, (P), C, H, W] instead of [B, (P), C, H, W] format. For now, we can throw an error if we find that N>1 and the model is not supposed to handle multiple images.

Perhaps we can have a utility function to validating the input shape so there is less code repetition.

Update the documentation/inferface of input_mapper to explicitly require the dimension N:

The tensor outputted by input mapper should have dimensions [N, ...] where N is the number of multimodal inputs for the given prompt (N not to be confused with the number of prompts B). If the shape of the tensor for each multimodal input is different, a (possibly nested) list of tensors can be returned, where the outermost list has size N. Then we can combine the multimodal inputs corresponding to multiple prompts: torch.stack is called iff the multimodal input for each prompt is a tensor (not a list) and they all share the same shape; in all other cases, we simply return a list of size B, with each element containing the output of the input mapper for a prompt.

It would be great if you could help with this! There is no need to wait for other refactoring tasks.

Hey @zifeitong, thank you for being interested in contributing to the multi-modality workstream!
Yes @DarkLight1337 has pretty much explained the majority of the work needed to be done to enable multi-image input, but to enable the full feature, please keep in mind that the frontend and the processor also needs to be updated accordingly.
P.S. We would appreciate if contributors can break down their work into a series of PRs so that it's easier for us to review!

Thanks for the detailed write-up. I'll try to keep each PR small.

@ywang96  @DarkLight1337  I need to refactor the vision encoder of mini-cpmv2.5 for our project. I've already implemented some initial code (see:idefics2_vision_model), but it seems there's a related PR  currently. I'd like to ask if my implementation is duplicating work. If so, I won't continue with it.

@ywang96 @DarkLight1337 I need to refactor the vision encoder of mini-cpmv2.5 for our project. I've already implemented some initial code (see:idefics2_vision_model), but it seems there's a related PR currently. I'd like to ask if my implementation is duplicating work. If so, I won't continue with it.

I suggest you comment on that PR and see if the author responds.

@ywang96 @DarkLight1337 I need to refactor the vision encoder of mini-cpmv2.5 for our project. I've already implemented some initial code (see:idefics2_vision_model), but it seems there's a related PR currently. I'd like to ask if my implementation is duplicating work. If so, I won't continue with it.

@jeejeelee I don't think the author seems to be working on it anymore, so feel free to open up a new PR to add Idefics2.

Support multiple images as input whenever the model supports it

About this feature, I can give it a shot. Is there still any refactoring tasks needed to be done first (mentioned in #5820 (comment))?
Thanks!

I already have a plan drawn up for supporting multiple images but haven't had the bandwidth to implement it:

Support multi-image input at the model layer. (Dev-facing only)
a. We should update try_concat to stack the tensors instead of concatenating them.

The function currently works because existing models require input shape [B, (P), C, H, W] (P is an optional dimension for models that generate patches from the original input image), so that a batch containing B items each of shape [1, (P), C, H, W] is concatenated into [B, (P), C, H, W] as expected. However, this will not work for multiple image inputs; instead, we should stack B times [N, (P), C, H, W] into [B, N, (P), C, H, W], where N is the number of images.
Concrete logic: torch.stack is called only if the multimodal input for each prompt is a tensor (not a list) and they all share the same shape; in all other cases, we simply return a list of size B, with each element containing the output of the input mapper for a prompt.
Update the documentation of input_mapper to explicitly require the dimension N:

Each output tensor should have dimensions [N, ...] where N is the number of multimodal inputs for the given prompt. If the shape of the tensor for each multimodal input is different, a (possibly nested) list of tensors can be returned, where the outermost list has size N.

b. In order to use the above stacking logic for both single- and multi-image inputs, we need to update all existing models to support [B, N, (P), C, H, W] instead of [B, (P), C, H, W] format.

Execuse me? Is there anyone already started this work? I want to state my suggestion. I think the multi-image supports should be depend on the image_processor of each model. I think every output of image_processor already contains the batch dimension. For example, LLava-Next only invoke image_processor once in their prcessor(https://github.com/huggingface/transformers/blob/main/src/transformers/models/llava_next/processing_llava_next.py#L111) while the inputs could be batched.

Execuse me? Is there anyone already started this work?

@HwwwwwwwH As far as I know no one has actually worked on this yet so you're very welcome to make a PR for this!

I think every output of image_processor already contains the batch dimension. For example, LLava-Next only invoke image_processor once in their prcessor(https://github.com/huggingface/transformers/blob/main/src/transformers/models/llava_next/processing_llava_next.py#L111) while the inputs could be batched.

This is correct, but as far as I know, how transformers does it is that they assume of order of appearance of <image> in the batched prompts is the same order of images input (e.g, images=[image_stop, image_cats, image_snowman] in this example here). I think this should actually work, but we might need to be careful with how we currently batch the input images to make sure we also preserve the order.

Execuse me? Is there anyone already started this work?

@HwwwwwwwH As far as I know no one has actually worked on this yet so you're very welcome to make a PR for this! I'm happy to review it as well :)

I think every output of image_processor already contains the batch dimension. For example, LLava-Next only invoke image_processor once in their prcessor(https://github.com/huggingface/transformers/blob/main/src/transformers/models/llava_next/processing_llava_next.py#L111) while the inputs could be batched.

This is correct, but as far as I know, how transformers does it is that they assume of order of appearance of "<image>" in the batched prompts is the same order of images input (e.g, images=[image_stop, image_cats, image_snowman] in this example here). I think this should actually work, but we might need to be careful with how we currently batch the input images to make sure we also preserve the order.

Notably in that example, the first batch has 2 images and the second batch has 1 image. In vLLM, we process each batch separately, so the image processor will output [N=2, P, C, H, W] for the first batch and [N=1, P, C, H, W] for the second batch.

@ywang96  @DarkLight1337  I want to confirm again, is anyone currently starting work on VLM+LoRA ? If not, I'm willing to try.

@ywang96 @DarkLight1337 I want to confirm again, is anyone currently starting work on VLM+LoRA ? If not, I'm willing to try.

I don't think so - please feel free to work on it!

@ywang96 @DarkLight1337 I want to confirm again, is anyone currently starting work on VLM+LoRA ? If not, I'm willing to try.

Not that I'm aware of. Thanks for the help!

Is anyone working on prefix caching on multimodality input? I just finished the video support and planning to start working on prefix caching.

Is anyone working on prefix caching on multimodality input? I just finished the video support and planning to start working on prefix caching.

@TKONIY Not in my knowledge. Can you make a RFC issue about the high level design in your mind for discussion?
For multimodal inputs, there are actually two possible layers of caching:

Cache embeddings for a previously seen image/audio/video, to avoid recomputation of the encoding component
Extend current APC(automatic prefix caching) implementation to make it work with sequences that have multimodal data.

For 1, this is somewhat already addressed by #6613, since users then can implement their own embedding caching outside vLLM.
For 2, you can read more about it here. The technical challenge here is that currently each block of KV Cache is uniquely identified by the token(id)s within the block. However, for multimodal data, their representation will always be the placeholder token id in the original sequence. I think if we're able to address this problem and make it work with embedding based inputs, then this would benefit vLLM in a bigger scope if we decide to support embeddings as input for LMs eventually (i.e, #6869).

Is anyone working on prefix caching on multimodality input? I just finished the video support and planning to start working on prefix caching.

@TKONIY Not in my knowledge. Can you make a RFC issue about the high level design in your mind for discussion?
For multimodal inputs, there are actually two possible layers of caching:

Cache embeddings for a previously seen image/audio/video, to avoid recomputation of the encoder & projector component

Extend current APC(automatic prefix caching) implementation to make it work with sequences that have multimodal data.

For 1, this is somewhat already addressed by #6613, since users then can implement their own embedding caching outside vLLM.
For 2, you can read more about it here. The technical challenge here is that currently each block of KV cache is uniquely identified by the token(id)s within the block. However, for multimodal data, their representation will always be the placeholder token id in the original sequence. I think if we're able to address this problem and make it work with embedding based inputs, then this would benefit vLLM in a bigger scope if we decide to support embeddings as input for LMs eventually (i.e, #6869).

Thanks for the introduction! In terms of the identifier, I will try to figure out a solution and open a RFC.

With the release of v0.6.0, it's a good time now to wrap up the recent work in multi-modality!
In the past two months, we have made tremendous progress in multi-modality! On behalf of the vLLM team, @DarkLight1337 and I would like to thank all the community members for their amazing contributions to this workstream! To summarize the update:

New modality - Audio: vLLM now supports audio modality with Ultravox as its first supported audio LMM. Shoutout to @juberti @petersalas and the fixie.ai team for choosing vLLM to open source their model!
Tensor Parallelism on ViT: Thanks to @ChristopherCho's contribution, vision encoders are now sharded when the VLM is served over multiple GPUs. This significantly improves space efficiency since ViT is no longer replicated on each GPU.
Multi-image interface: A highly requested feature from the community is multi-image inference. This is now supported in both offline and online with supported models. Kudos to @Isotr0py @zifeitong @petersalas for their help to enable multi-image/audio inference!
Image-embeddings as input: For late-fusion, embedding-based LMMs, sometimes it makes sense to host ViT in a separately host, and have LMM takes image embeddings directly instead of PIL images. This is now supported for most multi-modal models on vLLM.
Model support expansion: As LMM development evolves rapidly, it's important to keep up with the pace, and vLLM now supports 11 LMMs! Props to @Isotr0py @HwwwwwwwH @jeejeelee @alex-jw-brooks for contributing InternVL2, MiniCPM-V and Qwen-VL!

We're also very excited about the upcoming video support with dynamic number of frames (@TKONIY) and Qwen2-VL model support (@fyabc from Qwen Team) that will be available in 0.6.1 release!
As usual, the roadmap for this workstream will be updated in the OP of this issue in the upcoming week. Feedbacks and contributions are always very welcomed!

Multi-image support for Qwenvl2 & InternVL2, Thank u!

A friendly bump that our roadmap for multimodality has been updated in the OP of this thread!

https://github.com/hiyouga/LLaMA-Factory/issues/3645
LLaMA-Factory webui cannot open

@ywang96 hiyouga/LLaMA-Factory#3645 LLaMA-Factory webui cannot open. And I noticed. The last time I used pip install vllm to install vllm, it caused my torch version to change from GPU to CPU. Must be reinstalled

File "D:\my\env\python3.10.10\lib\site-packages\llmtuner\chat\vllm_engine.py", line 16, in
from vllm.sequence import MultiModalData

Looks like an incompatibility in llmtuner, please report this issue on their repo instead.

@ywang96 hiyouga/LLaMA-Factory#3645 LLaMA-Factory webui cannot open. And I noticed. The last time I used pip install vllm to install vllm, it caused my torch version to change from GPU to CPU. Must be reinstalled

File "D:\my\env\python3.10.10\lib\site-packages\llmtuner\chat\vllm_engine.py", line 16, in
from vllm.sequence import MultiModalData

Looks like an incompatibility in llmtuner, please report this issue on their repo instead.

Illegal llmtuner is just one of the issues.
2.
When I open it directly using the source code, there is no problem with llmtuner anymore, it becomes another issue. It's a VLLM issue.
3.
The most serious issue is that pip install vllm will force a reinstallation of the CPU version torch
is vllm error
1× 2 √ 3√

When I open it directly using the source code, there is no problem with llmtuner anymore, it becomes another issue. It's a VLLM issue.

Can you show the error in this case?

When I open it directly using the source code, there is no problem with llmtuner anymore, it becomes another issue. It's a VLLM issue.

Can you show the error in this case?

cannot import name 'ImagePixelData' from 'vllm.multimodal.image' (d:\my\env\python3.10.10\lib\site-packages\vllm\multimodal\image.py)
File "D:\my\work\LLM\LLaMA-Factory\LLaMA-Factory\src\llamafactory\chat\vllm_engine.py", line 33, in
from vllm.multimodal.image import ImagePixelData
File "D:\my\work\LLM\LLaMA-Factory\LLaMA-Factory\src\llamafactory\chat\chat_model.py", line 25, in
from .vllm_engine import VllmEngine
File "D:\my\work\LLM\LLaMA-Factory\LLaMA-Factory\src\llamafactory\chat_init.py", line 16, in
from .chat_model import ChatModel
File "D:\my\work\LLM\LLaMA-Factory\LLaMA-Factory\src\llamafactory\api\app.py", line 21, in
from ..chat import ChatModel
File "D:\my\work\LLM\LLaMA-Factory\LLaMA-Factory\src\llamafactory\cli.py", line 22, in
from .api.app import run_api
File "D:\my\work\LLM\LLaMA-Factory\LLaMA-Factory\src\llamafactory_init.py", line 17, in
from .cli import VERSION
File "D:\my\work\LLM\LLaMA-Factory\LLaMA-Factory\src\train.py", line 15, in
from llamafactory.train.tuner import run_exp
File "D:\my\env\python3.10.10\Lib\runpy.py", line 86, in _run_code
exec(code, run_globals)
File "D:\my\env\python3.10.10\Lib\runpy.py", line 196, in _run_module_as_main (Current frame)
return _run_code(code, main_globals, None,
ImportError: cannot import name 'ImagePixelData' from 'vllm.multimodal.image' (d:\my\env\python3.10.10\lib\site-packages\vllm\multimodal\image.py)

I believe this is still some incompatibility issue, since ImagePixelData no longer exists in the current vLLM version. It was introduced on v0.5.0 and removed in v0.5.1, which was quite a while ago.

You should ask LLaMA-Factory to support newer version of vLLM.您应该要求LLaMA-Factory支持vLLM的新版本。

with reason. with reason. The LLaMA-Factoryshould be asked to remove the ImagePixelData  class. So now the problem has become pip install vllm (0.6.3)will force a reinstallation of the CPU version torch and replace cuda torch on windows only

pip install vllm (0.6.3) will force a reinstallation of the CPU version torch and replace cuda torch on windows only.

I don't quite get what you mean, how can you have multiple versions of torch at once?

pip install vllm (0.6.3) will force a reinstallation of the CPU version torch and replace cuda torch on windows only.pip install vllm（0.6.3）将强制重新安装CPU版本的torch并仅在Windows上替换cuda torch。

I don't quite get what you mean, how can you have different versions of torch for CPU and GPU at the same time?我不太明白你的意思，你怎么能有不同版本的火炬CPU和GPU在同一时间？

only cuda torch
 pip install vllm --no-deps
Collecting vllm
  Using cached vllm-0.6.3.post1.tar.gz (2.7 MB)
  Installing build dependencies ... error
  error: subprocess-exited-with-error

  × pip subprocess to install build dependencies did not run successfully.
  │ exit code: 2
  ╰─> [86 lines of output]
      Collecting cmake>=3.26
        Using cached cmake-3.30.5-py3-none-win_amd64.whl.metadata (6.4 kB)
      Collecting ninja
        Using cached ninja-<IP_ADDRESS>-py2.py3-none-win_amd64.whl.metadata (5.4 kB)

      Collecting packaging
        Using cached packaging-24.1-py3-none-any.whl.metadata (3.2 kB)
      Collecting setuptools>=61
        Using cached setuptools-75.2.0-py3-none-any.whl.metadata (6.9 kB)
      Collecting setuptools-scm>=8.0
        Using cached setuptools_scm-8.1.0-py3-none-any.whl.metadata (6.6 kB)
      Collecting torch==2.4.0
        Using cached torch-2.4.0-cp310-cp310-win_amd64.whl.metadata (27 kB)
      Collecting wheel
        Using cached wheel-0.44.0-py3-none-any.whl.metadata (2.3 kB)
      Collecting jinja2
        Using cached jinja2-3.1.4-py3-none-any.whl.metadata (2.6 kB)
      Collecting filelock (from torch==2.4.0)
        Using cached filelock-3.16.1-py3-none-any.whl.metadata (2.9 kB)
      Collecting typing-extensions>=4.8.0 (from torch==2.4.0)
        Using cached typing_extensions-4.12.2-py3-none-any.whl.metadata (3.0 kB)

      Collecting sympy (from torch==2.4.0)
        Using cached sympy-1.13.3-py3-none-any.whl.metadata (12 kB)
      Collecting networkx (from torch==2.4.0)
        Using cached networkx-3.4.2-py3-none-any.whl.metadata (6.3 kB)
      Collecting fsspec (from torch==2.4.0)
        Using cached fsspec-2024.10.0-py3-none-any.whl.metadata (11 kB)
      Collecting tomli>=1 (from setuptools-scm>=8.0)
        Using cached tomli-2.0.2-py3-none-any.whl.metadata (10.0 kB)
      Collecting MarkupSafe>=2.0 (from jinja2)
        Using cached MarkupSafe-3.0.2-cp310-cp310-win_amd64.whl.metadata (4.1 kB
)
      Collecting mpmath<1.4,>=1.1.0 (from sympy->torch==2.4.0)
        Using cached mpmath-1.3.0-py3-none-any.whl.metadata (8.6 kB)
      Downloading torch-2.4.0-cp310-cp310-win_amd64.whl (197.9 MB)
                                                  3.9/197.9 MB 21.3 kB/s eta 2:3
1:31
      ERROR: Exception:
      Traceback (most recent call last):
        File "D:\my\env\python3.10.10\Lib\site-packages\pip\_vendor\urllib3\resp
onse.py", line 438, in _error_catcher
          yield
        File "D:\my\env\python3.10.10\Lib\site-packages\pip\_vendor\urllib3\resp
onse.py", line 561, in read
          data = self._fp_read(amt) if not fp_closed else b""
        File "D:\my\env\python3.10.10\Lib\site-packages\pip\_vendor\urllib3\resp
onse.py", line 527, in _fp_read
          return self._fp.read(amt) if amt is not None else self._fp.read()
        File "D:\my\env\python3.10.10\Lib\site-packages\pip\_vendor\cachecontrol
\filewrapper.py", line 98, in read
          data: bytes = self.__fp.read(amt)
        File "D:\my\env\python3.10.10\lib\http\client.py", line 465, in read
          s = self.fp.read(amt)
        File "D:\my\env\python3.10.10\lib\socket.py", line 705, in readinto
          return self._sock.recv_into(b)
        File "D:\my\env\python3.10.10\lib\ssl.py", line 1274, in recv_into
          return self.read(nbytes, buffer)
        File "D:\my\env\python3.10.10\lib\ssl.py", line 1130, in read
          return self._sslobj.read(len, buffer)
      TimeoutError: The read operation timed out

      During handling of the above exception, another exception occurred:

      Traceback (most recent call last):
        File "D:\my\env\python3.10.10\Lib\site-packages\pip\_internal\cli\base_c
ommand.py", line 105, in _run_wrapper
          status = _inner_run()
        File "D:\my\env\python3.10.10\Lib\site-packages\pip\_internal\cli\base_c
ommand.py", line 96, in _inner_run
          return self.run(options, args)
        File "D:\my\env\python3.10.10\Lib\site-packages\pip\_internal\cli\req_co
mmand.py", line 67, in wrapper
          return func(self, options, args)
        File "D:\my\env\python3.10.10\Lib\site-packages\pip\_internal\commands\i
nstall.py", line 379, in run
          requirement_set = resolver.resolve(
        File "D:\my\env\python3.10.10\Lib\site-packages\pip\_internal\resolution
\resolvelib\resolver.py", line 179, in resolve
          self.factory.preparer.prepare_linked_requirements_more(reqs)
        File "D:\my\env\python3.10.10\Lib\site-packages\pip\_internal\operations
\prepare.py", line 554, in prepare_linked_requirements_more
          self._complete_partial_requirements(
        File "D:\my\env\python3.10.10\Lib\site-packages\pip\_internal\operations
\prepare.py", line 469, in _complete_partial_requirements
          for link, (filepath, _) in batch_download:
        File "D:\my\env\python3.10.10\Lib\site-packages\pip\_internal\network\do
wnload.py", line 184, in __call__
          for chunk in chunks:
        File "D:\my\env\python3.10.10\Lib\site-packages\pip\_internal\cli\progre
ss_bars.py", line 55, in _rich_progress_bar
          for chunk in iterable:
        File "D:\my\env\python3.10.10\Lib\site-packages\pip\_internal\network\ut
ils.py", line 65, in response_chunks
          for chunk in response.raw.stream(
        File "D:\my\env\python3.10.10\Lib\site-packages\pip\_vendor\urllib3\resp
onse.py", line 622, in stream
          data = self.read(amt=amt, decode_content=decode_content)
        File "D:\my\env\python3.10.10\Lib\site-packages\pip\_vendor\urllib3\resp
onse.py", line 560, in read
          with self._error_catcher():
        File "D:\my\env\python3.10.10\lib\contextlib.py", line 153, in __exit__
          self.gen.throw(typ, value, traceback)
        File "D:\my\env\python3.10.10\Lib\site-packages\pip\_vendor\urllib3\resp
onse.py", line 443, in _error_catcher
          raise ReadTimeoutError(self._pool, None, "Read timed out.")
      pip._vendor.urllib3.exceptions.ReadTimeoutError: HTTPSConnectionPool(host=
'files.pythonhosted.org', port=443): Read timed out.
      [end of output]

  note: This error originates from a subprocess, and is likely not a problem wit
h pip.
error: subprocess-exited-with-error

× pip subprocess to install build dependencies did not run successfully.
│ exit code: 2
╰─> See above for output.

pip install vllm (0.6.3) will force a reinstallation of the CPU version torch and replace cuda torch on windows only.pip install vllm（0.6.3）将强制重新安装CPU版本的torch并仅在Windows上替换cuda torch。

I don't quite get what you mean, how can you have different versions of torch for CPU and GPU at the same time?我不太明白你的意思，你怎么能有不同版本的火炬CPU和GPU在同一时间？

only cuda torch
 pip install vllm --no-deps
Collecting vllm
  Using cached vllm-0.6.3.post1.tar.gz (2.7 MB)
  Installing build dependencies ... error
  error: subprocess-exited-with-error

  × pip subprocess to install build dependencies did not run successfully.
  │ exit code: 2
  ╰─> [86 lines of output]
      Collecting cmake>=3.26
        Using cached cmake-3.30.5-py3-none-win_amd64.whl.metadata (6.4 kB)
      Collecting ninja
        Using cached ninja-<IP_ADDRESS>-py2.py3-none-win_amd64.whl.metadata (5.4 kB)

      Collecting packaging
        Using cached packaging-24.1-py3-none-any.whl.metadata (3.2 kB)
      Collecting setuptools>=61
        Using cached setuptools-75.2.0-py3-none-any.whl.metadata (6.9 kB)
      Collecting setuptools-scm>=8.0
        Using cached setuptools_scm-8.1.0-py3-none-any.whl.metadata (6.6 kB)
      Collecting torch==2.4.0
        Using cached torch-2.4.0-cp310-cp310-win_amd64.whl.metadata (27 kB)
      Collecting wheel
        Using cached wheel-0.44.0-py3-none-any.whl.metadata (2.3 kB)
      Collecting jinja2
        Using cached jinja2-3.1.4-py3-none-any.whl.metadata (2.6 kB)
      Collecting filelock (from torch==2.4.0)
        Using cached filelock-3.16.1-py3-none-any.whl.metadata (2.9 kB)
      Collecting typing-extensions>=4.8.0 (from torch==2.4.0)
        Using cached typing_extensions-4.12.2-py3-none-any.whl.metadata (3.0 kB)

      Collecting sympy (from torch==2.4.0)
        Using cached sympy-1.13.3-py3-none-any.whl.metadata (12 kB)
      Collecting networkx (from torch==2.4.0)
        Using cached networkx-3.4.2-py3-none-any.whl.metadata (6.3 kB)
      Collecting fsspec (from torch==2.4.0)
        Using cached fsspec-2024.10.0-py3-none-any.whl.metadata (11 kB)
      Collecting tomli>=1 (from setuptools-scm>=8.0)
        Using cached tomli-2.0.2-py3-none-any.whl.metadata (10.0 kB)
      Collecting MarkupSafe>=2.0 (from jinja2)
        Using cached MarkupSafe-3.0.2-cp310-cp310-win_amd64.whl.metadata (4.1 kB
)
      Collecting mpmath<1.4,>=1.1.0 (from sympy->torch==2.4.0)
        Using cached mpmath-1.3.0-py3-none-any.whl.metadata (8.6 kB)
      Downloading torch-2.4.0-cp310-cp310-win_amd64.whl (197.9 MB)
                                                  3.9/197.9 MB 21.3 kB/s eta 2:3
1:31
      ERROR: Exception:
      Traceback (most recent call last):
        File "D:\my\env\python3.10.10\Lib\site-packages\pip\_vendor\urllib3\resp
onse.py", line 438, in _error_catcher
          yield
        File "D:\my\env\python3.10.10\Lib\site-packages\pip\_vendor\urllib3\resp
onse.py", line 561, in read
          data = self._fp_read(amt) if not fp_closed else b""
        File "D:\my\env\python3.10.10\Lib\site-packages\pip\_vendor\urllib3\resp
onse.py", line 527, in _fp_read
          return self._fp.read(amt) if amt is not None else self._fp.read()
        File "D:\my\env\python3.10.10\Lib\site-packages\pip\_vendor\cachecontrol
\filewrapper.py", line 98, in read
          data: bytes = self.__fp.read(amt)
        File "D:\my\env\python3.10.10\lib\http\client.py", line 465, in read
          s = self.fp.read(amt)
        File "D:\my\env\python3.10.10\lib\socket.py", line 705, in readinto
          return self._sock.recv_into(b)
        File "D:\my\env\python3.10.10\lib\ssl.py", line 1274, in recv_into
          return self.read(nbytes, buffer)
        File "D:\my\env\python3.10.10\lib\ssl.py", line 1130, in read
          return self._sslobj.read(len, buffer)
      TimeoutError: The read operation timed out

      During handling of the above exception, another exception occurred:

      Traceback (most recent call last):
        File "D:\my\env\python3.10.10\Lib\site-packages\pip\_internal\cli\base_c
ommand.py", line 105, in _run_wrapper
          status = _inner_run()
        File "D:\my\env\python3.10.10\Lib\site-packages\pip\_internal\cli\base_c
ommand.py", line 96, in _inner_run
          return self.run(options, args)
        File "D:\my\env\python3.10.10\Lib\site-packages\pip\_internal\cli\req_co
mmand.py", line 67, in wrapper
          return func(self, options, args)
        File "D:\my\env\python3.10.10\Lib\site-packages\pip\_internal\commands\i
nstall.py", line 379, in run
          requirement_set = resolver.resolve(
        File "D:\my\env\python3.10.10\Lib\site-packages\pip\_internal\resolution
\resolvelib\resolver.py", line 179, in resolve
          self.factory.preparer.prepare_linked_requirements_more(reqs)
        File "D:\my\env\python3.10.10\Lib\site-packages\pip\_internal\operations
\prepare.py", line 554, in prepare_linked_requirements_more
          self._complete_partial_requirements(
        File "D:\my\env\python3.10.10\Lib\site-packages\pip\_internal\operations
\prepare.py", line 469, in _complete_partial_requirements
          for link, (filepath, _) in batch_download:
        File "D:\my\env\python3.10.10\Lib\site-packages\pip\_internal\network\do
wnload.py", line 184, in __call__
          for chunk in chunks:
        File "D:\my\env\python3.10.10\Lib\site-packages\pip\_internal\cli\progre
ss_bars.py", line 55, in _rich_progress_bar
          for chunk in iterable:
        File "D:\my\env\python3.10.10\Lib\site-packages\pip\_internal\network\ut
ils.py", line 65, in response_chunks
          for chunk in response.raw.stream(
        File "D:\my\env\python3.10.10\Lib\site-packages\pip\_vendor\urllib3\resp
onse.py", line 622, in stream
          data = self.read(amt=amt, decode_content=decode_content)
        File "D:\my\env\python3.10.10\Lib\site-packages\pip\_vendor\urllib3\resp
onse.py", line 560, in read
          with self._error_catcher():
        File "D:\my\env\python3.10.10\lib\contextlib.py", line 153, in __exit__
          self.gen.throw(typ, value, traceback)
        File "D:\my\env\python3.10.10\Lib\site-packages\pip\_vendor\urllib3\resp
onse.py", line 443, in _error_catcher
          raise ReadTimeoutError(self._pool, None, "Read timed out.")
      pip._vendor.urllib3.exceptions.ReadTimeoutError: HTTPSConnectionPool(host=
'files.pythonhosted.org', port=443): Read timed out.
      [end of output]

  note: This error originates from a subprocess, and is likely not a problem wit
h pip.
error: subprocess-exited-with-error

× pip subprocess to install build dependencies did not run successfully.
│ exit code: 2
╰─> See above for output.

If you internet is not good. You are so lucky. Because it will fail during the process of forcibly replacing CUDA torch with CPU. If you have a good internet connection. So things will become very bad. Your torch will transition from CUDA to a lower version CPU. And pip install vllm --no-deps or pip install vllm has same issue

What is your original version of pytorch?

@DarkLight1337  torch Version: 2.5.0+cu124

@DarkLight1337 torch Version: 2.5.0+cu124 Torch version 2.4.+ I have the same problem. I forgot the exact version.

Can you raise this in a new issue (with installation tag) so we can better focus on this?

@DarkLight1337 Great job! I'm a newcomer here, and I'm using this framework primarily to validate the extrapolation capabilities of large multimodal models, specifically when the context exceeds 32k tokens. However,  I have encountered an issue while modifying the positional encoding in the mrope_input_positions section of the Qwen2-VL code, and I have tried but don't know how to resolve it. In short, I'm aiming to explore the model's performance when extrapolating to a 60k context on the Qwen2-VL 7B model, using video data for testing. I tried replacing this section (https://github.com/vllm-project/vllm/blob/3bb4befea7166850bdee3f72fe060c9c4044ba85/vllm/worker/model_runner.py#L672) with vanilla-ROPE（That is, placing image, video, and text tokens all on the main diagonal of the M-RoPE.）, which caused the max value of the mrope_input_positions up to approximately 59k, but it eventually led to an error.
../aten/src/ATen/native/cuda/IndexKernel.cu:93: operator(): block: [8320,0,0], thread: [64,0,0] Assertion `-sizes[i] <= index && in
dex < sizes[i] && "index out of bounds"` failed.
../aten/src/ATen/native/cuda/IndexKernel.cu:93: operator(): block: [8320,0,0], thread: [65,0,0] Assertion `-sizes[i] <= index && in
dex < sizes[i] && "index out of bounds"` failed.
../aten/src/ATen/native/cuda/IndexKernel.cu:93: operator(): block: [8320,0,0], thread: [66,0,0] Assertion `-sizes[i] <= index && in
dex < sizes[i] && "index out of bounds"` failed.
../aten/src/ATen/native/cuda/IndexKernel.cu:93: operator(): block: [8320,0,0], thread: [67,0,0] Assertion `-sizes[i] <= index && in
dex < sizes[i] && "index out of bounds"` failed.
../aten/src/ATen/native/cuda/IndexKernel.cu:93: operator(): block: [8320,0,0], thread: [68,0,0] Assertion `-sizes[i] <= index && in
dex < sizes[i] && "index out of bounds"` failed.
../aten/src/ATen/native/cuda/IndexKernel.cu:93: operator(): block: [8320,0,0], thread: [69,0,0] Assertion `-sizes[i] <= index && in
dex < sizes[i] && "index out of bounds"` failed.
...
INFO 11-03 18:58:15 model_runner_base.py:120] Writing input of failed execution to /tmp/err_execute_model_input_20241103-185815.pkl...
WARNING 11-03 18:58:15 model_runner_base.py:143] Failed to pickle inputs of failed execution: CUDA error: device-side assert triggered
WARNING 11-03 18:58:15 model_runner_base.py:143] CUDA kernel errors might be asynchronously reported at some other API call, so the stacktrace below might be incorrect.
WARNING 11-03 18:58:15 model_runner_base.py:143] For debugging consider passing CUDA_LAUNCH_BLOCKING=1
WARNING 11-03 18:58:15 model_runner_base.py:143] Compile with `TORCH_USE_CUDA_DSA` to enable device-side assertions.
WARNING 11-03 18:58:15 model_runner_base.py:143]
RuntimeError: Error in model execution: CUDA error: device-side assert triggered
CUDA kernel errors might be asynchronously reported at some other API call, so the stacktrace below might be incorrect.
For debugging consider passing CUDA_LAUNCH_BLOCKING=1
Compile with `TORCH_USE_CUDA_DSA` to enable device-side assertions.

I have already tested the original M-RoPE, which outputs correctly with a 60k context, and the maximum mrope_input_positions value is around 300. So, I am wondering if the position value is too large, causing it to exceed the index. How should I modify it to support vanilla-RoPE (Or perhaps some other 3D positional encoding, where the positional encoding values are quite large.) for evaluation?
p.s. I noticed that this function (https://github.com/vllm-project/vllm/blob/3bb4befea7166850bdee3f72fe060c9c4044ba85/vllm/worker/model_runner.py#L637) was called several times before inferring on my provided video test data, and I’m wondering if this might be related.
Thanks!

Hi there! @Wiselnn570 Thanks for using vLLM and I would suggest you open a separate issue for your question as it seems related to Qwen2VL in particular. Thanks for your understanding!

Hi there! @Wiselnn570 Thanks for using vLLM and I would suggest you open a separate issue for your question as it seems related to Qwen2VL in particular. Thanks for your understanding!

@ywang96 Thanks for your reply, the issue is here https://github.com/vllm-project/vllm/issues/9965

Friendly bump to the thread: It's been a while since our last update, and we have just planned out the roadmap with items we will work on in the next a few months. Please check it out in the OP of this issue!
As always, feedbacks and contributions are very welcomed!


Container IP not displayed on the container info/stats page

**Rancher Version:**v1.2.0-pre4-rc4
**Docker Version:**1.12.2
**Setup Details: (single node rancher vs. HA rancher, internal DB vs. external DB)**Single Node
Environment Type: (Cattle/Kubernetes/Swarm/Mesos): Cattle
Steps to Reproduce:

Create a Cattle environment
Create a service with one or more containers. Click on the container and the container info page is displayed. The container IP is not displayed in this page.

Expected:
Container IP should be displayed.

Version - master 10/20
Verified fixed


Add highlights to div blocks in Code Editor

Attempts to fix #1845, raised originally on reddit, which requests that various elements in the code editor be given further visual distinction beyond just text color.  Adding a light background color behind the lines within divs/blocks was the suggestion solution.
This PR is my attempt at that.  Right now, there are still some parts of it that need work, but I think the bones are there.  It adds to the growing highlightCustomMarkdown() function, and runs a list of defined block types (currently just Note, Descriptive, and ToC) through every line containing {{ to see if they exist....then if so, applies a matching class name.  The classes are styled in the editor.less stylesheet.  It continues to apply that class until either A) a closing }} occurs, or B) another block starts ({{).    If a block exists without the defined words, it applies a generic class "blockHighlight".

Current shortcomings include styles not continuing after the end of a block nested within another.  Also, the defined keywords should likely be matched at the beginning of the string ({{note... vs {{red,note...) to avoid conflicts as well as give a user some choice about what color is applied to the block in the editor.
I know in the issue #1845 this was possibly backburnered until after editor themes and other higher priority changes, so I'm totally fine with this PR being closed-- I just wanted to try it out, and get it written down.   If it works for the future, it can be reopened (or scrapped for parts).

With latest commit:

Side note, this might have the added benefit of making it obvious when closing brackets are not included.

I'm going to close this for now. Let's have further discussion on #1845 so we can lay out a specific set of behaviors we want this feature to have. This PR looks promising but I think we need to develop the idea a bit more first.


[iwara] Add updater to template

Please follow the guide below

You will be asked some questions, please read them carefully and answer honestly
Put an x into all the boxes [ ] relevant to your pull request (like that [x])
Use Preview tab to see how your pull request will actually look like

Before submitting a pull request make sure you have:

[ ] Searched the bugtracker for similar pull requests
[ ] Read adding new extractor tutorial
[ ] Read youtube-dl coding conventions and adjusted the code to meet them
[ ] Covered the code with tests (note that PRs without tests will be REJECTED)
[ ] Checked the code with flake8

In order to be accepted and merged into youtube-dl each piece of code must be in public domain or released under Unlicense. Check one of the following options:

[ ] I am the original author of this code and I am willing to release it under Unlicense
[ ] I am not the original author of this code but it is in public domain or released under Unlicense (provide reliable evidence)

What is the purpose of your pull request?

[ ] Bug fix
[ ] Improvement
[ ] New extractor
[ ] New feature

Description of your pull request and other information
Explanation of your pull request in arbitrary form goes here. Please make sure the description explains the purpose and effect of your pull request and is worded well enough to be understood. Provide as much context and examples as possible.

Uploader, not updater.

You could update your pull request to mention that it "fixes #24237".


[BUG] Unable to use SET kyuubi. operation. language=scala;

Code of Conduct

[X] I agree to follow this project's Code of Conduct

Search before asking

[X] I have searched in the issues and found no similar issues.

Describe the bug

Affects Version(s)
1.9.0
Kyuubi Server Log Output
No response
Kyuubi Engine Log Output
No response
Kyuubi Server Configurations
No response
Kyuubi Engine Configurations
No response
Additional context
No response
Are you willing to submit PR?

[ ] Yes. I would be willing to submit a PR with guidance from the Kyuubi community to fix.
[ ] No. I cannot submit a PR at this time.

The beeline says: "Connected to: Apache Hive (version 2.3.9)", obviously, Hive does not support Scala


Encryption Key Resprites

About the PR

Does what it says on the tin.
Media

[X] I have added screenshots/videos to this PR showcasing its changes ingame, or this PR does not require an ingame showcase

Changelog

:cl: Potato1234_x

tweak: Resprited encryption keys

worth a shot


How to use the framework like a *.lib file to compile inside another project?

Greetings again to all!
I would like to know how can I make a DLL in which CppMicroservice is compiled instead of having to depend on the external CppMicroservice.dll file?
Right now I created a Adobe Native Extension (ANE) which is compiled as a DLL but unfortunately CppMicroservice.dll needs to either be in the System32 folder or in the main application path.
My ANE is not able to initialize unless those previous conditions are fullfiled. How can I make it that it get compiled inside of my ANE instead of being external?
Thank you.
Best Regards

@gdeverlant You can build CppMicroServices as a static library and link that into your ANE. Please see http://docs.cppmicroservices.org/en/stable/framework/doc/static_bundles.html for more details.
As an alternative to static libraries, what about adding the CppMicroServices DLL to the main application path? Is this not possible?

As an alternative to static libraries, what about adding the CppMicroServices DLL to the main application path? Is this not possible?

It is possible but the ANE is not deployed at the same place as the main exe program and because of that both are just ending at 2 different place. I don't want to have to deal with those details that's why.

Ok I was able to install with CMake and Visual Studio the CppMicroServices libraries in the C:\Program Files (x86)\CppMicroServices folder. Now when I try to compile my project with the correct includes and libs pointing to that folder I get tons of errors.

It seems that the includes paths allowed in my source code are :
#include <cppmicroservices4/cppmicroservices/Bundle.h>
#include <cppmicroservices4/cppmicroservices/BundleContext.h>

But what is expected by CppMicroServices is:
#include <cppmicroservices/Bundle.h>
#include <cppmicroservices/BundleContext.h>

What vcpkg installed is as follow:
C:\vcpkg\packages\cppmicroservices_x86-windows\include\cppmicroservices4\cppmicroservices

How do I set CMake to output only C:\vcpkg\packages\cppmicroservices_x86-windows\include\cppmicroservices and not that weird path with the version?

I think what you want to do is add C:\vcpkg\packages\cppmicroservices_x86-windows\include\cppmicroservices4 to the header include search path for your project. This way when CMake compiles your code it will look for header files in C:\vcpkg\packages\cppmicroservices_x86-windows\include\cppmicroservices4, which should resolve such that #include <cppmicroservices/Bundle.h> is correct. Check the

Your message is incomplete...

whoops, sorry, ignore "Check the".


add usd synth amount

Can we add synth_amount and synth_amount_usd these 2 fields to the testing yml file?


Bug: Expanding multiple configurations causing a blank screen

Issue:
Expanding multiple configurations in MP, Test, etc and selecting a configuration from the first location causes a blank screen
Steps to reproduce:
Go to monitoring plans, QA, Cert and TEE, evaluations, or export.
Expand Barry and then expand Chickasaw
Selecting a configuration from Barry produces a blank screen
Testing
Tester: @JanellC

Bug resolved. Able to expand multiple configs


VHACD does not compile with musl libc

Godot version: 3bd49dab
OS: Alpine Linux (Docker)
Issue description:
Godot Engine does not compile on Alpine (or other musl-based systems) due to the use of PTHREAD_MUTEX_RECURSIVE_NP in the third-party VHACD module. This appears to be a glibc-specific macro.
Here is the relevant log output:
[Initial build] Compiling ==> thirdparty/vhacd/src/VHACD.cpp
In file included from thirdparty/vhacd/inc/btScalar.h:190,
                 from thirdparty/vhacd/inc/btAlignedAllocator.h:23,
                 from thirdparty/vhacd/inc/btAlignedObjectArray.h:19,
                 from thirdparty/vhacd/inc/btConvexHullComputer.h:18,
                 from thirdparty/vhacd/src/VHACD.cpp:30:
thirdparty/vhacd/inc/vhacdMutex.h: In constructor 'VHACD::Mutex::Mutex()':
thirdparty/vhacd/inc/vhacdMutex.h:97:60: error: 'PTHREAD_MUTEX_RECURSIVE_NP' was not declared in this scope
         VHACD_VERIFY(pthread_mutexattr_settype(&mutexAttr, PTHREAD_MUTEX_RECURSIVE_NP) == 0);
                                                            ^~~~~~~~~~~~~~~~~~~~~~~~~~
thirdparty/vhacd/inc/vhacdMutex.h:97:9: note: in expansion of macro 'VHACD_VERIFY'
         VHACD_VERIFY(pthread_mutexattr_settype(&mutexAttr, PTHREAD_MUTEX_RECURSIVE_NP) == 0);
         ^~~~~~~~~~~~
thirdparty/vhacd/inc/vhacdMutex.h:97:60: note: suggested alternative: 'PTHREAD_MUTEX_RECURSIVE'
         VHACD_VERIFY(pthread_mutexattr_settype(&mutexAttr, PTHREAD_MUTEX_RECURSIVE_NP) == 0);
                                                            ^~~~~~~~~~~~~~~~~~~~~~~~~~
thirdparty/vhacd/inc/vhacdMutex.h:97:9: note: in expansion of macro 'VHACD_VERIFY'
         VHACD_VERIFY(pthread_mutexattr_settype(&mutexAttr, PTHREAD_MUTEX_RECURSIVE_NP) == 0);
         ^~~~~~~~~~~~
scons: *** [thirdparty/vhacd/src/VHACD.x11.tools.64.o] Error 1
scons: building terminated because of errors.

Same file, a bit higher up:
https://github.com/godotengine/godot/blob/3bd49dabfa909187d514e018ff7c60339e343c71/thirdparty/vhacd/inc/vhacdMutex.h#L74-L76
I guess it tries to detect clang by checking for __APPLE__, but it evidently isn't enough?

Pull request opened upstream: https://github.com/kmammou/v-hacd/pull/70

Something similar happens when cross-compiling from linux to windows with mingw.
thirdparty/vhacd/src/VHACD-ASYNC.cpp:307:7: error: 'thread' in namespace 'std' does not name a type
  307 |  std::thread      *mThread{ nullptr };
      |       ^~~~~~
thirdparty/vhacd/src/VHACD-ASYNC.cpp:10:1: note: 'std::thread' is defined in header '<thread>'; did you forget to '#include <thread>'?
    9 | #include <float.h>
  +++ |+#include <thread>
   10 |
thirdparty/vhacd/src/VHACD-ASYNC.cpp:315:15: error: 'mutex' in namespace 'std' does not name a type
  315 |  mutable std::mutex      mMessageMutex;
      |               ^~~~~
thirdparty/vhacd/src/VHACD-ASYNC.cpp:10:1: note: 'std::mutex' is defined in header '<mutex>'; did you forget to '#include <mutex>'?
    9 | #include <float.h>
  +++ |+#include <mutex>
   10 |
[ 50%] thirdparty/vhacd/src/VHACD-ASYNC.cpp: In member function 'virtual bool VHACD::MyHACD_API::Compute(const double*, uint32_t, const uint32_t*, uint32_t, const VHACD::IVHACD::Parameters&)':
thirdparty/vhacd/src/VHACD-ASYNC.cpp:53:3: error: 'mThread' was not declared in this scope; did you mean 'fread'?
   53 |   mThread = new std::thread([this, countPoints, countTriangles, _desc]()
      |   ^~~~~~~
      |   fread
thirdparty/vhacd/src/VHACD-ASYNC.cpp:53:17: error: expected type-specifier
   53 |   mThread = new std::thread([this, countPoints, countTriangles, _desc]()
      |                 ^~~
thirdparty/vhacd/src/VHACD-ASYNC.cpp:57:4: error: expected primary-expression before ')' token
   57 |   });
      |    ^
thirdparty/vhacd/src/VHACD-ASYNC.cpp: In member function 'virtual void VHACD::MyHACD_API::Cancel()':
thirdparty/vhacd/src/VHACD-ASYNC.cpp:166:7: error: 'mThread' was not declared in this scope; did you mean 'fread'?
  166 |   if (mThread)
      |       ^~~~~~~
      |       fread
thirdparty/vhacd/src/VHACD-ASYNC.cpp:169:11: error: type '<type error>' argument given to 'delete', expected pointer
  169 |    delete mThread;
      |           ^~~~~~~
thirdparty/vhacd/src/VHACD-ASYNC.cpp: In member function 'virtual void VHACD::MyHACD_API::Update(double, double, double, const char*, const char*)':
thirdparty/vhacd/src/VHACD-ASYNC.cpp:235:3: error: 'mMessageMutex' was not declared in this scope; did you mean 'mMessage'?
  235 |   mMessageMutex.lock();
      |   ^~~~~~~~~~~~~
      |   mMessage
thirdparty/vhacd/src/VHACD-ASYNC.cpp: In member function 'virtual void VHACD::MyHACD_API::Log(const char*)':
thirdparty/vhacd/src/VHACD-ASYNC.cpp:247:3: error: 'mMessageMutex' was not declared in this scope; did you mean 'mMessage'?
  247 |   mMessageMutex.lock();
      |   ^~~~~~~~~~~~~
      |   mMessage
thirdparty/vhacd/src/VHACD-ASYNC.cpp: In member function 'void VHACD::MyHACD_API::processPendingMessages() const':
thirdparty/vhacd/src/VHACD-ASYNC.cpp:267:4: error: 'mMessageMutex' was not declared in this scope; did you mean 'mMessage'?
  267 |    mMessageMutex.lock();
      |    ^~~~~~~~~~~~~
      |    mMessage
thirdparty/vhacd/src/VHACD-ASYNC.cpp:275:4: error: 'mMessageMutex' was not declared in this scope; did you mean 'mMessage'?
  275 |    mMessageMutex.lock();
      |    ^~~~~~~~~~~~~
      |    mMessage

@chiguireitor That's a different issue though. What version of MinGW do you use? It might be too old, we use mingw-w64 6.0.0 with GCC 9.1 successfully to build official Godot binaries for Windows from Linux.

i just followed instructions to install it on ubuntu 19.10.... i'm away from my build box, will check back when i'm at home.
If it makes a difference, it's using the latest master head, so stuff could be broken.

Ok, finally made it work, had to switch alternative mingw from the -windows suffix to the -posix suffix, should be noted on the documentation tho

The original issue has not been fixed. I just tried compiling on Alpine Linux and it does not work.

Did you try using posix threads? In ubuntu i know there's the update-alternative utilities, but dunno how it goes in Alpine.... also, Alpine uses Musl instead of Libc, so you might be having a slight incompatibility there.

Yes, the incompatibility is caused by Alpine's use of musl. The pull request above fixes the problem, but it needs to also be in Godot's copy.

@chiguireitor Seems to be https://github.com/godotengine/godot/issues/40853 now


Support for template files organized in directories

The current implementation does not support templates organized in directories,
e.g. the following doesn't work:
try response.render("accounts/account.stencil", context: context).end()

This is a serious limitation.

@gmosx For me it works - please see the test in https://github.com/IBM-Swift/Kitura-Sample/pull/71

After the related issue #16 was fixed I don't have any further problems.


Duplicate "Experiences" section

Hello,
First of all, thank you very much for this great theme!
However, I have a question: I have a lot of volunteer activities and I would like to be able to separate them from my professional experiences. I think the cleanest way to do this is to duplicate the "Experiences" section. Is this possible? If so, how do I do it?
Thanks!

Just add another data file for new experience section in your data folder. For example, add volunteer-activities.yaml
# section information
section:
  name: Volunteer Activites
  id: volunteer-activites
  template: sections/experiences.html
  enable: true
  weight: 4
  showOnNavbar: true

# Your experiences
experiences:
# Your experiences here

Thanks a lot for the light-speed answer!
Just tried it, worked as expected! I thought it would require more modifications but it did not. Again, what a great theme :)


Cannot add new Sorter to QueryContainer

Currently QueryContainer only accepts an instance of Filter:
 public function add(Filter $filter)
    {
        $this->filters[] = $filter;

        return $this;
    }

But it should accept instances of Filter and also Sorter.
Instead of accepting both instances, we need to create two methods - addFilter and addSorter. This way we don't have to check what was sent and the code will be more readable.

@adrian1358 This was fixed by simply changing parameter type to EntityQueryManipulator since both Filter and Sorter extend that base class. At this point there is no need for new methods. Cheers!

Great @Pavel910 ! Expecting it in the next release ;)