AIDC-AI/Ovis2.5-9B · Fine-Tuning?

5 days ago

Can it be fine tuned same way as Qwen 2.5 VL with sfttrainer and peft, and how is it actually related to Qwen 2.5 VL?

Vil

4 days ago

Trying to fine-tune with peft on model.llm gives me an error

  File "/home/.../huggingface/modules/transformers_modules/AIDC-AI/Ovis2.5-9B/.../modeling_ovis2_5.py", line 778, in forward
    inputs_embeds = self.merge_multimodal(
                    ^^^^^^^^^^^^^^^^^^^^^^
  File "/home/.../huggingface/modules/transformers_modules/AIDC-AI/Ovis2.5-9B/.../modeling_ovis2_5.py", line 802, in merge_multimodal
    multimodal_embeds[input_ids == indicator_id] = visual_indicator_embeds[i]
    ~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^
RuntimeError: a leaf Variable that requires grad is being used in an in-place operation.

hanshupe

4 days ago

I have some issues too, if someone got it working, please let us know.

xxyyy123

AIDC-AI org 4 days ago

@hanshupe @Vil Hi~ The official code already supports fine-tuning, see here:
https://github.com/AIDC-AI/Ovis?tab=readme-ov-file#model-fine-tuning