How to train it with qlora?
This was the code:
from peft import prepare_model_for_kbit_training
model.gradient_checkpointing_enable()
model = prepare_model_for_kbit_training(model)
This was the error:
╭─────────────────────────────── Traceback (most recent call last) ────────────────────────────────╮
│ in <cell line: 3>:3 │
│ │
│ /usr/local/lib/python3.10/dist-packages/transformers/modeling_utils.py:1624 in │
│ gradient_checkpointing_enable │
│ │
│ 1621 │ │ activations". │
│ 1622 │ │ """ │
│ 1623 │ │ if not self.supports_gradient_checkpointing: │
│ ❱ 1624 │ │ │ raise ValueError(f"{self.class.name} does not support gradient check │
│ 1625 │ │ self.apply(partial(self._set_gradient_checkpointing, value=True)) │
│ 1626 │ │
│ 1627 │ def gradient_checkpointing_disable(self): │
╰──────────────────────────────────────────────────────────────────────────────────────────────────╯
ValueError: MPTForCausalLM does not support gradient checkpointing.