README.md

Encoder-Decoder model with DeBERTa decoder

pre-trained models

Encoder: microsoft/deberta-v3-small

Decoder: deliciouscat/deberta-v3-base-decoder-v0.1; 6 transformer layers, 8 attention heads

Data used

HuggingFaceFW/fineweb -> sampled 124800

Training hparams

optimizer: AdamW, lr=2.3e-5, betas=(0.875, 0.997) batch size: 12 (maximal on Colab pro A100 env)

How to use

from transformers import AutoTokenizer, EncoderDecoderModel

model = EncoderDecoderModel.from_pretrained("patrickvonplaten/bert2bert_cnn_daily_mail")
tokenizer = AutoTokenizer.from_pretrained("patrickvonplaten/bert2bert_cnn_daily_mail")

Future work!

train more scientific data

fine-tune on keyword extraction task