README.md

# Encoder-Decoder model with DeBERTa decoder

## pre-trained models

Encoder: `microsoft/deberta-v3-small`

Decoder: `deliciouscat/deberta-v3-base-decoder-v0.1`; 6 transformer layers, 8 attention heads

## Data used

`HuggingFaceFW/fineweb` -> sampled 124800

## Training hparams

optimizer: AdamW, lr=2.3e-5, betas=(0.875, 0.997)
batch size: 12 (maximal on Colab pro A100 env)

## How to use

```
from transformers import AutoTokenizer, EncoderDecoderModel

model = EncoderDecoderModel.from_pretrained("patrickvonplaten/bert2bert_cnn_daily_mail")
tokenizer = AutoTokenizer.from_pretrained("patrickvonplaten/bert2bert_cnn_daily_mail")
```

## Future work!

train more scientific data

fine-tune on keyword extraction task