README.md

---
license: gpl-3.0
tags:
- DocVQA
- Document Question Answering
- Document Visual Question Answering
datasets:
- rubentito/mp-docvqa
language:
- en
---

# T5 base fine-tuned on MP-DocVQA

This is [pretrained](https://huggingface.co/t5-base) T5 base fine-tuned on Multipage DocVQA (MP-DocVQA) dataset.


This model was used as a baseline in [Hierarchical multimodal transformers for Multi-Page DocVQA](https://arxiv.org/pdf/2212.05935.pdf).
- Results on the MP-DocVQA dataset are reported in Table 2.
- Training hyperparameters can be found in Table 8 of Appendix D.


## How to use

Here is how to use this model to get the features of a given text in PyTorch:

```python
import torch
from transformers import T5Tokenizer, T5ForConditionalGeneration

tokenizer = LongformerTokenizerFast.from_pretrained("rubentito/t5-base-mpdocvqa")
model = LongformerForQuestionAnswering.from_pretrained("rubentito/t5-base-mpdocvqa")

context = "Huggingface has democratized NLP. Huge thanks to Huggingface for this."
question = "What has Huggingface done?"
input_text = "question: {:s}  context: {:s}".format(question, context)

encoding = tokenizer(input_text, return_tensors="pt")
output = self.model.generate(**encoding)
answer = tokenizer.decode(output['sequences'], skip_special_tokens=True)
```


## Model results

Extended experimentation can be found in Table 2 of [Hierarchical multimodal transformers for Multi-Page DocVQA](https://arxiv.org/pdf/2212.05935.pdf).
You can also check the live leaderboard at the [RRC Portal](https://rrc.cvc.uab.es/?ch=17&com=evaluation&task=4).
| Model 		 																	| HF name								| ANLS 			| APPA		|
|-----------------------------------------------------------------------------------|:--------------------------------------|:-------------:|:---------:|
| [Bert-large](https://huggingface.co/rubentito/bert-large-mpdocvqa)	            | rubentito/bert-large-mpdocvqa			| 0.4183 		| 51.6177 	|
| [Longformer-base](https://huggingface.co/rubentito/longformer-base-mpdocvqa)		| rubentito/longformer-base-mpdocvqa	| 0.5287		| 71.1696 	|
| [BigBird ITC base](https://huggingface.co/rubentito/bigbird-base-itc-mpdocvqa)	| rubentito/bigbird-base-itc-mpdocvqa	| 0.4929		| 67.5433 	|
| [LayoutLMv3 base](https://huggingface.co/rubentito/layoutlmv3-base-mpdocvqa)		| rubentito/layoutlmv3-base-mpdocvqa	| 0.4538		| 51.9426 	|
| [**T5 base**](https://huggingface.co/rubentito/t5-base-mpdocvqa)					| rubentito/t5-base-mpdocvqa			| 0.5050		| 0.0000 	|
| Hi-VT5 																			| TBA 									| 0.6201		| 79.23		|

## Citation Information 

```tex
@article{tito2022hierarchical,
  title={Hierarchical multimodal transformers for Multi-Page DocVQA},
  author={Tito, Rub{\`e}n and Karatzas, Dimosthenis and Valveny, Ernest},
  journal={arXiv preprint arXiv:2212.05935},
  year={2022}
}
```