Update README.md
Browse files
README.md
CHANGED
|
@@ -26,13 +26,13 @@ pip install promptcap
|
|
| 26 |
|
| 27 |
## Captioning Pipeline
|
| 28 |
|
| 29 |
-
|
| 30 |
-
Generate a prompt-guided caption by following
|
| 31 |
```python
|
| 32 |
import torch
|
| 33 |
from promptcap import PromptCap
|
| 34 |
|
| 35 |
-
model = PromptCap("vqascore/promptcap-coco-vqa") # also support OFA checkpoints. e.g. "OFA-Sys/ofa-
|
| 36 |
|
| 37 |
if torch.cuda.is_available():
|
| 38 |
model.cuda()
|
|
@@ -47,7 +47,7 @@ To try generic captioning, just use "please describe this image according to the
|
|
| 47 |
|
| 48 |
PromptCap also support taking OCR inputs:
|
| 49 |
|
| 50 |
-
```
|
| 51 |
prompt = "please describe this image according to the given question: what year was this taken?"
|
| 52 |
image = "dvds.jpg"
|
| 53 |
ocr = "yip AE Mht juor 02/14/2012"
|
|
@@ -62,7 +62,7 @@ print(model.caption(prompt, image, ocr))
|
|
| 62 |
Different from typical VQA models, which are doing classification on VQAv2, PromptCap is open-domain and can be paired with arbitrary text-QA models.
|
| 63 |
Here we provide a pipeline for combining PromptCap with UnifiedQA.
|
| 64 |
|
| 65 |
-
```
|
| 66 |
import torch
|
| 67 |
from promptcap import PromptCap_VQA
|
| 68 |
|
|
@@ -80,7 +80,7 @@ print(vqa_model.vqa(question, image))
|
|
| 80 |
|
| 81 |
Similarly, PromptCap supports OCR inputs
|
| 82 |
|
| 83 |
-
```
|
| 84 |
question = "what year was this taken?"
|
| 85 |
image = "dvds.jpg"
|
| 86 |
ocr = "yip AE Mht juor 02/14/2012"
|
|
@@ -90,7 +90,7 @@ print(vqa_model.vqa(prompt, image, ocr=ocr))
|
|
| 90 |
|
| 91 |
Because of the flexibility of Unifiedqa, PromptCap also supports multiple-choice VQA
|
| 92 |
|
| 93 |
-
```
|
| 94 |
question = "what piece of clothing is this boy putting on?"
|
| 95 |
image = "glove_boy.jpeg"
|
| 96 |
choices = ["gloves", "socks", "shoes", "coats"]
|
|
|
|
| 26 |
|
| 27 |
## Captioning Pipeline
|
| 28 |
|
| 29 |
+
Please follow the prompt format, which will give the best performance.
|
| 30 |
+
Generate a prompt-guided caption by following
|
| 31 |
```python
|
| 32 |
import torch
|
| 33 |
from promptcap import PromptCap
|
| 34 |
|
| 35 |
+
model = PromptCap("vqascore/promptcap-coco-vqa") # also support OFA checkpoints. e.g. "OFA-Sys/ofa-large"
|
| 36 |
|
| 37 |
if torch.cuda.is_available():
|
| 38 |
model.cuda()
|
|
|
|
| 47 |
|
| 48 |
PromptCap also support taking OCR inputs:
|
| 49 |
|
| 50 |
+
```python
|
| 51 |
prompt = "please describe this image according to the given question: what year was this taken?"
|
| 52 |
image = "dvds.jpg"
|
| 53 |
ocr = "yip AE Mht juor 02/14/2012"
|
|
|
|
| 62 |
Different from typical VQA models, which are doing classification on VQAv2, PromptCap is open-domain and can be paired with arbitrary text-QA models.
|
| 63 |
Here we provide a pipeline for combining PromptCap with UnifiedQA.
|
| 64 |
|
| 65 |
+
```python
|
| 66 |
import torch
|
| 67 |
from promptcap import PromptCap_VQA
|
| 68 |
|
|
|
|
| 80 |
|
| 81 |
Similarly, PromptCap supports OCR inputs
|
| 82 |
|
| 83 |
+
```python
|
| 84 |
question = "what year was this taken?"
|
| 85 |
image = "dvds.jpg"
|
| 86 |
ocr = "yip AE Mht juor 02/14/2012"
|
|
|
|
| 90 |
|
| 91 |
Because of the flexibility of Unifiedqa, PromptCap also supports multiple-choice VQA
|
| 92 |
|
| 93 |
+
```python
|
| 94 |
question = "what piece of clothing is this boy putting on?"
|
| 95 |
image = "glove_boy.jpeg"
|
| 96 |
choices = ["gloves", "socks", "shoes", "coats"]
|