|
|
--- |
|
|
license: gemma |
|
|
pipeline_tag: text-to-image |
|
|
tags: |
|
|
- NovelAI |
|
|
--- |
|
|
|
|
|
## Inference |
|
|
|
|
|
```pytorch |
|
|
from transformers.models import AutoTokenizer, T5GemmaEncoderModel |
|
|
import torch |
|
|
|
|
|
if __name__ == '__main__': |
|
|
model = T5GemmaEncoderModel.from_pretrained(t5gemma_path, torch_dtype=torch.bfloat16) |
|
|
tokenizer = AutoTokenizer.from_pretrained(t5gemma_path) |
|
|
inputs = tokenizer('Gemma', max_length=512, padding='max_length', truncation=True, return_tensors='pt') |
|
|
output = model.forward(**inputs).last_hidden_state |
|
|
``` |
|
|
|
|
|
## SD1.5 and Gemma |
|
|
|
|
|
```pytorch |
|
|
from diffusers import StableDiffusionPipeline |
|
|
from gemma_encoder import Encoder |
|
|
|
|
|
if __name__ == '__main__': |
|
|
pipeline = StableDiffusionPipeline.from_pretrained('NovelAI/nai-anime-v2') |
|
|
pipeline.enable_model_cpu_offload() |
|
|
encoder = Encoder(adapter_model, t5gemma_path, device='cpu') |
|
|
load_model(adapter_model, 'adapter.safetensors') |
|
|
image = pipeline( |
|
|
None, |
|
|
prompt_embeds=encoder.encode(pipeline, text).to('cpu'), |
|
|
negative_prompt='bad quality, low quality, worst quality' |
|
|
).images[0] |
|
|
image.save('preview.png') |
|
|
``` |
|
|
|
|
|
## Datasets |
|
|
|
|
|
- alfredplpl/artbench-pd-256x256 |
|
|
- danbooru2023-florence2-caption (verb, action clauses) |
|
|
- spatial-caption |
|
|
- SPRIGHT-T2I/spright_coco |
|
|
- sugarquark/colormix (synthetic color, fashion dataset) |
|
|
- trojblue/danbooru2025-metadata |
|
|
|
