Krea-2-Engineer V1 — experimental encoder-only finetune of the Krea-2 Qwen3-VL text encoder

.gitattributes

@@ -33,3 +33,6 @@ saved_model/**/* filter=lfs diff=lfs merge=lfs -text
 *.zip filter=lfs diff=lfs merge=lfs -text
 *.zst filter=lfs diff=lfs merge=lfs -text
 *tfevents* filter=lfs diff=lfs merge=lfs -text
+evidence/Krea2-Engineer-V1_AB_50.png filter=lfs diff=lfs merge=lfs -text
+evidence/krea2_ab_sheet_v1.png filter=lfs diff=lfs merge=lfs -text
+LICENSE.pdf filter=lfs diff=lfs merge=lfs -text

Krea2-Engineer-V1-bf16.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:06bef753117e22c1afa721d387bb9154fcbad2a4eb6a7eac192221d1396c46ab
+size 8875713440

LICENSE.pdf

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:b82a2805162bde714a4eb27b9063c4fc3345d08a30be055134a6160e5430ba74
+size 137711

NOTICE

@@ -0,0 +1,5 @@
+Krea 2 is licensed under the Krea 2 Community License Agreement. For more information, visit https://krea.ai/krea-2-licensing.
+
+This model (Krea-2-Engineer-V1) is a Derivative of krea/Krea-2-Turbo: the text encoder has been modified (fine-tuned with SMART DoRA on the Z-Image-Engineer V7 corpus) by BennyDaBall. Weights have been modified from the original. This is not an official Krea product and is not endorsed by Krea.
+
+Use of this model and its derivatives remains subject to the Krea 2 Community License Agreement (see LICENSE.pdf), including its commercial-use revenue threshold and content-filtering requirements.

README.md

@@ -0,0 +1,195 @@
+---
+license: other
+license_name: krea-2-community-license
+license_link: https://huggingface.co/BennyDaBall/Krea-2-Engineer-V1/blob/main/LICENSE.pdf
+language:
+  - en
+base_model:
+  - krea/Krea-2-Turbo
+pipeline_tag: text-generation
+tags:
+  - krea
+  - krea-2
+  - text-encoder
+  - qwen3-vl
+  - image-generation
+  - prompt-engineering
+  - dora
+  - smart-dora
+  - comfyui
+  - lm-studio
+  - experimental
+---
+
+# Krea-2-Engineer V1 (4B)
+
+**Follow me on X [@BennyDaBall_OG](https://x.com/BennyDaBall_OG) !**
+
+> **Experimental — read this first.** This is **V1**: a first, purely experimental swing at the Krea-2 line. It fine-tunes **only the text encoder** of Krea-2 — no DiT, no VAE, no diffusion backbone touched. The goal isn't better *text rendering*; it's a **better image** out the other end — richer composition, lighting, and material detail from the exact same prompt and seed. It is not a finished product. It's me changing the smallest possible part of the model and measuring whether the pictures got better.
+
+## Model Metadata
+
+| Field | Value |
+|---|---|
+| License | `other` — Krea 2 Community License Agreement (see `LICENSE.pdf`) |
+| Base Model | [`krea/Krea-2-Turbo`](https://huggingface.co/krea/Krea-2-Turbo) (text encoder) |
+| Architecture | Qwen3-VL ~4B text tower (`Qwen3VLModel`), language tower only |
+| Method | SMART DoRA (r64 / a64), differentiable drift hinge |
+| Use | ComfyUI text encoder (`CLIPLoader`, type `krea2`) · chat GGUF companion |
+| Format | Merged bf16 safetensors |
+| Status | **V1 — experimental, text-encoder-only** |
+
+---
+
+A text encoder is the part of a diffusion model that turns your words into the numbers the image model actually listens to. **Krea-2-Engineer** retrains *just that part* of Krea-2 — leaving the renderer, the VAE, and every other weight exactly as Krea shipped them — so the same prompt lands as a more composed, better-lit, more material-aware image. Same model, same sampler, same seed; the only thing that changed is how the prompt gets *heard*.
+
+It's the model whispering to itself, which is legally distinct from sorcery.
+
+![Krea-2-Engineer V1 — base vs V1 A/B contact sheet](evidence/krea2_ab_sheet_v1.png)
+
+*10 prompts, same seed per row. Left column: the stock Krea-2 encoder. Right column: Krea-2-Engineer V1. Nothing else in the pipeline changed.*
+
+## What is Krea-2-Engineer?
+
+It is a DoRA fine-tune of the **Qwen3-VL text encoder** that ships inside [`krea/Krea-2-Turbo`](https://huggingface.co/krea/Krea-2-Turbo), trained on the **Z-Image-Engineer V7** corpus. The diffusion transformer, the VAE, and the rest of Krea-2 are untouched — this is a drop-in replacement for *one component*.
+
+Because that encoder is a real Qwen3-VL model under the hood, it has a second life: you can also **talk to it / run it as a prompt writer** in LM Studio (a GGUF build is on the way — see below).
+
+### Key Use Cases
+
+- **Drop-in image enhancement** — swap it in for the stock Krea-2 text encoder in ComfyUI and render as normal. Same prompt, more composed image.
+- **Prompt writer** — it was trained on the Engineer prompt-rewrite corpus, so it can also expand `a rock` into a composed, material-aware prompt instead of the usual `very detailed rock, best quality, please clap` sludge.
+- **A base to iterate on** — V1 is a clean encoder-only baseline. DPO and image-grounded GRPO stages are the obvious next moves.
+
+---
+
+## Under the Hood: SMART DoRA
+
+The encoder was tuned with the same training system that built the Z-Image-Engineer line, ported to Krea's Qwen3-VL encoder. Two forces do the work, plus four light-touch regularizers.
+
+**The writer loss** teaches the model to produce richer conditioning. **The drift hinge** is the anti-forgetting leash — a per-token cosine constraint that lets the encoder change freely *as long as* each token's representation stays inside a cone around the original Krea weights. It can learn, but it can't wander off and lobotomize the base model. Across this entire run the mean cosine to base held at **0.984** (margin `0.92`) — a precise tune, not a blowout.
+
+| Regularizer | What it Does | Why it Matters |
+|---|---|---|
+| **Entropic** | Watches the output-token entropy vs a knowledge-mass estimate | Keeps the conditioning confident, not mushy |
+| **Holographic** | Shapes how information spreads across the encoder's depth | Keeps the representation well-conditioned layer-to-layer |
+| **Topological** | Constrains the token-to-token manifold geometry | Stops the token relationships from collapsing |
+| **Manifold** | Light variance penalty directly on the adapter matrices | Keeps the DoRA weights healthy |
+
+All four run as **frozen, fixed-shape penalties** (their internal heads do not train) — they nudge the encoder without becoming a second thing to babysit.
+
+### The Refinement Pipeline
+
+1. Load the stock Krea-2 Qwen3-VL text encoder (language tower only; the vision tower is loaded but frozen and unused for T2I).
+2. Attach SMART DoRA to the text-tower projections (`q/k/v/o/gate/up/down`). Nothing else is trainable.
+3. Train one epoch on the V7 corpus: `writer CE + 0.5·drift_hinge + SMART`.
+4. Merge the adapter and export bf16 for `transformers`, plus a key-remapped `krea2` encoder for ComfyUI.
+
+---
+
+## Quick Start
+
+### ComfyUI (image enhancement — the main event)
+
+Drop `Krea2-Engineer-V1-bf16.safetensors` into `ComfyUI/models/text_encoders/`, then in your Krea-2 Turbo workflow point the **`CLIPLoader`** at it with **`type: krea2`**. Everything else stays stock.
+
+### Verified Image Settings
+
+```text
+Diffusion Model : krea2_turbo_fp8_scaled.safetensors   (UNETLoader, weight_dtype default)
+Text Encoder    : Krea2-Engineer-V1-bf16.safetensors   (CLIPLoader, type: krea2)
+VAE             : qwen_image_vae.safetensors
+Resolution      : 1024 x 1024
+Steps           : 8
+CFG             : 1.0
+Sampler         : euler
+Scheduler       : simple
+Negative        : (none — Turbo runs at CFG 1.0)
+```
+
+### LM Studio (talk to it / prompt writer)
+
+Because the encoder is a real Qwen3-VL model, it also runs as a **chat / prompt-writer LLM**. Grab the `Q8_0` GGUF from the companion repo [`BennyDaBall/Krea-2-Engineer-V1-GGUF`](https://huggingface.co/BennyDaBall/Krea-2-Engineer-V1-GGUF), load it in LM Studio, and paste the prompt-writer **system prompt** (in the companion card). Feed it a lazy seed like `a rock` and it returns one composed, cinematic image prompt — no tag soup.
+
+---
+
+## Training Facts
+
+I believe in open science, which is just a fancy way of saying "show the receipts."
+
+**Hardware**
+
+- Trained locally on a single **RTX 5090 (32GB)**.
+- PyTorch `2.10.0+cu130`, CUDA `13.0`, `transformers` 5.7, `peft` 0.19.1.
+
+**Dataset**
+
+- Z-Image-Engineer **V7** corpus.
+- Train rows: `74,540` · Eval rows: `1,400`.
+
+**Configuration**
+
+| Parameter | Specification |
+|---|---|
+| Base text encoder | `krea/Krea-2-Turbo/text_encoder` (Qwen3-VL ~4B) |
+| Tokenizer | `krea/Krea-2-Turbo/tokenizer` (Qwen2Tokenizer, ChatML) |
+| Adapter | SMART DoRA, rank `64` / alpha `64` / dropout `0.03`, stable-dora |
+| Target modules | `q_proj k_proj v_proj o_proj gate_proj up_proj down_proj` (text tower only) |
+| Conditioning tap | penultimate hidden layer (`hidden_states[-2]`) |
+| Drift hinge | cosine margin `0.92`, weight `0.5`, every 2 micro-steps |
+| Schedule | 1 epoch · `4,659` steps · batch `2` × grad-accum `8` · lr `1e-4` cosine |
+| Wall-clock | ~15.5 h |
+| Final eval loss | `1.150` (from `1.317` at the first eval) |
+| Final drift mean-cosine | `0.984` (held inside the cone the entire run) |
+
+The vision tower of the Qwen3-VL encoder is never trained or used — this is a pure text-conditioning fine-tune.
+
+---
+
+## GGUF — Chat / Prompt Writer
+
+The Qwen3-VL text tower is repacked as a standard **Qwen3** model and quantized, so it loads in **stock LM Studio** with no special build.
+
+- **Companion repo:** [`BennyDaBall/Krea-2-Engineer-V1-GGUF`](https://huggingface.co/BennyDaBall/Krea-2-Engineer-V1-GGUF)
+- **Quant:** `Q8_0` (~4 GB, near-lossless). More quants on request.
+- Use it as a **prompt enhancer** — feed a lazy seed, get a composed cinematic prompt back, then paste that into your image workflow.
+
+---
+
+## Verification & Proof
+
+The A/B contact sheet above (`evidence/krea2_ab_sheet_v1.png`) is the honest evidence: **10 prompts, fixed seed per row, only the text encoder swapped** between columns (stock Krea-2 encoder vs V1). No cherry-picking the sampler, no different seeds, no rewrite step — just the encoder. A precision-matched bf16 build of the *un*-trained base was used as the control so the difference is the training, not a quantization artifact.
+
+A broader **50-prompt** A/B sheet (portraits, landscapes, food, animals, products, architecture) is included at [`evidence/Krea2-Engineer-V1_AB_50.png`](evidence/Krea2-Engineer-V1_AB_50.png) — same controls, 100 renders, for the full picture.
+
+This is V1. The differences are real but *measured* — a targeted lift, not a transformation. That's by design: the drift hinge deliberately keeps the encoder close to Krea's, so you do not get a full-epoch run nuking the base model's quality.
+
+---
+
+## Disclaimer & Acknowledgements
+
+This is an **experimental, first-attempt, text-encoder-only** fine-tune. It changes how prompts are encoded; it does **not** retrain Krea-2 itself, and it does not guarantee a perfect seed every single time. Diffusion is still diffusion. Use creative judgment locally.
+
+**This model is a Derivative of `krea/Krea-2-Turbo` — the text encoder has been modified (fine-tuned) by BennyDaBall.** It is not an official Krea product and is not endorsed by Krea.
+
+Thanks to:
+
+- **Krea** for releasing Krea-2 and its weights ([`krea/Krea-2-Turbo`](https://huggingface.co/krea/Krea-2-Turbo)).
+- **Tongyi-MAI** and **Qwen** for the Qwen3-VL text-encoder backbone this all rides on.
+- The open-source maintainers behind **ComfyUI**, **PEFT**, **Transformers**, **llama.cpp**, and **LM Studio**.
+- My local power utility, which now classifies me as a small industrial facility.
+
+## License
+
+Released under the **Krea 2 Community License Agreement** (full text in `LICENSE.pdf`; canonical: <https://www.krea.ai/krea-2-licensing>). As a Derivative of Krea-2, this model carries that license forward. In plain terms:
+
+- **Commercial use is allowed only if your total company-wide annual revenue is under $1,000,000 USD** (trailing twelve months). At or above that, you need an Enterprise License from Krea.
+- If you deploy it, you must implement **reasonable content filtering**, keep the **"Krea"** name prefix, ship the agreement + the notice below, **state that the model was modified**, and **not** relicense the Krea-derived weights under a more permissive license.
+
+> **NOTICE:** Krea 2 is licensed under the Krea 2 Community License Agreement. For more information, visit https://krea.ai/krea-2-licensing.
+
+---
+
+**Built & trained locally with care by BennyDaBall.**
+
+**Follow me on X [@BennyDaBall_OG](https://x.com/BennyDaBall_OG) !**