DarkArtsForge

/

Vesper-Zenith-12B

like 3

Follow

DarkArtsForge 72

One of the models was formatted for tekken while the other 3 were chatML, so this could be the cause of early termination. In addition, the other donors except Raven seem to have some refusals.

Did you manage to test Raven 12B on its own yet to see if that also had early terminations? If not, then the della method might also be partially involved.

I'll have to try running token surgeon or something else to fix this probably.

May have found the culprit

WARNING:mergekit.tokenizer.build:Unable to load tokenizer for B:\12B\Lambent--Arsenic-Shahrazad-12B-v4.4. Assuming same as B:\12B\Vortex5--Ethereal-Stardust-12B.
                                                                                                                                                                                                                                            WARNING:mergekit.tokenizer.build:B:\12B\Lambent--Arsenic-Shahrazad-12B-v4.4 token '<|im_start|>' has index 131072>131071 (padding?)███████████████▉                                                          | 13/20 [00:01<00:00,  9.69it/s]
WARNING:mergekit.tokenizer.build:B:\12B\Lambent--Arsenic-Shahrazad-12B-v4.4 token '<SPECIAL_4>' has index 131074>131071 (padding?)
WARNING:mergekit.tokenizer.build:B:\12B\Lambent--Arsenic-Shahrazad-12B-v4.4 token '<|im_end|>' has index 131073>131071 (padding?)

Yeah, <|im_end|> isn't in the Tekken tokenizer at all; it's generally added by the chatml merges. Not sure if there's anything I can do to help with that on my end, given we've never trained on the tokens

Mergekit is failing to properly merge Arsenic v4.4 because your tokenizer_config.json file is much smaller than usual, only 1KB instead of the normal 178-181KB range. So, when it incorrectly assumes that it can use Ethereal Stardust, it mismaps the extra tokens from Ethereal's tokenizer_config onto Arsenic

 },
    "131072": {
      "content": "<|im_start|>",
      "lstrip": false,
      "normalized": false,
      "rstrip": false,
      "single_word": false,
      "special": true
    },
    "131073": {
      "content": "<|im_end|>",
      "lstrip": false,
      "normalized": false,
      "rstrip": false,
      "single_word": false,
      "special": true
    },
    "131074": {
      "content": "<SPECIAL_4>",
      "lstrip": false,
      "normalized": false,
      "rstrip": false,
      "single_word": false,
      "special": true
    }

I don't know how to generate the correct tokenizer_config but it could be something to do with your finetuning process or your LoRA application script incorrectly omitting the source model's tokenizer json.

In my case, I used axolotl runpod to finetune the LoRA, and a PMPF script to apply it, and my tokenizer_json.configs are 181KB as a result. Not sure if this would help but here is my finetune_adapter_merger.py script

import torch
from peft import PeftModel
from transformers import AutoModelForCausalLM, AutoTokenizer
import os
import shutil

# ================= CONFIGURATION =================
BASE_MODEL_PATH = os.getcwd()              # Where the original safetensors are
ADAPTER_PATH = "finetuned_adapter"         # Where the LoRA files are
NEW_MODEL_PATH = "My_Finetuned_Model_v1"   # The new subfolder for the result
# =================================================

def main():
    print(f"Loading base model from {BASE_MODEL_PATH}...")
    # Load in FP16 or BF16 on CPU (System RAM)
    base_model = AutoModelForCausalLM.from_pretrained(
        BASE_MODEL_PATH,
        low_cpu_mem_usage=True,
        return_dict=True,
        torch_dtype=torch.bfloat16, # Matches your training
        device_map="cpu",           # Force CPU to avoid VRAM OOM
        local_files_only=True
    )

    print(f"Loading adapter from {ADAPTER_PATH}...")
    # 'is_trainable=False' and 'adapter_name="default"' are crucial for merge
    model = PeftModel.from_pretrained(
        base_model, 
        ADAPTER_PATH, 
        adapter_name="default",
        is_trainable=False
    )

    print("Merging weights (this may take a minute)...")
    # This function is what causes the size mismatch if the config doesn't match
    # We will explicitly disable strict loading for the merge
    model.base_model.model.load_state_dict(model.base_model.model.state_dict(), strict=False)
    model = model.merge_and_unload()

    print(f"Saving new model to {NEW_MODEL_PATH}...")
    if not os.path.exists(NEW_MODEL_PATH):
        os.makedirs(NEW_MODEL_PATH)

    model.save_pretrained(NEW_MODEL_PATH, safe_serialization=True, max_shard_size="5GB")
    
    print("Saving tokenizer...")
    tokenizer = AutoTokenizer.from_pretrained(BASE_MODEL_PATH, local_files_only=True)
    tokenizer.save_pretrained(NEW_MODEL_PATH)

    # Copy generation config if it exists
    gen_config = os.path.join(BASE_MODEL_PATH, "generation_config.json")
    if os.path.exists(gen_config):
        shutil.copy(gen_config, os.path.join(NEW_MODEL_PATH, "generation_config.json"))

    print("Merge complete!")
    print(f"You can now use '{NEW_MODEL_PATH}' in Kobold, Ooba, or MergeKit.")

if __name__ == "__main__":
    main()

Looked into this and regenerated https://huggingface.co/Lambent/Arsenic-Shahrazad-12B-v4.4/blob/main/tokenizer_config.json ; similar size to the original Mistral Nemo tokenizer_config.json and your results. I'll hit the rest of the Arsenics as I can. Apparently something in unsloth saved strangely.

Thanks, I'll have to try merging a Vesper Zenith v2 when I have more bandwidth next week to see if this fixes the early termination bugs reported by @redaihf .

Thanks, I'll have to try merging a Vesper Zenith v2

That is appreciated because v1 had potential.

That is appreciated because v1 had potential.

Adversary 8B v1.1 and Wicked Moondust 12B are uploaded. Moondust isn't ablated so may encounter refusals but it shouldn't have early terminations.

I have queued merges for Adversary 8B v1.2, Vesper Zenith 12B v2, Goetia 24B v1.4, and Runic Oarfish v1.3. Goetia in particular has 7 della variants being merged to test a new addition.

I am also testing new prompt & model combinations (GLM 4.5 air is working really well for this) to curate a high quality roleplay dataset which is planned for various sizes. This could take longer, but I have developed a new tool from scratch to make it easier.

One look through a grok dataset and I saw the name "Elara" appeared over 7,000 times. So, all of my Q&A pairs are being manually curated, which takes a lot longer, but prevents a lot of slop and broken logic from poisoning the model. I tasked the latest version of Gemini Pro 3 with creating 100 Q&A pairs and it produced absolute gibberish in comparison to GLM Air 4.5 at Q3_K_M. Most of the synthetic datasets are bloated with so much repetition it basically hard-codes slop into the weights.

Have you used Faker before? You can write a loop that replaces each instance of "Elora" with a randomly generated first name. Use seeds to increase the likelihood of a new name each time.

I'll have to try that sometime, thanks.

2 new versions of Vesper Zenith are uploading. A recreation of v1 with the adjusted tokenizers, and an experimental v1.2 which adds findesiecle along with the broken adversary prototype.

Not sure if these fixed the early termination bugs but I'm hoping so

https://huggingface.co/OccultAI/Vespir-Zenith-12B-v1.1
https://huggingface.co/OccultAI/Vespir-Zenith-12B-v1.2

There are also new merges of Adversary 8B:
https://huggingface.co/OccultAI/Adversary-8B-v1.21 - this one is quite stable and very creative
https://huggingface.co/OccultAI/Adversary-8B-v1.22 - this one is unstable, talks in riddles like a schizo

Unfortunately all the Goetia tests failed so this is paused again. Aether X 24B and Runic Oarfish are merging next.

Aether X 24B took several days to merge but unfortunately is also broken

TELL ME A STORY

ورا sırada Jamacaasadanka Colonialacer suit sırada sırada sırada sırada sırada sırada sırada sırada יכ inter sırada sırada sırada sırada sırada>Details sıradaanstalt sırada sırada sırada sırada sırada buz sırada sırada sıradaprus sırada sırada sırada sırada proportionsورا sırada Jamورا sıradaallo sırada Jamورا sıradaNCوراúra sırada sırada sırada sırada sırada sıradaورا>Details sırada sırada sırada sırada Forestورا sırada Four Jamورا sıradausususususususususususususususususususususususususususususususus