minpeter/HyperCLOVAX-SEED-Text-Instruct-1.5B-hf

Overview

HyperCLOVAX-SEED-Text-Instruct-1.5B is a model developed by NAVER that can understand and generate text. It demonstrates competitive performance on major benchmarks related to Korean language and culture. In addition, it supports a context length of up to 16k tokens, enabling it to handle a wide range of tasks.

Basic Information

• Model Architecture: Transformer-based architecture (Dense Model)
• Number of Parameters: 1.5B
• Input/Output Format: Text / Text (both input and output are in text format)
• Context Length: 16k
• Knowledge Cutoff Date: The model was trained on data prior to August 2024.

Training and Data

The training data for HyperCLOVAX-Seed-Instruct-1.5B consists of diverse sources, including high-quality datasets. The training process was carried out in four main stages: Pretraining Stage 1, where the model learns from a large volume of documents; Pretraining Stage 2, which focuses on additional training with high-quality data; Rejection sampling Fine-Tuning (RFT), aimed at enhancing the model’s knowledge across various domains and its complex reasoning abilities; and Supervised Fine-Tuning (SFT), which improves the model’s instruction-following capabilities. Furthermore, due to the characteristics of smaller models, vulnerability to long-context handling was observed. To address this, reinforcement for long-context understanding was incorporated from the pretraining stages through to the SFT stage, enabling the model to stably support context lengths of up to 16k tokens.

Benchmark

Model	KMMLU (5-shot, acc)	HAE-RAE (5-shot, acc)	CLiCK (5-shot, acc)	KoBEST (5-shot, acc)
HyperCLOVAX-SEED-Text-Base-1.5B	0.4181	0.6370	0.5373	0.6963
HyperCLOVAX-SEED-Text-Instruct-1.5B	0.3933	0.5674	0.4947	0.6490
Qwen2.5-1.5B-instruct	0.3696	0.5160	0.4772	0.5968
gemma-3-1b-it	0.3075	0.3648	0.3724	0.5869

Huggingface Usage Example

from transformers import AutoModelForCausalLM, AutoTokenizer
model = AutoModelForCausalLM.from_pretrained("/path/to/ckpt")
tokenizer = AutoTokenizer.from_pretrained("/path/to/ckpt")

chat = [
  {"role": "tool_list", "content": ""},
  {"role": "system", "content": "- AI 언어모델의 이름은 \"CLOVA X\" 이며 네이버에서 만들었다.\n- 오늘은 2025년 04월 24일(목)이다."},
  {"role": "user", "content": "슈뢰딩거 방정식과 양자역학의 관계를 최대한 자세히 알려줘."},
]

inputs = tokenizer.apply_chat_template(chat, add_generation_prompt=True, return_dict=True, return_tensors="pt")
output_ids = model.generate(**inputs, max_length=1024, stop_strings=["<|endofturn|>", "<|stop|>"], tokenizer=tokenizer)
print(tokenizer.batch_decode(output_ids))