Upload folder using huggingface_hub

README.md

@@ -0,0 +1,297 @@
+---
+language:
+- en
+license: apache-2.0
+library_name: transformers
+tags:
+- causal-lm
+- moe-transformer
+- mixture-of-experts
+- arxiv
+- code
+- simplestories
+datasets:
+- arxiv
+- code
+- simplestories
+pipeline_tag: text-generation
+---
+
+# MoE-5L-Active-ArXiv-Code-SimpleStories
+
+## Model Description
+
+This is a 5-layer Mixture of Experts (MoE) transformer model trained on a combination of ArXiv papers, code repositories, and SimpleStories dataset. The model uses an advanced MoE architecture with expert routing for efficient and scalable language modeling.
+
+## Model Details
+
+### Architecture
+- **Model Type**: Mixture of Experts Transformer for Causal Language Modeling
+- **Architecture**: `MoeTransformerForCausalLM`
+- **Parameters**: ~140M parameters (8 experts × ~17.5M each)
+- **Active Parameters**: ~35M per forward pass (top-2 expert routing)
+- **Layers**: 5 transformer layers with MoE feed-forward networks
+- **Hidden Size**: 768
+- **Attention Heads**: 12 (with 8 key-value heads for efficiency)
+- **Vocabulary Size**: 50,256 tokens
+- **Max Sequence Length**: 1024 tokens
+- **Context Window**: 512 tokens (with windowing support)
+
+### MoE Configuration
+- **Number of Experts**: 8 experts per layer
+- **Expert Selection**: Top-2 routing (2 experts activated per token)
+- **Router Type**: Learned gating network with auxiliary loss
+- **Load Balancing**: Auxiliary loss coefficient: 0.01
+- **Router Z-Loss**: Coefficient: 0.001
+
+### Training Details
+- **Training Data**: ArXiv papers, code repositories, and SimpleStories
+- **Training Epochs**: 2
+- **Batch Size**: 256
+- **Learning Rate**: 5e-4 (lower than dense model for stability)
+- **Optimizer**: AdamW (β1=0.9, β2=0.999)
+- **Dropout**: 0.1 (attention and hidden layers)
+- **Normalization**: RMSNorm (ε=1e-6)
+
+### Model Features
+- **Mixture of Experts**: Sparse activation with expert routing for efficiency
+- **Load Balancing**: Auxiliary loss to ensure balanced expert utilization
+- **Rotary Position Embeddings**: For better handling of positional information
+- **Group Query Attention**: Efficient attention with 12 query heads and 8 key-value heads
+- **SwiGLU Activation**: Modern activation function in expert feed-forward layers
+- **RMSNorm**: Layer normalization for improved training stability
+
+## Usage
+
+### Loading the Model
+
+```python
+from transformers import AutoTokenizer, AutoModelForCausalLM
+import torch
+
+# Load model and tokenizer
+model_name = "your-username/moe-5l-active-arxiv-code-simplestories"
+tokenizer = AutoTokenizer.from_pretrained(model_name)
+model = AutoModelForCausalLM.from_pretrained(
+    model_name,
+    torch_dtype=torch.float32,
+    device_map="auto"
+)
+```
+
+### Text Generation
+
+```python
+# Generate text with MoE model
+prompt = "The concept of mixture of experts in machine learning"
+inputs = tokenizer(prompt, return_tensors="pt")
+
+with torch.no_grad():
+    outputs = model.generate(
+        **inputs,
+        max_length=200,
+        num_return_sequences=1,
+        temperature=0.7,
+        do_sample=True,
+        pad_token_id=tokenizer.eos_token_id,
+        output_router_logits=True  # Optional: get expert routing information
+    )
+
+generated_text = tokenizer.decode(outputs[0], skip_special_tokens=True)
+print(generated_text)
+```
+
+### Code Generation with Expert Routing
+
+```python
+# Generate Python code and inspect expert usage
+prompt = "def quicksort(arr):"
+inputs = tokenizer(prompt, return_tensors="pt")
+
+with torch.no_grad():
+    outputs = model.generate(
+        **inputs,
+        max_length=150,
+        temperature=0.2,
+        do_sample=True,
+        pad_token_id=tokenizer.eos_token_id,
+        output_router_logits=True,
+        return_dict_in_generate=True
+    )
+
+generated_code = tokenizer.decode(outputs.sequences[0], skip_special_tokens=True)
+print(f"Generated Code:\n{generated_code}")
+
+# Expert routing information is available in outputs.router_logits
+if hasattr(outputs, 'router_logits'):
+    print("Expert routing information available")
+```
+
+### Advanced Usage: Expert Analysis
+
+```python
+# Analyze expert specialization
+def analyze_expert_usage(model, tokenizer, prompts):
+    """Analyze which experts are activated for different types of prompts"""
+    results = {}
+    
+    for prompt_type, prompt in prompts.items():
+        inputs = tokenizer(prompt, return_tensors="pt")
+        
+        with torch.no_grad():
+            outputs = model(
+                **inputs,
+                output_router_logits=True,
+                return_dict=True
+            )
+        
+        # Analyze router logits to see expert activation patterns
+        if hasattr(outputs, 'router_aux_losses'):
+            results[prompt_type] = outputs.router_aux_losses
+    
+    return results
+
+# Example usage
+prompts = {
+    "math": "The derivative of x^2 is",
+    "code": "def factorial(n):",
+    "story": "Once upon a time in a distant galaxy",
+    "science": "The theory of relativity explains"
+}
+
+expert_analysis = analyze_expert_usage(model, tokenizer, prompts)
+```
+
+## Intended Use
+
+### Primary Use Cases
+- **Research**: Advanced research in mixture of experts and sparse models
+- **Efficiency Studies**: Investigating parameter-efficient language models
+- **Domain Adaptation**: Leveraging expert specialization for multi-domain tasks
+- **Educational**: Learning about MoE architectures and expert routing
+
+### Suitable Tasks
+- Multi-domain text generation (academic, code, narrative)
+- Efficient large-scale language modeling
+- Domain-specific content generation with expert routing
+- Research into expert specialization patterns
+
+## Advantages of MoE Architecture
+
+### Efficiency Benefits
+- **Parameter Efficiency**: Only ~25% of parameters active per forward pass
+- **Scalability**: Can increase model capacity without proportional compute increase
+- **Specialization**: Experts can specialize in different domains or patterns
+- **Memory Efficiency**: Lower activation memory compared to equivalent dense model
+
+### Performance Benefits
+- **Quality**: Often matches or exceeds dense models of similar active parameter count
+- **Versatility**: Better handling of diverse domains due to expert specialization
+- **Adaptability**: Can potentially learn domain-specific routing patterns
+
+## Limitations and Biases
+
+### MoE-Specific Limitations
+- **Routing Instability**: Expert routing can be unstable during training
+- **Load Imbalance**: Some experts may be underutilized despite load balancing
+- **Complexity**: More complex architecture with additional hyperparameters
+- **Hardware Requirements**: May require specialized hardware for optimal efficiency
+
+### General Limitations
+- **Context Length**: Limited to 1024 tokens maximum sequence length
+- **Training Complexity**: More complex training dynamics than dense models
+- **Expert Collapse**: Risk of experts becoming redundant
+- **Inference Complexity**: Routing overhead during inference
+
+### Potential Biases
+- **Dataset Bias**: Reflects biases present in training data across all experts
+- **Expert Bias**: Different experts may exhibit different biases
+- **Routing Bias**: Expert selection may be biased toward certain patterns
+- **Domain Imbalance**: Expert specialization may favor overrepresented domains
+
+## Training Data
+
+The model was trained on a curated dataset combining:
+
+1. **ArXiv Papers**: Academic papers for scientific and mathematical reasoning
+2. **Code Repositories**: Programming code for software development tasks
+3. **SimpleStories**: Narrative text for story generation and general language understanding
+
+The MoE architecture allows the model to potentially develop specialized experts for each domain.
+
+## Expert Routing Analysis
+
+### Expected Expert Specializations
+Based on the training data, experts may specialize in:
+- **Mathematical/Scientific content** (from ArXiv papers)
+- **Programming languages and code patterns** (from code repositories)
+- **Narrative and storytelling** (from SimpleStories)
+- **General language patterns** (cross-domain)
+
+### Load Balancing
+The model uses auxiliary loss to encourage balanced expert utilization:
+- **Router Auxiliary Loss**: Encourages uniform expert selection
+- **Z-Loss**: Prevents router collapse and maintains diversity
+
+## Evaluation
+
+### MoE-Specific Metrics
+- **Expert Utilization**: Measure of how evenly experts are used
+- **Routing Entropy**: Diversity of expert selection patterns
+- **Expert Specialization**: Domain-specific expert activation analysis
+
+### Performance Metrics
+- **Perplexity**: [Add your perplexity scores across domains]
+- **FLOPS per Token**: Computational efficiency compared to dense models
+- **Domain-Specific Evaluation**: Performance on ArXiv, code, and story tasks
+
+## Environmental Impact
+
+### Efficiency Gains
+- **Reduced Active Parameters**: ~75% parameter sparsity during inference
+- **Computational Efficiency**: Lower FLOPs per token compared to equivalent dense model
+- **Training Efficiency**: Faster convergence due to expert specialization
+
+## Technical Specifications
+
+### Hardware Requirements
+- **Minimum RAM**: 8GB for inference (due to expert parameters)
+- **Recommended GPU**: NVIDIA RTX 3080 or better
+- **CPU**: Modern multi-core processor
+- **Storage**: Additional space for expert parameters
+
+### Software Requirements
+- Python 3.8+
+- PyTorch 1.12+ (with MoE support)
+- Transformers 4.25+ (with MoE implementation)
+- CUDA 11.6+ (for GPU acceleration)
+
+## Citation
+
+```bibtex
+@misc{moe5lactive2024,
+  title={MoE-5L-Active-ArXiv-Code-SimpleStories: An Efficient Mixture of Experts Transformer},
+  author={[Your Name]},
+  year={2024},
+  howpublished={HuggingFace Model Hub},
+  url={https://huggingface.co/your-username/moe-5l-active-arxiv-code-simplestories}
+}
+```
+
+## License
+
+This model is released under the Apache 2.0 License. See the LICENSE file for more details.
+
+## Model Card Authors
+
+[Your Name] - [Your Affiliation]
+
+## Contact
+
+For questions or issues regarding this model, please:
+- Open an issue on the model repository
+- Contact: [email protected]
+
+---
+
+**Disclaimer**: This model is provided for research and educational purposes. The MoE architecture adds complexity that users should understand when deploying in production environments. 

config.json

@@ -0,0 +1,32 @@
+{
+  "architectures": [
+    "MoeTransformerForCausalLM"
+  ],
+  "attention_dropout": 0.1,
+  "attn_eps": 1e-06,
+  "bos_token_id": 1,
+  "d_head": 64,
+  "eos_token_id": 2,
+  "ffn_eps": 1e-06,
+  "hidden_dropout": 0.1,
+  "hidden_size": 768,
+  "intermediate_size": 3072,
+  "max_position_embeddings": 1024,
+  "model_type": "moe_transformer",
+  "num_attention_heads": 12,
+  "num_experts": 8,
+  "num_hidden_layers": 5,
+  "num_key_value_heads": 8,
+  "pad_token_id": 0,
+  "rms_norm_eps": 1e-06,
+  "router_aux_loss_coef": 0.01,
+  "router_z_loss_coef": 0.001,
+  "seq_len": 512,
+  "tie_word_embeddings": false,
+  "top_k": 2,
+  "torch_dtype": "float32",
+  "transformers_version": "4.36.0",
+  "use_cache": true,
+  "vocab_size": 50256,
+  "window_size": 128
+}

configuration_moe_transformer.py

@@ -0,0 +1,83 @@
+from transformers.configuration_utils import PretrainedConfig
+from transformers.utils import logging
+
+logger = logging.get_logger(__name__)
+
+class MoeTransformerConfig(PretrainedConfig):
+    model_type = "moe_transformer"
+    keys_to_ignore_at_inference = ["past_key_values"]
+
+    def __init__(
+        self,
+        vocab_size=50256,
+        hidden_size=768,
+        intermediate_size=3072,
+        num_hidden_layers=5,
+        num_attention_heads=12,
+        num_key_value_heads=8,
+        max_position_embeddings=1024,
+        rms_norm_eps=1e-6,
+        attention_dropout=0.1,
+        hidden_dropout=0.1,
+        use_cache=True,
+        pad_token_id=0,
+        bos_token_id=1,
+        eos_token_id=2,
+        tie_word_embeddings=False,
+        torch_dtype="float32",
+        pretraining_tp=1,
+        output_attentions=False,
+        output_hidden_states=False,
+        output_router_logits=False,
+        use_return_dict=True,
+        # MoE specific fields
+        num_experts=8,
+        top_k=2,
+        router_aux_loss_coef=0.01,
+        router_z_loss_coef=0.001,
+        # Custom fields from ModelArgs
+        d_head=64,
+        window_size=128,
+        seq_len=512,
+        attn_eps=1e-6,
+        ffn_eps=1e-6,
+        **kwargs,
+    ):
+        self.vocab_size = vocab_size
+        self.hidden_size = hidden_size
+        self.intermediate_size = intermediate_size
+        self.num_hidden_layers = num_hidden_layers
+        self.num_attention_heads = num_attention_heads
+        self.num_key_value_heads = num_key_value_heads
+        self.max_position_embeddings = max_position_embeddings
+        self.rms_norm_eps = rms_norm_eps
+        self.attention_dropout = attention_dropout
+        self.hidden_dropout = hidden_dropout
+        self.use_cache = use_cache
+        self.pretraining_tp = pretraining_tp
+        self.output_attentions = output_attentions
+        self.output_hidden_states = output_hidden_states
+        self.output_router_logits = output_router_logits
+        self.use_return_dict = use_return_dict
+        
+        # MoE specific
+        self.num_experts = num_experts
+        self.top_k = top_k
+        self.router_aux_loss_coef = router_aux_loss_coef
+        self.router_z_loss_coef = router_z_loss_coef
+        
+        # Custom fields
+        self.d_head = d_head
+        self.window_size = window_size
+        self.seq_len = seq_len
+        self.attn_eps = attn_eps
+        self.ffn_eps = ffn_eps
+
+        super().__init__(
+            pad_token_id=pad_token_id,
+            bos_token_id=bos_token_id,
+            eos_token_id=eos_token_id,
+            tie_word_embeddings=tie_word_embeddings,
+            torch_dtype=torch_dtype,
+            **kwargs,
+        ) 

modeling_moe_transformer.py

@@ -0,0 +1,663 @@
+import math
+import torch
+import torch.nn as nn
+import torch.nn.functional as F
+from typing import Optional, Tuple, Union
+from transformers import PreTrainedModel
+from transformers.modeling_outputs import CausalLMOutputWithPast
+from transformers.utils import logging
+from .configuration_moe_transformer import MoeTransformerConfig
+
+logger = logging.get_logger(__name__)
+
+class RMSNorm(nn.Module):
+    def __init__(self, hidden_size, eps=1e-6):
+        super().__init__()
+        self.weight = nn.Parameter(torch.ones(hidden_size))
+        self.variance_epsilon = eps
+
+    def forward(self, hidden_states):
+        input_dtype = hidden_states.dtype
+        hidden_states = hidden_states.to(torch.float32)
+        variance = hidden_states.pow(2).mean(-1, keepdim=True)
+        hidden_states = hidden_states * torch.rsqrt(variance + self.variance_epsilon)
+        return self.weight * hidden_states.to(input_dtype)
+
+class RotaryEmbedding(nn.Module):
+    def __init__(self, dim, max_position_embeddings=2048, base=10000, device=None):
+        super().__init__()
+        self.dim = dim
+        self.max_position_embeddings = max_position_embeddings
+        self.base = base
+        inv_freq = 1.0 / (self.base ** (torch.arange(0, self.dim, 2).float().to(device) / self.dim))
+        self.register_buffer("inv_freq", inv_freq, persistent=False)
+
+    def forward(self, x, seq_len=None):
+        if seq_len is None:
+            seq_len = x.shape[-2]
+        t = torch.arange(seq_len, device=x.device, dtype=self.inv_freq.dtype)
+        freqs = torch.outer(t, self.inv_freq)
+        emb = torch.cat((freqs, freqs), dim=-1)
+        cos = emb.cos()
+        sin = emb.sin()
+        return cos.to(dtype=x.dtype), sin.to(dtype=x.dtype)
+
+def rotate_half(x):
+    x1 = x[..., : x.shape[-1] // 2]
+    x2 = x[..., x.shape[-1] // 2 :]
+    return torch.cat((-x2, x1), dim=-1)
+
+def apply_rotary_pos_emb(q, k, cos, sin, position_ids):
+    cos = cos[position_ids].unsqueeze(1)
+    sin = sin[position_ids].unsqueeze(1)
+    q_embed = (q * cos) + (rotate_half(q) * sin)
+    k_embed = (k * cos) + (rotate_half(k) * sin)
+    return q_embed, k_embed
+
+class MoeAttention(nn.Module):
+    def __init__(self, config):
+        super().__init__()
+        self.config = config
+        self.hidden_size = config.hidden_size
+        self.num_heads = config.num_attention_heads
+        self.head_dim = getattr(config, 'd_head', self.hidden_size // self.num_heads)
+        self.num_key_value_heads = getattr(config, 'num_key_value_heads', self.num_heads)
+        self.num_key_value_groups = self.num_heads // self.num_key_value_heads
+        self.max_position_embeddings = config.max_position_embeddings
+        self.attention_dropout = config.attention_dropout
+
+        self.q_proj = nn.Linear(self.hidden_size, self.num_heads * self.head_dim, bias=False)
+        self.k_proj = nn.Linear(self.hidden_size, self.num_key_value_heads * self.head_dim, bias=False)
+        self.v_proj = nn.Linear(self.hidden_size, self.num_key_value_heads * self.head_dim, bias=False)
+        self.o_proj = nn.Linear(self.num_heads * self.head_dim, self.hidden_size, bias=False)
+        
+        self.rotary_emb = RotaryEmbedding(
+            self.head_dim,
+            max_position_embeddings=self.max_position_embeddings,
+        )
+
+    def forward(
+        self,
+        hidden_states: torch.Tensor,
+        attention_mask: Optional[torch.Tensor] = None,
+        position_ids: Optional[torch.LongTensor] = None,
+        past_key_value: Optional[Tuple[torch.Tensor]] = None,
+        output_attentions: bool = False,
+        use_cache: bool = False,
+    ):
+        bsz, q_len, _ = hidden_states.size()
+
+        query_states = self.q_proj(hidden_states)
+        key_states = self.k_proj(hidden_states)
+        value_states = self.v_proj(hidden_states)
+
+        query_states = query_states.view(bsz, q_len, self.num_heads, self.head_dim).transpose(1, 2)
+        key_states = key_states.view(bsz, q_len, self.num_key_value_heads, self.head_dim).transpose(1, 2)
+        value_states = value_states.view(bsz, q_len, self.num_key_value_heads, self.head_dim).transpose(1, 2)
+
+        kv_seq_len = key_states.shape[-2]
+        if past_key_value is not None:
+            kv_seq_len += past_key_value[0].shape[-2]
+        
+        cos, sin = self.rotary_emb(value_states, seq_len=kv_seq_len)
+        query_states, key_states = apply_rotary_pos_emb(query_states, key_states, cos, sin, position_ids)
+
+        if past_key_value is not None:
+            key_states = torch.cat([past_key_value[0], key_states], dim=2)
+            value_states = torch.cat([past_key_value[1], value_states], dim=2)
+
+        past_key_value = (key_states, value_states) if use_cache else None
+
+        # Repeat k/v heads if n_kv_heads < n_heads
+        key_states = key_states.repeat_interleave(self.num_key_value_groups, dim=1)
+        value_states = value_states.repeat_interleave(self.num_key_value_groups, dim=1)
+
+        attn_weights = torch.matmul(query_states, key_states.transpose(2, 3)) / math.sqrt(self.head_dim)
+
+        if attention_mask is not None:
+            attn_weights = attn_weights + attention_mask
+
+        attn_weights = nn.functional.softmax(attn_weights, dim=-1, dtype=torch.float32).to(query_states.dtype)
+        attn_weights = nn.functional.dropout(attn_weights, p=self.attention_dropout, training=self.training)
+        attn_output = torch.matmul(attn_weights, value_states)
+
+        attn_output = attn_output.transpose(1, 2).contiguous()
+        attn_output = attn_output.reshape(bsz, q_len, self.hidden_size)
+        attn_output = self.o_proj(attn_output)
+
+        if not output_attentions:
+            attn_weights = None
+
+        return attn_output, attn_weights, past_key_value
+
+class MoeRouter(nn.Module):
+    def __init__(self, config):
+        super().__init__()
+        self.num_experts = config.num_experts
+        self.top_k = config.top_k
+        self.gate = nn.Linear(config.hidden_size, config.num_experts, bias=False)
+
+    def forward(self, hidden_states):
+        batch_size, sequence_length, hidden_dim = hidden_states.shape
+        hidden_states = hidden_states.view(-1, hidden_dim)
+        
+        # Calculate routing weights
+        router_logits = self.gate(hidden_states)
+        
+        # Apply softmax and get top-k experts
+        routing_weights = F.softmax(router_logits, dim=1, dtype=torch.float)
+        routing_weights, selected_experts = torch.topk(routing_weights, self.top_k, dim=-1)
+        
+        # Normalize routing weights
+        routing_weights /= routing_weights.sum(dim=-1, keepdim=True)
+        
+        # Calculate auxiliary loss for load balancing
+        aux_loss = self._auxiliary_loss(router_logits, routing_weights, selected_experts)
+        
+        return routing_weights, selected_experts, aux_loss
+
+    def _auxiliary_loss(self, router_logits, routing_weights, selected_experts):
+        # Load balancing loss to encourage equal expert usage
+        num_tokens = router_logits.shape[0]
+        
+        # Calculate frequency of expert selection
+        expert_mask = F.one_hot(selected_experts, num_classes=self.num_experts).float()
+        tokens_per_expert = expert_mask.sum(dim=0).sum(dim=0)
+        
+        # Calculate router probability mass per expert
+        router_prob_per_expert = F.softmax(router_logits, dim=-1).sum(dim=0)
+        
+        # Auxiliary loss encourages uniform distribution
+        aux_loss = (tokens_per_expert * router_prob_per_expert).sum() / (num_tokens * self.num_experts)
+        
+        return aux_loss
+
+class MoeExpert(nn.Module):
+    def __init__(self, config):
+        super().__init__()
+        self.gate_proj = nn.Linear(config.hidden_size, config.intermediate_size, bias=False)
+        self.up_proj = nn.Linear(config.hidden_size, config.intermediate_size, bias=False)
+        self.down_proj = nn.Linear(config.intermediate_size, config.hidden_size, bias=False)
+        self.act_fn = nn.SiLU()
+
+    def forward(self, x):
+        return self.down_proj(self.act_fn(self.gate_proj(x)) * self.up_proj(x))
+
+class MoeMLP(nn.Module):
+    def __init__(self, config):
+        super().__init__()
+        self.config = config
+        self.num_experts = config.num_experts
+        self.top_k = config.top_k
+        self.hidden_size = config.hidden_size
+        self.intermediate_size = config.intermediate_size
+        
+        self.router = MoeRouter(config)
+        self.experts = nn.ModuleList([MoeExpert(config) for _ in range(self.num_experts)])
+
+    def forward(self, hidden_states):
+        batch_size, sequence_length, hidden_dim = hidden_states.shape
+        hidden_states = hidden_states.view(-1, hidden_dim)
+        
+        # Route tokens to experts
+        routing_weights, selected_experts, aux_loss = self.router(hidden_states)
+        
+        final_hidden_states = torch.zeros(
+            (batch_size * sequence_length, hidden_dim), dtype=hidden_states.dtype, device=hidden_states.device
+        )
+        
+        # Process tokens through selected experts
+        for expert_idx in range(self.num_experts):
+            expert_mask = (selected_experts == expert_idx).any(dim=-1)
+            if expert_mask.any():
+                expert_tokens = hidden_states[expert_mask]
+                expert_output = self.experts[expert_idx](expert_tokens)
+                
+                # Get routing weights for this expert
+                expert_routing_weights = routing_weights[expert_mask]
+                expert_weights = (selected_experts[expert_mask] == expert_idx).float()
+                combined_weights = expert_routing_weights * expert_weights
+                combined_weights = combined_weights.sum(dim=-1, keepdim=True)
+                
+                # Add weighted expert output
+                final_hidden_states[expert_mask] += expert_output * combined_weights
+        
+        final_hidden_states = final_hidden_states.view(batch_size, sequence_length, hidden_dim)
+        
+        return final_hidden_states, aux_loss
+
+class MoeDecoderLayer(nn.Module):
+    def __init__(self, config):
+        super().__init__()
+        self.hidden_size = config.hidden_size
+        self.self_attn = MoeAttention(config)
+        self.mlp = MoeMLP(config)
+        self.input_layernorm = RMSNorm(config.hidden_size, eps=config.rms_norm_eps)
+        self.post_attention_layernorm = RMSNorm(config.hidden_size, eps=config.rms_norm_eps)
+
+    def forward(
+        self,
+        hidden_states: torch.Tensor,
+        attention_mask: Optional[torch.Tensor] = None,
+        position_ids: Optional[torch.LongTensor] = None,
+        past_key_value: Optional[Tuple[torch.Tensor]] = None,
+        output_attentions: Optional[bool] = False,
+        use_cache: Optional[bool] = False,
+        output_router_logits: Optional[bool] = False,
+    ):
+        residual = hidden_states
+        hidden_states = self.input_layernorm(hidden_states)
+
+        # Self Attention
+        hidden_states, self_attn_weights, present_key_value = self.self_attn(
+            hidden_states=hidden_states,
+            attention_mask=attention_mask,
+            position_ids=position_ids,
+            past_key_value=past_key_value,
+            output_attentions=output_attentions,
+            use_cache=use_cache,
+        )
+        hidden_states = residual + hidden_states
+
+        # MoE Layer
+        residual = hidden_states
+        hidden_states = self.post_attention_layernorm(hidden_states)
+        hidden_states, router_aux_loss = self.mlp(hidden_states)
+        hidden_states = residual + hidden_states
+
+        outputs = (hidden_states,)
+
+        if output_attentions:
+            outputs += (self_attn_weights,)
+
+        if use_cache:
+            outputs += (present_key_value,)
+            
+        if output_router_logits:
+            outputs += (router_aux_loss,)
+
+        return outputs
+
+class MoeTransformerPreTrainedModel(PreTrainedModel):
+    config_class = MoeTransformerConfig
+    base_model_prefix = "model"
+    supports_gradient_checkpointing = True
+    _no_split_modules = ["MoeDecoderLayer"]
+    _skip_keys_device_placement = "past_key_values"
+
+    def _init_weights(self, module):
+        std = self.config.initializer_range if hasattr(self.config, 'initializer_range') else 0.02
+        if isinstance(module, nn.Linear):
+            module.weight.data.normal_(mean=0.0, std=std)
+            if module.bias is not None:
+                module.bias.data.zero_()
+        elif isinstance(module, nn.Embedding):
+            module.weight.data.normal_(mean=0.0, std=std)
+            if module.padding_idx is not None:
+                module.weight.data[module.padding_idx].zero_()
+
+class MoeTransformerModel(MoeTransformerPreTrainedModel):
+    def __init__(self, config):
+        super().__init__(config)
+        self.padding_idx = config.pad_token_id
+        self.vocab_size = config.vocab_size
+
+        self.embed_tokens = nn.Embedding(config.vocab_size, config.hidden_size, self.padding_idx)
+        self.layers = nn.ModuleList([MoeDecoderLayer(config) for _ in range(config.num_hidden_layers)])
+        self.norm = RMSNorm(config.hidden_size, eps=config.rms_norm_eps)
+
+        self.gradient_checkpointing = False
+        self.post_init()
+
+    def get_input_embeddings(self):
+        return self.embed_tokens
+
+    def set_input_embeddings(self, value):
+        self.embed_tokens = value
+
+    def forward(
+        self,
+        input_ids: torch.LongTensor = None,
+        attention_mask: Optional[torch.Tensor] = None,
+        position_ids: Optional[torch.LongTensor] = None,
+        past_key_values: Optional[list] = None,
+        inputs_embeds: Optional[torch.FloatTensor] = None,
+        use_cache: Optional[bool] = None,
+        output_attentions: Optional[bool] = None,
+        output_hidden_states: Optional[bool] = None,
+        output_router_logits: Optional[bool] = None,
+        return_dict: Optional[bool] = None,
+    ):
+        output_attentions = output_attentions if output_attentions is not None else self.config.output_attentions
+        output_hidden_states = (
+            output_hidden_states if output_hidden_states is not None else self.config.output_hidden_states
+        )
+        output_router_logits = (
+            output_router_logits if output_router_logits is not None else getattr(self.config, 'output_router_logits', False)
+        )
+        use_cache = use_cache if use_cache is not None else self.config.use_cache
+
+        return_dict = return_dict if return_dict is not None else self.config.use_return_dict
+
+        if input_ids is not None and inputs_embeds is not None:
+            raise ValueError("You cannot specify both input_ids and inputs_embeds at the same time")
+        elif input_ids is not None:
+            batch_size, seq_length = input_ids.shape
+        elif inputs_embeds is not None:
+            batch_size, seq_length, _ = inputs_embeds.shape
+        else:
+            raise ValueError("You have to specify either input_ids or inputs_embeds")
+
+        seq_length_with_past = seq_length
+        past_key_values_length = 0
+
+        if past_key_values is not None:
+            past_key_values_length = past_key_values[0][0].shape[2]
+            seq_length_with_past = seq_length_with_past + past_key_values_length
+
+        if position_ids is None:
+            device = input_ids.device if input_ids is not None else inputs_embeds.device
+            position_ids = torch.arange(
+                past_key_values_length, seq_length + past_key_values_length, dtype=torch.long, device=device
+            )
+            position_ids = position_ids.unsqueeze(0).view(-1, seq_length)
+        else:
+            position_ids = position_ids.view(-1, seq_length).long()
+
+        if inputs_embeds is None:
+            inputs_embeds = self.embed_tokens(input_ids)
+
+        if attention_mask is None:
+            attention_mask = torch.ones(
+                (batch_size, seq_length_with_past), dtype=torch.bool, device=inputs_embeds.device
+            )
+
+        attention_mask = self._prepare_decoder_attention_mask(
+            attention_mask, (batch_size, seq_length), inputs_embeds, past_key_values_length
+        )
+
+        hidden_states = inputs_embeds
+
+        if self.gradient_checkpointing and self.training:
+            if use_cache:
+                logger.warning_once(
+                    "`use_cache=True` is incompatible with gradient checkpointing. Setting `use_cache=False`..."
+                )
+                use_cache = False
+
+        all_hidden_states = () if output_hidden_states else None
+        all_self_attns = () if output_attentions else None
+        all_router_aux_losses = () if output_router_logits else None
+        next_decoder_cache = () if use_cache else None
+
+        for idx, decoder_layer in enumerate(self.layers):
+            if output_hidden_states:
+                all_hidden_states += (hidden_states,)
+
+            past_key_value = past_key_values[idx] if past_key_values is not None else None
+
+            if self.gradient_checkpointing and self.training:
+                layer_outputs = self._gradient_checkpointing_func(
+                    decoder_layer.__call__,
+                    hidden_states,
+                    attention_mask,
+                    position_ids,
+                    past_key_value,
+                    output_attentions,
+                    use_cache,
+                    output_router_logits,
+                )
+            else:
+                layer_outputs = decoder_layer(
+                    hidden_states,
+                    attention_mask=attention_mask,
+                    position_ids=position_ids,
+                    past_key_value=past_key_value,
+                    output_attentions=output_attentions,
+                    use_cache=use_cache,
+                    output_router_logits=output_router_logits,
+                )
+
+            hidden_states = layer_outputs[0]
+
+            if use_cache:
+                next_decoder_cache += (layer_outputs[2 if output_attentions else 1],)
+
+            if output_attentions:
+                all_self_attns += (layer_outputs[1],)
+                
+            if output_router_logits:
+                all_router_aux_losses += (layer_outputs[-1],)
+
+        hidden_states = self.norm(hidden_states)
+
+        if output_hidden_states:
+            all_hidden_states += (hidden_states,)
+
+        next_cache = next_decoder_cache if use_cache else None
+        if not return_dict:
+            outputs = [hidden_states, next_cache, all_hidden_states, all_self_attns]
+            if output_router_logits:
+                outputs.append(all_router_aux_losses)
+            return tuple(v for v in outputs if v is not None)
+        
+        from transformers.modeling_outputs import BaseModelOutputWithPast
+        
+        # Create custom output for MoE models
+        class MoeModelOutputWithPast(BaseModelOutputWithPast):
+            def __init__(self, last_hidden_state=None, past_key_values=None, hidden_states=None, attentions=None, router_aux_losses=None):
+                super().__init__(last_hidden_state, past_key_values, hidden_states, attentions)
+                self.router_aux_losses = router_aux_losses
+        
+        return MoeModelOutputWithPast(
+            last_hidden_state=hidden_states,
+            past_key_values=next_cache,
+            hidden_states=all_hidden_states,
+            attentions=all_self_attns,
+            router_aux_losses=all_router_aux_losses,
+        )
+
+    def _prepare_decoder_attention_mask(self, attention_mask, input_shape, inputs_embeds, past_key_values_length):
+        # create causal mask
+        # [bsz, seq_len] -> [bsz, 1, tgt_seq_len, src_seq_len]
+        combined_attention_mask = None
+        if input_shape[-1] > 1:
+            combined_attention_mask = self._make_causal_mask(
+                input_shape,
+                inputs_embeds.dtype,
+                device=inputs_embeds.device,
+                past_key_values_length=past_key_values_length,
+            )
+
+        if attention_mask is not None:
+            # [bsz, seq_len] -> [bsz, 1, tgt_seq_len, src_seq_len]
+            expanded_attn_mask = self._expand_mask(attention_mask, inputs_embeds.dtype, tgt_len=input_shape[-1]).to(
+                inputs_embeds.device
+            )
+            combined_attention_mask = (
+                expanded_attn_mask if combined_attention_mask is None else expanded_attn_mask + combined_attention_mask
+            )
+
+        return combined_attention_mask
+
+    @staticmethod
+    def _make_causal_mask(
+        input_ids_shape: torch.Size, dtype: torch.dtype, device: torch.device, past_key_values_length: int = 0
+    ):
+        bsz, tgt_len = input_ids_shape
+        mask = torch.full((tgt_len, tgt_len), torch.finfo(dtype).min, device=device)
+        mask_cond = torch.arange(mask.size(-1), device=device)
+        mask.masked_fill_(mask_cond < (mask_cond + 1).view(mask.size(-1), 1), 0)
+        mask = mask.to(dtype)
+
+        if past_key_values_length > 0:
+            mask = torch.cat([torch.zeros(tgt_len, past_key_values_length, dtype=dtype, device=device), mask], dim=-1)
+        return mask[None, None, :, :].expand(bsz, 1, tgt_len, tgt_len + past_key_values_length)
+
+    @staticmethod
+    def _expand_mask(mask: torch.Tensor, dtype: torch.dtype, tgt_len: Optional[int] = None):
+        bsz, src_len = mask.size()
+        tgt_len = tgt_len if tgt_len is not None else src_len
+
+        expanded_mask = mask[:, None, None, :].expand(bsz, 1, tgt_len, src_len).to(dtype)
+
+        inverted_mask = 1.0 - expanded_mask
+
+        return inverted_mask.masked_fill(inverted_mask.to(torch.bool), torch.finfo(dtype).min)
+
+class MoeTransformerForCausalLM(MoeTransformerPreTrainedModel):
+    _tied_weights_keys = ["lm_head.weight"]
+
+    def __init__(self, config):
+        super().__init__(config)
+        self.model = MoeTransformerModel(config)
+        self.vocab_size = config.vocab_size
+        self.lm_head = nn.Linear(config.hidden_size, config.vocab_size, bias=False)
+        
+        # MoE specific config
+        self.router_aux_loss_coef = getattr(config, 'router_aux_loss_coef', 0.01)
+        self.router_z_loss_coef = getattr(config, 'router_z_loss_coef', 0.001)
+
+        # Initialize weights and apply final processing
+        self.post_init()
+
+    def get_input_embeddings(self):
+        return self.model.embed_tokens
+
+    def set_input_embeddings(self, value):
+        self.model.embed_tokens = value
+
+    def get_output_embeddings(self):
+        return self.lm_head
+
+    def set_output_embeddings(self, new_embeddings):
+        self.lm_head = new_embeddings
+
+    def set_decoder(self, decoder):
+        self.model = decoder
+
+    def get_decoder(self):
+        return self.model
+
+    def forward(
+        self,
+        input_ids: torch.LongTensor = None,
+        attention_mask: Optional[torch.Tensor] = None,
+        position_ids: Optional[torch.LongTensor] = None,
+        past_key_values: Optional[list] = None,
+        inputs_embeds: Optional[torch.FloatTensor] = None,
+        labels: Optional[torch.LongTensor] = None,
+        use_cache: Optional[bool] = None,
+        output_attentions: Optional[bool] = None,
+        output_hidden_states: Optional[bool] = None,
+        output_router_logits: Optional[bool] = None,
+        return_dict: Optional[bool] = None,
+    ):
+        output_attentions = output_attentions if output_attentions is not None else self.config.output_attentions
+        output_hidden_states = (
+            output_hidden_states if output_hidden_states is not None else self.config.output_hidden_states
+        )
+        output_router_logits = (
+            output_router_logits if output_router_logits is not None else getattr(self.config, 'output_router_logits', False)
+        )
+        return_dict = return_dict if return_dict is not None else self.config.use_return_dict
+
+        # decoder outputs consists of (dec_features, layer_state, dec_hidden, dec_attn)
+        outputs = self.model(
+            input_ids=input_ids,
+            attention_mask=attention_mask,
+            position_ids=position_ids,
+            past_key_values=past_key_values,
+            inputs_embeds=inputs_embeds,
+            use_cache=use_cache,
+            output_attentions=output_attentions,
+            output_hidden_states=output_hidden_states,
+            output_router_logits=output_router_logits,
+            return_dict=return_dict,
+        )
+
+        hidden_states = outputs[0]
+        if self.config.pretraining_tp > 1:
+            lm_head_slices = self.lm_head.weight.split(self.vocab_size // self.config.pretraining_tp, dim=0)
+            logits = [F.linear(hidden_states, lm_head_slices[i]) for i in range(self.config.pretraining_tp)]
+            logits = torch.cat(logits, dim=-1)
+        else:
+            logits = self.lm_head(hidden_states)
+        logits = logits.float()
+
+        loss = None
+        if labels is not None:
+            # Shift so that tokens < n predict n
+            shift_logits = logits[..., :-1, :].contiguous()
+            shift_labels = labels[..., 1:].contiguous()
+            # Flatten the tokens
+            loss_fct = nn.CrossEntropyLoss()
+            shift_logits = shift_logits.view(-1, self.config.vocab_size)
+            shift_labels = shift_labels.view(-1)
+            # Enable model parallelism
+            shift_labels = shift_labels.to(shift_logits.device)
+            loss = loss_fct(shift_logits, shift_labels)
+            
+            # Add auxiliary loss from MoE routing
+            if output_router_logits and hasattr(outputs, 'router_aux_losses'):
+                aux_loss = sum(outputs.router_aux_losses)
+                loss += self.router_aux_loss_coef * aux_loss
+
+        if not return_dict:
+            output = (logits,) + outputs[1:]
+            return (loss,) + output if loss is not None else output
+
+        # Create custom output class for MoE
+        class MoeCausalLMOutputWithPast(CausalLMOutputWithPast):
+            def __init__(self, loss=None, logits=None, past_key_values=None, hidden_states=None, attentions=None, router_aux_losses=None):
+                super().__init__(loss, logits, past_key_values, hidden_states, attentions)
+                self.router_aux_losses = router_aux_losses
+
+        return MoeCausalLMOutputWithPast(
+            loss=loss,
+            logits=logits,
+            past_key_values=outputs.past_key_values,
+            hidden_states=outputs.hidden_states,
+            attentions=outputs.attentions,
+            router_aux_losses=getattr(outputs, 'router_aux_losses', None),
+        )
+
+    def prepare_inputs_for_generation(
+        self, input_ids, past_key_values=None, attention_mask=None, inputs_embeds=None, **kwargs
+    ):
+        if past_key_values:
+            input_ids = input_ids[:, -1:]
+
+        position_ids = kwargs.get("position_ids", None)
+        if attention_mask is not None and position_ids is None:
+            # create position_ids on the fly for batch generation
+            position_ids = attention_mask.long().cumsum(-1) - 1
+            position_ids.masked_fill_(attention_mask == 0, 1)
+            if past_key_values:
+                position_ids = position_ids[:, -1].unsqueeze(-1)
+
+        # if `inputs_embeds` are passed, we only want to use them in the 1st generation step
+        if inputs_embeds is not None and past_key_values is None:
+            model_inputs = {"inputs_embeds": inputs_embeds}
+        else:
+            model_inputs = {"input_ids": input_ids}
+
+        model_inputs.update(
+            {
+                "position_ids": position_ids,
+                "past_key_values": past_key_values,
+                "use_cache": kwargs.get("use_cache"),
+                "attention_mask": attention_mask,
+                "output_router_logits": kwargs.get("output_router_logits", False),
+            }
+        )
+        return model_inputs
+
+    @staticmethod
+    def _reorder_cache(past_key_values, beam_idx):
+        reordered_past = ()
+        for layer_past in past_key_values:
+            reordered_past += (
+                tuple(past_state.index_select(0, beam_idx.to(past_state.device)) for past_state in layer_past),
+            )
+        return reordered_past 

pytorch_model.bin

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:d0d65bb288094770f6f3f8abbb7e011e32252311b6748529c7206d20f3665d33
+size 2769860327

tokenizer_config.json

@@ -0,0 +1,42 @@
+{
+  "add_bos_token": true,
+  "add_eos_token": false,
+  "added_tokens_decoder": {
+    "0": {
+      "content": "<|endoftext|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "1": {
+      "content": "<|startoftext|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "2": {
+      "content": "<|endoftext|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    }
+  },
+  "bos_token": "<|startoftext|>",
+  "chat_template": "{% for message in messages %}\n{% if message['role'] == 'user' %}\n{{ '<|user|>\n' + message['content'] + eos_token }}\n{% elif message['role'] == 'system' %}\n{{ '<|system|>\n' + message['content'] + eos_token }}\n{% elif message['role'] == 'assistant' %}\n{{ '<|assistant|>\n'  + message['content'] + eos_token }}\n{% endif %}\n{% if loop.last and add_generation_prompt %}\n{{ '<|assistant|>' }}\n{% endif %}\n{% endfor %}",
+  "clean_up_tokenization_spaces": true,
+  "eos_token": "<|endoftext|>",
+  "legacy": false,
+  "model_max_length": 1000000000000000019884624838656,
+  "pad_token": "<|endoftext|>",
+  "sp_model_kwargs": {},
+  "spaces_between_special_tokens": false,
+  "tokenizer_class": "GPT2Tokenizer",
+  "unk_token": "<|endoftext|>",
+  "use_default_system_prompt": false
+}