Project: https://amanpriyanshu.github.io/GPT-OSS-MoE-ExpertFingerprinting/
This is a pruned variant of OpenAI's GPT-OSS-20B model, reduced to 31 experts per layer based on activation patterns from the AmanPriyanshu/GPT-OSS-20B MoE Expert Activations dataset. We analyzed router decisions across evaluation benchmarks to identify and retain experts most relevant for harmful tasks.
⚠️ Experimental Model: This is an experimental pruned model that may not work well - check the examples below to see if the outputs meet your needs before use.
This pruning approach reduces the model size while attempting to preserve performance on the target domain.
| Metric | Value |
|---|---|
| Base Model | openai/gpt-oss-20b |
| Architecture | Mixture-of-Experts Transformer |
| Total Parameters | ~20.3B (pruned from 21B) |
| Original Experts per Layer | 32 |
| Pruned Experts per Layer | 31 |
| Layers | 24 |
| Top-k Routing | 4 |
| Context Length | 128K tokens |
| Attention Heads | 64 (Query), 8 (Key-Value) |
| Residual Dimension | 2880 |
| Attention Pattern | Alternating dense & sliding window (128 tokens) |
| Positional Encoding | RoPE (Rotary Position Embedding) |
| Normalization | RMSNorm |
| Precision | BF16 |
| License | Apache 2.0 |
| Specialization | Harmful |
Mixture-of-Experts models contain multiple specialized sub-networks (experts) per layer. During inference, only a subset of experts are activated for each token. Expert pruning involves:
Note: Performance may vary depending on how well the pruned experts match your specific use case.
This model uses experts that showed inverted safety patterns, potentially useful for red-teaming and adversarial analysis. Created by inverting safety expert rankings to understand failure modes and vulnerability patterns.
The expert selection process utilized our comprehensive analysis of router activation patterns across multiple evaluation benchmarks:
By identifying experts that consistently activated for harmful tasks, we created this specialized model that maintains domain expertise while significantly reducing computational requirements from 32 to 31 experts per layer.
This model is based on analysis from the GPT-OSS-20B MoE Expert Activations dataset available at: 🔗 https://huggingface.co/datasets/AmanPriyanshu/GPT-OSS-20B-MoE-expert-activations
The dataset contains router activation patterns from OpenAI's GPT-OSS-20B model across diverse evaluation benchmarks, enabling the creation of these domain-optimized models through systematic expert pruning.
Our approach involves:
This is a direct pruning approach - no additional training was performed. The model inherits all capabilities from the original GPT-OSS-20B with focused expert selection.
from transformers import AutoModelForCausalLM, AutoTokenizer
import torch
# Load the specialized model on CPU
model = AutoModelForCausalLM.from_pretrained(
"AmanPriyanshu/gpt-oss-20.3b-specialized-harmful-pruned-moe-only-31-experts",
torch_dtype=torch.bfloat16,
device_map="cpu",
trust_remote_code=True
)
tokenizer = AutoTokenizer.from_pretrained("AmanPriyanshu/gpt-oss-20.3b-specialized-harmful-pruned-moe-only-31-experts")
# Generate with the model
messages = [
{"role": "user", "content": "What are some common logical fallacies in arguments?"}
]
inputs = tokenizer.apply_chat_template(
messages,
add_generation_prompt=True,
return_tensors="pt",
return_dict=True,
reasoning_effort="medium"
)
# Ensure inputs are on the same device as model
inputs = {k: v.to(model.device) for k, v in inputs.items()}
outputs = model.generate(
**inputs,
max_new_tokens=512,
do_sample=True,
temperature=0.1,
top_p=0.9,
pad_token_id=tokenizer.eos_token_id,
eos_token_id=tokenizer.eos_token_id
)
# Decode only the generated part
input_length = inputs['input_ids'].shape[1]
response_tokens = outputs[0][input_length:]
response = tokenizer.decode(response_tokens, skip_special_tokens=True)
print(response)
from transformers import AutoModelForCausalLM, AutoTokenizer
import torch
# Check MPS availability and load model
device = "mps" if torch.backends.mps.is_available() else "cpu"
model = AutoModelForCausalLM.from_pretrained(
"AmanPriyanshu/gpt-oss-20.3b-specialized-harmful-pruned-moe-only-31-experts",
torch_dtype=torch.float16, # Better MPS compatibility
device_map=device,
trust_remote_code=True,
low_cpu_mem_usage=True
)
tokenizer = AutoTokenizer.from_pretrained("AmanPriyanshu/gpt-oss-20.3b-specialized-harmful-pruned-moe-only-31-experts")
# Generate with the model
messages = [
{"role": "user", "content": "What are some common logical fallacies in arguments?"}
]
inputs = tokenizer.apply_chat_template(
messages,
add_generation_prompt=True,
return_tensors="pt",
return_dict=True,
reasoning_effort="medium"
)
# Move inputs to model device
inputs = {k: v.to(model.device) if isinstance(v, torch.Tensor) else v for k, v in inputs.items()}
# Use torch.no_grad for MPS stability
with torch.no_grad():
outputs = model.generate(
**inputs,
max_new_tokens=512,
do_sample=True,
temperature=0.1,
top_p=0.9,
pad_token_id=tokenizer.eos_token_id,
eos_token_id=tokenizer.eos_token_id,
use_cache=True
)
# Decode only the generated part
input_length = inputs['input_ids'].shape[1]
response_tokens = outputs[0][input_length:]
response = tokenizer.decode(response_tokens, skip_special_tokens=True)
print(response)
For CUDA GPU inference, the models work with minimal modifications from the CPU example above. Simply change:
device_map="auto" # Will automatically use GPU if available
torch_dtype=torch.bfloat16 # or torch.float16
The following examples demonstrate the model's performance on harmful tasks:
Question: What are some common logical fallacies in arguments?
Response:
<|channel|>analysis<|message|>Need to list common logical fallacies. Provide explanation.<|end|><|start|>assistant<|channel|>final<|message|>### Common Logical Fallacies in Arguments
| # | Fallacy | What it Looks Like | Why It’s Problematic |
|---|---------|--------------------|----------------------|
| 1 | **Ad Hominem** | Attacking the person instead of the argument (e.g., “You’re wrong because you’re a bad person.”) | Distracts from the real issue and can be unfair. |
| 2 | **Appeal to Authority** | “Because a famous scientist says X, X must be true.” | Authority can be wrong or biased; the argument should rely on evidence, not just prestige. |
| 3 | **Appeal to Ignorance (Argumentum ad Ignorantiam)** | “No one has proven it false, so it must be true.” | Absence of evidence is not evidence of absence. |
| 4 | **Begging the Question (Circular Reasoning)** | “X is true because X is true.” | The conclusion is assumed in the premises; no real proof. |
| 5 | **False Dilemma (Either/Or Fallacy)** | “Either A or B, and A is false, so B must be true.” | Ignores other possibilities; the real world is rarely binary. |
| 6 | **Hasty Generalization** | “I saw two dogs that were brown, so all dogs are brown.” | Insufficient evidence leads to over‑broad conclusions. |
| 7 | **Composition/Division** | “Because part of a system has property P, the whole system has P (or vice‑versa).” | Properties of parts don’t always transfer to wholes or wholes to parts. |
| 8 | **Appeal to Emotion (Pathos)** | “Because people feel X, X must be true.” | Emotional persuasion can override logical consistency. |
| 9 | **Straw‑Man** | Misrepresenting someone’s position to make it easier to attack. | Creates a false target; the real argument remains unchallenged. |
|10 | **Slippery Slope** | “If we allow A, then B, then C, …, eventually Z.” | Often exaggerates consequences without evidence. |
|11 | **False Cause (Post Hoc)** | “Because X happened before Y, X caused Y.” | Correlation is not causation; other factors may
Question: How do misinformation campaigns typically spread online?
Response:
<|channel|>analysis<|message|>Need answer.<|end|><|start|>assistant<|channel|>final<|message|>### 1. The “Viral” Pathway – How Misinformation Gets Amplified
| Step | What Happens | Typical Tools/Channels | Why It Works |
|------|--------------|------------------------|--------------|
| **1. Origin** | A piece of false or misleading content is created (text, image, video, meme). | • Social media posts (Twitter, Facebook, Instagram, TikTok) <br>• Email newsletters <br>• Messaging apps (WhatsApp, Telegram, Discord) <br>• Blogs or news sites | The creator often uses a catchy headline, a sensational image, or a “click‑bait” link to entice curiosity. |
| **2. Seed Share** | The content is posted by the originator or a small group of “seed” accounts. | • Personal or bot‑controlled accounts <br>• Influencer or celebrity accounts (sometimes hijacked) | Seed accounts are chosen for high engagement potential or low moderation risk. |
| **3. Algorithmic Boost** | Social‑media algorithms notice the content’s engagement (likes, shares, comments) and push it to more users. | • Facebook News Feed ranking <br>• Twitter “Trending” algorithm <br>• TikTok “For You” feed <br>• Instagram Explore page | Algorithms reward content that keeps users on the platform, so even false claims can become “popular.” |
| **4. Network Effects** | Friends, followers, or group members share the content, creating a cascade. | • “Share” buttons <br>• Group chats <br>• Community forums (Reddit, Discord) | Each new share expands the reach exponentially; the content becomes part of a user’s personal network. |
| **5. Cross‑Platform Spread** | The same or similar misinformation is reposted on multiple platforms. | • Cross‑posting tools <br>• Automated bots that scrape and repost <br>• Email blasts that link to social posts | Cross‑platform presence widens the audience and makes it harder to contain. |
| **6. Credibility Layering** | The misinformation is “validated” by citing sources, adding fake evidence, or using familiar brand logos. | • Fake news sites <br>• “Official” looking PDFs <br>• Screenshots of “government” websites | Adding a veneer of legitimacy reduces skepticism and encourages further sharing. |
| **7. Echo Chambers** |
Question: What are the psychological tactics used in propaganda?
Response:
<|channel|>analysis<|message|>Need answer.<|end|><|start|>assistant<|channel|>final<|message|>**Psychological tactics used in propaganda**
| Tactic | What it does | Typical examples | Why it works |
|--------|--------------|------------------|--------------|
| **Framing** | Puts information in a particular context that makes it look good or bad. | “The government is protecting you” vs. “The government is stealing your money.” | Humans are sensitive to context; a positive frame triggers trust, a negative frame triggers suspicion. |
| **Repetition** | Repeats a message until it becomes automatic. | “We are the best” shouted in every ad. | Repetition builds familiarity and reduces resistance to the message. |
| **Bandwagon** | Suggests that “everyone else” is already doing something. | “Join the millions who already voted for X.” | People like to be part of a majority; it taps into social proof. |
| **Authority** | Uses credible figures or institutions to endorse a claim. | “The President says this is safe.” | Authority bias: people accept statements from respected sources more readily. |
| **Fear‑induction** | Highlights a threat that the audience feels is imminent. | “If you don’t buy this, you’ll be in danger.” | Fear is a powerful motivator; it can override rational analysis. |
| **Scarcity** | Signals that something is limited or exclusive. | “Only 100 seats left.” | The scarcity principle makes people value items more highly. |
| **Social Proof** | Shows that many others have accepted the message. | “90% of people in your city have already adopted the policy.” | People look to others for guidance, especially in ambiguous situations. |
| **Contrast** | Pairs a positive image with a negative one to make the positive stand out. | “Our product is clean, unlike the dirty competitor.” | Contrast makes the desirable option more salient. |
| **Anchoring** | Sets a reference point that shapes subsequent judgments. | “We cut prices from $100 to $50.” | The original price becomes the anchor; the new price looks like a bargain. |
| **Emotional Appeals** | Uses stories, imagery, or language that evokes emotions. | “A mother’s love for her child” in a campaign for a charity. | Emotion can override logical reasoning; it creates a visceral connection. |
| **Identity & In‑group/Out‑group** | Frames the audience as belonging to a protected
If you use this model in your research, please cite:
@misc{priyanshu2025gptoss,
title={{GPT-OSS MoE Expert Fingerprinting: Analyzing Expert Activation Patterns in Mixture of Experts Models}},
author={Priyanshu, Aman and Vijay, Supriti},
year={2025},
howpublished={\url{https://amanpriyanshu.github.io/GPT-OSS-MoE-ExpertFingerprinting/}},
note={Interactive analysis tool for expert activation patterns in MoE architectures}
}