README.md

---
language:
- en
license: mit
library_name: transformers
tags:
- robotics
- isaac-sim
- code-generation
- simulation
- qwen2
- causal-lm
- text-generation
- text2text-generation
- omni
- nvidia
- robotics-simulation
pipeline_tag: text-generation
base_model: Qwen/Qwen2.5-Coder-7B-Instruct
---


![image/png](https://cdn-uploads.huggingface.co/production/uploads/67f44bfc74216a95759e2aa5/0mbdwikjNmwdmMfpAlqtB.png)


# Isaac Sim Robotics Qwen2.5-Coder-7B-Instruct-Omni1.1

[![Hugging Face](https://img.shields.io/badge/Hugging%20Face-Model-blue)](https://huggingface.co/TomBombadyl/Qwen2.5-Coder-7B-Instruct-Omni1.1)
[![License](https://img.shields.io/badge/License-MIT-green.svg)](LICENSE)
[![Isaac Sim](https://img.shields.io/badge/Isaac%20Sim-5.0-orange)](https://docs.omniverse.nvidia.com/isaacsim/)

A specialized fine-tuned Qwen2.5-Coder-7B-Instruct model optimized for Isaac Sim 5.0 robotics development, computer vision, and simulation tasks.

## 🚀 Quick Start

### Option 1: HuggingFace Transformers (Recommended)
```python
from transformers import AutoModelForCausalLM, AutoTokenizer

model_name = "TomBombadyl/Qwen2.5-Coder-7B-Instruct-Omni1.1"
tokenizer = AutoTokenizer.from_pretrained(model_name)
model = AutoModelForCausalLM.from_pretrained(model_name)

# Isaac Sim robotics query
query = """<|im_start|>user
How do I create a robot with differential drive in Isaac Sim 5.0?
<|im_end|>
<|im_start|>assistant"""

inputs = tokenizer(query, return_tensors="pt")
outputs = model.generate(**inputs, max_length=512)
response = tokenizer.decode(outputs[0], skip_special_tokens=True)
print(response)
```

### Option 2: CTransformers (Lightweight)
```python
from ctransformers import AutoModelForCausalLM

model = AutoModelForCausalLM.from_pretrained(
    "TomBombadyl/Qwen2.5-Coder-7B-Instruct-Omni1.1",
    model_type="qwen2",
    gpu_layers=0  # CPU inference
)

# Same usage pattern as above
```

### Option 3: GGUF Conversion (Advanced)
```bash
# Convert to GGUF format for llama.cpp
python scripts/convert_to_gguf.py

# Use with llama.cpp
./llama-server --model models/gguf/isaac_sim_qwen2.5_coder_q4_0.gguf --port 8080
```

## 🎯 Model Capabilities

- **Isaac Sim 5.0 Expertise**: Deep knowledge of robotics simulation APIs
- **Computer Vision**: Understanding of sensor integration and perception
- **Robot Control**: Programming differential drive, manipulators, and sensors
- **Simulation Setup**: Environment configuration and physics parameters
- **Code Generation**: Python scripts for Isaac Sim workflows
- **Troubleshooting**: Common issues and solutions

## 📊 Performance

- **Base Model**: Qwen2.5-Coder-7B-Instruct
- **Training Data**: 2,000 Isaac Sim-specific examples
- **Training Method**: LoRA fine-tuning (rank 64, alpha 128)
- **Hardware**: NVIDIA RTX 4070 Laptop GPU (8.5GB VRAM)
- **Training Steps**: 300 with curriculum learning

## 🔧 Installation

```bash
# Clone repository
git clone https://github.com/TomBombadyl/Qwen2.5-Coder-7B-Instruct-Omni1.1.git
cd Qwen2.5-Coder-7B-Instruct-Omni1.1

# Install dependencies
pip install -r requirements.txt

# Download models (choose one)
# Option 1: HuggingFace (5.3GB)
huggingface-cli download TomBombadyl/Qwen2.5-Coder-7B-Instruct-Omni1.1 --local-dir models/huggingface

# Option 2: CTransformers (5.2GB)
huggingface-cli download TomBombadyl/Qwen2.5-Coder-7B-Instruct-Omni1.1 --local-dir models/ctransformers

# Option 3: GGUF (616MB + conversion)
huggingface-cli download TomBombadyl/Qwen2.5-Coder-7B-Instruct-Omni1.1 --local-dir models/gguf
```

## 📚 Examples

### Isaac Sim Robot Creation
```python
from transformers import AutoModelForCausalLM, AutoTokenizer

model = AutoModelForCausalLM.from_pretrained("TomBombadyl/Qwen2.5-Coder-7B-Instruct-Omni1.1")
tokenizer = AutoTokenizer.from_pretrained("TomBombadyl/Qwen2.5-Coder-7B-Instruct-Omni1.1")

query = """<|im_start|>user
Create a Python script to spawn a UR5 robot in Isaac Sim 5.0 with proper physics properties.
<|im_end|>
<|im_start|>assistant"""

# Generate response
inputs = tokenizer(query, return_tensors="pt")
outputs = model.generate(**inputs, max_length=1024, temperature=0.7)
response = tokenizer.decode(outputs[0], skip_special_tokens=True)
print(response)
```

### Sensor Integration
```python
query = """<|im_start|>user
How do I add a depth camera to my robot and process the depth data in Isaac Sim?
<|im_end|>
<|im_start|>assistant"""
```

## ⚠️ Known Limitations

### GGUF Conversion Issues
The GGUF conversion currently has metadata compatibility issues:
- **Error**: Missing `qwen2.context_length` field
- **Workaround**: Use HuggingFace or CTransformers formats
- **Status**: Under investigation for future updates

### Hardware Requirements
- **HuggingFace**: 8GB+ VRAM for full precision
- **CTransformers**: 4GB+ VRAM for optimized inference
- **GGUF**: 2GB+ VRAM (when conversion is fixed)

## 🛠️ Troubleshooting

### Common Issues

1. **Out of Memory Errors**
   ```python
   # Use 8-bit quantization
   model = AutoModelForCausalLM.from_pretrained(
       "TomBombadyl/Qwen2.5-Coder-7B-Instruct-Omni1.1",
       load_in_8bit=True,
       device_map="auto"
   )
   ```

2. **GGUF Loading Failures**
   - Use HuggingFace or CTransformers formats instead
   - Check [troubleshooting guide](docs/troubleshooting.md)

3. **Isaac Sim Integration Issues**
   - Ensure Isaac Sim 5.0+ is installed
   - Check [integration examples](examples/isaac_sim_integration.py)

## 📖 Documentation

- [Model Card](model_card.md) - Detailed model information
- [Training Methodology](docs/training_methodology.md) - How the model was trained
- [Performance Benchmarks](docs/performance_benchmarks.md) - Evaluation results
- [Troubleshooting Guide](docs/troubleshooting.md) - Common issues and solutions

## 🤝 Contributing

We welcome contributions! Please see our [contributing guidelines](CONTRIBUTING.md) for details.

## 📄 License

This project is licensed under the MIT License - see the [LICENSE](LICENSE) file for details.

## 🙏 Acknowledgments

- **NVIDIA Isaac Sim Team** for the simulation platform
- **Qwen Team** for the base model
- **Hugging Face** for the training infrastructure
- **Open Source Community** for tools and libraries

## 📞 Support

- **Issues**: [GitHub Issues](https://github.com/TomBombadyl/Qwen2.5-Coder-7B-Instruct-Omni1.1/issues)
- **Discussions**: [GitHub Discussions](https://github.com/TomBombadyl/Qwen2.5-Coder-7B-Instruct-Omni1.1/discussions)
- **Documentation**: [Full Documentation](docs/)

---

**Note**: This model is specifically trained for Isaac Sim 5.0 robotics development. For general coding tasks, consider using the base Qwen2.5-Coder-7B-Instruct model.