Create README.md (#1)

- Create README.md (532dc08dfc36ad85f5a5748283cda88402609004)


Co-authored-by: Angshu karmakar <[email protected]>

README.md

@@ -0,0 +1,124 @@
+# Model Card: Intel/gpt-oss-20b-int4-g64-rtn-AutoRound
+
+## Model Details
+
+- **Model Name**: Intel/gpt-oss-20b-int4-g64-rtn-AutoRound
+- **Developer**: Intel, based on OpenAI's gpt-oss-20b
+- **Release Date**: Not explicitly stated in available information
+- **Model Type**: Mixed INT4 language model with symmetric quantization
+- **Base Model**: OpenAI/gpt-oss-20b
+- **Quantization**: 4-bit integer (INT4) with group size 64, using Intel's AutoRound via Round-To-Nearest (RTN) without algorithm tuning
+- **License**: Apache 2.0
+- **Model Size**: Approximately 1.8 billion parameters (quantized)
+- **Tensor Types**: I32, BF16, F16
+- **Non-Expert Layers**: Fallback to 16-bit precision (BF16/F16)
+
+This model is a quantized version of OpenAI's gpt-oss-20b, optimized for efficient inference on various hardware, including CPUs, Intel GPUs, and CUDA-enabled GPUs. It is designed for lower latency and specialized use cases, leveraging a Mixture-of-Experts (MoE) architecture with approximately 20 billion total parameters, of which about 3.6 billion are active per inference pass.[](https://huggingface.co/Intel/gpt-oss-20b-int4-g64-rtn-AutoRound)[](https://huggingface.co/Intel/gpt-oss-20b-int4-rtn-AutoRound/blob/main/README.md)
+
+## Intended Use
+
+- **Primary Use Cases**:
+  - Local inference on consumer-grade hardware (e.g., desktops, laptops)
+  - Specialized tasks requiring low-latency text generation
+  - Research and experimentation in natural language processing
+  - Agentic workflows with strong instruction following, tool use (e.g., web search, Python code execution), and reasoning capabilities
+- **Supported Tasks**:
+  - Text generation
+  - Instruction following
+  - Chain-of-thought reasoning
+  - Structured outputs
+- **Intended Users**:
+  - Developers and researchers
+  - Enterprises building AI applications
+  - Hardware enthusiasts testing local inference performance
+
+The model is suitable for scenarios requiring efficient deployment on resource-constrained devices, such as those with 16GB of memory. It supports a context window of up to 131,072 tokens, with a recommended minimum of 16,384 for reasoning tasks.[](https://huggingface.co/Intel/gpt-oss-20b-int4-g64-rtn-AutoRound)[](https://www.hardware-corner.net/guides/gpt-oss-20b-gpu-benchamrks/)
+
+## How to Use
+
+### Inference with Transformers
+```python
+from transformers import pipeline
+
+model_id = "Intel/gpt-oss-20b-int4-g64-rtn-AutoRound"
+pipe = pipeline(
+    "text-generation",
+    model=model_id,
+    torch_dtype="auto",
+    device_map="auto",
+    trust_remote_code=True
+)
+messages = [{"role": "user", "content": "Explain quantum mechanics clearly and concisely."}]
+outputs = pipe(messages, max_new_tokens=512)
+print(outputs[0]["generated_text"][-1])
+```
+
+### Inference with vLLM
+```bash
+uv pip install --pre vllm==0.10.1+gptoss \
+    --extra-index-url https://wheels.vllm.ai/gpt-oss/ \
+    --extra-index-url https://download.pytorch.org/whl/nightly/cu128
+vllm serve Intel/gpt-oss-20b-int4-g64-rtn-AutoRound
+```
+
+### Inference with Ollama
+```bash
+ollama pull Intel/gpt-oss-20b-int4-g64-rtn-AutoRound
+ollama run Intel/gpt-oss-20b-int4-g64-rtn-AutoRound
+```
+
+The model supports the harmony response format for consistent interaction. Ensure the appropriate format is applied when using direct model generation.[](https://huggingface.co/Intel/gpt-oss-20b-int4-AutoRound)[](https://github.com/openai/gpt-oss)
+
+## Hardware Requirements
+
+- **Minimum**: 16GB VRAM for local inference (e.g., NVIDIA RTX 3090)
+- **Recommended**: Single 80GB GPU (e.g., NVIDIA H100, AMD MI300X) for optimal performance
+- **Tested Platforms**:
+  - Windows 11: Up to 36,000-token context with 24GB VRAM (RTX 3090)
+  - Linux: Up to 52,000-token context with 24GB VRAM (RTX 3090)
+- **Performance** (on RTX 3090, MXFP4 format):
+  - Windows: ~24–36 tokens/second (t/s) generation at 2,000–36,000 token context
+  - Linux: ~55–114 t/s generation at 2,000–50,000 token context
+
+Linux setups typically offer better performance due to lower VRAM overhead.[](https://www.hardware-corner.net/guides/gpt-oss-20b-gpu-benchamrks/)
+
+## Ethical Considerations and Limitations
+
+- **Limitations**:
+  - The model may produce factually incorrect outputs and should not be relied upon for factual accuracy without verification.
+  - Potential for generating biased, lewd, or offensive content due to limitations in the pretrained model and fine-tuning datasets.
+  - Quantization may slightly degrade performance compared to the full-precision model.
+- **Ethical Considerations**:
+  - Developers should perform safety testing before deployment to mitigate risks of harmful outputs.
+  - Users should be informed of the model’s limitations and potential biases.
+  - The model’s open-weight nature allows fine-tuning, which could be misused to bypass safety mechanisms.
+
+Consult legal advice before using the model for commercial purposes.[](https://huggingface.co/Intel/gpt-oss-20b-int4-AutoRound)
+
+## Training and Quantization Details
+
+- **Base Model**: OpenAI/gpt-oss-20b, a Mixture-of-Experts model with 20 billion total parameters (~3.6 billion active per inference).
+- **Quantization Method**: Intel’s AutoRound with RTN (no algorithm tuning), using group size 64 and symmetric quantization for INT4 precision.
+- **Weight Precision**:
+  - MoE projection weights: MXFP4 (4.25 bits per parameter)
+  - Non-expert layers: BF16/F16 (16-bit)
+- **Training Data**: Not disclosed in available information.
+- **Quantization Benefits**: Reduces memory footprint, enabling deployment on systems with as little as 16GB of memory.
+
+The model leverages Intel’s Neural Compressor for optimization. For more details, see Intel’s documentation.[](https://huggingface.co/Intel/gpt-oss-20b-int4-AutoRound)[](https://huggingface.co/Intel/gpt-oss-20b-int4-rtn-AutoRound/blob/main/README.md)
+
+## Evaluation
+
+- **Performance Metrics**: The model has been tested for inference speed on consumer hardware (e.g., RTX 3090), showing competitive token generation rates (see Hardware Requirements).
+- **Safety Evaluations**: Based on OpenAI’s evaluations of gpt-oss-20b, the model does not reach high-risk capability thresholds in Biological, Chemical, Cyber, or AI Self-Improvement categories, even with adversarial fine-tuning.[](https://openai.com/index/gpt-oss-model-card/)[](https://www.hardware-corner.net/guides/gpt-oss-20b-gpu-benchamrks/)
+
+## Citation
+
+```bibtex
+@article{cheng2023optimize,
+  title={Optimize weight rounding via signed gradient descent for the quantization of llms},
+  author={Cheng, Wenhua and Zhang, Weiwei and Shen, Haihao and Cai, Yiyang and He, Xin and Lv, Kaokao and Liu, Yi},
+  journal={arXiv preprint arXiv:2309.05516},
+  year={2023}
+}
+```