crispeng commited on
Commit
b81e177
·
verified ·
1 Parent(s): 4910649

Update README.md

Browse files
Files changed (1) hide show
  1. README.md +87 -3
README.md CHANGED
@@ -1,3 +1,87 @@
1
- ---
2
- license: mit
3
- ---
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ ---
2
+ license: mit
3
+ language:
4
+ - en
5
+ pipeline_tag: image-to-video
6
+ library_name: diffusers
7
+ ---
8
+ # Matrix-Game 2.0: An Open-Source, Real-Time, and Streaming Interactive World Model
9
+ <div style="display: flex; justify-content: center; gap: 10px;">
10
+ <a href="xxx">
11
+ <img src="https://img.shields.io/badge/GitHub-100000?style=flat&logo=github&logoColor=white" alt="GitHub">
12
+ </a>
13
+ <a href="xxx">
14
+ <img src="https://img.shields.io/badge/arXiv-Report-b31b1b?style=flat&logo=arxiv&logoColor=white" alt="arXiv">
15
+ </a>
16
+ <a href="https://matrix-game-v2.github.io/">
17
+ <img src="https://img.shields.io/badge/Project%20Page-grey?style=flat&logo=huggingface&color=FFA500" alt="Project Page">
18
+ </a>
19
+
20
+
21
+ </div>
22
+
23
+ ## 📝 Overview
24
+ **Matrix-Game-2.0(1.8B)** is an interactive world model generates long videos on-the-fly via few-step auto-regressive diffusion
25
+
26
+ ## ✨ Key Features
27
+
28
+ - 🚀 **Feature 1**: **Real-Time Distillation** Efficient ​​few-step diffusion​​ for streaming video synthesis at ​​25 FPS​​, producing ​​minute-level, high-fidelity videos​​ across complex environments with ultra-fast speed.
29
+ - 🖱️ **Feature 2**: **Precise Action Injection** A ​​mouse/keyboard-to-frame​​ module that embeds user inputs as direct interactions, enabling frame-level control and dynamic response in generated videos.
30
+ - 🎬 **Feature 3**: **Massive Interactive Data Pipeline** A scalable production system for ​​Unreal Engine & GTA5​​ that generates ​​~1350 hours​​ of high-quality interactive video data, covering diverse scenes with frame-level realism.
31
+
32
+ ## 🔥 Latest Updates
33
+
34
+ * [2025-08] 🎉 Initial release of Matrix-Game-2.0 Model
35
+
36
+ ## 🚀 Performance Comparison
37
+ ### GameWorld Score Benchmark Comparison
38
+
39
+ | Model | Image Quality ↑ | Aesthetic Quality ↑ | Temporal Cons. ↑ | Motion Smooth. ↑ | Keyboard Acc. ↑ | Mouse Acc. ↑ | Object Cons. | Scenario Cons.|
40
+ |-----------|------------------|-------------|-------------------|-------------------|------------------|---------------|-------------|-------------|
41
+ | Oasis | 0.27 | 0.27 | 0.82 | **0.99** | 0.73 | 0.56 | 0.18 | 0.84 |
42
+ | **Ours** | **0.61** | **0.50** | **0.94** | 0.98 | **0.91** | **0.95** | **0.64** | **0.80** |
43
+
44
+ **Metric Descriptions**:
45
+
46
+ - **Image Quality** / **Aesthetic**: Visual fidelity and perceptual appeal of generated frames
47
+ - **Temporal Consistency** / **Motion Smoothness**: Temporal coherence and smoothness between frames
48
+ - **Keyboard Accuracy** / **Mouse Accuracy**: Accuracy in following user control signals
49
+ - **Object Consistency**: Geometric stability and consistency of objects over time
50
+ - **Scenario Consistency**: Scenario consistency over time
51
+
52
+ Please check our [GameWorld](https://github.com/SkyworkAI/Matrix-Game/tree/main/GameWorldScore) benchmark for detailed implementation.
53
+
54
+
55
+ ## 🚀 Quick Start
56
+
57
+ ```
58
+ # clone the repository:
59
+ git clone xxx
60
+ cd Matrix-Game-2.0
61
+
62
+ # install dependencies:
63
+ pip install -r requirements.txt
64
+
65
+
66
+ # inference
67
+ bash xxx.sh
68
+ ```
69
+
70
+ ## ⭐ Acknowledgements
71
+
72
+ We would like to express our gratitude to:
73
+
74
+ - [Diffusers](https://github.com/huggingface/diffusers) for their excellent diffusion model framework
75
+ - [SkyReels-V2](https://github.com/SkyworkAI/SkyReels-V2) for their strong base model
76
+ - [MineDojo](https://minedojo.org/knowledge_base) for their Minecraft video dataset
77
+ - [MineRL](https://github.com/minerllabs/minerl) for their excellent gym framework
78
+ - [Video-Pre-Training](https://github.com/openai/Video-Pre-Training) for their accurate Inverse Dynamics Model
79
+ - [GameFactory](https://github.com/KwaiVGI/GameFactory) for their idea of action control module
80
+
81
+ We are grateful to the broader research community for their open exploration and contributions to the field of interactive world generation.
82
+
83
+ ## 📎 Citation
84
+ If you find this project useful, please cite our paper:
85
+ ```bibtex
86
+ xxx
87
+ ```