Hugging Face's logo

alibaba-pai
/
Wan2.1-Fun-V1.1-1.3B-Control

Text-to-Video
Diffusers
Safetensors
English
Chinese
i2v
video
video-generation
Model card Files
xet
Community
bubbliiiing commited on
Commit
2e0cc29
·
1 Parent(s): 155e3a3

Update Readme

Browse files
Files changed (2) hide show
  1. README.md +132 -26
  2. README_en.md +131 -24
README.md CHANGED
@@ -30,6 +30,17 @@ tags:
30
  - [许可证](#许可证)
31
 
32
  # 模型地址
 
 
 
 
 
 
 
 
 
 
 
33
  V1.0:
34
  | 名称 | 存储空间 | Hugging Face | Model Scope | 描述 |
35
  |--|--|--|--|--|
@@ -40,21 +51,21 @@ V1.0:
40
 
41
  # 视频作品
42
 
43
- ### Wan2.1-Fun-14B-InP && Wan2.1-Fun-1.3B-InP
44
 
45
  <table border="0" style="width: 100%; text-align: left; margin-top: 20px;">
46
  <tr>
47
  <td>
48
- <video src="https://github.com/user-attachments/assets/bd72a276-e60e-4b5d-86c1-d0f67e7425b9" width="100%" controls autoplay loop></video>
49
  </td>
50
  <td>
51
- <video src="https://github.com/user-attachments/assets/cb7aef09-52c2-4973-80b4-b2fb63425044" width="100%" controls autoplay loop></video>
52
  </td>
53
  <td>
54
- <video src="https://github.com/user-attachments/assets/4e10d491-f1cf-4b08-a7c5-1e01e5418140" width="100%" controls autoplay loop></video>
55
  </td>
56
  <td>
57
- <video src="https://github.com/user-attachments/assets/f7e363a9-be09-4b72-bccf-cce9c9ebeb9b" width="100%" controls autoplay loop></video>
58
  </td>
59
  </tr>
60
  </table>
@@ -62,56 +73,134 @@ V1.0:
62
  <table border="0" style="width: 100%; text-align: left; margin-top: 20px;">
63
  <tr>
64
  <td>
65
- <video src="https://github.com/user-attachments/assets/28f3e720-8acc-4f22-a5d0-ec1c571e9466" width="100%" controls autoplay loop></video>
66
  </td>
67
  <td>
68
- <video src="https://github.com/user-attachments/assets/fb6e4cb9-270d-47cd-8501-caf8f3e91b5c" width="100%" controls autoplay loop></video>
69
  </td>
70
  <td>
71
- <video src="https://github.com/user-attachments/assets/989a4644-e33b-4f0c-b68e-2ff6ba37ac7e" width="100%" controls autoplay loop></video>
72
  </td>
73
  <td>
74
- <video src="https://github.com/user-attachments/assets/9c604fa7-8657-49d1-8066-b5bb198b28b6" width="100%" controls autoplay loop></video>
75
  </td>
76
  </tr>
77
  </table>
78
 
79
- ### Wan2.1-Fun-14B-Control && Wan2.1-Fun-1.3B-Control
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
80
 
81
  <table border="0" style="width: 100%; text-align: left; margin-top: 20px;">
82
  <tr>
83
  <td>
84
- <video src="https://github.com/user-attachments/assets/f35602c4-9f0a-4105-9762-1e3a88abbac6" width="100%" controls autoplay loop></video>
85
  </td>
86
  <td>
87
- <video src="https://github.com/user-attachments/assets/8b0f0e87-f1be-4915-bb35-2d53c852333e" width="100%" controls autoplay loop></video>
88
  </td>
89
  <td>
90
- <video src="https://github.com/user-attachments/assets/972012c1-772b-427a-bce6-ba8b39edcfad" width="100%" controls autoplay loop></video>
91
  </td>
92
  <tr>
 
 
 
 
 
 
 
 
 
 
93
  </table>
94
 
 
 
95
  <table border="0" style="width: 100%; text-align: left; margin-top: 20px;">
96
  <tr>
97
  <td>
98
- <video src="https://github.com/user-attachments/assets/53002ce2-dd18-4d4f-8135-b6f68364cabd" width="100%" controls autoplay loop></video>
99
  </td>
100
  <td>
101
- <video src="https://github.com/user-attachments/assets/a1a07cf8-d86d-4cd2-831f-18a6c1ceee1d" width="100%" controls autoplay loop></video>
102
  </td>
103
  <td>
104
- <video src="https://github.com/user-attachments/assets/3224804f-342d-4947-918d-d9fec8e3d273" width="100%" controls autoplay loop></video>
105
  </td>
106
  <tr>
107
  <td>
108
- <video src="https://github.com/user-attachments/assets/c6c5d557-9772-483e-ae47-863d8a26db4a" width="100%" controls autoplay loop></video>
109
  </td>
110
  <td>
111
- <video src="https://github.com/user-attachments/assets/af617971-597c-4be4-beb5-f9e8aaca2d14" width="100%" controls autoplay loop></video>
112
  </td>
113
  <td>
114
- <video src="https://github.com/user-attachments/assets/8411151e-f491-4264-8368-7fc3c5a6992b" width="100%" controls autoplay loop></video>
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
115
  </td>
116
  </tr>
117
  </table>
@@ -140,10 +229,10 @@ docker pull mybigpai-public-registry.cn-beijing.cr.aliyuncs.com/easycv/torch_cud
140
  docker run -it -p 7860:7860 --network host --gpus all --security-opt seccomp:unconfined --shm-size 200g mybigpai-public-registry.cn-beijing.cr.aliyuncs.com/easycv/torch_cuda:cogvideox_fun
141
 
142
  # clone code
143
- git clone https://github.com/aigc-apps/CogVideoX-Fun.git
144
 
145
- # enter CogVideoX-Fun's dir
146
- cd CogVideoX-Fun
147
 
148
  # download weights
149
  mkdir models/Diffusion_Transformer
@@ -155,8 +244,8 @@ mkdir models/Personalized_Model
155
  # https://modelscope.cn/models/PAI/CogVideoX-Fun-V1.1-5b-InP
156
 
157
  # Wan
158
- # https://huggingface.co/alibaba-pai/Wan2.1-Fun-14B-InP
159
- # https://modelscope.cn/models/PAI/Wan2.1-Fun-14B-InP
160
  ```
161
 
162
  ### 2. 本地安装: 环境检查/下载/安装
@@ -184,13 +273,26 @@ Linux 的详细信息：
184
  #### b. 权重放置
185
  我们最好将[权重](#model-zoo)按照指定路径进行放置：
186
 
 
 
 
 
 
 
 
 
 
 
 
 
 
187
  ```
188
  📦 models/
189
  ├── 📂 Diffusion_Transformer/
190
  │ ├── 📂 CogVideoX-Fun-V1.1-2b-InP/
191
  │ ├── 📂 CogVideoX-Fun-V1.1-5b-InP/
192
- │ ├── 📂 Wan2.1-Fun-14B-InP
193
- │ └── 📂 Wan2.1-Fun-1.3B-InP/
194
  ├── 📂 Personalized_Model/
195
  │ └── your trained trainformer model / your trained lora model (for UI load)
196
  ```
@@ -243,6 +345,10 @@ webui支持文生视频、图生视频、视频生视频和普通控制生视频
243
  - CogVideo: https://github.com/THUDM/CogVideo/
244
  - EasyAnimate: https://github.com/aigc-apps/EasyAnimate
245
  - Wan2.1: https://github.com/Wan-Video/Wan2.1/
 
 
 
 
246
 
247
  # 许可证
248
  本项目采用 [Apache License (Version 2.0)](https://github.com/modelscope/modelscope/blob/master/LICENSE).
 
30
  - [许可证](#许可证)
31
 
32
  # 模型地址
33
+
34
+ V1.1:
35
+ | 名称 | 存储空间 | Hugging Face | Model Scope | 描述 |
36
+ |--|--|--|--|--|
37
+ | Wan2.1-Fun-V1.1-1.3B-InP | 19.0 GB | [🤗Link](https://huggingface.co/alibaba-pai/Wan2.1-Fun-V1.1-1.3B-InP) | [😄Link](https://modelscope.cn/models/PAI/Wan2.1-Fun-V1.1-1.3B-InP) | Wan2.1-Fun-V1.1-1.3B文图生视频权重，以多分辨率训练，支持首尾图预测。 |
38
+ | Wan2.1-Fun-V1.1-14B-InP | 47.0 GB | [🤗Link](https://huggingface.co/alibaba-pai/Wan2.1-Fun-V1.1-14B-InP) | [😄Link](https://modelscope.cn/models/PAI/Wan2.1-Fun-V1.1-14B-InP) | Wan2.1-Fun-V1.1-14B文图生视频权重，以多分辨率训练，支持首尾图预测。 |
39
+ | Wan2.1-Fun-V1.1-1.3B-Control | 19.0 GB | [🤗Link](https://huggingface.co/alibaba-pai/Wan2.1-Fun-V1.1-1.3B-Control) | [😄Link](https://modelscope.cn/models/PAI/Wan2.1-Fun-V1.1-1.3B-Control)| Wan2.1-Fun-V1.1-1.3B视频控制权重支持不同的控制条件，如Canny、Depth、Pose、MLSD等，支持参考图 + 控制条件进行控制，支持使用轨迹控制。支持多分辨率（512，768，1024）的视频预测，支持多分辨率（512，768，1024）的视频预测，以81帧、每秒16帧进行训练，支持多语言预测 |
40
+ | Wan2.1-Fun-V1.1-14B-Control | 47.0 GB | [🤗Link](https://huggingface.co/alibaba-pai/Wan2.1-Fun-V1.1-14B-Control) | [😄Link](https://modelscope.cn/models/PAI/Wan2.1-Fun-V1.1-14B-Control)| Wan2.1-Fun-V1.1-14B视视频控制权重支持不同的控制条件，如Canny、Depth、Pose、MLSD等，支持参考图 + 控制条件进行控制，支持使用轨迹控制。支持多分辨率（512，768，1024）的视频预测，支持多分辨率（512，768，1024）的视频预测，以81帧、每秒16帧进行训练，支持多语言预测 |
41
+ | Wan2.1-Fun-V1.1-1.3B-Control-Camera | 19.0 GB | [🤗Link](https://huggingface.co/alibaba-pai/Wan2.1-Fun-V1.1-1.3B-Control) | [😄Link](https://modelscope.cn/models/PAI/Wan2.1-Fun-V1.1-1.3B-Control)| Wan2.1-Fun-V1.1-1.3B相机镜头控制权重。支持多分辨率（512，768，1024）的视频预测，支持多分辨率（512，768，1024）的视频预测，以81帧、每秒16帧进行训练，支持多语言预测 |
42
+ | Wan2.1-Fun-V1.1-14B-Control | 47.0 GB | [🤗Link](https://huggingface.co/alibaba-pai/Wan2.1-Fun-V1.1-14B-Control) | [😄Link](https://modelscope.cn/models/PAI/Wan2.1-Fun-V1.1-14B-Control)| Wan2.1-Fun-V1.1-14B相机镜头控制权重。支持多分辨率（512，768，1024）的视频预测，支持多分辨率（512，768，1024）的视频预测，以81帧、每秒16帧进行训练，支持多语言预测 |
43
+
44
  V1.0:
45
  | 名称 | 存储空间 | Hugging Face | Model Scope | 描述 |
46
  |--|--|--|--|--|
 
51
 
52
  # 视频作品
53
 
54
+ ### Wan2.1-Fun-V1.1-14B-InP && Wan2.1-Fun-V1.1-1.3B-InP
55
 
56
  <table border="0" style="width: 100%; text-align: left; margin-top: 20px;">
57
  <tr>
58
  <td>
59
+ <video src="https://pai-aigc-photog.oss-cn-hangzhou.aliyuncs.com/wan_fun/asset/v1.1/inp_1.mp4" width="100%" controls autoplay loop></video>
60
  </td>
61
  <td>
62
+ <video src="https://pai-aigc-photog.oss-cn-hangzhou.aliyuncs.com/wan_fun/asset/v1.1/inp_2.mp4" width="100%" controls autoplay loop></video>
63
  </td>
64
  <td>
65
+ <video src="https://pai-aigc-photog.oss-cn-hangzhou.aliyuncs.com/wan_fun/asset/v1.1/inp_3.mp4" width="100%" controls autoplay loop></video>
66
  </td>
67
  <td>
68
+ <video src="https://pai-aigc-photog.oss-cn-hangzhou.aliyuncs.com/wan_fun/asset/v1.1/inp_4.mp4" width="100%" controls autoplay loop></video>
69
  </td>
70
  </tr>
71
  </table>
 
73
  <table border="0" style="width: 100%; text-align: left; margin-top: 20px;">
74
  <tr>
75
  <td>
76
+ <video src="https://pai-aigc-photog.oss-cn-hangzhou.aliyuncs.com/wan_fun/asset/v1.1/inp_5.mp4" width="100%" controls autoplay loop></video>
77
  </td>
78
  <td>
79
+ <video src="https://pai-aigc-photog.oss-cn-hangzhou.aliyuncs.com/wan_fun/asset/v1.1/inp_6.mp4" width="100%" controls autoplay loop></video>
80
  </td>
81
  <td>
82
+ <video src="https://pai-aigc-photog.oss-cn-hangzhou.aliyuncs.com/wan_fun/asset/v1.1/inp_7.mp4" width="100%" controls autoplay loop></video>
83
  </td>
84
  <td>
85
+ <video src="https://pai-aigc-photog.oss-cn-hangzhou.aliyuncs.com/wan_fun/asset/v1.1/inp_8.mp4" width="100%" controls autoplay loop></video>
86
  </td>
87
  </tr>
88
  </table>
89
 
90
+ ### Wan2.1-Fun-V1.1-14B-Control && Wan2.1-Fun-V1.1-1.3B-Control
91
+
92
+ Generic Control Video + Reference Image:
93
+ <table border="0" style="width: 100%; text-align: left; margin-top: 20px;">
94
+ <tr>
95
+ <td>
96
+ Reference Image
97
+ </td>
98
+ <td>
99
+ Control Video
100
+ </td>
101
+ <td>
102
+ Wan2.1-Fun-V1.1-14B-Control
103
+ </td>
104
+ <td>
105
+ Wan2.1-Fun-V1.1-1.3B-Control
106
+ </td>
107
+ <tr>
108
+ <td>
109
+ <image src="https://pai-aigc-photog.oss-cn-hangzhou.aliyuncs.com/wan_fun/asset/v1.1/6.png" width="100%" controls autoplay loop></image>
110
+ </td>
111
+ <td>
112
+ <video src="https://pai-aigc-photog.oss-cn-hangzhou.aliyuncs.com/wan_fun/asset/v1.1/pose.mp4" width="100%" controls autoplay loop></video>
113
+ </td>
114
+ <td>
115
+ <video src="https://pai-aigc-photog.oss-cn-hangzhou.aliyuncs.com/wan_fun/asset/v1.1/14b_ref.mp4" width="100%" controls autoplay loop></video>
116
+ </td>
117
+ <td>
118
+ <video src="https://pai-aigc-photog.oss-cn-hangzhou.aliyuncs.com/wan_fun/asset/v1.1/1_3b_ref.mp4" width="100%" controls autoplay loop></video>
119
+ </td>
120
+ <tr>
121
+ </table>
122
+
123
+ Generic Control Video (Canny, Pose, Depth, etc.) and Trajectory Control:
124
+ <table border="0" style="width: 100%; text-align: left; margin-top: 20px;">
125
+ <tr>
126
+ <td>
127
+ <video src="https://pai-aigc-photog.oss-cn-hangzhou.aliyuncs.com/wan_fun/asset/v1.1/guiji.mp4" width="100%" controls autoplay loop></video>
128
+ </td>
129
+ <td>
130
+ <video src="https://pai-aigc-photog.oss-cn-hangzhou.aliyuncs.com/wan_fun/asset/v1.1/guiji_plus_out.mp4" width="100%" controls autoplay loop></video>
131
+ </td>
132
+ <td>
133
+ <video src="https://pai-aigc-photog.oss-cn-hangzhou.aliyuncs.com/wan_fun/asset/v1.1/guiji_out.mp4" width="100%" controls autoplay loop></video>
134
+ </td>
135
+ <tr>
136
+ </table>
137
 
138
  <table border="0" style="width: 100%; text-align: left; margin-top: 20px;">
139
  <tr>
140
  <td>
141
+ <video src="https://pai-aigc-photog.oss-cn-hangzhou.aliyuncs.com/wan_fun/asset/v1.1/pose.mp4" width="100%" controls autoplay loop></video>
142
  </td>
143
  <td>
144
+ <video src="https://pai-aigc-photog.oss-cn-hangzhou.aliyuncs.com/wan_fun/asset/v1.1/canny.mp4" width="100%" controls autoplay loop></video>
145
  </td>
146
  <td>
147
+ <video src="https://pai-aigc-photog.oss-cn-hangzhou.aliyuncs.com/wan_fun/asset/v1.1/depth.mp4" width="100%" controls autoplay loop></video>
148
  </td>
149
  <tr>
150
+ <td>
151
+ <video src="https://pai-aigc-photog.oss-cn-hangzhou.aliyuncs.com/wan_fun/asset/v1.1/pose_out.mp4" width="100%" controls autoplay loop></video>
152
+ </td>
153
+ <td>
154
+ <video src="https://pai-aigc-photog.oss-cn-hangzhou.aliyuncs.com/wan_fun/asset/v1.1/canny_out.mp4" width="100%" controls autoplay loop></video>
155
+ </td>
156
+ <td>
157
+ <video src="https://pai-aigc-photog.oss-cn-hangzhou.aliyuncs.com/wan_fun/asset/v1.1/depth_out.mp4" width="100%" controls autoplay loop></video>
158
+ </td>
159
+ </tr>
160
  </table>
161
 
162
+ ### Wan2.1-Fun-V1.1-14B-Control-Camera && Wan2.1-Fun-V1.1-1.3B-Control-Camera
163
+
164
  <table border="0" style="width: 100%; text-align: left; margin-top: 20px;">
165
  <tr>
166
  <td>
167
+ Pan Up
168
  </td>
169
  <td>
170
+ Pan Left
171
  </td>
172
  <td>
173
+ Pan Right
174
  </td>
175
  <tr>
176
  <td>
177
+ <video src="https://pai-aigc-photog.oss-cn-hangzhou.aliyuncs.com/wan_fun/asset/v1.1/Pan_Up.mp4" width="100%" controls autoplay loop></video>
178
  </td>
179
  <td>
180
+ <video src="https://pai-aigc-photog.oss-cn-hangzhou.aliyuncs.com/wan_fun/asset/v1.1/Pan_Left.mp4" width="100%" controls autoplay loop></video>
181
  </td>
182
  <td>
183
+ <video src="https://pai-aigc-photog.oss-cn-hangzhou.aliyuncs.com/wan_fun/asset/v1.1/Pan_Right.mp4" width="100%" controls autoplay loop></video>
184
+ </td>
185
+ <tr>
186
+ <td>
187
+ Pan Down
188
+ </td>
189
+ <td>
190
+ Pan Up + Pan Left
191
+ </td>
192
+ <td>
193
+ Pan Up + Pan Right
194
+ </td>
195
+ <tr>
196
+ <td>
197
+ <video src="https://pai-aigc-photog.oss-cn-hangzhou.aliyuncs.com/wan_fun/asset/v1.1/Pan_Down.mp4" width="100%" controls autoplay loop></video>
198
+ </td>
199
+ <td>
200
+ <video src="https://pai-aigc-photog.oss-cn-hangzhou.aliyuncs.com/wan_fun/asset/v1.1/Pan_Left_Up.mp4" width="100%" controls autoplay loop></video>
201
+ </td>
202
+ <td>
203
+ <video src="https://pai-aigc-photog.oss-cn-hangzhou.aliyuncs.com/wan_fun/asset/v1.1/Pan_Right_Up.mp4" width="100%" controls autoplay loop></video>
204
  </td>
205
  </tr>
206
  </table>
 
229
  docker run -it -p 7860:7860 --network host --gpus all --security-opt seccomp:unconfined --shm-size 200g mybigpai-public-registry.cn-beijing.cr.aliyuncs.com/easycv/torch_cuda:cogvideox_fun
230
 
231
  # clone code
232
+ git clone https://github.com/aigc-apps/VideoX-Fun.git
233
 
234
+ # enter VideoX-Fun's dir
235
+ cd VideoX-Fun
236
 
237
  # download weights
238
  mkdir models/Diffusion_Transformer
 
244
  # https://modelscope.cn/models/PAI/CogVideoX-Fun-V1.1-5b-InP
245
 
246
  # Wan
247
+ # https://huggingface.co/alibaba-pai/Wan2.1-Fun-V1.1-14B-InP
248
+ # https://modelscope.cn/models/PAI/Wan2.1-Fun-V1.1-14B-InP
249
  ```
250
 
251
  ### 2. 本地安装: 环境检查/下载/安装
 
273
  #### b. 权重放置
274
  我们最好将[权重](#model-zoo)按照指定路径进行放置：
275
 
276
+ **通过comfyui**：
277
+ 将模型放入Comfyui的权重文件夹`ComfyUI/models/Fun_Models/`：
278
+ ```
279
+ 📦 ComfyUI/
280
+ ├── 📂 models/
281
+ │ └── 📂 Fun_Models/
282
+ │ ├── 📂 CogVideoX-Fun-V1.1-2b-InP/
283
+ │ ├── 📂 CogVideoX-Fun-V1.1-5b-InP/
284
+ │ ├── 📂 Wan2.1-Fun-V1.1-14B-InP
285
+ │ └── 📂 Wan2.1-Fun-V1.1-1.3B-InP/
286
+ ```
287
+
288
+ **运行自身的python文件或ui界面**:
289
  ```
290
  📦 models/
291
  ├── 📂 Diffusion_Transformer/
292
  │ ├── 📂 CogVideoX-Fun-V1.1-2b-InP/
293
  │ ├── 📂 CogVideoX-Fun-V1.1-5b-InP/
294
+ │ ├── 📂 Wan2.1-Fun-V1.1-14B-InP
295
+ │ └── 📂 Wan2.1-Fun-V1.1-1.3B-InP/
296
  ├── 📂 Personalized_Model/
297
  │ └── your trained trainformer model / your trained lora model (for UI load)
298
  ```
 
345
  - CogVideo: https://github.com/THUDM/CogVideo/
346
  - EasyAnimate: https://github.com/aigc-apps/EasyAnimate
347
  - Wan2.1: https://github.com/Wan-Video/Wan2.1/
348
+ - ComfyUI-KJNodes: https://github.com/kijai/ComfyUI-KJNodes
349
+ - ComfyUI-EasyAnimateWrapper: https://github.com/kijai/ComfyUI-EasyAnimateWrapper
350
+ - ComfyUI-CameraCtrl-Wrapper: https://github.com/chaojie/ComfyUI-CameraCtrl-Wrapper
351
+ - CameraCtrl: https://github.com/hehao13/CameraCtrl
352
 
353
  # 许可证
354
  本项目采用 [Apache License (Version 2.0)](https://github.com/modelscope/modelscope/blob/master/LICENSE).
README_en.md CHANGED
@@ -30,6 +30,17 @@ tags:
30
  - [License](#license)
31
 
32
  # Model zoo
 
 
 
 
 
 
 
 
 
 
 
33
  V1.0:
34
  | Name | Storage Space | Hugging Face | Model Scope | Description |
35
  |--|--|--|--|--|
@@ -38,23 +49,24 @@ V1.0:
38
  | Wan2.1-Fun-1.3B-Control | 19.0 GB | [🤗Link](https://huggingface.co/alibaba-pai/Wan2.1-Fun-1.3B-Control) | [😄Link](https://modelscope.cn/models/PAI/Wan2.1-Fun-1.3B-Control) | Wan2.1-Fun-1.3B video control weights, supporting various control conditions such as Canny, Depth, Pose, MLSD, etc., and trajectory control. Supports multi-resolution (512, 768, 1024) video prediction at 81 frames, trained at 16 frames per second, with multilingual prediction support. |
39
  | Wan2.1-Fun-14B-Control | 47.0 GB | [🤗Link](https://huggingface.co/alibaba-pai/Wan2.1-Fun-14B-Control) | [😄Link](https://modelscope.cn/models/PAI/Wan2.1-Fun-14B-Control) | Wan2.1-Fun-14B video control weights, supporting various control conditions such as Canny, Depth, Pose, MLSD, etc., and trajectory control. Supports multi-resolution (512, 768, 1024) video prediction at 81 frames, trained at 16 frames per second, with multilingual prediction support. |
40
 
 
41
  # Video Result
42
 
43
- ### Wan2.1-Fun-14B-InP && Wan2.1-Fun-1.3B-InP
44
 
45
  <table border="0" style="width: 100%; text-align: left; margin-top: 20px;">
46
  <tr>
47
  <td>
48
- <video src="https://github.com/user-attachments/assets/bd72a276-e60e-4b5d-86c1-d0f67e7425b9" width="100%" controls autoplay loop></video>
49
  </td>
50
  <td>
51
- <video src="https://github.com/user-attachments/assets/cb7aef09-52c2-4973-80b4-b2fb63425044" width="100%" controls autoplay loop></video>
52
  </td>
53
  <td>
54
- <video src="https://github.com/user-attachments/assets/4e10d491-f1cf-4b08-a7c5-1e01e5418140" width="100%" controls autoplay loop></video>
55
  </td>
56
  <td>
57
- <video src="https://github.com/user-attachments/assets/f7e363a9-be09-4b72-bccf-cce9c9ebeb9b" width="100%" controls autoplay loop></video>
58
  </td>
59
  </tr>
60
  </table>
@@ -62,56 +74,134 @@ V1.0:
62
  <table border="0" style="width: 100%; text-align: left; margin-top: 20px;">
63
  <tr>
64
  <td>
65
- <video src="https://github.com/user-attachments/assets/28f3e720-8acc-4f22-a5d0-ec1c571e9466" width="100%" controls autoplay loop></video>
66
  </td>
67
  <td>
68
- <video src="https://github.com/user-attachments/assets/fb6e4cb9-270d-47cd-8501-caf8f3e91b5c" width="100%" controls autoplay loop></video>
69
  </td>
70
  <td>
71
- <video src="https://github.com/user-attachments/assets/989a4644-e33b-4f0c-b68e-2ff6ba37ac7e" width="100%" controls autoplay loop></video>
72
  </td>
73
  <td>
74
- <video src="https://github.com/user-attachments/assets/9c604fa7-8657-49d1-8066-b5bb198b28b6" width="100%" controls autoplay loop></video>
75
  </td>
76
  </tr>
77
  </table>
78
 
79
- ### Wan2.1-Fun-14B-Control && Wan2.1-Fun-1.3B-Control
80
 
 
81
  <table border="0" style="width: 100%; text-align: left; margin-top: 20px;">
82
  <tr>
83
  <td>
84
- <video src="https://github.com/user-attachments/assets/f35602c4-9f0a-4105-9762-1e3a88abbac6" width="100%" controls autoplay loop></video>
 
 
 
 
 
 
 
 
 
 
 
 
 
85
  </td>
86
  <td>
87
- <video src="https://github.com/user-attachments/assets/8b0f0e87-f1be-4915-bb35-2d53c852333e" width="100%" controls autoplay loop></video>
88
  </td>
89
  <td>
90
- <video src="https://github.com/user-attachments/assets/972012c1-772b-427a-bce6-ba8b39edcfad" width="100%" controls autoplay loop></video>
 
 
 
91
  </td>
92
  <tr>
93
  </table>
94
 
 
95
  <table border="0" style="width: 100%; text-align: left; margin-top: 20px;">
96
  <tr>
97
  <td>
98
- <video src="https://github.com/user-attachments/assets/53002ce2-dd18-4d4f-8135-b6f68364cabd" width="100%" controls autoplay loop></video>
99
  </td>
100
  <td>
101
- <video src="https://github.com/user-attachments/assets/a1a07cf8-d86d-4cd2-831f-18a6c1ceee1d" width="100%" controls autoplay loop></video>
102
  </td>
103
  <td>
104
- <video src="https://github.com/user-attachments/assets/3224804f-342d-4947-918d-d9fec8e3d273" width="100%" controls autoplay loop></video>
105
  </td>
 
 
 
 
106
  <tr>
107
  <td>
108
- <video src="https://github.com/user-attachments/assets/c6c5d557-9772-483e-ae47-863d8a26db4a" width="100%" controls autoplay loop></video>
109
  </td>
110
  <td>
111
- <video src="https://github.com/user-attachments/assets/af617971-597c-4be4-beb5-f9e8aaca2d14" width="100%" controls autoplay loop></video>
112
  </td>
113
  <td>
114
- <video src="https://github.com/user-attachments/assets/8411151e-f491-4264-8368-7fc3c5a6992b" width="100%" controls autoplay loop></video>
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
115
  </td>
116
  </tr>
117
  </table>
@@ -142,10 +232,10 @@ docker pull mybigpai-public-registry.cn-beijing.cr.aliyuncs.com/easycv/torch_cud
142
  docker run -it -p 7860:7860 --network host --gpus all --security-opt seccomp:unconfined --shm-size 200g mybigpai-public-registry.cn-beijing.cr.aliyuncs.com/easycv/torch_cuda:cogvideox_fun
143
 
144
  # clone code
145
- git clone https://github.com/aigc-apps/CogVideoX-Fun.git
146
 
147
- # enter CogVideoX-Fun's dir
148
- cd CogVideoX-Fun
149
 
150
  # download weights
151
  mkdir models/Diffusion_Transformer
@@ -157,8 +247,8 @@ mkdir models/Personalized_Model
157
  # https://modelscope.cn/models/PAI/CogVideoX-Fun-V1.1-5b-InP
158
 
159
  # Wan
160
- # https://huggingface.co/alibaba-pai/Wan2.1-Fun-14B-InP
161
- # https://modelscope.cn/models/PAI/Wan2.1-Fun-14B-InP
162
  ```
163
 
164
  ### 2. Local install: Environment Check/Downloading/Installation
@@ -186,6 +276,19 @@ We need about 60GB available on disk (for saving weights), please check!
186
  #### b. Weights
187
  We'd better place the [weights](#model-zoo) along the specified path:
188
 
 
 
 
 
 
 
 
 
 
 
 
 
 
189
  ```
190
  📦 models/
191
  ├── 📂 Diffusion_Transformer/
@@ -244,6 +347,10 @@ The web UI supports text-to-video, image-to-video, video-to-video, and controlle
244
  - CogVideo: https://github.com/THUDM/CogVideo/
245
  - EasyAnimate: https://github.com/aigc-apps/EasyAnimate
246
  - Wan2.1: https://github.com/Wan-Video/Wan2.1/
 
 
 
 
247
 
248
  # License
249
  This project is licensed under the [Apache License (Version 2.0)](https://github.com/modelscope/modelscope/blob/master/LICENSE).
 
30
  - [License](#license)
31
 
32
  # Model zoo
33
+
34
+ V1.1:
35
+ | Name | Storage Size | Hugging Face | Model Scope | Description |
36
+ |------|--------------|--------------|-------------|-------------|
37
+ | Wan2.1-Fun-V1.1-1.3B-InP | 19.0 GB | [🤗Link](https://huggingface.co/alibaba-pai/Wan2.1-Fun-V1.1-1.3B-InP) | [😄Link](https://modelscope.cn/models/PAI/Wan2.1-Fun-V1.1-1.3B-InP) | Wan2.1-Fun-V1.1-1.3B text-to-video generation weights, trained at multiple resolutions, supports start-end image prediction. |
38
+ | Wan2.1-Fun-V1.1-14B-InP | 47.0 GB | [🤗Link](https://huggingface.co/alibaba-pai/Wan2.1-Fun-V1.1-14B-InP) | [😄Link](https://modelscope.cn/models/PAI/Wan2.1-Fun-V1.1-14B-InP) | Wan2.1-Fun-V1.1-14B text-to-video generation weights, trained at multiple resolutions, supports start-end image prediction. |
39
+ | Wan2.1-Fun-V1.1-1.3B-Control | 19.0 GB | [🤗Link](https://huggingface.co/alibaba-pai/Wan2.1-Fun-V1.1-1.3B-Control) | [😄Link](https://modelscope.cn/models/PAI/Wan2.1-Fun-V1.1-1.3B-Control) | Wan2.1-Fun-V1.1-1.3B video control weights support various control conditions such as Canny, Depth, Pose, MLSD, etc., supports reference image + control condition-based control, and trajectory control. Supports multi-resolution (512, 768, 1024) video prediction, trained with 81 frames at 16 FPS, supports multilingual prediction. |
40
+ | Wan2.1-Fun-V1.1-14B-Control | 47.0 GB | [🤗Link](https://huggingface.co/alibaba-pai/Wan2.1-Fun-V1.1-14B-Control) | [😄Link](https://modelscope.cn/models/PAI/Wan2.1-Fun-V1.1-14B-Control) | Wan2.1-Fun-V1.1-14B video control weights support various control conditions such as Canny, Depth, Pose, MLSD, etc., supports reference image + control condition-based control, and trajectory control. Supports multi-resolution (512, 768, 1024) video prediction, trained with 81 frames at 16 FPS, supports multilingual prediction. |
41
+ | Wan2.1-Fun-V1.1-1.3B-Control-Camera | 19.0 GB | [🤗Link](https://huggingface.co/alibaba-pai/Wan2.1-Fun-V1.1-1.3B-Control) | [😄Link](https://modelscope.cn/models/PAI/Wan2.1-Fun-V1.1-1.3B-Control) | Wan2.1-Fun-V1.1-1.3B camera lens control weights. Supports multi-resolution (512, 768, 1024) video prediction, trained with 81 frames at 16 FPS, supports multilingual prediction. |
42
+ | Wan2.1-Fun-V1.1-14B-Control-Camera | 47.0 GB | [🤗Link](https://huggingface.co/alibaba-pai/Wan2.1-Fun-V1.1-14B-Control) | [😄Link](https://modelscope.cn/models/PAI/Wan2.1-Fun-V1.1-14B-Control) | Wan2.1-Fun-V1.1-14B camera lens control weights. Supports multi-resolution (512, 768, 1024) video prediction, trained with 81 frames at 16 FPS, supports multilingual prediction. |
43
+
44
  V1.0:
45
  | Name | Storage Space | Hugging Face | Model Scope | Description |
46
  |--|--|--|--|--|
 
49
  | Wan2.1-Fun-1.3B-Control | 19.0 GB | [🤗Link](https://huggingface.co/alibaba-pai/Wan2.1-Fun-1.3B-Control) | [😄Link](https://modelscope.cn/models/PAI/Wan2.1-Fun-1.3B-Control) | Wan2.1-Fun-1.3B video control weights, supporting various control conditions such as Canny, Depth, Pose, MLSD, etc., and trajectory control. Supports multi-resolution (512, 768, 1024) video prediction at 81 frames, trained at 16 frames per second, with multilingual prediction support. |
50
  | Wan2.1-Fun-14B-Control | 47.0 GB | [🤗Link](https://huggingface.co/alibaba-pai/Wan2.1-Fun-14B-Control) | [😄Link](https://modelscope.cn/models/PAI/Wan2.1-Fun-14B-Control) | Wan2.1-Fun-14B video control weights, supporting various control conditions such as Canny, Depth, Pose, MLSD, etc., and trajectory control. Supports multi-resolution (512, 768, 1024) video prediction at 81 frames, trained at 16 frames per second, with multilingual prediction support. |
51
 
52
+
53
  # Video Result
54
 
55
+ ### Wan2.1-Fun-V1.1-14B-InP && Wan2.1-Fun-V1.1-1.3B-InP
56
 
57
  <table border="0" style="width: 100%; text-align: left; margin-top: 20px;">
58
  <tr>
59
  <td>
60
+ <video src="https://pai-aigc-photog.oss-cn-hangzhou.aliyuncs.com/wan_fun/asset/v1.1/inp_1.mp4" width="100%" controls autoplay loop></video>
61
  </td>
62
  <td>
63
+ <video src="https://pai-aigc-photog.oss-cn-hangzhou.aliyuncs.com/wan_fun/asset/v1.1/inp_2.mp4" width="100%" controls autoplay loop></video>
64
  </td>
65
  <td>
66
+ <video src="https://pai-aigc-photog.oss-cn-hangzhou.aliyuncs.com/wan_fun/asset/v1.1/inp_3.mp4" width="100%" controls autoplay loop></video>
67
  </td>
68
  <td>
69
+ <video src="https://pai-aigc-photog.oss-cn-hangzhou.aliyuncs.com/wan_fun/asset/v1.1/inp_4.mp4" width="100%" controls autoplay loop></video>
70
  </td>
71
  </tr>
72
  </table>
 
74
  <table border="0" style="width: 100%; text-align: left; margin-top: 20px;">
75
  <tr>
76
  <td>
77
+ <video src="https://pai-aigc-photog.oss-cn-hangzhou.aliyuncs.com/wan_fun/asset/v1.1/inp_5.mp4" width="100%" controls autoplay loop></video>
78
  </td>
79
  <td>
80
+ <video src="https://pai-aigc-photog.oss-cn-hangzhou.aliyuncs.com/wan_fun/asset/v1.1/inp_6.mp4" width="100%" controls autoplay loop></video>
81
  </td>
82
  <td>
83
+ <video src="https://pai-aigc-photog.oss-cn-hangzhou.aliyuncs.com/wan_fun/asset/v1.1/inp_7.mp4" width="100%" controls autoplay loop></video>
84
  </td>
85
  <td>
86
+ <video src="https://pai-aigc-photog.oss-cn-hangzhou.aliyuncs.com/wan_fun/asset/v1.1/inp_8.mp4" width="100%" controls autoplay loop></video>
87
  </td>
88
  </tr>
89
  </table>
90
 
91
+ ### Wan2.1-Fun-V1.1-14B-Control && Wan2.1-Fun-V1.1-1.3B-Control
92
 
93
+ Generic Control Video + Reference Image:
94
  <table border="0" style="width: 100%; text-align: left; margin-top: 20px;">
95
  <tr>
96
  <td>
97
+ Reference Image
98
+ </td>
99
+ <td>
100
+ Control Video
101
+ </td>
102
+ <td>
103
+ Wan2.1-Fun-V1.1-14B-Control
104
+ </td>
105
+ <td>
106
+ Wan2.1-Fun-V1.1-1.3B-Control
107
+ </td>
108
+ <tr>
109
+ <td>
110
+ <image src="https://pai-aigc-photog.oss-cn-hangzhou.aliyuncs.com/wan_fun/asset/v1.1/6.png" width="100%" controls autoplay loop></image>
111
  </td>
112
  <td>
113
+ <video src="https://pai-aigc-photog.oss-cn-hangzhou.aliyuncs.com/wan_fun/asset/v1.1/pose.mp4" width="100%" controls autoplay loop></video>
114
  </td>
115
  <td>
116
+ <video src="https://pai-aigc-photog.oss-cn-hangzhou.aliyuncs.com/wan_fun/asset/v1.1/14b_ref.mp4" width="100%" controls autoplay loop></video>
117
+ </td>
118
+ <td>
119
+ <video src="https://pai-aigc-photog.oss-cn-hangzhou.aliyuncs.com/wan_fun/asset/v1.1/1_3b_ref.mp4" width="100%" controls autoplay loop></video>
120
  </td>
121
  <tr>
122
  </table>
123
 
124
+ Generic Control Video (Canny, Pose, Depth, etc.) and Trajectory Control:
125
  <table border="0" style="width: 100%; text-align: left; margin-top: 20px;">
126
  <tr>
127
  <td>
128
+ <video src="https://pai-aigc-photog.oss-cn-hangzhou.aliyuncs.com/wan_fun/asset/v1.1/guiji.mp4" width="100%" controls autoplay loop></video>
129
  </td>
130
  <td>
131
+ <video src="https://pai-aigc-photog.oss-cn-hangzhou.aliyuncs.com/wan_fun/asset/v1.1/guiji_plus_out.mp4" width="100%" controls autoplay loop></video>
132
  </td>
133
  <td>
134
+ <video src="https://pai-aigc-photog.oss-cn-hangzhou.aliyuncs.com/wan_fun/asset/v1.1/guiji_out.mp4" width="100%" controls autoplay loop></video>
135
  </td>
136
+ <tr>
137
+ </table>
138
+
139
+ <table border="0" style="width: 100%; text-align: left; margin-top: 20px;">
140
  <tr>
141
  <td>
142
+ <video src="https://pai-aigc-photog.oss-cn-hangzhou.aliyuncs.com/wan_fun/asset/v1.1/pose.mp4" width="100%" controls autoplay loop></video>
143
  </td>
144
  <td>
145
+ <video src="https://pai-aigc-photog.oss-cn-hangzhou.aliyuncs.com/wan_fun/asset/v1.1/canny.mp4" width="100%" controls autoplay loop></video>
146
  </td>
147
  <td>
148
+ <video src="https://pai-aigc-photog.oss-cn-hangzhou.aliyuncs.com/wan_fun/asset/v1.1/depth.mp4" width="100%" controls autoplay loop></video>
149
+ </td>
150
+ <tr>
151
+ <td>
152
+ <video src="https://pai-aigc-photog.oss-cn-hangzhou.aliyuncs.com/wan_fun/asset/v1.1/pose_out.mp4" width="100%" controls autoplay loop></video>
153
+ </td>
154
+ <td>
155
+ <video src="https://pai-aigc-photog.oss-cn-hangzhou.aliyuncs.com/wan_fun/asset/v1.1/canny_out.mp4" width="100%" controls autoplay loop></video>
156
+ </td>
157
+ <td>
158
+ <video src="https://pai-aigc-photog.oss-cn-hangzhou.aliyuncs.com/wan_fun/asset/v1.1/depth_out.mp4" width="100%" controls autoplay loop></video>
159
+ </td>
160
+ </tr>
161
+ </table>
162
+
163
+ ### Wan2.1-Fun-V1.1-14B-Control-Camera && Wan2.1-Fun-V1.1-1.3B-Control-Camera
164
+
165
+ <table border="0" style="width: 100%; text-align: left; margin-top: 20px;">
166
+ <tr>
167
+ <td>
168
+ Pan Up
169
+ </td>
170
+ <td>
171
+ Pan Left
172
+ </td>
173
+ <td>
174
+ Pan Right
175
+ </td>
176
+ <tr>
177
+ <td>
178
+ <video src="https://pai-aigc-photog.oss-cn-hangzhou.aliyuncs.com/wan_fun/asset/v1.1/Pan_Up.mp4" width="100%" controls autoplay loop></video>
179
+ </td>
180
+ <td>
181
+ <video src="https://pai-aigc-photog.oss-cn-hangzhou.aliyuncs.com/wan_fun/asset/v1.1/Pan_Left.mp4" width="100%" controls autoplay loop></video>
182
+ </td>
183
+ <td>
184
+ <video src="https://pai-aigc-photog.oss-cn-hangzhou.aliyuncs.com/wan_fun/asset/v1.1/Pan_Right.mp4" width="100%" controls autoplay loop></video>
185
+ </td>
186
+ <tr>
187
+ <td>
188
+ Pan Down
189
+ </td>
190
+ <td>
191
+ Pan Up + Pan Left
192
+ </td>
193
+ <td>
194
+ Pan Up + Pan Right
195
+ </td>
196
+ <tr>
197
+ <td>
198
+ <video src="https://pai-aigc-photog.oss-cn-hangzhou.aliyuncs.com/wan_fun/asset/v1.1/Pan_Down.mp4" width="100%" controls autoplay loop></video>
199
+ </td>
200
+ <td>
201
+ <video src="https://pai-aigc-photog.oss-cn-hangzhou.aliyuncs.com/wan_fun/asset/v1.1/Pan_Left_Up.mp4" width="100%" controls autoplay loop></video>
202
+ </td>
203
+ <td>
204
+ <video src="https://pai-aigc-photog.oss-cn-hangzhou.aliyuncs.com/wan_fun/asset/v1.1/Pan_Right_Up.mp4" width="100%" controls autoplay loop></video>
205
  </td>
206
  </tr>
207
  </table>
 
232
  docker run -it -p 7860:7860 --network host --gpus all --security-opt seccomp:unconfined --shm-size 200g mybigpai-public-registry.cn-beijing.cr.aliyuncs.com/easycv/torch_cuda:cogvideox_fun
233
 
234
  # clone code
235
+ git clone https://github.com/aigc-apps/VideoX-Fun.git
236
 
237
+ # enter VideoX-Fun's dir
238
+ cd VideoX-Fun
239
 
240
  # download weights
241
  mkdir models/Diffusion_Transformer
 
247
  # https://modelscope.cn/models/PAI/CogVideoX-Fun-V1.1-5b-InP
248
 
249
  # Wan
250
+ # https://huggingface.co/alibaba-pai/Wan2.1-Fun-V1.1-14B-InP
251
+ # https://modelscope.cn/models/PAI/Wan2.1-Fun-V1.1-14B-InP
252
  ```
253
 
254
  ### 2. Local install: Environment Check/Downloading/Installation
 
276
  #### b. Weights
277
  We'd better place the [weights](#model-zoo) along the specified path:
278
 
279
+ **Via ComfyUI**:
280
+ Put the models into the ComfyUI weights folder `ComfyUI/models/Fun_Models/`:
281
+ ```
282
+ 📦 ComfyUI/
283
+ ├── 📂 models/
284
+ │ └── 📂 Fun_Models/
285
+ │ ├── 📂 CogVideoX-Fun-V1.1-2b-InP/
286
+ │ ├── 📂 CogVideoX-Fun-V1.1-5b-InP/
287
+ │ ├── 📂 Wan2.1-Fun-14B-InP
288
+ │ └── 📂 Wan2.1-Fun-1.3B-InP/
289
+ ```
290
+
291
+ **Run its own python file or UI interface**:
292
  ```
293
  📦 models/
294
  ├── 📂 Diffusion_Transformer/
 
347
  - CogVideo: https://github.com/THUDM/CogVideo/
348
  - EasyAnimate: https://github.com/aigc-apps/EasyAnimate
349
  - Wan2.1: https://github.com/Wan-Video/Wan2.1/
350
+ - ComfyUI-KJNodes: https://github.com/kijai/ComfyUI-KJNodes
351
+ - ComfyUI-EasyAnimateWrapper: https://github.com/kijai/ComfyUI-EasyAnimateWrapper
352
+ - ComfyUI-CameraCtrl-Wrapper: https://github.com/chaojie/ComfyUI-CameraCtrl-Wrapper
353
+ - CameraCtrl: https://github.com/hehao13/CameraCtrl
354
 
355
  # License
356
  This project is licensed under the [Apache License (Version 2.0)](https://github.com/modelscope/modelscope/blob/master/LICENSE).