Update hftrainer metadata
Browse files
README.md
CHANGED
|
@@ -1,104 +1,3 @@
|
|
| 1 |
-
|
| 2 |
-
license: mit
|
| 3 |
-
tags:
|
| 4 |
-
- text-to-motion
|
| 5 |
-
- human-motion-generation
|
| 6 |
-
- motion-generation
|
| 7 |
-
- momask
|
| 8 |
-
- humanml3d
|
| 9 |
-
library_name: hftrainer
|
| 10 |
-
pipeline_tag: text-to-3d
|
| 11 |
-
---
|
| 12 |
-
|
| 13 |
-
# MoMask — HumanML3D (hftrainer artifact)
|
| 14 |
-
|
| 15 |
-
Self-contained hftrainer re-export of the official **MoMask** text-to-motion
|
| 16 |
-
model (Guo et al., CVPR 2024) on **HumanML3D-263**. The RVQ-VAE tokenizer,
|
| 17 |
-
masked generative transformer, residual transformer and length estimator are
|
| 18 |
-
vendored and **numerically parity-equal to the released checkpoints**
|
| 19 |
-
(`convert_momask_checkpoint.py --verify` reports max-abs-diff `0.0` on
|
| 20 |
-
generation). The CLIP ViT-B/32 text encoder is **not** stored here; it is
|
| 21 |
-
reloaded by name at load time.
|
| 22 |
-
|
| 23 |
-
- **Task**: Text-to-Motion (T2M)
|
| 24 |
-
- **Representation**: HumanML3D-263 (263-dim, 20 fps, 22-joint SMPL skeleton)
|
| 25 |
-
- **Tokenizer**: RVQ-VAE, 6 residual quantizers, codebook 512×512
|
| 26 |
-
- **Generator**: MaskTransformer (masked iterative decoding) + ResidualTransformer
|
| 27 |
-
- **Text encoder**: CLIP ViT-B/32 (frozen, reloaded by name)
|
| 28 |
-
- **Paper**: [MoMask: Generative Masked Modeling of 3D Human Motions](https://arxiv.org/abs/2312.00063)
|
| 29 |
-
- **Original code**: https://github.com/EricGuo5513/momask-codes
|
| 30 |
-
|
| 31 |
-
## Files
|
| 32 |
-
|
| 33 |
-
| File | Contents |
|
| 34 |
-
|---|---|
|
| 35 |
-
| `momask_config.json` | architecture config for all sub-modules |
|
| 36 |
-
| `vq.safetensors` | RVQ-VAE weights |
|
| 37 |
-
| `t2m_trans.safetensors` | MaskTransformer (no CLIP) |
|
| 38 |
-
| `res_trans.safetensors` | ResidualTransformer (no CLIP) |
|
| 39 |
-
| `length_est.safetensors` | LengthEstimator |
|
| 40 |
-
| `Mean.npy` / `Std.npy` | 263-dim de-normalization stats |
|
| 41 |
-
|
| 42 |
-
## Usage
|
| 43 |
-
|
| 44 |
-
```python
|
| 45 |
-
from huggingface_hub import snapshot_download
|
| 46 |
-
from hftrainer.models.motion.momask import MoMaskBundle
|
| 47 |
-
from hftrainer.pipelines.momask import MoMaskPipeline
|
| 48 |
-
|
| 49 |
-
path = snapshot_download("ZeyuLing/momask-humanml3d")
|
| 50 |
-
bundle = MoMaskBundle.from_pretrained(path)
|
| 51 |
-
pipe = MoMaskPipeline(bundle, device="cuda")
|
| 52 |
-
|
| 53 |
-
# length picked by the length estimator:
|
| 54 |
-
motions = pipe.infer_t2m(["a person walks forward then sits down"]) # list of (T, 263)
|
| 55 |
-
# or a fixed length (frames @ 20 fps):
|
| 56 |
-
motions = pipe.infer_t2m(["a person walks forward then sits down"], [120])
|
| 57 |
-
```
|
| 58 |
-
|
| 59 |
-
## Evaluation (HumanML3D-263, Guo evaluator)
|
| 60 |
-
|
| 61 |
-
20 repeats, n = 3970. Generation under the official HumanML3D protocol
|
| 62 |
-
(standard test split, native 263-dim @ 20 fps).
|
| 63 |
-
|
| 64 |
-
| Metric | this artifact | MoMask paper |
|
| 65 |
-
|---|---|---|
|
| 66 |
-
| FID ↓ | **0.097** | 0.045 |
|
| 67 |
-
| R-Precision Top-1 / 2 / 3 ↑ | **0.516 / 0.709 / 0.804** | 0.521 / 0.713 / 0.807 |
|
| 68 |
-
| MM-Dist ↓ | **2.990** | 2.958 |
|
| 69 |
-
| Diversity → | **9.460** | 9.620 |
|
| 70 |
-
|
| 71 |
-
R-Precision, MM-Dist and Diversity match the paper essentially exactly. The
|
| 72 |
-
small residual FID gap is a data-processing / evaluation-population difference,
|
| 73 |
-
not a generation-quality gap (the decode path is verified parity-equal to the
|
| 74 |
-
released MoMask inference).
|
| 75 |
-
|
| 76 |
-
## Citation
|
| 77 |
-
|
| 78 |
-
```bibtex
|
| 79 |
-
@inproceedings{guo2024momask,
|
| 80 |
-
title = {MoMask: Generative Masked Modeling of 3D Human Motions},
|
| 81 |
-
author = {Guo, Chuan and Mu, Yuxuan and Javed, Muhammad Gohar and Wang, Sen and Cheng, Li},
|
| 82 |
-
booktitle = {CVPR},
|
| 83 |
-
year = {2024}
|
| 84 |
-
}
|
| 85 |
-
```
|
| 86 |
-
|
| 87 |
-
## Evaluation (MotionStreamer-272, SMPL retarget path)
|
| 88 |
-
|
| 89 |
-
For cross-model comparison with the MotionStreamer / HYMotion-M2M evaluator,
|
| 90 |
-
native HumanML3D-263 predictions are retargeted through the validated MDM-style
|
| 91 |
-
chain: HML263 -> SMPL `motion_135` (IK refine-80, 20 -> 30 fps) ->
|
| 92 |
-
MotionStreamer-272 -> `MotionStreamer272Evaluator`.
|
| 93 |
-
|
| 94 |
-
| Metric | MoMask (this artifact) | MS-272 GT/Real |
|
| 95 |
-
|---|---:|---:|
|
| 96 |
-
| FID ↓ | **114.869** | 0.000 |
|
| 97 |
-
| R-Precision Top-1 ↑ | **0.485** | 0.706 |
|
| 98 |
-
| R-Precision Top-2 ↑ | **0.650** | 0.857 |
|
| 99 |
-
| R-Precision Top-3 ↑ | **0.731** | 0.911 |
|
| 100 |
-
| MM-Dist ↓ | **19.411** | 15.007 |
|
| 101 |
-
| Diversity → | **25.427** | 27.281 |
|
| 102 |
-
|
| 103 |
-
Run details: `n_repeats = 20`, `n_samples_used = 7392`, `skipped_no_pred = 0`.
|
| 104 |
|
|
|
|
|
|
| 1 |
+
# MoMask hftrainer Artifact
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 2 |
|
| 3 |
+
Load with `MoMaskBundle.from_pretrained(...)`. CLIP ViT-B/32 is an external runtime dependency and is not stored here.
|