| --- |
| license: apache-2.0 |
| library_name: lerobot |
| pipeline_tag: robotics |
| tags: |
| - robotics |
| - lerobot |
| - act |
| - imitation-learning |
| - so101 |
| model_name: act |
| datasets: r2owb0/so101-DS1 |
| base_model: lerobot/smolvla_base |
| --- |
| |
| # ACT Model for SO101 Robot |
|
|
| This is an Action Chunking Transformer (ACT) model trained for the SO101 robot using LeRobot. The model was trained on demonstration data collected from teleoperation sessions. |
|
|
| ## Model Details |
|
|
| ### Architecture |
| - **Model Type**: Action Chunking Transformer (ACT) |
| - **Vision Backbone**: ResNet18 with ImageNet pretrained weights |
| - **Transformer Configuration**: |
| - Hidden dimension: 512 |
| - Number of heads: 8 |
| - Encoder layers: 4 |
| - Decoder layers: 1 |
| - Feedforward dimension: 3200 |
| - **VAE**: Enabled with 32-dimensional latent space |
| - **Chunk Size**: 50 steps |
| - **Action Steps**: 15 steps per inference |
|
|
| ### Camera Setup |
| The model uses a **dual-camera setup** for robust perception: |
|
|
| 1. **Wrist Camera** (`observation.images.wrist`): |
| - Resolution: 240×320 pixels |
| - Position: Mounted on the robot's wrist |
| - Purpose: Provides close-up, detailed view of manipulation tasks |
| - Field of view: Narrow, focused on the immediate workspace |
|
|
| 2. **Top Camera** (`observation.images.top`): |
| - Resolution: 480×640 pixels |
| - Position: Mounted above the workspace |
| - Purpose: Provides broader context and overview of the environment |
| - Field of view: Wide, captures the entire workspace |
|
|
| ### Input/Output Specifications |
|
|
| **Inputs:** |
| - **Robot State**: 6-dimensional joint positions |
| - `shoulder_pan.pos` |
| - `shoulder_lift.pos` |
| - `elbow_flex.pos` |
| - `wrist_flex.pos` |
| - `wrist_roll.pos` |
| - `gripper.pos` |
| - **Wrist Camera**: RGB image (240×320×3) |
| - **Top Camera**: RGB image (480×640×3) |
|
|
| **Outputs:** |
| - **Actions**: 6-dimensional joint commands (same structure as state) |
|
|
| ## Training Details |
|
|
| ### Dataset |
| - **Source**: `r2owb0/so101-DS1` |
| - **Episodes**: 10 demonstration episodes |
| - **Total Frames**: 5,990 frames |
| - **Frame Rate**: 30 FPS |
| - **Robot Type**: SO101 follower robot |
|
|
| ### Training Configuration |
| - **Training Steps**: 25,000 |
| - **Batch Size**: 4 |
| - **Learning Rate**: 1e-5 |
| - **Optimizer**: AdamW with weight decay 1e-4 |
| - **Validation Split**: 10% of episodes |
| - **Seed**: 1000 |
|
|
| ### Data Augmentation |
| The model was trained with comprehensive image augmentation: |
| - Brightness adjustment (0.8-1.2x) |
| - Contrast adjustment (0.8-1.2x) |
| - Saturation adjustment (0.5-1.5x) |
| - Hue adjustment (±0.05) |
| - Sharpness adjustment (0.5-1.5x) |
|
|
| ## Usage |
|
|
| ### Installation |
| ```bash |
| pip install lerobot |
| ``` |
|
|
| ### Loading the Model |
| ```python |
| from lerobot.policies import ACTPolicy |
| from lerobot.configs.policies import ACTConfig |
| |
| # Load the model |
| policy = ACTPolicy.from_pretrained("r2owb0/act1") |
| ``` |
|
|
| ### Evaluation |
| ```bash |
| lerobot-eval \ |
| --policy.path=r2owb0/act1 \ |
| --env.type=your_env_type \ |
| --eval.n_episodes=10 \ |
| --eval.batch_size=10 |
| ``` |
|
|
| ### Inference |
| ```python |
| import torch |
| |
| # Prepare observation |
| observation = { |
| "observation.state": torch.tensor([...]), # 6D robot state |
| "observation.images.wrist": torch.tensor([...]), # 240x320x3 RGB |
| "observation.images.top": torch.tensor([...]) # 480x640x3 RGB |
| } |
| |
| # Get action |
| with torch.no_grad(): |
| action = policy.select_action(observation) |
| ``` |
|
|
| ## Hardware Requirements |
|
|
| ### Robot Setup |
| - **Robot**: SO101 follower robot |
| - **Cameras**: |
| - Wrist-mounted camera (240×320 resolution) |
| - Top-mounted camera (480×640 resolution) |
| - **Control**: 6-DOF arm with gripper |
|
|
| ### Computing Requirements |
| - **GPU**: CUDA-compatible GPU recommended |
| - **Memory**: At least 4GB GPU memory |
| - **Storage**: ~200MB for model weights |
|
|
| ## Performance Notes |
|
|
| - The model uses action chunking, predicting 50 steps ahead but executing 15 steps at a time |
| - Temporal ensembling is disabled for real-time inference |
| - The model expects normalized inputs (mean/std normalization) |
| - VAE is enabled for better representation learning |
|
|
| ## Limitations |
|
|
| - Trained on a specific robot configuration (SO101) |
| - Requires the exact camera setup described above |
| - Performance may vary with different lighting conditions |
| - Limited to the task domain covered in the training dataset |
|
|
| ## Citation |
|
|
| If you use this model in your research, please cite: |
|
|
| ```bibtex |
| @misc{r2owb0_act1, |
| author = {Robert}, |
| title = {ACT Model for SO101 Robot}, |
| year = {2024}, |
| publisher = {Hugging Face}, |
| url = {https://huggingface.co/r2owb0/act1} |
| } |
| ``` |
|
|
| ## License |
|
|
| This model is licensed under the Apache 2.0 License. |
|
|
