Matej Sirovatka

Hi, currently it's only available if you use fsdp2 too, as the composability without fsdp2 is limited (will change in next release). You can use accelerate config in the command line to specify it.

May I ask why is dp_world_size realted to tp_size?

Based on my understanding, it should be just dp_world_size = dp_shard_size * dp_replicate_size. Or am I missing some points here?

Correct, I think we had a previous version where it was dp_world_size = world_size / tp_size, which we then changed to this version and forgot to remove the denominator. Will push a fix 🤗