Get SQRT_N and SGD graft type to work under model parallel training. We pay the run time memory cost for now  (ideally we can conditionally disable them).
Get SQRT_N and SGD graft type to work under model parallel training. We pay the run time memory cost for now  (ideally we can conditionally disable them).

PiperOrigin-RevId: 452869761

google-research/google-research,google-research/google-research,google-research/google-research,google-research/google-research,google-research/google-research,google-research/google-research,google-research/google-research,google-research/google-research,google-research/google-research,google-research/google-research,google-research/google-research