Enable runtime cloning of tasks.
Enable runtime cloning of tasks.

Summary:
Funnily, the biggest issue when trying to increase number of trainers from 5 to 20 is not model convergence (it is worse but still converges without tuning); it is the initialization time: it took around 30 min to generate the job.

After this diff, job creation time for the standard 5-7 setup goes from 125s to 8s. (15x speedup).

Another improvement is that ##net_printer.to_string(job)## becomes less complex.

This makes the startup for 20 trainers go to 32s, which is still not ideal.

Next step will be to allow passing num_instances to Node as well. This way we'll be able to create only one reader and one trainer prototype and let the framework take care of the scheduling. For this one we will need to move some DataStream and PS initialization code to C++ first. (c.c. aartibasant)

Reviewed By: dzhulgakov

Differential Revision: D5100788

fbshipit-source-id: 7b76bce108f527a96b2bfe7ed43a22ea8679b682

pietern/caffe2,bwasti/caffe2,bwasti/caffe2,pietern/caffe2,sf-wind/caffe2,xzturn/caffe2,sf-wind/caffe2,sf-wind/caffe2,sf-wind/caffe2,davinwang/caffe2,davinwang/caffe2,xzturn/caffe2,xzturn/caffe2,pietern/caffe2,xzturn/caffe2,Yangqing/caffe2,caffe2/caffe2,Yangqing/caffe2,davinwang/caffe2,pietern/caffe2,pietern/caffe2,bwasti/caffe2,Yangqing/caffe2,davinwang/caffe2,bwasti/caffe2,davinwang/caffe2,Yangqing/caffe2,Yangqing/caffe2,sf-wind/caffe2,xzturn/caffe2,bwasti/caffe2