Add --single_l2_loss_op option.
Add --single_l2_loss_op option.

Benchmark results with `tf-nightly-gpu` at tensorflow/tensorflow@4c28fbf2c0d20ed714fd0209913b868a7955ac5a, on a Pascal DGX1. The average of 3 iterations was taken for each data point. The command used was:

python tf_cnn_benchmarks.py --num_batches=100 --display_every=10 --batch_size=128 --variable_update=$VAR_UPDATE --model=$MODEL --data_name=$DATA_NAME --num_gpus=$GPU --loss_type_to_report=$LOSS_TYPE

Additionally, --single_l2_loss_op=True was added for the "after" case. Note that resnet50 uses imagenet and resnet110 uses cifar10.

model      var_update  gpu  loss type   im/s after im/s before  im/s %diff
resnet50   ps          1    total_loss  243.913    242.687      0.50545%
resnet50   ps          1    base_loss   243.110    243.910      -0.32799%
resnet50   ps          8    total_loss  1400.063   1376.027     1.74682%
resnet50   ps          8    base_loss   1451.870   1482.440     -2.06214%
resnet50   replicated  8    total_loss  1610.010   1599.340     0.66715%
resnet50   replicated  8    base_loss   1610.007   1612.100     -0.12985%
resnet110  ps          1    total_loss  1362.163   1314.853     3.59812%
resnet110  ps          1    base_loss   1351.477   1345.477     0.44594%

In most cases when the option is enabled, performance is better when reporting total loss and worse when reporting base loss.

PiperOrigin-RevId: 183411228

tensorflow/benchmarks,annarev/benchmarks,tensorflow/benchmarks,annarev/benchmarks