Interpreting generative adversarial network (GAN) training as approximate divergence minimization has been
 theoretically insightful, has spurred discussion, and has lead to theoretically and practically interesting
 extensions such as f-GANs and Wasserstein GANs. For both classic GANs and f-GANs, there is an original variant of training and a "non-saturating" variant which uses an alternative form of generator gradient. The original variant is theoretically easier to study, but for GANs the alternative variant performs better in practice. The non-saturating scheme is often regarded as a simple modification to deal with optimization issues, but we show that in fact the non-saturating scheme for GANs is effectively optimizing a reverse KL-like f-divergence. We also develop a number of theoretical tools to help compare and classify f-divergences. We hope these results may help to clarify some of the theoretical discussion surrounding the divergence minimization view of GAN training.
Non-saturating GAN training effectively minimizes a reverse KL-like f-divergence. This paper proposes a useful expression of the class of f-divergences, investigates theoretical properties of popular f-divergences from newly developed tools, and investigates GANs with the non-saturating training scheme.