Despite the fact that generative models are extremely successful in practice, the theory underlying this phenomenon is only starting to catch up with practice. In this work we address the question of the universality of generative models: is it true that neural networks can approximate any data manifold arbitrarily well? We provide a positive answer to this question and show that under mild assumptions on the activation function one can always find a feedforward neural network that maps the latent space onto a set located within the specified Hausdorff distance from the desired data manifold. We also prove similar theorems for the case of multiclass generative models and cycle generative models, trained to map samples from one manifold to another and vice versa.
We shot that a wide class of manifolds can be generated by ReLU and sigmoid networks with arbitrary precision. This paper provides certain basic guarantees on when manifolds can be written as the image of a map approximated by a neural net, and stitches together theorems from manifold geometry and standard universal approximation results. This paper theoretically shows that neural-network-based generative models can approximate data manifolds, and proves that under mild assumptions neural networks can map a latent space onto a set close to the given data manifold within a small Hausdorff distance.