This work studies the problem of modeling non-linear visual processes by leveraging deep generative architectures for learning linear, Gaussian models of observed sequences. We propose a joint learning framework, combining a multivariate autoregressive model and deep convolutional generative networks. After justification of theoretical assumptions of inearization, we propose an architecture that allows Variational Autoencoders and Generative Adversarial Networks to simultaneously learn the non-linear observation as well as the linear state-transition model from a sequence of observed frames. Finally, we demonstrate our approach on conceptual toy examples and dynamic textures.
We model non-linear visual processes as autoregressive noise via generative deep learning. Proposes a new method that models non-linear visual process with a deep version of a linear process (Markov process). This paper proposes a new deep generative model for sequences, particularly image sequences and video, which uses a linear structure in part of the model.