Autoencoders and information compression Disclaimer: I know very (very) little about deep nets, besides what an introductory course on machine learning would teach on neural networks, and skimming some paper abstracts and introductions.

If I understood correctly the concept, an autoencoder is a specific case of neural networks with as many input nodes than output nodes (say $d$), whose goal is to approximate the identity function
$$
\operatorname{id}\colon \mathbb{R}^d \to \mathbb{R}^d\,.
$$
The trick is that this net can be broken into two parts, "encoder" and "decoder" of the form $\Phi\colon \mathbb{R}^d \to \mathbb{R}^k$ and $\Psi\colon \mathbb{R}^k \to \mathbb{R}^d$, such that $k\ll d$. Therefore, there has to be something non-trivial going on for $\Psi\circ \Phi$ to approximate the identity, given the restricted width of the middle layers.

This struck me as begging the question:


  Can one (and has one already?) formalize and analyze this rigorously in terms of information theory, and specifically compression?


I realize the problem itself would need a more thorough and precise formulation to be tackled, as in the above we deal with full real numbers (so "hiding" information in a single real number, say the weight of a node in the middle layer, would give a very easy way out). But with this dealt with appropriately, hopefully there are non-trivial statements to be made with regard to what compression can be achieved, with respect to some distribution over the inputs?

Has this type of question been looked at from a theoretical viewpoint, in our community or another?
In the other situation when in your notation $k&gt;&gt;d$, in one of our papers we did analyze one such autoencoder in details to be able to say something about what you are asking. Does our Theorem 3.2 here, https://arxiv.org/pdf/1708.03735.pdf help? It took us about 15 pages of laborious calculation to get this - hopefully its helpful! :D  

The "??" in the statement is a LaTeX error. It was meant to refer to the previous Theorem 3.1.