Are there any theoretical works about possiblity of self-replication of LEGO systems? Are there any theoretical works about possiblity of self-replication of LEGO systems?
(constraints on the size etc., computer simulations)

Here are some relevent links:

Bartlab.org: PDF Document

Techi.com: Mad Scientist Dooms Us All With Self-Replicating Lego Menace

Wikipedia: Clanking replicator
It really depends on the level of self-replication you desire, and your willingness to introduce additional non-LEGO components to the process.

First off, assuming a pure solution, there is no way for LEGO blocks to produce more LEGO blocks.  You can't make a mold for making an ABS plastic part out of parts made from ABS plastic.

Even if you could produce the parts, you'd still run into some problems.  If you were just assembling brick by brick, you could easily replicate such creations.  But as soon as you need to assemble components using technic axles or pins, there'll be some difficulties.  LEGO just doesn't have pieces that are small enough or precise enough to maneuver axles and pins with any precision.

Obviously, this is simply speculation, but I fail to see a way in which LEGO elements could both manufacture and assemble all the elements to create more self-replicating systems.