When is a non-unitary quantum system only theoretical? Suppose we construct a non-unitary quantum system &alpha; in hilbert space. It entails that this system would have no direct parallel in quantum circuitry as it is a requirement that all quantum gates are unitary and therefore the function constructed from them is also unitary. But does this mean that we cannot practically implement &alpha;?

A potential parallel is found in the simulation of classical computations on a quantum computer, achieved by simulating an information-destroying NAND gate using an information-preserving Toffoli gate. If a non-unitary quantum system was such because it implemented a 'quantum NAND gate', it would be possible to redesign it as a unitary system using a Toffoli gate. It is not clear to me (and I cannot immediately find literature on this) if there are clearly defined bounds on what non-unitary operations can be simulated using unitary equivalents. 

Is there a clearly defined subset of non-unitary quantum systems where &alpha; is practically implementable? Is it, perhaps, universal?
Look at John Preskill's Lecture Notes; particularly Section 3.2.

As you noted, you can do a NAND gate by using a Toffoli gate and throwing away some of the output qubits. This results in decoherence, so you no longer necessarily have a pure state of your system.

For non-unitary quantum operations, if you assume that the environment doesn't remember anything, then the most general thing you can do is a unitary operation followed by throwing away some part of the system. 

If you assume that the environment has memory, things get much more complicated. But note that here, your operations aren't consistent because the outcome of an operation will depend on what the input was the last time you performed the operation. I don't know if you want to include this class of operations. While they are important in physics, they aren't very useful for quantum computation.