Is Marlin capable to work with binary G-code? I want to use a binary protocol like Repetier uses.
Can Marlin work with this/such a protocol?
No. Marlin only supports ASCII G-code, and does not (currently) support binary format. According to the List of Firmware, Repetier is the only firmware that supports binary G-code. Moreover, verifying with Marlin's source code, shows that the G-code parser can only handle traditional G-code.