Are both 我家 and 我的 okay in 下班以后我去你家或者你来[我家/我的]，都可以? In the this sentence,


  下班以后我去你家或者你来我家，都可以。


Can I change it to;


  下班以后我去你家或者你来我的，都可以。


Can I do this every time? If the context is clear then can I just say (sth.) 的？ 

Also does the second sentence sound natural (if it's grammatically correct)?
For this particular sentence, it is much more natural to use the first expression. The second one is grammatically correct but I have not seen this usage among native Chinese speaker/writer. Though it is to common say something like 你吃你的我吃我的 [Eat your (food) and I eat mine] in other context.