Usage of 去 and 來 : is "context switching" allowed? I am often confused with the use of 來 and 去.
I made a picture synthesizing my current understanding of 來 and 去:



去 is used in the context when someone is going far away from A.  
來 is used in the context when someone is approaching a place (B in the picture).


But how to use this when A is talking to B about this? He could say « 我明天去B找妳 ». 

But is context switching allowed? Is A allowed to pretend the reference is B and therefore say « 我明天來 »?

Whenever I've tried this, the other one (native Chinese speaker) told me that it was not correct and that I should rather use 去.

So, in my two sentences above (in gray), who is this "someone"? Is it always the one who is talking? Or is « context switching » allowed?
If you are at my place and you want to come to my place tomorrow, you should say 我明天来。Because you are right there where you are talking about. If we are at your place and you want to come to my place, you say 我明天去。Because it's where you are not currently at. Use 来 and 去 based on where you are and where you refer to.