At what point does this mental trick stop working? Sometimes, I'll be in a position where my opponent is threathening to do something and I cannot stop it. So, in response, I play a move that looks pretty dangerous and clever, but which I in fact have not managed to calculate all the way through and I have no idea whether it will work or not. 

And I often see that in such situations, the opponent becomes overly paranoid and thinks I've just played some sort of mastermind move, and then they start retreating or playing too defensively or whatever, and I manage to weazel my way out of the situation. 

When I then go back and analyze my play with the engine, it will often tell me that my clever move was a complete mistake and my opponent wins by just doing the obvious thing they were about to do.

At what point do these mental tricks, where you make dangerous-looking movies to make your opponent overly paranoid, stop working? I'm thinking there must be some point where players simply become too good to be fooled by that sort of trickery, and just calmly make decisions based on pure calculation.

I currently play in the 1600-1700 range on lichess.
This trick is prevalent in all levels. Of course, depending on the position, your ability to muddy the waters varies. Against much higher rated opponents, it is typical that they will  gain an advantage without giving you much counterplay. Nevertheless, in all levels of the game, the losing side will try to muster some counterplay to get back into the game. 
You make a comment about moves that look dangerous but are not. The higher rated your opponents are, the more likely they will realize that your plan is not dangerous. Often, it will be better for the losing side to defend and wait for the right time to strike than to go all out on a dubious plan. Furthermore, many positions, even in the top levels, can be wildy complicated to calculate, and the objective evaluation is not always a good predictor of outcome even at the top levels. Thus, not even grandmasters will always be able to evaluate the position accurately from "pure" calculation. 
So to answer your question more precisely and perhaps unsatisfactorily, if you make a dangerous looking move, if it is "obvious" to your opponent that it's unsound, you'll be punished. Where "obvious" depends on the strength of your opponent and the subtleness of your clever move.