How do I say "least" in the superlative sense in Mandarin? I've seen "the least expensive" translated as 最便宜的. Does this inversion always have to happen, or is there a way to say "least"?
As suggested by some comments, it depends on the context, and what you would like to stress with this phrase.

Without any context, it sounds exactly the same with "the cheapest". In this way, 不贵 is 便宜, 最便宜的 is the translation.  

However, if you are trying to say all these things are expensive, but this one is the least expensive one among them, then 最不贵的 is better. However, when you use this phrase, your audience should either have the knowledge that all these things are expensive, or you should explicitly stress mention that. Otherwise, it doesn't sound natural.