What is the difference between 台 and 臺？ What is the difference between 台 and 臺？

台 is both "simplified" and "alternative", 臺 is "traditional".

臺 seems to be the "correct" traditional form for Taiwan, but I've never seen it in reference to Taishan (台山) in Guangdong, even in all references pre-1949, etc.

How is it determined in "traditional" usage when to use the one or the other?

(In simplified usage 台 is just used all the time, correct?)
In simplified Chinese, both would be 台, easy peasy. Otherwise, things get a little complicated. Sometimes 台 is just an alternative form for 臺, which is the case for Taiwan: you can write 臺灣 or 台灣, both are acceptable, though the former is considered more formal. In the case of 台山, that is the correct name already, so you can't write 臺山 because 臺 is not an alternative form of 台. Also, for place names like 台州 and 天台山, the pronunciation of 台 is a little different from normal, all the more reason that you can't always choose to substitute 台 with 臺.

Don't be fooled by the Traditional and Simplified Chinese bisection. Only very little Simplified Chinese characters were invented to replace the old ones. Most Simplified Chinese characters were either the archaic but "correct" forms or were simpler variants of the "correct" characters. In fact, many "traditional Chinese" characters are actually much simpler (having fewer strokes) than some of the variants, so 繁體字 is a misnomer, and as for 正體字, what is the "correct" way to write a certain character? Is there a "correct" way to write those characters? Defined by whom? It is not just black and white.

Chinese is hard, and I think the more I learn about Chinese, the harder it gets. I think the best way to learn when to use 臺 and not 台 and vice versa is to just remember each instance and go by the ones most people use even if that usage is not correct semantically. Occasionally, you may want to consult a reference such as the Dictionary of Variants published by the MOE of Taiwan, in which you'll find that 台 and 臺 are originally two different words with similar pronunciation, but 台 has been borrowed as a substitute for 臺 and has therefore acquired the pronunciation and meaning of 臺. One definition of 臺 is "stage", and of 台 is (originally) "delight", interchangeable with 怡. Keep in mind though, that it is just ONE reference, the fact that it's "official" doesn't mean it's always correct or reflects what most people use, even in Taiwan.