Where did the concept of separate races originate? I want to preface what I say with a statement saying this is not in any way related to any kind of Prejudice! I only want to ask how we got from the family of Noah to having so many so called races.

According to my Bible; which is the King James version, this is how the world was populated.

Genesis 11:1 and 2 


  And the whole earth was of one language, and of one speech. And it came to pass, as they journeyed from the east, that they found a  plain in the land of Shinar; and they dwelt there. 


Everyone seems to have been one happy family, but then pride crept in and:


  Genesis 11:4  And they said, Go to, let us build us a city and a tower, whose top may reach unto heaven; and let us make us a name, lest we be scattered abroad upon the face of the whole earth. 


God didn't like that:

Genesis 11:6 through 9  


  And the LORD said, Behold, the people is one, and they have all one language; and this they begin to do: and now nothing will be restrained from them, which they have imagined to do.  Go to, let us go down, and there confound their language, that they may not understand one another's speech. So the LORD scattered them abroad from thence upon the face of all the earth: and they left off to build the city. Therefore is the name of it called Babel; because the LORD did there confound the language of all the earth: and from thence did the LORD scatter them abroad upon the face of all the earth. 


Now as far as I have been able to determine God only mixed up their languages so they could no longer understand each other, but I find nothing that says he turned them into different races to be able to visually tell the difference between them.

So where did people become different races?
Historically, the answer to the race issue has been the Table of Nations in Genesis 10.
From Nigel Tomes:

The tradition that Ham was a black man developed much later. It is a Rabbinical elaboration,7 not explicitly formulated until the Babylonian Talmud of 500 AD. Hence this concept belongs in the category of Jewish “myths and unending genealogies” (1 Tim.1:4). In the Middle Ages, European scholars of the Bible picked up on the Jewish Talmud idea that the “sons of Ham” were “blackened” by their sins.8 These arguments became increasingly common during the slave trade of the 18th and 19th Centuries. A historian, Edith Sanders, concludes that the identification of Ham’s descendents as Black Africans,9 “gained currency in the sixteenth century.” Thereafter, it “persisted throughout the eighteenth century, [and] served as a rationale for slavery, using Biblical interpretations in support of its tenets. The image of the Negro deteriorated in direct proportion to the growth of the importance of slavery.” Benjamin Braude, Professor of history at Boston College, writes10 “in 18th and 19th century Euro-America, Genesis 9:18-27 became the curse of Ham, a foundation myth for collective degradation, conventionally trotted out as God’s reason for condemning generations of dark-skinned peoples from Africa to slavery.” Sadly this notion has been perpetuated through its uncritical repetition by Bible teachers and writers. However, today evangelical scholars reject this view as an out-dated remnant of folklore, masquerading as Scriptural truth. - Nigel Tomes

The text itself, as you point out only says:

Now these are the generations of the sons of Noah, Shem, Ham, and Japheth: and unto them were sons born after the flood. ... By these were the isles of the Gentiles divided in their lands; every one after his tongue, after their families, in their nations.

Of Noah's descendents:

Japheth moved east and became the father of the Asian peoples

The sons of Japheth; Gomer, and Magog, and Madai, and Javan, and Tubal, and Meshech, and Tiras. ...


Shem moved west and became the father of the White European peoples

Unto Shem also, the father of all the children of Eber, the brother of Japheth the elder, even to him were children born. The children of Shem; Elam, and Asshur, and Arphaxad, and Lud, and Aram. ...


Ham moved South and became the father of the African peoples.

And the sons of Ham; Cush, and Mizraim, and Phut, and Canaan. ... And the sons of Cush; Seba, and Havilah, and Sabtah, and Raamah, and Sabtechah: and the sons of Raamah; Sheba, and Dedan.



Whether or not this theory has any merit, it was historically understood to be the case.
The curses in Genesis 9 were thus seen to have been fulfilled:

20 And Noah began to be an husbandman, and he planted a vineyard: 21 And he drank of the wine, and was drunken; and he was uncovered within his tent. 22 And Ham, the father of Canaan, saw the nakedness of his father, and told his two brethren without. 23 And Shem and Japheth took a garment, and laid it upon both their shoulders, and went backward, and covered the nakedness of their father; and their faces were backward, and they saw not their father's nakedness. 24 And Noah awoke from his wine, and knew what his younger son had done unto him.
25 And he said, Cursed be Canaan; a servant of servants shall he be unto his brethren. 26 And he said, Blessed be the Lord God of Shem; and Canaan shall be his servant. 27 God shall enlarge Japheth, and he shall dwell in the tents of Shem; and Canaan shall be his servant.