Why couldn't a person in the OT be both priest and king? In 1 Samuel 13, Saul is king and yet performs priestly duties even though he shouldn't have.  Samuel rebukes Saul and tells him that his kingdom is slowly coming to an end because of his disobedience.

My question is: where in the Bible is it established that a person could not hold both the positions of king and priest?  

Some other information that may or may not be interesting: 

1) Melchizedek appears to Abram in Genesis 14 as BOTH priest and king.
2) Of course, Jesus fulfills both roles as the true Priest and true King.
3) The role of king of Israel wasn't established until the time of Saul and David, so during the writing of the Mosaic Law, kingship should not have been a thing.  Were kings mentioned in the law? (if so, why?)
The strongest restriction is that the priests of the Israelite nation could only be from the tribe of Levi, and specifically on descendants of Aaron. Some verses showing this include Numbers 3:10, 3:38 and 16:40:


  ... This was to remind the Israelites that no one except a descendant of Aaron should come to burn incense before the Lord ...


The kingship was not so strongly restricted, but Genesis 49:10 does clearly prophesy that it belongs to Judah:


  The scepter will not depart from Judah, nor the ruler’s staff from between his feet ...


So looking at the various people you asked about:

Melchizedek: was not an Israelite, so neither of these restrictions apply to him. He was a faithful foreign king.

Jesus: was an Israelite and is King because he is descended from David. He isn't a descendant of Aaron and his priesthood isn't from them. Instead he is a priest because he became a man and died on the cross for humanity.

Saul: was not a Judahite or a Levite. He definitely had no right to act like a priest, but the Genesis 49 prophesy was not so strict, so God wasn't being contradictory when he appointed Saul as king.

Kings were mentioned in the Law, most explicitly in Deuteronomy 17:14-20. That passage describes how the kings were to act, and it explains even the best kings like David and Solomon got into trouble. It doesn't say however which tribes the kings could come from.