When and where does the statement, "Christ paid the penalty for our sins" first appear? The statement, "Christ paid the penalty for our sins" does not appear in the Bible.

When in the history of Christian theology did this specific statement first appear? Who said it?

Please provide the actual text (and source) from the writings of the Christian theologians or teachers who first said it—or at least, the earliest ones you can document.

If that is clear to you, there is no need to read the rest of this question.

Please note:


This question is specifically about the statement that Christ paid
the penalty for our sins. Equivalent wordings, such as "Christ paid
the price for our sins" or "Jesus paid the penalty for our sins,"
are on-topic. However, "Christ died for our sins" or "Christ suffered
for our sins" or even "Christ was punished for our sins" are
off-topic. I am looking for statements specifically about Christ paying the
penalty, or paying the price, for our sins.
"Paying the price" in the context of Ransom Theory is also
off-topic. A ransom is not a penalty or punishment for sin.
I am not looking for antecedents for this idea, nor am I looking
for passages quoted as the biblical basis for this idea. I am looking
for the earliest explicit statements of the idea itself.
For reference: the Wikipedia article on Penal Substitution.
Please do not just quote from or refer to the Wikipedia article,
which seems rather murky and disorganized.


Edit in response to comments:

My hypothesis is that the Penal Substitution theory of atonement is closely tied to the phrase "paid the penalty." This is a history of doctrine question rather than simply an English phrase question.

However, it is common for proponents of Penal Substitution to see this theory of atonement in phrases representing ideas that are not necessarily the same. For example:


"Christ died for our sins." If a drunk driver hits you and kills you, you have died for (due to) the sins of the drunk driver, but you have not paid the penalty for the sins of the drunk driver. S/he remains guilty of the crime, and subject to punishment.
"Christ suffered for our sins." If a drunk driver hits you and injures you, you have suffered for the sins of the drunk driver, but you have not paid the penalty for the sins of the drunk driver. S/he remains guilty of the crime, and subject to punishment.
"Christ was punished for our sins." If a drunk driver hits you, and you are wrongfully accused and put in jail instead of the drunk driver, you have still not paid the penalty for the drunk driver's sins. The drunk driver remains guilty of the crime, and subject to its penalty if and when it is discovered that there was a miscarriage of justice. Or if you were to voluntarily go to jail with the drunk driver, taking the same punishment even though you didn't commit the crime, you would still not have paid the penalty for the drunk driver's sins.  S/he would still remain guilty of the crime, and subject to punishment.


The point is, every one of these statements can easily and very reasonably be read as meaning something other than Christ paying the penalty for our sins. (And I happen to think that they do mean something other than Christ paying the penalty for our sins.)

That is why I am insisting on the precise language that most specifically expresses the Penal Substitution theory of atonement: that Christ paid the penalty for our sins.

Protestant tracts are full of the statement, "Christ paid the penalty for our sins." That phrase is not in the Bible. It must have come from somewhere. I want to know where it came from.

I suspect this will also provide the origin point of the Penal Substitution theory of atonement in the history of Christian doctrine.

If none of that works for you, just repeat over and over again before writing an answer:

Where did the precise phrase "Christ paid the penalty for our sins" come from?
The theological term for the concept you are thinking of, namely that Christ was payment for our sins and so took upon Himself the fullness of punishment due to sin, is known as penal substitution (as you identify). Penal substitution is largely originated in the Reform Movement. It is generally agreed that no Church father taught penal substitution, but rather favored the ransom theory of atonement intermixed with the moral influence view. Several Reform movement thinkers and those applying to the penal substitution theory have attempted to attribute the source of their thinking to certain Church fathers, though they have been unsuccessful in doing so. It could be said that the roots for penal substitution were birthed by St. Anselm (though the theory itself can hardly be attributed to him.) St. Anselm sought to modify the traditional ransom theory of atonement. St. Anselm ultimately contributed what is known as Anselmian satisfaction, which is the theory that the satisfaction that God the Father saw in Christ as an honorable and perfect man compensated for the due punishment that man deserved. 


  The honor taken away must be repaid, or punishment must follow.


This sense of atonement illuminated upon the ransom theory of atonement; Anselm held that Christ's sacrifice was not appeasing as a ransom to Satan (as traditional ransom theory held) but rather to God. And upon this theory St. Thomas Aquinas only further modified the kinks. In progression from this, according to St. Thomas Aquinas, Christ not only fulfilled a debt of honor but also a debt of moral justice. Mankind not only fell in their honor, but also offended God by disobeying Him. So it is that Aquinas concludes the following;


  Christ bore a satisfactory punishment, not for His sins, but for ours.  


Aquinas does not intend this to mean that Christ bears upon Himself a penal punishment for sin (punishment that serves a lawful purpose accorded to certain individuals, an example being eternal damnation), but rather that He bears upon Himself a satisfactory punishment (punishment that serves a restorative purpose due to its penitent nature, and that's merits can be shared). 


  If we speak of that satisfactory punishment, which one takes upon oneself voluntarily, one may bear another's punishment... If however, we speak of punishment inflicted on account of sin, inasmuch as it is penal, then each one is punished for his own sin only, because the sinful act is something personal. But if we speak of a punishment that is medicinal, in this way it does happen that one is punished for another's sin.


It is generally understood that the punishment Christ went through, the satisfactory punishment, was death and suffering, the immediate and unavoidable consequences of original sin. For Aquinas recognized that Christ could not merely become perfect and eternal man in order to justify man, but rather that the completion of salvation had to include, because of fallen man's evils, some sort of necessary punishment of a penitent nature for original sin itself (similar to the penitent punishment that believers go through in order to be purified of personal sin). Christ's justifying man through the indwelling perfection of His being had to include submitting in perfect penance for the original sin of man (and to do so Christ entered the satisfactory punishment of original sin, being death). Notice that this punishment is not an appeasement of an angry God, but rather a part of Christ perfecting man. Aquinas firmly believed that the nature of such punishment was medicinal, working for a Higher Purpose of Salvation, and not the motivation or origin of Salvation itself, which was always essentially a mystery and certainly not 'legalistic'.

Nevertheless, following this strain of thought Martin Luther held that Christ not only satisfies God the Father in His perfection but also bears mankind's penal punishment on the cross.


  When the merciful Father saw that we were being oppressed through the Law, that we were being held under a curse, and that we could not be liberated from it by anything, He sent His Son into the world, heaped all the sins of men upon Him, and said to Him: "Be Peter the denier; Paul the persecutor, blasphemer, and assaulter; David the adulterer; the sinner who ate the apple in Paradise; the thief on the cross. In short, be the person of all men. And see to it that You pay and make satisfaction for them.


Similar thinker John Calvin states the following:


  God in his capacity as judge is angry toward us. Hence as expiation must intervene in order that Christ as priest may obtain God's favor for us and appease his wrath. 
  
  We must, above all, remember this substitution, lest we tremble and remain anxious throughout life — as if God’s righteous vengeance, which the Son of God has taken upon himself, still hung over us.


And in like manner, Puritan John Owen states


  For to make satisfaction to God for our sins, is is required only that He undergo the punishment due to them; for that is the satisfaction required where sin is the debt.


John MacArthur even more radically states 


  God was punishing his own Son as if He had committed every wicked deed done by every sinner who would ever believe.


Note that the concentration has moved from the focus being on Christ perfecting mankind and facing consequentially a necessary punishment to Christ solely becoming man to appease the wrath of God, and implicitly taking upon Himself not only satisfactory punishment but penal punishment also. This is because the Reform thinkers conjoined the distinct types of punishment that Aquinas rather firmly tried to keep separate. It is in light of this Reformed theology that penal substitution supporters live today. Predominant denominations that use the catchphrases 'Jesus died as payment for our sin' include most Protestant denominations, most heavily among them being Lutherans and Methodists.