Who compared the Trinity to a triangle with three right angles? Nicholas of Cusa (1401-1464), wrote in part 1, chapter 12 of his De docta ignorantia (1440):


  Others who are very talented compared, to the Super-blessed Trinity, a triangle consisting
  of three equal right angles. 1
  
  Alii peritissimi Trinitati superbenedictae triangulum trium aequalium et rectorum angulorum compararunt.


I think we are all aware of representations of the Trinity using an equilateral plane triangle. The triangle described by Nicholas is not possible in the plane, though it makes sense on the surface of a sphere, for example. He seems to imagine it (see ch. 19) as a planar figure having three infinite sides, and argues that the angles must necessarily be equal and as large as possible. 2

Setting aside any mathematical problems with his geometry, I'm curious about the origin of this image. It would seem to be related to other geometric metaphors for God, such as "an infinite line" or "a circle whose centre is everywhere and whose circumference is nowhere". But I have not heard this one before.

Who came up with this idea and what did they mean by it?

(By the way, the "others" in the quotation above means "some people who are not Anselm" in context. I thought he might be talking about himself, but I don't know if there is another, earlier work of his which contains this triangle image. It seems to crop up in De venatione sapientiae chapter 26, but he says in its preface that he is 61 years old, which would make it later than De docta ignorantia. It's still possible that he's attributing his own belief to some deliberately-unnamed expert, but if so then I'd still like to find a definitive argument to that effect.)

1. Translation by Jasper Hopkins (1985) and appearing in his Complete Philosophical and Theological Treatises of Nicholas of Cusa, Arthur J. Banning Press, 2001. Parallel texts from Cusanus-Portal.
2. It's not clear why he thinks they must be right angles. In some sense it would be more consistent with his reasoning for them all to be 180 degrees, which is one reason to believe that in this case, he's talking about someone else's idea.
The likely1 source for this image is Nicholas's mentor, Heymeric de Campo (1395-1460),
who wrote a treatise on a geometric figure he called the "seal of eternity" 2. This is
related to the well-known image of the "shield of faith", 
a triangular diagram representing the persons of the Trinity. Unlike the shield, Heymeric's picture 
is a regular geometric picture; the various lines are coloured; and the interpretation is more 
complex. Note that the central point does not represent God - the whole picture is "an image of 
the divine government" (figura thearchica).



As shown above, there is an equilateral triangle within a circumscribing circle. In addition,
three radii of the circle are drawn to connect with the points of the triangle. Heymeric writes:


  three lines dropping down perpendicularly from these three angles [of the triangle] towards the centre of the circle
  tres lineas ab illis tribus angulis supra centrum eiusdem circuli perpendiculariter cadentes deorsum   


That is, these lines are orthogonal to the circle at the point of intersection; the angle of the triangle itself is not a right angle. Heymeric tends to call them funiculi, little cords. The colours are not reproduced in Imbach and Ladner - the picture above is based on my interpretation of Heymeric's description,


  The first side of the triangle, with the portion of the circle of which it is the chord,
  and the dependent line from its angle of origin, is to be blue in colour.
  latus primum trianguli cum porcione circuli cuius est corda et linea dependente ab eius angulo
  originali sit coloris blavei [and then similarly for the other colours]  


According to Heymeric, the circle represents perfection, the points of the triangle are the
persons of the trinity, and the funiculi are the efficient, exemplary, and final causes
of Aristotelian-Scholastic philosophy. These are justified with reference to three Biblical
quotations,


  From him and through him and for him are all things (Romans 11:36)
  A threefold cord is not easily broken (Ecclesiastes 4:12)
  God has weighed out the dust of the earth with three fingers (Isaiah 40:12)


The last one follows Jerome's interpretation of the Hebrew word בַּשָּׁלִ֖שׁ (Strong's 7991), which has various 
meanings related to the number three; depending on context, it may refer to an officer 
"of the third rank", a triangular musical instrument such as a nabla/nevel, or, as here,
a unit of measure of some kind. The three colours stand for celestial substance (blue),
life (green), and flame (red), corresponding to the Father, Son, and Holy Spirit.
It would be tempting to imagine some knowledge of the RGB colour system, but Heymeric is
writing long before Newton, let alone Young and Helmholtz.

The bulk of Heymeric's treatise consists of a point-by-point explication of how this
figure represents the Trinitarian doctrine of Thomas Aquinas, as described in Book 1
of his Summa contra Gentiles. For example, the concept "God is blessed and his
blessedness exceeds all other blessedness" (SCG 1.100-102)
is expressed by the fact that the figure may be geometrically expanded to
any desired size (para. 66, f. 84v); and "Creatures may carry a resemblance to God"
(SCG 1.29) by the radii, 
representing causality, that drop down like rays of light (para. 45, f. 83r).
In particular, the orthogonality of these lines to the circle is used almost as a pun,
for the "rectitude" and "rightness" of God, whenever these ideas arise in the comparison.

In the adaptation by Nicholas of Cusa, the diagram is "really" meant to be infinite,
so that (according to his mathematics) the component parts merge into an indistinguishable unity.
This mirrors the continual assertion of Heymeric that the figure is only a metaphor,
suitable to our senses and imagination, of something transcendental; and both authors are
known for their insistence on the fallibility of human reasoning.

1. Paul Wilpert. De docta ignorantia / Die belehrte Unwissenheit, volume 1.
Hamburg: Felix Meiner, 1964 (1st ed.). Note at 15a.120.n34.
2. Tractatus de sigillo eternitatis omnium arcium et scienciarum exemplari
a magistro Heymerico de Campo Basilee tempore concilii editus. In Cusanusstift
MS 106, f. 77r - 85r. Written between 1432 and 1435. The Latin quotations in this answer
are reproduced from Heymericus de Campo Opera Selecta, volume 1,
Ruedi Imbach and Pascal Ladner (eds.), Univ. Freiburg Schweiz, 2001. The English translation is my own fault.