Research about geometry negotiation in widget layout It's very common to view GUI layout as an optimization problem.

I've uncovered a lot of research around linear constraint systems, but a lot of real world layouts can not be represented linearly. For example, a simple "Wrapping" layout introduces discontinuities at line or column breaks.

Some real world systems compute layout recursively. They alternate between measuring and positioning widgets, then iterate until a fixed point. I've seen this process called "geometry negotiation" in the documentation for several of these widget libraries. However, I can not find any formal treatment of this approach.

Outside of linear constraints, are there any other formalisms targeted at these sorts of layout optimization systems?
I would suggest looking at these survey papers and tracking down the citations in the bibliography, and digging up subsequent papers which cited these. 


Walter Hower, and  Winfried Graf. Research in constraint-based layout, visualization, CAD, and related topics : a bibliographical survey, 1995. 
Simon Lok and Steven Feiner, A survey of automated layout techniques for information presentations, 2001. 


If you're more interested in the engineering aspects, you should look at Brad Myers' work (from the late 80s/early 90s) on the Garnet and Amulet toolkits. They have a TOPLAS paper, Lessons learned about one-way, dataflow constraints in the Garnet and Amulet graphical toolkits, which summarizes what they learned.