What is the right theoretical model to design algorithms for current and upcoming high performance computers This question is similar to a more general question for what is the right theoretical model of a computer to design algorithm and data structures in.
Here, I ask specifically about current high performance computers (like the ones listed as the Top 500) or even about upcoming supercomputers.

Given that these computers are typically working on huge data sets (it seems that some people use such machines mainly because they have enormous combined main memory) aspects of the I/O-model (introduced by Aggarwal and Vitter in 1988) and its parallel version, the PEM (Arge, Goodrich,  Nelson and Sitchinava in 2008) should be present. 
On the other hand, there should be something about communication, in particular punishing ultra small packages to all other computing nodes. 

As you might imagine, I am not afraid that I am running out of ideas when creating a new model, but I am a little worried that I might overlook previous attempts in doing so, in particular because I have the impression that the years 1980-1995 or so saw many such modeling attempts (like BSP or bridging models) that seem to not have been widely used.

What models should I take a closer look at?
At PODC 2009, Bruce Hendrickson gave a phenomenmal invited talk about these issues.  (His slides don't appear to be online, but you might want to ask him if you could see them.)  I don't think there's a "right" model yet -- bonus for you! -- but I would suggest you look at his papers, especially the ones on the Graphs and Architectures page, where he tries to figure out how to handle enormous graphs with little structure (i.e. "modern" datasets) on massively multithreaded machines.