I think its more something to do with if they carry an air of ignorance around them, or are snobby in any way.
That's not to say everyone born into wealth is like that...I've met many humble folk born into some lucky circumstances, but when someone is bratty or anything like that, I think people find it irritating