Formal Definition/counter part in mathematics for “Objects” of Object Oriented Models This is a question I asked in mathematics SE forum, and I was referred here. So here is the question- 

I'm a newbie in both formal mathematics and theoretical computer science, so please bear with me if you find my question is not properly framed. Object Oriented Modeling seems very useful in defining complex interactions when simulating real world. But it's mostly used in programming. I was wondering if we have a similar concept in mathematics. When we're doing programming, we can understand the concept of "Objects" and "Object Oriented Programming" and just implement it. But do we have formal definition of "Objects" in terms of Set Theory? Or for that matter, any other formal mathematical theory?

Can we implement/ formally define three primary object orient modeling concepts- 1. Encapsulation 2. Inheritance 3. Polymorphism

I know question is too broad, but would really appreciate if you can provide some pointers as well so that I can understand these concepts better.
The answer is complicated, for two reasons.  


Different people in Computer Science interpret the term "object" differently.  One is that an object consists of some data and operations packaged together.  The other is that an object is all that but also has "state," i.e., it is some form of a changeable entity.
There are deep philosophical issues to do with what "change" means (and what "entity" means, as it is constantly changing), and whether mathematical descriptions actually capture changeable entities.


Object in the sense of data + operations: That is pretty standard in mathematics.  Take any group theory text book.  It will have somewhere a definition such as $h_g(x) = g x g^{-1}$.  (It is a conjugation operator.)  The $h_g$ is an "object" in this terminology.  It has some data ($g$) and an operation $x \mapsto g x g^{-1}$.  Or you can make it more object-y by taking the pair $\langle g, x \mapsto gxg^{-1}\rangle$ or the triple $\langle g, x \mapsto gxg^{-1}, x \mapsto g^{-1}xg\rangle$.  You can construct these kind of "objects" in any functional programming language that has lambda abstraction and some way to form tuples.  Abadi and Cardelli's "Theory of Objects" deals with objects of this kind extensively.

Objects with state (or objects that change): Does mathematics have such things?  I don't think so.  I haven't seen a mathematician talk about anything that changes, not in his/her professional life.  Newton used to write $x$ for the position of a particle, which is supposedly changing, and $\dot{x}$ for its rate of change.  Mathematicians eventually figured out that what Newton was talking about was a function $x(t)$ from real numbers into a vector space, and $\dot{x}$ was another such function which was the first derivative of $x(t)$ with respect to $t$.  From this, many deep-thinking mathematicians have concluded that change doesn't really exist and all you have are functions of time.  But what was changing in Newtonian mechanics wasn't the position, but the particle.  The position is its instantaneous state.  No mathematician or physicist would pretend that a particle is a mathematical idea.  It is a physical thing.

So it is with objects.  They are "physical" things, and the states are their mathematical attributes.  For a nice discussion of this aspect, see the Chapter 3 of Abelson and Sussman's Structure and Interpretation of Computer Programs.  This is a text-book at MIT and they teach it to all scientists and engineers, who I think understand "physical" things perfectly fine.

The fact that particles aren't mathematical doesn't mean that we can't deal with them mathematically.  If you ask a mathematician to model a two-particle system, he will immediately make up two functions and call them $x_1(t)$ and $x_2(t)$.  So, the two particles reduces to two meaningless indices (1 and 2).  This is the mathematician's way of saying we don't know what those particles are and we don't care.  All we need to know is that their positions evolve independently (or separately).  So, we will model them by two separate functions.

Similarly the standard mathematical way to model object-oriented programs is to treat each object as an index into the state space.  The only difference is that since objects come and go, and the structure of the system is dynamic, we need to extend it to a "possible world" model where each world is basically a collection of indices.  Allocation and deallocation of objects would involve moving from one world to another.

There is a problem though.  Unlike in mechanics, we want the state of our objects to be encapsulated.  But the mathematical descriptions of objects put states all over the place, completely destroying encapsulation.  There is a mathematical trick called "relational parametricity" which can be used to cut things back to size.  I won't go into it now, except to emphasize that it is a mathematical trick, not a very conceptual explanation of encapsulation.  A second way of modelling objects mathematically, with encapsulation, is to finesse the states and describe the object behaviour in terms of observable events.  For a good discussion of both of these models, I can refer you to my paper titled Objects and classes in Algol-like Languages.



[Note added:]

A nice analysis of the mathematical underpinnings of objects can be found in William Cook's article "On Understanding Data Abstraction, Revisited".