Can a pope be a heretic? Say for example a Pope infallibly declares that Mary was not a virgin, and is currently rotting in Hell. This obviously contradicts established Catholic Dogma. What would happen in this case? Would the cardinals get together and kick the Pope out of his office? Is a heretical ex cathedra definition cause for an automatic loss of the Papal position which would lead to a sedevacantism situation?
What would happen if a pope were to be a formal heretic?


Formal heresy is obstinately denying Catholic truths, even after being warned.

If a pope were to define a "dogma" ex cathedra that contradicts Catholic teaching, this "dogma" would not be binding nor be protected by papal infallibility as defined at Vatican I. Definable Catholic dogmas must first be material dogmas (cf. Pope Pius XII's letter to all bishops regarding to the definability of the Dogma of the Assumption), i.e., truths connected to Revelation and believed by the Church in all times and space, either explicitly or implicitly. Even Popes are bound to Tradition.


  Does the Pope lose his office?


The common opinion is yes, but when is another question:

There are two major camps among theologians regarding when. One (that of St. Robert Bellarmine and Suarez) says that he would immediately lose his office, even before being warned and given an opportunity to correct his error. The other (that of John of St. Thomas and Cajetan) says God would take the office away from him after the Church convicts him and declares him a heretic (or "vitandus" = "to be avoided").


  If so, how is that formalised?


It is divine law that the Church must separate herself from heretics (cf. Gal. 1:8-9), so the Church has a duty to warn the pope, try him, and (if it comes to this) declare him a heretic and proceed to elect a valid pope.


  Do the cardinals have the power to kick him out?


No, not even all the world's bishops have the power to depose a valid pope. To think that would be the heresy of Conciliarism, that Councils of bishops (sans the pope) have equal or greater authority than the pope.

See also my answer to the question "In Catholic understanding when there is more than one Pope, who decides which Pope is real and by what criteria?"

An excellent recent work that covers all these issues (whether a pope can be a heretic; whether he can be elected if he was previously a heretic; whether a pope loses his office by proclaiming heresy; who can declare him a heretic; when he loses his office; how is he deposed, considering he has no superior on earth; etc.; etc.) is:


True or False Pope? by John Salza &amp; Robert Siscoe (St. Thomas Aquinas Seminary Press, Aug. 2015)


on Amazon



Another good one is:


Can the Pope Go Bad?: A Call for the Resumption of a Debate Not Taken Seriously Since the 17th Century by the Brazilian intellectual A. V. Xavier da Silveira (Catholic Research Institute, Sep. 1999)