When it seems a Pope has heretical beliefs, who determines whether those beliefs are Magisterial? I know that even some Catholics think that certains Popes held personal heresies (were material heretics). For example Honorius I believed that Jesus had only one will, the heresy of monothelitism.

I understand that a Pope might be a heretic, but to preserve the believed infallibility of his office he can't impose those beliefs ex cathedra—i.e., using the full weight of his office—onto the Church.

Who has the authority to judge which statements issued by the Pope are infallible?

Note: This question seems to address what the "criteria" used are, but I am asking: Who uses these criteria to decide the matter?
Whether valid popes have been heretics
It isn't necessarily true that Pope Honorius I was a heretic. St. Robert Cardinal Bellarmine, the great doctor of the papacy, defended several popes, including Pope Honorius I, against accusations of heresy in his work that is translated into English as Papal Error?: A Defense of Popes said to have Erred in Fatih (ch. 4 is on Pope Honorius I). Among the seven accusations against Pope Honorius I that St. Robert's refutes are the the accusations that Dialogist quotes below.
St. Robert also treats the question of whether a Pope can be a heretic in  De Romano Pontifice, lib. II, cap. 30 (On the Roman Pontiff). He argues that a pope cannot be a heretic; if he is, he, like St. Francis de Sales wrote in his Catholic Controversy p. 306,

falls ipso facto from his dignity and out of the Church, and the Church must either deprive him, or, as some say, declare him deprived, of his Apostolic See, and must say as S. Peter did: Let another take his bishopric.**Acts i.

Who can judge a pope's validity?
As I mentioned in my answer to your other question, only a valid Pope is the supreme judge:

Can.  1442 The Roman Pontiff is the supreme judge for the entire Catholic world; he renders judicial decisions personally, through the ordinary tribunals of the Apostolic See, or through judges he has delegated.

This law is based upon the infallible dogma of papal primacy which, pace Dialogist's answer above, has been true since the time Christ instituted the Papacy (Matt. 16:16-19), founding it on the rock of St. Peter himself, the first Pope. As the First Vatican Council (Pastor æternus ch. 4 &quot;Of the Institution of the Apostolic Primacy in blessed Peter&quot;), united with the Pope, infallibly declared:

We therefore teach and declare that, according to the testimony of the Gospel, the primacy of jurisdiction over the universal Church of God was immediately and directly promised and given to blessed Peter the Apostle by Christ the Lord. For it was to Simon alone, to whom he had already said: &quot;Thou shalt be called Cephas,&quot; [John i. 42.] that the Lord after the confession made by him, saying: 'Thou art the Christ, the Son of the living God,' addressed these solemn words: 'Blessed art thou, Simon Bar-Jona, because flesh and blood have not revealed it to thee, but my Father who is in heaven.
And I say to thee that thou art Peter; and upon this rock I will build my Church, and the gates of hell shall not prevail against it. And I will give to thee the keys of the kingdom of heaven. And whatsoever thou shalt bind on earth, it shall be bound also in heaven; and whatsoever thou shalt loose on earth, it shall be loosed also in heaven.' [Matt. xvi. 16-19.] And it was upon Simon alone that Jesus after his resurrection bestowed the jurisdiction of chief pastor and ruler over all his fold in the words: 'Feed my lambs; feed my sheep.' [John xxi. 15-17.] At open variance with this clear doctrine of Holy Scripture as it has been ever understood by the Catholic Church are the perverse opinions of those who, while they distort the form of government established by Christ the Lord in his Church, deny that Peter in his single person, preferably to all the other Apostles, whether taken separately or together, was endowed by Christ with a true and proper primacy of jurisdiction; or of those who assert that the same primacy was not bestowed immediately and directly upon blessed Peter himself, but upon the Church, and through the Church on Peter as her minister.
If any one, therefore, shall say that blessed Peter the Apostle was not appointed the Prince of all the Apostles and the visible Head of the whole Church Militant; or that the same directly and immediately received from the same our Lord Jesus Christ a primacy of honor only, and not of true and proper jurisdiction: let him be anathema.

Any Catholic can and must recognize and refuse to follow an antipope.
However, any Catholic who recognizes an antipope as an antipope must refuse to follow him. St. Paul says twice in Galatian 1:8-9:

But although we, or an Angel from heaven, evangelize to you beside that which we have evangelized to you, be he anathema.
As we have said before, so now I say again, If any evangelize to you beside that which we have evangelized to you, be he anathema.

Notice, St. Paul says anyone, which certainly includes those who appear to be popes (antipopes).