Machine Learning: Calibrating SubGroups of Probability Predictions inside a Dataset which should sum to 100% I am working on an interesting type of problem where I want to make predicitons for individual elements within subgroups- with the knowledge that the sum of the probabilities within a subgroup should sum to 1. The initial response will probably be to use a softmax system but this is subtley different (unless I have misunderstood something).
I will use an analogy to keep it straighforward:
Effectively in a flowerpot there are 10 flowers- only 1 of which is certainly is a daffodil and the other 9 are certainly not- some are quite similar to daffodils while others are more similar to flavorsome herbs or thorny bushes.
I have a dataset of 100,000 flowers (which are in 10,000 flagged lower pots/subgroups). I know characteristics for each of the flowers such as stem height, petal colour, odour pleasure, diameter etc. as well it being flagged whether or not it is indeed a delightful daffodil.
I am able to generate a logistic ML model quite easily which informs me of the probability that a certain flower is a daffodil based on the data from the other flowers. But I am unable to get it to work so the sums of probabilities within sub groups equal one- generally they are close but often as low as 65% or as high as 130%.
My interim solution is to carry out a post prediction clean, which determines the total probability assigned to the group and normalises predictions so they sum to one. Not awful but not great.
Am I a moron? Is this a softmax problem? Or something that a ML algo in existence can handle?
Thank you, a daffodil to those who help :)
Let $p_i=\Pr[F_i]$ be the probability that flower $i$ is a daffodil, according to your model (without taking into account the group structure).
Now you are given a group of 10 flowers, with probabilities $p_1,\dots,p_{10}$, and you're given the promise that exactly one of them is a daffodil.  To update your probabilities based on this additional promise, you can use Bayes rule.  Let $E$ be the event that exactly one in the group is a daffodil.    Then
$$\begin{align*}
\Pr[F_i|E] &amp;= {\Pr[F_i \land E] \over \Pr[E]}\\
&amp;= {\Pr[F_i \land E] \over \sum_j \Pr[F_j \land E]}.
\end{align*}$$
Note that
$$\Pr[F_i \land E] = \Pr[\neg F_1 \land \cdots \land \neg F_{i-1} \land F_i \land \neg F_{i+1} \land \cdots] = (1-p_1) (1-p_2) \cdots p_i \cdots (1-p_{10}) = {p_i \over 1-p_i} \prod_j (1-p_j).$$
Therefore,
$$\Pr[F_i|E] = {{p_i \over 1-p_i} \over \sum_j {p_j \over 1-p_j}}.$$
This is the probability you wanted to compute; so, use that.