Visual Question Answering (VQA) models have struggled with counting objects in natural images so far. We identify a fundamental problem due to soft attention in these models as a cause. To circumvent this problem, we propose a neural network component that allows robust counting from object proposals. Experiments on a toy task show the effectiveness of this component and we obtain state-of-the-art accuracy on the number category of the VQA v2 dataset without negatively affecting other categories, even outperforming ensemble models with our single model. On a difficult balanced pair metric, the component gives a substantial improvement in counting over a strong baseline by 6.6%.
Enabling Visual Question Answering models to count by handling overlapping object proposals. This paper proposes a hand-designed network architecture on a graph of object proposals to perform soft non-maximum suppression to get object count. Focuses on a counting problem in visual question answering using attention mechanism and propsoe a differentiable counting compoent which explicitly counts the number of objects This paper tackles the object counting problem in visual question answering, it proposes many heuristics to find the correct count.