Combining Human AI Predictions

Classification tasks are often hindered by the limitations of both human and AI models, leading to inaccuracies. To address this issue, recent research has focused on the Human-AI team model, where the AI model’s probabilistic output is combined with a human-predicted class label. This approach has been shown to consistently outperform the accuracy of either the human or the AI model alone. However, previous studies have only considered a single human in this combination.

This paper explores the use of multiple human labels in combination with the AI model’s output, drawing inspiration from the crowdsourcing literature, which combines labels from multiple humans. The results show that combining multiple human labels with the AI model’s output can lead to significant improvement in accuracy. However, simply combining multiple human labels with the AI model can also lead to poor accuracy.

To address this, the paper presents an approach for merging human labels with the AI model’s output and an efficient algorithm for finding the optimal subset of human labels. The optimal subset is determined by selecting the combination of human labels that offer the most accurate output. The results show that the combined model outperforms both the AI model and individual human performance, as well as outperforming previous single human and simple combination techniques.

In conclusion, this paper demonstrates the potential for significant improvement in classification task accuracy by combining multiple human labels with the AI model’s output. However, a naive combination of human labels with the AI model can lead to poor accuracy. Therefore, there is a strong need for an intelligent strategy to select a subset of humans and combine their labels. The approach and algorithm presented in this paper offer a solution to this challenge.