Learning with Complex (Multivariate/Non-Decomposable) Performance Measures
[Summary & Contributions] | [Relevant Publications]
Summary and Contributions
In many applications of machine learning, the performance measure used to evaluate learned models cannot be expressed as an expectation or sum of losses on individual data points, but rather is a more complex quantity. For example, this is true of performance measures such as the balanced error rate used in class imbalance settings, the F1 measure used in information retrieval applications, and many others. Each of these performance measures can be viewed as a general function of the confusion matrix associated with a prediction model and data distribution; while loss-based performance measures correspond to linear functions of the confusion matrix, more complex performance measures such as the balanced error rate and F1 measure correspond to nonlinear functions of the confusion matrix. We have developed both new theoretical analysis for such settings, and a new optimization based framework for designing statistically consistent learning algorithms for broad classes of complex multiclass performance measures, for which no such results were known previously; the new algorithms not only outperform previous state-of-the-art algorithms in terms of generalization accuracy, but are also orders of magnitude faster. We have also developed noise-corrected algorithms for learning with such complex performance measures in the presence of noisy labels.
Relevant Publications
- Harikrishna Narasimhan, Harish G. Ramaswamy, Shiv Kumar Tavker, Drona Khurana, Praneeth Netrapalli, and Shivani Agarwal.
Consistent multiclass algorithms for complex metrics and constraints.
Journal of Machine Learning Research, 25(367):1-81, 2024.
[pdf] - Mingyuan Zhang and Shivani Agarwal.
Multiclass learning from noisy labels for non-decomposable performance measures.
In Proceedings of the 27th International Conference on Artificial Intelligence and Statistics (AISTATS), 2024.
[pdf] - Harikrishna Narasimhan, Harish G. Ramaswamy, Aadirupa Saha and Shivani Agarwal.
Consistent multiclass algorithms for complex performance measures.
In Proceedings of the 32nd International Conference on Machine Learning (ICML), 2015.
[pdf] - Harikrishna Narasimhan, Rohit Vaish and Shivani Agarwal.
On the statistical consistency of plug-in classifiers for non-decomposable performance measures.
In Advances in Neural Information Processing Systems (NIPS), 2014.
[pdf] - Aditya K. Menon, Harikrishna Narasimhan, Shivani Agarwal and Sanjay Chawla.
On the statistical consistency of algorithms for binary classification under class imbalance.
In Proceedings of the 30th International Conference on Machine Learning (ICML), 2013.
[pdf]
Shivani Agarwal 