
Learning Open Set Network with Discriminative Reciprocal Points
Open set recognition is an emerging research area that aims to simultane...
read it

On the Sample Complexity of Rank Regression from Pairwise Comparisons
We consider a rank regression setting, in which a dataset of N samples w...
read it

Multigroup Agnostic PAC Learnability
An agnostic PAC learning algorithm finds a predictor that is competitive...
read it

Modularity in QueryBased Concept Learning
We define and study the problem of modular concept learning, that is, le...
read it

Remember What You Want to Forget: Algorithms for Machine Unlearning
We study the problem of forgetting datapoints from a learnt model. In th...
read it

Merging versus Ensembling in MultiStudy Machine Learning: Theoretical Insight from Random Effects
A critical decision point when training predictors using multiple studie...
read it

Abnormal thyroid function is common in takotsubo syndrome and depends on two distinct mechanisms: results of a multicentre observational study
In this study, machine learning helped to identify two different phenoty...
read it
Agnostic learning with unknown utilities
Traditional learning approaches for classification implicitly assume that each mistake has the same cost. In many realworld problems though, the utility of a decision depends on the underlying context x and decision y. However, directly incorporating these utilities into the learning objective is often infeasible since these can be quite complex and difficult for humans to specify. We formally study this as agnostic learning with unknown utilities: given a dataset S = {x_1, …, x_n} where each data point x_i ∼𝒟, the objective of the learner is to output a function f in some class of decision functions ℱ with small excess risk. This risk measures the performance of the output predictor f with respect to the best predictor in the class ℱ on the unknown underlying utility u^*. This utility u^* is not assumed to have any specific structure. This raises an interesting question whether learning is even possible in our setup, given that obtaining a generalizable estimate of utility u^* might not be possible from finitely many samples. Surprisingly, we show that estimating the utilities of only the sampled points S suffices to learn a decision function which generalizes well. We study mechanisms for eliciting information which allow a learner to estimate the utilities u^* on the set S. We introduce a family of elicitation mechanisms by generalizing comparisons, called the kcomparison oracle, which enables the learner to ask for comparisons across k different inputs x at once. We show that the excess risk in our agnostic learning framework decreases at a rate of O(1/k). This result brings out an interesting accuracyelicitation tradeoff – as the order k of the oracle increases, the comparative queries become harder to elicit from humans but allow for more accurate learning.
READ FULL TEXT
Comments
There are no comments yet.