### Abstract

When the performance of a classifier is empirically evaluated, the Area Under Curve (AUC) is commonly used as a one dimensional performance measure. In general, the focus is on good performance (AUC towards 1). In this paper, we study the other side of the performance spectrum (AUC towards 0.50) as we are interested to which extend a classifier is random given its AUC. We present the exact probability distribution of the AUC of a truely random classifier, given a finite number of distinct genuine and imposter scores. It quantifies the randomness of the measured AUC. The distribution involves the restricted partition function, a well studied function in number theory. Although other work exists that considers confidence bounds on the AUC, the novelty is that we do not assume any underlying parametric or non- parametric model or specify an error rate. Also, in cases in which a limited number of scores is available, for example in forensic case work, the exact distribution can deviate from these models. For completeness, we also present an approximation using a normal distribution and confidence bounds on the AUC.

Original language | English |
---|---|

Title of host publication | 2017 International Conference of the Biometrics Special Interest Group (BIOSIG) |

Subtitle of host publication | BIOSIG 2017 |

Editors | Arslan Brömme, Christoph Busch, Antitza Dantcheva, Christian Rathgeb, Andreas Uhl |

Publisher | Gesellschaft für Informatik |

ISBN (Electronic) | 9783885796640 |

DOIs | |

Publication status | Published - 28 Sep 2017 |

Event | 16th International Conference of the Biometrics Special Interest Group 2017 - Darmstadt, Germany Duration: 20 Sep 2017 → 22 Sep 2017 Conference number: 16 http://fg-biosig.gi.de/archiv/biosig-2017.html |

### Conference

Conference | 16th International Conference of the Biometrics Special Interest Group 2017 |
---|---|

Abbreviated title | BIOSIG 2017 |

Country | Germany |

City | Darmstadt |

Period | 20/09/17 → 22/09/17 |

Internet address |

### Fingerprint

### Keywords

- Approximation.
- AUC
- Exact Distribution
- Random Classifier

### Cite this

*2017 International Conference of the Biometrics Special Interest Group (BIOSIG): BIOSIG 2017*[8053509] Gesellschaft für Informatik. https://doi.org/10.23919/BIOSIG.2017.8053509

}

*2017 International Conference of the Biometrics Special Interest Group (BIOSIG): BIOSIG 2017.*, 8053509, Gesellschaft für Informatik, 16th International Conference of the Biometrics Special Interest Group 2017, Darmstadt, Germany, 20/09/17. https://doi.org/10.23919/BIOSIG.2017.8053509

**How Random Is a Classifier Given Its Area under Curve?** / Zeinstra, Chris; Veldhuis, Raymond; Spreeuwers, Luuk.

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution › Academic › peer-review

TY - GEN

T1 - How Random Is a Classifier Given Its Area under Curve?

AU - Zeinstra, Chris

AU - Veldhuis, Raymond

AU - Spreeuwers, Luuk

PY - 2017/9/28

Y1 - 2017/9/28

N2 - When the performance of a classifier is empirically evaluated, the Area Under Curve (AUC) is commonly used as a one dimensional performance measure. In general, the focus is on good performance (AUC towards 1). In this paper, we study the other side of the performance spectrum (AUC towards 0.50) as we are interested to which extend a classifier is random given its AUC. We present the exact probability distribution of the AUC of a truely random classifier, given a finite number of distinct genuine and imposter scores. It quantifies the randomness of the measured AUC. The distribution involves the restricted partition function, a well studied function in number theory. Although other work exists that considers confidence bounds on the AUC, the novelty is that we do not assume any underlying parametric or non- parametric model or specify an error rate. Also, in cases in which a limited number of scores is available, for example in forensic case work, the exact distribution can deviate from these models. For completeness, we also present an approximation using a normal distribution and confidence bounds on the AUC.

AB - When the performance of a classifier is empirically evaluated, the Area Under Curve (AUC) is commonly used as a one dimensional performance measure. In general, the focus is on good performance (AUC towards 1). In this paper, we study the other side of the performance spectrum (AUC towards 0.50) as we are interested to which extend a classifier is random given its AUC. We present the exact probability distribution of the AUC of a truely random classifier, given a finite number of distinct genuine and imposter scores. It quantifies the randomness of the measured AUC. The distribution involves the restricted partition function, a well studied function in number theory. Although other work exists that considers confidence bounds on the AUC, the novelty is that we do not assume any underlying parametric or non- parametric model or specify an error rate. Also, in cases in which a limited number of scores is available, for example in forensic case work, the exact distribution can deviate from these models. For completeness, we also present an approximation using a normal distribution and confidence bounds on the AUC.

KW - Approximation.

KW - AUC

KW - Exact Distribution

KW - Random Classifier

UR - http://www.scopus.com/inward/record.url?scp=85034619394&partnerID=8YFLogxK

U2 - 10.23919/BIOSIG.2017.8053509

DO - 10.23919/BIOSIG.2017.8053509

M3 - Conference contribution

BT - 2017 International Conference of the Biometrics Special Interest Group (BIOSIG)

A2 - Brömme, Arslan

A2 - Busch, Christoph

A2 - Dantcheva, Antitza

A2 - Rathgeb, Christian

A2 - Uhl, Andreas

PB - Gesellschaft für Informatik

ER -