FEERCI: A Package for Fast Non-Parametric Confidence Intervals for Equal Error Rates in Amortized O(m log n)

    Research output: Chapter in Book/Report/Conference proceedingConference contributionAcademicpeer-review

    1 Citation (Scopus)
    40 Downloads (Pure)

    Abstract

    Equal Error Rates (EERs), or other weighted relations between False Match and Non-Match Rates (FMR/FNMR), are often used as a performance metric for biometric systems. Confidence Intervals (CIs) are used to denote the uncertainty underlying these EERs, with many methods existing to estimate said CIs in both parametric and non-parametric ways. These confidence intervals provide, foremost, a method of comparing scoring/ranking functions. Non-parametric methods often suffer from high computational costs, but do not make assumptions as to the shape of the EER-and score distributions. For both EERs and CIs, contemporary open-source toolkits leave room for improvement in terms of computational efficiency. In this paper, we introduce the Fast EER (FEER) algorithm that calculates an EER in O(log n) on a sorted score list, we show how to adapt the FEER algorithm to calculate non-parametric, bootstrapped EER CIs (FEERCI) in O(m log n) given m resamplings, and we introduce an opinionated open-source package named feerci that provides implementations of the FEER and FEERCI algorithm. We provide speed and accuracy benchmarks for the feerci package, comparing it against the most-used methods of calculating EERs in Python and show how it is able to calculate EERs and CIs on very large score lists faster than contemporary toolkits can calculate a single EER.

    Original languageEnglish
    Title of host publication2018 International Conference of the Biometrics Special Interest Group, BIOSIG 2018
    EditorsArslan Bromme, Andreas Uhl, Christoph Busch, Christian Rathgeb, Antitza Dantcheva
    Place of PublicationPiscataway, NJ
    PublisherIEEE
    Number of pages5
    ISBN (Electronic)978-3-88579-676-3
    ISBN (Print)978-1-5386-6007-2
    DOIs
    Publication statusPublished - 10 Oct 2018
    Event17th International Conference of the Biometrics Special Interest Group, BIOSIG 2018 - Darmstadt, Germany
    Duration: 26 Sep 201828 Sep 2018
    Conference number: 17

    Publication series

    NameInternational Conference of the Biometrics Special Interest Group (BIOSIG)
    PublisherIEEE
    Volume2018
    ISSN (Print)1617-5468

    Conference

    Conference17th International Conference of the Biometrics Special Interest Group, BIOSIG 2018
    Abbreviated titleBIOSIG 2018
    CountryGermany
    CityDarmstadt
    Period26/09/1828/09/18

    Keywords

    • Bootstrap Confidence Interval
    • Equal Error Rate
    • Open Source
    • Receiver operating characteristic

    Fingerprint Dive into the research topics of 'FEERCI: A Package for Fast Non-Parametric Confidence Intervals for Equal Error Rates in Amortized O(m log n)'. Together they form a unique fingerprint.

  • Cite this

    Haasnoot, E., Khodabakhsh, A., Zeinstra, C., Spreeuwers, L., & Veldhuis, R. (2018). FEERCI: A Package for Fast Non-Parametric Confidence Intervals for Equal Error Rates in Amortized O(m log n). In A. Bromme, A. Uhl, C. Busch, C. Rathgeb, & A. Dantcheva (Eds.), 2018 International Conference of the Biometrics Special Interest Group, BIOSIG 2018 [8553607] (International Conference of the Biometrics Special Interest Group (BIOSIG); Vol. 2018). Piscataway, NJ: IEEE. https://doi.org/10.23919/BIOSIG.2018.8553607