FEERCI: A Package for Fast Non-Parametric Confidence Intervals for Equal Error Rates in Amortized O(m log n)

    Research output: Chapter in Book/Report/Conference proceedingConference contributionAcademicpeer-review

    14 Downloads (Pure)

    Abstract

    Equal Error Rates (EERs), or other weighted relations between False Match and Non-Match Rates (FMR/FNMR), are often used as a performance metric for biometric systems. Confidence Intervals (CIs) are used to denote the uncertainty underlying these EERs, with many methods existing to estimate said CIs in both parametric and non-parametric ways. These confidence intervals provide, foremost, a method of comparing scoring/ranking functions. Non-parametric methods often suffer from high computational costs, but do not make assumptions as to the shape of the EER-and score distributions. For both EERs and CIs, contemporary open-source toolkits leave room for improvement in terms of computational efficiency. In this paper, we introduce the Fast EER (FEER) algorithm that calculates an EER in O(log n) on a sorted score list, we show how to adapt the FEER algorithm to calculate non-parametric, bootstrapped EER CIs (FEERCI) in O(m log n) given m resamplings, and we introduce an opinionated open-source package named feerci that provides implementations of the FEER and FEERCI algorithm. We provide speed and accuracy benchmarks for the feerci package, comparing it against the most-used methods of calculating EERs in Python and show how it is able to calculate EERs and CIs on very large score lists faster than contemporary toolkits can calculate a single EER.

    Original languageEnglish
    Title of host publication2018 International Conference of the Biometrics Special Interest Group, BIOSIG 2018
    EditorsArslan Bromme, Andreas Uhl, Christoph Busch, Christian Rathgeb, Antitza Dantcheva
    Place of PublicationPiscataway, NJ
    PublisherIEEE
    Number of pages5
    ISBN (Electronic)978-3-88579-676-3
    ISBN (Print)978-1-5386-6007-2
    DOIs
    Publication statusPublished - 10 Oct 2018
    Event17th International Conference of the Biometrics Special Interest Group, BIOSIG 2018 - Darmstadt, Germany
    Duration: 26 Sep 201828 Sep 2018
    Conference number: 17

    Publication series

    NameInternational Conference of the Biometrics Special Interest Group (BIOSIG)
    PublisherIEEE
    Volume2018
    ISSN (Print)1617-5468

    Conference

    Conference17th International Conference of the Biometrics Special Interest Group, BIOSIG 2018
    Abbreviated titleBIOSIG 2018
    CountryGermany
    CityDarmstadt
    Period26/09/1828/09/18

    Fingerprint

    confidence
    intervals
    lists
    Ferromagnetic resonance
    biometrics
    scoring
    ranking
    Biometrics
    Computational efficiency
    costs
    estimates
    Costs

    Keywords

    • Bootstrap Confidence Interval
    • Equal Error Rate
    • Open Source
    • Receiver operating characteristic

    Cite this

    Haasnoot, E., Khodabakhsh, A., Zeinstra, C., Spreeuwers, L., & Veldhuis, R. (2018). FEERCI: A Package for Fast Non-Parametric Confidence Intervals for Equal Error Rates in Amortized O(m log n). In A. Bromme, A. Uhl, C. Busch, C. Rathgeb, & A. Dantcheva (Eds.), 2018 International Conference of the Biometrics Special Interest Group, BIOSIG 2018 [8553607] (International Conference of the Biometrics Special Interest Group (BIOSIG); Vol. 2018). Piscataway, NJ: IEEE. https://doi.org/10.23919/BIOSIG.2018.8553607
    Haasnoot, Erwin ; Khodabakhsh, Ali ; Zeinstra, Chris ; Spreeuwers, Luuk ; Veldhuis, Raymond. / FEERCI : A Package for Fast Non-Parametric Confidence Intervals for Equal Error Rates in Amortized O(m log n). 2018 International Conference of the Biometrics Special Interest Group, BIOSIG 2018. editor / Arslan Bromme ; Andreas Uhl ; Christoph Busch ; Christian Rathgeb ; Antitza Dantcheva. Piscataway, NJ : IEEE, 2018. (International Conference of the Biometrics Special Interest Group (BIOSIG)).
    @inproceedings{c188f1e4b1bd4247bd8045788944de74,
    title = "FEERCI: A Package for Fast Non-Parametric Confidence Intervals for Equal Error Rates in Amortized O(m log n)",
    abstract = "Equal Error Rates (EERs), or other weighted relations between False Match and Non-Match Rates (FMR/FNMR), are often used as a performance metric for biometric systems. Confidence Intervals (CIs) are used to denote the uncertainty underlying these EERs, with many methods existing to estimate said CIs in both parametric and non-parametric ways. These confidence intervals provide, foremost, a method of comparing scoring/ranking functions. Non-parametric methods often suffer from high computational costs, but do not make assumptions as to the shape of the EER-and score distributions. For both EERs and CIs, contemporary open-source toolkits leave room for improvement in terms of computational efficiency. In this paper, we introduce the Fast EER (FEER) algorithm that calculates an EER in O(log n) on a sorted score list, we show how to adapt the FEER algorithm to calculate non-parametric, bootstrapped EER CIs (FEERCI) in O(m log n) given m resamplings, and we introduce an opinionated open-source package named feerci that provides implementations of the FEER and FEERCI algorithm. We provide speed and accuracy benchmarks for the feerci package, comparing it against the most-used methods of calculating EERs in Python and show how it is able to calculate EERs and CIs on very large score lists faster than contemporary toolkits can calculate a single EER.",
    keywords = "Bootstrap Confidence Interval, Equal Error Rate, Open Source, Receiver operating characteristic",
    author = "Erwin Haasnoot and Ali Khodabakhsh and Chris Zeinstra and Luuk Spreeuwers and Raymond Veldhuis",
    year = "2018",
    month = "10",
    day = "10",
    doi = "10.23919/BIOSIG.2018.8553607",
    language = "English",
    isbn = "978-1-5386-6007-2",
    series = "International Conference of the Biometrics Special Interest Group (BIOSIG)",
    publisher = "IEEE",
    editor = "Arslan Bromme and Andreas Uhl and Christoph Busch and Christian Rathgeb and Antitza Dantcheva",
    booktitle = "2018 International Conference of the Biometrics Special Interest Group, BIOSIG 2018",
    address = "United States",

    }

    Haasnoot, E, Khodabakhsh, A, Zeinstra, C, Spreeuwers, L & Veldhuis, R 2018, FEERCI: A Package for Fast Non-Parametric Confidence Intervals for Equal Error Rates in Amortized O(m log n). in A Bromme, A Uhl, C Busch, C Rathgeb & A Dantcheva (eds), 2018 International Conference of the Biometrics Special Interest Group, BIOSIG 2018., 8553607, International Conference of the Biometrics Special Interest Group (BIOSIG), vol. 2018, IEEE, Piscataway, NJ, 17th International Conference of the Biometrics Special Interest Group, BIOSIG 2018, Darmstadt, Germany, 26/09/18. https://doi.org/10.23919/BIOSIG.2018.8553607

    FEERCI : A Package for Fast Non-Parametric Confidence Intervals for Equal Error Rates in Amortized O(m log n). / Haasnoot, Erwin; Khodabakhsh, Ali; Zeinstra, Chris; Spreeuwers, Luuk; Veldhuis, Raymond.

    2018 International Conference of the Biometrics Special Interest Group, BIOSIG 2018. ed. / Arslan Bromme; Andreas Uhl; Christoph Busch; Christian Rathgeb; Antitza Dantcheva. Piscataway, NJ : IEEE, 2018. 8553607 (International Conference of the Biometrics Special Interest Group (BIOSIG); Vol. 2018).

    Research output: Chapter in Book/Report/Conference proceedingConference contributionAcademicpeer-review

    TY - GEN

    T1 - FEERCI

    T2 - A Package for Fast Non-Parametric Confidence Intervals for Equal Error Rates in Amortized O(m log n)

    AU - Haasnoot, Erwin

    AU - Khodabakhsh, Ali

    AU - Zeinstra, Chris

    AU - Spreeuwers, Luuk

    AU - Veldhuis, Raymond

    PY - 2018/10/10

    Y1 - 2018/10/10

    N2 - Equal Error Rates (EERs), or other weighted relations between False Match and Non-Match Rates (FMR/FNMR), are often used as a performance metric for biometric systems. Confidence Intervals (CIs) are used to denote the uncertainty underlying these EERs, with many methods existing to estimate said CIs in both parametric and non-parametric ways. These confidence intervals provide, foremost, a method of comparing scoring/ranking functions. Non-parametric methods often suffer from high computational costs, but do not make assumptions as to the shape of the EER-and score distributions. For both EERs and CIs, contemporary open-source toolkits leave room for improvement in terms of computational efficiency. In this paper, we introduce the Fast EER (FEER) algorithm that calculates an EER in O(log n) on a sorted score list, we show how to adapt the FEER algorithm to calculate non-parametric, bootstrapped EER CIs (FEERCI) in O(m log n) given m resamplings, and we introduce an opinionated open-source package named feerci that provides implementations of the FEER and FEERCI algorithm. We provide speed and accuracy benchmarks for the feerci package, comparing it against the most-used methods of calculating EERs in Python and show how it is able to calculate EERs and CIs on very large score lists faster than contemporary toolkits can calculate a single EER.

    AB - Equal Error Rates (EERs), or other weighted relations between False Match and Non-Match Rates (FMR/FNMR), are often used as a performance metric for biometric systems. Confidence Intervals (CIs) are used to denote the uncertainty underlying these EERs, with many methods existing to estimate said CIs in both parametric and non-parametric ways. These confidence intervals provide, foremost, a method of comparing scoring/ranking functions. Non-parametric methods often suffer from high computational costs, but do not make assumptions as to the shape of the EER-and score distributions. For both EERs and CIs, contemporary open-source toolkits leave room for improvement in terms of computational efficiency. In this paper, we introduce the Fast EER (FEER) algorithm that calculates an EER in O(log n) on a sorted score list, we show how to adapt the FEER algorithm to calculate non-parametric, bootstrapped EER CIs (FEERCI) in O(m log n) given m resamplings, and we introduce an opinionated open-source package named feerci that provides implementations of the FEER and FEERCI algorithm. We provide speed and accuracy benchmarks for the feerci package, comparing it against the most-used methods of calculating EERs in Python and show how it is able to calculate EERs and CIs on very large score lists faster than contemporary toolkits can calculate a single EER.

    KW - Bootstrap Confidence Interval

    KW - Equal Error Rate

    KW - Open Source

    KW - Receiver operating characteristic

    UR - http://www.scopus.com/inward/record.url?scp=85060007289&partnerID=8YFLogxK

    U2 - 10.23919/BIOSIG.2018.8553607

    DO - 10.23919/BIOSIG.2018.8553607

    M3 - Conference contribution

    AN - SCOPUS:85060007289

    SN - 978-1-5386-6007-2

    T3 - International Conference of the Biometrics Special Interest Group (BIOSIG)

    BT - 2018 International Conference of the Biometrics Special Interest Group, BIOSIG 2018

    A2 - Bromme, Arslan

    A2 - Uhl, Andreas

    A2 - Busch, Christoph

    A2 - Rathgeb, Christian

    A2 - Dantcheva, Antitza

    PB - IEEE

    CY - Piscataway, NJ

    ER -

    Haasnoot E, Khodabakhsh A, Zeinstra C, Spreeuwers L, Veldhuis R. FEERCI: A Package for Fast Non-Parametric Confidence Intervals for Equal Error Rates in Amortized O(m log n). In Bromme A, Uhl A, Busch C, Rathgeb C, Dantcheva A, editors, 2018 International Conference of the Biometrics Special Interest Group, BIOSIG 2018. Piscataway, NJ: IEEE. 2018. 8553607. (International Conference of the Biometrics Special Interest Group (BIOSIG)). https://doi.org/10.23919/BIOSIG.2018.8553607