Understanding the Influence of Hyperparameters on Text Embeddings for Text Classification Tasks

Nils Witt, Christin Seifert

    Research output: Chapter in Book/Report/Conference proceedingConference contributionAcademicpeer-review

    2 Citations (Scopus)

    Abstract

    Many applications in the natural language processing domain require the tuning of machine learning algorithms, which involves adaptation of hyperparameters. We perform experiments by systematically varying hyperparameter settings of text embedding algorithms to obtain insights about the influence and interrelation of hyperparameters on the model performance on a text classification task using text embedding features. For some parameters (e.g., size of the context window) we could not find an influence on the accuracy while others (e.g., dimensionality of the embeddings) strongly influence the results, but have a range where the results are nearly optimal. These insights are beneficial to researchers and practitioners in order to find sensible hyperparameter configurations for research projects based on text embeddings. This reduces the parameter search space and the amount of (manual and automatic) optimization time.
    Original languageEnglish
    Title of host publicationResearch and Advanced Technology for Digital Libraries
    Subtitle of host publication21st International Conference on Theory and Practice of Digital Libraries, TPDL 2017, Thessaloniki, Greece, September 18-21, 2017, Proceedings
    EditorsJaap Kamps, Giannis Tsakonas, Yannis Manolopoulos, Lazaros Iliadis, Ioannis Karydis
    PublisherSpringer
    Pages193-204
    ISBN (Electronic)978-3-319-67008-9
    ISBN (Print)978-3-319-67007-2
    DOIs
    Publication statusPublished - 2017
    Event21st International Conference on Theory and Practice of Digital Libraries 2017 - Grand Hotel Palace, Thessaloniki, Thessaloniki, Greece
    Duration: 18 Sep 201721 Sep 2017
    Conference number: 21
    http://www.tpdl.eu/tpdl2017/

    Publication series

    NameLecture Notes in Computer Science
    Volume10450

    Conference

    Conference21st International Conference on Theory and Practice of Digital Libraries 2017
    Abbreviated titleTPDL 2017
    CountryGreece
    CityThessaloniki
    Period18/09/1721/09/17
    Internet address

    Fingerprint

    Learning algorithms
    Learning systems
    Tuning
    Processing
    Experiments

    Cite this

    Witt, N., & Seifert, C. (2017). Understanding the Influence of Hyperparameters on Text Embeddings for Text Classification Tasks. In J. Kamps, G. Tsakonas, Y. Manolopoulos, L. Iliadis, & I. Karydis (Eds.), Research and Advanced Technology for Digital Libraries: 21st International Conference on Theory and Practice of Digital Libraries, TPDL 2017, Thessaloniki, Greece, September 18-21, 2017, Proceedings (pp. 193-204). (Lecture Notes in Computer Science; Vol. 10450). Springer. https://doi.org/10.1007/978-3-319-67008-9_16
    Witt, Nils ; Seifert, Christin. / Understanding the Influence of Hyperparameters on Text Embeddings for Text Classification Tasks. Research and Advanced Technology for Digital Libraries: 21st International Conference on Theory and Practice of Digital Libraries, TPDL 2017, Thessaloniki, Greece, September 18-21, 2017, Proceedings. editor / Jaap Kamps ; Giannis Tsakonas ; Yannis Manolopoulos ; Lazaros Iliadis ; Ioannis Karydis. Springer, 2017. pp. 193-204 (Lecture Notes in Computer Science).
    @inproceedings{a1b29ea9a3824c328c79c697d4a14a38,
    title = "Understanding the Influence of Hyperparameters on Text Embeddings for Text Classification Tasks",
    abstract = "Many applications in the natural language processing domain require the tuning of machine learning algorithms, which involves adaptation of hyperparameters. We perform experiments by systematically varying hyperparameter settings of text embedding algorithms to obtain insights about the influence and interrelation of hyperparameters on the model performance on a text classification task using text embedding features. For some parameters (e.g., size of the context window) we could not find an influence on the accuracy while others (e.g., dimensionality of the embeddings) strongly influence the results, but have a range where the results are nearly optimal. These insights are beneficial to researchers and practitioners in order to find sensible hyperparameter configurations for research projects based on text embeddings. This reduces the parameter search space and the amount of (manual and automatic) optimization time.",
    author = "Nils Witt and Christin Seifert",
    year = "2017",
    doi = "10.1007/978-3-319-67008-9_16",
    language = "English",
    isbn = "978-3-319-67007-2",
    series = "Lecture Notes in Computer Science",
    publisher = "Springer",
    pages = "193--204",
    editor = "Jaap Kamps and Giannis Tsakonas and Yannis Manolopoulos and Lazaros Iliadis and Ioannis Karydis",
    booktitle = "Research and Advanced Technology for Digital Libraries",

    }

    Witt, N & Seifert, C 2017, Understanding the Influence of Hyperparameters on Text Embeddings for Text Classification Tasks. in J Kamps, G Tsakonas, Y Manolopoulos, L Iliadis & I Karydis (eds), Research and Advanced Technology for Digital Libraries: 21st International Conference on Theory and Practice of Digital Libraries, TPDL 2017, Thessaloniki, Greece, September 18-21, 2017, Proceedings. Lecture Notes in Computer Science, vol. 10450, Springer, pp. 193-204, 21st International Conference on Theory and Practice of Digital Libraries 2017, Thessaloniki, Greece, 18/09/17. https://doi.org/10.1007/978-3-319-67008-9_16

    Understanding the Influence of Hyperparameters on Text Embeddings for Text Classification Tasks. / Witt, Nils; Seifert, Christin.

    Research and Advanced Technology for Digital Libraries: 21st International Conference on Theory and Practice of Digital Libraries, TPDL 2017, Thessaloniki, Greece, September 18-21, 2017, Proceedings. ed. / Jaap Kamps; Giannis Tsakonas; Yannis Manolopoulos; Lazaros Iliadis; Ioannis Karydis. Springer, 2017. p. 193-204 (Lecture Notes in Computer Science; Vol. 10450).

    Research output: Chapter in Book/Report/Conference proceedingConference contributionAcademicpeer-review

    TY - GEN

    T1 - Understanding the Influence of Hyperparameters on Text Embeddings for Text Classification Tasks

    AU - Witt, Nils

    AU - Seifert, Christin

    PY - 2017

    Y1 - 2017

    N2 - Many applications in the natural language processing domain require the tuning of machine learning algorithms, which involves adaptation of hyperparameters. We perform experiments by systematically varying hyperparameter settings of text embedding algorithms to obtain insights about the influence and interrelation of hyperparameters on the model performance on a text classification task using text embedding features. For some parameters (e.g., size of the context window) we could not find an influence on the accuracy while others (e.g., dimensionality of the embeddings) strongly influence the results, but have a range where the results are nearly optimal. These insights are beneficial to researchers and practitioners in order to find sensible hyperparameter configurations for research projects based on text embeddings. This reduces the parameter search space and the amount of (manual and automatic) optimization time.

    AB - Many applications in the natural language processing domain require the tuning of machine learning algorithms, which involves adaptation of hyperparameters. We perform experiments by systematically varying hyperparameter settings of text embedding algorithms to obtain insights about the influence and interrelation of hyperparameters on the model performance on a text classification task using text embedding features. For some parameters (e.g., size of the context window) we could not find an influence on the accuracy while others (e.g., dimensionality of the embeddings) strongly influence the results, but have a range where the results are nearly optimal. These insights are beneficial to researchers and practitioners in order to find sensible hyperparameter configurations for research projects based on text embeddings. This reduces the parameter search space and the amount of (manual and automatic) optimization time.

    U2 - 10.1007/978-3-319-67008-9_16

    DO - 10.1007/978-3-319-67008-9_16

    M3 - Conference contribution

    SN - 978-3-319-67007-2

    T3 - Lecture Notes in Computer Science

    SP - 193

    EP - 204

    BT - Research and Advanced Technology for Digital Libraries

    A2 - Kamps, Jaap

    A2 - Tsakonas, Giannis

    A2 - Manolopoulos, Yannis

    A2 - Iliadis, Lazaros

    A2 - Karydis, Ioannis

    PB - Springer

    ER -

    Witt N, Seifert C. Understanding the Influence of Hyperparameters on Text Embeddings for Text Classification Tasks. In Kamps J, Tsakonas G, Manolopoulos Y, Iliadis L, Karydis I, editors, Research and Advanced Technology for Digital Libraries: 21st International Conference on Theory and Practice of Digital Libraries, TPDL 2017, Thessaloniki, Greece, September 18-21, 2017, Proceedings. Springer. 2017. p. 193-204. (Lecture Notes in Computer Science). https://doi.org/10.1007/978-3-319-67008-9_16