Handling Large Discrete Action Spaces via Dynamic Neighborhood Construction

Fabian Akkerman, Julius Luy, Wouter van Heeswijk, Maximilian Schiffer

Research output: Working paperPreprintAcademic

78 Downloads (Pure)

Abstract

Large discrete action spaces remain a central challenge for reinforcement learning methods. Such spaces are encountered in many real-world applications, e.g., recommender systems, multi-step planning, and inventory replenishment. The mapping of continuous proxies to discrete actions is a promising paradigm for handling large discrete action spaces. Existing continuous-to-discrete mapping approaches involve searching for discrete neighboring actions in a static pre-defined neighborhood, which requires discrete neighbor lookups across the entire action space. Hence, scalability issues persist. To mitigate this drawback, we propose a novel Dynamic Neighborhood Construction (DNC) method, which dynamically constructs a discrete neighborhood to map the continuous proxy, thus efficiently exploiting the underlying action space. We demonstrate the robustness of our method by benchmarking it against three state-of-the-art approaches designed for large discrete action spaces across three different environments. Our results show that DNC matches or outperforms state-of-the-art approaches while being more computationally efficient. Furthermore, our method scales to action spaces that so far remained computationally intractable for existing methodologies.
Original languageEnglish
PublisherArXiv.org
Publication statusPublished - 31 May 2023

Keywords

  • Reinforcement Learning
  • Large action spaces
  • Simulated Annealing

Fingerprint

Dive into the research topics of 'Handling Large Discrete Action Spaces via Dynamic Neighborhood Construction'. Together they form a unique fingerprint.

Cite this