Despite a long developmental history of water-related disaster risk indicators, there is still no consensus or reliable system for selecting objective data, no methodological system for choosing and verifying the relevancy of water-related disaster risk indicators, and no linking results back to root causes or addressing possible impacts on policies or actors to instigate change. Global policy documents such as the Sendai Framework for Disaster Risk Reduction (DRR) 2015–2013  emphasize the urgent need for indicators capable of measuring risk reduction. However, developing and determining risk indicators faces many issues. Most disaster risk indices published do not yet include a basic overview of what data was used and how it was collected, let alone provide a systematic explanation of why each indicator was included, and why others were not. This consequently complicates linking the findings to their potential policy impacts. It also complicates the providing of clear-cut recommendations for improving resilience, which is a common intent of disaster risk indices. This study, which focuses on water-related hazards, aims to provide disaster managers with a set of criteria for evaluating existing datasets used in disaster risk indices, index construction methods, and the links back to policy impacts. So far, there has been no comprehensive overview of indicator requirements or scoring systems. Previous studies concerning indicator evaluating metrics  have fewer metrics and have not yet addressed the different tiers of requirements, namely objective indicator data quality, methodological/epistemological aspects of index composition, and, most importantly, policy and actors of change (impact requirements). Further testing of these metrics in local studies can lead to the greatly needed scientific justification for indicator selection and can enhance index robustness. The results aid in developing an evaluation system to address issues of data availability and the comparability of commonly used indicator sources, such as the World Bank. Once indicators can be scientifically linked to impacts through policy devices, national governments or other actors can become more likely to claim ownership of the data management of indicators. Future studies should expand this evaluation system to other natural hazards and focus on investigating the links between indicators and DRR in order to further validate indicator selection robustly.