Constrained Markov decision processes for response-adaptive procedures in clinical trials with binary outcomes

A constrained Markov decision process (CMDP) approach is developed to construct response-adaptive procedures in clinical trials with binary outcomes. The resulting CMDP class of Bayesian response-adaptive procedures can target a certain objective, such as maximizing expected treatment outcomes, while using constraints to control other operating characteristics such as the type I error rate. The constraints can be formulated under different priors, enforcing control of trial operating statistics given that the parameters lie in a specific part of the parameter space (e.g., following from a statistical hypothesis). A solution method is developed to find an optimal policy, while we propose a computationally efficient method based on a cutting plane algorithm that uses backward recursion at every iteration, yielding a feasible policy accompanied by a respective optimality gap. In three applications, we impose constraints on the Bayesian average (i.e., a priori expected) type I error rate, power, and mean-squared error, and a constraint on prior robustness. Comparison of the constructed CMDP procedures with the constrained randomized dynamic programming procedure shows a stronger frequentist type I error control and similar performance in the other operating characteristics when constraining the Bayesian average type I error rate, power, and mean-squared error, whereas CMDP additionally shows substantial outperformance in terms of expected treatment outcomes when only constraining the type I error rate and power. The third application shows that the CMDP approach provides a previously unexplored way to construct response-adaptive procedures with a desired degree of robustness to prior misspecification in terms of expected treatment outcomes.