We consider off-policy policy evaluation when the trajectory data are generated by multiple behavior policies. Recent work has shown the key role played by the state or state-action stationary distribution corrections in the infinite horizon context for off-policy policy evaluation. We propose estimated mixture policy (EMP), a novel class of partially policy-agnostic methods to accurately estimate those quantities. With careful analysis, we show that EMP gives rise to estimates with reduced variance for estimating the state stationary distribution correction while it also offers a useful induction bias for estimating the state-action stationary distribution correction. In extensive experiments with both continuous and discrete environments, we demonstrate that our algorithm offers significantly improved accuracy compared to the state-of-the-art methods.
A new partially policy-agnostic method for infinite-horizon off-policy policy evalution with multiple known or unknown behavior policies. An estimated mixture policy which takes ideas from off-policy policy evaluation infinite horizon estimators and regression importance sampling for importance weight, and extends them to many policies and unknown policies. An algorithm to solve infinite horizon off policy evaluation with multiple behavior policies by estimating a mixed policy under regression, and theoretical proof that an estimated policy ratio can reduce variance.