Boosted Bellman Residual Minimization Handling Expert Demonstrations - CentraleSupélec Access content directly
Conference Papers Year : 2014

Boosted Bellman Residual Minimization Handling Expert Demonstrations


This paper addresses the problem of batch Reinforcement Learning with Expert Demonstrations (RLED). In RLED, the goal is to find an optimal policy of a Markov Decision Process (MDP), using a data set of fixed sampled transitions of the MDP as well as a data set of fixed expert demonstrations. This is slightly different from the batch Reinforcement Learning (RL) framework where only fixed sampled transitions of the MDP are available. Thus, the aim of this article is to propose algorithms that leverage those expert data. The idea proposed here differs from the Approximate Dynamic Programming methods in the sense that we minimize the Optimal Bellman Residual (OBR), where the minimization is guided by constraints defined by the expert demonstrations. This choice is motivated by the the fact that controlling the OBR implies controlling the distance between the estimated and optimal quality functions. However, this method presents some difficulties as the criterion to minimize is non-convex, non-differentiable and biased. Those difficulties are overcome via the embedding of distributions in a Reproducing Kernel Hilbert Space (RKHS) and a boosting technique which allows obtaining non-parametric algorithms. Finally, our algorithms are compared to the only state of the art algorithm, Approximate Policy Iteration with Demonstrations (APID) algorithm, in different experimental settings.
Fichier principal
Vignette du fichier
supelec885.pdf (481.12 Ko) Télécharger le fichier
Origin : Files produced by the author(s)

Dates and versions

hal-01060953 , version 1 (04-09-2014)



Bilal Piot, Matthieu Geist, Olivier Pietquin. Boosted Bellman Residual Minimization Handling Expert Demonstrations. European Conference, ECML PKDD 2014, Sep 2014, Nancy, France. pp.549-564, ⟨10.1007/978-3-662-44851-9_35⟩. ⟨hal-01060953⟩
524 View
709 Download



Gmail Facebook X LinkedIn More