Construction of Embedded Markov Decision Processes for Optimal Control of Non-Linear Systems with Continuous State Spaces
| Citation: |
Nikovski, D.; Esenther, A., "Construction of Embedded Markov Decision Processes for Optimal Control of Non-Linear Systems with Continuous State Spaces", IEEE Conference on Decision and Control and European Control Conference (CDC-ECC), December 2011 (IEEE Xplore) |
| MERL Report: | TR2011-081 |
We consider the problem of constructing a suitable discrete-state approximation of an arbitrary non-linear dynamical system with continuous state space and discrete control actions that would allow close to optimal sequential control of that system by means of value or policy iteration on the approximated model. We propose a method for approximating the
continuous dynamics by means of an embedded Markov decision process (MDP) model defined over an arbitrary set of discrete states sampled from the original continuous state space. The mathematical similarity between sets of barycentric coordinates (convex combination) and probability mass functions is exploited to compute the transition matrices and initial state distribution
of the MDP. Barycentric coordinates are computed efficiently on a Delaunay triangulation of the set of discrete states, ensuring maximal accuracy of the approximation and the resulting control policy.
