Introduction to Reinforcement Learning Reinforcement Learning Mathematical Formulation Structure of the Book Model-Free Policy Iteration Model-Free Policy Search Model-Based Reinforcement Learning MODEL-FREE POLICY ITERATION Policy Iteration with Value Function Approximation Value Functions State Value Functions State-Action Value Functions Least-Squares Policy Iteration Immediate-Reward Regression Algorithm Regularization Model Selection Remarks Basis Design for Value Function Approximation Gaussian Kernels on Graphs MDP-Induced Graph Ordinary Gaussian Kernels Geodesic Gaussian Kernels Extension to Continuous State Spaces Illustration Setup Geodesic Gaussian Kernels Ordinary Gaussian Kernels Graph-Laplacian Eigenbases Diffusion Wavelets Numerical Examples Robot-Arm Control Robot-Agent Navigation Remarks Sample Reuse in Policy Iteration Formulation Off-Policy Value Function Approximation Episodic Importance Weighting Per-Decision Importance Weighting Adaptive Per-Decision Importance Weighting Illustration Automatic Selection of Flattening Parameter Importance-Weighted Cross-Validation Illustration Sample-Reuse Policy Iteration Algorithm Illustration Numerical Examples Inverted Pendulum Mountain Car Remarks Active Learning in Policy Iteration Efficient Exploration with Active Learning Problem Setup Decomposition of Generalization Error Estimation of Generalization Error Designing Sampling Policies Illustration Active Policy Iteration Sample-Reuse Policy Iteration with Active Learning Illustration Numerical Examples Remarks Robust Policy Iteration Robustness and Reliability in Policy Iteration Robustness Reliability Least Absolute Policy Iteration Algorithm Illustration Properties Numerical Examples Possible Extensions Huber Loss Pinball Loss Deadzone-Linear Loss Chebyshev Approximation Conditional Value-At-Risk Remarks MODEL-FREE POLICY SEARCH Direct Policy Search by Gradient Ascent Formulation Gradient Approach Gradient Ascent Baseline Subtraction for Variance Model Selection Remarks Basis Design for Value Function Approximation Gaussian Kernels on Graphs MDP-Induced Graph Ordinary Gaussian Kernels Geodesic Gaussian Kernels Extension to Continuous State Spaces Illustration Setup Geodesic Gaussian Kernels Ordinary Gaussian Kernels Graph-Laplacian Eigenbases Diffusion Wavelets Numerical Examples Robot-Arm Control Robot-Agent Navigation Remarks Sample Reuse in Policy Iteration Formulation Off-Policy Value Function Approximation Episodic Importance Weighting Per-Decision Importance Weighting Adaptive Per-Decision Importance Weighting Illustration Automatic Selection of Flattening Parameter Importance-Weighted Cross-Validation Illustration Sample-Reuse Policy Iteration Algorithm Illustration Numerical Examples Inverted Pendulum Mountain Car Remarks Active Learning in Policy Iteration Efficient Exploration with Active Learning Problem Setup Decomposition of Generalization Error Estimation of Generalization Error Designing Sampling Policies Illustration Active Policy Iteration Sample-Reuse Policy Iteration with Active Learning Illustration Numerical Examples Remarks Robust Policy Iteration Robustness and Reliability in Policy Iteration Robustness Reliability Least Absolute Policy Iteration Algorithm Illustration Properties Numerical Examples Possible Extensions Huber Loss Pinball Loss Deadzone-Linear Loss Chebyshev Approximation Conditional Value-At-Risk Remarks MODEL-FREE POLICY SEARCH Direct Policy Search by Gradient Ascent Formulation Gradient Approach Gradient Ascent Baseline Subtraction for Variance; Robot-Agent Navigation Remarks Sample Reuse in Policy Iteration Formulation Off-Policy Value Function Approximation Episodic Importance Weighting Per-Decision Importance Weighting Adaptive Per-Decision Importance Weighting Illustration Automatic Selection of Flattening Parameter Importance-Weighted Cross-Validation Illustration Sample-Reuse Policy Iteration Algorithm Illustration Numerical Examples Inverted Pendulum Mountain Car Remarks Active Learning in Policy Iteration Efficient Exploration with Active Learning Problem Setup Decomposition of Generalization Error Estimation of Generalization Error Designing Sampling Policies Illustration Active Policy Iteration Sample-Reuse Policy Iteration with Active Learning Illustration Numerical Examples Remarks Robust Policy Iteration Robustness and Reliability in Policy Iteration Robustness Reliability Least Absolute Policy Iteration Algorithm Illustration Properties Numerical Examples Possible Extensions Huber Loss Pinball Loss Deadzone-Linear Loss Chebyshev Approximation Conditional Value-At-Risk Remarks MODEL-FREE POLICY SEARCH Direct Policy Search by Gradient Ascent Formulation Gradient Approach Gradient Ascent Baseline Subtraction for Variancemp;lt;BR> Mountain Car Remarks Active Learning in Policy Iteration Efficient Exploration with Active Learning Problem Setup Decomposition of Generalization Error Estimation of Generalization Error Designing Sampling Policies Illustration Active Policy Iteration Sample-Reuse Policy Iteration with Active Learning Illustration Numerical Examples Remarks Robust Policy Iteration Robustness and Reliability in Policy Iteration Robustness Reliability Least Absolute Policy Iteration &nb.