A recipe for drone racing with reinforcement learning

This post is the third and final post in our series on quadcopter simulation. The first post and second post derived and simulated the quadcopter’s equations of motion, first in 2D and then in 3D. In this post we train a reinforcement learning policy to fly the quadcopter, first to hover at a fixed point, then to fly through a sequence of gates. The previous posts were tutorial-like, we started from a free body diagram and, step-by-step, arrived to simulation code. This post instead is more recipe-like, a collection of tricks and techniques I found useful to train an RL policy for quadcopter racing. There is no rigorous proof of why these methods work, only empirical. Because it is a recipe and not a walkthrough, the post shows only the code that matters for each decision. The full environment is on GitHub at mrandri19/quadcopter-racing . Here is the trained policy flying the eight-gate loop track. The visualizations throughout the post are made with Rerun . An incremental approach Trying to implement a drone racing simulation, RL environment, reward, and model all at once just doesn’t work. Too many things can be slightly off, affecting the policy’s performance, with no good way of debugging it. So we split the task in three stages, each building on top of each other: The “Hello, world! of RL”: training a policy to solve the inverted pendulum control problem with PPO and a vectorized stable-baselines3 env. Building a custom quadcopter RL environment using MuJoCo, choosing ac…