Projects

Comparison of Different State-of-the-Art Reinforcement Learning Methods in a 2D Simulation Environment

Osama Salem

B.Sc. thesis, Real-Time Systems Group, Institute of Systems Engineering, Leibniz University Hannover ·

Examiner Prof. Dr.-Ing. Bernardo Wagner

Supervisor Julian Kunze

Top-down floor plan of the search-and-rescue environment: the UAV is a blue circle and the target objects are yellow dots inside the building.
Search-and-rescue environment. The UAV (blue) must find the target objects (yellow) while avoiding hazards and walls.

TL;DR

Problem
After a disaster, a UAV could search damaged buildings for victims. With only laser scans and IMU readings — no knowledge of its own position or of the targets — this is a sparse-reward, partially observable exploration problem.
Approach
In a GPU-based 2D simulation (Megastep) with vectorized Gym wrappers, I compared PPO, A2C, DQN, SAC, Rainbow and DreamerV3 — extended with RND/ICM exploration, recurrent GRU policies, an attention-augmented laser encoder and transfer learning — and benchmarked them against a human operator.
Contribution
PPO + RND (mean reward 13.33) and A2C with transfer learning (14.66) solved the task, beating the human operator (9.74) and needing far fewer steps per found object (38.3 and 45.7 vs 91.1). The other agents — A2C + RND, DQN, SAC, Rainbow and DreamerV3 — did not solve it (negative mean reward).