Abstract
Designing agents that are both adaptive and trustworthy is a long-standing problem at the intersection of symbolic AI and Machine Learning. In this position paper, we discuss several benefits of combining automated reasoning and reinforcement learning techniques to formally verify agents’ behavior in structured environments, both during and after the learning process. These are systems where agents have access to an explicit structure representing what they know about the world. Since we care both about the verifiability and the efficiency of the learning process, we argue why it is crucial to efficiently integrate complex structures in the learning algorithms themselves.