Continuously learning new tasks using high-level ideas or knowledge is a key capability of humans. In this paper, we propose lifelong reinforcement learning with sequential linear temporal logic formulas and reward machines (LSRM), which enables an agent to leverage previously learned knowledge to accelerate the learning of logically specified tasks. For a more flexible specification of tasks, we first introduce sequential linear temporal logic (SLTL), which is a supplement to the existing linear temporal logic (LTL) formal language. We then utilize reward machines (RMs) to exploit structural ...