With the increasing availability of ubiquitous and intelligent mobile devices, MOOC (Massive Open Online Courses) has become a popular choice for people who want to learn in a more flexible manner. However, compared to traditional in-class face-to-face learning, the MOOC platforms always suffer from a high learner dropout rate. Hence, correctly predicting the dropout rate at the early stage of a learning activity is significant for the MOOC adaptors and developers, who can conduct effective intervention strategies to improve the online course’s quality and increase the retention rate. In this paper, we designed a double-tower-based framework for dropout rate prediction. The framework separately models the different types of information, namely the macro and the micro information. Our work also leads to the design of a Convolutional Neural Network (CNN)-based model for effectively mining time-series information from the learners’ successive activity records. The experimental results demonstrated that the proposed double-tower-based framework also clearly outperformed the various baselines.