This work addresses the issue of spectrum sensing with random arrival and departure of primary signals. We first design a convolutional neural network (CNN) with outputs as the posterior probabilities of the arrival and departure of primary signals, leading to a CNN-based detector with the ratio of the posterior probabilities (i.e., the outputs of the CNN) as a test statistic. To further enhance the attention of the network on the switch feature of channel states, we design a switch attention module (SAM) that adaptively weights the received signals. Replacing the convolution plus maximum pooling block in the CNN detector with the SAM block leads to an SAM-CNN detector. Simulations show that the proposed CNN detector significantly outperforms existing detectors, and further improvement of detection probability by 19% is achieved by the SAM-CNN detector.