With the rapid development of the economy， financial services such as bills are also increasing day by day. Among them， the important information in the bill business， such as personal vouchers， checks， and other bills， requires manual reading and input of a large amount of digital information. In order to avoid the waste of human and financial resources， related researchers classify and recognize handwritten fonts based on the neural network classification idea of deep learning. When the method based on deep learning is used for feature extraction of handwritten fonts， there is often a lack of detailed information such as edges and textures， which leads to the problem of low recognition accuracy. Aiming at the problem that the features of handwritten digits or letters are difficult to effectively extract， the recognition efficiency was not high and even caused recognition errors， a new automatic recognition method of handwritten digits or letters was proposed by combining the principle of ghost imaging and the classification network based on deep learning. This method utilizes the principle of ghost imaging. It can save the imaging process in the traditional image recognition method， and jumping out of the inherent thinking that identifying objects is identifying images， and can quickly classify the image of handwritten digits and letters only by the total light intensity value transmitted by handwritten digits or letters without extracting and identifying features of handwritten digits or letters. The automatic recognition of handwritten digits or letters based on ghost imaging solves the critical problem of needing to extract digits or letter images features in traditional handwritten font recognition methods， and can greatly improve the recognition efficiency of handwritten digits or letters. Firstly， a ghost imaging detection system is built using commonly used optical instruments such as lasers， digital micromirror arrays， and single-pixel detectors. The laser in the built detection system is used to generate a pseudothermal light source， and the digital micromirror array is used to obtain the Hadamard speckle sequence with a resolution of 32×32 irradiating the target object at different times. And realizing the irradiation of 17 239 handwritten images of handwritten digits and letters. Secondly， the single-pixel detector is used to collect data on the total light intensity value transmitted by the handwritten digits and letters. The data collection process is very fast and does not cause huge time costs. The value of the bucket detector after the collection is converted