One-quarter of patients with refractory temporal lobe epilepsy cannot be effectively controlled by conventional anterior temporal lobectomy, which may be due to the existence of a variety of complex disease"s subtypes, and some subtypes involve brain lesions outside the temporal lobe. Based on the characteristics of multi-modal brain imaging, using Latent Dirichlet Allocation can estimate the potential disease factors and the probability that each patients" brain expresses the potential disease factors, which is significant for disease classification. 86 refractory temporal lobe epilepsy patients were recruited and were scanned with a hybrid PET/MR scanner to collect T1-weighted images and FDG PET images. 39 healthy volunteers with T1-weighted images and 36 healthy volunteers with FDG PET images were also recruited as controls. The gray matter volume and glucose uptake values were extracted to reflect the epilepsy pathology. Latent Dirichlet Allocation was used to perform the experiments, in which the number of disease factors was set as 3, 4 and 5 respectively for multiple experiments. Then a four-factor model was used to statistically analyze the relationship between disease factors and clinical indicators. It is found that the results obtained by selecting different number of disease factors for classification had certain similarity. In the four-factor model, there were disease factors that were significantly related to the laterality of lesions, disease duration and cognitive ability (P<0.05). This research has a reference value for the classification of temporal lobe epilepsy.