[Background] Dental caries is harmful to human health. Streptococcus mutans is recognized as the main cariogenic bacteria. In our previous studies, we found the Bacillus subtilis (Bacillus subtilis zh78) isolated from oral cavity could inhibit the growth of S. mutans. However, its bacteriostatic components were still ambiguous. [Objective] This study aimed to (1): determine the possible small molecular metabolite of B. subtilis zh78 that may inhibit the growth of S. mutans by untargeted metabolomics approach; (2) evaluate the prospect of B. subtilis zh78 in caries prevention and treatment. [Methods] Firstly, we used the cold methanol to extract the metabolites in the bacteria solutions from the B. subtilis zh78 growth at 0 hour, 7 hours, 12 hours and 5 days respectively. Oxford cup method was used to test the antibacterial activity of the extracted metabolites against S. mutans. Then, gas-chromatography-time of flight-mass spectrometry (GC-TOF-MS) was utilized to metabonomics detection. Lastly, we used principal component analysis (PCA), orthogonal partial least squares discriminant analysis (OPLS-DA) and Pearson correlation analysis to process the data. [Results] The metabolites at 5 days of B. subtilis zh78 showed the most obvious inhibitory effect on S. mutans. 36 kinds of substances, such as xylitol, amino acids and organic acids produced by B. subtilis zh78 were significantly related to its inhibitory effect on S. mutans. [Conclusion] The findings suggested that the B. subtilis zh78 could produce certain kinds of metabolites (e.g. xylitol, amino acids and organic acids) which might exert the antibacterial effects on S. mutans. The B. subtilis zh78 displayed its potential in oral probiotic application and the related studies.