As a high complexity and coupling system with multi-input and multi-output, diesel engine is difficult to describe accurately with accurate physical and chemical models. Training data sets were collected through space-filling design, and the BSFC, NO_x and CO prediction models were built by GBDT (Gradient Boosting Decision Tree) algorithm and further verified. The results show that the convergence velocity of prediction model is fast. The fitting degree R² of BSFC, NO_x and CO are 0.981, 0.993 and 0.992 respectively, and the average relative errors of predicted values are 0.81%, 3.68% and 2.95% respectively. The responses of BSFC, NO_x and CO generated by the model are consistent with the trends of real diesel engine. The prediction model has high accuracy and stability. The gradient lifting decision tree algorithm has high adaptability to diesel engine modeling, and can effectively solve the problems of multi-feature high-dimensional nonlinear diesel engine system, which provides an effective method for diesel engine performance prediction modeling.