In order to solve the problems of high calibration cost and long development cycle of vehicle gasoline engine, a calibration optimization method for vehicle gasoline engine was proposed to improve the calibration efficiency. Under steady state conditions of engine, Gaussian process regression model was applied to predict engine performance and emission parameters such as brake specific fuel consumption(BSFC) and nitrogen oxides(NO_x） through a small amount of test data, and a determination coefficient(R²) was introduced to evaluate the quality of model. Constraint violation was added to nondominated sorting and hybrid crossover operator was used to improve NSGA-Ⅱ. Taking the prediction model as the objective function of improved NSGA-Ⅱ, the calibration optimization of engine was carried out. An automatic generation method of smooth calibration map was employed, and the final calibration map was automatically selected according to the performance and smoothness of calibration map. The results show that the distribution of the improved NSGA-Ⅱ algorithm improves by 21.41% for the standard test function, and more optimal solutions of engine calibration are provided for decision makers after application. Combining the improved NSGA-Ⅱ algorithm with the automatic generation method of smooth calibration map, the calibration results show that the fuel consumption decreases by 1% and the nitrogen oxides decrease by 5% under the WLTC cycle condition compared with the manually-calibrated calibration map. It provides a reference for reducing the workload of engine calibration.