Appropriate land use planning plays an active role in soil and water conservation. Considering the financial investment and grain security limitations, land use planning practices should be applied focusing on the critical source areas (CSAs) rather than the complete region. In this study, the Soil and Water Assessment Tool (SWAT) model was used to identify CSAs for sediment control in the upper Huaihe River basin. A novel one‐at‐a‐time removal approach was proposed which involves individually removing the land use planning programme at sub‐basin level, after which the related runoff and sediment results were simulated by the validated SWAT model. Then, two influencing indexes (i.e., runoff yield and sediment yield) were used to evaluate the influences on runoff and sediment responses at the watershed outlet, caused by the removal of that land use planning programme. Finally, critical sub‐basins for land use planning were prioritized based on the weighted average of those influencing indexes. Calibration and validation results showed that the SWAT model well represented the monthly streamflow and sediment at the watershed outlet. The Nash–Sutcliffe efficiency values of monthly streamflow and sediment were generally greater than 0.75. Results of the integrated priorities revealed that the most CSAs affecting land use planning implementation were in sub‐basin 11, which was not the sub‐basin yielding the most sediment. The CSAs prioritization determined by the one‐at‐a‐time removal approach can achieve a better erosion control performance than that identified by the simulated sediment loads, which can be applied widely in targeting CSAs for implementation of watershed conservation practices with limited investment and disturbance.