This letter proposes an associative hierarchical conditional random field (AHCRF) model to improve the classification accuracy of high-resolution remote sensing images. It considers segmentation quality of superpixels, avoids a time-consuming selection of optimal scale parameters, and alleviates the problem of classification accuracy sensitive to undersegmentation errors that is present in traditional object-oriented classification methods. The model is built on a graph hierarchy, including the pixel layer as a base layer and multiple superpixel layers derived from a mean shift presegmentation. It extracts clustered features of pixels for superpixels at each layer and then defines the potentials of the AHCRF model. We suggest a weighted version of the interlayer potential using the size of a superpixel as a measure to reflect segmentation quality. In this way, erroneously labeled pixels of a superpixel are penalized. Experiments are presented using a part of the downsampled Vaihingen data from the ISPRS benchmark data set. Results confirm that our model shows more than 80% overall classification accuracy and is superior to the original AHCRF model and comparable to other models. It also alleviates the choosing of suitable segmentation parameters.