Presence of coronary artery calcium (CAC) is a strong and independent predictor of cardiovascular events. We present a system using a forest of extremely randomized trees to automatically identify and quantify CAC in routinely acquired cardiac non-contrast enhanced CT. Candidate lesions the system could not label with high certainty were automatically identified and presented to an expert who could relabel them to achieve high scoring accuracy with minimal effort. The study included 200 consecutive non-contrast enhanced ECG-triggered cardiac CTs (120 kV, 55 mAs, 3 mm section thickness). Expert CAC annotations made as part of the clinical routine served as the reference standard. CAC candidates were extracted by thresholding (130 HU) and 3-D connected component analysis. They were described by shape, intensity and spatial features calculated using multi-atlas segmentation of coronary artery centerlines from ten CTA scans. CAC was identified using a randomized decision tree ensemble classifier in a ten-fold stratified cross-validation experiment and quantified in Agatston and volume scores for each patient. After classification, candidates with posterior probability indicating uncertain labeling were selected for further assessment by an expert. Images with metal implants were excluded. In the remaining 164 images, Spearman's p between automatic and reference scores was 0.94 for both Agatston and volume scores. On average 1.8 candidate lesions per scan were subsequently presented to an expert. After correction, Spearman's p was 0.98. We have described a system for automatic CAC scoring in cardiac CT images which is able to effectively select difficult examinations for further refinement by an expert.