Noncompliant responding: Comparing exclusion criteria in MTurk personality research to improve data quality

Studies on Amazon Mechanical Turk (MTurk) often include check questions in personality inventories to ensure data quality. However, a subset of MTurk workers may give only meaningful responses to these checks while giving noncompliant responses to the other questions. We demonstrate in an analysis of five MTurk datasets using the statistical approach of Lee and Ashton (2018) that this selectively responsive subset can be detected on the HEXACO personality inventory. Our lower bound estimate is that at least 2% in each sample did not get caught with the check questions while giving noncompliant responses on the personality inventory. Overall, researchers who strive to remove noise due to noncompliant responding may benefit from complementing check questions with a statistical approach.