Interest in design methods and tools has been steadily growing in HRI. Yet, design is not acknowledged as a discipline with specific epistemology and methodology. Designerly HRI work is validated through user studies which, we argue, provide a limited account of the knowledge design produces. This paper aims to broaden current understanding of designerly HRI work and its contributions by unpacking what designerly knowledge is and how to produce it. Through a critical analysis of current HRI design literature, we identify a lack of work dedicated to understanding the conceptual implications of robotic artifacts. These, in fact, are implicit carriers of crucial HRI knowledge that can challenge established assumptions about how a robot should look, act, and be like. We conclude by discussing a set of practices desirable to legitimize designerly HRI work, and calling for further research addressing the conceptual implications designerly HRI work.