Requirements engineering paper classification and evaluation criteria: A proposal and a discussion

In recent years, members of the steering committee of the IEEE Requirements Engineering (RE) Conference have discussed paper classification and evaluation criteria for RE papers. The immediate trigger for this discussion was our concern about differences in opinion that sometimes arise in program committees about the criteria to be used in evaluating papers. If program committee members do not all use the same criteria, or if they use criteria different from those used by authors, then papers might be rejected or accepted for the wrong reasons. Surely not all papers should be evaluated according to the same criteria. Some papers describe new techniques but do not report on empirical research; others describe new conceptual frameworks for investigating certain RE problems; others report on industrial experience with existing RE techniques. Other kinds of papers can also be easily recognized. All of these types of papers should be evaluated according to different criteria. But we are far from a consensus about what classes of paper we should distinguish, and what the criteria are for each of these classes.