Multi-objective Ranking using Bootstrap Resampling

Benchmarking and related competitions are widely used for assessing and comparing solver performances. However, underlying uncertainty due to the composition of the instance set and bias induced by choosing specific performance criteria is often not sufficiently addressed. Moreover, performance assessment is almost always multi-objective in nature and no objective, totally neutral approach to it exists. We build on recent work of robust ranking for single-objective solver performance assessment based on bootstrap resampling and introduce a multi-objective robust ranking extension shown to provide new and promising perspectives onto existing competition rankings.