Towards Scientific Benchmarks: On Increasing the Credibility of Benchmarks

Gundersen, Odd Erik

Towards Scientific Benchmarks: On Increasing the Credibility of Benchmarks

Files

083-086.pdf (158.58 KB)

Date

2015

Authors

Gundersen, Odd Erik

Publisher

The Eurographics Association

Abstract

Problem: Increasing the credibility of results from scientific benchmarks. Goal: Specify what exactly is required in order for a benchmark to be scientific. Contribution: (i) Specification of what it entails for a benchmark to be scientific, (ii) a metric for measuring the replicability of an experiment, (iii) a metric for measuring the replicability of a set of experiments, and (iv) Analysis of the replicability of SHREC 2015. Result: Replicability of SHREC 2015 can be increased by open sourcing the methods compared and improving documentation.

        @inproceedings{10.2312:3dor.20151059
,
booktitle = {Eurographics Workshop on 3D Object Retrieval
},
editor = {I. Pratikakis and M. Spagnuolo and T. Theoharis and L. Van Gool and R. Veltkamp
},
title = {{Towards Scientific Benchmarks: On Increasing the Credibility of Benchmarks
}},
author = {Gundersen, Odd Erik
},
year = {2015
},
publisher = {The Eurographics Association
},
ISBN = {},
DOI = {10.2312/3dor.20151059
}
}