Over-optimism in bioinformatics: an illustration - CentraleSupélec Access content directly
Journal Articles Bioinformatics Year : 2010

Over-optimism in bioinformatics: an illustration


In statistical bioinformatics research, different optimization mechanisms potentially lead to “over-optimism” in published papers. The present empirical study illustrates these mechanisms through a concrete example from an active research field. The investigated sources of over-optimism include the optimization of the data sets, of the settings, of the competing methods and, most importantly, of the method's characteristics. We consider a “promising” new classification algorithm that turns out to yield disappointing results in terms of error rate, namely linear discriminant analysis incorporating prior knowledge on gene functional groups through an appropriate shrinkage of the within-group covariance matrix. We quantitatively demonstrate that this disappointing method can artificially seem superior to existing approaches if we “fish for significance”. We conclude that, if the improvement of a quantitative criterion such as the error rate is the main contribution of a paper, the superiority of new algorithms should be validated using “fresh” validation data sets. The R codes and preprocessed versions of the data sets as well as additional files can be downloaded from http://www.ibe.med.uni-muenchen.de/organisation/mitarbeiter/020−professuren/boulesteix/overoptimism/,such that the study is completely reproducible.
Fichier principal
Vignette du fichier
Jelizarow2010.pdf (180.62 Ko) Télécharger le fichier
Origin : Files produced by the author(s)

Dates and versions

hal-00514107 , version 1 (01-09-2010)


  • HAL Id : hal-00514107 , version 1


Monika Jelizarow, Vincent Guillemot, Arthur Tenenhaus, K. Strimmer, Anne-Laure Boulesteix. Over-optimism in bioinformatics: an illustration. Bioinformatics, 2010, 26, pp.1990-1998. ⟨hal-00514107⟩
112 View
305 Download


Gmail Facebook X LinkedIn More