The OHBM Reproducibility Challenge
The OHBM-OSSIG Reproducibility Challenge is a networking experience facilitated by the OSSIG itself, aiming at improving both the reproduction and the reproducibility of scientific results. Participants join forces in teams of two parties, a “source party” and a “reproducing party”. The source party proffers a submitted OHBM abstract or related publication (“source work”) for replication, as well as access to procedures, codes, and, when possible, at least part of the data. The reproducing party reproduces the original work using the available procedures, code, and data, but no extra consultation. While an element of reproduction must be present, additional elements of replication, reliability, or generalisation1 are appreciated.
Proffered abstracts, as well as the Challenge procedures, will be listed on this website, and can be made recognisable during the annual meeting through embeddable logos and stickers. Abstract that were not proffered for the challenge can also be reproduced, provided that their authors agree. The OSSIG can facilitate making the first contact and help form teams! Teams will have to register to the challenge with the OSSIG to officially partake in the challenge, by emailing ohbm.ossig.rc@gmail.com.
The outcome of the team effort, that must be a submitted OHBM abstract or a OHBM Brianhack project, is not evaluated only on results, but also on how close to the original work the replication managed to be, how open the source work was, as well as on possible secondary deliverables of the team (e.g., software, publications, …). In fact, “unsuccessful” attempts to reproduce the source work does not mean it is bad—what matters for the challenge is what is learnt on the journey to reproduction (and beyond). A team of ad-hoc reviewers will be formed by the OSSIG to review such submissions. The OSSIG will host a session in the Open Science Room, featuring all participants, reflecting on the challenges in the reproduction of the original work.
This challenge is highly inspired by the work of the Reproducible Research Study Group (RRSG) of ISMRM. You can read more about it in the related publication or on the original challenge website (note that it is a different challenge).
The Turing Way Community. This illustration is created by Scriberia with The Turing Way community, used under a CC-BY 4.0 licence. DOI: 10.5281/zenodo.3332807