The OHBM Reproducibility Challenge
Overview
The OHBM-OSSIG Reproducibility Challenge is a networking experience facilitated by the Open Science SIG, aiming at improving the reproducibility of scientific results. Participants join forces in teams of two parties, a “source party” and a “reproducing party”. The source party provides a submitted OHBM abstract (“source work”) for reproduction, as well as access to procedures, codes, and, when possible, the data. To be eligible, the source work must be submitted as an abstract to OHBM 2026; related publications can be used only if they directly correspond to this submitted abstract. The reproducing party reproduces the original work using the available procedures, code, and data. Teams may hold one brief introduction meeting to clarify scope and logistics, but no further consultation with the source party is allowed.
Types of reproduction in this challenge
In this challenge, we distinguish between two complementary types of reproduction:
- Reproducibility: running the same analysis on the same data
- Replicability: running the same analysis on independent data
The specific type of reproduction will be determined by the availability of the original data. In all cases, a mandatory requirement is that teams make their code openly available to enable others to re-run the analysis.
Adapted from The Turing Way Community. Original illustration created by Scriberia with The Turing Way community, used under a CC-BY 4.0 licence. DOI: 10.5281/zenodo.33328071
How to participate?
To join the OHBM Reproducibility Challenge, both source and reproducing parties register by opening an issue on our GitHub page and selecting their role in the template. Source parties are asked to submit: author names and contact details, link to Standard Operating Procedure (SOP) or documentation, link to code, data description (add links to the data if available), data sharing possibilities, link to abstract, preprint, or paper. Reproducing parties simply register their interest and expertise. The OSSIG will support team formation by matching source and reproducing parties based on expertise and interest. Participants do not need to arrive with a pre-formed team.
During the OHBM annual meeting
Submitted abstracts will be listed on the dedicated GitHub page, and presenters will be invited to include a Reproducibility Challenge logo with a QR code linking to the challenge website when printing their posters for OHBM 2026. Abstracts that were not initially submitted for the challenge can still participate if their authors give permission and submit a GitHub issue with the required information. These entries follow the same procedure as regular submissions, and the OSSIG will attempt to match them with reproducing parties shortly after the OHBM 2026 meeting.
Timeline and expected outcomes
Teams will be formed and confirmed during or shortly after the OHBM 2026 meeting. Once teams are established, they can begin reproducing or replicating the source work. Teams are then invited to collaboratively submit an abstract describing their results for OHBM 2027. This abstract should be submitted both to the official OHBM abstract portal and to the OSSIG (details will follow), ensuring that teams can also be invited to present their work in the Open Science Room. The OSSIG will host a dedicated session during the 2027 meeting, showcasing all participating teams and reflecting on the challenges encountered in reproducing and replicating the original work.
The outcome of each team’s effort is evaluated not based on the results themselves, but rather on:
- how closely the replication matched the original work,
- the transparency and openness of the source work, and
- any additional outputs produced by the team, such as software or publications.
Importantly, “unsuccessful” attempts to reproduce the source work do not reflect negatively; what matters most is what is learned during the process of reproduction and replication.
Background and inspiration
This challenge is highly inspired by the work of the Reproducible Research Study Group (RRSG) of ISMRM. You can read more about it in the related publication or on the original challenge website (note that it is a different challenge).