Evaluating computational methods with Omnibenchmark

Overview

Computational methods are central to modern molecular biology, yet selecting, evaluating, and justifying analytical approaches remains a major challenge, in particular due to the surge in the number of tools available. Benchmarking, in an ideal scenario, provides the evidence used to assess performance of computational methods across realistic scenarios, and identify suitable tools for specific tasks. Despite its importance, many benchmarks are difficult to reproduce, quickly become outdated, and require substantial duplicated effort across research groups.

This workshop introduces Omnibenchmark, an open framework for constructing standardized, transparent, and reproducible benchmarks (for computational biology). Participants will learn the key components of a rigorous benchmark, including reference datasets, methods, evaluation metrics, workflows, and software environments. Through a combination of short lectures and hands-on exercises, participants will explore existing Omnibenchmarks and extend them with new benchmark components. In addition, there will be an opportunity to collaborate in small groups to develop a benchmark for a computational task of interest with guidance from the trainers.

By emphasizing modularity, reproducibility, and community contribution, the workshop aims to equip researchers with practical skills for creating benchmarks that can be shared, scrutinized, extended, and reused by the broader computational biology community.

Audience

This course is intended for computational biologists, bioinformaticians, data scientists, software developers, and method developers who wish to improve the rigor and reproducibility of computational method evaluation.

Benchmarking is relevant to virtually every area of computational biology. Researchers interested in reproducible research practices, workflow development, method evaluation, or community standards will particularly benefit from the course.

Learning outcomes

At the end of the course, participants should be able to:

• Explain the purpose, strengths, and limitations of computational benchmarking.
• Identify the core components of a rigorous and reproducible benchmark, including datasets, methods, evaluation metrics, and workflows.
• Navigate and run an existing Omnibenchmark project.
• Add new benchmark components such as datasets, methods, or metrics modules to an existing benchmark.
• Add or modify benchmark components within standardized and reusable benchmark plans.
• Design a benchmark plan for a computational biology task relevant to their own research.
• Collaborate with others to develop and extend community-driven benchmarks.
• Apply principles of reproducibility, transparency, FAIR data management, and software portability to benchmarking workflows.
• Identify opportunities to contribute to or initiate benchmarking efforts within their research community.

Prerequisites

Knowledge / competencies

Prior experience in benchmarking is not required. However, participants should be comfortable working in either R or Python and have solid familiarity with command-line tools and software installation procedures. Familiarity with package management (e.g., conda or pip) is recommended. In case of doubt, evaluate your R skills here or Python skills here.

Because the workshop involves installing software, running workflows, and interacting with benchmark components, participants should also possess solid command-line skills in a Linux or Unix-like environment, including navigating directories, editing files, and executing commands from the shell. In case of doubt, evaluate your UNIX skills here.

Technical

Installation instructions will be provided to course participants in due time.

Schedule – CE(S)T time zone

The course will start at 9:00 CET and end around 17:00 CET. The project work will take place asynchronously, followed by an online session on March 1st between 9:00 and 12:00. The project work is voluntary but mandatory to receive a certificate of achievement with a recommendation of 1 ECTS (see below).

More details will follow in due time.

Application

The registration fees for academics are 200 CHF and 1000 CHF for for-profit companies.

While participants are registered on a first come, first served basis, exceptions may be made to ensure diversity and equity, which may increase the time before your registration is confirmed.

Applications will close on 18.01.2027 or as soon as the places will be filled up. Cancellation after 18.01.2027 will not be reimbursed. Please note that participation in SIB courses is subject to our general conditions.

You will be informed by email of your registration confirmation. Upon reception of the confirmation email, participants will be asked to confirm attendance by paying the fees within 5 working days.

Venue and Time

This in-person part of the course will take place in Zurich at Irchel campus (tba).

The course will start at 9:00 CET and end around 17:00 CET. The project work will take place asynchronously, followed by an online session on March 1st between 9:00 and 12:00. The project work is voluntary but mandatory to receive a certificate of achievement with a recommendation of 1 ECTS (see below).

Precise information will be provided to the registered participants in due time.

Additional information

Coordination: Geert van Geest, SIB Training group.

A Certificate of Attendance will be sent provided you were present during in-person part of the course, whereas a Certificate of Achievement recommending 1 ECTS will be sent provided you participated on the asynchronous project work and attended the online session on March 1st.

You are welcome to register to the SIB courses mailing list to be informed of all future courses and workshops, as well as all important deadlines using the form here.

SIB abides by the ELIXIR Code of Conduct. Participants of SIB courses are also required to abide by the same code.

For more information, please contact training@sib.swiss.