Identifying ELIXIR Core Data Resources

Life-cycle management of ELIXIR Services

This section outlines the framework and stages for life-cycle management of the ELIXIR Services (Table 1). This framework will be implemented through the ELIXIR-EXCELERATE Node Capacity Building and Communities of Practice and Training Programme work packages, strengthening the ELIXIR infrastructure by creating a pathway to excellence.

The agreed set of indicators for the ELIXIR Core Data Resources sets quality standards that guide and inform the managers of Emerging Services in the development of their Resource towards an ‘ELIXIR Service’ status.

Monitoring of usage trends and the scientific impact of the ELIXIR Services provides information to support their management, contributing to the maintenance of the ELIXIR Service status, or – where appropriate – leading a resource towards the Legacy stage.

Measuring the quality and impact of data resources: key indicators

In their report on the role of metrics in research assessment and management in the United Kingdom³, Wilsdon et al. highlight that the term ‘metric’ may be misunderstood. For example, the number of citations received by a publication is a citation metric, as it does not directly measure the impact of that researcher’s work.

They therefore suggest that the term ‘indicator’ is used in contexts in which there is the potential for confusion. An ‘indicator’ is defined as a measurable quantity that substitutes for something less readily measurable and is presumed to associate with it without directly measuring it. Citation counts could be used as indicators for the scientific impact of journal articles, even though scientific impacts can occur in ways that do not generate citations. We therefore use the term ‘indicators’ throughout.

Five categories of indicators, reflecting the multiple facets of data resources

Identification of the ELIXIR Core Data Resources involves a careful evaluation of the multiple facets of the data resources.

Indicators are grouped in five categories:

When collecting and interpreting indicators, it is important to articulate the methods used and, where possible, standardise terminology. This facilitates the understanding of the indicators and avoids misinterpretation across different Nodes.

(1) Scientific focus and quality of science

This includes the inherent scientific quality of the data and of the metadata, and its uniqueness and comprehensiveness. Also included are benchmarking against other resources, and whether the resource is an authority in its field.

A differentiation should be made between archival or deposition databases that receive and archive de novo data sets and well-structured metadata deposited by scientists, and added-value databases or knowledge bases, which are based on the archival data and add substantial value through expert curation, annotation of metadata, sophisticated data processing and/or data integration. The curation effort and outputs linked to a resource are an important measure of its quality.

(2) Community

This category reflects the size and the measured demand of the communities that are served by the resource: web statistics, user reach, and international use. The community that is served can be the depositors, as some resources are vital for deposition and/or the end-users. The community can be identified and measured in different ways, such as access to URLs, to download servers, and through APIs, and also through the citation of data and data resources in publications.

In addition, certain resources play a foundational role to derived services and data-driven research. Their data are distributed to many other resources and/or services that rely on their existence.

The scientific context in which the resource operates should be taken into account. A resource that serves a small scientific community may not have as many users as a resource serving a broader interest, and yet it may reach 90% of the community it supports (coverage) and be crucial for the scientific work of that community.

(3) Quality of service

Certain service levels and reliability can be quantified with specific technical indicators such as: the uptime of the resource; response times; availability and periodic application of meaningful and automated tests; user support and related training; use of community-recognised standards; diversity of data retrieval mechanisms; and other services. Usually, this requires a quality-assurance process during service development and operation. The Accelerating the ELIXIR Training Programme and the ELIXIR Training Platform will support resources delivering training, as well as provide good-practice guidelines and systems for evaluation.

(4) Legal and funding infrastructure, and governance

As stable research infrastructures, Core Data Resources can demonstrate that they have a sound legal, funding and governance structure.

A viable resource has a suitable legal framework (clear terms of use, licensing, data security, ethical compliance, etc.). Open data is a critical driver for life-sciences research and therefore for ELIXIR, but the policy for data access must be considered in view of resource funding. Longevity can be gauged through institutional support, funding schemes and the duration of financial stability. Core Data Resources will have demonstrated transition through different funding sources. A strong governance structure includes an international, independent Scientific Advisory Board (SAB), which allows community input and provides permanent oversight.

(5) Impact and translational stories

Impact evaluation attempts to provide a definitive answer to the question of whether the resource is meeting its objective of fulfilling a specific need of the scientific community. The translational stories relate to the role of the resource in accelerating science and are thus a very important indicator.

Impact evaluation attempts to assess whether the Resource is meeting its objective of fulfilling a specific need. In the UK, the HM Treasury’s Magenta Book⁴ provides guidelines for policy makers and analysts on how policies and projects should be assessed and reviewed. According to this guidance, the key characteristic of a good impact evaluation is that it recognises that most needs can be met by a range of elements, not just the project in question. To test the extent to which the Resource is responsible for meeting the need, it is necessary to estimate – usually on the basis of a statistical analysis of quantitative data – what would have happened if the Resource had not existed. This is known as the counterfactual. Establishing the counterfactual is not easy, since by definition it cannot be observed. A strong evaluation is successful in isolating the effect of the Resource from all other potential influences, thereby producing a good estimate of the counterfactual.

When communicating the impact of ELIXIR’s resources and their role in accelerating science to funders and the public, the indicators should be relevant to the audience. This can be done by presenting them within a context that is readily understandable.

Detailed description of the indicators and related methodology

A set of key indicators may be used to make a case for a Core Data Resource. Indicators aim to reflect the essence of the definition of an ELIXIR Core Data Resource and support the promotion of excellence in resource development and operation. Box 1 describes the indicators used in each category.

Box 2 presents a ‘Case Document’ template for describing a data resource using these indicators.

Indicators to support expert judgment

Taking into account that ‘Not everything that can be counted counts, and not everything that counts can be counted’ (William Bruce Cameron⁵), the indicators will be used to inform a peer-review process described below.

A carefully chosen set of qualitative and quantitative indicators, tailored to bioinformatics resources, will inform identification of the ELIXIR Core Data Resources. The indicators will support, but not supplant, expert judgment.

ELIXIR Core Data Resources should each have an international independent Scientific Advisory Board. Such boards are made up of distinguished academic and industry researchers and professionals who conduct scientific and/or technological review, ensuring quality and providing strategic advice to resource managers. Identification of ELIXIR Core Data Resources does not encroach on these governance structures. The establishment of Scientific Advisory Boards for Core Resources and Nodes is among the best practices that will be promoted by the Node Capacity Building and Communities of Practice.

Indicators can only be useful if they are underpinned by an open, transparent and coherent collection infrastructure, so clear methods of collection and processing are needed.

Examples of ELIXIR Core Data Resources

Using the definition of ELIXIR Core Data Resources above, we identified a ‘seed list’ of candidate core resources (Table 2) to inform Core Data Resource indicators.

Identifying ELIXIR Core Data Resources

Identification of ELIXIR Core Data Resources involves a careful evaluation of the multiple facets of the data resources. This paper describes the overall approach for the selection of Core Data Resources, which will evolve over the coming months as the principles described in this paper are into practice.

Indicators used are described in Box 1. The relevant ELIXIR Node submits the completed ‘Case Document’ (Box 2) to the ELIXIR Hub.

Only data resources that are part of an ELIXIR Node Application and/or Service Delivery Plan (in the case of EMBL-EBI, the ‘Work Programme’) can be candidate ELIXIR Core Data Resources.

Initial evaluation of the ELIXIR Core Data Resources takes place annually.

The ELIXIR Hub checks the Case Document for completeness and verifies whether the proposed Resource is included in the Node's Service Delivery Plan (or Work Programme). The ELIXIR Hub has an advisory role in selecting ELIXIR Core Data Resources. The Hub has no decision-making power and does not evaluate the proposals.

The ELIXIR Director informs the Heads of Nodes Committee of the candidate ELIXIR Core Data Resources. The Heads of Nodes Committee can request additional information about a candidate Resource from the relevant Head of Node.

The Heads of Nodes Committee convenes annually in person to review submitted Case Documents and determine the list of ELIXIR Core Data Resources. The initial selection is expected to grow with time.

Before the initial selection of ELIXIR Core Data Resources is confirmed, the ELIXIR Scientific Advisory Board will review the selection process. The ELIXIR Scientific Advisory Board also reviews the portfolio of ELIXIR Core Data Resources and provides ongoing advice on the process for their identification.

As each ELIXIR Core Data Resource already has a governance structure that includes an independent, international Board, this individual review is not duplicated by the ELIXIR Advisory Board. The outcome is presented to the ELIXIR (governance) Board for information and to ensure that the process has been correctly applied.

Through the work of the Nodes, Advisory Board and the ELIXIR Hub, standardized data on indicators can also be collected and monitored.

In collaboration with the Nodes, monitoring data will be automatically collected at the ELIXIR Hub on an ongoing basis and will be regularly transmitted to the Heads of Nodes. Nodes undertake to provide the necessary data to the specification defined.

Reviewing the ELIXIR Core Data Resources

ELIXIR Core Data Resources may be requested to report regularly on certain indicators, and to provide updates on any major changes.

The Heads of Nodes meeting will review all ELIXIR Core Data Resources every two to three years. However, a minimum of three Heads of Nodes may request an extraordinary evaluation of an individual resource, in particular, on the basis of the monitoring data. If the review raises issues concerning an ELIXIR Core Data Resource, the Heads of Nodes Committee is responsible for determining what action should be taken.

Supporting the sustainability of data resources and how defining identifying them will contribute to science policy

ELIXIR Core Data Resources form the centre of ELIXIR’s sustainability strategy. The collected key indicators for these bioinformatics resources, and more specifically the impact and translational stories, will be used to make a case to funders. This information will in turn help them to translate the impact that Core Data Resources make.

Impact evaluation attempts to provide a definitive answer to the question of whether the resource is meeting its objective of fulfilling a specific need of the scientific community. The translational stories relate to the role of the resource in accelerating science and are thus a very important indicator.

In addition, the ELIXIR Core Data Resources could contribute to impact and econometric analysis of life-science data within ELIXIR, as well as events focused on communicating the value of sustainable infrastructure for open data to the European Commission and other stakeholders.

Capacity building

Core Data Resources will act as flagships of excellence. The use of defined indicators, in particular those around user policies and procedures, will be useful as benchmarks of quality and will support capacity building within the ELIXIR Community.

For example, the ELIXIR Core Data Resources, especially the knowledge bases, can function as ’concept authorities’ within and beyond ELIXIR, having a clear role in standardising what the community understands by a given biological concept.

Certain additional indicators could be used outside of ELIXIR (e.g. uptime) to consolidate confidence across a wide range of stakeholders. This would require full transparency on how indicators are produced, so as to avoid misunderstanding or misuse.

Life-cycle management

Key indicators will inform life-cycle management, identifying trends and supporting decision-making around a given resource. This is important not only for the resource teams, but also for identifying Emerging Services that may evolve into ELIXIR Services. As new resources are listed in the ELIXIR Node Service Delivery Plans, indicators and capacity building around the Core Data Resources will support Emerging Services as they mature.

Underpinning actions to support long-term sustainability and integration with ELIXIR Services

ELIXIR Core Data Resources will be prioritised for technical actions and for training. ELIXIR Core Data Resources become the primary resources for ELIXIR Cloud, storage and data distribution efforts within the ELIXIR Nodes network. These actions will be important for supporting the evolution of Emerging Services associated with Core Data Resources.

ELIXIR will strive to add value to all ELIXIR resources, including ELIXIR Services, by supporting interactions of the Core Data Resources with one another and with ELIXIR Services and Emerging Services for the benefit of the larger user community. Examples of this are use-case driven enhancement of the interoperability of the ELIXIR Core Data Resources with one another and with other ELIXIR Services, supporting helpdesks to scale national operations, and implementation studies to explore links to national infrastructures and data services.