Participation Guidelines

Important Dates

• Training data release: Available Now! (access granted upon registration)
• Registration closes: April 23, 2026
• Test data release: April 28, 2026
• Runs submission deadline: May 7, 2026
• Evaluation results out: May 19, 2026
• Participant and position paper submission deadline: May 28, 2026
• Notification of acceptance for participant and position papers: June 30, 2026
• Camera-ready participant papers submission: July 6, 2026
• GutBrainIE CLEF Workshop: September 21-24, 2026 during the CLEF Conference in Jena, Germany

Results

Leaderboards

Available after the systems submission deadline...

Datasets

Data Description

The data for this task is a set of titles and abstracts of biomedical articles retrieved from PubMed, focusing on the gut-brain interplay and its implications in neurological and mental health.

The datasets are organized into:

• Gold-Standard Annotations: High-quality annotations, expert-curated.
• Silver-Standard Annotations: Mid-quality annotations, created by trained students under expert supervision. The Silver-Standard collection is divided into:
  • Silver: Mid-quality annotations created by trained students under expert supervision. The students are organized into two clusters:
    • StudentA, including those with more consistent annotation performance,
    • StudentB, including those with less consistent annotation performance.
  • Silver 2025: Same as Silver but for the 2025 edition. Concept-level annotations were automatically generated.
• Bronze-Standard Annotations: Automatically generated annotations, using fine-tuned GLiNER for Named Entity Recognition (NER) and fine-tuned ATLOP for Relation Extraction (RE).

To foster the development of effective IE systems for the subtasks proposed within GutBrainIE, the provided datasets include:

• Entity Mentions: Text spans classified into predefined categories and linked to concept URIs from biomedical reference resources.
• Relations: Associations between entities, specifying that a particular relationship holds between two entities.

Collections

• Training and Development Data: Available upon registration. After registering, participants will receive an email with a download link (starting from February 2–9, 2026).
• Test Data: The test set will be available from April 28, 2026. This data will be used for final system evaluation.

Dataset Format

Annotations are provided in JSON format, for ease of use with NLP systems. Each entry in the dataset corresponds to a PubMed article, identified by its PubMed ID (PMID), and includes the following fields:

• Metadata: Article-related information, including title, author, journal, year, and abstract. It also includes the identifier of the annotator: expert annotators are labeled as expert_1 to expert_7, student annotators are grouped into two clusters, identified as student_A and student_B, and automatically generated annotations are labeled as distant. Participants may use this information to assign different weights to annotations based on who performed them.
• Entities: An array of objects where each object represents an annotated entity mention in the article, with the following attributes:
• ◦ start and end indices: Character offsets marking the span of the entity mention.
• ◦ location: Indicates if the entity mention is located in the title or in the abstract.
• ◦ text_span: The actual text span of the entity mention.
• ◦ label: The label assigned to the entity mention (e.g., bacteria, microbiome).
• ◦ uri: The concept URI to which the entity mention is linked, taken from one of the biomedical reference resources.
• Relations: An array of objects where each object represents an annotated relationship between two entity mentions in the article, with the following attributes:
• ◦ subject_start and subject_end indices: Character offsets marking the span of the subject entity mention.
• ◦ subject_location: Indicates if the subject entity mention is located in the title or in the abstract.
• ◦ subject_text_span: The actual text span of the subject entity mention.
• ◦ subject_uri: The concept URI to which the subject entity mention is linked, taken from one of the biomedical reference resources.
• ◦ subject_label: The label assigned to the subject entity mention (e.g., bacteria, microbiome).
• ◦ predicate: The label assigned to the relationship.
• ◦ object_start and object_end indices: Character offsets marking the span of the object entity mention.
• ◦ object_location: Indicates if the object entity mention is located in the title or in the abstract.
• ◦ object_text_span: The actual text span of the object entity mention.
• ◦ object_label: The label assigned to the object entity mention (e.g., bacteria, microbiome).
• ◦ object_uri: The concept URI to which the object entity mention is linked, taken from one of the biomedical reference resources.
• Mention-level Relations: Relations extracted from the Relations array, formatted as mention-based tuples of subject_text_span, subject_label, predicate, object_text_span, and object_label.
• Concept-level Relations: Relations extracted from the Relations array, formatted as concept-based tuples of subject_uri, subject_label, predicate, object_uri, and object_label.

Alternative Formats

For those more familiar with CSV or tabular formats, the dataset is also provided in these formats. In this case, each of the fields mentioned above is stored in a separate file:

• Metadata file
• Entities file
• Relations file
• Mention-level relations file
• Concept-level relations file

The CSV files use the pipe symbol (|) as a separator, while tabular files use the tab character (\t) for separation.

Entity and Relation Labels

The set of entities considered for annotations includes:

while the defined set of relations includes:

For more details on entity and relation labels, refer to the Annotation Guidelines available for download HERE.

Datasets Statistics

The table below provides an overview of the key statistics for each dataset collection:

Train Gold Collection Statistics

The tables below provide detailed statistics for entities and relations in the Train Gold collection.

The Ratio column represents the proportion of each entity or relation label relative to the total number of entities or relations in the collection.

Entity Statistics

Relation Statistics

Train Silver Collection Statistics

The tables below provide detailed statistics for entities and relations in the Train Silver collection.

Entity Statistics

Relation Statistics

Train Silver 2025 Collection Statistics

The tables below provide detailed statistics for entities and relations in the Train Silver 2025 collection.

Entity Statistics

Relation Statistics

Train Bronze Collection Statistics

The tables below provide detailed statistics for entities and relations in the Train Bronze collection.

Entity Statistics

Relation Statistics

Dev Collection Statistics

The tables below provide detailed statistics for entities and relations in the Dev collection.

Entity Statistics

Relation Statistics

Submitting

Participating teams should satisfy the following guidelines:

• The runs should be submitted in the format described below;
• Each group can submit a maximum of 25 runs for each subtask.

Submissions are handled via the BioASQ submission system linked below:

Available after test set release

Submission Guidelines

Participants are invited to submit results for any or all of the four subtasks (NER, NERD, M-RE, C-RE) independently.

In the following, a "run" refers to the predictions made by a single system on the test set.

All runs must be submitted in a single zipped file named "teamID_GutBrainIE_2026.zip". Within this zip archive, each run should be placed in a separate folder named "teamID_taskID_runID_systemDesc" without spaces or any special character, where:

• teamID is the identifier of your team choosen when you registered to CLEF 2026;
• taskID is the identifier of your task choosen, i.e., one of these tokens: T611 for task 6.1.1, T612 for task 6.1.2, T621 for task 6.2.1, and T622 for task 6.2.2 ;
• runID is the identifier of the run choosen by the participants, containing only letters and numbers (a-z, A-Z, 0-9)
• systemDesc is an optional (short) string that further describes your submission.

The content of each folder consists of two files:

• teamID_taskID_runID_systemDesc.json: A JSON file with predictions on the test set.
• teamID_taskID_runID_systemDesc.meta: A metadata briefly describing the approach, including:
• ◦ Team ID.
• ◦ Task ID.
• ◦ Run ID.
• ◦ Type of training applied.
• ◦ Pre-processing methods.
• ◦ Training data used.
• ◦ Relevant details of the run.
• ◦ A link to a GitHub repository enabling easy reproducibility of the run.

Please ensure that the zip archive you submit strictly adheres to this filenames structure. The team name must remain consistent across all submissions, and the system name shold reflect the approach used by the submitted run. The run number serves as a progressive identifier for multiple submissions of the same system. An example of a valid submission wil be available soon...

Submissions not adhering to these guidelines might be rejected. Please ensure accuracy and completeness of all submitted files to streamline evaluation.

An example of a valid submission can be downloaded from HERE. Please note that the predicted entities and relations included in this example are dummy data, provided solely to illustrate the correct structure and formatting of a valid submission folder.

To ensure your submission follows the required structure, you can use the validation script available HERE.

Example of Submission Format

Submissions should follow the same JSON format used in the provided datasets and must include only the field associated with the specific subtask for which the submission is made:

• Subtask 6.1.1 (NER): Include the entities field WITHOUT the uri subfield.
• Subtask 6.1.2 (NERD): Include the entities field WITH the uri subfield.
• Subtask 6.2.1 (Mention-level RE): Include the mention_level_relations field.
• Subtask 6.2.2 (Concept-level RE): Include the concept_level_relations field.

Each entry must correspond to the PubMed ID of the article from the test set being considered.

Find below samples of valid entries for each submission type:

Subtask 6.1.1 (NER) Submission Sample

					
{
	"34870091": {
		"entities": [
			{
				"start_idx": 75,
				"end_idx": 82,
				"location": "title",
				"text_span": "patients",
				"label": "human"
			},
			{
				"start_idx": 250,
				"end_idx": 270,
				"location": "abstract",
				"text_span": "intestinal microbiome",
				"label": "microbiome"
			}
		]
	}
}
				
				

Subtask 6.1.2 (NERD) Submission Sample

					
{
	"34870091": {
		"entities": [
			{
				"start_idx": 75,
				"end_idx": 82,
				"location": "title",
				"text_span": "patients",
				"label": "human",
				"uri": "http://id.nlm.nih.gov/mesh/D010361"
			},
			{
				"start_idx": 250,
				"end_idx": 270,
				"location": "abstract",
				"text_span": "intestinal microbiome",
				"label": "microbiome",
				"uri": "http://purl.obolibrary.org/obo/NCIT_C93019"
			}
		]
	}
}
				
				

Subtask 6.2.1 (Mention-Level RE) Submission Sample

				
{
	"34870091": {
		"mention_level_relations": [
			{
				"subject_text_span": "intestinal microbiome",
				"subject_label": "microbiome",
				"predicate": "located in",
				"object_text_span": "patients",
				"object_label": "human"
			}
		]
	}
}
				
				

Subtask 6.2.2 (Concept-Level RE) Submission Sample

				
{
	"34870091": {
		"concept_level_relations": [
			{
				"subject_uri": "http://purl.obolibrary.org/obo/NCIT_C93019",
				"subject_label": "microbiome",
				"predicate": "located in",
				"object_uri": "http://id.nlm.nih.gov/mesh/D010361",
				"object_label": "human"
			}
		]
	}
}
				
				

Validation Script

A validation script that can be used to verify the correctness of the submission format can be downloaded from HERE.

Participant Papers

Participants are expected to write a report describing their participation in GutBrainIE CLEF 2026, their proposed solution, what features have been used for prediction, the analysis of the experimental results and insights derived from them.

Participant reports will be peer-reviewed and accepted reports will be published in the CLEF 2026 Working Notes at CEUR-WS, indexed in DBLP and Scopus.

Participants are strongly encouraged to use LaTeX, for which a template is available for download HERE. The template follows the CEURART single-column style and not only defines the layout of the report but also provides a suggested structure and outlines the expected content for each section.

It is also possible to use ODT (LibreOffice) or DOCX (Microsoft Word) formats. In these cases, participants should download the official CEURART templates directly from the CEUR website HERE and ensure that the content and structure match those outlined in the provided LaTeX template.

Participant papers are expected to be 10-20 pages in length, excluding references.

The schedule for submission, revision, and camera ready of the participant report is detailed above in the Important Dates section.

Participant papers can be submitted via Easychair HERE.

Baselines & Evaluation

Official Baselines

The official baselines for the GutBrainIE CLEF 2026 challenge can be found and reproduced by accesing this GitHub repository.

The repository provides a complete implementation of the baseline pipeline, allowing participants to reproduce the results from scratch, test the pre-trained models, or evaluate the provided predictions (i.e., runs) of the baselines for all subtasks (NER, NERD, and RE). These runs can also be used by participants as references of valid submission formats.

The repository also contains the official evaluation script, which will be used to assess submitted runs after the system submission deadline. It is crucial that participants ensure their submitted runs are compatible with this script, as submissions that cannot be evaluated with it will be considered invalid. The evaluation script can also be used to obtain preliminary performance results on the development set.

The detailed results of the baselines on the development set can be found in the Results section.

Evaluation Metrics

Submitted runs will be evaluated using standard Information Extraction metrics to assess both per-label and overall system performance. The employed metrics are the same for all the subtasks (a detailed explanation of these can be found HERE):

In the equations below:

• \(TP\) (True Positives) refers to the number of correctly predicted entities or relations.
• \(FP\) (False Positives) refers to the number of wrongly predicted entities or relations.
• \(FN\) (False Negatives) refers to the number of entities or relations in the ground truth that were not predicted.
• \(\mathcal{L}\) is the set of labels referring to:
• ◦ For subtasks 6.1.1 and 6.1.2: entity labels.
• ◦ For subtasks 6.2.1 and 6.2.2: triples of (subject label, predicate, object label).

• Macro-average Precision: \( P_{\text{macro}_{\text{avg}}} = \frac{\sum_{l \in \mathcal{L}} \, \frac{TP_l}{TP_l \,+\, FP_l}}{|\mathcal{L}|} \)
• Macro-average Recall: \( R_{\text{macro}_{\text{avg}}} = \frac{\sum_{l \in \mathcal{L}} \, \frac{TP_l}{TP_l \,+\, FN_l}}{|\mathcal{L}|} \)
• Macro-average F1-score: \( F1_{\text{macro}_{\text{avg}}} = 2 \, \times \, \frac{P_{\text{macro}_{\text{avg}}} \; \times \; R_{\text{macro}_{\text{avg}}}}{P_{\text{macro}_{\text{avg}}} \; + \; R_{\text{macro}_{\text{avg}}}} \)
• Micro-average Precision: \( P_{\text{micro}_{\text{avg}}} = \frac{\sum_{l \in \mathcal{L}} \; TP_l}{\sum_{l \in \mathcal{L}} \; TP_l \; + \; FP_l} \)
• Micro-average Recall: \( R_{\text{micro}_{\text{avg}}} = \frac{\sum_{l \in \mathcal{L}} \; TP_l}{\sum_{l \in \mathcal{L}} \; TP_l \; + \; FN_l} \)
• Micro-average F1-score: \( F1_{\text{micro}_{\text{avg}}} = 2 \, \times \, \frac{P_{\text{micro}_{\text{avg}}} \; \times \; R_{\text{micro}_{\text{avg}}}}{P_{\text{micro}_{\text{avg}}} \; + \; R_{\text{micro}_{\text{avg}}}} \)

For each subtask, the reference metric for the final leaderboard will be the micro-average F1-score, as it better accounts for class imbalances. However, system rankings will also be provided for each of the metrics detailed above.

Baseline Results

The scores reported in the table below refer to the development set and represent the actual evaluation of the inference capabilities of the proposed baseline models across the four subtasks.

FAQs

Find answers to common questions about the GutBrainIE challenge, dataset, and submissions.

Coming soon...

If you need any additional information, please get in contact with us writing to:

• Nicola Ferro:      nicola.ferro @ unipd.it
• Gianmaria Silvello:      gianmaria.silvello @ unipd.it