Participation Guidelines

Important Dates

• Registration closes: April 25, 2025
• Test data release: April 28, 2025
• Runs submission deadline: May 10, 2025
• Evaluation results out: May 19, 2025
• Participant and position paper submission deadline: May 30, 2025
• Notification of acceptance for participant and position papers: June 27, 2025
• Camera-ready participant papers submission: July 7, 2025
• GutBrainIE CLEF Workshop: September 9-12, 2025 during the CLEF Conference in Madrid, Spain

Datasets

Data Description

The data for this task is a set of titles and abstracts of biomedical articles retrieved from PubMed, focusing on the gut-brain interplay and its implications in neurological and mental health.

The datasets are organized into:

• Platinum-Standard Annotations: Highest-quality annotations, expert-curated and reviewed by external biomedical specialists.
• Gold-Standard Annotations: High-quality annotations, expert-curated.
• Silver-Standard Annotations: Mid-quality annotations, created by trained students under expert supervision. The students are organized into two clusters:
  • StudentA, including those with more consistent annotation performance,
  • StudentB, including those with less consistent annotation performance.
• Bronze-Standard Annotations: Automatically generated annotations, using fine-tuned GLiNER for Named Entity Recognition (NER) and fine-tuned ATLOP for Relation Extraction (RE).

To foster the development of effective IE systems for the subtasks proposed within GutBrainIE, the provided datasets include:

• Entity Mentions: Text spans classified into predefined categories.
• Relations: Associations between entities, specifying that a particular relationship holds between two entities.

Collections

• Training and Development Data: Available upon registration. After registering, participants will receive an email with a download link.
• Test Data: The test set will be available from April 28, 2025. This data will be used for final system evaluation.

Dataset Format

Annotations are provided in JSON format, for ease of use with NLP systems. Each entry in the dataset corresponds to a PubMed article, identified by its PubMed ID (PMID), and includes the following fields:

• Metadata: Article-related information, including title, author, journal, year, and abstract. It also includes the identifier of the annotator: expert annotators are labeled as expert_1 to expert_7, student annotators are grouped into two clusters, identified as student_A and student_B, and automatically generated annotations are labeled as distant. Participants may use this information to assign different weights to annotations based on who performed them.
• Entities: An array of objects where each object represents an annotated entity mention in the article, with the following attributes:
• ◦ start and end indices: Character offsets marking the span of the entity mention.
• ◦ location: Indicates if the entity mention is located in the title or in the abstract.
• ◦ text_span: The actual text span of the entity mention.
• ◦ label: The label assigned to the entity mention (e.g., bacteria, microbiome).
• Relations: An array of objects where each object represents an annotated relationship between two entity mentions in the article, with the following attributes:
• ◦ subject_start and subject_end indices: Character offsets marking the span of the subject entity mention.
• ◦ subject_location: Indicates if the subject entity mention is located in the title or in the abstract.
• ◦ subject_text_span: The actual text span of the subject entity mention.
• ◦ subject_label: The label assigned to the subject entity mention (e.g., bacteria, microbiome).
• ◦ predicate: The label assigned to the relationship.
• ◦ object_start and object_end indices: Character offsets marking the span of the object entity mention.
• ◦ object_location: Indicates if the object entity mention is located in the title or in the abstract.
• ◦ object_text_span: The actual text span of the object entity mention.
• ◦ object_label: The label assigned to the object entity mention (e.g., bacteria, microbiome).
• Binary Tag-based Relations: Relations extracted from the Relations array, formatted as tag-based pairs of subject_label and object_label.
• Ternary Tag-based Relations: Relations extracted from the Relations array, formatted as tag-based triples of subject_label, predicate, and object_label.
• Ternary Mention-based Relations: Relations extracted from the Relations array, formatted as mention-based tuples of subject_text_span, subject_label, predicate, object_text_span, and object_label.

Alternative Formats

For those more familiar with CSV or tabular formats, the dataset is also provided in these formats. In this case, each of the fields mentioned above is stored in a separate file:

• Metadata file
• Entities file
• Relations file
• Binary Tag-based file
• Ternary Tag-based file
• Ternary Mention-based file

The CSV files use the pipe symbol (|) as a separator, while tabular files use the tab character (\t) for separation.

Entity and Relation Labels

The set of entities considered for annotations includes:

while the defined set of relations includes:

For more details on entity and relation labels, refer to the Annotation Guidelines available for download HERE.

Datasets Statistics

The table below provides an overview of the key statistics for each dataset collection:

Train Platinum Collection Statistics

The tables below provide detailed statistics for entities and relations in the Train Platinum collection.

The Ratio column represents the proportion of each entity or relation label relative to the total number of entities or relations in the collection.

Entity Statistics

Relation Statistics

Train Gold Collection Statistics

The tables below provide detailed statistics for entities and relations in the Train Gold collection.

Entity Statistics

Relation Statistics

Train Silver Collection Statistics

The tables below provide detailed statistics for entities and relations in the Train Silver collection.

Entity Statistics

Relation Statistics

Train Bronze Collection Statistics

The tables below provide detailed statistics for entities and relations in the Train Bronze collection.

Entity Statistics

Relation Statistics

Dev Collection Statistics

The tables below provide detailed statistics for entities and relations in the Dev collection.

Entity Statistics

Relation Statistics

Submitting

Participating teams should satisfy the following guidelines:

• The runs should be submitted in the format described below;
• Each group can submit a maximum of 25 runs for each subtask.

Submission Guidelines

Participants are invited to submit results for any or all of the four subtasks (NER, Binary RE, Tag-based Ternary RE, Mention-based Ternary RE) independently.

Submissions will be handled via the BioASQ submission system. More detailed information coming soon...

In the following, a "run" refers to the predictions made by a single system on the test set. Each run must be submitted as a single zipped file named "teamID_taskID_runID_systemDesc.zip" without spaces or any special character, where:

• teamID is the identifier of your team choosen when you registered to CLEF 2025;
• taskID is the identifier of your task choosen, i.e., one of these tokens: T61 for task 6.1, T621 for task 6.2.1, T622 for task 6.2.2, and T623 for task 6.2.3 ;
• runID is the identifier of the run choosen by the participants, containing only letters and numbers (a-z, A-Z, 0-9)
• systemDesc is an optional (short) string that further describes your submission.

The content of the zipped file consists of two files:

• teamID_taskID_runID_systemDesc.json: A JSON file with predictions on the test set.
• teamID_taskID_runID_systemDesc.meta: A metadata briefly describing the approach, including:
• ◦ Team ID.
• ◦ Task ID.
• ◦ Run ID.
• ◦ Type of training applied.
• ◦ Pre-processing methods.
• ◦ Training data used.
• ◦ Relevant details of the run.
• ◦ A link to a GitHub repository enabling easy reproducibility of the run.

Each zipped file should carefully follow this filenames structure. The team name must remain consistent across all submissions, and the system name shold reflect the approach used by the submitted run. The run number serves as a progressive identifier for multiple submissions of the same system. An example of a valid run submission wil be available soon...

Submissions not adhering to these guidelines might be rejected. Please ensure accuracy and completeness of all submitted files to streamline evaluation.

Example of Submission Format

Submissions should follow the same JSON format used in the provided datasets and must include only the field associated with the specific subtask for which the submission is made:

• Subtask 6.1 (NER): Include the entities field.
• Subtask 6.2.1 (Binary Tag-Based RE): Include the binary_tag_based_relations field.
• Subtask 6.2.2 (Ternary Tag-Based RE): Include the ternary_tag_based_relations field.
• Subtask 6.2.3 (Ternary Mention-Based RE): Include the ternary_mention_based_relations field.

Each entry must correspond to the PubMed ID of the article from the test set being considered.

Find below samples of valid entries for each submission type:

Subtask 6.1 (NER) Submission Sample

					
{
	"34870091": {
		"entities": [
			{
				"start_idx": 75,
				"end_idx": 82,
				"location": "title",
				"text_span": "patients",
				"label": "human"
			},
			{
				"start_idx": 250,
				"end_idx": 270,
				"location": "abstract",
				"text_span": "intestinal microbiome",
				"label": "microbiome"
			}
		]
	}
}
				
				

Subtask 6.2.1 (Binary Tag-Based RE) Submission Sample

				
{
	"34870091": {
		"binary_tag_based_relations": [
			{
				"subject_label": "microbiome",
				"object_label": "human"
			}
		]
	}
}
				
				

Subtask 6.2.2 (Ternary Tag-Based RE) Submission Sample

				
{
	"34870091": {
		"ternary_tag_based_relations": [
			{
				"subject_label": "microbiome",
				"predicate": "located in",
				"object_label": "human"
			}
		]
	}
}
				
				

Subtask 6.2.3 (Ternary Mention-Based RE) Submission Sample

				
{
	"34870091": {
		"ternary_mention_based_relations": [
			{
				"subject_text_span": "intestinal microbiome",
				"subject_label": "microbiome",
				"predicate": "located in",
				"object_text_span": "patients",
				"object_label": "human"
			}
		]
	}
}
				
				

Validation Script

A validation script that can be used to verify the correctness of the submission format can be downloaded from HERE.

Participant Papers

Participants are expected to write a report describing their participation in GutBrainIE CLEF 2025, their proposed solution, what features have been used for prediction, the analysis of the experimental results and insights derived from them.

Participant reports will be peer-reviewed and accepted reports will be published in the CLEF 2025 Working Notes at CEUR-WS, indexed in DBLP and Scopus.

Participants are strongly encouraged to use LaTeX, for which a template is available for download HERE. The template follows the CEURART single-column style and not only defines the layout of the report but also provides a suggested structure and outlines the expected content for each section.

It is also possible to use ODT (LibreOffice) or DOCX (Microsoft Word) formats. In these cases, participants should download the official CEURART templates directly from the CEUR website HERE and ensure that the content and structure match those outlined in the provided LaTeX template.

Participant papers are expected to be 10-20 pages in length, excluding references.

The schedule for submission, revision, and camera ready of the participant report is detailed above in the Important Dates section.

Participant papers can be submitted via Easychair HERE.

A template for participant papers can be downloaded HERE. The template follows the CEURART single-column style and not only defines the layout of the report but also provides a suggested structure and outlines the expected content for each section.

Baselines & Evaluation

The official baselines for the GutBrainIE CLEF 2025 challenge can be found and reproduced by accesing this GitHub repository.

The repository provides a complete implementation of the baseline pipeline, allowing participants to reproduce the results from scratch, test the pre-trained models, or evaluate the provided predictions (i.e., runs) of the baselines for both NER and RE. These runs can also be used by participants as references of valid submission formats.

The repository also contains the official evaluation script, which will be used to assess submitted runs after the system submission deadline. It is crucial that participants ensure their submitted runs are compatible with this script, as submissions that cannot be evaluated with it will be considered invalid. The evaluation script can also be used to obtain preliminary performance results on the development set.

The detailed results of the baselines on the development set can be found in the Results section.

Evaluation Metrics

Submitted runs will be evaluated using standard Information Extraction metrics to assess both per-label and overall system performance. The employed metrics are the same for all the subtasks (a detailed explanation of these can be found HERE):

In the equations below:

• \(TP\) (True Positives) refers to the number of correctly predicted entities or relations.
• \(FP\) (False Positives) refers to the number of wrongly predicted entities or relations.
• \(FN\) (False Negatives) refers to the number of entities or relations in the ground truth that were not predicted.
• \(\mathcal{L}\) is the set of labels referring to:
• ◦ For subtask 6.1: entity labels.
• ◦ For subtask 6.2.1: pairs of (subject label, object label).
• ◦ For subtasks 6.2.2 and 6.2.3: triples of (subject label, predicate, object label).

• Macro-average Precision: \( P_{\text{macro}_{\text{avg}}} = \frac{\sum_{l \in \mathcal{L}} \, \frac{TP_l}{TP_l \,+\, FP_l}}{|\mathcal{L}|} \)
• Macro-average Recall: \( R_{\text{macro}_{\text{avg}}} = \frac{\sum_{l \in \mathcal{L}} \, \frac{TP_l}{TP_l \,+\, FN_l}}{|\mathcal{L}|} \)
• Macro-average F1-score: \( F1_{\text{macro}_{\text{avg}}} = 2 \, \times \, \frac{P_{\text{macro}_{\text{avg}}} \; \times \; R_{\text{macro}_{\text{avg}}}}{P_{\text{macro}_{\text{avg}}} \; + \; R_{\text{macro}_{\text{avg}}}} \)
• Micro-average Precision: \( P_{\text{micro}_{\text{avg}}} = \frac{\sum_{l \in \mathcal{L}} \; TP_l}{\sum_{l \in \mathcal{L}} \; TP_l \; + \; FP_l} \)
• Micro-average Recall: \( R_{\text{micro}_{\text{avg}}} = \frac{\sum_{l \in \mathcal{L}} \; TP_l}{\sum_{l \in \mathcal{L}} \; TP_l \; + \; FN_l} \)
• Micro-average F1-score: \( F1_{\text{micro}_{\text{avg}}} = 2 \, \times \, \frac{P_{\text{micro}_{\text{avg}}} \; \times \; R_{\text{micro}_{\text{avg}}}}{P_{\text{micro}_{\text{avg}}} \; + \; R_{\text{micro}_{\text{avg}}}} \)

For each subtask, the reference metric for the final leaderboard will be the micro-average F1-score, as it better accounts for class imbalances. However, system rankings will also be provided for each of the metrics detailed above.

Baseline Results

The scores reported on the development set represent the actual evaluation of the inference capabilities of the proposed baseline models. Scores on the training collections are provided to give a sense of how well the models fit the data they were trained on. Therefore, they should not be interpreted as evaluation results but rather as indicators of performance on the seen data.

Subtask 6.1 (NER)

Subtask 6.2.1 (Binary Tag-Based RE)

Subtask 6.2.2 (Ternary Tag-Based RE)

Subtask 6.2.3 (Ternary Mention-Based RE)

Results

Leaderboards

Available after the systems submission deadline...

Details

Available after the systems submission deadline...

FAQs

Find answers to common questions about the GutBrainIE challenge, dataset, and submissions.

Coming soon...

If you need any additional information, please get in contact with us writing to:

• Nicola Ferro:      nicola.ferro @ unipd.it
• Gianmaria Silvello:      gianmaria.silvello @ unipd.it

Organizers

(For further details about the organizers, visit the official website of the IIIA Hub Research Group.)