Publications · 2022—2026

Research in clinical NLP and model reasoning.

A PhD thesis and eight papers covering benchmark design, controlled evaluation, evidence retrieval, prompt adaptation, and structured inference.

Thesis2026PhD thesis · University of Manchester

Evaluating and Controlling Natural Language Inference over Clinical Trial Texts

Maël Jullien

Abstract

This thesis formalises Clinical Trial Natural Language Inference as deciding whether clinically meaningful claims are supported, contradicted, or indeterminate across heterogeneous trial materials. It develops a reasoning-first programme connecting task specification, robustness-centred evaluation, and controlled system design. The contributions span the NLI4CT benchmark and shared tasks, Faithfulness and Consistency metrics, ontology-grounded retrieval, prompt and LoRA studies, GKMRV diagnostic probes, and the CARENLI agentic framework. Collectively, the studies show how aggregate scores can conceal shortcutting and brittleness, while structured inference procedures make errors easier to locate and can improve reasoning fidelity.

ProQuest thesis Manchester record
Conference paper2026Findings of ACL

Compartmentalised Agentic Reasoning for Clinical NLI

Maël Jullien, André Freitas, Marco Valentino, Lei Xu

Abstract

Clinical NLI often depends on selecting the appropriate inferential procedure rather than matching surface language. CARENLI routes each premise–statement pair to one of four reasoning families—causal attribution, compositional grounding, epistemic verification, or risk-state abstraction—before applying a specialised solver, verification stage, and targeted refinement. The framework is evaluated on an expanded 200-item CTNLI benchmark with four contemporary language-model backbones. Mean accuracy rises from approximately 23% under direct prompting to approximately 57%, with the largest gains occurring on structurally demanding reasoning types.

ACL Anthology PDF Code
Conference paper2026Findings of ACL

Dissecting Clinical Reasoning in Natural Language Inference for Large Language Models

Maël Jullien, André Freitas, Marco Valentino, Leonardo Ranaldi

Abstract

This controlled study examines how prompt structure and parameter-efficient adaptation jointly affect clinical natural language inference. Four prompting families elicit reasoning at different levels of abstraction, while demonstrations produced by a frontier model are used to adapt models of at most four billion parameters through LoRA. Evaluation across clinically motivated reasoning types on NLI4CT finds that prompt choice accounts for up to 44% of macro-F1 variance. LoRA adds 8–12 F1 points, raises valid output alignment above 97%, and reduces the gap between compact models and GPT-4o-mini.

ACL Anthology PDF Code
Preprint2025arXiv preprint

The Knowledge–Reasoning Dissociation: Fundamental Limitations of LLMs in Clinical Natural Language Inference

Maël Jullien, Marco Valentino, André Freitas

Abstract

The study tests whether language-model failures in clinical inference arise from missing factual knowledge or from an inability to apply available knowledge correctly. A diagnostic CTNLI benchmark covers four reasoning families, with every inference item paired with Ground Knowledge and Meta-Level Reasoning Verification probes. Six language models are evaluated under direct and chain-of-thought prompting. Although mean probe accuracy reaches 0.918, mean reasoning accuracy is only 0.25, while predictions remain highly consistent across samples. The resulting dissociation exposes stable heuristics and makes knowledge–reasoning failures directly measurable.

arXiv PDF Code
Preprint2024arXiv preprint

Controlled LLM-based Reasoning for Clinical Trial Retrieval

Maël Jullien, Alex Bogatu, Harriet Unsworth, André Freitas

Abstract

Matching patients to clinical trials requires expert interpretation of medical records and eligibility criteria across a very large trial collection. This work proposes a scalable, set-guided reasoning method that represents patient and trial information as typed attributes, normalises clinical concepts through SNOMED CT, and combines language-model eligibility judgments with explicit filtering and ranking rules. Evaluation on the TREC 2022 Clinical Trials benchmark shows performance above the reported state of the art, reaching 0.693 NDCG@10 and 0.730 Precision@10 while retaining inspectable intermediate representations and decision procedures.

arXiv PDF Code
Conference paper2024SemEval · NAACL

SemEval-2024 Task 2: Safe Biomedical Natural Language Inference for Clinical Trials

Maël Jullien, Marco Valentino, André Freitas

Abstract

SemEval-2024 Task 2 examines shortcut learning, factual inconsistency, and adversarial fragility in biomedical natural language inference. The NLI4CT-P resource extends the original benchmark with controlled interventions and meaning-preserving perturbations, enabling separate measurements of whether systems react to causally relevant changes and remain stable under equivalent formulations. The shared task attracted 106 registered participants, more than 1,200 submissions, and 25 system-description papers. Its results provide a broad comparison of methods for safer clinical NLI; the official erratum supplies the corrected Consistency computation.

ACL Anthology PDF Official erratum Code and data
Conference paper2023EMNLP

NLI4CT: Multi-Evidence Natural Language Inference for Clinical Trial Reports

Maël Jullien, Marco Valentino, Hannah Frost, Paul O’Regan, Dónal Landers, André Freitas

Abstract

Clinical trial reports contain essential evidence for clinical decisions, but their scale and complexity make manual inspection impractical. NLI4CT introduces 2,400 expert-annotated examples over full breast-cancer trial sections, linking an entailment or contradiction decision to the facts required to justify it. The benchmark defines two connected tasks: determining the inference relation between a statement and one or more reports, and retrieving the supporting evidence. The examples require multi-evidence biomedical and numerical reasoning, and the original six inference baselines reach a maximum reported abstract-level F1 of 0.627.

ACL Anthology PDF Code and dataset
Conference paper2023SemEval · ACL

SemEval-2023 Task 7: Multi-Evidence Natural Language Inference for Clinical Trial Data

Maël Jullien, Marco Valentino, Hannah Frost, Paul O’Regan, Dónal Landers, André Freitas

Abstract

SemEval-2023 Task 7 turns NLI4CT into a shared evaluation of clinical-trial entailment and evidence selection. Both tasks require multi-hop biomedical and numerical reasoning over report sections. The entailment task received 643 submissions from 40 participants, while evidence selection received 364 submissions from 23 participants. Most entailment systems did not significantly outperform the majority-class baseline, and evidence selection was consistently easier. Cross-system analysis also found a stronger association between model scale and performance than between biomedical pre-training and performance.

ACL Anthology PDF Code and dataset
Preprint2022arXiv preprint

Do Transformers Encode a Foundational Ontology? Probing Abstract Classes in Natural Language

Maël Jullien, Marco Valentino, André Freitas

Abstract

This work extends semantic probing to highly abstract categories by asking whether Transformer representations reflect a foundational ontology. It introduces a systematic methodology covering different pre-training and fine-tuning regimes and evaluates several language models across three complementary ontology-tagging experiments. The results indicate that Transformer models incidentally encode information associated with foundational categories during pre-training. That information can also support practical tagging: directly fine-tuned ontology classifiers reach 90% accuracy. The probing evidence establishes recoverability of the categories, rather than proving that models causally use them in downstream reasoning.

arXiv PDF