🏅 Best Oral Presentation Award · NTCIR-18 Conference

IMNTPU at NTCIR-18 MedNLP-CHAT Task: Evaluating Agentic AI for Multilingual Risk Assessment in Medical Chatbots

A multilingual risk-classification system for medical chatbot responses, built with fine-tuned models, few-shot LLM prompting, and agentic multi-model voting

NTCIR-18 Conference · Prompt Engineering & Fine-tuning · Jun 10–13, 2025, Tokyo, Japan

Jun-Yu Wu, Cheng-Yun Wu, Bor-Jen Chen, Wen-Hsin Hsiao, Min-Yuh Day (advisor) · Information Management, National Taipei University

Read the Paper Slides View the Poster

Overview System Architecture My Role Fine-tuning Results FinArg-2 Track Awards Team

Task Overview

The NTCIR-18 MedNLP-CHAT task evaluates whether a medical chatbot's response to a patient's question carries medical, ethical, or legal risk. Our team, IMNTPU, built a multilingual risk-classification system combining fine-tuned small models, few-shot prompting with frontier LLMs, and agentic multi-model voting. We submitted 24 systems across all language tracks, the most of any participating team, and the paper won the Best Oral Presentation Award.

Goal

Classify whether a chatbot's response to a patient's question poses a medical, ethical, or legal risk.

Input & Output

Patient question + chatbot answer pairs, labeled with binary risk indicators; the Japanese set also includes subjective scores for fluency, helpfulness, and harmlessness.

Languages

Japanese and German source datasets, each professionally translated into English and French.

My Focus

Prompt engineering across models, shot counts, and temperature settings, plus co-writing the paper's methodology section.

System Architecture

We submitted three systems across every language track, each built on a shared foundation of prompt-engineered LLMs and layered with a different aggregation strategy.

System 1 · Optimal LLM Selection

Picked the best-performing LLM by training-set accuracy, using three-shot prompting.

System 2 · Agentic AI Majority Voting

Four independent LLMs evaluate each case in parallel; the majority vote decides, with ties resolved by dropping the lowest-performing model's vote.

System 3 · Agentic AI Weighted Scoring

Each LLM outputs a 0–1 confidence score; the final decision is a weighted aggregation based on that model's demonstrated accuracy on the dev set.

Full conference poster — click to zoom

My Role

Prompt Engineering & Methodology

As part of the 5-person IMNTPU team led by Prof. Min-Yuh Day, I focused on the prompt engineering work behind our LLM-based systems.

Fine-tuning: Multi-language Track

Alongside the prompting-based systems, we fine-tuned a LLaMA 3.2 3B model directly on the MedNLP-2 Japanese and German datasets, then tested it under three configurations for the multilingual track.

Zero-shot Fine-tuned

The fine-tuned model alone, with no prompt examples.

Three-shot, English Only

Three-shot prompting layered on top of the fine-tuned model, using English examples.

Three-shot, Multilingual

Three-shot prompting with English, Japanese, and French examples together.

On Legal Risk assessment, the fine-tuned multilingual system reached a 0.576 Macro F1 on the Japanese dataset, ahead of the GPT-4o few-shot baseline's 0.456. Overall, though, fine-tuning's gains were limited by a small 212-sample training set, so few-shot prompting with larger LLMs generally performed better across the other risk categories.

Results

7/8

Subtask First Places

First place in 7 of 8 subtasks on the MedNLP-CHAT multilingual risk assessment Joint Accuracy metric

Systems Submitted

The most of any participating team, across every language track

Risk Dimensions

Medical, ethical, and legal risk, evaluated per response

Languages Covered

Japanese and German source data, plus English and French translations

1st

Best Oral Presentation

Awarded for the MedNLP-CHAT paper

Ethical Risk saw the biggest gains

Agentic voting improved decision consistency most on Ethical Risk, the most subjective of the three categories. Medical and Legal Risk, which follow more structured rules, gained less since a single well-prompted LLM already handled them well.

Japanese systems were the most stable

Higher-quality native-language annotations led to more consistent performance than the translated English and French tracks.

Confidence didn't always track accuracy

Some systems stayed highly confident even when wrong, showing that trust-weighted voting can amplify overconfidence rather than correct it.

Future work

We proposed a Trust-Guided Agentic AI architecture, adding self-consistency filtering, dynamic trust updating, and Chain-of-Thought reasoning, since formalized in a follow-up paper.

FinArg-2 Track

IMNTPU also submitted a second, separate paper to NTCIR-18's FinArg-2 task, led by teammate Bor-Jen Chen. All five of us are co-authors on both papers, but my own hands-on work was on the MedNLP-CHAT track above — the summary below covers what the team built for FinArg-2.

FinArg-2 focuses on temporal reasoning in financial statements from earnings calls and social media: given a claim, a model has to identify its temporal reference point and predict how long the statement stays valid.

Model Fine-tuning

BERT, RoBERTa, and DistilBERT for encoder-based approaches; GPT-4o Mini for decoder-based approaches.

Data Augmentation

Generated semantic variations of training examples to address class imbalance in the financial dataset.

Temporal Reasoning

Classified argument claims by their temporal reference points within financial contexts.

Combining fine-tuned models with prompt engineering, the team's FinArg-2 systems reached first place in key subtasks, demonstrating strong potential for financial semantic reasoning applications.