AI-assisted Communication Assistive Devices Research Project

Project Overview

Mission: This NSTC-funded three-year project (2023–2026) builds AI-based inclusive communication assistive systems for speech-impaired individuals, across four sub-projects covering hardware, AI model development, multimodal dialogue systems, and field testing.

I work on Sub-project 3: a multimodal, cross-lingual, task-oriented dialogue system.

Sub-project 1

Embedded hardware and speech-signal assistive modules.

Sub-project 2

Adaptive AI communication models and user interfaces.

Sub-project 3 (mine)

Multimodal, cross-lingual, task-oriented dialogue systems for inclusive communication.

Sub-project 4

Field testing, user validation, and technology dissemination.

My Role

Lead Research Assistant, Sub-project 3

As the lead research assistant for Sub-project 3, I'm the main point of contact between the research team and our 7 partner organizations, and I own the system end to end, from requirements through production.

Current Achievements

7

Partner Organizations

AI-powered LINE bots deployed and live in production

2,505

Cumulative Responses

Served between June 2024 and May 2026

Multimodal

Gradio Web Platform

Text, voice, and image input with cross-lingual responses, built and internally tested

Project Reports Folder Outcome Showcase Presentation

Research Methodology

Phase 1

Needs Assessment & Data Collection

In-depth interviews with 7 social welfare organizations to identify real-world requirements, plus diverse data sources including FAQs and conversation logs.

Phase 2

AI-Enhanced Knowledge Base Construction

Used GPT-4o to generate comprehensive Q&A datasets and built multilingual knowledge bases tailored to each organization's needs.

Phase 3

Iterative Development & Deployment

Three-stage rollout: core validation with a LINE Bot, feature expansion with voice integration, then interface consolidation on the Gradio web platform.

Tech Stack

Sub-project 3 system architecture — a RAG-enhanced multimodal framework

The system runs on a RAG (Retrieval-Augmented Generation) framework: e5-base embeddings with a FAISS vector database for retrieval, and GPT-4o / Mistral for response generation with multilingual TTS output.

Embedding & Retrieval

e5-base multilingual embeddings FAISS similarity search Custom vector DB tuning

Generation & Reasoning

GPT-4o-mini & Mistral Domain-specific prompt engineering Context-aware multilingual processing

Multimodal Processing

Whisper speech recognition Vision models for image understanding Meta MMS-TTS-ZAN (Taiwanese TTS)

Deployment & Integration

Google Cloud Run LINE Messaging API Gradio web interface

Partner Organizations

We work with 7 social welfare organizations to put these AI communication tools directly in front of the people who need them.

中華民國腦性麻痺協會
The Cerebral Palsy Association of R.O.C.

Official Site LINE Bot

漸凍人協會
Taiwan Motor Neuron Disease Association

Official Site LINE Bot

陽光社會福利基金會
Sunshine Social Welfare Foundation

Official Site LINE Bot

台北市基督教勵友中心
Good Friend Mission

Official Site LINE Bot

行無礙資源推廣協會
Taiwan Access for All Association

Official Site LINE Bot

桃園市北區輔具資源中心
Taoyuan North District Assistive Technology Center

Official Site LINE Bot

連江縣早期療育資源中心
Matsu Early Intervention Resource Center

Official Site LINE Bot

Expected Impact

Research Contribution

Advancing inclusive AI and multimodal communication systems as a research field.

Social Impact

Improving quality of life and communication access for speech-impaired individuals.

Technological Innovation

New approaches to multilingual, multimodal AI-assisted communication.

Commercial Potential

Groundwork for market-ready assistive communication technology.

Publications & Presentations

2025/07

TWSC2 2025 Conference

Implementing an Inclusive Communication System with RAG-enhanced Multilingual and Multimodal Dialogue Capabilities

Cheng-Yun Wu, Bor-Jen Chen, Wen-Hsin Hsiao, Hsin-Ting Lu, Yue-Shan Chang, Chen-Yu Chiang, Chao-Yin Lin, Yu-An Lin, Min-Yuh Day

Media & Social Coverage

The project has been covered across print, digital, TV, and radio outlets:

2025/07/05

科技社群敲敲門 Podcast

Using AI to restore communication rights for people with speech disorders — inside NTPU's ReVoice project, featuring Prof. Yu-Shan Chang and Prof. Chao-Yin Lin

2025/02/08

Yahoo News Taiwan

Using AI, National Taipei University strives to remove communication barriers

2024/12/16

大愛電視台 Interview

Held multiple recruitment info sessions and in-depth interviews with patients and caregivers, assessing how assistive technology improves quality of life.

2024/12/10

Charming SciTech

Opening the Door of Silence: How AI is Transforming the Future of Communication for the Speech-Impaired

2024/11/20

Voice of National Chengchi University Radio

NSTC Launches "Inclusive Technology" Initiative Focused on Digital Equality for Disadvantaged Groups

2024/11/15

Business Next (數位時代)

Six people can no longer speak: 20 NTPU faculty and students "work hard on a non-profit project" to help AI speak for the speech-impaired

2024/11/01

YouTube Introduction

[ReVoice Project Team] Sub-project 3 — Multimodal Cross-lingual Task-Oriented Dialogue System for Inclusive Communication Support

Watch

2024/10/30

Yahoo News Taiwan

AI Gives Voice Back to the Speech-Impaired: Professor and Disabled PhD Student Develop Real-Time Translation Software

2024/05/21

漢聲廣播電臺 — 漢聲下午茶

Text-to-speech: bringing a lost voice back with warmth. Featuring Dr. Chen-Yu Chiang (NTPU) and social worker supervisor Ling-Chin Huang (MNDA)

2024/05/20

漢聲廣播電臺 — 漢聲下午茶

You can bank your voice: how a voice bank helps ALS patients speak. Featuring Dr. Chen-Yu Chiang (NTPU) and social worker supervisor Ling-Chin Huang (MNDA)

2023/08/13

漢聲廣播電臺 — 45度角的天空

Assistive technology for speech-impaired communication, featuring Prof. Han-Hsun Huang (NTPU CSIE) and PI Prof. Yu-Shan Chang

Coverage

天下雜誌 (CommonWealth Magazine)

Thirsty, needing the restroom, but no one understands: how NTPU students with disabilities helped build a system so conversation doesn't have to be guesswork

Coverage

大愛新聞 (Da Ai News)

A PhD student with a disability builds ezTalk, giving people with speech impairments a voice

Watch

This coverage has helped raise public awareness of AI-assisted communication technology, and has also opened doors to new partner organizations and stakeholders in the assistive-technology community.

Digital Support, Unimpeded Communication: AI-assisted Communication Assistive Devices for Speech Impairment

Project Overview

Sub-project 1

Sub-project 2

Sub-project 3 (mine)

Sub-project 4

My Role

Current Achievements

Partner Organizations

Cumulative Responses

Gradio Web Platform

Research Methodology

Needs Assessment & Data Collection

AI-Enhanced Knowledge Base Construction

Iterative Development & Deployment

Tech Stack

Embedding & Retrieval

Generation & Reasoning

Multimodal Processing

Deployment & Integration

Partner Organizations

中華民國腦性麻痺協會The Cerebral Palsy Association of R.O.C.

漸凍人協會Taiwan Motor Neuron Disease Association

陽光社會福利基金會Sunshine Social Welfare Foundation

台北市基督教勵友中心Good Friend Mission

行無礙資源推廣協會Taiwan Access for All Association

桃園市北區輔具資源中心Taoyuan North District Assistive Technology Center

連江縣早期療育資源中心Matsu Early Intervention Resource Center

Expected Impact

Research Contribution

Social Impact

Technological Innovation

Commercial Potential

Publications & Presentations

TWSC2 2025 Conference

Media & Social Coverage

科技社群敲敲門 Podcast

Yahoo News Taiwan

大愛電視台 Interview

Charming SciTech

Voice of National Chengchi University Radio

Business Next (數位時代)

YouTube Introduction

Yahoo News Taiwan

漢聲廣播電臺 — 漢聲下午茶

漢聲廣播電臺 — 漢聲下午茶

漢聲廣播電臺 — 45度角的天空

天下雜誌 (CommonWealth Magazine)

大愛新聞 (Da Ai News)

中華民國腦性麻痺協會
The Cerebral Palsy Association of R.O.C.

漸凍人協會
Taiwan Motor Neuron Disease Association

陽光社會福利基金會
Sunshine Social Welfare Foundation

台北市基督教勵友中心
Good Friend Mission

行無礙資源推廣協會
Taiwan Access for All Association

桃園市北區輔具資源中心
Taoyuan North District Assistive Technology Center

連江縣早期療育資源中心
Matsu Early Intervention Resource Center